Value-Guided Search for Efficient Chain-of-Thought Reasoning
Authors: Kaiwen Wang
Jin Peng Zhou
Jonathan Daniel Chang
Zhaolin Gao
Nathan Kallus
Kianté Brantley
Wen Sun
Venue: NeurIPS 2025 poster ()
Keywords: Value-Guided Search
Chain-of-Thought Reasoning
Competition Math
Primary area: reinforcement_learning

Abstract: In this paper, we propose a simple and efficient method for value model training on long-context reasoning traces. Compared to existing process reward models (PRMs), our method does not require a fine-grained notion of ``step,'' which is difficult to define for long-context reasoning models.
By collecting a dataset of 2.5 million reasoning traces, we train a 1.5B token-level value model and apply it to DeepSeek models for improved performance with test-time compute scaling. We find that block-wise value-guided search (\texttt{VGS}) with a final weighted majority vote achieves better test-time scaling than standard methods such as majority voting or best-of-$n$.
Moreover, \texttt{VGS} significantly reduces the inference FLOPs required to achieve the same performance of majority voting. Our dataset, model and codebase are open-sourced at \codeurl.

Value-Guided Search for Efficient Chain-of-Thought Reasoning
Kaiwen Wang"!, Wan
Jin Peng Zhou" "I Jonathan Chang" "
Zhaolin Gao', Nathan Kallus Kianté Brantley", and Wen Sun'
Coomell University Harvard University
Netflix
*Databricks
Abstract
In this paper, propose simple and efficient method for value model training on long-context reasoning traces. Compared to existing process reward models (PRMs), method does not require fine-graine's notion of "step," which is difficult to define for long-context reasoning models. By collecting a dataset of 2.5 million reasoning traces, train 1.5B token-level value model and apply it to DeepSeek models for improved performance with test-time compute scaling. We find that block-wise value- value-guided search (VGS) with final weighted majority vote achieves better test-time scaling than standard methods such as majority voting or best-of-n. Moreover, VGS significantly reduces the inference FL.OPs required achieve the same performance of majority voting. Our dataset, model and codebase are open-sourced https: /ttp//gitttt github. com/ wwwp////wwwwaeeeeea/ee/a/a www.raalleeedee---eeeeeeee search.
PetaFLOPs to Reach Accuracy (-)
DeepSeek-R1 (671B)
32
High Budget Accuracy {->)
16
DS-R1-Distil--2B
90%
81%
2
45
68%
40
60%
0.5
1.58 (256)
14B (64)
35
37
Model Size (Test-time Budget)
Scaling Budget Accuracy (DeepSeek 1.5B)
Value Guided Search (VGS)
Majority Vating (MV)
Figure I: Performance and Efficiency of Value Guidance: (Left) Value Value-guided search improves the overall quality of DeepSeek- -Ri--istt1--i responses across combined competition math benchmarks (AIME 24, 25 & HMMT Feb 24, 25). The inference budget for 1.5B, 7B and 14B are 256, 128 and 64 generations, respectively. (Right) Value-guided search also reduces the inference FLOPs required to achieve the same accuracy levels as majority voting, standard TTC scaling baseline, showing value-guidance is promising improving efficiency.
'Equal contribution. Correspondence [kw437, (kw437,j25633) (25.3))ccooeeell..c..m edu.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Recent large language models (LLMs), such as OpenAl ol & o3, Claude Sonnet 3.7, Gemini Pro 2.5 and DeepSeck RI [19] are trained via reinforcement learning (RL) to "think" for many tokens before generating final answer. Through multi-step reasoning and self-corr,ction, these reasoning models have state-of-the-art performance in competition math, coding [16] and scientific research [38], often surpassing the average human. However, this enhanced capability comes at cost: each generation involves long
(CoT), thus requiring more inference compute. Further, these CoT traces can often be repetitive and get stuck in unproductive loops [31]. This raises two questions. Can we extract the same performance at a fraction of the inference compute by refining the thinking process? Can we improve the performance ceiling of these models with productive search methods?
Search with guidance models is a natural solution that addresses longer ch siiin--f-thuught reasoning by managing the exponential growth of possible paths with guidance models identifying optimal routes [15, 40, 36]. Prior works that combined search with LLMs proposed to guide search with process reward models (PRMs), predicting the correctness of each step (e.g., delimited by newlines) in the model- generated solution [24, 45, 51J. While PRM-guided search has been shown to improve test-time compute (TTC) [6, 37, 35, 26), it is challenging to scale existing PRM training techniques to long-context reasoning models. First, existing methods require a pre-defined notion of "step," but, per Guo et al. [19]. "it is challenging to explicitly define a fine-grain step in general reasoning." Second, even if we can define a "step." " collecting step-wise labels is prohibitively expensive, since requires annotations from humans [24], LLM-as a-------d- [51], or multiple Monte Carlo roll-outs [45, 28].
Thus, there has been limited success to scale PRMs to long- reasoning models [19].
We propose vulue-guided search (VGS)- a block-level search method guided by a token-level value model - as a promising approach to scale TTC for reasoning models. In Section 2, we present an ef- fective pipeline for value model training on tasks with outcome labels, such as competition math. Our data pipeline collects solution prefixes from various models and then, starting from random prefixes, generates completed solutions using a lean reasoning model (e.g.. DeepSeek-Ri RR--Dsstil-- 5B). Notably, our data collection does not require a pre-defined notion of step and is more efficient than existing techniques [45, 28]. With this pipeline, we collect dataset of 2.5 million math reasoning traces (over 30 billion tokens) from a filtered subset of the OpenR OpenRI-Mathe Maahh dataset [2]. Then, we train a 1.5B token-level value model called DeepSeek-V.t -M 5B by regressing (via classification) the final reward of the completed solution.
Next, in Section 3, we apply our value model to perform block-wise search with DeepSeek models [19] on competition math, where we evaluate on four prestigious high school math competitions in the US (AIME 2024 & 2025 and HMMT 2024 & 2025). Our experiments show that block- wise VGS significantly improves TTC compared to majority voting or weighted majority voting, strong baselines from the literature [46, 6]. We also show that VGS with DeepSeek-VM- 1akk-MM-1.55 1. leads to higher performance than searching with state-of-the-art PRMs, demonstrating that our value model can provide better feedback. When given an inference budget of 64 generations, VGS on DeepSeek-R1- DeepSeek-R1 R1 (67 IB) VGS reduces the amount of inference compute required to attain the same performance as majority voting (Fig. right). In summary, we find that block-wise VGS not only improves the performance ceiling of reasoning models, but also reduces the amount of inference compute required to match the performance of standard TTC methods. Our contributions are summarized below:
(total size with value model is 15.5B) is on par with competition math evaluations (Fig. left). Moreover, we show that
1. simple recipe for token-level value model training that does not require a pre-defined notion of "step" and scales to long-context reasoning traces (Section 2).
2. Block-wise search, guided by our 1.5B value model, achieves impressive performance on four challenging math competitions, outperforming standard TTC methods (e.g., best-of-N, majority voting) and search with existing PRMs (Section 3).
3. We open-source our dataset of 2.5 million reasoning traces, value model, and codebase (including data filtering and distributed training scripts) to support future work on applying VGS other domains. //g:: com/ hwm//aawweww// wwweee//vaaeeeeeee www.a/aaeeggdddddderrrelll
Please see Appendix A for detailed discussion of related works.

---

2 Methods
We present an end-to-end pipeline for training a token-level value model and applying it to guide block wiiee search. In Section 2.1, we introduce necessary notation and present regression- classification algorithm for learning the token-level value model [14]. Then, in Section 2.2, we outline an efficient data pipeline for creating our dataset of 2.5 million reasoning traces from DeepSeek models. Finally, in Section 2.3, we describe several TTC methods and baselines, e,g., best-of-N, (weighted) majority voting and search algorithms that can leverage our value model. While we focus on competition math in this paper, we remark that our pipeline can in principle be applied to any task with automated outcome supervision (e.g., a reward model). In Appendix B, we summarize simple recipe for applying VGS to other such domains.
2.1 Learning Algorithm for Value Model
We describe our training process for a language value model by performing regression via classifica- tion [7]. Let be the vocabulary and let S = UnEn 1 denote the input sequence space. Given a problem prompt
S and a response y S, let F(z, y) [0, I...., , K 1] denote its class label, where K is the number of classes. Furthermore, let r = R(xr, y) denote the scalar reward, which we assume to be binary since we focus on competition math (see Appendix B for the general case). For our value model, K = 2 if the response is an incomplete generation (i.e., exceeds max generation length), k = 0 if the response finished and incorrect, and and is correct. Thus, the event that corresponds to r = 0 (incorrect or exceeds max length). We adopt this convention in the rest of the paper. We remark that regression- via-classification is standard approach that leads to better down-stream decision making than regressing via squared error [7, 21, 17, 3, 43,444..
if the response finished (correct answer), and E {0, 2}
1 corresponds to
We employ datasets of the form D = {(Ti, Mi, 2i, NE(N) where x; is the problem prompt, y: partial response (which we call "roll-in""), 24 is a completion starting from y/s (which we call a "roll-out"), and K; = I[x;, Vi, 2) is the label of the full response, where x, V, 2 denotes the concatenation of x. y and 2. In this paper, we assume that the completions roll-outs 2; are generated by a fixed roll-out policy x=5 i.e., ~ (- I.,, ) for all i. We remark that good choice for ge a cost-efficient model which is capable of producing diverse responses with positive reward, egg., a distilled version of a large reasoning model.
We train a classifier fa : S + A(KK) via gradient descent on the following loss on data batch B:
L)O TBT (E |.a Eherange(==))) (eo(Jo( Vi, zith) ka),
where (ū, x) = -- -1nn (p(rs) is the standard cross-entropy loss for classification and 21th denotes the first tokens of 2;. The rationale for the inner average is analogous to next-token prediction training of autoregressive models: since 2; is generated autoregressively by π' any suffix 2/"h is also roll-out from ,d and hence can be viewed as another data-point. We found this to be an important training detail for performance, which is consistent with prior work who used a similar objective for training an outcome reward model [14, 24].
We can now view the classifier as a value model. Since x 1 corresponds the event that r = 1, we have that Vo(x) = [1] predicts the correctness probability of roll-outs from
Classifier Value Model Training
Block 1A
Block 2.
Slock
Value-Guided Search
Labal Connect
Prompt
Label Incorrect
Block 2C
Block
Weighted Mail Vosa
Firal Ansuee
Prompt
Book 1D
Block 2D
Block 4.
Label Incomplete
Value
reward D
Label Correct
reward 1
Figure 2: Summary of Methods. (Left) Diagrams how we collect multiple roll-ins (grey circles representing tokens) per problem, and branch off multiple roll-outs per roll-in at class label for each roll-out token is the outcome label at the very end. (Right) Shows the beam search process (beam width 2 and budget 4) guided by a value model.
random points. The
3

---

Algorithm 1 Beam Search with Width w
Algorithm 2 Best-of-N
1: Input: prompt T.
1: Input: prompt x, responses (ui
2: Set num beams B m²
2: return yoo = arg max,, y; ((r,
3: Initialize beams y1.... "1.....U8.
4: while Aj s.t. yj is not finished do
Algorithm 3 (Weighted) Majority Vote
5: For each. J s.t. 1/j is not finished, sample w i...d. blocks (br.jiisjw) [ej from ((- Mj).
1: Input: prompt x, responses {yi he()
6: Update unfinished beams to be the best
weights ( equivalence relation ~.
continuations with the highest (y.,.b.,))
2: Partition {yi}: into equiv. classes {ps)a,
7: end while
3: return A response from the highest weight partition arg maxp, Ey:Eps W/-
8: return BoN or WMV on , .BB).
rf Indeed, if f denotes the optimal classifier that minimizes the population-level loss, then (z, v)[1] = E -=> R[x, y, x) x; x. which is the expected reward of a completed response from rolling-out π starting from 2, y. In sum, our value model is learned via predicting labels (one
of which corresponds to reward 1), and the training objective is the standard cross-entropy loss.
2.2 Dataset Creation Process
We describe our process for creating OpenR1-VM, a novel dataset of 2.5 million reasoning responses from DeepSeek models, across 45k math problems from OpenR1-Math [2].
Pre-Filtering. We start from the OpenR1 -Math dataset (default split) [2] which contains 94k math problems with solutions that were already filtered for quality. Upon manual inspection, we found that this dataset still contained unsolvable problems (e.g., problems that require web browsing but unverifiable answers (e.g., multiple \boxed| expressions or unparsable answers). We filter out all such problematic problems, producing a cleaned subset of 50k problems with solutions verifiable by sympy or math-verify [23]. We call this pre-filtered dataset OpenR1-Cleanee
models cannot access the web) and ambiguous
Next, we collect roll-ins and roll-outs from DeepSeek models [19]. We fix the roll-out policy पΩ as DeepSeek-R1-Distie 1itsstll--weenn..-5 To ensure diversity in the roll- in distribution, we sample 14 independent roll-ins from four sizes: 1.5,, 7B, 14B, and 32B by generating until the end of thinking token <\think>. ! For each roll- in ys. we then sample four random prefix locations where we generate complete roll-outs { ]EE(4] from rf Finally, to compute the class label (incomplete, incorrect, or correct), we parse the response for the final answer (enelosed in \boxed )) and use math-verify check for correctness against the ground truth answer. In total, this process (illustrated in Fig. 2 left) yields 56 labeled roll-in, roll-out pairs per problem, leading to 2.8 million datapoints.
Response Generation.
model
FilteFiii We filter out problems where all 56 roll-outs for that problem were incomplete or incorrect (i.e., has reward 0). This post-filtering removes any ambiguous unanswerable problems that we missed during pre-filten pre-filtering, and also removes problems that are too difficult for -e and do not provide useful learning signal. This step filters roughly 10% of problems, yielding a final dataset of 2.5 million datapoints.
Notably, our approach does not require fine-grained notion of step and our data collection is cheaper than existing PRM techniques. Specificallyy, Lightman et al. [24] used per-step annotations by human experts, Zhang et al. [51] used per-step annotations via LLM-as-a-Judgee and Wang et al. [45] used multiple Monte Carlo roll-outs at every step. Since the number newlines in reasoning CoT traces can grow very quickly, per-step labels are quite expensive to collect for reasoning models. In contrast, our approach only requires a handful of roll-ins (from any policy) and roll-outs (from rh per problem, and this number can be flexibly tuned up or down to trade-off data coverage and data collection cost. Please refer to Appendix D for further details on each step. We also release our filtering code and datasets to support future research.
DeepSeek- R1 and its distilled variants output CoT reasoning between tokens <think> and <\think> followed by tinal solution, which is usually summarization of the CoT reasoning.

---

2.3 Algorithms for Test-Time Compute and Search
Equipped with value model V S + R. we can now apply it to scale test-time compute of generator model æ. For search-based approaches, we focus on block-wise search where a "block" refers sequence of tokens (e.g., blocks of 4096 tokens worked best in experiments). We let N denote the inference budget, which is the number of generations we can sample (e.g., generating four responses and taking a majority vote is N 4).
BFS. Breadth-firsse first-search (BFS) [48, 29] is natural search method that approximates the optimal KL-regularized policy given a good value model [53]. Given prompt æ. BFS samples N parallel blocks 6, from æ and selects the block with the highest value b' = arg max,, V V(a, b,), which gets added to the prompt, i.e., <- x, b" The process repeats until the response finishes. Note the number of tokens generated from æ is roughly equivalent to N independent generations from IT
Beam Search. One weakness of BFS is that parallel blocks are correlated because they share the same prefix, which limits diversity. Beam search with width w (Algorithm 1) is a generalization that keeps B = N/w (assume to be integer) partial responses and branches w parallel blocks from each one [27, 5, 41, 6, 37]. Given a prompt x. beam search first generates N parallel blocks. However, unlike BFS, beam search keeps the top B beams with the highest scores, and then samples w parallel blocks per beam at the next step. Since B x w = N blocks are sampled at each step, the compute budget is also N. We illustrate beam search with N = 4 and w = 2 in Fig. 2 (right).
DVTS. Diverse verifier tree search (DVTS) is a meta-algorithm that further increases diversity by running parallel searches each with smaller budgets [6]. Specifically, DVTS-M runs M parallel beam searches each with budget N/M (assume to be integer), and aggregates responses into final answer.
We remark crucial detail of beam search and DVTS is how the final set of beams/responses are aggregated. Prior works [6, 37, 35] select the response with the highest score, which is analogous to final best-of-N (BoN; Algorithm 2). Instead, we found that taking a weighted majority vote (WMV; Algorithm 3) led to much better performance, which is demonstrated by Fig. (left).
Computational Efficiency of Block-wise Search. Since value scores are only used at the end of each block or the end of the whole response, the FLOPs required for block-wise value model guidance is tiny fraction (< 1%) of the generation cost from т. We compute FLOP estimates in Appendix H to concretely show this.
3 Experiments
We extensively evaluate value-guided search (VGS) with our 1.5B value model -er-Seek-V--..... focusing on guiding the CoT reasoning of DeepSeek models [19]. The best VGS setup for our value model beam search with final WMV aggregation, beam width 2, block size 4096 and with DVTS (for larger inference budgets). We show this setup outperforms other test-time compute methods (e.g., MV, WMV, BoN) and other scoring models (e.g., existing 7B PRMs and a 1.5B Bradley- model trained on our dataset). We remark our search results use a fixed beam width and block size for all problems; this is more practical than prior works on "compute-optimal scaling" which vary search parameters for each problem and require estimating each problem's 's difficulty [6, 37, 26). Please see Appendices and for additional details on value model training and inference.
reward
Benchmarks. We evaluate on the American Invitational Mathematics Exam (AIME) and the February Harvard-MIT Mathematics Tournament (HMMT). Both AIME and HMMT are prestigious high school math competitions in the US that have also been used to evaluate frontier LLMs [32, 19, 1]. We use AIME & II and the individual part of HMMT, yielding 30 problems per competition. We selected the block size and beam width based on the performance AIME-24 as the validation set (ablations in Section 4.1). Then, we evaluate on AIME-25 and HMMT-25 since they happened after the release of DeepSeek and OpenR1. 1. This controls for data contamination since the test problems were released after the underlying reference models and datasets. We also ensure that there are no problems with > 50% 8-gram overlap between training and test sets as a further sanity check for contamination. Unless otherwise stated, we report the overall accuracy averaged across AIME-25 and HMMT-25. In Appendix C, we report individual per-benchmark results for the 2024 and 2025 editions of both math competitions.
htttps://ma.
wowpallloooootiiiiiioioooo tinns and https: / /wwp//wwwww/uaa/ hmmt org
5

---

Test-time scaling DeepSeek- (NV 256)
AIME-25
HMMT-25
AVG
VGS w/ DeepSeek-VW-11 (ours)
4.7
328
39.8 + 0.5
WMV DeepSeek-VM- VM (ours)
2.2
37.0 2.6
VGS w/ DeepSeek-BT-1. (ours)
4.8
2..
34.1
WMV w/ DeepSeek-BT-1 5B (ours)
40.5
24.6
12.611..
VGS w/ (wen2.5-Math-PPi)
1..
24.2
31.6
WMV Owen2.6-Math-PPA.
39.1
2..0
3.9
VGS w/ PathhShepherd-PPM-778
41.9
1.49
32.9 1.0
WMV w/ MachhShepherd-PPRK-77
40.0 + 2.5
25.6
32.8 + 2.0
MV @ 256
38.9 + 1.9
24.3 2.9
31.6 1.7
Test-time scaling DeepSeek-78 (N 128)
VGS w/ DeepSeek-WVee -...5.
59.4 + 0.8
41.1 1.6
50.3 + 0.9
MV
56.5 + 1.6
33.8 2.5
45.2 + 1.5
Pass@N baselines for various models
DeepSeek-1. 1.58 Pass @
22.4 1 4.1
13.0 3.9
17.7 1 2.8
DeepSeek-380 32B @ass@]
60.4 = 6.0
42.1
5.2
51.3 4.0
Deepseek-BB R1 (671B) Pass
70.0 = 0.9
46.7 7
2.4
58. 1.3
Table 1: (Top) Weighted majority vote (WMV) and VGS results for DeepSeek-1 with inference budget of N = 256, using various scoring models. (Middle) Compares MV and VGS for larger DeepSeek models guided with DeepSeek-VM-L1 -M....B. (Bottom) Lists performance of DeepSeek models and strong close-sourced reasoning models. For VGS, + indicates standard deviation across seeds; for MV, WMV, Pass @N, denotes bootstrap with 100 repetitions. We bold the highest avg. accuracy and underline second highest. Appendix C.1 contains more baselines.
Baseline Models. We evaluate two state-of-the-art 7B PRMs with distinct training styles: Seeb-Sheppeerd-kisttal-------
[45] and Fwen2.6-Math-PRM-77 [51]. Math-Shepherd uses Monte-Carlo roll-outs from each step to estimate per-step value while the Qwen2.5 PRM uses LLM- Judge annotation for each step, similar the per-step human annotation of PRмаоOK [24]. As step-level value model, Math Setth--hppeerrd-RP. Finally, we also evaluate a 1.5B Bradley-Terr Terry (BT) [8] model, called DeepSeek- -1 which we trained using our dataset (see Appendix G for training details).
is more related to our token- level value model.
3.1 Main Results (Table 1)
In the top section of Table 1, we fix the generator to DeepSeek-1. and test-time budget to N 256, and compare VGS to weighted majority voting (WMV), using value model, the BT model and baseline PRMs. We see that VGS and WMV with DeepSeek-VM-1 -M--.. achieve the two highest scores, outperforming the BT reward model and prior PRMs. This shows that value model not only a strong outcome reward model (ORM) but also an effective value model for guiding only trained
search. Intriguingly, while DeepSeek-BT-L improves performance relative to WMV, suggesting that BT models may also provide meaningful block-wise feedback to guide search. We also observe that accuracies for the 7B baseline PRMs (MathSheperd and Qwen2.5-Math) are only slightly higher than MV@256 and do not improve with search, which suggests that these PRMs are likely out-of-distribution (OOD) for the long CoTs generated by DeepSeek-1.58
ORM, find that VGS also
In the middle section of Table 1, we guide the stronger 7B DeepSeek model and compare VGS to MV, a standard TTC method that does not use an external scoring model. We see that VGS again achieves higher accuracy than MV for 7B, which suggests that SeepSeek-VI--1 VM-11.B 5B is also useful for guiding CoT of larger DeepSeek models. In Table 2, we also include results for guiding 14B DeepSeck, where observe that the gap between VGS and MV becomes much smaller. This suggests that DeepSeek-1e -14 CoT's may be becoming OOD for our value model trained on DeepSeek-1.. 1. CoT's. To guide more capable models, new value models should be trained rollouts from similarly capable models; we however do not foresee this being a practical concern given the scalability of our training process (described in Section and summarized Appendix B).
Throughout the paper, DeepSeek- XB shorthand for DeepSeek-Re -R1 Distill-Quen-e

---

70
40
Accuracy
35
50
30
25
30
4
16
64
256
1024
4
16
64
256
1024
Inference Budget (N)
Inference Budget (N)
BoN
WMV
Pass@N
VGS BoN
VGS WMV
VGS WMV
Figure Test-Time Compute with DeepSeck-VM-LL5B. (Left) Compares best-of-N (BoN), weighted majority voting (WMV) and VGS with either BoN or WMV for the final aggregation. (Right)
Compares VGS majority voting (MV), standard baseline that does not require scorer.
3.2 Test-Time Compute Scaling for Search
This section presents three experiments designed analyze the TTC scaling properties of VGS. Our investigation addresses three key research questions:
1.
Does VGS, with its block-wise guidance, demonstrate superior performance compared to response- level aggregation methods such as BoN or WMV?
How does the TTC scaling behavior of VGS compare to the standard score-free baseline MV?
How does the TTC scaling behavior of DeepSeek- -M---e
VM-1.5B 5B compare to baseline models?
Response-Leeve Selection Search-Based Block-I Level Selection. While BoN and WMV repre- sent standard approaches for selecting responses using an ORM, block-wise VGS guides response generation through sequential block-by-block selection. As Fig. 3 (left) illustrates, WMV consistently outperforms BoN across all inference budget scales, which demonstrates the benefits combining MV with value scores. Furthermore, VGS (with WMV final aggregation step) yields additional improvements beyond WMV alone. This confirms the benefits of search and aligns with conclusions from previous studies [6, 37, 26). Interestingly, do not observe the same benefits of search if BoN used as a final aggregation step. suggesting that WMV is a critical component to VGS.
Response Length for VGS. In addition to consistent performance gains, VCS also produces noticeably shorter responses compared to the base DeepSeek-1. model. Figure 15 (Appendix C.7), we present histograms of response lengths across all benchmarks. The results show that VGS consistently generates more concise outputs, whereas the base model often reaches the generation cap, with up to
WMV
Search
40
35
Accuracy
30
25
4
16
64
256
1024
4
16
64
256
1024
Inference Budget (N)
Inference Budget (N)
VM
BT
Math Shepherd 7B
Qwen PRM 7B
Figure 4: TTC Scaling of Various Scorers. Comparison of our 1.5B value model (VM), our 1.5B Bradley-Tere reward model (BT), and two state-of-the-art PRMs for two TTC scaling methods: (Left) WMV or (Right) VGS (with WMV as a final aggregation step).
7

---

50% of its responses being unfinished. On average, VGS responses are 1 11,219 tokens long, compared 12,793 for DeepSeek - 5B, representing reduction of over 12% in token and thus FLOPs usage.
VGS vs Majority Voting. As Fig. 3 (right) demonstrates, VGS consistently achieves higher accuracy than MV, attaining equivalent performance while requiring substantially lower inference budgets (also shown in Fig. right), Fully closing the gap with the oracle @ curve may require larger value model trained on more extensive datasets.
against consistently delivers superior performance when employed both as an ORM for WMV (left) and as a guidance mechanism for block-wise search (right). Note that we find our BT model to be surprisingly effective search guidance model which suggests the importance of our token-level dataset playing an important role in successful downstream search.
DeepSeek- VM-1.5B vs Baseline Scoring Models. Fig. benchmarks DeepSeek-VW-L existing PRMs and BT model. We observe that DeepSeck-VK- (tt-1.55 VM-
3.3 Scaling up the Generator Model Sizes
60
In Fig. 5, scale up experiments guide larger
7B and 14B DeepSeck models. Here, we run VGS with the same search parameters using the same value model DeepSeek- SeepSeek-VM-11 VM-1 5B. Although the 7B and 14B DeepSeek rollins are OOD for our value model (trained on 1.5B traces), we observe that VGS continues to scale without plateauing as test-time compute increases. This provides some evidence that a value model trained with a weaker verifier policy can generalize effectively and guide the Co'I's of stronger models, which is useful since is much cheaper to collect training data from smaller generaliza-
Accuracy
50
40
30
4
16
64
256
1024
Inference Budget (N)
DeepSeek 1.5B
14B
re models. This form of tion [10] appears to be a promising direction for future research.
Figure 5: VGS + WMV Performance when Guiding Larger Models. With the same DeepSeek-V.. ance, search continues to improve with more test-time compute.
providing guid-
Ablation Studies
AIME-24 Accuracy
To investigate the role of key hyperparameters in value- guided search, we perform ablation analyses of block
size and beam width on AIME-24 across varying in- ference budgets. We also ablate the amount of DVTS parallelism. These tests suggest that there is consistent choice of search hyperparameters that work well across inference budgets.
0.5
0.4
16
4096 16384 Block Size (num. tokens)
64
256
1024
4.1 Different Search Parameters and Methods
N 16
32
64
Block Size. We perform beam search with width using search block sizes from 16 to 16384. Fig. 6 shows AIME-24 accuracies across three inference budgets N, revealing that the optimal choice of 4096 stays consistent across different N. We see a decline performance when scarching with more fine-grained blocks.
Figure 6: Ablation: Search Block Size. AIME-24 accuracies for beam search (width 2) with varying block sizes from 16 16384. We found 4096 to be optimal across test-time budgets and benchmarks.
Beam Width. We perform beam search with block size 4096 using varying beam widths, with breadth-first-eeane (BFS) being special case where beam width is equal to N. Fig. (left) shows AIME-24 accuracies across five inference budgets, demonstrating that beam width is consistently optimal across different N. We note our optimal beam width is different from prior works' which found to work best [6, 37, 26].
DVTS Parallelism. Fig. 7 (right) shows the role of ablating DVTS from VGS. For each inference budget, we report average accuracies without DVTS and with the best DVTS parallelism M. We observe that DVTS becomes more effective at higher budgets and scales better than single search

---

75
40
AIME-24 Accuracy
65
Accuracy
35
60
55
30
50
45
16
32
64
128
8
32
128
512
Inference Budget (N)
Inference Budget (N)
Beam 2
Beam 4
Beam 8
BFS
With DVTS
Without DVTS
Figure 7: Ablations: Beam- Miith and DVTS. (Left) AIME-24 accuracies for beam search with various widths (Section 2.3) across inference budgets N. BFS is equivalent to setting width as N. We find that the optimal beam width is robust across multiple TTC budgets. (Right) Averaged accuracy for beam 2 with and without DVTS. For DVTS, we report the best result with parallelism M >> 1 per inference budget N, which we find scales better at higher budgets.
tree, which is consistent with findings from prior works [6]. However, we find that DVTS is never worse than single search tree even at smaller inference budgets, which is the opposite conclusion reached by prior works [6]. This discrepancy may be explained by the fact that we use WMV to combine the DVTS responses, which seems be a more robust way to perform DVTS than BoN (used in prior works) given our findings from Fig. 3.
4.2 Random Value-Guided Search
Finally, we directly ablate the role of our value model's guidance during the search process. We
Accuracy
35
perform VGS (w/ same width, block size and DVTS) but randomly select blocks instead of selecting blocks with the highest value. We still aggregate the final beams via WMV with our value model, so the only change is how inter- mediate blocks are chosen. We call this process "random search". Thus, if our value model is helpful for search, we should expect VGS to out- perform random search. Indeed, Fig. 8 validates this hypothesis. We also evaluate a hybrid ap- proach where half of DVTS's parallel trees use random search and the other half use VGS. We find that this hybrid approach lands roughly be- tween pure VGS and pure random search, again validating that block-selection from our value model improves over random selection.
25
4
16
64
256
1024
Inference Budget (N)
VGS
Random Search
- Hybrid
Figure 8: Ablation: Random Search. Random search is the same search process VGS except intermediate blocks are randomly selected instead of using our value model. Hybrid is mixture where we flip a fair coin at the start of search tree that decides whether to use random search or VGS. We see that selecting blocks with highest value improves accuracy across inference budgets.
Conclusion
In this paper, we introduced block-wise Value-Guided Search (VGS), simple yet effective strategy for steering long-context CoT reasoning models. We proposed a scalable token-level value model training pipeline that does not require a pre-defined notion of "step" or expensive per-step annotations. We collect large dataset of reasoning CoTs (OpenR1-Vic) and train lean 1.5B value model (DeepSeek- V.-11. 5B), which we show can effectively guide the CoTs of DeepSeck models up to 14B in size. With extensive experiments, we demonstrate that VGS with DeepSeek- -eepSeek-VM-111.. WM-1.5B enjoys better test-time compute scaling than standard methods (e.g., majority voting, best-of-NN and other scoring models (e.g., existing PRMs and BT model), achieving higher performance ceiling while reducing the FLOPs needed to extract the same performance as baseline methods (Fig. 1). Our results point to VGS promising approach scale TTC of emerging reasoning models.
Discussion of Limitations. Our value model is trained exclusively on completions roll-outs from Iean reasoning model x (e.g., DeepSeek- Sel--DDstll--pp 1.))). As frontier LLMs continue

---

to advance, the distribution of their generated responses may increasingly diverge from our training distribution, potentially degrading scoring and search performance. To maintain optimal performance. new value models will need to be retrained on rollouts from updated generator policies. However, we do not foresee this as major practical concern given the simplicity and scalability of our pipeline. To facilitate retraining and adaptation to similar verifialle domains, we open-source our codebase and provide a step-by-step recipe in Appendix B for data collection, training and search inference.
Acknowledgment
KW is supported by a Google PhD Fellowship. JPZ is supported by a grant from the Natural Sciences and Engineering Research Council of Canada (NSERC) (567916). ZG is supported by LinkedIn- Cornell Grant. Wen Sun is supported by NSF IIS-2154711, NSF CAREER 2339395 and DARPA LANCER: LeArning Network CybERagents. This research also supported by grants from the National Science Foundation NSF (US-1846210, IIS-2107161 161, and IIS- HDR-211 (8-22118310) the Cornell Center for Materials Research with funding from the NSF MRSEC program 171-97798775 DARPA, arXiv, LinkedIn, Google, and the New York Presbyterian Hospital.
References
[1] Marah Abdin, Sahaj Agarwal, Ahmed Awadallah, Vidhisha Balachandran, Harkirat Behl, Lingjiao Chen, Gustavo de Rosa, Suriya Gunasekar, Mojan Javaheripi, Neel Joshi, al. Phi-4- reasoning technical report. arXiv preprint arXiv:2304.21333 2025.
[2] Loubna Ben Allal, Lewis Tunstall, Anton Lozhkov, Elie Bakouch, Guilherme Penedo, Hynek Kydlicek, and Gabriel Martín Blázquez. Open rl: Update #2. attps://huggingtf co/ httog//pen-rr/ppaateeeeeeee 2, February 2025. Hugging Face Blog.
[3] Alex Ayoub, Kaiwen Wang. Vincent Liu, Samuel Robertson, James McInerney, Dawen Liang, Nathan Kallus, and Csaba Szepesvári. Switching the loss reduces the cost in batch (offline) reinforcement learning. arXiv preprint arXiv:2403.055. 2024.
[4] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022.
[5] Dhruv Batra, Payman Yadollahpour, Abner Siizman-Riverss and Gregory Shakhnarovich. Diverse m-best solutions in markov random fields. In Computer European Conference on Computer Vision Florence, Italy, October 7-13, 2012, Proceedings, Part V 12, pages 1-16. Springer, 2012.
2012: 12th
[6] Edward Beeching, Lewis Tunstall, and Sasha Rush. Scaling test-time compute with open models, 2024. URL https://huggingtaat http://ggggggacc.c/////////g/gg/ co/spaace//uuggingacccll/ blogpost- scalttssaainng tes--testttte time-tiimm-iee www-ttiimmemttteee compute.
[7] Mare G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforce- ment learning. In International conference on machine learning, pages 449-458. PMLR, 2017.
[8] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-345 1952.
[9] Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv: 2407.2277 2024.
[10] Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschen- brenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. alization: Eliciting strong capabilities with weak supervision. arXiv preprint arXiv:2312.09390. 2023.
gener-
[11] Jonathan D Chang, Kiante Brantley, Rajkumar Ramamurthy, Dipendra Misra, and Wen Sun. Learning to generate better than your 1lm. arXiv preprint 2XXi:2306.11816 2023.
10

---

[12] Kai-Wei Chang, Akshay Krishnamurthy, Alekh Agarwal, Hal Daumé III, and John Langford. Learning to search better than your teacher. In International Conference on Machine Learning. pages 2058-2066. PMLR, 2015.
[13] Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan. Alphamath almost zero: Process supervision without process. In The Thirty-eight Annual Conference Neural Information Processing Systems, 2024. URL http::/t////// /ttp::/:////eeeeieeeeee wwtpoviwwwi...eeeeeeeeeeeeeeeeee net/forum?? www.:iiiiiiaaaaaaaaa.a
[14] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint 1Xiv:21100.1166 2021.
[15] Scott Davies, Andrew Ng. and Andrew Moore. Applying online search techniques to continuous-state reinforcement learning, In AAAI/IAA,, pages 753-760, 1998.
[16] Ahmed El-Kishky, Alexander Wei, Andre Saraiva, Borys Minaicv, Daniel Selsam, David Dohan, Francis Song, Hunter Lightman, Ignasi Clavera, Jakub Pachocki, et al. Competitive programming with large reasoning models. arXiv preprint vrXiv:2502.0680) 2025.
[17] Jesse Farebrother, Jordi Orbay, Quan Vuong, Adrien Ali Taiga, Yevgen Chebotar, Ted Xiao, Alex Irpan, Sergey Levine, Pablo Samuel Castro, Alcksandra Faust, et al. Stop regressing: Training value functions via classification for scalable deep rl. arXiv preprint arXiv:2403.03350. 2024.
[18] Yichao Fu, Xuewei Wang, Yuandong Tian, and Jiawei Zhao. Deep think with confidence. arXiv preprint arXiv:2508. 15260, 2025.
[19] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and Others. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.
[20] Seungwook Han, Idan Shenfeld, Akash Srivastava, Yoon Kim, and Pulkit Agrawal. Value augmented sampling for language model alignment and personalization. arXiv preprint MXKi::2005.6663) 2024.
[21] Ehsan Imani, Kai Luedemann, Sam Scholnick-Hughes, Esraa Elelimy, and Martha White. Investigating the histogram loss in regression. arXiv preprint arXiv:2402.13425 2024.
[22] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:000.0036), 2020.
[23] Hynek Kydlíček. Math-verify: Math verification library, 2025. URL https: //tgt/ggttttt com/ huggingface//Mee MaatgffceeMaah----e Verify. library to rule-based verify mathematical answers.
[24] Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards. Bowen Baker, Teddy Lee, Jan Leike, John Schulman, llya Sutskever. and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
[25] Jiacheng Liu, Andrew Cohen, Ramakanth Pasunuru, Yejin Choi, Hannaneh Hajishirzi, and Asli Celikyilmaz. Don't throw away your value model! generating more preferable text with value-guided monte-carlo tree search decoding. arXiv preprint arXiv:2309. /5028, 2023.
[26] Runze Liu, Junqi Gao, Jian Zhao, Kaiyan Zhang, Xiu Li, Biqing Qi, Wanli Ouyang, and Bowen Zhou. Can 1b IIm surpass 405b Ilm? rethinking compute-optimal test-time scaling. arXiv preprint arXiv:2502.06703. 2025.
[27] Bruce T Lowerre. The harpy speech recognition system. Carnegie Mellon University, 1976.
[28] Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Meiqi Guo, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, et Improve mathematical reasoning in language models by automated process supervision. arXiv preprint arXiv:2406.06557 2024.

---

[29] Sidharth Mudgal, Jong Lee, Harish Ganapathy, YaGuang Li, Tao Wang, Yanping Huang, Zhifeng Chen, Heng-Tze Tze Cheng, Michael Collins, Trevor Strohman, et al. Controlled decoding from language models. arXiv preprint 1rXii:2310.17000 2023.
[30] Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Perey Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393 2025.
[31] Ivo Petrov, Jasper Dekoninck, Lyuben Baltadzhiev, Maria Drencheva, Kristian Minchev, Mislav Balunović, Nikola Jovanović, and Martin Vechev. Proof or bluff? evaluating Ilms on 2025 usa math olympiad. arXiv preprint arXiv25533.299.: 2025.
[32] Qwen Team. Qwen3: Think deeper, act faster, 2025, URL https://qwenlm.. blog/ blog/qwen3/. qwen3/. Accessed: 2025-05-08.
io/
[33] Nikhil Sardana, Jacob Portes, Sasha Doubov, and Jonathan Frankle. Beyond chinchilla-optimal: Accounting for inference in language model scaling laws. arXiv preprint arXiv:2401.00444 2023.
[34] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL https: org/ abs/ 1170776677777.7
[35] Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein. Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated process verifiers for LLM reasoning. In The Thirteenth International Conference on Learning Representations, 2025. URL httts:// net/ foet/forumm? A607=666774222
[36] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou. Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with general reinforcement learning algorithm. arXiv preprint 17Xiv:171..0/815. 2017.
[37] Charlic Victor Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test- time compute optimally can be more effective than scaling parameters for reasoning. In The Thirteenth International Conference on Learning Representations, 2025. URL https: / //openreview //operreviev..ee//trrr///ieeeeee net.:forrum/d--4///aaa
[38] Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, et al. Paperbench: Evaluating ai's ability to replicate ai research. arXiv preprint arXiv:2504. 018488 2025.
[39] Jiashuo Sun, Yi Luo, Yeyun Gong, Chen Lin, Yelong Shen, Jian Guo, and Nan Duan. Enhancing thain-of-thoughts prompting with iterative bootstrapping in large language models. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Findings of the Association for Computational Linguistics: NAACL 2024, pages 4074-4101 Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/v11 1/2024. 1/202..iin/.... .ow.ooyy..orro..oooo....o..
URL https:
//aclanthology.coo org/ 2024.0224442 windings-nnaadnnee naagg..naaal... 257/
[40] Richard S Sutton, Andrew G Barto, al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
[41] Ashwin K Vijayakumar, Michael Cogswell, Ramprasath R Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. Diverse beam search: Decoding diverse solutions from neural sequence models. arXiv preprint arXivv1610.02424, 2016.
[42] Junlin Wang, Shang Zhu, Jon Saad-Falcon, Ben Athiwaratkun, Qingyang Wu, Jue Wang. Shuai- wen Leon Song, Ce Zhang, Bhuwan Dhingra, and James Zou. Think deep. think fast: Investigat- ing efficiency of verifier-free paarencc-time-scaliinn methods. arXiv preprint arXiv:2504.1400 140477 2025.
[43] Kaiwen Wang, Kevin Zhou, Runzhe Wu, Nathan Kallus, and Wen Sun. The benefits of being distributional: Small-loss bounds for reinforcement learning. Advances in neural information processing systems, 36:2275-2312 2023.
12

---

[44] Kaiwen Wang, Nathan Kallus, and Wen Sun. The central role of the loss function in reinforce- ment learning. Statistical Science, 2025, Forthcoming.
[45] Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce Ilms step-by-step without human annotations. arXiv preprint arXiv:23/2.08993, 2023.
[46] Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprini ar arXiv:2203.11117 2022.
[47] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le. Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023. URL attps://openreviiss wwwt/teuur/rrre indddmiid--- 1PL1NIMMTW.
[48] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances in neural information processing systems, 36: 1809-11822, 2023.
[49] Di Zhang, Xiaoshui Huang, Dongzhan Zhou, Yuqiang Li, and Wanli Ouyang, Accessing gpt-4 level mathematical olympiad solutions via monte carlo tree self-refine with Ilama-3 8b. arXiv preprint arxiv:2406.0739/ 2024.
[50] Hanning Zhang, Pengcheng Wang, Shizhe Diao, Yong Lin, Rui Pan, Hanze Dong, Dylan Zhang, Pavlo Molchanov, and Tong Zhang. Entropy-regularized process reward model. arXiv preprint arXiv:2412. 1006, 2024.
[51] Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons developing process reward models in mathematical reasoning. arXiv preprint arXiv:2501.0730) 2025.
[52] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Livia Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. Sglang: Efficient execution of structured language model programs. Advances in Neural Information Processing Systems, 37:62557-62583, 2024.
[53] Jin Peng Zhou, Kaiwen Wang, Jonathan Chang, Zhaolin Gao, Nathan Kallus, Kilian Q Wein- berger, Kianté Brantley, and Wen Sun. qf: Provably optimal distributional rl for Ilm post-training. arXiv preprint arXhv:2502.20548 2025.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please see the second to last paragraph of our introduction which includes both our claims and the sections where the claim is supported.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Section 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide experimental details for reproduction in Section and Appendix B, E and F.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification: We provide guide and experimental details for reproducing the main results in Appendix. We will also release our data, models and codebase for distributed training and efficient search.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm co/
wtt.oodddesamissssooooooooooo..o for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: wttpcc/ggg/////eeeeeeeiec htte//CooSuumiieeeeeee wettimiissoooooiiiooooo for more details.
/ //nips.co.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide training and evaluation details for reproduction Section 3 and Appendix B, E and F.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report standard deviation of performance across seeds and bootstrapping repetition in Table
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The compute resources used are reported in Appendix E.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code ethics
Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: htttpp////errpps..pppp/////////
Question:
Answer: [Yes]
Justification:
The research conducted in the paper conforms with the NeurIPS Code of
Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper presents work whose goal is to advance the field Machine Learning. There many potential societal consequences of work, none which we feel must be specifically highlighted here.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The benchmarks and data splits are publicly available. All licenses respected.
Justification:
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasttssss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Assets will be released, and all instructions and details will be included for reproduction.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
19

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
20

---

Appendices
Table of Contents
Appendix A. Related Works
Appendix B. Summary of VGS Pipeline
Appendix C. Additional Experiment Results
Appendix D. Further Details of Data Collection
Appendix E. Further Details for Value Model Training
Appendix F. Further Details for Inference with Search
Appendix H. Inference FLOPS Computation
Note: In the appendix, we also provide additional empirical results in Appendix C. Two results are worth highlighting here. First, in Appendix C.6, we provide test-time scaling results for guiding DeepSeek-1. further trained with PPO on our math dataset. We find that VGS improves test-time scaling compared to MV and WMV, which shows that our method nicely complements policy-base RL training. Moreover, in Appendix C.7, we include three qualitative examples of contrastive blocks that were selected or rejected by our value model during beam search process. We see that our value model prefers blocks with more straightforward logical deductions, yielding more efficient and effective CoT for reasoning.
A Related Works
Test-Time Compute
Score-Free
Score-Based
Majarity Voting IMV)
Sequential CoT
Outcome-Level
Process-Level
Best-offN
(Wiighted MV
PRMs
Value Model
Figure 9: Taxonomy of TTC Methods. Score-free TTC methods do not require an external scoring model, e.g., by taking majority vote. Score-based TTC methods require external scoring model. The coarsest scoring model is an outcome reward model (ORM), which scores a whole response and can be used for best-of-NN or weighted MV. A more fine- -raiine scoring model process- scorers, which includes process reward models (PRMs) and value models; these more graiind scoring models can be used for search.
Test-time compute (TTC) broadly refers to algorithms that improve problem-solving performance when given more compute (i.e., FI.OPs) at test-time. Fig. 9 summarizes the taxonomy of TTC methods. The simplest TTC methods are score-free in the sense that they do not require access to an external scoring model. A notable example is majority voting (MV), which selects the most frequent answer among N responses, breaking ties randomly [14, 47, 9]. Also known as self-consistency, MV can be applied to tasks where the output space is equipped with equivalence relation, e.g., mathematical formulae that can be symbolically checked for equality. Other score-free TTC methods include sequentially revising the response via CoT prompting [39, 30] and hybrid methods [42].
There are also score-based TTC methods that employ an external scorer. The coarsest type of scorer is an outcome reward model (ORM), which takes the full prompt and response as input and produces a scalar that measures the quality correctness of the response. Popular examples of ORMs include Bradley- Herry reward models [8] or LLMeee-a-Judge [4]. ORMs can be used for
21

---

best-of-N (BoN), which selects the response with the highest score [14, 4]. ORMs can also be used for weighted majority voting (WMV), which generalizes MV where the strength of response's vote is proportional to its ORM score. Weighted MV (WMV) typically provides an improvement over vanilla (unweighted) MV [6. 26), which is also what we observe in our experiments (Fig. 3).
Outcome-level TTC methods (e.g. BoN, WMV) may be further refined with process-level scorers that guide the generation process in a fine-grained manner. We remark that our value model can act as both an outcome-level and a process-level scorer. When queried with partial response, the value model predicts the expected quality of future completions under n. When queried at the end of a full response, the value model predicts the quality of the final response. Indeed, our best performing setup for value-guides search (VGS) uses intermediate values to guide block-wise beam search and uses final values via WMV to aggregate the final beams, which employs both the process-level and outcome-level scoring capabilities of the value model. Finally, to the best of our knowledge, the combination of search with WMV is novel to this work, and we found this to be crucial ingredient to effectively scale TTC of DeepSeek models.
Prior works on process-level TTC largely focused on step-wise search with process reward models (PRMs), which measures the correctness of fine-grained step [24]. They showed that step-wise search can provide better performance than outcome-level TTC methods [29, 48, 25, 37, 49, 35, 13]. However, training step-wise PRMs requires a pre-defined notion of step. which is challenging to explicitly define for general reasoning [19]; e.g., prior works used newlines \n to separate steps, but DeepSeek's 's CoTs often contain newlines in-between coherent thoughts. Moreover, prior PRM training techniques require per-step annotations, via humans [24), LLM-Judges [51], or per-step MC rollouts [45, 28], which are expensive to collect for long reasoning traces; e.g., a single response from DeepSeek models typically contains hundreds of newlines.
These limitations make difficult to scale PRMs to long-context reasoning models [19], and all these prior works could only evaluate on short-context models with easier benchmarks such as GSM8K [14] and MATH [24]. In contrast, our paper focuses on scaling process-level guidance to long-context reasoning models, and we propose block-level search method that mitigates the above limitations. We train token-level value model by collecting rollouts from random solution prefixes, which requires neither pre-defined notion of step nor per-step annotations. We use our value model to guide a block-wise search process, where the block size a hyperparameter and we find there exists a consistent choice that works well across inference budgets (Fig. 6). Crucially, we are able to scale our value-guiee guidedd search (VGS) to long-context DeepSeek models and demonstrate impressive performance and efficiency gains on challenging math competitions (Fig. 1).
Closely related our work is Setlur al. [35], who propose train a token-level process advantage verifier (PAV), which is the sum of n Q-function and an (off-policy) expert's 's advantage function, to guide step-wise search. This method is similar to ours since the training process also occurs at a token-level and is agnostic to the definition of step. However, limitation of the PAV is that if the expert disagrees with the underlying policy, then maximizing the PAV can lead to suboptimal behavior [1 2]. Our approach of directly using the value model does not have this issue. Moreover, Setlur et al. [35] proposed to use PAVs to guide step-wise search, which still requires a definition of step at inference time; in contrast, we propose to use block-wiss search which does not require a definition of step at inference. At a technical level, Setlur et al. [35] trained the PAV by minimizing the mean-squared error (MSE) loss; in contrast, we propose to use the cross-entropy loss, which has been shown to work better for downstream decision making [17, 44, 53].
We remark that some prior works proposed to use token-level value models to reweight the next-token distribution of the generator [29, 50, 20, 53]. However, these methods require one classifier call per token, which is more expensive than block-wise search. Morcover, token-level guidance might also be less effective because the imperfect value model may introduce cascading errors if queried at every token. We highlight that Mudgal et al. [29] also experimented with wise BFS and found this to be more effective at scaling test-time compute than reweighting the next-token distribution (i.e., token-wise guidance). One drawback of block- wise BFS is that the blocks may all become correlated due to sharing the same prefix. Thus, we build upon Mudgal et al. [29] by proposing to use beam search, which we show yields better test-time scaling for reasoning models (Fig. 6).
Recently, Fu al. [18] showed that confidence as measured by the average next-token entropy along the thinking trace can also be used to guide the CoT generation process to improve TTC performance. An advantage of Fu et al. [18] that it does not require training a value model and can be applied
22

---

directly given generator policy. However, it does not leverage any reward signals and may fall short in planning long-term strategies [53]. It is promising future direction explore how to combine incertainty-baser methods with value-based methods.
B Summary of VGS Pipeline
We provide step-by-step recipe for running VGS on any verifiable domain of interest. This recipe is applicable any task with reward label for responses (i.e., outcome-level feedback). If the task has continuous rewards, standard trick from distributional RL is to discretize the reward distribution as histogram, and then the value model is simply the expected reward under the learned distribution [7, 21, 17, 43, 44].
Start with a verifiable domain, where responses are identified with a label and reward.
2. Identify good dataset of prompts.
3. Identify of roll-in policies and single roll-out policy. The roll-in policies should provide a diverse distribution of solutions, and the roll-out policy should be strong enough complete responses given partial roll-in.
4. For each prompt, sample roll-in responses from the set of roll-in policies.
5. For each roll-in response, sample m random indices {ix Ji[(m) and collect roll-out per index. Thus, there are nm roll-in, roll-out pairs per prompt.
6. Post-filter by removing prompts where all roll-out responses fail complete solve the prompt.
7. At this point, we have created our dataset of roll-in, roll-out pairs. We are now ready to train our value model.
8. Train classifier value model by following (Section 2.1). Sweep hyperparameters such as learning rate.
9. Choose generator policy be guided by the value model. The most in-distribution choice is to use the roll-out policy rf
10. Perform model selection by running outcome-level TTC (e.g., WMV) on some validation benchmark.
11. Sweep search parameters (e.g., block size, beam width, DVTS parallelism) on the validation benchmark.
12. Run the final model on the test benchmark with the best search parameters.
The sampling distribution for the cut-off index (Step 5) is also worth tuning. For example, values earlier or middle indices may be harder to predict than final indices, so is worth sampling more cut-off indices from these earlier regions.
23

---

Additional Experiment Results
C.1 Full Main Results Table
We reproduce Table 1 with additional results and baselines. The results with closed source models were obtained via API calls in May 2025. We see that with budget of 256, our 1.5B value model can guide DeepSeek-1.58 (total parameter count is 3B) to reach parity with the pass @ I of 03-mini- mediim. 03-mini-medium
Test-time scaling DeepSesk-1.55 1. (N 256)
AIME-24
AIME-25
HMMT-22
IIMMT-25
AVG
VGS w/ DeepSeek- VM- 58 (ours)
72.0 + 0.4
46.7 0.7
3..4
32.8 + 0.8
45.7 1.0
WMV w/ DeepSeek-VN- 1M-1. 5B (ours)
69.6 3.9
45.1 2.2
29.1 2.6
28.9 2.6
43.2 1.4
VGS w/ DeepSeek-BT-11.. . (ours)
73.1
40.6
28.1
27.5 0.0
42.3 0.5
WMV w/ DDeepSeek-BT11... 5B (ours)
72.0 3.3
40.5 2.9
25.3
2.3
24.6
40.6 1.6
VCS w/ Qwen2. 5-Math-PRM-77
1.0
1.
2.2
24.2
40.2
WMV 5
3.1
2.11
2..4
24.0
VGS WathhhheeeerddPP.-----..
3.1
1.4
1.
23.9 1.4
41.6 0.9
WMV w/ MathShepherd PRM-7B
3.2
4..0
2.3
25.6 3.1
41.2 1.4
MV @ 256
71.0 3.5
38.9
2.7
24.3
39.7 1.2
Test-time scaling larger models with our DeepSeek-VM- VM--1.B
DeepSeek-77 (N 128)
82.2 0.8
59.4
42.8 2.8
41.1
0.8
MV w/ DeepSeek-78 (N 128)
77.1 1.1
56.5 1.6
34.7 1.6
33.8
50.5 0.9
VCS w/ DeepSeek- 14B (N 64)
86.7 1 2.7
59.6 0.6
46.7 2.7
51.1 1.6
61.0 0.9
MV w/ DeepSeek-115 14B (VV
64)
80.6 1.2
67.0 17.0::20
40.6 -
59.6
Pass baselines for various models
DeepSeek-1.555 Pass @1
28.2 6.1
22.4 4.1
4.2
13.0 3.9
19.4 1.1
DeepSeck-1.50 Pass@2255
81.9 1.7
3.6
+..2
57.1 +
63.9
0.9
DeepSeek-7B Pass
54.8 6.0
40.9 6.1
31.5
144
25.5
38.2
1.3
DeepSeek-144 14B @ass
5.4
5.5
3..4
36.5 5.5
49.8
DeepSeek-32B 6
4.9
6.0
4.6
42.1
54.4
Deepscek-R1 (671B) Pass
8..0
0.9
46.7 3.5
60.8 0.5
o1-mini-media Pass@ of
63.3 1 6.6
52.3 6.88
3.11
34.0 5.9
45.7 1.5
Pass
83.7
+..7
5.00+
52.8
69.1
03-mini-medium Pass@]
49.2
45.8
3..4
3.0
41.0 1.5
03-mini-median Pass@8
83.0 4.6
3.9
4.3
64.9
70.3
o4-mini-medium @uss@]
85.4 4.3
4.5
5.0
61.1
69.8 2.5
04-mini mmdiimm Pass
95.4 = 2.6
0..
69.7
84.5
85.7
Table 2: (Top) Weighted majority vote (WMV) and VGS accuracies for DeepSeek-1 5B with an inference budget of N 256, with various scoring models. (Middle) Compares MV and VGS for larger DeepSeek models guided with our DeepSeek-Wis Mor-Seekk-M-1..55 5B. (Bottom) Lists performance of DeepSeek models and strong close-sourced reasoning models. For VGS, + indicates standard deviation across 3 seeds, and for MV, WMV, Pass N, it denotes bootstrap with 100 repetitions. We bold the highest avg. accuracy and underline second highest.
24

---

C.2
Per-benchmark Plots for Fig. 3
AIME-24
AIME-25
AIME-24
AIME-25
50
Accuraoy
80
80
30
HMMT-24
HMMT-25
HMMT-24
HMMT-25
Accuracy
25
40
44
20
1024
Budget (N)
Budget
Budget (N)
Budget
VGS
VGS
Pass@NN
WMV
BoN
Figure 10: Per-benchmark results for Fig. 3. (Left) Compares best-of-N (BoN), weighted majority voting (WMV) and VGS with either BoN WMV for the final aggregation. (Right) Compares VGS to majority voting (MV), a standard baseline that does not require a scorer.
C.3 Per-henchmark Plots for Fig. 4
Welghted Majority Voting
Value-Gulded Search
AIME-24
AIME-25
AIME-24
AIME-25
70
Accuraty
40
40
50
90
HMMT-24
HMMMT-25
HM-MT-24
HMMT-25
30
30
Acaredy
25
25
20
15
16
64
288
1024
16
64
254
1024
16
64
256
1024
16
64
256
1024
Inference Budget (N)
Inforence Budget (N)
Inference Budget (N)
Inference Budgal (N)
VM
BT
Math Shaphard 7B
- Owen PRM 7B
Figure 11: Per-benchmark results for Fig. 4. Comparison of our 1.5B value model (VM), our 1.5B Bradley-Terry reward model (BT), and two 7B state-of-the-art PRMs for two TTC scaling methods: (Left) WMV or (Right) VGS (with WMV as a final aggregation step).
25

---

C.4 Per-benchmark Plots for Fig. 5
AIME-24
AIME-25
80
Accuracy
50
50
30
HMMT-24
HMMT-25
50
Accuracy
40
40
30
30
16
1024
16
1024
Inference Budget (N)
Inference Budget (N)
DeepSeek 1.5B
Figure 12: Per-benchmark results for Fig. 5. With the same DeepSeek-VM- .5 providing guidance, search continues to improve with more test-time compute.
C.5
Per-benchmark Plots for Fig. 8
AIME-24
AIME-25
50
70
Accuracy
50
30
HMMT-24
HMMT-25
30
Accuracy
25
25
20
20
16
64
258
1024
16
256
1024
Inference Budget (N)
Inference Budget (N)
VGS
Random Search
Hybrid
Figure 13: Per-benchmark results for Fig. 8. Random search is the same search process as VGS except intermediate blocks are randomly selected instead of using our value model.
26

---

C.6 Results for Guiding a PPO Policy
Guo al. [19] mentions that the performance of distilled DeepSeek models can be further en- hanced through reinforcement learning (RL). In this section, we explore whether we could guide the generation of a RL trained policy. Specifically, we apply Proximal Policy Optimization (PPO) [34] to DeepSeek-1. 5B using prompts from OpenR1-Cleand and guide the trained model with DeepSeek-VN-1.555 VM-1.5B.
We perform full parameter training with 8 H100 GPUs and use the same model as the policy for critic. We use a rule-based reward function based solely on the correctness the response, assigning +I for correct answers and 0 for incorrect or incomplete ones. To ensure that the learned policy и remains close to the reference policy Tref, additional KL penalty is applied the reward:
r(z, v) 7KL (In (t x) In (yy y x)).
(I)
where r(x,8) y) is the rule- based reward for prompt z and response y, and 7Kl controls the strength of the KL penalty. To further encourage exploration, we apply standard entropy regularization by subtracting the policy entropy from the loss weighted by coefficient "
Lroo Tentropy H(T [r)]
(2)
The hyperparameter settings are shown below.
PPO Hyperparameter Setting
Setting
Parameters
Generation (train)
temperature: 1.0
top
PPO
batch size: 256
Yentropy: le-3
mini batch size: 128
TKL: 1c-4
micro batch size:
gae 7/ 1
policy learning rate: le-6
gae A:
critic learning rate: le-5
clip ratio: 0.2
train epochs: 25
Total number of steps: 2250
DeepSeek-1.58
Pass
Pass@8
Puss 16
Pass@ 32
Pass@64 64
Pass@ 128
Pass 256
AIME-24
48.7 1 5.0
58.9 4: 4,9
67.3 4.7
74.1 + 3.2
78.4 d 3.0
80.4 i 2.4
81.9 1.7
AIME-25
33.1 1 4.1
39.1 4.4
43.7 43.7.33. 3.9
49.0 : 3,8
54.0
58.8
3.4
62.6
3.6
HMMT-24
24.0 4.6
28.6 3.9
32.9 3.6
37.7 = 4.4
42.9 42.9
48.1 3.7
54.2
HMMT-25
21.6 4.6
26.7
30.1 4.8
37.8 5.1
45.6
52.6 4.2
57.1
DeepSeek-1.5B-PPO
Pass@4
Pass@8
Pass
Pass@32
Pass@64
Pass@
Pass@256
AIME-24
54.0 5.0
4.6
67.6 4.1
73.3 3.4
76.8 2.4
78.3 1.7
79.6
AIME-25
35.9
3.9
3.8
45.8 4.2
50.9 3.5
56.1 3.0
59.8 3.6
64.1
HMMT-24
27.2 + 4.2
32.8 4.4
37.6
37.6++3.6 3.6
41.5 + 3.6
45.0
48.8 + 3.3
52.4 + 3.4
HMMT-25
22.8 1: 4.0
26.8 4.3
32.5 4.4
36.9 1: 4.4
43.7 3.38
48.9 1. 3.8
52.6 -: 3.6
Table 3: Pass@N results for DeepSeek-1..B 1.0beek11.5 model and PPO-trained DeepSeek-1.52. 5B-PPP model.
Table 3 presents the comparison between the DeepSeek-1 .B model and the PPO-trained model (DeepSeek-1 5B-PPO). (.B-Seee-..B--PP) As N increases, the performance gap gradually narrows. While the PPO- trained model performs competitively at lower N values, it is surpassed by the base model at Pass @3 32 on both the AIMB-24 and HMMT-25 datasets. This decline in performance could be attributed the reduced entropy of the model after PPO training, which limits the diversity of model generations and negatively impacts performance higher @
27

---

256 We report our value- galle-guided results of the -POinrrieed model in Fig. 14. We observe that VGS nicely complements PPO training and provides additional test-time compute gains in performance compared to WMV and MV.
AIME-24
AIME-25
Overall
70
50
Accuracy
40
35
40
30
4
16
64
256
4
16
64
256
4
16
64
256
Inference Budget (N)
HMMT-24
HMMT-25
35
30
Accuracy
25
VGS
WMV MV
20
20
15
15
64
256
16
Inference Budget (N)
Inference Budget (N)
Figure 14: Guiding DeepSeek- 1.5B Trained with PPO. Comparison of VGS, WMV and MV for TTC scaling our PPO policy.
28

---

C.7 Qualitative Examples
Figures 17, 18, and 19 (at the end of the paper) show representative qualitative examples where value scores from VGS are used to guide beam search. At each step, two blocks of tokens are proposed, and the one with the higher value is selected to continue the solution. Due space constraints, parts of the beams are abridged with .... and for ease of visualization, blue highlights indicate correct reasoning steps, while red highlights denote incorrect ones.
Low-scoring beams exhibit different types of failure. In Figure 17, the rejected beam alternates between correct and incorrect steps, resulting in confused and ultimately incorrect reasoning. In Figure 18, the beam begins with plausible strategy involving GCD analysis but eventually resorts to ineffective trial and error. In Figure 19, the beam makes a critical error in the algebraic transformation early on and fails to recover. In contrast, the selected beams across all examples demonstrate systematic reasoning and successfully solve the problems.
Interestingly, despite the critical error in Figure 19, VGS assigns a moderately high score (0.337) to the rejected beam-higher than scores for less subtle failures in earlier examples- suggestinn that even significant mistakes can be difficult to detect when embedded in otherwise coherent reasoning.
Finally, empirically compare the distribution of generation lengths between the DeepSeek-1 1..5...5. base model and VGS with DeepSeek-11 5B across all benchmarks (Figure 15). On average, VGS generates noticeably shorter responses (11,219 tokens vs. 12,793 for the DeepSeek-1 1.nkk-1.50 .5B base model), suggesting that beam search not only enhances accuracy but also promotes more concise reasoning. This trend is consistent with our qualitative analysis, where beam search tends to favor token blocks that are direct and volution-oriento Notably, the sharp peak near 16,000 tokens corresponds to the maximum generation length of DeepSeek models (16,384). For the base model, as many as 50% of the generations reach this limit, often resulting in incomplete outputs.
rather than verbose or meandering reasoning.
AIME 24
AIME-25 25
0.5
0.4
Portion
0.3
0.2-
0.0
2000
4000
6000
8000
10000
12000
14000
16000
2000
4000
6000
8000
10000
12000 14000 16000
Generation Length
Generation Length
HMMT-24
HMMT-25
Base Model VGS
0.5
0.4
Portion
0.3-
0.2-
0.0
2000
4000
6000
8000
10000 12000 14000 16000
2000
4000
6000
10000 12000 14000 16000 Generation Length
8000
Generation Length
Figure 15: Histogram of generation lengths for the DeepSeek-1 -..o base model vs. VGS. VGS consistently produces shorter responses across benchmarks, with average lengths of 11,219 and 12,793 tokens for VGS and the base model, respectively. The peak around 16,000 tokens reflects the generation cap of DeepSeek models, which the base model frequently hits, often resulting in incomplete outputs.
29

---

D Further Details of Data Collection
Pre-filtering process. Here describe the pre-filterigg process for constructing OpenR1-Cleaned in more detail. Below are the sequence of filtering operations we performed on OpenR1-Mach [2]. We arrived at these rules by manually inspecting the data, by sampling 100 random problems from the dataset and checking if all problems' solutions looked reasonable to us. In OpenR1-Matin solution a fleshed-out solution to the math problem, an answer the final answer to the math problem.
1. Filter out all solutions with 0 or > boxed answers (enclosed in \boxed These are ambiguous and difficult to parse out the answer.
2. Filter out answers which are difficult automatically parse or verify. This includes answers containing: 'or', 'and' \mathrm, Nquad, answers with equal signs, commas, semicolons, cuup, capp. inequality symbols, approximation symbols.
3. Filter out multiple-choice questions, which are labeled with question_type 'MCQ'.
4. Filter out questions with multiple parts, as it ambiguous which part the answer is for.
5. Filter out questions containing links (http:// or https://), since the models we test cannot access the web.
Roll-in roll-out process. We also provide further intuition for the roll-in vs. roll-out process (illustrated Fig. left). The roll-in and then roll-out process a standard technique imitation learning [12] and reinforcement learning [11].
Roll-out. The roll-out process uses a fixed policy N roll-out from any partial solution provided by the roll-in process. The rationale for using fixed roll-out policy is to fix the target of the classification value regression problem. particular, the classifier trained predict the probability of each class under the roll-out policy, given the partial solution.
The main point of the roll-in process is create diverse distribution of partial solutions to roll-out from. By creating a diverse roll-in distribution with multiple roll-in policies, we can ensure diverse context distribution and will generalize better new traces. To select where to cut a roll-in (to start the roll-out), we sample a cut index from the distribution vi of p(i) Σ,eı v3 where L is the length of the roll-in. We chose this such that the cut index is more likely to occur at earlier positions of the roll-in. We want encourage more learning carlier positions since those prediction problems are more difficult than at later positions. The following figure (Fig. 16) illustrates the distribution of the length of roll-outs, which shows that this indexing scheme indeed yields many long roll-outs.
Roll-in.
that the classifier is trained
CDF of Max Tokens vs Count
0.6
0.4
2500
5000
7500
10000
12500
15000
Tokens
Figure 16: Distribution of roll-out length.
30

---

E Further Details for Value Model Training
Our value model DeepSeek-VM-1 -Mrteekk-M-1.5555 uses the same base architecture 1.aenn11.50 .B model, which is 1.5B parameter transformer model with 28 layers and 1536 hidden size. To turn this model into a value model, we replace the LM head with scoring head, parameterized by two-layer MLP, that outputs logits for three classes: 'correct', 'incorrect', and 'incomplete.. The number of classes can be modified to suit the task hand.
the
Category
Parameter
Value
Base Model Initialization
DeepSeek-R3 -atteeekk-i-iiiie Sestilll-wee---
Hidden size dmodel
1536
Model
Score Head
two-layer MLP with hidden size dmeddl
Score Head Bias
False
Score Head Labels
0=Incorrect, 1=Correct, 2=Incomplete
Dataset
OpenR1-VW
Data
Validation split
500
Max sequence length
16384
Batch size
1024
Learning- rate schedule
Cosine with max max_1r=ie-4
Warm-up steps
10% of total steps 0.05 5
Training
Dropout
Number of epochs
Optimizer type
AdamW 0.9
B1
Optimizer
B2
0.95
Weight decay Grad Norm Clip
0.1
5
GPUs
16 nodes of x NVIDIA H100
Compute
Wall-clock time (h)
24 hours
Tokens Throughput (tokens/s)
2.07M
Loss Tokens Throughput (loss tokens/s)
835k
Total Tokens Processed (per epoch)
35.7B
Total Loss Tokens Processed (per epoch)
14.4B
Table Value Model Training Parameters.
sweeped learning rates le-4, 3e-4, 7e-5 and we save checkpoints at every epoch. selected best checkpoint via WMV performance on AIME-24.
31

---

Further Details for Inference with Search
Given problem, the prompt used is:
<I Bengin__o_sentnnee >< ></User >probbem) Please think step-by- -tpeep and put your final answer within \boxed(: < Assistant >><think>>> \n
use the same decoding parameters as in the original DeepSeck paper [19].
Category
Parameter
Value
Inference Engine
SGLang [52]
Max generation length
16384
Temperature
0.6
Decoding
Top-p
0.95
Think Token
<think>
End of Think Token
</ </think>
Model
DeepSeek-VM-11
Beam width
2
Best Parameters for Search (VGS)
Block size (tokens)
4096
Parallel branches (DVTS)
budget dependent
Final aggregation rule
Weighted Majority Vote (WMV)
Table 5: Decoding and Search Parameters.
G Further Details for Training Bradley-Terry Reward Model
Dataset. Recall that our dataset for value model training (OpenR1-VM) contains 56 responses per problem. construct a Bradley-Terry dataset, we sample up to 4 response pairs per problem, where pair consists of response with reward 0 (the 'reject' response) and response with reward 1 (the 'chosen' response). Some prompts may have fewer than 4 responses with reward or 1, and those cases, include as many possible. This yields dataset of roughly 122k pairs.
Model. We the same model architecture as the value model (Appendix E) except that the score head outputs single scalar score instead of three-dimensional vector. We use the same training pipeline, except that the training loss swapped to the standard BT loss [8]:
Lav (0, B)
Σ
log (foo,, ve) (o(x,y)))
B
EE
where a is the sigmoid function, Br is the reject response and Ve is the chosen response. We use batch size of 128 pairs and we train for epoch. We sweeped many learning rates: 3e-4, le-4, 7e-5, 3e-5, 7e-6. We found that the BT loss drops and plateaus much quicker than the value model loss, and all learning rates yielded similar final losses. We consider the last ckpt of each run and we selected 1r=3e-5 as best ckpt with search (width 2, block size 4096, WMV aggregation) on aime-24. We note that one detail for getting BT work with weighted majority is score, i.e., take WMV with a (fo(z, y)) instead fo (z. y) itself. While this doesn't 't affect BoN performance, found that taking sigmoid was crucial for WMV performance scale well.
the sigmoid of the BT
32

---

H Inference FLOPS Computation
In this section, we compute the FLOPs for search and show that adding value model guidance at the block-level introduces negligible compute overhead, as the vast majority of the compute is from generator model. We follow the approach outlined Kaplan the FLOPs for single forward pass of the transformer, ignoring the embedding layers for simplicity. Consider a transformer with Nioger layers, dimensional residual stream, d dimensional feed- forward layer, and d dimensional attention outputs. Then the number of non-embedding parameters N 21 + dff), and the number of FLOPs for single forward pass over a context of length Retz 18
[22]. Sardana et al. [33] compute
((eetx) = 2NV + Maayyrr
Then, in the regime where dmodel > Neta/ 12, Kaplan al. [22], Sardana et [33] further approx- imate the above by ignoring the Natz term, i.e., C' becomes independent of Retz. We adopt this approximation when estimating the inference FLOPs of our generator models. Thus, for context length of nets 16, 384, the inference FLOPs for one complete generation for each generator model 2N naz:
1. DeppSeekRR--Ditttl1---e--------- x 1.5B X 16384 49.17'
2. ReepSeex-il-Diitiill----ee-----e x 7B x 16384 2297'.
3. DaegSeel-RL-DDittilll-eeee----e x 14B x 16384 4597.
4. DeepSeek-R1 (671B, with 37B activated params): 2 x 37 16384 12127.
We now compute the FLOPs needed for forward pass of the value model. Since block-size of 4096, there are at most (6384/4090 value model inferences per generation. Thus, the FLOPs from the value model is:
1. 1.5B classifier: 2 x 1.5B x 4 = 12B.
2. 7B classifier (baselines): 7B 4 56B.
Thus,
that the value model FLOPs is negligible compared to the generator model FLOPs. In particular, when guiding a 1.5B generator with 1.5B classifier, the classifier FLOPs is only 0.024% of the generator FLOPs. With a compute budget of 256, this amounts to total FLOPs of (49.17 19 + 12B) x 256 = 12.6P. When guiding with a 7B classifier, the total FLOPs (49.17 17 + 56B) x 256 12.6P. Note that the FLOPs required for generating 256 independent generations is 49.17 256 = 12.6P. Thus, search has negligible overhead comapred to (weighted)
majority voting best of
33

---

A1ME-24 Problem: Jen enters lottery by picking 4 distinct numbers from S = (1, 2,...... 9. numbers are randomly chosen from S. She wins a prize if at least two of her numbers were 2 of the randomly chosen numbers, and wins the grand prize if all four of her numbers were the randomly chosen numbers. The probubility of her winning the grand prize given that she won prize is where m and n are relatively prime positive integers. Find m n.
Rejected beam with score 0.009: Okay, so need to figure out the probability that Jen wins the grand prize given that she won prize. That means have to use conditional probability here. remember that conditional prohability is P[A[ ()) = P( n B P( (). In this case, event is her winning the grand prize, and event B her winning a prize. So, need to find the probability that she won both the grand prize and a prize, which is just her winning the grand prize, divided by the probability that she won a prize. First, let me figure out the total number possible ways Jen can pick numbers from S. which has numbers. That should the combination of 10 choose 4, right? So. C(10, 4) 210. So. the total number of possible combinations 210.
So, she can win a prize two scenarios: either exactly two of her numbers are the randomly chosen ones, or all four are.... ... The number of ways she can have exactly two numbers correct is C(4, 2) ways to choose the correct numbers, and then C16 2) ways to choose the incorrect ones from the remaining numbers. Similarly, the numher of ways she can have all four numhers correct is just C(4, 4) So, the total number of winning ways C14, 2) C(6, 2) + C4, 4). ..(,,..(4 2) is 6, and C(6, 2) is 15. So, 6515 90. Then, C(4, 4) is 1. So, total winning ways are 90 + 1= 91. Alteraatvely,, I can compute the probability or correct numbers. Let compute that and subtract from to if it matches.
C(4,
not winning a prize, which is the probability that she has
Number of ways to have 0 correct numbers: C(4, 0) C(6, 4) 1 + 15 15. Number of ways
have I correct number: C44, C(6 3) 4 20 80. So, total non-winning ways: 15 + 80 95, Therefore, probability of not winning a prize 95/210, probability of winning a prize 1 95/2200 (210 95)/210 115/210. Wait, that doesn't 't match with the previous result. Hmm, something's s wrong here.
Wait, why did I get 91 before? Let me I had C(4, 2) C(6 2) C(4, 4). So, that's 6*15 +1=91, But that's only the number of winning ways where she has exactly two exactly four correct numbers. So, 91 winning ways, 95 non-winning, which adds to 186, which is less than 210. So, the rest must he the impossible case? No. hecause in reality, she can't 't have 0 correct numbers if she has 4 numhers. Wait. she can have 0 correct numbers if all her numbers are among the 6 incorrect ones. So, that's 15 ways. Similarly, 1 correct number is 80 ways. and 2 correct numbers is 90 ways, and 4 correct numbers is I way. So, 15 +80 +90 +1=10 86. Wait, so actually, 186 winning ways and 24 non non-winning ways. So, probability of winning is 186/210, 10, which simplifies to 31/35.
Therefore, P ([A = (1/210) (186)/(11/211))) = 1/ 1/186. So, m is and n is 186. Are they relatively prime? and 186 share no common factors besides 1, so yes. So, m + n + 186- 187....
Selected Beam with score 0.996: Okay, so this is a conditional probability problem. The formula for conditional prohability is P( P(A[B) = P(A n B))P where A is the event that she wins the grand prize, and B is the event that she wins a prize. Since winning the grand prize is a specific case of winning a prize. A is a subset of B. So, P(AnB) B) is just P(A). Therefore, the probability we're 'rre looking for is P(A)/P(B).
Since she picks 4 numbers from 10, the total number of possible 4-number combinations is C(10, 4). The number of favorable outcomes for A is 1 (since there's 's only one exact set of four numbers that could be the randomly chosen ones). So, P(A) is IIO( 4). ...
To find P(B), it might be easier to compute the probability of the complementary event, which is that Jen wins zero prizes or one prize, and subtract that from 1. So, P(B) = 1- P(0 prizes)- - P(1 prize). Let me compute P(O prizes) and P(1 prize).
First, P(O prizes): This the probability that none of Jen's four numbers are among the four randomly chosen. So, all four numbers she picked are from the remuining six numbers (since has 10 numbers, and if she picks 4, the remaining are 6). So, the number of ways to choose 4 numbers from is C(6, 4). Therefore, P(O prizes) = C6, 4)/C(10, 4).
Similarly. P(1 prize): This the probability that exactly one of Jen's numbers is among the four randomly chosen. So, we need to choose number from her 4, and numbers from the remaining
(since 10 4 = 6). So, the number ways C(4, 1) * C(6 3). Therefore, P(I prize) = [C(4.
C(6. (T6,3) /C 4). ... So, putting it all together: P(B)
[C(6, 4)/C(10, 4)) [C(4,))
C(6.3) 3)/33)))11 4)
Therefore, P(B) = 1- 15/210 80/210 Therefore, P(H) is 23/42. ... Therefore, the conditional prohability P(A is (1/210) = (1/210) x (42/23) = (42)/(210 x 23) (.3)....TTreeeeeee Therefore, the
probability becomes (1/5)/23 1/(5 23) 1/115. But 151 can't he simplified further, so I and is 115, which coprime. Therefore, n 115 116....
Figure 17: Example of selected and rejected beams during beam search with VGS. The high-scoring beam (score 0.996) follows correct and coherent line of reasoning, arriving at the correct answer. In contrast, the rejected beam (score 0.009) contains several inconsistencies and incorrect steps, despite occasionally making correct logical model. Highlighting is added for clarity: blue indicates correct reasoning steps or results, while red indicates incorrect ones. Part of generations are abridged with... ... notation.
demcoiirn-dmmoostaaattinn the effectiveness of VGS as a value
34

---

AIME-25 Problem: Find the sum of all positive integers n such that w + 2 divides the product 3(n + (n² +9).
Rejected beam with score 0.395: First, let's parse the problem. need to find all positive integers n where
is a divisor of 3(n + 3](n" + 9). So, mathematically, this means that + divides 3(n + (n 2 + 9). In other words, when you divide 3(n + 3) + 9) by n + 2, there should be no remainder. So, n + must a factor this product.
One way to approach this is by using the Factor Theorem. The Factor Theorem states that n i factor of polynomial P(x), then P(-2) = So, if let P(n) = 3(m 3)(n² 9), then P(-2) should be equal zero + divides P(n). Let compute 2(-2):
P(-2) 3( 2- 2)(3 (+))((2)) 9) Simplify each part: (-2+ 3) (-2)²
4 + 13 Thus, 39 Wait, perhaps I'm misapplying the Factor Theorem here. Let me think
P(-2) 3x again.
2. Then, k 2. Substitute == into the expression 3(n 33(n" 2 9) and what conditions must hold for to divide this. ...
Alternatiely.. another upproach express + as k.
Therefore, the product becomes: 3(k 1)(k² 4k
Which implies that &s divides 3(k 1)(k2 4k 13). Since k divides the product, it must divide least of the factors. But &s divides kc, we can consider the other factors.
But wait, & divides 3(k: + [) 4k 13), & divides 3(k 1) 4k + 13). Since &s and & + consecutive integers, their GCD is 1. Similarly, k and k2 4k 13 may have GCD, Let me check:
Let me denote d god(k, k2 4k 13). Then d divides &t and k²² 4k + 13. So, d divides k.a 4k + 13 k(k: (1(::-4] 4) k2 4k 13 k2 4k = 13. So, d divides 13. Therefore, d = or 13. Wait, maybe I'm complicating things. Let me approach this differently.... ... But maybe there's another to approach this, Let mc try plugging in small positive integers for and see if n + divides the product 3(n + 3)(n" + 9). That might help confirm if n 1 is the only solution.
Testing n = I: As above, product is 120, which is divisible by 3. Testing = 2: + = 4. Product is (13) = 195. 195 divided by 4 is 48.75, which isn't an integer. So, 4 doesn't divide 195.
Testing 9: + 2 = 11. Product is ((1) 3240. 3240 divided by II is approximately 294.545, not integer.
Testing 10: n + 2 = 12. Product is (119) 4209. 4209 divided by 12 is 350.75, not integer.
Wait, so up to 10, only n works. Maybe = 1 is the only solution...
Selected Beam with score 0.998: First, let me understand what the problem is asking. It says that n +-- 2 divides 3(n + 3) + 9). In other words, when we divide the product 3(n + 3) + 9) by n +-2, 2, the result should be an integer. So, need to find all positive integers n for which this condition holds and then sum them up.
Let me write this out mathematically. n divides 3(n + (n2 + 9), then there exists some integer such that:
3(n + (n² + 9) k(n + 2)
But since we're looking for integer solutions, maybe it's better to think about this terms of divisibility. So, n must be divisor of 3(n 3) + 9). That suggests that n 2 divides this product.
remember that numher divides another number then b modulo zero. So, mayhe use modular arithmetic here. Let me see.
Since + divides 3(2 3)(n" 2
9). must be that:
3(n + 3) + 9) = (mod n 2)
... First, 4-3 modulo 2. Let's substitute -2+ but maybe that's necessary. Alternativly,
note that + (n + 2) 1. So, modulo
n
Similariy, n² + 9. Let compute n² modulo + 2. Since -2 (mod + 2). then n² (-2)² = 4 (mod 2). Therefore, n² + 4 + 13 (mad n 2).
So, putting it all together, modulo + the product becomes:
3(n (n2 +9
13 39 (mod 2)
So, this implies that 39 (mod n 2). which means that + 2 divides 39. Therefore, + must
be positive divisor 39. Now, the positive divisors 39 are 1, 3. 13, and 39. But since n is positive integer, n 2 must be at least 3 (because if n = 1, n + = 3). So, the possible values for n + are 3, 13, and 39. Therefore, the corresponding values for are: If + 2 3, then = 1. If + 2 13, then 11. If n 2 39, then 37. Therefore, the sum of all such positive integers is 11 + 37. Let me compute that: + 11 is 12, + 37 is 49....
Figure 18: Additional examples of selected and rejected beams under VGS. The high-scoring beam applies a concise and effective modular reasoning strategy whereas the rejected beam attempts several approaches-inee inncuuuinnnnnnnnngnn an incorrect use of the Factor Theorem and trial-and-error-baa ultimately fails. Color coding follows Figure 17. Part of generations are abridged with ... notation.
35

---

HMMT-225 Problem: Mark writes the expression v abed the board. where abed four-digit number and of O. Derek, toddler, decides to move the changing Mark's expression to ey bed. Surprisingly, these two expressions are equal. Compute the only possible four-digit number abed.
Rejected beam with score 0.337: Okay, so have this problem where Mark writes the square root of a four-digit number, which is written /[abedJ, where isa' Is''t zero. Then Derek moves the first digit 'a' to the end, making the expression av (bezf). And the interesting part is that hoth expressions are equal. need to find the only possible four-digit number abed that satisfics this condition.
First, let mc try to understand the problem step by step. Mark's expression is the square root of a four-digit number, that's (1000a 1006 10c d). Derek changes this by moving the first digit "a' the end, the number is 10006 + 100c 10d a, and Derek's expression
multiplied by the square root this new number, which is / (10006 + 100c + 10d + a). And both these are equal.
So, mathematically. can write this us: v(1000a + (006 + 10c + d) av (10006 + 100c 10d + a)
If square both sides to eliminate the square roots, I get:
So, 1000a + 1006 + 10c + d
(10006 + 100c + 10d a a)
Selected Beam with score 0.999: Let me write this down step by step. Let's denote the four-digit number as N 1000a 1006 10c d. So, Mark's expression is vN. Then, Derek changes it av s//bdd). which / / (1006 10c d). According to problem, these two expressions are equal,
Which means:
/ /(1000a 1006 + 10c + = a (100b + 10c d)
me square both sides to eliminate the square roots:
1000a 100b + 10c
a (100b 10c +
Hmm, that gives me equation:
1000a 100b +
(100b 10c so 1000a 100b 10c three-digit number equation becomes: =
me
four-digit number
as M 100b 10c d.
But since N four-digit number and is single digit (from to since o 0), and M three-digit number (from 0 999). So, let rewrite N 1000a M.
1000a
Let rearrange this equation:
1000a a² M M
Factor out M the right side:
1000a M(a2-
Therefore, M (1000a) (a2- 1)
Since M must be integer (as it's three-digit number), (1000a) must be divisible by (a²- 1). So, (a?- 1) must be divisor of 1000a.
Let note that and then check if M is a three-digit integer.
digit from to 9. So. I can each from to 9 and see if (a? 1) divides 1000a,
Let's compute for each a: Starting with a=1:
u=1: M 1000*1 / (1 1)= 1000 / 0. which undefined. So, u=1 is invalid.
a=2:
M 1000*2/(4-1 =)= 2000 3 = 666.666... which is not an integer. So, a=2 invalid.
a=3:
M 1000*3/(9-
= 3000 / 375. That's an integer. So, M=375.
a=4:
M 1000*4/(16- 4400 15 =s 266.666.., not integer. So, a=4 invalid.
a=5:
M 1000*5/(25 -1) 5000 24 Rv 208.333... not integer.
6=
M 1000*6 / (36-1 1) 6000 35 =s 171.428.., not integer.
a=7:
M 1000*7 (49 1) 7000 / 48 Rs 145.833... not integer.
a=8:
M 1000*8/ (64 l) 8000 / 63 zs 127.450.., not integer.
n=9:
M 10000000//0000 / (81 1)= 9000 80 112.5, which not integer.
So, only a=3 gives us an integer M=375, which is three-digit number. Therefore, the only possible four-digit number is 3375.
Figure 19: Additional examples of selected and rejected beams under VGS. The rejected beam misrepresents Derek's transformation, leading to an incorrect equation that derails the solution. In contrast, the high-scoring beam correctly models the relationship and systematically solves for the unique valid value of a. Color coding follows Figure 17. Part of generations are abridged with. notation.
36


Reviews:
[{"review_id": "NO5TgxHGCP", "reviewer": "Reviewer_oEdM", "summary": "This paper proposes Value-Guided Search (VGS), a method to improve the efficiency and effectiveness of long-context chain-of-thought (CoT) reasoning in language models. The central idea is to train a token-level value model that predicts the outcome of a reasoning trace without requiring a fine-grained definition of “step.” This value model is then used to guide block-wise search (e.g., beam search with blocks of 4096 tokens), enabling more effective inference-time compute scaling than standard test-time compute (TTC) methods such as majority voting or best-of-N. The authors build a large dataset of 2.5M math reasoning traces from DeepSeek models and demonstrate strong results on four high-school math benchmarks (AIME & HMMT, 2024/2025), showing that their method is competitive with or better than step-level PRM-based baselines. All code, models, and data are promised to be open-sourced.", "questions": "See weaknesses.", "limitations": "Yes", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n* Clear and focused motivation: The paper is framed around the growing cost and inefficiency of long-context reasoning in LLMs, a timely and important topic.\n\n* Strong empirical results: The proposed method consistently outperforms standard TTC baselines (e.g., MV, WMV, BoN) and state-of-the-art PRMs across multiple competition math datasets, even when guiding larger generators like DeepSeek-14B.\n\n* Ablation and sensitivity studies are comprehensive: The authors evaluate block size, beam width, DVTS parallelism, and even hybrid/random guidance settings, increasing the paper’s technical completeness.\n\nWeaknesses:\n\n* High training cost and limited generalization of value model: (1) Although the authors claim their training pipeline is scalable, their value model (trained on DeepSeek-1.5B rollouts) requires 96×H100 GPUs running for 24 hours, not including tuning iterations. This is non-trivial for most researchers. (2) The trained value model shows signs of out-of-distribution (OOD) degradation when guiding larger generators (e.g., DeepSeek-14B), which indicates that a separate value model may be needed for each generator model. This introduces a significant maintenance and compute burden, limiting real-world applicability despite the authors' claim (Line 217–218) that this is not a “practical concern.”\n\n* Unclear advantage over step-level supervision: (1) A key motivation of the paper is that step-level supervision is difficult to define and expensive to collect. However, the authors do not quantitatively compare the cost or quality of their token-level labeling approach to step-level PRMs. (2) In practice, step definitions via rule-based chunking (e.g., newline delimiters) have been shown to work well in PRM training. Therefore, it remains unclear whether the additional complexity of collecting roll-in/roll-out pairs and full-trace supervision is actually more cost-effective than existing PRM alternatives.", "was_revised": true, "final_justification": "I have read the author's rebuttal and the comments from other reviewers, and after a comprehensive evaluation, I have decided to maintain my original score.", "rebuttal": "Dear Reviewer oEdM,\n\nThank you for your constructive review. We really appreciate the opportunity to respond to your questions and concerns.\n\n### Regarding Value Model Training Cost and Generalization:\n\n**(1) On Training Cost & Scalability**: We apologize for the lack of clarity -- by \"scalable,\" we refer specifically to our data collection pipeline. Our method avoids the need for expensive, step-by-step human or LLM-as-a-Judge annotations required by prior PRM approaches. This simplification is what enabled us to collect a large-scale dataset of 2.5 million reasoning traces (each having up to 16k tokens). Training on a dataset of this magnitude indeed requires significant compute. To ensure this effort benefits the entire community, we are open-sourcing the full dataset, model checkpoints, and codebase, allowing other researchers to bypass this one-time cost.\n\n**(2) On Generalization**: Thank you for this point. While it is true that our 1.5B value model shows signs of OOD degradation when guiding DeepSeek-14B, we want to highlight that our value model still strongly improves MV when guiding DeepSeek-7B. We view this as a successful demonstration of weak-to-strong generalization: our value model can guide a ~5x larger model to strongly improve the MV baseline (Table 1, VGS guiding DeepSeek-7B has 56.4% accuracy vs. MV with DeepSeek-7B has 50.5% accuracy, so VGS shows a 6% improvement). \n\nMoreover, in practice, value guidance is only a small fraction of the total inference compute as we showed in Appendix H, since the value model is only queried at every block of tokens. Thus, we believe training a large value model (of the same size as the generator or even larger) is a reasonable investment and we believe VGS is a promising way to for scaling test-time compute of reasoning models.\n\n### Regarding the Advantage over Step-Level Supervision:\n\nWe agree it is difficult to provide a direct \"apples-to-apples\" cost comparison. The key advantage of our approach is that it mitigates the need to define a \"step,\" a notion that is difficult to formalize for complex reasoning traces (Guo et al. 2025). For example, if one were to use a simple heuristic like newlines, a single reasoning trace from DeepSeek can easily exceed 100 \"steps.\" Using an LLM-as-a-judge for our 2.5M trace dataset would have required over 250 million such judgments, a staggering and impractical cost. In contrast, our outcome-based method requires only one cheap verification per reasoning trace.\n\nWe note that our approach is philosophically aligned with Sutton's \"bitter lesson,\" favoring general methods that scale with data and compute over those relying on human-defined structures such as a \"step.\" Indeed, our VGS method using a 1.5B value model, empirically outperforms a 7B PRM baseline trained with step-level feedback (Table 1, VGS w/ DeepSeek-VM-1.5B has 45.7% accuracy vs. VGS w/ Qwen2.5-Math-PRM-7B has 40.2% accuracy). Ultimately, we hypothesize it may be useful to learn an appropriate notion of \"step\" beyond the newline heuristic, and we leave that as promising future work.\n\nWe hope these clarifications address your concerns and highlight the significance of our contributions. Thank you again for your time and valuable feedback."}, {"review_id": "aCrnb8qBVZ", "reviewer": "Reviewer_4pBh", "summary": "Based on a newly collected (and also released) large-scale dataset with reasoning paths a value model is learned for guiding DeepSeek LLMs on reasoning tasks. The value model is trained on partial responses and completions based on a more cost-efficient, distilled  reference LLM and can be combined with multiple blockwise, chain-of-thought reasoning strategies such as best-first search or beam search. The resulting value-model guided search achieves good performance and efficiency on standard math benchmarks like AIME or HMMT.", "questions": "* Regarding the Experiment in Section 4.2: If I understand it correctly, the blocks are selected uniformly at random. Selecting the blocks according to the likelihood under the LLM seems like a stronger -- and equally expensive and simple -- baseline to me.  Do I misunderstand the experiment or is there another reason for why you didn't use this ablation?\n* How does the method conceptually compare to tokenwise reward-guided text generation?\n\n  > **A Critical Look At Tokenwise Reward-Guided Text Generation**, Rashid et al., 2025, https://arxiv.org/pdf/2406.07780\n\n   The training on partial sequences as describe in Section 2.1 strongly reminds me of the method in the above reference, but the related work section does not discuss it. \n\n* I assume tree-of-thought based reasoning is in an entirely different league of computational budget than chain-of-thought based methods. Is that why you did not empirically compare the value model with tree-of-thought based approaches or are there other reasons?\n* Can the value model also be applied to tree-of-thought based reasoning?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 2, "strengths_and_weaknesses": "**Strengths**:\n* significant performance and efficiency gains on standard reasoning benchmarks\n* robustness towards hyperparameter choice across a  wide range of computational budgets\n* methods scales well to long contexts\n* release of large-scale dataset with reasoning-responses that might also be of use for future work\n\n**Weaknesses**:\n* The empirical evaluation exclusively focuses on LLMs from the DeepSeek family\n* Some parts of the paper are not fully clear to me. I list my questions below.\n\n**Minor comment**:\n* I think it would further enhance the readability of Table 1, if the best and 2nd best performance on each of the benchmarks would also be highlighted, and not just the average performance.", "was_revised": true, "final_justification": "The authors rebuttal mostly addressed my concerns. I also read the other reviews and I agree with Reviewer oEdM, that an empirical comparison with step-level or token-level reward guided models would be interesting. On the other hand, if there is already some evidence in prior work that block-wise guidance performs better than token-level guidance (e.g. the paper of Mudgal et al. 2024, which I was not familiar with), I am willing to accept that such a comparison is not absolutely necessary. I will therefore maintain my positive score.", "rebuttal": "Dear Reviewer 4pBh,\n\nThank you for your positive and insightful review! We are happy to clarify the points you raised and appreciate your suggestions for improving the paper.\n\n### On the Scope of Evaluation:\nWe chose to focus on the DeepSeek series because they were the only open-weights reasoning models -- LLMs trained with RL to perform CoT reasoning -- at the time of submission. This focus also provided the benefit of a controlled study across different model scales (1.5B, 7B, and 14B) from the same architectural family, allowing us to investigate properties like scaling up generator sizes (Section 3.3) in a controlled manner.\n\n### On the Minor Comment:\nThanks for this suggestion. We will update the table to highlight the best and second-best performing methods on each individual benchmark in the camera-ready version.\n\n### Responses to Questions:\n1. Regarding the Random Search Ablation (Section 4.2): Your understanding is correct. Our intent is to cleanly isolate the effect of using our value model for selecting intermediate blocks versus having no selection rule at all. By removing the value signal entirely, we can directly measure its contribution to the search process. We agree that a likelihood-based selection is a strong baseline for a verifier-free search, and we will run this comparison and add a discussion to the final version.\n\n2. Comparison to (Rashid et al., 2025): Thank you for this reference; the work is certainly related and merits a citation and discussion. There are two primary issues / challenges with applying reward guidance at the token level that make it less efficient and scalable:\n    * Computational Cost: Token-wise guidance requires a classifier forward pass for every generated token [1,2,3], which is substantially more expensive than our block-wise method that only queries the value model once per block (e.g., every 4096 tokens).\n    * Compounding Errors: An imperfect reward or value model, when queried too frequently, can introduce cascading errors that derail the generation process. Indeed, prior work by Mudgal et al. 2024 compared block-wise versus token-wise guidance and found that block-wise approaches performed better, which informed our decision to focus on these methods. \n\n    We will expand on this discussion and add a citation to Rashid et al. to our related work section.\n3. Comparison to Tree-of-Thought (ToT) Approaches: Thanks for the great question. Our Value-Guided Search can be viewed as an instantiation of the broader ToT framework, where our value model serves as the verifier to score different reasoning paths. Specifically, in the ToT framework, our policy model $\\pi$ corresponds to the \"thought generator\", and our value model corresponds to the \"state evaluator\", which scores the generated blocks to determine which reasoning paths are most promising to pursue. Thus, VGS can be viewed as a specific, highly effective instantiation of this general framework. We show this particular approach is an efficient and powerful method for improving long-context reasoning, but we agree that exploring more complex ToT search strategies with a value model evaluator can lead to even more efficient reasoning, which we believe is a promising direction for future work.\n\nThank you again for your supportive review and constructive feedback. We will incorporate these clarifications and additions into our camera-ready version and believe your suggestions will significantly strengthen the paper.\n\n---------------\nCitations:\n\n[1] Rashid, Ahmad, et al. \"A critical look at tokenwise reward-guided text generation.\" arXiv preprint arXiv:2406.07780 (2024).\n\n[2] Mudgal, Sidharth, et al. \"Controlled decoding from language models.\" arXiv preprint arXiv:2310.17022 (2023).\n\n[3] Zhou, Jin Peng, et al. \"$ Q\\sharp $: Provably Optimal Distributional RL for LLM Post-Training.\" arXiv preprint arXiv:2502.20548 (2025)."}, {"review_id": "918cAdtQ2i", "reviewer": "Reviewer_uxSh", "summary": "This paper proposes Value-Guided Search (VGS), a method to improve the performance and computational efficiency of chain-of-thought (CoT) reasoning in large language models. The core idea is to train a token-level value model that predicts the final outcome (correct, incorrect, or incomplete) of a reasoning trace given a partial generation. This value model is trained on a large-scale dataset of 2.5 million reasoning traces, collected via a novel and scalable pipeline that does not require pre-defined \"steps\" or expensive human annotations. At inference time, VGS uses this value model to guide a block-wise beam search, selecting the most promising reasoning paths.\n\nThe authors demonstrate through extensive experiments on challenging competition math benchmarks (AIME and HMMT) that VGS significantly outperforms standard test-time compute (TTC) methods like majority voting and search guided by existing process reward models (PRMs). Notably, they show that a 1.5B value model can effectively guide much larger generator models (up to 14B) and reduce the FLOPs required to achieve a target accuracy. The authors are open-sourcing their dataset, model, and code.", "questions": "My overall assessment of the paper is positive. The following questions are intended to probe the boundaries of the method and offer suggestions for strengthening the paper even further. A thoughtful response to these points, particularly with supporting analysis where feasible, would solidify my view of this work as a top-tier contribution.\n\n1.  **Probing the Limits of Weak-to-Strong Generalization:** The \"weak-to-strong\" generalization is one of the most significant claims of this work, but the data suggests it may have limits. In Table 1, the performance improvement of VGS over standard Majority Voting (MV) diminishes as the generator model gets stronger: the accuracy gain is **5.9 points** for the 7B model (56.4 vs 50.5), but drops to just **1.4 points** for the 14B model (61.0 vs 59.6). This suggests the 1.5B value model is becoming a less effective guide for the much more capable 14B model.\n    *   **Question:** What is your primary hypothesis for this shrinking performance gap? Is it that the 14B model generates reasoning paths that are so complex or novel that they are fundamentally out-of-distribution (OOD) for the 1.5B value model, causing the value predictions to become less reliable?\n    *   **Actionable Suggestion:** To make the paper's claims about scalability more robust, could you conduct a small-scale analysis for the rebuttal? For instance, you could analyze cases where the 14B model with VGS fails but with MV succeeds. Does the value model assign an erroneously low score to the correct path in these cases? A more ambitious but highly impactful experiment would be to train a new 1.5B value model on a small set of roll-outs from the 7B model and show if this new value model provides better guidance for the 14B generator.\n    *   **Impact on Review:** A convincing analysis or a positive result from the suggested experiment would strongly reinforce the paper's central claims about the practicality and scalability of your approach, addressing what appears to be its most significant limitation.\n\n2.  **The Critical Role of Block Size:** Your ablation on search block size (Figure 6) shows that a large block of 4096 tokens is consistently optimal. This is a fascinating and perhaps counter-intuitive result, as one might expect that more frequent guidance (i.e., smaller blocks) would be beneficial for correcting errors early.\n    *   **Question:** Why do you believe such a large block size is optimal? I see three competing hypotheses: (a) **Signal-to-Noise:** The value model requires a large context to make an accurate prediction, and its judgments on shorter blocks are too noisy to be useful guides. (b) **Task-Specific Coherence:** The reasoning steps in competition math problems are long and logically self-contained, so interrupting the generator mid-\"thought\" is detrimental. (c) **Amortized Cost:** The overhead of invoking the value model is non-trivial, and using fewer, larger blocks is simply more computationally efficient for a given budget. Could you provide your perspective on these or offer an alternative hypothesis?\n    *   **Actionable Suggestion:** This is a key parameter of your method. To help the reader understand this result, could you provide a qualitative example in the appendix showing a generation from a small-block-size search (e.g., 256) that fails, contrasted with a successful 4096-block-size search on the same problem? A clear discussion of these trade-offs in Section 4.1 would significantly improve the paper.\n\n3.  **From Binary Math Problems to Nuanced Domains:** The success of your data collection pipeline and value model training hinges on the availability of an unambiguous, binary correctness signal from `math-verify`. This is an ideal scenario that is not representative of many other complex reasoning tasks.\n    *   **Question:** How do you envision the VGS framework being adapted to domains with more nuanced or non-binary rewards? Consider code generation, where a program might pass some unit tests but not others. Would you adapt the value model to predict a continuous score (e.g., fraction of tests passed), or would you define a more complex set of discrete outcome classes (e.g., 'passes all tests', 'fails compilation', 'fails runtime tests', 'infinite loop', 'incomplete')? How would the learning objective and the final WMV aggregation handle such a signal?\n    *   **Actionable Suggestion:** While a full experimental evaluation is outside the scope of the rebuttal, a more detailed discussion of these necessary adaptations would greatly strengthen the paper by clarifying the boundaries and future trajectory of this line of research. I suggest adding a dedicated paragraph on this topic in your \"Limitations\" or \"Conclusion\" section. A thoughtful response here would demonstrate a deeper consideration of the method's generalizability.\n\n4.  **The Power of Final Aggregation via WMV:** The finding that using Weighted Majority Vote (WMV) at the end of the search is crucial, and significantly outperforms Best-of-N (BoN) (Figure 3), is a very important and subtle result. It implies that even after guiding the generation block-by-block, the final value scores across the surviving beams still contain useful, independent information. In a perfect world with a perfect value model and search, the single highest-scoring path should be the best.\n    *   **Question:** What is your explanation for why WMV is so much more effective? Does this suggest that the value model's scores are inherently noisy, and that a single high score on one beam might be an outlier, making aggregation across multiple diverse beams a more robust ensembling strategy?\n    *   **Actionable Suggestion:** This is a key insight into the practical use of process-level scorers. Could you provide a brief analysis to support your hypothesis? For example, you could measure the variance of value scores across the final beams, comparing cases where BoN fails and WMV succeeds. A high variance in these cases would support the \"noisy value model\" hypothesis. Highlighting this finding and its implications more explicitly in the main text would be a valuable addition.", "limitations": "The authors have a dedicated limitations section that appropriately identifies the main limitation: the value model is trained on roll-outs from a specific policy and will need to be retrained to maintain optimal performance as generator models advance.\n\nTwo areas for improvement would be:\n1.  **Elaborating on \"Verifiable Domains\":** The authors state their pipeline can be adapted to \"similar verifiable domains.\" It would be helpful to expand on what constitutes a \"verifiable domain\" beyond binary pass/fail and discuss the challenges for tasks with more nuanced, multi-faceted, or human-in-the-loop reward structures.\n2.  **Potential Negative Societal Impact:** While the risk for math problem solving is low, the paper contributes to the broader area of improving automated reasoning. A brief mention of the potential dual-use nature of such technologies (e.g., generating highly convincing but subtly flawed arguments at scale) would be appropriate for a paper at this level.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "This is a well-executed paper that makes a significant contribution to the important problem of scaling LLM reasoning.\n\n**Strengths:**\n\n1.  **Significance and Impact:** The paper tackles a critical challenge in the LLM landscape: the immense computational cost of generating long, high-quality reasoning chains. The proposed VGS method offers a practical and effective solution that demonstrably improves the trade-off between performance and compute. The finding that a small value model can effectively guide a much larger generator (\"weak-to-strong\" generalization) is particularly significant, as it suggests a cheaper and more scalable path to improving frontier models.\n\n2.  **Technical Quality and Rigor:** The experimental evaluation is exemplary.\n    *   **Challenging Benchmarks:** The use of recent and difficult benchmarks (AIME/HMMT 2024 & 2025) provides a convincing testbed for their method, moving beyond potentially saturated benchmarks.\n    *   **Thorough Ablations:** The authors conduct a comprehensive set of ablations that validate their key design choices. The analyses of block size (Fig. 6), beam width (Fig. 7), the benefit of DVTS, and the direct comparison between value-guided and random search (Fig. 8) provide strong support for their methodology.\n    *   **Strong Baselines:** The comparison against not just standard TTC methods but also state-of-the-art 7B PRM models strengthens the paper's claims. The fact that VGS outperforms these larger, more complex PRMs highlights the effectiveness of the authors' proposed value model training scheme.\n    *   **Novel Empirical Finding:** The discovery that using Weighted Majority Vote (WMV) for final aggregation is critical to the success of VGS is a subtle but important finding that is novel to this work.\n\n3.  **Originality and Novelty:** The primary contribution is the novel and scalable pipeline for training a token-level value model without relying on a pre-defined notion of a \"step.\" This elegantly sidesteps a major limitation of prior work on Process Reward Models (PRMs), making process-level guidance practical for long-context models like DeepSeek. While the components (beam search, value models) are known, their combination into the block-wise VGS framework, the specific step-free data generation strategy, and the successful application to long-form math reasoning is a novel and impactful contribution.\n\n4.  **Clarity and Presentation:** The paper is exceptionally well-written, clear, and easy to follow. The problem statement is well-motivated, and the proposed method is explained logically. Figure 2 provides an excellent, intuitive summary of the entire data collection and search pipeline. The distinction from and advantages over prior work are clearly articulated.\n\n5.  **Reproducibility:** The commitment to open-sourcing the dataset (OpenR1-VM), the trained 1.5B value model, and the codebase is a major strength and a great service to the research community.\n\n**Weaknesses:**\n\n1.  **Limited Generality of the Value Model:** The value model is trained on roll-outs from a single, fixed policy (DeepSeek-R1-Distill-1.5B). While the weak-to-strong results are promising, Table 1 shows that the performance gap between VGS and standard Majority Voting (MV) narrows as the generator model size increases from 7B to 14B. This suggests the 1.5B value model may be approaching the limits of its ability to effectively guide a much stronger generator. The paper would be strengthened by a deeper discussion of this OOD issue and the expected scaling limits.\n\n2.  **Domain Specificity:** The entire evaluation is conducted on competition math problems. This domain benefits from having an unambiguous, binary correctness signal that can be checked automatically (`math-verify`). It is unclear how the VGS pipeline, particularly the 3-class (correct, incorrect, incomplete) data labeling, would translate to more open-ended domains where rewards are nuanced or require human evaluation (e.g., creative writing, summarization quality, or complex code generation).\n\n3.  **Fixed Search Hyperparameters:** The authors use a fixed block size and beam width for all problems, which they frame as a practical advantage over methods that require per-problem tuning. However, this \"one-size-fits-all\" approach might be suboptimal. It's possible that different problems could benefit from different search strategies (e.g., a wider search for harder problems), and a discussion of this trade-off between practicality and potentially higher performance would be welcome.", "was_revised": true, "final_justification": "I thank the authors for an exemplary rebuttal that was both thorough and directly responsive to my questions. They have convincingly resolved all of my initial reservations, significantly strengthening my confidence in this work's contribution.\n\nThe key issues that have now been addressed are:\n\n1.  **Limits of Generalization:** My concerns about the diminishing performance gains (\"weak-to-strong\" generalization) have been fully addressed. The authors confirmed the out-of-distribution (OOD) challenge and importantly clarified the practical, intended use case (guiding with a similarly-sized value model), which resolves the apparent scaling limitation.\n2.  **Optimal Block Size:** The authors provided a compelling explanation for why a large block size is optimal, combining the need for a clear signal-to-noise ratio with the importance of coherent reasoning. Their commitment to add the suggested qualitative example to the appendix will make this core design choice much clearer to readers.\n3.  **Adaptation to Nuanced Domains:** The thoughtful discussion on adapting the VGS framework to more nuanced domains (e.g., using distributional RL) has effectively addressed my questions about the method's generalizability and clarifies the future trajectory of this research.\n4.  **The Power of WMV Aggregation:** The explanation for why Weighted Majority Vote (WMV) outperforms Best-of-N—and the commitment to add a new ablation to support this claim—provides a critical and novel insight into the practical application of value-based search methods.\n\nIn summary, the authors' comprehensive and actionable responses have turned a solid paper into an even stronger one. The promised additions will substantially improve the final version. Therefore, I have raised my score from 4 (Borderline Accept) to **5 (Accept)** and now strongly advocate for this paper's acceptance.", "rebuttal": "Dear Reviewer uxSh,\n\nThank you for your exceptionally thorough and insightful review. We are grateful for your detailed positive feedback on our work's significance, rigor, and clarity. We really appreciate the opportunity to respond to your questions and hopefully solidify your view of VGS as a top-tier contribution.\n\n# Responses to Questions:\n1. On the Limits of Weak-to-Strong Generalization: Your hypothesis is correct. The shrinking performance gap is likely an out-of-distribution (OOD) issue, as the 14B generator produces reasoning paths that are qualitatively different from the 1.5B rollouts our value model was trained on.\n\n    While retraining the value model on rollouts from a stronger generator is the ideal experiment, it is unfortunately too computationally expensive for us to perform during the rebuttal period. We also want to clarify that demonstrating weak-to-strong generalization, while an interesting result, is not the main intended use case of VGS. When applying VGS to guide frontier models, we envision training a value function of a similar size to the generator policy itself. Since the value model is only queried once per block, the computational overhead of this approach is negligible, as we show in Appendix H.\n\n2. On the Critical Role of Block Size (4096 tokens): We agree this is a fascinating result, and we believe the optimality of a large block size is driven by a combination of two of your hypotheses:\n    (a) Signal-to-Noise: The value model requires sufficient context to make an accurate prediction. A short block may not contain enough information to distinguish a promising path from a dead end, leading to noisy and unreliable guidance.\n    (b) Task-Specific Coherence: Competition math problems often require long, uninterrupted lines of reasoning. We found that interrupting the generator too frequently with smaller blocks is disruptive to this coherent thought process.\n    We don't think the amortized cost hypothesis (c) is a reason because the performance also degrades for blocks larger than 4096, which would not be explained by efficiency alone. We will add a qualitative example to the appendix, as you suggested, to illustrate this trade-off and expand this discussion in Section 4.1.\n\n3. On Adapting VGS to Nuanced Domains: This is a critical question. The notion of a value function is very general and can be applied to any type of reward signal. For domains with more nuanced rewards, we would adapt the value model's training as follows:\n    * For continuous rewards (e.g., the fraction of unit tests passed), we would suggest moving beyond simple regression and using a distributional RL (Bellemare et al. 2017) approach to model the entire distribution of possible rewards. A distributional approach to value-based RL has robustly shown improved downstream decision making and is supported by a large body of prior works (Imani et al. 2024, Farebrother et al. 2024, Wang et al. 2025).\n    * For multi-faceted discrete rewards, we would use a more complex set of outcome classes, as you suggest.\n    In general, we believe that VGS would be broadly helpful for tasks that require planning (in the RL sense), where a sequence of correct decisions is needed to reach a final goal. This includes domains like math reasoning, as explored in our paper, but could also extend to applications like creative writing, where one might guide a story's progression toward a specific ending. The primary challenge across these domains is defining a reasonable outcome-based verifier to generate the reward signal for training the value model. This is precisely why we focused our study on competition math, where the verifier is well-defined and can be reliably implemented with tools like `sympy`.\n\n4. On the Power of Final Aggregation via WMV: This is a key insight. Our explanation for why Weighted Majority Vote (WMV) is more effective than Best-of-N (BoN) is twofold:\n    * First, WMV is more scalable against the value model being \"hacked\" at large inference budgets. With many samples, BoN is more likely to select an outlier—a flawed path that receives an erroneously high score. WMV, by relying on consensus, is more robust to such outliers.\n    * Second, WMV provides an implicit filter for unfinished responses. Incomplete generations will not produce the same final answer and thus will not be part of a majority cluster. BoN does not have this filter, and our value model sometimes assigns an unduly high score to an incomplete trace.\n    As a concrete action, we will add results to the final version showing the performance of BoN after manually filtering for unfinished responses to further support this analysis.\n\nThank you again for your detailed and expert feedback. We believe that incorporating these analyses and discussions will significantly strengthen our paper. We are grateful for your review.\n\n--------------------\nCitations\n\n[1] Bellemare et al. 2017: \"A distributional perspective on reinforcement learning\", ICML 2017.\n\n[2] Imani et al. 2024: \"Investigating the histogram loss in regression\", arXiv 2024\n\n[3] Farebrother et al. 2024: \"Stop regressing: Training value functions via classification for scalable deep rl\", ICML 2024\n\n[4] Wang et al. 2025: \"The central role of the loss function in reinforcement learning\". Statistical Science, 2025"}, {"review_id": "ishR9iAmRx", "reviewer": "Reviewer_TuTX", "summary": "This paper proposes a recipe for value model training; the value model is token-level and does not require defining what a step is. Using the proposed value-model-guided block-wise search, the algorithm achieves good performance compared to a few baselines on math benchmarks. Baselines include best-of-N, weighted majority voting (selecting a response from the highest-weight partition), and search algorithms that use the value model. \n\n-\tValue model is trained using a standard cross-entropy loss for classification. Input: prompt, partial response (i.e., prefix), completion starting from the prefix; reference output: label of the full response. \n-\tTo create the dataset OpenR1-VM (2.5M reasoning responses from DeepSeek models), the authors rely on problems from OpenR1-Math. The steps involve pre-filtering, response generation (from a collection of models of different sizes; generation prefixes are determined by sampling four random prefix locations – see line 125), and post-filtering (throwing out prompts with zero accuracy from rollouts).", "questions": "See above", "limitations": "Discussed but could expand", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The method makes sense intuitively. The writing is exceptionally clear: research questions are clear in Section 3.2 for example. Figure 2 is helpful. \n\nIt’s great that the experiments are done on different base models (Qwen, Mistral) because recent work has shown that different models might have different training/fine-tuning behaviors.\n\nI appreciate that the authors included standard deviation across three seeds in reporting the performance for math benchmarks.\n\n\n\n---\n\n\n\n\nWhy does high beam width not work well? Is it due to similar issue as exposure bias in machine translation literature? \n\nWhat about greedy search (width 1)? \n\nIn general I’m curious what tasks/applications would this approach be good at, what tasks/applications would this approach would this approach not make a difference or have harmful effects.", "was_revised": true, "final_justification": "Tasks are limited; discussion of limitation is limited. The authors can include more baselines (e.g., other approaches based on PRMs or token-level rewards) but in general the ideas in the paper make sense intuitively.", "rebuttal": "Dear Reviewer TuTX,\n\nThank you for the positive review and your thoughtful feedback! We are happy to clarify the points you raised and appreciate your suggestions for improving the paper.\n\n1. On Beam Width: This is a great question. Our ablations in Figure 7 show that a beam width of 2 is consistently optimal for our method across many inference budgets. We also note that prior works with non-reasoning models [6, 32, 22] also found a low beam width of four to be optimal. We hypothesize the success of lower beam width may be due to two factors. First, it is likely easier for the value model to perform pairwise comparisons than to rank a larger set of candidate blocks. Second, given the high pass@k of DeepSeek models, it is possible that generating more diverse responses is particularly helpful for arriving the right response. \n\n2. On Greedy Search (Width 1): We did not explicitly run a width=1 experiment given the trend that we saw in our block size ablation (Figure 6). We see a clear trend that a 4k block size was optimal across many inference budgets for our value model. \n\n2. On the Generalizability of VGS: We believe that VGS would be broadly helpful for tasks that require planning (in the RL sense), where a sequence of correct decisions is needed to reach a final goal. This includes domains like math reasoning, as explored in our paper, but could also extend to applications such as creative writing, where one might guide a story's progression toward a specific ending. The primary challenge across these domains is defining a reasonable outcome-based verifier to generate the reward signal for training the value model. This is precisely why we focused our study on competition math, where the verifier is well-defined and can be reliably implemented with tools like sympy.\n\nThank you again for the insightful questions and your positive evaluation. We hope our answers are helpful, and we will incorporate these points into the final version of the paper."}, {"review_id": "VSPe3aUIMa", "reviewer": "Reviewer_YPfy", "summary": "The paper proposes a new method to train a value model which does not rely on a notion of \"step\" by training a correctness classifier on data of the form (prompt, roll-in, roll-out) generated from DeepSeek reasoning responses for competition math problems. The value model is also applied to guide test-time search (VGS) in a blockwise manner, i.e., the block with highest value when concatenated is selected according to BFS, beam search, etc. VGS is shown to outperform existing inference compute methods including BoN, majority vote and PRM-guided search.", "questions": "See Weaknesses", "limitations": "Yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\nThe proposed value-guided search appears to be novel, effective and scalable compared to costly PRM-based search methods. VGS improves on both accuracy and FLOP savings for fixed model size and performance budget. The experiments and results are clearly presented and most include estimates on error bars.\n\n**Weaknesses/Questions**\n\n* How can VGS overcome the issue of credit assignment given that there are no explicit steps to score? How strong is the empirical correlation between block-level value scores and eventual correctness?\n* For the competition math problem sets, how can data contamination be controlled for?\n* The dataset generation is directly dependent on the base reasoning model. Have other base models been tried and how might they affect the resulting value model?\n* It would be interesting to see how VGS performs on cross-domain tasks.", "was_revised": true, "final_justification": "The author rebuttal and other reviews have stregthened my positive assessment of the paper’s contributions towards utilizing test time compute for reasoning tasks requiring long CoT.", "rebuttal": "Dear Reviewer YPfy,\n\nThank you for your positive review and for your support of our work! We are happy to provide detailed answers to your questions below, and we hope they will further increase your confidence in your assessment.\n\n### 1. On Credit Assignment\nThanks for this great question. You are correct that there are no explicit steps to score since we train a token-level value model, which operates at the same granularity as the autoregressive LM. To see the benefit of selecting intermediate blocks with our value model, we performed the random search ablation presented in Figure 8. The results show that selecting blocks with a high value score leads to significantly improved final performance compared to random selection. This provides strong empirical evidence for a positive correlation between higher intermediate value scores and more promising reasoning blocks. \n\nMoreover, we remark that despite having sparse rewards, our value model estimates the expected return at any partial response, and thus can still provide dense signals about the likelihood of this state leading to high reward. In other words, even when rewards are sparse, there are partial responses and blocks associated with non-zero rewards. We note that our approach is philosophically aligned with Sutton's \"bitter lesson,\" favoring general methods that scale with data and compute over those relying on human-defined structures such as a \"step.\" Indeed, our VGS method using a 1.5B value model, empirically outperforms a 7B PRM baseline trained with step-level feedback (Table 1, VGS w/ DeepSeek-VM-1.5B has 45.7% accuracy vs. VGS w/ Qwen2.5-Math-PRM-7B has 40.2% accuracy). Ultimately, we hypothesize it may be useful to learn an appropriate notion of \"step\" beyond the existing heuristics (i.e. separating by newline), but we leave that as promising future work.\n\n### 2. On Controlling for Data Contamination \nThanks for this question. To ensure there is no data contamination, we will focus our evaluations on only AIME-25 and HMMT-25, which are math competitions that happened after the release dates of DeepSeek and OpenR1 (our dataset is a subset of OpenR1). This certainly controls for data contamination because we only evaluate on problems after the release dates of both the underlying model and the dataset. \n\nFor example, we will update the main comparison in Table 1 to the following:\n\n**Test-time scaling DeepSeek-1.5B ($N=256$)**\n| Model | AIME-25 | HMMT-25 | AVG |\n| :--- | :---: | :---: | :---: |\n| VGS w/ DeepSeek-VM-1.5B (ours) | 46.7 ± 0.7 | 32.8 ± 0.8 | **39.8 ± 0.5** |\n| WMV w/ DeepSeek-VM-1.5B (ours) | 45.1 ± 2.2 | 28.9 ± 2.6 | 37.0 ± 1.7 |\n| VGS w/ DeepSeek-BT-1.5B (ours) | 40.6 ± 0.8 | 27.5 ± 0.0 | 34.1 ± 0.4 |\n| WMV w/ DeepSeek-BT-1.5B (ours) | 40.5 ± 2.9 | 24.6 ± 4.7 | 32.6 ± 2.8 |\n| VGS w/ Qwen2.5-Math-PRM-7B | 38.9 ± 1.4 | 24.2 ± 0.2 | 31.6 ± 0.7 |\n| WMV w/ Qwen2.5-Math-PRM-7B | 39.1 ± 2.1 | 24.0 ± 3.2 | 31.6 ± 1.9 |\n| VGS w/ MathShepherd-PRM-7B | 41.9 ± 1.4 | 23.9 ± 1.4 | 32.9 ± 1.0 |\n| WMV w/ MathShepherd-PRM-7B | 40.0 ± 2.5 | 25.6 ± 3.1 | 32.8 ± 2.0 |\n| MV@256 | 38.9 ± 1.9 | 24.3 ± 2.9 | 31.6 ± 1.7 |\n\n**Test-time scaling larger models with our DeepSeek-VM-1.5B:**\n| Model | AIME-25 | HMMT-25 | AVG |\n| :--- | :---: | :---: | :---: |\n| VGS w/ DeepSeek-7B ($N=128$) | 59.4 ± 0.8 | 41.1 ± 1.6 | **50.3 ± 0.9** |\n| MV w/ DeepSeek-7B ($N=128$) | 56.5 ± 1.6 | 33.8 ± 2.5 | 45.2 ± 1.5 |\n\n**Pass@1 Baselines**\n| Model | AIME-25 | HMMT-25 | AVG |\n| :--- | :---: | :---: | :---: |\n| DeepSeek-1.5B Pass@1 | 22.4 ± 4.1 | 13.0 ± 3.9 | 17.7 ± 2.8 |\n| DeepSeek-32B Pass@1 | 60.4 ± 6.0 | 42.1 ± 5.2 | 51.3 ± 4.0 |\n| Deepseek-R1 (671B) Pass@1| 70.0 ± 0.9 | 46.7 ± 2.4 | 58.4 ± 1.3 |\n\nFor our camera ready, we will also update our main plots to evaluate only on AIME-25 and HMMT-25, and we will add a discussion regarding how this ensures there is no data contamination.\n\n### 3. On the Use of Other Base Models\nWe chose to focus on the DeepSeek series because they were the only open-weights reasoning models -- LLMs trained with RL to perform CoT reasoning -- at the time of submission. This focus also provided the benefit of a controlled study across different model scales (1.5B, 7B, and 14B) from the same architectural family, allowing us to investigate properties like scaling up generator sizes (Section 3.3) in a controlled manner.\n\n### 4. On Performance in Cross-Domain Tasks\nWe believe that VGS would be broadly helpful for tasks that require planning (in the RL sense), where a sequence of correct decisions is needed to reach a final goal. This includes domains like math reasoning, as explored in our paper, but could also extend to applications like creative writing, where one might guide a story's progression toward a specific ending. The primary challenge across these domains is defining a reasonable outcome-based verifier to generate the reward signal for training the value model. This is precisely why we focused our study on competition math, where the verifier is well-defined and can be reliably implemented with tools like sympy.\n\nThank you again for your valuable questions and positive assessment. Please let us know if you have any additional questions."}]

Author rebuttal:

Meta-review: (a) Summarize the scientific claims and findings of the paper based on your own reading and characterizations from the reviewers.

This paper introduces Value-Guided Search (VGS), a method for improving the efficiency and accuracy of long-context, chain-of-thought (CoT) reasoning in language models. The central contribution is a scalable pipeline for training a token-level value model that predicts the outcome of a reasoning trace without relying on a pre-defined, fine-grained notion of a "step," which has been a limitation of prior Process Reward Models (PRMs). The authors collected a large-scale dataset of 2.5 million reasoning traces to train a 1.5B parameter value model. At inference time, this model guides a block-wise beam search, and a final weighted majority vote (WMV) is used for aggregation.

The key findings are that VGS significantly outperforms standard test-time compute methods like majority voting and best-of-N sampling on challenging competition math benchmarks (AIME and HMMT). The method is shown to be more computationally efficient, reducing the FLOPs required to reach a target performance level. Furthermore, the authors demonstrate a "weak-to-strong" generalization capability, where their 1.5B value model successfully improves the performance of a larger 7B generator model. The authors commit to open-sourcing their dataset, model, and codebase.

(b) What are the strengths of the paper?

The paper's strengths were highlighted by a majority of the reviewers:

Significance and Timeliness: It addresses the critical and timely problem of improving the computational efficiency of long-context reasoning in LLMs.

Novel Methodology: The "step-free" approach to training the value model is an elegant and practical innovation that sidesteps a key difficulty in creating process-level supervision for complex reasoning tasks.

Strong Empirical Results: The method achieves substantial performance gains on difficult and recent benchmarks. The evaluation is rigorous, including comprehensive ablations on key design choices (e.g., block size, beam width) and comparisons to strong baselines, including existing PRMs.

Clarity and Reproducibility: The paper is well-written and easy to follow. The commitment to open-sourcing the large-scale dataset, the trained value model, and the implementation is a significant contribution to the community.

(c) What are the weaknesses of the paper? What might be missing in the submission?

The reviewers identified a few weaknesses, which were discussed at length during the rebuttal period:

Generalization of the Value Model: The value model is trained on data from a specific model family (DeepSeek) and shows diminishing returns when guiding a much larger, out-of-distribution generator (DeepSeek-14B). This suggests that the value model may need to be retrained or scaled up as generator models advance, which introduces a maintenance and compute burden.

Domain Specificity: The evaluation is confined to the domain of competition mathematics, which benefits from a clear, automatically verifiable correctness signal. How the data collection and training pipeline would adapt to more open-ended domains with nuanced rewards (e.g., summarization, creative writing) is not fully explored.

High Training Cost: While the authors emphasize the scalability of data collection, the one-time training cost for the value model is substantial (requiring 96 H100 GPUs for 24 hours), which could be a barrier for academic labs. However, this is mitigated by the open-sourcing of the final model.

(d) Provide the most important reasons for your decision to accept/reject.

The decision to accept is based on the paper's novel and practical solution to an important problem, backed by strong and thorough empirical evidence. The "step-free" value modeling approach is a significant conceptual contribution that advances the state of the art beyond existing PRMs. The demonstrated improvements in both performance and computational efficiency are substantial. While the weaknesses regarding generalization and domain specificity are valid, they represent limitations and avenues for future work rather than fatal flaws. The authors' commitment to releasing all artifacts further strengthens the paper's value and potential impact on the field. The overall positive assessment from four out of five reviewers, with one becoming a strong advocate after the rebuttal, confirms the paper's quality and readiness for publication.

(e) Summarize the discussion and changes during the rebuttal period. What were the points raised by the reviewers? How were each of these points addressed by the authors? How did you weigh in each point in your final decision?

The discussion period was highly productive and significantly clarified the paper's contributions and limitations.

Reviewer oEdM raised the primary dissenting points regarding the high training cost and the limited generalization of the value model to OOD generators. The authors clarified that their claim of "scalability" referred to data collection (avoiding expensive step-wise annotation) and acknowledged the OOD issue as a known trade-off. They argued that in practice, one would train a value model of a similar size to the generator, making the inference overhead small. While this reviewer was not fully convinced and maintained their score, their points were acknowledged as valid limitations of the approach.

Reviewer uxSh provided an exceptionally detailed and positive review, asking deep questions about the limits of generalization, the reason for the large optimal block size, adaptation to nuanced domains, and the effectiveness of the final WMV aggregation. The authors' rebuttal was exemplary, providing compelling hypotheses and promising additional analyses and qualitative examples in the final version. This exchange turned a solid paper into an even stronger one, leading the reviewer to raise their score and become a strong advocate for acceptance.

Reviewers 4pBh, TuTX, and YPfy raised other important points, including comparisons to related work (token-wise guidance, ToT), the choice of hyperparameters, data contamination, and generalizability. The authors addressed each of these points convincingly. They contextualized VGS within the ToT framework, differentiated it from more costly token-wise methods, proposed a clear strategy to control for data contamination, and discussed the challenges and potential solutions for applying VGS to other domains. These reviewers were all satisfied with the responses and maintained their positive scores.

In my final decision, I weighed the strong support from the majority of reviewers against the concerns of one. The authors' rebuttal was thorough and effective in addressing most concerns, particularly those from reviewer uxSh, which added depth to the paper's contributions. The concerns raised by reviewer oEdM are valid limitations but do not undermine the novelty and empirical strength of the work. The consensus among the other four reviewers, combined with the quality of the paper and the rebuttal, strongly supports acceptance, which is my recommendation.
Decision: Accept (poster)