Title: Value-Guided Search for Efficient Chain-of-Thought Reasoning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Value-Guided Search

Chain-of-Thought Reasoning

Competition Math
Abstract: In this paper, we propose a simple and efficient method for value model training on long-context reasoning traces. Compared to existing process reward models (PRMs), our method does not require a fine-grained notion of ``step,'' which is difficult to define for long-context reasoning models.
By collecting a dataset of 2.5 million reasoning traces, we train a 1.5B token-level value model and apply it to DeepSeek models for improved performance with test-time compute scaling. We find that block-wise value-guided search (\texttt{VGS}) with a final weighted majority vote achieves better test-time scaling than standard methods such as majority voting or best-of-$n$.
Moreover, \texttt{VGS} significantly reduces the inference FLOPs required to achieve the same performance of majority voting. Our dataset, model and codebase are open-sourced at \codeurl.


Peer reviews:
comments: SUMMARY:
The paper proposes a new method to train a value model which does not rely on a notion of "step" by training a correctness classifier on data of the form (prompt, roll-in, roll-out) generated from DeepSeek reasoning responses for competition math problems. The value model is also applied to guide test-time search (VGS) in a blockwise manner, i.e., the block with highest value when concatenated is selected according to BFS, beam search, etc. VGS is shown to outperform existing inference compute methods including BoN, majority vote and PRM-guided search.

STRENGTHS AND WEAKNESSES:
**Strengths**

The proposed value-guided search appears to be novel, effective and scalable compared to costly PRM-based search methods. VGS improves on both accuracy and FLOP savings for fixed model size and performance budget. The experiments and results are clearly presented and most include estimates on error bars.

**Weaknesses/Questions**

* How can VGS overcome the issue of credit assignment given that there are no explicit steps to score? How strong is the empirical correlation between block-level value scores and eventual correctness?
* For the competition math problem sets, how can data contamination be controlled for?
* The dataset generation is directly dependent on the base reasoning model. Have other base models been tried and how might they affect the resulting value model?
* It would be interesting to see how VGS performs on cross-domain tasks.

QUESTIONS:
See Weaknesses

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-06-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a recipe for value model training; the value model is token-level and does not require defining what a step is. Using the proposed value-model-guided block-wise search, the algorithm achieves good performance compared to a few baselines on math benchmarks. Baselines include best-of-N, weighted majority voting (selecting a response from the highest-weight partition), and search algorithms that use the value model. \n\n-\tValue model is trained using a standard cross-entropy loss for classification. Input: prompt, partial response (i.e., prefix), completion starting from the prefix; reference output: label of the full response. \n-\tTo create the dataset OpenR1-VM (2.5M reasoning responses from DeepSeek models), the authors rely on problems from OpenR1-Math. The steps involve pre-filtering, response generation (from a collection of models of different sizes; generation prefixes are determined by sampling four random prefix locations – see line 125), and post-filtering (throwing out prompts with zero accuracy from rollouts).\n\nSTRENGTHS AND WEAKNESSES:\nThe method makes sense intuitively. The writing is exceptionally clear: research questions are clear in Section 3.2 for example. Figure 2 is helpful. \n\nIt’s great that the experiments are done on different base models (Qwen, Mistral) because recent work has shown that different models might have different training/fine-tuning behaviors.\n\nI appreciate that the authors included standard deviation across three seeds in reporting the performance for math benchmarks.\n\n\n\n---\n\n\n\n\nWhy does high beam width not work well? Is it due to similar issue as exposure bias in machine translation literature? \n\nWhat about greedy search (width 1)? \n\nIn general I’m curious what tasks/applications would this approach be good at, what tasks/applications would this approach would this approach not make a difference or have harmful effects.\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nDiscussed but could expand", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Value-Guided Search (VGS), a method to improve the performance and computational efficiency of chain-of-thought (CoT) reasoning in large language models. The core idea is to train a token-level value model that predicts the final outcome (correct, incorrect, or incomplete) of a reasoning trace given a partial generation. This value model is trained on a large-scale dataset of 2.5 million reasoning traces, collected via a novel and scalable pipeline that does not require pre-defined \"steps\" or expensive human annotations. At inference time, VGS uses this value model to guide a block-wise beam search, selecting the most promising reasoning paths.\n\nThe authors demonstrate through extensive experiments on challenging competition math benchmarks (AIME and HMMT) that VGS significantly outperforms standard test-time compute (TTC) methods like majority voting and search guided by existing process reward models (PRMs). Notably, they show that a 1.5B value model can effectively guide much larger generator models (up to 14B) and reduce the FLOPs required to achieve a target accuracy. The authors are open-sourcing their dataset, model, and code.\n\nSTRENGTHS AND WEAKNESSES:\nThis is a well-executed paper that makes a significant contribution to the important problem of scaling LLM reasoning.\n\n**Strengths:**\n\n1.  **Significance and Impact:** The paper tackles a critical challenge in the LLM landscape: the immense computational cost of generating long, high-quality reasoning chains. The proposed VGS method offers a practical and effective solution that demonstrably improves the trade-off between performance and compute. The finding that a small value model can effectively guide a much larger generator (\"weak-to-strong\" generalization) is particularly significant, as it suggests a cheaper and more scalable path to improving frontier models.\n\n2.  **Technical Quality and Rigor:** The experimental evaluation is exemplary.\n    *   **Challenging Benchmarks:** The use of recent and difficult benchmarks (AIME/HMMT 2024 & 2025) provides a convincing testbed for their method, moving beyond potentially saturated benchmarks.\n    *   **Thorough Ablations:** The authors conduct a comprehensive set of ablations that validate their key design choices. The analyses of block size (Fig. 6), beam width (Fig. 7), the benefit of DVTS, and the direct comparison between value-guided and random search (Fig. 8) provide strong support for their methodology.\n    *   **Strong Baselines:** The comparison against not just standard TTC methods but also state-of-the-art 7B PRM models strengthens the paper's claims. The fact that VGS outperforms these larger, more complex PRMs highlights the effectiveness of the authors' proposed value model training scheme.\n    *   **Novel Empirical Finding:** The discovery that using Weighted Majority Vote (WMV) for final aggregation is critical to the success of VGS is a subtle but important finding that is novel to this work.\n\n3.  **Originality and Novelty:** The primary contribution is the novel and scalable pipeline for training a token-level value model without relying on a pre-defined notion of a \"step.\" This elegantly sidesteps a major limitation of prior work on Process Reward Models (PRMs), making process-level guidance practical for long-context models like DeepSeek. While the components (beam search, value models) are known, their combination into the block-wise VGS framework, the specific step-free data generation strategy, and the successful application to long-form math reasoning is a novel and impactful contribution.\n\n4.  **Clarity and Presentation:** The paper is exceptionally well-written, clear, and easy to follow. The problem statement is well-motivated, and the proposed method is explained logically. Figure 2 provides an excellent, intuitive summary of the entire data collection and search pipeline. The distinction from and advantages over prior work are clearly articulated.\n\n5.  **Reproducibility:** The commitment to open-sourcing the dataset (OpenR1-VM), the trained 1.5B value model, and the codebase is a major strength and a great service to the research community.\n\n**Weaknesses:**\n\n1.  **Limited Generality of the Value Model:** The value model is trained on roll-outs from a single, fixed policy (DeepSeek-R1-Distill-1.5B). While the weak-to-strong results are promising, Table 1 shows that the performance gap between VGS and standard Majority Voting (MV) narrows as the generator model size increases from 7B to 14B. This suggests the 1.5B value model may be approaching the limits of its ability to effectively guide a much stronger generator. The paper would be strengthened by a deeper discussion of this OOD issue and the expected scaling limits.\n\n2.  **Domain Specificity:** The entire evaluation is conducted on competition math problems. This domain benefits from having an unambiguous, binary correctness signal that can be checked automatically (`math-verify`). It is unclear how the VGS pipeline, particularly the 3-class (correct, incorrect, incomplete) data labeling, would translate to more open-ended domains where rewards are nuanced or require human evaluation (e.g., creative writing, summarization quality, or complex code generation).\n\n3.  **Fixed Search Hyperparameters:** The authors use a fixed block size and beam width for all problems, which they frame as a practical advantage over methods that require per-problem tuning. However, this \"one-size-fits-all\" approach might be suboptimal. It's possible that different problems could benefit from different search strategies (e.g., a wider search for harder problems), and a discussion of this trade-off between practicality and potentially higher performance would be welcome.\n\nQUESTIONS:\nMy overall assessment of the paper is positive. The following questions are intended to probe the boundaries of the method and offer suggestions for strengthening the paper even further. A thoughtful response to these points, particularly with supporting analysis where feasible, would solidify my view of this work as a top-tier contribution.\n\n1.  **Probing the Limits of Weak-to-Strong Generalization:** The \"weak-to-strong\" generalization is one of the most significant claims of this work, but the data suggests it may have limits. In Table 1, the performance improvement of VGS over standard Majority Voting (MV) diminishes as the generator model gets stronger: the accuracy gain is **5.9 points** for the 7B model (56.4 vs 50.5), but drops to just **1.4 points** for the 14B model (61.0 vs 59.6). This suggests the 1.5B value model is becoming a less effective guide for the much more capable 14B model.\n    *   **Question:** What is your primary hypothesis for this shrinking performance gap? Is it that the 14B model generates reasoning paths that are so complex or novel that they are fundamentally out-of-distribution (OOD) for the 1.5B value model, causing the value predictions to become less reliable?\n    *   **Actionable Suggestion:** To make the paper's claims about scalability more robust, could you conduct a small-scale analysis for the rebuttal? For instance, you could analyze cases where the 14B model with VGS fails but with MV succeeds. Does the value model assign an erroneously low score to the correct path in these cases? A more ambitious but highly impactful experiment would be to train a new 1.5B value model on a small set of roll-outs from the 7B model and show if this new value model provides better guidance for the 14B generator.\n    *   **Impact on Review:** A convincing analysis or a positive result from the suggested experiment would strongly reinforce the paper's central claims about the practicality and scalability of your approach, addressing what appears to be its most significant limitation.\n\n2.  **The Critical Role of Block Size:** Your ablation on search block size (Figure 6) shows that a large block of 4096 tokens is consistently optimal. This is a fascinating and perhaps counter-intuitive result, as one might expect that more frequent guidance (i.e., smaller blocks) would be beneficial for correcting errors early.\n    *   **Question:** Why do you believe such a large block size is optimal? I see three competing hypotheses: (a) **Signal-to-Noise:** The value model requires a large context to make an accurate prediction, and its judgments on shorter blocks are too noisy to be useful guides. (b) **Task-Specific Coherence:** The reasoning steps in competition math problems are long and logically self-contained, so interrupting the generator mid-\"thought\" is detrimental. (c) **Amortized Cost:** The overhead of invoking the value model is non-trivial, and using fewer, larger blocks is simply more computationally efficient for a given budget. Could you provide your perspective on these or offer an alternative hypothesis?\n    *   **Actionable Suggestion:** This is a key parameter of your method. To help the reader understand this result, could you provide a qualitative example in the appendix showing a generation from a small-block-size search (e.g., 256) that fails, contrasted with a successful 4096-block-size search on the same problem? A clear discussion of these trade-offs in Section 4.1 would significantly improve the paper.\n\n3.  **From Binary Math Problems to Nuanced Domains:** The success of your data collection pipeline and value model training hinges on the availability of an unambiguous, binary correctness signal from `math-verify`. This is an ideal scenario that is not representative of many other complex reasoning tasks.\n    *   **Question:** How do you envision the VGS framework being adapted to domains with more nuanced or non-binary rewards? Consider code generation, where a program might pass some unit tests but not others. Would you adapt the value model to predict a continuous score (e.g., fraction of tests passed), or would you define a more complex set of discrete outcome classes (e.g., 'passes all tests', 'fails compilation', 'fails runtime tests', 'infinite loop', 'incomplete')? How would the learning objective and the final WMV aggregation handle such a signal?\n    *   **Actionable Suggestion:** While a full experimental evaluation is outside the scope of the rebuttal, a more detailed discussion of these necessary adaptations would greatly strengthen the paper by clarifying the boundaries and future trajectory of this line of research. I suggest adding a dedicated paragraph on this topic in your \"Limitations\" or \"Conclusion\" section. A thoughtful response here would demonstrate a deeper consideration of the method's generalizability.\n\n4.  **The Power of Final Aggregation via WMV:** The finding that using Weighted Majority Vote (WMV) at the end of the search is crucial, and significantly outperforms Best-of-N (BoN) (Figure 3), is a very important and subtle result. It implies that even after guiding the generation block-by-block, the final value scores across the surviving beams still contain useful, independent information. In a perfect world with a perfect value model and search, the single highest-scoring path should be the best.\n    *   **Question:** What is your explanation for why WMV is so much more effective? Does this suggest that the value model's scores are inherently noisy, and that a single high score on one beam might be an outlier, making aggregation across multiple diverse beams a more robust ensembling strategy?\n    *   **Actionable Suggestion:** This is a key insight into the practical use of process-level scorers. Could you provide a brief analysis to support your hypothesis? For example, you could measure the variance of value scores across the final beams, comparing cases where BoN fails and WMV succeeds. A high variance in these cases would support the \"noisy value model\" hypothesis. Highlighting this finding and its implications more explicitly in the main text would be a valuable addition.\n\nLIMITATIONS:\nThe authors have a dedicated limitations section that appropriately identifies the main limitation: the value model is trained on roll-outs from a specific policy and will need to be retrained to maintain optimal performance as generator models advance.\n\nTwo areas for improvement would be:\n1.  **Elaborating on \"Verifiable Domains\":** The authors state their pipeline can be adapted to \"similar verifiable domains.\" It would be helpful to expand on what constitutes a \"verifiable domain\" beyond binary pass/fail and discuss the challenges for tasks with more nuanced, multi-faceted, or human-in-the-loop reward structures.\n2.  **Potential Negative Societal Impact:** While the risk for math problem solving is low, the paper contributes to the broader area of improving automated reasoning. A brief mention of the potential dual-use nature of such technologies (e.g., generating highly convincing but subtly flawed arguments at scale) would be appropriate for a paper at this level.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nBased on a newly collected (and also released) large-scale dataset with reasoning paths a value model is learned for guiding DeepSeek LLMs on reasoning tasks. The value model is trained on partial responses and completions based on a more cost-efficient, distilled  reference LLM and can be combined with multiple blockwise, chain-of-thought reasoning strategies such as best-first search or beam search. The resulting value-model guided search achieves good performance and efficiency on standard math benchmarks like AIME or HMMT.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n* significant performance and efficiency gains on standard reasoning benchmarks\n* robustness towards hyperparameter choice across a  wide range of computational budgets\n* methods scales well to long contexts\n* release of large-scale dataset with reasoning-responses that might also be of use for future work\n\n**Weaknesses**:\n* The empirical evaluation exclusively focuses on LLMs from the DeepSeek family\n* Some parts of the paper are not fully clear to me. I list my questions below.\n\n**Minor comment**:\n* I think it would further enhance the readability of Table 1, if the best and 2nd best performance on each of the benchmarks would also be highlighted, and not just the average performance.\n\nQUESTIONS:\n* Regarding the Experiment in Section 4.2: If I understand it correctly, the blocks are selected uniformly at random. Selecting the blocks according to the likelihood under the LLM seems like a stronger -- and equally expensive and simple -- baseline to me.  Do I misunderstand the experiment or is there another reason for why you didn't use this ablation?\n* How does the method conceptually compare to tokenwise reward-guided text generation?\n\n  > **A Critical Look At Tokenwise Reward-Guided Text Generation**, Rashid et al., 2025, https://arxiv.org/pdf/2406.07780\n\n   The training on partial sequences as describe in Section 2.1 strongly reminds me of the method in the above reference, but the related work section does not discuss it. \n\n* I assume tree-of-thought based reasoning is in an entirely different league of computational budget than chain-of-thought based methods. Is that why you did not empirically compare the value model with tree-of-thought based approaches or are there other reasons?\n* Can the value model also be applied to tree-of-thought based reasoning?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 2, "DATE": "2025-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Value-Guided Search (VGS), a method to improve the efficiency and effectiveness of long-context chain-of-thought (CoT) reasoning in language models. The central idea is to train a token-level value model that predicts the outcome of a reasoning trace without requiring a fine-grained definition of “step.” This value model is then used to guide block-wise search (e.g., beam search with blocks of 4096 tokens), enabling more effective inference-time compute scaling than standard test-time compute (TTC) methods such as majority voting or best-of-N. The authors build a large dataset of 2.5M math reasoning traces from DeepSeek models and demonstrate strong results on four high-school math benchmarks (AIME & HMMT, 2024/2025), showing that their method is competitive with or better than step-level PRM-based baselines. All code, models, and data are promised to be open-sourced.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n* Clear and focused motivation: The paper is framed around the growing cost and inefficiency of long-context reasoning in LLMs, a timely and important topic.\n\n* Strong empirical results: The proposed method consistently outperforms standard TTC baselines (e.g., MV, WMV, BoN) and state-of-the-art PRMs across multiple competition math datasets, even when guiding larger generators like DeepSeek-14B.\n\n* Ablation and sensitivity studies are comprehensive: The authors evaluate block size, beam width, DVTS parallelism, and even hybrid/random guidance settings, increasing the paper’s technical completeness.\n\nWeaknesses:\n\n* High training cost and limited generalization of value model: (1) Although the authors claim their training pipeline is scalable, their value model (trained on DeepSeek-1.5B rollouts) requires 96×H100 GPUs running for 24 hours, not including tuning iterations. This is non-trivial for most researchers. (2) The trained value model shows signs of out-of-distribution (OOD) degradation when guiding larger generators (e.g., DeepSeek-14B), which indicates that a separate value model may be needed for each generator model. This introduces a significant maintenance and compute burden, limiting real-world applicability despite the authors' claim (Line 217–218) that this is not a “practical concern.”\n\n* Unclear advantage over step-level supervision: (1) A key motivation of the paper is that step-level supervision is difficult to define and expensive to collect. However, the authors do not quantitatively compare the cost or quality of their token-level labeling approach to step-level PRMs. (2) In practice, step definitions via rule-based chunking (e.g., newline delimiters) have been shown to work well in PRM training. Therefore, it remains unclear whether the additional complexity of collecting roll-in/roll-out pairs and full-trace supervision is actually more cost-effective than existing PRM alternatives.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 4, "DATE": "2025-07-09"}