Title: Reasoning Is Not a Race: When Stopping Early Beats Going Deeper
Venue: NeurIPS 2025
Accepted: yes
Keywords: Long CoT Model

LLM Reasoning

Process Reward Model

Test time Scaling
Abstract: We study the use of Process Reward Models (PRMs) for guiding Long Chain-of-Thought (CoT) reasoning in large language models. Although PRMs deliver fine-grained feedback in standard tasks, PRM-guided beam search does not consistently outperform PRM-free approaches in long CoT reasoning. We trace this shortfall to a "step quality degradation''—the expected step quality shows concave behavior, yielding unimodal or monotonically declining trends. To counteract this, we propose Z-Score Guided Early Stopping (ZGES), which halts search at the detected quality peak using local PRM-reward z-scores. Across multiple math benchmarks and model scales, ZGES outperforms both standard PRM-guided beam search and the PRM-free methods. Ablation studies further highlight the advantages and robustness of ZGES’s adaptive stopping mechanism.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nFirst, this study has empirically found that the PRM quality during beam search degrades as step increases, making unimodal curve.\nTheoretical analysis also support the claim by deriving that the second order of the curve becomes minus value if we assume that the PRM’s re-ranking capacity gradually declines during beam search.\n\nSecond, it has proposed a method of early stopping beam search by setting the stopping trigger as the z-normalized PRM reward falling below a threshold. Experiments have demonstrated that the proposed method consistently outperform or competitive as the baseline methods (normal PRM beam search and self-consistency method), insisting the effectiveness of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- This study carefully observe the deficiency of PRM-based beam search and found that the cause of inferior performance was the consistent degradation of PRM quality as step increases. This finding is beneficial for the community.\n- The proposed method has shown to be effective across different size of models and multiple benchmarks, outperforming or competitive as major baselines.\n\nWeaknesses:\n- The theoretical analysis does not always support the claim that there exists unimodal peak. For instance, if the degradation happens at the begging step, the curve consistently decreases as step increases and there is no peak according to the theoretical consequence that second-order difference of the expected step quality is minus.\n- The proposed method is heuristic and not exciting, just early stopping the beam search.\n- Step 3 of the proposed method \"Compute the local z-score using the sequence of {x1, x2, ..., xt}\": In the early phase of the step, the size of the sequence is so small, so the normalized z-score becomes unstable.\n- Ideally, the baseline method should include manually annotated PRM approach to indicate an upper-bound of PRM.\n- I think there are some readability and clarity issues. Please see \"Questions\".\n\nQUESTIONS:\nI would suggest that the authors update manuscripts to increase readability and clarity.\n- Figure1: This overview lacks lots of important information. First, there is no separate explanation for each of (a) and (b) in the caption. Second, The figure itself does not illustrate what to do after the early stopping (so readers cannot grasp the overall concept of the proposal at first glance). Third, there is no explanation in the caption/figure about what is \"locally normalized z-score\" and why it degrades as the steps increases. The explanation should not be detailed, but it should be simple but to the point.\n- \"Step quality\" is an important concept in this paper, but the detailed calculation approach was moved to Appendix B.2. It is better to describe it in Main page for readability.\n- Step 4 of the proposed method \"stop Beam Search early and directly decode from the candidates\": what kind of decoding strategy is used?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the performance degeneration of Beam Search as the decoding step grows in searching for high-quality long CoT with a Process Reward Model (PRM). The authors define the step quality as the value of the state and estimate it using Monte Carlo simulation. Empirically, they observed that the step quality decreases significantly as the decoding step grows. They also theoretically show that the expected step quality is concave in the discrete sense.  To solve the degradation of step quality, the authors propose to use PRM scores as a proxy (based on a correlation analysis) and early terminate beam search when the z-score below a threshold. The merits of the proposed method are empirically verified.\n\nSTRENGTHS AND WEAKNESSES:\nI think the authors do a very well job in presenting their core ideas. The problem to be solved is important, the analysis are intuitive and the claims are well supported, either empirically or theoretically. \n\nI have only some minor concerns regarding the paper. \n - First, although your method for early stopping is intuitive, it seems to have nothing to do with the theoretical results you have derived in Section 3.3. Can you shed some lights on how these results (especially Proposition 3.8) could be used in developing early stopping/pruning algorithms for BS or more generally, tree-search-based test-time scaling methods? \n - Second, note that the hyperparameter $\\lambda$ can be sensitive to the PRMs. One might need to tune case-by-case for different scenarios.\n - Third, I suggest the authors adding the results of token usages or the number of PRM calls, otherwise line 279-280 is not very well supported.\n\nSome minor presentation issues\n\n - line 23: O1 -> o1\n - line 46: insert a space after word \"concavity\"\n - line 85: it seems that $r_h$ should map to {$0,1$}, not $[0,1]$\n - Figure 5: the negative sign is too thin and almost invisible\n - How $\\lambda$ is set in your main results (Figure 4)?\n\nQUESTIONS:\nSee my comments in Strengths And Weaknesses section\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work provides empirical evidence that Progress Reward Models (PRM) are ineffective for long Chain-of-Thought reasoning tasks and offers theoretical intuition and empirical analysis to provide the following explanation: The PRMs ability to correctly estimate the probability that the LLM finds the correct answer from a given token first increases and then decreases during the search process. An empirical analysis shows that this probability is highly correlated with the PRM score. This insight is leveraged to define an early stopping criterion for the search. The stopping criterion is based on the z-scores of the PRM scores: As soon as it drops below a certain threshold the search is stopped. A comparison with majority@N and standard PRM-guided beam search shows that the method achieves higher accuracy on reasoning tasks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n* The paper provides a theoretical explanation for the empirical phenomenon that the performance of PRM-guided beam search drops in the **long** chain of reasoning regime\n* The idea to use z-scores to define an early-stopping criterion is original as far as I know\n* The work is significant as early-stopping helps to reduce computational costs/increases the performance in LLM-reasoning tasks\n* The paper provides an ablation study for the involved hyper parameters (stopping threshold, beam size). The results show that the method is relatively robust towards these choices. \n\n**Weaknesses:**\n* Clarity: there are some parts of the submission that are unclear to me. I am open to increasing my score if these issues can be clarified and resolved. I listed my questions below. \n* Related work: The related work section does not touch upon other pieces of work on early-stopping (a.k.a length normalization) for beam search such as:\n  > Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation (Wu et al., 2016)\n\n  > When to finish? optimal beam search for neural text generation (modulo beam size). (Huang et al., 2017)\n\n  > Breaking the beam search curse: A study of (re-)scoring methods and stopping criteria for neural machine translation. \n\n**Minor comments:**\n* The legend in Figure 6 covers most of the plot. \n* lines 236-244 contain a proof for a theorem/proposition/statement that is never explicitly stated. It is stated informally \"... we demonstrate the the t-scores of the two variables remain consistent throughout the search process.\", but stating it explicitly would make it easier to refer to it/cite it in future work ect.\n* \"concave in the discrete sense\" $\\rightarrow$ isn't the discrete analogue to concavity simply submodularity? Is there are reason why you don't use this term?\n* For me, it would have been helpful if there was a definition of the \"Majority@N\" strategy included, in particular because there is no reference to a publication that introduces this method. I was not familiar with it.\n\nQUESTIONS:\n* In the experiment in Section 4.3: How did you define the stopping criterion for the baselines?\n* What is $\\tau$ in Assumption 3.1?\n* How is the correctness of a state defined?\n* In Section 3.1, Equation (1): $\\pi$ is mentioned to be the LLM policy. What is the LLM policy? Is it PRM-guided beam search in this context?\n* If I understood it correctly, I would have assumed that the pattern in Figure 2 looks like the pattern in Figure (b). But in Figure 2, the step quality doesn't have this peak. Can you elaborate on this?\n* The theoretical analysis assumes a beam size of 1 (see line 165). This essentially reduces it to greedy search. To me, this seems like quite a strong restriction. Can you elaborate on this?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the effectiveness of Process Reward Models (PRMs) in guiding test-time search for Long Chain-of-Thought (Long CoT) reasoning. The authors find that standard PRM-guided beam search does not consistently outperform PRM-free approaches like Majority@N. Through empirical and theoretical analysis, they observe a phenomenon of step quality degradation during beam search and propose Z-Score Guided Early Stopping (ZGES) as a remedy. ZGES monitors local trends in PRM reward signals and adaptively halts beam expansion when reasoning quality peaks, thereby reducing unnecessary computation and mitigating the diminishing returns of deeper search. The method is evaluated on Long CoT math benchmarks and shows improved accuracy and efficiency compared to conventional beam search methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. The paper targets an important and practical challenge in reasoning with LLMs: improving test-time search in Long CoT settings using PRMs.\n\n2. The authors identify and explain the degradation of step quality during beam search with theoretical justification (concavity of step quality trajectory).\n\n3. ZGES is a simple yet effective strategy that improves performance while reducing compute, and is well supported by experiments.\n\nWeaknesses\n\n1. The findings partially contradict prior conclusions in PRM literature (e.g., Scaling LLM Test-Time Compute Optimally, AlphaZero-style lookahead), which have shown benefits from deeper reward-guided exploration. The paper would be strengthened by applying ZGES to a wider range of PRMs (e.g., value estimation PRMs trained with lookahead) to ensure its generality and reconcile these differing observations.\n\n2. The training process of the PRM used in this study is under-explained. There is little analysis of PRM accuracy, which is critical given that poorly trained PRMs may introduce unreliable reward signals. Related benchmarks like PRMBench could help validate the PRM’s fidelity.\n\n3. All experiments are limited to math datasets. Testing on other domains such as scientific QA (e.g., GPQA) would be valuable to demonstrate the generalization of ZGES beyond math reasoning.\n\n4. Since the Long CoT outputs from models like DeepSeek-R1 are not cleanly step-separated, the process of segmenting steps is crucial for PRM-guided reasoning, yet the paper provides no technical detail or justification for how step boundaries are identified.\n\n5. Minor writing issues exist, e.g., line 87 contains a typo: “reasonoing”.\n\nQUESTIONS:\nSee above weakness\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}