Title: Rollout Roulette: A Probabilistic Inference Approach to Inference-Time Scaling of LLMs using Particle-Based Monte Carlo Methods
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Language Models

Inference Scaling

Particle Filtering
Abstract: Large language models (LLMs) have achieved significant performance gains via scaling up model sizes and/or data. 
However, recent evidence suggests diminishing returns from such approaches, motivating a pivot to scaling test-time compute.
Existing deterministic inference-time scaling methods, usually with reward models, cast the task as a search problem, but suffer from a key limitation: early pruning. Due to inherently imperfect reward models, promising trajectories may be discarded prematurely, leading to suboptimal performance. We propose a novel inference-time scaling approach by adapting particle-based Monte Carlo methods. Our method maintains a diverse set of candidates and robustly balances exploration and exploitation. 
Our empirical evaluation demonstrates that our particle filtering methods have a 4--16x better scaling rate over deterministic search counterparts on both various challenging mathematical and more general reasoning tasks. 
Using our approach, we show that Qwen2.5-Math-1.5B-Instruct surpasses GPT-4o accuracy in only 4 rollouts, while Qwen2.5-Math-7B-Instruct scales to o1 level accuracy in only 32 rollouts.
Our work not only presents an effective method to inference-time scaling, but also connects rich literature in probabilistic inference with inference-time scaling of LLMs to develop more robust algorithms in future work.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a particle filtering algorithm for inference-time scaling to solve early pruning and diminishing-returns issues in search-based LLM inference. It casts test-time compute as posterior inference over a state-space model—using the LLM as a transition model and a process reward model (PRM) as an emission model—and iteratively resamples, mutates, and reweights a diverse set of candidate sequences (particles) to maintain exploration–exploitation balance under uncertainty. The key insight is that preserving trajectory diversity prevents premature pruning and yields robust, efficient scaling without exponential rollout growth.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n1. **Quality / Clarity:** The writing is well structured and the figures clearly illustrate the algorithm (e.g., the particle update pipeline). Method details and advantages are presented in a direct, easy-to-follow manner.  \n2. **Significance:** Early pruning in search-based LLM inference is a critical bottleneck; the proposed particle filtering (PF) algorithm is well matched to this problem, as it preserves candidate diversity and balances exploration–exploitation under reward uncertainty.  \n3. **Originality:** Applying particle‐based Monte Carlo filtering to inference-time scaling in LLMs is novel, providing a principled Bayesian framing and opening a new axis of compute allocation beyond deterministic search.  \n4. **Quality:** Comprehensive experiments on MATH500 and AIME 2024  show 4–16× faster scaling rates than deterministic baselines. Ablation studies on compute allocation, PRM aggregation, and temperature yield useful insights into each module’s contribution.  \n\n### Weaknesses\n\n1. **Significance / Clarity:** Regarding Lines 43–46, there appears to be a trade‐off: early pruning can save compute by discarding unlikely trajectories and reallocating resources to high‐probability regions. Could you clarify under what conditions—such as problem difficulty or reward‐model uncertainty—additional exploration becomes beneficial?  \n2. **Originality / Significance:** Monte Carlo Tree Search (MCTS) also balances exploration and exploitation to avoid premature pruning. The PF algorithm proposed in this paper shares strong operational similarities with Monte Carlo Tree Search (MCTS). The paper does not discuss MCTS: how does PF differ from or relate to MCTS? Including MCTS as a baseline and comparing performance would strengthen the evaluation.\n\nQUESTIONS:\nPlease refer to the “Weaknesses” section above.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider using a particle filter (PF) based approach for the problem of inference-time scaling of language models (LMs). Specifically, the authors use process reward models (PRMs) in the PF, stochastically resampling sequences with higher PRM scores. The authors compare this against other inference-time methods such as best-of-n and beam search, testing empirically across a range of datasets (math, chemistry, finance) and models (LMs and PRMs), showing significantly improved performance with the PF method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nPaper tackles an important problem setting. Ideas presented are reasonable, worth exploring, and could lead to significant advances in state-of-the-art. The theory/math that I checked (not everything) seems correct. \n\nGood consideration of baselines, alternatives, ablations. Good evaluation over a wide range of different models (both generator/base LMs and PRMs). Fairly comprehensive experimental results across several different datasets. Experimental results appear to show very strong performance (though see comments below). \n\nPaper is reasonably structured and not too hard to understand.\n\nWeaknesses:\n\nLine 47 - \"shift-from\" dash should be em dash \n\nLine 97 - x1:T is improperly formatted\n\nLines 140-141: \"Deterministic search can irreversibly discard trajectories that may have low early scores but high posterior mass overall\" - actually this is true of particle filters as well. If you have a limited number of particles (which is always the case in practice), and high enough scores assigned by your PRM, then resampling can concentrate on only those high score sequences and discard low score sequences. And if this happens, that you've resampled and removed some early completions that had low score, you also cannot recover from this, since your propagation only generates subsequent tokens and cannot revise previous tokens.\n\nLines 145-146 - please use em dashes for readability \n\nLine 152 - PG should be defined here instead of later; here is the first usage\n\nWould be nice if you had MCTS as a baseline that you also compared against (this is a minor point).\n\nHaving no confidence intervals in any of the results is a big negative. It's understandable that an academic budget limits the number of experiments you could do, but at least for the final results (I'm not asking for the hyperparameter search/exploration phase), even if only for a few selected runs, you could do at least 3 seeds. This would at least give the reader a rough idea of variability/confidence interval width/statistical significance. With no confidence intervals/statistical significance tests, it's difficult to evaluate the validity and significance of results (having them isn't perfect either, but is much more informative than a single run).\n\nLines 210-213 - I appreciate testing on more benchmarks; that is good. However, I find the reasoning and exposition regarding these testing generalization beyond mathematical reasoning questionable. These are different domains, but they still rely on mathematical reasoning to a large degree. The authors themselves note that the NumGLUE task \"targets numerical reasoning in scientific contexts.\" Similarly, FinanceBench (https://github.com/patronus-ai/financebench/blob/main/data/financebench_open_source.jsonl) has a lot of numerical reasoning based questions. So I would argue that instead of, as the authors claim in lines 229-230 that \"such RMs implicitly learn broader reasoning evaluation capabilities during their training, not limited strictly to mathematical content\", there's a simpler explanation, which is that Qwen2.5-Math-PRM-7B as the reward model works well for these tasks because they're close enough to mathematical reasoning tasks.\n\nLines 260-263 - if using a PRM as an ORM works best, then why is the whole paper focused on PRMs? It seems to me the authors should compare against using ORMs directly in their PF. \n\nLines 301-309 - \"TSMC relies on a ground-truth verifier\"; this is false, TSMC can also be used with ORM/PRMs. \"requires joint training of a generator and value function\" - this is also not really correct, the generator (proposal) can be trained as in Zhao et al. (authors' reference [33]) but does not have to be trained; e.g., you could just use the base LM. The value function also does not necessarily have to be trained, for example, as in Lew et al. (authors' reference [15]). \"The authors of TSMC did not release code\" - this is also false, see for example https://github.com/Silent-Zebra/twisted-smc-lm for the code for [33]; if this comment is for only reference Feng et al. [8], then this should be clarified. Finally, with respect to comparing vs [8], for a fair comparison, you'd have to use the same PRM (I believe [8] uses weaker PRMs).\n\nPF/SMC-related works are the most relevant to this paper, and it's unclear how much originality this paper adds (the core novelty appears to me to stem from the application of PRMs in PFs to the language modelling (specifically mathematical reasoning) setting).\n\nOne other issue with the evaluations is, as the authors acknowledge, using a reward model at intermediate steps can be expensive. Baselines like BoN require only a single call to the PRM/ORM at the end, whereas the PF requires multiple intermediate calls. Therefore, while the authors control for the number of generated tokens/samples, the authors are likely giving the PF significantly more wall-clock time or total FLOPs. Ideally, the authors would be able to demonstrate that either under similar FLOPs or wall-clock time, PF still outperforms baselines.\n\nQUESTIONS:\nFigure 4c - why is the greedy line below the 2^0 point? Does this suggest that greedy is worse than a single random sample? And if so, why does Figure 4d show that greedy is better than a single random sample? Are these quirks of the models? Is the plot wrong? Or does random sampling have high variance (something that confidence intervals would be very useful to get an understanding of)? \n\nLines 246-249 - this is an interesting finding, and somewhat unintuitive. Why do the authors think this is happening? Also, a related question - why do the authors think that the Qwen PRM performance stagnates after just 4 samples? \n\nOverall, I'm on the fence about this work. There's a lot that I like, but there's also a lot that I don't like. I think the results could be quite significant for the broader community, but I have doubts about the results. I'm choosing borderline accept instead of borderline reject because I think the strength of the positives outweighs the negatives. If the authors address all my points convincingly I'd consider increasing to accept.\n\nLIMITATIONS:\nAuthors do discuss limitations, which is good, but only briefly; there is much more the authors could discuss (e.g., my points above). \n\nAuthors claim the work has no potential negative social impacts, which I don't think is fair. The critical assumption made here is that better LM performance (particularly on reasoning tasks, and for smaller models) can only lead to positive outcomes. While I would agree that the potential positives outweigh potential negatives, so overall the work would have positive social impact, there is potential for risks such as harm from more capable models in the hands of bad actors.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper formulates the process of PRM-guided LLM inference as a state-space models and applies particle filtering to infer a posterior distribution over optimal trajectory. Trajectories sampled from this posterior are scored with outcome reward model and the best trajectory is retuned. The method achieves higher performance in a range of reasoning tasks while using the same computational budget.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n* The method is theoretically sound and grounded in the principles of probabilistic inference. \n* The empirical performance gains in the experiments are significant and consistent across different settings.\n* Multiple ablations show that the performance remains high across a range of hyperparameter choices.\n\n**Weakness**:\n\n* It is not fully clear to me how the methods would compare in terms of runtime.  For the results in Table 1 and 2, it is mentioned that the budget (i.e. number of generated sequences) was the same for all methods. Does this also hold for the runtime? It is mentioned earlier that the total runtime is comparable to generating N full completions independently. This should be more than the runtime of beam search, if I understand it correctly.\n\n* The conclusion mentions: \n> We hope that the formal connection of inference scaling to probabilistic modeling established in this work will lead to systematic solutions for current limitations of these methods and pave the way for brining advances probabilistic inference algorithms into LLM inference-time scaling in future work. \n\n  Applying techniques from probabilistic inference to LLM inference-time scaling has already been explored in other work: \n\n   > Uncertainty-Guided Optimization on Large Language Model Search Trees (Grosse et al., 2016)\n\n  The settings differ to some extend (and obviously the probabilistic inference techniques also differ), but to claim that this connection is newly established in this work nevertheless is an exaggeration.\n\nQUESTIONS:\n* Which stopping criterion for the searches was used in empirical comparison? I would suspect that this has a strong influence on the performance in reasoning tasks.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nInference time scaling approaches based on deterministic rules suffer from early pruning -- pruning promising sequences due to incorrectly assigning low scores. This paper proposes a particle-based monte carlo method to alleviate this problem. The idea is to keep $N$ candidate incomplete sequences at each step, sample new tokens for every sequence from a LLM, re-weight each new sequence using a PRM, and finally re-sample with replacement from the candidate pool using the corresponding weights. This process is designed to trade-off exploration and exploitation by using PRMs to exploit but using sampling to explore. Experimental results are done using Qwen and Llama family of LLMs with a Qwen-based frozen PRM. The proposed sampling approach shows improvement on several reasoning benchmarks including math, chemistry, and finance compared to several baselines. On MATH500, the approach improves open source Qwen-2.5 Math 7B Instruct to reach the performance of o1-preview. While using other PRMs still give promising results, a strong PRM is necessary to best performance. The approach is also shown to be robust to different sampling temperatures.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n1. The paper is written clearly and easy to follow. The motivation is clear and the approach is explained using a clear language.\n\n2. The approach outperforms several baselines on multiple benchmarks while being simple to implement.\n\n# Weaknesses\n1. **The approach is similar to other particle-based MC methods.** Especially, [18] also introduces a particle-based MC sampling approach for inference time scaling. Based on Eq (4) in their paper, if you define the potential function as $\\phi(x_{<t}, x_t)=exp(\\sum_{i<t} w_i + w_t)$ where $w_i$s are defined as in your paper, you get a very similar rejection sampling approach. Currently, related work lack a detailed comparison and it needs improvement to highlight key differences. If they are closely related, they should also be added as baselines in empirical comparison.\n\n2. **Why do you update $w_t$ multiplicatively?** While you define $r$ as a probability, weights are treated merely as unnormalized logits; you use softmax over them for re-sampling. Multiplicative weights make softmax more complicated and requires at least some ablation studies for justification.\n\n3. **Some details are missing.** Please correct/address the followings:\n- $w_t$ is initialized to 1.0.\n- Why is it proportional to rather than equality in $w_t=w_{t-1} \\hat{r}$?\n- Sampling is without replacement.\n- In line 88, you define the bernoulli distribution by using $x_t$ but the left side has $x_{<t}$.\n- In Eq (2), subscript is missing, in line 111, subscript/superscript is missing from summation.\n\nQUESTIONS:\n1. Please compare to relevant work on MC methods in more detail and add related baselines to empirical comparison.\n\n2. Why weights are updated multiplicatively rather than additively?\n\n3. Please correct or address other issues explained in weaknesses.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}