Rollout Roulette: A Probabilistic Inference Approach to Inference-Time Scaling of LLMs using Particle-Based Monte Carlo Methods
Authors: Isha Puri
Shivchander Sudalairaj
Guangxuan Xu
Abhishek Bhandwaldar
Kai Xu
Akash Srivastava
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Models
Inference Scaling
Particle Filtering
Primary area: deep_learning
TL;DR: We introduce a Particle Filtering approach to Inference Scaling that is robust against inherently imperfect reward models and performs significantly better than previous methods.

Abstract: Large language models (LLMs) have achieved significant performance gains via scaling up model sizes and/or data. 
However, recent evidence suggests diminishing returns from such approaches, motivating a pivot to scaling test-time compute.
Existing deterministic inference-time scaling methods, usually with reward models, cast the task as a search problem, but suffer from a key limitation: early pruning. Due to inherently imperfect reward models, promising trajectories may be discarded prematurely, leading to suboptimal performance. We propose a novel inference-time scaling approach by adapting particle-based Monte Carlo methods. Our method maintains a diverse set of candidates and robustly balances exploration and exploitation. 
Our empirical evaluation demonstrates that our particle filtering methods have a 4--16x better scaling rate over deterministic search counterparts on both various challenging mathematical and more general reasoning tasks. 
Using our approach, we show that Qwen2.5-Math-1.5B-Instruct surpasses GPT-4o accuracy in only 4 rollouts, while Qwen2.5-Math-7B-Instruct scales to o1 level accuracy in only 32 rollouts.
Our work not only presents an effective method to inference-time scaling, but also connects rich literature in probabilistic inference with inference-time scaling of LLMs to develop more robust algorithms in future work.

Rollout Roulette: A Probabilistic Inference Approach to Inference-Time Scaling of LLMs using Particle-Based Monte Carlo Methods
Isha Puri
Shiv Sudalairaj
GX Xu
Kai Xu²
Akash Srivastava²
'MIT CSAIL
RedHat AI
Abstract
Large language models (LLMs) have achieved significant performance gains via scaling up model sizes and/or data. However, recent evidence suggests diminishing returns from such approaches, motivating pivot to scaling test-time compute. Existing deterministic inference-time scaling methods, usually with reward mod- els. cast the task as search problem, but suffer from key limitation: early pruning. Due to inherently imperfect reward models, promising trajectories may be discarded prematurely, leading to suboptimal performance. We propose novel inference-time scaling approach by adapting particle-based Monte Carlo methods. Our method maintains diverse of candidates and robustly bal- ances exploration and exploitation. Our empirical evaluation demonstrates that particle filtering methods have 16x better scaling rate over deterministic search counterparts on both various challenging mathematical and more general show that 1wren2.--Math-1555-Insse
reasoning tasks. Using approach, surpasses GPT-4o accuracy in only rollouts, while Qwwn2..5-Math-7--nnss scales ol level accuracy in only 32 rollouts. Our work not only presents an effective method to inference-tine scaling, but also connects rich literature in probabilistic inference with inference-time scaling of LLMs to develop more robust algorithms in future work. Code and further information is available at https://prrobbbllee inf inference-eee scanieeeccaiinn... github.io/
Introduction
Large language models (LLMs) continue to improve through larger architectures and massive training corpora [3 (2). In parallel, inference-time scaling allosaallcating more computation at inference time-has emerged complementary approach to improve performance without increasing model size. Recent work has shown that ITS enables smaller models to match or exceed the accuracy of larger
complex reasoning tasks B.. with proprietary systems like OpenAl's ol and o3 explicitly incorporating multi-trial inference for better answers [21].
Beyond answer-level scaling methods like self-consistency and best-of-n sampling 4 popular class of ITS methods formulates inference as search guided by a process reward model (PRM), which scores partial sequences step-by-step. This perspective has led to the adoption of algorithms like beam search 34 and Monte Carlo tree search [TT. These methods refine LLM outputs by prioritizing trajectories that score highly according the PRM. However, they also inherit major limitation from classical search: early pruning. Because the PRM is an imperfect approximation of true correctness, these methods often eliminate promising candidates too carly-based on noisy or miscalibrated partial scores. This failure mode illustrated Figure The PRM assigns a higher initial score Answer which ultimately turns out be incorrect, and lower score to the correct
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

BACKGROUND
Total (Jane) (Mark)
Each pencils
Final pencils
Jane torice many pancis
Score:
Score:
Scone:
Together they pencils. many
Stop Total parts u each part
Step Jane gets part -o Answer: 3 pencils
2
Score: 0.9453
Score: 0.0133
Figure I: A true example of PRM assigning a lower score to the first step of a solution that turns out to correct. deterministic scaling methods, this solution would have been discarded in favor for one that had higher initial PRM score but turned out to be incorrect.
Answer 1. A deterministic method like beam search would discard Answer after the first step, never recovering it- even though it is the correct solution. Such brittleness is inherent to greedy search: a path is pruned, it cannot be revisited.
To address this limitation, instead maintain a distribution over the possible paths during generation to represent the uncertainty we would like to account for due to the imperfection in reward modeling and propagate through sampling. We realize this approach in a probabilistic inference framework for inference- sinrence-timeeeeee scaling, in which we use particle filtering to sample from the posterior distribution over accepted trajectories. Our method maintains a weighted population of candidate sequences that evolves over time, balancing exploitation of high-PRM paths with stochastic exploration of alternatives. Unlike beam search, which targets the mode an approximate distribution, particle filtering samples from the typical set, making it inherently more robust to noise and multi-modality in the reward landscape. nigh-sccorig candidates are favored but not allowed to dominate, ensuring that low- wwwprobbbbiltty (but potentially correct) paths remain in play.
We demonstrate that this simple shift-from deterministic search sampling with uncertainty- produces strong empirical gains. On mathematical and reasoning tasks, our method significantly outperforms existing ITS approaches across multiple model families. For instance, using Qwen2.5- Math-1.55 1.5h Instruct and compute budget of 4, our method surpasses GPT-4o; with a budget of 32, 7B model surpasses ol accuracy.
Our key contributions are as follows.
We propose partiele filtering algorithm for inference-ting -ieeeneee-teeeeeee scaling that mitigates early pruning by maintaining exploration across trajectories with uncertainty. We formulate ITS as posterior inference over a state space model defined by an LLM (transition model) and PRM (emission model), enabling principled application of probabilistic inference tools.
2. We present strong results on mathematical and out-of-dommin reasoning tasks and study Particle Filtering's scaling performance.
3. We ablate compute allocation strategies (parallel vs. iterative), PRM aggregation methods, and generation temperature, proposing a new model-based reward aggregation method that improves stability and performance.
4. We demonstrate that our proposed methods have 4-16x faster scaling speed than previous methods based on search formulation on the MATH500 and AIME 2024 datasets, with small language models the Llama and Qwen families. We show that PF can scale to surpasses GPT-4o accuracy with only a budget of 4 and scale Qwen2.5-Math- Bithh-BB-Insstuee to ol accuracy with a budget 32.
Qwen2.5-Math--L5BB
2 Background
State space models describe sequential systems that evolve stepwise, typically over time [25 They consist sequence of latent states {x+)²= and corresponding observations {or] The evolution of states is governed by transition model p(æ, xc(-1), and the observations
2

---

3 METHOD
governed by the emission model p2)) The joint distribution of states and observations is given by: ((:r: 01:1)
p( p( M( [<ce [e)), where p(æ1) is the prior.
Probabilistic inference in SSMs involves estimating the posterior distribution of the hidden states
given the observations, p(z)r)o))))) [25], which is generally intractable due to the high dimension- ality of the state space and the dependencies in the model. Common approaches approximate the posterior through compling-based methods or variational approaches [19]. Particle filtering (PF) is a sequential Monte Carlo method to approximate the posterior distribution in SSMs (24 7 PF
represents the posterior using set of N weighted particles {x," w! } where x; denotes the 1t
particle at time i, and ( is its associated weight. The algorithm iteratively propagates particles using
(8) the transition model and updates weights based on the emission model: wi:" w/-1P(or 1tz =
3 Method
generate 1st stop score
Resampling
generate a next stop soore
Resampling
generate next stop / /score
Select particle with Highest reward as final answer
Step 1: Total parts [Jane|y [Jane)=10Mariqqea 10Mari/e3
Stop part pencils
Step Mark part = pencils, Jane other Anewer: & Pancils
Pantical
Step 1:
Score: 0.3198
Score:
Total parts 2 (Jane)
(Mark) -
Jane has baice many peneils Mark: Together
Total 1:
We partich hus
Step 2:
part-a33
comp
part -
they have if pencils How mony pencill dove June
each part
Answer: 3
pencils
................................
Step
Score:
has parts
0.9453
0.0133
libs partch jar completed is arown
Final Answer: pencis
Stap 3:
Partic 3
Stop 1: Let =
Step
Jane has parts 2x3-
2x
Mark = part
Final Answer: penciis
Score: 0.5898
Score: 0.03P2
Score: 0. 0.9883
Hi5 partice sompiitna aaomn
Figure 2: Inference-time scaling with particle filtering: initialize n particles, generate step for each, score with the PRM, resample via boftmax-weightnn scores, and repeat until full solutions are formed.
We formulate inference-ting time scaling for a language model pM as approximate posterior inference in a state space model (SSM) defined over token sequences. Let æ1.7 denote the sequence of generated tokens (or chunks, such as steps in math problem), and let 01:7 denote binary observations indicating whether the steps so far are accepted, given prompt
The forward model defines the joint distribution over latent states and observations
T PM (M(E,, O1. c) x II (M |c, II p(oe c, c,t)));
T
(I)
where the transition model (x | (.xe is given by the language model M, and the emission model p(o, c, is a Bernoulli distribution: p(or G, B(o x,), with reward function r(c, 24) encoding the acceptability of the step Zt. Figure iillustaates this SSM.
Given this model, our goal is to infer the posterior distribution over latent trajectories that yield fully accepted sequences, i.e., PM (F1rT c, 01.r = 1). This formulation makes particle filtering a natural and theoretically grounded choice for inference.
In practice, the true reward function r often unavailable. Following prior work, we approximate it using pre-trained preference or reward model (PRM) f suited to the target domain (e.g., math reasoning). This results in an approximate emission model: p(o: c, (.)) = B(o,; ((C,
Substituting this into the forward model, we obtain the approximate joint distribution:
T
T
PM(LLT OuT c) x pm c, (,, 1)
II
p(or c,
(2)
3

---

3.1 Particle Filtering for Estimating the Posterior
3 METHOD
language model
TT per ((ze))
Aarta
partan e,
reward
OT plo c,
model
plol c,21))
p(02 c2
Figure 3: State spate- model for inference-ting scaling. is prompt. 21,.... Iy are LLM outputs, and 01,..., or are "observed" acceptances from a reward model. We estimate the latent states conditioned on or for all
and correspondingly aim estimate the posterior PM T G, = =).
3.1 Particle Filtering for Estimating the Posterior
We now apply particle filtering (PF) to approximate the posterior over accepted sequences æ 1: under the model (2). Each particle represents a partial trajectory, and inference proceeds by iteratively generating, scoring, and resampling these particles based on their reward-induced likelihood. At each time step t, PF maintains a population of N weighted particles. The algorithm proceeds as follows:
Initialization (t 1): Each particle is initialized by sampling the first token from the LLM:
PMr (x1 c). Each particle's initial weight w, initialized to 1.0.
Propagation (t > 1): Each particle is extended by sampling the next token from the LLM conditioned on its history: (1) (M c, c,
Weight Update: Using reward model each particle assigned an unnormalized weight that reflects the likelihood of acceptance: w, x wi x()
Resampling: To focus computation promising trajectories, we sample new popula- tion particles from the current set using a softmax distribution over weights: P,(j =
i) exp( w (6) //E 1 exp(w(") (()), Our resampling is performed with replacement, consistent with standard practicle in particle filtering.
(N) ~ P, and retaining the The next generation of particles is formed by sampling indices j( " corresponding sequences.
This procedure balances exploitation of high-reward partial generations with stochastic exploration, increasing the chances of recovering correct completions even when early steps are uncertain.
The algorithm can be implemented efficiently: both token sampling and reward computation can be parallelized across particles. With prefix caching, the total runtime is comparable to generating N full completions independently.
Final Answer Selection Particle filtering yields a weighted set of samples approximating the poste- rior, enabling principled answer selection strategies. While selecting the seahest-weightdd particle or using (weighted) majority voting better reflects the typical set, we adopt the standard practice of scoring all final outputs with an outcome reward model (ORM) and selecting the top-scoring one for fair comparison with prior work. Additionally, the posterior samples allow richer evaluation- instance, estimating the expected correctness of the model under own distribution, rather than relying solely on point estimates. Notably, samples from Algorithm [ can be used to construct unbiased estimates of any expectation over (2). In the context math and reasoning, it guarantees the expected accuracy is unbiased, which we formulate in Theorem 2 (proof in Appendix C

---

4 EVALUATION
Theorem 1 (Unbiasedness of Expected Accuracy). Let (uu(),z(0)) be weighted particles from and is correct(z) is a function to check the correctness of response x. We have
Algorithm 7
Σ ( is co_orocc((()))
Σ [ (Zver |c, = 1) is c=rrrrcct((())
E
(3)
where the expectation over the randomness of the algorithm itself.
Reward Aggregation with PRMs To compute particle weights during generation, we aggregate per-step scores from the process reward model (PRM) f. We adopt multiplicative update, so that each particle's s weight reflects the cumulative quality of its entire partial trajectory rather than only the most recent step- consistent with standard particle-fitternng practice, where total weight corresponds to cumulative likelihood. Recognizing that multiplicatiie updates may also amplify noise, experiment with alternative aggregation schemes (e.g.. using only the final-step minimum score) and report their trade-offs between stability and expressivity in Appendix A and Section 4.5
Sampling v.s. deterministic search An alternative to our sampling-based approach is to treat inference-time scaling as an optimization problem under the approximate posterior (2), reducing to deterministic search methods like beam search or MCTS. However, these methods assume the reward model f is accurate at every step and prune aggressively based on early scores. In practice, PRMs noisy and their preferences often shift during unrolling. As result, deterministic search can irreversibly discard trajectories that may have low early scores but high posterior mass overall. Once pruned, such paths cannot be recovered.
In contrast, particle filtering maintains a distribution over partial sequences and uses stochastic resampling to balance exploration and exploitation. This allows recovery from early errors and robustness to reward noise. While beam search targets the mode of the approximate pnterior-making it sensitive to local errors-PF samples from the typical set, smoothing over inconsistencies in f. Unlike search heuristics, PF provides consistent, unbiased estimators under mild assumptions, and naturally captures multi-modal solutions critical when multiple valid completions exist. We validate these advantages empirically Section 4.5
Multiple iterations and parallel chains The PF approach to inference-time scaling can be used to define MCMC kernel that enables two new types of scaling: multiple iterations of complete answers inspired by PG (Particle Gibbs) and parallel simulations inspired by parallel tempering. We detail the methodology and results for both extensions in sections B.1 and B.2 in the appendix.
4
Evaluation
We evaluate our proposed methods in this section. We detail our experimental setup in Section 4.1 and start with highlighted results comparing against closed-source models and competitive inference- scaling methods with open-source models (Section 4.2). We then study how the main algorithm, particle filtering, scales with more computation and compare it with competitors (Section 4.4). We further perform an extensive ablation study on key algorithmic choices like reward models, reward aggregation, final answer aggregation, and LLM temperatures (Section 4.5). Finally, we study different allocations of the compute budget through iterative and parallel extensions (Section 4.5)
4.1 Setup
Models We consider two types of open-source small language models (SLMs) as our policy models for generating solutions. The first is general models, for which we evaluate 1.e.n.5--.5B-Insstii (2). 1rren255-7B-Instruee Llama-3.2- 2221BBInstruct and Llama-3. --8--- is math models, using Qwen2.5-Main In5t-Instr and Twwen.5-MMat--7B--
10] The second These small
models are well-suited for inference-time scaling, enabling efficient search of multiple trajectories.
BII, 7B process reward model. We selected this model for its superior performance over other PRMs we tested, including Math-Shepherd-i soahhend-mistral-b----r FurusPRM-SSS Stage2 RD). This result as an ablation study provided in Section 4.5 where we also
Process Reward Models To guide our policy models, we utilized Swen2.5-Math-PPe
(26 Llama3. 1 1-88--PM----ee De--PRRDeeeesee---Dt 28 and
study the different ways to aggregate step-level rewards from PRMs discussed in Section 3.]
5

---

4.2
Results on Mathematical Reasoning Datasets
4 EVALUATION
Baselines
Greedy: single greedy generation from the model, serving as the "bottom-line" performance.
simplest scaling method, majority voting across candidates BoN/WBoN | simple RM-based scaling method, scores outputs with outcome reward models
Self Consistency [27]:
Beam Search [3] structured search procedure that incrementally builds sequences by keeping the
top-k highest-scoring partial completions at each generation step.
DVTS
3
parallel extension of beam search that improves the exploration and performance.
Datasets We evaluate methods and baselines across wide variety of tasks that span difficulty level and domain to test basic and advanced problem-solving and reasoning.
MATH500 [16]: 500 competition-lett problems from various mathematical domains.
AIME 2024 II: 30 high difficulty problems from the American Invitational Mathematics Exami- nation (AIME and II) 2024.
NumGLUE Task 2 (Chemistry) [0 325 questions that test advanced reasoning across real-world
tasks, mostly centered on the chemistry domain.
FinanceBench [12 150 open-book financial question answering tasks grounded in real-world
financial documents and analysis.
Parsing and scoring
Details on parsing and scoring functions across datasets in the Appendix
4.2 Results on Mathematical Reasoning Datasets
LITT SA. IiR
0.6
Acorrity
Accuracy
0.5
Wegitel Buk (Ua 5 2-15 instrct
Waggoss BuN (Lana 3.1 88 Instruct
Oun-Fartt Farst a Thter ng diama-322 3.mm-32-100 intrudt
Durs Partide Tikering (1lema-3.33 3----------------- nn-nuttuct
DVTS aamm-12223 B2-3--
DVTS (Jlam 1-4Bimtuust
0.2
Basat (cresdy)
Hure maau
Baits Cot (0redda)
(u of mode genorations)
( of model generations!
(a) Llama-344 3.2- IB Instruct
(b) Llama-3 3.1-8B SB-Instruc
0.825
0.825
RECUracy
0.800
0.800
Owero instuct
0.755
597
0350
Curs Partide Flaiin (Owe 2. Muth 1.5 bntrod)
Durs Puide fit (Qwen2.5 Nath rstruct
Waigtted 10we12 Muth instrutt DVT'S (0wer22 Mors Sittstro 1102.5 No02 Soree Oshct DeT (Srsedy)
Widiited But rDwan2 Mazs instrutt
DVTS 10we12 5-Math 78 intruct
Danntt Cat (Geeso))
Budget 16 model generations)
Budget (# model generations)
(c) Qwen2.5-Math 1.5--Bnsttttt
(d) T.ven2.5-Math-7B-nnnttee
Figure Accuracy vs. Generation Budget across models using different inference-time strategies.
We present our main results in Table [ comparing Particle Filtering (PF) with a suite of strong inference-time scaling baselines on two challenging mathematical reasoning tasks: MATH500 and AIME 2024. All inference-tine scaling methods are evaluated under a fixed compute budget of 32 generations per instance, using Owen2.5-MMth--PRN. PRMhh as the reward model. Specifically, serves as the PRM for PF, Beam Search, and DVTS, and as the ORM in WBoN.
PF consistently achieves the best performance across all model sizes. Among all inference- time scaling methods, PF delivers the highest accuracy on both benchmarks, often outper- forming alternatives by significant margin.

---

4.3
Results on Generalized Reasoning Datasets
4 EVALUATION
Model
Method
MATTH500
AIME 2024
Closed Source LLMs
GPT-4o
76.2
4/30
ol-preview
87.0
12/30
Claude 3.5 Sonnet
78.2
5/30
Open Source LLMs
Llama-3.1 70B Instruct
65.6
5/30
2.e Math 72B Instruct
82.0
9/30
Open Source General SLMs Qwen-2.5 1,5B Instruct
Greedy
54.4
1/30
Self Consistency
61.0
2/30
BoN
67.8
1/30
WBoN
69.2
2/30
Beam Search
76.2
5/30
DVTS
76.6
4/30
Particle Filtering (urr))
79.3
6/30
Open Source Math SLMs Qwen-2.5 Math 1.5B Instruct
Greedy
70.0
3/30
Self Consistency
79.6
6/30
BoN
81.8
4/30
WBoN
82.6
4/30
Beam Search
83.0
4/30
DVTS
82.8
5/30
Particle Filtering (Ours)
84.6
7/30
Qwen-2.5 Math 7B Instruct
Greedy
79.6
5/30
Self Consistency
84.0
4/30
BoN
82.6
5/30
WBoN
83.0
5/30
Beam Search
86.9
7/30
DVTS
84.6
6/30
Particle Filtering (Ours)
87.7
10/30
Table 1: Results of various LLMs on MATH500 and AIME 2024, highlighting particle filtering performance. All methods used a compute budget of 32 generations with Bwen2.5-Math-PRM- as the reward model. Notably, Qwen2.5-Math-7B. with just 32 particles, matches ol -preview on MATH500, demonstrating PF's effectiveness.
PF unlocks competitive performance even for small models. For instance, Qwen2.5-1.3 Instruct, when scaled using PF, surpasses the much larger GPT-4o on both MATH500 and AIME 2024. This showcases the ability of inference-time compute to significantly improve performance without increasing model size.
PF on Qwen2.5-Me 5-Math-7B (aath-7B-Instiie Instrrctt outperforms ol orevvew on MATH500. Scaling Qwen2.5- Math-7B with PF results in new sate-of-the-art among open models: 87.7% on MATH500 and 10/30 on AIME. This surpasses the ol ol-preview model and highlights the potential of inference-time scaling to close the gap with-on or even exceed- the performance of proprietary frontier L.MS using smaller, open models.
For results additional model families and broader ablations, Appendix H
4.3 Results Generalized Reasoning Datasets
To evaluate whether our inference-time scaling method generalizes beyond mathematical reasoning, we test Particle Filtering on two diverse esstruction-oolowiiii benchmarks: FinanceBench [12] which evaluates financial QA over real-world documents, and NumGLUE Task 2 (Chemistry) 20 which targets numerical reasoning in scientific contexts.
As shown in Table 2 Particle Filtering consistently outperforms all other inference-time scaling baselines, achieving the highest accuracy on both datasets. This shows that our method effective not only for mathematical reasoning but also for tasks that combine numerical reasoning with domain-

---

4.4 Scaling with inference- time compute
4 EVALUATION
specific reading comprehension. We use Llama 1.1-8B-Instruue the policy model and Qwen2.5- Math-PRM-7B as the reward model, with particles for FinanceBench and 32 for NumGLUE.
We note that although we use Swen2.5-Math-PRM-775
NumGLUE Task 2
ward model trained primarily for mathematical process evaluation it performs surprisingly well as a reward model on these domains that require significant quantitative reading comprehension. We leave deeper exploration of this hypothe- sis and the development domain- specific or generalized PRMs for future work.
Method
FinanceBench
(Chemistry)
Greedy
62.67
71.69
BoN
68.00
80.92
Self Consistency
68.67
79.32
Beam Search
67.33
80.47
Particle
70.33
84.22
Filtering (Ours)
Table 2: Results on Non-Math Datasets
4.4 Scaling with inference-time compute
We now zoom in how PF scales with inference-time compute. Figure shows the change of perfor- mance (in terms of accuracy) with an increasing computation budget (N 4, 8, 16, 32, 64, 128) across Math SLMs and Non-Math SLMs. As competitor DVTS, e.g. DVTS requires budget of 32 to reach the same performance of PF with a budget of 8 with Llama-3.2-1B Instrucct and requires budget of 128 to reach the performance of PF with budget of 8 with LLama-3.. ----ruuttt
see, PF scales 16x faster than the next best
4.5 Ablation study
Performance of different PRMs Figure 5a shows an ablation on 100 MATH500 questions, comparing our method's accuracy across reward functions as the number of particles increases.
Accuneyy
Qwen2.5-Math 1-t.5---ath---M-7-- consistently outper- forms other models, making the natural choice for main results. Interestingly, while EurusPRM-Stagee performs poorly with smaller budgets, it improves and eventually matches Swen2.5-Math-PRM.. at higher budgets.
0.3
Owe 2, Mach PRM 78
Lomss 48 FRM Deepoeet Dats
EuruaPi Stuge2
math-hesteee 7hpm
Budget model generations)
(a) Results of ablation comparing the performance of PF across PRMs.
Reward aggregation within PRMs As men- tioned in Section 3.J Appendix A and reported by prior works 32 there are multiple ways use PRMs to calculate reward scores, which can significantly impact final performance. Fig- ure studiiss three existing methods for com- bining PRM scores-using the last reward, the minimum reward, and the product of all rewards, We also study "Model Aggregation,' where the PRM is used as an ORM with partial answers.
6.6
Accuwey
Last
Product
0:2
Budgat (# of modell generations)
(b) Effect of different aggregation strategies for Qwen2.5
As we can see, using Model Aggregation-ing -in essence, feeding into a PRM the entire partial answer alongside the question scales the best with an increasing budget. To clarify, however, since particle filtering requires scoring and resampling based on intermediate generations, PRMs remain the appropriate choice (over ORMs) even when used in aggregation modes resembling full-sequence evaluation.
5.6
Accurary
Tomp=ti
Tompwos Targwos
6.3
Budgat (6 of model generations)
(c) Results of ablation comparing the effect of temper- ature across different particle budget.

---

5 RELATED WORK
Controlling the state cunsition-temperatnn in LLM generation We investigate the effect of different LM sampling temperatures on the scaling of our method across varying numbers particles. The results of our ablation study on a 100-question subset of MATH questions are shown in Figure 5c
Our findings show that the common LLM temperature range of 0,4-1.0 performs well, with minimal accuracy variation across budgets. Following (3), we use temperature 0.8 for all experiments.
Budget allocation over iterations and parallelism The multi-iteration and parallel-chain exten- sions introduced in Section / Appendix [ provide two more axes to spend computation beyond the number of particles. We explore how different budget allocations affect performance in the appendix.
Further Exploration We include confidence intervals and runtime explorations in the B and of the Appendix.
5 Related Work
Process reward models (PRMs) aim to provide more granular feedback by evaluating intermediate steps rather than only final outputs. They are trained via process supervision, training approach where models receive feedback on each intermediate step of their reasoning process rather than only on the final outcome.
[17] propose step-by-step verification approach PRMs, improving the reliability of reinforcement learning. DeepSeek PRM [26] uses Mistral to annotate training data for PRMs. 32 introduces Qwen-PRM, PRMM, which combines both Monte Carlo estimation and model/human annotation approach to prepare training data for PRM. PRIME [6] proposes to train an outcome reward model (ORM) using an implicit reward objective. The paper shows that implicit reward objective directly learns a Q-function that provides rewards for each token, which can be leveraged to create process-level reward signal. This process eliminates the need for any process labels, and reaches competitive performance on PRM benchmarks.
Inference-time scaling a key training-free strategy for enhancing LLM performance. inves- tigates best-of-N (BoN) decoding, showing quality gains via selective refinement. [22] analyzes how scaling compute improves inference efficiency from a compute-optimality view. While not implementing full Monte Carlo Tree Search (MCTS), [64] explores tree-search-inspired approach within language models. Compared to such MCTS--tyle methods, our particle-filee filteee (PF) approach is inherently parallel, more robust to reward noise, and empirically outperforms MCTS under equal compute (PF 16.6% vs. MCTS 6.6% on AIME 2024 with LLaMA3.1 which combines MCTS for data generation and training to improve mathematical reasoning. [ discusses beam search and dynamic variable-time search (DVTS) as inference-tine scaling methods for open-source LLMs. DVTS runs multiple subtrees in parallel avoid all leaves getting stuck in local minima.
[I]] introduces ISTAR,
Particle-based Monte Carlo methods are powerful tools for probabilistic inference. Sequential Monte Carlo 7 or particle filtering 24 has been a classical way to approximate complex posterior distributions over state-space models. Particle Gibbs (PG) sampling [2] extends these approaches by integrating MCMC techniques for improved inference.
[15] and 18 use token-based SMC
within probabilistic programs to steer LLMs, while 9 apply token-based SMC for self-constranned generation. 33 and [ introduce Twisted SMC (TSMC) for probabilistic inference in language
models.
Our work builds on this shared foundation but differs from TSMC and related methods in several important ways. (I) Training-free vs. learned components: TSMC learns twist function-a value function trained with ground-truth supervision approximate expected future reward- -loon with proposal model. In contrast, our method is entirely training- fain:: both the generator and the process reward model (PRM) are fixed, with no additional tuning or supervision.
(2) Robustness to noisy reward models: Whereas TSMC assumes access to accurate value estimates or verifiers, our method explicitly targets the more practical regime where only noisy, imperfect PRMs are available, framing inference-time scaling as posterior inference under noisy supervision. (3) Broader applicability: Because our approach does not rely on verifiers or fine-tuned value functions, generalizes naturally beyond math reasoning other domains (c.g., chemistry, finance).

---

REFERENCES
Empirically, we find that this training-free design remains highly effective: using the same generator (DeepSeekMath7B) and dataset (MATH500) as reported by [8] our method achieves 75.4% accuracy with 128 samples- outperfrnnn TSMC by 14.6 points using fewer than half the samples and no fine-tuning-
Decision making with uncertainty The way of representing uncertainty using softmax is often referred as Boltzmann exploration in the multi-armed bandit (MAB) literature [14). While formulating it as a MAB problem allows it to use a scheduling on the softmax temperature and to derive regret we no longer have the same unbiasedness from the particle filtering / SMC formulation.
bounds 5
Conclusion
In this paper, we introduce a particle filtering algorithm for inference-time scaling. To address the limitations of deterministic inference-time naminn early pruning from imperfect reward models-we we adapt particle-based Monte Carlo methods to maintain diverse population of candidate sequences and balance exploration and exploitation. This probabilistic framing enables more resilient generation and opens principled path for integrating uncertainty into LLM inference. Our evaluation shows these algorithms consistently outperform search-based approaches by a significant margin.
However, inference-ting scaling comes with computational challenges. Hosting and running reward model often introduces high latency, making the process more resource-intensiog For smaller models, prompt engineering is often required to ensure outputs adhere to the desired format. Finally, hyperparameters like budget are problem-dependee
and may require tuning across domains.
We hope that the formal connection of inference scaling to probabilistic modeling established in this work will lead to systematic solutions for current limitations of these methods and pave the way for bringing advanced probabilistic inference algorithms into LLM inference-timg scaling in future work.
References
[1] AI-MO. Aimo validation aime dataset. http::///hggggggff....oottttttee aimo validation aime 2023. Accessed: 2025-01-24.
[2] Christophe Andrieu, Arnaud Doucet, and Roman Holenstein. Particle Markov Chain Monte
Carlo Methods. Journal of the Royal Statistical Society Series B: Statistical Methodology, 72(3):269 342, June 2010.
[3] Edward Beeching, Lewis Tunstall, and Sasha Rush. Scaling test-time compute with open models, 2024.
[4] Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoe Le, Christopher Ré, and Azalia Mirhoseini. Large Language Monkeys: Scaling Inference Compute with Repeated Sampling, July 2024.
[5] Nicolò Bianchi, Claudio Gentile, Gábor Lugosi, and Gergely Neu. Boltzmann exploration done right. Advances in neural information processing systems, 30, 2017.
[6] Ganqu Cui, Lifan Yuan, Zefan Wang. Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, Jiarui Yuan, Huayu Chen, Kaiyan Zhang, Xingtai Lv, Shuo Wang, Yuan Yao, Hao Peng, Yu Cheng, Zhiyuan Liu, Maosong Sun, Bowen Zhou, and Ning Ding. Process reinforcement through implicit rewards, 2025.
[7] Pierre Del Moral. Sequential Monte Carlo Methods for Dynamic Sys- tems: Journal of the American Statistical Association: Vol 93, No 443.
1998.. 10473999,77755, 1997.
[8] Shengyu Feng. Xiang Kong, Shuang Ma, Aonan Zhang, Dong Yin. Chong Wang. Ruoming Pang, and Yiming Yang. Step-by-step reasoning for math problems via twisted sequential monte carlo, 2024.
[9] Gabriel Grand, Joshua B. Tenenbaum, Vikash K. Mansinghka, Alexander K. Lew, and Jacob Andreas. Self-stcering language models, 2025.
10

---

REFERENCES
REFERENCES
[10] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ah- mad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song. Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia- Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang. Gabriel Synnaeve, Gabrielle Lec, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang. Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong. Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang. Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Va- suden Alwala. Karthik Prasad, Kartikeya Upasani, Kate Plawiak. Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kumal Bhalla, Kushal Lakhotia, Lauren Rantala- Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kam- badur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi. Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng. Prajiwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang. Saghar Hos- seini, Sahana Chennabasappa, Samjay Singh, Sean Bell, Seohyun Somia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do Vish Vogeti, Vítor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong. Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia. Xuewei Wang. Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaci, Yi Wen, Yiwen Song. Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan. Zhengxing Chen, Zoe Papakipos, Aaditya Singh. Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang. Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao. Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel. Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison

---

REFERENCES
REFERENCES
Rudolph. Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg. Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang. Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang. Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael I.. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang. Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub. Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li. Rebekkah Hogan, Robin Battey, Rocky Wang. Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng. Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shugiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max. Stephen Chen, Steve Kehoe, Steve Satterfield. Sudarshan Govindaprasad. Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ioneseu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenehen Wang, Wenwen Jiang, Wes Bouaziz, Will Con- stable, Xiaocheng Tang, Xiaojian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary Vito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The llama herd of models, 2024.
[11] Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. rathr-Math: Small LLMs Can Master Math Reasoning with SelfvExolved Deep Thinking, January 2025.
[12] Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, and Bertie Vidgen. Financebench: A new benchmark for financial question answering, 2023.
[13] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.
[14] Volodymyr Kuleshov and Doina Precup. Algorithms for multi-armed bandit problems. arXiv preprint arXiv: aetp:400266000 6028,, 2014.
[15] Alexander K. Lew, Tan Zhi-Xuan. Gabriel Grand, and Vikash K. Mansinghka. Sequential monte carlo steering of large language models using probabilistic programs, 2023.
[16] Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step, 2023.
[17] Hunter Lightman. Vineet Kosaraju. Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's Verify Step by Step, May 2023.
[18] Joāo Loula, Benjamin LeBrun, Li Du, Ben Lipkin, Clemente Pasti, Gabriel Grand, Tianyu Liu, Yahya Emara, Marjorie Freedman, Jason Eisner, Ryan Cotterell, Vikash Mansinghka,
12

---

REFERENCES
REFERENCES
Alexander K. Lew, Tim Vieira, and Timothy J. O' Donnelll. Syntactic and semantic control of large language models via sequential monte carlo. In The Thirteenth International Conference on Learning Representations, 2025.
[19] David JC MacKay. Information theory, inference and learning algorithms. Cambridge university press, 2003.
[20] Swaroop Mishra, Arindam Mitra, Neeraj Varshney, Bhavdeep Sachdeva, Peter Clark, Chitta Baral, and Ashwin Kalyan. Numglue: A suite of fundamental yet challenging mathematical reasoning tasks. ACL, 2022.
[21] OpenAI,; :, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Camey, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, Ananya Kumar, Andre Saraiva, Andrea Vallone, Andrew Duberstein, Andrew Kondrich, Andrey Mishchenko, Andy Applebaum, Angela Jiang, Ashvin Nair, Barret Zoph, Behrooz Ghorbani, Ben Rossen, Benjamin Sokolowsky, Boaz Barak, Bob McGrew, Borys Minaiev, Botao Hao, Bowen Baker, Brandon Houghton, Brandon McKinzie, Brydon Eastman, Camillo Lugaresi, Cary Bassin, Cary Hudson, Chak Ming Li, Charles de Bourcy, Chelsea Voss, Chen Shen, Chong Zhang, Chris Koch, Chris Orsinger, Christopher Hesse, Claudia Fischer, Clive Chan, Dan Roberts, Daniel Kappler, Daniel Levy, Daniel Selsam, David Dohan, David Farhi, David Mely, David Robinson, Dimitris Tsipras, Doug Li, Dragos Oprica, Eben Freeman, Eddie Zhang, Edmund Wong, Elizabeth Proehi, Enoch Cheung. Eric Mitchell, Eric Wallace, Erik Ritter, Evan Mays, Fan Wang, Felipe Petroski Such, Filippo Raso, Florencia Leoni, Foivos Tsimpourlas, Francis Song, Fred von Lohmann, Freddie Sulit, Geoff Salmon, Giambattista Parascandolo. Gildas Chabot, Grace Zhao, Greg Brockman, Guillaume Leclere, Hadi Salman, Haiming Bao, Hao Sheng, Hart Andrin, Hessam Bagherinezhad, Hongyu Ren, Hunter Lightman, Hyung Won Chung, Ian Kivlichan, Ian O' Connell, Ian Osband, Ignasi Clavera Gilaberte, Ilge Akkaya, Ilya Kostrikov. Ilya Sutskever, Irina Kofman, Jakub Pachocki, James Lennon, Jason Wei, Jean Harb, Jerry Twore, Jiacheng Feng, Jiahui Yu, Jiayi Weng, Jie Tang, Jieqi Yu, Joaquin Quiñonero Candela, Joe Palermo, Joel Parish, Johannes Heidecke, John Hallman, John Rizzo, Jonathan Gordon, Jonathan Uesato, Jonathan Ward, Joost Huizinga, Julie Wang. Kai Chen, Kai Xiao, Karan Singhal, Karina Nguyen, Karl Cobbe, Katy Shi, Kayla Wood, Kendra Rimbach, Keren Gu-Lemberg, Kevin Liu, Kevin Lu, Kevin Stone, Kevin Yu, Lama Ahmad, Lauren Yang. Leo Liu, Leon Maksin, Leyton Ho, Liam Fedus, Lilian Weng, Linden Li, Lindsay McCallum, Lindsey Held, Lorenz Kuhn, Lukas Kondraciuk, Lukasz Kaiser, Luke Metz, Madelaine Boyd, Maja Trebacz, Manas Joglekar, Mark Chen, Marko Tintor, Mason Meyer, Matt Jones, Matt Kaufer, Max Schwarzer, Meghan Shah, Mehmet Yatbaz, Melody Y. Guan, Mengyuan Xu, Mengyuan Yan. Mia Glaese, Mianna Chen, Michael Lampe. Michael Malek. Michele Wang. Michelle Fradin, Mike McClay, Mikhail Pavlov, Miles Wang, Mingxuan Wang, Mira Murati, Mo Bavarian, Mostafa Rohaninejad, Nat McAleese, Neil Chowdhury, Neil Chowdhury, Nick Ryder, Nikolas Tezak, Noam Brown, Ofir Nachum, Oleg Boiko, Oleg Murk, Olivia Watkins, Patrick Chao, Paul Ashbourne, Pavel Izmailov, Peter Zhokhov, Rachel Dias, Rahul Arora, Randall Lin, Rapha Gontijo Lopes, Raz Gaon, Reah Miyara, Reimar Leike, Renny Hwang, Rhythm Garg, Robin Brown, Roshan James, Rui Shu, Ryan Cheu, Ryan Greene, Saachi Jain, Sam Altman, Sam Toizer, Sam Toyer. Samuel Miserendino, Sandhini Agarwal, Santiago Hernandez, Sasha Baker, Scott McKinney, Scottie Yan, Shengjia Zhao, Shengli Hu, Shibani Santurkar, Shraman Ray Chaudhuri. Shuyuan Zhang. Siyuan Fu, Spencer Papay, Steph Lin, Suchir Balaji, Suvansh Sanjeev, Szymon Sidor, Tal Broda, Aidan Clark, Tao Wang. Taylor Gordon, Ted Sanders, Tejal Patwardhan, Thibault Sottiaux, Thomas Degry. Thomas Dimson, Tianhao Zheng. Timur Garipov, Tom Stasi, Trapit Bansal, Trevor Creech, Troy Peterson, Tyna Eloundou, Valerie Qi. Vineet Kosaraju, Vinnie Monaco, Vitchyr Pong. Vlad Fomenko, Weiyi Zheng, Wenda Zhou, Wes McCabe, Wojciech Zaremba. Yann Dubois, Yinghai Lu, Yining Chen, Young Cha, Yu Bai, Yuchen He, Yuchen Zhang, Yunyun Wang, Zheng Shao, and Zhuohan Li. Openai ol system card, 2024.
[22] Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters, August 2024.
[23] Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling IIm test-time compute optimally can be more effective than scaling model parameters, 2024.
13

---

REFERENCES
REFERENCES
[24] Robert H. Swendsen and Jian-Sheng Wang. Nonlinear filtering: Interacting particle resolution ScrenceDirect. http://wwww.aaaiiccccccccccottta 1986.
[25] Simo Särkkä. Bayesian Filtering and Smoothing. Institute of Mathematical Statistics Textbooks. Cambridge University Press, 2013.
[26] Peiyi Wang, Lei Li, Zhihong Shao, R. X. Xu, Damai Dai, Yifei Li, Deli Chen, Y. Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce Ilms step-by-step without human annotations, 2024.
[27] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models, 2023.
[28] Wei Xiong, Hanning Zhang, Nan Jiang, and Tong Zhang. An implementation of generative prm. https://githuut Cot. Reward-Modeling 2024.
[29] An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5 math technical report: Toward mathematical expert model via self-imprrvement. arXiv preprint arXiv:2409. 2tt.:2409//122., 2024.
[30] Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, Zhenghao Liu, Bowen Zhou, Hao Peng, Zhiyuan Liu, and Maosong Sun. Advancing Ilm reasoning generalists with preference trees. 2024.
[31] Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons of developing process reward models in mathematical reasoning. arXiv preprint arXiv:2501.0745 2025.
[32] Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The Lessons of Developing Process Reward Models in Mathematical Reasoning, January 2025.
[33] Stephen Zhao, Rob Brekelmans, Alireza Makhzani, and Roger Grosse. Probabilistic inference in language models via twisted sequential monte carlo, 2024.
[34] Andy Zhou, Kai Yan, Michal Stlapentokh-Rothmmnn Haohan Wang, and Yu-Xiong Wang. Language agent tree search unifies reasoning acting and planning in language models, 2024.
14

---

B MULTIPLE ITERATIONS AND PARALLEL CHAINS
A Aggregation Strategies for PRM Scores
To compute particle weights during generation, we aggregate per-step scores from the process reward model (PRM) f. Our default uses product of step-level rewards to align with the factorized likelihood structure, but alternative aggregation strategies (e.g., min, last-step, or model-based) may offer different trade-offs..
The weight update step particle filtering depends on how rewards are assigned partial trajectories using a preference or reward model (PRM) f. Since PRMs often provide per-step scores, aggregating them into a single weight requires a strategy that balances theoretical correctness and practical utility.
We consider the following four aggregation approaches:
Product (prod): Computes the product of step-level rewards across all tokens. This aligns directly with the factorized likelihood structure used in the PF objective ((2 enabling online weight updates as generation proceeds.
Minimum (min): Takes the minimum reward seen so far. This penalizes trajectories for weak intermediate steps, which may help in discouraging risky completions. However, prevents online updates because the entire prefix must be scored to determine the weight.
Last-step (last): Uses only the most recent step's reward. Although not aligned with a likelihood-based interpretation, this method is computationally efficient and reflects the scoring mode used in [3].
Model-based aggregation: Instead of relying on step-wise rewards, this method repurposes the PRM in black-box fashion assign single scalar score the full partial trajectory. This helps smooth over noisy token-level scores and can be more stable, especially when PRMs are inconsistent across steps. The model receives the prompt and prefix and returns scalar reward.
Appendix[[hoowe how the input format for this black-box mode differs from standard per-step PRM usage. We compare all strategies empirically 4.5 and find that the optimal choice varies with the PRM's training and evaluation objective.
B Multiple Iterations and Parallel Chains
Here, we explore how different ways to allocate budgets change the performance in the appendix. Specifically, we study for a fixed budget N T M, how the combination of N, M can yield the best performance, where N is the number of particles, T is the number of iterations, and M is the number of parallelism.
B.1 Particle Gibbs
Specifically, each iteration, PG samples a new set of particles using PF with a reference particle from the previous iteration. This integration combines the efficiency of PF with the theoretical guarantees of MCMC, making PG suitable for high-dimensional or challenging posterior distributions. The adaption of PG to inference-tine which works as follows: For each iteration, we run a modified PF step with an additional sampling step to sample 1 reference particle according to the equation in the resampling step of Section [.1] For any PF step that is not the initial step. the PF is executed with a reference particle: This reference particle is never replaced during the resampling step, but its partial trajectory during resampling. We detail the PG version of inference-time scaling in Algorithm of Appendix [ Note that typically, reasonably large number of particles is needed show the benefits of multiple iterations, which we also confirm in our results in Section 4.5
Particle Gibbs is type of MCMC algorithm that uses PF as transition kernel (2].
scaling is essentially a multi-iteration extension of the PF algorithm presented,
still be forked
Allocating budget between N and T F results of Llama-3.2 1B model when config- ured with various test-time compute budget allocations. Although the plot shows that various Particle Gibbs onfigurationss do not have marked benefit over an equivalently budgeted particle filtering run, PG experiment with 16 particles and 4 iterations powered by Qwen 2.5 7B Math Instruct
15

---

B.2 Parallel Tempering
B MULTIPLE ITERATIONS AND PARALLEL CHAINS
0.60
Accuracy
0.50
0.45
Particle Filtering
0.40
Gibbs particles per group)
Particle Gibbs particles per group)
0.35
Particle Gibbs (16 particles per group)
2
2
2
2
Budget (# of model generations)
Figure 6: Comparison of PF and Particle Gibbs with different numbers of iterations, evaluated 100-question subset of the MATH-500 dataset using Llama-3.2- 1--aa--.---------e IB-Instruct as the policy model.
policy model achieved a 87.2% accuracy on MATH500, beating ol performance. Configurations with larger N values typically do better than equivalently budgeted runs with less particles.
B.2 Parallel Tempering
Parallel tempering In parallel tempering (aka replica exchange MCMC sampling), multiple MCMC chains run in parallel different temperatures and swap states to allow better exploration. The key idea is that the chain running in high temperature can explore better, e.g. traversing between different modes the target, and the swap makes possible to let the low temperature chain exploit the new region found by the other chain. We detail the complete parallel tempering version of inference-time scaling in the Algorithms section of the Appendix below Algorithm of while we only explore special case of it (multiple chains with single iteration) in experiments.
Allocating budget between N and M Figure 7 shows PF and PT configurations over set of increasing numbers of budgets. First, as we can see, for any fixed N, increasing M also improves
0.48
0.44
Accuracy
0.42
0.40
0.38
Particle Filtering
Paralle! Tempering particles per group)
Parallel Tempering (8 particles per group)
Parallel Tempering (16 particles per group)
2²
23
2
Budget (# of model generations)
Figure 7: Comparison of PF and PT with different particle group sizes, evaluated on 100-question subset of the MATH500 dataset using Llama-3.2- 22-1B-Insttuct as the policy model.
the performance. This may be helpful when combining batch generation with distributed computing. Second, PT with N 16 has a better overall scaling than PF. This indicates that there is some optimal budget allocation over parallel chains that further improve the overall performance of main results.
We leave the exploration over the optimal configuration of N,T, M jointly future work.
16

---

RUNTIME EXPLORATION
C Proof of Theorem 2
Theorem 2 (Unbiasedness of Expected Accuracy). Let ((u(),z(0)) be weighted particles from and is s__oorect(z) is function to check the correctness of response x. We have
Algorithm 7
( is corrrec((aa)) )
E
Σ
Σ
(aa(Err e,, 0, = 1) is correct(z!1))
where the expectation is over the randomness of the algorithm itself.
Proof. This is direct result of applying the unbiasedness property of particle filtering on well- defined expectation p [x=p(f(x)} of any function f over distribution p. the Monte Carlo estimate using weighted samples from particle filtering is an unbiased estimate of this expectation. As is (-rrrocttt)) binary function, the expectation of the estimate finite thus well-defined and therefore the unbiasedness of accuracy holds.
D Side by Side Comparison of Particle Filtering vs Beam Search
math problem
math problem
top- N
softmax
-12
+/
distribution:
L
selected: 1,3
sample N: 1, 1,4,
expand Mf
(a) Particle filtering uses rewards to produce softmax distribution and does stochastic expansion of N based sampling.
(b) Beam search treats rewards as exact and per- forms deterministic expansion based on beam size N and beam width M.
Figure A side-by-side comparison between particle filtering and closest search-based counterpart, beam search. Compared with beam search in Figure [bb] where the selection and expansion is deterministic (implicitly assumes the rewards are correct), particle filtering in Figure Sa trusts the rewards with uncertainty and propagates the expansion via sampling.
E
Confidence Intervals
To assess the robustness of our results, we reran BoN, Beam Search, and Particle Filtering across three random seeds using three different generator models on the AIME 2024 benchmark. Table reporrts mean accuracy with standard deviation (#), providing a measure of variability and approximate confidence intervals. Across all models, Particle Filtering consistently outperforms alternative decoding methods with low variance, confirming that our improvements are statistically robust rather than artifacts of single seed.
F Runtime Exploration
We evaluate the inference-time efficiency of Particle Filtering (PF) relative to standard decoding baselines. Wall-clock comparisons were conducted on 50-question subset of the MATH500 dataset using the Qwen2.5-Math- Matth1.55- 1.5B- Instrrctt model with vLLM as the inference engine.
PF with sampling budget of 32 requires comparable time BoN with budget of 64, while achieving higher accuracy (84.6 vs. 82.8). This reflects a favorable compute accuracy tradeoff, indicating that PF yields more effective exploration per unit of compute.
ccmpute-accuracy
In terms theoretical cost, PF, BoN, and Beam Search have comparable FLOP complexity under idealized conditions with full KV-cache reuse (i.c., past token representations perfectly reused
17

---

ALGORITHMS
Model
Method
AIME 2024
BoN
0.033 = 0.019
Qwen-2.5 1.5B Instruct
Beam Search
0,122 # 0.022
Particle Filtering (Ours)
0.155 = 0.022
BoN
0.122 0.011
Qwen-2.5 Math 1.5B Instruct
Beam Search
0.167 + 0.019
Particle Filtering (Ours)
0.222 = 0.011
BoN
0.144 = 0.011
Qwen-2.5 Math 7B Instruct
Beam Search
0.222 = 0.001
Particle Filtering (Ours)
0.300 = 0.019
Table 3: Mean + standard deviation over 3 seeds on AIME 2024. Particle Filtering (Ours) consistently
achieves the highest accuracy across all generator models with low variance.
Method (Budget)
Wall-Clock Time (sec) heightBoN (32)
211.69 BoN (64)
392.14 14 Beam Search (32)
549.48 Particle Filtering (32)
401.63 height
Table 4: Wall-clock time a 50-question subset of MATH500. All methods use Qwen2.5-Math- I.5B-Instruct with vLLM.
during decoding). In practice, however, stepwise inference introduces reduced cache reuse, as current inference engines such vLLM are not optimized for token-by-toker decoding. This leads to duplicated computation and increased latency. These runtime differences therefore stem primarily from current implementation constraints rather than from the underlying algorithmic complexity.
Finally, we note that engineering optimizations can substantially affect apparent runtime: for instance, implementing more sophisticated search variants such as DVTS required several days to complete under the same hardware setup, underscoring the dominant role of inference engine design in end-to-end efficiency.
G Algorithms
Algorithm 1 Particle Filtering for Inference-Time Scaling
the number of particles N, reward model f, a LLM pm and the prompt Initialize N particles {ar() ~ Pas c) t ++1
Input:
while not all particles stop do
Update rewards w = ==(<= ) f(z (N)
Compute softmax distribution
softmax (w)
Sample indices {5} 1 J ~ P.(j i) = 0
Update the set of particles (aft Ji=1
((()))
Transition {x(2] PM( c, a 1 IN
1- ++
end while
Return: the set of weighted particles in the end
18

---

ALGORITHMS
Algorithm 2 Particle Gibbs for Inference- Time Scaling
Input: same as Algorithm
D with the number of Gibbs iterations
Run Algorithm to get set of particles {=1:t]i=1 do
for j =
Compute rewards w w- [f( ... ; f (2) )
Compute softmax distribution softmax (w)
Sample reference particle "i:t = o where j PPj = 0:
Initialize N particles {w1 pM( c))
t t+1
while not all particles stop do
Update w [ f( (N 1 f(x1tt)]
Compute softmax distribution
softmax(() w)
Sample indices ( ~ P.(j = i) = 0i
Update the set of particles as (aft (t ) Ji=1
Transition {2t4] PM( c, 2,141)) D
f 1
end while
end for Return: the set of particles in the end
For a set parallel chains with temperatures Ti > T2 > at each iteration, we swap the states of every pair of neighboring chains k, k + with the following probability
A = min (1, Thk (z( bー
( (zx(k)
(4)
TT. (x<))))) (x(++1))
where TA. #1+1 are the two targets (with different temperatures) 2k, T3+1 are their states before swapping.
19

---

H PARTICLE FILTERING RESULTS WITH MORE GENERATOR MODELS
Algorithm 3 Particle Gibbs with Parallel Tempering for Inference- Scaling
Input: same as Algorithm with the number of parallel chains M and a list of temperature
T1....; TM for j = 1..... T do
for k l........... .M do
if j = then
Run Algorithm I to get set of particles {=iati= for chain k
else
Initialize N particles {=] ~ps( [
& <-1
while all particles stop do
Update w = [f(11.), f ),
(N-1)), f(oT:1)
Compute softmax distribution 0 = softmax (wexxx/w///// (w/Th)
Sample indices 1 1 Pr(j = i) = 0;
Update the set of particles as {a1 (1 (2)
ji=1
Transition { ~ PM( c, 1111
1+1+1
end while
end if
Compute rewards w = (f(x f f(z (N)
Compute softmax distribution softmax (Tas
Sample reference particle re := I.t where P = = end for for l = L....., .M do Exchange the reference particle between chain & and with probability according
end for end for
Return: M of particles in the end
H Particle Filtering Results with More Generator Models
Below, we show further results using particle filtering to inference scale wider variety of generator models.
Model
Method
MATH500
AIME2024
JJama-3.2-IB-Insttee
Greedy
26.8
0.0
Particle Filtering (Ours)
59.6
10.0
LJama-3.2-8B--nsttuue
Greedy
49.9
6.6
Particle Filtering (Ours)
74.4
16.6
phi-4
Greedy
79.8
16.6
Particle Filtering (Ours)
83.6
26.6
Mistral-Small-244-Innnrreee
Greedy
69.2
Particle Filtering (Ours)
83.4
23.3
Table Performance of LLMs on MATH500 and AIME 2024 using greedy decoding and Particle Filtering (ours). Particle Filtering is run with 64 generations per problem.
20

---

EVALUATION DETAILS
I Inference Prompt Template
Evaluation System Prompt
Solve the following math problem efficiently and clearly:
For simple problems (2 steps or fewer):
Provide concise solution with minimal explanation.
For complex problems (3 steps or more): Use this step-by-step format:
## Step 1: [Concise description]
[Brief explanation and calculations]
## Step 2: [Concise description]
[Brief explanation and calculations]
Regardless of the approach, always conclude with:
Therefore, the final answer is: (nnsxed(an I hope it is correct.
Where [answer] is just the final number expression that solves the problem.
PRM Input Format
## Step [Concise description]
[Brief explanation and calculations]
Creward_token
## Step 2: [Concise description]
[Brief explanation and calculations] howward_toker
## Step 3: [Concise description]
[Brief explanation and calculations] kreward_token>
ORM Input Format
## Step [Concise description]
[Brief explanation and calculations]
## Step 2: [Concise description]
[Brief explanation and calculations] ## Step 3: [Concise description] [Brief explanation and calculations] creward_ token>
J Evaluation details
Parsing and scoring Following prior work on mathematical reasoning benchmarks 29 we apply their heuristic-based parsing and cleaning techniques robustly extract the boxed expression. These heuristics handle spacing variations, formatting inconsistencies, and other artifacts in model outputs. For answer verification, follow (3) converting responses canonical form. Ground truth and generated answers are transformed from LaTeX into SymPy expressions, simplified for normalization, and converted back LaTeX. Exact match is determined using two criteria: numerical equality, where expressions evaluate to the same float, and symbolic equality, where they are algebraically equivalent in SymPy [3. Accuracy is computed the fraction of problems where the generated answer exactly matches the ground truth.

---

EVALUATION DETAILS
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We show experiments that directly address and back up every claim make in the abstract and introduction. Please see the Evaluation section for our empirical results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss several limitations of our work the conclusion section. For clarity, we copy them here: "However, inference-ting cetrence-timeeeee scaling comes with computational challenges. Hosting and running reward model often introduces high latency, making the process more eesource-intessvve. For smaller models, prompt engineering is often required to ensure outputs adhere to the desired format. Finally, hyperparameters like budget are problem-dependent and may require tuning across domains."
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specificattin, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
22

---

EVALUATION DETAILS
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We have I theorem in our paper, and its proof is provided Appendix C
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We include full details of how exactly our overall methodology work. We also provide key details into the hyperparameter selection and ablation process. which significantly helps reproducability. We also provide details into several reward model aggregation techniques, as well as why we chose the one we chose, in our paper. We also include details on several different Process Reward Models and include ablations showing why we chose the one we chose. Our experiments are reproducible.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
23

---

EVALUATION DETAILS
(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We do not currently include code not break author confidentiality. How- ever, we will completely open source our code upon acceptance of the paper to encourage as many people possible
our work.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
httpppulbuuddeeeeeeepeeeppppeeee httpai//ggiidde//e/uuubbiiiieee http///oodsmssiiooooooo sosouoooisssiooooocccc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide full details all datasets we used, well their citations. We list exactly which models and versions we used as generator models. We also provide key details into the hyperparameter selection and ablation process, which significantly helps reproducability. We also provide details into several reward model aggregation techniques, well as why chose the one chose, in our paper. We also include details on several different Process Reward Models and include ablations showing why we chose the one we chose. Our experiments are reproducible.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
24

---

EVALUATION DETAILS
Question: Does the paper report enror bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Due to computational limitations on academic budget, we were not able to run every single experiment multiple times to produce accurate and fair error bars across every experiment in the paper. However, we have many experiments several times during the research and development process (both several times and by several different people) and very confident in our results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [No]
Justification: Although we include information about the computational burden of running inference time scaling experiments, we do not provide formal information about which exact computational resources we used, as we used different numbers GPUs for different experiments. That being said, we have developed (to be released upon acceptance) open source library for our work that is able to used completely off the shelf and contains very simple information about how to run it.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
25

---

EVALUATION DETAILS
Justification: We can confirm that, in every respect, we do not violate the NeurIPS Code of Ethics. Our research does not have negative societal consequences, nor does it involve human subjects. We do not use any private or sensitive data.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Our work does not have any negative societal consequences. We discuss the positive impacts of inference scaling, as it opens up higher level language model performance to those who are only able to access smaller models.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), faireess considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification: We do not release any data or models of our own. Instead, we only use off-the-sheff open source models, and therefore there are possibilities of misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
26

---

EVALUATION DETAILS
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Our work only open source models, and cite every model that Therefore, all creators the original models are credited this work.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com /dtassts has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA].
Justification: We do release any new assets in this paper instead, discuss how to enhance the performance of already existing open-sourced models.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our research does not include any human subject experiments or crowdsourcing experiments.
Guidelines:
27

---

EVALUATION DETAILS
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA].
Justification: Our work does not include any human subjects, and did not need IRB approvals.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLM usage did not impact the core methodology, scientific rigorousness, or originality of the research.
Guidelines:
The answer NA means that the core method development in this research does not involve L.LMs as any important, original, or non ntn-standard components.
Please refer to our LLM for what should or should not be described.
28


Reviews:
[{"review_id": "ixKWDSKunn", "reviewer": "Reviewer_APTW", "summary": "Inference time scaling approaches based on deterministic rules suffer from early pruning -- pruning promising sequences due to incorrectly assigning low scores. This paper proposes a particle-based monte carlo method to alleviate this problem. The idea is to keep $N$ candidate incomplete sequences at each step, sample new tokens for every sequence from a LLM, re-weight each new sequence using a PRM, and finally re-sample with replacement from the candidate pool using the corresponding weights. This process is designed to trade-off exploration and exploitation by using PRMs to exploit but using sampling to explore. Experimental results are done using Qwen and Llama family of LLMs with a Qwen-based frozen PRM. The proposed sampling approach shows improvement on several reasoning benchmarks including math, chemistry, and finance compared to several baselines. On MATH500, the approach improves open source Qwen-2.5 Math 7B Instruct to reach the performance of o1-preview. While using other PRMs still give promising results, a strong PRM is necessary to best performance. The approach is also shown to be robust to different sampling temperatures.", "questions": "1. Please compare to relevant work on MC methods in more detail and add related baselines to empirical comparison.\n\n2. Why weights are updated multiplicatively rather than additively?\n\n3. Please correct or address other issues explained in weaknesses.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "# Strengths\n1. The paper is written clearly and easy to follow. The motivation is clear and the approach is explained using a clear language.\n\n2. The approach outperforms several baselines on multiple benchmarks while being simple to implement.\n\n# Weaknesses\n1. **The approach is similar to other particle-based MC methods.** Especially, [18] also introduces a particle-based MC sampling approach for inference time scaling. Based on Eq (4) in their paper, if you define the potential function as $\\phi(x_{<t}, x_t)=exp(\\sum_{i<t} w_i + w_t)$ where $w_i$s are defined as in your paper, you get a very similar rejection sampling approach. Currently, related work lack a detailed comparison and it needs improvement to highlight key differences. If they are closely related, they should also be added as baselines in empirical comparison.\n\n2. **Why do you update $w_t$ multiplicatively?** While you define $r$ as a probability, weights are treated merely as unnormalized logits; you use softmax over them for re-sampling. Multiplicative weights make softmax more complicated and requires at least some ablation studies for justification.\n\n3. **Some details are missing.** Please correct/address the followings:\n- $w_t$ is initialized to 1.0.\n- Why is it proportional to rather than equality in $w_t=w_{t-1} \\hat{r}$?\n- Sampling is without replacement.\n- In line 88, you define the bernoulli distribution by using $x_t$ but the left side has $x_{<t}$.\n- In Eq (2), subscript is missing, in line 111, subscript/superscript is missing from summation.", "was_revised": true, "final_justification": "The authors clarified most of my concerns.", "rebuttal": "Thank you for your thoughtful review. We address your comments below. \n\n&nbsp;\n\n**Comparison to Other Works:**\nThank you for the thoughtful comment. \n\n-   We agree that **particle-based Monte Carlo methods are a shared foundation** across multiple works and note that **particle filtering is a decades-old concept** with long-standing use in probabilistic inference. As such, none of these papers - including ours or [18] - are unique in applying such techniques to language models.\n    \n\n&nbsp;\n\n-   Despite their shared theoretical inspirations, the **goals of the two works are quite distinct**.\n    \n    -   [18] focuses on **token-level SMC within probabilistic programs for constrained generation**.\n        \n    -   Our work develops **step-level particle filtering for inference-time scaling**, designed to be **robust to noisy and imperfect reward functions** when scaling LLM inference across diverse reasoning tasks.\n        \n&nbsp;\n\n-   Because [18] addresses a different setting and is not directly applicable to our evaluation domains (e.g., MATH, AIME), we did not include it as a baseline. We instead **compare to and outperform TSMC for Math [8]**, which also uses SMC for LLM reasoning and is closer in scope to our work. We **exceed [8]'s performance by 14.6 points using half the budget** on the same dataset (MATH500) and model (DeepSeekMath7B), and we do so **without needing fine-tuning or ground-truth verifiers**.\n\n\n&nbsp;\n\n- We also implemented the MCTS baseline from Zhang et al. and evaluated it against PF using the same base model (Llama 3.1 8B Instruct) and compute budget (32 rollouts). On the AIME 2024 benchmark, **PF achieved 16.6% accuracy, significantly outperforming MCTS, which achieved only 6.6%**. We will include this empirical comparison and accompanying discussion in the revised version.\n\n&nbsp;\n_______\n**Updating $w_t$ multiplicatively**\n\nThank you for raising this point — you're absolutely right that the choice to multiply weights across steps deserves clarification.\n    \n-   We use a **multiplicative update** so that each particle’s weight reflects the quality of the **entire partial trajectory**, not just the most recent step. This aligns with standard particle filtering practice, where the total weight captures cumulative likelihood.\n    \n-   That said, we acknowledge that multiplicative updates could potentially compound noise, so **we explore alternative reward aggregation schemes** (e.g., using only the final step score, minimum score, etc.) **as ablations in Appendix A and Section 4.5** (see Lines 131–135) and show that these offer different trade-offs between stability and expressivity. \n\nWe’ll make this connection more explicit in the main text.\n\n&nbsp;\n\n_______\n**Other Details**\n- $w_t$ is initialized to $1.0$: \n   - Yes, each particle’s initial weight w_t is initialized to 1.0. We will make this explicit in the revised algorithm description.\n   \n- Proportional rather than equality in $w_t \\propto w_{t-1} \\hat{r}$:\n  - We use a proportionality symbol rather than equality to reflect that the weights are unnormalized at each step. The normalization is performed later during the resampling step via a softmax over all particle weights. We will clarify this in the text to avoid confusion. \n\n\n\n- Sampling is without replacement: \n  - Thank you for raising this point. To clarify, our resampling is performed *with* replacement, consistent with standard practice in particle filtering. This means that a particle at step $t$ can be selected multiple times and continued by multiple particles at step $t+1$. An example of this behavior is illustrated in our main method Figure 2 of our paper. We will make this explicit in the revision.\n\n- Line 88 Bernoulli Distribution:\n  - Thank you for the careful reading - you're absolutely right that the emission distribution as written is formally incorrect. It should be written as $p(o_t \\mid c, x_{\\leq t})$, since the emission probability depends explicitly on the newly sampled token $x_t$.\n   - In our model, the reward function $r(c, x_t)$ directly determines the likelihood of observing $o_t$, and thus the correct conditioning set includes $x_t$. We’ll revise the notation to reflect this.\n   - This kind of dependency is common in state space models, where although emissions are conditioned on past latent states, they also rely on the current state sampled at each time step.\n\n- Subscript Formatting: \n\t- Thanks for catching this. We will fix both issues in the final version.\n\n\n&nbsp;\n\n\nReferences:\nZhang, D., et al. (2024). Accessing GPT-4 level Mathematical Olympiad Solutions via Monte Carlo Tree Self-refine with LLaMa-3 8B.\n\n_________\nThank you again for your thoughtful review. We sincerely hope these clarifications and updates address your concerns and strengthen your confidence in the work."}, {"review_id": "2BMJDx3915", "reviewer": "Reviewer_YTBD", "summary": "The paper formulates the process of PRM-guided LLM inference as a state-space models and applies particle filtering to infer a posterior distribution over optimal trajectory. Trajectories sampled from this posterior are scored with outcome reward model and the best trajectory is retuned. The method achieves higher performance in a range of reasoning tasks while using the same computational budget.", "questions": "* Which stopping criterion for the searches was used in empirical comparison? I would suspect that this has a strong influence on the performance in reasoning tasks.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n* The method is theoretically sound and grounded in the principles of probabilistic inference. \n* The empirical performance gains in the experiments are significant and consistent across different settings.\n* Multiple ablations show that the performance remains high across a range of hyperparameter choices.\n\n**Weakness**:\n\n* It is not fully clear to me how the methods would compare in terms of runtime.  For the results in Table 1 and 2, it is mentioned that the budget (i.e. number of generated sequences) was the same for all methods. Does this also hold for the runtime? It is mentioned earlier that the total runtime is comparable to generating N full completions independently. This should be more than the runtime of beam search, if I understand it correctly.\n\n* The conclusion mentions: \n> We hope that the formal connection of inference scaling to probabilistic modeling established in this work will lead to systematic solutions for current limitations of these methods and pave the way for brining advances probabilistic inference algorithms into LLM inference-time scaling in future work. \n\n  Applying techniques from probabilistic inference to LLM inference-time scaling has already been explored in other work: \n\n   > Uncertainty-Guided Optimization on Large Language Model Search Trees (Grosse et al., 2016)\n\n  The settings differ to some extend (and obviously the probabilistic inference techniques also differ), but to claim that this connection is newly established in this work nevertheless is an exaggeration.", "was_revised": true, "final_justification": "All questions were addressed and the authors provided additional results regarding the runtime. I am not 100% confident in the runtime-efficiency of the proposed method, but since Reviewer wjrz seems to think that the new results regarding the runtime are great, I am also not 100% confident that I did not misread something. All in all, I still lean towards acceptance due to the paper being theoretically well motivated.", "rebuttal": "Thank you for your thoughtful and constructive review—we’re grateful for your recognition of the paper’s theoretical grounding, strong empirical results, and robust ablations across hyperparameters.\n____\n**Runtime Exploration:**\nThank you for the thoughtful comment. We ran **wall-clock time comparisons** on a 50-question subset of MATH500 using Qwen2.5-Math-1.5B-Instruct and vLLM as the inference engine:\n\n-   BoN (budget=32): 211.69 sec  \n      \n    \n-   BoN (budget=64): 392.14 sec  \n      \n    \n-   Beam Search (budget=32): 549.48 sec  \n      \n    \n-   Particle Filtering (budget=32): 401.63 sec  \n      \n    \n\nWe find that **PF at budget 32 takes comparable time to BoN at budget 64, while achieving higher accuracy** (84.6 vs. 82.8), indicating a **favorable compute–accuracy** tradeoff.\n\nRegarding FLOPs: in theory, PF, BoN, and Beam Search should have similar cost under perfect KV-cache reuse (where past token KV-caches are fully reused in decoding). However, in practice, step-wise inference leads to low KV cache hit rates because inference engines like vLLM are not optimized for token-by-token generation. This results in duplicated compute and increased latency. Importantly, **these differences are driven more by current engineering constraints than by the underlying algorithm**.\n\n(As further evidence of how much these engineering implementation details can dominate: running DVTS, a better version of beam search, took days.)\n\nWe will include these latency results and a discussion of the practical vs. theoretical tradeoffs in the revised version.\n\n\n**Related Work**: Thank you for pointing this out and for highlighting the relevant prior work. We’ll revise the conclusion to better reflect this and adjust our claims accordingly.\n\n**Stopping Criterion:** Thank you for the question. For all methods, including particle filtering and the baselines, we used a consistent stopping criterion: each generation runs until the $< EOS >$ token is produced or the model hits a predefined maximum length. This ensures that completions are full solutions and not prematurely truncated. We will clarify this detail in the revised version.\n\n_____\nThank you again for your thoughtful comments. We hope the above information further solidifies your support of this work."}, {"review_id": "nZUXsZhHZX", "reviewer": "Reviewer_wjrz", "summary": "The authors consider using a particle filter (PF) based approach for the problem of inference-time scaling of language models (LMs). Specifically, the authors use process reward models (PRMs) in the PF, stochastically resampling sequences with higher PRM scores. The authors compare this against other inference-time methods such as best-of-n and beam search, testing empirically across a range of datasets (math, chemistry, finance) and models (LMs and PRMs), showing significantly improved performance with the PF method.", "questions": "Figure 4c - why is the greedy line below the 2^0 point? Does this suggest that greedy is worse than a single random sample? And if so, why does Figure 4d show that greedy is better than a single random sample? Are these quirks of the models? Is the plot wrong? Or does random sampling have high variance (something that confidence intervals would be very useful to get an understanding of)? \n\nLines 246-249 - this is an interesting finding, and somewhat unintuitive. Why do the authors think this is happening? Also, a related question - why do the authors think that the Qwen PRM performance stagnates after just 4 samples? \n\nOverall, I'm on the fence about this work. There's a lot that I like, but there's also a lot that I don't like. I think the results could be quite significant for the broader community, but I have doubts about the results. I'm choosing borderline accept instead of borderline reject because I think the strength of the positives outweighs the negatives. If the authors address all my points convincingly I'd consider increasing to accept.", "limitations": "Authors do discuss limitations, which is good, but only briefly; there is much more the authors could discuss (e.g., my points above). \n\nAuthors claim the work has no potential negative social impacts, which I don't think is fair. The critical assumption made here is that better LM performance (particularly on reasoning tasks, and for smaller models) can only lead to positive outcomes. While I would agree that the potential positives outweigh potential negatives, so overall the work would have positive social impact, there is potential for risks such as harm from more capable models in the hands of bad actors.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nPaper tackles an important problem setting. Ideas presented are reasonable, worth exploring, and could lead to significant advances in state-of-the-art. The theory/math that I checked (not everything) seems correct. \n\nGood consideration of baselines, alternatives, ablations. Good evaluation over a wide range of different models (both generator/base LMs and PRMs). Fairly comprehensive experimental results across several different datasets. Experimental results appear to show very strong performance (though see comments below). \n\nPaper is reasonably structured and not too hard to understand.\n\nWeaknesses:\n\nLine 47 - \"shift-from\" dash should be em dash \n\nLine 97 - x1:T is improperly formatted\n\nLines 140-141: \"Deterministic search can irreversibly discard trajectories that may have low early scores but high posterior mass overall\" - actually this is true of particle filters as well. If you have a limited number of particles (which is always the case in practice), and high enough scores assigned by your PRM, then resampling can concentrate on only those high score sequences and discard low score sequences. And if this happens, that you've resampled and removed some early completions that had low score, you also cannot recover from this, since your propagation only generates subsequent tokens and cannot revise previous tokens.\n\nLines 145-146 - please use em dashes for readability \n\nLine 152 - PG should be defined here instead of later; here is the first usage\n\nWould be nice if you had MCTS as a baseline that you also compared against (this is a minor point).\n\nHaving no confidence intervals in any of the results is a big negative. It's understandable that an academic budget limits the number of experiments you could do, but at least for the final results (I'm not asking for the hyperparameter search/exploration phase), even if only for a few selected runs, you could do at least 3 seeds. This would at least give the reader a rough idea of variability/confidence interval width/statistical significance. With no confidence intervals/statistical significance tests, it's difficult to evaluate the validity and significance of results (having them isn't perfect either, but is much more informative than a single run).\n\nLines 210-213 - I appreciate testing on more benchmarks; that is good. However, I find the reasoning and exposition regarding these testing generalization beyond mathematical reasoning questionable. These are different domains, but they still rely on mathematical reasoning to a large degree. The authors themselves note that the NumGLUE task \"targets numerical reasoning in scientific contexts.\" Similarly, FinanceBench (https://github.com/patronus-ai/financebench/blob/main/data/financebench_open_source.jsonl) has a lot of numerical reasoning based questions. So I would argue that instead of, as the authors claim in lines 229-230 that \"such RMs implicitly learn broader reasoning evaluation capabilities during their training, not limited strictly to mathematical content\", there's a simpler explanation, which is that Qwen2.5-Math-PRM-7B as the reward model works well for these tasks because they're close enough to mathematical reasoning tasks.\n\nLines 260-263 - if using a PRM as an ORM works best, then why is the whole paper focused on PRMs? It seems to me the authors should compare against using ORMs directly in their PF. \n\nLines 301-309 - \"TSMC relies on a ground-truth verifier\"; this is false, TSMC can also be used with ORM/PRMs. \"requires joint training of a generator and value function\" - this is also not really correct, the generator (proposal) can be trained as in Zhao et al. (authors' reference [33]) but does not have to be trained; e.g., you could just use the base LM. The value function also does not necessarily have to be trained, for example, as in Lew et al. (authors' reference [15]). \"The authors of TSMC did not release code\" - this is also false, see for example https://github.com/Silent-Zebra/twisted-smc-lm for the code for [33]; if this comment is for only reference Feng et al. [8], then this should be clarified. Finally, with respect to comparing vs [8], for a fair comparison, you'd have to use the same PRM (I believe [8] uses weaker PRMs).\n\nPF/SMC-related works are the most relevant to this paper, and it's unclear how much originality this paper adds (the core novelty appears to me to stem from the application of PRMs in PFs to the language modelling (specifically mathematical reasoning) setting).\n\nOne other issue with the evaluations is, as the authors acknowledge, using a reward model at intermediate steps can be expensive. Baselines like BoN require only a single call to the PRM/ORM at the end, whereas the PF requires multiple intermediate calls. Therefore, while the authors control for the number of generated tokens/samples, the authors are likely giving the PF significantly more wall-clock time or total FLOPs. Ideally, the authors would be able to demonstrate that either under similar FLOPs or wall-clock time, PF still outperforms baselines.", "was_revised": true, "final_justification": "As I mentioned in my comments, I am happy with the discussion with authors and have an improved evaluation of the paper now. I have increased to accept (but this is conditional on/assuming the authors making all the changes they have outlined in their responses).", "rebuttal": "Thank you for your detailed review and for highlighting the importance of our problem setting, the soundness of our approach, and the strength and breadth of our experimental evaluation - we appreciate your careful assessment. We address additional comments below. \n\n&nbsp;\n\n**Deterministic Search vs. Particle Filtering:**\nThank you for the thoughtful comment. While it's true that particle filtering can also discard low-scoring trajectories, the key distinction is that it does so **stochastically rather than deterministically**. This means there is always a non-zero probability of retaining low-scoring (but potentially correct) paths, unlike deterministic search which prunes them irrevocably. As a result, **particle filtering offers greater robustness to early reward noise, especially under imperfect PRMs, by maintaining exploration over diverse hypotheses.**\n\nAn **example of this behavior is illustrated in Figure 2** of our paper, which shows a **real example** of the full PF method: the first step of the trajectory that ultimately leads to the correct final answer receives the lowest initial score. Yet, because PF retains a non-zero chance of continuing such paths, this trajectory survives and ultimately yields the correct solution.\n\n&nbsp;\n\n______\n\n**MCTS:**\nWhile both Particle Filtering (PF) and Monte Carlo Tree Search (MCTS) aim to balance exploration and exploitation, they differ fundamentally in design and suitability for inference-time scaling in LLMs. To directly compare the two, we implemented the MCTS baseline from Zhang et al. and evaluated it against PF using the same base model (Llama 3.1 8B Instruct) and compute budget (32 rollouts). On the AIME 2024 benchmark, **PF achieved 16.6% accuracy, significantly outperforming MCTS, which achieved only 6.6%.** We will include this empirical comparison and accompanying discussion in the revised version.\n\nReferences:  \nZhang, D., et al. (2024). Accessing GPT-4 level Mathematical Olympiad Solutions via Monte Carlo Tree Self-refine with LLaMa-3 8B. \n\n&nbsp;\n\n____\n\n\n**Confidence Intervals:**\nThank you for this important suggestion — we completely agree. To address this, we ran **BoN, Beam Search, and Particle Filtering** across **3 different seeds using 3 different generator models** on the AIME 2024 benchmark. Below, we **report mean accuracy along with standard deviation (±)**, which we will include in the revised version to better reflect result variability and statistical significance.\n\n \n| Model| Method | AIME 2024      |\n| --------------------------- | ------------------------- | -------------- |\n|  Qwen-2.5 1.5B Instruct     | BoN                       | 0.033 ± 0.019  |\n|     | Beam Search               | 0.122 ± 0.022  |\n|   | Particle Filtering (Ours) | ***0.155 ± 0.022***  |\n| Qwen-2.5 Math 1.5B Instruct | BoN                       | 0.122 ± 0.011  |\n|                             | Beam Search               | 0.167 ± 0.019  |\n|                             | Particle Filtering (Ours) |  ***0.222 ± 0.011*** |\n| Qwen-2.5 Math 7B Instruct   | BoN                       |  0.144 ± 0.011 |\n|                             | Beam Search  |  0.222 ± 0.011 |\n|                             | Particle Filtering (Ours) | ***0.300 ± 0.019***  |\n\n\n&nbsp;\n\n____\n\n**Additional Benchmarks:**\nThank you for the observation — we agree that many questions in FinanceBench and NumGLUE involve significant numerical reasoning, and that this likely contributes to the strong performance of our math-trained PRM. While these tasks also include elements such as reading comprehension and scientific context, we acknowledge that the overlap with mathematical reasoning is substantial. We will revise the discussion to better reflect this and avoid overstating the generality of the PRM. \n\n&nbsp;\n\n_____\n**PRM vs. ORMs:**\nWe appreciate the question. While \"PRM as ORM\" scores entire partial trajectories, it is still fundamentally different from using an outcome reward model (ORM). PRMs are explicitly trained to evaluate partial solutions step-by-step, whereas ORMs are trained to judge only final outputs. Since **particle filtering requires scoring and resampling based on intermediate generations, PRMs remain the appropriate choice**—even when used in aggregation modes resembling full-sequence evaluation. We will clarify this distinction in the revision. Notably, as we show in Figure 5b, **other methods for using PRM scores**, including the Last score, **also perform very well**.\n\n&nbsp;\n\n_____\n**TSMC:**\nThank you for this thoughtful and detailed clarification—we appreciate the opportunity to improve the precision of our comparisons to TSMC. You are correct that our original statements could be clarified further.\n\n-   Our original statements referred specifically to the formal setup introduced in **Feng et al. [8], which trains an optimal value function via Contrastive Twist Learning. This approach relies on both a trained generator and supervision from a ground-truth verifier**, as detailed in Sections 3.4, 4, and Appendix C of [8]. You are correct that TSMC doesn't require a ground-truth verifier at inference time—the verifier is only used during training to shape the twist function.\n\n-   This highlights a key difference in our premises: while **TSMC (e.g., [8], [33]) focuses on learning theoretically optimal twist functions, our work assumes that PRMs are inherently imperfect and noisy**. We ask a **different question**—how to best scale LLM inference given an off-the-shelf PRM—with a method that requires no training or tuning, offering greater robustness in practice.  \n\n-   You're absolutely right that Zhao et al. [33] released their code. Our comment specifically referred to Feng et al. [8], which we focused on since it applies TSMC to math. To our knowledge, their implementation and models are not publicly available—we’ll clarify this in the final version.\n    \n-   Regarding empirical comparisons, we acknowledge that [8] uses a trained value function and argues that off-the-shelf PRMs are weaker. Our goal was to highlight that **even under this disadvantage—using a fixed off-the-shelf PRM (Qwen2.5-Math-PRM-7B)—our method outperforms [8]**, achieving 75.4% on MATH500 with 128 samples, compared to 60.8% in [8] using 240 samples, a trained generator, and a trained value function.  \n\nWe’ll revise the final version to clearly reflect that [8] is focused on learning optimal PRMs, while our work addresses how to conduct inference-time search when the PRM is fixed and noisy.\n\n&nbsp;\n\n____\n\n**Runtime Exploration:**\nThank you for the thoughtful comment. We ran **wall-clock time comparisons** on a 50-question subset of MATH500 using Qwen2.5-Math-1.5B-Instruct and vLLM as the inference engine:\n\n-   BoN (budget=32): 211.69 sec\n-   BoN (budget=64): 392.14 sec\n-   Beam Search (budget=32): 549.48 sec\n-   Particle Filtering (budget=32): 401.63 sec  \n\nWe find that **PF at budget 32 takes comparable time to BoN at budget 64, while achieving higher accuracy** (84.6 vs. 82.8), indicating a **favorable compute–accuracy** tradeoff.\n\nRegarding FLOPs: in theory, PF, BoN, and Beam Search have similar cost under perfect KV-cache reuse (where past token KV-caches are fully reused in decoding). However, in practice, step-wise inference leads to low KV cache hit rates because inference engines like vLLM aren't optimized for token-by-token generation. This causes duplicated compute and increased latency. Importantly, **these differences are driven more by current engineering constraints than by the underlying algorithm**.\n\n(As further evidence of how much these engineering implementation details can dominate: running DVTS, a better version of beam search, took days.)\n\nWe will include these latency results and a discussion of the practical vs. theoretical tradeoffs in the revised version.\n\n&nbsp;\n\n____\n\n**Greedy Lines in Figure 4c:**\nThe small discrepancy in Figure 4c arises because greedy decoding uses temperature 0, whereas the multi-sample methods—including the 1-sample case at $2⁰$—use temperature 0.8. Thus, the \"1 sample\" point reflects a stochastic sample that may occasionally outperform deterministic greedy decoding, depending on the model. In the 1.5B case (Fig. 4c), this occurred; for the 7B model (Fig. 4d), greedy slightly outperformed the sample.\n\nThat said, the differences are minute—on the order of 0.005 to 0.01 in accuracy—and well within the expected variance from sampling.\n\n&nbsp;\n\n___\n**Performances of Different PRMs:**\n-   Thank you for highlighting this insightful finding.\n-   **EurusPRM-Stage2 improves at higher budgets** likely due to greater robustness to noise with more particles:\n    -   At low budgets, PF relies heavily on accurate early-step scoring, and EurusPRM may be less reliable at early steps due to noisier or less calibrated partial scores.       \n    -   **With more particles, PF can better tolerate this noise—allowing good trajectories to survive and improve final accuracy**.   \n-   **Qwen2.5-Math-PRM-7B stagnates early** possibly because of **limited answer diversity**:\n    -   Most viable completions are already surfaced with a few samples. Additional samples contribute diminishing returns.\n-   We note that this ablation used only **100 MATH500 questions** (vs. 500 in main results), so **variability is higher**, especially at small budgets.\n-   We’ll clarify these points in the revision.\n\n&nbsp;\n\n____\n**Social Impact:**\nThank you for raising this important point. We will revise our broader impact statement to note that while our method improves inference robustness, it could amplify harm if misused in sensitive applications.\n\n&nbsp;\n\n____\n\nWe will be sure to fix all formatting related concerns in the camera ready version of this work. \n\n**We appreciate your thoughtful feedback. We hope that our clarifications on the novelty of our approach and the strength of our experimental results have further reinforced your confidence in the significance of this work.**"}, {"review_id": "jbh6TZ5bqK", "reviewer": "Reviewer_1y8J", "summary": "This paper introduces a particle filtering algorithm for inference-time scaling to solve early pruning and diminishing-returns issues in search-based LLM inference. It casts test-time compute as posterior inference over a state-space model—using the LLM as a transition model and a process reward model (PRM) as an emission model—and iteratively resamples, mutates, and reweights a diverse set of candidate sequences (particles) to maintain exploration–exploitation balance under uncertainty. The key insight is that preserving trajectory diversity prevents premature pruning and yields robust, efficient scaling without exponential rollout growth.", "questions": "Please refer to the “Weaknesses” section above.", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\n1. **Quality / Clarity:** The writing is well structured and the figures clearly illustrate the algorithm (e.g., the particle update pipeline). Method details and advantages are presented in a direct, easy-to-follow manner.  \n2. **Significance:** Early pruning in search-based LLM inference is a critical bottleneck; the proposed particle filtering (PF) algorithm is well matched to this problem, as it preserves candidate diversity and balances exploration–exploitation under reward uncertainty.  \n3. **Originality:** Applying particle‐based Monte Carlo filtering to inference-time scaling in LLMs is novel, providing a principled Bayesian framing and opening a new axis of compute allocation beyond deterministic search.  \n4. **Quality:** Comprehensive experiments on MATH500 and AIME 2024  show 4–16× faster scaling rates than deterministic baselines. Ablation studies on compute allocation, PRM aggregation, and temperature yield useful insights into each module’s contribution.  \n\n### Weaknesses\n\n1. **Significance / Clarity:** Regarding Lines 43–46, there appears to be a trade‐off: early pruning can save compute by discarding unlikely trajectories and reallocating resources to high‐probability regions. Could you clarify under what conditions—such as problem difficulty or reward‐model uncertainty—additional exploration becomes beneficial?  \n2. **Originality / Significance:** Monte Carlo Tree Search (MCTS) also balances exploration and exploitation to avoid premature pruning. The PF algorithm proposed in this paper shares strong operational similarities with Monte Carlo Tree Search (MCTS). The paper does not discuss MCTS: how does PF differ from or relate to MCTS? Including MCTS as a baseline and comparing performance would strengthen the evaluation.", "was_revised": true, "final_justification": "Thank you to the authors for the clear and direct rebuttal. Your responses successfully addressed my primary concerns regarding the trade-offs of exploration and the comparison to MCTS. I am maintaining my \"Borderline Accept\" score and now have stronger confidence in the paper's contributions.\n\nMy main reservations were about the lack of comparison to MCTS and the conditions under which the proposed exploration strategy is beneficial. The rebuttal addressed these points well:\n1.  **MCTS Comparison:** The clarification of the fundamental differences between Particle Filtering (PF) and MCTS—particularly PF's robustness to reward noise and superior parallelism—was insightful. The new empirical results showing PF outperforming a recent MCTS baseline on the AIME benchmark provide the necessary evidence to position this work effectively within the existing literature.\n2.  **Value of Exploration:** Your explanation that reward model uncertainty makes exploration valuable for most non-trivial problems is a reasonable and convincing argument. It clarifies the trade-off and better frames the problem setting where your method is most impactful.\n\nAfter the rebuttal, I believe the paper is much stronger. The key remaining weakness is that the experimental comparison to MCTS, while valuable, is still limited to a single setting. However, the conceptual arguments and the new data are sufficient to demonstrate the novelty and value of the proposed PF approach.", "rebuttal": "Thank you for your thoughtful and constructive review - we're glad you enjoyed the narrative clarity, novelty of applying particle filtering to inference-time scaling, and the strength of our experimental results and analysis.\n\n&nbsp;\n\n**Conditions Under Which Addional Exploration is Helpful:** \nIt’s true that in easier problem settings - where the language model has a very high probability of answering correctly - extensive exploration may not be necessary, and early pruning can save compute. That said, reward models are inherently uncertain: they’re not verifiers, but transformer models trained to assign scores, and scoring partial solutions is particularly noisy and difficult. Because of this built-in uncertainty, for most problems, and especially for non-trivial tasks, the stochasticity in particle filtering is beneficial - it helps retain potentially correct paths that might otherwise be prematurely discarded.\n\n&nbsp;\n\n______\n**MCTS Comparison**\nThank you for this thoughtful comment. While both PF and MCTS seek to balance exploration and exploitation, **they differ fundamentally in design and suitability for inference-time scaling of LLMs**.\n -   Design:  \n\t - MCTS builds a tree and backpropagates scores from full rollouts or value estimates.\n    \n\t - **PF maintains a flat particle population, updates weights step-by-step, and resamples stochastically**.  \n      \n    \n\n -   Value Function Sensitivity:  \n      \n    \n\n\t - MCTS can use value estimates to avoid full rollouts, but errors in these estimates backpropagate and bias future decisions.\n    \n\t - PF avoids this by using only local stepwise scores in a stochastic manner - **more robust to reward noise**.  \n      \n    \n\n -   Parallelism & Scalability:  \n      \n    \n\n\t - MCTS requires sequential updates and backtracking, limiting parallelization.  \n      \n    \n\t - **PF updates all particles in parallel, aligning better with modern inference engines** like vLLM.  \n      \n    \n\n -   Empirical Comparison:\n    \n\n\t - We ran MCTS (from Zhang et al. 2024) vs. PF on AIME 2024 with LLaMA3.1-8B and budget 32.  \n      \n    \n\t - PF: 16.6% accuracy, MCTS: 6.6%—**PF outperforms MCTS under equal compute**.  \n      \n    \n\nWe will include this empirical comparison and discussion in the final version.  \n  \n\nReferences:\n\nZhang, D., Huang, X., Zhou, D., Li, Y., & Ouyang, W. (2024). Accessing GPT-4 level Mathematical Olympiad Solutions via Monte Carlo Tree Self-refine with LLaMa-3 8B. https://arxiv.org/abs/2406.07394\n\n\n&nbsp;\n\n_________\nThank you again for your thoughtful review. We hope these clarifications and updates address your concerns and further strengthen your confidence in the work."}]

Author rebuttal:

Meta-review: This paper introduces a novel and effective method for inference-time scaling of LLMs by adapting particle-based Monte Carlo methods to address the issue of "early pruning" in deterministic search. The proposed particle filtering approach robustly maintains a diverse set of candidate solutions, balancing exploration and exploitation to achieve impressive performance gains on various reasoning tasks. The work is well-written, theoretically sound, and supported by a comprehensive set of experiments demonstrating a significantly better scaling rate than existing baselines. While the results are quite strong, the paper could be further improved by expanding its discussion on related work to more clearly differentiate the approach from other sequential Monte Carlo methods (like TSMC) and search algorithms (like MCTS), perhaps including them as empirical baselines. Additionally, including a more detailed analysis of runtime efficiency and confidence intervals for the results would add further weight to the already compelling findings. Nevertheless, the paper presents a valuable and impactful contribution to the field. I would like to recommend accept this paper.
Decision: Accept (poster)