Reasoning with Sampling: Your Base Model is Smarter Than You Think
Authors: Aayush Karan
Yilun Du
Venue: ICLR 2026 Oral (2026)
Keywords: LLMs
reasoning
MCMC
sampling
inference-time compute
Primary area: foundation or frontier models, including LLMs
TL;DR: We find a training-free sampling algorithm that achieves reasoning boosts on base models comparable to those obtained by RL techniques.

Abstract: Frontier reasoning models have exhibited incredible capabilities across a wide array of disciplines, driven by posttraining large language models (LLMs) with reinforcement learning (RL). However, despite the widespread success of this paradigm, much of the literature has been devoted to disentangling truly novel behaviors that emerge during RL but are not present in the base models. In our work, we approach this question from a different angle, instead asking whether comparable reasoning capabilities can be elicited from base models at inference time by pure sampling, without any additional training. Inspired by Markov chain Monte Carlo (MCMC) techniques for sampling from sharpened distributions, we propose a simple iterative sampling algorithm leveraging the base models' own likelihoods. Over different base models, we show that our algorithm offers substantial boosts in reasoning that nearly match and even outperform those from RL on a wide variety of single-shot tasks, including MATH500, HumanEval, and GPQA. Moreover, our sampler avoids the collapse in diversity over multiple samples that is characteristic of RL-posttraining. Crucially, our method does not require training, curated datasets, or a verifier, suggesting broad applicability beyond easily verifiable domains.

Published as a conference paper at ICLR 2026
REASONING WITH SAMPLING:
YOUR BASE MODEL IS SMARTER THAN YOU THINK
Aayush Karan
Yilun Du
Harvard University
Harvard University ydu seas harvard.edu
akaran 8g harvard.edu
ABSTRACT
Frontier reasoning models have exhibited incredible capabilities across wide array of disciplines, driven by posttraining large language models (LLMs) with reinforcement learning (RL). However, despite the widespread success of this paradigm, much of the literature has been devoted to disentangling truly novel behaviors that emerge during RL but not present in the base models. In our work, we approach this question from a different angle, instead asking whether comparable reasoning capabilities
be elicited from base models inference time by pure sampling, without any additional training. Inspired by Markov chain Monte Carlo (MCMC) techniques for sampling from sharpened distributions, we propose a simple iterative sampling algorithm leveraging the base models' own likelihoods. Over different base models, we show that our algorithm offers sub- stantial boosts in reasoning that nearly match and even outperform those from RL. on a wide variety of single-shot tasks, including MATH500, HumanEval, and GPQA. Moreover, our sampler avoids the collapse in diversity over multi- ple samples that is characteristic PL-posttaanning Crucially, our method does not require training, curated datasets, verifier, suggesting broad applicability beyond easily verifiable domains.
INTRODUCTION
Reinforcement learning (RL) has become the dominant paradigm for enhancing the reasoning ca- pabilities of large language models (LLMs) (Guo et al., 2025; Hu et al., 2025). Equipped with a reward signal that is typically automatically verifiable, popular RL techniques have been success- fully applied to posttrain frontier models, leading to sizeable performance gains in domains like math, coding, and science (Hendrycks al., 2021; Li et al., 2022; Rein et al., 2024).
Despite the widespread empirical success of RL for LLMs, large body of literature has centered around the following question: are the capabilities that emerge during RL-posttraining fundamen- tally novel behaviors that are not present in the base models? This is the question of distribution sharpening (He et al., 2025; Shao al., 2025; Yue al., 2025): that is, whether the posttrained distribution is simply "sharper" version of the base model distribution, instead of placing mass on reasoning traces the base model is unlikely generate.
Several works point towards the difficulty in learning new capabilities with RL-posttraining. He al. (2025); Song et al. (2025) compare the pass@k k (multi-shot) scores of base models with post- trained models, finding that for large k, base models actually outperform while the latter suffer from degraded generation diversity. In such cases, RL appears to redistribute pass@k performance to single-shot performance at the expense of multi-shot reasoning. Yue et al. (2025) also notes that the reasoning traces post-RL are tightly concentrated at high fikelitoods/coffdences under the base model. scemingly drawing from existing high- likellikel capabilities. We illustrate this point in own experiments in Figure 4. Regardless, the advantage of RL- posttrraiinng for single-shot reasoning has remained, of yet, undeniable.
In this paper, we present a surprising result: sampling directly from the base model can achieve single-sho reasoning capabilities on par with those from RL.
We propose a sampling algorithm for base models that leverages additional compute at inference time, achieving single-shot performance that nearly matches RL-posttaaining in-domain reason-

---

Published as a conference paper at ICLR 2026
Reasoning
General
80
78.5
2.88
70
60
$7.3
Accuracy (56
53.7
2.0
50
49.6
Score
1.61
1.5
40
39.9
38.9
32.9
30
20
0.5
10
MATI1500
HurnanEval
GPQA
AlpacaEval2.00
Basc (0wen25-MMab-77
GRPO (RL)
Ours (Trainiig-ffre)
Figure 1: Our sampling algorithm can match and outperform RL-posttraining Left: we compare our sam- pling algorithm (ours) against the base model (basc) and CL-posttraining (GRPO) three verifiable reasoning tasks (MATH500, HumanEval, GPQA). Right: compare them unverifiable general task (Alpacali- val2.0). Our algorithm achieves comparable performance to GRPO within the posttraining domain (MATH500) but can outperform on out-of-domain tasks such as HvmanEval and AlpacaEval.
ing tasks and can even oulperform on dom-ii- reasoning tasks. Furthermore, we observe that generation diversity does not degrade with our sampler; in fact, our pass@k k (multi-shot) perfor- mance strongly outperforms RL. We benchmark specifically against Group Relative Policy Opti- mization (GRPO), which is the standard RL algorithm for enhancing LLM reasoning (Shao et al., 2024).
Crucially, our algorithm training-free, dataset-free, and verifier-free, avoiding some of the inher- ent weaknesses of RL methods including extensive hyperparameter sweeps to avoid training insta- bilities, the need to curate a diverse and expansive posttraining dataset, and the lack of guaranteed access ground truth verifier/reward signal (Prabhudesai et al., 2025).
Our contributions can be summarized as follows:
We introduce the power distribution as useful sampling target for reasoning tasks. Since it can be explicitly specified with base LLM, no additional training is required.
ii) We further introduce an approximate sampling algorithm for the power distribution using a Markov chain Monte Carlo (MCMC) algorithm that iteratively resamples token subsequences according to their base model likelihoods.
iil) We empirically demonstrate the effectiveness of our algorithm over range of models (Qwen2.5- Math-7B, Qwen2.5-7B, Phi-3.5-mini-instre
and reasoning tasks (MATH500, HvlanEval, GPQA, AlpacaEval 2.0). Our results show that sampling directly from the base model can achieve results on par with GRPO. In fact, for performs the RI. baseline. Moreover, over multiple samples, we avoid the collapse in diversity afflicting RL-posttrainiigg achieving the best of both worlds in terms of singletto-few-shot rea- soning capabilities as well as sample diversity.
out-of-domain tasks, algorithm consistently out-
Our results collectively illustrate that existing base models are much more capable at single-shot reasoning than current sampling methods reveal.
2 RELATED WORKS
Reinforcement learning for LLMs. RL has been instrumental in posttraining LLMs. Early RL with human feedback (RLHF) (Ouyang et al., 2022) was developed as a technique to align LLMs with human preferences using trained reward model. Recently, RI. with verifiable rewards (RLVR)
2

---

Published as a conference paper at ICLR 2026
has emerged as powerful new posttraining technique, where many works (Guo al., 2025; Lam- bert et al., 2024; Hu et al., 2025: Zeng et al., 2025) discovered that a simple, end-of difficult reason- ing tasks in mathematics and coding. The Group Relative Policy Optimization (GRPO) algorithm was the center of these advances (Shao al., 2024). Building off of this success, many subsequent works have examined using reward signals derived from internal signals such as self-entropy (Zhao al., 2025), confidence (Prabhudesai et al., 2025), and even random rewards (Shao et al., 2025). Similar to these works, our paper examines base model likelihoods as a mechanism for improving reasoning performance, but crucially, our technique training-free.
reward given by automated verifier could substantially enhance performance end-of-generae geggeerrtaion
Autoregressive MCMC sampling with LLMs. Prior works have explored integrating classic MCMC techniques with autoregressive sampling. Many settings including red-teaming, prompt- engineering, and personalized generation can be framed targeting sampling from the base LLM distribution but rilted towards an external reward function. Zhao et al. (2024) proposes learning intermediate value functions that are used in a Sequential Monte Carlo (SMC) framework (Chopin, 2004), where multiple candidate sequences are maintained and updated according to their expected (MH) algorithm, which
future reward. Similarly, Faria et al. (2024) proposes a Hattropoli--axxtiinn instead of maintaining multiple candidates performs iterative resampling, again updating according to expected reward. Methodologically, our sampling algorithm is most similar to this latter work, but the crucial difference is that our target sampling distribution is completely specified by the base LLM, avoiding the need for an external reward.
3 PRELIMINARIES
Let X' be a finite vocabulary of tokens, and let X denote the set of finite sequences of tokens To:2" = (ro, 31,.... ..T) where n; E X' for all i and T 20 is some nonnegative integer. For convenience, for a given 1, let vi = (ro,.... It-1) and u = (It+1,.... ,, ITT), with similar definitions for zigr and Zz(- In general, x refers a token sequence Io.T. where T implicitly given.
Then LLM defines distribution p over token sequences X by autoregressively learning the conditional token distributions p(x, (x)))) for all t, giving the joint distribution via the identity
T
p(ZoT) = II p(x)(=e))
(1)
To sample a sequence from p, we simply sample from the LLM token by token using the conditional distributions, which by (1) directly samples from the joint distribution.
4 MCMC SAMPLING FOR POWER DISTRIBUTIONS
In this section, we introduce our sampling algorithm for base models. Our core intuition is derived from the notion of distribution sharpening posed in Section 1. Sharpen- ing reference distribution refers to reweighting the dis- tribution so that high likelihood regions are further up- weighted while low likelihood regions are downweighted, biasing samples heavily towards higher likelihoods under the reference. Then if RL posttrained models really are just sharpened versions of the base model, we should be able to explicitly specify a target sampling distribution that achieves the same effect.
We organize this section as follows. Section 4.1 presents this target sharpened distribution and provides some math- ematical motivation for why its samples are amenable for reasoning tasks. Section 4.2 introduces a general class of Markov chain Monte Carlo (MCMC) al- gorithms aimed at actually sampling from this target distribution, and finally, Section 4.3 details our specific implementation for L.Mss.
Figure 2: A toy example of distribution
sharpening. Here p is mixture of Gaus- sians, which we plot against pa (a 4.0).
3

---

Published as a conference paper at ICLR 2026
4.1 REASONING WITH POWER DISTRIBUTIONS
One natural way to sharpen distribution p is to sample from the power power distribution pº. Since
pxx)
p(x)
p(x) p(x')
p(x') Ea [1,
p(x')
follows that exponentiating p increases the relative weight on higher likelihood sequences (x)
while decreasing the relative weight on lower likelihood ones (x') (see Figure 2 for visualization).
related but well-known sharpening strategy is low-temperature sampling (Wang et al., 2020), which exponentiates the conditional next-token distributions at each step:
p 1 .....0))
Peomp [2 [o ................................ 21-1) ExjEx p(a//et-11... 1...
(3)
where the temperature is T = 1/a. A common misconception that sampling with (3) over T tokens is equivalent sampling from p"; however, this is false in subtle yet crucial way, illuminate the following.
Proposition 1. Low-temperature sampling does not sample from the power distribution pd.
Proof. We show that the associated conditional next-token distributions are distinct at each timestep t. The conditional distribution on De for pd is given by
Σ Exyr pl.............................. Zt It...., IT)
IT)
Poow (Ie 100 ... It-1) = Σ x2 plzo......
(4)
.I.......... II...., ... )
Using Bayes rule
plao: ,..,, [t,
Σ rzz , It, ,
p(r,12,-1... ... Ho) p(uo. xt-1)
(5)
Σ
Lzze p(zo)...,
we can rewrite the low-temperature marginal (3) as
Piemp (xe Ii ... 11-1) Eirs,(ato "h... a
(6)
En; Exy.
Ignoring normalizations for clarity, the relative weight token n, for sampling from p" is given by a sum of exponents
Poow (x)) x
Σ C(zo...... IT,...,TT)
Meanwhile, the relative weight for low-temperature sampling given by an exponent of sums
Premp (Te (x)))) xx
_pt "...,.,,,.
(8)
Since the relative weights of next-token prediction are distinct for each sampling strategy, it fol- lows that the joint distribution over seqeunces must also be distinct for each sampler. Hence, the distribution on sequences given by low-temperature sampling is not the same as one given by pa
One intuitive way to understand this difference is that low-temperature sampling does not account
for how exponentiation sharpens the likelihoods of "future paths" time step t, instead "greedily"
averaging all these future likelihoods (exponent of sums (8)). On the other hand, sampling from pd inherently accounts for future completions as it exponentiates all future paths (sum of exponents (7))
before computing the weights for next-token prediction. This has the following consequence:
Observation 1. The power distribution upweights tokens with few but high likelihood future paths, while low- tempereter sampling upweights tokens with several but low likelihood completions.
Example 1. We can observe this phenomenon with simple example. Let us consider the token vocabulary X' {a, b) and restrict our attention to two-token sequences (æo, I1): aa, ab, ba, bb. Let p(bb) 0.25,
p(aa) 0.00,
p(ab) 0.40, p(ba) 0.25,
4

---

Published as a conference paper at ICLR 2026
X
1d (x)
Ao
X
xyy
A x')
Naw
(x')
X'y
Ao
A2
Ayy
Figure 3: Illustrating ivetropolis-Hastingg with random resampling. A random index selected and new candidate is generated by resampling, Based on the relative likelihoods, the candidate is accepted or rejected, and the process repeats.
so that
p(1o = a) 0.40,
p(ro 0.50.
Let a = 2.0. Under p², we have
Prow (In = a) x 0.00² + 0.40² = 0.160, Pro b) x 0.25 + 0.25 = 0.125,
so pf prefers sampling over b. Under low- low-temperature sampling.
Premp a) (0.00 + 0.40) 0.160,
Promp (2o = b) x (0.25 + 0.25) 0.250,
preferring sampling b over a. If pd samples zo a. there is only one future path with likelihood 0.40. If Premp samples zo = 6, there are two future paths ba, 6b, but either choice has likelihood 0.25.
In other words, even though has lower conditional likelihood under both p and Premp. pd upweights a and samples the highest likelihood two-token sequence. 6 has many future paths contributing to higher likelihood under p and Piemp- but leads low likelihood sequences. We provide stronger formalization of this phenomenon in Appendix A.2.
Thus, sampling from pd encourages sampling tokens which have fewer but higher likelihood "future paths", as opposed to tokens with several lower likelihood completions. This type of behavior is immensely valuable for reasoning tasks. For example, choosing "wrong" tokens that have high average likelihoods but trap outputs in low likelihood individual futures are examples of critical windows or pivotal tokens (Li al., 2025; Abdin et al., 2024), a phenomenon where few tokens highly influential in the correctness of language model outputs. In fact, sharp critical windows have been shown to correlate strongly with reasoning failures (Li et al., 2025). Instead, embedded in sampling from the power distribution is implicit bias towards planning for future high likelihood tokens.
4.2 THE METROPOLIS SEAROPOLIS-HASTINSS HASTINGS ALGORITHM
Now that have how sampling from pd in theory assist the underlying LLM's ability to reason, our aim now turns towards proposing an algorithm to accurately sample from it. Given an LLM p. we have access the values pd over any sequence length; however, these values unnormalized. Direct sampling from the true probabilities requires normalizing over all sequences (Go...., E XT, which is computationally intractable.
To get around this, we invoke a Markov Chain Monte Carlo (MCMC) algorithm known as Eatropolis-Hastiine (MH) (Metropolis al., 1953), which targets exactly what we want: approx- imate sampling from an unnormalized probability distribution. The MH algorithm constructs Markov chain of sample sequences (x", x! "x.....,'" x") using an arbitrary proposal distribution (xx(x')
to select the next candidate x With probability
(x() q(x')x (x)
A(x, x') min
pa (x') ((x(x))
candidate x is accepted x otherwise, MH sets ++1 x'. This algorithm is especially convenient as it only requires the relative weights given by pd (as the normalization weights in A cancel) and works with any generic but tractable sampler q with minimal restrictions. Remarkably, for large enough n, this process converges to sampling from the target distribution pa under the following (quite minimal) conditions the proposal distribution (Neal, 1993);
5

---

Published as a conference paper at ICLR 2026
Definition 1. The proposal distribution q is irreducible if for any set X with nonzero mass under the target distribution p", q has nonzero probability of eventually sampling from X. The proposal is aperiodic if the induced chain of samples does not return the same sample after a fixed interval number of steps.
Thus, we must simply ensure that our proposal distribution satisfies irreducibility and aperiodicity, and Metropolis- Maastigs takes care of the rest. On practical level, we would also like both (x((x)) and its reverse q(x' x) to be easily computable.
Consider the following family of random resampling proposal distributions (see Figure 3). Let Proup be proposal LLM. With uniform probability J. select a random t E [1. and resample the se- quence starting at index / using Ppro-- Then the transition likelihood g(x[x*) ís simply the likelihood of the resampling. Note that at each candidate selection step, we have a nonzero probability of transitioning between any two sequences x' E X, since with some probability we can always sample as early as the beginning of x. This ensures our proposal distribution is both irreducible and aperiodic. Moreover, (x') (x) is easy calculate by symmetry, since can treat x' as resampled version of x.
With the flexibility endowed by Metropolis-Hastingg, any LLM with any sampling strategy (e.g., low-temperature sampling).
choose the proposal LLM Prrop to be
4.3 POWER SAMPLING WITH AUTOREGRESSIVE MCMC
A direct implementation of Metropolis- Hastings for LLMs would involve initializing with a sampled token sequence of length T, subsequently generating new candidates of length T with (9) over many, many iterations. This process is computationally expensive, however, due to the repeated, full sequence inference calls to the LLM.
In fact, the main downside to MCMC algorithms in practice is the potential for an exponential mixing time (Gheissari et al., 2017), where a poor choice of initialization or proposal distribution can result in an exponentially large number of samples required before convergence to the target distribution. This problem is exacerbated if the sample space has high dimensionality (Bandeira et al., 2022; Schmidler & Woodard, 2013), which is precisely exhibited by the sequence space of tokens XT. especially for long sequences/laree values T..
To remedy this, we propose algorithm that leverages the sequential structure of autoregressive sampling. We define a series of intermediate distributions which we progressively sample from, until converging to the target distribution p". In particular, samples from one intermediate distribution initiate a Metropolis- Settooolis-Haastinss process for the next, helping avoid pathological initializations.
Fix block size B and proposal LLM Pprop: and consider the sequence of (unnormalized) distributions T2B) (22 _ - p( ..., "T)"
0 p( .... BB) - p(
(10)
where p( FKB)) denotes the joint distribution over token sequences of length kB, for any k. For convenience, let ग. denote the distribution given by
TR (T(..)))) X p(
(11)
Suppose we have a sample from TE. To obtain a sample from TE+1, we initialize Metropolis- Hastings process by sampling the next B tokens 1 with Prop- We subsequently run the MCMC sampling procedure for NMMMC steps, using the random resampling proposal distribution q from the previous section. The full details are presented in Algorithm 1.
Note that Algorithm is single-shot: even though multiple inference calls are made, the decision to accept vs. reject new tokens is made purely by base model likelihoods to simulate sampling a single sequence from pa. We can interpret this as a new axis for inference-tie time scaling, as we expend additional compute during sampling to obtain higher quality/likelihood sample.
To quantify the scaling, we can estimate the average number of tokens generated by Algorithm 1. Note that each candidate generation step when sampling from n& (Zo:kB resamples an average of kB tokens, NMCMe times. Summing over all is, the expected number of tokens generated is
[T/B1
kB
NMCMC T
Eyokens = NMMMC
Σ
(12)
2
4B

---

Published as a conference paper at ICLR 2026
Algorithm 1: Power Sampling for Autoregressive Models
Input : base pč proposal Ppropi power o; length T
Hyperparams: block size B; MCMC steps NMCMC
Output : (Zo....,2T] pd
Notation: Define the unnormalized intermediate target
TT x p(ro.B))
2 for f
0 to
B I
do
Given prefix To.AB, we wish to sample from T4+11 Construct initialization x" by extending autoregressively with Pprop:
2t Perup (ar x<));
for kB +1sts st (A: 1)B.
Set the current state x + x".
for n +- 1 to NMCMC do
Sample an index m f1...., (k 1) uniformly.
Construct proposal sequence x' with prefix 20:-1 and resampled completion:
x/} Ppeup (z; x<t),
for m sts (k + 1)B.
Compute acceptance ratio (9)
A(x'x) + min {1, (xxxx) Pprup (x xx) Pprop (x' |x)
{1, Tek (x(x)
Draw u ~ Uniform/ (0, 1);
u S A(x', x) then accept and set < x'
end
10
Set (o:(k+1)B + X to fix the new prefix sequence for the next stage.
1I end
return T0:T
The key tradeoff here is between the block size and number of MCMC steps NMCMC- larger B requires larger "jumps" between intermediate distributions, requiring larger NMCMC to adequately transition. In Section 5, we empirically find value for that makes Algorithm performant for relatively small values of NMCMC.
EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Evaluation. We standard suite of reasoning benchmarks ranging across mathematics, coding, and STEM (MATH500, HumanEval, GPQA), along with ner-verifiable benchmark (AlpacaEval 2.0) evaluating general helpfulness. We evaluate all of our methods and baselines single-shor; i.e., on one final response string.
MATH500: The MATH dataset (Lightman et al., 2024) consists of competition math problems spanning seven categories including geometry, number theory, and precalculus. There are 12500 problems total, with 7500 training problems and 5000 test problems. MATH500 specific randomly chosen subset of the test set standardized by OpenAl.
HumanEval: HumanEval a set of 164 handwritten programming problems covering algorithms, reasoning, mathematics, and language comprehension (Chen et al., 2021). Each problem has an average of 7.7 associated unit tests, where solving the problem corresponds to passing all unit tests.
GPQA: GPQA (Rein et al., 2024) is a dataset multiple-choice science questions (physics, chem- istry, and biology) which require advanced reasoning skills to solve. We use subset GPQA Dia- mond for evaluation, which consists 198 questions which represent the highest quality subset of the GPQA dataset.
AlpacaEval 2.0: The AlpacaEval dataset is a collection 805 prompts (Dubois et al., 2024) that gauge general helpfulness with questions asking e.g., for movie reviews, recommendations,

---

Published as a conference paper at ICLR 2026
MATH500
HumanEval
GPQA
AlpacaEval2.0
Qwen2.5-Math-7B
Base
0.496
0.329
0.278
1.61
Low-temperature
0.690
0,512
0.353
2.09
Power Sampling (ours)
0.748
0.573
0,389
2.88
GRPO (MATH)
0.785
0.537
0.399
2.38
Qwen2.5-7B
Base
0.498
0.329
0,278
7.05
Low-temperature
0,628
0.524
0,303
5.29
Power Sampling (ours)
0.706
0.622
0.318
8.59
GRPO (MATH)
0.740
0.561
0.354
7.62
Phi-3.5-mini-nnstruct
Base
0,400
0.213
0.273
14.82
Low-temperature
0.478
0.585
0.293
18.15
Power Sampling (ours)
0.508
0.732
0.364
17.65
GRPO (MATH)
0.406
0.134
0.359
16.74
Table I: Power sampling (ours) matches and outperforms GRPO across model families and tasks. benchmark the performance of our sampling algorithm on MATH500, EvaanEval, GPQA. and AlpacaEval 2.0. bold the scores of both our method and GRPO, and underline whenever our method outperforms GRPO. Across models, we see that power sampling is comparable to GRPO in-domain reasoning (MATH500), and outperform GRPO
out-of-domain tasks.
and reading emails. The model responses are graded by an automated LLM judge (GPT-4-turbo), which determines preference for the model responses over those from baseline (also GPT-4- turbo). The resulting score is a win rate of model responses normalized for the length of the model response.
Models. To demonstrate the efficacy of sampling algorithm, Math-7B, Qwen2.5-7B, and Phi-3.5-mini-instuue For our RL baselines, we use the implementation of GRPO in Shao et al. (2025), which posttrains these models the MATH training split. For both the Qwen2.5 models, we use the default hyperparameters used to benchmark their performance in of hyperparameters selected from Abdin et al.
the base models Qwen2.5-
Shao et al. (2025). For the Phi-3.5 model, we (2024) that avoids training instabilities and converges to improvement over the base model over a large number of epochs.
Sampling Algorithm. For implementation of power sampling (Algorithm 1), set the max- imum T to be T = 3072 (termination can happen earlier with an EOS token) and block size B 3072 16 192. Empirically, we find 4.0 coupled with proposal LLM Pprop chosen as the base model with sampling temperature 1/a to most performant for reasoning tasks. For Al- pacalEval 2.0, find that having proposal distribution of higher temperature 0.5) improves performance.
5.2 RESULTS
Main results. We display our main results in Table 1, Across base models of different families, our sampling algorithm achieves massive, univ-universal boosts in single-shot accuracies and scores over different reasoning and evaluation tasks that reach, e.g., up to +51.9% Phi-3.5-mini and +25.2% % on MATH500 with Qwen2.5-Math, In particular, on MATH500, which in-domain for RL-posttraining, power sampling achieves accuracies that are par with those obtained by GRPO. Furthermore, on out-of-domain reasoning, our algorithm again matches GRPO GPQA and actually outprrorrs HumanEval by up +59.8%. Similarly, power sampling consistently outperforms on the non-verifiable AlpacaEval 2.0, suggesting a generalizability of our boosts to domains beyond verifiability.
HumanEval with
The surprising success of this fundamentally simple yet training-fret free sampling algorithm under- scores the latent reasoning capabilities of existing base models.
8

---

Published as a conference paper at ICLR 2026
Filter on imput list of strings only for ones that start with given prefix. (Phi-3.5-mini-insttue HumanFval)
Method
Response
Passed
Ours
return [s for ån atrings
s.stertsxich (prefiii)
GRPO
return (epring for string in strings if
seriin.starsswite. (prefix)" x2)
sal
Table Sample responses on Human Phi-3.5-mini-nnstruut.. We present an example where our method solves a simple coding question, but GRPO does not. 5.3 ANALYSIS
We analyze how the reasoning characteristics of power sampling relate to those of GRPO. We present an example in Table 2, with further examples in Appendix A.5.
Reasoning trace likelihoods and confidences. By design, power sampling targets sampling higher likelihood sequences from the base model. In Figure 4, the left graph plots histogram of the out- put sequence log-likelihoods (averaged by length) of the base model, power sampling, and GRPO responses on MATH500, where likelihoods are taken relative to the Qwen2.5-Math-7B base model. Our method samples from higher likelihood regions of the base model, as intended, but still main- tains noticeable spread. Meanwhile, GRPO samples are heavily concentrated the highest likeli- hood peak.
Base
Basc
GRPO
GRPO
Densly
Density
Average Log-Loogee Likelliddd
Average Confidence
Figure 4: Base model (Qwen2.5-Math-7B, likelihoods and confidences for MATH500 responses. Left: We plot the ligeliiissooods (relative the base model) original, power sampling, and GRPO responses over MATH500, Right: We do the same but for confidences relative to the base model. We observe that GRPO samples from the highest likelihood and confidence regions with power sampling close behind, which correlates with higher empirical accuracy.
We also plot the base model confidence of MATH500 responses, defined to be the average negative entropy (uncertainty) of the next-token distributions (Prabhudesai et al., 2025):
Conf(
Σ p(x(( log pgp(())))))
(13)
t=0 ZE
The right plot of Figure demonstrates that our method's and GRPO responses sample from sim- ilarly high confidence regions from the base model, which again correspond to regions of higher likelihood and correct reasoning.
Reasoning trace lengths. Another defining characteristic of RL-posttraining is fong-form reasoning (Guo et al., 2025), where samples tend to exhibit longer responses. On MATH500, --ahh.5-Math- 7B averages response length of 600 tokens, while GRPO averages 671 tokens. Surprisingly, power sampling achieves a similar average length of 679 tokens, without explicitly being encouraged to favor longer generations. This emerges naturally from the sampling procedure.
Diversity and pass @ k performance. Again, notice the peaked and highly concentrated likelihood- s/confidences of GRPO relative to the distributional spread of power sampling in Figure This suggests GRPO exhibits collapse in diversity while sampler does not, aligning with the obser-

---

Published as a conference paper at ICLR 2026
vation that RL- RL-posttrainnig strongly sharpens the base model distribution at the expense of diversity (Song et al., 2025). To quantify the comparative diversity of power sampling relative to GRPO, we can plot the pass @ l accuracy rate, where question is solved if at least of k samples is accu- rate. Figure 5 shows exactly this: unlike GRPO, whose pass @ performance tapers off for large k, power sampling strongly outperforms for k >> 1. Moreover, our performance curve supersedes that of the base model until finally converging in performance. In particular, we are able to achieve GRPO-level single-shot performance without compromising multi-shot performance (see Appendix A.4 for other domains), addressing a long-standing downside to RL- Rl-ttstta
MATH500
Pass @P Accuracy
0.8
0.7
0.6
Ours
0.
GRPO
2
7
10
11
12
14
15
16
k
Figure 5: Pass@k performance MATH500. We plot the @ accuracy (correct if at least of samples accurate) power sampling (ours) (GRPO) relative to the base model (Qwen2.5-Math) 7B). Our performance curve is strictly better than both GRPO and the base model, and our pass rate at high matches the base model, demonstrating sustained generation diversity.
CONCLUSION
In this work, we present an algorithm that samples directly from base model without any additional training or access to an external signal, achieving a single-shot reasoning performance that on par with, and sometimes even better than, that of a state-of tiiie-of-the-atte the-art RL- &L-posttrainiig algorithm. We use the discussion of RI. distribution sharpening to motivate defining the power distribution as a valuable target distribution for reasoning. Although exact power distribution sampling is intractable, we employ classic MCMC techniques alongside the sequential structure of autoregressive generation to define our power sampling algorithm, which demonstrates strong empirical performance.
Our results suggest that base model capabilities are underutilized at sampling time and point towards close relationship between high likelihood regions of the base model and strong reasoning capabil- ities. Employing additional compute at sampling-tin time with a stronger understanding of base model capabilities offers promising direction for expanding the scope of reasoning beyond verifiability.
7
ACKNOWLEDGEMENTS
We gratefully acknowledge support from the Paul and Daisy Soros Foundation, the Kempner Insi- tute, the DoD NDSEG Fellowship. Pickle Robotics, and the Amazon AGI Lab.
REFERENCES
Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Xin Wang, Rachel Ward, Yue Wu, Dingli Yu, Cyril Zhang, and Yi Zhang. Phi-4 technical report. arXiv preprint arXiv:2412.08900 2024. URL https:// ptops://areee wow..oogg/aaao abs alos 2412.242 08905. 8,
10

---

Published as a conference paper at ICLR 2026
Afonso S. Bandeira, Antoine Maillard, Richard Nickl, and Sven Wang. On free energy barriers in gaussian priors and failure of cold start meme for high-dimensional unimodal distributions. arXiv preprint arXiv: 220.:2200000001,,,, 2022. URL https::////... /ww.///aaivvv//aa///////a.../aia org/ haa///a////// 2209. 02001. 6
M. Chen, J. Tworek, H. Jun, Q. Yuan, H. de Oliveira Pinto, J. Kaplan, H. Edwards. Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, Power, L. Kaiser, M. Bavarian, C. Winter, Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Perbert-Voss, W. H. Guss,
B. Chan, Gray, Ryder, M. Pavlov,
A. Nichol, A. Paino, N. Tezak, J. Tang. I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba. Evaluating large language models trained code. CoRR, Abs/2107.033744 2021.
URL https:tt///// //apsi////xvvv.////// org/ /b//aaa////1/// 03374.
Nicolas Chopin. Central limit theorem for sequential monte carlo methods and its application to bayesian inference. The Annals of Statistics, 326))2385- URL https://ppzoecteeeee 2004. doi: 101214/0903344400000111
oxc///ournallsammall----------ap 10.122440090333000000000000
Yilun Du, Conor Durkan, Robin Strudel, Joshua B Tenenbaum, Sander Dieleman, Rob Fergus, Jascha Sohl-Dickstein, Arnaud Doucet, and Will Sussman Grathwohl. Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and meme. In International con-
ference on machine learning, pp. 8489-8510. PMLR, 2023. 14
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. Length-controlled alpacaeval: A simple way debias automatic evaluators. arXiv preprint arX1v:2404.04445, 2024. URL. htw.t::tp///// http:://axxvvvo abs /t.p////44. 2404. 04475.
Gonçalo R. A. Faria, Sweta Agrawal, António Farinhas, Ricardo Rei, José G. C. de Souza, and André R. T. Martins. Quest: Quality-aware metropolis-hhasinng sampling for machine translation. In NeurlPS, 2024. URL httpttt///pcccceeeeiiiiiee/ neurripp cc/ wwtppc/p.ppppcceceeeccce htppp/appprraaa/ firreliee////ee wwlfffeepaappaeeee pappr/ 2022//Elee2222222253333322222222 wwwpee7edddbbbppp-preeeeee Paper Saappr-Cone Coneer--oneeeenceee ppf.
Tomas Geffner, Kieran Didi, Zuobai Zhang, Danny Reidenbach, Zhonglin Cao, Jason Yim, Mario Geiger, Christian Dallago, Emine Kucukbenli, and Arash Vahdat. Proteina: Scaling flow-based protein structure generative models. arXiv preprint arXiv:2503.007 10, 2025. URL https:
//arxiv.org.. abs 2503 00710. 14
Reza Gheissari, Eyal Lubetzky, and Yuval Peres. Exponentially slow mixing in the mean-field swendsen-wang dynamics. arXiv preprint arXiv:1702.0579/1 2017. 6
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. 1,
Andre He, Daniel Fried, and Sean Welleck. Rewarding the unlikely: Lifting GRPO beyond distri- bution sharpening. arXiv preprint arXiv:2506.022 02355, 2025.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. arXiv preprint arXiv:2 103. 038744 2021.
Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Open-reasoner-zeree An open source approach to scaling up reinforcement learning on the base model. arXiv preprint arXiv:2503 24290, 2025. 1.3
Aayush Karan, Kulin Shah, and Sitan Chen. Reguidance: A simple diffusion wrapper for boosting sample quality on hard inverse problems. arXiv preprint arXiv:2506.10955, 2025. URL https: //axxiv.co wtwpwiiw.o./aaa..oo 6506. 10955. 14
11

---

Published as a conference paper at ICLR 2026
Sunwoo Kim, Minkyu Kim, and Dongmin Park. Test-time alignment of diffusion models without reward over-optimization. arXiv preprint arXiv:2501.05803, 2025. URL https: //arivv. org/abs/2501. 05803. 14
Lingkai Kong, Yuanqi Du, Wenhao Mu, Kirill Neklyudov, Valentin De Bortoli, Dongxia Wu, Haorui Wang, Aaron M. Ferber, Yian Ma, Carla Gomes, and Chao Zhang, Diffusion models as con- strained samplers for optimization with unknown constraints. In Yingzhen Li, Stephan Mandt, Shipra Agrawal, and Emtiyaz Khan (eds.), Proceedings of The 28th International Conference on Artificial Intelligence and Statistics, volume 258 of Proceedings of Machine Learning Re- search, pp. 4582 4590. PMLR, 2025. URL https: http:///ooccedddeeee egge wwlg..mmlr......... mlr. press///sss/////// kong25b. html. 14
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tülu 3: Pushing frontiers in open language model bost-training. arXiv preprint arXiv:241.11512, 2024. 3
Krzysztof Latuszyáski, Matthew T. Moores, and Timothée Stumpf-Fétizon. Meme for multi-modal distributions. arXiv preprint arXiv:2501.05908, 2025. URL https:// /ttppii////xvvv/o/////. org/ /bs/ 2501. 05908v1. 14
Marvin Li, Aayush Karan, and Sitan Chen. Blink of an eye: A simple theory for feature localization in generative models. arXiv preprint arXiv:2502.0092/ 2025. URL ttttps://arxiyy
abs/2502 00921. 5, 14
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d'Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel Mankowitz, Esme Sutherland Robson, Push- meet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-level code generation with AlphaCode. arXiv preprint 2tt.:22203.78144 2022.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2024. URL https: /oppnreview.....
net/fozee feet/forumaan (
Nicholas Metropolis, Arianna W. Rosenbluth, Marshall N. Rosenbluth, Augusta H. Teller, and Ed- ward Teller. Equation of state calculations by fast computing machines. Journal of Chemical Physics, 1166):107711092, 1953. doi: 10.1063/1.1699114
Radford M Neal. Probabilistic inference using markov chain monte carlo methods. Department of Computer Science, University of Toronto (review paper / technical report), 1993.
Radford M. Neal. Annealed importance sampling. arXiv preprint physics/9803008, 1998. URI alss
https://arxiv..r http:///rrr//a////scss////////// httprrsccc//00000 14
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin,
Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. In NeurIPS, volume 35, pp. 27730-27742 2022. 2
Mihir Prabhudesai, Lili Chen, Alex Ippoliti, Katerina Fragkiadaki, Hao Liu, and Deepak Pathak. Maximizing confidence alone improves reasoning. arXiv preprint arXiv:2505.22666, 2025. URL
https: arxiv. org/ abs 220.2225...2....2222222 22660. 2, 3.9
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. GPQA: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024. 1,7
Malcolm Sambridge. A parallel tempering algorithm for probabilistic sampling and optimiza- tion. Geophysical Journal International, 196(1):357-377 2014. doi: 10. 10.1093/gi//gtt374. URL https: //w.a//accmmcc/// c. oup wttpccccc/...aeeee http:///aaaaaaaa////////////////
12

---

Published as a conference paper at ICLR 2026
Scott C. Schmidler and Dawn B. Woodard. Lower bounds the convergence rates of adaptive meme methods. Technical report, Duke University Coreell University, 2013. URL https: AS pdf.
/ //www2 stat duke. aachu-ssca http:///ccceeeeeee/aaatttteteeee
Rulin Shao, Shuyue Stella Li, Rui Xin, Scott Geng, Yiping Wang, Sewoong Oh, Simon Shaolei Du, Nathan Lambert, Sewon Min, Ranjay Krishna, Yulia Tsvetkov, Hannanch Hajishirzi, Pang Wei Koh, and Luke Zettlemoyer. Spurious rewards: Rethinking training signals in RLVR. arXiv 3.
preprint arXiv:2506.10095 2025.
Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Wu, and Daya Guo. Deepseek-man math: Advancing mathematical reasoning through step-by-step exploration. arXiv preprint arXiv:2404.01144 2024. 2,
Marta Skreta, Tara Akhound-Sadegh. Viktor Ohanesian, Roberto Bondesan, Alán Aspuru-Guzia Arnaud Doucet, Rob Brekelmans, Alexander Tong, and Kirill Neklyudov. Feynman-kac correc- tors in diffusion: Annealing, guidance, and product of experts. arXiv preprini arX1v:2503.02819
2025. URL https: //wpiv/aaivvv///a////i//eaaaaiia wrg.abs///03...o.. 02819. 14
Yuda Song, Julia Kempe, and Rémi Munos. Outcome- based exploration for LLM reasoning. arXiv preprint arXiv:2509.0699 2025. URL
06941. 1,
Pei-Hsin Wang, Sheng-lou Hsieh, Shih-Chich Chang, Yu-Ting Chen, Jia- Yu Pan, Wei Wei, and Da- Chang Juan. Contextual temperature for language modeling. arXiv preprint 2rXiv:20/2.13555.
2020. URL https:///// arxiv org/ abs/////////// httgg//bs/022.11..3 22222
Yanwei Wang, Lirui Wang, Yilun Du, Balakumar Sundaralingam, Xuning Yang, Yu-Wei Chao, Claudia Pérez-D' Arpiio, Dieter Fox, and Julie Shah. Inference-time policy steering through human interactions. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pp. 15626-15633. IEEE, 2025.
Yanbo Xu, Yu Wu, Sungjae Park, Zhizhuo Zhou, and Shubham Tulsiani. Temporal score rescaling for temperature sampling in diffusion and flow models. arXiv preprint arXiv:2510.0118% 2025. URL https::/aaat
wwwpxvvv.rr//oo.o... abs/ 01184. 14
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does re- inforcement learning really incentivize reasoning capacity Ilms beyond the base model? arXiv preprint vrXiv:2504.1388 2025. URL /tttpp:://asaa org/ /bb/
13837..
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Sim- plerizoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503. 22.5503.18999 18892, 2025, URL https: / /aaxxvivvv.///////////aaaaaaaaii org/ abaa/bbs555 18892.
Xiangcheng Zhang, Haowei Lin, Haotian Ye, James Zou, Jianzhu Ma, Yitao Liang, and Yilun Du. Inference-time scaling of diffusion models through classical search. arXiv preprint arXiv:2505.23614 2025. 14
Stephen Zhao, Rob Brekelmans, Alireza Makhzani, and Roger Grosse. Probabilistic inference in language models via twisted sequential monte carlo. arXiv preprint arXiv:2404.17446 2024. URL htRps://s//aaat org/ abs/2404 17546.
Xuandong Zhao, Zhewei Kang, Aosong Feng, Sergey Levine, and Dawn Song. Learning to reason without external rewards. arXiv preprini arXiv:2505. 19590, 2025. URL https:
brg/abs/2505.0 19590.
13

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.I MORE RELATED WORKS
Annealed sampling for difflusion. In the statistical physics and Monte Carlo literature, sampling from pd is known as sampling from an annealed, or tempered, distribution (Neal, 1998) and has inspired a new wave of interest within the diffusion community. Indeed, in traditional MCMC sampling, annealing used as way to avoid mode-collapse during sampling and more accurately sample from complex multimodal distributions (Eatuszyński et al., 2025). This has re-emerged as inference-time sampling methods for diffusion that aim to steer a pretrained model towards "tilted distributions" (Du et al., 2023; Kim et al., 2025; Karan et al., 2025; Wang et al., 2025; Kong et al., 2025: Zhang ct al., 2025). Where traditional RL techniques cxhibit mode collapse, applications in the physical sciences (Sambridge, 2014) require multimodal sampling. To this end, works such as Du et al. (2023); Wang et al. (2025); Kim et al. (2025) construct sequences of annealed distributions to ease the transition from base diffusion distribution to tilted distribution. Other works (Skreta et al., 2025; Xu et al., 2025) intentionally target sampling from pd for a > 1 as a means generating higher quality samples from the base diffusion model, which is particularly popular for generating more designable proteins (Geffner et al., 2025).
A.2 ADDITIONAL THEORETICAL DISCUSSION
In this section, we provide a stronger formalization of the phenomenon that power sampling down- weights tokens that trap outputs in low-likelihood futures while low- low-temperature sampling does not. Proposition 2 (Informal). Power sampling upweights tokens with small support but high likelihood completions, while low-temperature sampling upweights tokens with large support but low likeli- hood completions.
Definition 2. For the rest of this section, fix a prefix 10 We say that T4 has marginal weight under the conditional next-token distribution if E(zo.( Et,, IT)
We consider a simplified model of the "critical window" or "pivotal token" phenomenon (Li et al., 2025; Abdin et al., 2024), which refers to intermediate tokens that strongly influence the quality of the final generation. We differentiate between pivotal tokens that lead to high-likelihood futures vs. low-likelihood ones.
Definition 3. At one extreme, a pivotal token maximally induces a high- lighllikelihood completion if it places its entire marginal weight 6 on one future (singular support); i.e., for only one choice of x > is p(zo. p(z.....,.I....... ... .T) nonzero. We call such a token a positive pivotal token.
Definition 4. At the other extreme, a pivotal token minimizes the likelihood of any future its entire marginal weight & is uniformly distributed across N future completions. In other words, there exist Tt....,, are all nonzero with likelihood N- We call
N completions x > such that such a token a negative pivotal token.
Our simplified model of high and liw-likelihood futures examines when positive pivotal tokens are favored over negative pivotal tokens under a given sampling distribution. In particular, we show that power sampling can upweight positive pivotal token over a negative one even if the latter has higher marginal weight, whereas low-temperature sampling always upweights the negative pivotal token in such a scenario.
Of course, whenever a positive pivotal token has higher marginal weight, both power sampling and now-temperature sampling will upweight it.
Proposition 3. Let x4 be a positive pivotal token with marginal weight e, and let x be a negative pivotal token with marginal weight e' and support N. Then if
NI-1/a <E<E',
(14)
the future likelihood of xt is higher than any future likelihood of x'c- Moreover, power sampling upweights En over x¹ while low-temperature sampling upweights xf over It-
Proof. Since a 2 follows that
(15)
N1 1/a
N
14

---

Published as a conference paper at ICLR 2026
and thus N. establishing that the future completion likelihood of xı is greater than that of z/ (i.e. the assignment of positive and negative pivotal tokens is consistent).
Now, i E', then under the teweeempertture distribution, the relative marginal weights aı and Iy are =" and E"a, so the probability of choosing F downweighted relative to x/z. However, for the power distribution, the relative marginal weights are Ppow (ze( = Ex and Poow (2) (<))) = N Then, as long as E" Na-1
N1-1/a token 2; will be upweighted relative token x/c-
In other words, the marginal weight on 2, can be less than the mass on xy under p. but if the completion for has higher likelihood than any individual completion for T/y power sampling favors Zt over x/y.
A.3 HYPERPARAMETERS FOR POWER SAMPLING
am 1.0
10.0
M-ath-B Qwar2.5-78
748
21.0
69.4
70,
6M2
630
amm 4.0
MA T1111000 Accuracy (1))
50.3
45.0
35.8
Qwen2.5 Math-BB
Qwen2.5-7B
Phi-3.5 1...3.5miiii-oem mini-insttratt
MCMC
Figure 6: Effect of hyperparameters power sampling. Left: We plot MATH500 accuracy across model familics for various values a. Right: We plot the increase in accuracy of power sampling on Qwen models as the number of MCMC steps increases.
The effect of power distributions. The two most important hyperparameters for power sampling the choice of a and the number of MCMC (resampling) steps during sequence generation NMCMe- At the extremes, choosing a = 1.0 samples from the base model directly, while taking - do has the effect of deterministically accepting any resampled sequence that strictly increases the likelihood. Of course, even though higher base model likelihoods correlate with better reasoning (Figure 4), directly optimizing for likelihood not necessarily optimal for reasoning, suggesting an ideal intermediate value of a.
In Figure 6, we display MATH500 accuracies across various values of and find that an intermediate 4.0 outperforms other values, as expected. Noticeably, the accuracies of power sampling remain relatively stable beyond a 2.0, suggesting that power sampling in practice is relatively robust to the choice of a.
Test-time scaling with MCMC steps. On the other hand, NMMMe toggles the inference-time com- pute expended by our algorithm, providing natural axis for test-time scaling. Section 4.3 we raised the notion of mixing time, or the number of MCMC steps required before adequately sam- pling from the target distribution. our case, we expect that the fewer MCMC steps we take, further our algorithm samples from the target n
We plot performance dependence on NиCVC Figure 6 and notice a steady increase in accuracy until NMCMC = 10, beyond which accuracy remains roughly stable (not plotted). The accuracy difference from using fewer MCMC steps noticeable but no more than 3-4% between NMCMC and NMCME 10. However, the jump in accuracy by using at least two steps opposed to none is substantial (3-4%).
We compute the total amount of tokens generated by our method relative running GRPO.
From (12), our sampler generates тø NMMMeT times as many tokens as standard inference to gen- erate sequence of length T. Plugging in experimental parameters NMCMC 10, T 679 (our
15

---

Published as a conference paper at ICLR 2026
average output length for MATH500) and B 192, running inference with power sampling incurs a multiplier of 8.84x the number of tokens as rumning standard inference. Since GRPO generates method incurs roughly the same inference cost one epoch of GRPO training, assuming 8 rollouts per sample with identical dataset sizes. Typically though, one GRPO epoch is still more expensive uses 16 rollouts and training set that is larger than MATH500.
multiple rollouts per example during training,
A.4 @ASS@K K ACCURACIES OVER MULTIPLE DOMAINS
In this section, we plot the pass performance of power sampling, GRPO, and the base model (Qwen2.5-Math-77B over MATH500, GPQA, and HumanEval demonstrate that our sampling algorithm is highly performant at both single-shot and multi-shot reasoning while maintaining re- sponse diversity. Power sampling is plotted with a 4.0 for MATH500 and GPQA and a 1.67 for HumanEval (this temperature exhibits slightly better results at earlier k). In all cases, both in- domain and out-of-domain for GRPO, power sampling has near universally better performance than both GRPO and the base model in pass @ for k and matches, if not exceeds, the base model upper bound large k.
MATH500
0.9
Pass @@ Accuracy
0.8
0.7
Ours
0.5
Base
GRPO
0.4
4
5
6
7
10
11
12
13
14
15
Figure 7: Pass@l performance on MATH500 (Qwen2.5-Math-7B)
One thing to note about these plots is that the loss in diversity varies noticeably from benchmark to benchmark. MATH500 and GPQA clearly show that GRPO has a significantly lower pass @ performance and diversity than the base model until & = 16. We speculate this might be due to the fact that while MATH500 "singular" answer, HumanEyal is based
for smaller while HumanEval GRPO exhibits better pass @k
and GPQA graded multiple solutions might be accepted, indicating a reduction in diversity may emerge but at much higher This of course varies model-to-model; in fact, from Table 1, GRPO leads reduction in performance on HumanEval relative to the base model Phi 3.5 mini. We noticed that a majority of failures are at least due formatting issues of the base model being sustained: 76.05% of failures for GRPO are marked for failing to adhere to unindents, compared to 81. 31.40% for the base model. Such errors not constitute failures of power sampling however.
completing code where
A.5 MORE QUALITATIVE EXAMPLES
16

---

Published as a conference paper at ICLR 2026
HumanEval
0.8
Pass @@ Accuracy
0.7
0.5
Ours
0.4
Base
GRPO
0.3
3
4
5
6
7
10
11
12
13
14
15
16
k
Figure 8: Pass@k performance on HumanEyal (Qwen2.5-Math-7B)
GPQA
1.0
Ours
0.9
Base
GRPO
Pass @k Accuracy
0.8
0.7
0.6
0.5
0.3
3
4
5
6
7
10
11
12
13
14
15
16
k
Figure 9: Pass@A l: performance on GPQA (Qwen2.5-Math-77)
17

---

Published as a conference paper at ICLR 2026
Can you solve the following math problem? Write 3 20
as decimal. Please reason step by step,
and put your final answer within |boxed (}. Correct answer: 0.15
System
Answer
convert the fraction
20 to decimal, we can follow steps:
1. Identify the numerator and the denominator the fraction. In this case, the numerator is and the denominator is 20. 2. Divide the numerator by the denominator. In this case, divide 3 by 20. Perform the division. The result is 0.15.
Ours
Therefore, the fraction 20 decimal is 0.15
To convert the fraction 20 to decimal,
follow these steps:
1. Understand the fraction: The fraction 3 means 3 divided by 20.
zú 2. Perform the division: We need to divide by 20. Since 3 is less than 20, will add decimal point and
3 make the division possible.
3. Set up the division: Write as 3.000 (adding three zeros for precision).
GRPO
4. Divide: 20 goes into 30 once (since 20 1 20). Subtract 20 from 30 to get a remainder of 10. Bring down the next 0 to make it 100. 20 goes into 100 five times (since 20 5 100). Subtract 100 from 100 get remainder of O.
5. Combine the results: The quotient from the division 0.15.
Therefore, the decimal representation of 3 is 0.15
20
Table 3: Comparison on MATH500: Qwen2.5-Math-7B We present an example where our method and GRPO are correct.
The Fib4 number sequence a sequence similar to the Fibonacci sequence defined as: fib4(0)=00, fib4(1)=0, fib4(2)=2, fib4(3)=0,, and for n 2 4, fib4(n) fib4(n-1) + fib4(n-2) fib4(n-3) + fib4(n-4). Write a function efficiently compute the n-th element without recursion.
System
Completion
Passed
Base case5
return 0
elif
return
elif 2:
return
elif su 3:
initialize the four fib4_valume values = [0, 0, 2, 3]
Ours
true
# Compute the for in range ntll: next_velue (Iibi_values[1
the sequence iteratively
ribe_values[i
-2]
fib4_values (i-3] fib4_value[i
-4))
sib4_veluss.o aibd_seluss.appen (next_value)
# Return the a-th element Eibd Fibl_values
d 0, 0, 2, for in range (n): d, e, return
GRPO
false
Table 4: HumanFval comparison on Phi-3.5-mini-nnstmue
18

---

Published as a conference paper at ICLR 2026
Can you solve the following math problem? What integer n satisfies S 18 and
-11213141 (mod 18) ?
Please reason step by step, and put your final answer within \boxed( Correct answer: 13 Method
Answer
First, we perform the division 11213141 by 18:
11213141 18 622952.277777
The integer part of the quotient 622952. Now, multiply 622952 by 18 find the product:
622952 18 11213136
Next, subtract this product from 11213141 find the remainder:
11213141 11213136 5
So, the remainder when 11213141 is divided by 18 is 5. This means:
Ours
11213141 = 5 (mod 18)
Therefore:
-11213141 = - (mod 18)
Since we need non-negative remainder, we convert to positive equivalent by adding 18:
-5
Thus:
- -11213141 13 (mod 18)
The integer n that satisfies 0 S n < 18 and n = 11211311 (mod 18) is: 13
Step Find remainder when 11213141 is divided by 18. We can use the property modular arithmetic that states a = 6 (mod m) if and only if a mod m = b mod m. So, we need to compute 11213141 mod 18.
To simplify this, can the fact that the remainder of number when divided by 18 is the same the remainder of the sum of its digits when divided by (although this is not always the most efficient method, can be used here for simplicity).
First, let's sum the digits of 11213141:
1+ 2 1 1-3+1=++11+ 14.
Now, we need to the remainder when divided by 18. Since remainder is simply 14. Therefore,
than 18, the
GRPO
11213141 = 14 (mod 18).
Step Find the remainder (mod 18), it follows that
11213141 divided by 18. Since 11213141
-11213141 = 144 (mod 18).
We need to convert -14 a positive remainder between and To this, we add to 14:
+ =
Therefore,
-11213141 = (mod 18).
The integer n that satisfics 0 S < 18 and n = -1 11213111 (mod 18) is 4
Table 5: MATH500 comparison between sampling algorithm and GRPO for Owen2.5-Math-2 Here is example where GRPO gets incorrect answer, while our sampling algorithm succeeds. Our sample answer uses a distinct method altogether.
19


Reviews:
[{"review_id": "eTj8b7DmBd", "reviewer": "Reviewer_L8G7", "summary": "- The paper shows that strong reasoning performance can be achieved from base LLMs without any additional training by changing how we sample from them.\n- It introduces a power sampling algorithm that draws from a sharpened distribution $p_{\\alpha}(\\mathbf{x}) \\propto p(\\mathbf{x})^\\alpha$, emphasizing higher-likelihood reasoning traces.\n- The method uses an MCMC-style resampling process to approximate sampling from this distribution efficiently at inference time.\n- Across several benchmarks such as MATH500, HumanEval, and GPQA, this training-free approach matches or surpasses GRPO-based post-training.\n- The results suggest that base models already encode strong reasoning abilities, and that better inference-time sampling can unlock them without retraining.", "questions": "- How can $\\alpha$ be selected practically and automatically without extensive tuning?\n- Does power sampling remain effective for longer reasoning chains or multi-turn dialogues?\n- How does the diversity of outputs change with increasing $\\alpha$ and MCMC steps, and is there a risk of collapsing to repetitive or overconfident generations?\n- Have you considered evaluating the proposed method on more challenging or diverse reasoning benchmarks such as MMLU?\n- Could you provide insights into why $\\alpha = 4.0$ consistently works well across tasks?\n- Have you considered using tempered transitions within your MCMC framework to improve chain mixing, especially when sampling from highly sharpened distributions for long reasoning sequences?\n- In Line 17, Line 46, and more, capabilites -> capabilities\n- In Line 356, algorihtms -> algorithms\n- The figures use multiple shades of blue that differ only in brightness, making it difficult to distinguish between lines or bars, especially when printed or viewed in grayscale. The authors are encouraged to adopt a more distinguishable color palette.\n- Use \\citep and \\citet appropriately.\n- Ensure that the citation formats are consistent, the capitalization is correct, and the reference information is up-to-date.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The method achieves RL-level reasoning performance without any additional training, data, or reward signals.\n- It provides a clear theoretical analysis showing why power sampling differs fundamentally from conventional low-temperature sampling.\n- The algorithm is simple, mathematically principled, and broadly applicable to existing base language models.\n- The work reframes reasoning enhancement as an inference-time sampling problem rather than a post-training or reward-learning problem.", "weaknesses": "- It depends on tuning the power factor, block size, and number of MCMC steps, which may limit plug-and-play usability.\n- The algorithm provides no theoretical guarantee of convergence to the true power distribution in large sequence spaces.\n- The paper evaluates only single-shot responses and does not test multi-turn reasoning or chain-of-thought extensions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BCLVWVAb5N", "reviewer": "Reviewer_mX2n", "summary": "This paper proposes a training-free method to improve reasoning in large language models (LLMs) by sampling from a sharpened distribution of the base model. Inspired by ideas from Markov chain Monte Carlo (MCMC), the authors introduce an inference-time sampling algorithm that approximates sampling from a power distribution, where higher-likelihood sequences under the base model are exponentially upweighted.", "questions": "- Have the authors evaluated Power Sampling on open-ended tasks like story writing or dialogue? If not, what properties of reasoning tasks (e.g., verifiability, objective correctness) make sharpening effective?\n- Is a higher base model likelihood always desirable? Could overly sharp distributions lead to degenerate or repetitive text, as seen in low-temperature sampling?\n- In my own experiments, sharpened sampling does not seem to improve autoregressive image generation. Do the authors believe that *reasoning tasks* uniquely satisfy conditions that make $p^{\\alpha}$ beneficial?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper offers a fresh conceptual lens by interpreting RLVR-based improvements as a form of distribution sharpening, reframing how we think about reasoning emergence in LLMs.\n- The authors demonstrate that a simple, training-free algorithm can match or surpass GRPO-trained models across multiple benchmarks and model families.", "weaknesses": "As noted by the authors, the method can only exploit existing capabilities of the base model—it cannot exceed its representational capacity. A deeper discussion or analysis of the top-k ceiling would help contextualize the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CMME9dcpy2", "reviewer": "Reviewer_4b7A", "summary": "This work investigates how distribution sharpening can improve reasoning without additional training. The authors introduce a simple iterative sampling algorithm that leverages the base model’s own likelihoods to refine generation. Across three different base models and a range of benchmarks, the proposed method achieves competitive performance with alignment-trained models—while requiring no post-training or additional data.", "questions": "- Why does applying GRPO to Phi-3.5-mini-instruct result in performance degradation on two math benchmarks? The authors should further analysis the underlying cause of this phenomenon.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper reframes alignment and reasoning improvement as a distribution-sharpening problem, bridging the gap between post-training RL methods and pure inference-time algorithms.\n- The proposed iterative sampling algorithms is simple, straightforward but exceptionally effective.\n- The authors conducted solid experiences across 3 base models and several benchmarks to demonstrate the proposed method's effectiveness.", "weaknesses": "- The experiments are only conducted on small models such as Qwen2.5-7B. Experiments on more advanced models and larger models can further solidify the empirical exploration. But it is understandable given the resources in academia.\n- The evaluation spans four benchmarks, which, while representative, do not fully capture the diversity of reasoning and generation tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bdEeeuGd0K", "reviewer": "Reviewer_nAW4", "summary": "a new inference scaling method that can match the RL performance by spending cost during decoding, but does not require training, verifiers, datasets.", "questions": "- why was there a decrease in performance when increasing the MCMC steps, was this more noisy as high likelihood samples could also be wrong? could be good to provide distributional stats for plot 5 b.\n- did you try your method on top of GRPO'd models, what will happen?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- the method is conceptually simple and straightforward\n- the method does not require any training, data or verifier, and is entirely formed by the base model's own distribution", "weaknesses": "- need to pay cost for each sample, and if not paying for any inference scaling cost degenerates to sampling from p.\n- the evaluated benchmarks are rather older. do we expect the method to be still tractable with harder datasets that require longer CoT lengths? at least evaluate AIME25 or better HMMT25/BRUMO25, which are common math reasoning benchmarks to be tested on", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)