Title: Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problems
Venue: NeurIPS 2025
Accepted: yes
Keywords: reinforcement learning

llm

pass at k

inference time compute

monte carlo

gradient estimation
Abstract: Reinforcement Learning algorithms commonly sample multiple ($n>1$) solution attempts for each problem and reward them independently. This optimizes for pass@1 performance and prioritizes individual sample performance over the diversity and collective utility of a set of samples. Such algorithms under-utilize the sampling capacity, limiting exploration and eventual improvement on harder examples. As a fix, we propose Pass-at-$k$ Policy Optimization (PKPO), a multivariate transformation on batches of rewards which leads to direct optimization of \passk\ performance, thus optimizing for sets of samples that feature a large maximum reward when considered jointly. Our primary contribution is to derive novel low variance unbiased estimators for the pass@k and its gradient, in both the binary and continuous reward settings. We show that optimizing with these estimators reduces to reinforcement learning with (batches of) rewards that have been jointly transformed by a function that is stable and efficient to compute.

While previous efforts propose transformations for $k=n$, our transformations are the first to enable robust optimization of the pass@k for any arbitrary $k \leq n$. Rather than simply trading off pass@1 performance for pass@k gains, our method allows annealing $k$ during training, optimizing both metrics and often achieving strong pass@1 performance alongside significant pass@k gains.

We validate our transformations on illustrative toy experiments, which reveal the variance reducing properties of our formulations. We also include real-world examples using the open-source models Gemma and Llama . We find that our transformation effectively optimizes for the target $k$. Furthermore, higher $k$ values enable solving more and harder problems, while annealing $k$ boosts both the pass@1 and pass@k. Crucially, for challenging task sets where conventional pass@1 optimization stalls, our pass@k approach unblocks learning, likely by improving exploration through the prioritization of joint utility over the utility of individual samples

Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problems

Abstract

Reinforcement Learning algorithms commonly sample multiple ( n > 1 ) solution attempts for each problem and reward them independently. This optimizes for pass@1 performance and prioritizes individual sample performance over the diversity and collective utility of a set of samples. Such algorithms under-utilize the sampling capacity, limiting exploration and eventual improvement on harder examples. As a fix, we propose Pass-atk Policy Optimization ( PKPO ), a multivariate transformation on batches of rewards which leads to direct optimization of pass@k performance, thus optimizing for sets of samples that feature a large maximum reward when considered jointly. Our primary contribution is to derive novel low variance unbiased estimators for the pass@k and its gradient, in both the binary and continuous reward settings. We show that optimizing with these estimators reduces to reinforcement learning with (batches of) rewards that have been jointly transformed by a function that is stable and efficient to compute.

While previous efforts propose transformations for k = n , our transformations are the first to enable robust optimization of the pass@k for any arbitrary k ≤ n . Rather than simply trading off pass@1 performance for pass@k gains, our method allows annealing k during training, optimizing both metrics and often achieving strong pass@1 performance alongside significant pass@k gains.

We validate our transformations on illustrative toy experiments, which reveal the variance reducing properties of our formulations. We also include real-world examples using the open-source models GEMMA2 and LLAMA3.1. We find that our transformation effectively optimizes for the target k . Furthermore, higher k values enable solving more and harder problems, while annealing k boosts both the pass@1 and pass@k . Crucially, for challenging task sets where conventional pass@1 optimization stalls, our pass@k approach unblocks learning, likely by improving exploration through the prioritization of joint utility over the utility of individual samples

1 Introduction

Recent years have seen the rapid rise of large language models (LLMs) trained with internet-scale pretraining data [RNS + 18] with post training using both supervised fine-tuning [WBZ + 21] and reinforcement learning (RL) [AAA + 23, Tea23, Ant, GYZ + 25]. The seminal paradigm of RL with human feedback [CLB + 17] is limited by the human-derived data it is based on and the reward hacking issues that arise from the use of subjective signals more generally [ABC + 21]. To enable progress toward superhuman capabilities, current work is focusing on grounded reward signals that are free of fine-grained human input as in code generation [SJTR23, LWG + 22, DLJ + 24, YTC + 23, GZC + 24] and mathematics [LCC + 22, AT, CTO + 25, YSG + 23].

0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

x

and

0.0

0.2

0.4

0.6

0.8

1.0

g

@

k

(a function of

)

k

=1

k

=2

k

=4

k

=8

k

=16

0.0

0.2

0.4

0.6

0.8

1.0

raw reward

g

(a function of

x

)

g

@

k

for

x

N

(

, 0.1) and various

k

g

(

x

)

(a)

g

(

x

)

and

g

@k

.

0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

x

and

7

6

5

4

3

2

1

0

1

2

g

@

k

(a function of

)

k

=2

k

=4

k

=8

k

=16

0.0

0.2

0.4

0.6

0.8

1.0

raw reward

g

(a function of

x

)

g

@

k

for

x

N

(

, 0.1) and various

k

g

(

x

)

(b)

g

(

x

)

and

∇

θ

g

@k

.

The policy gradients family of RL methods [Wil92] has proven effective in language model training [GYZ + 25]. To scale to new capabilities, model training needs to tackle challenging RL task sets with no known solutions but for which correctness may be verified, as in formal mathematics environments [AT]. In such settings, the RL training loop both updates model parameters and searches for solutions to problems at the continuously advancing frontier of model capabilities.

The specific search algorithm introduces a coupling between inference and model updates, which means that naively optimizing the expected single sample reward, or pass@1 may be suboptimal. While various inference-time search methods are possible [HYM + 24, KZA + 24, LKB + 23, WSL + 24], simply taking multiple independent samples from the model has proven rather effective [OIW + 23]. Our contribution is to couple this simple search method with model parameter updates by enabling robust optimization of the pass@k objective, which is the expected maximum reward over k independent samples.

Related Literature The pass@k was championed by [CTJ + 21a] who gave a popular unbiased estimator of the metric which we derive from a new perspective (to set up our gradient estimators) as Theorem 1, generalize to continuous rewards in Theorem 3, and provide additional characterisations in Corollaries 2 and 3. Concurrently with our work [TZSM25] offered an elegant variance reduction method for the gradient of the pass@k which corresponds to the special case n = k of our Equation (33). Interpreting pass@k in terms of a partial sort, [CTV19] and [XDC + 20] present elegant approximations that are rather general but less efficient in our setting. Others have provided variational approximations for handling the closely related Best-ofN [CTG + 24, AVAC24] and other more general [BSB + 24] inference-time algorithms. The contrasting idea of training a model to approximate the Best-ofN prediction with a single sample was addressed by [SDH + 24]. Our contribution can be interpreted as a generalization and variance reduction of [TZSM25]. For a general discussion of gradient estimation, variance reduction, and Monte Carlo, we recommend [MRFM20, Owe13].

Overview and Contributions Our theme is constructing robust estimators of the pass@k and its gradient given n ≥ k samples by averaging (over all ( n k ) subsets of size k ) simple estimators that are functions of k samples. This is straightforward for binary rewards (Section 2), using the counting proof of Theorem 4. We generalize to continuous rewards using the key trick of assuming without loss of generality that the rewards are sorted, as in Section 3. Finally, we give baselining methods that require more involved derivations due to averaging over all subsets that do not include a given element (to retain unbiasedness) but which boil down to the same easy-to-apply results in Section 4, yielding our Pass-atk Policy Optimization ( PKPO ). We present toy experiments in Section 5.1 which demonstrate the variance reduction afforded by our estimators. Finally, Section 5.2 demonstrates

that using our reward transformation solves more tasks and selectively optimizes pass@k through RL experiments on GEMMA2 [TRP + 24] and LLAMA3.1 [GDJ + 24], showcasing real-world impact.

How to Apply this Method It is easy to adapt any policy gradient algorithm to use our results. Assume a vector ( g ( x 1 ) , g ( x 2 ) , . . . , g ( x n )) glyph[latticetop] of per-sample rewards for a given task. For example, the x i could be model samples of source code addressing a specific task (which should be the same for all n samples), and g could provide a numeric score that measures how many tests the code passes, or an overall binary pass indicator, or some combination with additional stylistic or brevity terms, etc. Then in order to optimize the pass@k of Equation (1) (or the continuous analog max g @k of Equation (11)) we simply transform the vector of rewards using either the sloo or the sloo minus one function of Listing 1, which map R n ↦→ R n . 1

2 Binary Rewards

Given a binary reward function f : X → { 0 , 1 } on the action space X , the pass@k for the model p ( x | θ ) is the probability that at least one of k samples drawn i.i.d. is correct:

where the expectation is over i.i.d. x 1 , x 2 , · · · , x k ∼ p ( x | θ ) .

2.1 An Unbiased pass@k Estimator

An estimator for the pass@k was given in [CTJ + 21a]: given n ≥ k i.i.d. samples of which c are correct, the estimator is

The following was proven in [CTJ + 21a]; we give a different proof that sets up our gradient estimator.

Theorem 1. ρ ( n, c, k ) is an unbiased estimator of the pass@k .

Proof. Let x 1 , x 2 , · · · , x n ∼ p ( x | θ ) , f i = f ( x i ) , and I be a set of k elements sampled uniformly without replacement from { 1 , 2 , . . . , n } . Then

Averaging over all subsets of size k recovers ρ :

where (6) holds because the sum on the r.h.s. of (5) is the number of subsets of size k of the ( n -c ) incorrect elements. Since averaging in this way retains unbiasedness, this completes the proof. We show in Corollary 2 that no such unbiased estimator exists for n < k , and in Corollary 3 that the asymptotic variance of this estimator decreases at a rate of 1 /n .

0

1

2

3

4

5

6

7

ascending sort index

i

0.0

0.2

0.4

0.6

0.8

1.0

i

(normalized)

Element Weight for Various

k

k

=1

k

=2

k

=4

k

=8

2.2 An Unbiased pass@k Gradient Estimator

Given a mini-batch of n i.i.d. samples x 1 , x 2 , . . . , x n from p ( x | θ ) with corresponding correctness labels f i ∈ { 0 , 1 } , we want to optimize the pass@k w.r.t. the model parameters θ . Letting c = ∑ n i =1 f i be the number of correct samples, we will demonstrate unbiasedness of the estimator

that assigns more weight to correct samples, while also assigning some reward to incorrect samples to encourage exploration. The following well-known results will be used to show that (8) is unbiased.

Lemma 1 (Policy Gradients) . For any absolutely continuous distribution p ( x | θ )

Corollary 1. If c is constant w.r.t. both θ and x then E p ( x | θ ) [ c ∇ θ log p ( x | θ )] = 0 .

We can now give our first main result:

Theorem 2. ̂ ∇ is an unbiased estimator of the gradient of the pass@k :

See Appendix A.3 for a proof.

3 Continuous Rewards

We generalize the pass@k to non-binary rewards g : X → R as

3.1 An Unbiased max g @k Estimator

The following estimator for the max g @k is a direct analog of ρ : given n ≥ k i.i.d. samples, assuming w.l.o.g. that the rewards g i = g ( x i ) are sorted, so that g 1 ≤ g 2 ≤ · · · ≤ g n the estimator is

To compute this stably we cancel factors in the binomial coefficients to get 2

Theorem 3. ρ ( g ) ( n, c, k ) is an unbiased estimator of the max g @k .

Proof. The proof is similar to Theorem 1. Here we exploit the assumption that the g i are sorted, so

since µ i is the number of subsets of 1 , 2 , . . . , i -1 of size k -1 , which equals ( i -1 k -1 ) . The sum starts at k because all subsets of size k include elements that are greater than or equal to g k . See Line 17 of Listing 1 for an implementation of ρ ( g ) .

3.2 An Unbiased max g @k Gradient Estimator

We propose the gradient estimator

where if we assume w.l.o.g. that the g i are sorted, the s i are a weighted combination of them,

Theorem 4. ̂ ∇ ( g ) is an unbiased estimator of the gradient of the max g @k :

where where the diagonals are

and the off-diagonals are

Proof. The proof is analogous to that of Theorem 2. Here we have

By assumption the g i are sorted, so max j ∈I g j = g max j ∈I . Therefore m ij is the number of subsets I of { 1 , 2 , . . . , n } that

are of size k ,

have j ≥ i as the largest element (so that we can factor out g j ),

include i (so that (25) holds in expectation by Corollary 1).

Due to the second condition, the form of m ij depends on whether i = j .

The diagonals m ii are zero if i < k since the largest element of any subset of size k is at least k . If i ≥ k then we fix i and are left with i -1 elements from which to choose k -1 which we can do ( i -1 k -1 ) ways in line with (20).

glyph[negationslash]

The m ij for i = j are obtained by fixing i and j leaving j -2 elements 1 , 2 , . . . , i -1 , . . . , i + 1 , . . . , j -1 from which to choose k -2 which we can do ( j -2 k -2 ) ways in line with (21).

Theorem 5. s 1 , s 2 , . . . , s n can be computed in total time O ( k + n log n ) .

See Appendix A.4 for the proof and Line 36 of Listing 1 for an implementation based on it.

4 Variance Reduction

4.1 Leave-One-Out Baseline for the Simple Case

A popular variance reduction method [MRFM20, Owe13, GYZ + 25] for point-wise rewards g ( x ) subtracts the mean of the leave one out (LOO) rewards within each mini-batch x 1 , x 2 , . . . , x n :

glyph[negationslash]

Since the subtracted part does not depend on x i , by Corollary 1 this retains unbiasedness.

4.2 Leave-One-Out Baseline for max g @k

Baselining the s i of (19) in this way introduces bias, however, as each s i depends on all x 1 , . . . , x n . We instead apply LOO to the following form of s i that follows from Theorem 4 and the proof thereof:

We can then retain unbiasedness by excluding i from the baseline, by defining glyph[negationslash]

Theorem 6. s ( loo ) 1 , s ( loo ) 2 , . . . , s ( loo ) n can be computed in total time O ( k + n log n ) .

Proof. Given (5) it is sufficient to consider computing, for i = 1 , 2 , . . . , n , glyph[negationslash]

By assuming w.l.o.g. an ascending ordering of g ( x i ) , excluding the first index does not change the ordering of the remaining indices. The first term is therefore 2

where (31) follows from (49). From (49) we obtain for 1 ≤ i < n the left to right recursion

Similar arguments to the proof of Theorem 5 therefore imply the same time complexity.

Line 52 of Listing 1 implements s (loo) i using the recursion in the above proof.

4.3 max g @(k -1) Leave-One-Out Baseline for max g @k

The baseline b ( k ) i is an average of the max g @k estimates over sets of size k . For a number of samples n equal to k , there are no such subsets to construct the baseline. [TZSM25] recently overcame this issue for the specific case n = k by using max g @(k -1) as the baseline statistic. We generalize their approach to k < n and to averaging over all subsets by defining similarly to Equation (27)

Averaging smaller but more numerous subsets in the baseline reduces variance but introduces bias (in the baseline, not s (loo -1) i ). Given our previous results it is straightforward to show

Proof. By the linearity of the expectation we can split the two terms in the parentheses of Equation (33) into two separate sums. The first summation is by definition simply s i of Equation (19). The (negation of the) second summation can be computed efficiently using

where the final equality follows with a little algebra from Equation (28) and Equation (30). Listing 1 implements s (loo -1) i using (34); Figure 5 compares s i , s (loo) i , and s (loo -1) i .

5 Experiments

5.1 One-Dimensional Toy Example

We start with a policy that is Gaussian with a fixed standard deviation and mean parameter θ we wish to learn, so that x ∼ N ( θ, 0 . 1) . We set the raw reward to be

The optimal policy under the max g @k reward varies with k (see Figure 1). The variance of our estimators is compared in Figure 4 where s ( loo -1) is the strongest.

5.2 RL on Open Source LLMs

Wedemonstrate promising RL results with the 2B and 9B parameter variants of GEMMA2 [TRP + 24] and the 8B parameter variant of LLAMA3.1 on real-world problems in MATH [HBK + 21], code generation [AON + 21] [CTJ + 21b], and the easy public subset of ARC-AGI-1 [CKKL25]. The latter is a challenging reasoning task-set even for state-of-the-art models much larger than GEMMA2.

For GEMMA2-2B we use a v5litepod-128 [Goo] which needs around 4 hours per 1000 training steps. Each RL training run [SWD + 17] involves sampling a fixed n number of completions { x i } n i =1 for a given prompt at a given training step. For our experiments, we set n = 16 . The rewards are computed for every completion using a reward function g ( · ) . We transform these rewards { g ( x i ) } n i =1 using our unbiased estimator s (loo -1) of (33), which we favour due to Figure 4, and which we refer to as PKPO . We repeat the training for a selection of k opt , thus optimizing a different pass@k opt each time. Since k opt = 1 leads to no reward transformation, this is our baseline (al-

though we use basic LOO mean centering of Equation (26), without which the training diverges). For each run, we measure pass@k eval for every k eval ∈ { 1 , 2 , 4 , 8 , 12 , 16 } at each step. Additionally, we also track model entropy and cumulative solve rate during training. The latter is defined as the fraction of tasks from the task-set for which the model has sampled a correct solution at least once; this is a critical metric that reflects the success of the model's exploration and measures its ability to find novel solutions.

Entropy regularization baseline In addition to our PKPO and the special case thereof of [TZSM25], we also add the entropy regularization baseline, which is PPO with an additional entropy term in the objective. We give this baseline an arguably unfair advantage by performing a small sweep over the values 0 . 001 , 0 . 005 , 0 . 01 , 0 . 05 , 0 . 1 for the entropy coefficient for each (model, benchmark) pair and only report the best result as EntropyReg .

5.2.1 Choosing k opt selectively optimizes pass@k eval and solves more tasks

We use the training split of Hendrycks MATH [HBK + 21] which contains 12,000 problems as our task set. Figure 6a shows that a higher k opt in our transformation leads to a consistently higher cumulative solve rate throughout training, as well as a higher entropy. By optimizing pass@k instead of pass@1 , the model appears to better utilize the exploration budget thus finding more solutions.

glyph[negationslash]

In Figure 7, we compare pass@k eval across our runs ( k opt ∈ { 1 , 4 , 8 } ) for various k eval . We find the best pass@k eval when k opt = k eval (or k opt is closest to k eval among available k opt ). Nontransformed rewards optimize pass@1 , leading to sub-optimal pass@k eval for k eval = 1 , and the deficit worsens as k eval increases. Thus, our experiments also demonstrate that setting k opt := k eval in our transformation suffices to optimize pass@k eval for a k eval ≤ n . This generalizes the already powerful result of [TZSM25] by alleviating the coupling that restricts to optimizing either pass@n or pass@1 . In other words, since RL training of LLMs typically samples a large batch ( n glyph[greatermuch] 1 ), failing to use our transformation results in sub-optimal pass@k performance, especially for modest values of k .

As k opt -→ n , the variance of our estimator increases as there are fewer subsets in (33) (see Figure 4). We presume this is why 1) gains of k opt = 8 over k opt = 4 are more prominent when k eval ∈ { 12 , 16 } than when k eval = 8 . That is, when k eval is further away from k opt ∈ { 4 , 8 } than when it is closer, and 2) the special case n = k opt of [TZSM25] struggles to optimize the pass@n .

5.2.2 PKPO robustly improves pass@k on held out evaluations

Tables 1-4 above (and Tables 7-8 in the appendix) present performance on held-out sets for two tasks. We report the mean and standard error based on three runs with different random seeds. For math, we train on the train split and evaluate on the test split of Hendrycks MATH [HBK + 21]. To

0

1000

2000

3000

Step

0.14

0.16

0.18

0.20

0.22

pass@

k

eval

pass@

k

eval

,

k

eval =1

k

opt =1

k

annealed

TZSM25

(a)

pass@1

(

k

eval

= 1

)

0

1000

2000

3000

Step

0.34

0.36

0.38

0.40

pass@

k

eval

pass@

k

eval

,

k

eval =8

k

opt =1

k

annealed

TZSM25

(b)

pass@8

(

k

eval

= 8

)

0

1000

2000

3000

Step

0.40

0.42

0.44

pass@

k

eval

pass@

k

eval

,

k

eval =16

k

opt =1

k

annealed

TZSM25

(c)

pass@16

(

k

eval

= 16

)

evaluate coding, we use MBPP [AON + 21] for training and evaluate on HUMANEVAL [CTJ + 21b]. MBPP has multiple unit tests per problem and hence we use this not only as a proxy for additional benchmarks but also to showcase our handling of a continuous reward function (% unit tests passed).

5.2.3 Improving pass@k without sacrificing pass@1

Figure 3 demonstrates that as PKPO can use any arbitrary k opt ≤ n , this allows varying k opt over the course of training to good effect. We show a simple annealing procedure which starts training with a high k opt = 8 and reduces it to k opt = 1 after 1500 steps. This trains the model to initially prioritize exploration (optimize pass@k ) and then consolidate the single-sample policy (optimize pass@1 ). This switch is apparent in Figure 3a, at step 1500 where the slope of k annealed changes. While traditional methods like [TZSM25] suffer from a trade-off between pass@k and pass@1 , we get a final model which has higher pass@k eval for all k eval > 1 with no sacrifice in pass@1 .

5.2.4 PKPO is essential for learning on hard problems

Figure 8 shows the limitation of traditional pass@1 optimization through RL on an especially challenging task-set. We use the easy subset of ARC-AGI-1 [CKKL25]. We observe that conventional pass@1 optimization stalls. However, our pass@k approach unblocks learning, and results in higher pass@k eval across all k eval including k eval = 1 . Furthermore, we see higher k opt leads to more effective and faster learning. This is likely because the benefits of prioritizing joint utility over individual sample utility are more prominent on a harder task-set.

Tables 5 and 6 show more extensive experiments on ARC-AGI-1 . We make an 80:20 train:test split of the same easy subset as before and report the cumulative solve rate on the train set and pass@k rate on the test set. We train to saturation (no change in cumulative rate for 1k steps), and again use three random restarts to provide standard errors. By encouraging exploration in a direct and stable manner, our method unblocks learning unlike other methods. Entropy Regularization does indeed sacrifice pass@1 and slightly improves pass@k by promoting exploration, but it is hard to tune, and is significantly outperformed by our method. Moreover, it has no explicit way to optimize for a specific k eval . [TZSM25] targets the same objective as PKPO , but couples the minibatch size to k and thereby incurs higher variance than PKPO with k < n .

6 Conclusions and Outlook

In RL training with multiple independent samples per task, optimizing the pass@k maximizes the expectation of the best reward in the set of samples, rather than the average one. This preserves model output diversity, which leads to solving more problems and ultimately yields stronger policies. We provide drop-in replacements for more traditional RL reward transformations that robustly and efficiently optimize the pass@k . This work can be extended in various ways, such as to other inference-time search algorithms, and to more sophisticated baseline techniques.

References

[AAA + 23] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

[ABC + 21] Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861 , 2021.

[Ant] Anthropic. Claude 3.5 Sonnet. https://www.anthropic.com/news/claude-3 .

[AON + 21] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021.

[AT] DeepMind AlphaProof and AlphaGeometry Teams. AI achieves silver-medal standard solving international mathematical olympiad problems. https://tinyurl.com/alphaproof .

[AVAC24] Afra Amini, Tim Vieira, Elliott Ash, and Ryan Cotterell. Variational Best-of-N alignment. arXiv preprint arXiv:2407.06057 , 2024.

[BSB + 24] Ananth Balashankar, Ziteng Sun, Jonathan Berant, Jacob Eisenstein, Michael Collins, Adrian Hutter, Jong Lee, Chirag Nagpal, Flavien Prost, Aradhana Sinha, et al. Infalign: Inference-aware language model alignment. arXiv preprint arXiv:2412.19792 , 2024.

[CKKL25] Francois Chollet, Mike Knoop, Gregory Kamradt, and Bryan Landers. Arc prize 2024: Technical report, 2025.

[CLB + 17] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems , 30, 2017.

[CTG + 24] Yinlam Chow, Guy Tennenholtz, Izzeddin Gur, Vincent Zhuang, Bo Dai, Sridhar Thiagarajan, Craig Boutilier, Rishabh Agarwal, Aviral Kumar, and Aleksandra Faust. Inference-aware fine-tuning for Best-of-N sampling in large language models, 2024.

[CTJ + 21a] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. arXiv , 2021.

[CTJ + 21b] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021.

[CTO + 25] Yuri Chervonyi, Trieu H. Trinh, Miroslav Olsak, Xiaomeng Yang, Hoang Nguyen, Marcelo Menegali, Junehyuk Jung, Vikas Verma, Quoc V. Le, and Thang Luong. Goldmedalist performance in solving olympiad geometry with alphageometry2, 2025.

[CTV19] Marco Cuturi, Olivier Teboul, and Jean-Philippe Vert. Differentiable ranking and sorting using optimal transport. In Advances in Neural Information Processing Systems , volume 32, 2019.

[DLJ + 24] Shihan Dou, Yan Liu, Haoxiang Jia, Limao Xiong, Enyu Zhou, Wei Shen, Junjie Shan, Caishuang Huang, Xiao Wang, Xiaoran Fan, et al. Stepcoder: Improve code generation with reinforcement learning from compiler feedback. arXiv preprint arXiv:2402.01391 , 2024.

[GDJ + 24] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzm´ an, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur C ¸ elebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, V´ ıtor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yaz-

dan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaojian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The llama 3 herd of models, 2024.

[Goo] Google. Google cloud platform. https://cloud.google.com .

[GYZ + 25] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948 , 2025.

[GZC + 24] Jonas Gehring, Kunhao Zheng, Jade Copet, Vegard Mella, Quentin Carbonneaux, Taco Cohen, and Gabriel Synnaeve. Rlef: Grounding code llms in execution feedback with reinforcement learning. arXiv preprint arXiv:2410.02089 , 2024.

[HBK + 21] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021.

[Hoe48] Wassily Hoeffding. A class of statistics with asymptotically normal distribution. Annals of Mathematical Statistics , 19:308-334, 1948.

[HYM + 24] Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishabh Agarwal. V-star: Training verifiers for self-taught reasoners. arXiv preprint arXiv:2402.06457 , 2024.

[Kol50] A. N. Kolmogorov. Unbiased estimates. Izvestiya Akademii Nauk SSSR. Seriya Matematicheskaya , 14(4):303-326, 1950.

[KZA + 24] Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, et al. Training language models to self-correct via reinforcement learning. arXiv preprint arXiv:2409.12917 , 2024.

[LCC + 22] Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Remi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. Science , 378(6624):1092-1097, 2022.

[Leh98] George Lehmann, E. L. ; Casella. Theory of Point Estimation . Springer, 2nd edition, 1998.

[LKB + 23] Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations , 2023.

[LWG + 22] Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu Hong Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. Advances in Neural Information Processing Systems , 35:21314-21328, 2022.

[MRFM20] Shakir Mohamed, Mihaela Rosca, Michael Figurnov, and Andriy Mnih. Monte carlo gradient estimation in machine learning. Journal of Machine Learning Research , 21(132):1-62, 2020.

[OIW + 23] Theo X Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar-Lezama. Is self-repair a silver bullet for code generation? arXiv preprint arXiv:2306.09896 , 2023.

[Owe13] Art B. Owen. Monte Carlo theory, methods and examples . https://artowen.su.domains/mc/ , 2013.

[PYTG25] Nilay Pande, Sahiti Yerramilli, Jayant Sravan Tamarapalli, and Rynaa Grover. Marvlqa: A benchmark for mathematical reasoning over visual landscapes. arXiv preprint arXiv:2508.17180 , 2025.

[RNS + 18] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training, 2018.

[SDH + 24] Pier Giuseppe Sessa, Robert Dadashi, Leonard Hussenot, Johan Ferret, Nino Vieillard, Alexandre Rame, Bobak Shahriari, Sarah Perrin, Abe Friesen, Geoffrey Cideron, Sertan Girgin, Piotr Stanczyk, Andrea Michi, Danila Sinopalnikov, Sabela Ramos, Amelie Heliou, Aliaksei Severyn, Matt Hoffman, Nikola Momchev, and Olivier Bachem. BOND: aligning LLMs with Best-of-N distillation. CoRR , abs/2407.14622, 2024.

[SJTR23] Parshin Shojaee, Aneesh Jain, Sindhu Tipirneni, and Chandan K Reddy. Executionbased code generation using deep reinforcement learning. arXiv preprint arXiv:2301.13816 , 2023.

[SWD + 17] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.

[Tea23] Gemini Team. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 , 2023.

[TRP + 24] Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L´ eonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram´ e, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Stanczyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Jean-Bastien Grill, Behnam Neyshabur, Olivier Bachem, Alanna Walton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchison, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Perry, Chris Welty, Christopher A. Choquette-Choo, Danila Sinopalnikov, David Weinberger, Dimple Vijaykumar, Dominika Rogozi´ nska, Dustin Herbison, Elisa Bandy, Emma Wang, Eric Noland, Erica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Klimczak-Pluci´ nska, Harleen Batra, Harsh Dhand, Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Peng Zhou, Joana Carrasqueira, Joana Iljazi, Jocelyn Becker, Joe Fernandez, Joost van Amersfoort, Josh Gordon, Josh Lipschultz, Josh Newlan, Ju yeong Ji, Kareem Mohamed, Kartikeya Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhia, Kish Greene, Lars Lowe Sjoesund, Lauren Usui, Laurent Sifre, Lena Heuermann, Leticia Lago, Lilly McNealus, Livio Baldini Soares, Logan Kilpatrick, Lucas Dixon, Luciano Martins, Machel Reid, Manvinder Singh, Mark Iverson, Martin G¨ orner, Mat Velloso, Mateo Wirth, Matt Davidow, Matt Miller, Matthew Rahtz, Matthew Watson, Meg Risdal, Mehran Kazemi, Michael Moynihan, Ming Zhang, Minsuk Kahng, Minwoo Park, Mofi Rahman, Mohit Khatwani, Natalie Dao, Nenshad Bardoliwalla, Nesh Devanathan, Neta Dumai, Nilay Chauhan, Oscar Wahltinez, Pankil Botarda, Parker Barnes, Paul Barham, Paul Michel, Pengchong Jin, Petko Georgiev, Phil Culliton, Pradeep Kuppala, Ramona Comanescu, Ramona Merhej, Reena Jana, Reza Ardeshir Rokni, Rishabh Agarwal, Ryan Mullins, Samaneh Saadat, Sara Mc Carthy, Sarah Cogan, Sarah Perrin, S´ ebastien M. R. Arnold, Sebastian Krause, Shengyang Dai, Shruti Garg, Shruti Sheth, Sue Ronstrom, Susan Chan, Timothy Jordan, Ting Yu, Tom Eccles, Tom Hennigan, Tomas Kocisky, Tulsee Doshi, Vihan Jain, Vikas Yadav, Vilobh Meshram, Vishal Dharmadhikari, Warren Barkley, Wei Wei, Wenming Ye, Woohyun Han, Woosuk Kwon, Xiang Xu, Zhe Shen, Zhitao Gong, Zichuan Wei, Victor Cotruta, Phoebe Kirk, Anand Rao, Minh Giang, Ludovic Peran, Tris Warkentin, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, D. Sculley, Jeanine Banks, Anca Dragan, Slav Petrov, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Sebastian Borgeaud, Noah Fiedel, Armand Joulin, Kathleen Kenealy, Robert Dadashi, and Alek Andreev. Gemma 2: Improving open language models at a practical size, 2024.

[TZSM25] Yunhao Tang, Kunhao Zheng, Gabriel Synnaeve, and R´ emi Munos. Optimizing language models for inference time objectives using reinforcement learning. arXiv preprint arXiv:2503.19595 , 2025.

[WBZ + 21] Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652 , 2021.

[Wil92] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning , 8:229-256, 1992.

[WSL + 24] Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models. arXiv preprint arXiv:2408.00724 , 2024.

[XDC + 20] Yujia Xie, Hanjun Dai, Minshuo Chen, Bo Dai, Tuo Zhao, Hongyuan Zha, Wei Wei, and Tomas Pfister. Differentiable top-k with optimal transport. In Advances in Neural Information Processing Systems , volume 33, 2020.

[YSG + 23] Kaiyu Yang, Aidan Swope, Alex Gu, Rahul Chalamala, Peiyang Song, Shixing Yu, Saad Godil, Ryan J Prenger, and Animashree Anandkumar. Leandojo: Theorem prov-

ing with retrieval-augmented language models. Advances in Neural Information Processing Systems , 36:21573-21612, 2023.

[YTC + 23] Zishun Yu, Yunzhe Tao, Liyu Chen, Tao Sun, and Hongxia Yang. B-coder: Value-based deep reinforcement learning for program synthesis. arXiv preprint arXiv:2310.03173 , 2023.

A Additional Theoretical Statements and Proofs

A.1 Statement and proof that n ≥ k samples are required to unbiasedly estimate pass@k

This result is a direct consequence of a well-known theorem concerning the unbiased estimability of parametric functions for the Bernoulli distribution.

Theorem 8 (Kolmogorov [Kol50]) . Let Y 1 , . . . , Y n be i.i.d. Bernoulli random variables with success probability p ∈ [0 , 1] . A function ρ ( p ) is unbiasedly estimable from this sample if and only if it can be expressed as a polynomial in p of degree at most n .

A sketch of a proof of Theorem 8 can be found in Lehmann and Casella [Leh98].

Corollary 2. Given a sequence of n i.i.d. model samples x 1 , x 2 , . . . , x n , the pass@k is unbiasedly estimable if and only if n ≥ k .

Proof. It is sufficient to consider a single a fixed and observed correctness function f , so that the independence of the x i implies the independence of the correctness events [ f ( x i ) = 1] . Let p = P [ [ f ( x i ) = 1] ] be the probability that any single sample is correct. The pass@k is defined as the complement of the probability that all k samples are incorrect, which for the specific assumptions adopted in this proof is 1 -(1 -p ) k . Because this expression is a polynomial in p of degree k , the result follows immediately from Theorem 8.

A.2 Characterization of the Variance

Our proof of Theorem 1 identifies the pass@k estimator ρ ( n, c, k ) as a U -statistic. To characterize its variance, we apply Hoeffding's asymptotic theory.

Theorem 9 (Hoeffding [Hoe48]) . Let X 1 , . . . , X n be independent and identically distributed random variables with distribution F . Let h ( x 1 , . . . , x k ) be a symmetric kernel with E [ h ( X 1 , . . . , X k ) 2 ] < ∞ . Define the parameter µ = E F [ h ( X 1 , . . . , X k )] and the U -statistic:

Let h 1 ( x ) = E [ h ( x, X 2 , . . . , X k )] be the projection of the kernel onto a single variable. Hoeffding proved that if ζ 1 = Var ( h 1 ( X 1 )) > 0 , then as n →∞ :

In the standard application of pass@k we evaluate the estimator on a specific problem defined by a prompt and a correctness oracle. While the true pass rate ν is unknown to the observer, it is a fixed property of the model-problem pair. Consequently, the correctness outcomes of the generated samples are i.i.d. conditioned on the problem.

The following lemma derives the variance parameter ζ 1 under this conditioning. We abuse the notation by allowing the X i to denote correctness.

Lemma 2 (Conditional Variance of the Projection) . Fix a problem instance such that the correctnesses X i are i.i.d. Bernoulli ( ν ) . For the pass@k kernel h ( x 1 , . . . , x k ) = max( x 1 , . . . , x k ) , the variance of the first-order projection is:

Proof. The projection h 1 ( x ) is the expected value of the kernel given the first sample is fixed to x , while X 2 , . . . , X k remain random variates drawn from Bernoulli ( ν ) .

We evaluate this for the two possible realizations of x :

Case x = 1 (Success): The maximum is 1 regardless of the remaining samples.

Case x = 0 (Failure): The maximum is 0 if and only if all remaining k -1 samples fail. Since the remaining samples are i.i.d. with failure probability (1 -ν ) ,

The projection h 1 ( X 1 ) is thus a binary random variable taking value h 1 (1) with probability ν and h 1 (0) with probability 1 -ν , so that

We can now substitute this explicit form back into Hoeffding's general result.

Corollary 3. For a fixed problem with pass rate ν , as n → ∞ , the asymptotic variance of the estimator ρ ( n, c, k ) is:

A.3 Proof of Theorem 2

Although Theorem 2 is a special case of Theorem 4, we include both because the following proof uses a different approach from that of the more general statement, and is arguably the easier of the two.

Proof. By Lemma 1 the gradient ∇ θ pass@k has the unbiased estimator

where (42) substitutes the l.h.s. of (5). m i is the number of subsets I of { 1 , 2 , . . . , n } that

are of size k ,

contain at least one correct element, so that ( 1 -∏ i ∈I (1 -f i ) ) = 1 ,

contain i , so that (43) holds in expectation by Corollary 1.

Due to the second condition, m i therefore equals one of two values, which we denote by m (1) and m (0) , depending on whether f i = 1 or f i = 0 , respectively.

If f i = 1 then all subsets that include i also include at least one correct element ( i itself), so that m (1) is just the number of subsets of size k of { 1 , 2 , . . . , n } that include i , which equals the number of subsets of size k -1 of { 1 , 2 , . . . , n -1 } :

If f i = 0 then we assume w.l.o.g. that i = n , so that m (0) is the number of subsets of size k -1 of { 1 , 2 , . . . , n -1 } with at least one correct element,

where we again used (5), this time to get an expression in terms of ρ . Using m (0) and m (1) we can compute r (0) and r (1) using (43) as

in line with (8).

A.4 Proof of Theorem 5

Proof. The vector s = ( s 1 , s 2 , . . . , s n ) glyph[latticetop] can be written as s = M g where we have introduced g = ( g ( x 1 ) , g ( x 2 ) , . . . , g ( x n )) glyph[latticetop] as well as the matrix M with

diagonal elements m ii given by (20),

upper diagonals m ij for i < j given by (21) which is independent of i ,

lower diagonals m ij for i > j equal to zero.

Because of the structure of M , we have that

and, for 1 ≤ i < n , the right to left recursion

The ratios of m ii , m i,i +1 and m i +1 ,i +1 divided by ( n k ) can be simplified by cancelling factors in the binomial coefficients and writing the remaining factors as a product of k ratios similarly to (14), for a total cost of O ( nk ) ; this computation can be further simplified by noting that the required ratios can be lazily computed in sequence (for example to obtain m i +1 ,i +1 from m ii ) at a cost of O (1) after computing the first at a cost of O ( k ) , giving a total cost of O ( k + n ) . The additional O ( n log n ) comes from assuming the i are sorted in increasing order of g ( x i ) .

and

B Implementation

C Additional Figures

5

10

15

20

25

30

35

Number of model samples

n

10 0

10 1

10 2

Empirical Variance of the Gradient Estimator

Variance of Estimated Gradient of pass

g

@

k

for

k

=4

all subsets baselined

s

(loo)

all subsets no baseline

s

naive partitioned baselined

naive partitioned no baseline

loo minus one partitioned

loo minus one all subsets

s

(loo

1)

all subsets baselined: s (loo) -our novel estimator of Equation (29) that analytically sums over all subsets of size k of the n samples with our unbiased baseline method that subtracts for each element i the mean of the estimator over all subsets of size k that do not include i .

all subsets no baseline: s -our novel estimator of Equation (19) that analytically sums over all subsets of size k of the n samples but that does not include a variance-reducing baseline.

naive partitioned baselined - a naive transformation that sets all k transformed rewards in a subset of k samples equal to the largest raw reward in that subset. To extend this method to n > k we partition the n samples (for integer multiples of k ) into disjoint subsets of size k and average the estimated gradient obtained from each. Furthermore, as a simple variance reduction method, for each such set of k samples we subtract the mean of the transformed rewards from the other sets of k samples (thereby averaging over ( n -k ) samples and subtracting the result from the k samples and repeating n/k times in a leave-one-out fashion over the subsets of size k ). If we were to randomly sample an increasing number of partitions of the samples and average over all of them, then intuitively the resulting estimator would approach the variance of s (loo) , but this would be expensive and indeed the limiting case of considering all partitions is intractable for general n and k . Our estimators have the key property of summing over all such partitions while nonetheless being efficient to compute.

naive partitioned no baseline -asimilar method to the previous one, but without the naive mean subtraction based variance reduction step.

loo minus one partitioned -amethodthat uses the same partitioning approach as the previous two, but instead of using the naive estimate (which sets every transformed reward to simple max of the raw reward in a given set of k samples) it uses the s (loo -1) method applied separately to each disjoint set of k samples, and averages that over all such subsets. In this way, this is a trivial generalization of [TZSM25] which extends to n > k by applying the basic method to disjoint subsets and averaging the results. We do not subtract a baseline across sets as this did not improve the variance, possibly because the method within each k already includes a variance reduction baseline.

loo minus one all subsets: s (loo -1) -our novel estimator of Equation (33) that analytically sums over all subsets of size k of the n samples and uses all appropriate subsets of size k -1 to form the variance-reducing baseline that retains unbiasedness, thereby non-trivially generalizing [TZSM25] to all n > k with strong variance reduction.

0.4

0.2

0.0

0.2

0.4

raw reward

g

(

xi

)

0

1

2

3

4

5

6

7

Effective Reward

s

(loo)

i

Raw vs Effective pass

g

@k Rewards

k

=1

k

=2

k

=4

k

=8

k

=16

(a) All subsets without baseline:

s

.

0.4

0.2

0.0

0.2

0.4

raw reward

g

(

xi

)

0.4

0.2

0.0

0.2

0.4

0.6

0.8

Effective Reward

si

Raw vs Effective pass

g

@k Rewards

k

=1

k

=2

k

=4

k

=8

k

=16

(b) All subsets with LOO baseline:

s

(loo)

0.4

0.2

0.0

0.2

0.4

raw reward

g

(

xi

)

0.0

0.2

0.4

0.6

0.8

1.0

Effective Reward

s

(loo

1)

i

Raw vs Effective pass

g

@k Rewards

k

=2

k

=4

k

=8

k

=16

(loo

-

1)

(c) All

k

sized subsets with

k

-

1

sized subsets baseline:

s

0

2000

4000

6000

8000

10000

Step

0.0

0.2

0.4

0.6

Solve rate

Cumulative solve rate

k

opt =1

k

opt =4

k

opt =8

0

500

1000

1500

2000

2500

3000

Step

93

92

91

90

89

88

87

Entropy

Entropy when optimizing pass@

k

opt

k

opt =1

k

opt =4

k

opt =8

N

(a) Cumulative solve rate (

= 12000

problems)

(b) Effect of changing

k

on entropy during training

0

500

1000

1500

2000

2500

3000

Step

0.14

0.16

0.18

0.20

pass@

k

eval

pass@

k

eval

,

k

eval =1

k

opt =1

k

opt =4

k

opt =8

TZSM25

(a)

pass@1

(

k

eval

= 1

)

0

500

1000

1500

2000

2500

3000

Step

0.22

0.24

0.26

pass@

k

eval

pass@

k

eval

,

k

eval =2

k

opt =1

k

opt =4

k

opt =8

TZSM25

(b)

pass@2

(

k

eval

= 2

)

0

500

1000

1500

2000

2500

3000

Step

0.28

0.30

0.32

0.34

pass@

k

eval

pass@

k

eval

,

k

eval =4

k

opt =1

k

opt =4

k

opt =8

TZSM25

(c)

pass@4

(

k

eval

= 4

)

0

500

1000

1500

2000

2500

3000

Step

0.34

0.36

0.38

0.40

pass@

k

eval

pass@

k

eval

,

k

eval =8

k

opt =1

k

opt =4

k

opt =8

TZSM25

(d)

pass@8

(

k

eval

= 8

)

0

500

1000

1500

2000

2500

3000

Step

0.38

0.40

0.42

0.44

pass@

k

eval

pass@

k

eval

,

k

eval =12

k

opt =1

k

opt =4

k

opt =8

TZSM25

(e)

pass@12

(

k

eval

= 12

)

0

500

1000

1500

2000

2500

3000

Step

0.40

0.42

0.44

0.46

pass@

k

eval

pass@

k

eval

,

k

eval =16

k

opt =1

k

opt =4

k

opt =8

TZSM25

(f)

pass@16

(

k

eval

= 16

)

0

200

400

600

800

1000

Step

0.0

0.2

0.4

0.6

0.8

Solve rate

Cumulative solve rate

k

opt =1

k

opt =4

k

opt =8

(a) Cumulative solve rate

0

200

400

600

800

1000

Step

0.000

0.005

0.010

0.015

0.020

pass@

k

eval

pass@

k

eval

,

k

eval =1

k

opt =1

k

opt =4

k

opt =8

(b)

pass@1

(

k

eval

= 1

)

0

200

400

600

800

1000

Step

0.00

0.02

0.04

0.06

0.08

0.10

pass@

k

eval

pass@

k

eval

,

k

eval =16

k

opt =1

k

opt =4

k

opt =8

(c)

pass@16

(

k

eval

= 16

)

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: the abstract and introduction are typical and represent the paper's contribution and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The paper is largely theoretical, and the theorems include appropriate qualifiers and assumptions.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: we have made every effort to ensure that the results are precise and rigorous.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: the main contribution is a reward transformation method that need only modify a standard RL algorithm by mapping batches of scalar rewards to their transformed values. We have provided the code for this transformation in Listing 1.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: the main non-trivial code that is required to transform the rewards is provided in the document as Listing 1.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Every effort has been made to report this information to an appropriate level of detail.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: the paper is typical in that due to the computational cost of the experiments, we do not provide detailed condifence intervals, etc. However, we made every effort to provide a suitable level of detail on the scope and significance of the experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer 'Yes' if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: we have provided as much information as we are permitted to by our organization at this stage. The compute requirements are already strongly indicated by the fact that we specify that we are training with the open source 2B GEMMA2 model. We can expand on this for the camera ready.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The paper conforms rather comfortably, as it is mainly a theoretical paper with standard experimental results on existing publicly available data.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: this is a methodological work that is not tied to any specific applications. There is no new and direct path from this paper to specific societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: this paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: we have made every effort to include these details.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: the paper does not introduce new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: the paper does not use human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: see above.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: the LLM training in our experiments is standard and on standard publicly available tasks.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, \"Solving Harder RL Tasks with Direct Pass@k Optimization\", the authors introduce a specific transformation on the final rewards which leads to a direct optimization of the pass@k objective.\n\nThe primary contribution is the theoretical motivation for the proposed transformation. Specifically the authors introduce unbiased estimators for the pass@k objective and gradient in the case of binary rewards. These estimators are then extended to the case of real rewards. The authors further propose various baselines for variance reduction for both binary and real valued rewards, and show that these can be efficient computed.\n\nFinally the authors show some empirical results of training with the transformed rewards on some toy and real domains.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper is clearly presented and easy to follow without extraneous content\n2. The paper presents a timely and important contribution. The implicit misalignment between the pass@1 training and pass@k usage of models leaves much on the table. The paper's contribution, to provide a more general (and practical) reward transformation to support the direct pass@k, is, to my mind, a clearly valuable contribution with immediate application.\n\nWeaknesses:\n1. There is likely some missing literature. Specifically, there are results like (https://arxiv.org/abs/2502.07154) which would both help to further motivate the problem and better contextualize the current contribution.\n2. The Introduction's positioning of this work as \"...unblock(ing) the path to artificial super intelligence...\" seems unnecessary\n3. The paper title itself appears somewhat misleading. While the authors show _some_ results which, expectedly, demonstrate improvement on \"harder\" tasks, it is clear that the paper's primary contribution is a theoretical advancement. I would prefer to have seen the title more closely match the content. \n4. ^^ Relatedly, the empirical validation, while suitable for a theoretically focused paper, is weak when making the more general claim suggested by the title (small models, few datasets).\n\nQUESTIONS:\n1. Compute restrictions aside were there any considerations for running $k^{eval}$ up to just 16? I would expect some sort of asymptotic behavior here and it would be interesting to observe this on, say, MATH\n2. The ability to implement an annealing procedure for k<n feels like it has a lot of promise for practitioners. Extending these results might provide just the small-scale but robust empirical justification for this reward scaling suitable to a theoretical paper.\n3. The step-function improvement on ARC-AGI-1 is pretty surprising (especially for the pass@1 eval). What's going on there?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on optimising policies jointly over multiple samples, rather than following the usual single-sample optimisation paradigm. Anticipating the fact that multiple attempts will be made at inference time, it enables training policies that have a higher chance of producing at least one very good output. The paper proposes optimising over the best k out of n samples, thereby generalising prior work which fixes k = n. Additionally, it introduces an unbiased and low-variance gradient estimator for this setting, supported by mathematical proofs, and provides experiments to evaluate its effect on performance during training.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- The pass@k optimization paradigm is relevant and increasingly important in RL for generative models. This paper contributes to it by introducing a general formulation (arbitrary k≤n), supporting it with rigorous mathematical derivations, and providing empirical validation.\n- The paper is well written. The mathematical content is clearly presented, and the proofs are systematically included, either in the main text or in the appendix.\n- The experimental section includes a toy problem to validate and illustrate the key claims (i.e., improved sample performance and reduced gradient variance).\n- The paper also presents large-scale experiments using Gemma-2 on challenging benchmarks (e.g., MATH, ARC-AGI), demonstrating that the proposed method can be applied to realistic and difficult problems.\n\n\n\n**Weaknesses:**\n- The title feels too broad relative to the empirical scope of the paper. While it refers to “solving harder RL tasks,” the experiments are limited to two tasks in the LLM domain. The claim would be more accurate if it were restricted to LLMs.\n- There already exist methods designed to jointly optimize over multiple attempts in hard RL tasks, such as Grinsztajn et al. (2022), Chalumeau et al. (2023), and follow-ups like Hottung et al. (2024) and Bonnet et al. (2024). It is unclear whether the approach proposed here would outperform these methods in other domains (e.g., combinatorial optimization). To be fully fair, the claims should be qualified to the LLM context, where those baselines have not (to my knowledge) been evaluated.\n- Statistical rigor: It is unclear whether the experimental results are averaged over multiple seeds. If only a single seed is used throughout, this raises concerns about the statistical robustness of the findings.\n- Line 54/55 claims that “[CTV19] and [XDC+20] present elegant approximations that are rather general but less efficient in our setting.” However, no empirical or theoretical justification is provided for this efficiency claim.\n- Baselines (concurrent methods): I am surprised by the lack of baselines in the paper. Is there only one concurrent method to compare to? \n- I’m unclear on what the loss function contributes beyond encouraging high-entropy solutions. - It would be useful to compare against a baseline that optimizes pass@1 (or pass@n) with an added entropy regularization term. This would help disentangle the benefits of the proposed method from standard exploration techniques in RL, such as entropy bonuses, which are widely used and easy to implement.\n- In Section 5.2.3, the authors demonstrate promising results on a difficult benchmark, but do not compare against other methods. The results would be compelling if competing approaches were also shown to fail in this setting, similar to the k=1 baseline. It remains unclear how well [TZSM25] or other baselines would perform on this task.\n\nReferences:\n- Nathan Grinsztajn, Daniel Furelos-Blanco, Shikha Surana, Clément Bonnet, and Thomas D. Barrett. Winner takes it all: Training performant rl populations for combinatorial optimization. Neurips 2023.\n- Felix Chalumeau, Shikha Surana, Clément Bonnet, Nathan Grinsztajn, Arnu Pretorius, Alexandre Laterre, and Thomas D Barrett. Combinatorial optimization with policy adaptation using latent space search. Neurips 2023\n- Andre Hottung, Mridul Mahajan, and Kevin Tierney. Polynet: Learning diverse solution strategies for neural combinatorial optimization. ICLR 2024\n- Clement Bonnet, Matthew Macfarlane. Searching Latent Program Spaces.\n\nQUESTIONS:\nWhat is the statistical significance of the empirical results? Specifically, how many seeds were used per experiment?\n\nCan we reasonably expect this method to outperform a standard entropy-regularized baseline? It would be helpful to understand whether the benefits go beyond what entropy regularisation alone can achieve.\n\nLIMITATIONS:\nThere does not seem to be any discussion of the limitations. Could the authors add one?", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Pass-at-k Policy Optimization (PKPO), a method for optimizing the pass@k metric in Reinforcement Learning rather than the conventional pass@1 approach. The key insight is that when sampling multiple solutions per problem, optimizing for the probability that at least one of k samples succeeds can improve exploration and performance on challenging tasks. The authors provide mathematical foundations, including unbiased estimators for pass@k and its gradient in both binary and continuous reward settings. The method is validated on toy problems and a small subset of real-world experiments using Gemma-2 on mathematical reasoning tasks.\n\nSTRENGTHS AND WEAKNESSES:\nThis paper addresses an important problem in reinforcement learning: how to effectively optimize when multiple solution attempts per problem are sampled. The core idea of directly optimizing for pass@k is valuable. However, the proposed method suffers from several fundamental methodological and computational issues that undermine its theoretical claims and practical scalability.\n\n**Strengths:**\n*  The core insight of directly optimizing pass@k rather than pass@1 is valuable, as it addresses a real limitation in current RL practice where multiple samples are commonly used but optimization focuses only on individual sample quality[cite: 2, 4].\n* The approach of deriving unbiased estimators for pass@k and its gradient, applicable to both binary and continuous reward settings, is sophisticated and aims to provide a complete theoretical treatment.\n\n**Weaknesses:**\n*  The paper contains several mathematical flaws that compromise its theoretical rigor:\n    *  In Equation (8), the `r_i` definition for $f_i=0$ (incorrect samples) uses $\\rho(n-1,c,k-1)$. This formulation, particularly how the count `c` is used for samples that are assumed incorrect, appears to include the incorrect sample being evaluated, which violates the leave-one-out principle. This can introduce bias, especially for small `k`, directly contradicting the claim of `r_i` being unbiased.\n    * All theoretical proofs (e.g., Theorem 1, Theorem 2) assume `k` independent and identically distributed (i.i.d.) samples. However, in practical LLM fine-tuning settings, samples from the same mini-batch often share components like dropout masks and KV-cache. This makes them non-independent, directly violating a core assumption of the theoretical guarantees and potentially invalidating the unbiasedness claims in the empirical context.\n    * The paper claims that $s^{(loo-1)}$ is an unbiased estimator despite admitting a biased baseline. However, when a biased baseline appears directly inside the score-function product (as implied by the general form of policy gradient estimators), the overall gradient estimate itself becomes biased. This specific technical issue is not adequately addressed, undermining the claim of an unbiased gradient estimator.\n* **some computational complexity issues**\n    * Theorem 5 claims an $O(k + n \\log n)$ complexity for computing the reward transformations. However, the provided Python implementation (Listing 1) explicitly notes that it \"costs $O(nk + n \\log n)$ for simplicity\". This discrepancy is significant for scalability.\n    * For continuous rewards, the method requires $O(n \\log n)$ sorting per task (due to the assumption that rewards $g_i$ are sorted in Theorem 3 and its application to `s_i` computation), which would make the wall-clock time super-linear in `n` across multiple tasks. This significantly impacts scalability for large `k` or when processing many tasks.\n* The experimental evaluation is limited to one specific model architecture (Gemma-2 2B variant) and primarily two domains (mathematical reasoning and logic puzzles). This narrow scope limits the empirical contribution and does not validate the method's effectiveness on diverse RL applications or in settings where the theoretical assumptions might hold better.\n* The comparison largely focuses on vanilla pass@1 optimization. The paper lacks comprehensive comparisons with other contemporary inference-time optimization methods, advanced exploration techniques, or alternative approaches for handling multiple samples in RL literature.\n\nQUESTIONS:\n1. Given that practical RL fine-tuning often involves non-i.i.d. samples (e.g., due to shared dropout, KV-cache), how do these violations of the i.i.d. assumption affect the theoretical guarantees of unbiasedness and variance reduction? Can the authors provide theoretical analysis or empirical validation under these realistic, non-i.i.d. conditions to demonstrate the robustness of PKPO?\n2.   The theoretical claim of an unbiased gradient estimator is crucial. Can the authors provide a rigorous proof that the overall gradient estimator remains unbiased when using $s^{(loo-1)}$, specifically addressing how the potential admitted bias in the baseline term within the score-function product does not propagate to make the entire gradient estimate biased?\n3. * Can the authors provide a clear path towards achieving an implementation with $O(k + n \\log n)$ complexity as claimed in Theorem 5, rather than the currently noted $O(nk + n \\log n)$?\n    * How do you propose to address the super-linear scaling in wall-clock time across multiple tasks that require sorting, particularly for large-scale applications with many tasks?\n4.  Please provide a more detailed justification and proof for the specific form of $r_i$ in Equation (8), especially for the $f_i=0$ case and its interaction with the count `c`. How does this formulation ensure unbiasedness for all $k \\le n$, and specifically, address concerns about potential bias for small $k$ values?\n5. To strengthen the theoretical contributions and their applicability, can the authors provide experiments on domains or tasks where the i.i.d. assumption (or approximations thereof) is better satisfied or can be explicitly controlled? This would help validate the method's performance under its theoretical premises.\n\nLIMITATIONS:\nThe authors acknowledge some limitations but miss several critical issues. These unaddressed points fundamentally impact the method's theoretical soundness and practical applicability. Specifically, the paper does not adequately discuss: (1) the potential bias in the leave-one-out formulation for $r_i$ in Equation (8), (2) the violation of i.i.d. assumptions in practical RL settings for LLMs, (3) the computational scaling issues across multiple tasks due to sorting, and (4) the potential for gradient bias originating from the admitted bias in the baseline terms within score-function estimators. Addressing these fundamental problems is crucial for the work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 1, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors derive a set of new algorithms for direct pass@k optimization. Starting from the unbiased pass@k /max@k estimators in the case of binary / continuous rewards, the authors derive a gradient estimator (both without and with variance reduction via LOO baseline).\n\nThe empirical evaluation involves training Gemma-2-2B on MATH and ARC-AGI-1 (easy subset), and baseline against another pass@k estimator (Tang et al, 2025). These experiments show improved pass@k performance when directly optimizing for the same k, as well as improvement over pass@k estimator from (Tang et al, 2025). \n\nThe authors also propose a method in which k is annealed during training, leading to models that balance better pass@1 and pass@k performance.\n\nSTRENGTHS AND WEAKNESSES:\n**Summary:** the theoretical contribution of this work is clearly strong; the main crux is whether the much weaker empirical section is sufficient to justify acceptance. In this reviewer's opinion, this paper does not meet the bar for the strength of empirical support in RL-for-LMs theoretical papers published at NeurIPS, hence Borderline Reject. However, I'd happily change my rating to Accept with the addition of several new experiments (see the Weaknesses section).\n\n\n## Strengths\n\nThe paper is very well-written and addresses an important problem: improving the effectiveness of multiple trials by directly optimizing for pass@k / max@k. The derived estimators are (to my knowledge) all novel contributions, and provide a clear improvement over the prior work by generalizing to continuous rewards and cases where sample size $n \\neq k$, as well as reducing the variance of the estimator. I didn't find any issues with the proofs, although it is possible I missed some. \n\nThe method is also very straightforward to apply, since it just transforms the rewards (python implementation in Listing 1 is a nice addition).\n\nThe demonstrated results are quite promising: pass@k performance is generally better when optimizing for the same (or the at least closest) k and consistently outperforms another estimator (Tang et al, 2025). The results are particularly promising for ARC-AGI-1, where the advantages of directly optimizing for k>1 unstucks the training. I also appreciated the illustrative example in Figure 1. \n\n## Weaknesses\n\nThe evidence of the practical usefulness of the derived method is quite preliminary. The following is a list of weaknesses of the Experimental section, together with proposed additions/changes:\n\n1. only one small model (Gemma-2-2B) and two dataset (MATH, easy subset of ARC-AGI-1) (fix: add different and larger base models; possibly also more datasets)\n2. reporting train set performance only (fix: report test set performance instead, which is the more relevant metric for practical usefulness of the method)\n3. no experiments with continuous/max@k estimator (fix: add experiment on some continuous reward domain; could be a coding task, where g(x_j) = % unit tests passed)\n4. all results have only one random seed / no error bars, and the training curves are far from convergence -- this is all non-standard for a RL paper (fix: include $\\geq 3$ random seeds, train longer)\n5. lack of baselines: currently include ony pass@$k^{eval}$ comparison to the same algorithm with pass@$k^{opt}$ estimator (for different values of $k^{opt}$) or the pass@k estimator from (Tang et al, 2025) (fix: include more baselines, inc standard RLVR algorithms like GRPO, a best-of-N approximation method (Amini et al, 2024; Chow et al, 2024; Sessa et al, 2024), training with entropy bonus (which would show the proposed algorithm is incentivising something more interesting that increasing entropy))\n6. the claims of empirical diversity / higher exploration are unsupported: the only provided support is in Fig 2b, showing slightly higher entropy increase when optimizing for higher k -- but this is a poor proxy for diversity of attempted approaches. (fix: report some semantic measure of solutions obtained with pass@1 and pass@k optimization -- distribution of answers, semantic clustering of reasoning traces / programs / proofs --> greater diversity here would be a very exciting result!)\n7. reporting cumulative solve rate -- which seems less important and makes comparison to other RLVR work where this isn't used harder (fix: skip, report pass@k only)\n\n\n## Minor \n\n- the citation format is wrong for this venue\n- the illustrative 1-D example seems misplaced in the Experiments section? Perhaps move into its own section on intuitive explanation of the proposed method\n- specify the dataset in the Figure 2-5 captions\n- Fig 3-5: why not show the same $k^{eval}$ and $k^{opt}$ sets? \n- Fig 3-5: the learning curves matter less than final performance (fine in Fig 4 to show annealing point), perhaps summarize the results with bar plots and the learning curves could be in the appendix\n- in Fig 5: the y-axis cuts off very low -- we don't see the rate of improvement past 0.02/0.1\n- include a full list of training hyperparameters in the appendix\n- there's a tendency to provide interpretation of the results in the Experiments instead of a dedicated Discussion section\n- no proper related work section\n\nQUESTIONS:\n1. Can you give an intuitive explanation for the resulting adjustments to the gradient estimator? (Equations 8 and 18-21) \n\n2. Do you have evidence that optimizing for pass@k leads to higher diversity of solutions beyond simple metrics like entropy?\n\nLIMITATIONS:\nNone.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-08"}