Representation-Based Exploration for Language Models:  From Test-Time to Post-Training
Authors: Jens Tuyls
Dylan J Foster
Akshay Krishnamurthy
Jordan T. Ash
Venue: ICLR 2026 Poster (2026)
Keywords: Exploration
language models
reinforcement learning
test-time scaling
Primary area: reinforcement learning
TL;DR: We find representation-based exploration for language models is helpful both at test-time and post-training time.

Abstract: Reinforcement learning (RL) promises to expand the capabilities of language models, but it is unclear if current RL techniques promote the discovery of novel behaviors, or simply sharpen those already present in the base model. In this paper, we investigate the value of deliberate exploration---explicitly incentivizing the model to discover novel and diverse behaviors---and aim to understand how the knowledge in pre-trained models can guide this search. Our main finding is that exploration with a simple, principled, 
representation-based bonus derived from the pre-trained language model's hidden states significantly improves diversity and pass@k rates---both for post-training, and in a novel inference-time scaling setting we introduce. (1) For inference-time, exploration with representation-based diversity improves efficiency, consistently improving pass@k rates across a variety of models and reasoning tasks. For example, for Qwen-2.5-14b-Instruct we obtain over 50\% improvement in verifier efficiency on almost all considered tasks. (2) For post-training, we show that integrating this exploration strategy into an RL pipeline improves reasoning performance over that of the initial model and over standard RL post-training. For example, on AIME 2024, our post-trained Qwen-2.5-7b-Instruct's pass@80 matches the pass@256 of GRPO on the same model, demonstrating a 3x improvement in test-time sample efficiency. Overall, our findings suggest that deliberate exploration---with the right notion of diversity---is a practical path toward discovery of new behaviors beyond sharpening.

Published as a conference paper at ICLR 2026
REPRESENTATION-BAAED EXPLORATION FOR LANGUAGE MODELS: FROM TEST- TO POST-TREE TRAINING
Jens Tuyls Dylan J. Foster? Akshay Krishnamurthy Jordan T. Ash
Princeton University "Microsoft Research NYC
j jtuylsecs, stuylsees.princeton princeton. edu, [dylanfoster akshaykr, ash. jorddannmmiirroooftt.oo
ABSTRACT
Reinforcement learning (RL) promises to expand the capabilities of language mod- els, but unclear if current RI. techniques promote the discovery of novel behav- iors, or simply sharpen those already present in the base model. In this paper, we investigate the value of deliberate to discover novel and diverse behaviors- and aim to understand how the knowledge in pre-trained models can guide this search. Our main finding is that exploration with a simple, principled, representation-baae language model's hidden states significantly improves diversity and pass@k k rates- both for post- post-training, and in novel inference-tine scaling setting we introduce.
exppiiiinn incentivizing the model
bonus derived from the pre-trained
1. For inference-time exploration with representation-batt based diversity improves efficiency, consistently improving pass@l rates across variety of models and reasoning tasks. For example. for Qwen-2. 2. 5- ab-Instrcct we obtain over 50% improvement in verifier efficiency on almost all tasks.
2. For post-training, we show that integrating this exploration strategy into an RL pipeline improves reasoning performance over that of the initial model and over standard RL post-training. For example, AIME 2024, our post-trained Qwen-2.5 5- 7r77--Innuut''' pass@80 80 matches the pass @ 256 of GRPO on the same model, demonstrating 3x improvement in test-time sample efficiency.
Overall, our findings suggest that deliberate exploration- the right notion of diversity-is practical path toward discovery of new behaviors beyond sharpening.'
INTRODUCTION
Reinforcement learning (RL) promises to endow agents with the ability to discover valuable behaviors autonomously, via closed-loop trial and error. For language modeling tasks with verifiable rewards, such as mathematical reasoning and code generation, post-training with reinforcement learning has already enabled impressive breakthroughs (DeepSeek-AI, 2025; OpenAI, 2024). Still, it is unclear whether contemporary RL implementations for language models attain the full promise of reinforcement learning. Rather than unlocking capabilities not present in the pre-trained model, there is increasing evidence (Yue et al., 2025; Gandhi et al., 2025) that existing RL recipes (Schulman et al., 2017; Rafailov et al., 2023; Ae,, 2025) may simply amplify or sharpen (Huang et al., 2025) behaviors that the base model can already execute, albeit with modest probability. While this can be mitigated through deliberate data curation and some algorithmic interventions (He et al., 2025; Liu et al., 2025; Setlur et al., 2025), data scale and quality are rapidly becoming bottlenecks, particularly in complex, open-ended domains where existing interventions fall short of eliciting desired behavior.
We argue that deliberate exploration exploration-iiccentivzzii behavior-is is an essential ingredient in realizing the full potential of RL for language model reasoning. Exploration has rich history in both the theory and practice of RI., and exploration techniques tailored to deep networks (Tang et al., 2017; Pathak et al., 2017: Burda et al., 2018; Osband et al., 2019) have received extensive investigation in the context of embodied decision making, including game playing and robotic control. These algorithms proceed from scratch, without pre-training,
the model discover truly novel and diverse
Work partially completed during an intersship at Microsoft Research. Wehsite and code: https: hxtp///eppppppppeeeeeeeeep/eeeee pii.p.ggttuu... thub.

---

Published as a conference paper at ICLR 2026
Model
104
20*
Uama
medium
Phi
Reptip sain pamples..so. co--corrett
Repsko
Qwen 78
Owen 148
200
101
107
Task
MATH
MBPP-
1.0
AIME 2025
Re samgles
Game
Exploration helps
10
10a
"""
R samples-to-comeett
Random samples-to-correc
Figure I: Representation-basee inference-time exploration improves verifier efficiency. (Left) We plot the samples-to tilee--tt-corrctt the average number of samples until correct response selected, for a wide range of tasks and models. We compare two inference-time exploration methods: representation- based exploration (Section 3) and naive (random) sampling from the base model. (Right) We display cumples-to-comrect disaggregated to each question in the dataset, for model-task pairs. We find representation-
exploration improves over random sampling for most model-task pairs. For example, for Qwen-2. 111--IIIntruc on GSM8K, MATH, MBPP+, and Game-of-24 See Section 4.1 for details.
obtain over 50% improvement verifier efficiency
yet rapidly learn complex behaviors, demonstrating that they enable learning beyond the sharpening regime. we can equip language models with exploration in a similar fashion, we may be able to advance reasoning capabilities without incurring exorbitant data curation costs.
In spite of the potential benefits of exploration, is unclear which, any, exploration technique from deep RL can be scaled to modern language models. A central challenge involves the scalable quantifi- cation of novelty and behavior diversity- acting on this information-wiln under consideration is the combinatorially large space of language. At the same time, pre-trained language models contain tremendous prior knowledge compared to policies found traditional embodied settings, which may be the key to guiding efficient exploration. This leads us to ask:
the decision space
1. Can the knowledge in pre-trained representations guide the search for novel behaviors?
2. Does deliberate exploration have the potential to move beyond sharpening the base model?
1.1 CONTRIBUTIONS
Toward answering these questions, we focus on understanding whether exploration with diversity bonuses div(æ, y) derived from language model can effectively guide the search for diverse behaviors. We adopt novel methodology (Section 2) in which we first evaluate exploration in simple, purely inference-time setting, then integrate our findings into post-training.
The inference-time selection problem (Section 2). this setting, we aim to select a small set of responses .,, Uk from a large set of candidates V1.....14 MN for given prompt such that the chosen set is as diverse as possible, and has high probability including a positive response. This simple regime allows us disentangle the role of diversity div(x, y) from other complex RL mechanisms, such as optimization and generalization.
Representation- based exploration improves diversity and efficiency. Our main finding is that exploration with representation--based bonus (Section 3) derived from the pre-trained language model's 's hidden states significantly improves diversity and pass@k @k rates-both for our inference-time setting and for post-training. Our specific findings are as follows:
1. Inference-time (Section 4). Inference-time exploration with representation-bun based diversity improves verifier efficiency. For example, obtain over 50% improvement in verifier efficiency
2

---

Published as a conference paper at ICLR 2026
MATH
GSM8K
AIME 2024
0.95
0.99
0.6
0.90
0.98
0.5
Pass@k
0.85
XS.65
0.97
0.4
0.96
0.80
0.95
0.3
0.75
0.94
0.2
0.93
0.70
0.1
2
2
22
2
24
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
GRPO
RepExp (ours)
Unlikeliness
Base Model
Figure 2: @ass@k for RL post-training with exploration. We find that RL generally increases the pass @k k for small values of k compared to the base model, but that exploration is required to improve or even preserve base model pass rates for larger values k. For MATH and GSM8K, RepExp roughly matches improves upon Unlikeliness for k 2 2. For AIME 2024, RepExp is slightly worse than Unlikeliness until & = 64, after which it surpasses Unlikeliness for all larger values k. Shaded areas indicate one standard error. Horizontal arrows indicate the test-time sample efficiency improvement for pass @ 256 of RepExp over GRPO (blue) and Unlikeliness (orange). RepExp is 2.1-4.1x more sample-efficient than Unlikeliness and 3.2-13.4x more sample-efficient than GRPO.
over standard sampling for Qwen- 2-5-144-Instruee on GSM8K, MATH, MBPP+ and Game-of-22 See Figure 1 for overview of our results for this setting.
exploration can be incorporated into RL post- training, where its pass@k @ @k performance is competitive with both GRPO and the base model uniformly for all & (Figure 2). Notably, representation--aane based exploration completely eliminates the "diversity collapse" phenomenon where RL degrades pass @ & with respect to the base model for large (Dang et al., 2025; Yue et al., 2025; Wu et al., 2025). In addition, representation- based exploration induces responses that look much more novel under the base model (Figure 8).
Post-training (Section 5). Representation-
Our findings, particularly these last two points, suggest that deliberate exploration a practical path toward discovery of new behaviors beyond sharpening. Although our experiments focus exploration scheme-for which we already expect that our two-pronged evaluation approach
arguably the simplest principled representation-base substantial performance improvements will enable deeper understanding of the benefits and tradeoffs of more sophisticated strategies, which may help realize the full potential of reinforcement learning for language model reasoning.
Diversity- -uideed generation (Section 4.2). As proof of concept, we also evaluate inference- time exploration algorithm that uses representation-basee diversity to encourage exploration during the rugoregressive generation process isseff. We find that this improves pass@) @k for large k over naive sampling for Qwen-2. 2 on MATH.
PROBLEM SETUP: FROM INFERENCE- --TEEE TO POST-TRAINING
In this section, we describe the two problem settings we consider for exploration: inference-time selection and RL post-training. In what follows, π denotes language model that maps prompt x E X' to a distribution over responses E V, and r* (Ix 8) E {0, denotes a verifiable reward function that measures correctness at task of interest, such as whether the answer math question is correct, or whether a Python program passes unit tests.
Methodology and motivation. The goal of RL post-training to find policy æ that maximizes the expected reward By [r (z, v)]. Given budget k of verifier queries per question at each data collection round, post-training algorithms such as GRPO (Shao et al., 2024) update the model
iid. iteratively, where in each iteration they sample responses y1,..., . 8k T(- x)) per prompt from the current model n, query the verifier for a reward r* (z, y:) for each response, and use observed rewards update the model for the next iteration.
the initial model я has poor support over rewarding behavior-i.e. if rz y) = O, with high probability under y x. x) common RL algorithms such GRPO PPO (Schulman al., 2017)
3

---

Published as a conference paper at ICLR 2026
will not make any progress. This motivates interventions for exploration such as bonuses (Tang et al., 2017: Pathak et al., 2017: Burda et al., 2018; Osband et al., 2019) and alternative sampling strategies (Holtzman et al., 2020; Minh et al., 2025). However, understanding the benefits and tradeoffs of these interventions in RI. post-training is challenging because exploration interacts with optimization and generalization. To isolate exploration from these other considerations, we center our investigation around a task we refer to as inference-time selection, validating interventions in this setting before integrating them into the RL post-training pipeline.
2.1 INFERENCE-TIME SELECTION
In the inference-time selection problem, we aim to use fixed model π build of k responses to a given prompt z that are maximally diverse and have high probability of containing positive response. As a simple baseline, we may independently sample k responses from the model- pootenlltt with high-temperature sampling, nucleus or min-p sampling, or other modified sampling schemes. However, the limitations of these baselines are (i) they may not effectively capture the model's 's understanding of diversity, and (ii) by sampling independently, we may waste verifier queries on redundant responses.
Instead, we focus selection-basee approaches that initially sample large set of candidate responses to the prompt, then use a diversity bonus div(z,)) y) derived from the model to filter this set down to a smaller, more diverse "coreset" (Clarkson, 2010; Feldman et al., 2020) that is passed to the verifier. Formally, we consider the following protocol: For each prompt x, we (1) sample an initial batch of N responses y1....,, , V/N ~ x(. 2)), (2) use the inference-time selection algorithm Alg to select & of these responses (a subset S C [N] of size |S| = k), and (3) query the verifier and record if any of the selected responses are rewarding. That is, we measure pass k;
-y.-..1 [Es.. (1,,
:ES'
Importantly, the filtering algorithm operates without the verifier, and so successfully retaining high-quality responses translates to improved verifier efficiency (i.e., number of responses for which we query the verifier) over the initial set of responses. Thus, a useful diversity bonus div(z, y) should yield a coreset that is maximally "exploratory." in the sense that it is the most diverse set of responses that can be selected for a fixed budget of verifier queries. For example, in math reasoning settings, we would like to select the seriinttbbu-pllussbbi proof strategies for given problem, thus covering the space of potential proofs and maximizing the chance of selecting a correct one.
Remark 2.1. While we mainly introduce inference-time selection as a stepping stone to post-training (i.e., algorithms in this setting are not more compute-efficien than naive sampling, even if they are more -efffeetfffiient),.. we do expect inference- time exploration be useful in its own right for domains where querying verifier is costly or difficult (e.g.. collecting feedback from expert-level annotators), allowing for more sample- and hence cost-efficient data curation. For preliminary results in one such domain, please refer to Appendix D.
2.2 REINFORCEMENT LEARNING POST-TRAINING
As described earlier, RL post-training (e.g., with GRPO or PPO) proceeds by iteratively sampling batches of responses, querying the verifier, and using the feedback to update the current policy, After selecting checkpoint i,, we evaluate performance via pass @ under standard generation, Ey, Ey.....g.><((111 |max; [4] r" (x, y.)]. There are two natural approaches to integrate exploration methods into this process. The first is to adjust the independent sampling process only (c.g., through nucleus sampling or min-p sampling), and the second is to augment the training objective with an exploration bonus div(r,) y). Our experiments focus the latter approach; however, based results for inference-tie time selection, we expect that incorporating representation-batt based exploration into the sampling process will also improve RL post-training performance. Indeed, our two-pronged evaluation is motivated by the hypothesis that diversity bonuses div(r,)) y) that perform well at inference-tine time also perform well post- ttaaiiran
REPRESENTATION--AA
EXPLORATION: INFERENCE- AND RL
Having motivated our setup, we now turn to the question of what diversity bonuses div(r, y) are suitable for exploration with language models. While many metries have been proposed in the literature (Tang et al., 2017; Pathak et al., 2017; Burda et al., 2018; Osband et al., 2019). the challenge in adapting these techniques to language models is to simultaneously (i) capture the
4

---

Published as a conference paper at ICLR 2026
Step 1: Generate
Step 2: Embed
Step Select
Step 4: Output
Prompt ar
ho (zt yc)
Au
M2
Mi: L /
A
Selected: {3/2, Mo. 8/7)
Mai Set 2 6?..."
bg
17
95
Ms: "Note that...
Selection:
11+1 arg max, h(x,, v)) A, A, v).
Update:
M/x: "Firs, consider..."
hx
A,
Ar- 14-1h, 14, 67
1-h Me-1hie
RepExp: Iteratively select responses that maximize elliptical bonus h A, for diversity.
Figure 3: RepExp for inference-time exploration. Given a prompt, RepExp selects a diverse set of responses from a large pool by optimizing elliptical bonuses computed using representations from the language model.
model's understanding and (ii) allow for efficient computation at scale. For example, count- exploration (Tang et al.. 2017) is simple, but unsuited large decision spaces. On the other hand, approaches based on intrinsic curiosity (Pathak et al., 2017), random network distillation (Burda et al., 2018), and posterior sampling (Osband al., 2019) are better suited large or continuous spaces, but require additional learning machinery (i.e., auxiliary networks), which introduces significant complexity when scaling language models.
exploration strategy that avoids these shortcomings: An adaptation of elliptic bonuses and sampling a de facto standard for linear bandits and active learning (Abbasi- Yadkori al., 2011; I; Chu al., 2011; Ash et al., 2021; Henaff et al., 2022: Saran et al., 2023; Foster -ithh a representation derived from the language model's hidden states. This approach is arguably the simplest principled strategy that is appropriate for language models, and already yields significant performance improvements in our experiments. Our use of elliptic bonuses is particularly inspired by Foster al. (2025), who prove that test-time exploration with such bonuses has provable computational benefits in simplified language model setting with frozen features.
We focus experiments
et al., 2025)2 with
At high level, elliptical bonus methods operate over a d-dimensional feature space and adopt linear-algebraic measure of novelty: given previously seen feature vectors h h4-1 the novelty (or bonus) of new feature vector defined as
div(h 11 = h 1
= hygh jCi
These bonuses are grounded in the theory of linear regression: If we a linear model fo = (0, h) on features hy..... h4-1 (with associated regression targets), the prediction error on h will be bounded by div(h h1:11) (Lattimore & Szepesvári, 2020). Thus, div(h h1x=1) reflects novelty, as it will be large for features h that are poorly represented by the training dataset.
To adapt elliptical bonuses to language models, we use representations extracted from the model itself as the feature vectors. Formally, given prompt x and response yi v!..... ...... yf of T tokens, we form the feature vector as ho(a, y:) = + Σt= (2, v/ where ho(r, v/ E IRd is the last-layer hidden state of the model on input (x, y : (the activation prior to the unembedding matrix).
IIdded,, elliptical bonuses are ubiquitous linear bandits and reinforcement learning, the simplest non- tabular RL setting. where they have strong provable guarantees. Beyond this, elliptic bonuses and iterative schemes such as Algorithm have long history in the theory of optimal experimental design (Kiefer & Wolfowitz, 1 1960; Pukelsheim, 2006; Allen-Zhu et al., 2021) and active learning (Cesa-Bianchi et al., 2009; Agarwal, 2013; Gu al., 2014; Chaudhuri al., 2015)
5

---

Published as a conference paper at ICLR 2026
Game
Algorithm 1 RepExp
Mion
Aug. camples-to-comeet
Last
1: input: Embeddings he (abbrv. h),
800
Second to last
generations y/ for prompt z,
budget k; regularization param. A.
600
2: Initialize L + {3ı J. y1 Unif())).
3: Initialize inverse covariance Ao = X Id-
400
4: for 6- 1 to k do
200
A Ma... Ae-1 1+h(a,)) A1-1)(233)))
5: Ar f- A-1
Phi
Owen
Qwen 14B
6: Mt+1 argmax v) T A,,,z.)) yey
Figure 4: Representation ablation. We compare averaging all token representations to using those from the penultimate or final token. Averaging over 2x more sample efficient.
7: LU (1++1).
8:
return: L.
In Figure 4, we ablate this choice by comparing to the effectiveness of using representations at the last token (z, y!!T") or penultimate token (a y; 1-T-1) instead. We reduce dimensionality to 512 using sparse random projection (Li et al., 2006); Appendix B for details.
Representation-basee exploration for inference-time selection. Figure presents RepExp, our main algorithm for inference-time selection using representation--assed elliptical bonuses. Here, given a single prompt z and a set candidate generations y = {MI,.... .,. yN we iteratively select generation that maximizes elliptical bonus via y/++ = arg max ho(a, ( v), leveraging the representations (z, y) described above. We efficiently update the inverse covariance matrix Σ using the Woodbury identity for (d2) time per step (Vetterling Press, 1992). We formally present our procedure in Algorithm 1.
Representation-basse exploration for RL post-training. For post-training experiments, we use the same representations ha y) as above, but directly augment the rewards with el- liptic bonuses instead of performing coreset selection. Concretely, given the current iterate ma iid.
Ta(- x) for each prompt in GRPO, we first sample group of k responses VI..... V/k ~
Letting Σ Ala Et he(a, (i)ho( vı) we define the reward for response y4 as² could
r* (1,,, Wi) + B (1, v:) ( also imagine performing inference- cerrence--ime coreset selection in the loop with GRPO, this approach is more practical and efficient, and it achieves significant improvements in performance. We refer the reader to Section 5 and Appendix C for further details.
hh E- yi), where B > 0 is bonus parameter. While
Why representation-ben
elliptical bonuses? We summarize several desirable properties of these bonuses. First, by leveraging the hidden state of the model in featurization, the bonuses capture rich information about the generations, thereby incorporating the language model's prior knowledge. Second, the method is history-aware". the covariance matrix summarizes previously selected generations, and redundancy with previous selection (in representation space) is penalized. Finally, the method is simple and scalable, involving no additional learning machinery and using rank-one updates to avoid costly matrix inversions.
INFERENCE-TE TIME EXPLORATION: EXPERIMENTAL RESULTS
In this section, investigate the performance of representation--bssee exploration for the inference- time selection problem. We detail the experimental setup in Section 4, present main findings in Section 4. and present additional experiments with "token-level" variant in Section 4.2.
Datasets.
We use the test splits of the following five datasets: MATH (Hendrycks et al., 2021), GSM8K (Cobbe et al., 2021), I), MBPP+ (Liu et al., 2023), Game-of-24 (Yao et al., 2023), and AIME 2025. We chose these tasks as they
easy (GSM8K), medium (MATH), and harder (Game ffame-of-220 ff224, AIME)
'The bonus here can interpreted as leverage score for y (Drineas et al., 2006; Cohen et al., 2015).
'In the RI. setting, we do not let the covariance matrix persist across multiple iterations of the same question, and hence there it is only group-aware.

---

Published as a conference paper at ICLR 2026
Qwen 2.551148--nssttiie
800
RepExp
Model
Samples- to/oorct
700
Random
Lamna 30
600
Lama
500
Phi mediuu Phi
%6 Improvement over random
4
400
300
200
Qwen 0.5B
100
Owen
Qwen
10
20
30
40
50
70 80 100
Qwen 14B
1,400
Phi-4
24
32B
Sample-to--coree -otto-corect
1200
Rep
Random
1000
-100
800
Task MATH GSMBK
Weaker models
Stronger models
600
MBPP+
-150
200
AIME
Game of 24
0
4 Relative rank (based pass@1)
10 20 30 40 50 60 70 00 90 100
Hardness quantile (%)
Figure 5: closer look into when RepExp provides improvement. (Left) For each task, we rank models according to their pass @1 rate (the weakest model has rank O, and the strongest has rank 8). We then plot relative improvement (%) of RepExp over random sampling, sorting by rank on the x-axis. While RepExp hurt weaker models (e.g., Qwen-2. 2. 5B Instrucu,,,
find stronger models almost always benefit from exploration (e.g., Qwen-2. S-14B-Instruch (Right) For two different model-task pairs, we plot the samples-to-corret Hardness is measured by the samples-to to-corect from a high-quality third-pary model (GPT-4o mini). We find that RepExp has the greatest benefit on harder examples (e.g., the hardest 20% of questions MATH). Shaded areas indicate one standard error.
function of question hardness.
difficulty levels in math. In addition, include MBPP+ to verify that findings transfer to the coding domain. For more detailed overview of these datasets, please refer to Appendix B.1.
Models. We consider range of model families and sizes: Phi-3-Medium (Abdin al., 2024a) and Phi -4, Llama-3, 3. 2-3B-Ime 18--nttruct and Llama-3. 3. 1-B 1-8--IInstuc (Dubey et al., 2024), (Abdin al., 2024b), Qwen-2. 5- Mistral-78 (Jiang et al., 2023).
(Qwen et al., 2024) for E {0.5B, 3B, 7B, 14B, 32B), and
Algorithms. In our experiment protocol, we initially draw a pool of N candidate generations from the base model, where unless otherwise specified which we refer to as vanilla settings (for MBPP+, we set top-p = 0.95). Then we compare RepExp with budget & with the baseline of random sampling (without replacement) of k responses from this pool. We consider generating the pool using different samplers such as nucleus and min-p sampling in Figure 6, but always use random sampling without replacement the baseline. See Appendix B. for further details.
temperature
1.0 and top-p 1.0,
4.1 BENEFITS OF REPRESENTATION--AAE BASED EXPLORATION
We present our results as a series of Research Findings (RF), expanding on the findings in Figure 1.
RF1: RepExp improves verifier efficiency across models and tasks. In Figure 1, plot the samples-to-correct defined as the expected number of samples k with which we query the verifier before finding correct answer, for all model-task pairs. We compare RepExp, which picks responses to fixed question according to Algorithm 1, with the random sampling baseline. For both algorithms, we average the samples-to-cos to-correct across all questions in the dataset. Our results show the bulk of the data fall below the line y x. indicating exploration improves over random sampling in most cases. For example, we find RepExp obtains 50% improvement in samples-to-corees for Qwen-2, 2. 5-14--14h-- -eb-Innttuct in MATH, GSMBK, MBPP+, and Game-of-24,
RF2: The benefits of RepExp grow with model strength. Since RepExp relies on the model's internal representations, it is natural to hypothesize that weaker models might have worse represen- tations and thus benefit less from exploration. To validate this hypothesis, we expand the collection of models in Figure include additional weaker models (e.g., Qwen-2, 5-0. BB 5B- Instruct and
7

---

Published as a conference paper at ICLR 2026
0.8
0.8
0.8
0.7
Pass@k
0.7
X4.
0.7
0.6
0.6
0.4
0.5
0.5
2
22
24
2
2
2
2
2
24
2
2
20
2
2
2
2
2
200
Data pool
Method
0.8
0.8
Vanilla
Random
Pass@k
0.7
X4.6
Low temp
RepExp
High temp
0.6
Min-p
Nucleus
0.5
0.5
2a
22
24
2
2
20
29
2
24
2
2
2
k
k
Figure 6: Benefits of RepExp across data pools, for the inference-time exploration setup in Figure (Section 4.1). We plot the pass@k & curve for random RepExp across five different data pools (base samplers). RepExp on top of vanilla generation outperforms random sampling on top of any of the generation strategies. Moreover, except for the high-temperature pool, RepExp over a pool improves verifier efficiency, with 3x to 6x improvement over random sampling for that pool.
Mistral-7b). For each task, we rank models according to their pass@ performance, and plot the relative improvement of representation--asee exploration over random sampling in Figure We indeed observe a strong correlation between model strength and the benefit from representation-bast exploration: weaker models (e.g.. 2wen-2.5-0. 5-0.58 5B) experience benefit or even degradation, while the strongest models (e.g., 2. 5-32B) almost uniformly benefit.
RF3: RepExp provides more improvement for harder questions. Beyond RF2-which provides insight into the benefits of RepExp across models-we also evaluate the benefits across question difficulty, for fixed model and task. To this end, we sort all questions for a given task by their samples-to-coree toerreettt under random sampling with reference model (GPT-40-mini). We then group the questions in bins, each containing 10% of the dataset, and plot the average samples-to-correct for each bin for both RepExp and random sampling. As displayed in Figure (right), RepExp matches or improves verifier efficiency across all bins, with the largest improvements (e.g.. the hardest 20% of questions on MATH). Concretely, on the hardest Game-of-24 questions, we find that RepExp with Phi-4 provides 3x improvement in verifier efficiency.
the hardest bins
RF4: RepExp improves verifier efficiency over standard generation modifications. We now investigate the effect alternative base sampling strategies that might already induce diversity. Using -wen-2.5-78-Inse
on MATH, change the underlying generation strategy to one of five different generation settings: vanilla (no changes), low temperature (r 0.6), high temperature (T 1.5), min-p (Minh et al., 2025) (T 1.5, p 0.05), and nucleus sampling (Holtzman al., 2020) (top-p 0.9). In Figure 6, we find that RepExp improves verifier efficiency in all settings, except for when paired with high-temperature sampling. We suspect this is because high-temperature sampling tends to produce less coherent responses, which may look novel in representation space, yet do necessarily contain correct answers.
4.2 EXTENSION: REPRESENTATION- BASSE EXPLORATION AT THE TOKEN LEVEL
While useful its own right as a testbed for benchmarking viability of exploration methods, one drawback of the inference-time selection setting is that compute-an measured by N, the size of the per-question data pool- pool-may As extension. conduct preliminary investigation into algorithms that use elliptic bonuses to guide the autoregressive generation process itself, removing the need to generate such pool at all.
may need to be rather large relative to for selection to yield improvements.
Representation--asee exploration for autoregressive generation. To guide sampling for improved diversity, given budget k, use features from responses 1,..., generation of the ith response by modifying the logits at every generation step. Specifically, consider the ith generation for given prompt At each position & within the generation, perturb the
guide the
8

---

Published as a conference paper at ICLR 2026
|Vlddieenioonal token-level logit vector as:
(x, V<t) = z (a, vee) + 3- 3- b Vee, V), (x) ERMM
where the bonus ((a) (z, Veri V) is token-level elliptic bonus, defined as:
b (z. Ne.. V) Vho(x, Vcc. o4) (() hz V<c. Uy),
for "/ V. Here (2r V.ce: vyy) is mean-centered Transformer representation for sequence (Vea, 2;); see Appendix B.2 for fürther details.
RFS: RepExp for autoregressive generation improves solve rate. In Figure 7, we visualize the effect of token- level representation-biee based exploration 5-
for on the MATH task. We use two values for the bonus parameter 33 (0.5, 1.0) and compare the @ass@k to vanilla autoregressive gener- ation for the 200 hardest (but solvable) questions in MATH, as judged by GPT-4o-mini. While token-level exploration tends to solve fewer problems compared to vanilla generation when given small budget, this trend reverses when the budget exceeds 512-640 (de- pending on choice for 3). Figure further shows that the improvement in solve rate is largest on the hardest questions. These results are encouraging, though further research is required (e.g., on more tasks and models) before one can draw a definitive conclusion. Further, our implementation is not optimized for efficiency and requires least one additional forward pass per generation step compared to naive sampling.
0.34
-8-
Vanila 1
Low RepExp (050.5)
0.32
Reptap
0.30
0.28
0.26
0.24
Figure 7: Representation-basee exploration at the token level, compared to naive autore- gressive generation (3 0) for inference- time exploration. RepExp improves pass @ for large over naive sampling.
EXPLORATION FOR RL
Following the methodology in Section 2, we now investigate the use of representation- explo- ration to guide the RL post-training process.
Tasks and models. Because there are only 30 questions in AIME 2024, we follow Yu et al. (2025) and use the DAPO-Math-177 17K dataset for training, leaving AIME 2024 for evaluation only. Please refer to Ap- pendix C for exact details on train, validation, and test splits for all tasks.
use Qwen-2. 5-75---nstruse evaluated on MATH, GSM8K, and AIME 2024.
Baselines. We compare our method with three baselines: (1) Unlikeliness (He et al., 2025) modifies GRPO by scaling the extrinsic rewards by value inversely related to the likelihood of a generation under the current policy. (2) GRPO is simply an unmodified version of the original (3) Base Model is the original untrained model included as reference point to see methods can improve upon it, especially at high values of k.
GRPO algorithm.
Representation--asse Exploration (RepExp). We augment the rewards in GRPO with representation- based bonuses as described in Section 3. Concretely, we add sequence-level elliptic bonuses to the binary extrinsic rewards provided by the verifier: r; R(x, y.) b(x, v:) for the jttt rollout yi of given prompt x. As mentioned in Section 3, we specifically use leverage score-like elliptic bonuses, which allow easier control over their scale as they are bounded in [0, 1]. The covariance matrix E used to compute bonuses is re- re-initialized for each batch of RL training. This way, the bonus b(x, vi) measures the novelty y/s only with respect to the other rollouts in the batch pee--pprevioyly generated sequences for æ are not considered. To better maximize the bonus along all relevant directions in representation space, we draw a new random projection of (2, yi) at each optimization step. For further details, please refer to Appendix C.
RF6: RepExp improves pass@k. k. Figure 2 compares @ass@k @k curves after training for all methods. In line with earlier work, we find that all instantiations of GRPO improve the pass @a for small values of k. and that standard GRPO degrades performance relative to the base model for large values of k (Yue et al., 2025). Exploration appears to be essential part of mitigating the latter effect: policies fit using RepExp preserve or improve pass@* k for large values with limited reductions for small k. This phenomenon is pronounced for RepExp than for Unlikeliness.

---

Published as a conference paper at ICLR 2026
MATH
RepExp responses look novel. To provide further insight into whether RepExp is able to move beyond merely sharpening the base model, we
RF7:
Base
RepExp (ours)
run an additional experiment inspired by Figure 4 in Karan & Du (2025). Specifically, we sample single response from the base model, the GRPO post-trained model, and the RepExp post-trained model for each question in the full test set of MATH. We then score all responses under the base model in terms of log likelihood. We plot the resulting histogram Figure We find that responses from RepExp tend to be less likely under the base model, indicating that it generates more novel responses and is therefore not merely performing sharpening. In contrast, notice that standard GRPO exhibits sharp- ening behavior, demonstrated by the movement of probability mass towards the right with respect to the base model. Taken together with RF6, these results suggest that with the right exploration strategy, we may be able to escape the sharpening regime and discover novel model behaviors.
Densty
6
0
Average Log kellhood (base)
Figure 8: Anti-sharpening behavior of RepExp. We plot the histogram of average log-likelihoods evaluated under the base model. While GRPO exhibits sharpening, the responses from RepExp are significantly more novel. See Appendix for results on GSM8K.
DISCUSSION
Related work. Several recent works aim to encourage exploration in language models, either by adapting exploration techniques from deep reinforcement learning, or by augmenting PPO or GRPO ways that are more specialized language models (He et al., 2025; Cheng et al., 2025; Chen et al., 2025b; Zhou et al., 2025; Setlur et al., 2025; Liu et al., 2025). Examples of the former include count- based exploration via pseudo-counts (Bai et al., 2025) and at the outcome level (Song al., 2025), random network distillation (Liu et al., 2024b; Gao et al., 2025), and posterior sampling (Dwaracherla et al., 2024). Examples of the latter include rewarding unlikely-but-corred responses (He et al., 2025), entropy bonuses (Cheng al., 2025), adapting the number of rollouts based question difficulty (Yang et al., 2025), training the model to explore in-context (Setlur et al., 2024), using a learned classifier jointly optimize diversity and quality (Li et al., 2025), adding diversity term based on determinantal point processes to the RL objective (Chen et al., 2025a), and reformulating post-training more-directly maximize the pass N objective (Balashankar et al., 2024; Chow et al., 2025; Chen et al., 2025b; Walder & Karkhanis, 2025; Tang et al., 2025). Among these, we experiment with the unlikeliness reward approach of He al. (2025) as a baseline, due its robust performance and clean implementation. More generally, our work is unique in (1) the specific representation-basee objective, and (2) our focus on inference-tine with minimal confounding factors. See Appendix A for detailed overview.
means to validate methods
Final remarks. Our work shows that deliberate exploration a viable path toward expanding the reasoning capabilities of language models, offering the possibility of discovering novel behaviors that diversity
would be unlikely under naive sampling. While our results show that representation-batt is effective at incentivizing exploration, the algorithm design space for exploration techniques is vast, and there is still much to understand regarding how to best use the knowledge encoded foundation models to guide exploration. Along these lines, natural directions for future work include:
1. Scaling up RL compute, and combining exploration with other techniques known to improve reasoning behavior in RL post-training, such as prolonged reinforcement learning (Liu et al., 2025).
Exploration for autoregressive generation. Our results in Section 4.2 show that incentivizing diversity during autoregressive generation is promising approach to reducing the computational burden exploration, but much remains to be done in terms of (1) understanding which diversity metrics most helpful, and (2) optimizing the implementation to close the compute gap.
3. Beyond verifiable rewards. How can we deliberately incentivize exploration in domains without verifiable rewards, while simultaneously mitigating reward hacking?
10

---

Published as a conference paper at ICLR 2026
ACKNOWLED
We thank Adam Block, Qinghua Liu, and Max Simchowitz for valuable feedback on this work. We thank Manan Tomar, Audrey Huang. Spencer Whitehead, Prabhat Nagarajan, and Karthik Narasimhan for support and encouragement throughout the project.

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
To ensure reproducibility of our results, we have listed all relevant details (hyperparameters, ex- periment resources, etc.) for the inference-tine time experiments in Appendix B, and those for RL post-training in Appendix C. In addition, the code for all our experiments and plots can be found at https://rep-aa gi thub. io. We used Weights Biases for experiment tracking and visualiza- tions to develop insights for this paper.
REFERENCES
Yasin Abbasi- Yadkori, Dávid Päl, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. In Advances in Neural Information Processing Systems, 2011.
Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report. arXiv preprint 24 /2 22208905, 2024a.
Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report. arXiv preprint arXiv:2412. 08905, 2024b.
Alekh Agarwal. Selective sampling algorithms for cost-sensitive multiclass prediction. In Interna- tional Conference on Machine Learning, pp. 1220-1228. PMLR, 2013.
Alekh Agarwal, Mikael Henaff, Sham Kakade, and Wen Sun. Pc-pg: Policy cover directed exploration for provable policy gradient learning. Advances in neural information processing systems, 33: 13399-13412, 2020a.
Alekh Agarwal, Sham Kakade, Akshay Krishnamurthy, and Wen Sun. FLAMBE: Structural com- plexity and representation learning low rank MDPs. Neural Information Processing Systems (NeurlPS), 2020b.
Zeyuan Allen-Zhu, Yuanzhi Li, Aarti Singh, and Yining Wang. Near-optimal discrete optimization for experimental design: regret minimization approach. Mathematical Programming, 186(1): 439-478 2021.
Dilip Arumugam and Thomas L Griffiths. Toward efficient exploration by large language model agents. arXiv preprint arXiv:2504.2098 2025.
Jordan Ash, Cyril Zhang, Surbhi Goel, Akshay Krishnamurthy, and Sham Kakade. Anti- concentrated confidence bonuses for scalable exploration. arXiv preprint arXiv:2110.11 1202, 2021.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, et al. Program synthesis with large language models. arXiv preprint
2021.
Chenjia Bai, Yang Zhang, Shuang Qiu, Qiaosheng Zhang, Kang Xu, and Xuelong Li. Online prefer- ence alignment for language models via count-based exploration. arXiv preprint arXiv:2501.12735 2025.
Ananth Balashankar, Ziteng Sun, Jonathan Berant, Jacob Eisenstein, Michael Collins, Adrian Hutter, Jong Lee, Chirag Nagpal, Flavien Prost, Aradhana Sinha, et al. Infalign: Inference-aware language model alignment. arXiv preprint urXiv:24 /9792, 2024.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv: 12894, 2018.
Shicong Cen, Jincheng Mei, Katayoon Goshvadi, Hanjun Dai, Tong Yang, Sherry Yang, Dale Schuurmans, Yuejie Chi, and Bo Dai. Value-incentiiiied preference optimization: A unified approach online and offline rlhf, 2024.
Nicolo Cesa-Biana Bianchi, Claudio Gentile, and Francesco Orabona. Robust bounds for classification via selective sampling. In Proceedings of the 26th annual international conference on machine learning, pp. 121-128, 2009.
12

---

Published as a conference paper at ICLR 2026
Kamalika Chaudhuri, Sham M Kakade, Praneeth Netrapalli, and Sujay Sanghavi. Convergence rates of active learning for maximum likelihood estimation. Advances in Neural Information Processing Systems, 28, 2015.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint arXiv:2107.03374. 2021.
Ruizhe Chen, Xiaotian Zhang, Meng Luo, Wenhao Chai, and Zuozhu Liu. Pad: Personalized alignment at decoding-time. arXiv:2410.040700 2024.
Yilei Chen, Souradip Chakraborty, Lorenz Wolf, Ioamnis Ch Paschalidis, and Aldo Pacchiano. Enhancing diversity in large language models via determinantal point processes. arXiv preprint arXiv:2509.04784 2025a.
Zhipeng Chen, Xiaobo Qin, Youbin Wu, Yue Ling, Qinghao Ye, Wayne Xin Zhao, and Guang Shi. Pass@k training for adaptively balancing exploration and exploitation of large reasoning models, 2025b. URL https: /
10751.
Daixuan Cheng, Shaohan Huang, Xuckai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. Reasoning with exploration: An entropy perspective. arXiv preprint arXiv:2506. 1...28886..478,, 2025.
Yinlam Chow, Guy Tennenholtz, Izzeddin Gur, Vincent Zhuang, Bo Dai, Aviral Kumar, Rishabh Agarwal, Sridhar Thiagarajan, Craig Boutilier, and Alcksandra Faust. Inference-aware fine-tuning for best-of-n sampling in large language models. In The Thirteenth International Conference on Learning Representations, 2025.
Wei Chu, Lihong Li, Lev Reyzin, and Robert E. Schapire. Contextual bandits with linear payoff functions. In International Conference on Artificial Intelligence and Statistics, 2011.
Kenneth L Clarkson, Coresets, sparse greedy approximation, and the frank-wolfe algorithm. ACM Transactions Algorithms (TALG), 6(4):1-30) 2010.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reicchiro Nakano, et al. Training verifiers to solve math word problems. arXiv:21 14168, 2021.
Michael B Cohen, Yin Tat Lee, Cameron Musco, Christopher Musco, Richard Peng, and Aaron Sidford. Uniform sampling for matrix approximation. In Proceedings of the 2015 conference on innovations in theoretical computer science, pp. 181-190, 2015.
Xingyu Dang, Christina Baek, Kaiyue Wen, Zico Kolter, and Aditi Raghunathan. Weight ensembling improves reasoning in language models. arXiv:2504. 10478, 2025.
DeepSeek-AI. DeepSeek-R1: Incentivizing Reasoning Capability LLMs via Reinforcement Learn- ai/DeepSeek-RR -R1 www.://aan/eeepeee......
ing. 2025. URL http::/:: t/w.thub..ttttt/e com/ deepseek- brrepseek--i/DeepSeeeee pdf.
Petros Drineas, Michael W Mahoney, and Muthukrishnan. Subspace sampling and relative-error matrix approximation: Column-row-base methods. In European Symposium Algorithms, pp. 304-314. Springer, 2006.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The Ilama 3 herd of models. arXiv:2407.2178 1783, 2024.
Vikranth Dwaracherla, Seyed Mohammad Asghari, Botao Hao, and Benjamin Van Roy. Efficient exploration for llms. In Forty-first International Conference on Machine Learning. 2024.
ESM Team. Esm cambrian: Revealing the mysteries of proteins with unsupervised learning, 2024. URL https::/eevolutionasssccll ai/bog// htobbllo//eeeeeo -atpaaaaal.aaallloooommmmmmmbbba
Dan Feldman, Melanie Schmidt, and Christian Sohler. Turning big data into tiny data: Constant-size coresets for k-means, pea, and projective clustering. SIAM Journal on Computing, 49(3):601-657, 2020.
13

---

Published as a conference paper at ICLR 2026
Dylan Foster, Zakaria Mhammedi, and Dhruv Rohatgi. Is good foundation necessary for efficient reinforcement learning? the computational role of the base model in exploration. Conference on Learning Theory (COLT), 2025.
Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars. arXiv preprint wwrXiv:2003.13000 2025.
Jingtong Gao, Ling Pan, Yejing Wang, Rui Zhong, Chi Lu, Qingpeng Cai, Peng Jiang, and Xiangyu Zhao. Navigate the unknown: Enhancing 11m reasoning with intrinsic motivation guided exploration. arXiv preprint arXiv:2505. 17621, 2025.
Quanquan Gu, Tong Zhang, and Jiawei Han. Batch-mode active learning via error bound minimization. In UAI, pp. 300-309, 2014.
Thomas Hayes, Roshan Rao, Halil Akin, Nicholas J. Sofroniew, Deniz Oktay, Zeming Lin, Robert Verkuil, Vincent Q. Tran, Jonathan Deaton, Marius Wiggert, Rohil Badkundri, Irhum Shafkat, Jun Gong, Alexander Derry, Raul S. Molina, Neil Thomas, Yousuf A. Khan, Chetan Mishra, Carolyn Kim, Liam J. Bartie, Matthew Nemeth, Patrick D. Hsu, Tom Sercu, Salvatore Candido, and Alexander Rives. Simulating 500 million years of evolution with language model. Science, 38766736) 858, 2025. doi: 10. 1126//cience.ads// ads0018 18. URL https: doi/abs/10.. 100./..1265550 wwt1/////e//en..ee ads0018.
Andre He, Daniel Fried, and Sean Welleck. Rewarding the unlikely: Lifting grpo beyond distribution sharpening. arXiv preprint arXiv:2506.0235: 2025.
Mikael Henaff, Roberta Raileanu, Minqi Jiang, and Tim Rocktäschel. Exploration via elliptical episodic bonuses. Advances in Neural Information Processing Systems, 35:37631-37646 2022.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In Thirty-fifth Conference on Neural Information P Processing Systems Datasets and Benchmarks Truck (Round 2), 2021.
Ari Holtzman, Jan Buys, Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. International Conference on Learning Representations, 2020.
Audrey Huang, Adam Block, Dylan Foster, Dhruv Rohatgi, Cyril Zhang, Max Simchowitz, Jor- dan T Ash, and Akshay Krishnamurthy. Self-improvement in language models: The sharpening mechanism. International Conference on Learning Representations (ICLR), 2025.
Hamish Ivison, Muru Zhang, Faeze Brahman, Pang Wei Koh, and Pradeep Dasigi. Large-scale data selection for instruction tuning. arXiv preprint 2503. 2025.
AQ Jiang, A Sablayrolles, A Mensch, C Bamffrr, DS Chaplot, D de Las Casas, F Bressand, G Lengyel, G Lample, I. Saulnier, et al. Mistral 7b. corr, (in/2330.06825,, 2023. doi: 10.48550. arXiv preprint ARXIV.2310.06225 10, 2023.
Yuu Jinnai, Tetsuro Morimura, Kaito Ariu, and Kenshi Abe. Regularized best-of-n sampling to mitigate reward hacking for language model alignment. arXiv:2404.01054 2024.
Aayush Karan and Yilun Du. Reasoning with sampling: Your base model is smarter than you think. arXiv preprint arXiv:2510.1490 2025.
Maxim Khanov, Jirayu Burapacheep, and Yixuan Li. Args: Alignment as reward-guides search. arXiv:2402.01694, 2024.
Jack Kiefer and Jacob Wolfowitz. The equivalence of two extremum problems. Canadian Journal of Mathematics, 12:363-366 1960.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposiun on Operating Systems Principles, 2023.
14

---

Published as a conference paper at ICLR 2026
Jack Lanchantin, Angelica Chen, Shehzaad Dhuliawala, Ping Yu, Jason Weston, Sainbayar Sukhbaatar, and Ili Kulikov. Diverse preference optimization. arXiv preprint 2xxi:2550118101
2025.
Tor Lattimore and Csaba Szepesvári. Bandit algorithms. Cambridge University Press, 2020.
Ping Li, Trevor J. Hastie, and Kenneth Ward Church. Very sparse random projections. In Knowl- edge Discovery and Data Mining, 2006. URL https://apt. semanticscholar, org/CorpusID: 7995734.
Tianjian Li, Yiming Zhang, Ping Yu, Swarnadeep Saha, Daniel Khashabi, Jason Weston, Jack Lan- chantin, and Tianlu Wang. Jointly reinforeing diversity and quality in language model generations. arXiv preprini arXiv:2509.02534 2025.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20050 2023.
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, Allan dos Santos Costa, Maryam Fazel-Zarandi. Tom Sercu, Salvatore Candido, and Alexander Rives. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science, 37966637) 2023. doi: 10. 126/ science.ade2574. URL https: www.:.sciene.... wttp:oorg//oooooooottttooo/ttott 11t.:/.111/ssscceeeeee ade2574.
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36:21558 21572, 2023.
Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models. arXiv preprint 250.:::505..2864 2025.
Tianlin Liu, Shangmin Guo, Leonardo Bianco, Daniele Calandriello, Quentin Berthet, Felipe Llinares, Jessica Hoffmann, Lucas Dixon, Michal Valko, and Mathieu Blondel. Decoding-ting time realignment of language models. arXhv:2402.02992, 2024a.
Zichen Liu, Changyu Chen, Chao Du, Wee Sun Lee, and Min Lin. Sample-efficient alignment for llms. arXiv preprin! 2rXiv:24/1.014933 2024b.
Nguyen Nhat Minh, Andrew Baker, Clement Neo, Allen G Roush, Andreas Kirsch, and Ravid Shwartz-Ziv. Turning up the heat: Min-p sampling for creative and coherent Ilm outputs. In The Thirteenth International Conference on Learning Representations, 2025.
OpenAl. Introducing openai ol. Blog, 2024. URL httpss//openai..oo com/o1/.
Ian Osband, Benjamin Van Roy, Daniel Russo, and Zheng Wen. Deep exploration via randomized value functions. Journal of Machine Learning Research, 20(124):1-62, 2019.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration
by self-supervised prediction. In International conference on machine learning, pp. 2778-2787 PMLR, 2017.
Friedrich Pukelsheim. Optimal design of experiments. SIAM, 2006.
Yang Qwen, Baosong Yang, B Zhang, B Hui, B Zheng, B Yu, Chengpeng Li, D Liu, Huang, H Wei, et al. Qwen2.5 technical report. arXiv preprint, 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 2023.
V. K. Rohatgi and A. K. Md. Ehsanes Salch. An Introduction to Probability and Statistics. John Wiley & Sons, Inc., 3rd edition, 2015.
15

---

Published as a conference paper at ICLR 2026
Akanksha Saran, Safoora Yousefi, Akshay Krishnamurthy, John Langford, and Jordan T Ash. Stream- ing active learning with deep neural networks. In International Conference on Machine Learning, pp. 30005-30021. PMLR, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv: (at:::7770034444 2017.
Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated process verifiers for Ilm reasoning. arXiv preprint arXiv:2410.08146. 2024.
Amrith Setlur, Matthew YR Yang, Charlie Snell, Jeremy Greer, Ian Wu, Virginia Smith, Max Simchowitz, and Aviral Kumar. e3: Learning explore enables extrapolation of test-time compute for llms. arXiv preprint arXiv:2506.0900 2025.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits mathemat- ical reasoning in open language models. arXiv preprint 20020:2402.03300, 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng. Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409. / 19256, 2024.
Ruizhe Shi, Yifang Chen, Yushi Hu, ALisa Liu, Noah Smith, Hannanch Hajishirzi, and Simon Du Decoding-time language model alignment with multiple objectives. MXXv:24006.1883. 2024a.
Ruizhe Shi, Runlong Zhou, and Simon S Du. The crucial role of samplers in online direct preference optimization. arXiv preprint arXiv:2409.19605 2024b.
Yuda Song, Julia Kempe, and Remi Munos. Outcome-based exploration for IIm reasoning. arXiv preprint arXiv:2509.06941 2025.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAI Xi Chen, Yan Duan, John Schulman, Filip DeTurck, and Pieter Abbeel. # exploration: A study of count-based exploration for deep reinforcement learning. Advances in neural information processing systems, 30, 2017.
Yunhao Tang, Kunhao Zheng, Gabriel Synnaeve, and Remi Munos. Optimizing language models for inference time objectives using reinforcement learning. In Forty-second International Conference on Machine Learning, 2025.
William T Vetterling and William H Press. Numerical recipes: example book C. Cambridge University Press, 1992.
Christian Walder and Deep Karkhanis. Pass k policy optimization: Solving harder reinforcement learning problems. arXiv preprint (tl.:2555.//20.. 2025.
Fang Wu, Weihao Xuan, Ximing Lu, 7aid Harchaoui, and Yejin Choi. The invisible leash: Why rlvr may not escape its origin. arXiv preprint arXiv:2507.14843, 2025.
Tengyang Xie, Dylan Foster, Akshay Krishnamurthy, Corby Rosset, Ahmed Awadallah, and Alexander Rakhlin. Exploratory preference optimization: Harnessing implicit Q gppapproximttion for sample-efficient RLHF arXiv:2405.21046 2024.
Wanqiao Xu, Allen Nie, Ruijie Zheng, Aditya Modi, Adith Swaminathan, and Ching-An Cheng. Provably learning from language feedback. arXiv preprint arXiv:2506. 10341, 2025.
Zhicheng Yang, Zhijiang Guo, Yinya Huang, Yongxin Wang, Dongchun Xie, Yiwei Wang, Xiaodan Liang, and Jing Tang. Depth-breadth synergy in rlvr: Unlocking Ilm reasoning gains with adaptive exploration. arXiv preprint arXiv:2508. 13755, 2025.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models, Advances in neural information processing systems, 36:1 1809-11822 2023.
16

---

Published as a conference paper at ICLR 2026
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint 2503 2025.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in Ilms beyond the base model? arXiv preprint arXiv:2504. 138.:35504.13377,, 2025.
Shenao Zhang. Donghan Yu, Hiteshi Sharma, Ziyi Yang, Shuohang Wang. Hany Hassan, and Zhaoran Wang. Self-exploring language models: Active preference elicitation for online alignment, 2024.
Ruiyang Zhou, Shuozhe Li, Amy Zhang, and Liu Leqi. Expo: Unlocking hard reasoning with self-explanation-guiine guidedd reinforcement learning. arXiv preprint arXiv:2507.02884 2025.
17

---

Published as a conference paper at ICLR 2026
A ADDITIONAL RELATED WORK
Exploration at test time. Test-time alignment techniques for language models are an active area of research with many complementary threads (Khanov et al., 2024; Chen et al., 2024; Shi et al., 2024a; Liu et al., 2024a; Jinnai et al., 2024; Shi et al., 2024b), but exploration has not typically been the focus of this line of work.
Most closely related to work, Setlur al. (2025), propose a test-time exploration approach based on the idea of learning to explore in-context. They propose to encourage exploration within a long chain of thought by training the LLM to chain operations such as generation, verification, and refineentt together in search of a solution. This is somewhat complementary to our inference-tine exploration framework, which aims to improve diversity across parallel generations once the model is fixed; these techniques could potentially be combined.
Also related, Xu et al. (2025) consider the problem of learning from language (non-verifiable) feedback, and propose iterative prompting approach to enable exploration test time; their work focuses on simpler exploration domains, but with more difficult, implicit feedback.
Exploration in RL post-training Exploration in RL post-training for reasoning is a growing area of research, motivated by the observation that standard techniques tend simply sharpen responses already covered by the base model (Yue et al., 2025; Gandhi et al., 2025; Wu et al., 2025). A number of recent works, discussed below, aim to improve diversity and expand the reasoning frontier by incorporating bonuses into the GRPO objective or by otherwise augmenting it. Briefly, our work unique in terms of (1) the specific representation-batt based diversity objective we focus on, and (2) our focus on inference-time exploration as means to validate diversity metrics before applying them to post-training.
He et al. (2025) introduce an unlikeliness reward to GRPO, which reweights the reward by ranking generations according to their unlikeliness under the sampling policy. Unlikeliness reward is a form of diversity metric, similar our representation-base based diversity metries. Cheng et al. (2025) observe that high-entropy (high uncertainty) tokens in the model's output often correspond to critical reasoning steps, and augment the GRPO objective with entropy bonuses to encourage exploration at these high-entropy steps. Entropy can be seen as another form of diversity metric in our setup. Another option is to learn the diversity metric as in Li et al. (2025), who use learned classifier to determine whether pair of responses is semantically equivalent. They then use the classifier score to scale the reward in GRPO to joinly optimize quality and diversity. Our work, in contrast, does not require training any auxiliary models for computing diversity.
Various works (Balashankar et al.. 2024; Chow et al., 2025; Chen et al., 2025b; Walder & Karkhanis, 2025; Tang et al., 2025) formulate the problem of directly post-training maximize the pass@ N ob- jective, deriving approximate gradient estimators and using them for policy optimization. As discussed in Chow et al. (2025); Chen al. (2025b), these gradient estimators implicitly encourage exploration. since they allow the model to distribute probability mass across a more diverse range of responses when it is uncertain about the correct answer. Our work instead focuses on using the language model representations to deliberately incentivize novel behaviors, including in novel inference-time setting.
Zhou et al. (2025) consider a setting where ground truth answers are available (as opposed to just rewards), and propose to encourage exploration by prompting the model generate self-explanattins for the ground truth answers and incorporating this as an SFT term in the GRPO loss. Unlike our method, their approach does not directly optimize for diversity, and cannot be used in settings where ground truth answers are unavailable (c.g. coding).
Yang et al. (2025) investigate the role of "breadth" " (batch size) and "depth" (number of rollouts) in RLVR. They show that increasing breadth through full batch updates and increasing depth through more rollouts for harder questions has complementary benefits and overall improves pass@) I and pass @k performance. We view this as orthogonal and potentially complementary to our work.
Lanchantin et al. (2025) introduce diverse preference optimization (DivPO), an alignment method to optimize for both quality and diversity. In contrast our work, their method designed for the RLHF setting and applied to non-reasoning tasks (e.g. creative writing).
Concurrent work of Song et al. (2025) adapts tabular UCB-style bonuses to language model post- training with GRPO, but their saproach-unlike representation-base exploration-is only suitable
18

---

Published as a conference paper at ICLR 2026
for domains with small, discrete set of possible outcomes. Other concurrent work of Chen et al. (2025a) optimizes a diversity term along with the rewards, where the diversity term captures the volume of the responses in representation space by computing the determinant of the gram matrix. While related, our method instead adds a leverage-based score to the reward of each individual response.
Lastly, Liu et al. (2025) take a complementary approach and aim incentivize reasoning beyond the base model through (1) prolonged RL training (increasing the overall amount of training steps), and (2) periodically resetting the reference model; they show that this can increase pass N performance beyond the base model in a variety of reasoning tasks. This approach is complementary. and could likely be combined with our techniques.
Representation-basee diversity. Our findings regarding benefits of inference-time exploration with appresentationnbasee diversity parallel the findings of Ivison al. (2025), who evaluated the effectiveness of various data selection schemes for instruction tuning, and found similar representation-basse based scheme be the most effective when normalized for compute. In addition, there is a long line of work relying on representation-batt non- non-LLM settings (Agarwal et al., 2020a;b; Henaff et al., 2022; Ash et al., 2021).
exploration for RL through elliptic bonuses in
Adapting exploration techniques from deep reinforcement learning. Various papers have adapted exploration techniques from deep learning language models, including Bai et al. (2025) (count-based exploration), Gao et al. (2025) (random network distillation), and Liu et al. (2024b); Dwaracherla al. (2024) (posterior sampling). 5 These works show initial promise in terms sample complexity benefits, but their potential to explore beyond the base model in reasoning domains has not been evaluated to our knowledge. In addition, these methods require additional learning machinery (e.g., auxiliary networks), which introduces significant complexity when scaling to language models.
Theoretical analysis of language model exploration. On the theoretical side, our work draws on Foster et al. (2025). who prove that test-time exploration with representation- provable computational benefits in simplified linear setting. Our RepExp algorithm for test-time exploration can be viewed as simplified, practical adaptation of their theoretical algorithm.
diversity has
Other theoretical works on exploration with language models include the XPO algorithm of Xie et al. (2024) and related algorithms by Cen al. (2024); Zhang et al. (2024), 6 which augment the Online DPO objective with exploration bonuses inspired by the optimism principle. To our knowledge, these techniques have only been evaluated on RLHF tasks, and Foster et al. (2025) show that there may be computational barriers implementing them in way that is faithful to the theoretical guarantees.
B DETAILS FOR INFERENCE- TIME EXPERIMENTS (SECTION 4)
B. DETAILS FROM SECTION 4.1
Hyperparameters. In Algorithm 1, we set À 1.0. For all models and tasks, we perform a sparse projection from the respective model hidden dimension to = 512.
Preprocessing. In Algorithm 1, after obtaining the representations he for every generation y for fixed prompt x, we sparse project all representations down and then mean-center where the mean is taken across the response-level representations.
Datasets. Below we provide a brief overview of all datasets along with relevant numerical details. Note that "vanilla" sampling settings refer to 7 = 1.0, top-p 1.0, and min-p = 0.0. We also do not use top-k sampling in any of the coreset experiments. Finally, we only use the test split of every dataset for all our inference-time experiments, unless specified otherwise.
MATH. This dataset contains 12.5k problems from high school math competitions, split into 7.5k training examples and 5k test examples. For each question the test split, we generate 6400 responses using vanilla settings and set the maximum response length per generation to 512 tokens.
SSee also Arumugam & Griffiths (2025), which uses pre-trained model simulate posterior sampling in-context for multi-turn sequential decision making tasks.
*Cen et al. (2024); Zhang et al. (2024) concurrently proposed similar algorithms to XPO, but did not provide non-trivial theoretical guarantees (e.g., guarantces that indicate benefits over purely passive exploration).
19

---

Published as a conference paper at ICLR 2026
GSM8K. This dataset contains 8.79k grade school math word problems, split into 7.47k training examples and 1.32k test examples. For each question in the test split, we generate 6400 responses using vanilla settings and set the maximum response length per generation to 512 tokens.
MBPP+ This dataset contains 378 basic Python programming problems that are a curated subset of the full MBPP (Austin al., 2021) dataset with more test cases. Since the dataset does not come with any train or test splits, we use the full set of questions for our experiments. For each problem, we generate 6400 responses using vanilla settings, except that we top-p 0.95. We set the maximum response length per generation to 768 tokens.
Game of 24. This dataset contains 1.36k questions that specify four integers that need to be combined using basic arithmetic operations (+, -, zz/) to equal 24. For each question, we generate 6400 responses using vanilla settings and set the maximum response length per generation 512 tokens. We use the version available at https: htt.://uugggggaeeeeeeee co co/daaaastt///lllee///ee -amme.
AIME 2025. This dataset contains the 30 problems taken directly from the 2025 edition of the American Invitational Mathematics Examination (AIME). For each question, we generate 8192 responses using vanilla settings and set the maximum response length per generation to 8192 tokens.
Experiment resources. We used vLLM (Kwon et al., 2023) on 2 (depending on the size of the model) NVIDIA A100 40GB GPUs per model-task pair to generate the data pools for all questions in the dataset.
For random sampling. we estimate the average number of samples to take (without replacement) from the data pool to find the first correct one as:
Estimating samples-to-corres
N
samples-to-correct c + 1
where N indicates the size the data pool and e indicates the number of correct samples in the pool. Please refer to Rohatgi & Salch (2015) for proof. For the representation-bust -asedd exploration algorithm described in Algorithm 1, we perform trials per question where we record the samples-to-coren for each and take their average.
Estimating pass @ess@k To compute the pass@1 @k values plotted in Figure 6, we follow Chen et al. (2021) and use the following unbiased estimator:
D
pass @k = ED
10
where DD indicates the dataset and [D] indicates its size.
B.2 DETAILS FROM SECTION 4.2
Hyperparameters. Similarly to Section 4.1, initialize 2T0) -- We set À = 0.1. For all values of 8, we use top-p = 0.95 and top-k 128. At every time step we use batch size of 64 to compute he (x, yet. ) for all dy V where ;; > oo (note that a logit uj is set to -oo if it gets filtered out by either the top-p or top-k filters mentioned earlier).
Computational expense. Note that computing the bonus b(x, Vet, V) requires forward pass through the model per token in the vocabulary at every time step f. While these can be batched together since all tokens share the same prefix, this is still prohibitively time and memory intensive. To mitigate this, we combine this method with nucleus and top-k sampling such that the bonus will only need to be computed for most k V tokens.
Dataset construction. Due to the large computational cost of experiments, we only focus on MATH using Qwen-2. 5-bb-Inst In addition, we do not evaluate on the full test split of MATH, but instead use a subset consisting of the 200 hardest questions as ranked by GPT-40 mini. Specifically, we sampled 1024 responses for each question in the MATH test split using GPT-4o mini to estimate the per-question pass @ 1. We threw out all questions for which the pass@) I was 0 (indicating not a single response was correct among all 1024). sorted the remaining questions, and kept the 200 questions with the lowest p
20

---

Published as a conference paper at ICLR 2026
Estimating samples-to-correet. We collected up to 1024 generations per question (stopping early once the correct answer was found). Since the resulting conples-to-corret value found can have high variance due to the inherent randomness in the generation process, repeat this process 5 times with a different seed every time. This then results in a total of 200 x = 1000 data points, minus a few data points that didn't finish running in time, for effective total of 985 data points used in Figure 7.
Experiment resources. Experiments were run on a combination of NVIDIA A100 40GB, NVIDIA A100 80GB, and NVIDIA H100 80GB GPUs. Every run indicates one seed for fixed question and performs up to 1024 generations. We used one GPU (one of the several mentioned earlier) per run, adjusting the forward batch size compute the elliptical bonus down from 64 to when using the 40GB GPUs.
Computing the inverse covariance. The inverse covariance matrix (3) includes all mean-centered hidden representations for all generated tokens in all - complete sequences generated so far for a fixed prompt r. Note that the mean ((() of the raw hidden representations hg is computed as:
T3
ho veth
H
T j=0
H
==
where Tj indicates the length (in number of tokens) of response j. Because of this, the mean changes after every generation This means some care required when computing the inverse covariance matrix ( with mean-centered hidden representations. To account for this, we will separately keep
track of the inverse covariance matrix (x) with non-mean-centenn hidden representations well as
the mean (())) Then, we compute the mean-centered inverse covariance matrix [(t)
10 ((a 1n 1-1
= E-1
(
1/H + 1T TT) 100
This result is immediate from Proposition B.1.
Proposition B.1. Given the current generation step i, the non-mean-cemtn centeeed inverse data covariance matrix 5t-1 and the current mean (1) the mean- -neaeecenteeed inverse data covariance matrix 2() is
(i) given as
Σー 0T
=
=
Σ T3. 5=0
(
-1/H+
Here, T₁ indicates the length (in number of tokens) of response j.
Proof of Proposition B.1. We can write
1-1 T3
Σイ
(ht
14
(00T
1=
i-1
i-1 T
i-1 I
i-1 T2 ΣV
ΣΣ
( (1
(
11)T
(
(n0
-=
f=
1=
1=0
[-1 ΣΣ (
T, ΣΣ
= =
ΣΣ (1)
(WOOTT (n0)
(1
(T)
1=
j=
500 Hp
H(p
(4(07
= 5000 2Hp ByT
+
H
= =
21

---

Published as a conference paper at ICLR 2026
RepExp 0.5] High RepExp (
Replixp 0.51
0.32
-
High RepExp 1 1.0)
Solve (46)
Passak
0.28
0.26
0.22
0.20
Hardness quantile (%)
Figure 9: Representation--uusdd exploration at the token level, compared to naive autoregressive generation (3 0) for inference-time exploration. (Left) We added shaded areas indicating one standard error to Figure 7. (Right) When binning the questions by hardness (judged by samples-to- correct for GPT-4o-ming 95% paired bootstrap Cls.
solve rate improves the most the hardest questions. Error bars indicate
Inverting E we conclude that
(0 500
ptts)
1 + +
where used the Woodbury matrix identity lemma? in the last step with U = =1() C - H, and V
P())
We exclude representations from the current sequence in { to keep the generation from veering off topic, and update (4) after each generation i using T, consecutive applications the Sherman- Morrison update Algorithm 1, one for each hidden representation of the output tokens. Finally, we found it necessary for numerical stability to perform all covariance related computations in double precision.
Additional plots. Figure 9, we provide revised version of Figure 7 where we add shaded areas indicating standard error the left plot, and we additionally add 8 = 0.5 to the right plot.
C DETAILS FOR RL TRAINNG EXPERIMENTS (SECTION 5)
Hyperparameters. We use verl for training (Sheng et al., 2024), and provide full overview of all common hyperparameters in Table 1, all hyperparameters specific to unlikeliness in Table 2, and all hyperpameters specific to RepExp Table 3. Note that for AIME 2024, we adjusted the maximum prompt length to 2048, the maximum response length to 8 8192, the train batch size to 512, the ppo mini batch size to 128, and the ppo micro batch size per gpu to 8. Also, for RepExp on AIME 2024, we increased the sparse projection dimension from 32 128.
Algorithm details. We note that mean-center the representations ho that used compute the elliptic bonuses as described in Section 3, where the mean is taken over all the response-level representations of the current group of rollouts for fixed prompt æ. In addition, bonus for questions where all rollouts in the batch are incorrect, as we found this to empirically hurt performance.
do add
Dataset splits. For MATH, use the original 7.5k train split for training, MATH-500 (Lightman et al., 2023) for validation, and the 4.5k (originally 5k minus the problems in MATH-500)) test split
hhttps://en http:p/dii...gg.iiii////o//ooooe
identity
22

---

Published as a conference paper at ICLR 2026
MATH
GSMBK
AIME 2024
0.95
0.6
0.98
0.5
Pass@kk
0.85
0.97
X3.0
0.4
0.96
K2.1
0.80
0.95
0.3
0.94
0.2
0.93
k
k
--- GAPG
-e- RepExp (ours)
-B- Uniieelneess
-B- Base Model
Figure 10: @ass@k for RL post-training with exploration. We plot Figure on linear axes to provide additional perspective.
Table Common RL hyperparameters.
Hyperparameter
Value
Table 2: Unlikeliness hyperparameters.
Learning rate
1e-6
Hyperparameter
Value
Maximum prompt length
1024
Maximum response length (tokens)
1024
Brank
0.25
Train batch size
1024
No bonus if all rollouts correct
True
PPO mini batch size
256
PPO micro batch size per gpu
16
PPO epochs
Table RepExp hy perparaaasssss..
KL loss coefficient
0.0
GRPO group size (rollouts)
8
Hyperparameter
Score
Entropy coefficient
0.0
8
0.01
Log prob micro batch size per gpu
16
Sparse projection dimension
32
Tensor model parallel size Number of GPUs
2
No bonus if all rollouts incorrect
True
8
Validation frequency
20
for testing. For GSM8K, we use the original 8.79k train split for training, the first 512 examples from the test split for validation, and the remaining 807 examples from the test split for testing. Finally for AIME 2024, we use 4096 examples randomly chosen from the full DAPO-Math-17 dataset for training and use the full AIME 2024 dataset both for validation and testing.
Modifications to unlikeliness baseline. The original unlikeliness method from He et al. (2025) combines the reward modification described in Section 5 along with several other modifications to the underlying GRPO mechanics:
They only include samples for which the resulting rollouts have nonzero advantages in the batch sent for training. Specifically, questions where either none of the rollouts are correct or all of the rollouts are correct are thrown out. ensure batch sizes stay roughly equal, the authors implement a buffer mechanism that collects samples until the buffer reaches a target batch size.
They increase the number of ppo epochs from to 2 as they find this helps further increase the @.
They use high KI. penalty coefficient of 0.1 they find this helps prevent the pass from decreasing.
Since our primary aim in this section is to isolate and compare the exploration mechanisms of our method with others, leave out the additional changes described above when running the unlikeli- ness baseline. Furthermore, this allows us to use the exact same underlying GRPO hyperparameters for all methods, making the comparison much more clean.
23

---

Published as a conference paper at ICLR 2026
Protein sequence generation with ESM3-open
1.0
Random RepExp
x56.9
0.8
0.6
Pass@k
0.4
0.2
0.0
2
21
2
23
24
25
26
27
2
2
2
Figure 11: Benefits of RepExp for protein sequence generation. We plot the pass@k curve for random vs. RepExp when performing inference-time selection on protein sequences from ESM3-open. Compared to random sampling, RepExp provides up to 56.9x improvements in verifier efficiency.
Checkpoint picking. For each seed per method, pick the checkpoint during training that achieves the highest pass@ @ on the respective task's validation set and use the resulting checkpoint for evaluation.
Evaluation. We evaluate each final checkpoint (picked in the way described earlier) on the test split of each respective task by sampling 256 responses per question using vanilla sampling parameters (r 1.0, top-p 1.0). We then estimate the pass @k k exactly as described in Appendix B. We run seeds for methods on all tasks and average the resulting pass@k curves. In addition, we provide an alternative view of Figure 2 in Figure 10.
Experiment resources. We run all methods on all tasks using 8 NVIDIA H100 80GB GPUs per seed per method for day.
Computational overhead. On MATH, we estimated RepExp to reach about 0.73x the steps/hour throughput of GRPO, In other words, RepExp requires about 1.37x higher wall-clock time per step. Note that this overhead comes from an extra forward pass that's required to compute the hidden representations of the sampled responses at each iteration of the algorithm. However, if either (I) one were to use KL constraint (which we do not in our experiments) or (2) one were to use the hidden representations from Ni (the current policy iterate) instead of Reef (the base model), then the extra forward pass would not be needed and RepExp would have the same throughput as GRPO.
D PROTEIN SEQUENCE GENERATION
As pointed out in Remark 2. I, we believe inference-time exploration can be useful in own right for domains where verification is expensive. While extensive study of this merits its own paper, provide some preliminary experiments the domain of protein sequence generation. In this domain, verification requires lab work, which is much more time consuming than sampling protein sequences from a generative protein model. Since we can't 't actually perform the lab work, we use two proxy metrics as described below.
24

---

Published as a conference paper at ICLR 2026
MATH
GSMBK
GRPD
Repēxp (ours)
Density
Average Log. ligeliiklddddd (base)
Average Log likellhoad (base)
Figure 12: Anti-sharpening behavior of RepExp. We plot the histogram of average log-likelihoods evaluated under the base model for test responses sampled from base, GRPO, and RepExp. While the response likelihoods from GRPO either roughly stay the same or increase, the responses from RepExp are significantly more novel as indicated by the low log- logeliilisoods under base.
For our experimental setup, we follow the unconditional generation setup in Hayes et al. (2025) and sampled 2048 protein sequences for fully masked sequence prompts ranging length from 64 to 916 with increments of size 4 (i.e. 64, 68, ... 916) using ESM3-open. Then, we perform structure prediction for all sampled sequences using ESMFold (Lin et al., 2023), which returns pLDDT and pTT scores per sequence. We count sampled sequence as plausible when pLDDT > 0.8 and pTM 0.8. To get representations for every sequence, we use esmc-600m, parameter version of ESM C (Hayes et al., 2025; ESM Team, 2024), and average the last layer hidden representations. Finally, we perform inference- time selection for every prompt and its 2048 corresponding sampled sequences, using both random selection and RepExp. We find the average samples-to-con to-correct for random to be 240.3 and for RepExp to be 6.7. This corresponds to 35.9x verifier efficiency improvement. We also plot the corresponding pass @ curves in Figure 11, for which we find up to 56.9x improvements in verifier efficiency.
600M
We find these results to be very promising and suggestive that method be practical in domain where verification is genuinely expensive.
E BEYOND SHARPENING: FULL RESULTS
In Figure 12, we provide results for both MATH and GSM8K. We find that on GSM8K the shift toward lower likelihood responses from RepExp evaluated under the base model is even more dramatic.
25


Reviews:
[{"review_id": "6HJp0t7uvu", "reviewer": "Reviewer_ppuG", "summary": "This paper investigates representation-based exploration as a means to improve reasoning and diversity in language model behavior, both at inference time and during reinforcement learning post-training. The authors propose a simple yet principled elliptic bonus derived from a model’s hidden states to encourage exploration. Across several benchmarks (MATH, GSM8K, MBPP+, Game of 24, AIME), the approach yields significant improvements in verifier efficiency and maintains performance or improves at large Pass@k. The effect holds for larger models and, specifically, harder tasks.", "questions": "- Why did you choose Qwen-2.5-7b-Instruct for the post-training experiments?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Adapting elliptical bonuses to language-model representations is well motivated, interesting, conceptually elegant, and grounded in prior exploration theory.\n- Comprehensive empirical evaluation across multiple model families, scales, and sampling strategies.\n- Significant improvements in sample efficiency.", "weaknesses": "- The proposed exploration strategy significantly degrades performance for smaller language models (e.g., Qwen-0.5B, Mistral-7B).\n- The paper’s main narrative emphasizes discovering novel behaviors beyond sharpening, yet the results primarily reflect improvements in verifier sample efficiency. For large k, performance remains comparable to the base model, suggesting that the method broadens coverage rather than uncovering qualitatively new capabilities. The work would benefit from being reframed explicitly as a study in verifier-efficiency optimization rather than behavioral discovery.\n- In the post-training experiments, the method consistently underperforms GRPO and Unlikeliness for small values of k.\n- All benchmarks employed (GSM8K, MATH, MBPP+, AIME, Game of 24) feature cheap, automatic verifiers where verification cost is negligible relative to model inference. The paper would benefit from experiments in domains where verification is more expensive", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KopLyHJMPF", "reviewer": "Reviewer_gWc2", "summary": "The paper studies representation based exploration for language models. An timely and highly important problem. Discovery of new and diverse behaviours would be a critical component for future revolutions in many AI4 science applications for example. \n\nThe paper is exploring this question using many established based models and a decent selection of downstream tasks including AIME.", "questions": "See the question for code and data in weaknesses?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The paper is really nicely written and structured. There was a dedicated effort to make it easy to understand and follow the structure. \n\nThe models studied include Llama, Mistral, Qwen and Phi and the tasks are representative.", "weaknesses": "Given the importance of the question in many domains I would have loved to see a similar study for protein language models and or how far the gap is in that area. That probably will however merit its own paper and is out of scope. \n\nRight now it states that \"In addition, we have uploaded a zip file with the complete, anonymized\ncode for all our experiments and plots.\" Will the code, the data and the experimental setup be made publicly available after acceptance?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cYAAxQLAV6", "reviewer": "Reviewer_VhJg", "summary": "•\tPrincipled and simple approach: The use of elliptical bonuses derived from model representations is conceptually clean and grounded in existing exploration theory, avoiding the complexity of auxiliary networks.\n•\tBroad empirical coverage: The paper evaluates across multiple tasks, datasets, and model families, providing a comprehensive empirical picture.\n•\tClear dataset justification: The chosen benchmarks are well-motivated and represent a diverse set of reasoning and coding tasks.", "questions": "1.\tCan the authors clarify why weaker models degrade under representation-based exploration, and whether this correlates with representational quality or another factor?\n2.\tIn the RL setting, what is the computational overhead of computing bonuses compared to standard GRPO training?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "•\tPrincipled and simple approach: The use of elliptical bonuses derived from model representations is conceptually clean and grounded in existing exploration theory, avoiding the complexity of auxiliary networks.\n•\tBroad empirical coverage: The paper evaluates across multiple tasks, datasets, and model families, providing a comprehensive empirical picture.\n•\tClear dataset justification: The chosen benchmarks are well-motivated and represent a diverse set of reasoning and coding tasks.", "weaknesses": "•\tOverstated claims relative to results: The introduction makes strong assertions about “moving beyond sharpening” and “substantial improvements,” but empirical gains are modest and often inconsistent—particularly in RL post-training, where RepExp sometimes performs on par or worse than baselines (e.g., Figure 6).\n•\tLimited discussion of negative results: The method degrades performance for weaker models, yet this is not adequately analyzed or contextualized.\n•\tUnclear motivation and structure: The introduction is hard to follow due to numerous forward references and misplaced motivation (appearing in Section 2). The “Contributions” subsection mixes results with claims, making it difficult to distinguish novelty from outcomes.\n•\tLack of practical relevance: The benefits at inference-time (reducing verifier calls) are meaningful only when verifier cost dominates, but the paper fails to demonstrate this in a realistic application scenario.\n•\tInconclusive post-training value: For large k, RepExp sometimes underperforms Unlikeliness or GRPO; thus, it is unclear when and why this exploration actually helps.\n•\tInsufficient alignment between claims and framing: The paper promises exploration as a path toward discovering new capabilities, but the experiments focus narrowly on pass@k efficiency, not on qualitatively novel behaviors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DPi7KExaPK", "reviewer": "Reviewer_iReu", "summary": "This paper focuses on the problem of exploration in the context of LLM reasoning. Existing RL methods are known to “sharpen” the distribution of responses v/s learning novel strategies for reasoning. To overcome this limitation, the authors propose RepExp—a simple yet principled exploration technique that leverages the internal representations of LLMs. Inspired by the literature of exploration in deep RL, RepExp encourages diversity in reasoning by providing an elliptical exploration bonus based on the model’s internal representations. The authors demonstrate the effectiveness of RepExp in two distinct settings: (1) a novel inference-time selection task, and (2) RL-based post-training. Across multiple datasets and problem types, RepExp consistently outperforms existing methods.", "questions": "1. Have you considered persisting the covariance matrix across training generations? — Can the LLM then explore novel strategies that are applicable across multiple problems?\n\n2. Does RepExp lead to “novel” reasoning patterns? Moreover, is there a way to quantitatively assess the diversity of the generated responses—for instance, analogous to how state coverage is used in reinforcement learning?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. The paper is clearly written and highly accessible. Moreover, the exploration problem in the context of LLM reasoning is both timely and highly relevant, addressing a critical challenge within LLM reasoning.\n\n2. The paper’s introduction of the inference-time selection task—serving both as an evaluation tool for exploration and as a standalone research problem—is an interesting and valuable addition to the study.\n\n3. The proposed method RepExp is well motivated.\n\n4. The results presented in this paper are both strong and compelling, and are effectively communicated through the well-structured Research Findings (RFs).", "weaknesses": "1. Section 2.1: (Line 143) ““Maximally diverse and high-probability of containing a positive response”: The inference-time selection problem is defined as selecting generations based on both diversity and quality. However, from my understanding, RepExp appears to select only for diversity. Could this lead to cases where the LLM produces hallucinated or incorrect reasoning that is nonetheless selected by RepExp due to its diversity?\n\n2. In the RL post-training setting, the covariance matrix is computed using the hidden reprs of all the responses first, and then used to calculate div(x,y). From the motivation of elliptical bonuses (Lines 201-204), div(h,h<i), is the prediction error for a new “h” which is not in the training set. In this setting, isn’t “h” already in the training set? Is this approach then still principled? (prediction error is bounded by div(h|h<i))\n\n3. “this method is history aware” – this line is a bit unclear, as it could be interpreted as div(x,y) is also conditioned on past-optimisation timesteps (which I assume is not the case). I think “history” can be a bit confusing for referring to responses which are often generated in parallel (which are therefore assigned the same generation timestep). \n\n4. How do other exploration-based methods, such as unlikeness or entropy, perform on the inference-time selection tasks? As the authors note, this setting provides a nice way to isolate and assess the impact of the exploration bonus.\n\n5. (Line 403) “not optimized … hence does not yet give a wall-clock time improvement“ – doesn’t this method have a complexity of k (forward passes) * k generations * T? How can this be improved to be faster than naive autoregressive generation?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)