Reward Is Enough: LLMs Are In-Context Reinforcement Learners
Authors: Kefan Song
Amir Moeini
Peng Wang
Lei Gong
Rohan Chandra
Shangtong Zhang
Yanjun Qi
Venue: ICLR 2026 Poster (2026)
Keywords: Test-Time Scaling
Inference-Time Improvement
LLMs
RL
Primary area: foundation or frontier models, including LLMs
TL;DR: LLMs can learn from scalar reward signals at inference time to improve beyond their training data, outperforming existing test-time scaling methods across reasoning, creative, and scientific tasks.

Abstract: Reinforcement learning (RL) is a framework for solving sequential decision-making problems. In this work, we demonstrate that, surprisingly, RL emerges during the inference time of large language models (LLMs),  a phenomenon we term in-context RL (ICRL). To reveal this capability, we introduce a simple multi-round prompting framework, we call ICRL prompting, for inference-time self-improvement. The goal of ICRL prompting is to guide LLMs to perform reinforcement learning during inference for self-improvement on a given task. After each response, the model receives numerical scalar feedback, denoted as a reward. In the next round, we prompt the LLM again together with a context that concatenates all prior responses and their associated rewards. We consistently observe that response quality improves as the context grows. In other words, the LLM can optimize scalar reward signals during inference, exhibiting behavior analogous to reinforcement learning. We evaluate ICRL prompting on Game of 24, creative writing, ScienceWorld, and Olympiad-level math competitions (AIME and HMMT), demonstrating significant improvements over baselines such as Self-Refine and Reflexion. Notably, even when the reward signals are generated by the same LLM, ICRL prompting still improves performance, highlighting a promising new paradigm for test-time scaling.

Published as a conference paper at ICLR 2026
REWARD Is ENOUGH: LLMs ARE IN-CONTEXT REINFORCEMENT LEARNERS
Kefan Song"
Amir Moeini
Peng Wang University of Virginia
University of Virginia
University of Virginia
Lei Gong University of Virginia
Rohan Chandra
Shangtong Zhang University of Virginia
University of Virginia
Yanjun Qi University of Virginia
ABSTRACT
Reinforcement learning (RL) is framework for solving sequential decision- making problems. In this work, we demonstrate that, surprisingly. RL emerges during the inference time of large language models (LLMss, phenomenon we term in-context RL (ICRL). To reveal this capability, we introduce a simple multi- round prompting framework, we call ICRL prompting, for inference-time self- improvement. The goal of ICRL prompting is to guide LLMs to perform rein- forcement learning during inference for self-improvement on a given task. After each response, the model receives numerical scalar feedback, denoted as reward. In the next round, we prompt the LLM again together with context that concate- nates all prior responses and their associated rewards. We consistently observe that response quality improves as the context grows. In other words, the L.LM optimize scalar reward signals during inference, exhibiting behavior analo- gous to reinforcement learning. We evaluate ICRL prompting on Game of 24, creative writing, ScienceWorld, and Olympiad-level math competitions (AIME and HMMT), demonstrating significant improvements over baselines such as Self- Refine and Reflexion. Notably, when the reward signals are generated by the same LLM, ICRL prompting still improves performance, highlighting promising new paradigm for test-time scaling.
INTRODUCTION
For Large Language Models (LLMs) act effective agents novel tasks. they must be able to improve during inference time, a capability often referred to as test-time scaling (Zhang ct al. (025). Learning and search the
general methods that can leverage scaling computation for perfor-
mance improvement (Sutton) 2019), reaching superhuman performance on Chess (Campbell et al.)
2002) and Go (Silver et al. 2016). Search has been successfully applied to LLM self-improvement
in test-time scaling, starting from the simple Best-of-N (Stiennon et al. 2022) to Tree of Thoughts
Yao al. 2023 and Monte Carlo Tree Search (Ding et al. 2024).
Learning, however, has yet to receive the same attention for L.M self-improvement at inference time. In-context (supervised) learning (ICL;Brown et al. (2020)),
supervised learning paradigm, requires expert demonstrations as ground-truth labels. However, such demonstration data are not easily scalable during inference time, which restricts the applicability of ICL to test-time scaling- Thus, LLMs must instead learn from their own generated experience for continual self-immprvemmnt (Silver & Sutton 2024).
"gqual contribution.
"hhnggong Zhang and Yanjun Qi contributed equally as supervising authors. Correspondence to: ke refan@emai i 1. vi rginia annia..ddee edu, amoe @vi rgin nia .du., shangt.ongtvi edu, yg ni edu.

---

Published as a conference paper at ICLR 2026
Reinforcement learning is perhaps the most successful algorithm capable of self- celf-improvement in- dependent of human knowledge (Silver et al.] (217). However, its major successes have primarily appeared in simulated environments (Mnih et al. 2015 Silver al. (016) during the training time of LLMss Luo et al. (202)). These current RL settings fall short in the big world setting (Javed & Sutton 2024), where the real environment is vastly more complex than the agent itself. In such environments, agents will encounter numerous situations far beyond their prior training data and must adapt and improve their solutions on the fly. Bridging this gap requires models that (I) can handle diverse tasks in the real world, where natural language often constitutes an essential ac- tion space (Silver& & Sutton 2025). and (2) can continually improve their solutions during inference, rather than relying on costly retraining for every novel situation.
This naturally raises the question: can reinforcement learning emerge during the inference phase of LLMs? Enabling LLMs to perform RL purely in context provides an elegant mechanism to meet both requirements: L.M provides a general-purposs initial policy, while RL. introduces the capabil- ity for continual self-improvement. Inspired by the first surprising evidence that LLMs can act as in-context learners in supervised settings (Brown et al. 2020), growing body of work has begun to explore in-context reinforcement learning (ICRL: Moeini et al. (2025). However, current instantia- tions are largely restricted to bandit or simulated environments (Monea et al.] [225]] Krishnamurthy et al.) 2024), failing short of addressing many diverse open-ended tasks where natural language is the action space.
In this paper, we bridge this critical gap by demonstrating that LLMs act effective in-context reinforcement learners, an emergent capability that improves performance on diverse, language- based tasks ranging from conducting scientific experiments creative writing to solving olympiad- level mathematics. To reveal this capability, we introduce a simple multi-round prompting frame- work, ICRL prompting. The goal of ICRL prompting to guide LLMs to perform reinforcement learning for self-immpovemmnnt on a task. Initially, the prompt is only the task description. After the L.M generates a response, we give numerical scalar feedbacks for the response, called the rewards. Then in the next round, we prompt the L.M again with the same task description and a context consisting of all previous responses and rewards. So on and so forth. We observe that the quality of the LLM's response increases the context grows. In other words, the LLM is able to maximize the scalar reward signal during the inference time, just like an RL algorithm.
A key design principle of ICRL prompting is minimality. To ensure that the observed gains arise from the emergent RL capacity of LLMs rather than auxiliary mechanisms, we deliberately exclude textual gradients (Yuksekgonul et al. 2025), prioritized experience replay, sampling- based heuris- ties (Zhang et al., 2024] Yang et al. 2024), or additional engineered modules (Brooks et al., 2024). The only supervision provided is the scalar reward itself. This complies with both the reward hy- pothesis (Sutton) 2004]. "that all of what we mean by goals and purposes can be well thought of as maximization of the expected value of the cumulative sum of a received scalar signal (reward)", and the "reward is enough" hypothesis (Silver et al.) 2021). "intelligence, and its associated abilities, can be understood as subserving the maximisation of reward".
To summarize, this paper makes three contributions:
(1): We introduce the ICRI. prompting framework, a minimal design that elicits inference-time self- improvement in LLMs using only scalar rewards. Just as ICI. places (æ. v) pairs in context, our framework places
aataaacttooowaarr tuples with simple meta-instructions in context. This design isolates the LLM's 's intrinsic capacity for ICRL, free from external code or engineered mechanisms. (2): We provide strong evidence suggesting the emergence of RL in LLM's inference time when the ICRL prompting framework is used. Specifically, we demonstrate the maximisation of the scalar reward signal, the provements from the growth of the context, the performance drop with short context, and the perfor- mance drop when the reward is absent. All those observations are well expected for an RI. algorithm.
trade-off in LLM's ' inference time, the performance im-
Essentially, this a "duck test" (Heim 2007 (1 for the inference process.
(3): We demonstrate that ICRL prompting yields significant improvements over self-revision meth- ods such as Self-Refine (Madaan benchmarks including Game of 24, creative writing, ScienceWorld, and Olympiad- mathemat- ies (AIME and HMMT). In Game of 24 and creative writing, the rewards are generated by the LLM itself, yet consistent performance gains are still observed.
al., 2023) and Reflexion (Shinn et all 2023). across diverse
If it looks like duck, swims like duck, and quacks like duck, then it prohahly is duck.
2

---

Published as a conference paper at ICLR 2026
2 BACKGROUND
Reinforcement Learning. RL uses Markov Decision Processes (MDPs) model task, consisting of a state space S, an action space A, reward function S + R, an initial distribution po e ((S) with (SS) denoting the set of probability distributions over S, and a transition function p : S x A + A(S). At time step O, an initial state Sn sampled from Po- At time t, an agent at S takes an action A, according its policy π S + ((A) with ((A) denoting the set of probability distributions over A. i.e., A, ~ (S,).. The action A, is then executed, after which the agent transitions to a successor state Se+1 ~ p(Sr, Ar) and recieves a reward Re+1 = r(S++1). This agent-environment interaction continues until time T, which marks the end of an episode. The
goal of the agent is to adapt its policy such that the expected total rewards [(T) = EIΣ?=1 Ri] is maximized. In modern deep RL (Mnih et al. 2015 SShhlman et al.) 2017), the policy т is usually parameterized by neural network. We 0 to denote the network parameter and write the policy as тe. Typically, RL algorithms update 0 to adapt its policy. For example, at time I, the action A, sampled from ne, (St). The RL algorithm then update 6, b.+1 based on available information such as So. Ao, Ri...., St, Ar, Re+1, S1+1- Then at time + 1, the action A441 is sampled from the updated policy T0.41 (S++1). Essentially, the typical RI. process is reflected in the updates of 0,.
In-Context Reinforcement Learning. ICRL (Moeini et al. 2025), first coined by Laskin et al. (2023), is an emerging inference time compute paradigm where the RL process occurs in the infer- ence time (i.e., the forward pass) of the network without any parameter update. In ICRL, the policy Ty is additionally conditioned on context called Ct, i.e., A, ~ T Ct). The construction of C. is an active research area but one example is to use all previous state- sate-action-rewan pairs obtained in the task. Notably, this usually includes state-action-rewart pairs from all previous episodes, not just the current episode (Laskin et al. (2023). In ICRL, there is a pretraining stage where the network 0 is pretrained in a wide range of tasks (MDPs). We use 8, to denote the parameter after the pretraining. After the pretraining stage, the policy no, is evaluated in new tasks. In other words, in the new MDP, the action Ae is sampled from (S,, Ct). Importantly, the 0. is kept fixed. Nevertheless, is observed that the quality of A; increases as C. grows in the new task. Since 0. is fixed, this improvement can only from the increase of the context. This is thus called in-context policy im- provement. Notably, this in-context policy improvement is also observed even when the new task is out of the distribution of the pretraining tasks, e.g.. Laskin et al. (2023) demonstrate in-context policy improvement new bandit problems that have the opposite optimal arms to the pretraining bandit problems. Thus this in-context policy improvement cannot be attributed to the hypothesis that 0, memorizes the pretraining tasks. The only plausible hypothesis seems to be that the forward pass of the network parameterized by 0. implements some RL algorithm to process the information in the context C2 to generate the action A₁. This inference-time (forward pass) RL is called in-context RL.
LLMS as RL Agents. The token generation process of LLMs can be modeled via RL. In short, the state is all generated tokens and the action is the next token to generate. Namely, let V be the set of all possible tokens. We consider a state space S = U V and an action space AA = V. At time step an initial prompt given, denoted as So E S. In this work, So contains description of a task. We refer to the LLM with parameter 0 as To. At time t, given the current tokens S,, a new token A is sampled from To(s)). The new state then S [S,At], i.e., the new state obtained by concatenating current tokens and the new token. A reward signal R++1 = r(S++1) is then emitted according to a reward function r. This token generation process continues until a time T, where either T is the maximal allowed response length or Ar is special way, this marks the end of an episode and the final state Sr, called the terminal state, contains both the initial task description and the LLM's response. There are types of reward functions. One is sparse (the outcome reward model, Ouyang et al. (202)). where r(s) is nonzero only when s is a terminal state. The other is dense (the progress reward model, Lightman al. (2023), where r(s) can also be nonzero for non-terminal states.
token. Either
IN-CONTEXT REINFORCEMENT LEARNING PROMPTING
We now present our main contribution, the ICRL prompting framework (Algorithm | Figure consisting of the following ingredients.
3

---

Published as a conference paper at ICLR 2026
Algorithm ICRL Prompting
Require: An LLM A reward function r. Number of episodes K. An experience buffer B. A task description Stask E The ICRL instruction SiCRI. E S.
1: for k k= to K do
2: Construct the initial prompt So by concatenating all the tokens in B, Susks and SICRL.
3: t 0 // Execute the policy we starting from So
4: while Se is not terminal do
5: At TT(()), Se+1 = (St A.]. R = =r(
6: end while
7: // [AA AI...., Ar-1] 1] the LLM's response to Stask at the current episode
8: Push (Au, R2, A1. R2, A2, Rr....Ar-1 Rr)) into B.
Ar-1,
9:
end for
LLM as the Policy, An LLM, denoted as иæ, serves as the policy network. The goal is prompt the LLM to solve a task. We assume a natural language description of the task is available and we denote it $task E S. At the beginning of each episode, construct the initial prompt by concatenating the LLM's own previous attempts together with the corresponding rewards, the task description, and some meta instruction denoted SICRL. The details of the concatenation of previous attempts and the choice of the meta instruction will be discussed shortly. With this initial prompt, the LLM generates the response. Both the response and the rewards are stored in the buffer for future episodes.
Reward Function. A numerical scalar re- ward feedback is provided for each S₁ in the episode. Notably, the reward can be either sparse (i.e., only Rr is nonzero) or dense. The reward function can be rule-based, learned sep- arately, instantiated via the same LLM for self-evaluation. The flexibility of using LLM's self- self-evaluation the reward function allows the ICRL prompting framework to be applied wide range of tasks. Notably, this scalar reward is the only feedback we provide to the LLM. But we do tell the LLM that this scalar is a reward. We do so by explicitly writing down the word "Reward" before this number when constructing the initial prompt. Notably, if the reward function rule-based and learned sepa- rately, the reward signal constitutes external feedback. But if the reward function is just the LLM's own evaluation of the answer, there is no external feedback at all in the ICRL prompt- ing framework. Yet still expect the L.MM's response to improve over the episode. This is because of the widely believed hypothesis that evaluation is eaiser than generation. But we do hypothesize that the ceiling with self-evaluation is lower than that with external feedback.
Context Window
R
A
A
2
R
A R
A
ARAR A
k+1 A AARARA A A
Pusk SICRL
Cancatonate Rewerds
LLM b
LLM b
OR
Env
Reward Function
Figure 1: ICRL Prompting. At each episode k k- 1, LLM generates action tokens based on previous experiences to k, and receives numerical re- wards either from itself as the evaluator or from the environment. At the end of the episode, the rewards are then concatenated with the action to- kens and placed back into the context.
Memory for Experience. We use an experience buffer B store the LLM's responses and rewards for the task in previous episodes. Our underlying hypothesis is that pretrained L.M already has the ICRL ability. To use this innate ICRI. to improve L.LM's response the task, we concatenate its previous attempts and rewards as many the context window allows. We expect that the LLM can reinforcement learn from the experiences in the context during the inference time.
ICRL Instructions. To facilitate LLM's inference time RL, we additionally provide some instruc- tions in initial prompt Sp at each episode. The instruction is in natural language and denoted as SICRL. We consider three types of instructions: (1) the exploration instruction (Figure 4 in App. A),, (2) the exploitation instruction (Figure in App. A), (3) the exploration or exploitation instruction (Figure 6 in App. A). For exploration instruction, the model is asked to provide response that is
4

---

Published as a conference paper at ICLR 2026
different from all its previous responses. For exploitation instruction, the model is asked generate the best response based on the the previous responses with the highest reward. We consider two strategies. (1) ICRL Preset: We alternate between the exploration and exploitation instructions. When the episode number is even, we use the exploration instruction. When the episode number K is odd, ploration or exploitation" instruction at each episode and let the LLM itself to decide on which to use.
the exploitation instruction. (2) ICRL Autonomous: We always provide the "ex-
4 RELATED WORKS
4.1 IN-CONTEXT REINFORCEMENT LEARNING.
The study of inference-time RL algorithms dates back to Duan et al. (2016): Wang et al. (2016), with Laskin et al.] (2023) later coining the term in-context reinforcement learning (ICRL), spurring
rapid growth in the field (Kirsch et al., 2023 Raparthy et al.
2023
Schmied et al. 2024 Lee al.
2024
Zisman et al. 2023 Grigsby et al.) 2024, Lu et al. 2023 Bauer al., 2023] Wang et al,
2025
Cook et al.) 2024; Xu et al./ 2024 Shi et al. [024 Huang et al. 2024 Liu & Abbeel 2023 Dai et al. 2024). See Moeini et al. (2025) for a survey.
Most existing ICRL works, as a subarea small models trained from scratch in games robotics. Some
of meta-RI. (Beck et al.) 2023),
2024 Mirchandani et al. 2023 Resendiz & Klinger (025) in bandit tasks (Krishnamurthy al.
employ pretrained LLMs, e.g., as simulators (Brooks et al.
2024 Nie et al.) 2024 Park et al. 2024]
Monea et al. 2025), where artificial interventions are often needed and LLMs remain uncompetitive with algorithmic baselines.
4.2 INFERENCE-TING LLM SELF-IMPROVEMENT
Existing methods often rely on natural language self-revision, e.g., Self-Refine (Madaan et al. 2023), Reflexion (Shinn et al. 2023), and Textual Gradient (Yuksekgonul et al.) 2025). Since the quality of self-revision is dependent upon the model's parametric knowledge of the task, such ap- proaches are prone to hallucinated feedback that accumulates across iterations, leading to perfor-
mance collapse (Stechly et al. 2025). In essence, they resemble language-guided search (Liu et al. 2025), where feedback serves
explicit instructions for the next revision.
By contrast, ICRL requires only numerical rewards, without prescribing new instructions. The model must infer a better response by recognizing patterns from its past experience, making the process akin to reinforcement learning. Crucially, such rewards can also originate directly from the environment, providing strong source of verification signals.
A parallel line of work improves I.LMs at inference via search, e.g., Tree-of-Thoughee (ToT) (ao al. 2023). Graph-of-Thoughte (GoT) (Besta
2024), Monte Carlo Tree Search (MCTS) (Ding et al. 2024). and Intelligent Go-Explore (Lu et al.) 2025 These methods largely depend on exter- nally engineered components such as heuristics or memory management, rather than leveraging the model's 's intrinsic learning ability. Our work is also related to previous work on prompt optimization (Yang et al. 2024), where numerical scores guide prompt refinement, though through top-k selection and error filtering. Thus, it is more aligned with in-context supervised learning (e.g., filtered behav- ior cloning, Grigsby et al. (2024)) than reinforcement learning. In contrast, ICRL, enables learning from failure experiences.
5 EXPERIMENT
In this section, we evaluate ICRL prompting on three benchmarks: Game of 24, creative writing from Yao et al. 2023). and ScienceWorld (Wang et al. (022). We compare several baselines in- cluding CoT-only. Long-CoT style prompting, Best-of-N, Self-Refine, and Reflexion. Notably. in all the experiments, we allow the prompt of Self-Refine and Reflexion to grow as long as the LLM
allows.
5

---

Published as a conference paper at ICLR 2026
E-. Pour Kaing
5.5
Survess fatte
0.5
Retur
Trial Number
oo Number
Thlal Number
Figure 2: Baseline Method Comparison. (Left) Mean Success Rate on Game of 24. (Middle) Mean Coherence Reward on Creative Writing. Both ICRL Preset and Self-Refine went through an additional run of 50 episodes. (Right) Mean Return on Science World. A running max version of the plots available in Figure 13 in App. B This plot shows quality of the response at the current trial while the running max version shows the quality of the best response until now. The shaded region represents #1 standard error of the performance calculated across the evaluated tasks.
5.1 GAME OF 24
Task Setup. Given four input numbers, the model must use each number exactly once and apply only addition, subtraction, multiplication, or division in any order reach 24. We choose the GPT-4.1 model for this experiment because of its excellent long-context capacity (OpenAl (025), accessed through api calls. Following Yao et al. (2023), we use CoT prompting elicit the model to provide a step-by-step solution, where each step the model picks two numbers from the remaining numbers and an operation to perform the calculation, and obtains one complete solution per LLM query, containing total of 4 thinking steps. To encure that the LLM generates response with correct format, we additionally provide 5 in-context supervised learning demonstrations. The CoT instruction and the 5 demonstrations together form the task description Susk (Figure in App. A
Evaluation. To verify the correctness of the solutions, leverage SymPy (Meurer et al. 2017 a library for symbolic mathematics, by extracting operands and operators and evaluating the reconstructed expression to confirm it equals 24, and report the mean success rate the 100 problems. This rule-based success rate is referred to as т- (i.e., the ground truth reward function) the rest of the paper. We use r to denote the reward function that the algorithms actually have access to. In particular, we use 1PT-4.1 as the the same LLM as the policy LLM but prompted differently (see Figure T2] in App. A). After the policy LLM generates the response, for each thinking step, GPT-4.1 scores the likelihood of reaching 24 with the remaining numbers on 0-3 sure). The task in challenging that no algorithm has access to r.. Instead, they have rely on their own (possibly imperfect) evaluation, generated by the same LLM,, to improve the response.
scale (0- = impossible,
Baselines. We compare our method with CoT-onll, Long-CoT style prompting, Best- of-N, Reflexion, and Self-Refine. In CoT-only prompting, the model receives only the task description Susk and produces single step-by- step solution. In Long-CoT -oT style prompting, we explicitly ask the LLM to generate long chain-of-thought and keep retrying if the so- lution is incorrect in "<think>..<<thinkk>>> tags, before finally providing the answer. Although GPT-4.1 is not specifically trained for long- form CoT reasoning, we find that Long-Col style prompting elicit significantly longer and self-correcting thought traces compared to zero-shot prompting, making it strong baseline for the Game of 24. Both methods cannot make
Table 1: Game of 24 Success Rate. The running max success rate of the last episode is reported.
Method
Success Rate
CoT-only
6%
Long-Col CoT
47%
Reflexion
44%
Best-of-N
49%
Self-Refine
47%
ICRL Preset (Ours)
90 %
ICRL Autonomous (Ours)
84%

---

Published as a conference paper at ICLR 2026
of reward signal and are run for one pass. For Best-of-N, to make it even stronger, ground truth reward r, to select the best response. Self-Refine does not require reward function. It instead asks the LLM itself to provide textual verbal feedback. Reflexion generates reflection according to r. ICRL prompting is different from Self-Refine and Reflexion in that it uses the reward r directly, without any verbal feedback. So the comparision hetween ICRL prompting and Self-Refine/ Reflexion is essentially the comparision between scalar feedback and verbal feedback.
the
ICRL prompting. As discussed before, both Ta and r in Algorithm | are the same LLM GPT-4.1 (prompted differently). We now clarify how we compute So at each episode. Since each action is token, not all actions receive a reward. In fact, since we use CoT to prompt the LLM for 4 thinking steps, only rewards are available for each episode. We thus only include those rewards in So- We add a "Reward: " tag before the actual scalar reward and then concatenate the tagged reward immediately after the corresponding action (i.e., token).
The success rate (i.e., r.) against the number trials (i.e., the episodes Algorithm is reported in Figure 2 The ICRL Preset method achieves the highest performance, and the observed oscillations success rate reflect the model's alternating phases of exploration and exploitation. The mean of rumning max is also plotted in Figure 13 in App. B For each problem, compute its running maximum success rate up to each episode and then average these values across all problems every episode. As summarized in Table after 50 trials, methods achieve success rate of 90% which is significantly higher than 49% from Best-of-N sampling, 47% from Self-Refine, and 44% from Reflexion.
Results.
5.2 CREATIVE WRITING
Task Setup. We consider the creative writing task from Yao et al. 2023 where four sentences are randomly sampled from pool of sentences. The task for LLMs is to generate four paragraphs, each ending with a sentence, while ensuring that the generated passage is coherent. This is difficult task, as it challenges the LLMs to craft a unified storyline that logically justifies each of the four sampled sentences by weaving them into a single narrative. A total of 100 problems are evaluated. An example of Susk Figure App. A
Evaluation. We evaluated outputs using the Length- Controlled Alpaca-Eval 2 (Tatsu Lab) 2025) framework, a widely used proxy for human evaluation (Hong et al.
Table 2: Length-Controlled Win Rate (LC) and Standard Error (SE) from Alpaca-Eval 2.0 on Creative Writing.
2024 Ethayarajh al. 2024; Meng et al. 2024) with up to 0.98 Pearson correlation with human judgments. For present each method's top response: for Reflexion and Best-of-N, the highest- reward output among 50 trials; for ICRL and Self-Refine, the 50th episode output. Alpaca-Eval then computes pair- wise win rates, denoted as r..
100 creative writing problems,
Comparison
LC + SE (%)
Ours vs Reflexion
59.48 + 3.47
Ours vs Long CoT
78.36 + 1.99
Ours vs Self-Refine
86.32 + 3.03
Ours vs Best-of-N
93.81 + 1.01
We next introduce the reward function r accessible to the algorithms. We follow standard pairwise comparison (heng et al. 2023). using GPT-4.1 with coherent reference paragraph to score each response from
-0 (see Fig. ITI. Notably, although both r and r, use an LLM as a judge, they serve distinct pur- poses. compares responses against fixed reference text with emphasis on coherence. By contrast, r. performs pairwise comparison between two responses generated by our method and a baseline method.
Baselines. We compare our method with Best-of-N, Reflexion, and Self-Refine. We allow Best-of- N. Reflexion and ICRL prompting use Self-Refine do not use r and instead asks GPT-4.1 to provide verbal feedback. Since it is hard to distinguish CoT and Long-CoT style prompting for this task, we include Long- Mon style prompting as the baseline.
ICRL prompting. GPT-4. used as both the policy LLM na and the reward model r in Algo- At each episode, the initial prompt So is constructed by concatenating all of the previous
rithm I
7

---

Published as a conference paper at ICLR 2026
generations along with their coherence scores from Notably, this reward is sparse and only Rr can be nonzero. We, therefore, only include Rr in constructing Sp.
Our method achieves a length-controlled win rate of 59.48% against Reflexion, 78.36 % against Long-CoT style prompting, 86.32 % against Self-Refine, and 93.81 % against Best-of-N as shown in Table 2 This shows the responses generated by our method outperform the ones by baselines in terms of following the instruction to write coherent paragraphs and achieving better human preference. The return curve from reward model is plotted in Figure [ and running max of the return is plotted in Figure [3] in App. B Although Self-Refine initially matches ICRL in terms of coherence reward, extending both methods by 50 additional episodes. our methods keep improving, whereas self-refine first plateaus, then declines, likely due to the significant growth of its context.
Results.
5.3 SCIENCEWORLD
Task Setup. ScienceWorld (Wang al. 2022) is an interactive, text-based benchmark consisting of 30 science-experiment tasks set in a multi-room environment populated with diverse objects. The environment is challenging due to sparse rewards, large action spaces, and the requirement for sci- entific knowledge and efficient exploration. At each step, the agent observes the result of its action and receives zero reward unless it completes predefined subgoal. This reward signal is used both for evaluation and for inference-time self-improvement (i.e., r and r. are identical in this task). Completing all subgoals yields a cumulative reward of 100 and marks the episode successful. An episode ends in failure if the agent either reaches the maximum number of steps or executes an incorrect terminating action. The input Stask provided to the agent describes the environment, the task, and the template of all possible actions. An example of Susk is provided in App- Evaluation. We use the cerroomment-rrovidde trajectories used in the context (r), and to evaluate the model (r*). We benchmark each method on all 30 tasks and aggregate the results. GPT-4.1 mini is used as the policy for all compared algo- rithms.
reward function for each task both construct the
Baselines. In Reflexion, at the end of each episode, the agent is prompted to reflect on its attempt. The reflection is then sanitized and appended to reflection buffer, which is formatted into the con- text for subsequent trials. Self-Refine similarly generates self-feedback, but appends it to a trajectory summary, which is then added to the buffer. To ensure fair comparison, we allow these methods access to the reward signals of the current episode (unlike ICRL) before prompting for reflection. ICRL Setup. Each trial corresponds to single episode in the environment. After the trial, the new trajectory added to the buffer is constructed by concatenating the actions, observations, rewards, and the final outcome (success or failure). As each episode typically yields only a few rewards, we include only those. At the start of each trial, Sp is constructed by concatenating the task description Susk, the collected trajectories, and then the instruction SICRL. An example of Sp is provided in A
Results. The mean return at each trial, is presented in Figure 2 Right. Steady improvements are ob- served for methods that make use of some form of history of interactions similar to ICRL prompting. However, ICRL prompting outperforms baseline methods by about 20% after enough iterations. To make the comparison fair for efficient basclines such Best-of-N, in App. B compare baslines as we scale test-time compute budget and observe that ICRL also scales better than the baselines not only in terms of number of trials but also the test-time compute budget (in dollar amounts).
ANALYSIS
Ablation Study. To better understand ICRL prompting, we consider following ablations. (1) Zero Rewards: We set all rewards to O. (2) Short Context: In Algorithm [ the buffer B is essentially a queue of infinite length. Instead, we now make
deque of length 3. In other words, only the recent 3 episodes are used in constructing Sp. (3) Exploration Only: We simply ask the LLM to provide a different response than the ones in context, using the exploration instruction SICRL, without the reward signal. (4) Exploitation Only: We always use the exploita- tion instruction as $ICRL, with the reward signal. (5) No ICRL Instruction: We entirely remove SICRL-
8

---

Published as a conference paper at ICLR 2026
Running Mex Success Paate
Sarrmni Mir Wetturr
mone (ha))
lnbo
uptin ohe Beith
uarce city
0/1/3 mdaee
Dmry3 Traemtato
lestix
Trial Number
Tral Number
Trial Number
Figure 3: Ablation Studies (Running Max). (Left) The mean of running max success rate on Game of 24. (Middle) The mean of running max coherence reward on creative criting. (Right). The mean of running max return on ScienceWorks World. The shaded region represents #1 standard error of the mean (SEM) of the performance calculated across the evaluated tasks within each benchmark.
The running max results of the ablation study plotted in Figure Both our methods and the exploitation only with reward signals demonstrate the best-performing curves. This demonstrates our ICRL prompting framework is quite robust to the different prompts setup. We have also ob- served performance drop with short context and performance drop when the reward is absent. A key finding is that the "exploration only without reward signal" method (shown green) performs significantly worse than our approach when comparing the maximum performance achieved over time (running max). This demonstrates that our method's improvement is not just due to exploring various responses and then picking the best one previously seen doing Best-of-N, Instead, ICRL can genuinely generate novel responses that are better than the ones during the exploration phase.
Context Length Analysis. To assess compute efficiency under different input scales, evaluate ICRL on Qwen3-32B (Yang et al. 2025) across context lengths of 8k, 16k, and 32k (Table 3p. The results show that ICRL consistently surpasses Self-Refine and Reflexion in both Creative Writing and AIME, demonstrating superior performance per unit of compute.
Evaluating ICRL on Open-Source Models and Olympiad-level Math- ematies. demonstrate the broad applicability of ICRL, evaluated its performance across two key dimensions: model architecture and task complexity. First, we tested ICRL, on a range open-source models, including Phi-4 (Abdin et al. 2024), Llama-4 Maverick (025) owen3-32B, and Owen3-32B- thinking mode (Yang et al. (025)) on the creative writing task. Second, we applied it to challenging Olympiad-level mathe- matics (AIME (Mathematical Association of America). HMMT (Harvard-MIT Mathematics Tournament)) )). For details of the experimental setup, please refer to Appendix [..]
Table 3: Performance of Qwen/Qwen3-322B with- out reasoning on Creative Writing (LC-WR: length- controlled win rate) and AIME (% solved) across dif- ferent context lengths.
CW (LC- WR)
AIME (% solved)
Method
8k
16k
32k
8k
16k
32k
Self-Refine
46.33
45.83
44.41
33.33
43.33
43.33
Reflexion
42.73
40.87
40.72
30.00
30.00
33.33
ICRL
50.00
50.00
50.00
40.00
43.33
46.66
ICRL consistently outperforms baselines like self-refine and Reflexion in all settings. Notably, the performance gains are substantial across the board, with improvements of up to 10-20 points over the base model both creative and mathematical benchmarks. These results demonstrate that ICRL is a robust capability that exists across diverse models and proves effective in challenging task domains.

---

Published as a conference paper at ICLR 2026
Table 4: Performance across benchmarks (HMMT, AIME, Creative Writing) for different models and inference-time improvement methods.
Qwen3 32B (32k)
Qwen3 32B think (32k)
Llama 4 Maverick (32k)
Phi-4 (16k)
Method
HMMT
AIME
CW
HMMT
AIME
CW
HMMT
AIME
CW
HMMT
AIME
CW
Base
9.14
22.54
34.14
52.00
66.58
1.24
8,50
17.58
0.98
5.55
20.00
10.85
Self-Refine
16.66
43.33
46.00
56.66
83.33
30.23
13.33
20.00
45.52
13.33
33.33
51.98
Reflexion
23.33
33.33
41.17
60.00
70.00
38.33
10.00
23.33
24.96
13.33
40.00
33.30
ICRL
33.33
46.66
50.00
60.00
80.00
50.00
20.00
35.00
50.00
13.33
40.00
50.00
Test-Time Learning vs. Test-Time Search. To verify that ICRL truly learns from external re- wards rather than merely searching within the model's parametric knowledge, we evaluated it generating abstracts for arXiv papers published after the model's training cutoff. In this setting, where the ground truth is absent from the model's 's training data, standard search methods like Best- of-N and self-correction methods like Reflexion plateau quickly (Figure (7). In contrast, ICRL con- tinues improve ROUGE-Reeall scores over 200 iterations, demonstrating ability to uncover unseen information solely by exploiting the scalar reward signal. Detailed results are provided in Appendix C
Reward-Aware Attention Heads. To understand how the model internally processes reward sig- nals, we conduct mechanistic analysis on the creative writing task using Qwen3-32B. Each re- sponse in the ICRL trajectory received either a low reward (I) or high reward (10). For every attention head across the last 32 layers, we computed the mean attention over all response tokens and measured its relationship with the reward using Pearson correlation. We found that many heads consistently track successful examples, placing significantly higher attention on high-reward re- sponses, while other heads track failures, attending consistent with classical reinforcement learning where models learn not only from successes but also from failures. Of 2,048 layer-head combinations, 597 (29.1%) show statistically significant correlations, far exceeding the 5% expected by chance. Detailed per-layer results are reported in Appendix
to low-reward responses. This pattern is
CONCLUSION
In this paper, we demonstrate that reinforcement learning is an emergent capability of LLMs inference time. We show that our minimal, collar-reward-aarr based ICRL prompting framework unlocks this ability across diverse models and general-purpose tasks, outperforming self-revision methods. A key direction in future work is to investigate how training-tine time interventions might further enhance this in-context RL capability LLMs. This surprisingly effective capability points toward a future of more autonomous agents that can explore, adapt, and self-improve in open-ended settings by learning from their own experience.
ACKNOWLEDGMENTS
This work is supported part by the National Science Foundation under the awards III-2128019, SLES-233 1904, and CAREER-2442098, the Commonwealth Cyber Initiative's Central Virginia Node under the award VV-1Q26-001, a Cisco Faculty Research Award, and an Nvidia academic grant program award.
This material is based upon work supported part by the National Science Foundation under Grant No. 2124538. Any opinions, findings, and conclusions recommendations expressed in this ma- terial are those of the author(s) and do not necessarily reflect the views the National Science Foundation.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck. Ronen Eldan, Suriya Gunasckar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee. Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Xin Wang, Rachel Ward, Yue Wu, Dingli Yu, Cyril Zhang, and Yi Zhang. Phi-4 technical report, 2024. 08905
Meta AI. The Ilama 4 herd: The beginning of a new era of natively multimodal ai in- novation. attpss:/at meta. www.cowwo/////c..ooooooooo /oogg//lam alama 4 4-mu mwl.llmmooa -oneedallinnellieennee 2025. Accessed: 2025-09-25.
Jakob Bauer, Kate Baumli, Feryal Behbahani, Avishkar Bhoopchand, Nathalie Bradley-Schmieg, Michael Chang. Natalie Clay, Adrian Collister, Vibhavari Dasagi, Lucy Gonzalez, et al. Human- timescale adaptation in open-ended task space. In Proceedings of the International Conference on Machine Learning, 2023.
Jacob Beck, Risto Vuorio, Evan Zheran Liu, Zheng Xiong, Luisa Zintgraf, Chelsea Finn, and Shi- mon Whiteson. A survey of meta-reinforcement learning. ArXiv Preprint, 2023.
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Giani- nazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, and Torsten Hoe- fler. Graph of thoughts: Solving elaborate problems with large language models. Proceedings of the AAAI Conference on Artificial Intelligence, 2024.
Ethan Brooks, Logan Walls, Richard Lewis, and Satinder Singh. Large language models can implement policy iteration. In Advances in Neural Information Processing Systems, 2024.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, 2020.
Murray Campbell, Feng-Hsiung Hsu, and A. Joseph Hoane. Deep blue. Artificial Intelligence, 134 (1):57-83, 2002. doi: (000////0004-7000000000000
Jonathan Cook, Chris Lu, Edward Hughes, Joel Z Leibo, and Jakob Foerster. Artificial generational intelligence: Cultural accumulation in reinforcement learning. ArXiv Preprint, 2024.
Zhenwen Dai, Federico Tomasi, and Sina Ghiassian. In-context exploration-exploitaaion for rein-
forcement learning. ArXiv Preprint, 2024.
DeepSeck-AL, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao. Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen. Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei. Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao. Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong
11

---

Published as a conference paper at ICLR 2026
Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xi- aosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang. Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo. Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang. Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in llms via reinforce- ment learning. ArXiv Preprint, 2025.
Ruomeng Ding, Chaoyun Zhang, Lu Wang, Yong Xu, Minghua Ma, Wei Zhang. Si Qin, Saravan Ra- jmohan, Qingwei Lin, and Dongmei Zhang. Everything of thoughts: Defying the law of penrose triangle for thought generation. In Findings of the Association for Computational Linguistics: ACL 2024, 2024.
Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and Pieter Abbeel. RI2: fast reinforcement learning via slow reinforcement learning, ArXiv Preprint, 2016.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. ArXiv Preprint, 2024.
Jake Grigsby, Linxi Fan, and Yuke Zhu. Amago: Scalable in-context reinforcement learning for adaptive agents. In Proceedings of the International Conference Learning Representations, 2024.
Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in lims via reinforcement learning. ArXiv Preprint, 2025.
HIrvard-MIT Mathematics Tournament. Hmmt (harvard-mit mathematics tournament). URL
https:// http///www... hrmnt org/
Michael Heim. Exploring Indiana Highways. 2007.
Jiwoo Hong, Noah Lee, and James Thomn. ORPO: Monolithic preference optimization without ref-
erence model. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024.
Sili Huang, Jifeng Hu, Hechang Chen, Lichao Sun, and Bo Yang. In-context decision transformer: Reinforcement learning via hierarchical chain-of- tfti--gghthhht.... ArXiv Preprint, 2024.
Khurram Javed and Richard S. Sutton. The big world hypothesis and its ramifications for artifi- cial intelligence. In Finding the Frame Workshop the Reinforcement Learning Conference. OpenReview, 2024. URL https////pppeeeeeeeeeeeeee/e/ees 4.0.
CC BY
Louis Kirsch, James Harrison, C. Freeman, Jascha Sohl-Dickstein, and Jürgen Schmidhuber. To- wards general-purpose in-context learning agents. In NeurIPS Foundation Models for Decision Making Workshop, 2023.
Akshay Krishnamurthy, Keegan Harris, Dylan Foster, Cyril Zhang, and Aleksandrs Slivkins. Can large language models explore in-context? ArXiv Preprint, 2024.
Michael Laskin, Luyu Wang, Junhyuk Oh, Emilio Parisotto, Stephen Spencer, Richie Steigerwald, DJ Strouse, Steven Stenberg Hansen, Angelos Filos, Ethan Brooks, maxime gazeau, Himanshu Sahni, Satinder Singh, and Volodymyr Mnih. In-context reinforcement learning with algorithm distillation. In Proceedings of the International Conference Learning Representations, 2023.
12

---

Published as a conference paper at ICLR 2026
Jonathan Lee, Annie Xie, Aldo Pacchiano, Yash Chandak, Chelsea Finn, Ofir Nachum, and Emma Brunskill. Supervised pretraining can learn in-context reinforcement learning. In Advances in Neural Information Processing Systems, 2024.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. ArXiv Preprint, 2023.
Hao Liu and Pieter Abbeel. Emergent agentic transformer from chain of hindsight experience. In Proceedings of the International Conference on Machine Learning, 2023.
Max Liu. Chan-Hung Yu, Wei-Hsu Lee, Cheng- Wei Hung, Yen-Chur Chen, and Shao-Hua Sun. Synthesizing programmatic reinforcement learning policies with large language model guided search. In Proceedings of the International Conference on Learning Representations, 2025.
Chris Lu. Yannick Schroecker, Albert Gu, Emilio Parisotto, Jakob Foerster, Satinder Singh, and Fer- yal Behbahani. Structured state space models for in-context reinforcement learning. In Advances in Neural Information Processing Systems, 2023.
Cong Lu, Shengran Hu, and Jeff Clune. Intelligent go-explore: Standing the shoulders of giant foundation models. In Proceedings of the International Conference on Learning Representations, 2025.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback. In Advances in Neural Information Processing Systems, 2023.
Mathematical Association of America. American invitational mathematics examination (aime). URL
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-free reward. ArXiv Preprint, 2024.
Aaron Meurer, Christopher P. Smith, Mateusz Paprocki, Ondřej Certik, Sergey B. Kirpichev, Matthew Rocklin, AMiT Kumar, Sergiu Ivanov, Jason K. Moore, Sartaj Singh, Thilina Rath- nayake, Sean Vig, Brian E. Granger, Richard P. Muller, Francesco Bonazzi, Harsh Gupta, Shivam Vats, Fredrik Johansson, Fabian Pedregosa, Matthew J. Curry, Andy R. Terrel, Simon Roucka, Akshay Saboo, Ishaan Fernando, Shravan Kulal, Robert Cimrman, and Anthony Scopatz. Sympy: symbolic computing in python. PeerJ Computer Science, 2017.
Suvir Mirchandani, Fei Xia, Pete Florence, Brian lchter, Danny Driess, Montserrat Gonzalez Are- nas, Kanishka Rao, Dorsa Sadigh, and Andy Zeng. Large Language Models as General Pattern Machines. ArXiv Preprint, 2023.
Volodymyr Mnih Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wier- stra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 2015.
Amir Moeini, Jiuqi Wang, Jacob Beck. Ethan Blaser, Shimon Whiteson, Rohan Chandra, and Shang- tong Zhang. A survey of in-context reinforcement learning. ArXiv Preprint, 2025.
Giovanni Monea, Antoine Bosselut, Kianté Brantley, and Yoav Artzi. Lims are in-context bandit reinforcement learners. ArXiv Preprint, 2025.
Allen Nie, Yi Su, Bo Chang, Jonathan N. Lee, Ed H. Chi, Quoc V. Le, and Minmin Chen. EVOLVE: Evaluating and Optimizing LLMs For Exploration. ArXiv Preprint, 2024.
OpenAI. Introducing gpt-4.1 in the api. htttps://openai.com///neee
2025.
13

---

Published as a conference paper at ICLR 2026
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, 2022.
Chanwoo Park, Xiangyu Liu, Asuman Ozdaglar, and Kaiqing Zhang. Do Ilm agents have regret? case study in online learning and games. ArXiv Preprint, 2024.
Sharath Chandra Raparthy, Eric Hambro, Robert Kirk, Mikael Henaff, and Roberta Raileanu. Gen- eralization new sequential decision making tasks with in-context learning. ArXiv Preprint, 2023.
Yarik Menchaca Resendiz and Roman Klinger. Parl: Prompt-based agents for reinforcement learn- ing, 2025. URL /attps://arxiv.ort abs/2510..01... 21306
Thomas Schmied, Thomas Adler, Vihang Patil, Maximilian Beck, Korbinian Pöppel, Johannes Brandstetter, Günter Klambauer, Razvan Pascanu, and Sepp Hochreiter. Large Recurrent Ac- tion Model: xLSTM enables Fast Inference for Robotics Tasks. ArXiv Preprint, 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ArXiv Preprint, 2017.
Lucy Xiaoyang Shi, Yunfan Jiang. Jake Grigsby, Linxi Fan, and Yuke Zhu. Cross-episodic curricu- lum for transformer agents. In Advances in Neural Information Processing Systems, 2024.
Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems, 2023.
David Silver and Richard S. Sutton. Welcome to the of experience, 2024.
David Silver and Richard S Sutton. Welcome to the era of experience. Google AI, 2025.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George van den Driess- che, Julian Schrittwieser, loannis Antonoglou, Vedavyas Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 2016.
David Silver, Julian Schrittwieser, Karen Simonyan, loannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of go without human knowledge. Nature, 550(7676):354-55, 2017. doi: 10.1038/ nature24270.
David Silver, Satinder Singh, Doina Precup, and Richard S. Sutton. Reward is enough. Artificial Intelligence, 2021.
Kaya Stechly, Karthik Valmeekam, and Subbarao Kambhampati. On the self-verification limitations of large language models reasoning and planning tasks. In Proceedings of the International Conference on Learning Representations, 2025.
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Rad- ford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. ArXiv Preprint, 2022.
Yi Su, Dian Yu, Linfeng Song, Juntao Li, Haitao Mi, Zhaopeng Tu, Min Zhang, and Dong Yu. Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains, April 2025. http: org/ abs./503.... 23829
Richard S. Sutton. The reward
ualberta. ca /RLAA/AA//AA
2004.
Richard S. Sutton. The bitter lesson.
Bitterfesson html 2019.
14

---

Published as a conference paper at ICLR 2026
Tatsu Lab. alpaca eval: An evaluation framework for llms. Latsu-lab/alle alpaca eval 2025.
bttps://githie com/
Jane Wang, Zeb Kurth-Nelspe Dhruva Tirumala, Hubert Soyer, Joel Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. ArXiv Preprint, 2016.
Jiuqi Wang, Ethan Blaser, Hadi Daneshmand, and Shangtong Zhang. Transformers can learn tempo- ral difference methods for in-context reinforcement learning. Proceedings of the International Conference Learning Representations, 2025.
Ruoyao Wang, Peter Jansen, Marc-Alexandre Côté, and Prithviraj Ammanabrolu. Science World: Is your Agent Smarter than a 5th Grader? ArXiv Preprint, 2022.
Tengye Xu, Zihao Li, and Qinyuan Ren. Meta-reinforcement learning robust distributional shift via performing lifelong in-context learning. In Proceedings of the International Conference on Machine Learning, 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang. Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL httph:tt///mm /ttpss//aaaivv.v. wwwpxxiww.rrgab//555555 09388
Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoe V. Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. ArXiv Preprint, 2024.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. ArXiv Preprint, 2023.
Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Pan Lu, Zhi Huang, Carlos Guestrin, and James Zou. Optimizing generative by backpropagating language model feedback. Nature, 2025.
Qiyuan Zhang, Fuyuan Lyu, Zexu Sun, Lei Wang, Weixu Zhang, Zhihan Guo, Yufei Wang, Niklas Muennighoff, Irwin King, Xue Liu, and Chen Ma. What, how, where, and how well? survey on test-time scaling in large language models, 2025. URL httttpps//aaxii..og//bbb////////
24235
Shimao Zhang, Yu Bao, and Shujian Huang. Edt: Improving large language models' generation by entropy-based based dynamic temperature sampling. ArXiv Preprint, 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu. Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. ArXiv Preprint. 2023.
Ilya Zisman, Vladislav Kurenkov, Alexander Nikulin, Viacheslav Sinii, and Sergey Kolesnikov. Emergence of in-context reinforcement learning from noise distillation. ArXiv Preprint, 2023.
15

---

Published as a conference paper at ICLR 2026
Instruction: Examine all the cnreemptt..</atemeen examples, each showing candidate Response, and the Rewards for each step of the Response. Provide response that is completely different for any steps from every single one of the previous attempts demonstrated in the context.
Figure 4: The Exploration Instruction (SICRL.).
Instruction: You will he given multiple battempt...</attempp examples, each showing candidate Response, and the Rewards for each step of the Response. Your task: Based the previous attempts, uy your best to produce response that can achieve higher rewards.
Figure 5: The Exploitation Instruction (SICKL).
A PROMPT EXAMPLES
Prompt: Write a coherent passage of short paragraphs. The end sentence of cach paragraph must be: For some unfathomable reason, the response team didn't 't consider a lack of milk for my cereal as proper emergency. You realize you'l not alonc you sit in your bedroom massaging your calves after long day of playing lug-of-war with Grandpa Joe in the hospital. He poured rocks in the dungeon of his mind. I'm living furnace. Make plan then write, Your output should of following format: Plan: Your plan here. Passage: Your passage here.
Figure 8: An example of susk for creative writing.
Instruction: Examine all the <attempt>..<<attemppp examples, each showing candidate Response and its Reward. You have two options: exploration exploitation.
exploration, provide a response is completely different any steps from every single one of the previous attempts demonstrated in the context, while making sure correctly follows the task instruction.
For exploitation, based the previous attempts, try your best produce a response that achieve higher rewards.
Pick one option to follow.
Figure 6: The Exploration Exploitation Instruction (SICRL..
16

---

Published as a conference paper at ICLR 2026
<attempt>
Input: 6
Stepl: 8 (left: 46 12)
Step2: 4-4 2 (left: 12)
Step3: * 12 24 (left: 24)
Answer: (6-4) 4) (4 8)
</attempt>
<attempt> Input: 29 10 12
Stepl: 12 * 2 24 (left: 9 24)
Step2: 10 9 (left: 1 24)
Step3: 24 # = 24 (left: 24)
Answer: (12 2) (10 90 9) 24
</attempt>
<attempt> Input: 49 10 13
Stepl: 13 10= 3 (left: 94
Step2: 9 6 (left: 4 6)
Step3: * 6 24 (left: 24)
Answer: 4 * (9 (13 10)) = 24
</attempt>
<attempt>
Input: 1488
Stepl: 8 8/4=2 (left: 1 8)
Step2: 4 2 = (left: 3 8)
Step3: # 8 24 (left: 24)
Answer: + 4/ =424
</attempt>
<attempt>
Input: 55559
Stepl: 5 (left: 59 10)
Step2: 50+5= 15 (left: 9 15)
Step3: 15 + 9 24 (left: 24)
Answer: ((5 + 5)
24
</attempt>
**aak** Use numbers and basic arithmetic operations (+ /) to obtain 24. Put your an- swer in this format < ...) as #nswer** <math operations of the 4 input numbers, if does not equal 24></answer>>
Stepl: (left: ...) Step2: (left: ...) Step3: (left:
Whether it correct or not, not try again.
"*Prompt**; Input: 10 11
Figure 7: An example of sunk for Game of 24 with few-shot CoT prompting.
17

---

Published as a conference paper at ICLR 2026
You helpful assistant to do some scientific experiment in environment. <Environment description>> 8usk
the environment, there are several rooms: kitchen, foundry, workshop, bathroom, outside, living room, bedroom, greenhouse, art studio, hallway
You should explore the environment and find the items you need to complete the experiment.
available actions are: Available Actions
FOCUS is extremely critical action that can he only used the number of times 'focus' is mentioned in the task description. Using it more than that inappropiately (such wrong object) will terminate the session and the task WILL FAIL.
Task Description:
Your task change the state of matter of water. First, focus the substance. Then, take actions that will cause change state of matter. </Favironment description>
<Instruction>> StCRL </Instruction>
<Attempts> Buffer
Attempt YE:
teleport bathroom -> Observation: You teleport to the bathroom. (reward=3)
>> focus on substance toilet Observation: You focus on the water. (reward=66)
>> use on substance in toilct -> Observation: I'm not sure how to do that. (reward=0)
>> activate sink -> Observation: The sink is now activated. (reward=0)
dunk cup into sink Observation: No known action matches that input. (reward=0)
move cup to sink -> Observation: You move the glass cup to sink. (reward=0)
>> teleport to kitchen > Observation: You teleport to the kitchen, (reward=0)
>>
use cup stove -> Observation: I'm not sure how to do that. (reward=0) activate
Observation: The stove is activated. (reward=2)
move cup to table Ohservation: You
the glass cup the chair. (reward=0)
-> examine cup Observation: a glass cup (containing nothing) (reward=()
Task Failed. You have exceeded the maximum number of steps. (reward=0) Total reward: 71
Attempt
teleport to hathroom -> Ohservation: You teleport to the bathroom, (reward=3)
>> focus on substance in toilet -> Observation: You focus on the water. (reward=66)
>> pick up cup Observation: You move the glass
the inventory. (reward=0)
move cup to sink Observation: You move the glass cup the sink. (reward=0) activate sink Observation: The sink now activated. (reward=0)
>>
dunk cup into sink Observation: No known action matches that input. (reward=0)
>> pour substance in cup into sink -> Observation: You pour the water into the sink. (reward=0) activate stove -> Your generated action "activate stove" cannot be matched to valid action. (re- ward=0)
-> teleport kitchen Observation: You teleport to kitchen. (reward=0)
-> pick up cup containing nothing in table ~> Observation: You move the glass cup to the inventory.
(reward=0)
activate sink Observation: The sink is now activated. (reward=0)
Task Failed. You have exceeded the maximum number of steps. (reward=0) Total reward:
</Attempis>
Figure 9: An example of So from Science
18

---

Published as a conference paper at ICLR 2026
Buffer:
<attempt>
Input: 4.9 10 13.
Response:
Stepl: 10 -0 = (left: 69 13)
<Reward: 3.00>
Step2: 13 6 (left: 79)
<Reward: 0.00>
Step3: * 63 (left: 63)
<Reward: 0.00>
"Answer**: (13 (10 4)) = 63
<Reward: 3.00>
</attempt>
<attempt>
Input: 9 10 13.
Response:
Stepl: 10 = 14 (left: 9 13 14)
<Reward: 0,00>
Step2: 9 (left: 23)
<Reward: 0.00>
Step3: 13 36 (left: 36)
<Reward: 0.00>
*Answert*: (10 + + 13 36
<Reward: 0.00>
</attempt>
<attempt>
Input: 9 10 13.
Response:
Stepl: 9 10 19 (left: 4 13 19)
<Reward: 3.00>
Step2: 19 6 (left: 4 6)
<Reward: 3.00s
Step3: 4 (left: 10)
<Reward: 0.00>
*Answer**: ((9 + 10) 13) +
10
<Reward: 6.00>
</attempt>
SICRL.
Stack
Figure 10: An example of So from Game of 24.
Instruction: You are a seasoned text coherence evaluator. Read the TEXT below and rate its overall coherence scale from I to 10, where 1 means significantly less coherent than the Base Answer, 5 means equally coherent, and 10 means significantly evaluator-only assign high
coherent. Be strict and conservative when the TEXT is clearly better than the Base Answer.
Base Answer:
lAt dawn, golden light slips past pale curtains, rousing the world in quiet celebration. A lone robin greets the morning with a clear, cheerful trill, its song drifting across dew-laden grass. A gentle breeze stirs the leaves, carrying the fresh, earthy scent of new growth. Nearby, rooftops and empty streets 1ic poised between night's calm and the city's stirring pulse, promising simple conforts like a warm cup of coffee. In this tranquil pause, one senses life's renewal and the gentle invitation to greet the day with hope and gratitude.)
TEXT: modellanswer)
Return your answer in exactly this format: Coherency score: <integer 1-10>. Response:
Figure 11: Prompt for Pairwise Coherence Evaluation for Reward Model
19

---

Published as a conference paper at ICLR 2026
Rule of the Game of 24: Use all four numbers provided in the input, without repetition, and only basic arithmetic operations (+, - x, +) to obtain 24, Only three steps allowed.
Given the following remaining numbers from previous step in the Game of 24, the current step is: (step). Evaluate this step.
Examine numbers shown in "left: ... after step and reason whether is still possible reach 24: Sure - 3 Likely - Impossible -
Return the score the following format: **Answer**: <integer score>
Response:
Figure 12: Prompt for single-step evaluation used in the reward model r for Game of 24.
B ADDITIONAL EXPERIMENTAL RESULTS
ES. Novt Pard Arhoroe
Bureing Max SucCEaA Pate
eng Man Reeetur
Rumninn
(Dant puticts
Nete
46 6
102
Trial Number
Triil Number berb
Thia Number
Figure 13: Benchmark results: Mean of Running Max. (Left) The mean of running max success rate Game of 24. (Middle) The mean of running max coherence reward on creative writing. Both ICRL Preset and Self-Refine went through an additional run of 50 episodes. (Right). The mean of running max return on Worln.
und
2.0
Orig Pagedaaiss
Stcccan Rate
Crep No 1TUl Bntteeit
4.5
Thial Number
Tria Number
Trial Number
Figure 14: Ablation study results: Original Curves. (Left) The mean of success rate Game of
24 ablation studies. (Middle) The mean coherence reward on creative writing ablation studies. (Right). The mean return Science World ablation studies.
20

---

Published as a conference paper at ICLR 2026
Table 5: Running max of return averaged over all the tasks in Science
Method
Return (max 100)
ReAct
69 1.4
Reflexion
74 = 1.1
Best-of-N
75 1.2
Self-Refine
83 0.9
ICRL Preset (Ours)
88 0.7
ICRL Autonomous (Ours)
87 0.8
Running Max Success Rate
0.7
Running Max Return
70
0.5
50-
Autoromous (Gun)
ICRU Autonomoux (Ouns)
Best ofN
Best of
40
Reflecon
0.2
Refleston
Soff-Refine
Refine
Rehd
DwAct
20
Cumulative Cost (USD)
Trial Number
Figure 15: Additional WcilnceWorld Results. (Lefft Although ICRL's context (comprising the experience buffer) is longer than that of random sampling methods, it still outperforms them and other experience- approaches given additional compute budget. (Right) ICRL's 's superior return improvement in other results, also leads to greater increase in success rate.
B.1 ADDITIONAL BENCHMARK: MATH COMPETITIONS
Task Setup. AIME (Mathematical Association of Americal and HMMT (Harvard-MIT Mathemat- ics Tournament) are of the most difficult mathematics competitions in the U.S. for high school students. They require not only strong mathematical knowledge but also advanced problem-solving skills. Training language models for reasoning has been one of the most effective methods for im- proving performance on these benchmarks (DeepSeek-Ai et al. 2025).
We aim test whether reward signals derived from previous reasoning traces can improve subse- quent reasoning attempts. To access these traces, we rely on open-source models. In addition to evaluating reasoning models on these benchmarks, we also include broader set of open-source models to demonstrate the prevalence of ICRL capability across wide spectrum.
asked to produce an answer. The only additional instruction concerns formatting: the model must place its final answer within <answer> </ </answer> tags. The dataset includes the ground-tre truth answer for each question.
In this benchmark, the model given a single question
For the reward model, we use the model proposed by Su al. (2025). which provides denser signal than simply checking with SymPy whether the model's 's output matches the ground truth. Using such model aligns with belief that the growing ecosystem of judge and reward models, primarily designed for reinforcement fine-tuning language models (Lightman et al. (023), will also play a key role in enabling ICRL test-time scaling method.
Evaluation. To verify correctness, we parse the model's output and check whether its answer matches the ground truth answer provided in the datasets.
21

---

Published as a conference paper at ICLR 2026
Baselines. We compare method with Reflexion and Self-Refine, the two strongest baselines from the other benchmarks. Their implementations are consistent with those used in the prior ex- periments.
ICRL Setup. For attempts after the first, we insert the model's previous answer (including reasoning or CoTT' tokens) into the context, along with its associated reward. We then prompt the model either to try a new method or to refine its best previous approaches. We truncate past answers to ensure that at least 32 prior attempts can fit in the context.
Results. Our method outperforms the baselines in almost all cases. For the reasoning mode of Qwen3-32B, approach remains competitive with Self-Refine AIME and surpasses it HMMT, which is the harder benchmark. Notably, this is achieved using scalar rewards computed with three orders of magnitude less computation, while Reflexion and Self-Refine are allowed nearly twice the compute budget for generating long-CoT verbal feedback.
22

---

Published as a conference paper at ICLR 2026
C ADDITIONAL ANALYSIS RESULTS
C.I UNSEEN PAPER ABSTRACT GENERATION
To isolate whether ICRL truly learns from external rewards rather than merely searching or selecting from the model's s parametric knowledge, we design task where search within parametric knowledge alone is expected be ineffective.
Setup. We fetch 30 arXiv papers published after GPT-4.1-ming training cutoff. Provided only the title of the papers, the model instructed to generate the abstract. The goal to uncover the unseen expert-written abstract much as possible. We measure performance using ROUGF-recall between the model's generated abstract and the ground-truth abstract, and we use the same score as the external reward. Because these abstracts lie outside the model's pre-training distribution, success requires learning from external rewards rather than searching through memorized content.
Best-of-1024 sampling reaches only 0.44 ROUGE-recall, indicating that direct search over the model's base distribution cannot recover the missing content. Self- Refine plateaus within a few rounds 0.45 because it does not external reward. Reflexion performs slightly better at 0.46 but largely mirrors Self-Refine, suggesting that its revisions are dominated by the model's self-verbal feedback, which carries little useful information in this setup. In contrast, ICRL continues to improve over 200 iterations and achieves substantially higher ROUGE-recall at 0.59, demonstrating that it can effectively learn from the external reward signal and not limited by the model's pre-training knowledge.
Results. As shown in Fig.
C.2 REWARD-SENSITIVG ATTENTION HEADS
We conduct a mechanistic analysis to investigate how Qwen3-32B internally processes reward sig- nals during ICRL the creative writing task.
Setup. For each of 100 questions, we select 5 trials from the ICRL trajectory and construct two prompts that differ only in the reward labels: test prompt where each trial's reward is independently randomized to 1 or with equal probability, and a baseline prompt where all rewards are set to 1. For every attention head across the last 32 layers (2,048 heads total), we compute the mean attention over all response tokens in each trial, subtract the baseline to isolate the reward-dependent component of attention, and measure its relationship with the reward label using Pearson correlation (500 observations per head: 100 questions x 5 trials).
Results. Figure 16 shows the number of significant heads per layer at 0.05. Of 2,048 heads tested, 597 (29.15 %)) are significant, far exceeding the 102 (5%) expected by chance (dashed line). Many heads consistently track successful examples, placing significantly higher attention on high- reward responses (249 positively correlated heads). Other heads track failures, attending more to low-reward responses (348 negatively correlated heads). This pattern consistent with classical reinforcement learning where models learn not only from successes but also from failures.
Across layers, reward sensitivity increases toward earlier layers: layers -27 to -32 average 29.7 significant heads per layer, while layers closest to the output (-1 to -6) average 12.2. The balance between the two types of heads also varies by depth. In layers closest to the output, most significant heads attend to low-reward responses. In middle layers (-14 to -21), more heads attend to high- reward responses. The earliest analyzed layers contain a high density of both. These results provide mechanistic evidence that the model actively processes the scalar reward signal across broad set of attention heads.
23

---

Published as a conference paper at ICLR 2026
Reward Raward-cansttive sonsitive attention heads acioss
Qwen3- loyers
Weere reventtrail LitaNEta lighomaard
Numberd sun Feads DAR
14 15
Layar Datativv
laal Aeyert
Figure 16: Number of attention heads with statistically significant Pearson correlation (p < 0.05) between bascline-adjusted attention and reward label, per layer Qwen3-32B. Blue: positive cor- relation (attend more to high-reward trials). Red: negative correlation (attend more low-reward trials). Dashed line: expected count under chance (3.2 heads per layer).
0.60
Best-of-N
self-refine
0.55
reflexion
ICRL
0.50
louge-recall
0.45
0.40
0.35
0.30
0.25
200
400
600
800
1000
Round
Figure 17: ROUGE-Recall Generating Unseen Paper Abstracts
24


Reviews:
[{"review_id": "pEFQInnPwV", "reviewer": "Reviewer_J3p5", "summary": "This paper introduces a multi-round prompting framework, ICRL Prompting, to argue that LLMs exhibit an emergent capability for in-context reinforcement learning (ICRL). The method appends a history of the LLM's prior (response, scalar reward) pairs to the context. The authors demonstrate that this minimal mechanism, which relies solely on scalar feedback, achieves performance improvements on several benchmarks and outperforms baseline methods that rely on textual feedback.", "questions": "1. The ablation study tests zero rewards but not noisy rewards. Could the authors provide a sensitivity analysis showing how ICRL's performance degrades if 20% or 50% of the scalar rewards in the context are intentionally replaced with incorrect values?\n\n2. The claim of emergent RL capability is currently based on external behavior. To provide mechanistic evidence, could the authors provide a preliminary interpretability analysis (e.g., attention maps) to show whether the model systematically attends more to tokens associated with high-reward history when generating a new response?\n\n3. The performance gap between ICRL (90% on Game of 24) and the Best-of-N baseline (49%), which used the true reward r*, is striking. What is the authors' intuition for this gap? Does it imply that correct solutions are so sparse in the model's base distribution that simple sampling even with a perfect r* selector fails to find them, whereas ICRL is actively learning to generate them?\n\n4. Could the authors please clarify the context truncation strategy mentioned in Appendix B.1 (used to fit \"at least 32 prior attempts\")? Is this a simple FIFO queue, or a more complex strategy based on reward (e.g., prioritizing high-reward experiences)?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The primary strength of this paper is its novelty. It reveals an interesting phenomenon: simple scalar reward signals within the context may be sufficient to drive self-improvement in LLMs. This claim is supported by impressive empirical results, where the method consistently outperforms strong baselines like Self-Refine and Reflexion on several benchmarks. The experimental design includes rigorous and fair baseline comparisons, notably by allowing baselines like Self-Refine to use an equally growing context and by comparing ICRL against an overpowered Best-of-N baseline that uses the ground-truth reward for selection. The inclusion of a computational cost analysis in the appendix further strengthens the paper's claims of practical utility.", "weaknesses": "The paper's most significant weakness is a methodological flaw in its core experiments: a self-referential bias. In several key benchmarks, the reward signal r is generated by the LLM itself. This introduces a high risk of reward hacking, where the policy model may simply be learning to overfit the biases of its own evaluation model rather than optimizing the true task metric r*. The paper fails to provide sufficient evidence to rule out this possibility.\n\nThis methodological issue is compounded by the paper's reliance on phenomenological observation (a \"duck test\") for its central claim that LLMs are RL learners. The paper offers no mechanistic analysis explaining how the Transformer architecture actually utilizes the (response, reward) history to achieve policy improvement.\n\nIn addition to this lack of mechanistic depth, the paper does not include a systematic robustness analysis of reward signal quality. The ablation study tests zero rewards, but this is not equivalent to testing the method's tolerance for noisy or incorrect rewards, which is critical for real-world application. This limited scope of evidence leads to the final weakness: the paper's conclusion is overclaimed. All empirical evidence is derived from text-based environments, making the extension of its claims to general-purpose tasks unsupported.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yVS2LTMFBs", "reviewer": "Reviewer_nrTz", "summary": "The paper proposes an in-context RL framework that provides a task description, and then judges the answers of the model to provide a reward that is used by the LLM to improve its answers. The authors evaluate the method on different text-based RL problems, where the method outperforms other self-revision techniques like Self-Refine and Reflexion.", "questions": "1. Figure 2/3 middle + right: The intuition of the returns is not clear, which makes it hard to assess the performance level and the significance of the performance differences between the different configurations. Is the agent actually able to solve the task? What is the difference between two agents that have X difference in return? Are both solving the tasks more or less, or is there a perceptible difference in the quality of the solutions? Maybe there is a more intuitive metric for these tasks than the return that would make it easier to assess the performance of the agents.\n\n2. Figures 2 and 3: What exactly do the shaded areas represent? \n\n3. Line 370 mentions a \"length-controlled win rate\": How is that defined?\n\n4. Figure 3: It is surprising that the \"zero rewards\" configuration actually improves and even reaches somewhat decent performance. Perhaps the authors could elaborate on how the agent is able to improve without any kind of feedback.\n\n5. Figure 3 middle: Some of the curves seem incomplete. ICRL Autonomous is still improving, so it is unclear whether it would eventually reach the performance of ICRL Preset and Self-Refine. Is there a reason why some curves only go to trial 50 here?\n\nTypos:\n\nLine 124: \"can only from\": word missing\n\nLine 139: \"funciton\" --> \"function\"\n\nLine 298: \"The task in challenging\"\n\nLine 444: \"criting\" --> \"writing\"", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "The paper is well-written and easy to follow. The method seems sound and is able to achieve good performance. There is good diversity in the tasks that the method is evaluated on, ranging from math puzzles to creative writing.", "weaknesses": "I am unsure about the novelty of the method. It seems that a number of works on ICRL already exist in the literature. The authors state that existing works are \"restricted to bandit or simulated environments [...], falling short of addressing complex open-ended tasks in the real world\", which seems a bit vague. There are other papers that do in-context reinforcement learning that also tackle complex sequential decision-making tasks, e.g., [1-4]. One of the claimed contributions is the \"ICRL prompting framework\", but it is unclear how this framework differs from existing frameworks used in ICRL. I would greatly appreciate it if the authors could elaborate on the exact novelty in their method and setting.\n\n[1] Michael Laskin et al. \"In-context Reinforcement Learning with Algorithm Distillation.\" The Eleventh International Conference on Learning Representations.\n[2] Yarik Menchaca Resendiz, and Roman Klinger. \"PARL: Prompt-based Agents for Reinforcement Learning.\" arXiv preprint arXiv:2510.21306 (2025).\n[3] Ethan Brooks et al. \"Large Language Models can Implement Policy Iteration.\" Advances in Neural Information Processing Systems 36 (2023).\n[4] Sili Huang et al. \"In-context decision transformer: Reinforcement learning via hierarchical chain-of-thought.\" arXiv preprint arXiv:2405.20692 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oUczTOB6tS", "reviewer": "Reviewer_iBtQ", "summary": "The authors propose a minimal in-context RL prompting framework where an LLM iteratively generates responses to a task, receives scalar numerical rewards, and conditions future outputs on previous responses and rewards. Over multiple rounds prompting, the model’s behavior exhibits classical RL traits, e.g., reward maximization and context-dependent improvement, demonstrating that reward-driven self-improvement can emerge from prompting.", "questions": "1. typo: “At time step 0, an initial state S_0 sampled from p_0.” at line 101.\n2. In line 110, using θ_t and θ_{t+1} to distinguish the policy network before and after weight updates is inappropriate, because t represents the reinforcement learning time step. If the authors' method does not perform policy updates at every interaction with the environment, then using t and t+1 to make this distinction is not suitable.\n3. At line 113, I think it is inappropriate to arbitrarily name the reinforcement learning policy update process (in-weight RL), as this only adds to the burden of understanding and does not substantially improve the method.\n4. Since the state transition of the reinforcement learning environment can be constructed by continuously adding prompts, why not directly fine-tune the parameters of the LLM, rather than only using the inference capability of the LLM?\n5. Without a strict compute-matched comparison against Best-of-N, ToT/MCTS, or tuned self-revision, it’s hard to isolate where the gains come from. Though, ICRL provides some scaling and context-length analysis, stronger budget-normalized comparisons would be useful.\n6. If the environments in the experimental section can be modeled as reinforcement learning environments, why is it necessary to use LLMs to complete these tasks? These tasks are not complex; for example, in the game of 24, I believe that as long as a reward function is set, classical RL methods can also accomplish such tasks.\n7. I suggest the authors add credit-assignment analysis to better demonstrate which outputs in the decision sequence the model's performance is strongly correlated with.\n8. In Figure 2, the performance variance of ICRL Preset in the game of 24 is very large. Does this indicate that the method is highly unstable? Why does the Self-Refine method have stronger learning efficiency than the method proposed in this paper on Creative Writing? Why does the ICRL Autonomous method only perform excellently on Science World?\n9. For creative writing, Alpaca-Eval is a proxy. Pairwise win-rates are useful but should be complemented with human studies or cross-judge robustness to prevent evaluation overfitting. For game of 24, Best-of-N uses a ground-truth solver to pick the best sample, while ICRL uses its own reward r. So, some comparisons are not fully like-for-like across selection rules.\n10. I suggest the authors to add Compute-matched comparisons against Best-of-N, ToT/MCTS, and tune Self-Refine/Reflexion under equal token/cost budgets.\n11. I suggest the authors to Mask or scramble the “Reward” semantics to test whether numeric tags (without the word “Reward” or with randomized tokens) still work, which will probe whether improvements require semantic understanding vs. mere structure.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "The paper is clearly written, and the method can be adapted to various environments with minimal modifications. Results span arithmetic puzzle solving, open-ended writing, and interactive text environments, suggesting some generality.", "weaknesses": "The method is extremely simple and does not fine-tune the model via RL training. It relies only on inference-time prompting with scalar rewards. If the authors could directly fine-tune the parameters of the large language model, just like training a policy network in reinforcement learning, it would increase the complexity of the method and also enhance the novelty of the paper.\n\nThe reward provider is the same LLM as the policy, which will amplify bias and risks reward hacking by style, for example, the model may learn to write to please itself rather than to solve the task objectively. I suggest the authors use an additional large model as the reward model.\n\nThe “explore” and “exploit” text labels are strong control signal and might explain some gains independent of any learned internal RL mechanism. The ablation shows the method’s reliance on hand-crafted prompt semantics. More neutral or masked forms of reward or context (e.g., numeric tags without the word “Reward”, randomized labels) could test how much the model depends on semantic priors. I suggest removing the \"reward\" prompt during the prompting process, or placing the reward signal in other positions of prompting to test the performance of this algorithm.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)