LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making Abilities
Authors: Thomas Schmied
Jörg Bornschein
Jordi Grau-Moya
Markus Wulfmeier
Razvan Pascanu
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement Learning
Fine-tuning
Agents
Decision-Making
Exploration
Analysis
Primary area: foundation or frontier models, including LLMs

Abstract: The success of LLMs has sparked interest in various agentic applications. A key hypothesis is that LLMs, leveraging common sense and Chain-of-Thought (CoT) reasoning, can effectively explore and efficiently solve complex domains.  However, LLM agents have been found to suffer from sub-optimal exploration and the knowing-doing gap, the inability to effectively act on knowledge present in the model. In this work, we systematically study why LLMs perform sub-optimally in decision-making scenarios. In particular, we closely examine three prevalent failure modes: greediness, frequency bias, and the knowing-doing gap. We propose mitigation of these shortcomings by fine-tuning via Reinforcement Learning (RL) on self-generated CoT rationales. Our experiments across multi-armed bandits, contextual bandits, and Tic-tac-toe demonstrate that RL fine-tuning enhances the decision-making abilities of LLMs by increasing exploration and narrowing the knowing-doing gap. Finally, we study both classic exploration mechanisms, such as $\epsilon$-greedy, and LLM-specific approaches, such as self-correction and self-consistency, to enable more effective fine-tuning of LLMs for decision-making.

Published as a conference paper at ICLR 2026
LLMS ARE GREEDY AGENTS: EFFECTS OF RL ON DECISION-MAKING ABILITIES
Thomas Schmied 1++ Jörg Bornschein Jordi Grau-Moya?
Markus 2ulfmeier2 Razvan Pascanu²
1 ELLIS Unit, LIT Al Lab, Institute for Machine Learning, JKU Linz, Austria
2 Google DeepMind, Work done Google DeepMind
ABSTRACT
The success of LLMs has sparked interest in various agentic applications. A key hypothesis is that LLMs. leveraging common sense and Chain-of-Thought (CoT) reasoning, can effectively explore and efficiently solve complex domains. However, LLM agents have been found to suffer from sub-optimal exploration and the nhowing-doing gap, the inability to effectively act on knowledge present in the model. In this work, we systematically study why LLMs perform sub-optimally in decision-makee making scenarios. In particular, we closely examine three prevalent failure modes; greediness, frequency bias, and the knowing-doing gap. We propose mitigation of these shortcomings by fine-tuning via Reinforcement Learning (RL) self- generated CoT rationales. Our experiments across multi-armed bandits, contextual bandits, and Tic-tac-toe demonstrate that RL. fine-tuning enhances the decision- maaiinnn abilities of LLMs by increasing exploration and narrowing the knowing-doing gap. Finally, we study both classic exploration mechanisms, such as e-greedy, and LLM-specific approaches, such as self-correction and self- consistency, to enable more effective fine-tuning of LLMs for decision-making.
INTRODUCTION
Large Language Models (LLMs) pre- pre-trained on massive internet-scale datasets have demonstrated success across diverse domains, including text generation and language understanding (Radford et al., 2019; Brown et al., 2020b; Team et al., 2023b; 2024a; Dubey et al., 2024). Their broad pre-training distribution enables generalization to wide range of scenarios, including coding assistance (Li et al., 2022), education (Team et al., 2024d), and medicine (Saab et al., 2024). Therefore, their success has sparked interest in using LLMs for decision-making problems (Chen et al., 2023; Krishnamurthy et al., 2024; Nie et al., 2024) at the core of agentic Al systems (Durante et al., 2024).
One key hypothesis is that LLMs can generate informed action predictions without extensive vironment interaction (Lu et al., 2024) due to "world knowledge" present in the model. Moreover, Chain-of-Though (CoT) (Wei al., 2022) reasoning equips models with the ability to reason about the observed history and their actions, which facilitates environment interaction. However, these advantages often do not seem to materialize into strong performance when LLMs are faced with decision-making scenarios. Notably, Krishnamurthy et al. (2024) and Nie et al. (2024) found that LLMs do not robustly engage in exploration, resulting in sub-optimal behavior. Similar shortcomings of LLMs have been observed by Paglieri et al. (2024) and Ruoss et al. (2024) on stateful environments commonly used in RL (e.g., gorddw, Atari). Both works broadly attribute the shortcomings the knowing-doing gap, which states that models can possess knowledge about a task or can describe the consequences their behavior (i.e., they know what to do), but cannot materialize this knowledge when acting (i.c., incapable of doing). Consequently, sub-optimal exploration and the knowing-doing gap are considerable obstacles towards more powerful and robust agentic LLMs.
In this work, we aim to understand why LLMs often perform sub-optimally in simple decision-making scenarios. In particular, we systematically study three prevalent failure modes in small-to to-toooomm scale LLMs across different model families: greediness, frequency bias, and the (see Section 4.2). Our analysis shows that final performance often remains sub-optimal, because L.MSs prematurely commit greedy action selection strategies, leading to stagnating action coverage
gap

---

Published as a conference paper at ICLR 2026
Tuning isput Template
Fref
Update
Se signnt playing following (a
Store
Outpat instractions Cor Think step hy-stop and output tourts enswer the furmat ACTION-SX
Srx Sga Frx Siel
π 0
Interaction History fr O Ki Ri
Extratt
What do you prodict next?
Enviromment
Figure Illustration of our Reinforcement Learning Fine Tuning (RLFT) pipeline. We fine-tune pre-trained LLM To via ceff-generated
(CoT) rationales environment rewards.
that leaves large part of the action space unexplored (up to 55%). Moreover, we observe that small-scale LLMs (2B) tend copy the most frequent actions in the context regardless of their respective reward, which we refer to as frequency bias. In contrast, larger LLMs (27B) mostly diminish the frequency bias, yet they remain prone to greedy behavior at the cost of exploration. Similarly, we quantify the knowing-doing gap and find that LLMs often know how to solve task (87% correct rationales) but fail at acting on this knowledge as they prioritize greedy actions (64% of actions when the rationale is correct).
To overcome these shortcomings, we propose Reinforcement Learning Fine-Tunee Tuning (RLFT) on self- generated CoT rationales. RL is the predominant learning paradigm in decision-making scenarios and has been successful game-playing (Silver et al., 2016; Vinyals et al., 2019), robotics (Tirumala al., 2025), plasma-control (Degrave al., 2022), navigating stratospheric balloons (Bellemare et al., 2020). We study the effects of RLFT on pre-trained Gemma2 models (Team et al., 2024b:c) in three sizes (2B, 9B, 27B) in multi-arm bandit (MAB) and contextual bandit (CB) settings proposed by Nie et al. (2024), and the textual Tic-tac-toe environment released by Ruoss et al. (2024). Across environments, we find that RLFT enhances the decissionmmainng abilities of LLMs by increasing exploration and narrowing the knowing-doing gap. While RI.FT positively affects exploration of ILM agents, their exploration strategies remain sub-optimal. Therefore, we empirically evaluate both "classic" exploration mechanisms commonly employed in RL, such e-greedy, and LLM-specific approaches, such as self-consistency, to enable more effective fine-tuning for decision-making Finally, in our ablations. we investigate the importance of CoT reasoning for decision-makinn -making.. highlight the effectiveness of leveraging expert data, and show the benefits of giving the agent more reasoning tokens to solve the decision-make makinn problem. We emphasize that the central aim of this work is provide an in-depth analysis of LLM behavior in agentic scenarios, illustrating potential shortcomings and avenues for improvement. We do not argue believe that solely improving performance via RLFT is a complete solution to the shortcomings outlined.
In summary, we make the following contributions:
We systematically examine three failure modes of small-to welllltto-meiiun scale LLMs in decision- making scenarios: greediness, frequency bias, and the knowing-doing gap.
We study how RL fine-tuning on self-generated CoT rationales affects these shortcomings, highlighting positive effects of RLFT exploration and decision makion abilities.
We evaluate variety of exploration mechanisms (e.g., e-greedy) and LLM-specific ap- proaches (e.g., self-consistency), to enable more effective RLFT for LLMs.
2 RELATED WORK
Exploration in RL and LLMs. The trade-off between exploration and exploitation is long-standing challenge in the field of RL (Schmidhuber, 1991a;b; Still Precup. 2012; Oudeyer et al., 2007). Widely used RI. agents have often relied random schemes (Mnih al., 2015), heuristics such
2

---

Published as a conference paper at ICLR 2026
as -iste-visitation counts (Ecoffet et al., 2019; Raileanu & Rocktäschel. 2020), intrinsic curiosity (Pathak et al., 2017: Burda et al., 2018; Groth et al., 2021), behavior priors (Rao et al., 2021), 1). or maximum entropy regularization (Schulman et al., 2017: Haarnoja et al., 2018). Naturally, number of works looked into leveraging L.LMs for improving exploration of RI. agents either as a source of rewards (Klissarov al., 2023; Lu et al., 2024) or orchestrate exploration strategies (Klissarov et al., 2024). Krishnamurthy et al. (2024) investigate the in-context exploration abilities of LLMs when acting directly as policy. Similarly, Nie et al. (2024) study the exploration abilities of LLMs when fine-tumed on expert trajectories. In contrast, our work particularly investigates the effects of RLFT on the exploration abilities of LLMs and focuses on why models fail.
In-context Learning for Decision-Making. ICL is a form of Meta- Meta-learning, also referred to as beaning-to-leear (Schmidhuber, 1987). While meta-learning is targeted via a meta-training phase (Santoro et al., 2016; Mishra et al., 2018; Finn et al., 2017; Wang et al., 2016; Duan et al., 2016; Kirsch et al., 2019; Flennerhag et al., 2019; Team et al., 2023a), ICL emerges as a result of the pre- training data distribution (Chan et al., 2022; Kirsch et al., 2022). ICL has been rediscovered in LLMs (Brown et al., 2020a) after initial observations by Hochreiter et al. (2001) in LSTMs (Hochreiter & Schmidhuber, 1997). Mirchandani al. (2023) leverage the ICL abilities of LLMs to operate as general pattern machines. A number of works leverage the CoT abilities (Wei et al., 2022) of LLMs in simple text-based scenarios (Shinn al., 2023; Yao et al., 2022). Similar in-context decision-making abilities have been observed in models trained from scratch, alheit in restricted environments (Laskin al., 2022; Lee et al., 2022; Kirsch al., 2023; Raparthy et al., 2023; Schmied et al., 2024b;a).
Self-Correction in LLMs. A critical component for LLM agents is the ability to self-correct over previously explored attempts. Existing works focus primarily on math benchmarks (Cobbe et al., 2021; Hendrycks et al., 2021; Welleck et al., 2022). Zelikman et al. (2022) leverage hints to iteratively generate correct answers and fine-tune on the respective CoT rationales. Kumar et al. (2024) employ RLFT over multiple trials to induce colf-correctio.. Similarly, Zelikman et al. (2024) make use of RL fine-tuning, but instead generate rationales every token position. Instead of imitation, Wang et al. (2025) rely on critique fine-tuning to induce self-correction. Wulfmeier et al. (2024) make use of inverse RL to avoid compounding errors. Other works rely on ICL abilities to learn from previous mistakes (Zhang et al., 2024; Monea et al., 2024). In contrast to Monea et al. (2024), who use ICL and contexts that span multiple previous episodes, we rely on reasoning abilities. Moreover, the effects of RLFT, rather than proposing a new method. While conceptual corrections are possible, exact token-level correction is usually difficult for autoregressive generation (Cundy & Ermon, 2023).
contexts and CoT focus analyses of the maakinnn-making abilities of LLMs and
METHODOLOGY
3.1 BACKGROUND
Reinforcement Learning. We assume the standard RL formulation via a Markov Decision Process (MDP) represented by a tuple of (S, P, R.,, where S and A denote state and action spaces, respectively. At every timestep t the agent observes state st E S, predicts action a; E A, and receives a reward Ft given by the reward function R(5t, at). P(s 8t, az) defines the transition dynamics constituting probability distribution over next states 86+1- The goal of RL is to learn policy To(a; 8t) with parameters 0 that predicts an action 0₂ in state that maximizes cumulative reward.
Reinforcement Learning from Human Feedback. RLHF aims to fine-tune pre-trained models towards human preferences (Christiano et al., 2017). Preferences are typically encoded via a reward model r with parameters á learned from -annoa-a dataset D consisting of query-response pairs z and y, respectively. RLHF optimizes constrained REINFORCE estimator (Williams, 1992):
max E.- [(, y) - V
a log o x) - (DDr x)) Tpe xx]] (1)
Here Tref is a reference policy, which is typically the frozen pre-trained model, and 3 is a weighting term. The baseline b represents baseline to reduce variance and is commonly instantiated by a value function (Schulman et al., 2017: Ouyang et al., 2022) or a Monte-Carle (MC) estimate of the returns (Ahmadian et al., 2024; Ramesh et al., 2024; Shao et al., 2024).
3

---

Published as a conference paper at ICLR 2026
3.2 REINFORCEMENT LEARNING TIN-IUNNIG (RLFT)
Our RLFT approach relies on environment interaction. During RLFT, the model learns iteratively refine its reasoning process, favoring CoT patterns and actions that lead to higher rewards (see Figure I), Our approach is akin to Guo et al. (2025) and targets decision-making scenarios similar to Zhai et al. (2025).
on self-generated CoT rationales on rewards obtained from
Context Representation. The input tokens to our model at step consists of input instructions 4", output instructions egu,, and the most recent interaction history o Ctt (see Figure 1). The history representation contains the trajectory Tt-Car (st-C+ C, Mt-C,Tr-C:.... 8t. Ut, of the C most recent states, actions, and rewards. We opt for task-specific instructions for of rather than a generic instruction template. providing the agent with information about the observations, the possible actions, and its objective. Consequently, et is represented by the concatenation of the instruction and history tokens ot = [ct" :e:tt C4 G.).
Factorization of Action Tokens. At every interaction step f. the agent generates action tokens 4 [=[oo 0s] containing both the CoT reasoning tokens 2 _CoT' and the action to be executed in the environment ar- To extract 0r from 2,, we make use of an extraction function a = g(x). ). In practice, g consists of regular expressions match the output pattern given by equl, If no valid action is found, a random action is executed. To allow for flexibility in refining the reasoning process, we opt for a permissive output template (i.e., ACT ACTION=X), rather than enforcing structured output template (e.g.. <action> blocks). We employ a token generation budget of G tokens (=256), therefore |zel S G.
Reward Shaping for Valid Actions. In addition the environment reward rę"", we employ reward shaping term p.aalid to encourage the model to adhere to the output template, = prne + yaalidd More specifically, we make use of reward penalty of 5 if g cannot extract valid action, r ryoll -5- 1 (g(afet) & A). To ensure that the reward penalty does not overly bias optimization, we employ reward normalization to the environment rewards.
Fine-tuming objective. We fine-tune using the clipping objective introduced by Schulman et al. (2017) with and additional KL constraint to the reference policy Tref:
(z(c)) TToad (2|c) Aade, clip,
( To(2lc) Toaaco (z|c) ) Aado -BDKs
max B (c,s)DD min
(2)
Here Thaia refers to the rollout generating policy, D is the rollout buffer, and is hyperparameter. fine-tuning in environments with fixed episode lengths (bandits), we
To allow for memory-efficiee make use of a Monte Carlo baseline to estimate Aade Instead of exploiting multiple rollouts, used by Ahmadian et al. (2024) and Ramesh et al. (2024), we compute rewards-to-go. For environments with variable episode lengths (Tic-tac-toe), we learn separate state-value head top of the last layer LLM representations and make use of generalized advantage estimation (Schulman et al., 2015) (see Appendix B) We provide additional implementation and training details in Appendix B.
4 LLMS FOR DECISION-MAKING
We study the effect of fine-tuning LLMs in MABs and CBs settings proposed by Nie et al. (2024), and on a text-based version of Tic-tac-toe released by Paglieri al. (2024). For our experiments, we primarily focus on Gemma2 (Team et al., 2024c) at three model scales (2B, 9B, and 27B) and report analyses for Llama3 (Dubey al., 2024) and Qwen2.5 (Qwen al., 2025) in Appendix C.4. In Section 4.2, we first analyze three common failure modes of LLM agents in MABs: (1) greediness. (2) frequency bias, and (3) the knowing- doing gap. Then we investigate the effects of fine-tuning on self-generated CoT rationales or expert rationales in MABs/CBs (Section 4.3), and in Tic-tac-toe (Section 4.5). In Section 4.4, we study the effects of exploration mechanisms on the fine-tuning performance. Finally, in Section 4.5 we empirically examine important components of our approach.
4.1 ENVIRONMENTS & BASELINES
Multi-armed and Contextual Bandits. MABs (Slivkins et al., 2019; Lattimore & Szepesvári, 2020) are a classic problem setting in RL that isolates the experiments, we leverage the bext- bandit scenarios released by Nie et al. (2024). We focus
trade-off. For our MAB
4

---

Published as a conference paper at ICLR 2026
on the continuous and button variants, as illustrated in Figure 2. We report results for MAB with E {5, 10, 20} arms (I.A) = k) and payoffs of the arms being either Gaussian or Berouuli distributed. In addition, consider three levels of stochasticity (low/mediumhhigh) that determine the standard deviation or delta gap in Gaussian or Berouulii bandits, respectively. For all MAB settings, we limit the horizon T to 50 interaction steps. We provide more details MAB setup in Appendix A.I.
We compare against two commonly used base- lines for MABs: Upper-confidence Bound (UCB) (Auer, 2002) and a random agent that selects actions uniformly at random. UCB is considered optimal and represents the upper bound for agent performance, whereas the ran- dom baseline represents the lower bound. We want to emphasize that we do not aim to out- perform UCB with LLMs, but instead aim for better understanding. We provide more details our baselines and Appendices A. and A.2, respectively.
Button Multi-armed Bandit (Gaussian)
You are a bandit algorithm room with 5 buttons labeled red, green, blue, yellow, orange. I...].
Think step-by-step and output your final answer format ACTION=X where of the arms listed above. IMPORTANT: Provide your (SHORT!) thinking process and your ACTION=X
setup for CBs in
Tic-tac-toe. In addition, use the text-based Tic-tac-toe environment released by Ruoss et al. (2024), which exhibits proper state transitions. Ruoss et al. (2024) demonstrated that frontier models struggle achieve strong performance in this environment and often barely beat a ran- dom opponent. Consequently, it is good target to investigate the efficacy of RLFT. In Appendix A.3, we provide additional details on our Tic- tac-toe environment and training setup.
So far you have tried/scen:
Step=0 Action=green Reward=0.3
Step=1 Action=blue Reward=0.1
Step=2 Action=orange Reward=00.5
What do you predict next?
Figure 2:
Button MAB from (Nie et al., 2024)
using our context representation and instructions.
4.2 WHY DO LLMs PERFORM SUBOPTIMALLY IN CECISION-MAKINGS
Prior works found that LLM agents often perform suboptimally and fail explore sufficiently in interactive settings (Paglieri et al., 2024; Ruoss et al., 2024). Therefore, we first examine why models perform suboptimally and identify three prevalent failure modes: (1) greediness, (2) frequency bias, and (3) the knowing-doing gap. In this section, we present analyses of LLMs models when given input contexts that elucidate the failure corresponding modes. We conduct our analyses the button instance of our MAB experiments at three model scales, and find that the failure modes persist across model scales (see Appendix C.1 for additional results on continuous MAB instance).
Turemciars 25
0,7
Ta Nanemncirrs
1.0
Coverade
0.6
0.6
25
Model
270
0.5
Action Coverage
0.6
28
0.5
95
True
0.4
True
278
0.4
Space
Random
0.3
0.3
CoT
False
0.2
Tue
0.1
0.1
0.0
100
50
40
Cumulative Regret
(a) Action Coverage: 10 arms
(b) Action Coverage: 20 arms
(c) Coverage Regret: 10 arms
Figure 3: Illustration of greediness. We show action coverage for Gemma2 2B/9B/27B w/ and
w/o CoT for (a) and (b) 20 arms over 50 interaction steps. Agents favor the best-performing action among the set of selected actions, leading to stagnating action coverage, despite benefits of larger models and COT. (c), we plot cumulative regret against action coverage. The agents exhibit suboptimal regret because of greedy action selection strategies.
5

---

Published as a conference paper at ICLR 2026
Greediness. The first and most pervasive failure mode is greediness, which is characterized by the LLM overly favoring the best besf-performing action among a small set of actions seen so far. To illustrate this failure mode, we show the average action coverage achieved by Gemma2 2B/9B/27B with and without CoT across 64 MABs with 10 and 20 arms over 50 interaction steps (see Figure 3 a and b). We define action coverage C, at step & the fraction of available actions that have been selected at least once, C. [aEAA: N=(a)=0) with Ne(a) representing the number of times action a É A has been selected until t. For 10 arms and averaged over 64 parallel environments, we find that Gemma2 2B covers 40% of all actions, while 9B/27B cover 65% (i.e., 6.5 actions), leaving significant part of the action space unexplored. Note that without COT, all models explore merely 25% of all actions in the 10 arms setting. The suboptimal coverage is caused by the model overly favoring high-reward actions (see Figure 15 in Appendix C.1.1). Consequently, the model prematurely commits to greedy strategy, leading to a stagnating action coverage beyond 10 steps. Increasing the number of arms makes the greediness even more apparent, with the largest models only covering 45% of all actions. Due to this, the regret remains high compared UCB, even though the models improve significantly over random agent (see Figure 3c). We repeat our analyses with the Llama3 (Dubey et al., 2024) and Qwen2.5 (Qwen et al., 2025) model families in Appendix C.4 and show that these biases persist.
[AI
green
red
bluc
orange
black
122
Frequent
Corelationn-@
-0.96
hadionnpe per abodv
12
Frequent 6.11
Comclation-
1.0
Achin Energey
0.5
0.6
Frocian our window
278
Actinn
graddy
100
0-10
90-150
100
(a) 2B: Correlations
(b) Strategy per Window
(c) 27B: Correlations
Figure 4: Illustration of frequency bias. We plot the frequency of the repeated action in the context against the action entropy across all actions for 10 armed MABs in the button scenario (actions are colors). (a) Gemma2 2B heavily suffers from frequency bias, becoming more certain of the most frequent action, the more often it occurs in the context. (c) Gemma2 27B overcomes the frequency bias, but instead behaves greedily. In (b), show the action strategies for three repetition windows.
Frequency Bias. The next prevalent failure mode is frequency bias, characterized by repeatedly select- ing the most frequent action in the context, when that action gives low reward. To understand how the model's 'ss behavior is influenced by the fre- quency, we construct prefix histories using a ran- dom policy, vary the number of repetitions of the last action in the history (0-100) and record the en- tropy over all actions (Figure 4a and c). See Ap- pendix C.1.2 for details on the context generation. To quantify frequency bias, we categorize an ac- tion as frequent a, arg max (a), greedy ag arg max A: (a)>0) (a),), or other if greedy. Note that an
"Doing" Action Strategy optimal
gready
other
inowtngg Correctness
21.6
136w
Falso
24h
Figure 5: Confusion matrix for the knowing- doing gap of Gemma2 27B. The agent "knows" how to solve the task (87% correct rationales, sum of top row), but fails at "do- ing" (58% greedy actions among correct ratio- nales). See Figure 26 for the CoT instructions and an agent response.
they are neither frequent action is optimal with 10% probability. Subsequently, compute the frequent Fj, greedy Fy. and other F, fractions as reported in Figure 4 (see Appendix for additional definitions).
Gemma2 2B heavily suffers from repeated actions, exhibiting decreasing entropy with increasing repetitions (96% Fy, see Figure 4a). In contrast, 27B

---

Published as a conference paper at ICLR 2026
escapes the frequeney bias (14%, see Figure 4c) and interestingly becomes less certain of its action prediction with increasing repetitions. To examine this further, we show the bucketized fractions with 0-10, 45-55, and 90-100 repetitions for 2B and 27B in Figure 4b. Indeed, for 2B F keeps increasing with increasing repetitions. While 27B escapes the frequency bias, it suffers heavily from greediness. Similar biases have been identified in Behavior Cloning (BC) settings using small models and termed copycat bias (Wen et al., 2020; Schmied et al., 2024b). This suggests that the persistent frequency bias in smaller models may be an artifact supervised pre-training or insufficient model capabilities, and motivates the use of RL as counter- measurement.
Knowing-Doing Gap. The knowing-doing gap has been observed by Paglieri al. (2024) and Ruoss et al. (2024). Our simple MAB setting is well-suited to quantify the knowing- To investigate this gap our setting, we first task Gemma2 27B to produce the UCB algorithm, to compute the relevant quantities accordingly (" ("knowing"), and finally to act according to the computed quantities ("doing". see Figure 26 for the instructions and an agent response). We let Gemma2 27B interact with the environment (64 instances) for 50 timesteps with G 2048 per step, and extract the UCB quantities from the generated rationales. To quantify "knowing". we compare the UCB values computed by the model against the real UCB values, and consider the rationale 2CoT as correct if the arm with the highest UCB values matches (see Appendix C.1.3 for details). To quantify "doing", ", we categorize the generated actions optimal action if the model selects the action with the highest UCB value, as greedy if it selects the action with the highest UCB value among the set of actions tried so far, and as other if the action is neither optimal nor greedy. Subsequently, we compute the percentages of greedy/optimal/olie rationales being correct (see Figure 5). However, even for correctly computed rationales, the model often selects the greedy action (58%) over the optimal action (21%). This discrepancy highlights the shortcomings of the LLM when it comes to "acting" even when "knowing" the algorithm.
gap precisely.
actions. The agent clearly knows how to solve the task, with 87% of all
4.3 EPFECTIVENESS OF RL FINE-TUNING
- 28, RLFT
-- 28, ICL
98, ICL
98, RLFT
278,
UCB
Random
ACTIOM
Amc-10
Ams+20
70
75-
63
Dum detee Ma..oeeeggggeege
43
33
20
10
Step
Figure 6: Main Comparison Gaussian MABs button scenario in the medium noise (a 1) setting. We compare cumulative regrets (lower is better) of classic basclines against ICL and RLFT performances for 5, 10, and 20 arms. See Figure 20 for 0.1 and o = 3.
Next, we study the effects of RLFT on cumulative regret (w.r.t. optimal policy) and whether alleviates the highlighted failure modes. We fine-tune Gemma2 2B and 9B on self-generated CoT rationales for 30K updates with an (accumulated) batch size of 128. To avoid memorization of reward distributions, maintain a pool of 512 MABs and randomly select subset of 16 MABs per rollout during training, and report evaluation results on hold-out MABs (see Appendix B for details).
RLFT lowers regret. In Figure 6, we report the cumulative regrets across model sizes and arms for a medium noise o = 1.0 scenario (see Appendix C.2 for low/high noise). Across environments, the LLMs clearly outperform the random baseline, and RLFT lowers regret for both 2B and 9B. For 2B, RLFT narrows the gap to its larger counterparts and UCB. Similarly. RLFT lowers regret for Gemma2 9B. Note that the lower cumulative regret of Gemma2 9/27B compared to UCB after 50 environment steps in the 20 arms scenario is an artifact of the limited interaction steps, but the trends remain clear. We repeat RLFT for CBs, and observe similar performance improvements for Gemma2 2B (see Appendix C.3). Consequently, reinforcing self- aeff-geeratee CoT rationales towards environment rewards improves performance on simple decision-makinn scenarios.
7

---

Published as a conference paper at ICLR 2026
RLFT mitigates greediness. In Figure 7, we report the action erage for 2B after RLFT at different numbers of gradient steps (10K, 20K, 30K). Indeed, coverage (+ ++1% after 30K updates. Interestingly, we first observe a decrease (at 10K) followed by increase in action coverage (20K, 30K). We observe similar effects for the 20 arms scenario (see Figure 17). Via RLFT, the agent learns to explore its options and mitigate greediness.
Model
0.
observe that RLFT results in increased action
Spare Covergg
dOOc Thxt
12H
Astion Spaton
0.2
RLFT counteracts frequency bias. We find that RLFT counteracts frequency bias (see Figure 19). In particular, for 0-10 repetitions, we observe strong decrease the fraction of frequent actions (70% + 35%) and an increase in "other" actions (8% - 35%). However, F remains elevated for high repetitions. Consequently, RLFT counteracts frequency bias, but does not fully alleviate it.
Figure 7: Effect of RLFT on greediness for Gemma2 2B.
4.4 EFFECTS OF EXPLORATION MECHANISMS
For RLFT, we relied solely the exploration properties for CoT reasoning. However, in RL it is common practice to employ additional exploration strategies (Mnih al., 2015; Schulman et al., 2017: Haarnoja et al., 2018). Therefore, study the effects of classic exploration mechanisms and LLM- specific strategies to encourage exploration. We compare: (1) try-all actions initially similar to UCB, (2) e-greedy, (3) context randomization, (4) context summary similar to Krishnamurthy et al. (2024) and Nie et al. (2024). (5) self-correttion similar to Kumar et al. (2024), (6) self colsissssstt (Wang al., 2022), and (7) incorporating exploration bonus during RLFT. We provide details each mechanism in Appendix B.4. Across model scales, we observe that the mechanisms result in varied effects on action coverage (see Figure 8). First, we find that the simple try-all strategy, which reduces the need for additional exploration by trying all actions, results in large performance improvements. This suggests that only given sufficient information about the (sub-)optinn cubb)ootimallty of actions, LLMs are able to select actions accordingly. underscoring their exploration shortcomings. Second, a simple exploration bonus (+1 reward for untried actions) during RLFT significantly increases exploration (50% + 70%) and lowers regret towards the expert compared to regular RLFT. This highlights the importance of reward shaping for fine-tuning LLMSs to elucidate desired behavior.
Iry-all
e-greedy
Context Rindomization
Context Summary
Self Correction
Self Consistency
Exploration Bonus
UC3
Model Gemma2 28
Model Gemma2 28, RLFT
Model Gemma2 98
Model
Gemma2 278
1.0
Actian Coverage
0.8
X
X
0.4
70
60
50
70
50
40
30
70
60
50
70
60
40 30
Aug. Cumulattve Regret
Avg. Cumulative Regret
Avg. Cumulative Regret
Avg. Cumulative Regret
Figure 8: Effect of exploration mechanisms on action coverage and cumulative regret.
4.5 ABLATIONS
RLFT in Tic-tac-toe. To investigate the efficacy of RLFT in stateful environments, we evaluate on Tic-tac-toe from Ruoss al. (2024), which frontier models struggle to achieve strong performance (see Appendix B for training details). We fine-tune against three opponents: random agent, Monte Carlo Tree Search (MCTS) (Coulom, 2006), and noisy MCTS (50% of actions selected at random). We find that RLFT significantly enhances the win-rate of Gemma2 2B against all opponents compared to ICL (see Figure 9a). Against the random agent, RLFT elevates the average return from 0.15 (i.c., winning 15% of games) to 0.75. Notably, the agent even manages draw against the optimal MCTS
8

---

Published as a conference paper at ICLR 2026
baseline (-0.95 0.0), underscoring the effectiveness of RLFT for decision-makee making. However, for high performance, it essential to provide the currently legal actions the context (see Figure 25).
Model
Tokens
60
16
0.50
0.25
...r
Ropret
50
SFT
UCB
Cumurative
1o
256
512
RUFT
-0.50
20
20
MCTS
25 v5. MCTS, Done +05
Bandom
0.2 0.4
0.5
08
1.0
10
Envirommnn Staps
le5
Step
(a) Tic-tac-toe: Opponents
(b) MAB: Effect of CoT
(c) MAB: "Thinking Time
Figure 9: Ablations. (a) Effect of RLFT in Tic-tac-toe from Ruoss al. (2024). (b) Effect of CoT on ICL, RLFT and SFT (expert data) performance on MABs. (c) Effect of increasing the number of "thinking" tokens to generate during RLFT.
Importance of CoT for RLFT. CoT reasoning is critical for ICL performance (see Figure 3), but the question remains how CoT influences RLFT. Therefore, we run RLFT on Gemma2 2B on the 10 arms Gaussian MAB both with and without CoT (see Figure 9b, RLFT). Indeed, without CoT, RLFT barely attains the performance of ICL with COT. This highlights the function of CoT vital exploration and rationalization mechanism for decision-making.
Expert Behavior Cloning Thought Cloning. BC is prevalent approach in sequence models for decision-making (Pomerleau, 1988; Brohan et al., 2022; 2023) and relies on expert datasets. Consequently, we construct two UCB expert datasets comprising 32K rollouts (1.6M transitions) across different MABs either with or without CoT traces (see Figure 12 for an illustration) and perform SFT on them. Notably, both SFT variants successfully mimic the expert, achieving comparable regret to UCB (see Figure 9b, SFT). This result underscores the effectiveness of expert data in decision-making echoing findings in reasoning tasks (Muennighoff al., 2025).
Effect of Thinking Time. Finally, we investigate the effect of giving the agent more/less time to "think" during RLFT by varying the generation budget G (see Figure 9c and Appendix D). Decreasing G results in poor performance, as the agent is unable to rationalize its decisions. Increasing G to 512 improves performance to the level of 9B w/ RLFT. The agent effectively leverages the additional tokens, which reflects recent observations in mathematical reasoning (Guo et al., 2025). However, when increasing G, rollout generation can make up the majority of the training time due to the multi-step nature of decision-makinn tasks (e.g., for H 50, G 500, agent generates 25K tokens).
CONCLUSION
We study why LLMS perform sub-optimally in decision-making scenarios and examine three prevalent and independent failure modes: greediness, frequency bias, and the knowing-doing gap. We empha- size that we focus on in-depth analyses to provide fundamental understanding of LLM behavior in decision-making scenarios and aid future research, rather than solely improving algorithm performance via RLFT. In particular, our rigorous empirical analysis aims to inform design decisions of future LLM agents. For example. when building tool-use or coding agents, it may be essential to restrict the breadth of available tools initially due to pre-existing greediness biases (Yang et al., 2025). We show that RLFT on CoT rationales mitigates greediness, counteracts frequency bias, and improves performance. While RLFT improves exploration abilities, it remains sub-optimal compared to bandit algorithms. Therefore, we investigate a varicty of mechanisms to enhance exploration. Models act optimmlpy if provided with sufficient information, underscoring their shortcomings in exploration. Finally, we highlight the importance of reward shaping for RLFT.
Future Work. We primarily focused our evaluation the Saaimaa//llmaa///veaaa meall-to-medium scale models. While we expect that our findings transfer to larger models, we deem research into frontier models important. Moreover, MAB experiments were conducted with
series and

---

Published as a conference paper at ICLR 2026
limited horizon of 50 environment steps, which is sufficient for 5 and 10 arms, but insufficient for 20 arms. Due to its simplicity, the MAB setting also allows us to rigorously study how the learned decision-making and exploration abilities transfer to unseen environments in future work (c.g., by varying the number of arms or shifting the reward distribution at test time). We expect that RLFT may increase robustness compared to SFT in such scenarios, which would reflect findings by Chu et al. (2025). For future work, we believe that evaluating the exploration abilities of LLM agents particularly interesting in environments that require targeted exploration towards an end-goal. First, this includes other stateful environments from Paglieri et al. (2024), Ruoss et al. (2024), and Tajwar et al. (2025). A deeper investigation into the knowing-doing gap would be particularly interesting in these environments. Second, we deem systematic investigation into the exploration abilities of LLMMs existing agentic and computer-use benchmarks (Mialon et al., 2023; He et al., 2024; Zhou et al., 2023) interesting. While RIFT positively influenced the agent's decision-making abilitics, our experiments demonstrated that performance remained suboptimal. Simple exploration mechanisms, such as an extrinsic reward bonus, considerably improved performance. Similarly, intrinsic reward mechanisms (Pathak et al., 2017; Burda et al., 2018) to encourage autonomous exploration could be valuable components of future agents in multi-turn applications. Moreover, in our ablations, we found that LLMs benefit from additional "thinking" time and believe that allowing for larger generation budget becoming increasingly important for agentic scenarios, especially for scenarios that involve high-stakes decisions (e.g., economics or AI safety). We deem investigations into such high-stakes scenarios fruitful for future work. While increasing "thinking" time improves performance, it comes with excessive computational cost at training time due to the rollout generation and the multi-step nature of decision-making. Therefore, modern recurrent architectures (Gu Dao, 2023; De et al., 2024; Beck et al., 2025) that allow for faster inference may be promising alternatives for recision-making and scaling-up RLFT.
REPRODUCIBILITY STATEMENT
To ensure reproducibility, we provide comprehensive details in the Appendix. We describe environment setup, the generated datasets, and our baselines in Appendix A. The primary benchmark we focus on, BanditBench, was released by Nie et al. (2024) and available on GitHuu. Furthermore, in Appendix B, we provide the implementation details for all RLFT and SFT experiments, along with comprehensive list of hyperparameters. Moreover, in Appendix B.4, we describe and provide details for the exploration mechanisms compared in Section 4.4. Finally, in Appendix C, we provide additional experimental results that we could not fit in the main text.
USE OF LARGE LANGUAGE MODELS
In preparing this manuscript, we used LLMs to polish existing text, specifically for improving grammar and some phrasings. No LLMs were used for research ideation or exploration.
REFERENCES
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34:29304-29320 2021.
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXiv:2402. /4740, 2024.
Jose A Arjona-Meding Medina, Michael Gillhofer, Michael Widrich, Thomas Unterthiner, Johannes Brand- stetter, and Sepp Hochreiter. Rudder: Return decomposition for delayed rewards. Advances in Neural Information Processing Systems, 32, 2019.
Peter Auer. Using confidence bounds for exploitation-explorai trade-offs. Journal of Machine Learning Research, 3(Nov):397-47) 2002.
Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xlstm: Extended long short-term memory. Advances in Neural Information Processing Systems, 37:107547-007603, 2025.
10

---

Published as a conference paper at ICLR 2026
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Marc G Bellemare, Salvatore Candido, Pablo Samuel Castro, Jun Gong, Marlos C Machado, Sub- hodeep Moitra, Sameera Ponda, and Ziyu Wang. Autonomous navigation of stratospheric balloons using reinforcement learning. Nature, 588(7836:77--22 2020.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control scale. arXiv preprint arXiv:22/2.0000
2022.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chehotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language-action models transfer web knowledge robotic control. arXiv preprint arXiv:2307.15878 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020a.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020b.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint 1rXiv: 12844, 2018.
Stephanic Chan, Adam Santoro, Andrew K. Lampinen, Jane Wang, Aaditya Singh, Pierre H. Richemond, James L. McClelland, and Felix Hill. Data distributional properties drive emer- gent in-context learning in transformers. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28- December 2022, 2022.
Liting Chen, Lu Wang, Hang Dong, Yali Du, Jie Yan, Fangkai Yang, Shuang Li, Zhao, Si Qin, Saravan Rajmohan, et al. Introspective tips: Large language model for in-context decision making. arXiv preprint ar arXiv:2305.1 11598, 2023.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc Le, Sergey Levine, and Yi Ma. Sit memorizes, generalizes: A comparative study of foundation model post-training arXiv preprint arXiv:2501.17161, 2025.
Wei Chu, Lihong Li, Lev Reyzin, and Robert Schapire. Contextual bandits with linear payoff functions. In Proceedings of the fourteenth international conference artificial intelligence and statistics, pp. 208-214. JMLR Workshop and Conference Proceedings, 2011.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jemry Tworek, Jacob Hilton, Rejichiro Nakano, al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 14168, 2021.
Rémi Coulom. Efficient selectivity and backup operators monte-carlo tree search. In International conference on computers and games, pp. 72-83. Springer, 2006.
Chris Cundy and Stefano Ermon. Sequencemateh: Imitation learning for autoregressive sequence modelling with backtracking. arXiv preprint wwXXi:23066054266, 2023.
Soham De, Samuel L Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Murant Albert
Gu, Ruba Haroun, Leonard Berrada. Yutian Chen, Srivatsan Srinivasan, et al. Griffin: Mix- ing gated linear recurrences with local attention for efficient language models. arXiv preprint arXiv:2402.1 /9427, 2024.
11

---

Published as a conference paper at ICLR 2026
Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, et al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897)444 419- 2022.
Yan Duan, John Schulman, Xi Chen, Peter L Bartleu, Ila Sutskever, and Pieter Abbeel. RI2: Fast 1.02779, 2016.
reinforcement learning via slow reinforcement learning, arXiv preprint
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad AI-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, al. The llama herd of models. arXiv preprint arXiv:2407.2 17833 2024.
Zane Durante, Qiuyuan Huang, Naoki Wake, Ran Gong, Jac Sung Park. Bidipta Sarkar, Rohan Taori, Yusuke Noda, Demetri Terzopoulos, Yejin Choi, et al. Agent ai: Surveying the horizons of multimodal interaction. arXiv preprint arXiv:2401.0m 4rX1v:2401.0356, 03568, 2024.
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Go-explore: new approach for hard-exploration problems. arXiv preprint 1r01v
2019.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agmostic meta-learning for fast adaptation of deep networks. International conference on machine learning. pp. 1126-1135. PMLR, 2017.
Sebastian Flennerhag, Andrei Rusu, Razvan Pascanu, Francesco Visin, Hujun Yin, and Raia Hadsell. Meta-leaming with warped gradient descent. arXiv preprint arXiv: 1xXiv:1909000025 2019.
Oliver Groth, Markus Wulfmeier, Giulia Vezzani, Vibhavari Dasagi, Tim Hertweck, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Is curiosity all you need? on the utility of emergent behaviours from curious exploration. arXiv e-prints, pp. arXiv-2109, 2021.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. /2948, 2025.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint 1812::811.0090000 2018.
F Maxwell Harper and Joseph A Konstan. The movielens datasets: History and context. Acm Iransactions on interactive intelligent systems (lus), 5(4):1-19, 2015.
Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models. arXiv preprint arXiv:2401.13919, 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint ar 1XXx:2103303874 2021.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Sepp Hochreiter, A Steven Younger, and Peter R Conwell. Learning to learn using gradient descent. In Artificial Neural Networks-ICAr 2001: International Conference Vienna, Austria, August 21-25, 200] Proceedings 11, pp. 87-94. Springer, 2001.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhil Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
12

---

Published as a conference paper at ICLR 2026
Shengran Hu and Jeff Clune. Thought cloning: Learning think while acting by imitating human thinking. Advances in Neural Information Processing Systems, 36:44451 2023.
Louis Kirsch, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Improving generalization in meta reinforcement learning using learned objectives. arXiv preprint arXiv:1910.04098, 2019.
Louis Kirsch, James Harrison, Jascha Sohl-Dickstein, and Luke Metz. General-purpose in-context learning by meta-learning transformers. arXiv preprint UrXiv:2212.04455. 2022.
Louis Kirsch, James Harrison, C Freeman, Jascha Sohl-Dickstein, and Jürgen Schmidhuber. Towards general-purpose in-context learning agents. NewrIPS 2023 Workshop on Generalization in Planning, 2023.
Martin Klissarov, Pierluca D'Oro, Shagun Sodhani, Roberta Raileanu, Pierre-Luc Bacon, Pascal Vincent, Amy Zhang, and Mikael Henaff. Motif: Intrinsic motivation from artificial intelligence feedback. arXiv preprint arXiv:23 10.00166, 2023.
Martin Klissarov, Mikael Henaff, Roberta Raileanu, Shagun Sodhani, Pascal Vincent, Amy Zhang, Pierre-Luc Bacon, Doina Precup, Marlos C Machado, and Pierluca D' O'Or. Maestromotif: Skill design from artificial intelligence feedback. arXiv preprint arXlv:2412.08542, 2024.
Akshay Krishnamurthy, Keegan Harris, Dylan Foster, Cyril Zhang, and Aleksandrs Slivkins. Can large language models explore in-context? arXiv preprint arXiv:2403. 15371, 2024.
Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co- Co-Reyes, Avi Singh, Kate Baumli, Shariq lqbal, Colton Bishop, Rebecca Roelofs, et al. Training language models to self-correct via reinforcement learning. arXiv preprint arXiv:2409. 12917, 2024.
Michael Laskin, Luyu Wang, Junhyuk Oh, Emilio Parisotto, Stephen Spencer, Richie Steigerwald, DJ Strouse, Steven Hansen, Angelos Filos, Ethan Brooks, et al. In-context reinforcement learning with algorithm distillation. arXiv preprint arXiv:2210.1 122:2220.. 2022.
Tor Lattimore and Csaba Szepesvári. Bandit algorithms. Cambridge University Press, 2020.
Kuang-Huei Lee, Ofir Nachum, Mengjiao Yang, Lisa Lee, Daniel Freeman, Winnie Xu, Sergio Guadarrama, lan Fischer, Eric Jang. Henryk Michalewski, et al. Multi-game decision transformers. 2022.
arXiv preprint arXiv:2205.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, al. Competition-level code generation with alphacode. Science, 108(6624) 1092-1097, 2022.
Cong Lu, Shengran Hu, and Jeff Clune. Intelligent go-explore: Standing on the shoulders of giant foundation models. arXiv preprint arXiv:2405. 2024.
Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: benchmark for general ai assistants. In The Twelfth International Conference Learning Representations, 2023.
Suvir Mirchandani, Fei Xia, Pete Florence, Brian Ichter, Danny Driess, Montserrat Gonzalez Arenas, Kanishka Rao, Dorsa Sadigh, and Andy Zeng. Large language models general pattern machines. arXiv preprint 200ivv2307.0472), 2023.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta- learner. 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https://openreviee net/forum? d-B1DmBBBWW......
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 5 51877540):5995533 2015.
Giovanni Monea, Antoine Bosselut, Kianté Brantley, and Yoav Artzi. Llms are in-context reinforce- ment learners. arXiv preprint arxiv:2410.05362, 2024.
13

---

Published as a conference paper at ICLR 2026
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannanch Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling, arXiv preprint arXiv:2501.1939) 2025.
Allen Nie, Yi Su, Bo Chang, Jonathan N Lee, Ed H Chi, Quoc V Le, and Minmin Chen. Evolve: Evaluating and optimizing lims for exploration. arXiv preprint arXiv:2410.06238,, 2024.
Pierre-X Yves Oudeyer, Frdric Kaplan, and Verena V Hafner. Intrinsic motivation systems for au- tonomous mental development. IEEE transactions evolutionary computation, 11(2):655-266
2007.
Long Ouyang. Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Davide Paglieri, Bartlomiej Cupial, Samuel Coward, Ulyana Piterbarg, Maciej Wolczyk, Akbir Khan, Eduardo Pignatelli, Fukasz. Kucinski, Lerrel Pinto, Rob Fergus, et al. Balrog: Benchmarking agentic I1m and vlm reasoning on games, arXiv preprint arXiv:2411.13543 2024.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pp. 2778-2787. PMLR, 2017.
Dean A Pomerleau. Alvinn: An autonomous land vehicle in neural network. Advances in neural information processing systems, 1, 1988.
Qwen, An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang. Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL petpss//arsee wxvvvv..vvvvv org/ wwwg/rs////////ooooo..ooo...ssre /bbs/2412. 15115.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Roberta Raileanu and Tim Rocktäschel. Ride: Rewarding impact-driven exploration for procedurally- generated environments. arXiv preprint arXiv:2002. /2292, 2020.
Shyam Sundhar Ramesh, Yifan Hu, Iason Chaimalas, Viraj Mehta, Pier Giuseppe Sessa, Haitham Bou Ammar, and Ilija Bogunovic. Group robust preference optimization in reward-free rlhf. arXiv preprint 2rXiv:2405.20300 2024.
Dushyant Rao, Fereshteh Sadeghi, Leonard Hasenclever, Markus Wulfmeier, Martina Zambelli, Giulia Vezzani, Dhruva Tirumala, Yusuf Aytar, Josh Merel, Nicolas Heess, et al. Learning transferable motor skills with hierarchical latent mixture policies. In International Conference Learning Representations, 2021.
Sharath Chandra Raparthy, Eric Hambro, Robert Kirk, Mikael Henaff, and Roberta Raileanu. Gener- alization to sequential decision making tasks with in-context learning, 2023.
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo. Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.061 000.0....65.... 2022.
Anian Ruoss, Fabio Pardo, Harris Chan, Bonnie Li, Volodymyr Mnih, and Tim Genewein. Lmact: A benchmark for in-context imitation learning with long multimodal demonstrations. arXiv preprins arXiv:2412.01144 2024.
Khaled Saab, Tao Tu, Wei-Hung Weng, Ryutaro Tanno, David Stutz, Ellery Wulezyn, Fan Zhang, Tim Strother, Chunjong Park, Elahe Vedadi, et al. Capabilities of gemini models in medicine. arXiv preprint 4rXiv:2404. 244.:2404..8416,,, 2024.
14

---

Published as a conference paper at ICLR 2026
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with semory-augmenter neural networks. In International conference on machine learning, pp. 1842-1850. PMLR, 2016.
Jurgen Schmidhuber. Evolutionary principles in self self-referential learning. on learning now to learn: The meta seata--mtt-mmtt..hhhe Diploma thesis, Technische Universitat Munchen, Germany, 14 May 1987.
Jürgen Schmidhuber. Curious model-bbilding control systems. In Proc. international joint conference on neural networks, pp. 1468-1463, 1991a.
Jürgen Schmidhuber. A possibility for implementing curiosity and boredom in model-building neural controllers. In Proc. of the international conference on simulation of adaptive behavior: From animals to animats, pp. 222-227, 1991b.
Thomas Schmied, Markus Hofmarcher, Fabian Paischer, Razvan Pascanu, and Sepp Hochreiter. Learning to modulate pre-trained models in rl. Advances in Neural Information Processing Systems, 36:38231-38265, 2023.
Thomas Schmied, Thomas Adler, Vihang Patil, Maximilian Beck, Korbinian Pöppel, Johannes Brandstetter, Günter Klambauer, Razvan Pascanu, and Sepp Hochreiter, A large recurrent action model: xlstm enables fast inference for robotics tasks. arXiv preprint arXiv:2410.22391 2024a.
Thomas Schmied, Fabian Paischer, Vihang Patil, Markus Hofmarcher, Razvan Pascanu, and Sepp Hochreiter. Retrieval-augmeettd decision transformer: External memory for in-context rl. arXiv preprint arXiv:2410.0707,, 2024b.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint 1rXiv:1506.000 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 170.:0707.06347, 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint 2tt:::4400.030000 2024.
Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pp. 4596-4604, PMLR, 2018.
Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement (aaming.(023)) arXiv preprint cs.A1/2303.1 11366, 2023.
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484 489 2016.
Aleksandrs Slivkins et al. Introduction to multi-armed bandits. Foundations and Trends in Machine Learning, 12(1-2):1-288 2019.
Susanne Still and Doina Precup. An information-theoreiic approach to curiosity-driven reinforcement learning. Theory in Biosciences, (31(3):19-114) 2012.
Fahim Tajwar, Yiding Jiang, Abitha Thankaraj, Sumaita Sadia Rahman, Zico Kolter, Jeff Schneider, and Ruslan Salakhutdinov. Training generally curious agent. arXiv preprint 25X::2502.17543 2025.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li. Diego de Las Casas, David Budden, Abbas Abdolmalcki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv: 00090, 2018.
15

---

Published as a conference paper at ICLR 2026
Adaptive Agent Team, Jakob Bauer, Kate Baumli, Satinder Baveja, Feryal M. P. Behbahani, Avishkar Bhoopchand, Nathalie
Michael Chang. Natalie Clay, Adrian Collister, Vibhavari Dasagi. Lucy Gonzalez, Karol Gregor, Edward Hughes, Sheleem Kashem, Maria Loks- Soks-Thompson, Hannah Openshaw, Jack Parker- Holder, Shreyaan Pathak, Nicolas Perez Nieves, Nemanja Rakice- vic, Tim Rocktäschel, Yannick Schroecker, Jakub Sygnowski, Karl Tuyls, Sarah York, Alexander Zacherl, and Lei M. Zhang. Humaan-timescal adaptation in an open-ended task space. In Interna- tional Conference on Machine Learning, 2023a.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Saan-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.1 1805, 2023b.
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05550, 2024a.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifree, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.0888 0882295, 2024b.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, al. Gemma 2: Improving open language models at practical size. arXiv preprint arXiv:2408.001 18, 2024c.
LearnLM Team, Abhinit Modi, Aditya Srikanth Veerubhotla, Aliya Rysbek, Andrea Huber, Brett Will- shire, Brian Veprek, Daniel Gillick, Daniel Kasenberg. Derek Ahmed, et al. Learll: Improving gemini for learning. arXiv preprint 24Xi::2412.164299 2024d.
Dhruva Tirumala, Markus Wulfmeier, Ben Moran, Sandy Huang, Jan Humplik, Guy Lever, Tuomas
Haarnoja, Leonard Hasenclever, Arunkumar Byravan, Nathan Batchelor, Neil sreendra, Kushal Patel, Marlon Gwira, Francesco Nori, Martin Riedmiller, and Nicolas Heess. Learning robot soccer from egocentric vision with deep reinforcement learning. In Pulkit Agrawal, Oliver Kroemer, and Wolfram Burgard (eds.), Proceedings of The 8th Conference on Robot Learning, volume 270 of Proceedings of Machine Learning Research, pp. 165-184. PMLR, 06-09 Nov 2025. URL mlr wwwpppssss22000 htwp:////trrmaaaa....mmm html.
Oriol Vinyals, Igor Babuschkin, Wojciech M Czamecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung. David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. nature, 575(7782)3500-334 2019.
Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z. Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. arXiv preprint arXiv: /6/ .05763, 2016.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- cry, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arxiv 2022.
Yubo Wang, Xiang Yue, and Wenhu Chen. Critique fine-tuning: Learning to critique is more effective than learning to imitate. arXiv preprint arXiv:2501.17703. 2025.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thoughh prompting elicits reasoning in large language models, Advances in neural information processing systems, 25-24724-24837, 2022.
Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. Generating sequences by learning to self-correct. arXiv preprint vrXiv:2211.0055: 2022.
Chuan Wen, Jierui Lin, Trevor Darrell, Dinesh Jayaraman, and Yang Gao. Fighting copycat agents in behavioral cloning from observation histories. Advances in Neural Information Processing Systems, 33:2564-2555 2020.
16

---

Published as a conference paper at ICLR 2026
Ronald J Williams. Simple statistical
algorithms for connectionist reinforcement
learning. Machine learning, 8:229 256, 1992.
Markus Wulfmeier, Michael Bloesch, Nino Vieillard, Arun Ahuja, Jörg Bornschein, Sandy Huang, Artem Sokolov, Matt Barnes, Guillaume Desjardins, Alex Bewley, Sarah Maria Elisabeth Bech- tle, Jost Tobias Springenberg, Nikola Momchev, Olivier Bachem, Matthieu Geist, and Martin Riedmiller. Imitating language via scalable inverse reinforcement learning.
A. Glober- son, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Ad- vances Neural Information Processing Systems, volume 37, pp. 90714-90735. Curran Asso- ciates, Inc., 2024. URL https: / http///poooeeeeeeeeeeeeeee neurips cc/paperr httpl/apppee/ppp//ellllaapaaep wtt.ee//pape//22////5/./ 45036c1664447312211f246 3334433 waw.:4400---app---o-eeeeee
pdf.
Zonghan Yang, Shengjie Wang, Kelin Fu, Wenyang He, Weimin Xiong, Yibo Liu, Yibo Miao, Bofei Gao, Yejie Wang. Yingwei Ma, al. Kimi-dev: Agentless training skill prior for swe-agents. arXiv preprint arXiv:2509.23344 2025.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting language models, arXiv preprint arXiv:2210.03629, 2022.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476-15488, 2022.
Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D Goodman. Quiet-star: Language models can teach themselves to think before speaking. arXiv preprint arXiv:2403.09629, 2024.
Simon Zhai, Hao Bai, Zipeng Lin, Jiayi Pan, Peter Tong. Yifei Zhou, Alane Suhr, Saining Xie, Yann LeCun, Yi Ma, et al. Fine-tuning large vision-language models as decision-making agents via reinforcement learning, Advances in Neural Information Processing Systems, 37:1 17:110935-11097, 2025.
Tianjun Zhang, Aman Madaan, Luyu Gao, Steven Zheng, Swaroop Mishra, Yiming Yang. Niket Tan- don, and Uri Alon. In-context principle learning from mistakes. arXiv preprint arXiv:2402.05403, 2024.
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint a30 /3854, 2023.
17

---

Published as a conference paper at ICLR 2026
CONTENTS
A Environments & Datasets
18
A.I Multi-arm Bandits: BanditBench
18
A.1.1
Baselines
19
A.1.2 SFT Datasets
20
A.2 Contextual Bandits
20
A.3
Tic-tac-toe
Experimental & Implementation Details
23
B.1
Training Evaluation
23
B.2 RLFT
23
B.3 SFT
24
B.4
Exploration Mechanisms
24
Additional Results
26
C.I Failure Modes
26
C.I.1 Greediness
26
C.1.2 Frequency Bias
27
C.1.3
Knowing- Doing Gap
27
C.2
Multi-armed Bandits
28
C33
Contextual Bandits
29
C,4 Other Model Families: Qwen- 2.5 and Llama3
30
C,5 Percentage of invalid actions.
C.6 Prompt Template Variations.
C.7 Effect of Exploration Mechanisms
32
Ablations
32
D.I RLFT in Tic-tac-toe.
................................
32
D.2 Tic-tac-toe: Effect Legal Actions State
32
D.3 Importance of CoT for RLFT
33
D.4 Expert Behavior Cloning vs. Thought Cloning
33
D.5 "Thinking" Time
33
A ENVIRONMENTS & DATASETS
We conduct experiments three sets of environments: multi-armed bandits, contextual bandits, and Tic-tac-toe. For the SFT experiments reported Section 4.5, we generate our own expert datasets, In this section, we provide additional details on our environments and datasets.
A.1 MULTI-ARM BANDITS: BANDITBENCH
MABs (Slivkins et al., 2019; Lattimore & Szepesvári, 2020) are classic problem setting in RL that isolates the ietoorratio-explitattti trade-off. In contrast, commonly used RL environments (Bellemare al., 2013; Tassa al., 2018) often conflate exploration with other RL-specific aspects, such as delayed rewards (Arjona-N) Aedinaa et al., 2019). We rely on the MAB scenarios released in BanditBench (Nie et al., 2024) and also used by (Krishnamurthy et al., 2024). MABs come with number of variable dimensions, including the scenario type (textual description of the task), the type of reward distribution (Gaussian, Bernoulli) and its corresponding noise level (low/mediun/high),
18

---

Published as a conference paper at ICLR 2026
the number of arms (i.e., actions), and the number of interaction steps per episode. Consequently, MABs are good testbed for LLM agents.
We focus on the continuous and button variants released by Nie al. (2024) using their benchmark available on Github. We report results for MAB with k E {5. 10, 20) arms ([.A) k) for three levels of stochasticity (low/medium/high)) In our experiments, for every arm the corresponding reward sampled from a Gaussian distribution 7 ~ (H,, where μ ~ NN, 1) and a fixed scalar {0.1, 1, 3} for the three levels of stochasticity, respectively. For all MAB settings, limit the horizon to 50 interaction steps. Limiting the horizon is necessary to handle the increasing lengths and, consequently, RAM requirements for fine-tuning. While we consider 50 interaction steps sufficient for 5 and 10 arms, it is insufficient for the 20-arms scenario. However, note that the general trends are well observable for the 20 arms scenario. In Figure 10, we show the continuous and button Gaussian MABs with CoT instructions for the agent. Similarly, in Figure 11 show the same instances without CoT instructions.
Continuous MAB (Gaussia))
Button MAB (Gaussian)
You bandit algorithm and interact with 5 arms labeled 0,1,2.3,4.
You bandit algorithm in room with buttons labeled red, green, blue, yellow, orange. Each button is associated with Bernoulli/Gaussian distribution with fixed but unknown mean; the means for the buttons could be different. For either button, when you it, you will get reward that is sampled from the button's 's associated distribution. You have 50 timesteps and, each MUST choose one of the buttons and receive the reward. Your goal is to maximize the total reward over the 50 timesteps.
arm is associated with a BernoulliGGassian distribution with fixed but unknown mean; the means for the arms could be different. For either arm, when you get reward that sampled from the arm's associated distribution. You have 50 time steps and, each time step. you MUST choose one of the reward. Your goal is to maximize the total reward.
you will
step, you
and receive the
[More Instructions]
[More Instructions]
Think step-by-step and output your final answer in the format ACTION=X where is one the arms listed above. IMPORTANT: Provide your (SHORT!) thinking process and your answer ACTION=X
Think step-by steppp and output your final answer in the format ACTION=X where X one of the arms listed above. IMPORTANT: Provide your (SHORT!) thinking process and your answer ACTION=X
So far you have tried/seen:
So far you have tried/scen:
Step=0
Action=1
Reward=0.3
Step=0
Action=green Reward=0.3
Step=1
Action=2
Reward=0.1
Step=1
Action=blue Reward=0.1
Step=2
Action=0
Rew
Step=2
Action=orange Reward=-0.5
Step=3
Action=3
Reward=0.5
Step=3
Action=red Reward=0.5
Step=4
Action=1
Reward=0.24
Step=4
Action=green Reward=0.24
What do you predict next?
What do you predict next?
Figure 10: Illustration of continuous and button Gaussian multi-armed bandits scenarios from BanditBench (Nie et al., 2024) using context representation and with CoT instructions.
A.1.1 BASELINES
We compare against two commonly used baselines for MABs: (Auer, 2002) and a random agent that selects actions uniformly at random. Generally, UCB aims to balance exploitation and exploration. The exploitation value in UCB:
Upper-confidence Bound (UCB)
V peplooi (a)
Σ
((ap=a)"" Ni(a)
for a given action a at timestep represents the empirical mean of the rewards obtained when that action executed, where Na(a) indicates the action frequency. The exploration bonus in UCB:
pexplore (a) og N.(a)
log
(4)
19

---

Published as a conference paper at ICLR 2026
Continuous MAB (Gaussian)
Button MAB (Gaussian)
You are bandit algorithm and interact with 5 arms labeled 0,1,2,3,4, Each arm is associated with distribution with a fixed mean; the mcans for the arms could be different. For either arm, when you it. you will get reward that is sampled from the arm's ' associated distribution. You have 50 time steps and, on each time step, you MUST choose one of arms and receive the reward. Your goal is maximize the total reward.
You are bandit algorithm in room with 5 buttons labeled red, green, blue, yellow, orange. Each button is associated with a Bernolli/Gaussial distribution with fixed but unknown mean; the means for buttons could be different. For cither button, when you press it, you will get a reward that is sampled from the button's 's associated distribution. You have 50 time steps and, each lime step, MUST choose one of the buttons and receive the reward. Your goal reward the 50 time steps.
CammouliiGaassiiie
unknown
maximize the total
[More Instructions]
[More Instructions]
Output ONLY your final answer in the
Output ONLY your final answer in the format ACTION=X,
format ACTION=X.
So you have tried/seen:
you have tried/seen:
Step=0
Action=1
Reward=0.3
Step=0
Action=green Reward=0.3
Step=1
Action=2
Reward=0.1
Step=1
Action=blue Reward=0.1
Step=2
Action=0
Reward=-0.5
Step=2
Action=orange Reward=-0.5
3tep=3
Action=3
Reward=0.5
Step=3
Action=red Roward=0.5
Step=4
1
Reward=0.24
Step=4
Action=green Reward=0.24
What do you predict next?
What do you predict next?
Figure 11: Illustration of continuous and button Gaussian multi-armed bandits scenarios from BanditBench (Nie et al., 2024) using our context representation without CoT instructions.
proportional to how frequently that action has previously been selected, and a is hyperparameter. UCB considered optimal and represents the upper bound for agent performance, whereas the random baseline represents the lower bound. Consequently, we want to emphasize that the purpose this work is not to outperform UCB or to propose better bandit algorithm with LLMs and RLFT. Instead, we aim to better understand the beeision-making abilities of LLMs and the effects of RLFT on exploration.
A.1.2 SFT DATASETS
In our main experiments, we focused on seff-generated CoT rationales and action predictions produced by our fine-tuned agents, which do not require a pre-collected dataset. In contrast, for our SFT experiments reported in Section 4.5, we generated UCB expert datasets. In particular, construct two dataset instances: behavior cloning dataset that only contains expert actions and thought cloning (TC) dataset that incorporates expert actions alongside a thought process for coming up with the respective action (i.e., w/ CoT), For every arm and stochasticity level combination, we construct 32K rollouts from different MABs, which amounts to 1.6M transitions (state, action, reward) per dataset.
To provide the thought process in the TC datasets, we reconstruct the computations of the UCB values conducted by the UCB expert in textual format. The thought process ends with final conclusion of why particular action was selected (i.e., highest UCB value exploratory action). Consequently, the action 2; at step contains the thought process #Cor and the action to execute ag (see Section 3.2). We illustrate the actions contained in the dataset for trajectory at steps 4 and 11 in Figure 12. The BC datasets do not contain the thought process. Instead, they only contain the final predictions made by the model, for example OCTION=yellow low as shown in Figure 12.
A.2 CONTEXTUAL BANDITS
MABs do not emit states. In contrast, CBs emit state representations at every interaction step, making them contextual. Consequently, CBs are interesting to test the abilities of LLMs to make use of the given context when predicting the next action. For our CB experiments, we leverage the MovieLens environment released by Nie al. (2024), semisynthetic bandit task based on the Movieleens
20

---

Published as a conference paper at ICLR 2026
Multi-armed Bandit step
Let's think step-by-step. We calculate the counts and means for every action.
[More Thoughts]
Count for action blue = 1, Mean = (-1.91)/1= -1.91 + sqr(( (git(111) = -0.9
Count for action green 1, Mean (I.41)/ 1.41, UCB 1.41 sqri(1/ =241
Count for action red 1, Mcan (0.45)/1 00.55, UCB = 0.45 + sqri(1 = 1.45
Count for action yellow Mean NaN. UCB NaN
Count for action orange 0, Mean = NaN, UCB NaN
We have not yet selected all actions. Therefore, we select the next one.
ACTION=yellow
Multi-armed Bandit at step 11
Let's think step-by-step. calculate the counts and means for every action.
[More Thoughts]
Count for action blue 1, Mean (-1.91)/ UCB squi( -0.91
Count for action green = 4, Mean = (1.41 + 0.17 + 0.67
UCB = 0.54 + sqri(1 /4))
1.04
Count for action red = Mcan = (0.45 0.78 +
UCB 1.13 +
Count for action yellow Mean = (-1.03)/1==IImm
-1.03 + sqri(1 1)) -0.03
Count for action orange Mcan (-1.2)/1=-11 UCB = -1.2 sqit(1 1)) = -022
We select actions according to the highest UCB value. Therefore, action red is selected.
ACTION=red
Figure 12: Illustration of UCB rationales contained in our SFT expert datasets at two timesteps (4
and 11) in the same trajectory. Both examples show the Thought Cloning dataset instance containing both the produced CoT rationale and the predicted action. The Behavior Cloning instances contain only the final action prediction (in red).
movie recommendation dataset (Harper & Konstan, 2015). In this setting, the agent operates
engine given contextual description of a user (10K users in total) and a list K possible movies. The context representation provides textual description of the user to recommend the movie to. This description includes the user's gender, age, profession, location, and numeric description of the user's preferences for each of the possible movies. As for MABs,
report results for K (5, 10, 20). and limit the horizon to 50 interaction steps. In Figure 13, we provide an example for Movielens CB with 5 actions with our context representation and CoT instructions.
Baselines. Similar to MABs, compare against LinUCB (Chu et al., 2011) and an agent selecting actions uniformly at random. We provide implementation details on our baselines Appendix
A.3 TIC-TAC-TOR
Finally, we use the text-based Tic-tat Tic-tac-toe environment released by Ruoss et al. (2024) (see Figure 14 for an example). Unlike MABs and CBs. Tic-tac-toe toe is stateful environment with proper state transitions (i.e., action predicted at step i affects the state observed at step t + 1). The agent receives scalar rewards of 1, and for winning, drawing, and losing against its opponent, respectively. Episodes last until either of the players wins, draws, or loses. To enable easy extraction of actions from the generated rationales, we represent the action space as a discrete set of 9 actions, corresponding to the grid positions episode are all 9 actions valid. Subsequently, only a subset valid because board positions, We (optionally) provide the set of valid actions at a particular step in textual form in the context given to the agent. Ruoss et al. (2024) demonstrated that frontier models struggle to achieve strong performance in this environment and barely beat random opponent. Consequently, deem it a good target to investigate the efficacy of RLFT.
the 3 3 grid used in Tic-tac-toe ( A ]= 9). However, only the start of an
the currently taken
21

---

Published as a conference paper at ICLR 2026
MovieLens Contextual Bandit
You are an Al movie recommendation assistant for streaming platform powered bandit algorithm that offers wide variety of films from different studios and genres. There are unique movies you recommend, named
(1977), (ontact_(1997)) fargo_(1996). (1996), return of the the_jedi (1983), liar liar (199_)999) When visits the streaming platform, you assess their demographic description to choose a movie to suggest. You aim to match the user with movies they are most likely to watch and enjoy.
[More Instructions]
Think step-by-slep and output your final answer the format ACTION=X where X is one of the arms listed above. IMPORTANT: Provide your (SHORT!) thinking process your answer AC-
TION=X
So far you have tried/scen:
Step=4 This person is a 28-ycar-old man, working as an administrator and living in Santa Clara county, CA. The user has some numeri- cal values that represent their true implicit preference or taste for all movics: [-0.04, 0.02, -0.02, -0.0, 0.02] What do you predict next?
Figure 13: Illustration of contextual Moviel scenario from BanditBench (Nie et al., 2024) using context representation and instructions.
Baselines. Following Ruoss et al. (2024), compare against random agent by default. In addition, we also compare against (MCTS) (Coulom, 2006), and noisy variant of MCTS that selects an action randomly 50% chance and according to MCTS otherwise.
Tic-tac-toe
You agent playing tic-tac-toe. You observe board with 9 entries that looks like this:
100
002
1 indicates that player has placed stone in that square. indicates that player 2 has placed a stone in that square. indicates that no stone has been placed in that square. You play as 1.
There are 9 possible actions: 0. 1, 2, 4, 5, 6, 7, 8. The actions correspond to the following board locations
012
345
678
[More Instructions]
Think step-by-step output your final answer in format ACTION=X where is one arms listed above. IMPORTANT: Provide your (SHORT!) thinking process and your answer ACTION=X
So far you have tried/seen:
Step=0 Statc=00000000 Action=0 Reward=0
Step=1 State=10200000
Action=4 Reward=0
Step=2 Statc= 10:201002
Action=5 Reward=0
What do you predict next?
Figure 14: Illustration of the text-based Tic-tac-toe environment.
22

---

Published as a conference paper at ICLR 2026
B EXPERIMENTAL & IMPLEMENTATION DETAILS
B. TRAINING & EVALUATION
In our experiments, fine-tune Gemma2 models in three model sizes (2B/9B/27B). For all exper- iments, we use the instruction-tunet versions of Gemma2 and leverage the respective instruction pre- and post-fixes. For bandits, we fine-tune all models for total of 30K updates and evaluate after every 10K steps. with an accumulated batch size of 128. Similarly, we fine-tune for 12K updates and evaluate every 4K updates on Tic-tac-toe. We report the mean and 95% confidence intervals over three seeds, suggested by Agarwal et al. (2021).
General. We train all agents with an accumulated batch size of 128. We use learning rate of le le-t, 100 linear warm-up steps followed by cosine decay le-6: To enable memory-efficien fine-tuning of 2B and 9B models, we utilize the AdaFactor optimizer (Shazeer & Stern, 2018). We experiment with LoRA (Hu et al., 2022) for fine-tuning the 9B and 27B models, but found it insufficient for improving the agent's becision-making abilities in our setting. However, LoRA considerably reduces the amount of memory required for RLFT and has been shown to work well for supervised fine-tuning of fecision-making agents (Schmied et al., 2023). Therefore, we deem a promising candidate for RLFT decision-making scenarios. Furthermore, we employ gradient clipping of 1.0. We list hyperparameters in Table 1.
Table 1: Default hyperparameters used in our experiments.
Name
Value
Description
Training
training_stepp
30K or 12K
Number of training steps. Evaluation frequency (in updates). Accumulated batch size. Learning rate scheduler Warmup steps. Maximum learning rate. Optimizer.
eval_freq
10K or 4K
batch_size
128
1r_scheduler
Linear cosine
warmup steps
100
1 to 8
optimizer
AdaFactor
Sequence Length & Generation Budget
context_ length
1792 256
Input context length. Generation budget.
num_tokens
RLFT
rollout_sten
800 or 2048
Rollout steps in in-between updates. Update epochs rollout- Reward penalty for invalid actions. Objective function.
or
rollout-buffer. buffer.
reward_penalty
-5
loss
PPO PPO clipping objective + KI constraint
baseline
MC-baseline or state-value head
Baseline.
envs
16
Number of parallel cnvs. Clipping value. KI. coefficient. Whether reward normalization is used. Sampling temp during rollouts. Sampling temp during evaluation. Sampling top-p.
0.2
8
0.05
reward_nore norm
True
train_temp
1.0
eval_temp
0.0
top_p
1.0
Hardware
accelerator
x H100
Hardware accelerator.
Context Lengths& Generation Budget. For all model sizes and tasks, we use a context length of 1792 for the input context. By default, we set the generation budget to 256 tokens, except for the knowing-doiin gap analyses reported in Section 4.2, which require a larger budget of 2048 tokens, and our generation budget ablation. Consequently, the effective sequence length for fine-tuning is 2048.
Hardware Setup. We train all models on server equipped with 8 x H100 GPUs.
B.2 RLFT
For our RLFT experiments on bandits, we employ the context representation, action factorization, reward shaping terms, and training objectives described in Section 3.2. To extract the target action a
23

---

Published as a conference paper at ICLR 2026
from 22, we make use of stack of regex expressions against the target pattern (i.e., ACTION=X) and consider the last match in the generated tokens as or. In addition to being fairly robust, we found that this approach allows for more flexibility during the RLFT process and led to better outcomes than a more structured approach. We opt for this choice, as preliminary experiments showed it was more robust to varied, sometimes malformed CoT outputs, which as particularly important during early stages of RLFT. This design choice prioritized more reliable signal extraction during training over strict format adherence. However, we acknowledge that depending on the model hand and its associated post-training approach, it makes sense to consider different established extraction approaches, such as boxed().
Across model sizes, we additionally found it essential to introduce a reward shaping term to penalize rationales that contain no valid actions. By default, we use reward penalty of -5 for invalid actions. To ensure that the reward penalty does not overly bias the optimization, we employ reward normalization to the environment rewards. In preliminary experiments. we found that a range of values between [-10, -1] resulted in stable training. In contrast. a high penalty of -100 caused instabilities. In case no valid action is found within the generated tokens, we exccute random action. While this occurs seldomly, this mechanism a useful safeguard to ensure robust interaction with the environment during RLFT (see Table 2 for percentage of random actions). Empirically, we found that the reward shaping term is sufficient for the models to produce exclusively valid actions early on in the training. Consequently, this simple mechanism enables a consistent signal throughout RLFT, and we add it to our recipe.
We fine-tune using the clipping objective introduced by Schulman et al. (2017) with an additional KL constraint to the reference policy Tref- We set 3 = 0.05 and E = 0.2 for all experiments. We make use of the approximated per-token KL divergence instead of computing the full KL. While we found that computing the full KL slightly improves performance, it slows down training considerably. In contrast to Ahmadian al. (2024) and Ramesh et al. (2024), we do not rely on producing multiple rollouts, because it is impractical for the multi-step nature of decision-making tasks. We opt for this choice because obtaining multiple rollouts from the same state, as are required for GRPO (Ramesh et al., 2024) and RLOO (Ahmadian et al., 2024), is difficult in decision-making scenarios for two reasons. First, due to the multi-step nature of different actions corresponding to ht rollouts at every interaction step. This difficult due to the already high associated costs of generating tokens on every step. Incorporating multiple rollouts would therefore drive up cost by a factor of k. Second, in stateful environments, the environment naturally action. Consequently, obtaining multiple rollouts from
tasks, it would be necessary to execute
transitions to the next state after executing the same state would require environment resets. In certain scenarios, such as simulations, resetting the environment is possible, but in many scenarios, this is not feasible. Therefore, we rely on standard MC-baselines to estimate Ande-
For bandit experiments, we maintain a pool of 512 stochastic MABs. For every rollout, we let the agent interact with a subset of 16 bandits for a single episode (50 timesteps). Consequently, every rollout contains 800 transitions. Similarly, for Tic-tac-toe, we maintain 16 parallel environments and collect 2048 rollout steps. We conduct and 2 update epochs over the rollout buffer for bandits and Tic-tac-toe, respectively.
B.3 SFT
For our SFT experiments on MABs, we fine-tune on either the expert action or expert rationales produced by UCB. We employ standard SFT training on the SFT datasets described in Appendix A.1.2 using a cross-entropy objective on the target tokens.
B.4 EXPLORATION MECHANISMS
In Section 8, we compare a variety of classic exploration mechanisms and L.M--speifiic approaches and study their effects on agent performance on Gaussian MABs with 10 arms. Here, we provide a description for each mechanism.
The try-all strategy is inspired by UCB, which incorporates an initial phase in which all untried actions are executed. This is because the UCB values for all untried actions are co. Therefore, for try-all, we incorporate the same exploration phase when performing ICL and RLFT at the beginning oy every episode. To enable fine-tuning on exploration actions, we provide an action rationale template to the model (c.g., Action X has not been tried yet, let's explore it. ACTION=X). While simple. we find that this try-all strategy is effective for lowering regret across all model sizes (see Figure 8). This suggests that the model is able to select appropriate actions if given sufficient information about the consequences of all actions, but struggles to explore actions by itself.
Try-all.
24

---

Published as a conference paper at ICLR 2026
e-greedy, e-greedy is classic exploration mechanism and commonly used in RL algorithms (Mnih al., 2015; Hessel et al., 2018). For our experiments, we use 0.1 both during training and evaluation. Note that e-greedy exploration is performed on a per-action level, not on a per-token level. Consequently, at every step, there is a 10% chance that the executed action is sampled uniformly from the action space, instead of being predicted by the agent. In preliminary experiments, over different e values. However, with e-greedy, sufficient action coverage and low regret in our setting. Overall, = 0.1 enabled stable learning and minor exploration benefits. As for the try-all strategy, we construct a synthetic action rationale template ("Action X has not been tried yet, let's explore it. ACTION=X") to enable RLFT on the random actions.
swept found it difficult strike a good balance between
Context Randomization. Context Randomization is an LLM-specific mechanism designed to introduce randomness in the action predictions by modifying the context representation. At every interaction step, construct mapping from the original action labels to shuffled list of the same action labels. Subsequently, we remap action in the context history according to the constructed mapping. Finally, the predicted action is mapped back to the original action label space and executed environment. Besides introducing randomness, context randomization acts as a control mechanism to ensure that the observed biases do not only stem from biases towards particular ection-tokens (e.g.. blue occurs more often than magenta in the pre-training dataset).
Context Summary. Similar Krishnamurthy et al. (2024) and Nie al. (2024), we evaluate the effects of providing a context summary to the agent. After the context history, we provide the model with a summary of that history that contains the number of times every action has been selected so far, along with their respective mean rewards.
Self-Con Correcteoe.... Inspired by Kumar et al. (2024) and Wang et al. (2025), we employ self-correction to the model's predicted actions. First, we let the model generate its initial rationale and corresponding action prediction. Then we append the generated rationale along with self-correction message (similar to Kumar et al. (2024)) to the input context, and repeat the action generation. Finally, we extract the action from the final response and execute it in the environment. For RLFT, we only fine-tune on the final response, but retain the initial response along with the self-correction message in the context.
Self- Coff-Consisteny,. Instead of generating a single answer, self-consistency (Wang et al., 2022) relies on generating multiple responses. Subsequently, self-consistency employs majority voting mechanism to determine the final response. For our experiments in Figure 8, we report results for ceff-consistency with 16 generated responses. Instead of majority voting, we experimented with sampling from the respective response distribution or random mechanisms.
Exploration Bonus. Finally, we evaluate a reward shaping mechanism in the form of an exploration bonus. In particular, we give an exploration bonus of +1 if the agent selects an action not yet tried within the respective episode. While simple, find that the exploration bonus effectively narrows the gap the UCB expert. This highlights the importance of reward shaping for fine-tuning LLMs in maiiing scenarios.
Button MAB
1.0
Ratmitts
[Instructions]
20
9B
S=0 A=bluc R=1
S=1 A=blue R=1
S=2 A=blue R=
0.4
S=3 A=green R=0
S=4 A=red R=2
0.2
What do you predict next?
0.0
blue green red
yallow
orange
purple
pard
betsee
(a) Example: History
(b) Example: Probability Mass
Figure 15: Illustration of action probabilities leading to greediness behavior. Models exhibit overly high action probabilities in the presence of rewards, potentially resulting in repeatedly selecting sub-optimal actions.
25

---

Published as a conference paper at ICLR 2026
C ADDITIONAL RESULTS
C.I FAILURE MODES
C.I.1 GREEDINESS
Greediness is characterized by the LLM overly favoring the best- persf-reforming action among a small set of actions seen so far. We define action coverage Ct at step as the fraction of available actions that have been selected at least once, C2 (oEA:N((u)20) with Ni(a) representing the number of [A]
times action has been selected until t.
Action probabilities. The suboptimal action coverage reported Section 4.2 is caused by the model overly favoring high-reward actions (i.e.. overly high action probabilities), In Figure C.I.I, we provide an illustration of the action probabilities for given input history. Across model sizes, Gemma2 exhibits overly high action probabilities in the presence of reward, which results in repeatedly selecting potentially suboptimal action.
0.8
96
2B
Action Space Coverage
0.6
270
Tue
0.4
Truc
False
Falie
Action
0.2
0.2
0.0
Step
Step
(a) Action Coverage: 10 arms
(b) Action Coverage: 20 arms
Figure 16: Illustration of greediness for the numbers scenario.
Greediness on Continuous MABs. We repeat the analyses conducted in Section 4.2 using numbers instead of buttons as the possible actions. Indeed, we find that the same trends hold. Without CoT, the performance remains low. For Gemma2 27B, we observe an increase the action coverage to almost 90% for the 10 arms scenario, and to 60% for the 20 arms scenario.
Post RLFT. In line with Figure 17a, we present the post-RLFT action coverage on the 20 arms scenario in Figure 17b. Similar to the effects on the 10 arms scenario, we observe that RLFT improves the action coverage by 13%.
Action Space Coverage
Model
0.3
28
28, @10K
CoT
True
0.1
2
4
45
10
20
30
40
50
Step
Figure 17: Effect of RLFT on greediness for 2B on 20 arms (medium noise).
Throughout experiments, limit the horizon 50 interaction steps (see Figure 3) because practical reasons. In our setup, the agent has to produce a decision rationale every step because of the multi-step nature of decision- decision-making Every step involves both a prefilling stage (encoding instructions, history) and a generation stage for producing the rationale/action, which is costly over multi-turn interactions. Given the flattening of the action coverage curves in Figure 3, we do not expect that the agent would continue to explore a longer horizon. To verify this
100-step Horizon.
26

---

Published as a conference paper at ICLR 2026
hypothesis, conduct additional ablation in which the smallest and largest Gemma2 models the 10 arms scenario with an extended horizon of 100 interaction steps (see Figure 18). Indeed, we find that the action coverage remains flat over the longer horizon, which indicates that the agent does not continue to explore due to greedy action selection strategies.
0.7
Action Coverngl
0.6
0.5
0.4
0.3
270
CuT
0.1
Step
Figure 18: Greediness over longer horizon of 100 interaction steps. We compare Gemma2 2B and 27B with and without CoT. The agents do not continue to explore over longer horizons.
C.1.2 FREQUENCY BIAS
Frequency bias is characterized by repeatedly selecting the most frequently occurring actions in the context, even when the dominant action gives a low reward. To measure frequency bias, we first construct variety of interaction histories (occurred during environment interaction) containing between 2 and 10 transitions. This interaction history is collected using random policy. Given an initial interaction history. we repeat the last action the history, which we also refer to as the target action, between 0 and 100 times. Finally, we report the entropy all actions, (0) = EaEA aa (a
T) log (a r)- To achieve this, we conduct a separate forward pass for every possible action in the action space and report the respective log probabilities. We repeat the same procedure for different interaction histories and target actions (see Figure 4a and c). For the 10 arms scenario, every interaction history therefore results in 1000 (10 arms 100 repetitions passes. We repeat this procedure for the target actions reported in Figure (i.e., 'green', 'red', 'blue', 'orange', 'black' buttons) using 5 interaction histories per action, accumulating to total of 25K model forward passes (1000 5 5) per figure.
the target action) forward
To quantify frequency bias, we categorize the resulting actions frequent action, greedy, other if they are neither frequent nor greedy. Subsequently, compute the frequent Fy, greedy Fg and other F. fractions as reported in Figure 4:
Nr(aa), N
(rg( N
F. = Eaeallaraat Nr(a) with N = Σ Nr(a). (5)
F
F==
N
aEA
Note that there can be an overlap between greedy and frequent actions. these (rare) cases, the greedy action category is dominant, i.c., we categorize the action as greedy even if it would also be the frequent action. This implies that the actions classified as frequent in Figure 4, are always suboptimal/bad compared the respective greedy action. Consequently, high F' indicates that the model prefers the most frequent action even when observing a better action in the context.
Post RLFT. In Section 4.3, we observed that RLFT counteracts frequency bias. In addition to frequency buckets reported in Figure 19a, we plot frequency against action entropy post RLFT in Figure 19b. Compared to Figure 4a, we observe that after RLFT the models maintain higher action entropy for longer. Only at high repetition frequencies does the action entropy decrease severely. Consequently, RLFT counteracts frequency bias, but does not completely alleviate it.
C.1.3 KNOWING-DOING GAP
The knowing- gap has been observed by Paglieri et al. (2024) and Ruoss al. (2024). It states that models possess knowledge about task or consequences of their behavior (i.e., they know what to do), but cannot materialize this knowledge when acting (i.c., they are incapable of doing). However, in these works, it was difficult to characterize the knowing-doing gap empirically, due to the complexity of their environments. In contrast, our simple MAB setting is well-suited to quantify their observations precisely. We illustrate the knowing-doing gap empirically in Figure 5. To this
27

---

Published as a conference paper at ICLR 2026
1.0
Strategy
1.6
Frequeg 0.667
Correltions
Fraction per window
frequent
0.8
greedy
14
other
0.6
1.0
0.4
Artion
0.2
0.6
0.4
0.0
45-55
90-100
103
Action Repetitions
(a) Action Strategies
(b) Corrleations
Figure 19: Effect of RLFT on frequency bias for 2B (10 arms, medium noise).
end, first task Gemma2 27B to produce the UCB algorithm and to compute the relevant quantities required according to UCB ("knowing"). This involves counting how often every action was selected, computing the mean rewards for every action, and computing the final UCB values. After producing the quantities, the model is tasked to act according to them (i.c., "doing"). In Figure 26, we present an example of the final instructions given to the model along with response produced by Gemma2 27B.
To evaluate performance empirically, we let Gemma2 27B interact with the environment (64 parallel instances) for 50 timesteps. We extend the token generation budget to 2048 tokens per step to accommodate the additional required computations. Every produced action contains both the CoT rationale 2CoT and the final selected action a. We first extract the computed UCB values from the produced rationale 2COT. To achieve this, we task Gemma2 27B to enclose the computed values by sucb_values> values> and in the environment. For this experiment, we use Gemma2 27B, because we found that 2B and 9B struggled with computing the relevant UCB quantities and with enclosing them appropriately under the desired blocks.
creb_values>> blocks. Then we extract the selected action a and execute it
Quantifying "Knowing". To quantify "knowing", we compare the UCB values computed by the model and extracted from 2Cof against the real UCB values. To this end, we recompute the real UCB values for every action at every time step given the observed history. We consider the rationale as correct if the arm with the highest UCB values matches. We opt for this choice rather than checking for exact equality because we observed that the model struggles with exact calculations for complex operations. This is expected because the necessary computations involve logarithms and square roots of floating-poin values. While tool use (e.g., calculator) could mitigate this issue, observed that Gemma2 27B gets the quantities approximately right, resulting valid rationales. Thus, the fraction of correct rationales is F. N ( given a classifier g.
Quantifying "Doing". To quantify "doing". we categorize the generated actions optimal action the model selects the action with the highest UCB value, as greedy if it selects the action with the highest UCB value among the set of actions tried so far, and as other if the action neither optimal nor greedy. It is possible that the greedy action the optimal action. However, in this case, the action is considered optimal instead of greedy. Subsequently, we compute the percentages of greedy/optimall/thee actions (c.g., Fg 100). We find that the model clearly knows how to solve the task, with 89% of rationales being correct (see Figure 5).
To ensure that our findings represent a fundamental model bias and not a prompt artifact, we investigated a variety of prompting strategies in preliminary experiments to encourage the LLM to strictly adhere to the UCB algorithm. For example, this involved strong language, including explicit instructions to prioritize untried actions and to prefer actions with high/infinite UCB values. Moreover, we provided hints to the model that actions with infinite UCB are strictly better than other actions and should always be preferred. Across prompting strategies, our findings remained similar.
C.2 MULTI-ARMED BANDITS
In Figure 6, we report the cumulative regrets across model sizes and arms for a medium noise (a = 1.0) scenario. In addition, we repeat the same experiment in the low-noise (o = 0.1) and the high-noise 3.0 setting in Figure 20. For both noise levels, observe similar trends to those
28

---

Published as a conference paper at ICLR 2026
for the medium noise setting. In particular, observe that LLMs clearly outperform the random baseline, and RLFT lowers the cumulative regret for Gemma2 2B across all arm scenarios.
We note that in Figure 6, we observed that 9B, ICL outperforms 9B, RLFT. One possible hypothesis for this behavior is that the higher short-term regret may be a direct consequence of RLFT. RLFT encourages the agent to explore more by rewarding reasoning processes that lead to exploratory actions. In this specific scenario, the increased exploration can be costly in the short term over the 50-step horizon, even though it may be essential for long-term performance.
RIFT
2B,
9R,
9B,
27A,
Random
Arms:s
Arms=10 10
Arms=:20
200
200
200
Cumuletive Ragret
150
150
150
1.00
100
100
0-
6
10
20
30
40
50
30
20
30
40
50
10
20
30
40
Step
Step
Step
(a) High Noise (a 3)
Armsmn
Armsn10
Armsm 20
60
Cumulative Regree
50
50
50
30
30
20
20
20
0
01
0
10
20
30
40
50
10
20
50
50
6
10
20
50
40
Step
Step
Stop
(b) Low Noise (a 0.1)
Figure 20: Main Comparison on Gaussian MABs button scenario the (a) high a and (a) low a = 0.1 noise settings. We compare cumulative regrets (lower is better) of classic baselines against ICL and RLFT performances for Gemma2 2/9/27B for 5, 10, and 20 arms.
C.3 CONTEXTUAL BANDITS
We repeat the same fine-tuning experiment for the contextual MovieLens bandits described in Section A.2. In Figure 21, we report the cumulative regrets attained by Gemma2B across different model sizes and for 5, 10, and 20 arms. Furthermore, we compare against a LinearUCB and random baseline. Overall, we observe similar performance improvements for RLFT on CBs as on MABs. While the ICL performances barely attain the same performance as a random agent, RLFT fine-tuned Gemma2 2B performs similarly to UCB.
28, ICL
28, RLFT
98, ICL
278, ICL
UnUCB
Random
Arms
Arms
10
Arms 20
150
150
150
Avg. Cumulative Regret
125
125
125
100
100
75
75
75
50
50
50
0
0
Figure 21: Main Comparison on Gaussian MovieLens CBs for (a) 5, (b) 10, and (c) 20 arms. We
compare classic baselines against ICL and RLFT performances for Gemma2 2/9/27B.
29

---

Published as a conference paper at ICLR 2026
C.4 OTHER MODEL FAMILIES: QWEN-2.5 2.5 AND LLAMA3
To ensure that our results are not specific to the Gemma2 model family, we repeat the greediness analysis reported in Figure with the widely-used instruction- iisoonttnedd/eee Llama3 (Dubey et al., 2024) roportd and Qwen-2.5 (Qwen et al., 2025) models. We report the action coverages for 10 arms and 20 arms along with the action coverages plotted against cumulative regret for 10 arms in Figure 22 and 23 for Qwen-2.5 and Llama3, respectively.
For Llama3, we evaluate the latest checkpoints for three available model sizes: 3B, 8B, and 70B. Specifically, we use the instruction-uu 3.3 for 70B. Similarly, we report scores for the instruction-tuned variants of Qwen-2.5 in 4 model sizes: 3B, 7B, 14B, and 32B. We evaluate all checkpoints both with and without CoT instructions using their respective instruction templates.
variants of Llama 3.2 for 3B, Llama 3.1 for 8B, and Llama
0.8
0.40
Parmettar
Adton space Covemde
0.7
Sction Spacce Conerox
0.6
0.30
Mosel
2B
0.5
0.25
0.0
140
75
0.20
320
32
0.3
0.15
0.4
0.10
0.2
0.1
0.05
0
10
20
30
40
50
10
20
30
40
50
70
60
50
40
30
Step
Step
Aug. Cumulative Regret
(a) Action Coverage: 10 arms
(b) Action Coverage: 20 arms
(c) Coverage vs. Regret: 10 arms
Illustration of Greediness. We show action coverage for Qwen-2.5 3B/7B/14B/32B instruct models with and without CoT for (a) 10 and (b) 20 arms over 50 interaction steps. The agents favor the best-performing action among the set of selected actions, leading to stagnating action coverage, despite the benefits of larger models and COT. In (c), plot cumulative regret against action coverage. The general trends are similar to Gemma2 and Llama3, suggesting that the limitations persist across model families.
Figure 22:
Generally, we observe that Llama3 and Qwen-2.5 models exhibit similar trends in terms of action coverage and regret as Gemma2 models (see Figures 22 and 23). For both model families, CoT considerably improves action coverage and therefore regret. One exception Qwen-2.5 3B w/o CoT in the 10 arm scenario, which always first selects 5 actions before committing to a particular action. Overall, the smaller-scale Llama3 models (3B, 8B) tend to achieve lower action coverages compared to the Gemma2 and Qwen-2.5 models of similar sizes, both in the 10 and 20-arms scenario. Llama3 70B achieves the highest action coverage across all model sizes, potentially highlighting the benefits of the larger model size. However, in the 20 arms scenario, we still observe a considerable gap to full action coverage for both model families, with Qwen-2.5 and Llama3 selecting 35% and 65% of all actions at maximum, respectively.
We note that for Qwen-2.5, the larger models do not always outperform the smaller models. For example, in 7B attains 108 higher action coverage than 14B in the 10 arms scenario. This is an interesting observation of non- mon-monoooi scaling, where performance on certain tasks does not strictly increase with model size. We believe that this may stem from differences in the pre-training corpora, token budgets, or alignment techniques applied to specific model checkpoints. We leave a deeper analysis of this phenomenon for future work. Across model sizes, it is, however, apparent that the action coverage flattens out over the 50-step horizon. This indicates that all model sizes suffer from the same fundamental bias.
C.5 PERCENTAGE OF INVALID ACTIONS.
In the case that no valid action is extracted from the decision rationale, a random action is executed. This mechanism serves as safeguard to ensure that the environment interaction continues without interruptions. To verify that the safeguard does not bias our experimental results, we report the percentage of invalid and consequently random actions performed across the 50-step horizon for all model families in Table 2.
For the experiments reported in Figures 3, 22, and 23, we recorded whether the executed action was valid or invalid. To compute the percentages reported in Table 2, we first calculate the fraction of random actions per episode (i.e., over the 50-step horizon) and subsequently average over the 64 bandits to get more robust statistical estimate. Indeed, we find that only small percentage of all
30

---

Published as a conference paper at ICLR 2026
space Covembo
8 0.0
Mosel
0.5
Paraneters
Action Coverse
Action Space
9.0
0.4
0.4
0.2
0.
0.2
0
10
20
30
40
50
10
20
40
50
90
80
70
80
30
40
30
Step
Step
Aug. Cumulative Regret
(a) Action Coverage: 10 arms
(b) Action Coverage: 20 arms
(c) Coverage vs. Regret: 10 arms
Figure 23: Illustration of Greediness. We show action coverage for Llama3 3B/8B/70B instruct models with and without CoT for (a) 10 and (b) 20 arms over 50 interaction steps. The agents favor the best-performing action among the set of selected actions, leading to stagnating action coverage, despite the benefits of larger models and COT. In (c), plot cumulative regret against action coverage. We use Llama 3.2, 3.1, and 3.3 for 3B, 8B, and 70B, respectively. Note that we do not report results for Llama3 70B w/o CoT, as we did not get the model to produce valid actions without CoT instructions. The general trends are similar to Gemma2 and Qwen-2.5, suggesting that the limitations persist across model families.
actions is invalid. For example, 2% of the 50 interaction steps indicates that, on average, only single action is invalid per episode. Consequently, the safeguard does not introduce any meaningful bias into our results. Interestingly, we found that for many trials, the very first interaction in the trial resulted an invalid action, with the model often producing ACTION=random as its final output. We assume that this is due to the empty interaction history in the first step.
Table 2:
Percentages of invalid/random actions performed over the 50-step horizon on the Gaussian button bandit with 10 arms for all model families.
Model
Chain of Thought
% of invalid/random actions
Gemma2 2B
2.00
Gemma2 2B
3.19
Gemma2 9B
2.00
Gemma2 9B
2.00
Gemma2 27B
2.00
Gemma2 27B
0.00
Llama3 3B
X
2.00
Llama3 3B
1.50
I.aaa3 8B
2.00
Llama3 8B
1.50
Llama3 70B
0.00
Owen 2.5 3B
X
0.00
Owen 2.5 3B
2.00
Qwen 2.5 8B
X
0.00
Owen 2.5
0,00
Owen 2.5 14B
X
2.00
Owen 2.5 14B
0.00
Qwen 2.5 32B
X
0.25
Qwen 2.5 32B
0.00
C.6 PROMPT TEMPLATE VARIATIONS.
To better understand the influence of the position of the interaction history final performance, conduct an additional ablation study using Gemma2 2B with different prompt templates. In our setup, the input context contains 3 sections, as illustrated in Figure 1: (1) task instructions, (2) output instruc- tions, and (3) the interaction history. Accordingly, we mix up the order of these sections, resulting in 3 context variations: task cete--oouuttiisooryyyy
history-task-outpll
Across prompt templates, we observe similar patterns with action coverages flattening out over the 50-step horizon (see Figure 24). Nevertheless, some differences exist among the templates.
31

---

Published as a conference paper at ICLR 2026
For example, history-task-output performs worst overall without CoT. Overall, positioning the interaction history last (our default setting) performed best in the CoT setting.
0.40
Coverage
0.30
0.25
Orteat Unkeeststnntoory hastayyyypuuu
0.0
0:15
0:10
Bille
20
30
40
50
Step
Illustration of the effect of different prompt templates. We compare Gemma2 2B with and without CoT using three different context orderings: task-output-bistory, While differences exist, the variations exhibit
Figure 24:
dask-history-outphe history-task-outge similar patterns with action coverages flattening out after a handful of steps.
C.7 EFFECT OF EXPLORATION MECHANISMS
For RLFT, relied solely on the exploration properties for CoT reasoning. Therefore, in Section 4.4 studied the effects of classic exploration mechanisms and LLM-specific strategies to encourage exploration. In particular, we compare: (1) try-all actions initially similar to UCB, (2) e-greedy, (3) context randomization, (4) context summary similar to Krishnamurthy et al. (2024) and Nie et al. (2024), (5) self-correction similar to Kumar et al. (2024), (6) self ceff-consistency (Wang et al., 2022), and (7) exploration bonus.
In Figure 8, we observe that the mechanisms result in varied effects action coverage First, we find that the simple try-all strategy, which reduces the need for additional exploration by trying all actions, results in the biggest performance improvements. This suggests that given sufficient information about the hub-)ooptimalit of actions, LLMs are better able to select actions accordingly, underscoring their exploration shortcomings. Second, simple exploration bonus (+1 reward for untried actions during RLFT) significantly increases exploration (50% + 70%) and lowers regret towards the expert compared to regular RI.T.. This highlights the importance of reward shaping for fine-tuning L.MSs to elucidate a desired behavior.
D ABLATIONS
D.1 RLFT TIC-TAC-TOE.
To investigate the efficacy of RLFT in stateful environments, evaluate Tic-tac-toe from Ruoss al. (2024), in which frontier models struggle achieve strong performance (see Appendix B for training details). We fine-tune against three opponents: a random agent, Monte Carlo Tree Search (MCTS) (Coulom, 2006), and noisy MCTS (50% of actions selected at random). We find that RLFT significantly enhances the win-rate of Gemma2 2B against all opponents compared to ICL (see Figure 9a). Against the random agent, RLFT elevates the average return from 0.15 (i.e., winning 15% of games) 0.75. Notably, the agent even manages draw against the optimal MCTS baseline (-0.95 + 0.0), underscoring the effectiveness of RLFT for decision-make makiinn.. However, for high performance, it essential to provide the legal actions the context (see Figure 25).
D.2 TIC-TAC-TOE: EFFECT OF LEGAL ACTIONS IN STATE
By default, provided the legal actions available at the current turn within the input context to the agent. We found this design choice to be essential for effective fine-tuning compared to training without legal actions (see Figure 9b). Without legal actions in the context, the average return drops from 0.75 (w/ legal actions) to 0.45. This suggests that the LLM fails at identifying the appropriate actions among the set of all possible actions when not given legal actions at the current state. In contrast, when provided with sufficient information (i.e., legal actions), the LLM is able to select actions appropriately (similar Section 4.4). Providing the legal actions in the agent's
32

---

Published as a conference paper at ICLR 2026
context alleviates the need to explore/identify invalid actions. Consequently, this shortcoming further highlights the need for principled exploration strategies for LLMs in meciiinn--makin scenarios.
with legal actions
0.7
-
w/ legal actions
0.8
0.4
0.2
0.0
D100
0.2
0.8
Environment Stops
1e5
Figure 25: Effect of removing legal actions from the context in Tic-tac-toe.
D.3 IMPORTANCE OF COT FOR RLFT
CoT reasoning is critical for ICL performance (see Figure 3), but the question remains how CoT influences RLFT. Therefore, we run RLFT on Gemma2 2B on the 10 arms Gaussian MAB both with and without COT (see Figure 9b, RLFT). Indeed, without COT. RLFT barely attains the performance of ICL w/ COT. This highlights the function of CoT as vital exploration and rationalization mechanism for mecision-making. For our results without CoT reported Figure 9b, we remove the CoT instructions given to our agents. Instead, we instruct the agents not to perform any reasoning steps and to only produce the action to execute in the environment a. We limit the token generation budget G' to 16 to avoid the model ignoring the instructions. Furthermore, this considerably speeds up training due to faster rollout times and shorter context lengths.
D.4 EXPERT BEHAVIOR CLONING VS. THOUGHT CLONING
prevalent approach sequence models for decision-making is behavior cloning (BC) (Pomerleau, 1 1988; Reed et al., 2022; Brohan et al., 2022; 2023), which relies on expert datasets. Consequently, we construct two UCB expert datasets comprising 32K rollouts either w/o CoT (behavior cloning) or w/ CoT (thought cloning), as described in Appendix A.1. Notably, both SFT variants successfully mimic the expert, achieving comparable regret to the UCB expert (see Figure 9b, SFT). This result underscores the efficacy of training on expert data in decision-making scenarios when available, echoing recent findings in reasoning tasks (Muennighoff et al., 2025). While BC and TC attain similar performance levels on the simplistic MABs, we anticipate that TC is advantageous in more complex decision-maker scenarios as found by Hu & Clune (2023).
D.5 "THINKING" TIME
We investigate the effect of giving the agent more time "think" in Figure 9c. To achieve this, we vary the maximal number of tokens that the agent can generate per action G E {16, 256, 512). By default, we G to 256. Indeed, we observe that the performance improves consistently with more thinking tokens. Decreasing G to 16 or 64 results in poor performance, because the agent is unable to rationalize its decisions within the restricted generation budget. This is similar to the performance without CoT, but in contrast, the agent is instructed to produce the reasoning process. Over the course of RLFT, the agents learn to produce short rationales &CoT, including the action due to our reward shaping mechanism (see Section 3.2). However, the produced short rationales unhelpful to improving agent performance.
In contrast, doubling from 256 to 512 results in a considerable performance increase to the level of Gemma2 9B with RLFT (see Figure 6). We observe an increase in the average sequence length over the course of the RLFT process. This suggests that the agent learns to effectively leverage the additional "thinking time" and reflects recent observations in mathematical reasoning (Guo et al., 2025). However, the increased performance comes with additional training cost due to the multistep nature of recision-making scenarios. In fact, we observed that rollout generation can make up the majority of the training time required by the RLFT process. This is because the agent has to produce more tokens at every environment interaction step. For example, for our default horizon of 50 timesteps and generation budget of 500, the agent produces 25K tokens (at maximum).
33

---

Published as a conference paper at ICLR 2026
Instructions for Gemma2 27B UCB agent
[Instructions]
Your task to act according to the Upper- Confidence Bound (UCB) algorithm. First, write down the UCB algorithm. Then compute the relevant UCB values for every button (approximate if necessary). Finally, select your action according to the computed quantities.
[More Instructions]
So you tried/seen:
Step=0
Action=blue Reward=1.06
Step=1
Action=blue Reward=1.82
Step=2
Action=green Reward=1.0
Step=3
Action=green
Rewardz-0.26
Step=4
Action=blue
Reward=-0.58
Step=5
Action=blue Reward=-0.34
6tep=6
Action=green Reward=1.19
Step=7
Action=green Reward=2.21
Step=8 Action=green Reward=0.07
Step=9
Action=green Reward=1.45
Step=10
Action=green Reward=:0.11
What you predict next?
Generated response by Gemma2 27B
Here's how I'd approach this as a UCB bandit algorithm: UCB Algorithm (Short Version)
Initialization: For each button, set count (number of times pressed) to and an estimated reward average to O.
Iteration:
For each button, calculate the UCB value: UCB average_rewae +
Ju [topa____stepe) connt
Choose the button with the highest UCB value.
Update the button's 's count average reward based on the observed reward.
UCB Calculations (Approximate)
Let's denote the buttons by their colors. Here's breakdown after 10 steps:
blue:
Count = 4
Average Reward (1.06 1.82 0.58 0434)/4=0.4
UCB x +
2-In(11)
green:
Count 7
Average Reward (1.0 -0.26 1.19 1111:+2222++000044444000000 0.92
UCB = 0.92 + / 2-1m(11) 1.47
Other buttons: Count = 0, Average Reward = 0, UCB = / 2210(11) oo (We'll explore
these later)
Action Selection: The button with the highest UCB value is green.
Therefore, ACTION=green
Explanation: The UCB algorithm balances exploration (trying buttons we haven't seen much) and exploitation (choosing buttons that seem to have high rewards).
Green has higher average reward than blue, but blue has been explored less.
The UCB formula takes this into account, giving higher value to green because its average reward is more reliable due to more observations.
The other buttons have infinite UCB values because we haven't it tried them yet, so they are highly prioritized for exploration. However, green's 's current UCB is high enough to warrant another try.
Figure 26: Illustration of the knowing-doing gap. (a) Instructions for the agent, which is prompted to act like UCB algorithm. (b) The response generated by Gemma2 27B with greedy decoding (temperature=O). The LLM "knows" the UCB algorithm and computes the relevant quantities approximately correctly, but acts erroncously by selecting the next action greedily.
34


Reviews:
[{"review_id": "YO14zX0Cr1", "reviewer": "Reviewer_ytZh", "summary": "This paper empirically investigates why LLMs perform sub-optimally in decision making settings and identifies three primary failure modes across different model families: greediness, frequency bias, and the knowing-doing gap. Through experiments on multi-armed bandits, contextual bandits, and a text-based Tic-tac-toe environment, this work provides an empirical characterization of these issues across model scales (Gemma2 2B/9B/27B, Llama3 3B/8B/70B and Qwen-2.5 3B/7B/14B/32B). The authors have designed a suite of experiments exploring whether Reinforcement Learning Fine Tuning (RLFT) on self-generated Chain-of-Thought (CoT) rationales can help mitigate these issues, showing that RLFT increases exploration, reduces the greediness and frequency bias in decision making and also narrows the knowing-doing gap with an increased reasoning token budget. Additionally, classical and LLM-specific exploration mechanisms (like $\\epsilon$-greedy, self-correction, self-consistency) are used to further improve the post-RLFT performance. Ablation studies highlight the roles of CoT traces in RLFT, expert data for behavior cloning vs thought cloning in SFT, and reasoning token budgets for inference time computational efficiency. Overall, this paper offers an insightful analysis into the decision-making behavior of LLMs and the impact of RLFT in addressing some of the current drawbacks of decision making with different language models.", "questions": "- Beyond empirical results, do authors have any intuitive understanding or insights into why RLFT improves exploration (maybe changes in CoT token distributions or entropy)?\n\n- How would the observations in this work transfer to other recent reinforcement-based fine-tuning frameworks (GRPO, DPO, or self-critique fine-tuning)?\n\n- Do the RLFT improvements transfer across unseen bandit configurations or to different reasoning prompts?\n\n- In both Fig 21 (a) and (b), a larger model does not always outperform a smaller one in terms of action space coverage. Is there any ablation experiment or empirical observation explaining this behavior? \n\n- The current evaluation for the knowing-doing gap is not extended to the Llama3 70B model, which is shown to have less severe issues from greediness and frequency bias. It would help to include empirical evidence of the relationship, if any, between the knowing-doing gap and model size.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper identifies and rigorously quantifies three key failure modes (greediness, frequency bias, knowing-doing gap) in LLMs for decision making applications, specifically focused on bandit-style decision making with instruction-tuned large language models. \n\n- The authors use RLFT to analyze how reinforcement feedback influences reasoning and exploration, with appropriate baselines and ablations across different LLM model families.  \n\n- The paper is overall well-written with thorough description of relevant implementation details and baselines, improving reproducibility.", "weaknesses": "- RLFT itself is a straightforward application of PPO-style RLHF training to CoT outputs, therefore the methodological novelty lies mainly in the empirical analysis and not in a new algorithm.\n- The environments are relatively simple (bandits, Tic-tac-toe) and while appropriate for controlled study, stronger claims about decision-making or agentic abilities would benefit from evaluation on richer, stateful domains.\n- The 50-step interaction limit may underestimate the difficulty of exploration and long-term adaptation with LLMs.\n- Unclear scalability to larger LLMs: Most analyses are conducted on $\\leq$27B models and the implications for larger model scales (e.g. Llama3 70B) are briefly discussed but not thoroughly tested.\n- RLFT results seem sensitive to design choices for reward shaping and action space exploration strategies. Ablation experiments in this paper demonstrate the effects qualitatively (in most likely a single trial) but they are not statistically quantified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vrQ15AtPns", "reviewer": "Reviewer_Sto4", "summary": "This paper evaluates LLMs in decision-making scenario, focusing on three failure modes: greediness, frequency bias and the knowing-doing gap. The authors illustrate these failure modes on a multi-armed bandit environment, and further show that RL finetuning can help mitigate them. \n\nThe paper is generally well-written and easy to follow. However, the novelty of the work is not clear to me. LLMs have been well studied in decision-making settings, and the failure points studied here (lack of exploration, knowing-doing gap) have been noted in previous works (see weaknesses). Using reinforcement learning finetuning to improve the abilities of LLMs is not novel either. I think this work would be better suited as a workshop contribution.", "questions": "- Figure 3: what do the cross markers correspond to?\n- Figure 4: the plots (a) and (c) are unclear to me: as I understand the y-axis (action entropy) refers to the entropy of the distribution over predicted actions. But then why do the plots show markers for each individual action? If they correspond to the action that is repeated N times across the x-axis, why would we expect the results to look different when only the action name is changing?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "- The multi-armed bandit environment allows to illustrate the three failure modes in a controlled setting", "weaknesses": "Major:\n\n- W1: The results presented in this work are not particularly novel.\n    1. LLMs have been studied as agents in decision-making environment in numerous works (see for example [1] for a recent review). As the authors themselves note in the introduction, previous works have observed failure modes such as lack of exploration (cf l.41) and the knowing-doing gap (cf l.43-44). The main results of the paper (Section 4.2) illustrate these failure modes on a multi-armed bandit setting but do not provide further insight beyond observing these failure modes in a controlled environment. \n    2. The benefits of RL finetuning with verifiable rewards have been established in previous works, for example [2-5]. The authors mention [4-5] saying that in contrast their approach is “specialized for decision-making scenarios” (l.163), even though I’d argue that [5] is precisely focused on decision-making scenarios already.  \n    3. Results in the Section 4.5 merely illustrate well-studied methods like RL finetuning, behavior cloning/distillation, and test-time scaling. \n- W2: The related works do not cover previous works on RL finetuning and LLM-as-an-agent, which makes the novelty compared to previous works unclear.\n- W3: In method l.175, “If no valid action is found, a random action is executed”. Weak LLMs might run into invalid actions often, which introduces an unwanted form of epsilon-greedy exploration where epsilon is the rate at which the LLM doesn’t output a valid action. Given the amount of exploration done by the LLM is precisely what is being studied here, it would be preferable to filter out these invalid actions instances instead.\n- W4: In Figure 6, results for 20 arms show that RLFT leads to higher regret for the 9B model, but this is not discussed in the text. Also the plots with lower noise in appendix C.2 are missing the 9B-RLFT model.\n- W5: In Section 4.2, the conclusion that “this suggest that frequency bias is an artifact of supervised pre-training” (l.326) is misleading: results for the 27B model shows (as the authors note l.324) that a strong model can escape the frequency bias, even though it is trained with supervised pre-training.\n- W6: In Section 4.2, the illustration provided Figure 24 suggests that the reason the LLM doesn’t pick previously unseen actions might be due to an imprecision in the prompt phrasing. Does the effect remain when specifying that the model should strictly according to the UCB algorithm? Does the result depend on most actions not having been seen previously like in Figure 24?\n\nMinor:\n\n- l.246: “LLMs model” is redundant\n- Why is the result on Tictactoe in the ablation section? This just seems to me like it is looking into the effectiveness of RL finetuning as in Section 4.3.\n\n\n[1] Cao, Yuji, et al. \"Survey on large language model-enhanced reinforcement learning: Concept, taxonomy, and methods.\" IEEE Transactions on Neural Networks and Learning Systems (2024).\n\n[2] Lambert, Nathan, et al. \"Tulu 3: Pushing frontiers in open language model post-training.\" arXiv preprint arXiv:2411.15124 (2024).\n\n[3] Luong, Trung Quoc, et al. \"Reft: Reasoning with reinforced fine-tuning.\" arXiv preprint arXiv:2401.08967 (2024).\n\n[4] Guo, Daya, et al. \"Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.\" arXiv preprint arXiv:2501.12948 (2025).\n\n[5] Zhai, Simon, et al. \"Fine-tuning large vision-language models as decision-making agents via reinforcement learning.\" Advances in neural information processing systems 37 (2024): 110935-110971.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p6Cdo7NDDC", "reviewer": "Reviewer_HhsF", "summary": "This paper presents a systematic investigation into the causes of suboptimal performance in Large Language Model (LLM) agents on decision-making tasks. The authors convincingly identify and quantify three prevalent failure modes: greediness, frequency bias, and the knowing-doing gap. To address these issues, the paper proposes a method of Reinforcement Learning Fine-Tuning (RLFT) on the model's self-generated Chains of Thought (CoT). Through meticulously designed experiments in environments such as multi-armed bandits, contextual bandits, and text-based Tic-Tac-Toe, the paper demonstrates that RLFT effectively enhances the model's exploration capabilities and bridges the knowing-doing gap, thereby improving the overall quality of the LLM's decisions. The study is further enriched by an in-depth analysis of various classical and LLM-specific exploration mechanisms, along with insightful ablation studies.", "questions": "- Regarding Figure 1, how significant is the position of the \"interaction history\" within the prompt on the LLM's performance? A robustness and sensitivity analysis of the prompt template would be highly beneficial for the community.\n- The paper shows that explicit exploration rewards are highly effective. As a follow-up, have the authors considered integrating intrinsic exploration motives (e.g., using an entropy bonus or curiosity-driven rewards) directly into the RLFT objective? A comparison with intrinsic exploration mechanisms would further enhance the paper's comprehensiveness.\n- On line 192, you mention: \"Instead of exploiting multiple rollouts, as used by Ahmadian et al. (2024) and Ramesh et al. (2024), we compute rewards-to-go.\" Could you please elaborate on the reasoning for this choice, perhaps in the appendix?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- A key strength of this paper is its thorough and systematic diagnosis of the problem before proposing a solution. The clear definition and empirical quantification of greediness, frequency bias, and the knowing-doing gap provide a valuable conceptual framework and a strong motivation for the proposed method.\n- The authors evaluate their approach across multiple model scales (Gemma2 2B, 9B, 27B) and environments of varying complexity. The inclusion of strong baselines like UCB and extensive ablation studies (e.g., on the importance of CoT, \"thinking time\") significantly strengthens the credibility of the findings. The analysis is also extended to other model families (Llama3, Qwen2.5) in the appendix, further demonstrating the generality of the results.\n- The proposed RLFT on self-generated CoT is a logical and effective approach. By directly acting on the model's internal reasoning process, it offers a more fundamental way to improve external behavior compared to fine-tuning on actions alone. The experimental results clearly demonstrate the method's positive impact on increasing action coverage and reducing cumulative regret, directly linking the solution to the identified problems.", "weaknesses": "- The paper effectively demonstrates that \"external\" exploration scaffolds (e.g., the try-all strategy or exploration bonuses) improve performance. This suggests that the model may still lack a fundamental, intrinsic drive for exploration. A valuable direction for future work could be to explore integrating intrinsic motivation directly into the RLFT objective. For instance, adding an entropy bonus on the action distribution or novelty-based rewards (e.g., based on state visitation counts) might encourage the model to develop a more generalizable and autonomous exploration policy.\n- The paper employs a regex-based method for action extraction and applies a fixed penalty (-5) for invalid outputs. While practical, the robustness of this approach could be further discussed. A comparison with increasingly standard structured output formats (e.g., JSON objects) would be a valuable addition. Furthermore, a sensitivity analysis on the magnitude of the penalty for invalid actions would offer valuable insights into its impact on the learning dynamics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "25Elo2NPWX", "reviewer": "Reviewer_8Kgn", "summary": "This paper investigates the decision-making capabilities of LLMs under uncertainty using multi-armed and contextual bandit settings. Through analyses including action coverage tracking and comparison against optimal algorithms like UCB, the authors identify three primary failure modes: greediness, frequency bias, and the knowing-doing gap.\n\nTo mitigate these, the paper fine-tunes Gemma2 models (2B, 9B, 27B) using PPO trained with CoT reasoning traces on environment rewards. The PPO method significantly improves performance over the baseline ICL methods. The study further shows that augmenting PPO with classic exploration techniques (like try-all, ε-greedy, and exploration bonuses) yields additional gains, with methods like adding an exploration bonus or an initial \"try-all\" phase proving most effective among the tested PPO variations, and provided ablation results on CoT, thinking time, and additional experiments on Tic-tac-toe.", "questions": "Have we explored prompting methods to ensure LLM strictly follows UCB algorithm? \n\nFor the 𝜖-greedy method: is it done at token level or at action level? If it’s the latter, how is it implemented?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper provides a valuable diagnostic analysis of why LLMs fail in decision-making tasks, systematically identifying and quantifying specific issues like greediness, frequency bias, and the knowing-doing gap using bandit environments. This clear characterization of common failure patterns is a useful contribution.\nThe work demonstrates that PPO fine-tuning with CoT traces effectively improves LLM decision-making across various tasks (MAB, CB, Tic-tac-toe) and model scales, outperforming baseline in-context learning without requiring expert data. Furthermore, the study evaluates integrating simple exploration strategies during PPO, showing that techniques implemented via prompt/context modifications, such as initializing with a \"try-all\" phase, yield additional performance gains. Overall the paper offers novel insights into practical methods for enhancing exploration of PPO-tuned LLM agents under related decision making settings.", "weaknesses": "Limited Generalizability: The paper claims to analyze general LLM agent behavior but bases its core analysis almost entirely on simple bandit tasks (MAB and CB). The setting is somewhat narrow (also, SOTA/commercial models were not evaluated), making it hard to generalize the findings to more complex agentic situations. The knowing-doing gap analysis is a particularly interesting topic, but the analysis is tied to the UCB structure in bandits; it would be interesting to explore a more generic analysis that can characterize the knowing-doing gap across a variety of decision-making tasks (possibly with some form of optimal algorithm as part of the LLM’s internal knowledge) and provide deeper insights (e.g., could this be a value alignment problem where the LLM’s default option is not using an optimal algorithm?).\n\nMethod Performance and Novelty: While the proposed PPO fine-tuning method improves LLM performance over the ICL baseline, it doesn't match the results of SFT using expert UCB data. Given that SFT data is relatively easy to synthesize for MAB tasks, the proposed method is less useful in this context. The PPO solution itself, while possibly the first published work on MAB/CB, is a well-established method, making it less novel. The authors are encouraged to expand the method to more decision-making settings (e.g., more text-games as in https://arxiv.org/abs/2502.17543) and showcase more solid results on improving exploration there.\n\nFinally, the Tic-tac-toe game feels less like an ablation to me. It would be interesting to have full coverage of the analysis there.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)