When Greedy Wins: Emergent Exploitation Bias in Meta-Bandit LLM Training

## Abstract

Abstract While Large Language Models (LLMs) hold promise to become autonomous agents, they often explore suboptimally in sequential decision-making. Recent work has sought to enhance this capability via supervised fine-tuning (SFT) or reinforcement learning (RL), improving regret on the classic multi-armed bandit task.
However, it remains unclear how these learning methods shape exploration strategies and how well they generalize.
We investigate both paradigms by training LLMs with SFT on expert trajectories and RL with a range of tailored reward signals including a strategic, regret-shaped reward to reduce variance, and an algorithmic reward that enables oracle imitation.
The resulting agents outperform pre-trained models and achieve performance comparable to Upper Confidence Bound (UCB) and Thompson Sampling, with robust generalization to 6 × \times longer horizons and across bandit families.
Behavioral analysis reveals that gains often stem from more sophisticated but greedier exploitation:
RL/SFT agents are more prone to early catastrophic failure than pre-trained models, prematurely abandoning exploration.
Furthermore, agents trained to imitate UCB learn to outperform their teacher by adopting more exploitative variants.
Our findings clarify when each training paradigm is preferable and advocate tailored reward design and evaluation beyond average regret to promote robust exploratory behavior.

## 1 Introduction

A fundamental challenge in sequential decision-making problems lies in the exploration-exploitation trade-off, where an agent must balance exploiting known good actions with exploring new ones to discover potentially better options.
The multi-armed bandit (MAB) problem serves as a classic, formalized testbed for studying this critical behavior.
Despite their sophisticated capabilities, Large Language Models (LLMs) often struggle here, defaulting to short-sighted, greedy behavior that over-exploits known rewards at the expense of exploration (Krishnamurthy et al., 2024; Schmied et al., 2025).
While certain prompting configurations can elicit better performance from frontier models like GPT-4, this inherent suboptimal bias remains a significant hurdle for most models.

To address this, two primary training paradigms have emerged for shaping LLM exploration behavior: Supervised Fine-Tuning (SFT) and RL. SFT teaches the LLM to mimic the behavior of an optimal exploration algorithm, such as Upper Confidence Bound (UCB), by training on trajectories of expert demonstrations. In contrast, RL enables the model to learn an effective policy directly from environmental rewards.
When trained to solve bandit instances that differ from those they encountered during training, LLMs effectively become meta-bandit agents, acquiring meta-policy capable of exploring novel environments (Kveton et al., 2020).
Prior works suggest that both methods can improve exploration capabilities in LLMs on in-distribution tasks, with SFT showing more consistent results (Nie et al., 2024; Schmied et al., 2025).
However, a deeper understanding of how these training methods shape an agent’s strategy is lacking.
It is unclear whether the policies induced by SFT and RL differ mechanistically.
More critically, how do these policies generalize to longer horizons and out-of-distribution environments?

In this work, we train LLMs to perform MAB tasks using both SFT on expert trajectories and RL with a spectrum of task-specific reward signals. We evaluate the performance of learned policies on a range of MAB environments, under length generalization and cross-distribution transfer (e.g., Gaussian to Bernoulli). In addition to the standard stochastic reward of bandits, we propose two additional reward signals: a strategic reward based on the notion of regret to reduce training variance, and an algorithmic reward, which incentivizes imitation learning of an oracle policy like UCB via RL.
We find that both SFT and RL improve the base model’s performance on MAB tasks in achieving lower regret and higher rewards, achieving comparable performance to theoretical optimal baselines like UCB and Thompson Sampling.
For RL, the strategic reward improves training efficiency in high-variance environments, while the algorithmic reward consistently outperforms other learned policies due to the ease of credit assignment.
Moreover, RL policies yield more robust generalization across different bandit families compared to SFT.
The policies also exhibit strong generalization on 6$\times$ longer (compared to training) and out-of-distribution environments.

While achieving lower regret is the canonical measure of success in MAB, classical literature cautions that relying solely on this aggregate statistic can obscure important characteristics of the agent’s behavior (Lattimore & Szepesvári, 2020). An agent might achieve a superior average performance with a high-risk policy prone to catastrophic failure, a nuance that the expected outcome can overlook. This prompts a deeper question: does a lower average regret achieved by the LLM policies indicate the acquisition of a robust exploration strategy?

To answer this question, we analyze the agents’ action patterns and compare them to pre-trained models and baselines like UCB and Greedy policies. We utilize surrogate statistics such as suffix failure rate, which is highly suggestive of the long-term prospects of the agent (Krishnamurthy et al., 2024). We find that the agents’ impressive improvements in performance are linked to learning more sophisticated forms of exploitative behavior.
For instance, agents trained via RL to imitate an optimal UCB policy often outperform their teacher by implementing variants of UCB that can prematurely stop exploring an action after unsatisfactory short-term rewards.
This suggests that while the training process maximizes average performance with reasonable generalization, it incentivizes short-term reward seeking that can be counterproductive in the long run.
The suitability of these learned policies ultimately depends on whether an application prioritizes long-term robustness over immediate returns, or average performance over worst-case scenarios.

In summary, we present a unified study of how SFT and RL shape LLM exploration in MAB, treating trained models as meta-bandit agents. We introduce two principled reward designs—strategic (regret-shaped) rewards that stabilize learning in high-variance settings and algorithmic rewards that enable efficient RL-based imitation of oracle policies, which both improve over baselines from prior work (Schmied et al., 2025), with algorithmic rewards yielding the most consistent gains. Evaluations demonstrate robust generalization to 6$\times$ longer horizons and across bandit families, with RL policies transferring more reliably than SFT. Beyond aggregate regret, our behavioral analysis reveals mechanistic differences: learned policies often implement exploitative strategies that boost average returns but can sacrifice long-term robustness.
(^1^11We will release our code and data in [https://github.com/sanxing-chen/meta-bandit-llm](https://github.com/sanxing-chen/meta-bandit-llm).)

## 2 Related Work

The multi-armed bandit problem, despite being a classical abstraction, embodies the fundamental exploration-exploitation trade-off central to sequential decision-making and has wide real-world applications (Bouneffouf et al., 2020; Bouneffouf & Feraud, 2025). As LLMs are increasingly deployed in interactive settings, the MAB problem has become a key testbed for evaluating their ability to incrementally gather information and improve over time, a paradigm known as In-Context Reinforcement Learning (ICRL) (Moeini et al., 2025).

Bandit problems have long been used to evaluate the generalizable ICRL capabilities of sequential models like RNNs and Transformers (Duan et al., 2016; Laskin et al., 2023; Lee et al., 2023). In the LLM era, initial benchmarks found that pre-trained models can learn to explore simple MAB problems in-context (Binz & Schulz, 2022; Wu et al., 2024; Coda-Forno et al., 2023; Park et al., 2025). However, they exhibit unsatisfactory exploratory behavior in complex environments without careful prompt engineering (Krishnamurthy et al., 2024; Monea et al., 2024). Subsequent work has sought to address this through activation steering (Rahn et al., 2024) and fine-tuning (Tajwar et al., 2025). Nie et al. (2024) uses supervised fine-tuning (SFT) on expert trajectories to improve performance, demonstrating successful generalization to different reward distributions within the same bandit class. More recently, Schmied et al. (2025) applies reinforcement learning to train LLMs for bandit tasks, showing positive but weaker in-distribution results compared to SFT.

Our work provides a systematic comparison of these two learning paradigms. We demonstrate that RL-trained agents, while matching SFT performance in-distribution, generalize more effectively to out-of-distribution environments. More importantly, we move beyond simple performance comparisons to conduct a behavioral analysis that uncovers subtle but critical failure modes in how LLMs learn to explore, highlighting previously unaddressed challenges.

## 3 Methodology

Figure: Figure 1: An instruction provided to the LLM agent for the MAB task.

A MAB problem $\mathcal{B}=(\mathcal{A},R)$ is defined as a set of arms $\mathcal{A}=\{1,\ldots,K\}$, where each arm $i\in\mathcal{A}$ is associated with a reward distribution $R_{i}$ and mean $\mu_{i}$. The goal of the agent is to maximize the expected cumulative reward $\mathbb{E}[\sum_{t=1}^{T}r_{t}]$ over $T$ trials. During training, the agent learns from bandit instances sampled from an unknown task distribution $\mathcal{D}$. We can evaluate the learning agent’s performance in-distribution by sampling bandit instances from $\mathcal{D}$ or out-of-distribution (OOD) on instances from a different distribution $\mathcal{D}^{\prime}$.
In training an agent to solve various bandit instances from a task distribution, we are effectively searching for a meta-bandit policy (Kveton et al., 2020), which is a reinforcement learning problem.

### 3.1 Reinforcement Learning of Meta-Bandit LLM Agents

At each bandit turn $t$, the LLM agent takes as input the interaction history consisting of past actions and rewards in the observation $o_{t}$, and generates a sequence of tokens $s_{t}$ which contains the action of the next arm to pull $a_{t}$. The environment then returns the stochastic reward $r_{t}\sim R_{a_{t}}$. The interaction history is then updated with $o_{t+1}=f(o_{t},a_{t},r_{t})$, where $f$ can be a simple concatenation or, in our case, a summarizer that extracts sufficient statistics as shown in [Figure 1](https://arxiv.org/html/2509.24923v1#S3.F1).
The process is repeated for $T$ turns for each episode. As the agent learns over a history to build its belief about the environment (e.g., distribution family and variance), this process forms a Partially Observable Markov Decision Process (POMDP).
It can be trained using on-policy RL to maximize episodic return and thus learns an amortized exploration strategy over histories.

Unlike traditional RL policies that directly select actions, LLM agents operate in the token space. This implementation converts the problem into a two-level hierarchical MDP (Hauskrecht et al., 2013; Xue et al., 2025), where a high-level policy operates at the turn level to select a local policy that generates the entire response $s_{t}$ and receives the external reward $r_{t}$.
The low-level policy operates at the token level to implement the selected local policy. The probability of generating the token $s_{t,j}$ at position $j$ is given by:
$\pi_{\theta}(s_{t,j}|o_{t},s_{t,<j})$
where $s_{t,<j}$ is the sequence of tokens generated in turn $t$ up to position $j-1$. At turn $t$, the token index $j$ ranges from $J_{t,\text{start}}=|o_{t}|+1$ to $J_{t,\text{end}}=|o_{t}|+|s_{t}|$. We pass $r_{t}$ as the reward signal to the low-level policy at $J_{t,\text{end}}$, while there is no reward signal for intermediate tokens.

To learn $\pi_{\theta}$, we adopt PPO (Schulman et al., 2017) and compute token-level advantages with a dual-($\gamma$, $\lambda$) Generalized Advantage Estimator (Schulman et al., 2016). We use separate discount factors and trace-decay coefficients for intra-turn and inter-turn steps, denoted $\gamma_{\text{intra}}$, $\gamma_{\text{inter}}$ and $\lambda_{\text{intra}}$, $\lambda_{\text{inter}}$, respectively.
For simplicity, we define the token-level state at step $j$ as $h_{t,j}=(o_{t},s_{t,<j})$. The one-step temporal difference (TD) error for each generated token index $j$ is:

$$ $\delta_{t,j}=\begin{cases}\gamma_{\text{intra}}V(h_{t,j+1})-V(h_{t,j})&\text{if }J_{t,\text{start}}\leq j<J_{t,\text{end}}\\ r_{t}+\gamma_{\text{inter}}V(o_{t+1})-V(h_{t,j})&\text{if }j=J_{t,\text{end}}\end{cases}$ (1) $$

For the final token at index $J_{t,\text{end}}$, the error incorporates the external reward $r_{t}$ and bootstraps from the value of the next turn’s initial state, $V(o_{t+1})$, using the inter-turn discount factor $\gamma_{\text{inter}}$.
In practice, since we can only optimize over a truncated horizon for this infinite-horizon problem, we infer the value of one more turn, $V(o_{T+1})$ for the last turn $T$.

The GAE advantage for token index $j$ now accumulates TD errors over all subsequent generated-token positions across the entire episode.
Let $\kappa(\tau)$ denote the starting generated-token index in turn $\tau$ as seen from $(t,j)$: $\kappa(\tau)=\begin{cases}j&\text{if }\tau=t\\
J_{\tau,\text{start}}&\text{if }\tau>t\end{cases}$. Define the step-weighting product from $(t,j)$ to $(\tau,k)$ as:

$$ $P(t,j,\tau,k)=\left[\prod_{p=t}^{\tau-1}\left(\lambda_{\text{inter}}\gamma_{\text{inter}}\right)\left(\prod_{u=\kappa(p)}^{J_{p,\text{end}}-1}\lambda_{\text{intra}}\gamma_{\text{intra}}\right)\right]\left(\prod_{u=\kappa(\tau)}^{k-1}\lambda_{\text{intra}}\gamma_{\text{intra}}\right).$ $$

The token-level GAE advantage for $(t,j)$ is then:

$$ $\hat{A}_{t,j}=\sum_{\tau=t}^{T}\sum_{k=\kappa(\tau)}^{J_{\tau,\text{end}}}P(t,j,\tau,k)\;\delta_{\tau,k}.$ $$

With token-level advantages defined only for generated tokens, the clipped PPO objective is:

$$ $\mathcal{L}^{\text{PPO}}(\theta)=\hat{\mathbb{E}}_{t,j}\left[\min\!\left(r_{t,j}(\theta)\,\hat{A}_{t,j},\;\text{clip}\!\left(r_{t,j}(\theta),1-\epsilon,1+\epsilon\right)\,\hat{A}_{t,j}\right)\right],$ $$

where the per-token probability ratio is
$r_{t,j}(\theta)=\frac{\pi_{\theta}(s_{t,j}\mid h_{t,j})}{\pi_{\theta_{\text{old}}}(s_{t,j}\mid h_{t,j})}.$
Here $\theta_{\text{old}}$ is the reference policy parameter at the previous iteration.
This objective trains the policy at token level using the two-scale GAE that respects intra-turn and inter-turn dynamics. We intentionally omit the KL-divergence term, which is often employed in PPO as we find it to be unnecessary for our setting without a learned reward model.

### 3.2 Reward Design

As described above, the meta-bandit agent relies solely on the past interaction history $o_{t}$ to generate the next action. The interaction history $o_{t}$ is a summary of the past actions and rewards, which is tied to the stochastic bandit rewards $r_{t}$ and cannot be changed.
We can however opt for a different reward signal for the PPO optimization in [Equation 1](https://arxiv.org/html/2509.24923v1#S3.E1).
The original bandit rewards (RL-OG), although a natural choice of reward signal for PPO optimization, contribute to credit assignment difficulty and learning inefficiency due to their intrinsic stochasticity.

On the other hand, we can more accurately measure the optimality of an action based on the notion of immediate regret. At each time step, the immediate regret is defined as the difference between the expected reward of the optimal arm and the expected reward of the arm selected by the agent. $\Delta_{t}=\mu^{*}-\mu_{A_{t}}.$ We define the strategic reward (RL-STR) based on the immediate regret of the agent’s action:

$$ $\tilde{r}_{t}=1-\frac{\Delta_{t}}{\Delta_{\text{max}}}=\frac{\mu_{A_{t}}-\min_{i}\mu_{i}}{\mu^{*}-\min_{i}\mu_{i}}\in[0,1].$ $$

This reward signal directly optimizes an action’s utility, which simplifies credit assignment. Using the realized regret as the reward is a form of baseline subtraction (Kveton et al., 2020). We further use the (pseudo) regret, which is analogous to introducing a control variate. While this approach reduces variance, it theoretically does not alter the optimal policy to which the agent converges.

A third approach moves beyond extrinsic environmental rewards, instead using a reward function optimized by an expert oracle (Ciosek, 2022). While this reward function often needs to be learned via inverse reinforcement learning (Abbeel & Ng, 2004), we can bypass this by deriving it directly from well-established optimal algorithms for MAB. We select the Upper Confidence Bound (UCB) algorithm as the oracle policy, as its deterministic and distribution-agnostic properties provide a consistent and unambiguous learning signal.
We therefore define the algorithmic reward (RL-ALG) as a binary signal $r_{t}=1$ if the agent’s action matches the oracle’s decision $\pi_{\mathrm{oracle}}(o_{t})$, and $r_{t}=0$ otherwise.
Because the UCB oracle is a reactive algorithm, this myopic reward is sufficient for on-policy learning and sidesteps the need for return-based credit assignment.
This imitation learning setup leaves the agent free to discover its own internal algorithm for processing the interaction history to match the oracle’s choice at each step, without any supervision on the reasoning process.

Figure: Figure 2: An example of the UCB calculations for the state in [Figure 1](https://arxiv.org/html/2509.24923v1#S3.F1), used in SFT.

On top of these task specific rewards, we also consider a reward shaping term that encourages the LLM agent to generate valid responses. Specifically, we set reward to zero if our parser cannot extract a valid action and rationale from the response. For the stochastic reward setting (RL-OG),
because the unbounded reward is sometimes negative, we subtract $0.5$ from the reward as the penalty for invalid responses.

### 3.3 Supervised Learning

We also consider a supervised fine-tuning (SFT) baseline, where the LLM agent is trained on observation-response pairs. The response includes synthetic CoT demonstrations to explicitly calculate UCB values and the UCB action ([Figure 2](https://arxiv.org/html/2509.24923v1#S3.F2)).
Here, both the rationales and the actions are directly supervised. Since the states embodied by the observation are sampled from the UCB policy, the learning process is off-policy.

## 4 Experimental Setup

#### Language Model Configuration.

We use Qwen 2.5 3B and 7B Instruct (Qwen et al., 2024) as the base model for fine-tuning. The observation at each time step consists of a natural language instruction of the MAB task and the interaction history presented as a summary of the number of pulls and average reward for each arm ([Figure 1](https://arxiv.org/html/2509.24923v1#S3.F1)).
We use this sufficient statistics to summarize the interaction history, which has been shown to be more effective than using a cumulative context, e.g., a raw list of actions and rewards (Krishnamurthy et al., 2024).
In the instruction, the agent is asked to think step-by-step using chain-of-thought reasoning, which is critical for eliciting the sequential decision-making ability of LLMs (Yao et al., 2023).

#### RL Configuration.

We build our RL training code on top of the VeRL framework (Sheng et al., 2024).
At each training iteration, we first sample a batch of 64 random environments from the training task distribution $\mathcal{D}$. From each environment, we collect a rollout of length $T=50$, resulting in a batch of $64\times 50$ transitions $(o_{t},s_{t},r_{t})$. This batch is then used to compute policy gradients and perform PPO updates.
We sample another set of environments for the next batch of rollouts.

#### Supervised Fine-Tuning (SFT).

For the SFT experiments, we train the model for 6 epochs on 32k transitions sampled from UCB rollouts in environments drawn from the training task distribution $\mathcal{D}$.
Transitions are uniformly sampled across the length of training horizon $T$.
We synthesize a templated response for each transition by demonstrating the step-by-step UCB value calculation for each arm and the comparison process which leads to the final action.
We perform full fine-tuning minimizing the cross-entropy loss between the predicted and ground-truth responses.

**Table 1: Generic families of $k$-armed MAB environments and some specific parameterizations used in our study. Asterisk indicates the training task distributions.**
| Family | Reward Dist. | Mean Dist. | Example Instantiation |
| --- | --- | --- | --- |
| Gaussian$k$_Var$\sigma^{2}$_MeanN$m$ | $r\sim\mathcal{N}(u_{i},\sigma^{2})$ | $u\sim\mathcal{N}(m,\sigma_{u}^{2})$ | Gaussian5_Var1_MeanN0^∗ |
| Gaussian$k$_Var$\sigma^{2}$_MeanU | $r\sim\mathcal{N}(u_{i},\sigma^{2})$ | $u\sim\mathcal{U}(0,1)$ | Gaussian5_Var1_MeanU |
| Bernoulli$k$_Uniform | $r\sim\mathcal{B}(u_{i})$ | $u\sim\mathcal{U}(0,1)$ | Bernoulli5_Uniform^∗ |
| Bernoulli$k$_Delta$\Delta$ | $r\sim\mathcal{B}(u_{i})$ | $u_{i^{*}}=p$,<br>$u_{i}=p-\Delta,\forall i\neq i^{*}$ | Bernoulli5_Delta0.2 |

#### Bandit Environments.

We consider MAB environments listed in [Table 1](https://arxiv.org/html/2509.24923v1#S4.T1). The environments can be generally grouped into two families: Gaussian and Bernoulli, based on the reward distribution.
The Gaussian environments have continuous reward distributions, while the Bernoulli environments have discrete binary reward distributions.
We select one from each family (i.e., Bernoulli5_Uniform and Gaussian5_Var1_MeanN0) as the two training task distributions, under which we train two set of policies to test out-of-distribution generalization.

#### Baselines.

We compare learning agents against the following standard baselines:
1) Upper Confidence Bound (UCB) (Auer et al., 2002) selects the action $A_{t}=\operatorname*{arg\,max}_{a}\left(Q_{t}(a)+C\times\sqrt{\frac{\log(t)}{N_{t}(a)}}\right)$, where $Q_{t}(a)$ is the mean reward of action $a$ up to time $t$, $N_{t}(a)$ is the number of times action $a$ has been selected up to time $t$, and $C$ is a constant.
2) Thompson Sampling (TS) (Thompson, 1933) is a Bayesian method that samples from the posterior reward distribution of each action and selects the one with the highest sample. We use Beta and Gaussian posteriors for Bernoulli and Gaussian rewards, respectively.
3) $\epsilon$-Greedy chooses a random action with probability $\epsilon$ and the action with the highest current mean reward otherwise. While simple, its constant exploration leads to linear regret. The purely exploitative Greedy policy is a special case where $\epsilon=0$.

For UCB, which sometimes serves as a teacher, we use an exploration constant of $C=0.5$, which performs well for both training environments. For $\epsilon$-Greedy, we use a standard $\epsilon=0.1$; while likely suboptimal, it provides a consistent anchor for comparison. The direct performance comparison between learned agents and baselines is *not* the central focus of this study. The one exception is the evaluation of our imitation learning agents against their UCB teacher.

#### Evaluation.

We evaluate the policy over 64 episodes, each with a maximum of 300 steps. We use a fixed set of 64 different seeds for initialization of evaluation environments and baseline policies. To compare the policies and test for length generalization, we follow standard practice to report cumulative regret at $t\in\{50,300\}$.
MAB instances, even when they are drawn from the same distribution, can be quite different in terms of challenge level. Conventional empirical evaluation aggregates from exccessive number of rollouts (e.g., ten of thousands) and long horizons, which although provides a more stable estimate is prohibitively costly for LLM inference.
We therefore utilize distribution plots to visualize this variation in regret and focus on the typical performance in comparison.
To provide a more comprehensive evaluation, we supplement this with two additional metrics: time-averaged reward and best arm frequency, which measure the proportion of times the optimal arm is selected.

## 5 Experimental Results

Figure: Figure 3: Comparison of LLM policies against baselines on cumulative regret at 300 steps. The first row shows the results of 7B models trained on Gaussian5_Var1_MeanN0, and the second row shows the results of 7B models trained in Bernoulli5_Uniform. Evaluation is performed both in- (first column) and out-of-distribution (other columns). The boxplots depict the median, interquartile range (IQR) from the 25^th to the 75^th percentile, and whiskers extending to 1.5$\times$IQR. 3B model results can be found in [Appendix B](https://arxiv.org/html/2509.24923v1#A2).
Refer to caption: x1.png

### 5.1 LLM Agents are Meta-Bandit Learners

As shown in [Figure 3](https://arxiv.org/html/2509.24923v1#S5.F3), across both training setups, RL-trained policies improve upon pre-trained models to be comparable with classical baselines (UCB, TS, $\epsilon$-Greedy), achieving lower cumulative regret and length generalization to a 6$\times$ longer horizon (50 $\rightarrow$ 300).
The time-averaged reward (AvgReward) and best arm frequency (BestArmFreq) in [Table 2](https://arxiv.org/html/2509.24923v1#S5.T2) indicate steady performance gains over time.
Learning agents remain competitive under OOD evaluation, exhibiting non-trivial cross-distribution transfer from Gaussian-trained policies to Bernoulli environments and vice versa.
However, RL agents that trained on environmental feedback (i.e., RL-OG and RL-STG) show weaker cross-distribution generalization, with greater variability in worst-case performance.

#### Learning from UCB signals.

Overall, policies optimized using teacher UCB signals, whether through reinforcement learning (RL-ALG) or supervised fine-tuning (SFT), consistently outperform policies trained solely on the task reward signal (RL-OG). This underscores the difficulty of training RL LLM policies for long-horizon exploration where credit assignment is challenging. The imitation policies match or achieve lower cumulative regret compared to the teacher UCB policy in all evaluation environments, revealing a seemingly exciting result: policies trained on expert-generated data can ultimately outperform the very expert policy that produced the data.

#### Improving training with strategic rewards.

To learn RL policies from environmental feedback, while theoretically aligned with the original bandit reward signal (RL-OG), optimizing for strategic rewards (RL-STG) empirically improves the performance of the policy in the Gaussian training setup, despite certain instabilities observed in OOD evaluation.
As the variance of the reward distribution decreases, RL-STG becomes equivalent to RL-OG, which explains why their performance is more closely matched when training in the Bernoulli5_Uniform environment.

#### SFT vs RL for imitation.

SFT with UCB expert demonstrations can achieve similar regret to UCB in-domain, consistent with prior work (Schmied et al., 2025). We additionally find SFT policies to be surprisingly competitive out-of-distribution.
Part of the reason is that UCB is a distribution-agnostic policy–the same calculation can be applied to different reward distributions as long as the LLM follows the arithmetic operations.
This generalization is however fragile. SFT policies can overfit to the training distribution and cause a degradation of basic arithmetic capability. Together, these factors lead to higher variability and worst-case regret in OOD evaluation.
For example, in [Figure 3](https://arxiv.org/html/2509.24923v1#S5.F3), the SFT policy trained in the Bernoulli5_Uniform environment exhibits unsatisfactory worst-case performance in Gaussian5_Var1_MeanN0, while the RL-ALG policy based on UCB reward signal remains robust.

Figure: Figure 4: Training performance of RL-STG policy on GaussianK_Var1_MeanN0 with 3B and 7B models. We additionally include RL-ALG of 3B model (5 arms) for comparison.
Refer to caption: x2.png

#### Small models struggle to learn without teachers.

[Figure 4](https://arxiv.org/html/2509.24923v1#S5.F4) illustrates the training dynamics of the RL-STG policy on the GaussianK_Var1_MeanN0 environment using 3B and 7B parameter models, measured by the frequency of selecting the best arm. For the 7B models, performance improves over iterations across varying numbers of arms (K=2, 3, 5), with higher accuracy for smaller K. In contrast, the 3B model exhibits stagnant accuracy, starting comparably or even higher than the 7B counterpart in simpler 2- and 3-arm settings pre-training but failing to improve with RL updates. Neverthless, we observe that the 3B model can learn with teacher guidance using RL-ALG or SFT.
This highlights the challenges of training smaller models with RL on task rewards, as learning effective exploration policies from environmental feedback demands long-horizon credit assignment.

We will explore in the following section why the LLM policies excel, what strategies drive their success, and how different learning paradigms lead to different behaviors, to better understand their potential and limitations.

### 5.2 Analyzing LLM Exploration Strategies

We additionally include two surrogate statistics used in Krishnamurthy et al. (2024) as diagnostics for long-term exploration failure: GreedyFreq@$t$ measures the relative frequency of rounds that selects the greedy action up to time $t$, and SuffixFail@$t$ measures the frequency of suffix failures. Specifically, in an episode of length $T=300$, a suffix failure at $t$ indicates that the policy never selects the optimal arm again for rounds $t,\ldots,T$.

**Table 2: Analytics of baselines and 7B LLM policies trained on Gaussian5_Var1_MeanN0, evaluated in-distribution.**
| Metric | AvgReward | BestArmFreq | GreedyFreq | SuffixFail |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| @$t$ | 50 | 300 | 50 | 300 | 50 | 300 | 50 | 150 |
| Baselines |  |  |  |  |  |  |  |  |
| Greedy | 0.91 | 1.01 | 65.4 | 71.7 | 90.0 | 98.3 | 25.0 | 25.0 |
| $\epsilon$-Greedy | 0.76 | 0.90 | 47.9 | 67.6 | 91.3 | 91.6 | 0.0 | 0.0 |
| TS | 0.77 | 1.00 | 55.8 | 78.5 | 67.0 | 85.2 | 0.0 | 0.0 |
| UCB | 0.91 | 1.04 | 67.7 | 80.6 | 83.3 | 95.4 | 3.1 | 4.7 |
| Learned Agents |  |  |  |  |  |  |  |  |
| Pretrain | 0.55 | 0.79 | 45.4 | 63.1 | 48.7 | 65.4 | 0.0 | 0.0 |
| SFT | 0.92 | 1.05 | 69.4 | 81.3 | 83.5 | 95.5 | 6.2 | 6.2 |
| RL-OG | 0.81 | 1.01 | 61.1 | 79.8 | 78.3 | 91.7 | 1.6 | 4.7 |
| RL-STG | 0.84 | 1.01 | 63.7 | 81.1 | 83.7 | 95.8 | 3.1 | 6.2 |
| RL-ALG | 0.92 | 1.05 | 70.7 | 85.7 | 85.4 | 97.0 | 7.8 | 9.4 |

#### Learned policies exhibit greedy tendencies.

While LLM policies achieve lower regret, our qualitative analysis reveals concerns about suboptimal exploration. The first warning sign, shown in [Table 2](https://arxiv.org/html/2509.24923v1#S5.T2), is that the learned agents exhibit higher suffix failure frequency than both the pre-trained model and theoretical optimal policies.
This indicates premature abandonment of the best arm, a pattern absent in the pretrained model. Learning also alters the distribution of best arm selection frequency from an approximately normal distribution for the pre-trained model to a bimodal distribution, where the agent either almost always selects or very rarely selects the best arm within an episode, a characteristic of Greedy behavior ([Figure 5](https://arxiv.org/html/2509.24923v1#S5.F5)). Direct measurement of greedy-arm selection frequency further confirms that the learning agents reach the exploitation phase more quickly than the pre-trained model.

Figure: Figure 5: Suffix failure frequency and best arm selection histogram of the two sets of LLM policies described in [Figure 3](https://arxiv.org/html/2509.24923v1#S5.F3), evaluated in-distribution.
Refer to caption: x3.png

Figure: Figure 6: Match rates of the RL-ALG policy against decisions of the oracle UCB and a UCB-like algorithm discovered in LLM rationales, at different timesteps. SFT policy shows a jump in calculation errors at the 51st step.
Refer to caption: x4.png

#### Dissecting imitation of the UCB oracle: RL vs. SFT.

Although the UCB policy is itself greedy in construction, student policies trained under UCB teacher often amplify this tendency.
This partially explains why LLM policies trained to imitate UCB decisions can sometimes perform better than the oracle. To analyze this phenomenon, we compare how often the choices of each UCB-mimicking policy diverge from the oracle’s decision given the same state, a metric we refer to as the “match rate”. For SFT policies that explicitly calculate confidence-bound values, we additionally report the absolute difference between the policy’s predicted UCB value and the corresponding oracle value, averaged across the arms.

As shown in [Figure 6](https://arxiv.org/html/2509.24923v1#S5.F6), when both policies trained with UCB supervision in the same environment Gaussian5_Var1_MeanN0, the SFT policy maintains a higher match rate than the RL policy from the beginning, indicating that it more faithfully imitates the teacher’s decisions. Both sustain a high match rate above 80% in the first 50 steps in-distribution, with the SFT policy tracks the oracle’s UCB decisions more closely.
However, this stronger imitation capability also makes the SFT policy more susceptible to overfitting, meaning its high match rates are only guaranteed when evaluation conditions closely resemble the training data.

This sensitivity becomes apparent when training on different data. While the SFT policy trained on Gaussian5_Var1_MeanN0 sustains high match rates across all tested environments, an SFT policy trained on Bernoulli5_Uniform achieves this *only* within the same Bernoulli family of tasks ([Figure 18](https://arxiv.org/html/2509.24923v1#A3.F18)). We find this failure is a result of systematic errors in simple calculations involving negative rewards, which are unseen during training—a sign of catastrophic forgetting of basic arithmetic skills (Chu et al., 2025; Shenfeld et al., 2025). The agent frequently miscalculates the UCB values and subsequently disregards its own calculations. This leads to asymmetric generalization, where performance degrades sharply outside the training distribution, consistent with the higher worst-case regret we previously observed in [Figure 3](https://arxiv.org/html/2509.24923v1#S5.F3). These results highlight the critical importance of training data selection to balance imitation fidelity with robustness.

#### What incentivizes RL-ALG to prioritize exploitation over imitation?

The adaptive behavior of the RL-ALG policy is a subtle consequence of the bandit learning structure and a fundamental change in the UCB teacher’s behavior over an episode. Initially, the UCB algorithm balances high uncertainty (exploration) and high observed rewards (exploitation). As an episode progresses, the uncertainty bounds shrink, and the teacher’s policy converges. Its decisions become increasingly dominated by the empirical means, causing it to select the greedy arm. In this regime, the RL objective, though formally defined in terms of imitation, becomes highly correlated with a reward for exploitation. The agent discovers that it can optimize more easily by directly picking the greedy arm, rather than faithfully internalizing the teacher’s complex exploration logic.

#### LLM rationales reveal flawed, exploitative heuristics.

The LLM’s generated rationales can reveal its underlying decision-making process.
RL policies trained on bandit rewards converge to templated heuristics that most oftenly compare and choose the arm with the highest mean reward. Their explorative actions are driven by rationales that explicitly evaluate the uncertainty of the arms, sometimes with UCB-like calculation.
RL-ALG trained in Gaussian5_Var1_MeanN0 however converges to a UCB-like algorithm and mentions in its rationales 98% of the time:
$Q_{t}(a)+C\times\sqrt{\frac{\log(N_{t}(a)+1)}{N_{t}(a)}}.$
In the standard UCB algorithm, the numerator of the exploration term $\log(t)$ grows with the total number of pulls, ensuring that no action is ever abandoned permanently. In contrast, this learned variant’s exploration term depends only on $N_{t}(a)$, the pulls of a specific arm. This allows the policy to prematurely stop exploring an action if it appears unprofitable in the short run, embodied an exploitative tendency.

[Figure 6](https://arxiv.org/html/2509.24923v1#S5.F6) shows that this UCB-like algorithm describes the LLM policy better than the oracle UCB. However, the LLM does not follow the algorithm strictly, as its decisions are also affected by frequent numerical inaccuracies such as miscalculating the $\log$ term.
We can also observe that when the policy diverges from the UCB variant’s decisions, it opts for the greedy action more than 86% of the time.
In Bernoulli5_Uniform, the LLM converges to another variant $Q_{t}(a)+\frac{C}{\sqrt{N_{t}(a)}}$ with similar greedy behavior.
These findings reveal that the RL-ALG policy learns approximate, error-tolerant variants of UCB, blending imitation with opportunistic exploitation that lowers regret in certain environments.
Intriguingly, we observe that the LLM generates the correct UCB formula with inaccurate calculations during early training stages. Its eventual convergence to greedy variants suggests failures in credit assignment.

## 6 Conclusion

We fine-tune LLM agents via SFT and RL with novel reward signals, achieving strong performance with lower regret and robust generalization to 6$\times$ longer horizons and new reward distributions in the multi-armed bandit task.
However, behavioral analysis reveals that training elicits short-sighted, exploitative policies.
This emergent greediness is a consequence of the fundamental imbalance in training data, where sparse exploration signals are easily overwhelmed by frequent exploitation. Compounded by the complex credit assignment problem, this challenge highlights the need for methods that explicitly amplify exploration signals.
Future work could explore focused replay techniques that re-weight experiences based on information gain and surprise or design adversarial and curriculum-based environments that make robust long-horizon planning a necessity for success.

## Acknowledgments

We thank Ronald Parr, Shuyan Zhou, and Fan Yao for providing insightful feedback and suggestions to the manuscript.
We also thank Qian Liu, Weiting Tan, and Junlin Wang for helpful discussions.
This work was supported by NSF award IIS-2211526.

## References

- Abbeel & Ng (2004)
Pieter Abbeel and Andrew Y. Ng.
Apprenticeship learning via inverse reinforcement learning.
In Carla E. Brodley (ed.), *Machine Learning, Proceedings of the Twenty-first International Conference (ICML 2004), Banff, Alberta, Canada, July 4-8, 2004*, volume 69 of *ACM International Conference Proceeding Series*. ACM, 2004.
[10.1145/1015330.1015430](https:/doi.org/10.1145/1015330.1015430).
URL [https://doi.org/10.1145/1015330.1015430](https://doi.org/10.1145/1015330.1015430).
- Auer et al. (2002)
Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer.
Finite-time analysis of the multiarmed bandit problem.
*Machine learning*, 47(2):235–256, 2002.
- Binz & Schulz (2022)
Marcel Binz and Eric Schulz.
Using cognitive psychology to understand gpt-3.
*arXiv preprint arXiv: 2206.14576*, 2022.
- Bouneffouf & Feraud (2025)
Djallel Bouneffouf and Raphael Feraud.
Multi-armed bandits meet large language models.
*arXiv preprint arXiv: 2505.13355*, 2025.
- Bouneffouf et al. (2020)
Djallel Bouneffouf, Irina Rish, and Charu Aggarwal.
Survey on applications of multi-armed and contextual bandits.
In *2020 IEEE congress on evolutionary computation (CEC)*, pp. 1–8. IEEE, 2020.
- Chu et al. (2025)
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V. Le, Sergey Levine, and Yi Ma.
Sft memorizes, rl generalizes: A comparative study of foundation model post-training.
*arXiv preprint arXiv: 2501.17161*, 2025.
- Ciosek (2022)
Kamil Ciosek.
Imitation learning by reinforcement learning.
In *The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022*. OpenReview.net, 2022.
URL [https://openreview.net/forum?id=1zwleytEpYx](https://openreview.net/forum?id=1zwleytEpYx).
- Coda-Forno et al. (2023)
Julian Coda-Forno, Marcel Binz, Zeynep Akata, Matt M. Botvinick, Jane X. Wang, and Eric Schulz.
Meta-in-context learning in large language models.
In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), *Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023*, 2023.
URL [http://papers.nips.cc/paper_files/paper/2023/hash/cda04d7ea67ea1376bf8c6962d8541e0-Abstract-Conference.html](http://papers.nips.cc/paper_files/paper/2023/hash/cda04d7ea67ea1376bf8c6962d8541e0-Abstract-Conference.html).
- Duan et al. (2016)
Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and Pieter Abbeel.
Rl^2: Fast reinforcement learning via slow reinforcement learning.
*arXiv preprint arXiv: 1611.02779*, 2016.
- Hauskrecht et al. (2013)
Milos Hauskrecht, Nicolas Meuleau, Leslie Pack Kaelbling, Thomas L. Dean, and Craig Boutilier.
Hierarchical solution of markov decision processes using macro-actions.
*arXiv preprint arXiv: 1301.7381*, 2013.
- Kingma & Ba (2015)
Diederik P. Kingma and Jimmy Ba.
Adam: A method for stochastic optimization.
In Yoshua Bengio and Yann LeCun (eds.), *3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings*, 2015.
URL [http://arxiv.org/abs/1412.6980](http://arxiv.org/abs/1412.6980).
- Krishnamurthy et al. (2024)
Akshay Krishnamurthy, Keegan Harris, Dylan J. Foster, Cyril Zhang, and Aleksandrs Slivkins.
Can large language models explore in-context?
In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang (eds.), *Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024*, 2024.
URL [http://papers.nips.cc/paper_files/paper/2024/hash/d951f73c521d069fefbb73396df01424-Abstract-Conference.html](http://papers.nips.cc/paper_files/paper/2024/hash/d951f73c521d069fefbb73396df01424-Abstract-Conference.html).
- Kveton et al. (2020)
Branislav Kveton, Martin Mladenov, Chih-Wei Hsu, Manzil Zaheer, Csaba Szepesvari, and Craig Boutilier.
Meta-learning bandit policies by gradient ascent.
*arXiv preprint arXiv: 2006.05094*, 2020.
- Kwon et al. (2023)
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica.
Efficient memory management for large language model serving with pagedattention.
In *Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles*, 2023.
- Laskin et al. (2023)
Michael Laskin, Luyu Wang, Junhyuk Oh, Emilio Parisotto, Stephen Spencer, Richie Steigerwald, DJ Strouse, Steven Stenberg Hansen, Angelos Filos, Ethan A. Brooks, Maxime Gazeau, Himanshu Sahni, Satinder Singh, and Volodymyr Mnih.
In-context reinforcement learning with algorithm distillation.
In *The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023*. OpenReview.net, 2023.
URL [https://openreview.net/pdf?id=hy0a5MMPUv](https://openreview.net/pdf?id=hy0a5MMPUv).
- Lattimore & Szepesvári (2020)
Tor Lattimore and Csaba Szepesvári.
*Bandit algorithms*.
Cambridge University Press, 2020.
- Lee et al. (2023)
Jonathan Lee, Annie Xie, Aldo Pacchiano, Yash Chandak, Chelsea Finn, Ofir Nachum, and Emma Brunskill.
Supervised pretraining can learn in-context reinforcement learning.
In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), *Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023*, 2023.
URL [http://papers.nips.cc/paper_files/paper/2023/hash/8644b61a9bc87bf7844750a015feb600-Abstract-Conference.html](http://papers.nips.cc/paper_files/paper/2023/hash/8644b61a9bc87bf7844750a015feb600-Abstract-Conference.html).
- Moeini et al. (2025)
Amir Moeini, Jiuqi Wang, Jacob Beck, Ethan Blaser, Shimon Whiteson, Rohan Chandra, and Shangtong Zhang.
A survey of in-context reinforcement learning.
*arXiv preprint arXiv: 2502.07978*, 2025.
- Monea et al. (2024)
Giovanni Monea, Antoine Bosselut, Kianté Brantley, and Yoav Artzi.
Llms are in-context reinforcement learners.
*arXiv preprint arXiv: 2410.05362*, 2024.
- Nie et al. (2024)
Allen Nie, Yi Su, Bo Chang, Jonathan N Lee, Ed H Chi, Quoc V Le, and Minmin Chen.
Evolve: Evaluating and optimizing llms for exploration.
*ArXiv preprint*, abs/2410.06238, 2024.
URL [https://arxiv.org/abs/2410.06238](https://arxiv.org/abs/2410.06238).
- Park et al. (2025)
Chanwoo Park, Xiangyu Liu, Asuman E. Ozdaglar, and Kaiqing Zhang.
Do LLM agents have regret? A case study in online learning and games.
In *The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025*. OpenReview.net, 2025.
URL [https://openreview.net/forum?id=qn9tBYQHGi](https://openreview.net/forum?id=qn9tBYQHGi).
- Qwen et al. (2024)
Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu.
Qwen2.5 technical report.
*arXiv preprint arXiv: 2412.15115*, 2024.
- Rahn et al. (2024)
Nate Rahn, Pierluca D’Oro, and Marc G. Bellemare.
Controlling large language model agents with entropic activation steering, 2024.
URL [https://arxiv.org/abs/2406.00244](https://arxiv.org/abs/2406.00244).
- Schmied et al. (2025)
Thomas Schmied, Jörg Bornschein, Jordi Grau-Moya, Markus Wulfmeier, and Razvan Pascanu.
Llms are greedy agents: Effects of rl fine-tuning on decision-making abilities.
*ArXiv preprint*, abs/2504.16078, 2025.
URL [https://arxiv.org/abs/2504.16078](https://arxiv.org/abs/2504.16078).
- Schulman et al. (2016)
John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel.
High-dimensional continuous control using generalized advantage estimation.
In Yoshua Bengio and Yann LeCun (eds.), *4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings*, 2016.
URL [http://arxiv.org/abs/1506.02438](http://arxiv.org/abs/1506.02438).
- Schulman et al. (2017)
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.
Proximal policy optimization algorithms.
*arXiv preprint arXiv: 1707.06347*, 2017.
- Shenfeld et al. (2025)
Idan Shenfeld, Jyothish Pari, and Pulkit Agrawal.
Rl’s razor: Why online reinforcement learning forgets less.
*arXiv preprint arXiv: 2509.04259*, 2025.
- Sheng et al. (2024)
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu.
Hybridflow: A flexible and efficient rlhf framework.
*arXiv preprint arXiv: 2409.19256*, 2024.
- Sutton et al. (1998)
Richard S Sutton, Andrew G Barto, et al.
*Reinforcement learning: An introduction*.
MIT Press, 1998.
- Tajwar et al. (2025)
Fahim Tajwar, Yiding Jiang, Abitha Thankaraj, Sumaita Sadia Rahman, J Zico Kolter, Jeff Schneider, and Ruslan Salakhutdinov.
Training a generally curious agent.
*ArXiv preprint*, abs/2502.17543, 2025.
URL [https://arxiv.org/abs/2502.17543](https://arxiv.org/abs/2502.17543).
- Thompson (1933)
William R Thompson.
On the likelihood that one unknown probability exceeds another in view of the evidence of two samples.
*Biometrika*, 25(3/4):285–294, 1933.
- Wu et al. (2024)
Yue Wu, Xuan Tang, Tom M. Mitchell, and Yuanzhi Li.
Smartplay : A benchmark for llms as intelligent agents.
In *The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024*. OpenReview.net, 2024.
URL [https://openreview.net/forum?id=S2oTVrlcp3](https://openreview.net/forum?id=S2oTVrlcp3).
- Xue et al. (2025)
Zhenghai Xue, Longtao Zheng, Qian Liu, Yingru Li, Xiaosen Zheng, Zejun Ma, and Bo An.
Simpletir: End-to-end reinforcement learning for multi-turn tool-integrated reasoning.
*arXiv preprint arXiv: 2509.02479*, 2025.
- Yao et al. (2023)
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao.
React: Synergizing reasoning and acting in language models.
In *The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023*. OpenReview.net, 2023.
URL [https://openreview.net/pdf?id=WE_vluYUL-X](https://openreview.net/pdf?id=WE_vluYUL-X).
- Zhao et al. (2023)
Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li.
Pytorch FSDP: experiences on scaling fully sharded data parallel.
*Proc. VLDB Endow.*, 16(12):3848–3860, 2023.
[10.14778/3611540.3611569](https:/doi.org/10.14778/3611540.3611569).
URL [https://www.vldb.org/pvldb/vol16/p3848-huang.pdf](https://www.vldb.org/pvldb/vol16/p3848-huang.pdf).

## Appendix A Implementation Details

We report additional details for the environment settings, the RL and SFT training of LLM policies.

### A.1 Environment Settings

**Table 3: Generic families of $k$-armed MAB environments and a complete list of 15 parameterizations used in our study. Asterisk indicates the training task distributions.**
| Family | Reward Dist. | Mean Dist. | Example Instantiation |
| --- | --- | --- | --- |
| Gaussian$k$_Var$\sigma^{2}$_MeanN$m$ | $r\sim\mathcal{N}(u_{i},\sigma^{2})$ | $u\sim\mathcal{N}(m,\sigma_{u}^{2})$ | Gaussian5_Var1_MeanN0^∗<br>Gaussian10_Var1_MeanN0<br>Gaussian5_Var3_MeanN0<br>Gaussian5_Var1_MeanN$\pm 1$<br>Gaussian5_Var3_MeanN$\pm 1$ |
| Gaussian$k$_Var$\sigma^{2}$_MeanU | $r\sim\mathcal{N}(u_{i},\sigma^{2})$ | $u\sim\mathcal{U}(0,1)$ | Gaussian5_Var1_MeanU<br>Gaussian5_Var3_MeanU<br>Gaussian5_Var5_MeanU |
| Bernoulli$k$_Uniform | $r\sim\mathcal{B}(u_{i})$ | $u\sim\mathcal{U}(0,1)$ | Bernoulli5_Uniform^∗<br>Bernoulli10_Uniform |
| Bernoulli$k$_Delta$\Delta$ | $r\sim\mathcal{B}(u_{i})$ | $u_{i^{*}}=p$,<br>$u_{i}=p-\Delta,\forall i\neq i^{*}$ | Bernoulli5_Delta0.2<br>Bernoulli10_Delta0.2<br>Bernoulli5_Delta0.1 |

We evaluate our policies on a comprehensive set of environments from the Gaussian and Bernoulli families, as listed in Table [3](https://arxiv.org/html/2509.24923v1#A1.T3).
The Bernoulli$k$_Delta$\Delta$ class, studied by Krishnamurthy et al. (2024), allows for instance difficulty to be easily adjusted by changing the reward gap between the optimal and sub-optimal arms. However, this environment is unsuitable for training, as a policy can simply explore each action sequentially until finding one with a mean reward above 0.5.
The Gaussian$k$_Var$\sigma^{2}$_MeanN$m$ class is a popular benchmark introduced in Sutton et al. (1998) and later used by Nie et al. (2024). In these environments, the variances of both the reward and mean distributions are tied to a single hyperparameter, $\sigma^{2}$. Increasing the variance does not necessarily make the environment more challenging; while the rewards become noisier, the means also become more dispersed. These two effects offset each other, so this comparison group primarily tests a policy’s robustness to multiplicative rescaling and shifts in rewards.
The Gaussian$k$_Var$\sigma^{2}$_MeanU class, from Schmied et al. (2025), maintains a fixed uniform distribution for the means, allowing the reward variance to be adjusted via $\sigma^{2}$ to control for different difficulty levels.

The diverse distribution shifts across these test environments—including changes in mean, variance, and distributional family—are designed to ablate different aspects of generalization. For instance, transitioning from Gaussian5_Var1_MeanN0 to Gaussian5_Var1_MeanN$\pm 1$ provides a targeted test of a policy’s ability to handle shifted reward distributions while all other properties remain constant. To assess scalability, we additionally test the policies on environments with an increasing number of arms (from $k=5$ to $k=10$).

### A.2 RL Settings

**Table 4: Hyperparameters for the PPO training of LLM policies (RL-OG and RL-STG).**
| Category | Hyperparameter | Value |
| --- | --- | --- |
| Model & Environment |  |  |
| Model | Base Language Model | Qwen/Qwen2.5-3/7b-Instruct |
|  | Max Response Length | 1024 tokens |
|  | Temperature | 1.0 |
| Environment | Type | Various (Gaussian, Bernoulli) |
|  | Number of Arms ($k$) | 5 |
|  | Episode Length ($T$) | 50 |
|  | Number of Parallel Environments | 64 |
| PPO Algorithm |  |  |
| Optimization | Optimizer | AdamW (Kingma & Ba, 2015) |
|  | Actor Learning Rate ($\alpha_{\pi}$) | $1\times 10^{-6}$ |
|  | Critic Learning Rate ($\alpha_{V}$) | $1\times 10^{-5}$ |
|  | Gradient Clipping | 1.00 |
|  | Response-level Discount Factor ($\gamma_{\text{intra}}$) | 1.00 |
|  | Response-level GAE Lambda ($\lambda_{\text{intra}}$) | 1.00 |
|  | Episode-level Discount Factor ($\gamma_{\text{inter}}$) | 0.95 |
|  | Episode-level GAE Lambda ($\lambda_{\text{inter}}$) | 0.95 |
|  | PPO Clipping Coefficient ($\epsilon$) | 0.20 |
|  | PPO Mini-batch Size | 128 |
| Regularization | Weight Decay | $1\times 10^{-2}$ |
|  | Entropy Coefficient | $5\times 10^{-4}$ |
| Training Infrastructure |  |  |
| Training | Total Training Steps | 500 |
| Hardware | Number of GPUs | 4 |
|  | Tensor Parallelism (Rollout) | 4 |

[Table 4](https://arxiv.org/html/2509.24923v1#A1.T4) presents the hyperparameters used for PPO training of the LLM policies (RL-OG and RL-STG). For the algorithmic-reward variant (RL-ALG), we retain all settings except that we set the episode-level discount factor and GAE lambda to zero, since cumulative rewards are not required.

We use vLLM (Kwon et al., 2023) for asynchronous rollouts across parallel environments and FSDP (Zhao et al., 2023) for fully sharded training under the VeRL framework. At the start of each iteration, all environments are reinitialized with fresh random seeds to ensure diverse experience collection.

Model checkpoints are saved every 100 training steps. Each checkpoint is evaluated on the same set of environments (matching the training type) to guarantee fair comparison. The checkpoint with the lowest cumulative regret is selected as the final model.

### A.3 SFT Settings

We generate training data for supervised fine-tuning by sampling $N$ trajectories from the environment using a UCB policy. To expose the model to a broad spectrum of environment configurations and exploration behaviors, we uniformly sample states and actions across each trajectory’s horizon.

As in our reinforcement learning experiments, we save model checkpoints at the end of every epoch and evaluate them on the same set of environments to ensure a fair comparison.

**Table 5: Hyperparameters for Supervised Fine-Tuning (SFT).**
| Category | Hyperparameter | Value |
| --- | --- | --- |
| Model & Data |  |  |
| Model | Base Language Model | Qwen/Qwen2.5-3/7b-Instruct |
| Data | Type | Various (Gaussian, Bernoulli) |
|  | Number of Arms ($k$) | 5 |
|  | Max Episode Length ($T$) | 50 |
|  | Number of Examples ($N$) | 32768 |
| Optimization |  |  |
| Optimizer | Type | AdamW |
|  | Learning Rate | $1\times 10^{-5}$ |
|  | Betas ($\beta_{1},\beta_{2}$) | (0.9, 0.95) |
|  | Weight Decay | 0.01 |
|  | LR Scheduler | Cosine Decay |
|  | Warmup Ratio | 0.1 |
|  | Gradient Clipping | 1.0 |
| Training Details |  |  |
|  | Batch Size | 256 |
|  | Epochs | 6 |

## Appendix B Details of Performance Comparison

We provide comprehensive experimental results of the LLM policies compared against baselines over a range of environments and model sizes on cumulative regret at 50 and 300 steps. The observation is consistent with the analysis in main text.

### B.1 7B Model Comparisons

The cumulative regret trends are consistent across evaluations at 50 and 300 steps. However, the longer 300-step horizon more prominently exposes the weaknesses of simple heuristics like $\epsilon$-greedy. At 50 steps, the imitation learning policies (RL-ALG and SFT) show some instability, likely due to the stronger mimicking effect of the UCB oracle in the initial exploratory phase.

The learned policies demonstrate effective generalization to higher variance in the Gaussian5_Var3_MeanN0 environment, where the difficulty of identifying the optimal arm is comparable to the Gaussian5_Var1_MeanN0 training setting. This generalization fails, however, when increased variance makes the task harder, as is the case in the Gaussian5_Var$\sigma^{2}$_MeanU environments. Here, exploration strategies developed in low-variance settings are hindered by their greedy bias, leading to poor performance under high uncertainty.
Amidst this, the RL-OG policy begins to show a slight advantage. Finally, we note that the SFT policy, when trained on Bernoulli5_Uniform, consistently fails to generalize to any Gaussian environment ([Figure 10](https://arxiv.org/html/2509.24923v1#A2.F10)).

Figure: Figure 7: Generalization to environments with 10 arms for Gaussian-trained LLM policies (7B base model). Best arm selection frequency is reported.
Refer to caption: x5.png

#### Generalization to Larger Action Space.

The pre-trained model’s performance degrades significantly as the action space increases. In the three 10-arm environments (Bernoulli10_Uniform, Bernoulli10_Delta0.2, and Gaussian10_Var1_MeanN0), it exhibits excessive regret, while the policies we trained maintain stable performance. [Figure 7](https://arxiv.org/html/2509.24923v1#A2.F7) also illustrates this failure: the base LLM’s best-arm selection frequency stops improving at an early stage, showing that it commits to a suboptimal action. Our training, therefore, successfully enhances the policy’s generalization to an increased action space.

Figure: Figure 8: Comparison of LLM policies (7B base model) against baselines on cumulative regret at 300 steps (outliers are trimmed). Results on training environment has a colored border.
Refer to caption: x6.png

Figure: Figure 9: Comparison of LLM policies (7B base model) against baselines on cumulative regret at 50 steps (outliers are trimmed). Results on training environment has a colored border.
Refer to caption: x7.png

Figure: Figure 10: Comparison of LLM policies (7B base model) against baselines on cumulative regret at 300 steps (outliers are trimmed). Results on training environment has a colored border.
Refer to caption: x8.png

Figure: Figure 11: Comparison of LLM policies (7B base model) against baselines on cumulative regret at 50 steps (outliers are trimmed). Results on training environment has a colored border.
Refer to caption: x9.png

### B.2 3B Model Comparisons

**Table 6: Analytics of baselines and 3B LLM policies trained on Gaussian5_Var1_MeanN0, evaluated in-distribution.**
| Metric | AvgReward | BestArmFreq | GreedyFreq | SuffixFail |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| @$t$ | 50 | 300 | 50 | 300 | 50 | 300 | 50 | 150 |
| Baselines |  |  |  |  |  |  |  |  |
| Epsilon-Greedy e=0.1 | 0.76 | 0.90 | 47.9 | 67.6 | 91.3 | 91.6 | 0.0 | 0.0 |
| Thompson Sampling | 0.77 | 1.00 | 55.8 | 78.5 | 67.0 | 85.2 | 0.0 | 0.0 |
| Greedy | 0.91 | 1.01 | 65.4 | 71.7 | 90.0 | 98.3 | 25.0 | 25.0 |
| UCB c=0.5 | 0.91 | 1.04 | 67.7 | 80.6 | 83.3 | 95.4 | 3.1 | 4.7 |
| Learned Agents |  |  |  |  |  |  |  |  |
| 3b Pretrain | 0.39 | 0.43 | 29.3 | 32.7 | 85.9 | 67.8 | 12.5 | 12.5 |
| 3b SFT | 0.92 | 1.04 | 69.7 | 80.8 | 83.5 | 93.9 | 6.2 | 9.4 |
| 3b RL-OG | 0.41 | 0.69 | 31.5 | 49.1 | 81.0 | 76.0 | 6.2 | 7.8 |
| 3b RL-STG | 0.51 | 0.75 | 39.3 | 56.9 | 77.7 | 71.5 | 0.0 | 1.6 |
| 3b RL-ALG | 0.81 | 0.95 | 60.2 | 71.2 | 81.7 | 89.0 | 14.1 | 17.2 |

Consistent with our main findings, results in [Figure 13](https://arxiv.org/html/2509.24923v1#A2.F13) and [Figure 15](https://arxiv.org/html/2509.24923v1#A2.F15) confirm that smaller models benefit less from reinforcement learning optimized directly for environmental reward signals. The RL-OG and RL-STG policies trained this way perform on par with the pre-trained model at 50 steps and achieve only small gains at 300 steps with RL-STG generally outperforming RL-OG.

In contrast, both the imitation learning policies, RL-ALG and SFT, demonstrate a significant improvement over the pre-trained model. The SFT policy, in particular, emerges as the top-performing method, achieving reliably lower regret across nearly all environments. This suggests that, even in the imitation learning setting, smaller models struggle with reinforcement learning optimization process itself. According to [Table 6](https://arxiv.org/html/2509.24923v1#A2.T6), SFT achieves AvgReward and BestArmFreq comparable to the UCB teacher in-distribution. Its performance in identifying the best arm continues to improve with more trials, even in most out-of-distribution environments ([Figure 12](https://arxiv.org/html/2509.24923v1#A2.F12)). We once again observe the previously noted generalization failure of SFT at this model size, where it fails to transfer from the Bernoulli5_Uniform environment to Gaussian environments with negative rewards ([Figure 15](https://arxiv.org/html/2509.24923v1#A2.F15)).

The pre-trained Qwen 2.5 3B model exhibits a distinct exploration pattern compared to its 7B counterpart. While the pre-trained 7B model starts an episode with high exploration and becomes more exploitative over time, the 3B model begins with a highly greedy strategy (GreedyFreq $\approx$ 86%) and becomes more explorative. This causes its ability to identify the best arm plateaus very early in the Gaussian5_Var1_MeanN0 environment.
These behavioral differences lead to different training dynamics: the 7B model consistently reduces exploration throughout the training iterations, while the 3B model first undergoes a phase of increasing exploration before reducing it until convergence.

Across the board, all learned agents show lower GreedyFreq at 50 steps than the pre-trained model. As trials progress, the two successful imitation learning policies (RL-ALG and SFT) adopt a more greedy exploitation strategy. As a result, they both suffer from a higher suffix failure rate compared to RL policies trained on environmental feedback. This reinforces our conclusion that their performance gains are associated with more sophisticated greedy policies.

Figure: Figure 12: Generalization to environments with 10 arms for Gaussian-trained LLM policies (3B base model). Best arm selection frequency is reported.
Refer to caption: x10.png

Figure: Figure 13: Comparison of LLM policies (3B base model) against baselines on cumulative regret at 300 steps (outliers are trimmed). Results on training environment has a colored border.
Refer to caption: x11.png

Figure: Figure 14: Comparison of LLM policies (3B base model) against baselines on cumulative regret at 50 steps (outliers are trimmed). Results on training environment has a colored border.
Refer to caption: x12.png

Figure: Figure 15: Comparison of LLM policies (3B base model) against baselines on cumulative regret at 300 steps (outliers are trimmed). Results on training environment has a colored border.
Refer to caption: x13.png

Figure: Figure 16: Comparison of LLM policies (3B base model) against baselines on cumulative regret at 50 steps (outliers are trimmed). Results on training environment has a colored border.
Refer to caption: x14.png

## Appendix C Details of Imitation Learning Analysis

We investigate why an imitation learning policy might outperform its teacher by analyzing its adherence to key decision-making heuristics. This section expands upon the main text by presenting results from a complete set of experimental environments.

A key finding is that both imitation learning policies (RL-ALG and SFT) make fewer (imitation) errors in high-variance environments. This is attributed to the teacher UCB policy (C=0.5) itself behaving more greedily in these settings, matching the exploitative bias of the imitation learning policies.

We find that the SFT agent’s mistakes reveal errors in both simple arithmetic (summation, subtraction) and complex calculations (logarithms, square roots). A prominent failure mode emerges when the Bernoulli-trained policy observes negative rewards: it often struggles with summations involving these numbers and subsequently disregards its own UCB calculations. For instance, in the Gaussian5_Var3_MeanN0 environment, the agent chooses an arm different from the one with the highest calculated UCB value 78% of the time. This divergence is sensitive to the reward distribution; lowering the environment’s mean reward by 1 increases this deviation rate to 89%, while raising the mean by 1 reduces it to 44%. This behavior indicates a regression in the LLM’s capabilities, leading to hallucinations in its reasoning.
Future work can explore mixed training with mathematical data to alleviate this issue.

We previously discovered that the RL-ALG agents converge to suboptimal variants of the UCB algorithm. This finding is both interesting and disappointing. On one hand, it demonstrates that agents can discover novel solutions from sparse reward signals received only at the end of a response. On the other hand, it suggests that either the oracle policy is not encountered during RL exploration or that credit assignment is a significant challenge. By manually inspecting rollouts from early training iterations, we find that the correct UCB formula did appear, but its calculations were frequently incorrect due to the base model’s weakness in complex operations like square roots and logarithms ([Figure 21](https://arxiv.org/html/2509.24923v1#A3.F21)). This points to a credit assignment issue, where the agent incorrectly attributes poor outcomes to the formula itself, rather than to flawed calculations or suboptimal hyper-parameter choices. Future work could explore more fine-grained RL signals to address this problem.

Figure: Figure 17: 7B SFT agent trained on Gaussian environments: UCB error by step.
Refer to caption: x15.png

Figure: Figure 18: 7B SFT agent trained on Bernoulli environments: UCB error by step.
Refer to caption: x16.png

Figure: Figure 19: 7B RL-ALG agent trained on Gaussian environments to optimize UCB reward signal: match rate by step. UCB_Var here is the UCB variant $Q_{t}(a)+C\times\sqrt{\frac{\log(N_{t}(a)+1)}{N_{t}(a)}}$, which the agent discovered and consistently used.
Refer to caption: x17.png

Figure: Figure 20: 7B RL-ALG agent trained on Bernoulli environments to optimize UCB reward signal: match rate by step. UCB_Var here is the UCB variant $Q_{t}(a)+\frac{C}{\sqrt{N_{t}(a)}}$ that the agent discovered and consistently used.
Refer to caption: x18.png

Figure: Figure 21: An example generated by RL-ALG policy at iteration 50 (7B Gaussian)

## Appendix D LLM Use Disclosure

#### Using LLMs to help with paper writing.

Commercial LLMs were used to correct typos and grammar, suggest alternative phrasings, and provide insights on the clarity and readability.
All LLM-generated text was reviewed, edited, and approved by the human authors.

#### Using LLMs as a research assistant.

LLMs assisted with brainstorming experimental designs, suggesting analysis approaches, searching potentially relevant prior work, and producing code scaffolding and completion. The human authors provided the research context, validated the literature identified by LLMs, verfied all analysis and results, and adapted or often rewrote the LLM-generated content before inclusion.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: Exploration bias mitigation – Could future training include explicit exploration bonuses (e.g., curiosity or entropy regularization) to test if the emergent greediness persists?

Do the authors have an intuition of how findings might transfer to contextual or Markovian tasks where exploration requires memory and state estimation?

The finding that RL-ALG outperforms its UCB teacher is quite interesting. Could the authors analyze whether this over-exploitation emerges from token-level credit assignment or from the episodic reward structure?
soundness: 2
strengths: - **Originality and significance**: This work provides a unified framework comparing SFT and RL fine-tuning for LLM agents on the same controlled MAB setup. The strategic and algorithmic rewards are well-motivated and address variance and credit-assignment issues in RL with LLMs. 
- **Quality**: This paper includes experiments across Gaussian and Bernoulli bandit families, including cross-distribution generalization and longer horizons, which strengthen the empirical claims. Metrics like suffix failure and greedy frequency are used to reveal qualitative behavioral shifts alongwith quantitative evaluation.
- **Clarity**: The paper is generally well-written with clear implementation details and code release commitments.
summary: This paper studies how different fine-tuning paradigms shape the exploration and exploitation behaviors of LLMs in multi-armed bandit (MAB) settings. The authors train Qwen-2.5 3B and 7B models on MAB tasks with different reward signals:
> RL-OG: standard stochastic rewards, 

> RL-STG: regret-shaped strategic rewards, and

> RL-ALG: imitation of the UCB oracle via algorithmic rewards.

The work demonstrates that both SFT and RL reduce regret and achieve performance comparable to theoretical baselines like UCB and Thompson Sampling, with generalization to unseen environments and 6$\times$ longer horizons. However, behavioral analysis shows that these gains arise from emergent exploitative tendencies since trained agents become greedier and sometimes abandon exploration prematurely. RL-ALG agents trained to imitate UCB outperform their teacher by adopting more exploitative variants of it.
The paper concludes that while both SFT and RL enhance decision-making, they also introduce bias toward short-term exploitation, highlighting the need for reward and data designs that explicitly sustain exploration.
weaknesses: - Limited task diversity: Experiments are confined to simple bandit environments, therefore claims about general agentic learning remain somewhat speculative. 

- Model scalability: There is no discussion of whether the observed training and inference dynamics hold even with larger models and across different model families. 

- While the paper uncovers behavioral tendencies, it stops short of offering causal or mechanistic explanations of why greediness emerges under RL/SFT objectives.

- Compared to the analysis of RL approaches in LLM training, the SFT analysis is less thorough, especially regarding data sampling effects and robustness to arithmetic errors. 

- No explicit exploration-promoting baseline: Methods like intrinsic reward shaping or information gain are not compared, though they would provide valuable context.

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 3
questions: What are you trying to show when evaluating these fine-tuned models on MAB tasks? It is not very surprising that a transformer will behave like UCB if you train it to.
soundness: 3
strengths: Using LLMs to solve decision-making tasks under uncertainty is an important next frontier, so the problem being studied is well-motivated. While the authors are not the first to fine-tune LLMs to solve these sorts of bandit tasks, their proposed reward designs are novel, from what I can tell. The finding that gains often stem from more greedy behavior is also interesting, and largely matches the findings of Krishnamurthy et al for vanilla models which are not fine-tuned.
summary: This submission studies how LLMs learn to balance exploration and exploitation when trained to solve multi-armed bandit (MAB) problems, using both supervised fine-tuning and reinforcement learning. They train LLMs on various MAB environments using RL with three forms of reward design: original reward (i.e. the raw environment rewards), strategic reward (a normalized, regret-shaped reward that reflects how close the agent’s action is to the optimal arm at each tilmestep), and algorithmic reward (a signal that rewards the agent for matching the decisions of an expert oracle, specifically the UCB algorithm). For supervised fine-tuning, they train the LLM on synthetic data generated from UCB expert trajectories, including step-by-step chain-of-thought demonstrations of UCB value calculations. 

They evaluate the performance of these methods (when fine-tuning Qwen models) on Gaussian and Bernoulli MAB environments. They find that the fine-tuned models outperform vanilla models on these tasks, and that policies trained with UCB imitation perform best. These methods also perform comparably to UCB and Thompson sampling. 

Upon further analysis, the authors find that these gains often come from the models adopting greedy behaviors and under-exploring.
weaknesses: I am not entirely sure what this paper is trying to accomplish. The typical motivation for studying LLMs’ ability to solve bandit tasks is because we want to design AI models capable of solving complex, text-based, decision making tasks under uncertainty. This is because we don’t currently have good algorithms for solving these types of tasks. Bandits get to the essence of the exploration/exploitation trade-off, and so they can be a useful abstraction. 

However, I am not sure what utility we get from fine-tuning LLMs to be more like existing algorithms, without evaluating their performance on some complex task that existing algorithms do not do well in (or are not even applicable to). (We don't need to reinvent the wheel to solve MAB tasks, as we have good algorithms for these types of tasks already.)

---

confidence: 3
contribution: 4
initial_rating: 10
presentation: 4
questions: Typo 1: In Figure 3, it says RL-STG, it should be RL-STR 

1. RL-STR is not significantly better than RL-OG. Just comparing the average, in 3 out of 8 domains (Figure 3), it's better. It's worse in 3 out of 8. In 2 it's about the same? For people who might want to train exploration policy in a realistic setting, how would you give them suggestions on what reward design to use? 

2. Table 2, you did some behavioral analysis and citing a prior work. Krishnamurthy et al. (2024) also proposed MinFrac. Do you think this is a metric worth including? 

3. [1] also proposed to fit a function form over steps $T$ on average cumulative regret -- and try to describe the slope/speed of learning/unlearning. Would such analysis yield additional insights for your setting?

4. What future directions are opened up by your analysis, besides moving to more realistic domain? In your GRPO training (with RL-OG), do you see length hacking (i.e., with training continues, the length in thinking tokens becomes longer and longer)? In appendix, we saw that RL with ALG reward learns to compute UCB. Can you:
   - Provide sampled generations with base model but sampled using your prompt/pipeline? Do they already know that they should calculate UCB values?
   - Can you provide sampled generation from RL-OG and RL-STR for us so we know if these also learned (on their own) to compute UCB values?
soundness: 4
strengths: 1. The paper has extremely clear writing
2. The authors disambiguated different training strategies and setups very well. 
3. This paper introduced token-level reward derived from GAE, which improved upon prior work.
4. Each reward design is principled and reasonable -- incorporating all possible learning signals. 
5. The paper is well-situated compared to prior works, investigating areas where prior works didn't, adding critical knowledge/understanding to the entire teaching LLM to explore space.
summary: This paper investigates how Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) training paradigms shape the exploration strategies of Large Language Models (LLMs) when solving Multi-Armed Bandit (MAB) tasks, treating the LLMs as meta-bandit agents.

I currently think this paper provides a lot of value to the "learning to learn" / "learning to explore" community. However, I raised several questions and based on the authors' rebuttal response, I reserve the right to change/update my score.
weaknesses: The paper doesn't have obvious flaws/weaknesses, but here is some nitpicking weaknesses:

1. As a field, the learning to explore community needs to move beyond toy setting. [1] proposed and [2] included the MovieLens environment, which is a good first step. Behaviors of RL vs SFT and whether we need $\pi^*$ (optimal exploration policy) need to be answered in slightly more realistic setting. Many algorithms (like SGD) might be flawed in constructed setups, but they work really well in practice. Even though I think this paper has enough value without exploring those more realistic settings, I would encourage the authors to extend the paper to cover those settings.

2. The conclusion from the experiment seems a bit unclear (see the question part).

[1] Nie, Allen, et al. "Evolve: Evaluating and optimizing llms for exploration." arXiv preprint arXiv:2410.06238 (2024).

[2] Schmied, Thomas, et al. "Llms are greedy agents: Effects of rl fine-tuning on decision-making abilities." arXiv preprint arXiv:2504.16078 (2025).


Decision: Accept (Poster)