Can large language models explore in-context?
Authors: Akshay Krishnamurthy
Keegan Harris
Dylan J Foster
Cyril Zhang
Aleksandrs Slivkins
Venue: NeurIPS 2024 poster (2024)
Keywords: in-context learning
exploration
multi-armed bandits
reinforcement learning
Primary area: bandits
TL;DR: We investigate the extent to which contemporary LLMs can engage in exploration.

Abstract: We investigate the extent to which contemporary Large Language Models (LLMs) can engage in exploration, a core capability in reinforcement learning and decision making. We focus on native performance of existing LLMs, without training interventions. We deploy LLMs as agents in simple multi-armed bandit environments, specifying the environment description and interaction history entirely in-context, i.e., within the LLM prompt. We experiment with GPT-3.5, GPT-4, and Llama2, using a variety of prompt designs, and find that the models do not robustly engage in exploration without substantial interventions: i) Only one configuration resulted in satisfactory exploratory behavior: GPT-4 with chain-of-thought reasoning and an externally summarized interaction history; ii) All other configurations did not result in robust exploratory behavior, including those with chain-of-thought reasoning but unsummarized history. While these findings can be interpreted positively, they suggest that external summarization—which may not be possible in more complex settings—is essential for desirable LLM behavior. We conclude that non-trivial algorithmic interventions, such as fine-tuning or dataset curation, may be required to empower LLM-based decision making agents in complex settings.

Can Large Language Models Explore In-Context?
Akshay Krishnamurthy' Keegan Harris² Dylan J. Foster' Cyril Zhang' Aleksandrs Slivkins'
Microsoft Research
CCanneiie Mellon University
keeganhēcs. cm.. edu, (akshaykr, eete sywp swl.kiinsjee..croofff..om
Abstract
We investigate the extent to which contemporary Large Language Models (LLMs) can engage in exploration, a core capability in reinforcement learning and decision making. We focus native performance of existing LLMs, without training inter- ventions. We deploy LLMs as agents in simple multi-armed bandit environments, specifying the environment description and interaction history entirely in-context, i.c. within the LLM prompt. We experiment with GPT-3.5, GPT-4, and LLAMA2, using variety of prompt designs, and find that the models do not robustly engage in exploration without substantial interventions: i) Only one configuration resulted in satisfactory exploratory behavior: GPT-4 with chain-of offin--f-thuuhh reasoning and an externally summarized interaction history; ii) All other configurations did not result in robust exploratory behavior, including those with chain-of-thoug!! reasoning but unsummarized history. While these findings can be interpreted positively, they suggest that external summarization-a -whih may not be possible in more complex settings-is is essential for desirable LLM behavior. We conclude that non-trivial algorithmic interventions, such as fine-tuning or dataset curation, may be required to empower LLM- --ddeddaddd- decision making agents in complex settings.
1 Introduction
In-context learning is an important emergent capability of Large Language Models (LLMs) whereby one can use pre-trained LLM solve problem by specifying the problem description and relevant data entirely in-contexl, i.e., within the LLM prompt, with no updates to LLM parameters [16]. For ex- ample, one can prompt LLM with numeric covariate vectors and scalar targets and subsequently ob- tain regression-style predictions from the model by including new covariate vectors in the prompt [28]. Perhaps surprisingly, LLMs are not explicitly trained for this behavior; instead the underlying algo- rithms employed for in-context learning are extracted from the training corpus and emerge at scale.
Since its discovery in the GPT-3 model [16], in-context learning has been actively studied, from the- oretical investigations into the underlying mechanisms [e.g.. 78, 71 to empirical probes [e.g.. 28, 40] to leveraging in-context learning in applications [e.g., 79, 67, 25]. This literature predominantly concerns prediction or supervised learning tasks, and while theoretical progress is in its infancy, our understanding of how to use in-confext supervised learning (ICSL) in practice is rapidly taking shape.
While ICSL. is an important capability, many applications demand the use of MI. models for down- stream decision making. Thus, in-context reinforcement learning (ICRL) is natural next frontier. LLMs are already being used as decision making agents in applications ranging from experimental design in the natural sciences [45] to game playing [63, 72], but our waweestttnninngttorttttt and operationally- of ICRL is far less developed than for ICSL. To date, we lack a systematic understanding to whether LLMs be considered general-purpose decision-male makinn agents.
Decision making agents must possess three core capabilities: generalization (required for supervised learning), exploration (making decisions that may be suboptimal in the short term for the sake of gath-
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

ering more information) and planning (to account for long-term consequences of decisions). In this paper, we focus on exploration, the capability to deliberately gather information in order to evaluate alternatives and reduce uncertainty. A recent series of papers [42, 44, 57] demonstrates in-context re- inforcement learning behavior (including exploration) in transformer models when they are explicitly trained to produce this behavior using data from reinforcement learning agents or expert demonstra- tions on related tasks. Such training tends to be laborious, expensive, and possibly task-specific. In particular, these findings do not shed light into whether exploratory behavior manifests in general- purpose LLMs obtained via standard training methods, which suggests the following basic question:
Do contemporary LLMs exhibit the capability to explore in-context?
Contributions. We investigate this question by deploying LLMs as agents in simple synthetic reinforcement learning problems, namely multi-armed bandits (MABs) [65, 43J, specifying the environment description and interaction history entirely within the LLM prompt. MABs are a well-studied type of RL problem that isolates the tradeoff between exploration and exploitation, i.c., making the best decision given the available data. They are also fundamental in that the ability to solve MABs is prerequisite for more challenging RL tasks. These considerations make MABs natural choice for systematically studying the in-context exploration abilities of LLMs.
We evaluate the in-context exploration behavior of GPT-3.5 [16], GPT-4 [54], and LLAMA2 [69] in MAB environments, using variety of prompt designs. In our experiments, we find that only single configuration (i.e., a prompt design and LLM pair) results in satisfactory exploratory behavior. All other configurations exhibit exploration failures, failing to converge to the best decision (arm) with significant probability. We find that this typically happens due to suffix where the LLM fails to select the best arm even once after some initial rounds (i.e., in some "time suffix"). This scenario is reflected in Figure (a)): in particular, GPT-4 with our basic prompt design experiences suffix failure in > 60% of the replicates. An alternative failure mode we identify is where the L.M behaves "uniformly", selecting all arms near-equally often and failing to narrow down to the better ones.
The single configuration that succeeds in our experiments involves a combination of GPT-4 and an "enhanced" prompt that (a) provides a suggestive hint to explore, (b) externally summarizes the history of interaction into per-arm averages, and (c) asks the LLM to use zero-shot chain-of-thought reasoning [74, 4 This configuration is visualized in Figure 1(b). One can interpret this finding positively: state-of-the-are art LLMs do possess the capability to robustly explore, provided that the prompt is carefully designed to elicit this behavior. On the other hand, the same configuration without external summarization fails, leading to a negative interpretation: LLMs may fail to explore in more complex environments, where external summarization is a non-trivial algorithmic problem."
We conclude that while the current generation of LLMs can perhaps explore in simple RL envi- ronments with appropriate prompt engineering, training interventions -in the spirit of Lee et al. [44], Raparthy al. [57]- may be required endow LLMs with more sophisticated exploration capabilities required for more complex settings.
Methodology. An underlying technical challenge in assessing LLM capabilities and limitations is that one must search a combinatorially large space of prompt designs while obtaining statistically meaningful results, all while meeting the financial and computational constraints associated with LLMs. Assessing in-context bandit learning is even more challenging because (a) stochasticity in the environment demands high degree of replication for statistical significance and (b) the sample com- plexity of learnin/eepllortioon demands that even a single experiment involve hundreds or thousands of LLM queries to obtain meaningful effect sizes (i.e., separation between successful and failing methods). To address these issues, our core technical contribution is to identify surrogale statistics as diagnostics for long-term exploration failure. The surrogate statistics consider characterize long- term exploration failure, yet can be measured at moderate scale with few replicates and short learning horizons, even when the standard performance measure (namely, reward) too noisy to be useful.
2 Experimental setup
Multi-armed bandits (MAB). We consider basic multi- bandit variant, stochastic Bernoulli bandits. There are K possible actions (arms), indexed as [K] :=
K}. Each arm a is
E.g.. there are many arms, or if we are considering contextual bandits with many contexts, then we may only play each arm (context-ang arm pair) few times, so averaging reward separately for each as we do in our experiments does not provide much summarization. (See Section 4 for further discussion.)

---

Buttons, 5-arms, Delta=0.2
1.0
Greedy
Sulta Piquarty
Tipe Eevergget reaas
0.55
Gready
0.6
0.4
T5
Greedy
JCB
G774
0.4
0.2
0.45
0.0
50
103
190
200 250 160
10. 450 450
0.0
50
100
150
250
250
100
15a
430
0.40
100
200
350
400
503
Plys of the best am munds (0,500]
Time slep
Time step 03
15 UCB
Eneedy GPF1-CST
Greedy
0.60
T5
GPT6-D0
0.8
0.6-
JCB
GF71-311
Time wutrogedd manee
UCB
Groedy
OFF
0.50
0.1-
0.2
Sulte
0.2
0.0
125
25
1.40
0.40
roundd (0,200)
stee
Figure 1: Representative experiments: Two prompt configurations for GPT-4 on a 5-armed bandit problem, with exploration failure (top) and success (bottom). The baselines are two standard bandit algorithms with performance guarantees, Upper Confidence Bound (UCB) and Thompson Sampling (TS), well the GREEDY algorithm (see Footnote 5). Visualizations are: (Left) histogram over replicates of the number of times the best arm is chosen, (Center) for each I, we plot the suffix failure frequency, the fraction of replicates for which the best arm is never chosen after time-step and (Right) cumulative time-averaged rewards, averaged over replicates (+22 standard errors)
(a) Top row. GPT-4 with basic prompt design and zero temperature. The experiment runs for T = 500 rounds, and is replicated N = 20 times, varying environment randomness. We see highly bimodal behavior: large (> 60%) fraction of replicates pick the best arm only few times, exhibiting suffix failures similar to GREEDY and very unlike UCB and TS. This is suggestive of a long-term failure explore; indeed, we see a substantial drop in rewards.
(b) Bottom row. GPT-4 with a suggestive framing, summarized history, and chain-of-thought with zero temperature. The experiment runs for T = 200 rounds and N = 40 replicates. We observe unimodal distribution of plays of the best arm, very few suffix failures, and reward comparable to TS.
associated with mean reward Ma E [0, 1). which is unknown. An agent interacts with the environment for T time steps, where in each time step E [T] the agent selects an arm a E [K] and receives reward F' E {0, 1} drawn independently from Bernoulli distribution with mean Ma.. Thus, the MAB instance is determined by the mean rewards (Ha a E [K]) and the time horizon I', The goal to maximize the total reward, which roughly corresponds identifying the best arm: arm with the highest mean reward. A key feature of the MAB setup is that rewards for arms not chosen by the agent not revealed, exploration is necessary to identify the best arm.
We focus on MAB instances where the best arm has mean reward μr 0.5 + A/2 for a parameter A > O, while all other arms have mean reward μ = 0.5 2 (so, A ==* 1u is the gap between the best and the second-best arm). The main instance we consider has K = 5 arms and gap = 0.2. We call this the hard instance, as we also consider easy instance with K = 4 and A = 0.5.
Prompts. We employ LLMS operate as decision making agents that interact with MAB instances by prompting them with description of the MAB problem (including the time horizon T) and the history of interaction thus far. Our prompt design allows several independent choices. First is "scenario", which provides a grounding for the decision making problem. positioning the LLM either a) as an agent choosing buttons to press, or b) as a recommendation engine displaying advertisements to users. Second, we specify a "framing" as either a) explicitly suggestive of the need to balance exploration and exploitation, or neutral. Third, the history can be presented as a) raw list over rounds, or it can b) be summarized via number of plays and average rewards of each arm. Fourth, the requested final answer can be a) single arm, or distribution over arms. Finally, we either a) request the answer only, or b) also allow the LLM to provide a "chain-of-thooue" -ff-uuhghh (CoT) explanation. Altogether, these choices lead 2 = 32 prompt designs, illustrated in Figure 2. More details about the prompt design, including examples, are provided in Appendix B.
Larger gap A makes it easier distinguish arms, while smaller K means there fewer arms to explore.
3

---

The most basic prompt design from the options above uses the buttons scenario, neutral framing. and raw history, and requests the LLM to return only an arm with no COT. Each of the five possible modifications to this prompt can potentially help the L.M,, and our experiments evaluate this. For example, both the advertising scenario and suggestive framing might help invoke the L.M's knowledge of bandit algorithms (as bandit algorithms are commonly used in content recommendation). History summarization might help if the LLM cannot reliably summarize history itself (perhaps due to arithmetic errors²) and/or does not fully realize that it should. Returning a distribution might help if the LLM can identify a good distribution, but fails to correctly sample from it. Finally, chain-of oftthoughtthtttttt is known to help in wide variety of LLM scenarios [74, 50], even when used in zero-shot manner [41] as we do here.
Prompts are presented to each LLM using both system and user messages (exposed by all three LLM APIs). The system message presents information about the scenario and framing and prompts the LLM about whether to use CoT and whether (and how) to return a distribution. The user message presents the history and reminds the LLM about how to format its response. For GPT-4 only, we found that prompting the LLM to use CoT in the system prompt did not reliably elicit CoT outputs, for GPT-4 only- we also consider reinforced Co'I' prompt design that additionally reminds the LLM to CoT at the end of the user prompt. See Appendix B for examples.
buttons scenario
advertisements scenario
neutral framing
suggestive framing
description of multi-armed bandit problem
raw history
summarized history
retur:: action
return: distribution over actions
final prompt: reply-only
final prompt: chain-of-thought
Prompt designs; see Figure 9 for more detailed view. A prompt is generated by traversing the graph from top to bottom.
Figure 2:
LLM configurations and baselines. We experiment with three LLMs: GPT-3.5, GPT-4, and LLAMA2.4 4 addition to the prompt variations above, we also consider two choices for the temperature parameter, O and 1. A temperature of O forces the LLM to be deterministic and therefore isolates the "deliberate" exploration behavior the LLM itself. A temperature of 1 provides a source of external randomness in the LLM responses, which may may not result in randomization among the arms. Allowing the LLM to return a distribution instead of a single arm also provides external randomness (as we sample from the returned distribution); to isolate sources of randomness, do not consider temperature 1 with "return distribution" prompt designs.
We refer to the tuple (prompt design, temperature) as the LLM configuration. We identify each configuration with a 5-letter "code" I2 I.s,, with letters L, denoting the choices:
Ly: 'B' or 'A' for, resp.. buttons or advertisements scenario;
. L2: 'N' or 'S' for, resp., neutral or suggestive framing;
. Ly: 'R' or 'S' for, resp., raw summarized history;
.
L4: 'C' or 'C' or 'N' for, resp., chain-of-thought, reinforced CoT, or no COT.
. Ls: '0', 'I' or 'D' for, resp., temperature and returning a distribution (with temperature 0).
We refer "BNRNO" as the basic configuration going forward. Most of our experiments consider the "buttons" scenario, and we use the "udvertisements" scenario primarily as a robustness check.
For GPT-3.5 and LLAMA2, we do not consider reinforced CoT as it is not required to reliably elicit CoT outputs; thus, we have 48 configurations total. For GPT-4, we primarily used reinforced CoT, but did experiment with some standard CoT prompt designs; thus, there are 72 configurations total.
For baselines, we consider two standard MAB algorithms, UCB [9] and Thompson Sampling (TS) [68], which are optimal in a certain theoretical sense and also reasonably effective in practice, We also consider the GREEDY algorithm, which does not explore and is known to fail. While all three
'E.g.. LLMS sometimes fail at basic arithmetic [27, 48), though this is likely to improve in the near future via better training and/or integrating calculator like tools.
"Specifically: GPT-3.5. 3-P-3.5-TURBO-- TURBO-0611 (released 06/13/2023), GPT-4-0613 (released 06/13/2023), and LLAMA2- 1--2-13B-CHAA quantized to 4-bits [24].
"n each round, GREEDY chooses an arm with the largest average reward so far. It is initialized with one sample of each arm. It fails in that with constant probability, it never chooses the best arm after initialization.

---

baselines have tunable parameters, we perform no parameter tuning (see Section A. for detailed description of each algorithm with parameter settings). In addition to these baselines, some of our experiments include the the E-GREEDY algorithm" with various choices of to quantitatively demonstrate tradeoff's between exploration and exploitation. We ran 1000 replicates for each baseline and each MAB instance (with rewards realized independently across the replicates).
Scale of the experiments. Our main set of experiments has time horizon T 100. To account for randomness in rewards (and possibly in the LLM,, via temperature) we ran N E (10,00000 replicates for each I.LM configuration and each bandit instance, with rewards generated independently across the replicates, As robustness check, we ran single experiment on GPT-4 with the basic configuration for T' 500 rounds (with N 20), and obtained consistent/stronger conclusions, see Figure ((a).
In more detail, for GPT-3.5 we used N = 20 replicates across all 48 prompt configurations, resulting order of magnitude more expensive, considerably slower
in = 200K queries in total. GPT-4 on throughput, and subject to unpredictable throttling. As such, we only used N = 10 replicates ran four GPT-4 across 10 representative prompt configurations. For additional robustness checks, configurations with T = 200, two for N = 20 replicates and two for N = 40 replicates. In total, this resulted in <55K queries issued to GPT-4. ILLAMA2 was essentially free from our perspective (since was locally hosted), but its performance was consistently sub-par; we limited our experiments to the hard MAB instance, 32 configurations, and N 10 replicates.
We emphasize that bandit experiments with LLMs are quite costly in terms of money and time. They take N ' LLM queries for each LLM configuration and each MAB instance being tested. Both N and T must be relatively large to obtain statistically meaningful results: N governs the significance level and must be large overcome randomness in reward realizations, while T governs the effect size and must be large so that good algorithms have enough time to identify the optimal arm. Both issues more pronounced in harder MAB instances (many arms K and/or small gap A), but exploration failures also tend to be less frequent in (very) easy MAB instances. Further, we need cover the space of possible prompt designs, which is essentially infinitely large, ensure that our findings do not overfit to one particular design. Thus, ideally we would take N, T, the number of MAB instances, and the number of prompts to be rather large, but doing so is not practically feasible.* Instead, we use moderately small gap A = 0.2, moderately large choices for N E {10, 20} and T 100, and the prompt design space as described above.
As we see below, these choices (N E {10, 20). T = 100, A = 0.2) do not provide enough statistical power to distinguish between successful and unsuccessful methods based solely accumulated rewards. In lieu of further increasing the scale of the experiments, which is not practically feasible, we rely on surrogate statisties which can be detected at our moderate scale, and are highly suggestive of tong-term/pprsisterr exploration failures. Our robustness checks with larger T and N, as well as qualitative findings that we report below provide supporting evidence for this methodology.
3 Experimental results
In this section, we present our experimental findings, beginning with a summary. In Section 3.1 we investigate failing LLM configurations in detail. In Section 3.2, we focus on the single successful LLM configuration we identified. In Section 3.3, we attempt to diagnose root causes for failures.
Overview. All but one LLM configurations considered exhibit exploration failures, not converging to the best arm with significant probability. This happens either due to suffix failures, where the LLM never selects the best arm after a small number of initial rounds, or (in a few configurations) due to uniform-like failures, where the LLM selects all arms at an approximately uniform rate, failing to eliminate poorly performing arms The one exception is GPT-4 with the BSSCO configuration, i.c.. the buttons scenario, suggestive framing, summarized history, reinforced CoT, and temperature O.
We summarize our key findings in Figures 3-4. Figure 3 summarizes the main set of experiments (on the hard MAB instance), mapping each LLM configuration to a single point on a scatter plot.
GREEDY is standard MAB algorithm which in each round chooses an arm uniformly at random with a given probability c, and exploits (i.e., mimics GrEEDy) otherwise.
7 N 10 for the buttons scenario, and N = 3 for the rohustness check with the advertisements scenario.
Raw-history prompts and ohain-of-thought outputs are particularly expensive, as LLM APIs bill per token.
5

---

Greedy
BSSCo
Mediankoward
Suffaliree
0.01
0.02
0.48
0.50
0.40
0.00
0.50
0.60
0.70
0.30
0.20
0.00
0,00
KrNinfrac
0.28
0.18
0.05
0.03
0.04
0.41
0.09
0.07
0.05
0.09
0.19
0.49
0.33
Greedyrrac
1.00
0.52
0.88
0.69
Figure 4: GPT-4 for T 100: per-configuration summary table on the hard MAB instance 10 replicates. Only three GPT-4 configurations not exhibit suffix failures; two of these (BNRND and BSSCD) exhibit uniform-like failures. The final configuration (BSSCO) succeeds.
with =
Buttons, K=5, Delta=0.2, T=100
The axes correspond to two surro- gate statistics, SuffFailFreq and K MinFrac, which represent the strength of the two failure modes (suffix failures and uniform-like failures), and are described in detail in the sequel. Figure 4 displays SuffFailFred MinFrac, GreedyF rac (which measures how similar a method
MinFrac(T) (exploitation fail)
eps=0.99
TS
UCB
0.6
Greedy
Eps Gaee-rr
0.4
GPT-3.5
Llama-2-138
GPT-4
0.2
GREEDY), and additional summary statistics for each GPT-4 configuration in the main set of experiments. These statis- tics reveal that all of the LLM configura- tions, except for GPT-4-BSSCY (the blue star in Figure 3), behave fundamentally differently from the baseline algorithms UCB and TS, and we find that these dif- ferences result in large, persistent drop in performance. Conversely, we find that GPT-4-BSSE --S-CO successfully explores and (hence) converges the best arm.
apa
0.0
0.6 SuffFallFree(T/22) (exploration fail)
Figure 3: Scatter plot summarizing all experiments 100. We plot suffix failures (via SuffFailFreq( vs. uniform-lik- failures (via K MinFrac(T)), Each LLM/configuration pair maps to dot (some dots overlap). The only successful GPT-4 con- figuration (BSSCO) is labeled with star. We also plot G-GREEDY, tracing out the tradeoffs for different
with
3.1 Identifying failures
We now give precise overview of the exploration failures illustrated in Figure 3 and Figure 4, and provide additional results and figures that illustrate failure in greater detail. We focus on GPT-4, as GPT-3.5 and LLAMA2 perform worse (and often much worse) in all experiments; detailed results for GPT-3.5 and LLAMA2 are included in Appendix C. We begin with detailed background on the surrogate statistics, SuffFailF and MinFrac, used to quantify failures in Figures 3 and 4 and beyond, providing evidence that exploration failure-as as quantified by these statistics -resulss in persistent drop in performance.
Suffix failures. Most of the LLM configurations we consider exhibit highly bimodal behavior, whereby large fraction of the replicates choose the best arm very rarely, and a few replicates converge to the best arm extremely quickly. Consistent with this bimodal behavior, we observe large incidence of suffix failures, where the best arm is not selected even once after small number initial of rounds (i.c., in some "time suffix"). Suffix failures are suggestive of long-term failure to explore which cannot be improved by running the algorithm for longer, because, without playing the optimal arm, one cannot acquire information to learn that it is indeed optimal. Such behaviors are qualitatively similar to those of GREEDY and qualitatively very different from those of UCB and Thompson Sampling.
Our surrogate statistic for measuring suffix failures is defined as follows: For an experiment replicate R and round let SuffFail(t, R) be binary variable that if the best arm is never chosen in rounds T). Then let SuffFail SuffFailFredu 1Fre((!) failures manifest in most of our experiments at 100. In the scatter plot in Figure 3, the X-axis plots SuffFailFregie /2) for each LLM configuration, and we find that all but five configurations
mean ( [SufFaal(( R) replicates R)). Suffix

---

Buttons, S-arms, Delta-0.2
SPF448300 GPF4
GPT4-MPNC GPT4-NNN
0.50
UCB-3.0
LCB-1.3
Poddon opicates
Greeder)
Greedy11
The Www.AAWEES www.Aneeeeeddde mouANes
0.55
0.5-
0.50
0.2
5.tt:
10
20
40
30
64
80 [0.1000]
910
20
20
40
50
80
20
60
60
80
100
Plas.000
bast
Stop
Time
Figure 5: Bimodal behavior and suffix failures for GPT-4 with T = 100, same visualizations as in Figure 1. Shown: the basic configuration (BNRNO) and the ablation with temperature 1 (BNRNI).
Buttons, S-arms, Deltam0.2
LCD3
T5
GTT GPT-4NEND
UCB-11 1.
CPT 4 MMD
0.00
GFT4 MAND
0.8
3rouddy
Gruedly1
Thre- con-suereeeee TWANER
1..5
UCB 2.0
Groedy
077
C7T4-N5ND
0.C
0.4
0.4
0.2
300 380
25
30 120 250
200
20
125
Figure 6: Detailed view of uniform-like failures for GPT-4 (the BNRND and BNSND configurations) with T 200. Visualizations are: (Left) suffix failure frequency, (Center) K MinFrac(t) function of and (Right) cumulative time-averaged rewards. These configurations exhibit uniform-like failures but not suffix failures, and uniform-like failures are detrimental to long-term rewards.
have Suf SufFFiiFFee()/2) 2 15%. Recalling the definition of suffix failures, this means that 2 15% of the time, these configurations do not pull the best arm even once in the last half of the rounds.
more detailed view of suffix failures and bimodal behavior can be obtained by focusing on individual LLM configurations. We visualize this for the basic configuration for T 500, and in Figure for GPT-4 (BNRNO and BNRN1) T = 100. In these detailed views, the middle panels plot SuffFaill (uffFailFeeq()) at each time t for the given LLM configurations, as UCB, TS, and GREEDY. We find that these LLM configurations have much higher suffix failure rates than both UCB and TS. Bimodal behavior is visualized in the left panel of each plot, where for each configuration, a large fraction of replicates rarely pulls the best arm. while the remaining fraction almost always pulls the best arm. Because of this bimodal behavior (particularly because constant fraction of replicates by chance almost always pull the best arm), suffix failures are not fully reflected in the total reward plots in the right panels of Figure 5, since the time horizon T 100 is not large enough. However, as mentioned, suffix failures are suggestive of an irrecoverable failure to explore which leads to stark differences in reward for larger T. This is precisely what find at T = 500 in Figure 1, which suggests that suffix failures indeed lead to poor long-term performance.
in Figure 1 (top)
well
Uniform-like failures. Returning to the left panel of Figure 3. we see that three GPT-4 configurations avoid suffix failures. Two of these configurations exhibit a different type of failure, where the LLM selects arms in roughly equal proportions for the entirety of the T rounds and fails to exploit the acquired information to focus on the better arms. We call this a uniform-like failure.
Our surrogate statistic for measuring such failures is defined as follows: For particular experiment replicate R and round t. let fa(t, R) be the fraction of rounds in [1.t] in which a given arm is cho- sen, MinFrac(t, R) ;= min, fa (t, R), and MinFrac(t)) := mean( (Minf raa(f R) replicates R)). Since MinFrac(t) S 1/K, Vt [TT.. we always plot K MinFrac(t)) so as to rescale the range to [0, 1]. Larger MinFr (t) corresponds to more uniform selection of arms at time t. When an LLM's MinFrac(t) does not decrease over time and stays substantively larger than that of the baselines (especially as t approaches the time horizon T), we take it as an indication of a uniform-like failure.
The Y-axis of Figure 3 records K MinFrac([]) for each configuration, where we see that of the three GPT-4 configurations that avoid suffix failures, two configurations have very high MinFrac(T))
relative to UCB and TS (the third configuration is GPT-4-BSSČO, which is successful). These two
7

---

configurations are GPT-4-BNRND and GPT-4-BSSCD, both of which use the distributional output format. We provide a more detailed view of GPT-4-BNRND (as well as GPT-4-BNSND, which also exhibits uniform-like failures, but only differs from in Figure 6, which considers a longer horizon and more replicates (7 = 200 and N 20). The middle panel reveals that K MinFFac(t) does not decrease over time for these LLM configurations, while it does for the baselines. This behavior results in no suffix failures, but leads to much lower reward than the baselines. In particular, we obtain a clear separation in total reward, showing that uniform-like failures indeed result poor long-term performance.
in the use of summarized history)
Generality of the failures. To summarize, Figure 3 shows that all LLM configurations except GPT-4-BaR GPT-4-BSSCO exhibit either a suffix failure or a uniform failure for the hard MAB instance and the buttons scenario. Scatter plots for the other three experiments (i.e., the advertisements scenario and/or the easy MAB instance) are qualitatively similar and are deferred to Appendix C.
The same data, but with attributions specific LLM configurations, are presented for all GPT-4 configurations in Figure 4; analogous tables for other LLMs and experimental settings are given in Appendix C. As it is not instructive to present detailed plots such as Figure 5 for every LLM configuration, Figure 4 summarizes the performance of each configuration with just a few statistics. We include: SuffFail 1 (Freq(T/2) and MinFrac(T), defined above; MedianReward: the rescaled median (over replicates) of the time-averaged total reward; GreedyFrac: the fraction of greedy rounds (where an arm with largest average reward is selected), averaged over the replicates. GreedyFrac is one way to quantify the extent to which a configuration behaves like GREEDY.
j
We now summarize further findings from the scatter plots (Figures 3 and 10) and the summary tables (Figures 11 to 17). First, GPT-4 performs much better than GPT-3.5, and LLAMA2 performs much worse (in particular, the suffix failure frequency for LLAMA2 ranges from that of GREEDY to much larger). Second, we observe that all LLMs are sensitive to small changes in the prompt design. However, the different modifications we consider appear to interact with each other, and it difficult to identify which individual modifications improve performance and which degrade it.
3.2 Investigating successes
On the hard MAB instance, the only configuration in our experiments that avoids both suffix failures and uniform-lik like failures is GPT-4 with the BSSCO prompt design. As can be seen from Figure 4, at T 100, this configuration has no suffix failures, the K MinFrac value is only slightly larger than TS, and the reward is comparable to TS. These statistics suggest that this configuration succeeds.
For more statistically mean- ingful results supporting this claim, we run GPT-4- BSSCO on the hard MAB instance with T = 200 and N = 40. We also con- sider GPT-4 swaps summarized history for raw history, as an abla- tion. Figure 7 summarizes this experiment, while Fig- ure 1(b) provides a detailed view of the BSSČO configu- ration. We see that BSSCO continues to avoid suffix failures and perform relatively well in terms of reward for larger T' On the other hand, the ablation BSRCO exhibits a non-trivial fraction of suffix failures, fundamentally different behavior.
T5
UCB
Greedy
8SRCO
B58C0
MedianReward
0.59
0.70
0.60
0.65
0.54
SufifalFree(T122
0.00
0.02
0.47
0.12
0.03
KtMinfrac
0.23
0.12
0.03
0.11
0.29
which
Groedyfrac
0.66
0.81
1.00
0.75
0.68
Figure 7: Summary statistics of two GPT-4 configurations with rein- forced CoT (BSRCO and BSSCO), on the hard MAB instance with T 200 and N 40 replicates. BSRČO shows suffix failures. BSSČO has neither suffix nor uniform-like failures and reasonable reward.
We provide additional visualizations with some qualitative evidence toward the success of BSSČO, well as the failure of other configurations. In Figure 8, we plot the fraction of rounds in [0, t] where the optimal arm was pulled; we plot this for individual replicates, as function of BSRCO is
"Specifically, let @ be the time- lineraeergged total reward for given replicate H. Then E (+()) ranges over 1//2 2 1/2 2/2). We rescale &e ((R). by translating and multiplying, that E (+()) ranges in [0, 1].
8

---

Buttons, S-arms, Delta=0.2
100
350
300 130
250
200
35
220
320
206
59
106 150
289
26
300
336
209
Figure 8: Per-replicate behavior: two reinforced-Co GPT-4 configurations & the baselines, For each algorithm, replicate and round we plot the fraction of rounds in [0, t] when the best arm was pulled.
visually similar UCB, except that a non-trivial fraction of runs exhibit suffix failures (the curves that converge to 0 on the plot). Meanwhile, BSSCO is visually similar to TS, with almost all replicates slowly converging 1. Another visualization, presented in Appendix D, shows the arm chosen at each time step for particular replicates; this is for several LLM configurations ("successful" and not), as well as the baselines. These visualizations, along with the summary statistics, suggest that BSSCO behaves most similarly to TS, which further suggests similar convergence in the long run.
3.3 Root causes
Why do LLMs behave the way they do? Particularly, can one explain their failures via flaws in their per-round decisions? Two natural hypotheses that the failing LLM configurations either a) too greedy, or b) too uniform-like. Indeed, most GPT-4 configurations behave much like GREEDY on the easy MAB instance; yet, they avoid suffix failures and accrue large rewards, and does GREEDY. However, on the hard instance, most GPT-4 configurations seem to be doing something non-trivial.
A secondary experiment studies this further: Each agent (LLM or baseline) faces "data source" (distribution of bandit histories) and makes a single decision. We used GPT-3.5 and several data sources. We find it difficult to separate LLMs from the baselines based on the per-round performance, as the latter is very sensitive to the data source. While a deeper investigation is needed, we report this difficulty non-trivial finding. All these results discussed in Appendix E.
4 Discussion and open questions
Our investigation suggests that contemporary LLMs do not robustly engage in exploration required for very basic statistical RL and decision making problems, at least without further intervention. Let us identify several natural next steps. First, experiment with other prompts: as in many other settings [61]. small changes to our prompt template might improve performance; but sensitivity to prompt design is already concerning. Second, experiment with few-shot prompting, where the prompt contains examples of exploratory behavior, or use such examples to fine-tune the L.M.M Third, train the LLM to use auxiliary tools, such as a calculator for basic arithmetic or a "randomizer" to correctly sample from a distribution. We emphasize that cost, access to models, and compute pose significant barriers to further study, particularly because of the need to employ long horizons T and many replicates N to obtain statistically meaningful results. this end, we believe that further methodological and/or statistical advancements to enable cost-effective diagnosis and understanding of LLM-agent behavior (c.g., our surrogate statistics) are essential.
Implications for more complex problems. Our focus on simple MAB problems provides a clean and controllable experimental setup to study the exploratory behavior of LLMss. Exploration failures here suggest that similar failures will also occur in more complex RL and decision-making settings. On the other hand, mitigations must be developed with caution, as solutions that succeed for the MAB setting may not generalize to more complex settings. For example, while GPT-4 with summarized interaction history and reinforced CoT seems to successfully explore our MAB setting, is not clear how one should externally summarize the history in settings with complex, high high-dimensional observations such as contextual bandits (see Footnote 1). Indeed, even for linear contextual bandits, the approach may not be applicable without substantial algorithmic intervention (such as, e.g., linear regression computed externally and included in the prompt) and the many explicit modeling and algorithmic interventions essential to understand the extent to which LLMS can operate as decision-make agents.
algorithmic choices involved therein. We believe deeper investigation of

---

References
[1] Jacob Abernethy, Alekh Agarwal, Teodor Marinov, and Manfred K Warmuth. A mechanism for sample-efficient in-context learning for sparse retrieval tasks, arXiv:2305.17040, 2023.
[2] Shipra Agrawal and Navin Goyal. Analysis of Thompson Sampling for the multi-armed bandit problem. In 25nd Conference on Learning Theory, 2012.
[3] Shipra Agrawal and Navin Goyal. Near-optimal regret bounds for thompson sampling. J. of the ACM, 34(5):30:1330222 2017. Preliminary version AISTATS 2013.
[4] Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra. Transformers learn to implement preconditioned gradient descent for in-context learning. arXiv:2306.00299) 2023.
[5] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes. Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Daniel Herzon, Alexand Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang. Rosario Jau- regui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Bhanng-Huci Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. Do as I can, not as I say: Grounding language in robotic affordances. arXiv:220.0016% 2022.
[6] Kabir Ahuja, Madhur Panwar, and Navin Goyal. In-context learning through the bayesian prism. arXiv:2306. 04891, 2023.
[7] Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? Investigations with linear models. urXiv:2211.1566)) 2022.
[8] Ekin Akyürekk, Bailin Wang, Yoon Kim, and Jacob Andreas. In-context language learning: Architectures and algorithms. arXiv:2401.12973, 2024.
[9] Peter Auer, Nicolò Cesa- Bisachi and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine Learning, 47(2-3):235-256, 2002.
[10] Peter Auer, Nicolò Cesa-Bianchi, Yoav Freund, and Robert E. Schapire. The nonstochastic multiarmed bandit problem. SIAM J. Compur., 32(1):48-77 2002. Preliminary version in 36th
IEEE FOCS, 1995.
[11] Yu Bai, Fan Chen, Huan Wang. Caiming Xiong, and Song Mei. Transformers as statisticians: Provable in-context learning with in-context algorithm selection. arXiv:2306.04637 2023.
[12] Kiarash Banihashem, MohammadTagh Hajiaghayi. Suho Shin, and Aleksandrs Slivkins. Bandit social learning: Exploration under myopic behavior. In 37th Advances in Neural Information Processing Systems, 2023.
[13] Satwik Bhattamishra, Arkil Patel, Phil Blunsom, and Varun Kanade. Understanding in-context learning in transformers and LLMs by learning to learn discrete functions. arXiv:2310.030166 2023.
[14] Mareel Binz and Eric Schulz. Using cognitive psychology to understand GPT-3. Proceedings of the National Academy of Sciences, 2023.
[15] Ethan Brooks, Logan A Walls, Richard Lewis, and Satinder Singh. Large language models can implement policy iteration. In Advances in Neural Information Processing Systems, 2023.
[16] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, 2020.
[17) Sébastien Bubeck and Nicolo Cesa-Bine -isacBianchi. Regret Analysis of Stochastic and Nonstochas- tic Multi-armed Bandit Problems. Foundations and Trends in Machine Learning, 5(1): 1-122, 2012. Published with Now Publishers (Boston, MA, USA). Also available at https:/ //tp:i///xvv..v.a httti..or//bs///00/00 2204. 5721.
10

---

[18] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li. Scott Lundberg, Harsha Nori, Hamid Palangi. Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with gpt-4, arXiv:2303. /27/2, 2023.
[19] Yuji Cao, Huan Zhao, Yuheng Cheng, Ting Shu, Guolong Liu, Gaoqi Liang, Junhua Zhao, and Yun Li. Survey on large language model-enhanced reinforcement learning: Concept, taxonomy, and methods. arXiv:2404. 00282, 2024.
[20] Xiang Cheng, Yuxin Chen, and Suvrit Sra. Transformers implement functional gradient descent to learn non-linear functions in context. arXiv:2312.06522 2023.
[21] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems. arXiv:2110.14168, 2021.
[22] Julian Coda-Forno, Marcel Binz, Zeynep Akata, Matt Botvinick, Jane Wang. and Eric Schulz. Meta-in-context learning in large language models. Advances in Neural Information Processing Systems, 2023.
[23] Julian Coda-Forno, Marcel Binz, Jane X Wang, and Eric Schulz. Cogbench: a large language model walks into a psychology lab. arXiv:2402.
2024.
[24] Tim Dettmers and Luke Zettlemoyer. The case for 4-bit precision: k-bit inference scaling laws. In International Conference on Machine Learning, 2023.
[25] Carl N Edwards, Aakanksha Naik, Tushar Khot, Martin D Burke, Heng Ji, and Tom Hope. Synergpt: In-context learning for personalized drug synergy prediction and drug design. wrXiv2337.11194, 2023.
[26] Deqing Fu, Tian-Qi Chen, Robin Jia, and Vatsal Sharan. Transformers learn higher-order optimization methods for in-context learning: A study with linear models. arXiv:2310.17086 2023.
[27] Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang. Jamie Callan, and Graham Neubig. Pal: Program-aides language models. In International Conference on Machine Learning, 2023.
[28] Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems, 2022.
[29] Tianyu Guo, Wei Hu, Song Mei, Huan Wang, Caiming Xiong. Silvio Savarese, and Yu Bai. How do transformers learn in-context beyond simple functions? A case study on learning with representations. arXiv:2310. 10616, 2023.
[30] Michael Hahn and Navin Goyal. A theory of emergent in-context learning implicit structure induction. 230307230 2023.
[31] Chi Han, Ziqi Wang, Han Zhao, and Heng Ji. Explaining emergent in-context learning as kernel regression. ar0iv:2305. /2766, 2023.
[32] Xiaochuang Han, Daniel Simig, Todor Mihaylov, Yulia Tsvetkov, Asli Celikyilmaz, and Tianlu
Wang. Understanding in-context learning via supportive pretraining data. 2rXiv:2336.15091 2023.
[33] William M Hayes, Nicolas Yax, and Stefano Palminteri. Relative value biases in large language models. arXiv:2401. /4530, 2024.
[34] Roee Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors. arXiv:2310.1 15916, 2023.
[35] Chien-Ju Ho, Aleksandrs Slivkins, and Jennifer Wortman Vaughan. Adaptive contract design for crowdsourcing markets: Bandit algorithms for repeated principal-agent problems. of Artificial Intelligence Research, 55:317-359, 2016. Preliminary version appeared in ACM EC 2014.
[36] Yu Huang, Yuan Cheng, and Yingbin Liang. In-context convergence of transformers. arXiv:2310.0224, 2023.
[37] Hong Jun Jeon, Jason D Lee, Qi Lei, and Benjamin Van Roy. An information--heereet analysis of in-context learning, 27X.5:2401.15530 2024.

---

[38] Emilie Kaufmann, Nathaniel Korda, and Rémi Munos. Thompson sampling: An asymptotically optimal finite-tima time analysis. In 23rd International Conference on Algorithmic Learning Theory, pages 199-213, 2012.
[39]
Emre Kiciman, Robert Ness, Amit Sharma, and Chenhao Tan. Causal reasoning and large language models: Opening new frontier for causality. arXiv:2305.00050 2023.
[40] Louis Kirsch, James Harrison, Jascha Sohl-Dickstein, and Luke Metz. General-purper in- context learning by meta-learning transformers. arXiv:22/2.04455 2022.
[41] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 2022.
[42] Michael Laskin, Luyu Wang. Junhyuk Oh, Emilio Parisotto, Stephen Spencer, Richie Steiger- wald, DJ Strouse, Steven Hansen, Angelos Filos, Ethan Brooks, Maxime Gazeau, Himanshu Sahni, Satinder Singh, and Volodymyr Mnih. In-context reinforcement learning with algorithm distillation. arXiv:2210.1422
2022.
[43] Tor Lattimore and Csaba Szepesvári. Bandit Algorithms. Cambridge University Press, Cam- bridge, UK, 2020.
[44] Jonathan N Lee, Annie Xie, Aldo Pacchiano, Yash Chandak, Chelsea Finn, Ofir Nachum, and Emma Brunskill. Supervised pretraining can learn in-context reinforcement learning. arXiv:2306. 14892, 2023.
[45] Peter Lee, Carey Goldberg, and Isaac Kohane. The Al revolution in medicine; GPT-4 and beyond. Pearson, 2023.
[46] Yingcong Li, Muhammed Emrullah Ildiz, Dimitris Papailiopoulos, and Samet Oymak. Trans- formers as algorithms: Generalization and stability in in-context learning. In International Conference on Machine Learning, 2023.
[47] Licong Lin, Yu Bai, and Song Mei. Transformers as decision makers: Provable in-context reinforcement learning via supervised pretraining. arXiv:2310.08566 2023.
[48] Bingbin Liu, Jordan Ash, Surbhi Goel, Akshay Krishnamurthy, and Cyril Zhang. Exposing attention glitches with flip-flop language modeling. Advances in Neural Information Processing Systems, 2024.
[49] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv:2310.02255, 2023.
[50] Eran Malach. Auto-regressive next-token predictors are universal learners. arXiv:2309.066 069999 2023.
[51] Ida Momennejad, Hosein Hasanbeig, Felipe Vieira, Hiteshi Sharma, Robert Osazuwa Ness, Nebojsa Jojic, Hamid Palangi, and Jonathan Larson. Evaluating cognitive maps and planning in large language models with cogeval. arXiv:2309. 15129, 2023.
[52] Giovanni Monea, Antoine Bosselut, Kiant'e Brantley, and Yoav Artzi. LLMs are in-context reinforcement learners. arxiv:2410.053622 2024.
[53] Allen Nie, Yi Su, Jonathan N. Lee, Ed H. Chi, Quoe V. Le, and Chen Minmin. EVOLVE: Evaluating and optimizing LLMs for exploration. arxiv:2410.06238, 2024.
[54] OpenAI. Gpt-4 technical report. urXiv:2303.08774, 2023.
[55] Chanwoo Park, Xiangyu Liu, Asuman Ozdaglar, and Kaiqing Zhang. Do LLM agents have regret? A case study in online learning and games. arX1v:2403.16843, 2024.
[56] Joon Sung Park, Joseph O' OrBrin, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. In Symposiam on User Interface Software and Technology, 2023.
[57] Sharath Chandra Raparthy, Eric Hambro, Robert Kirk, Mikael Henaff, and Roberta Raileanu. Generalization to new sequential decision making tasks with in-context learning. arXiv:2312.03801, 2023.
[58] Allan Raventós, Mansheej Paul, Feng Chen, and Surya Ganguli. Pretraining task diversity and the emergence of bay-bayesian in-context learning for regression. arXiv:2306.1506, 2023,
12

---

[59] Daniel Russo, Benjamin Van Roy, Abbas Kazerouni, Ian Osband, and Zheng Wen. A tutorial on thompson sampling. Foundations and Trends in Machine Learning. 11(1): 1-96, 2018. Published with Now Publishers (Boston, MA, USA). Also available at https://arxizs httpss///aaxii.oogab/////77 wtgg//////////..../ 020338.
[60] Johannes A Schubert, Akshay Jagadish, Marcel Binz, and Eric Schulz. In-context learning agents are asymmetric belief updaters. arXiv:2402.03969, 2024.
[61] Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models' sensitivity to spurious features in prompt design or: How learned to start worrying about prompt formatting. arXiv:2310. 13324, 2023.
[62] Lingfeng Shen, Aayush Mishra, and Daniel Khashabi. Do pretrained transformers really learn in-context by gradient descent? arXiv:2310.088
2023.
[63] Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. arXiv:2303. 10.300311330 /1366, 2023.
[64] Max Simchowitz, Christopher Tosh, Akshay Krishnamurthy, Daniel J Hsu, Thodoris Lykouris, Miro Dudik, and Robert E Schapire. Bayesian decision-making under misspecified priors with applications to meta-learning Advances in Neural Information Processing Systems, 2021.
[65] Aleksandrs Slivkins. Introduction to multi-armed bandits. Foundations and Trends in Machine Learning, 12(1-2):111)) 1111-)):22866, November 2019. Published with Now Publishers (Boston, MA, USA). Also available at ht http://aaawvv...aaaabbbo 1901.07272 Latest online revision: Jan 2022.
[66] Aleksandrs Slivkins, Filip Radlinski, and Sreenivas Gollapudi. Ranked bandits in metric spaces: Learning optimally diverse rankings over large document collections. J. of Machine Learning Research (JMLR), 14(Fcb):399-433)) 436, 2013. Preliminary version in 27th ICML, 2010.
[67] Anirudh Som, Karan Sikka, Helen Gent, Ajay Divakaran, Andreas Kathol, and Dimitra Vergyri. Demonstrations are all you need: Advancing offensive content paraphrasing using in-context learning. arXiv:2310. 10707, 2023.
[68] William R. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3-4):285-294, 1933.
[69] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Maric-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog. Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang. Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.0922. 2023.
[70] Karthik Valmeekam, Matthew Marquez. Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change. Advances in Neural Information Processing Systems: Datasets and Benchmarks Track, 2023.
[71] Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers lear in-context by gradient descent. In International Conference Machine Learning, 2023.
[72] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv:2305.1629. 2023.
[73] Lucas Weber, Elia Bruni, and Dieuwke Hupkes. The ICI. consistency test, arXiv:2311 1200445, 2023.
13

---

[74] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, and Denny Zhou, Chain-of thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 2022.
[75] Noam Wies, Yoav Levine, and Amnon Shashua. The learnability of in-context learning. arXiv:2303.07895, 2023.
[76] Jingfeng Wu, Difan Zou, Zixiang Chen, Vladimir Braverman, Quanquan Gu, and Peter L Bartlett. How many pretraining tasks are needed for in-context learning of linear regression? drXiv:2310.08391, 2023.
[77] Yue Wu, Xuan Tang. Tom Mitchell, and Yuanzhi Li. Smartplay: A benchmark for LLMs as intelligent agents. In International Conference on Learning Representations, 2024.
[78] Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. arXiv:2111.02080, 2021.
[79] Mengdi Xu, Yikang Shen, Shun Zhang, Yuchen Lu, Ding Zhao, Joshua Tenenbaum, and Chuang Gan. Prompting decision transformer for few-shot policy generalization. In International Conference on Machine Learning, 2022.
[80] Mengdi Xu, Peide Huang, Wenhao Yu, Shiqi Liu, Xilun Zhang, Yaru Niu, Tingnan Zhang, Fei Xia, Jie Tan, and Ding Zhao. Creative robot tool use with large language models. AKIv:2310.130635 2023.
[81] Eunice Yiu, Eliza Kosoy, and Alison Gopnik. Imitation versus innovation: What children can do that large language and language-and-vision models cannot (yet)? 7rXiv:2305.07666 2023.
[82] Dingli Yu, Simran Kaur, Arushi Gupta, Jonah Brown-Cohen, Anirudh Goyal, and Sanjeev Arora. Skill-mix: A flexible and expandable family of evaluations for ai models. 10Xiv:2310.17560 2023.
[83] Ruiqi Zhang, Spencer Frei, and Peter Bartlett. Trained transformers learn linear models in-context. MXXi:2306.00922) 2023.
[84] Yufeng Zhang, Fengzhuo Zhang, Zhuoran Yang, and Zhaoran Wang. What and how does in-context learning learn? bayesian model averaging, parameterization, and generalization. arXiv:2305. 19420, 2023.
14

---

A Related work
This paper belongs to recent body of work that aims to understand the capabilities of LLMs, i.e. what they can and cannot do well, and why. Capabilities that have received considerable attention, but are peripheral to the present paper, include general intelligence [18. 14), causal [39, 81] and mathematical reasoning [21, 49), planning [70, 51, 15], and compositionality [82].
In more detail, our work contributes to the broader literature on capabilities of in-context learning.
Prior studies of in-context learning include theoretical [78, 7. 84, 1, 83, 31, 20, 4, 75, 26. 76, 36, 34, 46, 71, 11, 30, 37] and empirical [28, 40, 6, 32, 58, 73, 13, 29, 62, 8] investigations, though as mentioned in the prequel, the vast majority of this work pertains to in-context supervised learning; in- context reinforcement learning has received far less attention. The small collection of empirical works that study in-context RL [42, 44, 57, 79] focus on models trained from scratch using trajectory data collected from another agent (either an RL algorithm or an expert); theoretically, Lee et al. [44] and later Lin et al. [47] justify this approach with Bayesian meta-reinforcement learning perspective [64], and show that pre-trained transformers can implement classical exploration strategies like Thompson sampling and upper confidence bounds (UCB). However, these works require interventions to the pre-training phase of the language model, and do not study whether existing LLMs exhibit exploration capabilities under standard training conditions.
Perhaps closest to the present paper, Coda-Forno et al. [22] evaluates the performance of in-context learning with GPT-3.5 on two-armed bandit task and an associated meta-learning task. As with our study, they find that GPT-3.5 performs similarly (in fact, slightly worse) than GREEDY; however, they do not consider long enough time horizons to distinguish GREEDY from successful baselines like UCB.
In parallel, there is a rapidly growing line of work that applies LLMs to real-world decision-making applications. Beyond previously mentioned works [63, 72, 45], which consider applications to gaming, programming, and medicine, highlights include Park et al. [56]. who introduce generative agents which simulate human behavior in an open- world environment, Ahn et al. [5]. Xu et al. [80], who develop LLM-enabled robots.
Concurrent work. Two closely related concurrent works [77, 55] also study in-context LLM performance in bandit tasks. Wu et al. [77] considers a battery of tasks that aim to characterize "intelligent agents" with two-armed bandits
specific task of interest. Their bandit experiments differ in several key respects: They consider a very easy MAB instance (with 2 arms and a gap A = 0.6, which is much easier than both of our instances), focus on a single prompt design (similar to our basic prompt), and compare to human players rather than algorithmic benchmarks. These differences lead to very different experimental findings. In particular, they find that GPT-4 performs well on their simple MAB instance, converging very quickly to the best arm, while we find that GPT-4 with similar prompt fails on harder MAB instance. However, their finding is consistent with ours, as we also find that several configurations of GPT-4 do well on the easy MAB instance. As we discuss in Section 3.3, this instance is too simple to provide compelling evidence for principled exploratory behavior.
Park et al. [55] primarily focus on full-information online learning and repeated game settings but also evaluate LLMs in bandit settings. Their experiments differ from ours in two significant ways. First, although some of their data generation protocols are stochastic in nature, they are primarily interested in adversarial settings. Consequently they compare with adversarial bandits baselines and present the history the LLM via importance-weighteee losses [10]. Second, they mostly consider shorter time horizons (T = 25 for bandits and up to 7 = 50 for full-information) In an updated version of their paper (announced on arXiv in Fall 2024), they also include longer horizon experiments of their original settings, where they find that LLMs continue to perform well, as well as experiments with our hard MAB instance with horizon T 100, where they evaluate uniform and suffix failures. Interestingly, they find that both GPT-4 and GPT-40 succeed (with high reward, no suffix failures, and low HinFrac) when using their default prompt which asks for distributional output, chain-of-thoup and which presents the history via importance weighting. They further find that removing importance weighting results failures, specifically, higher MinFrac for GPT-4 and suffix failures for GPT-40. These findings are perhaps consistent with ours: both results highlight that pre-processing the history (either via summarization or via importance weighting) is crucial for eliciting exploratory behavior from LLMs.
15

---

Other concurrent work includes Schubert et al. [60], Hayes et al. [33], Coda-Forno et al. [23] who use in-comtext bandit and other tasks to study whether LLMs exhibit human- maman-like behavior (particularly. biases) in decision making tasks.
We also refer the interested reader to recent survey of methods for using LLMSs in reinforcement learning settings [19].
Follow-up work. Monea et al. [52] and Nie et al. [53] follow up on our results with several new experimental findings. Both works consider contextual bandits (and Nie al. [53] also considers vanilla MAB). and find that LLMs fail to explore without non-trivial interventions. In this sense, these works corroborate our main findings. Further, both works propose interventions that improve LLM exploration. In particular, Monea et al. [52] propose training-free intervention whereby the interaction history subsampled uniformly before it included in the LLM prompt, while Nie et al. [53] consider few-shot prompting and fine-tuming with optimal demonstrations. These interventions improve performance, but are still not competitive with standard algorithmic baselines.
A.1 Further background on multi-armed bandits
Here, provide additional background the multi-armed bandit problem, and on the baseline algorithms used in this paper. Deeper discussion can be found in Bubeck and Desa-Bianchi [17], Slivkins [65], Lattimore and Szepesvári [43].
The UCB algorithm [9] explores by assigning each arm an index, defined as the average reward from the arm so far plus bonus of the form /C/ng,, where C (log T') and na the number of samples from the arm so far. In each round, it chooses an arm with the largest index. The bonus implements the principle of optimism under uncertainty. We use a version of UCB that sets = (a heuristic), which has been observed to have favorable empirical performance [e.g., 66, 35).
Thompson Sampling [68, 59, for a survey] proceeds if the arms' mean rewards were initially drawn from some Bayesian prior. In each round, it computes a Bayesian posterior given the history so far, draws sample from the posterior, and chooses arm with largest mean reward according this sample (i.e., assuming the sample were the ground truth). In our setting, the prior is essentially parameter to the algorithm. We choose the prior that draws the mean reward of each arm independently and uniformly at random from the [0, 1] interval. This is one standard choice, achieving near- optimal regret bounds, as well as good empirical performance [38, 2, 3]. Each arm is updated independently as a Beta- Bernoulli conjugate prior. Further optimizing UCB and Thompson Sampling is non-essential to this paper, as they already perform quite well in our experiments.
Provable guarantees for bandit algorithms are commonly expressed via regret: the difference in expected total reward of the best and the algorithm. Both baselines achieve regret OVVKT log T), which is nearly minimax optimal as a function of T and K. They also achieve a nearly instance- optimal regret rate, which scales as O (k/a log T') for the instances we consider.
The e-GREEDY algorithm (Footnote 6) is fundamentally inefficient in that does not adaptively steer its exploration toward better-perfon setter-performig arms. Accordingly, its regret rate scales as 72/3 (for an optimal setting of 1/3). Fixing such e, regret does not improve for easier instances.
The GREEDY algorithm (Footnote 5) does not explore at all, which causes suffix failures. This is obvious when the algorithm is initialized with single sample (n 1) of each arm: suffix failure happens when the good arm returns 0, and one of the other arms returns 1. However, suffix failures are not an artifact of small n: they can happen for any n, with probability that scales 221/ (1///)) [12].
16

---

B
Prompt designs
neutral framing
suggestive framing
You are a room with 5 buttons labeled blue, green, red, yellow, purple.
You are bandit algorithm a room with 5 buttons labeled blue, green, red, yellow, purple.
Each button is associated with Bernoulii distribution with a fixed but unknown mean; the means for the buttons could be different. For cach button, when you press it, will get reward that is sampled from the button's associated distribution. You have 100 time steps and, each time step, you can choose any button and receive the reward. Your goal is to maximize the total reward over the 100 time steps.
raw history
summarized history
far you have played times with the following choices and rewards:
far you have played times with your past choices rewards summarized as follows:
green button, reward I red button, reward blue button, reward green button, reward 0 yellow button, reward
button: pressed times with average reward 0.2
green button: pressed times with average reward 0.0
red button: pressed 0 times
yellow button: pressed 0 times purple button: pressed times
return: action
return: distribution actions
You must provide final answer within the tags <Answer> COLOR </Answer> where COLOR one blue, green, red, yellow, purple.
You may output distribution over the buttons formatted
EXACTLY like "bheee.grennbb.eecccellooo...ppp
final prompt: reply-only
final prompt: chain-of-thought
Which button will you choose next? You must provide your final answer immediately <..> and with no text explanation.
Let's think step by step to make sure make good choice. Which button will you choose next? You must provide your final answer S..>
Figure 9: Prompt designs with text in the buttons scenario, expanding Figure 2.
17

---

B.1 Prompt examples
Let us present three full examples of our prompts. We remove the blank lines for the sake of readability.
(a) Our basic prompt design (i.e., BNRN, as temperature is specified elsewhere): the buttons scenario with neutral framing and raw histories, asking the LLM to return an action without chain-of -f-hhugghtt reasoning.
[SYSTEM] You are in a room with 5 buttons labeled blue, green, red, yellow, purple. Each button is associated with a Bernoulli distribution with fixed but unknown mean; the means for the buttons could be different. For each button, when you press it, you will get a reward that sampled from the button's 's associated distribution. You have 10 time steps and, on each time step, you can choose any button and receive the reward. Your goal to maximize the total reward over the 10 time steps.
At each time step, will show you your past choices and rewards. Then you must make the next choice, which must be exactly one of blue, green, red, yellow, purple. You must provide your final answer immediately within the tags <An- Swer>COLORR</AnSe
where COLOR is one of blue, green, red, yellow, purple and with text explanation.
[USER] So far you have played times with the following choices and rewards: blue button, reward green button, reward (
Which button will you choose next? Remember, YOU MUST provide your final answer within the tags <Answer>COLLOR<Anweee where COLOR is one of blue, green, red, yellow, purple.
(b) The adverts scenario with all the "advanced" modifications (i.e., ASSCD) : suggestive framing, summarized histories, asking the LLM return distribution over actions, and asking for a chain-of- thought reasoning.
[SYSTEM] You are recommendation engine that chooses advertisements to display when they visit your webpage. There 5 advertisements you can choose from, named A, B, C, D, F. When a user visits the webpage you can choose an advertisement to display and you will observe whether the user clicks the ad not. You model this by assuming that each advertisement has a certain click rate and users click on advertisements with their corresponding rates.
You have a budget of 10 users to interact with and your goal to maximize the total number of clicks during this process.
A good strategy optimize for clicks in these situations requires balancing explo- ration and exploitation. You need to explore to try out all of the options and find those with high click rates, but you also have to exploit the information that you have to accumulate clicks.
When each user visits the webpage, will show you a summary of the data you have collected far.
Then you must choose which advertisement to display. You may output a distribu- tion over the 5 choices formatted EXACTLY like "A:nl 2::n,,,:n3,,D... C:n4,,Dn4,.EnnS
Let's think step by step make sure we make good choice. Then, you must provide your final answer within the tags &AnswersDIST</Answee where DIST is the distribution in the format specified above.
[USER] So far you have interacted with users. Here is summary of the data you have collected:
Advertisement A was shown to 1 users with an estimated click rate of 1.00 Advertisement was shown to users with an estimated click rate of 0.00 Advertisement C has not been shown Advertisement D has not been shown Advertisement E has not been shown
18

---

Which advertisement will you choose next? Remember, YOU MUST provide your final answer within the tags <Answer>DiST<Annswer>> where DIST is formatted like
(c) The successful configuration for GPT-4 (i.e., BSSC, temperature is specified elsewhere), which uses the buttons scenario, suggestive framing, summarized histories, and reinforced chain-of-thought reasoning.
[SYSTEM] You are bandit algorithm in room with 5 buttons labeled blue, green, red, yellow, purple. Each button is associated with Bernoulli distribution with a fixed but unknown mean; the means for the buttons could be different. For each button, when you press it, you will get a reward that is sampled from the button's 's associated distribution. You have 10 time steps and, each time step, you choose any button and receive the reward. Your goal is to maximize the total reward over the 10 time steps.
each time step, will show you a summary of your past choices and rewards. Then you must make the next choice, which must be exactly one of blue, green, red, yellow, purple. Let's think step by step to make sure we make a good choice. You must provide your final answer within the tags cAnsver>CoLoR<<Ansverr where COLOR one of blue, green, red, yellow, purple
[USER] So far you have played 2 times with your past choices and rewards summarized as follows:
blue button: pressed times with average reward 1.00
green button: pressed times with average reward 0.00 red button: pressed () times
yellow button: pressed 0 times
purple button: pressed 0 times
Which button will you choose next? Remember, YOU MUST provide your final answer within the tags <Answe>COLLRR</nswee where COLOR is one blue, green, red, yellow, purple. Let's think step by step to make sure we make a good choice.
19

---

C Scatter plots and summary tables
Buttons, K=5, Delta=0.2, T=100
Advertisements, K=5, Delta=0.2, T=100
* UCB
0.6
0.6
Greedy
Greedy
K MinFrac(T)
Eps- Greedy
Eps-Greede
0.4
GPT-3.5
0.4
GPT-3.5
Llama-2-13b
Llama-2-13b
GPT-4
GPT-4
0.2
0.2
0.0
0.2
0.6
0.8
1.0
0.0
0.4
0.6
0.8
1.0
Buttons, Delta=0.5,
Advertisements, K=4, Delta=0.5, T=100
0.8
0.8
TS
TS
UCB
UCB
Greedy
Greedy
Min
Eps- Greedy
Eps-Greedy
GPT-3.5
GPT-3.5
0.4
GPT-4
0.4
GPT-4
0.2
0.2
0.0
0.0
0.1
0.2
0.3 SuffFalIFre((/22)
0.4
0.5
0.6
0.0
0.2
0.4
0.6
SuffFallFreq(T12)
Figure 10: All scatter plots for the main experiments (T 100): suffix failures vs. uniform-lide like failures. Specifically: SuffFailFreq(T/2) vs K MinFrac(T)). Each LLM//onfigurration pair maps to dot on this plane. (However, some dots may be hidden by some others.) We also plot e-GREEDY, tracing out the different tradeoffs obtained for different values of c.
20

---

(a) Hard MAB instance (A 0.2), buttons scenario, N
10 replicates.
UPCB
Greedy
BARNN
BARNI
BNNWO
BNACC
BNSN6
BSSRN
BSSCO
BSSC1
BSSCD
85820
Meciankkweed
0.47
0.55
0.40
0.63
0.70
0.33
0.35
0.60
0.45
0.68
0.28
0.37
0.47
SutfaalheetTiz)
FrMi
0.28
0.18
0.05
0.03
0.04
0.41
0.09
0.07
0.05
0.09
0.19
0.49
0.33
Greedyfrac
0.62
0.76
1.00
0.52
0.46
0.45
0.78
0.59
0.93
0.88
0.49
0.69
1000
10
(b) Hard MAB instance (A = 0.2), advertisements scenario, = 3 replicates.
T5
UCB
Greedy
ANRNO
ANNN1
ANRND
ANFCO
ANENO
ASANO
ASSCo
ASSC1
ASSCD
ModianReward
0.47
0.35
0.40
0.00
-0.05
-0.15
0.35
0.40
0.45
0.15
0.60
-0.15
0.67
1.00
f-Minfrac
0.05
Groedyfrac
0.62
0.76
1.00
0.47
0.23
1.00
0.86
0.99
0.91
0.68
0.70
1.00
Replicates
(c) Easy MAB instance (Á 0.5), buttons scenario, N = replicates.
TS
UCB
Greedy
BM
BNRNI
BNRND
BNACO
BNSK
BSAND
895CO
BBSC1
BSSCD
Medlankeward
0.84
0.88
0.92
0.90
0.92
0.56
0.92
0.96
0.92
0.92
0.90
0.58
SufFalFroel122
0.00
0.00
0.19
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
GroedyFrac
0.88
0.94
1.00
0.97
0.99
0.56
0.99
1.00
0.73
0.99
0.93
0.63
Replicates
1000
1000
1000
3
3
(d) Easy MAB instance (A 0.5), advertisements scenario, N 3 replicates.
Greedy
Median&eward
SUffFalProeT113
0.00
0.00
0.19
0.33
0.33
0.67
0.00
0.33
0.00
0.00
0.00
0.67
KnMin Frac
GreedyPrac
0.88
0.94
1.00
0.81
0.95
1.00
0.94
1.00
0.96
0.81
0.73
1.00
Reglicates
1000
1000
1000
3
3
3
3
Figure II: GPT-4 for = 100: the per-configuration summary tables. The "fails" row indicates that all replicates completed successfully.
21

---

MedianReward
Sufffallfre((T12) 0.01
K*MinFrac
GreedyFrac
Replicates
TS
0.47
0.28
0.62
1000
UCB
0.55
0.02
0.18
0.76
1000
Greedy
0.40
0.48
0.05
1.00
1000
BNRNO
0.22
0.50
0.16
0.30
20
BNRN1
0.22
0.00
0.41
0.28
20
BNRND
0.12
0.55
0.07
0.40
20
BNRCO
0.12
0.80
0.01
0.51
20
BNRC1
0.10
0.50
0.03
0.57
20
BNRCD
0.65
0.45
0.01
0.75
20
BNSNO
0.12
0.85
0.00
1.00
20
BNSN1
0.22
0.25
0.04
0.76
20
BNSND
0.20
0.20
0.52
0.38
20
BNSCO
0.12
0.85
0.00
0.95
20
BNSC1
0.22
0.70
0.01
0.88
20
BNSCD
0.05
0.50
0.11
0.50
20
BSRNO
0.17
0.30
0.25
0.32
20
BSRN1
0.25
0.00
0.66
0.29
20
BSRND
0.42
0.25
0.12
0.33
20
BSRCO
0.10
0.65
0.03
0.44
20
BSRC1
0.05
0.25
0.12
0.47
20
BSRCD
0.28
0.15
0.11
0.60
20
BSSNO
0.12
0.85
0.00
1.00
20
BSSN1
0.25
0.30
0.03
0.78
20
BSSND
0.25
0.15
0.45
0.42
20
BSSCO
0.17
0.85
0.00
1.00
20
BSSC1
0.17
0.55
0.02
0.83
20
BSSCD
0.20
0.35
0.10
0.78
20
Figure 12: GPT-3.5 for T 100: the per-configuration summary table. The buttons scenario, hard MAB instance.
22

---

Rewian
SufffailFreq(T/22
K*MinFrac
GreedyFrac
Replicates
TS
0.47
0.01
0.28
0.62
1000
UCB
0.55
0.02
0.18
0.76
1000
Greedy
0.40
0.48
0.05
1.00
1000
ANRNO
0.22
0.65
0.03
0.48
20
ANRN1
0.22
0.50
0.05
0.33
20
ANRND
0.15
0.70
0.00
1.00
20
ANRCO
0.15
0.85
0.00
0.98
20
ANRC1
0.20
0.50
0.00
0.80
20
ANRCD
0.15
0.70
0.00
1.00
20
ANSNO
0.12
0.85
0.00
1.00
20
0.12
0.20
0.04
0.93
20
ANSND
0.15
0.70
0.00
1.00
20
ANSCO
0.17
0.80
0.00
1.00
20
ANSC1
0.12
0.55
0.01
0.93
20
0.15
0.70
0.00
1.00
ASRNO
0.25
0.70
0.03
0.48
20
ASRN1
0.05
0.42
0.06
0.28
20
ASRND
0.15
0.70
0.00
1.00
20
ASRCO
0.37
0.40
0.06
0.64
ASRC1
0.30
0.25
0.11
0.65
20
ASRCD
0.15
0.70
0.00
1.00
20
ASSNO
0.15
0.85
0.00
1.00
20
ASSN1
0.25
0.42
0.05
0.92
ASSND
0.15
0.70
0.00
1.00
20
ASSCO
0.12
0.80
0.01
0.99
20
ASSC1
0.30
0.15
0.14
0.83
20
ASSCD
0.15
0.70
0.00
1.00
20
Figure 13:
GPT-3.5 for T
100: the per-configuration summary table. The advertisements
scenario, hard MAB instance.
23

---

RedianReward
Sufffallfrec(T12)
K*MinFrac
GreedyFrac
Replicates
TS
0.84
0.00
0.14
0.88
1000
UCB
0.88
0.00
0.09
0.94
1000
Greedy
0.92
0.19
0.04
1.00
1000
BNRNO
0.23
0.55
0.02
0.85
20
BNRN1
0.72
0.05
0.16
0.62
20
BNRND
0.14
0.25
0.17
0.46
20
BNRCO
0.84
0.25
0.03
0.56
20
BNRC1
0.81
0.05
0.08
0.77
20
BNRCD
0.88
0.10
0.04
0.92
20
BNSNO
0.18
0.65
0.00
1.00
20
BNSN1
0.60
0.40
0.02
0.89
20
BNSND
0.26
0.10
0.54
0.52
20
BNSCO
0.18
0.65
0.00
1.00
20
BNSC1
0.16
0.55
0.01
0.95
20
BNSCD
0.62
0.35
0.03
0.77
20
BSRNO
0.73
0.30
0.11
0.57
20
BSRN1
0.35
0.00
0.48
0.42
20
BSRND
0.21
0.25
0.09
0.43
20
BSRCO
0.87
0.05
0.06
0.72
20
BSRC1
0.73
0.05
0.16
0.72
20
BSRCD
0.81
0.05
0.11
0.76
20
BSSNO
0.18
0.65
0.00
1.00
20
BSSN1
0.17
0.25
0.02
0.89
20
BSSND
0.26
0.30
0.39
0.60
BSSCO
0.19
0.60
0.00
0.99
20
BSSC1
0.53
0.35
0.03
0.82
20
BSSCD
0.78
0.25
0.02
0.90
20
Figure 14: GPT-3.5 for T 100: the per-configuration summary table. The buttons scenario, easy MAB instance.
24

---

RedianReward
Sufffallfrec(T/22
K*MinFrac
GreedyFrac
Replicates
TS
0.84
0.00
0.14
0.88
1000
UCB
0.88
0.00
0.09
0.94
1000
Greedy
0.92
0.19
0.04
1.00
1000
ANRNO
0.18
0.65
0.01
0.81
20
ANRN1
0.10
0.35
0.03
0.47
20
ANRND
0.10
0.55
0.00
1.00
20
ANRCO
0.13
0.60
0.00
0.96
20
ANRC1
0.77
0.35
0.03
0.89
20
ANRCD
0.10
0.55
0.00
1.00
20
ANSNO
0.18
0.65
0.00
1.00
20
ANSN1
0.69
0.15
0.03
0.97
20
ANSND
0.10
0.55
0.00
1.00
20
ANSCO
0.23
0.60
0.00
1.00
20
ANSC1
0.71
0.20
0.03
0.96
20
ANSCD
0.10
0.55
0.00
1.00
20
ASRNO
0.08
0.75
0.01
0.81
20
ASRN1
0.08
0.45
0.05
0.40
20
ASRND
0.10
0.55
0.00
1.00
20
ASRCO
0.68
0.10
0.08
0.86
20
ASRC1
0.74
0.00
0.13
0.86
20
ASRCD
0.10
0.55
0.00
1.00
20
ASSNO
0.29
0.00
0.04
0.92
20
ASSN1
0.79
0.10
0.05
0.93
20
0.10
0.55
0.00
1.00
20
ASSCO
0.89
0.20
0.01
1.00
20
ASSC1
0.82
0.10
0.11
0.92
20
ASSCD
0.10
0.55
0.00
1.00
20
Figure 15: GPT-3.5 for T 100: the per-configuration summary table. The adverts scenario, easy MAB instance.
25

---

Rewaad
(ufffailFreq(1/22)
K*MinFrac
GreedyFrac
Replicates
0.47
0.01
0.28
0.62
1000
UCB
0.55
0.02
0.18
0.76
1000
Greedy
0.40
0.48
0.05
1.00
1000
BNRNO
-0.05
0.90
0.00
1.00
10
BNRN1
0.07
0.90
0.00
1.00
10
BNRCO
0.10
0.80
0.01
0.62
10
BNRC1
0.28
0.90
0.00
0.89
10
BNSNO
0.60
0.50
0.00
1.00
10
BNSN1
0.60
0.50
0.00
1.00
10
BNSCO
0.07
1.00
0.00
1.00
10
BNSC1
0.47
0.60
0.00
1.00
10
BSRNO
-0.03
0.90
0.00
1.00
10
BSRN1
-0.08
1.00
0.00
0.93
10
BSRCO
0.10
0.80
0.01
0.72
10
BSRC1
-0.08
1.00
0.01
0.67
10
BSSNO
0.60
0.50
0.00
1,00
10
BSSN1
0.60
0.50
0.00
1.00
10
BSSCO
0.07
1.00
0.00
1.00
10
BSSC1
0.22
0.90
0.00
1.00
10
Figure 16: LLAMA2 for T 100: the per-configuration summary tables. The buttons scenario, hard MAB instance.
26

---

RewianReward
(ufffai/Freq(1/22)
K*MinFrac
GreedyFrac
Replicates
T5
0.47
0.01
0.28
0.62
1000
UCB
0.55
0.02
0.18
0.76
1000
Greedy
0.40
0.48
0.05
1.00
1000
BNRNO
-0.05
0.90
0.00
1.00
10
BNRN1
0.07
0.90
0.00
1.00
10
BNRCO
0.10
0.80
0.01
0.62
10
BNRC1
0.28
0.90
0.00
0.89
10
BNSNO
0.60
0.50
0.00
1.00
10
BNSN1
0.60
0.50
0.00
1.00
10
BNSCO
0.07
1.00
0.00
1.00
10
BNSC1
0.47
0.60
0.00
1.00
10
BSRNO
-0.03
0.90
0.00
1.00
10
BSRN1
-0.08
1.00
0.00
0.93
10
BSRCO
0.10
0.80
0.01
0.72
10
BSRC1
-0.08
1.00
0.01
0.67
10
BSSNO
0.60
0.50
0.00
1,00
10
BSSN1
0.60
0.50
0.00
1.00
10
BSSCO
0.07
1.00
0.00
1.00
10
BSSC1
0.22
0.90
0.00
1.00
10
Figure 17: LLAMA2 for T' scenario, hard MAB instance.
100: the per-configuration summary tables. The advertisements
27

---

D Investigating successes: additional visualization for Subsection 3.2.
We provide an additional visualization for Subsection 3.2, with some qualitative evidence toward the success BSSCO, as well as the failure of other configurations. In Figure 18 we visualize the arm chosen at each time step for various replicates of several different methods (LLMs and baselines). Specifically, we have four replicates for the basic configuration (BNRNO) and the two configurations with reinforced CoT (BSRCO and BSSCO), as well as one replicate of each of the baseline algorithms. We that the basic configuration BNRNO tends to commit behavior that is similar to that of GREEDY and very different from both UCB and TS. BSRČO also commits for long periods, but to a lesser extent than the basic configuration. In contrast, BSSCO switches arms much more frequently, and qualitatively appears much more similar to TS.
single arm for several rounds,
Figure 18: Traces of the arm chosen each time step for (a) 4 of the replicates of the basic configuration (GPT-4-BNRNO) (left four cells in top row), (b) 4 of the replicates of GPT-4-BSRCO (left four cells of the middle row), (c) of the replicates of GPT-4-BSSCU (left four cells of the bottom row), as well as one replicate of GREEDY (red border), UCB (green border) and TS (orange border). For each of the T 100 time steps (X-axis) we indicate which of the five arms was chosen (Y-axis). The best arm is the top row of each plot, highlighted with blue boxes.
28

---

E
Root causes
Our experimental findings above shed light on how
GreedyFrac
LeastFrac
LLM- decision making agents behave, but it is also worthwhile to understand why they behave the
TS
0.60
0.54
0.53
0.30
0.12
0.12
UCB
0.84
0.66
0.55
0.46
0.09
0.26
way they do (and particularly, why they fail). This question rather challenging to answer decisively, but two natural hypotheses are that the configurations we consider (outside of GPT-4-BSSCO) are either a) too greedy, or b) too uniform-like. In this section, we describe how our experiments offer some insight into this hypotheses.
BNRNO
0.34
0.36
0.50
0.30
0.30
0.24
BNRCO
0.50
0.84
0.58
0.12
0.04
BNSNO
0.82
0.94
0.84
0.28
BSRNO
0.20
0.18
0.22
0.60
0.38
0.38
Data source
Unif
UCB
TS
Unit
UCB
TS
Figure 19: Per-round decisions with some GPT-3.5 configurations. T = 100, histories of length / 30, hard MAB instance.
First, focusing on GPT-4, our experiments reveal qualitatively different behavior between the easy and hard instances (Figure 11(a) and Figure I1(c)). Indeed, the easy instance appears to be much casier; most GPT-4 configurations avoid suffix failures and accrue large rewards this instance, and the GreedyFrac statistic offers a potential explanation as to why. On the easy instance, most GPT-4 con- figurations have very high GreedyFras Fac values, so they behave similarly GREEDY, which performs quite well (even though GREEDY provably fails with small constant probability and, empirically, has many suffix failures this instance)." A plausible hypothesis from this is that GPT-4 performs quite well in low-noise settings, which is precisely when GREEDY also performs well.
A stronger hypothesis would be that most GPT-4 configurations (except perhaps those using reinforced CоT) behave like GREEDY on all instances, but this hypothesis is invalidated by the GreedyFrac statistics for our experiments on the hard instance. On the hard instance, it seems that most GPT-4 configurations are doing something non-trivial (albeit flawed); their behavior is neither completely GREEDY-like nor like uniform-at-randon at-random.
Toward more fine- graingd understanding, we ran a collection of small-scale secondary experiments focusing on the per-round decisions of LLM-agents. The experiments focus on a single round t in bandit problem. Each experiment considers a particular "data source" (a distribution of bandit histories), samples N = 50 bandit histories of length / from this distribution, and presents them to the agents (the LLMs and the baselines) and asks them to output an arm or distribution over arms. We track two statistics for each agent: GreedyFras Frac and LeastFrac, the fraction of replicates in which the agent chose, resp., an empirically best arm so far and least-chosen arm so far. We vary the data source, i.e., the algorithm which generates the history. In particular, we consider histories generated by sampling uniformly at random (Unif) and by running our baselines UCB and TS for rounds.
Results are summarized in Figure 19. Unfortumately, we find that per-round performance of both the LLMs and the baselines is very sensitive to the particular data source. For example, the MinFrac statistic of UCB can vary from as high as 0.46 on histories generated uniformly at random to as low as 0.09 on histories generated by UCB itself. seems plausible to conclude the BNSNO is too greedy while BSRNO is too uniform, but the statistics for the other two LLM configurations (BNRNO of which fail our longitudinal experiments- fall within the reasonable range provided by the baselines. Thus, we find that it is challenging to assess whether LLM agents too greedy or too uniform-like based on per-round decisions, even though these agents behave rather differently from the baselines in the longitudinal experiments.
and BNRCO)
IIndeed, Figure II(c) we see that most GPT-4 configurations have very high GreedyFrac but no suffix failures. Apparently, even a very small amount of exploration suffices for easy instances (and makes a big difference, relative to GREEDY). However. this should not be construed as evidence for the more general and robust exploratory behavior necessary for harder bandit instances.
29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
30

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
31

---

Answer: [Yes]
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cc Bealiicggiies/Codee
wwwiiissoooooiiio for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips. perips.cc/pppi cc/publ
gegide//eoodeee Submissionti ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
32

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics /tt////eurriiii httperrrsss.cppplllc/hhhttsseeee
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work studies the exploration capabilities of the current generation of large language models. such, there no societal impact of this work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
33

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
34

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
35


Reviews:
[{"review_id": "FeqyBeXIWi", "reviewer": "Reviewer_6C6Y", "summary": "This paper investigates whether popular LLMs can engage in exploration in a in-context manner (all experiences are stored as context/prompt). To achieve, the paper deploys LLMs (GPT-3.5, GPT-4, LLAMA2) as agents in multi-armed bandit environments, using various prompt designs to specify the environment description and interaction history entirely in-context. The results show that only GPT-4 with chain-of-thought reasoning and externally summarized interaction history exhibited satisfactory exploratory behavior. Other configurations failed to robustly explore. The authors suggest that non-trivial algorithmic interventions, such as fine-tuning or dataset curation, may be required for LLMs to effectively explore in more complex settings.", "questions": "1. Only one configuration (GPT-4 with chain-of-thought reasoning and an externally summarized interaction history) resulted in satisfactory exploratory behavior. Why this specific configuration was successful and others were not? Are there any insights into the underlying mechanisms?\n2. The paper concludes that external summarization is essential for desirable LLM behavior in simple environments. How do the authors envision scaling this approach to more complex settings where external summarization might be infeasible? Are there any preliminary ideas or experiments that could address this limitation?", "limitations": "I do not find any limitations discussed in the main body.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper investigates an interesting question: can LLMs explore in-context. Human holds the ability to explore in-context, with their self-summarized histories or abstractions. The results may indicate the intelligence embedded in LLMs is still not aligned with human.\n\nThe findings have meaningful implications for the use of LLMs in decision-making tasks (especially when directly employing LLM as the agent). The identification of the need for external summarization and potential training interventions makes sense.", "weaknesses": "As an experimental article, the findings are naive and obvious: the LLMs are not designed for solving decision-making tasks. It is consistent with intuition that LLMs can explore in-context when CoT with summarized history.\n \nI feel that experiment results on more challenging tasks are necessary, such as some textual interactive games. In practical, the scenarios human encounters and explores are more complex than a bandit.\n\nMore discussions about related work are appreciated.\n\nThere lacks a more systematic analysis of which specific prompt design elements are most critical for successful exploration? This could help in understanding the sensitivity of LLMs to prompt variations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> *As an experimental article, the findings are naive and obvious: the LLMs are not designed for solving decision-making tasks. It is consistent with intuition that LLMs can explore in-context when CoT with summarized history.*\n\nWhether LLMs are “designed” for decision-making tasks is not very relevant to our study. LLMs are already being deployed as decision-making agents (e.g. [40, 54, 63, 48, 5, 71]), and so it is crucial to understand their capabilities as such. Besides, LLMs are not explicitly “designed” for many tasks that they are good at.\n\nWe do not understand how LLMs not being designed for (or failing at) decision-making is consistent with the success of CoT and history summarization. Indeed, it was unclear if CoT would help much and LLMs are quite capable of performing the simple arithmetic required for history summarization.)\n\nRe. “naive and obvious”: we emphasize that the scale of our experiments (as discussed in Section 2) was already near-prohibitive given the LLM costs, but still insufficient with the standard statistics (cumulative rewards), and hence necessitated a considerable subtlety (e.g., tracking suffix and uniform-like failures). The “prompt space” also required a somewhat careful design.\n\n> *I feel that experiment results on more challenging tasks are necessary, such as some textual interactive games. In practical, the scenarios human encounters and explores are more complex than a bandit.*\n\nReal-world decision-making problems are indeed much more complicated than ours. However we emphasize that our MAB setting distills the essence of exploration (as justified by decades of theoretical research) and embeds as a special case into essentially all RL/decision-making formulations. If an agent cannot solve a simple task, what hope does it have for more complex ones? \n\nAlso: simpler settings require the experimenter to make fewer arbitrary choices, and can lead to more generalizable findings. \n\n> *More discussions about related work are appreciated.*\n\nYou might have missed our detailed related work section, deferred to Appendix A due to page limit. Nevertheless, we’d be happy to add any specific references you’d like.\n\n> *There lacks a more systematic analysis of which specific prompt design elements are most critical for successful exploration? This could help in understanding the sensitivity of LLMs to prompt variations.*\n\nWe already consider a non-trivial space of prompt designs, see Fig2. (And we note that it is at least comparable to that considered in concurrent work on bandit tasks for LLMs.) \n\nWe totally agree a more systematic analysis of prompt designs could be very useful. However, it is also extremely challenging because the prompt space (even just for simple bandit problems) is already massive. It might be more tractable to do this in simpler (non-sequential) settings, but LLM sensitivity suggests it is difficult to generalize the findings. \n\n> [Q1] *Only one configuration (GPT-4 with chain-of-thought reasoning and an externally summarized interaction history) resulted in satisfactory exploratory behavior. Why this specific configuration was successful and others were not? Are there any insights into the underlying mechanisms?*\n\nAn advanced LLM (GPT-4) and summarization are obvious benefits. The latter avoids occasional arithmetic errors that LLMs are known to make (and we found in our logs). CoT is known to help in many other tasks, but the reason is unobvious. One hypothesis (from other work) is that CoT gives the model access to more compute, which improves reasoning capabilities. In our experiment logs,  CoTs often did describe correct algorithm behavior, like the correct posterior or upper confidence bound computations. We can describe this in more detail in the final version.\n\n> [Q2] *The paper concludes that external summarization is essential for desirable LLM behavior in simple environments. How do the authors envision scaling this approach to more complex settings where external summarization might be infeasible? Are there any preliminary ideas or experiments that could address this limitation?*\n\nThis is a fundamental open question for future work. One perhaps naive idea is via “orchestration”: we invoke the LLM once and ask it to summarize the history, then we use this “self-produced” summarization in a second invocation where we request a decision. While not provably correct, such summaries seem simple enough to experiment with. \n\n> *The results may indicate the intelligence embedded in LLMs is still not aligned with human.* \n\nTrue. However, we focus on comparing LLMs to algorithms rather than humans.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wmtmhFFqNQ", "reviewer": "Reviewer_yw69", "summary": "This paper investigates the exploration capabilities of contemporary large language models. The authors use LLMs as agents in multi-armed bandit environments, describing the environment and interaction history in-context without any training interventions. Their experiments involve a variety of configurations and models, and they conclude that only GPT-4 with chain-of-thought reasoning and an externally summarized interaction history demonstrates satisfactory exploratory behavior.", "questions": "- Why isn't parameter tuning performed for the baselines with tunable parameters?\n- In line 97, it is mentioned that all the arms, except for the best arm, have the same mean reward. What is the reason for this choice?\n- If the summary is not available for more complex situations, can few-shot examples help instead?", "limitations": "- Longer horizons T and replicates N might be required to obtain statistically significant results.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper comprehensively analyzes the LLMs in various configurations and prompt choices to understand their exploratory capabilities.", "weaknesses": "- In line 161, it is mentioned that no parameter tuning is performed for the baselines with tunable parameters. In this case, we are not sure that the baselines are performing to their highest capacity and therefore, the comparison might not be fair.\n- Longer horizons T and replicates N might be required to obtain statistically significant results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> *In line 161, it is mentioned that no parameter tuning is performed for the baselines with tunable parameters. In this case, we are not sure that the baselines are performing to their highest capacity and therefore, the comparison might not be fair.”* (Also, Q1)\n\nThis is rather standard in bandit experiments. It puts the baselines at a disadvantage, and even then they (for the most part) outperform the LLMs. So even without parameter tuning there is strong evidence that LLMs do not effectively explore. \n\n> *Longer horizons T and replicates N might be required to obtain statistically significant results.*\n\nThis is a fair concern. As we discuss in Section 2 (\"Scale of the experiments\"), our scale (T x N x #configs x #instances) is near-prohibitive given the high cost of invoking modern LLMs, yet insufficient if one relies on standard statistics such as accumulated rewards. This necessitates a more subtle study focusing on surrogate statistics focusing suffix and uniform-like failures. \n\nWe note that most/all follow-up or concurrent studies of LLMs in bandit problems were smaller-scale (e.g. [68]; [1-6] mentioned by Reviewer Jht5; Wu et al, \"A benchmark for LLMs as intelligent agents\".ICLR'24; Park et al., \"Do LLM agents have regret?A case study in online learning and games. arXiv:2403.16843, 2024). \n\n> [Q2] *In line 97, it is mentioned that “all the arms, except for the best arm, have the same mean reward”. What is the reason for this choice?*\n\nThis choice (a) is the standard lower bound construction for MABs, and (b) has the fewest degrees of freedom, leading to a simpler and more defensible experiment setup. Of course other designs are also reasonable. \n\n> [Q3] *If the summary is not available for more complex situations, can few-shot examples help instead?*\n\nWe suspect so, and mention few-shot prompting as a natural direction for future work in Section 4.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1iCioZtlsa", "reviewer": "Reviewer_qwyY", "summary": "This paper investigates whether large language models (LLMs) like GPT-3.5, GPT-4, and LLAMA2 can perform exploration in reinforcement learning settings without additional training. The authors focus on in-context learning, where the environment description and interaction history are provided entirely within the LLM prompt. They evaluate LLMs on multi-armed bandit problems, comparing their performance to standard bandit algorithms like Thompson Sampling (TS) and Upper Confidence Bound (UCB).\n\nThe key findings are: (1) Most LLM configurations fail to robustly explore, exhibiting either \"suffix failures\" (converging to a suboptimal arm) or \"uniform-like failures\" (selecting arms uniformly at random). (2) Only one configuration (GPT-4 with chain-of-thought reasoning and externally summarized history) showed satisfactory exploratory behavior comparable to TS. (3) LLM performance is highly sensitive to prompt design and varies significantly across model versions.\n\nThe authors conclude that non-trivial algorithmic interventions may be necessary for LLMs to function effectively as decision-making agents in complex settings.", "questions": "None.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "-  The study uses multiple LLMs, various prompt designs, and compares against established baselines. The use of both easy and hard multi-armed bandit instances provides a comprehensive evaluation.\n- The introduction of \"suffix failure frequency\" and \"MinFrac\" as surrogate statistics for identifying exploration failures is innovative and allows for more efficient evaluation of LLM performance.\n- The work addresses an important question in the rapidly evolving field of LLM capabilities, with implications for using LLMs in decision-making tasks.", "weaknesses": "- While the focus on multi-armed bandits provides a clean experimental setup, it may not fully represent the challenges of more complex reinforcement learning problems. The generalizability of the findings to broader settings is unclear.\n-  The paper is predominantly empirical. A theoretical framework for understanding why LLMs struggle with exploration could provide deeper insights and guide future research.\n- While the authors attempt to explore reasons for LLM failures in Section 3.3, this analysis feels underdeveloped. A more systematic investigation of failure modes could yield valuable insights.\n- While the authors compare summarized vs. raw history, more systematic ablations of different prompt components could help isolate the factors contributing to successful exploration.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> *While the focus on multi-armed bandits provides a clean experimental setup, it may not fully represent the challenges of more complex reinforcement learning problems. The generalizability of the findings to broader settings is unclear.*\n\nFailures on a fundamental special case such as MAB plausibly carry over to the broader scenarios. But we completely agree for the positive findings. However, this would be the case for any experiment setup: the findings do not necessarily generalize beyond the precise experiment setting. Among all possible experiment setups, we argue that simple/clean/fundamental special cases are preferable. MAB is precisely such a setup. \n\n> *The paper is predominantly empirical. A theoretical framework for understanding why LLMs struggle with exploration could provide deeper insights and guide future research.*\n\nIndeed! But we view it as a next step in this line of research. Our contribution is the prerequisite to this: we identify that there is a need for such a framework. Nevertheless, we hope that the suffix failure and MinFrac statistics provide some hint as to how one might develop a deeper understanding. \n\n> *While the authors attempt to explore reasons for LLM failures in Section 3.3, this analysis feels underdeveloped. A more systematic investigation of failure modes could yield valuable insights.*\n\nAgreed! We included section 3.3 primarily to point out the difficulty of such investigation, particularly of using  single-round experiments to isolate long-horizon issues. We can emphasize this in a revision. \n\n> *While the authors compare summarized vs. raw history, more systematic ablations of different prompt components could help isolate the factors contributing to successful exploration.*\n\nAgain we agree. For this we were primarily limited by cost for GPT-4. Note that we have reported results for all 24 prompt configurations for GPT-3.5-Turbo in the appendix, so “systematic ablations” on a weaker LLM are included in the manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "i1ck4zMhXQ", "reviewer": "Reviewer_Jht5", "summary": "In this study, the authors investigate the impact of diverse prompt designs, including environmental descriptions, summarizing interaction history, and chain of thought reasoning, on the exploration behavior of three distinct LLMs in a multi-armed bandit task. The results demonstrated that none of the models engaged in exploration without substantial interventions. The only configuration that demonstrated efficacy was the GPT-4 model, which was prompted with change of thought reasoning and an externally summarised interaction history. The authors demonstrate that the typical failure mode for these models is either suffix errors, wherein the LLM fails to select the optimal arm even once in the initial stages, or uniform sampling of all options. In conclusion, the authors raise concerns about the external validity of the working configuration, noting that it may not be scalable to more complex task settings. They also speculate on the potential algorithmic interventions that could be employed to encourage LLMs to explore in such settings.", "questions": "What factors within the model (size, architecture, RLHF, internal representation), its training dataset, and training protocol (at a specific point during training) could be preventing LLMs to fail at exploration? It may be beneficial for the authors to consider investigating some of these factors further, which could potentially enhance the quality of the paper. It would be beneficial to investigate the emergence of exploration behavior in LLMs by examining smaller models, such as GPT-2/LLaMa-2. \n\nIt has been demonstrated that LLM models can be fine-tuned for reinforcement learning tasks, enabling them to function as decision-makers capable of generalizing to novel tasks with high efficiency [1-2]. These findings suggest that, despite their limited capacity for exploration when tested off-the-shelf, LLMs can be effectively fine-tuned to enhance their ability to explore. It would be valuable for the authors to investigate whether fine-tuning LLMs (such as the GPT-2 /LLaMa-2 model) on simple RL tasks could facilitate their exploration capabilities.\n\n[1] Cao, Y., Zhao, H., Cheng, Y., Shu, T., Liu, G., Liang, G., ... & Li, Y. (2024). Survey on large language model-enhanced reinforcement learning: Concept, taxonomy, and methods. _arXiv preprint arXiv:2404.00282_.\n\n[3] T. Carta, C. Romac, T. Wolf, S. Lamprier, O. Sigaud, and P.-Y. Oudeyer, “Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning,” Sep. 2023", "limitations": "The failure mode of an agent following an epsilon greedy policy with a high epsilon value is analogous to that of Suffix Failure. Conversely, KMinFrac is analogous to the failure mode of an agent following a random policy or an epsilon greedy policy with a low epsilon value. Therefore,  I believe the metrics themselves are quite trivial and do not constitute a novel contribution.\n\nRather than relying on a qualitative comparison between LLM's choices and those of alternative bandit algorithms, the authors may find it beneficial to fit the models of exploration to LLM's choices and conduct a more quantitative evaluation [1] \n\n[1] Gershman, S. J. Deconstructing the human algorithms for exploration. Cognition, 173:34–42, 2018.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Whether or not LLMs can explored in context in a reinforcement learning task and if so, how is a question worth investigating and approach to study it including the paradigm used are tried and tested.", "weaknesses": "The research question and paradigm used to study that question have been previously investigated in the literature [1-5]. However, in contrast to previous studies, the authors have conducted a more comprehensive search of prompt designs, included a greater number of options in the multi-armed bandit task, and analyzed the failure modes of the models using two novel measures. Nevertheless, I have significant concerns regarding the work's novelty and its contribution's strength. While the results are interesting, the main contribution of the work is somewhat limited, as the subset of evaluations performed has already been studied by previous works (see questions and limitations for details). \n\nAdditionally, the authors have not adequately cited related works [1-8].\n\n[1] Binz, M., & Schulz, E. (2023). Using cognitive psychology to understand GPT-3. _Proceedings of the National Academy of Sciences_, _120_(6), e2218523120.\n\n[2] Schubert, J. A., Jagadish, A. K., Binz, M., & Schulz, E. (2024). In-context learning agents are asymmetric belief updaters. _arXiv preprint arXiv:2402.03969_.\n\n[3] Hayes, W. M., Yax, N., & Palminteri, S. (2024). Large Language Models are Biased Reinforcement Learners. _arXiv preprint arXiv:2405.11422_.\n\n[4] Coda-Forno, J., Binz, M., Akata, Z., Botvinick, M., Wang, J., & Schulz, E. (2023). Meta-in-context learning in large language models. _Advances in Neural Information Processing Systems_, _36_, 65189-65201.\n\n[5] William M Hayes, Nicolas Yax, and Stefano Palminteri. Relative value biases in large language models. arXiv preprint arXiv:2401.14530, 2024.\n\n[6] Julian Coda-Forno, Marcel Binz, Jane X Wang, and Eric Schulz. Cogbench: a large language model walks into a psychology lab. arXiv preprint arXiv:2402.18225, 2024\n\n[7] Thilo Hagendorff. Machine psychology: Investigating emergent capabilities and behavior in large language models using psychological methods.\n\n[8] Lampinen, A. K., Dasgupta, I., Chan, S. C., Matthewson, K., Tessler, M. H., Creswell, A., ... & Hill, F. (2022). Can language models learn from explanations in context?. _arXiv preprint arXiv:2204.02329_.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> *Additionally, the authors have not adequately cited related works [1-8].*\n\nThanks for the references! We actually became aware of some of these between the submission and now and have already incorporated appropriate discussions into the manuscript. But we’ll be sure to discuss them in a revision.\n\nLet us comment on [1-8], as per your list. \n\n[2,3,5,6] are concurrent, unpublished works (from 2024), and should not be considered as novelty issues. (Note that our paper has appeared on arxiv since March 2024.) We’ll provide a detailed comparison in a revision*.\n \nOf the remaining papers, [1,4] consider bandit tasks at a very small scale (T=10, 2 arms), which is really insufficient to distinguish between good and bad algorithms, and [7,8] have no bandit tasks at all.\n \nAll bandit tasks but [4] focus on LLM vs human comparisons, whereas we focus on LLM vs algorithms. We believe this is an important distinction. \n\n*Two brief notes, though: [2] only experiments with Claude v1.3, which is considered roughly on the level of GPT-3.5 and hence probably not indicative of GPT-4 performance;  [5] only considers bandit tasks at a very small scale.  \n \n> [Q1] *What factors within the model (size, architecture, RLHF, internal representation), its training dataset, and training protocol (at a specific point during training) could be preventing LLMs to fail at exploration? It may be beneficial for the authors to consider investigating some of these factors further, which could potentially enhance the quality of the paper. It would be beneficial to investigate the emergence of exploration behavior in LLMs by examining smaller models, such as GPT-2/LLaMa-2.*\n\nThanks for the suggestion. We did experiment with LLaMA-2 in the paper (results are summarized in Fig 3 and reported with more details in the appendix) and found that it is much worse than GPT-3.5-Turbo, which itself is much worse than GPT-4. In a sense, this already demonstrates that exploration behavior is emergent, and that it perhaps can be improved further by scaling. \n\n> [Q2] *It has been demonstrated that LLM models can be fine-tuned for reinforcement learning tasks, enabling them to function as decision-makers capable of generalizing to novel tasks with high efficiency [1-2]. These findings suggest that, despite their limited capacity for exploration when tested off-the-shelf, LLMs can be effectively fine-tuned to enhance their ability to explore. It would be valuable for the authors to investigate whether fine-tuning LLMs (such as the GPT-2 /LLaMa-2 model) on simple RL tasks could facilitate their exploration capabilities.*\n\nThanks for these references. In our related work section (Appendix A), we discussed some works that train language models from scratch on trajectory data from another agent, resulting in suitable decision-making (and exploratory) behavior. Based on this, it’s not too surprising that LLMs can be fine-tuned to improve their decision-making capabilities; we’ll definitely include the citations. However, it’s worth emphasizing that fine-tuning requires expertise & money and/or infrastructure & access.  Many practitioners are using (or will use) off-the-shelf LLMs for decision-making tasks without fine-tuning, and so it is critical to understand what is possible with standard training. \n\n> *The failure mode of an agent following an epsilon greedy policy with a high epsilon value is analogous to that of Suffix Failure. Conversely, KMinFrac is analogous to the failure mode of an agent following a random policy or an epsilon greedy policy with a low epsilon value. Therefore, I believe the metrics themselves are quite trivial and do not constitute a novel contribution.\nRather than relying on a qualitative comparison between LLM's choices and those of alternative bandit algorithms, the authors may find it beneficial to fit the models of exploration to LLM's choices and conduct a more quantitative evaluation* \n\nBoth approaches seem perfectly reasonable to us but we respectfully disagree that one is “qualitative” while the other is “quantitative.” Our approach is completely quantitative: we suggest quantitative statistics to measure, demonstrate that they correlate strongly with performance (measured via reward over a much longer time scale), and quantitatively compare to state-of-the-art methods. Compared with the suggested modeling approach, our statistics allow for direct measurement of exploratory behavior without imposing any modeling assumptions (which would likely be wrong/erroneous given the erratic behavior of LLMs that we witnessed in our experiments).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thanks for taking the time to review our submission. To summarize our contributions, we perform a systematic analysis of the extent to which LLMs are capable of exploration, a core component of reinforcement learning/decision-making, by deploying LLMs as agents in multi-armed bandit environments. Out of all configurations we tried, we found that only one configuration resulted in satisfactory exploratory behavior. Along the way, we develop several statistics which allow us to measure the degree to which LLMs can explore in a sample-efficient manner.

 We hope our responses have adequately addressed your concerns. If not, we are happy to engage further in the discussion period.

Meta-review: This paper explores whether large language models (LLMs) can engage in in-context exploration using various prompt designs within multi-armed bandit environments. The study found that only GPT-4, when combined with chain-of-thought reasoning and externally summarized interaction history, demonstrated effective exploration, while other configurations failed. 

The reviewers generally acknowledge the paper’s interesting direction and practical value but criticize its lack of deeper insights and innovation in the field. Despite these criticisms, they recognize the potential impact of the paper, which outweighs its weaknesses.
Decision: Accept (poster)