Can large language models explore in-context?

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 15:29:22

Abstract: We investigate the extent to which contemporary Large Language Models (LLMs) can engage in exploration, a core capability in reinforcement learning and decision making. We focus on native performance of existing LLMs, without training interventions. We deploy LLMs as agents in simple multi-armed bandit environments, specifying the environment description and interaction history entirely in-context, i.e., within the LLM prompt. We experiment with GPT-3.5, GPT-4, and Llama2, using a variety of prompt designs, and find that the models do not robustly engage in exploration without substantial interventions: i) Only one configuration resulted in satisfactory exploratory behavior: GPT-4 with chain-of-thought reasoning and an externally summarized interaction history; ii) All other configurations did not result in robust exploratory behavior, including those with chain-of-thought reasoning but unsummarized history. While these findings can be interpreted positively, they suggest that external summarization—which may not be possible in more complex settings—is essential for desirable LLM behavior. We conclude that non-trivial algorithmic interventions, such as fine-tuning or dataset curation, may be required to empower LLM-based decision making agents in complex settings.

Main paper: 
 introduction: 
 In-context learning is an important emergent capability of Large Language Models (LLMs) that enables one to use a pre-trained LLM to solve a problem by specifying the problem description and relevant data entirely in-context, i.e., within the LLM prompt, with no updates to the LLM parameters (Brown et al., 2020) . For example, one can prompt an LLM with numeric covariate vectors and scalar targets and subsequently obtain regression-style predictions from the model by including new covariate vectors in the prompt (Garg et al., 2022) . Perhaps surprisingly, LLMs are not explicitly trained for this behavior; instead the underlying algorithms employed for in-context learning are extracted from the training corpus and emerge at scale. Since its discovery in the GPT-3 model (Brown et al., 2020) , in-context learning has been the subject of a growing body of research. These works include theoretical investigations into the underlying mechanisms (e.g., Xie et al., 2021; Akyürek et al., 2022) , empirical probes (e.g., Garg et al., 2022; Kirsch et al., 2022) , and works leveraging in-context learning in applications (e.g., Xu et al., 2022; Som et al., 2023; Edwards et al., 2023) . This literature predominantly studies in-context learning for prediction or supervised learning tasks, and while theoretical progress is in its infancy, our understanding of how to use in-context supervised learning (ICSL) in practice is rapidly taking shape.Although supervised learning is an important capability, many applications demand the use of ML models for downstream decision making. Thus, in-context reinforcement learning (ICRL) and sequential decision making is a natural next frontier. LLMs are already being used as decision making agents in applications ranging from experimental design in the natural sciences (Lee et al., 2023b) to game playing (Shinn et al., 2023; Wang et al., 2023) , but our understanding-theoretically and operationally-of ICRL is far less developed than for ICSL. To date, we lack a systematic understanding as to whether LLMs can be considered general-purpose decision making agents.Decision making agents must possess three core capabilities: generalization (required for supervised learning), exploration (making decisions that may be suboptimal in the short term for the sake of gathering more information) and planning (to account for long-term consequences of decisions). In this paper, we focus on exploration, the capability to deliberately gather information in order to evaluate alternatives and reduce uncertainty. A recent series of papers (Laskin et al., 2022; Lee et al., 2023a; Raparthy et al., 2023) demonstrates in-context reinforcement learning behavior (including exploration) in transformer models when they are ex-plicitly trained to produce this behavior using data from reinforcement learning agents or expert demonstrations on related tasks. Such training tends to be laborious, expensive, and possibly task-specific. In particular, these findings do not shed light into whether exploratory behavior manifests in general-purpose LLMs obtained via standard training methods, which suggests the following basic question: Do contemporary LLMs exhibit the capability to explore in-context?Contributions. We investigate this question by deploying LLMs as agents in simple synthetic reinforcement learning problems, namely multi-armed bandits (MABs) (Slivkins, 2019; Lattimore & Szepesvári, 2020) , specifying the environment description and interaction history entirely within the LLM prompt. Multi-armed bandits are a classical and well-studied type of RL problem that isolates the tradeoff between exploration and exploitation, i.e., making the best decision given the available data. They are also a fundamental building block toward general sequential decision making; the ability to solve MABs is a prerequisite for more challenging reinforcement learning tasks. Their simplicity, centrality to RL, and focus on exploration versus exploitation make MABs a natural choice for systematically studying the in-context exploration abilities of LLMs.We evaluate the in-context exploration behavior of GPT-3.5 (Brown et al., 2020) , GPT-4 (OpenAI, 2023) , and LLAMA2 (Touvron et al., 2023) in MAB environments, using a variety of prompt designs. In our experiments, we find that only a single configuration (i.e., a prompt design and LLM pair) results in satisfactory exploratory behavior. All other configurations exhibit exploration failures, failing to converge to the best decision (arm) with significant probability. We find that typically this happens due to suffix failures, where the LLM fails to select the best arm even once after some initial rounds (i.e., in some "time suffix"). This scenario is reflected in Figure 1 (a): in particular, GPT-4 with our basic prompt design experiences a suffix failure in > 60% of the replicates. An alternative failure mode we identify is where the LLM behaves "uniformly", selecting all arms nearequally often and failing to narrow down to the better ones.The single configuration thato succeeds in our experiments involves a combination of GPT-4 and an "enhanced" prompt that (a) provides a suggestive hint to explore, (b) externally summarizes the history of interaction into per-arm averages, and (c) asks the LLM to use zero-shot chain-of-thought reasoning (Wei et al., 2022; Kojima et al., 2022) . This configuration is visualized in Figure 1 (b). One can interpret this finding positively: state-of-the-art LLMs do possess the capability to robustly explore, provided that the prompt is carefully designed to elicit this behavior. On the other hand, we find that the same configuration without external summarization fails, which leads to a negative interpretation: LLMs may fail to explore in more complex environments, where externally summarizing the history is a non-trivial algorithm design problem. 1We conclude that while the current generation of LLMs can perhaps explore in simple RL environments with appropriate prompt engineering, training interventions-in the spirit of Lee et al. (2023a) ; Raparthy et al. (2023) -may be required to endow LLMs with more sophisticated exploration capabilities required for more complex settings.Methodology. An underlying technical challenge in assessing LLM capabilities and limitations is that one must search a combinatorially large space of prompt designs while obtaining statistically meaningful results, all while meeting the financial and computational constraints associated with LLMs. Assessing in-context bandit learning is even more challenging because (a) stochasticity in the environment demands a high degree of replication for statistical significance and (b) the sample complexity of learning/exploration demands that even a single experiment involve hundreds or thousands of LLM queries to obtain meaningful effect sizes (i.e., separation between successful and failing methods).To address these issues, our core technical contribution is to identify surrogate statistics as diagnostics for long-term exploration failure. The surrogate statistics we consider characterize long-term exploration failure, yet can be measured at moderate scale with few replicates and short learning horizons, even when the standard performance measure (namely, reward) is too noisy to be useful. 
experimental setup: 
 Multi-armed bandits (MAB). We consider a basic multiarmed bandit variant, stochastic Bernoulli bandits. There are K possible actions (arms), indexed as [K] := {1, . . . , K}. Each arm a is associated with mean reward µ a ∈ [0, 1], which is unknown. An agent interacts with the environment for T time steps, where in each time step t ∈ [T ] the agent selects an arm a t ∈ [K] and receives a reward r t ∈ {0, 1} drawn independently from a Bernoulli distribution with mean µ at . Thus, the MAB instance is determined by the mean rewards {µ a : a ∈ [K]} and the time horizon T . The goal is to maximize the total reward, which roughly corresponds to identifying the best arm: an arm with the highest mean reward. A key feature of the MAB setup is that rewards for arms not chosen by the agent are not revealed, so exploration is necessary to identify the best arm. (a) Top row. GPT-4 with our basic prompt design with zero temperature. The experiment runs for T = 500 rounds, and is replicated N = 20 times, varying environment randomness. This configuration exhibits highly bimodal behavior: a large (> 60%) fraction of replicates choose the best arm only a handful of times and exhibit suffix failures, similar to GREEDY, and very unlike UCB and TS. This is suggestive of a long term failure to explore and, indeed, this configuration underperforms substantially in terms of reward.(b) Bottom row. GPT-4 with a suggestive framing, summarized history, and chain-of-thought with zero temperature. The experiment runs for T = 200 rounds and is replicated N = 40 times. This configuration exhibits a unimodal distribution of plays of the best arm, very few suffix failures, and reward that is comparable to TS.We focus on MAB instances where the best arm has mean reward µ ⋆ = 0.5 + ∆/2 for a parameter ∆ > 0, while all other arms have mean reward µ = 0.5 -∆/2 (so, ∆ = µ ⋆ -µ is the gap between the best and the second-best arm).The main instance we consider has K = 5 arms and gap ∆ = 0.2. We call this the hard instance, as we also consider an easy instance with K = 4 and ∆ = 0.5.2 Prompts. We employ LLMs to operate as decision making agents that interact with MAB instances by prompting them with a description of the MAB problem (including the time horizon T ) and the history of interaction thus far. Our prompt design allows several independent choices. First is a "scenario", which provides a grounding for the decision making problem, positioning the LLM either a) as an agent choosing buttons to press, or b) as a recommendation engine displaying advertisements to users. Second, we specify a "framing" as either a) explicitly suggestive of the need to balance exploration and exploitation, or b) neutral. Third, the history can be presented as a) a raw list over rounds, or it can b) be summarized via number of plays and average rewards of each arm. Fourth, the requested final answer can be a) a single arm, or b) a distribution over arms. Finally, we either a) request the answer only, or b) also allow the LLM to provide a "chain-of-thought" (CoT) explanation.Altogether, these choices lead to 2 5 = 32 prompt designs, illustrated in Figure 2 . More details about the prompt design, including examples, are provided in Appendix D.The most basic prompt design from the options above uses the buttons scenario, neutral framing, and raw history, and requests the LLM to return only an arm with no CoT. Each of the five possible modifications to this prompt can potentially help the LLM, and our experiments evaluate this. For example, both the advertising scenario and suggestive framing might help invoke the LLM's knowledge of bandit algorithms (as bandit algorithms are commonly used in content recommendation). History summarization might help if the LLM cannot reliably summarize history itself (perhaps due to arithmetic errors3 ) and/or does not fully realize that it should. Returning a distribution might help if the LLM can identify a good distribution, but fails to correctly sample from it. Finally, chain-of-thought is known to help in a wide variety of LLM scenarios (Wei et al., 2022; Malach, 2023) , even when used in a zero-shot manner (Kojima et al., 2022) as we do here.Prompts are presented to each LLM using both system and user messages (exposed by all three LLM APIs). The system message presents information about the scenario and framing and prompts the LLM about whether to use CoT and whether (and how) to return a distribution. The user message presents the history and reminds the LLM about how to format its response. For GPT-4 only, we found that prompting the LLM to use CoT in the system prompt did not reliably elicit CoT outputs, so-for GPT-4 only-we also consider a reinforced CoT prompt design that additionally reminds the LLM to use CoT at the end of the user prompt. See Appendix D for examples.LLM configurations. We experiment with three LLMs: GPT-3.5, GPT-4, and LLAMA2. 4 In addition to the prompt variations above, we also consider two choices for the temperature parameter, 0 and 1. A temperature of 0 forces the LLM to be deterministic and therefore isolates the "deliberate" exploration behavior of the LLM itself. A temperature of 1 provides a source of external randomness in the LLM responses, which may or may not result in randomization among the arms. Allowing the LLM to return a distribution instead of a single arm also provides external randomness (as we sample from the returned distribution); to isolate sources of randomness, we do not consider temperature 1 with "return distribution" prompt designs.We refer to the tuple (prompt design, temperature) as the LLM configuration. We identify each configuration with a 5-letter "code" L 1 L 2 L 3 L 4 L 5 , with letters L i denoting the choices:• L 1 : 'B' or 'A' for, resp., buttons or advertisements scenario;• L 2 : 'N' or 'S' for, resp., neutral or suggestive framing;• L 3 : 'R' or 'S' for, resp., raw or summarized history;• L 4 : 'C' or ' C' or 'N' for, resp., chain-of-thought, reinforced CoT, or no CoT.• L 5 : '0', '1' or 'D' for, resp., temperature and returning a distribution (with temperature 0).We refer to "BNRN0" as the basic configuration going forward. Most of our experiments consider the "buttons" scenario, and we use the "advertisements" scenario primarily as a robustness check.For GPT-3.5 and LLAMA2, we do not consider reinforced CoT as it is not required to reliably elicit CoT outputs; thus, we have 48 configurations total for these two LLMs. For GPT-4, we primarily used reinforced CoT, but did experiment with some standard CoT prompt designs; thus, there are 72 configurations total for GPT-4.Baselines. For baselines, we consider two standard MAB algorithms, UCB (Auer et al., 2002) and Thompson Sampling (TS) (Thompson, 1933) , which are optimal in a certain theoretical sense and also reasonably effective in practice. We also consider the GREEDY algorithm, which does not explore and is known to fail.5 While all three baselines have tunable parameters, we perform no parameter tuning (see Appendix A.1 for a detailed description of each algorithm with parameter settings).In addition to these baselines, some of our experiments include the the ϵ-GREEDY algorithm 6 with various choices of ϵ to quantitatively demonstrate tradeoffs between exploration and exploitation. We ran 1000 replicates for each baseline and each MAB instance (with rewards realized independently across the replicates).Scale of the experiments. Our main set of experiments has time horizon T = 100. To account for randomness in rewards (and possibly in the LLM, via temperature) we ran N ∈ {10, 20} replicates for each LLM configuration and each bandit instance, with rewards generated independently across the replicates. As a robustness check, we ran a single experiment on GPT-4 with the basic configuration for T = 500 rounds (with N = 20), and obtained consistent/stronger conclusions, depicted in Figure 1(a) .In more detail, for GPT-3.5 we used N = 20 replicates across all 48 prompt configurations, resulting in ≈ 200K queries in total. GPT-4 was an order of magnitude more expensive, considerably slower on throughput, and subject to unpredictable throttling. As such, we only used N = 10 replicates across 10 representative prompt configurations. 7For additional robustness checks, we ran four GPT-4 configurations with T = 200, two for N = 20 replicates and two for N = 40 replicates. In total, this resulted in ≈50K queries issued to GPT-4. LLAMA2 was essentially free from our perspective (since it was locally hosted), but its performance was consistently sub-par; we limited our experiments to the hard MAB instance, 32 configurations, and N = 10 replicates.We emphasize that bandit experiments with LLMs are quite costly in terms of money and time. They take N • T LLM queries for each LLM configuration and each MAB instance being tested. Both N and T must be relatively large to obtain statistically meaningful results: N governs the significance level and must be large to overcome randomness in reward realizations, while T governs the effect size and must be large so that good algorithms have enough time to identify the optimal arm. Both issues are more pronounced in harder MAB instances (many arms K and/or small gap ∆), but exploration failures also tend to be less frequent in (very) easy MAB instances.8 Further, we need to cover the space of possible prompt designs, which is essentially infinitely large, to ensure that our findings do not overfit to one particular design. Thus, ideally we would take N , T , the number of MAB instances, and the number of prompts to be rather large, but doing so is not practically feasible.9 Instead, we use moderately small gap ∆ = 0. We plot suffix failures (expressed via SuffFailFreq(T /2)) vs. uniform-like failures (expressed via K • MinFrac(T )). Each LLM/configuration pair maps to a dot on this plane (some dots may overlap). The GPT-4 configuration labeled with a star is BSS C0, which is the only configuration that succeeds. We also plot ϵ-GREEDY, tracing out the different tradeoffs obtained for different values of ϵ.and the prompt design space as described above.As we will see below, these choices (specifically, N ∈ {10, 20} and T = 100 and ∆ = 0.2) do not provide enough statistical power to distinguish between successful and unsuccessful methods based solely on accumulated rewards.In lieu of further increasing the scale of the experiments, which is not practically feasible, we rely on surrogate statistics which can be detected at our moderate scale, and which are highly suggestive of long-term/persistent exploration failures. Our robustness checks with larger T and N , as well as qualitative findings that we report below provide supporting evidence for this methodology. 
experimental results: 
 In this section, we present our experimental findings, beginning with a summary in Section 3.1. In Section 3.2 we investigate failing LLM configurations in detail, and in Section 3.3 we focus on the single successful LLM configuration our experiments identified. Finally, in Section 3.4 we attempt to diagnose the underlying causes for exploration failures.We find that all but one of the LLM configurations we consider exhibit exploration failures, not converging to the best arm with significant probability. This happens either due to suffix failures, where the LLM never selects the best arm after a small number of initial rounds, or (in a smaller num-ber of configurations) due to uniform-like failures, where the LLM selects all arms at an approximately uniform rate, failing to eliminate poorly performing arms. The only one exception is GPT-4 with the BSS C0 configuration, i.e., with the buttons scenario, suggestive framing, summarized history, reinforced CoT, and temperature 0.We summarize our key findings in Figure 3 and Figure 5 . Figure 3 summarizes the main set of experiments (which we recall consider the hard MAB instance), visualizing each LLM configuration with a single point on a scatter plot where the axes correspond to two surrogate statistics, SuffFailFreq and MinFrac, which represent the strength of the two failure modes (SuffFailFreq measures suffix failures, and K • MinFrac measures uniform-like failures); these statistics are described in detail in the sequel. Figure 5 displays SuffFailFreq, MinFrac, GreedyFrac (which measures how similar a method is to GREEDY), and additional summary statistics for each of the GPT-4 configurations in the main set of experiments. These statistics reveal that all of the LLM configurations, except for GPT-4-BSS C0 (the blue star in Figure 3 ), behave fundamentally differently from the baseline algorithms UCB and TS, and we find that these differences result in a large, persistent drop in performance. Conversely, we find that GPT-4-BSS C0 successfully explores and (as a result) converges to the best arm.We now give a precise overview of the exploration failures illustrated in Figure 3 and Figure 5 , and provide additional results and figures that illustrate failure in greater detail. We focus on GPT-4, as we find that GPT-3.5 and LLAMA2 perform worse (and often much worse) in all experiments; detailed results for GPT-3.5 and LLAMA2 are included in Appendix E for completeness. We begin with detailed background on the surrogate statistics, SuffFailFreq and MinFrac, used to quantify failures in Figures 3 and 5 and beyond, providing evidence that exploration failure-as quantified by these statistics-results in a persistent drop in performance.Suffix failures. Most of the LLM configurations we consider exhibit highly bimodal behavior, whereby a large fraction of the replicates choose the best arm very rarely, and a few replicates converge to the best arm extremely quickly.Consistent with this bimodal behavior, we observe a large incidence of suffix failures, where the best arm is not selected even once after a small number initial of rounds (i.e., in some "time suffix"). Suffix failures are suggestive of a long-term failure to explore which cannot be improved by running the algorithm for longer, because, without playing the optimal arm, one cannot acquire information to learn that it is indeed optimal. Such behaviors are qualitatively similar to those of GREEDY and qualitatively very different from those of UCB and Thompson Sampling.Our surrogate statistic for measuring suffix failures is defined as follows: For an experiment replicate R and round t, let SuffFail(t, R) be a binary variable that is 1 if the best arm is never chosen in rounds [t, T ].Then let SuffFailFreq(t) := mean({SuffFail(t, R) : replicates R}). Suffix failures manifest in most of our experiments at T = 100. In the scatter plot in Figure 3 , the X-axis plots SuffFailFreq(T /2) for each LLM configuration, and we find that all but five configurations have SuffFailFreq(T /2) ≥ 15%. Recalling the definition of suffix failures, this means that ≥ 15% of the time, these configurations do not pull the best arm even once in the last half of the rounds.A more detailed view of suffix failures and bimodal behavior can be obtained by focusing on individual LLM configurations. We visualize this for the basic configuration (GPT-4-BNRN0) in Figure 1 (top) for T = 500, and in Figure 6 for GPT-4 (BNRN0 and BNRN1) at T = 100. In these detailed views, the middle panels plot SuffFailFreq(t) at each time t for the given LLM configurations, as well as UCB, TS, and GREEDY. We find that these LLM configurations have much higher suffix failure rates than both UCB and TS. Bimodal behavior is visualized in the left panel of each plot, where for each configuration, a large fraction of replicates rarely pulls the best arm, while the remaining fraction almost always pulls the best arm. Because of this bimodal behavior (particularly because a constant fraction of replicates by chance almost always pull the best arm), suffix failures are not fully reflected in the total reward plots in the right panels of Figure 6 , since the time horizon T = 100 is not large enough. However, as mentioned, suffix failures are suggestive of an irrecoverable failure to explore which leads to stark differences in reward for larger T . This is precisely what we find at T = 500 in Figure 1 , which suggests that suffix failures indeed lead to poor long-term performance.Uniform-like failures. Returning to the left panel of Figure 3 , we see that three GPT-4 configurations avoid suffix failures. Two of these configurations exhibit a different type of failure, where the LLM selects arms in roughly equal proportions for the entirety of the T rounds and fails to exploit the acquired information to focus on the better arms. We call this a uniform-like failure.Our surrogate statistic for measuring such failures is defined as follows: For a particular experiment replicate R and round t, let f a (t, R) be the fraction of rounds in which a given arm a is chosen, MinFrac(t, R) := min a f a (t, R), and MinFrac(t) := mean({MinFrac(t, R) : replicates R}). Since MinFrac(t) ≤ 1/K, ∀t ∈ [T ], we always plot K • MinFrac(t), so as to rescale the range to [0, 1].Larger MinFrac(t) corresponds to a more uniform selection of arms at time t. When an LLM's MinFrac(t) does not decrease over time and stays substantively larger than that of the baselines (especially as t approaches the time horizon T ), we take it as an indication of a uniform-like failure.The Y-axis of Figure 3 records K • MinFrac(T ) for each configuration, where we see that of the three GPT-4 configurations that avoid suffix failures, two configurations have very high MinFrac(T ) relative to UCB and TS (the third configuration is GPT-4-BSS C0, which is successful). These two configurations are GPT-4-BNRND and GPT-4-BSSCD, both of which use the distributional output format. We provide a more detailed view of GPT-4-BNRND (as well as GPT-4-BNSND, which also exhibits uniform-like failures, but only differs from GPT-4-BNRND in the use of summarized history) in Figure 7 , which considers a longer horizon and more replicates (T = 200 and N = 20).The middle panel reveals that K • MinFrac(t) does not decrease over time for these LLM configurations, while it does for the baselines. This behavior results in no suffix failures, but leads to much lower reward than the baselines.In particular, we obtain a clear separation in total reward, showing that uniform-like failures indeed result in poor long-term performance.Generality of the failures. To summarize, Figure 3 shows that all LLM configurations except GPT-4-BSS C0 exhibit either a suffix failure or a uniform failure for the hard MAB instance and the buttons scenario. Scatter plots for the other three experiments (i.e., the advertisements scenario and/or the easy MAB instance) are qualitatively similar and are deferred to Appendix E.The same data, but with attributions to specific LLM configurations, are presented for all GPT-4 configurations in Figure 5 ; analogous tables for other LLMs and experimental settings are given in Appendix E. As it is not instructive to present detailed plots such as Figure 6 for every LLM configuration, Figure 5 summarizes the performance of each configuration with just a few statistics. We include:• SuffFailFreq(T /2) and MinFrac(T ), defined above.• MedianReward: the rescaled median (over replicates) of the time-averaged total reward.10 • GreedyFrac: the fraction of greedy rounds, averaged over the replicates. A greedy round is one in which an arm with a largest average reward is selected. This is one way to quantify the extent to which a configuration behaves like GREEDY.We now summarize further findings from the scatter plots (Figures 3 and 12 ) and the summary tables (Figures 13 to 19). First, GPT-4 performs much better than GPT-3.5, and LLAMA2 performs much worse (in particular, the suffix failure frequency for LLAMA2 ranges from that of GREEDY to much larger). Second, we observe that all LLMs are sensitive to small changes in the prompt design. However, the different modifications we consider appear to interact with each other, and it is difficult to identify which individual modifications improve performance and which degrade it.On the hard MAB instance, the only configuration in our experiments that avoids both suffix failures and uniform-like failures is GPT-4 with the BSS C0 prompt design. As can be seen from Figure 5 , at T = 100, this configuration has no suffix failures, the K • MinFrac value is only slightly larger than TS, and the reward is comparable to TS. These statistics suggest that this configuration succeeds, and in this section we present further evidence supporting this claim.To do so, we run GPT-4-BSS C0 on the hard MAB instance with T = 200 and N = 40 to obtain more statistically meaningful results. We also consider GPT-4-BSR C0, which swaps summarized history for raw history, as an ablation. Figure 8 provides a summary of the results from this experiment, while Figure 1 (b) provides a detailed view of the BSS C0 configuration. The figures reveal that BSS C0 continues to avoid suffix failures and performs relatively well in terms of reward for larger T . On the other hand, we see that BSR C0 exhibits a non-trivial fraction of suffix failures, demonstrating that this ablation results in fundamentally different behavior.We also provide two additional visualizations that provide some qualitative evidence toward the success of BSS C0, as well as the failure of other configurations. These are presented in Figure 9 and Figure 10 . In Figure 9 we visualize the arm chosen at each time step for various replicates of several different methods (LLMs and baselines). Specifically, Figure 9 shows four replicates for the basic configuration (BNRN0) and the two configurations with reinforced CoT (BSR C0 and BSS C0), as well as one replicate of each of the baseline algorithms. We see that the basic configuration BNRN0 tends to commit to a single arm for several rounds, a behavior that is similar to that of GREEDY and very different from both UCB and TS. BSR C0 also commits for long periods, but to a lesser extent than the basic configuration.In contrast, BSS C0 switches arms much more frequently, and qualitatively appears much more similar to TS.In Figure 10 , we plot the fraction of rounds in [0, t] where the optimal arm was pulled as a function of t for individual replicates. BSR C0 is visually similar to UCB, except that a non-trivial fraction of runs exhibit suffix failures (the curves that converge to 0 on the plot). Meanwhile, BSS C0 is visually similar to TS, with almost all replicates slowly converging to 1. These visualizations, along with the summary statistics, suggest that BSS C0 behaves most similarly to TS, which further suggests it will successfully converge to the optimal arm given a long enough time horizon. Our experimental findings above shed light on how LLMbased decision making agents behave, but it is also worthwhile to understand why they behave the way they do (and particularly, why they fail). This question is rather challenging to answer decisively, but two natural hypotheses are that the configurations we consider (outside of GPT-4-BSS C0) are either a) too greedy, or b) too uniform-like. In this section, we describe how our experiments offer some insight into this hypotheses.First, focusing on GPT-4, our experiments reveal qualitatively different behavior between the easy and hard instances (Figure 13 (a) and Figure 13(c) ). Indeed, the easy instance appears to be much easier; most GPT-4 configurations avoid suffix failures and accrue large rewards on this instance, and the GreedyFrac statistic offers a potential explanation as to why. On the easy instance, most GPT-4 configurations have very high GreedyFrac values, so they behave similarly to GREEDY, which performs quite well (even though GREEDY provably fails with small constant probability and, empirically, has many suffix failures on this instance). 11 A plausible hypothesis from this is that GPT-4 performs quite well in low-noise settings, which is precisely when GREEDY also performs well.A stronger hypothesis would be that most GPT-4 configurations (except perhaps those using reinforced CoT) behave 11 Indeed, in Figure 13 (c) we see that most GPT-4 configurations have very high GreedyFrac but no suffix failures. Apparently, even a very small amount of exploration suffices for easy instances (and makes a big difference, relative to GREEDY). However, this should not be construed as evidence for the more general and robust exploratory behavior necessary for harder bandit instances.like GREEDY on all instances, but this hypothesis is invalidated by the GreedyFrac statistics for our experiments on the hard instance. On the hard instance, it seems that most GPT-4 configurations are doing something non-trivial (albeit flawed); their behavior is neither completely GREEDY-like nor like uniform-at-random.Toward a more fine-grained understanding, we ran a collection of small-scale secondary experiments focusing on the per-round decisions of LLM-agents. The experiments focus on a single round t in a bandit problem. Each experiment considers a particular "data source" (a distribution of bandit histories), samples N = 50 bandit histories of length t from this distribution, and presents them to the agents (the LLMs and the baselines) and asks them to output an arm or distribution over arms. We track two statistics for each agent: GreedyFrac and LeastFrac, the fraction of replicates in which the agent chose, resp., an empirically best arm so far and a least-chosen arm so far. We vary the data source, i.e., the algorithm which generates the history. In particular, we consider histories generated by sampling uniformly at random (Unif) and by running our baselines UCB and TS for t rounds.Results are summarized in Figure 4 . Unfortunately, we find that per-round performance of both the LLMs and the baselines is very sensitive to the particular data source. For example, the MinFrac statistic of UCB can vary from as high as 0.46 on histories generated uniformly at random to as low as 0.09 on histories generated by UCB itself. It seems plausible to conclude the BNSN0 is too greedy while BSRN0 is too uniform, but the statistics for the other two LLM configurations (BNRN0 and BNRC0)-both of which fail in our longitudinal experiments-fall within the reasonable range provided by the baselines. Thus, we find that it is challenging to assess whether LLM agents are too greedy or too uniform-like based on per-round decisions, even though these agents behave rather differently from the baselines in the longitudinal experiments.This paper belongs to a recent body of work that aims to understand the capabilities of LLMs, i.e., what they can and cannot do well, and why. Capabilities that have received considerable attention, but are peripheral to the present paper, include general intelligence (Bubeck et al., 2023) , causal (Kıcıman et al., 2023; Yiu et al., 2023) and mathematical reasoning (Cobbe et al., 2021; Lu et al., 2023) , planning (Valmeekam et al., 2023; Momennejad et al., 2023; Brooks et al., 2023) , and compositionality (Yu et al., 2023) .In more detail, our work contributes to the broader literature on capabilities of in-context learning. Prior studies of in-context learning include theoretical (Xie et al., 2021; Akyürek et al., 2022; Zhang et al., 2023b; Abernethy et al., 2023; Zhang et al., 2023a; Han et al., 2023a; Cheng et al., 2023; Ahn et al., 2023; Wies et al., 2023; Fu et al., 2023; Wu et al., 2023; Huang et al., 2023; Hendel et al., 2023; Li et al., 2023; Von Oswald et al., 2023; Bai et al., 2023; Hahn & Goyal, 2023; Jeon et al., 2024) and empirical (Garg et al., 2022; Kirsch et al., 2022; Ahuja et al., 2023; Han et al., 2023b; Raventós et al., 2023; Weber et al., 2023; Bhattamishra et al., 2023; Guo et al., 2023; Shen et al., 2023; Akyürek et al., 2024) investigations, though as mentioned in the prequel, the vast majority of this work pertains to in-context supervised learning; in-context reinforcement learning has received far less attention. The small collection of empirical works that study in-context RL (Laskin et al., 2022; Lee et al., 2023a; Raparthy et al., 2023; Xu et al., 2022) focus on models trained from scratch using trajectory data collected from another agent (either an RL algorithm or an expert); theoretically, Lee et al. (2023a) and later Lin et al. (2023) justify this approach with a Bayesian meta-reinforcement learning perspective (Simchowitz et al., 2021) , and show that pre-trained transformers can implement classical exploration strategies like Thompson sampling and upper confidence bounds (UCB). However, these works require interventions to the pre-training phase of the language model, and do not study whether existing LLMs exhibit exploration capabilities under standard training conditions.In parallel, there is a rapidly growing line of work that applies LLMs to real-world decision-making applications. Beyond previously mentioned works (Shinn et al., 2023; Wang et al., 2023; Lee et al., 2023b) , which consider applications to gaming, programming, and medicine, highlights include Park et al. (2023) , who introduce generative agents which simulate human behavior in an open-world environment, Ahn et al. (2022) ; Xu et al. (2023) , who develop LLM-enabled robots. Wu et al. (2024) studies LLM performance in a battery of tasks that aim to characterize "intelligent agents", with two-armed bandits as a specific task of interest. Their bandit experiments differ in several key respects: They consider a very easy MAB instance (with 2 arms and a gap ∆ = 0.6, which is much easier than both of our instances), focus on a single prompt design (similar to our basic prompt), and compare to human players rather than algorithmic benchmarks. These differences lead to very different experimental findings. In particular, they find that GPT-4 performs well on their simple MAB instance, converging very quickly to the best arm, while we find that GPT-4 with a similar prompt fails on a harder MAB instance. However, their finding is consistent with ours, as we also find that several configurations of GPT-4 do well on the easy MAB instance. As we discuss in Section 3.4, this instance is too simple to provide compelling evidence for principled exploratory behavior.Here, we provide additional background on the multi-armed bandit problem, and on the baseline algorithms used in this paper. Deeper discussion can be found in Bubeck & Cesa-Bianchi (2012) ; Slivkins (2019); Lattimore & Szepesvári (2020) .The UCB algorithm (Auer et al., 2002) explores by assigning each arm a an index, defined as the average reward from the arm so far plus a bonus of the form C/n a , where C = Θ(log T ) and n a is the number of samples from the arm so far. In each round, it chooses an arm with the largest index. The bonus implements the principle of optimism under uncertainty. We use a version of UCB that sets C = 1 (a heuristic), which has been observed to have a favorable empirical performance (e.g., Slivkins et al., 2013; Ho et al., 2016) .Thompson Sampling (Thompson, 1933; Russo et al., 2018 , for a survey) proceeds as if the arms' mean rewards were initially drawn from some Bayesian prior. In each round, it computes a Bayesian posterior given the history so far, draws a sample from the posterior, and chooses an arm with largest mean reward according to this sample (i.e., assuming the sample were the ground truth). In our setting, the prior is essentially a parameter to the algorithm. We choose the prior that draws the mean reward of each arm independently and uniformly at random from the [0, 1] interval. This is one standard choice, achieving near-optimal regret bounds, as well as good empirical performance (Kaufmann et al., 2012; Agrawal & Goyal, 2012; 2017) . Each arm is updated independently as a Beta-Bernoulli conjugate prior. Further optimizing UCB and Thompson Sampling is non-essential to this paper, as they already perform quite well in our experiments.Provable guarantees for bandit algorithms are commonly expressed via regret: the difference in expected total reward of the best arm and the algorithm. Both baselines achieve regret O( √ KT log T ), which is nearly minimax optimal as a function of T and K. They also achieve a nearly instance-optimal regret rate, which scales as O{ K /∆ log T } for the instances we consider.The ϵ-GREEDY algorithm (Footnote 6) is fundamentally inefficient in that it does not adaptively steer its exploration toward better-performing arms. Accordingly, its regret rate scales as T 2/3 (for an optimal setting of ϵ ∼ T -1/3 ). Fixing such ϵ, regret does not improve for easier instances.The GREEDY algorithm (Footnote 5) does not explore at all, which causes suffix failures. This is obvious when the algorithm is initialized with a single sample (n = 1) of each arm: a suffix failure happens when the good arm returns 0, and one of the other arms returns 1. However, suffix failures are not an artifact of small n: they can happen for any n, with probability that scales as Ω(1/ √ n) (Banihashem et al., 2023) .Our investigation suggests that contemporary LLMs do not robustly engage in exploration required for very basic statistical reinforcement learning and decision making problems, at least without further intervention. In what follows, we identify several next steps to further evaluate this hypothesis and search for interventions to mitigate this behavior.Basic interventions and the need for methodological advancements. In light of our negative results, the most obvious interventions one might consider include:1. Experiment with other prompts. As with many other settings (Sclar et al., 2023) , it is possible that small changes to our prompt template might improve performance. However, sensitivity to prompt design is already concerning. 3. Train the LLM to use auxiliary tools, such as a calculator for basic arithmetic or a "randomizer" to correctly sample from a distribution.While these steps are quite natural, cost, access to models, and compute pose significant barriers to further study, particularly because of the need to employ long horizons T and many replicates N to obtain statistically meaningful results. To this end, we believe that further methodological and/or statistical advancements to enable cost-effective diagnosis and understanding of LLM-agent behavior (e.g., our surrogate statistics) are essential.Implications for complex decision making problems. Our focus on simple multi-armed bandit problems provides a clean and controllable experimental setup to study the exploratory behavior of LLMs and potential algorithmic interventions.Exploration failures here suggest that similar failures will also occur in more complex RL and decision making settings. On the other hand, caution must be exercised in developing mitigations, as solutions that succeed for the MAB setting may not generalize to more complex settings. For example, while GPT-4 with summarized interaction history and reinforced CoT seems to successfully explore in our MAB setting, it is not clear how one should externally summarize the history in settings with complex, high-dimensional observations such as contextual bandits (see Footnote 1). Indeed, even for linear contextual bandits, the approach may not be applicable without a substantial algorithmic intervention (such as, e.g., a linear regression computed externally and included in the prompt) and the many explicit modeling and algorithmic choices involved in such interventions. We believe a deeper investigation of algorithmic interventions is essential to understand the extent to which LLMs can operate as decision making agents. Anonymous Institution, Anonymous City, Anonymous Region, Anonymous Country. Correspondence to: Anonymous Author <anon.email@domain.com>. Preliminary work. Under review by the 1st In-context Learning Workshop at the International Conference on Machine Learning (ICML). Do not distribute.E.g., if there are many arms, or if we are considering contextual bandits with many contexts, then we may only play each arm (context-arm pair) a few times, so averaging reward separately for each-as we do in our experiments-does not provide much summarization. (See Appendix B for further discussion.)A larger gap ∆ makes it easier to distinguish arms, while smaller K means there are fewer alternatives to explore.E.g., LLMs sometimes fail at basic arithmetic(Gao et al., 2023;Liu et al., 2024), though this is likely to improve in the near future via better training and/or integrating calculator-like tools.Specifically: GPT-3.5-TURBO-0613 (released 06/13/2023), GPT-4-0613 (released 06/13/2023), and LLAMA2-13B-CHAT quantized to 4-bits(Dettmers & Zettlemoyer, 2023).In each round, GREEDY chooses an arm with the largest average reward so far. The algorithm is initialized with one sample of each arm. It fails in that with constant probability, it never chooses the best arm after initialization.6 ϵ-GREEDY is a standard MAB algorithm which in each round chooses an arm uniformly at random with a given probability ϵ, and exploits (i.e., mimics GREEDY) otherwise.Precisely, N = 10 for the buttons scenario, and N = 3 for the robustness check with the advertisements scenario.For example, GREEDY always succeeds when the gap is ∆ = 1, i.e., there is no noise, and trivially succeeds with probability at least (1 + ∆) 2 /4 when the initial samples evaluate to 1 for the good arm and 0 for the bad arm.Raw-history prompts and chain-of-thought outputs are particularly expensive, as LLM APIs bill per token.More precisely, let Φ(R) be the time-averaged total reward for a given replicate R. Then E{Φ(R)} ranges in the interval [ 1 /2 -∆/2, 1 /2 + ∆/2]. We rescale Φ(R), by translating and multiplying, so that E{Φ(R)} ranges in [0, 1]. 
 
 Appendix:  


Decision: This paper explores whether large language models (LLMs) can engage in in-context exploration using various prompt designs within multi-armed bandit environments. The study found that only GPT-4, when combined with chain-of-thought reasoning and externally summarized interaction history, demonstrated effective exploration, while other configurations failed. 

The reviewers generally acknowledge the paper’s interesting direction and practical value but criticize its lack of deeper insights and innovation in the field. Despite these criticisms, they recognize the potential impact of the paper, which outweighs its weaknesses.

Reviews: review: paper_summary: paper_summary: This paper investigates whether popular LLMs can engage in exploration in a in-context manner (all experiences are stored as context/prompt). To achieve, the paper deploys LLMs (GPT-3.5, GPT-4, LLAMA2) as agents in multi-armed bandit environments, using various prompt designs to specify the environment description and interaction history entirely in-context. The results show that only GPT-4 with chain-of-thought reasoning and externally summarized interaction history exhibited satisfactory exploratory behavior. Other configurations failed to robustly explore. The authors suggest that non-trivial algorithmic interventions, such as fine-tuning or dataset curation, may be required for LLMs to effectively explore in more complex settings.
strength_weakness: strength_weakness: As an experimental article, the findings are naive and obvious: the LLMs are not designed for solving decision-making tasks. It is consistent with intuition that LLMs can explore in-context when CoT with summarized history.
 
I feel that experiment results on more challenging tasks are necessary, such as some textual interactive games. In practical, the scenarios human encounters and explores are more complex than a bandit.

More discussions about related work are appreciated.

There lacks a more systematic analysis of which specific prompt design elements are most critical for successful exploration? This could help in understanding the sensitivity of LLMs to prompt variations.
questions: questions: 1. Only one configuration (GPT-4 with chain-of-thought reasoning and an externally summarized interaction history) resulted in satisfactory exploratory behavior. Why this specific configuration was successful and others were not? Are there any insights into the underlying mechanisms?
2. The paper concludes that external summarization is essential for desirable LLM behavior in simple environments. How do the authors envision scaling this approach to more complex settings where external summarization might be infeasible? Are there any preliminary ideas or experiments that could address this limitation?
limitations: limitations: I do not find any limitations discussed in the main body.
paper_summary: paper_summary: This paper investigates the exploration capabilities of contemporary large language models. The authors use LLMs as agents in multi-armed bandit environments, describing the environment and interaction history in-context without any training interventions. Their experiments involve a variety of configurations and models, and they conclude that only GPT-4 with chain-of-thought reasoning and an externally summarized interaction history demonstrates satisfactory exploratory behavior.
strength_weakness: strength_weakness: - In line 161, it is mentioned that no parameter tuning is performed for the baselines with tunable parameters. In this case, we are not sure that the baselines are performing to their highest capacity and therefore, the comparison might not be fair.
- Longer horizons T and replicates N might be required to obtain statistically significant results.
questions: questions: - Why isn't parameter tuning performed for the baselines with tunable parameters?
- In line 97, it is mentioned that all the arms, except for the best arm, have the same mean reward. What is the reason for this choice?
- If the summary is not available for more complex situations, can few-shot examples help instead?
limitations: limitations: - Longer horizons T and replicates N might be required to obtain statistically significant results.
paper_summary: paper_summary: This paper investigates whether large language models (LLMs) like GPT-3.5, GPT-4, and LLAMA2 can perform exploration in reinforcement learning settings without additional training. The authors focus on in-context learning, where the environment description and interaction history are provided entirely within the LLM prompt. They evaluate LLMs on multi-armed bandit problems, comparing their performance to standard bandit algorithms like Thompson Sampling (TS) and Upper Confidence Bound (UCB).

The key findings are: (1) Most LLM configurations fail to robustly explore, exhibiting either "suffix failures" (converging to a suboptimal arm) or "uniform-like failures" (selecting arms uniformly at random). (2) Only one configuration (GPT-4 with chain-of-thought reasoning and externally summarized history) showed satisfactory exploratory behavior comparable to TS. (3) LLM performance is highly sensitive to prompt design and varies significantly across model versions.

The authors conclude that non-trivial algorithmic interventions may be necessary for LLMs to function effectively as decision-making agents in complex settings.
strength_weakness: strength_weakness: - While the focus on multi-armed bandits provides a clean experimental setup, it may not fully represent the challenges of more complex reinforcement learning problems. The generalizability of the findings to broader settings is unclear.
-  The paper is predominantly empirical. A theoretical framework for understanding why LLMs struggle with exploration could provide deeper insights and guide future research.
- While the authors attempt to explore reasons for LLM failures in Section 3.3, this analysis feels underdeveloped. A more systematic investigation of failure modes could yield valuable insights.
- While the authors compare summarized vs. raw history, more systematic ablations of different prompt components could help isolate the factors contributing to successful exploration.
questions: questions: None.
limitations: limitations: Yes.
paper_summary: paper_summary: In this study, the authors investigate the impact of diverse prompt designs, including environmental descriptions, summarizing interaction history, and chain of thought reasoning, on the exploration behavior of three distinct LLMs in a multi-armed bandit task. The results demonstrated that none of the models engaged in exploration without substantial interventions. The only configuration that demonstrated efficacy was the GPT-4 model, which was prompted with change of thought reasoning and an externally summarised interaction history. The authors demonstrate that the typical failure mode for these models is either suffix errors, wherein the LLM fails to select the optimal arm even once in the initial stages, or uniform sampling of all options. In conclusion, the authors raise concerns about the external validity of the working configuration, noting that it may not be scalable to more complex task settings. They also speculate on the potential algorithmic interventions that could be employed to encourage LLMs to explore in such settings.
strength_weakness: strength_weakness: The research question and paradigm used to study that question have been previously investigated in the literature [1-5]. However, in contrast to previous studies, the authors have conducted a more comprehensive search of prompt designs, included a greater number of options in the multi-armed bandit task, and analyzed the failure modes of the models using two novel measures. Nevertheless, I have significant concerns regarding the work's novelty and its contribution's strength. While the results are interesting, the main contribution of the work is somewhat limited, as the subset of evaluations performed has already been studied by previous works (see questions and limitations for details). 

Additionally, the authors have not adequately cited related works [1-8].

[1] Binz, M., & Schulz, E. (2023). Using cognitive psychology to understand GPT-3. _Proceedings of the National Academy of Sciences_, _120_(6), e2218523120.

[2] Schubert, J. A., Jagadish, A. K., Binz, M., & Schulz, E. (2024). In-context learning agents are asymmetric belief updaters. _arXiv preprint arXiv:2402.03969_.

[3] Hayes, W. M., Yax, N., & Palminteri, S. (2024). Large Language Models are Biased Reinforcement Learners. _arXiv preprint arXiv:2405.11422_.

[4] Coda-Forno, J., Binz, M., Akata, Z., Botvinick, M., Wang, J., & Schulz, E. (2023). Meta-in-context learning in large language models. _Advances in Neural Information Processing Systems_, _36_, 65189-65201.

[5] William M Hayes, Nicolas Yax, and Stefano Palminteri. Relative value biases in large language models. arXiv preprint arXiv:2401.14530, 2024.

[6] Julian Coda-Forno, Marcel Binz, Jane X Wang, and Eric Schulz. Cogbench: a large language model walks into a psychology lab. arXiv preprint arXiv:2402.18225, 2024

[7] Thilo Hagendorff. Machine psychology: Investigating emergent capabilities and behavior in large language models using psychological methods.

[8] Lampinen, A. K., Dasgupta, I., Chan, S. C., Matthewson, K., Tessler, M. H., Creswell, A., ... & Hill, F. (2022). Can language models learn from explanations in context?. _arXiv preprint arXiv:2204.02329_.
questions: questions: What factors within the model (size, architecture, RLHF, internal representation), its training dataset, and training protocol (at a specific point during training) could be preventing LLMs to fail at exploration? It may be beneficial for the authors to consider investigating some of these factors further, which could potentially enhance the quality of the paper. It would be beneficial to investigate the emergence of exploration behavior in LLMs by examining smaller models, such as GPT-2/LLaMa-2. 

It has been demonstrated that LLM models can be fine-tuned for reinforcement learning tasks, enabling them to function as decision-makers capable of generalizing to novel tasks with high efficiency [1-2]. These findings suggest that, despite their limited capacity for exploration when tested off-the-shelf, LLMs can be effectively fine-tuned to enhance their ability to explore. It would be valuable for the authors to investigate whether fine-tuning LLMs (such as the GPT-2 /LLaMa-2 model) on simple RL tasks could facilitate their exploration capabilities.

[1] Cao, Y., Zhao, H., Cheng, Y., Shu, T., Liu, G., Liang, G., ... & Li, Y. (2024). Survey on large language model-enhanced reinforcement learning: Concept, taxonomy, and methods. _arXiv preprint arXiv:2404.00282_.

[3] T. Carta, C. Romac, T. Wolf, S. Lamprier, O. Sigaud, and P.-Y. Oudeyer, “Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning,” Sep. 2023
limitations: limitations: The failure mode of an agent following an epsilon greedy policy with a high epsilon value is analogous to that of Suffix Failure. Conversely, KMinFrac is analogous to the failure mode of an agent following a random policy or an epsilon greedy policy with a low epsilon value. Therefore,  I believe the metrics themselves are quite trivial and do not constitute a novel contribution.

Rather than relying on a qualitative comparison between LLM's choices and those of alternative bandit algorithms, the authors may find it beneficial to fit the models of exploration to LLM's choices and conduct a more quantitative evaluation [1] 

[1] Gershman, S. J. Deconstructing the human algorithms for exploration. Cognition, 173:34–42, 2018.

Comments: title: Response to rebuttal
Response to rebuttal
comment: Thanks for the helpful feedback. We would be happy to include such a mixed-effects regression analysis in our next revision.
Thank the authors for their detailed responses to our queries. 

Re. Q1: Can the authors include a mixed-effects regression analysis, where they perform a regression from the abovementioned factors onto the surrogate measures to see how much each factor determines the extent of exploration and lack thereof? This allows us to understand the contributions of the factors at a more fine-grained level. 

Re. Q2: We agree with the authors that expertise & money and/or infrastructure prevent fine-tuning to be a plausible option for everyone. But we differ from them in that there is a sufficient audience who would be interested if there is a simple fine-tuning protocol that makes LLMs explore optimally.  We think smaller models can be a good test bed for such approaches and are quite confident that the protocol can even be standardized if shown to provide sufficient gains.

Re. external validity concerns as raised by two other reviewers: 
We agree with the authors that the bandit task provides a well-studied, controlled setting to study exploration in LLMs.
However, given that these models are typically made to solve tasks vastly different from the bandit setting, We still have concerns about whether the findings will hold for those settings. If authors find the same effects in coding or some problem-solving tasks with the help of their prompting strategy, that would greatly increase the potential impact of the findings.

Keeping these points in mind, I am happy to increase my score from 3 to 5.
It seems that major concerns raised by other reviewers are the complexity of the evaluation tasks. While I expect to see the performance of LLM in-context exploration in challenging tasks, I agree with the authors that MAB setting presents significance in exploration. Exploration in simple tasks is the foundation for more complicated tasks.

Overall, I feel that LLM in-context exploration is an interesting direction. Thus I raise my score from 6 to 7.