Behavioral Exploration: Learning to Explore via In-Context Adaptation
Authors: Andrew Wagenmaker
Zhiyuan Zhou
Sergey Levine
Venue: ICML 2025 poster (2025)
Keywords: in-context learning
exploration
adaptive agents
behavior cloning
Primary area: reinforcement_learning
TL;DR: We train adaptive policies that utilize in-context learning to enable online exploration.

Abstract: Developing autonomous agents that quickly explore an environment and adapt their behavior online is a canonical challenge in robotics and machine learning. While humans are able to achieve such fast online exploration and adaptation, often acquiring new information and skills in only a handful of interactions, existing algorithmic approaches tend to rely on random exploration and slow, gradient-based behavior updates. How can we endow autonomous agents with such capabilities on par with humans? Taking inspiration from recent progress on both in-context learning and large-scale behavioral cloning, in this work we propose behavioral exploration: training agents to internalize what it means to explore and adapt in-context over the space of ''expert'' behaviors. To achieve this, given access to a dataset of expert demonstrations, we train a long-context generative model to predict expert actions conditioned on a context of past observations and a measure of how ''exploratory'' the expert's behaviors are relative to this context. This enables the model to not only mimic the behavior of an expert, but also, by feeding its past history of interactions into its context, to select different expert behaviors than what have been previously selected, thereby allowing for fast online adaptation and targeted, ''expert-like'' exploration. We demonstrate the effectiveness of our method in both simulated locomotion and manipulation settings, as well as on real-world robotic manipulation tasks, illustrating its ability to learn adaptive, exploratory behavior.

Behavioral Exploration: Learning to Explore via In-Context Adaptation
Andrew Wagenmaker' Zhiyuan Zhou' Sergey Levine'
Abstract
fully directed attempts, use the information collected from these attempts infer the correct behavior, and then success- fully solve the objective. In other words, humans exhibit fast online exploration and adaptation addptarion-attemmiing different be- haviors to acquire useful information and adjusting their behavior based on this information to solve the objective. fundamental goal in the fields of robotics and machine learn- ing is to develop autonomous agents which exhibit such fast online exploration and adaptation. While much work has been devoted to achieving this, existing methods of- ten fall short, typically relying on expensive trial-and-error approaches to exploration that require an infeasibly large number of interactions, and slow, gradient- behavior updates which are unable to quickly adapt behaviors online.
Developing autonomous agents that quickly ex- plore environment and adapt their behavior online is a canonical challenge in robotics and ma- chine learning. While humans are able to achieve such fast online exploration and adaptation, often acquiring new information and skills in only a handful of interactions, existing algorithmic ap- proaches tend to rely on random exploration and slow, gradient- based behavior updates. How can we endow autonomous agents with such capabili- ties on par with humans? Taking inspiration from recent progress on both in-context learning and large-scale behavioral cloning, in this work we propose behavioral exploration: training agents internalize what it means explore and adapt in-context over the space of "expert" behaviors. To achieve this, given access to dataset of expert demonstrations, we train a long-contex, genera- tive model to predict expert actions conditioned on a context of past observations and a measure of how "exploratory" the expert's behaviors are relative this context. This enables the model not only mimic the behavior of an expert, but also, by feeding its past history of interactions into its context, to select different expert behaviors than what have been previously selected, thereby al- lowing for fast online adaptation and targeted. "expert-like" exploration. We demonstrate the ef- fectiveness of our method in both simulated loco- motion and manipulation settings, as well as on wall-wrldd robotic manipulation tasks, illustrating its ability to learn adaptive, exploratory behavior.
What would successful exploration and adaptation behavior look like for autonomous agent? As example, consider telling a robot "hand me a cup". In this scenario, the robot should attempt to pick up whichever cups are in the scene until it successfully picks up the right one. It should not, however, move its arm randomly, try to pick up a plate, or repeatedly pick up the same cup when there are other cups in the scene it has not yet attempted to pick up. Instead, the correct behavior requires first leveraging knowledge of the scene and what it means to "hand me cup" to direct actions to the most relevant features, and then quickly adapt- ing based on the attempts that have already been made in order to continue attempting novel, potentially correct, ac- tions. Such a semantic, informed approach to exploration, coupled with fast online adaptation, is much more akin to how humans behave and dramatically more efficient than the "uninformed" novelty-sekking exploration strategies that are more often considered in the existing literature.
In this work we take steps towards developing autonomous agents which exhibit such behavior, focusing in particular on settings where we have access to expert demonstration datasets that provide a prior on what behaviors may be "rea- sonable" for task of interest. We inspired by the recent success of in-context learning to enable fast online adaptation in language domains (Brown et al., 2020), and seek to leverage similar in-context learning capabilities in our setting. We propose training a long-context policy to internalize both which actions an expert would take a given state, and which expert actions are most exploratory, To instill the policy with in-context adaptation capabilities,
1. Introduction
When tasked with achieving some objective in a novel sce- nario, humans are often able to make small number of care-
'Dpaatmeet of Electrical Engineering & Computer Science, University of California, Berkeley. Correspondence to: Andrew Wagenmaker <ajjaggn@berkttey.dduss
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Behavioral Exploration: Learning to Explore via In-Context Adaptation
History,
Future:
T BE
BC Öt
51
ät
Coverage(Futurrs
History,) + cOV:
COV+ St
History,
Given offiine demonstration data D, we sample history (History,), state-action pair (sy, dt), and future trajectory proceeding from s, (Futurez), and compute the total coverage of the states visited in the future combined with those visited in the past. Center: We then train long-context policy ThE to minimize a standard behavioral cloning loss, but we include in TEE 's context the history and total coverage, in addition to the state, enabling Tss to learn to infer "in-context" which actions are likely to increase the future coverage given the past. Right: At deployment, Tar can then adapt its behavior by conditioning on the past states visited, and we can regulate how "exploratory" it by setting the desired coverage conditioning value, allowing for effective exploration over the space of behaviors present in the demonstration data.
Figure 1: Overview of our proposed approach, behavioral exploration. Left:
in addition to the current state, we condition context which includes a history of previous observations, and a measure of how exploratory the expert action in this state is given these observations. This enables the model to not only infer which actions an expert is likely to take, but also which expert actions are exploratory and, by feeding its past observations into its context, allows it to quickly adapt its behavior online to attempt different behaviors than what it has already attempted. Critically, our approach is trained to be exploratory over the space of expert behaviors and thus naturally restricts its exploration to coherent, reason- able
Shyam et al., 2019; Lee et al., 2021; Henaff al., 2022). The majority of these works, however, aim to explore fully online, and do not make use of prior information or offline data, which is the focus of this work. To our knowledge, only several works seek to leverage offline data to learn exploratory behaviors. Li al. (2023) and Wilcoxson et al. (2024) study the setting where the offline data does not con- tain reward labels, and utilize the offline data to warm-start an RL algorithm to explore online. Hu et al. (2023) trains policies to maximize random reward functions offline, then deploys these policies to explore online, coupling them with online RL. Though not explicitly tackling the exploration challenge, variety of works have considered utilizing offline data speed up online RL in the setting where both offline and online data contain reward labels (Lee al., 2022b; Zhang et al., 2023; Uchendu et al., 2023; Zheng et al., 2023; Ball et al., 2023; Nakamoto et al., 2024). Com- pared to these methods, which all rely on gradient- based online RL training, our approach learns online in-context, allowing for much faster adaptation and exploration.
an expert is likely to play, and therefore behaviors likely to solve a given task.
To the best of our knowledge, our approach, which we refer to as behavioral exploration (BE), is the first to enable learning of data-driven exploratory behavioral policies that can quickly adapt online as they collect new experience, and achieves this with fully offline training and in-context online adaptation. Furthermore, our approach relies on easy-to-train behavioral cloning methodology (as compared to reinforcement learning approaches, that are often much more challenging to train), and can be efficiently scaled to large-scale datasets. Focusing particularly on robotic and continuous control domains, we demonstrate the effectiveness of our approach on a variety of simulated reinforcement learning (RL) and imitation
Learning for fast online adaptation. Developing learning algorithms that quickly adapt to new observations broadly falls under the domain of meta-learning. In particular, meta- RL aims to learn policy on some set of train environments that can quickly adapt to a new test environment; see Beck et al. (2023) for a complete overview. A key challenge here is learning to explore effectively in the test environment, and variety of methods have been proposed to achieve this (Duan et al., 2016; Wang et al., 2016; Mishra et al., 2017; Zintgraf et al., 2019; Rakelly et al., 2019; Kamienny et al., 2020; Zhang et al., 2020; Liu et al., 2021; Norman & Clune, 2023; Jiang et al., 2024). These works require online access to train environments, however, while in contrast we only assume access to offline data. Several works have considered meta-RL from only offline data (Li et al., 2020; Mitchell et al., 2021; Dorfman et al., 2021; Rafailov et al., 2021; Ghosh et al., 2022; Pong et al., 2022), including
learning benchmarks, as well as a real-world robotic setting, showing that yields a significant gain over standard behavioral cloning (BC) and RL approaches.
2. Related Work
Exploration in decision-min maaiin.....ainn...... Exploring an unknown environment is canonical problem in RL and control, and sigmificant amount of attention has been devoted to (Stadie et al., 2015; Osband et al., 2016; Bellemare al., 2016; Burda et al., 2018; Choi et al., 2018; Ecoffet et al., 2019;
2

---

Behavioral Exploration: Learning to Explore via In-Context Adaptation
recent line of work that aims to train transformer-style models which learn online adaptation strategies in-context (Laskin et al., 2022; Liu & Abbeel, 2023; Lee et al., 2024), or utilizes the in-context learning abilities of LLMs to solve simple decision-making problems (e.g., multi-armed bandits) (Krishnamurthy et al., 2024; Nie et al., 2024; Monea et al., 2024). While our work is related to meta-RL in that we also aim to learn fast adaptation behaviors, our setting is fundamentally different: all of the above works assume access to a reward function, and seek to learn policy maximizing this reward function, while we instead assume access to a set of demonstrations, and aim to learn exploratory behaviors. We highlight as well the meta-imitation learning setting, where the goal is to learn a policy that can imitate a very small number of expert demon- strations (Duan al., 2017; Finn al., 2017: James al., 2018; Dasari & Gupta, 2021; Raparthy et al., 2023; Fu et al., 2024). The objective of these works is fundamentally dif- ferent than ours, however: they aim to imitate the "history", while our goal is to behave differently than the history.
as a "return", and our approach therefore bears resemblance to work on retum-conditioned policies for decision-making. Typically, these works condition on the desired reward-to-go (Schmidhuber, 2019; Srivastava et al., 2019; Kumar et al., 2019; Emmons et al., 2021). 1), but have also considered goal conditioning (Ghosh et al., 2019; Ding et al., 2019; Lynch et al., 2020). More modern variants have proposed training higher-capacity transformer or diffusion models (Chen et al., 2021; Furuta et al., 2021; Ajay et al., 2022; Lee et al., 2022a; Huang et al., 2024; Wu et al., 2024; Schmied et al., 2024; Yan et al., 2024). While our work bears resemblance to these works methodologically, our focus is quite different-rathie than training a model to maximize reward, we train the model to explore, requiring a very different return structure and much more intricate history dependence.
3. Preliminaries
We consider decision-making in the context of reward-free Markov decision processes (MDPs), which we denote by a tuple .M- := A, P.Po), where S is the set of states, A the set of actions, P S A - Ag the transition kernel, and Po E As the initial state distribution. A trajectory denotes interaction sequence where we sample initial state so po. take some action co. transition to state 81 ~ (u)) and this process continues; for simplicity, we assume each trajectory is of length K. Throughout we use T to denote such a trajectory, T = (50, dg 81, ai....,, 5K, and we let S 5 denote some set of states typically this will correspond to the set of previous trajectories an agent has observed, its history. We denote policy by x S X + A where X' could be some other conditioning space (in particular, we will condition π on its history of past interactions. h, as we describe below). We say that a trajectory T is generated by policy ग ag ~ x (8k, for each step k in T.
Learning control policies from expert demonstrations. Learning control policies by mimicking expert demonstra- tions, behavioral cloning, has long history (Argall et al., 2009; Ross et al., 2011: Bojarski. 2016; Zhang et al., 2018; Rahmatizadeh et al., 2018; Mandlekar et al., 2021). Over the last several years, significant attention has been given to applying BC to real-world control and robotic domains, and much progress has been made in deploying modern learning paradigms (Shafiullah et al., 2022: Cui et al., 2022b; Zhao al., 2023; Chi et al., 2023; Dasari al., 2024) and training controllable policies by conditioning on human language and other modalities (Stepputtis et al., 2020; Brohan et al., 2022; Nair et al., 2022; Tcam et al.. 2024; Kim et al., 2024; Black et al., 2024; Gu et al., 2023; Cui et al., 2022a;b; Black et al., 2023; Sundaresan et al., 2024). Another line of work aims to learn "skills" from expert data, and compose these with high-level skill policy (Ajay et al., 2020; Singh et al., 2020; Pertsch et al., 2021). Our work is somewhat related to this line of work, but instead of training policies to directly mimic an expert in solving tasks as these works do, we focus on training policies that can effectively explore and adapt. We also mention the work of Zhou et al. (2024), which trains a goal-conditioned BC policy and deploys this online to col- lect data and perform policy improvement. While this work can be seen as tackling the exploration problem, the pro- posed approach is quite different, relying on a multi-stage system composing several models trained on internet-scale data; in contrast, we aim to learn a single model from only demonstration data that internalizes the ability to explore.
Demonstration data. We assume access to a demon- stration dataset D = [T')T= where for each t, T' := (N6- ai. : ,'x axx) denotes trajectory collected by some behavior policy TB in M (note that Te may be mixture policy over multiple demonstrators). We do not make ex- plicit assumptions on the optimality of the behavior of Tg. but we do assume that the behaviors exhibited by ma are a reasonable prior over potentially "useful" behaviors, as we formalize in the following.
4. Learning to Explore from Demonstrations
Given demonstration data D, our goal is to learn a policy that, when deployed in a test environment, explores its set- ting and adapts its behavior online in order to continue to explore. In particular, rather than exploring uniformly over the space of all possible behaviors, we would like the ex-
Decision-making with return-condittiond policies. Our proposed methodology relies on training a generative model that conditions a measure of the coverage an action should lead to. This measure of coverage can be thought of
3

---

Behavioral Exploration: Learning to Explore via In-Context Adaptation
ploration to be focused over the space of "useful" behaviors represented in D. In this section we formalize this objective, and outline our proposed approach.
us a prior over where to focus exploration, and, rather than exploring uniformly, we might focus exploration on the space of behaviors spanned by ms. For instance, returning to the example given in the introduction, instead of explor- ing all possible robot state configurations-tte the behavior that would result when maximizing (1) we should explore only within the space of motions represented in a dataset of expert robot demonstrations, attempting different manip- ulation behaviors likely to solve our task. More generally, by focusing our exploration on the behaviors present in an effective demonstration policy TT. we can efficiently collect highly targeted data that could be utilized for downstream learning objectives, or, if v exhibits diverse task- solvino behaviors, explore over these behaviors to arrive at a solu- tion to a task of interest.
4.1. Exploration Over Demonstration Coverage
Canonically, the goal of exploration is to interact with environment in order to collect data that achieves high coverage-vistting as many useful and informative states as possible. To formalize this, assume that we have access to some fcaturization of our environment, ((s) E R.. For example, in a finite-state setting we might have ((s) = e. in a continuous-sten setting ((s) might correspond to a vectorized representation of the state (e.g., the final layer of a neural network state encoder), in an image-based setting ((s) could correspond to feature vector of our image given by some image feature extractor. In short, ((s) denotes some mapping which allows us to relate states based on their vector representations, and we can therefore quantify coverage in terms of the degree to which this representation space is spanned. For the purposes of this work, we take inspiration from classical estimation theory, and propose quantifying coverage via the trace of the inverse feature m Precisely, given some set of states h- == { letting (h)) := Σim ((s))((ss)T we define the coverage of h as:
To formalize coverage over behaviors in xg, denote Ag := E o (()))((s)))) "]. U3ΣBUJ the singular-value decomposition of Ag. and U3,, E Rd the matrix of singu- lar vectors of A, corresponding to singular values of at least c. UB,, then corresponds to the span of features covered by Th (with weight at least e) and, augmenting our uniform coverage metric, (1), to:
c = (()U)A(h)
(2)
we can quantify how well h covers the space of features spanned by тy with chv For example, in the finite- state setting, c- corresponds to the coverage only over states visited by न with probability at least e, rather than coverage over all states, or in the robotic manipulation set- ting, this may correspond to coverage over all coherent manipulation motions. Critically, achieving coverage over Ug.. could be much easier than achieving coverage over the entire feature space, and allows to focus our exploration only on the most salient aspects of the environment. Given this, we propose the following objective, which formalizes our goal of exploring over the behaviors exhibited by TT--
+ cov(h) tr (r((A()++ I)-)
In the finite-state setting, for example, this reduces to cov(h) = (E. M=b))x) for N(s; h) the number of times state s appears in h, recovering a standard notion of total state coverage. In more general, potentially stte settings, (1) corresponds to how well we span the entire feature space.
Standard exploration approaches would typically aim plore uniformly over the space, collecting data with the goal of (approximately) maximizing (1). While achieving high uniform coverage would provide a rich dataset for future downstream learning, for many problems of interest the en- tire reachable space is extremely large, and achieving such coverage may be infeasible or, at the very least, extremely wasteful. Instead, we would like focus our exploration only the most relevant parts of the space. If we have access to demonstrations from some policy Te which exhibits be- haviors spanning the space of "useful" behaviors, this gives
Objective 4.1 (Behavior Policy Coverage). For covg(.) the coverage over the features spanned by the behavior policy Ta. (2), and A == E E a policy n which maximizes COV 3 (AAz).
1 Φl 8h (d($)) T), use D to find
4.2. Learning Exploratory Behaviors via Conditional Distribution Modeling
A natural choice for a policy maximizing cov- erage over the space spanned by πa would be та itself-by defi- nition, Ty covers the space spanned by --- which suggests simply training a BC poliiy on D to solve
Ø1
'For example, the trace of the inverse information matrix, which we can think of Σi= (>s)))(s):) as approximating, cor- responds to the estimation error of the maximum likelihood esti- mator (Van der Vaart, 2000), and, by the Cramer- Rao lower bound, lower bounds the estimation eиror of any unbiased estimator (Pron- zato & Pázman, 2013). Furthermore, this expression is standard in the experiment design literature; maximizing (1) is commonly known as A-optimal experiment design (Pukelsheim, 2006).
T
80
a1
02
T2
U2
Figure 2: Illustrative example.
4

---

Behavioral Exploration: Learning to Explore via In-Context Adaptation
Objective 4.1. To illustrate the shortcomings of BC, how- ever, consider Figure 2. In this example, assume that we start at state so and that there are two goals of interest, 91 and 92. both within the coverage of Tg. At Su, we can either take action 01, which will lead us along trajectory T to 91, or d2 which will lead us along T22 to 92- Assume that T | (80 = - p and T=((2 | 88) = p. for p < 1. If we clone Tg. since 3h only takes 02 with small probability, the vast majority of the time we will take 01 and follow T to 91- foregoing the opportunity to reach 92. In contrast, if we were to take di and 02 with equal probability, we would much more quickly reach both goals. Thus, while xy will eventu- ally achieve coverage over 91 and 92. we can achieve this much more efficiently with a different weighting of actions.
conditioned distribution over the actions of the behavior policy, we can arrive at an optimal solution Objective 4.1. Critical to achieving this is that we condition on the history and adapt our behavior based on the history in order to in- crease coverage-as as the previous example illustrates, only by adapting to our history can we ensure the exploratory action is taken. We note that this result only requires condi- tioning on cov and not on go thereby avoiding explicit de- pendence on Ug the policy of (3) explores over Ug., by construction since its action distribution is just a reweighting of zT.. Please see Section A for the proof of Proposition 4.2.
To learn a policy from D which explores over the space of features spanned by Tg. the above argument motivates the following simple supervised learning problem, which we refer to as behavioral exploration.
While naive BC is therefore suboptimal, we see that does place mass both actions of interest and, if we could re- weight this distribution in some way in order to increase mass on 02, it may offer feasible method to collect high- coverage data. Let us assume that we are in so and, on the previous trajectory, already took 01 and observed Tι denote by our history of past observations, h Ti. In this case we would expect cov(h U T2) cov(h U T1) since T2 covers states not previously encountered in h, it would, in general, increase coverage relative to h more than observing Tı again. If we let P*8 [-] denote the distribution over actions induced by T and t the trajectory that results from taking actions under TT from state sp, then we see that
Behavioral Exploration (BE). Given expert demonstration data D, find a policy The maximizing the following:
K
Σ> k=1
EA B ((D) (a I st h, u) (4)
where Tt. := (s (s denotes the subtrajectory of I' starting at step k. and H(D) some distribution over demonstration trajectories.
We can think of this objective as corresponding to the max- imum likelihood estimate of the conditional distribution given in (3) (see e.g. Foster et al. (2024)). We empha- size that aee naturally focuses its exploration on the space spanned by Ty since, ultimately, it is still fitting the action distribution induced by Tg.
[a2 so. h, cov(h UT) = cov(h T2)] p=* [a1 | so h, cov(h U r) = cov(h U T-)]
In other words, by conditioning on the subsequent trajectory achieving high coverage relative to h, the likelihood of T taking 02 and therefore reaching d2 increases significantly, as 02 is much more likely to increase coverage than 01 The following result makes this formal, and shows that (in cer- tain cases), this -eeendeed conditional distribution over actions is in fact an optimal solution to Objective 4.1.
Given a policy aas solving (4), at deployment, if we are at step & and state 8t, and have already visited states hg, we sample our next action as a Tae(- sg, hy. exp). where exp E IR is a measure of how exploratory the sampled action should be relative to hg- If we set exp to be large, then this will increase the likelihood of sampling actions that lead to higher coverage data, while if exp is small, the sampled action is incentivized to induce behaviors close to the already collected observations.
Proposition 4.2 (Informal). Assume that we are in a de- terministic environment, that for any terminal state sk, ((8k) E dei,...,,a-il. and that (6s) = ea for 8 not a terminal state. Let he denote the history of states visited up to trajectory t. Then under several additional technical assumptions, the policy which takes actions at trajectory with probability
4.3. Practical Implementation
The behavioral exploration objective, (4), forms the basis for our proposed approach. Here we make several comments on the practical implementation of BE.
[ s, hu, cov(h, T) max, cov UT']] (3)
an optimal solution to Objective 4.1.
Architectural considerations. Our goal in (4) is to learn policy modeling a potentially complex and multi-modal conditional distribution, in particular in control settings with continuous actions. Furthermore, we must condition on a significant amount of information- the current state as well as a history of previous states. Effectively modeling this distribution therefore requires an architecture able to handle
Thus, while naive behavioral cloning is suboptimal, Propo- sition& 4.2 shows that by setting our policy to a carefully
2 More precisely, if we consider the distribution over trajectories induced by rolling out R on M, then pas denotes the marginal of this distribution over actions.
5

---

Behavioral Exploration: Learning to Explore via In-Context Adaptation
such complex continuous distributions, as well as a large conditioning space. We propose utilizing diffusion model with a transformer backbone to address these challenges. Diffusion models are known to effectively handle complex, continuous, multi-modal distributions in policy learning set- tings (Chi et al., 2023), and the long-context capabilities of transformers allow us to efficiently capture the large condi- tioning space. In practice, for all experiments we use the state-of-the-an transformer-based diffusion policy proposed in Dasari et al. (2024). We allocate single token to each conditioning variable-the current state, coverage-to and each state in the history-enabling the model to infer the relationship between these modalities, and at deployment adapt its behavior in-context based on the observed history.
world robotic tasks. Additional details on all experimental setrings are given in Section B.
5.1. Fast Exploration in RL Benchmarks
For our RL experiments, we evaluate BE on a subset of the environments in the D4RL benchmark (Fu et al., 2020), fo- cusing in particular on settings that require exploration. For each D4RL setting, standard dataset exists to enable offline learning; we utilize these datasets to train our approach, and evaluate online. Notably, the data in these datasets is from a scripted policy and is not optimal. While D4RL is pri- marily an offline RL benchmark, a variety of recent works have used it benchmark online learning and exploration approaches initialized from offline data (Zhang et al., 2023; Uchendu et al., 2023; Li et al., 2023; Wilcoxson et al., 2024). Our focus in this section will be to compare with such RL ap- proaches. In particular, we consider the setting proposed by Li et al. (2023) and Wilcoxson et al. (2024), where reward labels are removed from the offline data and the agent must explore online in order to find reward; note that standard offline RL and offline-to-onlin RL algorithms do not di- rectly apply here as they require reward information offline.
Selecting the history distribution. At deployment, our history h may contain a wide distribution of different states, correponniig to those encountered by the policy online, and we would like the policy to infer, for any such set of states, where it should direct its exploration. As such, we should ideally choose the training history distribution, HD)), to correspond to the induced distribution of online states. Es- timating this induced distribution is challenging, however, and requires solving a fxed-point problem. We note, though, that this distribution is simply a reweighting of the distribu- tion induced by T.. Given this, in practice we simply choose H(D) to be a uniform distribution over trajectories in D, thereby enabling Res in the ideal case, to learn exploratory behaviors over all histories it is likely to encounter.
Environment details. We focus our experiments on the state- basee Antmaze and Kitchen environments from D4RL, where the tasks, respectively, are to navigate an ant agent in maze in order to find goal location, and use robotic arm to accomplish various tasks in kitchen. For Antmaze, we evaluate on the medi um and large variants of the maze using the derse offline dataset, and for each test with four distinct goal locations-which are initially unknown to the agent and must be found by exploring the maze- utilizing the same goal locations as are used by Wilcoxson et al. (2024). For Kitchen, we utilize the partial variant of the offline data. As our objective is to evaluate the ability of our approach to explore and adapt online, rather than the standard RL objective of learning an optimal policy, we modify the standard setup somewhat. First, for both environ- ments, as a metric of success we consider the time the agent takes to achieve each task at least once, and for Antmaze we also evaluate maze traversal ability, measuring this in terms of the number of regions reached (where we define a region to be a block in the maze grid, see Figure 12). Second, to increase the challenge of exploration and test fast-adaptation ability, we shorten the episode length in the Antmaze envi- ronment and evaluate on a relatively short number of total environment steps compared to standard RL evaluations (20k and 3k for Antmaze and Kitchen, respectively).
Incorporating task conditioning. Assume that for each r' we have corresponding task label, for example, language command. We can easily modify (4) to incorporate such task labels by simply conditioning on y" as well:
T K En-H(D) [log (a she v¹ h, cov(h U U.))).
Maximizing this objective enables us to restrict our behavior to within the space of given task, thereby further focusing our exploration to the most relevant parts of the space.
5. Experiments
In our experimental evaluation, our focus is on understand- ing (a) whether BE is able to learn effective exploration strategies from offline demonstration data and adapt quickly online, (b) if BE is able to effectively focus its exploration over the space of behaviors present in the demonstration data, and (c) if BE scales to large-scale, real-world imitation learning (IL. settings. Since our method combines elements of both RL and IL, we aim to show that it can perform well in both settings. We first focus on RL benchmarks, where we compare against RL- basdd approaches to exploration, and then on II., where we consider both simulated and real-
Benchmark algorithms. We compare against four repre- sentative algorithms from those evaluated in Wilcoxson et al. (2024), as well as a BC baseline.
Online RND: The RND algorithm (Burda et al., 2018) is a canonical approach for exploration that incentivizes ex-
6

---

Behavioral Exploration: Learning to Explore via In-Context Adaptation
BE [Ours]
50
BC
Accompiished
Goals Reached
SUPE
20
75
15
RND
Regions
50
25
10
25
0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
Steps
x10
Steps
x10*
Steps
x10'
Figure 3: Aggregated performance
Figure Aggregated performance across D4RL Antmaze medi um and large, measuring number of regions reached.
Performance on D4RL Kitchen, measuring percentage of tasks accomplished.
Figure 5:
across D4RL Antmaze medium um and large, measuring percentage of goals reached.
ploratory behavior by labeling observations with a pseudo- reward intended to reflect uncertainty. To achieve this, neural network is randomly initialized and, online, the agent trains another network to match the predictions of this net- work over the observed states. The pseudo-reward is the mis- match between the predictions of the networks; intuitively, unvisited states will have larger prediction error, and there- fore pseudo-reward, than visited states. We run RND using standard online RL, ignoring the offline data completely.
the checkpoints provided by Wilcoxson et al. (2024) (which are trained on the provided offline datasets with reward la- bels removed), and run each method exactly as described in Wilcoxson et al. (2024). We train and run BE as described in Section 4.2, training with 8 different random seeds on the provided offline datasets, and in online deployment feeding the data collected in previous episodes into the learned pol- icy's 's context. We remark that BE relies only on in-context online adaptation, while all other baselines considered here (with the exception of BC) rely on more expensive online gradient-basee based updates. All methods are evaluated online with 80 trials (10 per offline seed). Error bars denote 1 standard error.
ExPLORe: The ExPLORe algorithm (Li et al., 2023) is variant of online RND, but makes use of the offline data well. This approach initializes an online RI. algorithm by feeding the offline data into its replay buffer, and labels this offline data with the RND rewards.
Results. Our results on D4RL are illustrated in Figures to 5. On Antmaze, BE reaches the goals much more quickly and achieves significantly higher coverage than the con- sidered baselines. In particular, possible search space of Antmaze is significantly larger than simply covering the (æ,, y)-locations in the grid- we can also attempt to cover the entire space of possible ant configuration states. However, the demonstration data con- tains a much lower-dimensional space of ant configurations, corresponding to standard walking motions, and we ob- that BE focuses its exploration on such behaviors, ulti- mately enabling effective exploration over the (z, y)-space. In Figure 6 we fur- ther illustrate the cali- bration the learned BE policy, plotting the total number of regions visited against the cov- erage value condi- tion on at deployment as this illustrates, the policy has learned cali- brated behavior and is able to adjust the degree to which it explores by tuning the coverage value. On Kitchen, our approach performs comparably the state-of-the-ar skill- based RL approaches. We emphasize that able to achieve this via simple supervised-leammin based-
HILP: HILP is an offline unsupervised skill discovery method recently proposed by Park al. (2024). It first trains set of "skill" policies the offline data, learning skills that efficiently traverse learned representation space. Skills are indexed by latent vector 2, which can be utilized as a transformation of the action space. That is, rather than taking actions in the original action space, we can think of as the action, and execute it by playing the corresponding skill policy. We take this approach for the HILP baseline, running online RL algorithm over the skill space. In particular, we use the "HILP with offline data" variant given in Wilcoxson et al. (2024), which feeds the offline data into the replay buffer of the online RI. algorithm.
note that the entire
Regiens Visited
20
SUPE: SUPE, proposed by Wilcoxson et al. (2024), an- other Akkillbbseee approach. Similar to HILP trains a skill policy on the offline data, and then online learns policy over skill space. In addition, SUPE utilizes RND rewards to incentivize exploration online, and feeds the offline data, labeled with the RND rewards, into its replay buffer.
16
50
100
Coverage Conditioning Value (1)
Figure 6: Calibration of BE policy on Antmaze medium um.
BC: As a final baseline, we compare against standard BC. We utilize the same diffusion policy base as is utilized by our method, conditioning only on state. At deployment, we simply run this leameed policy online.
For all methods which require offline training, utilize

---

Behavioral Exploration: Learning to Explore via In-Context Adaptation
approach, coupled with a policy that adapts online in- context, in contrast to the more complex RL training of SUPE and HILP. Furthermore, our approach yields sig- nificant improvements over BC, the standard non-adaptive supervised learning approach.
tion learning benchmark, we focus on comparing with other imitation learning approaches. We compare against BC baseline, which utilizes the same diffusion policy base as our approach, and several variants of standard BC: BC with ac- tion noise, where we add a small amount of Gaussian noise to the actions produced by the BC policy, and BC with his- tory conditioning, where we train a BC policy conditioned on a history of past observations. For the hidden-tass eval- uation, we also consider deploying a task-conditioned BC policy with a randomly sampled task at each attempt (sim- ulating the use of task-conditioned policy in settings where the task is ambiguous) and SUPE, the rest-performance RL baseline considered in the previous section. We again deploy BE as described in Section 4.2, feeding observations from the past episodes collected online into its context. For each method considered here, with the exception of SUPE, we train 5 policies with different random seeds. For each policy, we evaluate on each of the 90 tasks in the benchmark 3 times. For SUPE we evaluate it once on each of the 90 tasks.
5.2. Exploration in Imitation Learning
We next consider simulated and real-world vision- -bsee imitation learning settings. In simulation, we utilize the Libero benchmark (Liu et al., 2024), which simulates a vari- city of robotic manipulation and pick-and-place tasks, while in the real world, we train a policy for object manipulation on the Bridge dataset (Walke et al., 2023), and evaluate the WidowX robot. Both settings cover wide range of environments and tasks, and demonstrate the ability of our approach to learn from such diverse data. Our focus here is to test the ability of our method to explore over the space of expert behaviors represented in the training data, while still learning useful task-solving behaviors.
Results. Our results are given in Figures 7 and 8, BE much more quickly solves the correct task in the hidden-task evaluation-aachieinng
5.2.1. SIMULATED RESULTS ON LIBERO BENCIMARK
2-3x reduction in the number of attempts required compared to standard BC-and similarly solves the task more quickly (and achieves higher final success rate) than BC in the evaluation with task conditioning. This demonstrates the ability of BE to explore over the space of behaviors represented in the expert demonstrations-inssee of exploring randomly, BE explores by selecting different task-solving behaviors from the demonstration dataset. This exploration does not hurt its ability to successfully solve tasks, but instead enables BE to attempt diverse behaviors likely to solve the task, achieving higher task success rate than BC. We see that this is in contrast to augmenting BC with action noise. While BC with action noise does induce some amount of additional exploration, improving the BC policy's 's success rate in the hidden-ter evaluation, action noise significantly hurts BC's performance in the evaluation with task conditioning, likely due to the action noise causing the behavior to deviate too far from the demonstrated task-solving behavior. This highlights the critical advantages of behavioral exploration over random exploration in inducing behaviors that are exploratory, but which still lead to high task success rates. Similarly, we see that naively conditioning on history can significantly hurt BC performance, phenomenon that has been observed in the existing literature (De Haan et al., 2019), yet the history conditioning employed by BE enables The to utilize the history to actually improve its performance.
Environment details. The Libero benchmark contains a variety of scenes, simulat- ing different robotic manipula- tion and lock-and-place tasks. Figure 10 visualizes such example tasks in the lower left scene include "turn on the stove and put the frying pan on it" or "put the moka pot on the stove" (see Section B.2 for additional task descriptions). on the Libero 90 dataset, which includes 90 tasks spread across 21 distinct scenes. For each task, the dataset provides 50 human demonstrations of successful completion, which we utilize our demonstration dataset. We consider two evaluation settings. In the first, we choose a task we would like the agent to solve, but do not instruct the policy which task we have chosen. As each scene contains from 2-7 dif- ferent tasks, to successfully solve the hidden task, the agent must attempt and successfully complete different tasks until it has completed the hidden one. This simulates settings where the goal is ambiguous and we would like the agent to explore possible solutions until it arrives at the correct one. We count a trial as a success if the hidden task has been solved at least once across 15 attempts. In our second evaluation, we do provide a task designation, and evaluate the ability of each policy to solve the task after 6 attempts, testing the ability of policy to try diverse behaviors for given task.
Figure 10: Example Libero tasks.
We run all experiments
5.2.2. REAL- WORLD RESULTS ON WIDOWX ROBOT
For our real-world experiments, we train a policy on the BridgeData V2 dataset (Walke et al., 2023), which contains diverse set of over 60,000 robotic teleoperation demon-
Benchmark algorithms. As Libero is primarily an imita-
8

---

Behavioral Exploration: Learning to Explore via In-Context Adaptation
100
Hidden Success (%6
Attempts Success 3
Trial Success (%)
80
60
Task Success
80
60
40
70
40
8C lAction Notoo!
60
(Hitoryl
(Random
50
SUPE
10
15
0 (0ural KG
Operit
6z (Pur)
OpenPuA
Number Attempts
Number Attempts
Figure 7: Performance on Libero in evaluation with task provided.
Figure 8: Performance on Libero in Figure 9: Performance on WidowX robot in reaching evaluation.
evaluation with task hidden.
strations across a wide range of environments and tasks. This setting demonstrates the ability of our approach to scale to real-world deployment and large-scale Fision-based datasets, and learn from such diverse demonstration data.
in Figure 9, where we plot the average number of success- ful trials, and average number of attempts to success (in cases where the policy did not interact with both objects in attempts, we count this as attempts). We see that our ap- proach succeeds roughly 40% more than BC or OpenVLA. and requires on average 2 fewer attempts succeed. As in the Libero experiments, this illustrates the ability of BE to focus its exploration over semantically meaningful features rather than exploring the entire space of robot configura- tions, explores only over the types of reaching and grasp- ing behaviors present in the demonstration dataset- and demonstrates the scalability of our approach to large-scale, real-world settings.
Environment details. We eval- uate policy the WidowX 250 6-DoF robot arm. We con- sider 3 different reaching tasks, where for each task, we place? 2 objects in front of the robot, and the goal is to interact with each ob- ject in some way (see Figure 11). A "trial" consists of 5 consecutive episodes in the same scene-we count a trial as success if the pol- icy interacts with both objects at least once across the attempts (and consider any instance of the robot's end-effector mak- ing contact with one of the objects as a successful "inter- action"). Similar to Libero, this simulates settings where the robot is given an ambiguous task specification, and we would like it to attempt different behaviors to arrive at the correct one.
6. Discussion
Figure 11: Example WidowX setup.
In this work we have proposed behavioral exploration, a novel approach to training policies that explore over the space of behaviors contained in a demonstration dataset. BE requires only a small modification to the standard BC objective train, and experimental results demonstrate that BE leads to substantial gains over both standard BC as well as RI-based exploration approaches. We believe our work opens several directions for future work:
Benchmark algorithms. We again compare against a BC baseline, using the same base diffusion model as our method. We also compare against OpenVLA, a state-of- the-art wanguugge-ondiiiionnn et al., 2024). We note that OpenVLA is trained on dataset much larger than Bridge, and has roughly 100x more pa- rameters than our policy. While this gives it an advantage in a certain sense, we nonetheless include it as a represen- tative example of generalist robot policy, and to evaluate the ability of generalist policies "exxlore". In particular, seek to understand whether OpenVLA, deployed with standard prompting, is able to solve the goal task in settings where the goal task is ambiguous given the prompt, and exploration over possible solutions is required (please see Section B.3 for additional discussion on how we prompt OLe) For each task and method, we evaluate on 5-episode trials.
While BE explores over the behaviors present in the demonstration data, there may exist scenarios where the exploration necessary learn to solve task requires go- ing beyond the space of demonstration data. How best utilize the prior information encoded in the demon- strations to speed exploration in such settings?
generalist robot policy (Kim
Much attention has been devoted to investigating the ability LLMs to explore, with mixed results (Krish- namurthy al., 2024; Nie al., 2024; Monea al., 2024). Could BE be combined with LLMs for example, by finetuning LLM the BE objective-to enable effeective exploration in language-based domains?
Our WidowX results demonstrate that BE effectively scales to real-world settings. Scaling this further and combining BE with state of-the-art generalist robot poli- cies (Kim et al., 2024; Black et al., 2024; Bjorck et al., 2025) is of great interest.
Results. The results for our real-world experiment given

---

Behavioral Exploration: Learning to Explore via In-Context Adaptation
Acknowledgements
Black, K., Nakamoto, M., Atreya, P., Walke, H., Finn, C., Kumar, A., and Levine, S. Zero-shot robotic manipulation with pretrained image-editing diffusion models. arXiv preprint arXiv:2310. 10639, 2023.
The authors would like to thank the members of RAIL Lab and Max Simchowitz for helpful conversations. This re- search was partly supported by the RAI Institute, ONR N00014-22-1-2773 and N00014-25-1-2060 and NSF IIS- 21 2150826. This research used the Savio computational clus- ter resource provided by the Berkeley Research Computing program at UC Berkeley.
Black, K., Brown, N., Driess, D., Esmail, A., Equi, M., Finn, C., Fusai, N.. Groom, L., Hausman, K., Ichter, B., et al. Tp: A vision-language-actinnn flow model for general robot control. arXiv preprint arXiv:2410221664, 2024.
Impact Statement
Bojarski, M. End to end learning for self-driving cars, arXiv preprint / 004.07316, 2016.
Exploration can potentially lead to unsafe behaviors- by incentivizing an agent to attempt new behaviors, it is pos- sible some of the behaviors attempted will lead to unsafe actions. While we believe our approach should mitigate this as compared to standard exploration approaches, care should nonetheless be taken. More broadly, advancing agentic sys- tems may have variety of societal applications. However, do not feel need to highlight any in particular here.
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Dabis, J.,
Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A.,
Hsu, J., et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.068177 2022.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
References
Ajay, A., Kumar, A., Agrawal, P., Levine, S., and Nachum, O. Opal: Offline primitive discovery for acceler- ating offline reinforcement learning. arXiv preprint arXiv:2010.136111 2020.
Burda, Y., Edwards, H., Storkey, A., and Klimov, O. Ex- ploration by random network distillation. arXiv preprint arXiv: /810. 1810.18894, 2018
Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. De- cision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:1 34:15084-15097 2021.
Ajay, A., Du, Y., Gupta, A., Tenenbaum, J., Jaakkola, T., and Agrawal, P. Is conditional generative model- ing all you need for decision-making? arXiv preprint arXiv:2211.15677 2022.
Chi, C., Xu, Z.. Feng, S., Cousineau, E., Du, Y., Burchfiel, B., Tedrake, R., and Song, S. Diffusion policy: Visuomo- tor policy learning via action diffusion. The International Journal of Robolics Research, pp. 027836492412736668 2023.
Argall, B. D., Chernova, S., Veloso, M., and Browning, B. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469 483, 2009.
Ball, P. J.. Smith, L., Kostrikov, I., and Levine, S. Ef- ficient online reinforcement learning with offline data. In International Conference on Machine Learning, pp. 1577-1594. PMLR, 2023.
Choi, J., Guo, Y., Moczulski, M., Oh, J., Wu, N., Norouzi, M., and Lee, H. Contingency-awan exploration in re- inforcement learning. arXiv preprint 1eXiv:18/1.011 2018
Beck, J., Vuorio, R., Liu, E. Z., Xiong, Z., Zintgraf, L., Finn, C., and Whiteson, S. A survey of reitrreinfor learning. arXiv preprint
Cui, Y., Nickum, S., Gupta, A., Kumar, V., and Rajeswaran, A. Can foundation models perform zero-shot task specifi- cation for robot manipulation? In Learning for dynamics and control conference, pp. 893- 905. PMLR, 2022a.
2023.
Bellemare, M., Srinivasan, S., Ostrovski, G., Schaul, T., Saxton, D., and Munos, R. Unifying count-based ex- ploration and intrinsic motivation. Advances in neural information processing systems, 29, 2016.
Cui, Z. J., Wang, Y., Shafiullah, N. M. M., and Pinto, L. From play to policy: Conditional behavior gen- eration from uncurated robot data. arXiv preprint arXiv:2210. 10047, 2022b.
Bjorck, J., Castañeda, F., Cherniadev, N., Da, X., Ding, R., Fan, L., Fang, Y., Fox, D., Hu, F., Huang, S., et al. Gr0Ot nl: An open foundation model for generalist humanoid robots. arXiv preprint arXiv:2503.1477 2025.
Dasari, S. and Gupta, A. Transformers for one-shot visual imitation. In Conference on Robot Learning, pp. 2071- 2084. PMLR, 2021.
10

---

Behavioral Exploration: Learning to Explore via In-Context Adaptation
Dasari, S., Mees, O., Zhao, S., Srirama, M. K., and Levine, S. The ingredients for robotic diffusion transformers. arXiv preprint htt:::441..008888 2024.
Ghosh, D., Gupta, A., Reddy, A., Fu, J., Devin, C., Eysenbach, B., and Levine, S. Learning reach goals via iterated supervised learning. arXiv preprint arXiv: htth:/912.00088. 2019.
De Haan, P., Jayaraman, D., and Levine, S. Causal confu- sion in imitation learning. Advances in neural information processing systems, 32, 2019.
Ghosh, D., Ajay, A., Agrawal, P., and Levine, S. Offline rl policies should be trained to be adaptive. In Interna tional Conference on Machine Learning, pp. 7513-7530. PMLR, 2022.
Ding, Y., Florensa, C., Abbeel, P., and Phielipp. M. Goal- conditioned imitation learning. Advances in neural infor- mation processing systems, 32, 2019.
Gu, J., Kirmani, S., Wohlhart, P., Lu, Y., Arenas, M. G., Rao, K., Yu, W., Fu, C., Gopalakrishnan, K., Xu, Z., et al. Rt-trajectory: Robotic task generalization via hindsight trajectory sketches. arXiv preprint arXiv:2311.01977, 2023.
Dorfman, R., Shenfeld, I., and Tamar, A. Offline meta reinforcement leamiing-identfiabiliyy challenges and ef- fective data collection strategies. Advances in Neural Information Processing Systems, 34:4607-465 4618, 2021.
Henaff, M., Raileanu, R., Jiang, M., and Rocktäschel, T. Exploration via elliptical episodic bonuses. Advances in Neural Information Processing Systems, 35:37631- 37646, 2022.
Duan, Y., Schulman, J., Chen, X., Bartlett, P. L., Sutskever, I, and Abbeel, P. RI2: Fast reinforcement learn- ing via slow reinforcement learning. arXiv preprint arXiv: 1611,02779 2016.
Hu, H., Yang, Y., Ye, J., Mai, Z., and Zhang, C. Unsu- pervised behavior extraction via random intent priors. Advances in Neural Information Processing Systems, 36: 51491-51514, 2023.
Duan, Y., Andrychowicz. M., Stadie, B., Jonathan Ho, O., Schneider, J., Sutskever, I., Abbeel, P., and Zaremba, W. One-shot imitation learning. Advances in neural information processing systems, 30, 2017.
Huang, S., Hu. J., Chen, H., Sun, L.. and Yang, B. In-context decision transformer: Reinforcement learn- ing via hierarchical chain-of of-thought. arXiv preprint 2rXiv:2405.20692 2024.
Ecoffet, A., Huizinga, J., Lehman, J., Stanley, K. O., and Clune, Go-explore: new approach for hard- exploration problems. arXiv preprint 1rXiv:1901.10995 2019.
James, S., Bloesch, M., and Davison, A. J. Task-embedded control networks for few-shot imitation learning. In Con- ference on robot learning, pp. 783-795. PMLR, 2018.
Emmons, S., Eysenbach, B., Kostrikov, I., and Levine, S. Rys: What is essential for offfine rl via supervised learn- ing? arXiv preprint arXiv:21 12. 12. 10751, 2021.
Jiang, Y., Kolter, J. Z., and Raileanu, R. On the importance of exploration for generalization in reinforcement learn- ing. Advances in Neural Information Processing Systems, 36, 2024.
Finn, C., Yu, T., Zhang, T., Abbeel, P., and Levine, S. One- shot visual imitation learning via meta-learning. In Con- ference on robot learning, pp. 357-368. PMLR, 2017.
Kamienny, P.-A., Pirotta, M., Lazaric, A., Lavril, T. Usunier, N., and Denoyer, L. Learning adap- tive exploration strategies in dynamic environments through informed policy regularization. arXiv preprint arXiv:2005.00 02934, 2020.
Foster, D. J., Block, A., and Misra, D. Is behavior cloning all you need? understanding horizon in imitation learning. arXiv preprint arXiv:2407.15000 2024.
Fu, J., Kumar, A., Nachum, O., Tucker, G. and Levine, S. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.072199 2020.
Kim, M. Pertsch, K., Karamcheti, S., Xiao, T., Balakr- ishna, A., Nair, S., Rafailov, R., Foster, E., Lam, G., San- keti, P., al. Openvla: An open-source fision-language action model. arXiv preprint arXiv:2406.09246, 2024.
Fu, L., Huang, H., Datta, G., Chen, L. Y., Panitch, W. C.- H., Liu, F,, Li, H., and Goldberg, K. In-context imita- tion learning via next-token prediction. arXiv preprint arXiv:2408. 15980, 2024.
Krishnamurthy, A., Harris, K., Foster, D. J., Zhang, C., and Slivkins, A. Can large language models explore in-context? arXiv preprint arXiv:2403. 2024.
Furuta, H. Matsuo, Y., and Gu, S. S. Generalized decision transformer for offline hindsight information matching. arXiv preprint arXiv:2 11.2111111364 2021.
Kumar, A., Peng, X. B., and Levine, Reward-ccnditione conditioned policies. arXiv preprint arXiv: 13465, 2019.

---

Behavioral Exploration: Learning to Explore via In-Context Adaptation
Laskin, M., Wang, L., Oh, J., Parisotto, E., Spencer, S., Steigerwald, R., Strouse, D., Hansen, S., Filos, A., Brooks, E., et al. In-context reinforcement learning with algorithm distillation. arXiv preprint arXiv:2210.14215, 2022.
Mandlekar, A., Xu, D., Wong, J., Nasiriany, S., Wang. C., Kulkarni, R., Fei-Fei, L.,, Savarese, S., Zhu, Y., and Martín-Martín, R. What matters in learning from offline human demonstrations for robot manipulation. arXiv preprint arXiv:2108.0322) 2021.
Lee, J. Xie, A., Pacchiano, A., Chandak, Y., Finn, C., Nachum, O., and Brunskill, E. Supervised pretraining can learn cn-tontext reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Mishra, N., Rohaninejad, M., Chen, X., and Abbeel, P. A simple neural attentive meta-learner. arXiv preprint vrXiv:1707.000 2017.
Mitchell, E., Rafailov, R., Peng, X. B., Levine, S., and Finn, C. Offline meta-reinforcemenn learning with advan- tage weighting. In International Conference on Machine Learning, pp. 7780-7791, PMLR, 2021.
Lee, K., Laskin, M., Srinivas, A., and Abbeel, P. Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning. In International Conference Machine Learning, pp. 6131-6141. PMLR, 2021.
Monea, G., Bosselut, A., Brantley, K., and Artzi, Y. Llms are in-context reinforcement leamners 2024.
Lee, K.-H., Nachum, O., Yang, M. S., Lee, L.,, Free- man, D., Guadarrama, S., Fischer, I., Xu, W., Jang, E., Michalewski, H., et al. Multi-game decision transformers. Advances in Neural Information Processing Systems, 35: 27921-27936, 2022a.
Nair, S., Mitchell, E., Chen, K., Savarese, S., Finn, C., et al. Learning Language-condttionn robot behavior from offline data and crowd-sourced annotation. In Conference on Robot Learning, pp. 1303-1315. PMLR, 2022.
Lee, S., Seo, Y., Lee, K., Abbeel, P., and Shin, J. Offline- to-online reinforcement learning via balanced replay and pessimistic qeensebble. In Conference on Robot Learn- ing, pp. 1702-1712. PMLR, 2022b.
Nakamoto, M., Zhai, S., Singh, A., Sobol Mark, M., Ma, Y., Finn, C., Kumar, A., and Levine, S. Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning, Ad- vances in Neural Information Processing Systems, 36. 2024.
Li, L.,, Yang, R., and Luo, D. Focal: Efficient fully- offline meta-reinffrrcement learning via distance met- ric learning and behavior regularization. arXiv preprint arXiv:2010.0111 112, 2020.
Nie, A., Su, Y., Chang, B., Lee, J.N Chi, E. H. Le, Q. V., and Chen, M. Evolve: Evaluating and optimizing llms for exploration. arXiv preprint arXiv:2410.06238 2024.
Norman, B. and Clune, J.
First-explore, then exploit: Meta-learning intelligent exploration. arXiv preprint WrXi::2307002276, 2023.
Li, Q., Zhang, J., Ghosh, D., Zhang, A., and Levine, S. Accelerating exploration with unlabeled prior data. Ad- vances in Neural Information Processing Systems, 36: 67434-67458, 2023.
Osband, I., Blundell, C.,, Pritzel, A., and Van Roy, B. Deep exploration via bootstrapped dqn. Advances in neural information processing systems, 29, 2016.
Liu, B., Zhu, Y., Gao, C., Feng, Y., Liu, Q. Zhu, Y., and Stone, P. Libero: Benchmarking knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems, 36, 2024.
Park, S., Kreiman, T., and Levine, Foundation policies with hilbert representations. arXiv preprint arXiv: 2024.
Liu, E. Z., Raghunathan, A., Liang, P., and Finn, C. Decou- pling exploration and exploitation for menerreeiforr learning without sacrifices. In International conference machine learning, pp. 0925-6935. PMLR, 2021.
Pertsch, K., Lee, Y., and Lim, J. Accelerating reinforcement learning with learned skill priors. In Conference on robot learning, pp. 188-28 288-204. PMLR, 2021.
Pong, V. H., Nair, A. V., Smith, L. M., Huang, C., and Levine, S. Offline meta-reinforcement learning with on- line self-supervision. In International Conference Machine Learning, pp. 17811-17829, PMLR, 2022.
Liu, H. and Abbeel, P. Emergent agentic transformer from chain of hindsight experience. In International Confer- ence on Machine Learning, pp. 21362-21374 PMLR, 2023.
Pronzato, L. and Pázman, A. Design of experiments in nonlinear models. Lecture notes in statistics, 212(1), 2013.
Lynch, C., Khansari, M., Xiao, T., Kumar, V., Tompson, J.,
Levine, S., and Sermanet, P. Learning latent plans from play. In Conference on robot learning, pp. 1113-1132. PMLR, 2020.
Pukelsheim, F. Optimal design of experiments. SIAM, 2006.
12

---

Behavioral Exploration: Learning to Explore via In-Context Adaptation
Rafailov, R., Vijay, V. K. Yu, T., Singh, A., Phielipp, M., and Finn, C. The reflective explorer: Online meta- exploration from offline data in realistic robotic tasks. In Deep RL Workshop NeurlPS 2021, 2021.
Stepputtis, S., Campbell, J., Phielipp. M., Lee, S., Baral, C., and Ben Amor, H. Language--onditioonn imitation learning for robot manipulation tasks. Advances in Neural Information Processing Systems, 33:13139-1111 2020.
Rahmatizadeh, R., Abolghasemi, P., Bölöni, L., and Levine, S. Vision-based multi-task manipulation for inexpen- sive robots using end-to-end learning from demonstration. In 2018 IEEE international conference on robotics and automation (ICRA). pp. 3758-3766 IEEE, 2018.
Sundaresan, P., Vuong, Q. Gu, J., Xu, P., Xiao, T., Kirmani, S., Yu, T., Stark, M., Jain, A., Hausman, K., et al. Rt- sketch: Goal-conditioned imitation learning from hand- drawn sketches. 2024.
Team, O. M., Ghosh, D., Walke, H., Pertsch, K., Black, K., Mees, O., Dasari, S., Hejna, J., Kreiman, T., Xu, C., et al. Octo: An open-source generalist robot policy, arXiv preprint 2.0.::::405.222133 2024.
Rakelly, K., Zhou, A., Finn, C., Levine, S., and Quillen, D. Efficient off- ooflcy meta-reinforcemment learning via prob-
abilistic context variables. In International conference on machine learning, pp. 5331-5340. PMLR, 2019.
Uchendu, I., Xiao, T., Lu, Y., Zhu, B., Yan, M., Simon, J., Bennice, M., Fu, C., Ma, C., Jiao, J., et al. Jump-start reinforcement learning. In International Conference on Machine Learning, pp. 34556-34583 PMLR, 2023.
Raparthy, S. C., Hambro, E., Kirk, R., Henaff, M., and Railcanu, R. Generalization to new sequential decision making tasks with in-context learning. arXiv preprint arXiv:2312.0388) 2023.
Van der Vaart, A. W. Asymptotic statistics, volume Cam- bridge university press, 2000.
Ross, S., Gordon, G., and Bagnell, D. A reduction of imita- tion learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 627-635. JMLR Workshop and Conference Proceedings, 2011.
Walke, H. R., Black, K., Zhao, T. Z., Vuong, Q. Zheng, C., Hansen-Estruch, P., He, A. W., Myers, V., Kim, M. Du, M., et al. Bridgedata v2: A dataset for robot learning at scale. In Conference on Robot Learning, pp. 1723-1736. PMLR, 2023.
Schmidhuber, J. Reinforcement learning upside down: Don't predict rewards-just map them to actions. arXiv preprint
Wang. X., Kurth-Nelson, Z., Tirumala, D., Soyer, H., Leibo, J. Z., Munos, R., Blundell, C., Kumaran, D., and Botvinick, M. Learning reinforcement learn. arXiv preprint arXiv:1611.05700 2016.
2019.
Schmied, T., Paischer, F., Patil, V.. Hofmarcher, M., Pas- canu, R., and Hochreiter, S. aetrivval-augmentee de- cision transformer: External memory for in-context rl. arXiv preprint ar arXiv:2410.0707// 2024.
Wilcoxson, M., Li, Q., Frans, K., and Levine, S. Leverag- ing skills from unlabeled prior data for efficient online exploration. arXiv preprint arXiv:2410. 18076, 2024.
Shafiullah, N. M., Cui, Z., Altanzaya, A. A., and Pinto, L. Behavior transformers: Cloning k modes with one stone. Advances in neural information processing systems, 35: 22955 -2968, 2022.
Wu, Y.-H., Wang, X., and Hamaya, M. Elastic decision transformer. Advances in Neural Information Processing Systems, 36, 2024.
Yan, K. Schwing, A. G., and Wang, Y.-.. Reinforcement learning gradients vitamin for online finetuning de- cision transformers. arXiv preprint arXiv:2410.24108. 2024.
Shyam, P., Jaskowski,, W., and Gomez, Model-based ac- tive exploration. In International conference on machine learning, pp. 5779-5788. PMLR, 2019.
Singh, A., Liu, H., Zhou, G., Yu, A., Rhinehart, N., and Levine, S. Parrot: Data-driven behavioral priors for inforcement learning, arXiv preprint arXiv:2011.1003. 2020.
Zhang, H., Xu, W., and Yu, H. Policy expansion for bridging offline-to-onle reinforcement learning. arXiv preprint wwXi:2330.000355, 2023.
Zhang, J.. Wang. J., Hu, H., Chen. Y., Fan, C., and Zhang, C. Leaan to effectively explore in context-based meta-rl. arXiv preprint arXiv:2006.08170 2020.
Srivastava, R. K., Shyam, P., Mutz, F., Jaskowski, W., and Schmidhuber, J. Training agents using upside-down re- inforcement learning, arXiv preprint arXiv:1912.02877 2019.
Zhang, T., McCarthy, Z.,, Jow, O., Lee, D., Chen, X., Gold- berg, K., and Abbeel, P. Deep imitation learning for complex manipulation tasks from virtual reality teleoper- ation. In 2018 IEEE international conference on robotics and automation (ICRA), pp. 5628-5633 IEEF, 2018
Stadie, B. C., Levine, S., and Abbeel, P. Incentivizing ex- ploration in reinforcement learning with deep predictive models. arXiv preprint 107,0000 2015.
13

---

Behavioral Exploration: Learning to Explore via In-Context Adaptation
Zhao, T. Z., Kumar, V., Levine, S., and Finn, C. Learn- ing fine-grand fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304.13700 2023.
Zheng, H., Luo, X., Wei, P., Song, X., Li, D., and Jiang. Adaptive policy learning for offline-to-online reinforce- ment learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pp. 11372-11380, 2023.
Zhou, Z., Atreya, Lee, A., Walke, H., Mees, O., and Levine, S. Autonomous improvement of instruction following skills via foundation models. arXiv preprint arXiv:2407.20633, 2024.
Zintgraf, L.. Shiarlis, K., Igl. M., Schulze, S., Gal, Y., Hof- mann, K., and Whiteson, S. Varibad: very good method for bayes-adaptin deep rl via meta-learning. arXiv preprint arXiv: 1900.0344 2019.
14

---

Behavioral Exploration: Learning to Explore via In-Context Adaptation
A. Proof of Proposition 4.2
[sk = s] and Ω2 := {s (*) > 03. Let C3 == { Es with =** (s) > 0 and (b(s = ), and n2 = |Cyl denote the number of directions in feature space V spans at step K. We then have the following.
For the purposes of the following result, define (**(s)) == Pr
in deterministic environment and that our feature is one-hot encoding and only non-zero at the final state. Furthermore, assume that |A] <, 0o, that each episode starts from state Su, that we only consider n rounds of interaction, and take < min.en w* (s). Let h denote the set of states visited in episodes
Proposition A.1 (Full version of Proposition 4.2). Assume that
. ,,...,-- 1, and let it denote the policy which at episode plays actions according to
a 1,, cov(h; UT) max cov(hi UT)].
Then म. is an optimal solution to Objective 4.1.
Proof. Note that in our setting we have that the optimal solution has Pr [Ø(sk) = = 1/nz for each i E C3 (and does not depend on the visitation to non-terminal states since they all affect coverage equally). Thus, it suffices to show that the policy से in the result statement satisfies Pr [Ø(*x) = i] = 1/n after playing for ni episodes.
We will complete this proof by induction, where we show that at each episode t, we reach a previously unreached terminal state in Cg. Define h, denote the set of states visited episodes 1, ....., and let C/2 Ca 3s E h with w (s) > and ((s) = e.). It suffices then to show that |C' = ng - + 1 for each The base case, t 1, is trivial, since C3 Cg.
Now assume that |C n2 - 1+ 1 for some k < nig, and wish show that IC n2 t. By construction, there exists some sequence of actions that will take us from 8p to a state Sy with ((sx) E Cz. If we can show that we will reach such a state at step H of episode k, the inductive step follows.
To show this, we introduce second inductive step. In partieular, we seek to show that for each step h of episode k; we will be in state sg for which path exists some šk playing only actions in the support of Te, and with Φ(sx) E Cj-- The base case is trivial and is true by assumption, as stated above. For the inductive case, assume that we are in such a state at sk-1. Then we want to show that we will play an action which will cause us to transition to such state at sy. As we condition on the event that (cov(h U {sK) }) = max, cov(h U {s]}, then गे by definition will play an action that will allow us to reach some state for which ((*x) + (ss) for any s E h2 (we note that such an action in the support of T exists at step h by the inductive assumption). Note also that से will only play actions in the support of my at 8k-1 and, furthermore, that since we have played fे up to step h 1, we have always played actions in the support of TT, and therefore T (sk-1) 1) 0.
Let A(sk-) 1) denote the set of actions that would lead to state 8 from which it is not possible to get to terminal state in the set {s dbs) E C3} by playing actions in the support of ky. For notational convenience, let & denote the event cov(h, U {sx]) }) = max, cov U {s}. Then we can show that
n la A(sk-1) 8k-1, 14, E4 =
(5)
we are done. By the definition of conditional probability, we have
pa E A(Sk-1) 88-1, pe E A(sh-1)) N Sk-1, hel
px [ 8k-1, hi)
Note that
pa E A(8k-1)) n 8k-1; hy] pTo A(8 nEn (am....,.... E sup)ort(TT)) 18-11 hn)
by definition of pas, Furthermore, note that the event
(a
ag-1 support(Ta))
implies the event
(Ø(sk) & Cs, sx he}
15

---

Behavioral Exploration: Learning to Explore via In-Context Adaptation
since 1(5=-1) cannot lead any šк with ((sk) Cg. by definition, if only take actions in the support of zg. but since condition on the event & must reach terminal state not in hy. This implies that the above be bounded
P [}(SK) & CB, SK & h2) sk-1,
pnd E 11 he]
However, since (8k-1) O, the final state reached by playing as from this state must satisfy ((8x) E Ca. Thus, we have [{((sk) & Cp.sk & hr) Sk-1, hr] 0. Furthermore, by the inductive assumption we have p | 8k-1, h₂] > 0. Together this proves that (5) holds, which proves the inner inductive step.
The inner induction then immediately implies the outer induction step, that we reach terminal state episode & in Cg so that
|0 = ng - Repeating this for each k ng. we have that ग will visit terminal states {sk} such that {i 3k, (sx)) e Co. which implies that Pr" (Ø((k)
1/ng after playing for ng episodes, thus proving the
result.
16

---

Behavioral Exploration: Learning to Explore via In-Context Adaptation
B. Experiment Details
We make several notes on hyperparameters for all experiments. For all experiments, for both BE and BC, we use the diffusion policy architecture proposed by Dasari al. (2024) and utilize their code base as the starting point for our method, We experiment with several different parameter settings for architecture size, and for all BC evaluations run the same parameter sweep consider for BE, reporting results for the res--performing architectures. For all experiments, chunking, where the model predicts several future actions (in evaluation, we roll out all actions before recomputing new set of actions)-we specify action chunk length used for each setting below. For BE, in addition to standard hyperparameters common with BC, it also requires specifying the context length (history length), and future trajectory length (in settings where r' is very long, how far in the future we should consider when calculating coverage). In deployment, in cases where the context length shorter than the history we have collected, we randomly downsample the history. For all experiments use the definition of cov(-) stated in the main text-we specify what we choose for o(+) in each case below, All BC and BE models were trained with the following hyperparameters in common specify other hyperparameters in the following.
action
Table 1: Common hyperparameters for all BE and BC experiments.
Hyperparameter Value
Learning rate
3e-4
LR scheduler
cosine
Warmup steps
2000
B.1. D4RL Antmaze and Kitchen Experiments
Visualization of D4RL Antmaze medium and large environments. Red squares denote single "region", used to calculate the region count in Figure 4. Please see Figure 2(a) of Wilcoxson et al. (2024) for goal locations.
Figure 12:
Figure 13: Visualization of D4RL Kitchen environment
We utilize the default D4RL enviromments and datasets in all respects except for the episode length Antmaze: for Antmaze medium um we shorten the episode length from 1000 to 500, and for Antmaze Large we shorten from 1000 to 750. The shorter horizon better tests the ability of each approach to realize directed exploration; with a longer horizon randomly exploring over the maze itself an affective strategy. As stated in the text, we evaluate based on the number of goals reached (for Antmaze) tasks completed (for Kitchen). For Antmaze, we utilize the goal locations given in Wilcoxson et al. (2024), and count the goal reached if the agent reaches with distance of 0.5 of a square of side length 1.5 centered at the goal. For Kitchen, we utilize the environments default success detector to determine whether goal is reached. We do not require that these are completed in the same episode, simply over the entire sequence of episodes (so over 20k steps for Antmaze and 3k steps for Kitchen).
For all RL. baselines, we use the implementations and pretraining checkpoints provided by Wilcoxson al. (2024). We
17

---

Behavioral Exploration: Learning to Explore via In-Context Adaptation
utilize the default hyperparameters for each method as given in Wilcoxson et al. (2024) with the exception of the length of the online warmup phase before RL training start, given that time horizon is shorter than is used in Wilcoxson et al. (2024). For the warmup length, for each method in each environment sweep over several different values, and choose the value that leads to the best final performance.
For BE, we utilize the definition for coverage given in the main text, and as a feature vector ((s), feed the state into a randomly initialized neural network, taking the last layer as the feature vector. all cases we use 3-layer fully connected network with hidden dimension 128 and output dimension 32 (and set X 0.01 for our regularization). In deployment, we the history be a randomly selected subset of past states visited (which resampled the start of each episode).
Figure 3 and Figure 4 gives the performance averaged across all goals and both Antmaze medium and large, and Table 2 provides the final values achieved in each plot. Please see Figure 14-19 for individual results.
Table 2: Final success rate from Figure 7 and 8.
BE
BC
SUPE
HILP
ExPLORe
RND
Antmaze (goals)
0.556 1 0.013
0.285
0.015
0.195 1 0.013
0.128 1 0.012
010
0=
Antmaze (regions)
24.118 1 0.253
20.056 1 0.150
16.963 1 0.258
18.934 0.218
3.606 1 0.040
3,475 1 0.044
Kitchen
0.922 1 0.014
0.306 0.017
0.956 1 0.011
0.913 1 0.015
0.075 1 0.013
0.081 1 0.013
Table 3: Hyperparameters for D4RL BE experiments.
Hyperparameter
Antmaze Medium
Antmaze Large
Kitchen
Batch size
256
256
256
Action chunk size
10
10
Hidden size
256
128
64
Number of Heads
4
Number of Layers
4
Feedforward dimension
256
512
512
Coverage future length
200
100
50
Context history length
100
50
50
Table 4: Hyperparameters for D4RL. BC experiments.
Hyperparameter
Antmaze Medium & Large
Kitchen
Batch size
256
256
Action chunk size
10
5
Hidden size
256
64
Number of Heads
8
1
Number of Layers
4
3
Feedforward dimension
512
128
18

---

Behavioral Exploration: Learning to Explore via In-Context Adaptation
0.6
(Ours)
Number of Regions Visited
Fraction of Goals Reached
BC
20
0.5
SUPE
HILP
0.4
15
RND
0.3
10
0.2
0.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
Steps
x10
Steps
x10
Performance on Antmaze medi um, measuring number of goals reached.
Figure 14:
Figure 15:
Performance on Antmaze
medi um, measuring number of regions reached.
100
Stemi
% Thits Reacch Goal
SIPE
Thas Rog
73
56 Thes Seath
50-
25
25
1.0 Stops
1.5
3.5
0.5
15
2.5
0.0
0.5
15
0.
0.5
1.5
20
5taps
(a) Goal 1
(b) Goal
(c) Goal 3
(d) Goal
Figure 16: Performance on individual goals on Antmaze
Fraction of Goals Reached
BE (Ours)
Number of Regions Visited
25
0.5
SUPE
20
0.4
ExPLORe
RND
15
0.3
0.2
10
0.1
0.0
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
Steps
x10
Steps
x10*
Figure
17:
Performance on Antmaze
Figure 18:
Performance on Antmaze
large, measuring number of goals reached.
large, measuring number of regions reached.
100
Tride Bendhinng Cran
Trals doigiing Gial
75
75
75
50
50
Traas
Imalk
25
Stog5
x10'
30055
X
S0.05
>300
ST055
(a) Goal
(b) Goal
(c) Goal
(d) Goal
Figure 19: Performance on individual goals Antmaze large.
19

---

Behavioral Exploration: Learning to Explore via In-Context Adaptation
B.2. Libero Experiments
Table 5:
Libero tasks for scenes in Figure 10.
Scene
Task Commands
Upper Left
pick up the alphabet soup and put in the tray pick up the butter and put it in the tray pick up the cream cheese and put it in the tray pick up the ketchup and put it in the tray pick up the tomato sauce and put it in the tray
Upper Right
close the top drawer of the cabinet put the black bowl in the top drawer of the cabinet
put the black bowl on the plate put the black bowl on top the cabinet put the ketchup in the top drawer of the cabinet
Lower Left
put the frying on the stove put the moka pot on the stove turn on the stove on the
and put the frying pan it
Lower Right
pick up the book and place in the front compartment of the caddy pick up the book and place it in the left compartment of the caddy pick up the book and place in the right compartment of the caddy pick up the red mug and place to the right of the caddy pick up the white mug and place it to the right of the caddy
For the hidden-task evaluation, for both our approach and BC we train the model without task conditioning, but we do provide scene conditioning vector for each (a one-hot 21 21-dimensional vector indicating which of the 21 scenes the agent is operating in). For each task, we run with a horizon 300 steps, and utilize the environment's built-in success detector. We condition all methods on the timestep, proprioceptive state, and both camera views (using a ResNet- 18 encoder to encode images). For the aask-conditioned evaluation, we use a one-hot encoding for each task. We set (bs) = cos(As AASproprio + b), where sproprio IR denotes the proprioceptive state, and A |6 and 1 are randomly initialized parameters. At deployment, for BE, sample single state from each of the last three episodes and repeat them each for 1/3 of the total history length.
For BE and BC, we swept over the stated hyperparameters, and include the results for the best-performing ones. For SUPE we utilize the default hyperparameters for the vision-based experiments from Wilcoxson et al. (2024). For BC with action noise, at every step we sample noise w ~ (0, 0.15 I) which we add to the action computed by the BC policy. For BC with history conditioning, we utilize a history length of 1 100 (the same as that utilized by BE) and otherwise use the same hyperparameters as standard BC.
Figure 7 and 8 provide results averaged across all 90 tasks, and state the final success rates for these evaluations in Table 6. Please see Figure 20-23 for individualized results.
Table 6: Final success rate from Figure and 8.
BE
BC
BC (Action Noise)
BC (History)
BC (Random Task)
SUPE
Task hidden
0.824 = 0.010
0.590 + 0.013
0.684 +: 0.013
0.643 + 0.013
0.359 +: 0.013
0.267 +:: 0.047
Task provided
0.993 0.002
0.977 = 0.004
0.950 + 0.006
0.804 0.011
20

---

Behavioral Exploration: Learning to Explore via In-Context Adaptation
Table 7: Hyperparameters for Libero BE and BC models.
Hyperparameter
BE
BC
Batch size
150
1.5
Action chunk size
20
Image encoder
ResNet-34
ResNett34
Hidden size
Number of Heads
8
8
Number Layers
4
Feedforward dimension
512
512
Coverage future length
200
Context history length
100
21

---

Behavioral Exploration: Learning to Explore via In-Context Adaptation
Tuxt
Thxt
Tluk:
46
Stacomn
-0
10
8
10
11doon
88
70
el
or Alleppo
Alleppoa
atlammp
Thisk
Thsk
Thyk
Thisk 10
22
LEdeo Success
2
20
120
10
60
Iidan
V3
16
10
(eda: 123
1.05
110
1:30
K:d0an Suronn 1h1
Sucomm
20
60
11 0000a
Number 21
Number Attempia
Kumber kttempta
Mumber
Number
3.30
320
120
01
80
30
Sucoee
10
60
20
Sumber of Attempps Task
oumbero Apsipps
Number kttampo Tack
Mumber Attampts Taak 29
Muncer acsempos
1.00
130
160
Tauk 30
01
80
10
8
40
24
1350mm
36
16
100
Mumber el Alteengs Taxt: 31
Number of Attengrs
Kurber attempes Teok 31
Mumber Atterrets "Tesk
Mumser ancerss
307
Thak 32
300
Taak 35
Succomn
8
86
10
8
8
133tmn 1020n Trobo
20:
20-
20
of Attempga
Numberol Altmpgaa Task 37
Number Adamppa
Attamppa
Acampoa
Task 36
Tack
Task 35
Truk 40
Succumn
20
40
18
200
Teak-1h
116
Mumber
Semberat
T
Mumber
Figure 20: Performance individual Libero tasks (1-45) in evaluation with task hidden.
22

---

Behavioral Exploration: Learning to Explore via In-Context Adaptation
Thxkk 4f.
Thet
Trak 48
Thrk-49
Thusk 50
Stacomns
10
50
00
Hedten 11doon
20
el Taask 51
or Alleppo
Allempoa Thick
allampts Thask
Thik 52
Thisk
22
20
11:te Succos
Success
60
23
Idaan Suczeer
22
16
10
(elda 123
1.05
110
130
Suronn 196)
150
20
60
26
20
Number
Number
Kumber kttamppa
Mumber
Numcer
11000 3e0.000000000
3.55
120
Tuxs Sucoee 201
100
Tral Sucoess
80
10
60
60
11 Teak
50
241
Number of Attempps
oumbero Abtipps Took 72
Number kttampo Tack 73
Mumber Attampts Task 74
Muncer of Amsempos Tauk 75
Task
1.0
110
011
80
60
trobe redinn
24-
30
20
135mm
20
Mumber el alteengs Taxk: 78
oumber of Altenors Thik 77
Kumber attemees
Mumber Atterrets "Tesk: 79
Munser arcerors
Teok 7B
Taok 8D
20
80
40
Hrddon
20
20
of Attampga Task 81
Numbero! Allmppaa Task B2
Number Atamppa Tack
Attamppa
Acamppa
Task
Thuk 85
07
succums
116
madeen Stncess
16
26
Mumber
Stmberr
Mimber
Figure 21: Performance individual Libero tasks (46-90) in evaluation with task hidden.
23

---

Behavioral Exploration: Learning to Explore via In-Context Adaptation
105-
Thst-
T
126
Thek
Tl-k
5a000o 26
Talk Sucomm 260
260
Taak S.cOMe 261
30
46-
10
5
10
50
el Alterpoo
of Allerpio Thik
Allempo
Allampt Task
Thisk
Task
Thisk 10
3.00
Succesn
Tork Suzconttii
03
42
40
10
20
26-
.sumberof
aetempps
Atterpps Task
Taok
100
1.00
120
1.00
300
8
The Swc.oeee 1h11
Tex Swwcooa
8
10
30
20
20
26-
20
100
320
132-
100
12a Sac.ba 260
30-
Tol Sacoee:14
Tul sucoees
Succaa
8
10
10
25
20
20-
20
Number 21
Number
Kumber Artemppa
Mumber Adterpgts
Nuumzer
1.0
1:00
130-
40-
42
8
16
60-
20
20
Sumbbr of Attemps Task
Sumbero Abbippo
Number Attempps Tack 20
.Mubbar Atterppt Task
Stumcar of Acrempcs
T7
Trokk 30
100
132-
8 8
20
Tho: CAcCOme740
60
00
Thx: Caccome
60
60
40
10
Suubbr el Aiterggs
Sumber Alterges Tuck 17
Kumber Alterngs
MMmbbr Aiterret "Tok 34
Text 31
Teok 33
Took 35
Tal 10k550000000
Tork 140
20
Tonk S.coes 140
02
8
20
20-
60
8
40
Tab
60
Mumbbr of Attempaa
Numberol Altrppi Thsk 37
number Attmmpia Thck 38
Murbbrr Adtamppa Task 35
Nlumzer Acempp
Taak 36
Truk 40
300
1.00
320
132-
16
60
40
08
46
46-
40
20
Arterppes
dfterress Taak
Tesk-43
100
120
120.
Teak Sucosso 150
Thal: Sucoess
The Sucomss
8
10
8
20
of Aherges
Mumber dnteergts
Figure 22: Performance on individual Libero tasks (1-45) in evaluation with task provided.
24

---

Behavioral Exploration: Learning to Explore via In-Context Adaptation
Thxkk 4G
47
Tudk-4n
Thutk 50
100
500000 260
Tal Sucomm 260
Thak SuCOME 2502
8 8
60
10
10
el Altempo
Allerpio Thik 52
MMmberr Allampt Thakk
Task 51
Thick 53
Thisk 55
1.0
110
120
100
Teak Succenn
46-
Taa Sutteoa
Tork Suzcennri
42
10
25
20
20
ditempps
"Mumber Attempt Task
Taak 56
100
122
10
The 1h1
Tox
Texr Sww.ooo
8
85
8
10
20
20
20
120
130-
100
260
86
30-
Tul Sucaees
50
60
10
10
20
20
20-
20
Number
Number
Kumber irtemppaa
Mumbbrr Atterppts Task
Suumzer
66
102
01
26
Success
8
10
Seccess
-5-
60
20
.umbbr of Adtempss Ta
Sumbero Absippo Took 72
Number Attemppa Tack 73
.Murbar Atterppts Task 74
Suumcar Amsempcs Took 75
3.05
Tho: CAcOome140
60
8
Tex Doccooo
80
8
10
Tex:
Sumbor el Aitenggs Taxt 7B
of Alterges Thok 77
Kurber Atterres
MMmbbr Aiterrets "Tak 79
Mumsee Alrerros
1.00
Teok: 7B
160
Taok 8D
Tork Success 140
Tonk S.coess 140
8
10
20-
42
8
40
60
Mumbbr of Attempaa Task 91
Numberol Altrppa Thsk B2
Number Attrmpia Thok 83
Murbbrr Adtamppa Task
Nuumzer Acemppo
Trok 85
3.00
320
130-
120
60
42-
00
40
40-
60
20
20
Arterppe
Afterress
100
Texk 05
1100
Taxt:
120
126
150
40
Teak Sucosoo 150
4G
Thal: Sucomss
Thel Sucomms
8
26
of
Mambbrr anteergss
Figure 23: Performance on individual Libero tasks (46-90) in evaluation with task provided.
25

---

Behavioral Exploration: Learning to Explore via In-Context Adaptation
B.3. WidowX Experiments
Figure 24: WidowX Task 1.
Figure 25: Widow Task
Figure 26: Widow Task
100
100
100
of Trials Succeeded
80-
Number Attempts
% of Trials Succeeded
80
Number of Attempts
% of Trials Succeeded
80
Number Attempts
60
60
40
40
40-
2-
20-
20
20
Bc
oz (Gun)
Openvix
oc (Oura)
Bt tows)
OSaMLL
St (ours) ODeNALA
Figure 27: Results WidowX Task
Figure 28: Results WidowX Task
Figure 29: Results WidowX Task
1.
2.
3.
Individual results on each WidowX task are given in Section B.3. We note the BC fails to succeed a single time on 2/3 tasks, and that our approach achieves the highest success rate and lowest average number of attempts on all tasks.
We use delta end-effector control with frequency of 5 Hz. We use an RGB camera to capture the top-down third-person view of the robot workspace, and use 128x128 images input our control policy.
While the Bridge dataset contains text labels for the demonstrations, since behaviors, for both the BC baseline and BE we do not use text conditioning. We use the same form of ((s) as with Libero, but instead of conditioning on the full proprioceptive state we condition only on the relative (x, v,2) coordinates (that is, if we are at state s', then we define <b 8") cos( A(8zyz -82z)) b).
evaluating the ability to select diverse
evaluate on a reaching task, as stated main text. consider a reach to successful if the end-effector makes contact with the object.
As Open VLA requires language conditioning, chose objects in the allow them both to correspond to the same language command. For Task 1, we use the command "pick up the cloth", for Task 2 "pick up the silver pot", and for Task 3 "pick up the object". We note that OpenVLA is not able to condition history of past observations and, as such, at deployment, each episode is effectively independent from past episodes. Our aim in this experiment therefore to test whether OpenVLA tends to focus its behavior on a single possible solution, or if attempts diverse behaviors over multiple episodes.
that relatively similar in order to
We also experiment with modifying the prompt for OpenVLA induce more diverse behaviors. In particular, on Figure 25, experiment with prompting OpenVLA with either "pick up the silver pot" "pick up the other silver pot". We hypothesize that by adding the word "other" to the prompt, this may encourage OpenVLA to attempt to pick up the pot that it is initially biased away from. We provide the results of our evaluation on these prompts in Table 8, where we see that OLA exhibits a strong bias to the right pot and, while changing the prompt does cause OpenVLA to reach for the left more frequently, still exhibits a strong rightward bias, and the prompt modification also causes it to fail to reach any pot more frequently. We hypothesize that this may be due to the modified command being more out-of-distribution for
26

---

Behavioral Exploration: Learning to Explore via In-Context Adaptation
OLe hurting its overall performance. These results illustrate that, while modifying the prompt may help somewhat in encouraging OpenVLA explore, it is far from exhibiting robust exploratory behavior.
Table 8: Performance of OpenVLA with different prompts. We consider running 8 rollouts on the task illustrated in Figure 25, and indicate how many episodes the end effector made contact with the left, right, neither pot.
Command
Left
Right
Neither
"pick up silver pot"
7/8
0/8
"pick up the other silver pot"
2/8
4/8
2/8
Table 9: Hyperparameters for WidowX BE and BC models.
Hyperparameter
BE
BC
Batch size
2560
1200
Action chunk size
4
4
Image encoder
ResNet-34
ResNet-34
Hidden size
512
512
Number of Heads
8
8
Number of Layers
6
6
Feedforward dimension
2048
2048
Coverage future length
20
Context history length
50
B.4. Understanding Behavioral Exploration
5
Figure 30: BE trajectory rollouts (blue) when the BE policy is conditioned on particular history segment (green). Red "x" denotes the starting point for the BE rollouts. In all cases BE adapts its behavior to explore the regions not covered by the conditioning trajectory.
Finally, we seek to provide some insight into the behavior of BE, in particular to what extent BE adapts to the history conditioning, and how critical that adaptation is for effective exploration. We consider here the Maze2D medi um environment from D4RL.
We first consider BE's ability adapt its history conditioning. In Figure 30, we plot the trajectories induced by BE when it is conditioned history of single trajectory, shown in green, and value of exp corresponding to high future coverage. With this conditioning we would expect BE to visit different states than those in the history. We see that this is indeed the casc-in the first two examples, where the conditioning trajectory is on the right side of the maze, the trajectories induced by BE concentrate on the left side, while in the final two examples where the conditioning trajectory is on the top or bottom, the trajectories induced by BE concentrate in the central part of the maze. This illustrates the adaptivity of BE to its history, and BE's ability to take actions leading to the collection of observations that are novel relative its history.
27

---

Behavioral Exploration: Learning to Explore via In-Context Adaptation
(a) BE (online history)
(b) BE (first-state history)
(c) BC
(d) Demonstration data density
Figure 31: Comparison of BE conditioned with history of past online observations, BE conditioned only the first state, and BC. We see that conditioning on the history of past states visited is critical to achieving high coverage, and that this induces coverage over low-density regions in the demonstration data that BC fails to reach.
We next seek to understand how critical this history adaptation is to overall exploration behavior. Figure 31a shows rollouts from BE in its standard operation, where we condition on a history sampled from the past states it has visited in online rollouts. Figure 31b shows rollouts from BE but where, instead of conditioning it on its history of past state, we simply condition on a history that only contains the first state visited the start of rollouts. We see that the coverage is significantly greater in Figure 3 a1a than Figure 31b-while trajectories in Figure 31b primarily visit points in the central part of the maze, in Figure 31a BE achieves coverage throughout the maze. This highlights the role of adaptivity in inducing effective exploration-by adapting to its history, BE is able to achieve much more diverse coverage than if it only considers a fixed history, and much more diverse coverage than BC (Figure 31c). Furthermore, if we compare the coverage achieved by BE vs BC relative to the demonstration data coverage density (Figure 31d), we see that BC primarily visits the high-density regions the training data particclullllllly through the central part of the saze-while BE achieves significantly higher coverage low-density regions in the training data pacticppatlllarlyy the lower left corner, This illustrates BE's ability increase coverage in the low-density regions in the demonstration data.
28


Reviews:
[{"review_id": "nWcruUcgdy", "reviewer": "Reviewer_fhX3", "summary": "The manuscript presents \"Behavioral Exploration\" (BE), a novel formalization for learning from expert demonstrations, while giving the ability to the policy/user to explore in a controlled manner (via some parameters). The main innovation is the Behavior Policy Coverage metric, and how to use it to condition on previous historical data to enhance exploration (or exploitation). The authors provide a practical algorithm and the results showcase that BE outperforms several baselines in RL and imitation learning scenarios.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments. In the following we provide additional clarification on the success metrics and RL experiments, as well as several other comments the reviewer had. We will update the final version of the paper to include all these details. \n\n## Clarification of Success Metrics\nFor the real-world WidowX experiments, we define an “interaction” with an object as the end-effector of the WidowX making contact with the object (as stated in the supplemental: “We consider a reach to be successful if the end-effector makes contact with the object”). We note that in the settings we considered this is straightforward and unambiguous to evaluate. As we are evaluating the ability of each approach to explore the possible tasks in the scene, we run the policy 5 times, and count the overall sequence as a success if across these 5 trials both objects in the scene were interacted with—the end-effector made contact with each object at least once across the 5 episodes. \n\nFor Libero and D4RL Kitchen, we utilize the built-in success detector in the simulator to identify success. For D4RL Antmaze, we utilize the same goal locations as in (Wilcoxson et al., 2024), and count a goal as reached if the agent reaches within distance 0.5 of a square of side length 1.5 centered at the goal point. For each setting, we evaluate on each task separately (e.g. each goal for Antmaze, or task for Libero), and the plots given in the paper are performance averaged across each task (see below for further clarification on aggregation of results). Figures 2, 4, 6, 7 therefore show what fraction of trials have resulted in a success up to a particular time (averaged across tasks and random seeds).\n\nWe will add all these details to the final version of the paper. If the reviewer has any additional questions on the success metrics we are happy to provide further clarification. \n\n## Clarification on RL Problem Setting and Experimental Details\nFor all RL baselines, we run them as described in their original works. In particular, for each setting the algorithm is given an offline dataset of transitions from the environment, but with no reward labels. With the exception of “Online RND”, each method we consider pretrains on this data (Online RND only trains online). After pretraining, each method is deployed online, and is evaluated on the number of steps it takes to reach the specified goal at least once. \n\nThe BE policy is therefore used only for exploration, and is not fine-tuned at all during online deployment. However, as described in Section 4, the history of previous (online) interactions is fed into the context of the BE policy online, so that its behavior can adapt to the past experience—this adaption is only “in-context”, however; no gradient steps are taken on the BE policy. For every other approach considered, with the exception of BC (which is not updated at all online), the policy is updated online with standard gradient-based RL updates (in particular, which seek to maximize RND bonuses, inducing exploration). \n\n## Aggregated Performance Curves\nWe observe similar trends in the individual results as in the aggregated results, but for the final version will include in the upplemental individualized results on Antmaze (for each maze type and goal location) and Libero (for each of the 90 tasks). For the WidowX results, the supplemental already provides individualized results (see Figures 20-22). In all cases where results were aggregated (Antmaze, Libero, and WidowX settings), the aggregated results are the mean of the individual results.\n\n## Fine-Grained Control\nIn addition to Antmaze, which, as noted by the reviewer, does require joint-space control, the Franka Kitchen environment considered in Figure 4 also requires joint-space control. We believe that these results indicate that our method does scale effectively to settings with joint-space control.\n\nMore generally, we would expect our method to scale effectively to any setting on which BC performs well. As our method is essentially training on a BC objective augmented with a particular conditioning structure, we would expect it to, in general, perform at least as well as BC, which we found to be the case in all our experiments. The degree to which it learns effective exploratory behavior is a function of several additional factors, in particular the distribution of behaviors present in the offline data, so there may be cases where it is not able to learn more effective exploratory behavior than BC—we do not necessarily expect the type of control to affect this, however.", "claims_and_evidence": "Overall the authors' claims are matched and supported by the evidence provided. The authors provide proof for the main theoretical result, and many experiments and explanations for the empirical results.", "methods_and_evaluation_criteria": "The D4RL and LIBERO datasets/scenarios and the real world experiments are suited to evaluate the task and provide meaningful comparisons overall. The RL and real-world experiments require some clarifications (see \"Experimental Designs Or Analyses\"), but otherwise the experiments are well-suited for validating the algorithm.", "theoretical_claims": "I did not check the correctness thoroughly, but the claim and proof make sense.", "experimental_designs_or_analyses": "I have a few comments here (tbh I was confused if those comments fit this section or the section on \"Methods And Evaluation Criteria\"):\n\n1) The RL setting is not described at all (the explanation in the supplementary is not enough). I understand that there is a lack of space, but we need to be more specific here. It is implied that the authors do something similar with two other papers, but we never see the exact pipeline. This is important as the authors might not even do what the other papers describe. Reading the paper alone (and not the other papers), it is not clear how the RL scenarios are set up. In other words, the policy learned by BE is used only for exploration? Is there policy adaptation based on newly acquired rewards/data? How is this incorporated with the BE policy? Overall, this part needs more explanation.\n\n2) I am not sure I understand the real-world experiment. The authors say that this is \"a reaching task\", but then they say \"we count a trial as a success if the policy interacts with both objects\". Again the setting here is poorly described.\n\n3) The task success definitions and metrics (e.g. \"we count a trial as a success if the policy interacts with both objects\") are quite vague. Thus, this makes us skeptical about the results. If they are actually not so vague (but due to lack of space, the authors had to come up with a sentence), then they need to be defined more precisely in the text (or even supplementary). Otherwise, the authors should perform new experiments with more rigorous metrics.\n\n4) Aggregated performance curves can lead to misleading results. The authors should at least describe how they aggregated the results from different tasks.", "relation_to_broader_scientific_literature": "The ability to effectively explore vast spaces can beneficial in many fields. Moreover, making it possible to \"ground\" exploration to real-world experience (via demonstrations and the conditioning) can be even more useful and beneficial, since it can shrink the exploration time while not losing interest parts of the state space. Overall, the problem tackled by the paper is very interesting, and the paper provides one solution to this.", "essential_references_not_discussed": "I did not find any.", "other_strengths_and_weaknesses": "Strengths\n-------------\n\n- Well written paper\n- The objective is clearly conveyed\n- The BE policy learning pipeline is nice and imho novel\n\nWeaknesses\n-----------------\n\n- The authors could \"give up\" a few experiments (e.g. D4RL kitchen) and provide more meaningful discussions of the results\n- The aggregated performance curves", "other_comments_or_suggestions": "Very well written paper.", "questions_for_authors": "- How would BE work in tasks where we require more fine control? E.g. when we require torque control? Or low-level joint space control? (The ant maze experiment does not require fine control imho)\n- What would be some more quantitative metrics for evaluating performance? \"we count a trial as a success if the policy interacts with both objects\" (and other similar metrics in the paper) are quite vague and cannot really be trusted", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W1YQ1BR9Zs", "reviewer": "Reviewer_kepy", "summary": "This paper addresses the problem of in-context RL -- in particular, learning how to *explore* through in-context adaptation.\n\nThe proposed method (Behavioral Exploration, BE) performs reweighted behavior-cloning over large expert datasets, with a long-context policy that also conditions on a variable representing how exploratory the expert’s behavior is.\n\nThe pretrained BE policy is validated in downstream RL / IL tasks, and demonstrates improved performance over standard BC and RL approaches.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. We have run an additional requested baseline on Libero—BC with history conditioning, which we found performed worse than BC—and additional experiments illustrating that BE adapts to its history online; please see below for further discussion and the results [here](https://drive.google.com/file/d/19kFpxrczDTiXL1S0rL1FfX2zXwAT1qVZ/view?usp=share_link).\n\n## New Experiment Showcasing BE’s Ability to Adapt Online\nPlease see this Figures 2 and 3 [here](https://drive.google.com/file/d/19kFpxrczDTiXL1S0rL1FfX2zXwAT1qVZ/view?usp=share_link) for illustration of the BE’s ability to adapt to history. To illustrate adaptivity, we run a BE policy trained on Antmaze Large, conditioning on either (a) history sampled from states visited in past online episodes (the strategy used in our experiments), or (b) histories containing only a single fixed trajectory. As our results illustrate, the history can have significant impact on the agent’s performance: when conditioning on a single trajectory, the agent largely takes routes that avoid the space traversed by this trajectory, adapting its behavior to visit novel states. Given this, we see that the agent achieves significantly higher coverage conditioning on the past states it has observed, rather than a single fixed trajectory.\n\n## New Experiment Showing History Conditioning Hurts Standard BC\nWe have a trained a BC policy with history conditioning on Libero—please see Figure 1[here](https://drive.google.com/file/d/19kFpxrczDTiXL1S0rL1FfX2zXwAT1qVZ/view?usp=share_link). As can be seen, conditioning the BC policy on history actually hurts performance. We believe this is an example of “causal confusion” [1], a known phenomenon where conditioning on additional information can decrease performance of BC by introducing spurious correlations. \n\n## Learning on Downstream Tasks\nThe reviewer states that: “A key claim is that BE learns... The experiments... downstream tasks”. We want to highlight that the main focus of the paper is not training on downstream tasks, but rather pretraining a policy that can explore effectively online, focusing its exploration on behaviors exhibited by the demonstration data. While existing work (e.g. [2]) has shown that a richer supply of data does improve policy learning on downstream tasks—and given this we believe that data collected with BE may indeed be useful for learning downstream tasks—this is not the main focus of the paper. We are instead primarily interested in how to pretrain a policy for exploration, and our experiments were chosen to reflect this. We will update the exposition in the paper to make this clear.\n\n## Additional Clarifications\n*History Distribution*: At deployment, we condition the BE policy on the history of states it has already collected online. As any policy $\\pi$ induces some distribution over states visited, the distribution of history encountered at test time depends on the learned BE policy. Ideally, we would like the distribution the BE policy was trained on to match this online distribution. As described in Section 4.3, this is challenging as we do not know what the state visitation distribution of the learned policy will be. Addressing this is a fixed-point problem in that we must first optimize a BE policy on some history distribution, deploy it to find what history distribution it induces, then re-optimize a policy on this new history distribution, and repeat until convergence. To avoid this complication, we suggest choosing the history distribution in training to be a uniform distribution over trajectories in the offline dataset, which we found worked well in practice.\n\n*Number of Demonstrators*: The offline data may be collected by different demonstrators; in this case $\\pi_{\\beta}$ is a mixture policy over all demonstrator policies. \n\n*How can we decouple the performance improvement…?*: \nWhile our approach can learn to explore effectively with any sufficiently diverse set of offline data, it focuses its exploration on the behaviors exhibited in the offline data. Therefore, while diverse low-quality data may be sufficient for training a policy that collects diverse data, it is not necessarily sufficient for successfully achieving a task; the demonstration data must contain examples of successful task completion for BE to learn these behaviors. Thus, both the learning to explore and the type of demonstration data are important for our approach. \n\n*How is the coverage objective…*: Would the reviewer be able to provide additional clarification for what they mean by “feature matching” here? One potential difference is that BE aims to induce *different* behaviors, while typical BC usually aims to induce the *same* behaviors. \n\n[1] De Haan, Pim, et al. \"Causal confusion in imitation learning.\" NeurIPS, 2019.\n\n[2] Yarats, Denis, et al. \"Don't change the algorithm, change the data: Exploratory data for offline reinforcement learning.\" arXiv preprint arXiv:2201.13425 (2022).", "claims_and_evidence": "A key claim is that BE learns to explore over the space of expert behaviors from an expert demonstration dataset, which should speed learning on downstream tasks.\n\n- The experiments, however, do not evaluate BE as a pretraining method for any downstream tasks.\n    \n\nAnother key claim is that BE performs in-context adaptation.\n\n- The provided explanations and experiments do not totally convince me that BE is adaptively exploring in-context.\n    \n    - The reuslts on D4RL show that BE learns a better policy and explores more than baselines, but do not demonstrate any *adaptive* exploration ability -- these results could also occur simply because BE is better at modeling the behaviors shown in the offline dataset.\n        \n    - I would be more convinced if the authors demonstrate that BE is able to learn faster on a completely unseen task than baselines.", "methods_and_evaluation_criteria": "The authors consider naive BC as a baseline, conditioning on state only. How well would BC with the diffusion transformer policy backbone + conditioning on history, but without the reweighting objective, perform?", "theoretical_claims": "I did not check the theory.", "experimental_designs_or_analyses": "yes. See claims.", "relation_to_broader_scientific_literature": "This paper examines how to equip a policy with exploration abilities based on contextual information. This relates to the emerging area of in-context RL and meta-RL.", "essential_references_not_discussed": "None that I am aware of.", "other_strengths_and_weaknesses": "Strengths\n\n- Overall, the paper seems well motivated\n    \n- The proposed approach is based on BC, making it applicable for robotics settings.\n    \n- Strong empirical performance in both simulated and real robot experiments\n    \n\nWeaknesses\n\n- There are some places where a lot of jargon is used, making it difficult to grasp what the authors mean:\n    \n    - In particular, the section on selecting the history distribution (pg 5) is quite confusing.\n        \n    - L257: what is the induced distribution of online states? Induced by what?\n        \n    - Why is estimating the distribution of online states a fixed point problem? How is the fixed point problem speciied?\n        \n- In-context learning is not clearly defined. What criteria should be used to determine if a policy has successfully adapted its behavior online?", "other_comments_or_suggestions": "Typo in Line 880: affective -> effective", "questions_for_authors": "1. The notation in the paper suggests that the demonstration data is generated by a single expert, but the text argues that the demonstration dataset should contain multiple behaviors. Is there any reason why the demo dataset can only contain behavior from one expert?\n    \n2. How is the coverage objective, which attempts to cover behaviors shown in $pi_\\beta$ diffrent from imitation learning approaches such as feature matching?\n    \n3. How can we decouple the performance improvement due to learning to explore, versus the performance improvement due to access to demonstration data?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0QzwSqEfKn", "reviewer": "Reviewer_Qbgd", "summary": "This paper proposes a method—Behavioral Exploration (BE)—that is capable of online in-context task adaptation while learning offline from a set of reward-free expert demonstrations. Such an ability is achieved by conditioning the policy on the coverage measure, which reweights the probabilities of less frequent trajectories while remaining under the distribution of expert policies. This approach is compared with existing exploration algorithms, zero-shot RL methods, and a modern VLA in maze-like and robotic environments, and also on a real-world robotic task. The claimed contributions of the paper are the method’s ability to generalize to new goals faster than baseline approaches and the scalability to both simulated and real-world robotic tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. We have added an additional unsupervised RL baseline to the [Libero experiment](https://drive.google.com/file/d/19kFpxrczDTiXL1S0rL1FfX2zXwAT1qVZ/view?usp=share_link), which we found performed worse than both BE and BC, and have also run additional experiments with OpenVLA, testing out the reviewer’s suggestion for enabling OpenVLA to explore. See below for discussion on these points and additional clarifications.\n\n## [Concern 2] New RL Baseline on Libero\nWe have added SUPE, the most effective unsupervised RL baseline, on Libero. Please see Figure 1 [here](https://drive.google.com/file/d/19kFpxrczDTiXL1S0rL1FfX2zXwAT1qVZ/view?usp=share_link). As illustrated, SUPE performs significantly worse than both BC and our approach. We note that the number of episodes we consider for Libero (15) is significantly less than what is typically required by RL algorithms to effectively learn, so we do not find this result surprising—it further illustrates the necessity for fast in-context adaptation as compared to standard RL gradient-based updates.\n\n\n## [Concern 4] New OpenVLA Experiments \nWe make several comments on our inclusion of OpenVLA as a baseline. First, our goal in including it is to demonstrate that existing state-of-the-art methods for robotic control do not effectively explore over the space of possible tasks when the goal specification is ambiguous. In other words, OpenVLA does not already solve the problem we are attempting to solve—if the specified goal is ambiguous, OpenVLA does not attempt different behaviors until it finds the correct one, it simply chooses the behavior it deems most likely.\n\nSecond, as the reviewer has highlighted, OpenVLA receives no feedback on whether it has succeeded. This is fundamental to OpenVLA, however, as well as most other current state-of-the-art BC methods for robotic control: OpenVLA is not trained to condition on a history of past interactions or observations, and therefore fundamentally cannot adapt online. Our approach aims to fill precisely this gap, and we hope it will be incorporated in future policy training to enable effective adaptation to history. We note as well that, as discussed in our reply to Reviewer kepy and illustrated by our new BC with history conditioning baseline on Libero, naively conditioning on history can actually hurt performance—our approach may be a way to instead improve performance when conditioning on history.\n\nGiven this, instructing OpenVLA to pick up “another” object is an ambiguous command—since OpenVLA has no history dependence, it has no grounding for what “another” object is. Nonetheless, we ran an additional trial with OpenVLA on the command “pick up the other silver pot”. We evaluate on the task illustrated in Figure 18, and give the results below, indicating which fraction of episodes the policy goes to the pot on the left, the right, or fails to grasp at either, and compare to our original command.\n\n| Command | Left | Right | Neither |\n|-|-|-|-|\n| “pick up the silver pot” | 1/8 | 7/8 | 0/8 |\n| “pick up the other silver pot” | 2/8 | 4/8 | 2/8 |\n\nAs this illustrates, while this command does cause OpenVLA to move to the left pot somewhat more frequently, it also hurts its ability to move to either pot at all (likely because this command is somewhat outside the training distribution for OpenVLA). \n\n## [Concern 1] Ease of Training and Evidence for Faster Adaptation\nTo clarify our claim that the method is easier to train, this is indeed somewhat tangential to the claim of faster exploration and adaptation online, the primary claim of our paper and the main focus of our experiments. However, our approach does lead to lower computational overhead and simplicity at deployment compared to the baseline RL methods, all of which require online gradient updates and hyperparameter tuning. While this is an advantage of our approach, we see this as complementary to our main objective.\n\nWe would also like to emphasize that BE is able to explore significantly faster on Antmaze and Libero than all RL baselines, supporting our main claim of faster adaptation and exploration. For example, to reach the final performance of SUPE, the best RL baseline, it requires only 15% of the samples. While the performance between BE and the baseline RL methods is similar on Franka Kitchen, we believe the strong performance on Antmaze and Libero highlights that in many settings, our approach leads to a substantial gain.\n\n## [Concern 3] Scene Conditioning\nAs noted in the paper, Libero 90 consists of 90 tasks distributed across 21 scenes. The scene conditioning vector is then simply a one-hot vector of dimension 21 indicating which scene the agent is operating in. While this information can be inferred entirely from visual information (the scenes appear visually distinct) we found that adding this conditioning improved the performance of both our approach and BC slightly. This is not critical to performance, however.", "claims_and_evidence": "There are three main claims in the paper:\n\n- [C1] The proposed method is able to learn from possibly not optimal RL data to quickly adapt to the task\n\nThis claim is supported by experiments on two D4RL benchmarks: AntMaze and Franka Kitchen. The proposed method is seems to be more sample efficient while exploring the AntMaze environment, but acts on par with zero-shot HILP and falls short of SUPE. To my mind, the evidence authors provide is not enough to make conclusions of faster adaptation due to similar performance of the BE with the existing ones. \n\n[Concern 1] At the same time, authors argue that their method is easier to train, compared to unsupervised zero-shot methods. Although I have an intuition this might be true, I find this explanation unrelated to the original claim of fast adaptation. Thus, I find the evidence not convincing enough to support the claim of faster adaptation.\n\n- [C2] The method can learn to explore from diverse robotic demonstrations\n\nTo show the method is capable of in-context exploration, authors choose the Libero 90 dataset, which consists of human demonstrations of different tasks in the Libero robotic simulator. Authors employ two approaches; first, they hide the final goal and measure the number of attempts of their method and the baselines to achieve the goal. In the second, the task is provided. For a baseline, the behavior cloning (BC) algorithm is used. In both scenarios, the BE approach finds the goal in fewer attempts compared to BC.\n\n[Concern 2] Although the experimental results are positive, the selection of baselines appears insufficient, as only BC is included. Based on the previous experiment, it would be reasonable to include unsupervised zero-shot RL methods, provided they do not require rewards during training. Without additional experiments, the claim is only weakly supported.\n\n[Concern 3] Besides that, I have a question to the authors. In Appendix B.2 they mention that when the goal is hidden, BC and BE are still provided with “a scene conditioning vector”. What does this vector consist of?\n\n- [C3] The method scales to the real-world environments\n\nTo support this clam, authors train their method on the BridgeData V2 dataset and set up a robotic arm WidowX 250 to solve three tasks, where it needs to reach the right object on a table. The trial is counted as a success if the right object was picked at least once in five attempts. Two methods are chosen for the comparison; OpenVLA, the recent Vision-Language-Action model that receives an image and language instruction to produce an action. The second baseline is behavior cloning. It is shown that BE has more successful trials compared to baselines, which indicates it is able to demonstrate exploration behavior.\n\n[Concern 4] While the comparison with OpenVLA provides some insights, its relevance may be questioned. According to Appendix B.3, the language command for VLA was fixed for all trials, whereas BE, being an explorative algorithm, adapts by conditioning on a coverage value to generate new trajectories. In contrast, VLA is designed to follow the language commands commands without feedback from the environment. Thus, after touching one of the object once, VLA does not receive any meaningful signal whether the episode was done successfully or not, which eliminates the incentive to try another object. For instance, it would be interesting to see how the results change if, after an initial unsuccessful trial, the model were prompted to select  “*another”* object. Given these differences, the motivation for the current comparison could be further clarified.\n\nOverall, the claim appears plausible; however, it could benefit from a more comprehensive comparison and further investigation.", "methods_and_evaluation_criteria": "I have raised some concerns regarding the baselines in [C2; C3]. Apart from them, the evaluation metrics and data seem reasonable.", "theoretical_claims": "I read the informal proposition in the main text. I did not check correctness of the proof in the Appendix A.", "experimental_designs_or_analyses": "I have raised my concerns in [C3] regarding the experiment design.", "relation_to_broader_scientific_literature": "The paper introduces a novel method that learns to explore from offline data and can adapt to new tasks by using the exploration strategy. To my mind, this is the first method that do not use the rewards or learn its approximation during inference, working entirely with states and actions. However, novelty is not the only criterion for a paper to be of high quality; this paper could be improved by clarifying its claims, adding missing baselines, and justifying some of its evaluation choices.", "essential_references_not_discussed": "None that I am aware of.", "other_strengths_and_weaknesses": "Strengths:\n\n1. The novel method which proposes in-context exploration entirely without rewards\n2. Well-written with accessible explanations\n3. Evaluation in the real-world setting\n\nWeaknesses:\n\n1. No discussion or conclusion sections are found.\n\n Other weaknesses are listed in the section “Claims And Evidence”.", "other_comments_or_suggestions": "None", "questions_for_authors": "I have numbered my concerns as [Concern N] in the 'Claims And Evidence' section. Discussing these concerns could potentially change my opinion and affect the score.\n\n[Edit] I have updated my score to 4.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YdvOvoE3Hs", "reviewer": "Reviewer_RYNM", "summary": "This paper presents a method to predict expert actions based on past observations and predict how \"exploratory\" the expert's behaviors are relative to the context. This design enables the model to mimic the behavior of an expert, and select the most relative experiences to explore.\n\n---\n\nUpdate after rebuttal: Many important results, such as tabular results, have not been provided. The supplemented videos lack descriptions and comparisons. This paper still needs more polish. The reviewer's score is unchanged.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. We have added videos illustrating the performance of our approach and baselines on our real-world WidowX setting [here](https://drive.google.com/drive/folders/1-B7kgD9lXVR41WyhtR1XuLepQXI8QR7W?usp=share_link), and provide clarification on additional points below. \n\n### The claim in the introduction is that existing works are \"slow and expensive”…\nIt is well-known that randomized exploration, on which the majority of practical RL approaches are based, is “expensive” in the sense that it requires many interactions before it discovers the correct behavior (see e.g. [1]). Furthermore, most RL algorithms update their behavior “slowly” by gradually fitting a value function to newly collected data, and updating their policy to maximize this value function. While this approach can fit the data after sufficiently many gradient steps, to avoid training instability one must typically limit the number of gradient steps per sample, causing the policy to lag behind the behavior that could in principle be learned from the data (see e.g. [2]). We believe these statements are backed up by our experiments (Figures 2 and 3, in particular), which illustrate that our approach of in-context adaptation leads to much faster exploration and task-solving ability than RL approaches, and furthermore these observations are common throughout the RL literature. \n\n### The introduction example claims that the robots should not explore…\nWhile we concur that in certain settings—in particular, when very little prior knowledge is available—taking “wrong” actions is indeed useful, in settings where one already knows an action is “wrong” taking this action is of limited usefulness. For example, if the agent knows what a “cup” is, then it already knows that picking up a plate is incorrect, so attempting to pick up the plate does not help it learn anything new about which cup it should pick up. Furthermore, as there are many possible \"wrong\" behaviors, exploring them all is very sample inefficient.\n\nThe focus of this paper is on scenarios where the prior knowledge (in our case, in the expert demonstrations) is rich enough to avoid many “wrong” behaviors, and where we want to quickly explore over potentially correct behaviors. We believe this is useful in many real-world settings. For example, ideally we would want a robot policy that follows instructions (when we tell it to “pick up the pot” it picks up a pot, and not some other object) but that can try different behaviors until it picks up the correct pot. Our experiments illustrate that OpenVLA, a state-of-the-art generalist robot policy, can do the former yet not the latter, and existing work on exploration can achieve the latter (after many samples attempting every possible behavior) but not the former—our work fills a gap in the literature by enabling a policy to achieve both simultaneously.\n\n### D4RL is a relatively new benchmark…\nWe would like to emphasize that D4RL is nearly 5 years old and has become the standard benchmark for nearly all work in offline RL since its release (see e.g. [3]-[5]). Furthermore, other RL benchmarks (such as Atari) do not have standard offline datasets—which is critical in our setting—and are therefore not amenable to this work. \n\n### Variances are not shown…\nFigures 2-4 do include error bars—see, for example, the BC curve of Figure 2. As stated in the main text, these error bars denote 1 standard error. \n\n### Tabular results…\nWe will include tabular results in final version of the paper, as requested.\n\n### Consistent results in Figures 2-4\nFigures 2 and 3 consider different metrics—Figure 2 plots the number of goals reached, while Figure 3 the total number of regions reached—and Figure 4 a different environment entirely, and so we would not necessarily expect the ordering to be the same. \n\n### Measuring coverage…\nAs noted in footnote 1 in our paper, measuring coverage through the inverse feature matrix is a canonical technique in statistics and, furthermore, is common in the RL theory literature (e.g. [6]-[8]).\n\n[1] Dann, Chris, et al. \"Guarantees for epsilon-greedy reinforcement learning with function approximation.\" ICML, 2022.\n\n[2] Li, Qiyang, et al. \"Efficient deep reinforcement learning requires regulating overfitting.\" ICLR, 2023.\n\n[3] Chen, Lili, et al. \"Decision transformer: Reinforcement learning via sequence modeling.\" NeurIPS, 2021.\n\n[4] Kostrikov, Ilya, et al. \"Offline reinforcement learning with implicit q-learning.\" arXiv, 2021.\n\n[5] Kumar, Aviral, et al. \"Conservative q-learning for offline reinforcement learning.\" NeurIPS, 2020.\n\n[6] Jin, Chi, et al. \"Provably efficient reinforcement learning with linear function approximation.\" COLT, 2020.\n\n[7] Wagenmaker, Andrew J., et al. \"Reward-free rl is no harder than reward-aware rl in linear markov decision processes.\" ICML, 2022.\n\n[8] Zhou, Dongruo, et al. \"Nearly minimax optimal reinforcement learning for linear mixture markov decision processes.\" COLT, 2021.", "claims_and_evidence": "1. The claim in the introduction is that existing works are \"slow and expensive.\" This is not supported by any evidence or citations.\n2. The introduction example claims that the robots should not explore \"pick up a plate or pick up the same cup.\" The reviewer doesn't agree with this because exploring (although not successful attempts) can be quite beneficial. Exploring wrong experiences can be quite beneficial in avoiding erroneous actions. For example, exploring the action of picking up a plate can help avoid making such a mistake.", "methods_and_evaluation_criteria": "1. Measuring coverage through the inverse feature matrix needs better motivation. Compared to similar lines of work in this domain (the coverage scheme) is highly recommended.\n\n2. D4RL is a relatively new benchmark while evaluating other existing standard benchmarks (e.g., Atari or Mujoco) are recommended.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "1. Variances are not shown in Figure 2-Figure 4.\n2. Tabular results should be provided as well, at least in the appendix.\n3. Supplementary videos are recommended.\n4. It seems that Figure 2 - Figure 4 do not present consistent results. For example, HILP doesn't always outperform SUPE.", "relation_to_broader_scientific_literature": "The reviewer cannot agree with the motivations behind this work (see the above comments). Therefore, the significance of this work is questionable.", "essential_references_not_discussed": "The literature review is reasonably extensive.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "See above sections.", "questions_for_authors": "Please answer the questions mentioned above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces "Behavioral Exploration", where an offline demonstration dataset is used to train a policy that can condition on how "exploratory" it is. The connection to in-context learning is that the policy conditions on a long history. Results on standard benchmarks as well as an actual physical robot provide empirical support for the method.

Although not all the reviewers support publication at this time, upon closer inspection, it's not clear that the specific criticisms should doom the paper: "a lot of jargon" [kepy] and one who cannot "agree with the motivations" [RYNM]. The other positive reviewers (and myself) find value in the motivation of the work, and it is normal for researchers to sometimes disagree about the significance over particular problem statement/motivation.

Therefore, I recommend accepting the paper.
Decision: Accept (poster)