The Challenges of Exploration for Offline Reinforcement Learning
Authors: Nathan Lambert
Markus Wulfmeier
William F Whitney
Arunkumar Byravan
Michael Bloesch
Vibhavari Dasagi
Tim Hertweck
Martin Riedmiller
Venue: Submitted to ICLR 2023 (2023)
Keywords: Offline Reinforcement Learning
Exploration
Robotics

Abstract: Offline Reinforcement Learning (ORL) enables us to separately study the two interlinked processes of reinforcement learning: collecting informative experience and inferring optimal behaviour. The second step has been widely studied in the offline setting, but just as critical to data-efficient RL is the collection of informative data. The task-agnostic setting for data collection, where the task is not known a priori, is of particular interest due to the possibility of collecting a single dataset and using it to solve several downstream tasks as they arise. We investigate this setting via curiosity-based intrinsic motivation, a family of exploration methods which encourage the agent to explore those states or transitions it has not yet learned to model. With Explore2Offline, we propose to evaluate the quality of collected data by transferring the collected data and inferring policies with reward relabelling and standard offline RL algorithms. We evaluate a wide variety of data collection strategies, including a new exploration agent, Intrinsic Model Predictive Control (IMPC), using this scheme and demonstrate their performance on various tasks. We use this decoupled framework to strengthen intuitions about exploration and the data prerequisites for effective offline RL.

Under review as a conference paper at ICLR 2023
THE CHALLENGES OF EXPLORATION FOR OFFLINE REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Offline Reinforcement Learning (ORL) enables us to separately study the two in- terlinked processes of reinforcement learning: collecting informative experience and inferring optimal behaviour. The second step has been widely studied in the offline setting. but just as critical to data-efficient RL is the collection of infor- mative data. The task-agnostic setting for data collection, where the task not known priori, is of particular interest due to the possibility of collecting single dataset and using it to solve several downstream tasks as they arise. We investi- gate this setting via vuriosity-basee' intrinsic motivation, family of exploration methods which encourage the agent to explore those states or transitions it has not yet learned to model. With Explore2Offline, we propose to evaluate the quality of collected data by transferring the collected data and inferring policies with re- ward relabelling and standard offline RL algorithms. We evaluate a wide variety of data collection strategies, including a new exploration agent, Intrinsic Model Predictive Control (IMPC), using this scheme and demonstrate their performance on various tasks. We use this decoupled framework to strengthen intuitions about exploration and the data prerequisites for effective offline RL.
INTRODUCTION
The field of offline reinforcement learning (ORL) is growing quickly, motivated by its promise to use mreviousyy-oollecte datasets to produce new high-quality policies. It enables the disentangling of collection and inference processes underlying effective RL (Riedmiller al., [021]. To date, the
majority of research in the offline RL setting has focused on the inference side the extraction of performant policy given a dataset, but just as crucial is the development of the dataset itself. While challenges of the inference step are increasingly well investigated (Levine et al. 2020 AAgarwal al., 2020), we instead investigate the collection step. For evaluation, we investigate correlations between the properties of collected data and final performance, how much data is necessary, and the impact of different collection strategies. Whereas most existing benchmarks for ORL (Fu et al.) 2020 Gulcehre et al. 2020) focus the single-tat tanklettskkae setting with the task known a priori, evaluate the potential of task agnostic exploration methods to collect datasets for previously-
tasks. Task-agnostic data is exciting avenue to pursue to illuminate potential tasks of interest in space via unsupervised learning. In this setting, we transfer information from the unsupervised pretraining phase not via the policy (Yarats et al. (021) but via the collected data.
Historically the question of how to act and therefore collect data in RL has been studied through the exploration-explooitaioo trade-off, which amounts a balance of an agent's goals in solving task immediately versus collecting data to perform better in the future. Task-agnostic exploration ex- pands this well-studied direction towards how explore in the absence of knowledge about current or future agent goals (Dasagi et al.) 2019). In this work, we particularly focus on intrinsic moti- vation (Oudeyer& & Kaplan| 2009), which explores novel states based on rewards derived from the agent's 's internal information. These intrinsic rewards can take many forms, such as serriosity-basee methods that learning a world model (Burda et al. 2018bb Pathak et al. 2017 Shyam et al. 2019) data-based methods that optimize statistical properties of the agent's 's experience (Yarats al. 2021
or competence- based metrics that extract skills (Eysenbach et al. 2018). In particular, we perform wide study of data collected via curiosity- based exploration methods, similar to EXORL Yarats jet al.] 2022). In addition, we introduce a novel method for effectively combining curiosity-based rewards with model predictive control.

---

Under review as a conference paper at ICLR 2023
Intrinsic Exploration Agent Task- agnostic Intrinsic. Rewards
Offine Reinforcement Learning via Critic Regularized Regression (CRR)
Agent's Experience
back-ignostic transitions
St
+ aa
Reward Model
at
int. t
Extrinsic Reward Function
Figure 1: The Explore20ftinee framework for evaluating data-efficient intrinsic agents. First the agent acts in the environment task-agnostically to search for novel states. After set lifetime, the agent experience stored in a replay buffer is labeled with the rewards of task of interest. This replay buffer is used train an RL policy with the offline reinforcement learning algorithm Critic Regularized Regression in order to finally evaluate the quality of exploration in the environment.
In Explore20ftline, we use offline RL as a mechanism for evaluating exploration performance of these curiosity- models, which separates the fundamental feedback loop key RL in order to disentangle questions of collection and inference Riedmiller et al. (2021) as displayed in Fig. With this methodology, our paper has series of contributions for understanding properties and applications of data collected by curiosity-based agents.
We propose Explore20filine to combine offline RL and reward relabelling for transferring information gained in the data from task-agnostic exploration to downstream tasks. Our results showease how experiences from intrinsic exploration can solve many tasks, partially reaching similar performance to state-of state-of-the-ae of-the-art online RL data collection.
Contribution 1:
Contribution We propose Intrinsic Model Predictive Control (IMPC) which combines learned dynamics model and curiosity approach to enable online planning for exploration to minimize potential of stale intrinsic rewards. large sweep over existing and new methods shows where task-agnostic exploration succeeds and where it fails.
Contribution 3: By investigating multi-task downstream learning, we highlight a further strength of task-agnostic data collection where each datapoint be assigned multiple rewards in hindsight.
RELATED WORKS
2.1 CURIOSITY-DRIVER EXPLORATION
Intrinsic exploration is a well studied direction in reinforcement learning with the goal of enabling agents to generate compelling behavior any environment by having an internal reward represen- tation. Curiosity-driven learning uses learned models to reward agents that reach states with high modelling error or uncertainty. Many recent works use the prediction error of learned neural net- work model to reward agents' that see new states Burda et al. (2018b); Pathak et al. (017)). Often, the intrinsic curiosity agents are trained with on-policy RL algorithms such as Proximal Policy Op- timization (PPO) to maintain recent reward labels for visited states. Burda et al.) (2018a) did wide study on different intrinsic reward models, focusing on pixel-based learning. Instead, we use off- policy learning and ra-label the intrinsic rewards associated with tuple when learning the policy. Other strategies for using learned dynamics models to explore is to reward agents based on the vari- ance of the predictions Pathak et al., [Sekar et al. the value function (Lowrey et al.) 2018). We build on recent advancements in intrinsic curiosity with the Intrinsic Model Predictive Control agent that has new properties: online planning of states explore and using separate reward model from the dynamics model used for control.
2.2 UNSUPERVISED PRETRAINING IN RL
Recent works have proposed a two-phase RL setting consisting of a long "pretraining" phase in a version of the environment without rewards, and cample-limites "task learning" phase with visible rewards (Schwarzer et al. 202T). . In this setting the agent attempts to learn task-agnostic information about the environment in the first phase, then rapidly re-explore find rewards and produce a policy
2

---

Under review as a conference paper at ICLR 2023
specialized to the task. Various methods have addressed this setting with diverse policy ensembles, such as a policy conditioned on a random variable whose marginal state distribution exhibits high coverage (Eysenbach et al., 2018) finding a set of policies with diverse successor features (Hansen et al. 2020). Similarly| Liu & Abbeel (202T) learn a single policy which approximately maximizes the estimated entropy of its state distribution in contrastive representation space. Another strategy collects diverse data during the pretraining phase and uses it to learn representations and exploration rewards that are beneficial for downstream tasks (Yarats al. 2021). central focus of all of these methods is delivering agents which can explore efficiently at task learning time. While the pretrain- ing and data collection phases of unsupervised RL pretraining and Explore2Offine (respectively) are similar, in Explore2Offline the task learning phase is performed on relabeled offline transitions, enabling us to use information acquired throughout training and not only the final policy.
2.3 EVALUATING TASK-AGNOSTED EXPLORATION
While there are many proposed methods for exploration, evaluation of exploration methods is varied. Recent work in exploration have proposed variety of evaluation metrics, including fine-tuning of agents post-exploration (Laskin et al. 2021), sample-efficiency and peak performance of online RL (Whitney al. 2021), zero-shot transter of learned dynamics models (Sekar et al.), multi- environment transfer (Parisi et al., 202T), and skill extraction to a separate curriculum (Groth et al., 2021]. Task-agnostic exploration has been investigated via random data (Cabi al.) 2019) and intrinsic motivation as a source of data for offline RL (Dasagi et al. [019 Endrawis et al.) [021), but has only been evaluated in the single-task setting and limited by current ORL, implementations. Offline RL is compelling candidate for evaluating exploration data because of its emerging ability to generalize across experiences in addition to imitating useful behaviors.
Complementary work echoes the importance of data collection for offline RL from the perspective of unsupervised RL (Yarats et al. 2022), while our work focuses more on the relationship between the exploration challenges of an environment and how new exploration algorithm could address current data generation shortcomings.
2.4 OFFLINE REINFORCEMENT LEARNING
With Offline Reinforcement Learning, we decouple the learning mechanism from exploration by training agents from fixed datasets. Various recent methods have demonstrated strong performance in the offline setting (Wang et al. [2000] Kumar et al. 10020/Penge et al.) 2019 Fujimoto & Gu 2021)
In Explore2Offline we use a variant Critic Regularised Regression (Wagg et al., (020).
Many datasets and benchmarks such as D4RL (Fu et al. 2020) and RL Unplugged (Gulcehre et al., 2020 have been proposed to investigate different approaches. The use of offline datasets has even been extended to improve online RL performance (Nair et al., (020). Our goal is related; instead of investigating multiple offline RL approaches, we investigate mechanisms to generate datasets for downstream tasks. Analysis over the desired state-action and reward distributions for ORL are studied, but little work is done to address how best to generate this data (Schweighofer et al.) 2021
On the theory side, recent works have investigated the Explore2Offline setting, which they call "reward- exploration" " (Jin et al., 22000] Kaufmann et al., 2021). These works study algorithms which guarantee the discovery of e-optimal policies after polynomially many episodes of task- agnostic data collection, though the algorithms they study are not straightforwardly applicable to the high-dimensional deep RL setting with function approximation.
2.5 REWARD RELARELLING
By using off-policy or even offline learning, data generated for one task and reward can be applied
to learn a variety of potential tasks. In off-policy RL, we can identify useful rewards for an existing trajectory based on later states from the same trajectories (Andrychowicz et al. (017)), uncertainty over trajectory (Nasiriany et al. 2021), distribution of goals (Nasiriany al. [202 related tasks
(Riedmiller et al. 2021 Wulfmeier et al.] 2019). agent-intrinsic tasks (Wulfmeier et al. (02T). via inverse reinforcement learning (Eysenbach et al. 2020) as well as other mechanisms (Li et al. 2020
3

---

Under review as a conference paper at ICLR 2023
In the context of pure offline RL, go step further as not required to find the optimal tasks for stored trajectory data. In this setting, data can be used for learning with a massive set of rewards such all states visited along stored trajectories (Chebotar al.) (22T)). We will evaluate our approaches for exploration across downstream tasks and relabel data with all possible tasks.
METHODOL.OGY
3.1 REINFORCEMENT LEARNING
Reinforcement Learning (RL) is framework where agent interacts with environment solve a task by trial and error. The objective of an agent is often to maximize the cumulative future reward on a predetermined task, E[ Σ 7 1 = s,]. We utilize the setting where an agent's interactions modeled as Markov Decision Process (MDP). A MDP is defined by
with environment state of the environment s, an action a that is taken by an agent according to a policy To(ss) transition function p(*++1|* Sc, ae) governing the next state distribution, and discount factor y [0, I] weighting future rewards. With a transition in dynamics, the agent receives reward r, from the environment and stores the SARS data in a dataset D (5) ag. l'k $k-1). Alternatively to this environment-centrr reward formulation is the concept of intrinsic rewards, where the agent maximizes an internal notion of reward in an task agn-atgootiic manner collect data.
3.2 CURIOSITY-DRIVEN EXPL.RATIOON
Existing Methods Reaching new, valuable areas of the state-space is crucial to solving sparse tasks with RL. One method to balance attaining new experiences, exploration, with the goal of solving a task, exploitation, is using curiosity models. Curiosity models are a subset of intrinsic rewards agent use explore by creating reward signal, int. These models encourage exploration by optimizing the signal from a learned model that corresponds to a modeling error or uncertainty, which often occurs states that have not been visited frequently. We deploy series of intrinsic models: the simplest, Next Step Model Error maximizes the error of learned one-step model .int [| s1+1ll", Random Network Distillation (RND) maximizes the distance of learned state encoding to that of static encoding n [l !12 (Burda al. 2018b), the Intrinsic Curiosity Module (ICM) maximizes the error on a forward dynamics model learned in the latent space, d(s), of a inverse dynamics model n 2| ツ0|² PPthhk et al. 2017)) and Dynamics Disagreement (DD) maximizes the variance an ensemble of learned one-step dynamics Pathak et al.).
models r = (8
Intrinsic Model Predictive Control Model Predictive Control (MPC) on learned model has been used for con- trol across a variety of simulated an real world settings (Wieber; [Camacho & Alba 2013). including recently with model- based reinforcement learning (MBRL) algorithms (Williams et al.] 2017] CCua let al., 2018 Lambert et al.] 2019). MBRL using MPC is an iterative loop of learning a predictive model of environment dynamics fe(+) (e.g. a one-step transition model), and acting in the environment through the use of model based planning with the learned model. This planning step usually involves optimizing for sequence of actions that maximizes the expected future reward (Eqn. [T] for example, via sample based optimization; the MPC loop executes the first action of this sequence followed by replanning.
Uninteresting
Memory Novel State
Cumust Novel State
Execoned Imagined Beluvio
2.
Trajectones
Online Intrinsic Planning Agent
A conceptual depiction of an exploration with foresight via planning. By simulating the fu- ture with a learned dynamics model, ideally an agent should be able to avoid damaging states before hav- ing experienced them to collect interesting data in sample-efficient manner.
Figure 2:
a. arg max Σ (((r. at). s.t. 8τ41 fo(s,,
(1)
4

---

Under review as a conference paper at ICLR 2023
Curiosity Model
Training Labels {8)
Eval. Labels {s}
MPC Compatible
Random Network Distillation (RND)
Yes
Intrinsic Curiosity Module (ICM)
as,
{s, (s,a,s) {s, a, {)
No
Next Step Model Error (NSM)
a, s)
No
Dynamies Dissimilarity (DD)
(s,
Yes
Table I: The intrinsic models used with the components of a tuple needed for training and evaluation. To function with planning, an intrinsic model cannot need access the next state to compute reward.
The reward function defines the behavior of the planned sequence of actions in model-basee plan- ning. For task-specific RL this can be the task reward function (known or estimated from data). Instead, our Intrinsic MPC (IMPC) agent uses curiosity based reward for planning in order to en- courage task agnostic exploration by reasoning about what states are currently interesting and novel. The goal of planning being used to visit new interesting states, rather than states that were recorded with high intrinsic reward in the replay memory is shown in Fig. 2 This evaluation occurs by sam- pling action sequences, unrolling them using the forward dynamics model, scoring the rollouts with the learned curiosity model, and finally taking the first action of the sequence with the highest score. Given that this evaluation happens with access to only imagined states and a proposed action, only subset of intrinsic models can be used with planning, summarized in Tab. D We primarily evaluate IMPC using the RND curiosity model, but we also present results with the DD model.
We use the Cross Entropy Method (CEM) (De Boer et a..), sample based optimization procedure for planning. Inspired by prior work Byravan et al. (2021) y we use a policy to generate action can- didates for the planner; this policy is trained using the Maximum a-posteriori Policy Optimization (MPO) algorithm (Abdolmaleki et al. 2018) from data generated by the MPC actor. Additionally, to amortize the cost of planning interleave planning with directly executing actions sampled from the learned policy. This is achieved by specifying a planning probability 0 < 1; at each step in the actor loop we choose either plan or execute the policy action according p (we use p 0.9). Additional algorithmic details are included in Appendix A.1
3.3 OFFLINE REINFORCEMENT LEARNING
To train an agent offline from task-agnostic exploration data, we determine rewards from observa- tions in hindsight. While the approach relies on the ability to compute rewards based on observa- tions, a large set of tasks can be described in this manner (Li et al., 2020). In comparison to online learning, we have the benefit that we do not need to determine for which task data is most infor- mative given a commensurate number of tasks. Instead we can relabel the data with all possible rewards to maximise its utility. Given the new task rewards, we replace the intrinsic reward in our trajectory data and apply a variant of a recent state-of-the-ar offline RL algorithm, Critic Regu- larised Regression (CRR) Wang al. (2020). While we apply CRR for our investigation, the overall method is general in that it could be applied with other approaches. We iteratively update critic and actor optimising their respective losses following Equation and
Le Es D(Qo(s), az), (r+ -B Q ((()))))))))
Since we use a distributional categorical critic, we apply the divergence measure instead of the squared Euclidean loss, following (Bellemare et al.). With f ReLU(((ss, ax)) and Á the advan- tage estimator via Qa (St, az) - -1/m EN,, Qu(s), the policy is optimized as:
T(a(l1)) arg max E a- S(Q 8, a.) log (als))
(3)
EXPERIMENTS
4.1 EXPERIMENTAL. SETTING
Exploration Agents In this work we benchmark a variety of task-agnostic exploration agents.
We classify agents reactive, selecting actions with a policy, planning, selecting actions by
5

---

Under review as a conference paper at ICLR 2023
Explort Stze
*
Clotroll State
1032
JAN0
NPQ
Loec
Obwerved Expiraatin Revard
Reward
692
402
4Bc
202
Evebutton
.02
=00
002
sec-
51
14
Stoo
Figure 4: Comparing the collected rewards of an exploration agent (left) to the reward achieved when training an offline RL algorithm on the same data (right). There is a substantial gain in performance with offline RI. algorithms simply by having more data, irregardless as to if the data has higher density of observed rewards. The median, 90th and 10th percentiles are shown for each agent, combined across tasks. Note: the MPO agent learns online and is the only task-aware method.
optimizing over trajectories. The reactive agents are trained with Maximum Apriori Optimization (MPO) (Abdolmaleki et al.] 2018)) and include the curiosity models RND, DD, ICM, and NS. We compare these to IMPC with DI and RND optimized with CEM and fixed random agent that sam- ples from the action distribution. Some figures will include a label of MPO, which corresponds to the benchmark of the task-aware RL agents, which provides interesting context to Explore2Offline. The online agent represents the state-of-the-art performance when the task is known priori - matching it without an environment reward function would highlight the potential of Explore2Ofitine.
Environments In this work, we investigate the exploration performance of a variety of DeepMind Control Suite tasks (Tassa al.]
Reacher, Easy
Walker, Stand
1000
2018). We evaluate domains (Ball-in-Cup, Finger, Reacher, and Walker) including 14 tasks with a variety of state-action sizes, with further details included in the Appendix. In or- der to include more challenging environments for task agnostic exploration, we use modifica- tions proposed by [Whitey et al.] (2021). Ex- plore Suite, which include constrained initial states and sparser reward functions.
500-
Epieode Reward
250
0
Reacher, Easy Explore
Walker, Stand Explore
1000
250
4.2 TASK-AGNOSTIC DATA COLLECTION
0
RAND
Plan-DD
Plan RND
React-DD
Reast JCM
Beact-VS
Reart RND
RAND
Plan-DD
Plan-RND
React-DD
Roact ICM 3
React-VS [+
React-NND
We compare the amount of task-reward re- ceived per-episode across a variety of agents and tasks, which is intuitive metric for ex- ploration agent performance but can only act proxy for exploration. To evaluate task- agnostic reward, the exploration agents are run without access the environment rewards, with the rewards are relabelled after. The dis- tributions of normalized reward achieved dur- ing the 5000 training episodes for four example tasks are documented for all of the exploration agents in Fig. 3 Crucially, the Explore Suite variants are challenging for the random agent across its lifetime, where all of the examples shown have median episode-reward of O. There a wide diversity in the agent-task pairings by proxy of measuring experienced reward, showcasing the large potential of future work to better understand this area.
Figure 3: Collected reward per-episode (1000
env. steps) distributions across subset of tasks and their Explore Suite variants. Boxplots show the median episode reward (red line), 25th and 75th percentiles (box), and the maximum and minimum reward computed over 5000 training episodes (whiskers). Only horizontal line indi- cates that the episode reward equal or near 0.
To give overall view of data collection for each agent, we show in Fig. 4 (left) the Control and Explore Suite average collected reward across all tasks. The median, 90th and 10th percentiles

---

Under review as a conference paper at ICLR 2023
Domain, Task
Best Task-Agnostic
Random Agent
IMPC-RND
Task-Aware
Ball-in-Cup, Catch
973.28
943.04
973.40
984.11
Ball-in-Cup, Catch Exp.
962.56
0.00
938.59
974.27
Finger, Turn Easy
981.76
822.27
981.76
983.16
Finger, Turn Easy Exp.
947.94
915.97
944.22
968.67
Finger, Turn hard
975.33
543.57
975.33
977.27
Finger, Turn hard Exp.
952.63
690.79
595.06
965.60
Reacher, Easy
955.01
955.01
900.67
766.79
Reacher, Easy Exp.
813.91
612.77
740.70
973.91
Reacher, Hard
719.90
428.38
719.90
587.37
Reacher, Hard Exp.
930.95
261.20
410.78
973.91
Walker, Stand
534.99
297.63
302.12
991.73
Walker. Stand Exp.
258.55
243.53
81.28
982.79
Walker, Walk
445.53
146.76
96.88
978.36
Walker, Walk Exp.
426.07
47.54
78.88
975.06
The best performance with x 10t transition datasets for all 7 task-agnostic exploration agents, the specific performance of Random Agent and IMPC-RND (to illustrate strengths and weak- nesses), and the task-aware MPO agent (the reported number is the median across ORL seeds on a fixed dataset). Bold represents the best reward across task and the median of near-best agents being within the 10th percentile range of the best agent. This shows where exploration agents perform similarly and consistently online RL for dataset collection. Even for some tasks where intrinsic agents are near to task-aware exploration, the online RL has much tighter performance across seeds. Results for all agents are shown in Appendix A.3
Table 2:
RAND MPO
Plan-DD Flan-RND
React-DD React-ICM Walker, Walk
Readt-NS React-RND Walker, Walk Explor
107.
Finger, Tem Easy
Fiager, Tum Easy Explose
Offine Rt. Reward
3.0.
100
10
During Expsodes
Figure 5: Correlation of final offline RL performance to cumulative reward in the training set on two example tasks and their explore-suite variants. The ORL performances are the median across trials for each task and agent. As the dataset sizes we use span many orders of magnitude of samples, both axes are plotted on log-scale. Across all tasks, there is trend of more reward the training distribution relating to better performing CRR agent.
shown for each agent, combined across tasks. Here also show what reward a task-aware agent (MPO) will collect in lifetime. included to indicate an upper target for exploration, rather than competitive baseline. An important artifact here is the random agent's flatness of reward achieved across time the other exploration agents show an increase in median reward as the dataset size grows (especially Explore Suite). This change in reward distribution across dataset size indicates a diversity of behaviors in the intrinsic exploration agents, while the random agent receives reward from the same distribution repeatedly. This can be seen as curiosity-based agent such as IMPC with RND achieves varied rewards and the random agent has repeated reward distribution.
While the collected reward be indicator of the usefulness of exploration agent, it is directly transferable to a task-focused policy capable of solving tasks. Without careful environment design, task-agnostic agents focusing on novel states will cover the entire state-space regardless of the predefined downstream task.
4.3 EVALUATING OFFLINE RI ON EXPLORATION DATA

---

Under review as a conference paper at ICLR 2023
Turget task Medium Transter
We study the performance of intrinsic agents via a SOTA offline reinforcement learning algorithm, Critic Regularized Re- gression (CRR) [Wang et al. 2020). For each agent, policies were trained on dataset sizes within the range of 10
1,00
Ttaining
Easy
300)
Pointmess
to 5 x 10 environment steps. The mean of the performance across all explore and control suite tasks with confidence inter- vals is shown in Fig. 4 (right). Due the width of this study, we were only able evaluate the offline RL performance on 3 seeds per each collected dataset - the median, max and min across these three policies from CRR are shown in each sub- plot. Here, we see three core findings that we will continue to detail: 1) for dataset sizes 1 X 10", there is little benefit to task-aware learning for data collection and the random agent is strong baseline versus other intrinsic model- model-based agents; 2) on larger datasets, the task-aware RL method, MPO, jumps ahead of the explor- ers, but the exploration agents all continue to improve in offline RL performance with more data; the novel IMPC approach with RND, along with existing methods of MPO with RND or DD, performs best on average with the largest datasets.
1000
Finger
MP0-
MPO- RND
MRO NatSSe
XPO ICM
NPC BND
X01O
MTO DD
XIO RND
AIPO ICM
MIC: DID
MPC: RND
XEO Oalise
MRO DD
MPO RND
MIO Netdhop
XIO ICM
MPC DD
MPC BND
XPO Online
MIO DD
XIO RDD
MEO NeaSeer
XPO ICK
MPC DI
MPC: BND
MPC
MPC Exploration MPO Explonaioo
Online
Figure 6: Performance across tasks in the Finger and Pointmass domains for offline RL agents trained on single dataset. From left to right the tasks become qual- itatively less similar to the initial task. Task agnostii exploration regimes show the potential for better per- formance over task-aware learning, but the results are varied across task.
To showcase which tasks are solved by the Explore20ffine framework, we document the median performance of the exploration agents with the full 5 10 steps training set size versus the task- aware MPO in Tab. Explore20ffline with these agents solve all but the Walker domain tasks, with further results included in the Appendix.
To highlight why dataset size and observed reward are such powerful indicators of ORL perfor- mance, we show in Fig. 5 the correlation for the Finger Turn and Walker Walk tasks of the cumu- lative reward in dataset versus the offline RL performance for that dataset. There is clear trend of more reward resulting in a better policy for the tasks paired with the environment. Fig. 7 Juees Spearman's rank correlation visualise how dataset size is considerably better predictor of perfor- mance than any reward statistics including mean, sum or 80% quantile. This further emphasises the importance to transfer via increasingly large datasets instead instead of the final exploration policy. In the next section we will evaluate how re-using data from task aggoosic exploration can enable multi-task performance.
4.4 EVALUATING AGNOSTII DATA FOR MULTITASK RI.
A key motivation for collecting task-agnostic data is its applicability when there are a variety of downstream tasks of interest, including those which might not be known at data collection time. In the ideal case, task-agnostic exploration could collect single dataset, then offline RL could consume that data (along with relabeled rewards) to solve arbitrary tasks. We evaluate the quality of datasets collected by various exploration agents for with multiple downstream tasks.
For this evaluation we collected one dataset for the Pointmass and Finger environments using each of seven exploration agents, then trained policies for downstream tasks by relabeling the data with different reward functions. Each of the tasks is defined by a sparse +1 reward corresponding to a particular goal state. As baseline, the Online agent collects experience using standard online RL algorithm as it learns to solve a "Training" task, while all of the other data collection agents are task agnostic. The datasets collected by each agent are evaluated with offline RL on the "Training" task
8

---

Under review as a conference paper at ICLR 2023
and three others: "Easy Transfer", "Medium Transfer", and "Hard Transfer". These datasets ranked in the level of challenge that task-aware agent has in generalization. Tasks increase in chal- lenge when goals require moving farther away from the training goal, which can be either travelling further in the same direction (often easier) or entirely misaligned (for harder generalization) Due to the dynamics of the systems, farther-away targets may be easier to discover, e.g. the "Medium Transfer" target for Pointmass lies in the corner of the arena. Full details of these environments, along with experiments on three more, are available in Appendix A.4
The performance of the exploration agents and the task-aware MPO agent are varied across the tasks as shown in Fig. 6 Depicted is the mean reward achieved of offline RI. policies trained on ran- dom seeds for each of datasets of 5 x 10 transi- tions. While collecting data specifically for the tar- get downstream task the best option when the data will be used only that task, the task-aware per- formance can degrade on even slightly misaligned test-task when compared to task-agnostic counter- parts. The potential for multitask transfer of ex- ploration agents highlighted, but further work needed in more open ended environments to show the potential of Explore2Oftine.
1.00
Spearmentt ' Corelaaioon
0.80
0.80
0.33
0.45
0.37
0.97
0.49
Mean
Cumutativn
anda Qunatile
Dataset Stea
0.20
0.00
Figure 7: Spearman's rank correlation be- tween performance of the ORL policy and data statistics such reward (mean, cumula- tive and 80th quantile) as well as dataset size for task-agnostic datasets.
DISCUSSION
Explore20ffline points to interesting directions for further understanding and utilizing task agnostii exploration agents. To start, there are two trends that point to need for further work explo- ration methods. On average across our evaluation suite, the random agent performs very closely to the curiosity-ban based methods, and any particular exploration method varies substantially across task. The performance of the random agent suggests some similarities in the data collected by the random agent and the curiosity-based methods. As mentioned previously, curiosity-based methods ex- haustive (given enough time), and do not consider useful trends that may be common in downstream tasks. This shows need for future exploration methods to be able prioritize interesting subsets of a state-space and generalize across domains to create flexible agents.
Although our evaluation demonstrates the potential of using offline RL task-agnostic data, there is substantial variation across task-agent pairings with the chosen static offline RI. algorithm (CRR). This variation needs to be studied in more detail better understand the limitations posed by the algorithm, and differentiate them from the quality of the data itself.
The intrinsic MPC agent can be progressed by utilizing in other forms of deep RL evaluation. By transferring a learned dynamics model, this flexible exploration agent could also be evaluated as task-aware agent (i.e. zero-shot learning of new task) or online RL by weighting the intrinsic reward model and the environment reward (i.e. better explore-explos balance).
6 CONCLUSION
We introduce Explore20filine, a method for utilizing task-agnostic data for policy learning of un- known downstream tasks. We describe how an agent can be used collect the requisite data once for solving multiple tasks, and demonstrate performance comparable to an online learning agent. Additionally, we show policies trained on task-agnostic data may robust to variations to the ini- tial task compared to task-aware learning, resulting in better transfer performance. Finally, data from the new exploration agent, Intrinsic Model Predictive Control, performs strongly across many tasks. As offline RL emerges data for learning will be needed. Directions for future work include developing better exploration methods specifically for offline training, and identification of experiences with high information for effective datasets.
useful tool in more domains, a deeper understanding of the required

---

Under review as a conference paper at ICLR 2023
REFERENCES
Abhas Abdolmaleki, Jost Tobias Springenherg, Yuval Tassa, Remi Munos, Nicolas Heess, and Mar- tin Riedmiller. Maximum posteriori policy optimisation. arXiv preprint arXiv:1806.06%: 2018.
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning, pp. 104-114. PMLR, 2020.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay, arXiv preprint arXiv: 2017.
Mare G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. International Conference on Machine Learning.
Zdravko I Botev, Dirk P Kroese, Reuven Y Rubinstein, and Pierre L'Ecuyer. The cross-entropy method for optimization. In Handbook of statistics, volume 31.
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros. Large-scale study of curiosity-driven learning. preprint arXiv: 04355, 2018a.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In Int. Conference on Learning Representations, 2018b.
Arunkumar Byravan, Leonard Hasenclever, Piotr Trochim, Mehdi Mirza, Alessandro Davide Ia- longo, Yuval Tassa, Jost Tobias Springenberg, Abbas Abdolmaleki, Nicolas Heess, Josh Merel, et al. Evaluating model- based planning and planner amortization for continuous control. arXiv preprint arXiv:21 10.03363, 2021.
Serkan Cabi, Sergio Gomez Colmenarejo, Alexander Novikov, Ksenia Konyushkova, Scott E. Reed, Rae Jeong, Konrad Zoina, Yusuf Aytar, David Budden, Mel Vecerík, Oleg O. Sushkov, David Barker, Jonathan Scholz, Misha Denil, Nando de Freitas, and Ziyun Wang. A framework for data-driven robotics. ArXiv, abs/1909. 12200, 2019.
Eduardo F Camacho and Carlos Bordons Alba. Model predictive control. Springer science, 2013.
Yevgen Chebotar, Karol Hausman, Yao Lu, Ted Xiao, Dmitry Kalashnikov, Jake Varley, Alex Irpan, Benjamin Eysenbach, Ryan Julian, Chelsea Finn, et al. Actionable models: Unsupervised offline reinforcement learning of robotic skills. arXiv preprint arXiv:2104.07749 2021.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learn- ing in a handful of trials using probabilistic dynamics models. Advances in Neural Information Processing Systems, 31, 2018.
Vibhavari Dasagi. Robert Lee, Jake Bruce, and Jürgen Leitner. Evaluating agntiic exploration for fixed-batch learning of arbitrary future tasks. arXiv preprint arXiv:1911.08660 2019.
Pieter-Tjert De Boer, Dirk P Kroese, Shie Mannor, and Reuven Rubinstein. tutorial on the cross-entropy method. Annals of operations research, 134(1).
Shadi Endrawis, Gal Leibovich, Guy Jacob, Gal Novik, and Aviv Tamar. Efficient self-supervised data collection for offline robot learning. In 2021 IEEE International Conference on Robotics and Automation (ICRA). pp. 4650-4650 IEEE, 2021.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint 10XX:08002.00700,, 2018.
Benjamin Eysenbach, Xinyang Geng, Sergey Levine, and Ruslan Salakhutdinov. Rewriting history with inverse rl: Hindsight inference for policy improvement. arXiv preprint arXiv:2002. 1089, 2020.
Justin Fu. Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. preprint ANiv:2004.002199 2020.
10

---

Under review as a conference paper at ICLR 2023
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. arXiv preprint Seti::20060068666 2021.
Oliver Groth, Markus Wulfmeier, Giulia Vezzani, Vibhavari Dasagi, Tim Hertweck, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Is curiosity all you need? on the utility of emergent 2021.
behaviours from curious exploration. arXiv preprini
Caglar Gulcehre, Ziyu Wang, Alexander Novikov, Tom Le Paine, Sergio Gomez Colmenarejo, Kon- rad Zolna, Rishabh Agarwal, Josh Merel, Daniel Mankowitz, Cosmin Paduraru, et al. RI un- plugged: Benchmarks for offline reinforcement learning. arXiv e-prints, 2020.
Steven Stenberg Hansen, Will Dabney, André Barreto, Tom Van de Wiele, David Warde-Farley, and Volodymyr Mnih. Fast task inference with variational intrinsic successor features. ArXiv, abs/1906.05030, 2020.
Chi Jin, Akshay Krishnamurthy, Max Simchowitz, and Tiancheng Yu. Reward-free exploration for reinforcement learning. In International Conference on Machine Learning, pp. 4870-4877
PMLR, 2020.
Emilie Kaufmann, Pierre Ménard, Omar Darwiche Domingues, Anders Jonsson, Edouard Leurent, and Michal Valko. Adaptive reward-free exploration. In Algorithmic Learning Theory, pp. 865- 891. PMLR, 2021.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint (tt::20000047779 2020.
Nathan O. Lambert, D. S. Drew, J. Yaconelli, Levine, R. Calandra, and K. S. J. Pister. Low- level control of quadrotor with deep model-based reinforcement learning. IEEE Roboties and Automation Letters, 4(4), 2019.
Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Lerrel Pinto, and Pieter Abbeel. Urlb: Unsupervised reinforcement learning benchmark. Advances in Neural Information Processing Systems, 2021.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020.
Alexander C Li, Lerrel Pinto, and Pieter Abbeel. Generalized hindsight for reinforcement learning. arXiv preprint 20Xv::0002.11788 2020.
Hao Liu and Picter Abbeel. Behavior from the void: Unsupervised active pre-training. arXiv preprint 1rXiv:2103.04531, 2021.
Kendall Lowrey Aravind Rajeswaran, Sham Kakade, Emanuel Todorov, and Igor Mordatch. Plan online, learn offine: Efficient learning and exploration via model-based control. arXiv preprint arXiv: 2018.
Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Mare G Bellemare. Safe and efficient off- policy reinforcement learning. preprint arXiv:1606.02647 2016.
Ashvin Nair, Abhishck Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online rein- forcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020.
Soroush Nasiriany, Vitchyr H Pong, Ashvin Nair, Alexander Khazatsky, Glen Berseth, and Sergey Levine. Disco rl: Distribution-connitionee reinforcement learning for general-purpose policies. arXiv preprint arXiv:2 104. 1707, 2021.
Pierre-Yve Oudeyer and Frederic Kaplan. What is intrinsic motivation? typology of computa- tional approaches. Frontiers in neurorobotics, 1:6, 2009.
Simone Parisi, Victoria Dean, Deepak Pathak, and Abhinav Gupta. Interesting object, curious agent: Learning task-agnostic exploration. Advances Neural Information Processing Systems, 34, 2021.
11

---

Under review as a conference paper at ICLR 2023
Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagreement. In Interaational conference on machine learning.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pp. 2778-2787, PMLR, 2017.
Xue Bin Peng. Aviral Kumar, Grace Zhang, and Sergey Levine. Advantago-weighted regression: Simple and scalable off-policy reinforcement learning, arXiv preprint arXiv:1910.00117,, 2019.
Martin Riedmiller, Jost Tobias Springenberg, Roland Hafner, and Nicolas Heess. Collect & infer-a fresh look data-efficient reinforcement learning. In 5th Annual Conference on Robot Learning, 2021.
Max Schwarzer, Nitarshan Rajkumar, Michael Noukhovitch, Ankesh Anand, Laurent Charlin, R De- von Hjelm, Philip Bachman, and Aaron C Courville. Pretraining representations for data-efficient reinforcement learning. Advances in Neural Information Processing Systems, 34:12686-128 12699, 2021.
Kajetan Schweighofer, Markus Hofmarcher, Marius- Constantin Dinu, Philipp Renz, Angela Bitto- Nemling, Vihang Patil, and Sepp Hochreiter. Understanding the effects of dataset characteristics on offline reinforcement learning. arXiv preprint arXiv:2111.0477 2021.
Ramanan Sekar.
Oleh Rybkin, Kostas Danilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via supervupe world models. In International Conference on Machine Learning.
Pranav Shyam, Wojciech Jaskowski, and Faustino Gomez. Model-based active exploration. In International conference on machine learning, pp. 5779-5788. PMLR, 2019.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmalcki, Josh Merel, Andrew Lefrancq, et Deepmind control suite. preprint arXiv: 10X1v08000000990 2018.
Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott E Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, et al. Critic regularized regression. Advances in Neural Information Processing Systems, 33, 2020.
William F Whitney, Michael Bloesch, Jost Tobias Springenberg, Abbas Abdolmaleki, Kyunghyun Cho, and Martin Riedmiller. Decoupled exploration and exploitation policies for sample-efficien reinforcement learning. arXiv preprint arXiv:2101.0942 2021.
Pierre-Brice Wieber. Trajectory free linear model predictive control for stable walking in the pres- ence of strong perturbations. In TEEE-RAS International Conference on Humanoid Robots.
Grady Williams, Nolan Wagener, Brian Goldfain, Paul Drews, James M Rehg. Byron Boots, and Evangelos A Theodorou. Information theoretic mpe for model-based reinforcement learning. In 2017 IEEE International Conference Robotics and Automation (ICRA), pp. 1714-1721. IEEE, 2017.
Markus Wulfmeier, Abbas Abdolmaleki, Roland Hafner, Jost Tobias Springenberg. Michael Ne- unert, Tim Hertweck, Thomas Lampe, Noah Siegel, Nicolas Heess, and Martin Riedmiller. Com- positional transfer in hierarchical reinforcement learning. arXiv preprint
11228, 2019.
Markus Wulfmeier, Arunkumar Byravan, Tim Hertweck, Irina Higgins, Ankush Gupta, Tejas Kulka- mi, Malcolm Reynolds, Denis Teplyashin, Roland Hafner, Thomas Lampe, et al. Representation matters: Improving perception and exploration for robotics. In 2021 IEEE International Confer- ence on Robotics and Automation (ICRA), pp. 6512-6519 IEEE, 2021.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement learning with pro- totypical representations. arXiv preprint 2xXiv:2102.11277 2021.
Denis Yarats, David Brandfonbrener. Hao Liu, Michael Laskin, Pieter Abbeel, Alessandro Lazaric, and Lerrel Pinto. Don't t change the algorithm, change the data: Exploratory data for offline reinforcement learning, 2022.
12

---

Under review as a conference paper at ICLR 2023
Domain
Task
dg. da
Description & Explore Suite Modifications Agent moves cup swing ball attached via string into it. The explore suite version restricts the initial states to avoid trivial so- lutions.
Catch
Ball-in-Cup
8,2
Catch Explore
Turn-Easy
Agent controls finger, similar to a manip- ulator, interact with objects. The explore suite version restricts the initial state of the finger be away from the object.
Turn-Easy Explore
Finger
12,2
Turn-Hard
Turn-Hard Explore
Turn-Easy
Agent controls a 2d manipulator to reach an arbitrary target state.
Turn-Easy Explore
Reacher
6,2
Turn-Hard
The explore suite version restricts the pos- sible targets to positive cone in front of the manipulator.
Turn-Hard Explore
Stand
Agent controls a 2d humanoid robot to move forward or stand. The explore suite version allows the robot to start on the ground rather than standing and has sparse walking reward.
Stand Explore
Walker
24, 6
Walk
Walk Explore
Table 3: Extra information on the tasks used in the paper in DM Control Suite.
A
APPENDIX
Here we include additional experimental context and results.
A.1 ALGORITHMIC DETAILS
A summary of the exploration algorithm, Intrinsic Model Predictive Control is shown in Alg. [ utilize a distributed setup where multiple actors and learners can be deployed concurrently.
We
Algorithm Intrinsic MPC
Given: Randomly initialized proposal ma, dynamics model mgs reward model rs, random critic Q [Modules to be learned)
Given: Planning probability Palans replay buffer B, MPO loss weight learning rates & optimizers (ADAM) for the different modules. (Known modules and parameters]
{ Exploration loop Asynchronously on the actors)
while True do
Initialize ENV and observe state
while episade is terminated do
Choose between intrinsic planner random action depending on Pptan (Use learned proposal (Tt,) as proposal for planner).
sample r ~ U[O,
PLANNER (e: iTo mo,
Pplan
Ta()))
otherwise.
Step ENV(S,, a.) (Ste1) and write transition replay buffer B end while end while
(Asynchronously the learner)
while True do
Sample batch B of trajectories, each sequence length T from the replay buffer B
Label rewards with reward model r..
Update action-value function 2, based on B using Retrace (Munos et al. 2016).
Update model m based on using multi-step.
Update reward model r; based on
Update proposal my based on B using (Byravan et al. 2021)
end while
13

---

Under review as a conference paper at ICLR 2023
The PLANNER subroutine takes in the current state St, the action proposal na. dynamics model Ms that predicts next state $t+1 given current state 8, and action O,, the reward function model (1 a.). Optionally, learned state-value function (s) (with parameters e)) that predicts the expected return from state 8 can be provided. We use the Cross-Entropy Method (CHM) (Botev a. shown in Alg. 2
Algorithm 2 CEM planner
Given: state So, action proposal Ta. dynamics model ma. reward model planning horizon H, number of samples S, elite fraction E, noise standard deviation Tnits and number of iterations
(Rollout proposal distribution using the model.}
(Sp, ag $1....... 9H) + proposal(mp. No. H)
1 < [ag, 01. [
Şinitial plan)
- Jnit
{Evaluate candidate action sequences open loop according to the model and compute associated returns.)
for i 1 do
for 1...Sod do o) Fk -evaluate actions(m Pk: H, r;)
{Sample candidate actions.
end for
Rank candidate sequences by reward and retain top E fraction.
Compute mean Melite and per-dim standard deviation elite based on the retained elite sequences.
11 (1 meean μ + O'mean/Meite
{Update mean; O'nean 0.9} {Update standard deviation; and 0.5)
(1 agd ) Orate
end for
return first action in μ.
A.2 ADDITIONAL ENVIRONMENT DETAILS
The state-actions dimensions (d,, d,,) and descriptions for the environments used in this paper detailed in Tab. 3. Additional collected reward distributions are shown in Fig. 11
A.3 FULL OFFLINE RL AGENT-TASK PERFORMANCES
To supplement the results discussed in Sec, 4.3 we have included the performance per dataset size for all agents across all tasks. The results are shown in Fig. and show the considerable variation when studying any given agent or task. There is substantially more variation across task than across agent, showing the value in continuing to fine-tune set of tasks for benchmarking task-agnostic agents. The mean performance across all tasks is shown in Fig. 9 with per-task breakdown shown in Table 4 A subset of agents and tasks are shown in Fig. show the convergence on set of tasks.
A.4 ADDITIONAL MULTI-TASK LEARNING EXPERIMENTS
To compliment Sec. 4.4 we have included additional experiments for multi-task learning in the Reacher, Cheetah, and Walker environments, shown in Fig. 10 For these tasks, there is less clear of benefit of using task agnostii learning generate data for offline RL policy generation. In our experience, this limited performance can be due to the fact that the environments are designed with specific behaviors and algorithms in mind, reducing the need for diverse exploration method. A description of the starting state and the goal state for each task, as well as for the Pointmass and Finger environments from the main text, is available in Table 4.
14

---

Under review as a conference paper at ICLR 2023
MPO
RAND
Plan-RND
React-
Tissk owaaee
1000
Ball-in-Cup, Catch
Finger, Tum-Easy
Offline RL Evaluation Episode Reward
750
500
250
0
Reacher, Easy
Walker, Stand
1000
750
500
250
9
1e4
5e4
1e5
1e6
5e6 2e3
1e4
5e4
1e6
5e6
Dataset Size (Environment Steps)
Figure 8: Highlighting performance subset of tasks and agents to showcase the relative differ- ences that can emerge across tasks. The median, max and min across training seeds shown for one input dataset.
Offline RL Evaluation Episode Reward
Dataset Size
1000
2 10
x 10
5 10
1 10*
1 10
5
10
600
400
200
RAND
MPO
Plan-DD
Plan-RND
React-DD React-ICM React-NS React-RND
Task-awwae
Figure 9: The median, max and min across training seeds are shown for different dataset sizes. This figure shows how most of the exploration methods we evaluated performed similarly, yet our new IMPC agent with RND does have the best overall performance.
15

---

Under review as a conference paper at ICLR 2023
Target task
Training
Easy Transfer
Medium Transfer
Hard Transfer
1,000
800
Reacher
600
400
200
1,000
Cheetah
600
400
200
1,000
Wiler
200
MPO NextStep
MPO NexiStep
MPO + NextStep
MPO NextStep
MPO Online
MPO DD
MPO + RND
MPO ICM
MPC DD
MPC + RND
MPO Online
MPO + DD
MPO RND
MPO ICM
MPC+ + DD
MPC + RND
MPO Online
MPO DD
MPO RND
MPO + ICM
MPC+ DD
MPC RND
MPO Online
MPO DD
MPO + RND
MPO ICM
MPC+ DD
MPC + RND
MPC + Exploration
MPO + Exploration
Online
Agent type
Figure 10: Performance across multiple tasks for offline RL agents trained on single task-agnostic
dataset.
16

---

Under review as a conference paper at ICLR 2023
Ball-in-Cup, Catch
Finger, Turn-Hard
1000
Episode Reward
750
500
250
1
Ball-in-Cup. Catch Explore
Finger, Turn-Hard Explore
1000
Episode Reward
750
500
250
Plan-DD 4TH
0
RAND
Plan-DD
Plan-RND
React-DD
React-ICM
React-NS
React-RND
RAND
Plan-RND
React-DD
React-ICM
React-NS
React RND
Figure 11: Collected reward per-episode (1000 environment steps) distributions across a subset of tasks and their Explore Suite variants. Boxplots depict the median episode reward (red line), 25th and 75th percentiles (box), and the maximum and minimum reward over 5000 episodes (whiskers). Across the suite of tasks studied, there is often comparable variation among the exploration agents as the variation among tasks.
Task-Aware Aware
Random
IMPC-DD
IMPC-RND
ICM
NSM
RND
DD
Domain, Task
Ball-in-Cup, Catch
984
943
973
973
975
901
972
922
Ball-in-Cup, Catch Explore
974
886
939
963
882
955
927
Finger, Turn Easy
983
822
768
982
577
541
689
630
Finger, Turn Easy Explore
969
916
636
944
850
170
920
948
Finger, Turn hard
977
544
486
975
513
366
510
567
Finger, Turn hard Explore
966
691
86
595
152
235
953
863
Reacher, Easy
767
955
881
901
626
612
550
868
Reacher, Easy Explore
974
613
254
741
98
89
235
814
Reacher, Hard
587
428
105
720
705
143
128
266
Reacher, Hard Explore
974
261
407
411
931
542
773
148
Walker, Stand
992
298
504
411
242
535
519
Walker, Stand Explore
983
244
259
81
188
232
179
256
Walker, Walk
978
147
446
103
60
107
84
Walker, Walk Explore
975
48
426
79
76
79
91
37
Table 4:
The best performance with 5 x 10 transition datasets for all task-agnostic exploration To complement the table in the main
agents and the task-aware MPO agent compliment Table text, bold represents the best reward among only task-agnostic agents across a task being within the range of 10th and th percentiles.
17

---

Under review as a conference paper at ICLR 2023
Domain
Start state
Training
Easy Transfer
Medium Transfer
Hard Transfer
Pointmass (r, y)
(0, 0)
(0.1,0.1)
(-0.1,-0.1
(0.3, 0.3)
(0.2. 0.2)
Finger (0)
(0)
(0.5)
(1.0)
(2.6)
(-2.6)
Reacher (x, y)
(0.24, 0)
(0.0.1)
(0.0.2)
(0. (0.-0.1)
(-0.1, 0.15)
Cheetah (v)
(0)
(2.5)
(-1.1)
(1.1)
(4.3)
Walker (v)
(0)
(1.5)
(0.5)
(-0.5)
(2.5)
Table 5: Starting and goal states for each of the multitask environments. For Finger, represents the angle of spinner which the finger can turn, and for Cheetah and Walker represents forward velocity.
MPO
Plan- RND
React-ICM
Beact-RND
RC--C
F-TEXX
F-TH
102
10a
Rewaga
00
00
107
W-S
W--X
X
10a
10a
10
10ta Exploration
10'
1d
Figure 12: Correlation of final offline RL performance to cumulative reward in the training set. As dataset sizes we use span many orders of magnitude of samples, both axes are plotted on log- scale. Across all tasks, there is trend of more reward in the training distribution relating to better performing CRR agent.
18

---

Under review as a conference paper at ICLR 2023
RAND
MPO
Plan DD
RND
React DD
Reart-ICM
React-NS
React-RND
Mean per Task
1000
500
500
500
1000
500
1.00
1000
500
Agret
500
20p
Figure 13: All agent-task pairings for final offline reinforcement learning performance. All tasks in the Explore and Control Suite occupy the rows while the Agents occupy the columns. The average results highlight the increased variation across tasks when compared agents.
19


Reviews:
[{"review_id": "zct_0f1WCZe", "reviewer": "Reviewer_58zj", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a study of a reinforcement learning setting consisting of a task-agnostic data collection phase and a task-aware offline optimization phases, on a set of continuous control tasks. Comparing the performance obtained when using different exploration criteria (combined with a new MPC-based method) in the data collection phase, as well as different dataset sizes, this work tries to draw conclusions about the features we should expect to find in offline RL datasets for a downstream algorithm to perform well.", "summary_of_the_review": "Overall, I appreciate the direction presented in this work, as well as the general methodology that has been attempted. However, I believe there are some serious drawbacks that might overly reduce the scope of the conclusions drawn in the paper, and thus for now I tend to recommend rejection.", "strength_and_weaknesses": "Strengths:\n- The problem addressed in the paper is of paramount importance: increasing our understanding of the kind of datasets that are needed to reach good performance with offline reinforcement learning has the potential to both shape the design of future offline RL algorithms and to have practical implications;\n- The proposed MPC-based exploration method, which could be of independent interest, stands out as a clean way (probably cleaner than the reactive counterpart) of comparing the different exploration criteria.\n- Despite being probably already known to some researchers in the field, having clearly written down some of the conclusions about the influence of dataset size and reward distribution could be useful to practitioners.\n\nWeaknesses:\n- The paper only uses the CRR offline RL algorithm for evaluating the downstream effects of the construction of different types of datasets. Despite this is a start, the actual outlook of offline RL algorithms used in the community is very vast, and there is really no baseline as established as for in the online RL case. This severely limits the scope of the work, since the reader cannot know a priori whether the conclusions of the study only hold for that specific offline RL algorithm or are indeed more general than that;\n- The paper determines dataset size is possibly the most important variable to predict the performance for a downstream offline RL algorithm. However, this could be a bit misleading: one could imagine having the same trajectory over and over again in the dataset and clearly increasing the dataset size without any increase in the downstream performance. Despite this is a trivial counterexample, I think it is really epistemologically important to pin down the effect of diversity, more than size, on the resulting policy, because that one is more likely to be the underlying cause of the observed performance increase.\n- The comparison with previous and parallel work is not very insightful from the perspective of the reader. For instance, it would be nice to understand more the relationship with (Yarats et al., 2022), instead of just briefly mentioning its existence. Moreover, there is some work similar in flavour about pretraining (https://arxiv.org/abs/2106.04799): it seems important to have a discussion on how the study presented in this work is different compared to that.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "y316O2cJpnO", "reviewer": "Reviewer_BjG9", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies offline RL from a less-studied perspective - collecting informative experiences. The authors investigate the task-agnostic setting based on curiosity-based intrinsic motivation methods. The authors propose the Explore2Offline framework, and conduct an extensive empirical study to investing the effect of data collection strategies. ", "summary_of_the_review": "The paper studies offline RL from an interesting and less-studied perspective (data collection instead of policy learning), and propose a new exploration agent, Intrinsic Model Predictive Control that have strong performance. The authors also conduct extensive and in-depth analysis and investigation in standard offline RL tasks with different qualities. My main concern is its novelty, which is quite related to [1], but without enough discussion about their differences.", "strength_and_weaknesses": "### Strengths\nThe paper is clearly written and easy to follow. The authors also conduct extensive experiments to investigate data collection in offline RL based on a number of intrinsic motivation based exploration algorithms to reveal interesting findings. \n\n### Weakness\nMy main concern for the paper is its novelty. Although I acknowledge that the paper studies offline RL from a less-studied perspective (data collection instead of policy learning that aims to address extrapolation error) which is very interesting, it seems that it is very related to [1] without enough discussion about the differences. Specifically, [1] also studies data collection in offline RL, and it is worth discussing the differences between them.\n\n[1] Denis Yarats, David Brandfonbrener, Hao Liu, Michael Laskin, Pieter Abbeel, Alessandro Lazaric, and Lerrel Pinto. Don’t change the algorithm, change the data: Exploratory data for offline reinforcement learning, 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "n-bVbhyQZy", "reviewer": "Reviewer_QZYQ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper explores how to collect informative data for offline RL methods. Many curiosity-based methods are considered to explore the environment. Intrinsic Model Predictive Control (IMPC) approach is proposed to improve the performance.", "summary_of_the_review": "Please see my questions above.", "strength_and_weaknesses": "The idea and setting are novel and interesting. However, the method part is a little difficult to follow. the description of the IMPC is not clear.\n1. What are the input and output of IMPC? How do you train it? What is the main contribution?\n2. The experiments only choose MPO to verify the quality of the samples, which is not convincing enough. Since you propose a new framework and use the offline RL methods as a mechanism for evaluating exploration performance, I think more than 2 most popular methods should be used,\n3. The paper does not give a deeper analysis of the data requirement of the offline RL. In other words, in order to get better performance, which kind of data is required by the offline RL methods? How to construct such a dataset?\n4. There are some papers focusing on offline training and online fine-tuning to improve performance via fewer data. They also use exploration methods to guide online data collocation. Their methods can also be compared after slight modification.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject