Don’t Start From Scratch: Leveraging Prior Data to Automate Robotic Reinforcement Learning
Authors: Homer Rich Walke
Jonathan Heewon Yang
Albert Yu
Aviral Kumar
Jędrzej Orbik
Avi Singh
Sergey Levine
Venue: CoRL 2022 Poster (2022)
Keywords: autonomous RL
offline RL
reset-free manipulation

Abstract: Reinforcement learning (RL) algorithms hold the promise of enabling autonomous skill acquisition for robotic systems. However, in practice, real-world robotic RL typically requires time consuming data collection and frequent human intervention to reset the environment. Moreover, robotic policies learned with RL often fail when deployed beyond the carefully controlled setting in which they were learned.
In this work, we study how these challenges of real-world robotic learning can all be tackled by effective utilization of diverse offline datasets collected from previously seen tasks. When faced with a new task, our system adapts previously learned skills to quickly learn to both perform the new task and return the environment to an initial state, effectively performing its own environment reset. 
Our empirical results demonstrate that incorporating prior data into robotic reinforcement learning enables autonomous learning, substantially improves sample-efficiency of learning, and enables better generalization.

Don't Start From Scratch: Leveraging Prior Data to Automate Robotic Reinforcement Learning
Homer Walke', Jonathan Yang', Albert Yu', Aviral Kumar', Jødrzej Orbik', Avi Singh', and Sergey Levine
UC Berkeley UT Austin 'Google
Abstract: Reinforcement learning (RI.) algorithms hold the promise of enabling autonomous skill acquisition for robotic systems. However, in practice, real- world robotic RL typically requires time consuming data collection and fre- quent human intervention to reset the environment. Moreover, robotic policies learned with RL often fail when deployed beyond the carefully controlled set- ting which they were learned. In this work, we study how these challenges can all be tackled by effective utilization of diverse offline datasets collected from previously seen tasks. When faced with new task, our system adapts previously learned skills to quickly learn to both perform the new task and re- turn the environment to an initial state, effectively performing its own environ- ment reset. Our empirical results demonstrate that incorporating prior data into robotic reinforeement learning enables autonomous learning, substantially proves sample-efficiency of learning, and enables better generalization. Project website:
Keywords: autonomous RL, offline RL, reset-free manipulation
Introduction
Reinforcement learning (RL) provides general learning-based framework that, in principle, could be utilized to acquire any goal-directed behavior. However, in practice, the standard RL formulation overlooks many of the challenges that arise in real-world robotic learning. RL problems are classi- cally (though not exclusively) situated in settings where ample exploration can be performed from scratch, the environment can be reset episodically, and the focus is more on attaining the highest possible performance rather than good generalization, such in the of playing a video game. Real-world robotic learning problems have very different constraints. With real-world robots, online interaction and exploration are often ronment itself between attempts, and the natural variability and uncertainty of the real world makes generalization far more essential than squeezing out every bit of final performance.
premium, the robot must figure how to reset the envi-
Humans and animals handle each these challenges in the real world. However, in contrast to standard robotic RL settings, humans do not approach each new problem with a blank slate: leverage prior experience to help us acquire new skills quickly, scaffold the process of learning that skill (e.g.. by using our existing skills practice and try again), and utilize representations learned from prior experience to ensure that the new skill is represented in a robust and generalizable way. We study analogous approach in this work and propose complete system for extracting useful skills from prior data and applying them to learn new tasks autonomously. Although the individual components of system are based on previously proposed principles and methods for offline RL [1], multi-task learning with task embeddings [2, 3J. and reset-free learning with forward- backward controllers [4, 5, 6, 7, 8], our system combines these components into a novel framework that effectively enables real-world robotic learning and leverages prior data for all of these parts.
Our system consists of two phases: offline learning using the prior data, and mostly autonomous online fine-tuning on new task (see Figure 1), where only occasional resets are provided. In the offline phase, we offline RL extract skills and representations from the prior data. Then, in the online phase, we adapt the learned skills to the new task, alternating between attempting the task and resetting the environment. If the behaviors required to perform the new task and reset the
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Phose 1. Offine Leoming
Phase Online Turing on New Task
Now Tock
Oftine Frior
Exploratio Trigecloriss
Fonward
Mulltass Office AL
CEM
RL
Ry. Ri
Bacowart
Fe a &s 2)
Forward Folels, sy:
Bacinars Consoter Talolee (4)
senee Muis Pollcy
Task Human Donos
Figure Overview of our system. the offline phase (middle), we train a multi-task policy that captures prior knowledge from an offline dataset of previously experienced tasks and a handful of human-teleoperated demos of new task (left). Then the online phase (right) the multi-task policy is used to initialize both forward policy and backward (reset) skill for learning the new task, improving learning speed and generalization.
environment are structurally similar to those in the prior data, the skills learned offline will succeed with non-negligible probability, providing learning signal that allows the agent to adapt its behavior with only small amount of online data. Moreover, where polices learned from narrow data may overfit to unimportant details of the environment and fail when those details change. polices learned from diverse multi-task data can generalize to new conditions with little to no adaptation.
Our main contribution is demonstrating that incorporating prior data into reinforcement learning system simultaneously addresses several key challenges real-world robotic RL; sample-efficiency. zero-shot generalization, and autonomous non-episodic learning. We validate our approach on real- world robotic manipulation, where we show that our method makes it possible to learn new ma- nipulation skills via RL in settings where prior approaches struggle make progress, reach fi- nal performance that is comparable to what the algorithm can attain when provided with "oracle" demonstration data for the new task, and acquire policies that generalize more broadly than those trained only on single-task data.
2 Related Work
Prior work has explored learning behaviors from offline datasets, leveraging multi-task data, and reinforcement learning with minimal resets. We build on techniques from each area to devise complete system for automated robot learning. We briefly review some related works below.
Accelerating online RL with offline data. We an offline RL method [9, 10, 11, 12, 13, 14, 15, 16, 1, 17, 18, 19] to extract useful skills and representations from prior data, but our system includes additional components that allow us to use multi-task offline data and autonomously fine-tune the policy online by learning to both perform the task and reset.
Multi-task RL. Similar to methods for multi-task RL [20, 21, 22, 23, 24, 25, 26, 27, 28, 2, 29, 3], we adopt the strategy learning a single policy conditional on a space of tasks. However unlike multi-task RL methods, our focus is adapting this multi-task policy to unseen tasks. Perhaps the closest prior system to ours MT-Opt [2], which addresses multi-task learning for similar class of pick-and-place robotic manipulation tasks. However, in contrast to this prior work, our focus is not on how to train a multi-task policy, but specifically on how prior offline data from varied tasks can be leveraged to improve the autonomy and generalization when learning Kalashnikov et al. [2] also evaluates fine-tuning to new tasks, our system goes significantly further, aiming to automate resets (which the prior work does not address, instead using instrumented bin setup), and evaluating the benefits in terms of generalization and efficiency for the new task.
task. While
Reset-free RL. Our system aims to utilize prior data to autonomously learn robotic skills, leveraging prior experience to both accelerate the learning of new behavior and the process learning how to reset between attempts. Prior work has tackled this "reset-free" learning problem [30] in a number of ways. Some prior work uses a curriculum-based approach, which relies on the observation that when learning several tasks simultaneously, some tasks reset others, thus forming curriculum

---

[31, 32, 33, 4, 34]. While Gupta et al. [31] also uses a multi-task setup to automate resets, this prior paper involves a small number of tasks that all focus on enabling reset-free learning for a particular (single) skill, whereas our focus is specifically on using prior data for other tasks to enable new task to be learned as autonomously as possible. Other approaches learn separate controllers for performing the task and resetting the environment [4, 5, 6, 7, 8). Though our method also learns separate controllers, we initialize them with potentially useful skills extracted from prior data, Most similar to our work, Sharma et al. [8] use prior data in the form of demonstrations to accelerate learning forward and reset behaviors. Our system can use expert demonstrations of the target task, but main focus is using potentially sub-optimal data from other tasks.
3 Autonomous Robotic Reinforcement Learning with Prior Data
First, we formalize our problem statement and describe our evaluation setup. Our goal is to utilize data from previous tasks to address three key challenges in robotic reinforcement learning: sample- efficiency, autonomous learning, and generalization and robustness unseen test conditions. We assume access to prior dataset D Donior of experience collected from multiple training tasks: Dprior
- Dj where D { "/, task Tj- Our goal will be to extract a rich repertoire of skills from this diverse, multi-task prior dataset Dprior to quickly solve a new target task T". For the prior data to be useful, the training tasks must share structural similarity with the new task. We evaluate two settings, based on the level of similarity between the tasks in Pprior and 7*. In the first setting, which we call direct transfer, I' is not seen at all during training. For direct transfer to succeed, the training tasks must share enough similarity with the target task that a policy trained on Pprior achieves non- zero success rate on r" If this condition does not hold, we evaluate in a second setting which we call transfer with demonstrations. We provide additional guidance by augmenting Dprioe with a small number of demonstrations of the target task D... (collected by human teleoperation) so that Pprur = Ppriur UD--. (This is labeled as Stage 0.5 in Fig. 1.) In both settings, when the robot is practicing on the target task 7", it is provided with minimal external resets, in accordance with the autonomous RL paradigm [30]. The learned agent is then evaluated on the same task T", but this time the environment is reset after every episode.
j ))
denotes the chunk of the prior data corresponding to
Our method, which we call Autonomous RobotIc REinforcement Learning (ARIEL), consists of two phases: offline learning using the prior data followed by mostly autonomous online adaptation to target task (see Figure 1).
3.1 Learning From Prior Data with Offline RL
Given a dataset from previous tasks, we first aim extract skills and representations that could be useful for learning downstream tasks. The multi-task dataset Dprioe can come from many dif- ferent sources: human demonstrations, sub-optimal data from previous reinforcement learning ex- periments, or even data collected using (imperfect) hand-engineered policies. To make use of this multi-task dataset for downstream tasks, we train a conditional policy (a)as z) using offline RL; we utilize the AWAC [1] algorithm in our experiments. This policy is conditioned on a task index z (represented as a one-hot vector) that informs the agent which task is performing. Training single multi-task policy offers several advantages over training separate policies on the prior dataset. First, we are able learn a representation of policy inputs and outputs (in this case, images and desired change in end end-effector pose, respectively) which is shared across tasks, and is therefore more likely to generalize well when faced with new inputs: both through interpolation (if the new task similar to previously seen tasks), and extrapolation (for novel tasks). Second, when attempting new task, we only need to sample actions from a single pre-trained policy, as opposed to choosing from an ensemble of pre-trained policies. Finally, this also allows our method to gracefully scale with the number of tasks in the prior dataset. After the offline learning phase, we have a multi-task policy To ( s, z) that can perform each of the tasks in the prior data. In Section 4 we discuss the specific tasks we use in our prior data and how we generate the prior dataset for our experiments.
3.2 Autonomous Online Training
After extracting skills from the offline data, our systems adapts the skills to efficiently learn new task with minimal externally provided environment resets. Since the environment is not frequently reset, the agent must simultaneously learn to perform the task and reset the environment to the initial state so it can continue practicing. Prior work has tackled this problem by learning distinct forward and backward controllers, alternating between applying the forward controller to perform the task and applying the backward controller to reset the task [4, 5, 6, 7, 8). We adopt an analogous strategy,
3

---

but we leverage the multi-task policy described in the previous section to bootstrap learning in both the forward and backward direction, which both helps to overcome the exploration challenge and provide automated resets more quickly than if the backward controller were trained from scratch.
To enable this, we fine-tune the parameters of the policy To (als, z) obtained after the offline phase with online data from the new task r", which we denote as D,-, similarly to prior work that uses online fine-tuning to improve upon policies learned via offline RL alone [1]. However, since we have a multi-task policy, we must confront the question of which skill(s), parameterized by task indices z, should be chosen to start fine-tumin aen-ttning for the forward and backward controller. If demonstrations of the target task were provided (in the transfer with demonstrations setting), we can use the task indices associated with these demonstrations. Otherwise, if no demonstrations were provided (the direct transfer setting), we must automatically determine which prior skill (or combination of prior skills) to fine-tune. To do this, we treat the task index z as continuous, learnable parameter and use data collected during online reinforcement learning to select the z's best suited for the given new task. In this sense, we treat z as a task embedding [3] that is automatically adapted for the task at hand, similar to prior work in meta-reinforcemen learning [35].
Since we aim to learn both a forward and backward controller, we optimize for two skills, parame- terized by two distinct task embeddings 2, and Zy, that respectively perform the task and reset the environment. Conditioning the policy on / results in the forward controller aa(a|s. zy) that per- forms the task, whereas conditioning the policy on Zy results in the backward controller ITe (als, zb)) that resets the environment according to the initial state distribution of the task T*. During fine- tuning we alternate between applying the forward controller and backward controller, optimizing the forward controller with rewards for completing the task y a) and the backward controller with rewards for returning to the initial state distribution rs(s, a). We optimize the parameters z and z, in tandem with the policy and value function parameters 0 and ø. The parameters of the policy and value function are updated using using an actor-critic algorithm (we use AWAC [1]). The embeddings f and z, are optimized differently, as we discuss next.
Optimizing task embeddings z and Zb. Since our goal is to learn the target task 5" as quickly as possible, and with minimal re- sets, we need to effectively explore the environ- ment during online training. An ideal approach that explores effectively must not prematurely commit to particular value of z, and 2u until the online experience clearly indicates so, and at the same time, it should be fast at collaps- ing to the correct values of 2y and 2y. when it is apparent from the online experience gath- ered until then. We found that the cross- cntsspyy method (CEM) [36] provided us with a favor- able balance between exploration and exploita- tion. CEM provides way to initialize the task embedding to a distribution, which ensures that we do not have to commit to fine-tuning any one particular skill. Since we want to optimize both the forward and backward embeddings y and Za. we fit two sampling distributions with CEM, (z) and (z).. The distribution of tasks in the prior data D Ppio provides us with a natural (and hopefully informative) prior to initialize qy (z) and (z(z). On each iteration of training. we sample a z from either q, (z) or (z(z) and roll out the policy conditioned on this z value for M steps. These M M-step trajectories essentially correspond
Algorithm ARIEL online adaptation
Input: pre-trained policy no (alas 2). Q-function (s.
1: Initialize buffer 8, CEM models qy.
2: Fit (z and qu(e (2) to offiine takk indioes
3: d <-- f // usk direction, i forward, b backward
4: while done do
5:
p(Bu) // sample initial state (i. external reset)
6:
for N steps do
7:
If sample new task embedding at max trajectory length if steps % .M == 0 then qa end if
8:
9:
10:
11:
Sample a ~u (- z). observe s' ~ p(-(s,a) B + 8 U {(s, ra(s', a).s',z) Ta a) then
12:
13:
14:
svitch(d) switch task direction end
15:
16: update ne. Q7 w/ RL. update oy (zs). qo (x) CEM
17:
x
+-+ x
18:
end for
19:
end while
an episode in episodic RL, however the environment is (typically) not reset afterward. We alternate between sampling the from gf (z) or q (z(z) to alternate between applying the forward and reset controllers. The reward function correspondingly alternates between (s, a) and rs(s, a).. We periodically refit (z) and (z(2) to the J most recently sampled z's that resulted in trajectories that successfully completed or reset the task, respectively. We update the policy and value function parameters 0 and á using the update steps from RL algorithm as summarized in Algorithm 1. Every N steps, we provide an external reset to the environment. How- ever, N >> > M, so the external resets are infrequent. For example. in one of our real world tasks, M- = 20, and N = 400. In theory, we could run our method without any manual resets at all, but we

---

(a) Objects and Containers.
(b) Downstream Tasks.
Figure 2: (a): Objects used to construct the train tasks (upper) and test tasks (lower) in our experi-
ments. (b): Selected test tasks on our real-world robotic setup.
found small number of manual resets necessary to recover from certain states, similarly to prior work on reset-free RL [5].
3.3 Algorithm Summary
To summarize, first use offline RL on Peiio to obtain a multi-task policy (a z) and value function (s a, z). Then, in the online phase (see Algorithm 1), we adapt this policy and value function target task. Notably, the offiin phase occurs only once, but the online phase can be run many times to efficiently learn many new tasks. During online adaptation, we use To Ta(als, z) to initialize forward and backward controller that perform and and reset the task respectively. Alternating between applying the forward and backward controller allows the agent to autonomously practice the target task. Our system enables environment resets by leveraging prior data from previous tasks. We provide further implementation details in the Appendix.
learn the new task efficiently and with minimal
4 Experiment Setup
We now describe our experimental setup for data collection, training, and evaluation.
Robotic platform. We evaluate our approach using a 6-DoF Widow robotic arm (see Figure 2b). The 7D action space of the robot consists of 6D Cartesian end-effector motion, corresponding to relative changes in pose, as well as one dimension to control the opening and closing of the parallel jaw gripper. The state observations consist of the joint angles and RGB images from an overhead camera, which are 48 x8 in simulation and either 64 or 128x 128 in the real world.
Evaluation tasks. To effectively leverage data from prior tasks solve new tasks, the prior tasks must share structural similarity with the new tasks. In our evaluation, we study a variety of pick and place tasks, where the shared structure is that each task involves picking up an object and placing it into or on top of some container or object, and the variability between the tasks corresponds different containers or placement targets, different objects to be picked, and potentially even additional manipulations that must be performed on the container (e.g., opening a drawer prior to putting something inside). The tasks are performed in one of two scenes: a simpler scene where the objects and containers are in tray, and more visually complex kitchen scene. Both scenes contain distractor objects that are not involved in the tasks, and the robot required to correctly learn which object to interact with, how to pick it, and where to place it. All tasks have sparse rewards evaluated using a simple hand-designed vision system, with a reward of +1 the robot completes the forward (or backward) task, and 0 otherwise. While such rewards are casy to define, they present a substantial challenge for RL.
The test tasks that we fine-tune some of which are shown in Fig. 2b, span range of difficulties and consist of picking and placing with objects and containers that were not seen in the prior data. Three of the tasks are in-distribution and hence similar to the prior data: placing toy tiger in drawer (which requires also opening the drawer, then closing again to reset), placing toy elephant in pot, and placing toy tiger lid. The other four tasks are out-oo-disttibbution: placing a cup on saucer, inserting ring onto a peg, sorting block into correct slot, and moving pot to the stove. The cup, ring, and pot require significantly different grasping behaviors than the objects seen
5

---

in the prior data since they must be grasped from the edge. Additionally, successfully placing the ring on the peg and the block in the slot requires greater precision than any of the prior tasks.
Generating prior data. Our system can use prior data from any reasonable source, including demonstrations and prior RL runs, but for simplicity and consistency we employ a simple scripted policy to collect to collect data for placing a wide variety of objects into containers (see Fig. 2a). This provides a largely autonomous strategy to collect mediocre data for many objects. While this scripted policy fails very frequently, offline RL can make use of such mediocre data effectively to pretrain value functions and policies for downstream fine-tuning. We provide further details of the data we generated in the Appendix.
Providing demonstrations. In the transfer with demonstrations setting, augment the scripted data with around 40 demonstrations of the target task collected by teleoperation with a 3DConnexion SpaceMouse Compact. Rewards specified by query at the end of each trajectory. This system allows specification of new tasks with no additional engineering effort.
Online training setup. Online training includes infrequent resets, to address the case where ob- ject becomes stuck in a hard-to-reach (or unreachable) part of the workspace, but otherwise requires the robot to train autonomously. The resets are provided every 80 trials in simulation (about every hour), and roughly every 20-30 trials (about every 20-30 minutes) in the real world.
5 Experimental Results
Our experiments aim to evaluate the hypothesis that leveraging prior data from other tasks can simultaneously address multiple challenges in autonomous real-world robotic RL. To this end, we evaluate ARIEL on seven real-world downstream tasks and a suite of simulated robotic manipulation tasks. Our goal to answer the following questions: (1) Does leveraging prior data enable mostly autonomous learning of both in-distribution and out-of-distributin
tasks in the real-world? (2) Does (3) How does ARIEL compare to prior methods for learning with minimal resets that do not use prior data? (4) Does leveraging prior data via ARTEL lead to gains in sample-efficiency for new task? Videos of the experiments can be found on the project website: https: / //sttes.//eeee/e/ googee..oooo...oooooooooo..o.... wooggle..comweee www/wwwwwwww//ee//ee aaraarrereeeeeeeeee
ARIEL produce policies that better generalize to new conditions?
5.1 Evaluating ARIEL in the Real World
We first evaluate the performance of ARIEL when training online to solve new instances of the pick and place tasks in the real world. After pretraining on the datasets discussed in Section 4, we evaluate ARIEL on both in-distribution and out-of-distribution tasks using the direct transfer and transfer with demonstrations settings, respectively. We evaluate success rates of the policy learned in the offline phase, the policy learned after 100 trials (1.5 hours) of online training, and the final policy after 600 trials (9 hours).
Autonomous fine-tuning with infrequent resets with ARIEL significantly improves real-world performance. Table 1 shows the success rate of the forward controller for both the in-distribution and out-of-distribution tasks increases moderately after 100 trials (c.g., from 2/10 to 4/10 on put tiger in drawer) and substantially by the end of online training (c.g., from 4/10 to 9/10). This indicates that ARIEL successfully optimizes the embeddings and policy parameters, with minimal resets, to perform the new task. Similar results for the performance of the backward controller are included in the Appendix. Notably, fine-tuning from only the small set of demonstrations (for the out-of-distributt
tasks) largely fails, meaning that the agent is successfully transferring skills from the other tasks in the prior data. Additionally, the autonomous fine-tuning phase does not simply robustify the previously seen behaviors, but can also lead to the emergence of new behaviors. In the insert ring onto peg task, the fine-tuned reset policy also learned useful retrying behaviors, such as shifting the ring to a more favorable grasping position so that the peg would not obstruct the robot from taking off the ring, a behavior it did not exhibit from offline training alone. This is best seen in the accompanying video.
Training on diverse prior data improves zero-shot generalization in the real world. Next, we investigate whether initializing from prior data provides ARIEL with more robust representations that generalize more effectively to new conditions without any further adaptation. To study this hypothesis, we additionally evaluate the fine-tuned policies for the liger on lid, ring on peg, and cup on saucer tasks in setting where we swap out the object (but keep the same container). We do not expect the policies to succeed consistently in this case, but we see in Table that ARIEL policies exhibit some generalization, attaining an average success rate of 54 % at the best epoch. A baseline

---

Task
Offline Only
100 Trials
600 Trials
Only Demos
Put Tiger Drawer
2/10
4/10
9/10
N/A
Direct Transfer
Put Elephant in Pot
1/10
2/10
7/10
N/A
Put Tiger on Lid
0/10
4/10
7/10
N/A
Put Cup on Saucer
7/10
9/10
0/10
Transfer with Demos
Insert Ring onto Peg
4/10
10/10
0/10
Put Block in Slot
5/10
8/10
0/10
Move Pot to Stove
2/10
9/10
2/10
Table 1: Real-world evaluation of the forward controller. We evaluate the success rates of the forward controller over the course of online fine-tuning, The first tasks are in-distribution and use the direct transfer setting. The next 4 tasks are out-of-distribution and use the transfer with demonstrations setting. For tasks in the transfer with demonstrations setting, we also evaluate a policy trained only the demonstrations. Success rates are calculated using 10 trials where each trial starts with different initial object positions.
ARIEL by # Trials
Target Data Only
Target Task
Unseen Task
100
360
600
Best Epoch
Put Tiger on Lid Put Monkey on Lid
5/10
6/10
5/10
6/10
Put Tiger Lid Put Tiger on Lid Put Cup on Saucer Insert Ring on Peg
Put Rabbit on Lid
2/10
4/10
0/10
0/10
Put Hippo on Lid Put Orange Pot on Saucer
3/10
5/10
0/10
0/10
6/10
0/10
Insert Black Ring on Peg
6/10
0/10
Table 2: Testing zero-shot generalization. We evaluate the zero-shot generalization performance of policies fine-tuned with ARIEL. We take the policies obtained at different stages of online fine- tuning on the liger on lid, cup on saucer, and ring on peg target tasks and evaluate them zero-shot on unseen tasks with interchanged objects. We compare with baseline trained on only the data collected for the target tasks, without first training on the prior data (Target Data Only) and find that policies initialized with diverse prior data (ARIEL) generalize better unseen tasks. We show success rates calculated using 10 trials where each trial starts with different initial object positions.
that uses only the data collected for each target task (i.e,, either only tiger on lid, ring on peg, or cup saucer) only succeeds in 12% of trials on the new objects. This suggests that pretraining on multi-task prior data does effectively boost generalization.
Interestingly, we observe that while fine-tuning improves the performance of ARIEL policies on the target tasks (see Table 1), fine-tuning on the target tasks adversely affects the zero-shot generaliza- tion performance on a new object. This is to be expected, since even though our online training procedure replays the prior data, we train with an increasing proportion of online data as online training goes on, making the policy more specialized to the target task. This observation is also con- sistent with prior RL works [37, 38] that note drop in performance on test tasks as more updates are made on the training tasks. We anticipate that more principled early stopping or replay of the prior data could alleviate this issue, which an interesting avenue for future work.
5.2 Evaluating ARIEL in Simulation
Lastly, we briefly present simulated experiments to compare ARIEL to prior autonomous learning approaches and ablations. Further details and analysis can be found in the Appendix.
ARIEL outperforms prior approaches for autonomous learning, We compare R3L [6]. which alternates between training forward controller to optimize a task-completion reward and training a perturbation controller that optimizes a novelty exploration bonus. This method has been proposed specifically to handle reset-free learning for real-world RL problems. We also compare to an ab- lation (Multi-task RL) that does not optimize the task embeddings z and Zù over the course of autonomous online fine-tuning, but instead fine-tunes the policy using fixed task index (that was unused during pre-training). Finally, we compare to an "oracle" version of our approach, labeled ARIEL + resets, which assumes access to external resets at the end of each episode. Figure 3 shows that ARIEL and its oracle reset variant are the only methods that succeed at learning the new tasks, and the full ARIEL method closely matches the oracle.
7

---

PuA us Cup in Tray
Put Vase Tray
Put Gluss Bocile tray
S20005 hate
Success kate
Timestups
Timestoos
Timesops
ANEL lours)
AREL "maata corecte)
RJL G2T 2020)
Mult-tan
Figure 3: Limited Resets. Simulated experiments comparing our method to prior works in reset- free and multi-task learning. Ours is the only method that learns in this limited-reset setting. Results are averaged over 4 random seeds.
Success
2-6 300
25. 1066 200 L/0
158
30%
25
100
BOS IoE LCE t01
204 3i
O6 1866 250 2/0 212 1854
ARIEL
Cracie
AWAC
Figure 4: Sample Efficiency. Simulated experiments comparing our method to (1) an oracle with offline data for the downstream task, and (2) learning without any prior data. Unlike in Figure 3, these experiments had resets at the end of every episode. Our method adapts almost as quickly as the oracle approach, whereas RL from scratch fails to make any progress. Results are averaged over random seeds.
ARIEL enables sample-eflicient learning of the new task. Next, we compare our method to simply applying RL to the target task, using the same RL algorithm as our system (AWAC (no prior data)), as well as an oracle method which is trained offline on data coming from only the single new task of interest and then trained online on the same task. Perhaps unsurprisingly, we found that learning from scratch fails in the challenging setting of learming from images and sparse rewards. However, our method learns just as efficiently as the oracle despite only having access to offline data for other tasks and not the new task of interest. Thus, our method demonstrates that properly leveraging prior data can improve sample-effiienncy even when data is only available for other tasks.
6 Discussion and Limitations
In this work, we proposed overcoming the numerous challenges of real-world robotic reinforce- ment learning through better leveraging prior datasets. While reinforcement learning often involves acquiring skill from scratch, such formalism is not well-suited to real world conditions where running online data collection expensive, and previously collected datasets are often accessible. While the use of previously collected datasets for tackling particular challenge (such as that of sample efficiency [1], or better generalization [39]) has been addressed in prior work, our work aims to address such challenges jointly in a single robotic learning system, which leverages prior data as the primary mechanism to facilitate autonomy during online training, improve efficiency, and boost generalization. In our experiments, we show that our method is able to make consistent progress in online inn-ttnnig despite having to learn from sparse rewards, matching or exceeding the final performance of an oracle baseline that receives prior data of the actual test task, and significantly outperforming a baseline without prior data. This supports our central hypothesis that prior data can be a significant facilitator for real-world robotic RL.
Limitations. The primary limitation of our approach that our online target tasks are structurally quite similar to the prior data used for offline pretraining (particularly in the direct transfer setting). While the objects seen during online training are different, they do not require significantly distinct physical skills. We believe that increasing the diversity and breadth of the prior data will make the offline pretrained model useful for a broader range of new tasks. Exploring this direction by scaling up the proposed approach is an exciting avenue for future work.
8

---

Acknowledgments
This research was supported by the Office of Naval Research, NSF CAREER IIS- 165184,, and ARO 191122-1111 with computing support from Google.
References
[1] A. Nair, M. Dalal, A. Gupta, and Levine. Accelerating online reinforcement learning with offline datasets, arXiv preprint arXiv:2006.0900 09359, 2020.
[2] D. Kalashnikov, J. Varley, Y. Chebotar, B. Swanson, R. Jonschkowski, C. Finn, S. Levine, and
K. Hausman. Mt-opt: Continuous multi-task robotic reinforcement learning at scale. arXiv preprint 104Kv::2044082/22 2021.
[3] K. Hausman, J. T. Springenberg, Z. Wang, N. Heess, and M. Riedmiller. Learning an embed- ding space for transferable robot skills. In International Conference Learning Representa- tions, 2018.
[4] W. Han, S. Levine, and P. Abbeel. Learning compound multi-step controllers under unknown dynamics. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 6435-6442. IEEE, 2015.
[5] B. Eysenbach, S. Gu, Ibarz, and S. Levine. Leave no trace: Learning reset for safe and au- tonomous reinforcement learning. In International Conference on Learning Representations, 2018. URL https: /ttp:://o/eeeiieeeeeeee net::forummid==Ivvu bCW.
[6] H. Zhu, Yu, A. Gupta, D. Shah, K. Hartikainen, A. Singh, Kumar, and Levine. The ingredients of real-world robotic reinforcement learning. In ICLR, 2020.
[7] K. Xu, S. Verma, C. Finn, and Levine. Continual learning of control primitives: Skill discovery via reset-games. In NeurlPS, 2020.
[8] A. Sharma, A. Gupta, S. Levine, K. Hausman, and C. Finn. Autonomous reinforcement learn- ing via subgoal curricula. Advances in Neural Information Processing Systems, 34, 2021.
[9] J. Peters and S. Schaal. Reinforcement learning by reward- weighted regression for operational space control. Proceedings of the 24th international conference on Machine learning, pages 745-750, 2007.
[10] D. Kalashnikow, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for vision- based robotic manipulation. arXiv preprint arXiv: /806. 10293, 2018
[11] Y. Wu, G. Tucker, and O. Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv: 1911. 1361, 2019.
[12] Fujimoto, D. Meger, and D. Precup. Off-policy deep reinforcement learning without explo- ration. In International Conference on Machine Learning, pages 2052-2062. PMLR, 2019.
[13] A. Kumar, Fu, G. Tucker, and S. Levine. Stabilizing off-policy q-learning via bootstrapping enror reduction. In NeurlPS, 2020.
[14] X. B. Peng, A. Kumar, G. Zhang, and S. Levine. wevintage-weighhee regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.000/77, 2019.
[15] Z. Wang. A. Novikov, K. Zolna, J. T. Springenberg, S. Reed, B. Shahriari, N. Siegel, J. Merel, C. Gulcehre, N. Heess, et al. Critic regularized regression. In NewrIPS, 2020.
[16] A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforce- ment learning. In NeurlPS, 2020.
[17] Fujimoto and S. S. Gu. A minimalist approach to offline reinforcement learning. NeurIPS, 2021.

---

[18] I. Kostrikov, R. Fergus, J. Tompson, and O. Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pages 5774-5783. PMLR, 2021.
[19] I. Kostrikov, A. Nair, and S. Levine. Offline reinforcement learning with implicit q-learning. In ICLR 2022, 2021.
[20] A. Wilson, Fern, S. Ray, and P. Tadepalli. Multi-task reinforcement learning: a hierarchical bayesian approach. In Proceedings of the 24th international conference on Machine learning, pages 1015-1022, 2007.
[21] E. Parisotto, J. Ba, and R. Salakhutdinov. Actor-mimic: Deep multitask and transfer reinforce- ment learning. In ICLR, 2016.
[22] Y. W. Teh, V. Bapst, W. M. Czamecki, J. Quan, J. Kirkpatrick, R. Hadsell, N. Heess, and R. Pascanu. Distral: Robust multitask reinforcement learning. NeurIPS, 2017.
[23] L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, I. Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. International Conference on Machine Learning, pages 1407- 1416. PMLR, 2018
[24] M. Hessel, H. Soyer, L. Espeholt, W. Czarnecki, S. Schmitt, and H. van Hasselt. Multi-task deep reinforcement learning with popart. Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3796-3803, 2019.
[25] T. Yu, S. Kumar, A. Gupta, S. Levine, K. Hausman, and C. Finn. Gradient surgery for multi- task learning. In NeurlPS, 2020.
[26] Z. Xu, K. Wu, Z. Che, Tang, and Ye. Knowledge transfer in multi-task deep reinforcement learning for continuous control. In NeurIPS, 2020.
[27] R. Yang, H. Xu, Y. Wu, and X. Wang. Multi-task reinforcement learning with soft modular- ization. arXiv preprint arXiv:2003. 13661, 2020.
[28] S. Sodhani, A. Zhang, and J. Pineau. Multi-task reinforcement learning with context-based representations. In ICML., 2021.
[29] T. Yu, A. Kumar, Y. Chebotar, K. Hausman, S. Levine, and C. Finn. Conservative data shar- ing for multi-task offline reinforcement learning. Advances in Neural Information Processing Systems, 34, 2021.
[30] A. Sharma, K. Xu, N. Sardana, A. Gupta, K. Hausman, S. Levine, and C. Finn. Autonomous reinforcement learning: Formalism and benchmarking. ICLR 2022, 2021.
[31] A. Gupta, J. Yu, Z. Zhao, V. Kumar, A. Rovinsky, K. Xu, T. Devlin, and S. Levine. Reset-free reinforcement learning via multi-task learning: Learning dexterous manipulation behaviors without human intervention. ICRA, 2021.
[32] K. Lu, A. Grover, Abbeel, and Mordatch. Reset-free lifelong learning with skill-space planning. In 2021.
[33] S. Ha, P. Xu, Z. Tan, S. Levine, and J. Tan. Learning to walk in the real world with minimal human effort. In CoRL, 2020.
[34] A. Gupta, C. Lynch, B. Kinman, G. Peake, Levine, and K. Hausman. Bootstrapped au- tonomous practicing via multi-task reinforcement learning, arXiv preprint arXiv:2203.15755, 2022.
[35] K. Rakelly, A. Zhou, Finn, Levine, and D. Quillen. Efficient off-policy meta- reinforcement learning via probabilistic context variables. In International conference on ma- chine learning, pages 5331-5340. PMLR, 2019.
[36] R. Rubinstein. The cross-entropy method for combinatorial and continuous optimization. Methodology and computing in applied probubility, 1(2):127-190, 1999.
10

---

[37] K. Cobbe, C. Hesse, J. Hilton, and J. Schulman. Leveraging procedural generation to bench- mark reinforcement learning. In International conference on machine learning, pages 2048- 2056. PMLR, 2020.
[38] D. Ghosh, J. Rahme, A. Kumar, A. Zhang, R. P. Adams, and S. Levine. Why generalization in rl is difficult: Epistemic pomdps and implicit partial observability. Advances in Neural Information Processing Systems, 34, 2021.
[39] Y. Lin, A. Zeng, S. Song, P. Isola, and T. Lin. Learning to see before learning to act: Visual pre-training for manipulation. CoRR, 0ass2107.0006//4 2021.


Reviews:
[{"review_id": "v6t7t3u9HVA", "reviewer": "Reviewer_YCu7", "summary": "Pre-training with large offline data and fine-tuning on a small amount of online data is a sample-efficient way of learning a new task. However, most recent approaches require explicit environment resetting for online data collection. This work proposes to minimize manual resetting of the environment when learning a new task in the real world leveraging offline multi-task data. First, from the offline multi-task data, the proposed method, ARIEL, learns a multi-task policy conditioned on a task embedding $\\mathbf{z}$ using offline RL. Then, during the online fine-tuning phase, ARIEL searches for a task embedding for the forward task as well as another task embedding for the backward task (environment reset). ARIEL iterates the forward and backward tasks, which ideally resets the environment and performs the target task without human intervention. The real-world experiments show that ARIEL achieves more than 70% success rates with only 5 to 30 manual resets.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strenghts\n\n* The idea of utilizing a multi-task policy for both forward and backward (reset) policies makes sense and the paper empirically shows its effectiveness.\n\n* The real-world experiments are exhaustively conducted with many different objects and in three different environment setups.\n\n### Weaknesses\n\n* In RL, covering diverse initial states is important for robustness and generalization of the learned policy. However, without explicit randomized resetting, the proposed backward controller may learn to reset to a few (easy to reset) initial states, which maximizes its own reward. It would be great if the authors discuss this potential issue.\n\n* The proposed alternating scheme is appealing in automating online reinforcement learning with less human intervention. However, one of the critical assumptions in this work is that both forward and backward controllers can be achieved with a few random shootings. The reviewer is wondering whether this is a practical assumption for robot learning.\n\n* To show the sufficient difference between training tasks and testing tasks, another interesting point could be freezing the policy after pre-training and only finding appropriate $\\mathbf{z}$ using CEM. If this works as well as the proposed method, it could mean the prior data is too similar to the testing tasks, which means the experimental setup might be too easy.\n\n* The proposed method assumes a task embedding corresponds to a task. However, in many cases, the offline data trajectories can be compositional (combination of multiple behaviors) and the target task may require stitching fo multiple of these trajectories. The proposed method does not seem trivial to scale to this common scenario.\n\n### Questions and suggestions\n\n* Figure 2b is too small, compact, and not descriptive without enough explanation. It would be better to include only two examples (one from in-distribution and another one from out-of-distribution tasks) and provide sufficient descriptions.\n\n* L242-243: placing the ring on the peg does not seem to require \"greater precision\" than the prior tasks.\n\n* In L244-245, while the paper claims that the proposed method can utilize data from any reasonable source, the proposed method requires reward annotation, which is not available in reward-free demonstrations (e.g. play data, multi-task expert demonstrations). Given that the proposed method can potentially work with sparsely-labeled data using offline RL, it may not be a big issue, but it should be clarified.\n\n* In L253, what does it mean that rewards are specified at the end of the trajectory?. The reviewer thinks rewards for the target demonstrations should be all 1 at the end. Or, does it mean that the reward is labeled depending on the forward or backward tasks?\n\n* What is the definition of \"trial\" in Section 5 and 6? Does this mean $M$ steps or $2M$ steps?\n\n* In Table 1, why do in-distribution tasks perform worse than out-of-distribution tasks?\n\n* In Table 1, including a baseline with episodic reset as an upper bound can be helpful to understand how much \"reset\" information is important for learning these tasks. \n\n* In Table 1 and Table 2, which experiment uses 40 downstream task demonstrations? This is not clear.\n\n* The explanation in L291-L293 about the experiments for Table 2 is a bit confusing. Are the testing tasks unseen during online fine-tuning? More specifically, \"Put X in Container\" tasks are evaluated using a policy pre-trained from multi-task prior data and fine-tuned for \"Put Tiger on Lid\"? \n\n* In Table 2, how many trials were used for training \"Single Task Data Only\"?\n\n* In A.2.1, there are three different environments, \"Tray Container\", \"Tray Drawer\", and \"Kitchen\". For offline training of ARIEL, are data from all three environments used together? or data from the same environment with the target task is used for offline training?\n\n* In Algorithm 1, \"steps\" should be reset to 0 and new $\\mathbf{z}$ should be sampled, like L9, when the task direction $d$ is reversed in L14. \n\n* In Algorithm 1, L2, what does it mean by \"fit $q_f(\\mathbf{z})$ and $q_b(\\mathbf{z})$ to offline task indices?\n\n* In Algorithm 1, L5, $d$ should be also reset to $f$ as the environment state is reset. Simply moving L3 to L5 can work.\n\n* How much randomness in object poses and robot poses for each trial of evaluation in Table 1 and 2?\n\n* In Table 1, how is \"Offline Only\" evaluated? More specifically, which $\\mathbf{z}$ is used for this experiment? And, how can out-of-distribution tasks have such high success rates only with offline only?\n\n* A bit more detailed description of the task indices $\\mathbf{z}$ can be helpful to understand the proposed method. The paper mentions that $\\mathbf{z}$ is a task embedding from Hausman et al. in L159, but more details about how it is trained can be elaborated in the paper.\n\n* It could be interesting to see how success rates of the forward and backward controllers over the online training. \n\n* Many references are from arXiv. Please cite their conference or journal versions.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper proposes a (semi) autonomous reinforcement learning leveraging offline multi-task data, and the empirical results demonstrate its effectiveness and autonomy (less than 30 manual resets required). Although the experimental tasks look a bit simple and the presentation should be improved, the proposed idea is novel and the results show some potential for autonomous RL.\n\n\n**After rebuttal**\n\nThank the authors for the response. The rebuttal clarified all my confusion, but the weaknesses are still there and a more in-depth discussion about the reviewers' concerns would make the paper stronger. Overall, I believe the proposed method shows an interesting research direction and empirically demonstrated its potential in real-world scenarios; thus, I would like to accept this paper.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "Please refer to the above weaknesses and questions.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}, {"review_id": "VgshAi75N-F", "reviewer": "Reviewer_CCLn", "summary": "The paper proposes to use a multi-task RL policy, trained offline with (i) data (state-action-state-reward tuples) for a collection of tasks, as a starting point to fine-tune, online, 2 new policies for a new, similar task (one policy to solve the task itself and one policy to autonomously reset the environment).\nIn case the new task is not sufficiently similar to the tasks for which the multi-task policy was pre-trained, (ii) additional data from operator-guided demonstrations are also used.\nThe approach is demonstrated both in real-world experiments and simulation for 6 pick-and-place tasks with a robotic manipulator.\nThe results in real-world experiments show how online fine-tuning improves the success rate of the multi-task policy on new tasks.\nThe paper interestingly brings together 3 ideas (offline learning, multi-task transfer learning, and learned automated reset policies) into a single approach.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper makes simultaneous use of three very interesting and promising methods (offline learning, multi-task transfer learning, and learned automated reset policies), and treats the one-hot task representation of the multi-agent policy as a continuous, learnable parameter that also represents the task embedding of new tasks (and the reset policy), although not entirely new, is a neat approach to address the paper's problem.\nBesides the marginal/incremental novelty, one possible weakness of the paper is that it seems that the new tasks need to be such (similar enough) that the old multi-task policy can already achieve success. The role of human demonstrations (initially presented as optional, then used for so-called out-of-distribution new tasks) should be better introduced and more formally justified.\nThe way data and results are presented in the tables should be improved for clarity and stand-alone readability", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper is easy to read, enjoyable, and summarizes very interesting ideas for real-world robotics RL. My main concerns are the lack of novelty and the not overly clear presentation of the real-world experimental results.\n", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "- I would recommend being very clear from early in the paper (e.g. Fig 1) and what is the motivation for the use of human demos and how they play into the learning pipeline, for what tasks, etc. (the explanation comes into multiple parts in sections 4 and 5)\n- If extra space is needed (for clarification, additional results), Section 3 is very textbook-like and could be removed\n- I would add a clarifying statement about frequent vs unfrequent and automated vs manual resets\n- Provide some numerical figure when saying that the scripted policy fails \"very frequently\"\n- Tables 1 and 2 should be improved, e.g. clarifying in the captions what the ratios stand for, separating in and out of distributions tasks, etc.\n- line 300, should it be 360 instead of 600? what was the \"offline only\" zero-shot generalization performance?"}}, {"review_id": "HZXlAlHK8Mt", "reviewer": "Reviewer_rVQq", "summary": "Real-world robotic reinforcement learning (RL) typically requires time-consuming data collection and frequent human intervention to reset the environment. The offline-RL approach provides a way to utilize previously collected data to bootstrap the learning process so that a new task can be learned with a small number of interactions. In this work, the authors study a problem where such offline data is available from several tasks (along with their task labels). The data may be collected with optimal or sub-optimal policies.  The authors propose a complete system for extracting useful skills from prior data and applying them to learn new tasks autonomously. When faced with a new task, the proposed system adapts previously learned skills to quickly learn to both perform the new task and return the environment to an initial state, effectively performing its own environment reset. The authors show the effectiveness of their system with extensive physical and simulated experiments.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n- The paper addresses an important problem of data efficiency and reset-free learning for real robotic applications.\n- The paper clearly mentions the prior works on which the proposed system is based.\n- Extensive experimentation to assess the effectiveness of the system both in simulated and physical environments.\n\nWeakness:\n\nThe main weakness of the paper is that it is a bit difficult to see what the actual contribution of the paper is and how different the approach is compared to the previously published work. Several prior works already demonstrated how online RL can be accelerated with prior offline data. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "in my opinion, the work is not significantly different from the prior work. However, the paper focuses on an important problem and demonstrated the effectiveness of their proposed approach through experimentation.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "Maybe the contribution of the paper could have been clearly mentioned instead of \"demonstrating already known\" facts as a contribution. For instance, we already know using prior data makes RL samples efficient, there is nothing to demonstrate here again in my opinion.\n\nLine 65: \"we adopt the strategy learning a single policy conditional on a space of tasks\" => we adopt the strategy *of* learning a single policy condition*ed* on a space of tasks.\n\nThere are several citations that are now published. But the arxiv versions are still cited on the paper. Please use the published versions. For instance, [31] should be ICRA 2021. \n\nline 246: Typo here \"...policy to collect to collect data for....\""}}, {"review_id": "9H_VOHQlzEFz", "reviewer": "Reviewer_BwFL", "summary": "The paper discusses a system to leverage prior data from related but distinct tasks to increase learning efficiency and generalization in new task. The system works in two steps: 1. Using offline data with consisting of different tasks with task ids to learn a task-conditional policy using offline RL. 2. Optimize task-conditioning embedding during deployment for the new task. To reduce the human supervision involved in resets the method proposes to extract two different embeddings one that accomplishes the tasks and the other that resets the task. The experiments are shown in both real world robot and simulation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The paper presents a relatively novel combination of existing known modules for learning with offline data and learning reset policies.\n2. The paper evaluates the results on real world and simulated domains and provide comprehensive results to show the validity of the method. The experiments demonstrate the superior generalization ability of the system arising as a result of learning from multiple tasks and increased learning efficiency.\n\nWeaknesses:\n\n1. Limited baselines: The paper evaluates ARIEL on the real world tasks but does not present a comparison with any baseline. Potential baselines to be compared to are: [2,29,31,32,33](references from paper) and common RL baselines which are more suited to sparse rewards like HER[1,2,3] (references below).\n2. The experiments use a short horizon of M=20 which is small for a number of locomotion and navigation tasks. It is not clear how this method would scale with horizon since the start-state distribution for learning the reset policy can grow exponentially.\n\n[1] :Hindsight Experience Replay [Marcin Andrychowicz] et al\n\n[2]: [Replacing rewards with examples: Example-based policy search via recursive classification],Eysenbach et al\n\n[3] [C-Learning: Learning to Achieve Goals via Recursive Classification], Eysenbach et al", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper presents a novel combination of existing modules for autonomous robot learning and demonstrates that near-autonomous learning is possible with similar tasks in the short-horizon manipulation domains. The paper can be strengthened by more extensive comparisons to baselines to properly situate how their work progresses the field.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "1. Limited baselines: Can the authors present baselines for their real-world and simulated robot experiments. If the comparisons are not feasible, can the authors describe in the paper what makes such a comparison not possible.\n2. Technical novelty: The work seems to borrow heavily from previous works [8,31] (references from paper) on reset free learning and offline multitask learnings. Can the authors comment on the significance of the new proposed methods in comparison to previous work?"}}]

Author rebuttal:

Decision: Accept (poster)