Self-Improving Robots: End-to-End Autonomous Visuomotor Reinforcement Learning
Authors: Archit Sharma
Ahmed M Ahmed
Rehaan Ahmad
Chelsea Finn
Venue: CoRL 2023 Poster (2023)
Keywords: reinforcement learning
autonomous
reset-free
manipulation

Abstract: In imitation and reinforcement learning (RL), the cost of human supervision limits the amount of data that the robots can be trained on. While RL offers a framework for building self-improving robots that can learn via trial-and-error autonomously, practical realizations end up requiring extensive human supervision for reward function design and repeated resetting of the environment between episodes of interactions. In this work, we propose MEDAL++, a novel design for self-improving robotic systems: given a small set of expert demonstrations at the start, the robot autonomously practices the task by learning to both do and undo the task, simultaneously inferring the reward function from the demonstrations. The policy and reward function are learned end-to-end from high-dimensional visual inputs, bypassing the need for explicit state estimation or task-specific pre-training for visual encoders used in prior work. We first evaluate our proposed system on a simulated non-episodic benchmark EARL, finding that MEDAL++ is both more data efficient and gets up to 30% better final performance compared to state-of-the-art vision-based methods. Our real-robot experiments show that MEDAL++ can be applied to manipulation problems in larger environments than those considered in prior work, and autonomous self-improvement can improve the success rate by 30% to 70% over behavioral cloning on just the expert data.

Self-Improving Robots: End-to-End Autonomous Visuomotor Reinforcement Learning
Archit Sharma" Ahmed M. Ahmed
Rehaan Ahmad Chelsea Finn
Stanford University
Abstract: imitation and reinforcement learning (RL), the cost of human super- vision limits the amount of data that the robots can be trained on. While RL offers framework for building self-improving robots that can learn via trial-and-error autonomously, practical realizations end up requiring extensive human supervi- sion for reward function design and repeated resetting of the environment between episodes of interactions. In this work, we propose MEDAL++, novel design for self-improving robotic systems: given a small set of expert demonstrations at the start, the robot autonomously practices the task by learning to both do and undo the task, simultaneously inferring the reward function from the demonstrations. The policy and reward function are learned end-to-end from nigh-dimensional visual inputs, bypassing the need for explicit state estimation task-specific pre- training for visual encoders used in prior work. We first evaluate our proposed system simulated non-episodic benchmark, EARL, finding that MEDAL++ is both more data efficient and gets up to 30% better final performance compared to sfate-of-the-art usion-based methods. Our real-robot experiments show that MEDAL++ can applied to manipulation problems larger environments than those considered in prior work, and autonomous self-imprrvement can improve the success rate by 30% to 70% over behavioral cloning just the expert data. Code, training and evaluation videos along with brief overview is available at: www.haaboooeffffimroiiiiiii
Keywords: reinforcement learning, autonomous, reset-free, manipulation
Introduction
While imitation learning methods have shown promis- ing evidence for generalization via large-scale teleoper- ated data collection efforts [1, 2], human supervision is expensive and collected datasets are still incommensu- rate for learning robust and broadly performant control. In this context, the aspirational notion of self-improving robots becomes relevant: robots that learn and im- prove from their own interactions with the environment autonomously. Reinforcement learning (RL) is natural framework for such self-improvement, where the robots can learn from trial-and-error autonomously in principle. However, RI. deployment requires domain expertise and extensive supervision in practice for state estimation, de- signing reward functions, and repeated resetting of the en- vironments after every episode of interaction.
Robut macts Be envincement tell
Robet by thuf
Goul
Initid
Cancst roleeio
regalies crleester Rannie seperssiie
Human rests the eusioonment Sor robrooot
Figure 1: from goal state in contrast to a human resetting the environ- ment for the robot (bottom). While latter is the norm in robotic reinforcement learning, rohot that can reset the environment and practice the task autonomously can train on more data, and thus, be more competent.
A robot resets the environment
initial state (top),
In particular, the human supervision for repeatedly reset- ting the environment through training is an impediment to
Authors contributed substantially real-robot results. Correspondence
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

building autonomous robots, visualized in Figure 1. Several prior works 43,4,3 5] show that standard RL algorithms can fail catastrophically when the reset frequency is decreased. While recent works address the lack of supervision for repeated resetting [4, 6, 7, 8, 9], learning efficiently in the absence of frequent environment resets remains a challenge in the real-world. Task-relevant states constitute only small subset of all possible states, especially in larger real-world environments, and the robot can drift far from these task-relevant states when learning without repeated resets. In view of this, using a small set of demonstrations can be an effective choice to construct self-improving systems. Expert demonstrations can alleviate challenges related to exploration [10] and enable efficient au- tonomous RL by encouraging agent to stay close to task-relevant states in the demonstrations [11]. And since the human supervision required for collecting the demonstrations is front-loaded, i.e., before the training begins, the robot can collect data autonomously and self-improve thereon. Im- portantly, we make the observation that the terminal states in expert trajectories effectively represent the goal distribution. This allows us learn goal-reaching reward function without any additional data collection reward engineering.
In this work, our main contribution is MEDAL++, a carefully designed system that can train au- tonomously in the real-world with minimal ssecifpeciiic engineering. MEDAL++ builds upon [11] with several crucial components that enable efficient end-to-end autonomous training in the real- world: First, we learn an encoder for nigh-dimensional visual inputs end-to-end using DrQ-v2 [12], bypassing the need for state estimation or task-specific pre-training of visual encoders. Second, we judiciously use expert demonstrations by learning a goal-reaching reward function [13, 14), elimi- nating the need for engineered reward functions. Finally, we improve the sample efficiency by using an ensemble of Q-value functions and increasing the update steps per sample collected [15], using BC regularization on expert data to regularize policy learning towards demonstration data [16], and oversampling transitions from demonstration data when training the Q-value function [10]. We eval- uate MEDAL++ on a pixel-based control version of EARL [5], a non-episodic learning benchmark and observe that MEDAL+ + is more data efficient and gets up to 30% better performance compared to competitive methods [4, 11]. Most importantly. we conduct real-robot evaluations using a Franka Panda robot arm on four manipulation tasks, such as hanging a cloth on hook, covering bowl with a cloth and peg insertion, all from RGB image observations. We observe that MEDAL++ can improve the success rate of the policy by 30% to 70% over a behavioral cloning policy learned only on the expert data, suggesting that MEDAL++ is a step towards self-improving robotic systems.
2 Related Work
Several works have demonstrated the emergence of complex skills on a variety of problems using reinforcement learning on real robots [17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27], However, these prior works require the environment to be reset to a (narrow) set of initial states for every episode of interaction with the environment. Such resetting of the environment either requires repeated human interventions and constant monitoring [28, 29, 30, 31, 32] or scripting behaviors [19, 23, 33, 34, 24, 35] which can be time-intensive while resulting in brittle behaviors. Some prior works have also designed the task and environment to bypass the need for resetting the environment [36, 37, 21J, but this applies to a restricted set of tasks.
More recent works have identified the need for algorithms that can work autonomously with min- imal supervision required for resetting the environments [38, 4, 5]. Several recent works propose to learn a backward policy undo the task, in addition learning a forward policy that does the task [38, 6, 7, 8, 39, 40]. In this work, we build upon MEDAL [11], where the backward policy learns to match the distribution of states visited by an expert to solve the task. While the results from these prior papers are restricted to simulated settings, some recent papers have demonstrated autonomous training on real robots [4, 41, 42, 43]. However, the results on real robots have either re- lied on state estimation [41, 42], pre-specified reward functions [43] or tpsk-specific decomposition into subgoals [44]. R3L [4] also considers the setting of learning from image observations without repeated resets and specified reward functions, similar to our work. uses backward policy that optimizes for state-novelyy while learning the reward function from set of goal images collected prior to training [14]. However, R3L relies on frozen visual encoders trained independently data

---

collected in the same environment, and optimizing for state-novelty does not scale to larger environ- ments, restricting their robot evaluations to smaller, easier to explore environments. Our simulation results indicate that MEDAL++ learns more efficiently than R3L, and real robot evaluations in- dicate the MEDAL++ can be used on larger environments. Overall, our work proposes a system that can learn end-to-end from visual inputs without repeated environment resets, with real-robot evaluations four manipulation tasks.
3 Preliminaries
Problem Setting. We consider the autonomous RL. problem setting [5]. We assume that the agent in Markov Decision Process represented by (S, po, po, where S is the state space, potentially corresponding to high-dimensional observations such as RGB images, A denotes the robot's action space, T : S x A 8 + R>0 denotes the transition dynamies of the environment, : x A - R is the (unknown) reward function, Pu denotes the initial state distribution, and yy denotes the discount factor. The objective is to learn policy that maximizes E 1o== ('r(st, 4z)] when deployed from Po during evaluation. There are two key differences from the standard episodic RL setting: First, the training environment is non-episodic, i.e., the environment does not periodi- cally reset to the initial state distribution after a fixed number of steps. Second, the reward function not available during training. Instead, we assume access to a set of demonstrations collected by an expert prior robot training. Specifically, the expert collects small set of forward trajectories Dj {(s a;) . demonstrating the task and similarly, set of backward demonstrations Dī undoing the task back to the initial state distribution Po.
Autonomous Reinforcement Learning via MEDAL. To enable a robot to practice the task au- tonomously, MEDAL [11] trains a forward policy my to solve the task, and a backward policy Ty to undo the task. The forward policy 5 executes for fixed number of steps before the control switched over to the backward policy m6 for a fixed number of steps. Chaining the forward and backward policy reduces the number of interventions required reset the environment. The for- ward policy is trained to maximize E. 1E 7'r("1, at)], which can be done via any RL algorithm. The backward policy Ti is trained to minimize the Jensen-Shanne between the stationary dittrdiistiiuttinn of the backward policy p² and the state-distribution of the expert policy p" By training a classifier C2 S - (0, 1) to discriminate between states visited by the expert (i.e. s ~ p²) and states visited by ne (i.e., 8 ~ p²), the divergence minimization problem can be rewritten as max,, -E 7 log (1 - (st))) [11]. The classifier used in the reward function for Tú is trained using the cross-entropy loss, where the states Dj are labeled 1 and states visited by Ts online are labeled O, leading to a minimax optimization between mo and Cj.
divergence (S(p"(s) II p (s))
Learning Reward Functions with VICE. Engineering rewards can be tedious, especially when only image observations are available. Since the transitions from the training environment are not labeled with rewards, the robot needs to learn a reward function for the forward policy Ty. In this work, we consider VICE [45], particularly, the simplified version presented by Singh et al. [14] that is compatible with off pollcy RL. VICE requires small set of states representing the desired outcome (i.e., goal images) prior to training. Given a set of goal states G, VICE trains classifier Cf S + (0, 1) where Cy is trained using the cross entropy loss on states E G labeled as 1, and states visited by T labeled as O. The policy n is trained with a reward function of log (s) - log (1 (s)), which can be viewed as minimizing the KL-divergence between the stationary state distribution of r and the goal distribution [46, 47, 48). VICE has two benefits over pre-trained frozen classifier-based rewards: first, the negative states do not need to be collected by person and second, the VICE classifier is harder to exploit as the online states are iteratively added to the label 0 set, continually improving the goal-reaching reward function implicitly.
4 MEDAL++: Practical and Efficient Autonomous Reinforcement Learning
The goal of this section is to develop RL method that can learn from autonomous online interaction in the real world, given just a (small) set of forward Dj and backward demonstrations Dj without reward labels. Particularly, we focus on design choices that make MEDAL++ viable in the real
3

---

world in contrast to MEDAL: First, we describe how to learn from visual inputs without explicit state estimation. Second, we describe how to learn a reward function from the expert demonstrations to eliminate the need for ground truth rewards when training the forward policy TT. Third, we describe the algorithmic modifications for training the Q-value function and the policy a more efficiently. Finally, we describe how to construct MEDAL++ using all the components described here, training forward policy f and a backward policy %e to learn autonomously.
Encoding Visual Inputs. We embed the high- high-dimensional RGB images into a low-dimensions feature space using a convolutional encoder& E. The RGB images are augmented using random crops and shifts (up to 4 pixels) to regularize Q-value learning [12]. While some prior works incorporate explicit representation learning losses for visual encoders [49, 50], Yarats et al. [12] suggest that reg- ularizing Q-value learning using random crop and shift augmentations is both simpler and efficient, allowing end-to-end learning without any explicit representation learning objectives. Specifically, the training loss for Q-value function on an environment transition (s, a, s',r) can be written as:
{(Q, 8) - (Q (E (aug(s)) a) -r -r--V (E(ug(()))) 2
(I)
where aug(.) denotes the augmented image, and + vV)) is the TD-target. Equation 2 describes the exact computation of V using slow-moving target networks Q and the current policy æ.
Learning the Reward Function. To train a VICE classifier Cy, we need to specify a set of goal states that can be used as positive samples. Instead of collecting a separate set of goals, we observe that the last K states of every trajectory in Dj approximate the goal distribution, and thus, can be used as the goal set G. The trajectories collected by the robot's policy y will be used generate negative states for training Cy. The policy is trained to maximize log (1 Cy(-)) the reward function, encouraging the policy to reach states that have a high probability of being labeled under Cy, and thus, similar to the states in the set G. The reward signal from the classifier can be sparse the classifier has high accuracy on distinguishing between the goal states and states visited by the policy. Since the classification problem for C2 is easier than the goal-matching problem for xy, especially early in the training when the policy not as successful, it becomes critical regularize the discriminator C2 [51]. We use spectral normalization [52], mixup [53] regularize Cj, and apply random crop and shift augmentations to input images to create a broader training distribution.
As we have access to expert demonstrations Dj, why do we match the policy's state distribution to the goal distribution, instead of GAIL [13, 54], which matches policy's 's state-action distribution to that of the expert? In a practical robotic setup, actions demonstrated by an expert during tele- operation and optimal actions for a learned neural network policy will be different. The forward pass through policy network introduces delay, especially as the visual encoder & becomes larger. Matching both the state and actions to that of the expert, as is the case with GAIL, can lead to sub- optimal policies and be infeasible in general. In contrast, VICE allows the robotic policies to choose actions that are different from the expert as long as they lead to a set of states similar to those in g. The exploratory benefits of matching the actions can be recovered, as described in the next section.
Improving the Learning Efficiency. To improve the learning efficiency over MEDAL, we incor- porate several changes in how we train the Q-value function and the policy First, train an ensemble of Q-value networks (Qn. and corresponding set of target networks (Qn (. When training an ensemble member Qn, the target is computed by sampling subset of target networks, and taking the minimum over the subset. The target value V(s') in Eq 1 can be computed as
((s')= = En'nx min (s'''aa').
(2)
JEM
where A is a random subset of the index set (1...N} of size M. Randomizing the subset of the ensemble when computing the target allows more gradient steps to be taken to update Qn on f(Q,.. 8) [15] without overfitting to a specific target value, increasing the overall sample efficiency of learning. The target networks Qn are updated as an exponential moving average of Qn in the where
weight space over the course of training. At iteration 4, Q + =Q + (1 T T E (0, 1] determines how closely Qn tracks Qn-

---

Importantly, we want to leverage the expert demonstrations to counteract the exploration challenge, especially because the training signal from VICE reward can be sparse. Q-value networks are typi- cally updated on minibatches sampled uniformly from a replay buffer D. However, the transitions in the demonstrations are generated by an expert, and thus, can be more informative about the actions for reaching successful states [10]. To bias the data towards the expert distribution, we oversample transitions from the expert data such that for batch of size B, p. transitions sampled from the expert data uniformly and (1 B) transitions are sampled from the replay buffer uniformly for p 1). Finally, we regularize the policy learning towards expert actions by introducing behavior cloning loss in addition to maximizing the Q-values [16, 10]:
((T)) (g
1 N
Qn (n(((aug(s)), m)) 1=1
+ AB (-))))))ppp log ग (a" [((augss)))
})}}
where À 2 0 denotes the hyperparameter controlling the effect of BC regularization. Note that the parameters of the encoder are frozen with respect to [(T), and are only trained through E(Qn, E).
Putting it Together: MEDAL++. MEDAL++ trains a forward policy that learns to solve the task and backward policy that learns undo the task towards the expert state distribu- tion. The parameters and data buffers for the forward policy are represented by the tuple F ET,,E, (Q (Q Cy, Gy. where the symbols retain their meaning from the previous sections. Similarly, the parameters and data buffers for the backward policy are repre- sented by the tuple B Tb, 1 1h Co2 Dj, Du, - F and B have similar structure: Both o and πe are trained using with - log((1 C()) as the reward function (using their respective classifiers C and Ct), with both classifiers trained to discriminate between the states visited by the policy and their target states. The primary difference is the set of positive target states G and G, used to train Cj and C. respectively, visualized in Figure 8. The VICE classifier Cy is trained to predict the last K states of every trajectory from Dÿ as positive, whereas we train the MEDAL classifier C. to predict all the states of forward demonstrations except the last K states positive. Optionally, we can also include the last K states of backward demonstrations from D positives for training Cs. The pseudocode for training is given in Algorithm 1, and more detailed description is available in Appendix A.1. Some key details: the control switches between the forward Ty and the backward policy Ta after a fixed number of steps. When executing in the real world, humans are allowed to intervene and reset the environment intermittently, switching the control over to y after the intervention to restart the forward-backward cycle.
Noticeably,
5 Experiments
The goal of our experiments is to determine whether MEDAL++ can be a practical method for self-improving robotic systems. We benchmark MEDAL++ against competitive methods [11, 4] on EARL [5] benchmark for non-episodic RL to evaluate the learning efficiency from nigh-dimensional observations, in Section 5.1. Our primary experiments in Section 5.2 evaluate MEDAL++ on four real robot manipulation tasks, primarily tasks with soft-body objects such as hanging cloth on hook and covering howl with cloth. The real robot evaluation considers the question of whether self-improvement is feasible via MEDAL++, and if so, how much self-improvement can MEDAL++ obtain? Finally, we run ablations to evaluate the contributions of different components to MEDAL++ in Section 5.3.
5.1 Benchmarking MEDAL++ on EARL
First, we benchmark MEDAL+ continuous-control environment from EARL against state-of- the-art non-episodic autonomous RL methods. To be consistent with the benchmark, we use the ground truth reward functions for all the environments.
Environments. We consider three sparse-reward continuous-contre environments from EARL benchmark [5], shown in Appendix, Fig 6. Tabletop organization is a simplified manipulation en- vironment where gripper tasked to move the mug one of the four coasters from wide set of initial states, sawyer door closing task requires sawyer robot arm to learn how to close door starting from various positions, and finally the sawyer peg insertion task requires the sawyer robot
5

---

Peg Insertion
Door Closing
Tabletop Orgenization
Evaluation
Evaluation
Eusluation
MEDAL
Figure 2: Comparison of autonomous RL methods on vision-based manipulation tasks in simulated environ- ments from EARL [5]. MEDAL+ is both more efficient and learns a similarly or more successful policy compared to other methods.
arm to grasp the peg and insert it into goal. Not only does the robot have to learn how to do the lask (i.e. close the door or insert the peg), but it has to learn how to undo the task (i.e. open the door or remove the peg) to try the task repeatedly in non-episodic training environment. All tasks are setup to return x 84 RGB observations with sparse goal reaching reward functions. The training environment reset to 5p mV Po every 25,000 steps of interaction with the environment. This is ex- tremely infrequent compared to episodic settings where the environment is reset to the initial state distribution every 200-1000 steps. EARL comes with 5-15 forward and backward demonstrations for every environment to help with exploration in these sparse reward environments. We report the average success of the forward policy every 10, 000 training steps over 10 trials. More details can be found in Appendix A.3.
Comparisons. We compare MEDAL++ to four methods: (1) MEDAL [11] uses a backward policy that matches the expert state distribution by minimizing Js(p"(s) p² (s)), similar to ours. How- ever, the method is designed for low-dimensional states and policy/Q-value networks and cannot be naïvely extended to RGB observations. For a better comparison, we improve the method to use visual encoder with random crop and shift augmentations during training, similar to MEDAL++. (2) R3L [4] uses a perturbation controller as backward policy which optimizes for state-novelty computed using random network distillation [55]. Unlike our method, R3L also requires a sepa- rately collected dataset of environment observations pre-train VAE [56] based visual encoder, which is frozen throughout the training thereafter. (3) We consider oracle RL method that trains just forward policy and gets privileged training environment that resets every 200 steps (i.e., the same episode length as during evaluation) and finally, (4) we consider a control method naïve RL, that similar to oracle trains just a forward policy, but resets every 25,000 steps similar to the non- episodic methods. We additionally report the performance on the forward demonstrations used in the EARL environments. The implementation details and hyperparameters can found in Appendix A.2.
behavior cloning policy, trained
Results. Figure 2 plots the evaluation performance of the forward policy versus the training sam- ples collected in the environment. MEDAL++ outperforms all other methods on both the sawyer environments, and is comparable to MEDAL on tabletop organization, the best performing method. While R3L does recover a non-trivial performance eventually on door closing and tabletop organi- zation, the novelty-seeking perturbation controller can cause the robot to drift to states farther away from the goal in larger environments, leading to slower improvement in evaluation performance on states starting from Sn Po- While MEDAL and MEDAL++ have the same objective for the backward policy, optimization related improvements enable MEDAL++ to learn faster. Note, BC performs worse on tabletop organization environment with 45% success rate, compared to the sawyer environments with a 70% and 80% success rate on peg insertion and door closing respec- tively. So, while BC regularization helps speed up efficiency and can lead to better policies, it can hurt the final performance of MEDAL++ if the BC policy itself has a worse success rate (at least, when true rewards are available for training, see ablations in Section 5.3). While we use the same hyperparameters for all environments, reducing the weight on BC regularization when BC policies have poor success reduce the bias in policy learning and improve the final performance.

---

5.2 Real Robot Evaluations
In line with the main goal of this paper, our experiments aim to evaluate whether self-improvement through MEDAL++ can enable real-robots to learn more competent policies autonomously. On four manipulation tasks, we provide quantitative and qualitative comparison of the policy learned by behavior cloning on the expert data to the one learned after self-improvement by MEDAL++. We recommend viewing the results on our anonymized website in the supplementary material for training and evaluation videos, which provides a more comprehensive overview.
Robot Setup and Tasks. We use Franka Emika Panda arm with Robotiq 2F-85 gripper for all our experiments. We use RGB camera mounted on the wrist and third person-camera, shown in Figure 3. The final obser- vation space includes two 100 100 RGB images, 3- dimensional end-effector position, orientation along the z-axis, and the width of the gripper. The action space is set up as either 4 DoF end-effector control, or 5 DoF end-effector control with orientation along the z-axis de- pending on the task (including one degree of freedom for the gripper). Our evaluation suite consists of four nipuation tasks: grasping a cube, hanging cloth hook, covering bowl with a piece of cloth, and (soft) peg insertion. Real world data and training is more perti- nent for soft-body manipulation as they are harder to sim- ulate, and thus, we emphasize those tasks in our evaluation suite. The tasks are shown in Figure
Figure 3: MEDAL++. The image observations include fixed third person view and first person view from a wrist camera mounted above the gripper. The evaluation tasks going clockwise: cube grasping, covering a bowl with cloth, hanging a cloth on the hook and, peg insertion.
The training sctup for
Training and Evaluation. For every task, we first collect set of 50 forward demonstrations and 50 backward demonstrations using Xbox controller. We chain the forward and backward demonstra- tions to speed up collection and better approximate autonomous training thereafter. After collecting the demonstrations, the robot is trained for 30 hours using MEDAL++, collecting about 300, 000 environment transitions the process. For the first 30 minutes of training, we reset the environment create enough (object) diversity in the initial data collected in the replay buffer. After the initial collection, the environment is reset intermittently approximately every hour of real world training on an average, though, it is left unattended for several hours. More details related to hyperparameters, network architecture and training can be found in Appendix A.2. For evaluation, we roll-out the policy from varying initial states, and measure the success rate over 50 evaluations. To isolate the role of self-improvement, we compare the performance to a behavior cloning policy trained on the forward demonstrations using the same network architecture for the policy as MEDAL++. For both MEDAL and BC, we evaluate multiple intermediate checkpoints and report the success rate of the best performing checkpoint.
Results. The success rate of the best per- forming BC policy and MEDAL++ policy is reported in Table 4. MEDAL++ substan- tially increases the success rate of the learned policies, with approximately 30-70% improve- ments. We provide an abridged version of the analysis here, and defer more detailed analysis to Appendix A.4: First, we con- sider a cuue--grasple late a potential source of improvement from autonomous reinforcement learning, forward demonstrations are collected from narrow ini- tial state distribution but the robot is evaluated starting from both in-distribution (ID) states and out-of-distributio (OOD) states, visualized in Appendix, Figure (only for this experiment). MEDAL++ improves the ID performance by 15% over the BC policy, but we see large im-
Task
Behavioral Cloning
MEDAL+
Cube Grasping
ID
0.85
1.00
OOD
0.08
Cloth Hanging
0.26
0.62
Bowl Cloth Cover
0.12
0.46
Peg Insertion
Figure 4: Evaluation performance the best check- point learned by bchavior cloning and MEDAL++. Ta- ble shows the final success rates over 50 trials from ran- domized initial states, normalized to |0, 1). MEDAL++ substantially improves the final performance compared behavior cloning, validating the self-improvement.
experiment. To iso-
7

---

provement of 74% on OOD performance. Autonomous training allows the robot to practice the task from a diverse set of states, including states that were OOD relative to the demonstration data. This suggests that we expect improvement in success rate to result partly from being robust to the ini- tial state distribution, as a small set of demonstrations unlikely to cover all possible initial states robot can be evaluated from. Next, we evaluate MEDAL++ on grasping cloth and putting through a fixed hook. Here, MEDAL++ improves the success rate over BC by 36%, improving the grasp success, reducing drift and collision with the hook and importantly, reducing memorization as MEDAL++ learns a policy that re-tries grasping the cloth if it fails the first time, rather than fol- lowing a memorized trajectory observed in the forward demonstrations. We observe similar trends on bow.=-oveering-witheee
and the peg-insertion (5DoF) where we observe that MEDAL++ improves 34% and 48% in success rate over BC, with similar sources of improvement as cloth-on- tat---oo-ttehoottt- hook. Overall, we observe that not only is MEDAL++ feasible to run in the real world with minimal task engineering, but it can also substantially improve the policy over BC.
5.3 Ablations
tatrstop Organiadiin
We benchmark four variants on the tabletop organization and peg insertion tasks in Figure 5: (1) MEDAL++, (2) MEDAL++ with the true reward function instead of the learned VICE reward, (3) MEDAL++ without the en- semble of Q-value functions, but, using SAC [57], and (4) MEDAL++ with neither BC-regularization nor over- sampling expert transitions when training Q-value func- tions. The learned reward in MEDAL++ can recover or exceed the performance with true rewards. Both en- semble of Q-values and BC-regularization + oversampled expert transitions improve the performance, though the latter makes a larger contribution to the improvement in performance. Note, when using the true rewards, BC- expert transitions can hurt the final performance (as discussed in Section 5.1). How- ever, when using learned rewards, they both become more important for better performance. We hypothesize that as the learned reward function becomes noisier, other com- ponents become more important for efficient learning and better final performance.
Dest attion Success
Inaerito
Evsluitton
MODALL 118C EVENUT
MOSAL weerd
regularization/overaappiii
Figure 5: Ablation identifying con-
tributions from different components of MEDAL++. Improvements from BC reg- ularization and oversampled expert transi- tions important for learning efficiency,
6 Discussion
We proposed MEDAL++, a method for learning autonomously from high-dimensional image ob- servation without engineered reward functions or human oversight for repeated resetting of the en- vironment. MEDAL++ takes a small set of forward and backward demonstrations as input, and autonomously practices the task to improve the learned policy, as evidenced by comparison with behavior cloning policies trained on just the demonstrations.
Limitations and Future Work: Real robot data collection is slow, even when autonomous. While the control frequency is 10 Hz, training data collected at approximately 3.5 Hz because policy updates and collection steps are done sequentially. Asynchronous and parallel data collection and training can substantially increase the amount of data collected. Several algorithmic extensions can improve the learning efficiency: sharing the visual encoder and the environment transitions between forward and backward policies, using better network architectures and exploration specifically designed for learning autonomously can improve the sample efficiency. Additionally, reducing the number of demonstrations required per task while learning an effective reward function and minimizing the exploration challenge would lead to greater autonomy. Our work assumes the environments to be reversible. Extending MEDAL++ to environments with irreversible states, for example, using PAINT [40], is an exciting opportunity. Intermittent human interventions to reset the environment can still be important learn successfully. The robotic system can get stuck in a specific state when collecting data autonomously due to poor exploration. Developing and using better methods for exploration pretraining on more offline data can further reduce human interventions in training.
8

---

7 Acknowledgements
We would like to acknowledge Tony Zhao, Sasha Khazatsky and Suraj Nair for help with setting up robot tasks and control stack, Eric Mitchell, Joey Hejna, Suraj Nair for feedback on an early draft, Abhishek Gupta for valuable conceptual discussion, and members of IRIS and SAIL. for listening AS drone about this project on several occasions, personal and group meetings. This project was funded by ONR grants N00014-20-1-2677 and N00014-21-1-268. and, Schmidt Futures.
References
[1] E. Jang, A. Irpan, M. Khansari, Kappler, F. Ebert, C. Lynch, Levine, and C. Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learn- ing, pages 991-1002. PMLR, 2022.
[2] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, et al. RT-1: Robotics transformer for real-world control at scale. arXiv preprint urXiv:2212.0689
2022.
[3] J. D. Co-Reyes, S. Sanjeev, G. Berseth, A. Gupta, and S. Levine. Ecological reinforcement learning. arXiv preprint arXiv:2006. /2478, 2020.
[4] H. Zhu, Yu, Gupta, Shah, K. Hartikainen, A. Singh, Kumar, and Levine. The ingredients of real-world robotic reinforcement learning, 2020. URL https://arxiv. org/abs/ 200/ 2004. 12570.
[5] A. Sharma, K. Xu, N. Sardana, A. Gupta, K. Hausman, S. Levine, and C. Finn. Autonomous reinforcement learning: Formalism and benchmarking. International Conference on Learning Representations (ICLR), 2021. URL https: / /ttraaiiivv/..///////ea///aaaavm org/ abs 2212. 09605.
[6] B. Eysenbach, S. Gu, J. Ibarz, and S. Levine. Leave trace: Learning reset for safe and autonomous reinforcement learning. International Conference on Learning Representations (ICLR), 2018. URL httpthttp//// /taaaaxvvvoorr..//a.o atrgg/bbb/11111.. 06782.
[7] A. Sharma, A. Gupta, Levine, Hausman, and Finn. Autonomous reinforcement learn- ing via subgoal curricula. Advances in Neural Information Processing Systems, 34:18474 18486, 2021.
[8] K. Xu, S. Verma, C. Finn, and S. Levine. Continual learning of control primitives: Skill discovery via reset-games, Neural Information Processing Symposium (NeurIPS), 2020. URL https: //tp:///axiiivv//i org/ abs 2011 05286.
[9] J. Kim, J. hyeon Park, D. Cho, and H. J. Kim. Automating reinforcement learning with example-based resets. IEEE Robotics and Automation Letters, 7(3):6006-6613, 2022.
[10] A. Nair, B. McGrew, M. Andrychowicz, W. Zaremba, and Abbeel. Overcoming exploration in reinforcement learning with demonstrations. In 2018 IEEE international conference robotics and automation (ICRA), pages 6292-6299, IEEE, 2018.
[11] A. Sharma, R. Ahmad, and C. Finn. A state state-distribution matching approach to non-episodic re- inforcement learning. International Conference on Machine Learning, pages 19645-19657. PMLR, 2022. URL https: //arxiv///ivv org/ abs 2205. 05212.
[12] D. Yarats, R. Fergus, A. Lazaric, and L. Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning, 2021. URL /tR.ltttsss//aaw....oo//bbbbbb// 09645.
[13] J. Ho and S. Ermon. Generative adversarial imitation learning, 2016. URL https: //arxiv. //arxiv.org/abal wrt. /1606. 03476.

---

[14] A. Singh, L. Yang, K. Hartikainen, C. Finn, and S. Levine. End-to-end robotic reinforcement learning without reward engineering. ArXiv, abs/1904.07854, 2019.
[15] X. Chen, C. Wang, Z. Thou, and Ross. Randomized ensembled double q-learning: Learning fast without model, 2021. URL httph:tttttttiiiie org/ abaaabs211111 05982.
[16] A.
Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstra- tions, arXiv preprint arX0v:/709.10087, 10087, 2017.
[17] Lange, M. Riedmiller, and Voigtländer. Autonomous reinforcement learning on raw visual input data real world application. In The 2012 international joint conference on neural networks (ICCNN), pages 1-8. IEEE, 2012.
[18] J. Kober, J. A. Bagnell, and J. Peters. Reinforcement learning in robotics: A survey, The International Journal of Robotics Research, (2(1 2013.
[19] S. Levine, C. Finn, Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334-13 2016.
[20] F. Ebert, S. Dasari, A. X. Lee, S. Levine, and C. Finn. Robustness via retrying: Closed-loop robotic manipulation with self-supervised learning. In A. Billard, A. Dragan, J. Peters, and J. Morimoto, editors, Proceedings of The 2nd Conference on Robot Learning, volume 87 of Proceedings of Machine Learning Research, pages 983-993. PMLR, 29-31 Oct 2018. URL https: http:////ccceeiingg. mlr. www..press//............ senass/v8//e wtt.v/////////ee /treebeeerttBt 18..111..mmmm...
[21] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, and S. Levine. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation, 2018. URL https: htapps///axxv...// 1 10293.
[22] H. Zhu, A. Gupta, A. Rajeswaran, S. Levine, and V. Kumar. Dexterous manipulation with deep reinforcement learning: Efficient, general, and low-cost, 2018. URL https::/arre org/abs/181 06045.
[23] A. Nagabandi, K. Konoglie, S. Levine, and V. Kumar. Deep dynamics models for learning dexterous manipulation, 2019, URL https: / /ttp//axxvvv..//// aass//99. 11652.
[24] A. Zeng, S. Song, J. Lee, A. Rodriguez, and T. Funkhouser. Tossingbot: Learning to throw arbitrary objects with residual physics, 2019. URL https::/arxxv..rrg//// 903.. 11239.
[25] D. Kalashnikov, J. Varley, Y. Chebotar, B. Swanson, R. Jonschkowski, C. Finn, S. Levine, and K. Hausman. Mt-opt: Continuous multi-task robotic reinforcement learning at scale, 2021. URL https: http://xxvvv.o./ org/ /2ss///1444 2144 08212.
[26] L. Smith, Kew, X. B. Peng. Ha, Tan, and Levine. Legged robots that keep on learning: Fine-tuning locomotion policies in the real world. 2022 International Conference on Robotics and Automation (ICRA), pages 1593-1599, IEBE, 2022.
[27] M. Bloesch, J. Humplik, V. Patraucean, R. Hafner, T. Haarnoja, A. Byravan, N. Y. Siegel, S. Tunyasuvunakool, F. Casarini, N. Batchelor, et al. Towards real robot learning in the wild: case study in bipedal locomotion. In Conference on Robot Learning, pages 1502-1511. PMLR, 2022.
[28] C. Finn, X. Tan, Duan, T. Darrell, S. Levine, and Abbeel. Deep spatial autoencoders for visuomotor learning. In 2016 IEEE International Conference Robotics and Automation (ICRA), pages 512-519, IEEE, 2016.
10

---

[29] S. Gu, E. Holly, T. Lillicrap, and S. Levine. Deep reinforcement learning for robotic manipula- tion with asynchronous off-policy updates. In 20/7 IEEE international conference on robotics and automation (ICRA), pages 3389-3396 IEEE, 2017.
[30] A. Ghadirzadeh, A. Maki, D. Kragic, and M. Björkman. Deep predictive policy training using reinforcement learning. In 2017 TEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 2351-2358. IEEE, 2017.
[3]] Y. Chebotar, K. Hausman, M. Zhang, G. Sukhatme, S. Schaal, and S. Levine. Combining model-based and model-free updates for trajectory-centric reinforcement learning. In Interna- tional conference on machine learning, pages 703-711. PMLR, 2017.
[32] T. Haarnoja, S. Ha, A. Zhou, J. Tan, G. Tucker, and S. Levine. Learning walk via deep reinforcement learning, arXiv preprint 1rX.i:1812..1111 2018.
[33] H. Zhu, A. Gupta, A. Rajeswaran, S. Levine, and V. Kumar. Dexterous manipulation with deep reinforcement learning: Efficient, general, and low-cost. In 2019 International Conference on Robotics and Automation (ICRA), pages 3651-3657. IEEE, 2019.
[34] A. Sharma, M. Ahn, S. Levine, V. Kumar, K. Hausman, and S. Gu. Emergent real-world robotic skills via unsupervised off- off-policy reinforcement learning. arXiv preprint arXiv:2004.12974, 2020.
[35] P. Agrawal, A. V. Nair, P. Abbeel, J. Malik, and S. Levine. Learning to poke by poking: Experiential learning of intuitive physics. Advances in neural information processing systems, 29, 2016.
[36] I.. Pinto and A. Gupta. Supersizing self-supervision: Learning grasp from 50k tries and 700 robot hours. In 2016 IEEE international conference robotics and automation (ICRA), pages 3406-344 3413. IEEE, 2016.
[37] C. Finn and S. Levine. Deep visual foresight for planning robot motion. In 2017 IEEE Inter- national Conference on Robotics and Automation (ICRA), pages 2786-2793. IEEE, 2017.
(38) W. Han, S. Levine, and Abbeel. Learning compound multi-step controllers under unknown dynamics. In 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2015, Hamburg, Germany, September 28- October 2, 2015, pages 6435-6442 IEEE, 2015. doi: 10. 109//ROS..011..354222 2015 7354297.
URL https: /ooi org/10. 11t///.0..199/.RR..
[39] K. Lu, A. Grover, Abbeel, and Mordatch. Reset-free lifelong learning with skill-space planning, arXiv preprint arXiv:2012.03544 2020.
[40] A. Xie, F. Tajwar, A. Sharma, and C. Finn. When to ask for help: Proactive interventions in autonomous reinforcement learning. Neural Information Processing Systems (NeurIPS), 2022.
URL hwtph:ttp////mc arxiv. wwwpxxvvovrrr/eoooooo abs 2210.1070660
[41] A. Gupta, Yu, Zhao, V. Kumar, A. Rovinsky, K. Xu, T. Devlin, and S. Levine. Reset-free reinforcement learning via multi-task learning: Learning dexterous manipulation behaviors without human intervention, 2021. URL htttp:://axxiv aarrivivivv.////a//.aaa///aaavva http::///wwwwww./a///////o/aoa//
[42] A. Gupta, C. Lynch, B. Kinman, G. Peake, S. Levine, and K. Hausman. Bootstrapped au- tonomous practicing via multi-rask reinforcement learning. arXiv preprint 15.xx: 2022.
[43] H. Walke, J. Yang, A. Yu, A. Kumar, Orbik, A. Singh, and Levine. Don't start from scratch: Leveraging prior data automate robotic reinforcement learning. Conference Robot Learning (CoRL), 2022, URL https: arxiv.iiivv.//..i org.//iiooorooooooooooooooooeeee org/ abs 220/ 047203..........

---

[44] K. Xu, Z. Hu, R. Doshi, A. Rovinsky, V. Kumar, A. Gupta, and S. Levine. Dexterous ma- nipulation from images: Autonomous real-world rl via substep guidance. arXiv preprint arXiv:2212.09902, 2022.
[45] J. Fu, A. Singh, D. Ghosh, L. Yang, and S. Levine. Variational inverse control with events: A general framework for data-driven reward definition, 2018. URL https: http://aaxvvv.ooooo abs/ / 18800..5 116116666 686.
[46] Fu, K. Luo, and S. Levine. Learning robust rewards with adversarial inverse reinforcement learning. arXiv preprint arXiv:1710.11246 2017.
[47] Nowozin, B. Cseke, and R. Tomioka. f-GAN: Training generative neural samplers using variational divergence minimization. Advances in neural information processing systems, 29, 2016.
[48] S. K. S. Ghasemipour, R. Zemel, and S. Gu. divergence minimization perspective on imita- tion learning methods. In Conference on Robot Learning, pages 1259-1277. PMLR, 2020.
[49] A. X. Lee, A. Nagabandi, P. Abbeel, and S. Levine. Stochastic latent actor-critic: Deep rein- forcement learning with a latent variable model. Advances in Neural Information Processing Systems, 33:741-752, 2020.
[50] M. Laskin, A. Srinivas, and P. Abbeel. CURL: Contrastive unsupervised representations for reinforcement learning. In International Conference on Machine Learning, pages 5639-5650. PMLR, 2020.
[51] I. J. Goodfellow, Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, Ozair, A. Courville, and Y. Bengio. Generative adversarial networks, 2014. URL 1406.2661
[52] Miyato, Kataoka, M. Koyama, and Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957, 2018.
[53] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz. mixup: Beyond empirical risk mini- mization, 2017. URL https: /ttp/aaaxvvvv/////.//.a http///ss11111 09412.
[54] I. Kostrikov, K. K. Agrawal, D. Dwibedi, S. Levine, and J. Tompson. Discriminator-actor- critic: Addressing sample inefficiency and reward bias in adversarial imitation learning, 2018. URL hww.h:ttp///......m http//aivvvv//////////// org/ 1809, 02925.
[55] Y. Burda, H. Edwards, A. Storkey, and O. Klimov. Exploration by random network distillation. arXiv preprint 1rXiv:1810.12880 2018.
[56] D. Kingma and M. Welling. Auto-encoding variational 13 14,61111400011 2013.
bayes.
arXiv preprint
[57] T. Haarnjja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[58] K. Hsu, M. J. Kim, R. Rafailov, J. Wu, and C. Finn. Vision-based manipulators need to also see from their hands. arXiv preprint 17217 2022.
12

---

Figure 6: Environments from the EARL benchmark used for simulated experiments. From left right, the environments are: Peg insertion, Door closing and Tabletop organization.
Image Observation O
CNNS
Fused Embedding
Agent
action d
Propeloception
Discriminator
Environment
reward
Figure 7: (lef)) Randomized position of the cube in the grasping task. The position marked by violet boundary are within the distribution of expert demonstrations, and the rest are outside the distribution. (right) Architecture overview for MEDAL++.
A Appendix
A.1 Algorithm Overview
The pscudocode for training is given in Algorithm 1. First, the parameters and data buffers in F and B are initialized and the forward and backward demonstrations are loaded into D7 and DØ respec- tively. Next, we update the forward and backward goal sets, as described above. After initializing the environment, the forward policy T interacts with the environment and collects data, updating the networks and buffers in F. The control switches over to the backward policy T. after a fixed number of steps, and the networks and buffers B are updated. The backward policy interacts for fixed number of steps, after which the control is switched over the forward policy and this cycle repeated thereon. When executing in the real world, humans are allowed intervene and reset the environment intermittently, switching the control over to f after the intervention to restart the forward- backward cycle.
We now expand on how the networks are updated for f during training (also visualized in Figure 9); the updates for т. analogous. First, the new transition in the environment added to Dy. Next, sample batch of states from D, and label them 0, and sample batch of equal size from Dj and label them 1. The classifier Cy is updated using gradient descent on the combined batch to minimize the cross-entropy loss. Note, the classifier is not updated for every step collected in the environment. As stated earlier, the classification problem is easier than learning the policy, and therefore, it helps to train the classifier slower than the policy. Finally, the policy Tý. Q-value networks (Q,, Q V and the encoder are updated on a batch of transitions constructed by sampling (1 p) tran- sitions from D and pB transitions from D7. The Q-value networks and the encoder are updated by minimizing N Σm= E(Qn. E) (Eq 1), and the target Q-networks are updated exponential
1
moving average of Q-value networks. The policy Ty is updated by maximizing [(T)). We update
13

---

Algorithm 1: MEDAL+
initialize F, BB; forward, backward parameters
<- load demonst natonsstaati ons ()
F.G + get_states (F.Dj -K)) // last states
exclude last states from D,, use only the last K states from D
B.G. get states (F.D),:- K) get states (B.D, -:))
Po: A F; initialize environment
while not done do
A.a T( s,
duspdate.classiffern.
eeypdate-parameten
// switch policy after fixed interval
if switch then
swi twitch(A, (F)
// allow intermittent human interventions
interrupt then
Po:
F+F;
else
+ s';
Backward Coal
Expert Data
classifier
Forward Goal States
States
predict
rewards
BC
Online
RL
Initial State Distribution
Goal Distribution
Figure An overview of MEDAL++ training. The classifier is trained to dis- criminate states visited by an expert from the visited online. The robot rein- forcement leams on a combination of self- collected and expert transitions, and the policy learning is regularized using the be- havior cloning loss.
Visualizing the positive target states for forward clas- sifier Cj and backward classifier C from the expert demonstra- tions. For forward demonstrations, last K states are used for Cy (orange) and the used for Cs (pink). For backward demonstrations, states are used for C..
Figure 8:
the Q-value networks multiple times for every step collected in the environment, whereas the policy network is updated once for every step collected the environment [15].
A.2 Implementation Details and Practical Tips
An overview of the architecture used by the forward and backward networks is shown in Figure
Visual Encoder: For the encoder, we use the same architecture DrQ-v2 [12]: 4 convolutional layers with 32 filters of size (3, 3), stride followed by ReL.U non-linearities. The high-dimensional output from the CNN is embedded into 50 dimensional feature using fully-connected layer, followed by LayerNorm and tanh non-linearity (to output the features normalized to [-1, 1)). For real-robot experiments, the first person and third person views are concatenated channel wise before being passed into the encoder. The output of the encoder fused with proprioceptive information, this case, the end-effector position, before being passed to actor and critic networks.
Actor and Critic Networks: Both actor and critic networks parameterized as 4 layer fully- connected networks with 1024 ReLU hidden units for every layer. The actor parameterizes Gaus-
14

---

sian distribution over the actions, where a tanh non- non-linearity on the output restricts the actions to [-1,1]. We use an ensemble size of 10 critics.
Discriminators: The discriminator for the forward and backward policies use a similar visual en- coder but with 2 layers instead of 4. The visual embedding is passed fully connected network with 2 hidden layers with 256 ReLU units. When training the network, we mixup and spectral norm regularization [53, 52] for the entire network.
Training Hyperparameters: For all our experiments, K = 20, i... the number of frames used as goal frames. The forward policy interacts with the environment for 200 steps, then the backward policy interacts for 200 steps. In real world experiments, we also reset the arm every 1000 steps to avoid hitting singular positions. Note, this reset does not require any human intervention as the controller just resets the arm to a fixed joint position. We use batch size of 256 to train the policy and critic networks, out of which 64 transitions are sampled from the demonstrations (oversampling). We use a batch size of 512 to train the discriminators, 256 of the states come from expert data and the other 256 comes from the online data. Further, the discriminators are updated every 1000 steps collected in the environment. The update-to-data ratio, that is the number of gradient updates per transition collected in the environment We use linearly decaying schedule for behavior cloning regularization from 1 to 0.1 over the first 50000 steps which remains fixed at 0.1 onwards throughout training.
for simulated environments and 1 for the real-robot experiments.
For real world experiments, we use a wrist camera to improve the overall performance [58], and provide only the wrist-camera view to both discriminators. We find that this further regularizes the discriminator. Finally, we provide no proprioceptive information for the VICE discriminator, but we give MEDAL discriminator the proprioceptive information, as it needs stronger notion of the robot's localization to adequately reset to varied number of initial positions for improved robustness.
Teleoperation: To collect our demonstrations on the real robot, we use an Xbox controller that manipulates the end-effector position, orientation and the gripper state. Two salient notes: (1) The forward and backward demonstrations are collected together, one after the other and (2) the initial position for demonstrations is randomized to cover large state-space as feasible. The increased coverage helps with exploration during autonomous training.
A.3 EARL Environments, Training and Evaluation
Environments. We consider three sparse-reward contimuous-contro) environments from EARL benchmark [5], shown in Appendix, Fig 6). Tabletop organization is a simplified manipulation environment where a gripper is tasked to move the mug to one of the four coasters from a wide set of initial states, sawyer door closing task requires a sawyer robot arm learn how close a door starting from various positions, and finally the sawyer peg insertion task requires the sawyer robot arm to grasp the peg and insert it into a goal. Not only does the robot have to learn how to do the task (i.e. close the door or insert the peg), but it has to learn how to undo the task (i.e. open the door or remove the peg) to try task repeatedly in the non-episodic training environment. The sparse reward function is given by r(s, a) = [[]ss - gll I e), where g denotes the goal, and e is the tolerance for the task to be considered completed.
Training and Evaluation. The environments are setup to return 84 x 84 RGB images as obser- vations with a 3-dimensional action space for the tabletop organization (2D end-effector deltas in the XY plane and 1D for gripper) and a 4-dimensional action space for sawyer environments (3D end-effector delta control + 1D gripper). The training environment is reset to 8 ~ Po every 25,000 steps of interaction with the environment. This is extremely infrequent compared to episodic set- tings where the environment is reset to the initial state distribution every 200-1000 steps. EARL comes with 5-15 forward and backward demonstrations for every environment to help with explo- ration in these sparse reward environments. We evaluate the forward policy every 10, 000 training steps, where the evaluation approximates Esor Er 7'r(st, a,)] by averaging the return of the
15

---

Expert Trajectories
Target states for backward policy
Tanget shates for forward policc
Autonomous Online Data
Forward Policy
Backward Policy
Forward Policy
Figure 10: An overview MEDAL++ on the task inserting the peg into the goal location. (top) Starting with a set of expert trajectories, MEDAL++ learns forward policy to insert the peg by matching the goal states and a backward policy to remove and randomize the peg position by matching the rest of the states visited by expert. (bottom) Chaining the rollouts of forward and backward policies allows the robot to practice the task autonomously. The rewards indicate the similarity their respective target states, output by discriminator trained classify online states from expert states.
policy over 10 episodes starting from 5g ~ po. These roll-outs are used only for evaluation, and not for training.
A.4 Real-world Experiment Analysis
We discuss the four manipulation tasks in detail. We recommend viewing the supplemental website for training and evaluation videos:
(1) Cube Grasping: The goal in this task grasp the cube from varying initial positions and con- figurations and raise it. For this task, consider controlled setting to isolate one potential source of improvement from autonomous reinforcement learning: robustness to the initial state distribution. Specifically, all the forward demonstrations are collected starting from narrow set of initial states (ID), but, the robot evaluated starting from both ID states and out-of-distribution (OOD) states, visualized in Appendix, Figure 7. BC policy is competent on ID states, but it performs poorly on states that are OOD. However, after autonomous self- eef--improvement using MEDAL++ we see an improvement of 15% ID performance, and large improvement of 74% on OOD performance. Autonomous training allows the robot to practice the task from diverse of states, including states that were OOD relative the demonstration data. This suggests that improvement in success rate results partly from being robust to the initial state distribution, unlikely to cover all possible initial states a robot can be evaluated from.
small set of demonstrations
(2) Cloth on the Hook: In this task, the robot tasked with grasping the cloth and putting it through a fixed hook. To practice the task repeatedly, the backward policy has to remove the cloth from the hook and drop on platform. Here, MEDAL++ improves the success rate over BC by 36%. The BC policy has several failure modes: (1) it fails grasp the cloth, (2) it follows through with hooking because of memorization, or (3) it hits into into the hook because it drifts from the right trajectory and could not recover. Autonomous self-improvement improves upon all these issues, but particularly, it learns to re-try grasping the cloth if it fails the first time, rather than following a memorized trajectory observed the forward demonstrations.
(3) Bowl Covering with Cloth: The goal of this task to cover bowl entirely using the cloth. The cloth can be wide variety of initial states, ranging from 'laid out flat' to 'scrunched up' in varying locations. The task is challenging as the robot has to grasp the cloth the correct location to successfully cover the entire bowl (partial coverage is counted improves the performance over BC by 34%. The failure modes of BC are similar to previous task, including failure to grasp, memorization and failure to re-try, and incomplete coverage due to wrong initial grasp. Autonomous self eel--improvement substantially helps with the grasping (including re- trying) and issues related to memorization. While it plans the grasps better than BC, there room for improvement to reduce failures resulting from partially covering the bowl.
failure). Here, MEDAL++
(4) Peg Insertion: Finally, and orientation of the peg is randomized, in service of which successful insertion requires the toy to be perpendicular to the goal before insertion, and the error
consider the task of inserting peg into goal location. The location 5DoF control for this task. A
16

---

margin for a successful insertion is small given the size of the peg and the goal. Additionally, the peg here is soft toy, it can be grasped while being in the wrong orientation. Here, MEDAL++ improves the performance by 48% over BC. In addition to failures described in the previous tasks, a common cause failure is the insertion itself where the agent takes an imprecise trajectory and unable insert the peg. After autonomous self-improvement. the robot employs an interesting strategy where it keeps retries the insertion till it succeeds. The policy is also better at grasping, though the failures of insertion often result from orienting the gripper incorrectly before the grasp which makes insertion infeasible.
17


Reviews:
[{"review_id": "zvfMh4Ah3n", "reviewer": "Reviewer_uNtL", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces MEDAL++, an extension of MEDAL. In MEDAL, forward and backwards policies perform autonomous reinforcement learning by learning to reset the environment. MEDAL++ incorporates RGB images and trains a policy end-to-end on real robot data using a small number of demonstrations. To enable this process, a host of ideas are implemented from existing works to improve efficiency, including image augmentation, ensembles of Q functions, learning reward functions from goal images, and incorporating behaviour cloning objectives.\n\nThe experiments are performed on a simulation benchmark for autonomous reinforcement learning (EARL), and on a real Franka arm performing tasks with soft objects (cloth and squishy blocks). The results show significant improvement over behaviour cloning after 30 hours of real-world interaction.\n", "summary_of_recommendation": "The implementation details of this work can be useful for others, hopefully leading towards simpler networks succeeding with fewer interactions on hardware (both demonstrations and training) and more complex tasks.\n\nThe authors efforts in the rebuttal were appreciated, sticking with the original recommendation (weak accept).\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "This is a significant number of networks to update in sequence with data collection, particularly with an update-to-data ratio of 1. What is the compute used? Were there any comparative results from hyperparameter search where a smaller setup was not sufficient, e.g. ensemble number, layer number and size etc. \n\nIn Figure 5, ablation (4), is it the regularization or the oversampling that has the greater effect?\n\nIn Equation 1, should it be r + \\gamma V(.)? \n\nSimilarly, the parameters and data buffers are represented by the tuple -> Similarly, the parameters and data buffers for the backward policy are represented by the tuple\n\ncan enable real-robots to be learn -> can enable real-robots to learn\n\nregularize policy learning towards [16], -> regularize policy learning towards demonstration data [16],\n\nas shown in Figure 4 -> should be as shown in Figure 3. Figure 4 should be Table 4 (or more preferably Table 1).\n\nReferences should include their conference acceptance if applicable, e.g:\nLeave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning -> ICLR 2018", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper was well presented and easy to read. Several practical ideas from previous works were effectively brought together here. While there were no notable new ideas in algorithms, novelty was introduced through the online learning of complex behaviours. To train a visuo-motor policy from scratch using real-world experience and a handful of demonstrations to interact with soft objects is impressive. \n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "CMvwsu1Bx-v", "reviewer": "Reviewer_xqXG", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper develops MEDAL++, and improvement over MEDAL, which seeks to learn robot skills through interaction with a user by learning to \"do\" and \"undo\" the task. The paper argues for the importance of removing the need for a user to reset the task to a start state. MEDAL++ improves upon MEDAL with an pixel-based encoder, a reward function-learning algorithm (VICE), and additional tweaks. The paper reports empirical results in simulation (including an ablation study) and in a real-world robot setup.", "summary_of_recommendation": "This paper shows positive results against a baselines that take a similar approach; however, the paper does so with incremental improvements by swapping out components of an existing baselines for existing component replacements. It is unclear that the research community would gain much additional insight into the design of these systems.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "-In addition to responding to the weaknesses above, can the authors also address what kinds of problems MEDAL++ (or really VICE) would fail due to not learning a good-enough goal state classifier?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n\n-The introduction to MEDAL is clear and concise.\n\n-The use of VICE, among many other reward learning (e.g., inverse RL) approaches is reasonably justified.\n\n-MEDAL++ makes MEDAL more realistic by including a convolutional autoencoder from pixel space to a state-embedding space.\n\n-Instead of requiring a reward function, MEDAL++ learns one using VICE, which learns a classifier over goal states.\n\n-The paper provides strong empirical results in a computational evaluation, an ablation study, and a real robot setup (compared to Behavioral Cloning).\n\n-The paper includes a discussion of limitations.\n\n-The paper is very well written.\n\n-The paper includes code for reproducing the results.\n\n\nWeaknesses:\n-Many papers are incremental improvements over prior work. Having said that, this paper seems more of an engineering contribution than even an incremental research contribution. The swapping out of various components for other existing components brings into question whether this paper really brings a full-paper's worth of new insights. At a minimum, this paper should be a journal paper; however, it is unclear that this paper is a standalone conference paper without significantly more empirical or theoretical analysis showing the justification of these engineering choices in a way that can inform research in this area writ large.\n\n-The proposed approach takes hundreds of thousands of episodes to outperform BC. This seems to stand in opposition to the goals of the paper of making a deployable algorithm.\n\n-The related works section, at just two paragraphs, is a bit short considering the breadth of work in RL and LfD.\n\n-\"benchmark EARL\" should have a comma before \"EARL\"\n\n-\"[11,4]\" -- can be nice to order citations numerically\n\n-The real-world robot evaluation does not include an RL-based baseline (e.g., MEDAL); however, this is a relatively minor point.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "0h67K8RXwNl", "reviewer": "Reviewer_Y525", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a system that can do nearly autonomous self improvement by reinforcement learning without requirement of reward engineering and frequent episodic resets. The key is to combine several techniques including MEDAL for learning a backward policy for automatic reset, VICE for learning a reward function from demonstration, drq-v2 for direct policy learning from high-dimensional pixels, and improvements on Q function learning to increase sample efficiency. The system is evaluated on 3 simulation tasks and 3 real-world tasks, and is shown to outperform several baselines. ", "summary_of_recommendation": "As summarized above.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "The method still requires 50 demonstrations for both forward and backward policy learning. The authors are encouraged to test or add discussions on if the number of demonstrations can be further reduced. ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strength:\n- Autonomous real-world robotic RL is of great important to the community, so the paper is well-motivated.\n- The paper is clearly written and easy to follow.\n- As a system paper, although the technical novelty is limited, the method is well evaluated on 3 real-world tasks with certain complexities that involve contacts and manipulation of deformables. It's good to see the methods working on these tasks.\n\nWeakness:\n- As mentioned, the major weakness is the lack of technical novelty, as the system is a combination & small improvements on prior work. However, this is fine when viewing the paper as a practical system.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose MEDAL++ as a novel design for visuomotor RL on robotic systems. It combines a number of existing techniques to achieve this, and while technical novelty is limited, the authors show how this combination can be leveraged in a novel system design to achieve impressive real-world performance. This is also how most reviewers evaluated the paper. In addition, they appreciated the experimental evaluation on three simulated and three real-world tasks with varying complexity.
Decision: Accept (Poster)