REBOOT: Reuse Data for Bootstrapping Efficient Real-World Dexterous Manipulation
Authors: Zheyuan Hu
Aaron Rovinsky
Jianlan Luo
Vikash Kumar
Abhishek Gupta
Sergey Levine
Venue: CoRL 2023 Poster (2023)
Keywords: Dexterous Manipulation
Reinforcement Learning
Sample-Efficient RL

Abstract: Dexterous manipulation tasks involving contact-rich interactions pose a significant challenge for both model-based control systems and imitation learning algorithms. The complexity arises from the need for multi-fingered robotic hands to dynamically establish and break contacts, balance forces on the non-prehensile object, and control a high number of degrees of freedom. Reinforcement learning (RL) offers a promising approach due to its general applicability and capacity to autonomously acquire optimal manipulation strategies. However, its real-world application is often hindered by the necessity to generate a large number of samples, reset the environment, and obtain reward signals. In this work, we introduce an efficient system for learning dexterous manipulation skills with RL to alleviate these challenges. The main idea of our approach is the integration of recent advancements in sample-efficient RL and replay buffer bootstrapping. This unique combination allows us to utilize data from different tasks or objects as a starting point for training new tasks, significantly improving learning efficiency. Additionally, our system completes the real-world training cycle by incorporating learned resets via an imitation-based pickup policy and learned reward functions, to eliminate the need for manual reset and reward engineering. We show the benefits of reusing past data as replay buffer initialization for new tasks, for instance, the fast acquisitions of intricate manipulation skills in the real world on a four-fingered robotic hand. \href{https://sites.google.com/view/reboot-dexterous}{https://sites.google.com/view/reboot-dexterous})

REBOOT: Reuse Data for Bootstrapping Efficient Real-World Dexterous Manipulation
Zheyuan Hul Aaron Rovinsky'*,
JJinnlan Luo', Vikash Kumar", Abhishek Gupta', Sergey Levine*
UC Berkeley
Meta AI Research
University of Washington
Across Objects
Across Tasks
Ofline-OOline Data
REBOOT achieves 2X sample efficiency boost learning variety of contact-rich real-world dexterous manipulation skills three different objects autonomously bootstrapping prior data across different objects and tasks with sample-efficient and imitation learning-based based reset policies.
Figure 1:
Abstract: Dexterous manipulation tasks involving contact-rich interactions pose a significant challenge for both model-based control systems and imitation learning algorithms. The complexity arises from the need for multi-fingered robotic hands dynamically establish and break contacts, balance non-prehensile forces, and control large degrees of freedom. Reinforcement learning (RL) offers promising approach due to its general applicability and capacity to autonomously acquire optimal manipulation strategies. However, its real-world application is often hin- dered by the necessity to generate a large number samples, reset the environ- ment, and obtain reward signals. In this work, we introduce an efficient system for learning dexterous manipulation skills with RL alleviate these challenges. The main idea of our approach is the integration of recent advances in sample- efficient RL and replay buffer bootstrapping. This combination allows to utilize data from different tasks or objects starting point for training new tasks, sig- nificantly improving learning efficiency. Additionally, our system completes the real-world training cycle by incorporating learned resets via an imitation mnisation-based pickup policy as well as learned reward functions, eliminating the need for man- ual resets and reward engineering. We demonstrate the benefits of reusing past data replay buffer initialization for new tasks, for instance, the fast acquisition of intricate manipulation skills in the real world on four-fingent robotic hand. (Videos: www.://ss.gollle.oooeeeeeeeeeeee
Keywords: Dexterous Manipulation, Reinforcement Learning, Sample-Efficient
Introduction
Dexterous manipulation tasks involving contact-rich interaction, specifically those involving multi- fingered robotic hands and underactuated objects, pose significant challenge for both model-based control systems and imitation learning algorithms. The complexity arises from the need for multi-
Both authors contributed cqually
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

RGB 128x126x3
Online Buffer
By 9,.)
Previous Buffers
Propricceptive States
50:50 Replay Sampling
Inhand Practice
((a)
Prefrained
Ax4x578
UTD-1
GENEET
(is.a)
Reset
Encoder
Leamed Spatial Embeddings
(s)
Figure 2: REBOOT System Overview: Our method learns various dexterous manipulation skills the real world using raw image observations. This is enabled by using sample-efficient RL and bootstrapping with data from other tasks and even other objects, with autonomous resets.
fingered robotic hands to dynamically establish and break contacts, balance non-p non-prehensile forces, and control high number of degrees of freedom. Reinforcement learning (RL) offers a promising solution for such settings. In principle, RL enables a robot refine its manipulation skills through process of trial-and-error, alleviating the requirement for strong modeling assumptions. However, making RL methods practical for learning such complex behaviors directly in the real world presents a number of obstacles. The main obstacle is sample efficiency: particularly for tasks that require complex interactions with many possibilities for failure (e.g., in-hand reorientation where the robot might drop the object), the number of trials needed for learning a skill with RL from scratch might be very high, requiring hours even days of training. Additionally, real-world learning outside of the lab requires the robot to perform the entire training process using its sensors and actuators, evaluating object state and rewards using camera observations, and resetting autonomously between trials. Because of these challenges, many prior works on RL for dexterous manipulation have ex- plored alternative solutions, such as sim-to-real transfer [1, 2, 3], imitation learning [4, 5, 6]. or the use of tools like motion capture [7, 2] or separately-engineerred reset mechanisms [8, 9].
In this paper, we instead propose system that is designed to make direct RL in the real world practical without these alternatives, so to take step toward robots that could one day learn under assumptions that are sufficient for autonomously acquiring new skills in open- open-world settings, even outside the laboratory. This means that the entire learning process must be conducted using the robot's own sensors and actuators, without simulation or additional instrumentation, and be efficient enough to learn skills quickly. We posit that key enabling factor for this goal is to reuse data from past skills, and we instantiate this with a simple buffer initialization method, where the replay buffer of each skill is initialized with data from other tasks vision-based method for learning reward functions from useviido images and learned reset procedure automatically pick up an object between trials, we demonstrate that our system enables robotic hand to learn in hand reorientation skills in just few hours of fully autonomous training, using only camera observations and joint encoder readings.
other objects. In combination with
Our main contribution is REBOOT, a system to Reuse Data for Bootstrapping Real- World Dex- terous Manipulation, which we illustrate in Figure By simply preloading the replay buffer using prior data from other objects and tasks, our system avoids starting from scratch for every new task. By combining recent advances in sample-efficient online RL [10] with buffer initialization to boot- strap learning from prior tasks and objects, we show that in-hand manipulation behaviors can be learned in a few hours of autonomous practicing.
We additionally use learned reset skills to make training autonomous, and extend adversarially learned rewards to handle buffer initialization method, allowing users to specify tasks with few examples of desired object poses and without manual reward engineering. Some of the skills

---

learned by our system, shown in Figure 3, include in-hand reorientation of a three- saree-pronged object, handling a T-shaped pipe, and manipulating a toy foothall.
2 Related Work
A number of designs for anthropomorphic hands have been proposed in prior work [11, 12, 13]. Prior learning-base based methods to control such hands utilize trajectory optimization [14, 15], policy search [16, 17, 18), demonstration inttrrattonnbbbced based learning [19, 20, 21, 22]. simulation to real-world transfer [3, 23, 24, 25], reinforcement learning directly in the real world [26, 8, 27, 28, 29], or a combination of these approaches [30].
Most of the aforementioned works leveraged accurate simulations engineered real-world state- estimation systems to provide compact state representations. In contrast, seek to learn visuomo- tor policies autonomously and entirely in the real world without access to privileged state informa- tion, under assumptions that more closely reflect what robots might encounter when learning "on the job" outside of laboratory settings. Prior work has explored learning these policies in simulation [31, 32), where autonomy not of concern due to the ability to reset the simulated environment. Most real-world methods either rely on instrumentation for state estimation [28] or deal with sim- pler robots and tasks [27]. An important consideration in system the ability specify task without manual reward engineering. Although task specification has been studied extensively, most prior works make variety of assumptions, ranges from having humans- humans-provided demonstrations for enabling imitation learning [4. 33, 34J, using inverse RL [35, 36, 37J. active settings where users can provide corrections [38, 39, 40], or ranking- based preferences [41, 42]. Our in-hand RL. training phase learns from raw high-dimensional pixel observations in an end-to-end fashion using DrQ[43] and VICE[44], although our system could use any reward inference algorithm based upon success examples [45]. With users defining the manipulation task by providing small number of image goals instead of full demonstrations,
method not only removes the barrier to orchestrate high-dimensional finger motions [46, 47] but also accelerates robot training progress by offering suf- ficient reward shaping for RL in real-world scenarios without per-task reward engineering. While AVAIL [29] also learns dexterous manipulation skills from raw images, we show in our comparison that our system is faster, and our buffer initialization approach significantly speeds up the acquisition of in-hand manipulation skills compared to starting from scratch.
Buffer initialization has also been employed by Smith al. [48] in the context of transfer learning for robotic locomotion, where similar approach was used to create a curriculum for locomotion skills or adapt to walking on new terrains. Our method differs in several significant ways. First, our method learns from raw image observations with learned reward functions defined through a few example images, whereas [48] uses prog-programmed rewards. Second, our focus on learning intricate dexterous manipulation skills from scratch the real world, whereas [48] uses initializa- tion in simulation. Although the methodology is closely related, our proposed system extends the methodology in significant ways, enabling the use of vision and learned rewards in very different domain.
Reset-free learning is essential for autonomous real-world training of dexterous skills (see [49] for a review of reset-free methods). Most of the prior works [27, 28, 29, 50, 51, 52, 53, 54] rely on "backward" policies reset the environment so the "forward" policy can continue learning the task of interest. Similarly, divide training into phases due to different skills having unique demands for control complexities and user- user-provided supervision. Specifically, the skill needed to pick up objects in reset is better studied and developed for immediate usage through imitating user- provided demonstrations [55].
3 Robot Platform and Problem Overview
custom-built, built, f-niger, 16-DoF robot hand mounted to 7-DoF Sawyer robotic arm for dexterous object manipulation tasks. Our platform is shown in Figure 3. Our focus is on
In this work, we custom-
3

---

learning in in-hand reorientation skills with reinforcement learning, During the in-hand manipulation phase, the RL policy controls the 16 degrees of freedom of the hand, setting target positions at 10 Hz, with observations provided by the joint encoders in the finger motors and two RGB cameras, one overhead and another embedded in the palm of the hand. To facilitate autonomous training, we also use im- itation learning learn reset policy to pick up the ob- ject from the table in-between in-hand manipulation tri- als. This imitation policy uses a 19-dimensional action space, controlling the end effector position of the wrist and 16 finger joints to pick up the object from any loca- tion.
Our tasks are parameterized by images of desired object poses in the palm of the hand. Since the reset policy can grasp the object in variety of poses, the in-hand policy must learn to rotate and translate the object care- fully to achieve the goal pose. We train and evaluate our method entirely in the real world. In the following sec- tions, we describe how data from different objects can be used to bootstrap manipulation skills for more effi- cient learning.
Depiction of our hardware plat- form and tasks. (a) custom-built 16 DoF robotic hand (c) teleoperation using the 3-D mouse, to interact with the following objects in-hand (b) blue football, (d) 3-pronged valve, (e) T-shaped pipe.
Figure 3:
4 Reinforcement Learning with Buffer Initialization
In this work, we propose system for efficiently learning visuomotor policies for dexterous ma- nipulation tasks via bootstrapping with prior data. We describe our learning method and real-world considerations for system in the following subsections.
Problem setting. Our method leverages the framework of Markov decision processes for rein- forcement learning described in [56]. In RI, the aim is to learn policy ((ae)s) (sz) that obtains the maximum expected discounted sum of rewards R(st, ar) under initial state distribution po. dynamics P(S4+1151, az), and discount factor The formal objective is as follows:
Σ xt R(se. at)
J(x) EF
eprupo (ac (8c)
The particular reinforcement learning algorithm that we build on in this work is RLPD [10], sample-efficient RL method that combines design based on soft actor-critic (SAC) [57] with a number design decisions to enable fast training. This approach trains a value function Q* (st, ae) combination with an actor or policy т T[ae (t), though principle our system could be compatible with a variety of off-policy RL. algorithms that utilize a replay buffer. For more details on RLPD, we refer readers to prior work [10].
Reinforcement learning with buffer initialization. While using a sample-efficient RL algorithm such RLPD to acquire in-hand manipulation skills can be feasible in the real world, the training process take a very long time (see Section 5). A central component of system design is to utilize data from other tasks or even other objects to bootstrap the acquisition of new in-hand manipulation skills. In our experiments, we will show that very simple procedure can make this possible: for every RL update, we sample half the batch from the growing buffer for the current task, and half the batch from a buffer containing the experience from all of the prior tasks. Thus, if skills have been learned, to learn a new n-th skill, we pre-load the replay buffer with trajectories from each of the n 1 prior skills and sample half of each training batch from prior data and the other half from the new agent's 's online experience. This 50-50 sampling method has been used in some prior works, including

---

RLPD [10. 58], in order to initialize online RL with offline data from the same task. However, in our system, we adapt this procedure to bootstrap a behavior from other skills. Since all of the tasks use visual observations, the generalization ability of the value function and policy networks can then learn to make use of this prior experience to assist in learning the new task. Note that it is not at all obvious that prior experience like this should be directly useful. as other tasks involve visiting very different states or manipulating different objects. However, if the networks are able to extract for example a general understanding of contacts physical interactions, then we would expect this to accelerate the acquisition of the new task.
Demonstration-Dase reset-free learning. In-between in-hand manipulation trials, the robot may drop the object and need to pick it back up again to attempt the task again. To automate training, we must also acquire an autonomous pick-up policy to serve as reset mechanism for the in-hand task, retrieving objects that may have fallen out of the hand during in-hand manipulation. We observe that the reset task is composed of essentially the same reaching, power grasping, and lifting up skills across different objects. Unlike complex manipulation tasks in the in-hand phase, a human operator can provide demonstrations for these skills more conveniently and effectively, overcoming the wide initial state distribution issue due to the fact that objects can fall to anywhere in the environment. As shown in prior work [27]. exploration is especially challenging for RI. such settings. Thus, we use behavioral cloning (BC) to train policies for the reset phase from simple demonstrations provided with 3D mouse and discrete finger closing command. Note that no demonstrations are used for the actual in-hand reorientation skill (which is difficult to teleoperate), only for the comparatively simpler reset skill, which only requires picking up the object.
Reward learning via goal images with buffer initialization. Our aim to enable our system to learn under assumptions that are reasonable outside of the laboratory: the robot should use the sen- sors and actuators available to it for all parts of the learning process, including using an autonomous reset policy and eschewing ground truth state estimation (e.g., motion capture) in favor of visual observations that are used to train an end-to-end visuomotor policy. However, this requires us to be able to evaluate a reward function for the in-hand RL training process from visual observations as well, which is highly non-trivial. We therefore use an automated method that uses goal examples provided by a person (e.g., positioning the object into the desired pose and placing it on the hand) to learn a success classifier, which then provides a reward signal for RL. Thus, for each in-hand manipulation task Tú we assume a set G; consisting of few goal images depicting the successful completion of the task. Naively training classifier and using it reward signal is vulnerable to exploitation, as RL can learn to manipulate the object so as to fool the classifier [44]. We therefore adapt VICE [44] to address this challenge, which trains an adversarial discriminator pre-defined goal images as positives (y = 1) and observation samples from the replay buffer as negatives (y = 0). However, it is necessary to adapt this method to handle our buffer initialization approach, since VICE is by design an on-policy [44]. We first summarize the VICE algorithm and the regularization techniques we employ to make it practical for vision-based training, and then discuss how we adapt it to handle buffer initialization.
A common issue with adversarial methods such as VICE is instability and mode collapse. We found strong regularization techniques based on mixup [59] and gradient penalty [60] to be essential to stabilize VICE for learning image-based tasks, and these regularizers additionally aid the RI. process by causing the classifier to produce a smoother, more shaped reward signal. The VICE classifier predicts log (gllo),, the log probability that the observation o corresponds to the goal g, which can then be used as reward signal for RL training. The VICE classifier parameterized by 8, Do, is then optimized by minimizing regularized discriminator loss:
[(x; 6) = X (xx + (1- -1-
a (oo=)lll 1)2
where the input æ is batch of evenly mixed user-defined goal images and observations collected during training, Lx and L1-A are the Binary Cross Entropy (BCE) loss terms for mixed-up samples and labels, a 10 is the weight for Gradient Penalty loss.
5

---

Figure 4: Successful rollouts in-hand object manipulation policies for the three objects: purple 3-pronged object (Pose B). black T-shaped pipe. and blue football. The boxes on the right (outlined in green) are repre- sentative user-provided success state examples for each task. Note that the autonomous pickup policy picks up the object in variety of different poses across episodes, requiring the in-hand manipulation skill to reorient into the target pose from many starting configurations.
Applying this method with buffer initialization, where prior data from other tasks and objects is included in the replay buffer, requires additional care. Naïvely, if we train a new VICE classifier with user-provided goal images for the current task as positives, then almost all previous experiences from other tasks and objects are likely to assigned a negligible reward during training, which would not result in beneficial learning signals for the RL agent. Instead, for tasks from other objects in the prior dataset, rewards are labeled using task-specific VICE classifier which was trained when that data collected for its task. These classifier rewards computed and saved prior to training a new skill, and they remain static throughout training, in contrast to the rewards for online data and offline data from the same object, which depend on the changing VICE classifier.
We hypothesize that initializing the buffer in this way with data from other objects, or other tasks for the same object, will allow the RL algorithm learn more quickly by transferring knowledge about finger-object interactions, actuation strategies for the hand, and other structural similarities between the tasks. Of course, the degree to which such transfer can happen depends the degree of task similarity, but we will show in the next section, we empirically observe improvement from prior data even when transferring to an entirely new object.
5 Experimental Results
In our experiments, we aim to study the following questions:
1. Can our system learn dexterous manipulation skills autonomously in the real world?
2. Can prior data from one object improve the sample efficiency of learning new skills with the same object?
3. Can data from different objects be transferred to enable faster acquisition of skills with new objects?
We perform experiments with objects of various shapes and colors: purple 3-pronged object, black T-shaped pipe, and blue football. For each manipulation task, collected set of 400 success example images, described in the Appendix E.

---

We also provide demonstrations per object for the reset policy to enable in-hand training. We present details of demonstration collection, training procedure, and success rates Appendix F. Each demonstration takes roughly 30 seconds to collect, totaling less than 2 hours to collect the necessary demonstrations. Please check our website https///iieeggogggee..ovvvvvvvee for videos and further experimental details.
Learning to Reorient Three-Pront Object
Training Needed Reach Target Success Rabe
20.2
Whll Clack Time (hours)
673
2.5
0.2
2.5
50
15
263
125
15.0
37.5
20.
Clock
REBOOT scratch (UTD=4)
Tasks
from scratch (AVAIL, UTD
From Scretcc (AVAIL, UTD=1)
Bulfer Initialization (Ours)
Figure 5: Learning curve showing performance as a function of training time reorienting the 3- prong object into different poses. Even though hoth method and training from scratch eventually reach success rate of 80%, our method gets there about two times faster.
Bar plot displaying the training time re- quired for each object to reach their respective target performance. Buffer initialization leads to more than 2x speedup across all of the objects compared train- ing from scratch.
Figure
Task transfer. answer Question we evaluated our method on each of the objects with varying amounts of prior data. We first trained 3-prong object manipulation policy (for goal pose call Pose A, shown in Figure 5) without prior data in order gather data to initialize training for subsequent objects/tasks. We then trained another 3-prong manipulation policy for a different goal pose (Pose B, shown in Figure 4) as well as a T-pipe manipula- tion policy, both using prior data from the first 3-prong experiment. Finally, we trained football manipulation policy using the 3-prong and T-pipe experiments as prior data. Our method's success rate is shown in Figure 6, and film strips of various manipulation pol- icy successes during training are shown in Figure 4. Our behavior- cloned reset policy was sufficient as reset mechanism for in-hand training. Furthermore, our in-hand policies are able to successfully pose the 3-prong and T-pipe objects more than 50% of the time.
Figure pronged object is approximately 60" offset from Pose B, with any leg pointing straight to the wall.
Pose
3-
To answer Question 2, consider the Pose B 3-prong experiment deseribed previously. Since reorienting to both Pose A and B uses the same 3-prong object, expect the task difficulty to be similar for both poses. A comparison between training Pose A from scratch and training Pose with a pre-loaded replay buffer is shown in Figure 5. The Pose B experiment with our method outperforms the Pose experiments training from scratch in terms of training time. Our method reaches 80% success around hours while training from scratch yields poor performance that point. takes more than 10 hours for learning from seratch to achieve comparable success rate. This suggests that method can significantly reduce training time when using prior data from the same object for new manipulation task.
Object transfer. To answer Question 3, we consider the T-pipe and football experiments described above. We compare our method learning from scratch without prior data and display the results in Figure 8. Our method with prior data from other objects is significantly faster than learning from scratch for both objects. For the T-pipe experiments, our method achieves 60% success rate 6 hours compared to 13 hours for training from scratch. Furthermore, the from-scratch runs have

---

Learning to Rearient T-Pipe
Learning to Reorient Foothall
Succoess
Clock
Wall Clock
16.8
13%
17.5
With butfeer altotion (Oung)
From seratch (AANLL LTDD-1 From sonatch (WAN), UTD lours
intrallnecien (Oung)
From wcretsh (ANNA, JTD=11 wcratch (HA0,, UTD lours
From wcnatch (UTD=44
wcrtthh (UTD-1))
Figure 8: Learning curve showing the performance objects. both cases, buffer initialization is about two times faster than learning from scratch, though particu- larly the football object is harder to reorient for all methods.
function of training time for the T-pipe and football
absolutely no success in evaluation prior to 5 hours of training, while our method achieves some initial success as early as hour into training. The football task appears to be significantly more challenging than the 3-prong and T-pipe tasks, as shown in Figure 8, with no methods performing above 30% success rate. However, our method still outperforms learning from scratch, achieving 30% success rate with 5 hours of training; the from-scratch runs required at least 16 hours of training to achieve a lower 20% success rate.
Ablation Studies. Finally, conduct ablation experiments in both simulation and the real world to compare the effects of varying the initial buffer size, the order in which the buffer is initialized, transfer learning from a trained policy, and training for an extended period of time. Results and in-depth analysis are provided in Appendix C and Appendix D.
6 Discussion, Limitations, and Future Work
We presented system for learning in- hand manipulation skills directly by training the real world with RL, without simulation, and using only onboard sensing from encoders and cameras. Our system enables sample-efficient and autonomous learning by initializing the replay buffer of an effi- cient online RL method with data from other tasks and even other objects. We extend adversarially- learned classifier-based rewards into this setting to make it possible for users to define tasks with a collection of goal images, and implement automated resets using an imitation-learmed reset pol- icy, providing pipeline for fully autonomous training. The complete system avoids any strong instrumentation assumptions, using the robot's own sensors and actuators for every part of training, providing proof-of-concept for an efficient real-world RL system that could operate outside of laboratory conditions.
Limitations: Our experimental evaluation does have number of limitations. Although we show that reusing data from one or two prior tasks improves learning efficiency, a more practical general- purpose robotic system might use data from tens, hundreds, or even thousands of skills. Evaluating the potential for such methods at scale might require additional technical innovations, buffer initialization with very large datasets will be as effective. Additionally, our evaluation is limited to in-han reorientation skills. While such skills exercise the robot's dexterity and physical capabilities, many other behaviors that require forceful interaction with the environment and other manipulation skills could require a different reset process or might require a different method for reward specification (for example to handle occlusions). Exploring these more diverse skills is an exciting direction for future work. The current manipulation setup is training with fairly robust objects where fragility or wear and tear are not major concerns. As we move to more dexterous tasks, more directed approach may be required handle fragile objects or perform tasks that require force-sensitive interaction. Studying how integrate system with tactile sensing is another exciting avenue to explore.
is unclear
8

---

Acknowledgments
This research was partly supported by the Office of Naval Research (N00014-00-1-2383), and ARO W91 19-22-111-- We would like thank llya Kostrikov for the initial versions of the simulator and codebase, and everyone at RAIL. for their constructive feedback.
References
[1] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 JEEE/RS/ international conference on intelligent robots and systems (IROS), pages 23-30. IEEE, 2017.
[2] X. B. Peng, M. Andrychowiez, W. Zaremba, and P. Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference on robotics and automation (ICRA), pages 3803-3810, IEEE, 2018.
[3] OpenAL, M. Andrychowicz, B. Baker, M. Chociej, R. Józefowicz, B. McGrew, J. W. Pachocki, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray. J. Schneider, S. Sidor, J. Tobin, P. Welinder, L. Weng, and W. Zaremba. Learning dexterous in-hand manipulation. CoRR, abs/1 081777 2018. URL
wrriii.org/aaie httg//bb/1100..00000
[4] B. D. Argall, S. Chernova, M. Veloso, and B. Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469-483, 2009.
[5] A. Hussein, M. M. Gaber, E. Elyan, and C. Jayne. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR), 50(2):1-35, 2017.
[6] T. Osa, J. Pajarinen, G. Neumann, J. A. Bagnell, P. Abbeel, and J. Peters. An algorithmic perspective imitation learning. arXiv preprint /0.iv:////106////// 2018.
[7] V. Kumar, Gupta, Todorov, and Levine. Learning dexterous manipulation policies from experience and imitation. arXiv preprint arXiv:1611.05095, 2016.
[8] H. Zhu, A. Gupta, A. Rajeswaran, S. Levine, and V. Kumar. Dexterous manipulation with deep reinforcement learning: Efficient, general, and low-cost. In 2019 International Conference on Roboties and Automation (ICRA), pages 3651-3657. IEEE, 2019.
[9] A. Nagabandi, K. Konolige, S. Levine, and V. Kumar. Deep dynamics models for learning dexterous manipulation. In Conference on Robot Learning, pages 1101-1112, 2020.
[10] P. J. Ball, L. Smith, I. Kostrikov, and S. Levine. Efficient online reinforcement learning with offline data, 2023.
[11] Z. Xu, V. Kumar, and E. Todorov. A low-cost and modular, 20-dof anthropomorphic robotic hand: Design, actuation and modeling. In 2013 13th IEEE-RAS International Conference on Humanoid Robots (Humanoids), pages 368-375. IEEE, 2013.
[12] R. Deimel and O. Brock. A novel type of compliant and underactuated robotic hand for dex- terous grasping. The International Journal of Robotics Research, 35(1-3).111-185, 2016.
[13] R. Bhirangi. A. DeFranco, J. Adkins, C. Majidi, A. Gupta, T. Hellebrekers, and V. Kumar. All the feels: A dexterous hand with large area sensing, 2023.
[14] I. Mordatch, Z. Popović, and Todorov. Contact-invariant optimization for hand manipula- tion. In Proceedings of the ACM SIGGGAAPt/Eaungraphiiss symposium computer anima- lion, pages 137-144, 2012.
[15] V. Kumar, Tassa, T. Erez, and E. Todorov. Real-time behaviour synthesis for dynamic hand- manipulation. In 2014 IEEE International Conference Robotics and Automation (ICRA), pages 6808-6815. IEEE, 2014.

---

[16] J. Kober and J. Peters. Policy search for motor primitives in robotics. Advances in neural information processing systems, 21, 2008.
[17] M. Posa, C. Cantu, and R. Tedrake. A direct method for trajectory optimization of rigid bodics through contact. The International Journal of Robotics Research, 33(1):69-81) 2014.
[18] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstra- tions. arXiv preprint arXiv: 10087, 2017.
[19] D. Jain, A. Li, S. Singhal, A. Rajeswaran, V. Kumar, and E. Todorov. Learning deep visuomo- tor policies for dexterous hand manipulation. 2019 International Conference on Robotics and Automation (ICRA), pages 3636-3643. IEEE, 2019.
[20] C. Zeng, S. Li, Y. Jiang, Q Li, Z. Chen, C. Yang, and J. Zhang. Learning compliant grasping and manipulation by telcoperation with adaptive force control, 2021.
[21] C. Zeng, S. Li, Z. Chen, C. Yang, F. Sun, and J. Zhang. Multifingered robot hand compliant manipulation based on vision-based demonstration and adaptive force control. IEEE Trans- actions on Neural Networks and Learning Systems, pages 1-12, 2022. doi:10. 1.s10..119/TNNL... 2022.3184255
[22] P. Arunachalam, Güzey, S. Chintala, and Pinto. Holo-dex: Teaching dexterity with immersive mixed reality, 2022.
[23] K. Lowrey, S. Kolev, J. Dao, A. Rajeswaran, and E. Todorov. Reinforcement learning for non-prehensile manipulation: Transfer from simulation to physical system. In 2018 IEEE International Conference on Simulation, Modeling, and Programming for Autonomous Robots (SIMPAR), pages 35-42. IEEE, 2018
[24] B. Wu, I. Akinola, J. Varley, and Allen. Mat: Multi-fingered adaptive tactile grasping via deep reinforcement learning, 2019.
[25] A. Allshire, M. Mittal, V. Lodaya, V. Makoviychuk, D. Makoviichuk, F. Widmaier, M. Wüthrich, S. Bauer, A. Handa, and A. Garg. Transferring dexterous manipulation from gpu simulation to remote real-world trifinger. arXiv preprint arXiv:2108.09779, 2021.
[26] H. Van Hoof, T. Hermans, G. Neumann, and J. Peters. Learning robot in-hand manipulation with tactile features. 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids), pages 121-127 IEEE, 2015.
[27] H. Zhu, J. Yu, A. Gupta, D. Shah, K. Hartikainen, A. Singh, V. Kumar, and S. Levine The ingredients of real-world robotic reinforcement learning, arXiv preprint arXiv:2004.1233 2020.
[28] A. Gupta, Yu, Z. Zhao, V. Kumar, A. Rovinsky, K. Xu, T. Devlin, and S. Levine. Reset-free reinforcement learning via multi-task learning: Learning dexterous manipulation behaviors without human intervention. arXiv preprint arXiv:2104.1 2021.
[29] K. Xu, Z. Hu, R. Doshi, A. Rovinsky, V. Kumar, A. Gupta, and S. Levine. Dexterous manipu- lation from images: Autonomous real-world rl via substep guidance, 2022.
[30] A. Gupta, C. Eppner, S. Levine, and P. Abbecl. Learning dexterous manipulation for a soft robotic hand from human demonstrations. In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3786-3793 IEEE, 2016.
[31] P. Mandikal and K. Grauman. Learning dexterous grasping with object-centric visual affor- dances. arXiv preprint arXiv:2009.0433) 2020.
10

---

[32] I. Akinola, J. Varley, and D. Kalashnikov. Learning precise 3d manipulation from multiple uncalibrated cameras. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 8616-4622. IEEE, 2020.
[33] S. Ross, N. Melik Barkhudarov, K. S. Shankar, Wendel, Dey, J. A. Bagnell, and M. Hebert. Learning monocular reactive UAV control in cluttered natural environments. In 2013 IEEE International Conference Robotics and Automation, 2013. doi:10. 109/ICA.. 2013.6630809.
[34] S. Reddy, A. D. Dragan, and S. Levine. Sqil: Imitation learning via reinforcement learning with sparse rewards. arXiv preprint arXiv:1905. 1005.//00 10000 2019.
[35] B. D. Ziebart, A. L. Maas, J. A. Bagnell, and A. K. Dey. Maximum entropy inverse reinforce- ment learning. AAAI, AAAI Press, 2008. ISBN 378-1-57735-368-3
[36] M. Wulfmeeerr P. Ondruska, and I. Posner. Maximum entropy deep inverse reinforcement learning. arXiv preprint arXiv:1507.0440 2015.
[37] N. D. Ratliff, J. A. Bagnell, and M. Zinkevich. Maximum margin planning. In Machine Learming. Proceedings of the Whenty-Third International Conference ICML, 2006. d0i:10. 143936.
[38] D. P. Losey and M. K. O'Malley. Including uncertainty when learning from human corrections. In Conference on Robot Learning, pages 123-132. PMLR, 2018.
[39] Y. Cui and S. Nickum. Active reward learning from critiques. 2018 IEEE international conference robotics and automation (ICRA), pages 6907-6914, IEEE, 2018.
[40] J. D. Co-Reyes, A. Gupta, S. Sanjeev, N. Altieri, J. DeNero, P. Abbeel, and S. Levine. Guiding policies with language via meta-learning CoRR, abs/1811.07883 2018. URL http://arxiv org/abs// 1at.s//111..888111
[41] V. Myers, E. Biyik, N. Anari, and D. Sadigh. Learning multimodal rewards from rankings. In Conference on Robot Learning, pages 342-352, PMLR, 2022.
[42] D. S. Brown, W. Goo, and S. Niekum. Better-hhan-dmoonsraao imitation learning via automatically-ranker demonstrations. In Conference robot learning, pages 330-359. PMLR, 2020.
[43] I. Kostrikov, D. Yarats, and R. Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint crXiv:2004.13649 2020.
[44] J. Fu, A. Singh, D. Ghosh, L. Yang, and S. Levine. Variational inverse control with events: A general framework for data-driven reward definition. arXiv preprint arXiv: 1686, 2018.
[45] K. Zolna, A. Novikov, Konyushkova, Gulcehre, Wang. Y. Aytar, M. Denil, N. de Fre- itas, and S. Reed. Offline learning from demonstrations and unlabeled experience. arXiv preprint arXiv:2011. 13885, 2020.
[46] B. Akgun, M. Cakmak, J. W. Yoo, and A. L. Thomaz. Trajectories and keyframes for kines- thetic teaching: human-robot interaction perspective. In Proceedings of the seventh annual ACM/IEEE international conference on Human-Role Interaction, pages 391-398, 2012.
[47] V. Villani, Pini, Leali, and C. Secchi. Survey on human-robot collaboration in industrial settings: Safety, intuitive interfaces and applications. Mechatronics, 55:248-266, 2018.
[48] L. Smith. J. C. Kew, T. Li, L. Luu, X. B. Peng, S. Ha, J. Tan, and S. Levine. Learning and adapting agile locomotion skills by transferring experience, 2023,

---

[49] A. Sharma, K. Xu, N. Sardana, A. Gupta, K. Hausman, S. Levine, and C. Finn. Autonomous reinforcement learning: Formalism and benchmarking, 2022.
[50] B. Eysenbach, S. Gu, J. Ibarz, and S. Levine. Leave trace: Learning reset for safe and autonomous reinforcement learning. arXiv preprint arXiv:1711.06782) 2017 2017.
[51] K. Xu, S. Verma, C. Finn, and S. Levine. Continual learning of control primitives: Skill discovery via reset-games, arXiv preprint arXiv:2011.05220 2020.
[52] A. Sharma, A. Gupta, S. Levine, K. Hausman, and C. Finn. Autonomous reinforcement learn- ing via subgoal curricula, 2021.
[53] W. Han, S. Levine, and P. Abbeel. Learning compound multi-step controllers under unknown dynamics. In 2015 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 6435-6444 IEEE, 2015.
[54] A. Sharma, A. M. Ahmed, R. Ahmad, and C. Finn. Self-improving robots: End-to-end tonomous visuomotor reinforcement learning, 2023.
[55] A. Mandlekar, D. Xu, R. Martín-Martin, Zhu, L. Fei-Fei, and S. Savarese. Human-in inten-inntthe ththe- loop imitation learning using remote teleoperation, 2020.
[56] R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. 2018.
[57] Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, Ha, J. Tan, V. Kumar, H. Zhu, A. Gupla, Abbeel, el. Soft actor-critic algorithms and applications. arXiv preprint arXiv: 18/200505, 2018.
[58] Y. Song, Zhou, A. Sekhari, A. Bagnell, Krishnamurthy, and W. Sun. Hybrid rl: Using both offline and online data can make efficient. arXiv preprint arXiv:2210006118. 2022.
[59] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz. mixup: Beyond empirical risk mini- mization, 2018.
[60] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. Courville. Improved training of wasserstein gans, 2017.
12

---

Appendix
A Evaluation Success Criteria
We evaluated the trained policy success rate every 12000 steps for the three real-world in-hand manipulation tasks considered in the paper. The success criteria are defined as follows to stay con- sistent with the goal images collected to learn the VICE classifiers:
Task
Success Criteria
3- 3-Prong Object Pose A&B
I Oany leg
gool posel 5%)
T-Shaped Pipe
{1 [verttiont,tg
O goont.poael S 50
Football
0longaaxis
0 ggoal-posel
S 50
3-Pronged Object:
Pose A is successful if any leg is pointing straight forward (to the wall) with less than or equal to 5 degrees deviations.
Pose B is successful if any leg is pointing straight backward (to the robot) with less than or equal to 5 degrees deviations.
2. T-Pipe: The T-Pipe is successful if the vertical pipe is pointing straight backward (to the robot) with less than equal to degrees deviations.
3. Toy Football: The toy football is successful if its long axis is pointing straight to both the wall and the robot with less than or equal to 5 degrees deviations.
For the reset policies, the success criteria intuitively defined whether the hand grasps and picks up the object in mady-to-manipulate pose, such that in-hand training can begin without objects falling out of the palm immediately.
13

---

B
Algorithm Details
In this section, we describe details related to our RL learning algorithms and our imitation learning algorithm and also provide hyperparameters used in experiments for each method.
REuse Data For BOOTstrapping Efficient Real-World Dexterous Manipulation
Algorithm REBOOT
1: Given: A replay buffer 'D with prior data, a set of reset demos Dreset, set of goal images G, and a start state 80-
2: Initialize empty replay buffer B, RLPD(SAC)[1) with policy Ty and value function Q
reset policy Tæ, and VICE classifier [44] De
3: Train reset policy ne using Preset via Behavior Cloning
4: for iteration j = 1,2.... do
5: Execute F to perform reset
6: Execute n in environment, storing data in the online replay buffer
7: Update the RLPD's policy and value functions Te. Qu using 50/50 batch of samples from
B and D, assigning reward based on Da using SAC [57].
8: Update the VICE classifier Da using samples from B and goal images from G, using eq2.
9: end for
Shared RL Hyperparameters
Value
Shared Images Encoder for DrQ
MobileNetV3-SSnanle --Saael Aimall-100 100 with ImageNet-INe weights
Learned Spatial Embedding
Actor Architecture
FC(256, 256) FC(256, 19)
Critic Architecture
REDQ with 10 Ensembles FC(256, 256) FC(256, 1)
Optimizer
Adam (3e-4)
Learning
Discount 7
0.99
REBOOT UTD
4
AVAIL UTD
Target Update Frequency
Actor Update Frequency
Batch size
VICE batch size
512 (256 Goals + 256 Replay Samples)
VICE Classifier Hyperparameters
Value
Optimizer
Adam {3c-4]
Learning rate
Classificr steps per iteration
Mixup Augmentation
Label Smoothing
0.2
Gradient Penalty Weight À
10
VICE update interval
episode
Classifier Architecture
SobileNerV3--Salll--e with ImageNet- K weights
Leamed Spatial Embedding
Dro5
FC(256, 256) LeakyReLU@ - Dropout(0.) 1).
FC(I)
14

---

C
Ablation Studies
Order of tasks to initialize the buffer. To investigate whether the ordering of tasks to initialize the replay buffer impacts the learning performance in our method, i.e. bootstrapping on one object's or task's data leads to better performance than others, we designed and ran two additional experiments.
The experimental setup follows our method in Figure 5a, where the robot autonomously learns to reorient the 3-prong object to pose B. In the paper. we experimented and reported the performance using replay experiences from pose A's training to bootstrap pose B. Then we bootstrapped the learning of the T-pipe and football using the 3-prong object's s data.
We consider bootstrapping the 3-prong object's pose B learning with replay data from the T- pipe and football training, while keeping the amount of prior data, replay sampling ratio, and UTD the same. We report the evaluation success rate every 12000 steps.
Learning to Reorient Three-Prong Object with Different Order of Buffer Init
0.8
Success Rate
0.6
0.4
Wall Clock Time in Hours
from scratch (AVAIL, UTD=1) at 13 Hours Pose with 60k buffer initialization from pose A
Pose B with 60k buffer initialization from T-pipe
Pose B with 60k buffer initialization from football
Figure 9: Ablation studying the effect of initializing the replay buffer with prior experience from different objects. Initializing with experience from the same object results the best performance, but initializing using football experience provides a similar benefit.
For the same task (reorient 3-prong object) under the same training hours, bootstrapping from the same object but different task data yields the best performance, initialized with football task data achieves similar results, T-pipe data's performance follows, and no buffer initialization performed the worst. We note two potentially significant differences between these objects:
1. The T-pipe fully black colored while the 3-prong object and football are more vividly colored.
2. The in-hand dexterous motions required to solve the tasks are similar between the 3-prong object and the football (planar rotation) but different from the T- pipe (vertical flipping). This can be visualized better on the project website link http:///ste..ooggg..om////wwwwbt
15

---

Initial buffer size. To investigate how different initial replay buffer sizes affect the performance, we performed additional real-world experiments for the 3-prong object reorient task. The experi- mental setup follows our method in Figure 5a, where the robot is tasked with autonomously learning to reorient the 3-prong object to pose using buffer initialization from reorienting to pose A. compare initializing the buffer with 60k 30k randomly selected transitions and apply our method.
Learning to Reorient Three-Prome Object with Different Buffer Init Size 1.0
0.8
Success Rate
0.4
0.2
0.0
0.0
2.5
5.0
7.5
10.0
12.5
15.0
17.5
20.0
Wall Clock
from scratch (AVAIL, UTD=1)
Pose with 60k buffer initialization from pose (Ours) Pose B with 30k buffer initialization from pose A (Ours)
from scratch (AVAIL, UTD=1) at 13 Hours
Figure 10: Ablation studying the effect of reducing the amount of data used for buffer initialization (30k vs. 60k transitions pre-loaded into replay buffer). Our result demonstrates that there some benefit to pre-loading with less data, but the 60k setting still learns considerably faster.
The 30k and 60k transitions used to initialize the replay buffers for these experiments were both sampled uniformly from the same Pose replay buffer (168k transitions), and both experiments use a 50/50 sampling ratio between prior and new data. However, the run initialized with 60k transitions contains more diverse replay experiences, accelerating the online sample efficiency while achieving higher success rate under the same training time.
16

---

Comparison to transfer learning. To compare whether our method is more effective at learning real-world dexterous tasks than alternative approaches such as transfer learning, we ran an additional experiment with the 3-prong object task (Pose B) by transferring a baseline policy (Pose A, UTD=1, no initialization) that trained for longer period of time (21 hours, 70% evaluation success rate). We initialized the training by reloading the actor and critics network parameters with the trained checkpoints from the baseline policy and finetuned with the same experimental setups and no replay buffer initialization. We report the evaluation success rate here.
Learning to Reorient Three-Prong Object with Alternative Approach
10
Success Rate
0.4
Wall Clock Time in Hours
Pose B with 60k buffer initialization from pose
from scratch (AVAIL, UTD=1) at 13 Hours
Pose B finetuning from pose policy w/o source data
Figure 11: Ablation studying the effect of fine-tuning previously trained policy for different goal pose with the same object, rather than pre-loading the replay buffer. We find that pre-loading the replay buffer improves sample efficiency significantly more than fine-tuning an existing policy.
While the policy transfer + finetuning approach outperformed the baseline that learns from scratch under the same training time, our method with buffer initialization still achieves the highest success rate,
17

---

Comparison of all ablations. Here we visualize a summary of all ablations and comparisons in one plot. Our method is the most sample-efficient among all experiments on reorienting the 3-prong object.
Learning Reorient Three-Prong Object
1.0
0.8
Success Rate
0.6
0.2
0.0
Wall Clock Time in Hours
Pose B with 60k buffer initialization from pose A
Pose B with 60k buffer initialization from football
Pose B with 30k buffer initialization from pose
Pose B finetuning from pose A policy wlo source data from scratch (AVAIL, UTD=1) at Hours
with buffer initialization from T-pipe
Figure 12: Evaluation plots showing the performance of checkpoints at different points in training for number of ablation experiments, all learning to reorient the 3-pronged object into Pose B. This figure compares the initial replay buffer size, data used to initialize the replay buffer, and policy initialization ablations against our method. Our method, initialized with 60k transitions from previous experiment with the same object, clearly learns faster than the ablations.
18

---

D Longer Training in Simulation
Simulation Environment: For testing and iterating our algorithms, we developed a simulation replica our real robot setup using Mujoco and dm-control. This simulation model consists of same 16 DoF 4-fingered DHand attached to DoF Sawyer robot arm as the one built the real world.
The simulation task considered here is to reposition the 3-pronged object from anywhere on the tabletop back to the center. In this environment, the robot correctly solving the task corresponds to a ground-truth episode reward -20.
evaluation/return
G Group No Buffer Init Baseline
0 Group Buffer Init (Ours)
0
-20
-40
-60
-80
-100
-120
Step
100k
200k
300k
400k
500k
Figure 13: Ground reward training steps for method and baseline without buffer initialization simulation, demonstrating the performance our method remains stable after training long period of time.
Results for simulated experiments are shown in Figure 13. The red line represents average eval performance of our method across seeds using buffer initialization (UTD=4, 60k transitions initialization, same real-world), while the brown line represents the average eval per- formance of the baseline method (UTD=1) w/o buffer initialization across 4 seeds. Both lines smoothed using an EMA of 0.9. Our method is notably more sample efficient at solving the task than the baseline method and is more stable at convergence than the baseline when trained up to 500k steps.
Results:
19

---

E
Goal Images Collection Procedure
For each task considered in the experiment section, we collect a set of 400 goal images by placing the object the palm of the robot hand in desired pose, closing the fingers for second, and executing random actions for 1.5 seconds. We repeat this procedure multiple times, collecting 25 goal images per iteration until we reach 400 total images
F Behavior-Cloned Reset Policy Details
We began demonstration collection with the 3-prong object for which we collected 160 demonstra- tions for the reset policy. We provided only 30 additional demonstrations per new object, for total of 220 reset demonstrations across all objects. This was sufficient to train a universal reset policy for all objects with a high enough success rate to enable in-hand training.
In most cases, our behavior cloned reset policy is capable of resetting the environment, or at least of making contact with the object, but there are few states where the policy unable pick up perturb the object in any way. In order avoid getting stuck attempting unsuccessful resets in these states, train two different reset policies. One is trained with reset demonstrations for multiple objects, while the other is trained with demonstrations for only the current experiment's object. For example, when running an experiment with the football, one policy is trained using reset demonstrations for the 3-pronged object, the T-shaped pipe, and the football, while the other trained only with demonstrations for the football. At start of each training episode, we select the multi-object reset policy with an 80% probability and the single-object reset policy with a 20% probability. Since the policies behave differently due being trained different data, states in which one policy might get stuck unlikely to cause the same issue for the other policy, which enables training to continue even if one of the two policies is sub-optimal.
Here we report the success rate of each reset policy measured when performing evaluations for the in-hand policies.
Objects
3- 3-Pronged Object
T-Pipe
Football
Success Rate
0.608
0.667
0.367
Table 1: Success Rate Reset Policies
The poor success rate of the toy football could be attributed to its reset success rate. While the 3 3-pronged object and the T-pipe are more challenging to reorient in-hand due to more complex geometries and more contacts during manipulation than the toy football, is harder to pick up the toy football by the robot hand due to its slim and small shape. With nearly half the success rate compared to the other two objects, the football in in-hand training had fewer opportunities to practice meaningfully. Hence, the football experiment with our method able to achieve a 30% success rate compared to the near zero success in runs without prior data initialization under the same amount of training time.
20


Reviews:
[{"review_id": "n5Z5UgTpLWv", "reviewer": "Reviewer_ufd7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces an approach to enhance the efficiency of reinforcement learning (RL) in tackling complex in-hand reorientation tasks. It addresses various challenges associated with deploying RL methods in real-world scenarios, such as sample efficiency, environment resets, and reward signals. To address these issues, the approach reuses prior data to initialize the replay buffer to improve sample efficiency. It also leverages a behavior cloning (BC) policy to reset the environment for continuous real-world training. Additionally, an adversarial discriminator is used to obtain rewards for online RL training in the real world. The authors conduct real-world experiments on in-hand manipulation of a 3-prong object, a T-shaped pipe, and a football to demonstrate the effectiveness of the approach.", "summary_of_recommendation": "After careful consideration of the points raised, I suggest a weak reject. The main concern is the focus of the experiments which primarily demonstrate the benefits of using replay buffer initialization with prior data in terms of sample efficiency, a known insight from prior works. Moreover, the paper lacks insights on how to enhance in-hand reorientation performance. Additionally, it is essential to include a comparison with state-of-the-art approaches to evaluate the proposed method's advancements. Without such a comparison, it is difficult to determine the method's competitiveness and potential impact in the field.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "In this paper, my main concerns revolve around the weaknesses identified. Furthermore, there are several important pieces of information that require clarification. \n* It is essential to explain how the success rate is measured in detail within the experimental results section.\n* Regarding Figure 5, it would be helpful to explain the meaning of UTD in the caption. \n* Regarding Figure 5(a), based on my understanding, the yellow dotted line represents pose A from scratch (solid yellow line) at 13 wall clock time in hours. However, the current plot is missing vital information as it omits the plots between hours 8 and 13, where the line increases from 0.0 to 0.8. Therefore, I suggest including this information in Figure 5(a) to provide a better understanding of the data\n\nTypos:\n- Line 113 - “””\n- Line 206 - “additional” to “additionally”", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n* This paper introduces a new system that learns in-hand reorientation skills autonomously with RL.\n* Extensive, convincing hardware experiments are shown.\n* The paper is well-structured and written.\n\nWeaknesses:\n* In the experimental results section, the emphasis is mainly on showcasing the improvement in sample efficiency resulting from the replay buffer initialization with prior data, which is already known in prior works. It would be interesting to see if a learned policy can handle unseen objects or reorient different objects.\n* No state-of-the-art comparison.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "hUqu6xGeY5", "reviewer": "Reviewer_7J5e", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This practical study explores the role of replay buffer initialization in accelerating RL skill learning, particularly by leveraging prior data from other tasks.\n\nMore specifically, this work also proposed a modified visual reward learning algorithm considering the random replay buffer initialization, which is to keep the VICE reward from other tasks static. \n\nReal-world experiments validate the effectiveness of this simple yet powerful technique, demonstrating significant improvements in sample efficiency when learning new skills, even across different object domains.", "summary_of_recommendation": "I really appreaciate the idea of using prior non-tasl related data to accelerate the skill learning. It also surprisingly accelerate the skill learning practically. However, more ablation studies are need to make this work complete.\n\n-----------------------------------------\n\nThe authors added the additional ablation studies and comparison experiments. After the Rebuttal I would change my rating to accept", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "My main concern is that the experiment part are not complete. \n\nQ1-1: Why 50/50 of current task buffer and prior tasks buffer? Maybe practically it works the best, but is there any comparison experiment that supports this choice? \n\nQ1-2: Is this 50/50 ratio fixed during the training? if so, could there be any adaptive strategy for mixing the buffer? e.g., gradually reduce the usage of previous (other tasks') buffer. \n\nQ2: I think several ablation study is missing, e.g.,: How does initialization buffer size affect the performance? In another word, for example, how 10/50/100 episode-size initial buffer affect the training performances? \n\nQ3: As this approach uses prior tasks' skills to initialize the buffer, I would like to see the comparison with transfer learning and meta-learning based solutions.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work presents an interesting and practical exploration, confirming that leveraging prior non-task specific data can significantly enhance skill-learning efficiency in real-world dexterous manipulation tasks.\n\nOne notable limitation is its confinement to image-based domains, where different tasks share the same observation space (dimension). This commonality enables the utilization of prior data effectively, possibly due to the properties of image signals. Additionally, the existence of shared visual features in the same dexterous manipulator allows for the transfer of visual-based control skills across various tasks.\n\nHowever, to further enhance the work's credibility, a more comprehensive analysis could be beneficial. Additional details regarding this aspect can be found in the questions section. Addressing these points would strengthen the overall contribution of the study.\n\nFor details please refer to the question part.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "RFaTwUN0NT_", "reviewer": "Reviewer_puFB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a novel system for learning real-world dexterous in-hand manipulation policies in an end-to-end fashion (REBOOT). By leveraging a combination of buffer initialization and reset-free learning techniques, the proposed method is able to achieve desired target goals with greater sample efficiency compared to learning policies from scratch. The method is evaluated against three different objects and is shown to have success when transferred across the different objects.", "summary_of_recommendation": "The proposed method clearly exhibits greater sample efficiency compared to learning individual in-hand policies from scratch by leveraging prior task data. Moreover, the method requires minimal assumptions, and does not require manual resets or rewards. However, the overall novelty of the system remains unclear, with more analysis and discussion needed to better justify the technical contributions of the overall system.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "Questions:\n- Does the ordering of the tasks matter for sample efficiency? E.g.: if the ordering was reversed, where the football was trained first, then the pronged items, would the resulting bootstrapped data be as useful? I could imagine that data from the pronged items may be more useful for downstream tasks as it contains clear features that can be learned.\n- How does varying the split between prior / current data for training impact the resulting policy? Is the method sensitive to the relative proportion of prior data used?\n- How is success measured? Naively I would imagine the VICE output is used as a surrogate for task success – but if this is the case, how has VICE been validated to produce correct reward annotations for the rollout frames, which may or may not be out of distribution of the frames seen from the previous replay buffer?\n- What is the success rate of the learned reset policy? The policy needs to be very robust, and achieve virtually 100% success rate in order to remove the need for a standby human operator. Were there any issues with learning the reset policy?\n\nTechnical Concerns:\n- Typo in Fig5a: Should dotted y yellow line be Pose B?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n\n- The paper is well written and easy to understand.\n- The quantitative results showcase the sample efficiency of the method when transferred to new goals and objects.\n- The end-to-end, reset free training paradigm is exciting to see, and promising for learning real-world robot dexterous manipulation at scale.\n- The method requires minimal human intervention and engineering, and does not require hand-engineered rewards.\n\nWeaknesses:\n\n- It is not clear what is the paper’s most significant contribution. It appears that minor variations of multiple pre-established methods have been compiled into a single end-to-end method for learning dexterous manipulation, which is impressive, but for such a systems paper it would be more ideal if there were stronger empirical results showcasing the strength of the proposed system. \n- One of the paper’s major focuses is the relative sample efficiency compared to learning from scratch; however, this has already been shown in the prior work cited, and while the proposed method utilizes a novel implementation (images as input applied to robot manipulation domain), that alone does not seem to warrant a strong technical contribution.\n- A clear limitation is that the proposed method still requires at least 1 task to be learned from scratch (i.e.: the task that will then seed the resulting data buffer). It is unclear how much this limits the scope of the method – given that even for relatively simple geometries (e.g.: the football), both learning from scratch and even applying REBOOT results in relatively poor success, it is unclear whether the method can be expected to perform well across a wide range of object geometries (especially given that the other two experimental objects share very similar geometries).", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "DKtenG7D3e", "reviewer": "Reviewer_tREV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper, titled REBOOT, proposes a method for reusing data for bootstrapping real-world dexterous manipulation, and creates an autonomous pipeline to reset and collect data that can be used across multiple tasks for training task-specific policies. This includes skills such as in-hand manipulation of a triangular valve, a T-shaped pipe, and a toy football. Their method trains a reset policy (via imitation learning) and an RL policy in tandem to train a system that can autonomously collect data in the real world.", "summary_of_recommendation": "After the rebuttal period, the authors produced more results indicating that their approach is not handicapped by the amount of training they performed on the real robot. After adding simulated experiments, they confirmed that their method's results were not limited by the robot hours, and that REBOOT has a stronger effect on the learned policy success rate than policy transfer and fine-tuning. For this reason, I recommend the paper as a weak accept, as it produces some interesting insights for reset-free methods and fine-tuning for dexterous grasping tasks.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "The experiments aim to answer whether this system can learn skills autonomously, whether using REBOOT improves the sample efficiency of learning new skills, and whether faster acquisition of skills is possible. While these are indeed essential questions, another one worth investigating is whether, given enough training time, if learning from scratch end-to-end (with the reset policy), can perform better than when regularized with tasks which may limit upperbound performance. This would be a good additional question to validate, and can be done with the existing runs, with longer training runs on the robot (or in simulation).", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall, the real world demonstrations of this robot platform and approach are one of the strongest points of this method. They seem to show fairly strong results in the four tasks they propose, learning within a few hours. However, for a method like this, training in simulation for speed of testing and verifying that their method can scale to the chosen tasks is fairly critical. This is especially true since some of the experimental runs require hours of real-time training to evaluate a single model, which is bound to be bottle-necked by robot hours of operation. It is highly possible that the unusual drops in performance as more training time progresses In Figures 5 and 6 is an artifact of the object resetting approach used, but this goes unmentioned in the experimental section.\nAdditionally, the link does not seem to work.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an efficient system for learning dexterous manipulation skills through sample-efficient RL and replay buffer bootstrapping. After discussion and clarification during the rebuttal phase by the authors, all the reviewers have issued 3 weak accepts and 1 accept.
Decision: Accept (Poster)