TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correction
Authors: Yunfan Jiang
Chen Wang
Ruohan Zhang
Jiajun Wu
Li Fei-Fei
Venue: CoRL 2024 (2024)
Keywords: Sim-to-Real Transfer
Human-in-the-Loop
Robot Manipulation
TL;DR: We present TRANSIC, a holistic human-in-the-loop method to tackle sim-to-real transfer for contact-rich robotic arm manipulation tasks.

Abstract: Learning in simulation and transferring the learned policy to the real world has the potential to enable generalist robots. The key challenge of this approach is to address simulation-to-reality (sim-to-real) gaps. Previous methods often require domain-specific knowledge *a priori*. We argue that a straightforward way to obtain such knowledge is by asking humans to observe and assist robot policy execution in the real world. The robots can then learn from humans to close various sim-to-real gaps. We propose TRANSIC, a data-driven approach to enable successful sim-to-real transfer based on a human-in-the-loop framework. TRANSIC allows humans to augment simulation policies to overcome various unmodeled sim-to-real gaps holistically through intervention and online correction. Residual policies can be learned from human corrections and integrated with simulation policies for autonomous execution. We show that our approach can achieve successful sim-to-real transfer in complex and contact-rich manipulation tasks such as furniture assembly. Through synergistic integration of policies learned in simulation and from humans, TRANSIC is effective as a holistic approach to addressing various, often coexisting sim-to-real gaps. It displays attractive properties such as scaling with human effort. Videos and code are available at https://transic-robot.github.io/.

TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correction
Yunfan Jiang, Chen Wang, Ruohan Zhang, Jiajun Wu, Li Fei-Fei
Stanford University
Abstract: Learning in simulation and transferring the learned policy to the real world has the potential to enable generalist robots. The key challenge of this ap- proach is address simulation-to-reality (sim-to-real) gaps. Previous methods often require domain-specific knowledge priori. We argue that straightfor- ward way to obtain such knowledge is by asking humans to observe and assist robot policy execution in the real world. The robots can then learn from humans to close various sim-to-real gaps. We propose TRANSIC, a data-driven approach to enable successful sim-to-real transfer based on a human-in-the-loop framework. TRANSIC allows humans to augment simulation policies to overcome various unmodeled sim-to-real gaps holistically through intervention and online correc- tion. Residual policies can be learned from human corrections and integrated with simulation policies for autonomous execution. We show that our approach can achieve successful sim-to-real transfer in complex and contact-rich manipula- tion tasks such as furniture assembly. Through synergistic integration of policies learned in simulation and from humans, TRANSIC is effective as a holistic ap- proach to addressing various, often coexisting sim-to-reet gaps. displays attrac- tive properties such scaling with human effort. Videos and code available at transic-robor gi github.io.
Keywords: Sim-to Real Transfer, Human-in -eeetiin-tttheeee tho-Loop. Human Correction
(a) Simulation Policy
(b) Direct Deployment
(c) Human-in-the-Loog Correction
(d) Successful Transfer with Learned Residual Policy
Figure 1: TRANSIC for sim-to-real transfer in contact-rich robotic manipulation tasks. a) and b) Naïvely deploying policies trained in simulation usually fails due various sim-to-real gaps. Here, the robot attempts to first align the light bulb with the base and then insert and screw the light bulb into the base. e) A human operator monitors robot behaviors, intervenes, and provides online correction through teleoperation when necessary. Human data are collected to train a residual policy to tackle various sim-to-real gaps in holistic manner. d) The simulation and the residual policies are integrated together during test time to achieve a successful sim-to-real transfer for contact-rich tasks, such screwing light bulb into the base.
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Introduction
Learning in simulation is a potential approach to the realization of generalist robots capable of solving sophisticated decision-making tasks [1, 2]. Learning to solve these tasks requires a large amount of training data [3-5]. Providing unlimited training supervision through state-of-the-ar simulation [6-10] could alleviate the burden of collecting data in the real world with physical robots [11, 12). Therefore, it is crucial seamlessly transfer and deploy robot control policies acquired in simulation, usually through reinforcement learning (RL), to real-world hardware. Suc- cessful demonstrations of this simulation-to-realiyy (sim-to-real) approach have been shown in dex- terous in-hand manipulation [13-17], locomotion [18-27], and quadrotor flight [28, 29].
Nevertheless, replicating similar success manipulation tasks with robotic arms remains surpris- ingly challenging, with only a few cases in simple non-prehensile manipulation [30-33], industry assembly under restricted settings [34-38], and peg swinging [39]. The difficulty mainly stems from the unavoidable sim-to-real gaps [40], including but not limited perception gap [41-43], embodiment mismatch [18, 44, 45), controller inaccuracy [46-48], and dynamics realism [49]. Tra- ditionally, researchers tackle them through system identification [18, 30, 50, 51], domain random- ization [13, 52-55), real-world adaptation [56, 57], and simulator augmentation [58-60]. Many of these approaches require explicit, domain-specific knowledge and expertise in tasks or simulators. Although for a particular simulation-reality pair, there may exist specific inductive biases that can be hand-crafted post hoc to close the sim-to-real gap [18], this knowledge is often available priori. Identifying its effects task completion is also intractable.
We argue that straightforward and feasible way for humans to obtain such knowledge is to observe and assist policy execution in the real world. humans can assist the robot to successfully accom- plish the tasks in the real world, sim-to-real gaps are effectively addressed. This naturally leads to a generally applicable paradigm that can cover different priors across simulations and realities- -uman-in-the-loop learning [61-63] and shared-autonomy [64, 65].
Our key insight is that the human-in-the-loop framework is promising for addressing the sim-to- real gaps as a whole, in which humans directly assist the physical robots during policy execution by providing online correction signals. The knowledge required to close sim-to-real gaps can be learned from human signals. We present TRANSIC (transferring policies sim-to-real by learning from online correction, Fig. 1), data-driven approach enable the successful transfer of robot manipulation policies trained with RL in simulation to the real world. In TRANSIC, once the base robot policies are acquired from simulation training, they are deployed to real robots where human operators monitor the execution. When the robot makes mistakes or gets stuck, humans interrupt and assist robot policies through teleoperation. Such human intervention data are collected to train a residual policy, after which the base policy and the residual policy are combined to solve contact- rich manipulation tasks. Unlike previous approaches that heavily rely on domain knowledge, since humans can successfully assist the robot trained in silico complete real-world tasks, sim-to-real gaps are implicitly handled and addressed by humans in a domain-agnostic manner.
To summarize, our key contribution is a novel, holistic -uman-in-thee-oop method called TRANSIC to tackle sim-to-real policy transfer for manipulation tasks. Through extensive evalua- tion, we show that our method leads to more effective sim-to-real transfer compared to traditional methods [50, 52] and requires less real-robot data compared to the prevalent imitation learning and offline RL algorithms [66-69]. We demonstrate that successful sim-to-real transfer of short-horizon skills can solve long-horizon, contact-rich manipulation tasks, such as furniture assembly.
2 Sim-to-Real Policy Transfer by Learning from Online Correction
An overview of TRANSIC is shown in Fig. 2. This section starts with a brief preliminary review, followed by a description of simulation training. We then introduce residual policies learned from human intervention and online correction and present integrated framework for deployment dur- ing testing. Lastly, we provide implementation details.

---

Operational Space Controller
Buse Policy O Human Operotor
Human in the Loop Data Collection
Correction Datust
Correst or nut?
Action Space Distillatonn
Residbal Policy Training
Sprbust Goad
Besidiaa Pollyy
Successful Transfer
Policy
Base Pulicy
Integrated Degloyment
(a) Simulation Policy Training through Action Space Distillation
(b) Residual Policy Learning from Human Correction
Figure 2: TRANSIC method overview. a) Base policies are first trained in simulation through action space distillation with demonstrations generated by RL teacher policies. Base policies take point cloud as imput reduce perception gap. b) The acquired base policies are first deployed with human operator monitoring the execution. The human intervenes and corrects through teleoperation when necessary. Such correction data are collected learn residual policies. Finally, both residual policies and base policies are integrated during test time to achieve a successful transfer.
2.1 Preliminaries
We formulate robot manipulation task cess (MDP) M = (S R, 7, Po), where S is the state space, and A is the action space. At time step a robot observes $t E S, executes an action ag and receives scalar reward from the reward function R(*;, az). The environment progresses to the next state following the transition function ((8++118,, Qz). The robot learns a parameterized policy To (-1s) to maxi- mize the expected discounted return J == Eroyy 1- X'r:] over induced trajectory distribu- tion == (so, ag. .o.). ~ Pxa. where so Po sampled from the initial state distribution and y [0, 1) is discount factor. We consider simulation and real environments two different MDPs. We adopt an intervention- -basd learning fraamwork [66, 67] where human operator can intervene and take control during the execution of the robot policy.
lnfinite-horizon discrete-time Markov Decision Pro-
2.2 Learning Base Policies in Simulation with RL
Policy Learning with 3D Representation Object geometry matters for contact-rich manipulation. For example, robot should ideally insert light bulb into the lamp base with the thread facing down. To retain such 3D information and facilitate sim-to-real transfer, propose to use point cloud the main visual modality. Typical RGB observation used in visuomotor policy training [70] suffers from several drawbacks that hinder successful transfer, such as vulnerability to different camera poses [71] and discrepancies between synthetic and real images [42]. Well-calibrated point cloud observation can bypass these issues and has been successfully demonstrated [14, 72]. During the simulation RL training phase, we synthesize point cloud observations for higher throughput. Concretely, given the synthetic point cloud of the m-th object ((m) IR xx3 transform it into the global frame through (m ((m) (R())) + (p(m)) Here, R(m) E R3x3 and p (m) R3x1 denote the object's orientation and translation in the global frame. Further, the point cloud representation of
scene with M objects is aggregated as pS... = U p(m) and subsequently used as policy input.
Action Space Distillation A suitable action abstraction is critical for efficient learning [46, 47] as well as sim-to-real transfer [48]. A high-level controller such as the operational space controller (OSC) [73] facilitates RL exploration [46] but may hinder sim-to-real transfer because it requires accurate modeling of robot parameters, such as joint friction, mass, and inertia [74]; on the other hand, a low-level action space such as the joint position ensures consistent deployment in simulation and real hardware, but renders trial-and-error RL impractical. We draw inspiration from the teacher- student framework [15, 75-77] and propose to first train the teacher policy gleacher through RL with OSC and then distill successful trajectories into the student policy or student with joint position

---

control. Specifically, we roll out geeacher and record the robot's joint position at every simulated time interval öt to construct a dataset Dieacher = [-( We then relabel actions from the end- effector's 's poses to joint positions. Such a relabeled dataset is ready to train student policies through behavior cloning. We name this approach as action space distillation and find crucial to overcome the sim-to-real controller gap. Furthermore, teacher policies directly receive privileged observations for ease of learning, while student policies learn on synthetie point-cloud inputs match real- world measurements. The student policy parameterized by 0, Th ngtudent, is trained by minimizing the loss function: Letadent = - -Rptccheer [log mg rggtudent]] + p [Ile(P)ea)) <(Psim))l' >|(]] where <6() denotes the point cloud encoder of कह ggfudent and Decd (Preal ())m)(() J.N. /i=1 is separate dataset that contains N pairs of matched point clouds in simulation and reality for regularization purpose. Further justifications of this distillation phase can be found in Appendix D.1.
2.3 Learning Residual Policies from Online Correction
Human-in-thh-Loop Data Collection Once the student policy is obtained from simulation, is used directly as the base policy xB to bootstrap the data collection. Naively deploying the base policy on real robots usually results in inferior performance and unsafe motion due to various sim-to-real gaps. In TRANSIC, the base policy is instead deployed in a way that is fully synchronized with the human operator. Concretely, at time step once af 7 is deployed, human operator needs to decide whether intervention is necessary, indicated as 1,". Intervention is not necessary for most task execution when the robot is approaching objects of interest. However, when the robot tends to behave abnormally, the human operator intervenes and takes full control through teleoperation to correct robot errors. In these cases, the robot's pre- and post-intervention states, as well as intervention indicator 1. are collected to construct the online correction dataset DH < DD U (1, qf .. post). This procedure is illustrated in Appendix Algorithm 1.
H.
Human Correction as Residual Policies Properly modeling human correction can be challeng- ing. This is because humans usually solve tasks not purely based on current observation, hence the non-Markovian decision process [68]. Therefore, directly fine-tuning the base policy x on human correction dataset DH leads to large motions and even model collapse (Sec. 3). Inspired by prior work on learning residuals to compensate for unknown dynamics and noisy observations [78-80], we propose to incorporate human correction behaviors with residual policies. Concretely, at the time intervention, a residual policy T parameterized by vá learns to predict human intervention as the difference between post- and pre-intervention robot states: a q @ qute, where θ denotes generalized subtraction. For continuous variables such as joint position, it computes the numerical difference; for binary variables such as opening and closing the gripper, it computes exclusive nor. The residual policy is then trained to maximize the likelihood human correction: gresidual -Ep logg n R (a
2.4 An Integrated Deployment Framework
In practice, we find that learning gating function to control whether to apply residual actions or not leads to better empirical performance (Appendix D.3). We call this learned gated residual policy. Denote the gating function (-). It shares the same feature encoder with the residual policy T and is jointly learned through classification on the same correction dataset D At the inference time, we effectively use as an indicator function 1, to determine whether to apply the predicted residual actions. The policy effectively being deployed to autonomously complete tasks is an integration of base policy = and residual policy R gated by g: deployed n + 1g A joint position controller is used during deployment.
2.5 Implementation Details
We use the Isaac Gym [9] simulator. Proximal policy optimization (PPO) [81] is used to train teacher policies from scratch with task-specific reward functions and curricula. Student policies

---

Stabilize
(h) Reach and Grasp
(c) Insert
(d) Screw
Figure 3: Four tasks benchmarked in this work. They are fundamental skills required assemble a square table from FurnitureBnnch [85]. The task definition can be found in Appendix C.I.
parameterized as Gaussian Mixture Models (GMMs) [68]. See Appendix A for more details. During human-in-the-loop data collection, we use Deonnexion SpaceMouse as the teleoperation interface. Residual policies use state-of-the-art point cloud encoders [82-84] and GMM as the action head. More training hyperparameters are provided in Appendix B.4.
3 Experiments
We answer the following research questions through experiments.
Q1: Does TRANSIC lead to better transfer performance while requiring less real-world data?
Q2: How effectively can TRANSIC address different types of sim-to-real gaps?
Q3: How does TRANSIC scale with human effort?
Q4: Does TRANSIC exhibit intriguing properties, such as generalization to unseen objects, pol- icy robustness, ability to solve long-horizon tasks, and other emergent behaviors?
3.1 Tasks, Baselines, and Evaluation Protocol
As shown in Fig. 3, we consider complex contact-rich manipulation tasks (Stabilize, Reach and Grasp, Insert, and Screw) that require high precision FurnitureBench [85]. These tasks are chal- lenging and ideal for testing sim-to-real transfer, since perception, embodiment, controller, and dy- namics gaps all need to be addressed. We collect 20, 100, 90, and 17 real-robot trajectories with human correction, respectively. These amount to 62, 434, 489, and 58 corrections for each task. See Appendix B.1 for the detailed system setup. We compare with three groups of baselines. 1) Tradi- tional sim-to-real methods: It includes domain randomization and data augmentation [52] ("DR. & Data Aug."), real-world fine-tuning through BC ("BC Fine-Tune") and implicit Q-learning [69] ("IQL Fine-Tune"). To estimate the performance lower bound, we also include "Direct Transfer" without any data augmentation or real-world fine-tuning.
Interactive imitation learning (IL): It includes HG-Dagee [66] and IWR [67]. 3) Learning from real-robot data only: includes BC [86], BC-RNN [68], and IQL [69] that are trained real-robot demonstrations only. All evalua- tions are conducted on the real robot and consist of 20 trials starting with different objects and robot poses. See Appendix C for details.
3.2 Results
TRANSIC is effectivee for sim-to-real transfer and requires significantly less real-world data (Q1). As shown in Fig. 4 and Table A.XI, TRANSIC achieves the best performance on average and in all four tasks with significant margins. What are the reasons for successful transfer? We observe that adding real-world human correction data does not guarantee improvement. For example, among traditional sim-to-real methods, the best baseline BC Fine-Tune outperforms DR. & Data Aug. by 7%, but IQL Fine- Tune leads to worse performance. In contrast, Transic effectively uses human correction data, which boosts average performance by 1.24x. Not only does it achieve
5

---

Direct Transfer
IQL Fine-Tune
BC
Success Rate (%)
DR. & Data Aug.
HG- HG-Dogger
BC-RNN
BC Fine- Tune
IWR
IQL
20
0
TRANSIC
Traditional Sin-to-Real
Interactive IL
Real Data Only
Figure 4: Average success rates over four benchmarked tasks. Numerical results Table A.XI.
the best transfer performance, but it also improves simulation policies the most among various sim-to-real approaches.
Furthermore, TRANSIC outperforms interactive methods, including HG-Dagger and IWR, by 0.75x average. Although both of them weigh the intervention data higher during training, find that they tend to erase the original policy and lead to catastrophic forgetting. In contrast, by incorporating human correction with separate residual policy and integrating both base and residual policies through gating, TRANSIC combines the best properties of both policies during deployment. relies on the simulation policy for robust execution most of the time; when the base policy is likely to fail, automatically applies the residual policy to prevent failures and correct mistakes.
Finally, TRANSIC only requires dozens of real-robot corrections achieve superior performance. However, methods such as BC-RNN and IQL trained on such limited number of trajectories suffer from overfitting and model collapse. TRANSIC achieves 3.6x better performance than them. This result highlights the importance of first training in simulation and then leveraging sim-to-real transfer for robot learning practitioners.
In summary, show that in sim-to-real transfer, a good base policy learned from the simulation can be combined with limited real-world data to achieve success. However, effectively utilizing human correction data to address the sim-to-real gap is challenging, especially when we want to prevent catastrophic forgetting of the base policy.
Arsct
Embodiment
TRANSIC is effective in addressing different sim-to- real gaps (Q2). We shed light on its ability to close each individual sim-to-real gap by creating five different simulation-really pairs. For each of them, intention- ally create large gaps between the simulation and the real world. These gaps are applied to real-world settings, including perception error, underactuated controller, embodiment mismatch, dynamics difference, and object asset mismatch. Note that these are artificial settings for a controlled study. See Appendix C.3 for detailed setups.
Figure 5: Robustness to different sim- to-real gaps. Numbers averaged success rates (%). Polar bars represent performances after training with data collected specifically to address par- ticular gap. Dashed lines are zero-shot performances. Shaded circles show av- erage performances.
As shown in Fig. 5, TRANSIC achieves average success rate of 77% across five different simulation-realie reality pairs with deliberately exacerbated sim-to-real gaps. This indicates its remarkable ability to close these individual gaps. In contrast, the best baseline method, IWR, only achieves an average success rate 18%. We attribute this effectiveness in addressing different sim-to-real gaps to the residual policy design. Zeng finding that residual learning an effective tool to compensate for domain factors that cannot be explicitly modeled. Furthermore, training with data specifically collected from particular setting generally increases TRANSIC's performance. However, this is not the case for IWR, where fine-
[80] echos our

---

tuning on new data can even lead to worse performance. These results show that TRANSIC is better not only in addressing multiple sim-to-real gaps as a whole but also in handling individual gaps of a very different nature.
TRANSIC
TRANSIC scales with human effort (Q3). We demon-
IWIt
Sucvess Rate (%)
strate that TRANSIC scales better with human data than the best baseline, IWR, as shown in Fig. 6 and Table A.XII. If we increase the dataset size from 25% to 75% of the full size, TRANSIC improves average by 42%. In contrast, IWR only achieves 23% relative improvement. Additionally, for tasks other than Insert, IWR performance plateaus at an early stage and even starts to decrease as more human data becomes available. We hypothesize that IWR suffers from catastrophic forgetting and struggles to properly model the behavioral modes of humans and trained robots. On the other hand,
30
Correction Dataact Size (%)
Figure 6: Scalability with human rection data. Per-task results are shown in Table A.XII.
TRANSIC bypasses these issues by learning gated residual
policies only from human correction.
Intriguing properties and emergent behaviors of
TRANSIC
IWR
TRANSIC (Q4). As shown in Fig. 7, TRANSIC achieve an average success rate of 75% when zero-shot evaluated on assembling lamp. However, IWR can only succeed once every three attempts. This evidence suggests that TRANSIC is not overfitting to a partic- ular object; instead, it has learned reusable skills for category-level object generalization. Further, with the ablation results shown in Table 1, TRANSIC exhibits intriguing properties including effective gating, policy robustness against reduced cameras and suboptimal correction data, and consistency in learned visual features. See Appendix C.5 for detailed setups and discussions. Qualitatively, TRANSIC shows several rep- resentative behaviors that resemble humans. For instance, they include error recovery, unsticking, safety-awan actions, and failure prevention, shown in Fig. A.12. Finally, demonstrate that successful sim-to-real transfer of individual skills can be effectively chained together to enable long-horizon contact-rich manipulation (Fig. 8). See videos at transic-robot, github. io.
Success Rate (%)
60
40
20
Square Table (Seen)
Lamp (Unseen)
Figure 7: objects from a new category. Success rates are averaged over tasks Reach and Grasp and Screw.
Generalization to unseen
4 Related Work
Robot Learning via Sim-to-Real Transfer Physics-based simula- tions have become driving force for developing robotic skills [6-10]. However, the domain gap between the simulators and reality not neg- ligible [40]. Successful sim-to-real transfer includes locomotion [18-
Table 1: Ablation study results. Numbers are success rates.
Reach and Grasp
Average
Stahilize
Insert
Screw
Original
81%
100%
95%
45%
85%
w/ Human Gating
82%
100%
95%
50%
85%
Reduced Cameras
80%
95%
90%
40%
95%
Noisy Correction
76%
85%
80%
45%
95%
w/o Regularization 55%
85%
55%
20%
60%
27], dexterous in-hand manipulation [13-17], and simple non-prehensile manipulation [30-39]. In
this work, we tackle more challenging sim-to-real transfer for complex whole-arm manipulation tasks and successfully demonstrate that our approach can solve sophisticated contact-rich tasks. More importantly, it requires significantly fewer real-robot data compared to the behavior cloning approach [68]. This makes solutions based on simulators and sim-to-real transfer more appealing to roboticists.

---

Sim-to-Real Gaps in Manipula- tion Tasks The sim-to-real gaps can be coarsely categorized as fol- lows: a) perception gap [41-43],
where synthetic sensory observa- tions differ from those measured in the real world; b) embodiment mismatch [18, 44, 45), where the robot models used in simulation do not match the real-world hardware precisely; c) controller inaceu- racy [46-48], meaning that the results of deploying the same com- mands differ in simulation and real hardware; and d) poor physical real- ism [49], where physical interactions such as contact and collision are poorly simulated [87]. Traditional methods to address them include system identification [18, 30, 50, 51],
Assemble lamp (160 seconds I x speed).
domain randomization [13, 52-55],
real-world adaptation [56]. and
(b) Assemble square table (550 seconds in 1x speed).
simulator augmentation [58-60].
Figure 8: a) The robot assembles a lamp. b) The robot as- sembles a square table from FurnitureBench [85]. Videos are available at transic-robot, mansic-rooot.github.. io.
However, system identification is mostly engincered on a case-by-case basis. Domain randomization suffers
from the inability to identify and randomize all physical parameters. Methods with real-worke adaptation, usually through meta-learning [88], incur potential safety concerns during the adaptation phase. In contrast, method leverages human intervention data to implicitly overcome the gap in domain-agnostic way, leading to safer deployment.
Human-in- -imna--i-TTh-Loop Robot Learning summan-in-the-loyy machine learning is a prevalent framework inject human knowledge into autonomous systems [62, 89, 90]. The recent trend fo- cuses on continually improving robots' capability with human feedback [91] and autonomously gen- erating corrective intervention data [92]. Our work further extends this trend by showing that sim-to- real gaps can be effectively eliminated by using human intervention and correction signals. In shared autonomy, robots and humans share the control authority to achieve a common goal [64, 65, 93-95]. This control paradigm has been largely studied in assistive robotics and human-robot collabora- tion [96-98]. this work, we provide novel perspective by employing it in the sim-to-real transfer of robot control policies and demonstrating importance attaining effective transfer.
5 Limitations and Conclusion
Limitations 1) Current tasks are in a single-arm tabletop scenario. Though TRANSIC can poten- tially be applied to more complicated robots with recent teleoperation interfaces [99-103]. 2) Human operators still manually decide when intervene. This could be automated using failure detection techniques [104, 105). 3) TRANSIC requires simulation policies with reasonable performance. Nevertheless, it is compatible with recent advances in synthesizing manipulation data [106. 107].
In this work, we present TRANSIC, a human-in-the-loop method for sim-to-real transfer in contact- rich manipulation tasks. We show that combining a strong base policy from simulation with limited real-world data can be effective. However, utilizing human correction data without causing catas- trophic forgetting of the base policy challenging. TRANSIC overcomes this by learning gated residual policy from small amount of human correction data. We show that TRANSIC effectively addresses various sim-ter to-rtol gaps, both collectively and individually, and scales with human effort.

---

Acknowledgments
We are grateful to Josiah Wong, Chengshu (Eric) Li, Weiyu Liu, Wenlong Huang, Stephen Tian, Sanjana Srivastava, and the SVL. PAIR group for their helpful feedback and insightful discussions. This work is in part supported by the Stanford Institute for Human-Centered AI (HAT), ONR MURI N00014-22-1-27400 ONR MURI 100014-21-1-2800 and Schmidt Sciences. Ruohan Zhang is par- tially supported by the Wu Tsai Human Performance Alliance Fellowship.
References
[1] H. Choi, C. Crump, C. Duriez, A. Elmquist, G. Hager, D. Han, F. Hearl, J. Hodgins, A. Jain, F. Leve, C. Li, Meier, D. Negrut, L. Righetti, A. Rodriguez, Tan, and J. Trinkle. On the use of simulation in robotics: Opportunities, challenges, and suggestions for moving forward. Proceedings the National Academy of Sciences, 118()))e19088561188,, 2021. doi:10.1073/pnat 1907856118. URL https: : 1907856118.
//wp///wwwwwwo http//wwwwwaa.aa.aaaaaaaaaaaaaaa wwtp..an/adi//pape.ppppppopa http:////ooaaaa....1/.//////////
[2] C. K. Liu and D. Negrut. The role of physics-based simulators in robotics. An- nual Review of Control, Robotics, and Autonomous Systems, 4(1):35-58, 2021. doi:10.1
URL https://doi.org wwtpooi.org//.... 1146/
annurev- cawpreeooo-ooooooo2220-----000
[3] P. Abbeel and A. Y. Ng. Apprenticeship learning via inverse reinforcement learn- ing. In Proceedings of the Twenty- First International Conference on Machine Learning, ICML '04, page 1, New York, NY, USA, 2004. Association for Computing Machinery. ISBN 1581138385. doi:10.1145/10053000155533 1015330. 1015430.
URL
1145/
[4] D. III, Langford, and D. Marcu. Search-based structured prediction. arXiv preprint arXiv: sriv-0907.0786, 2009.
[5] S. Yang. O. Nachum, Y. Du, Wei, Abbeel, and Schuurmans. Foundation models for decision making: Problems, methods, and opportunities. arXiv preprint arXiv: Arxiv- 2303.04129, 2023.
[6] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physies engine for model-based control. 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033, 2012.
[7] E. Coumans and Y. Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. http::/pybulll org,
[8] Y. Zhu, J. Wong, A. Mandlekar, R. Martin-Martín, A. Joshi, S. Nasiriany, and Y. Zhu. ro- bosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint arXiv: Arxiv-2009, 10xi=22009.122933 12293, 2020.
[9] V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, and G. State. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv: Arxiv-2108.10445 2021.
[10] C. Li, C. Gokmen, G. Levine, R. Martin-Martin S. Srivastava, C. Wang. J. Wong, R. Zhang, M. Lingelbach, J. Sun, M. Anvari, M. Hwang, M. Sharma, A. Aydin, D. Bansal, Hunter, K.-Y. Kim, A. Lou, C. Matthews, Villa-Renteria, J. H. Tang, C. Tang, F. Xia, S. Savarese, H. Gweon, K. Liu, Wu, and L. Fei-Fei. BEHAVIOR-IK kk: A benchmark for embodied AI with 1,000 everyday activities and realistic simulation. In 6th Annual Conference on Robot Learning, 2022. URL https: http////ppeeeeeeeeeeeeeeeeeeeeet wwweeerrveewnnee/oooee www/teetturnneee id= ddde___De8e3.....

---

[11] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan,
K. Hausman, A. Herzog, J. Hsu, J. Iharz, B. Ichter, A. Irpan, T. Jackson, S. Jesmonth,
N. J. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, I. Leal, K.-H. Lee, S. Levine, Y. Lu, U. Malla, D. Manjunath, Mordatch, O. Nachum, C. Parada, Peralta, E. Perez, K. Pertsch, J. Quiambao, K. Rao, M. Ryoo, G. Salazar, Sanketi, K. Sayed, J. Singh, S. Sontakke, A. Stone, C. Tan, H. Tran, V. Vanhoucke, S. Vega, Q. Vuong, F. Xia, T. Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv: Arxiv-22/2.06817, 2022.
[12] K. Bousmalis, G. Vezzani, D. Rao, C. Devin, A. X. Lee, M. Bauza, T. Davchev, Y. Zhou, A. Gupta, A. Raju, A. Laurens, C. Fantacci, V. Dalibard, M. Zambelli, M. Martins, R. Pevce- viciute, M. Blokzijl, M. Denil, N. Batchelor, T. Lampe, E. Parisotto, K. Zoina, S. Reed, S. G. Colmenarejo, J. Scholz, A. Abdolmaleki, O. Groth, J.-B. Regli, O. Sushkov, T. Rothörl, J. E. Chen, Y. Aytar, D. Barker, J. Ortiz, M. Riedmiller, T. Springenberg. R. Hadsell, F. Nori, and N. Heess. Robocat: A self-improving generalist agent for robotic manipulation. arXiv preprint arXiv: Arxiv-2306.1 270..2300..1706, 2023.
[13] OpenAI, Akkaya, M. Andrychowicz, M. Chociej. M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, J. Schneider, N. Tezak, J. Tworek, P. Welin- der, L. Weng, Q. Yuan, W. Zaremba, and L. Zhang. Solving rubik's cube with a robot hand. arXiv preprint arXiv: Arxiv-1910.071133 2019.
[14] T. Chen, M. Tippur, Wu, V. Kumar, E. Adelson, and P. Agrawal. Visual dexterity: In-hand reorientation of novel and complex object shapes. Science Robotics, 8(84):aadc9244, 2023. doi:10.1 100011////crbbbticssadeoo/ scirobotics 4.c.944.....dc9244...
URL https: http::/wwwwww.in..e./eee science
www.ience.rrg//oooooo.oooooo wwtp/o//aa///o/oooo/oo/ooo 1126/
[15] Y. Chen, C. Wang. L. Fei-Fei, and C. K. Liu. Sequential dexterity: Chaining dexterous policies for long-hone horizon manipulation. arXiv preprint arXiv: Arxiv-230900090 2023.
[16] H. Qi, A. Kumar, R. Calandra, Y. Ma, and J. Malik. In-hand object rotation via rapid motor adaptation. arXiv preprint arXiv: Arxiv-22 20.-22200.888777 2022.
[17] H. Qi, B. Yi, S. Suresh, M. Lambeta, Y. Ma, Calandra, and Malik. General in-hand object rotation with vision and touch. arXiv preprint arXiv: 199999930 2023.
[18] J. Tan, T. Zhang, E. Coumans, A. Iscen, Bai, D. Hafner, S. Bohez, and V. Vanhoucke. Sim-to-real: Learning agile locomotion for quadruped robots. arXiv preprint arXiv: Arxiv- 1804. 10332, 2018.
[19] Kumar, Z. Fu, D. Pathak, and Malik. RMA: rapid motor adaptation for legged robots. In D. A. Shell, M. Toussaint, and M. A. Hsieh, editors, Robotics: Science and Systems XVII, Virtual Event, July 12-16, 2021, 2021. doi:
URL https:
//doi. ongg 15607 /RSS. 2021.
[20] Z. Zhuang. Z. Fu, J. Wang. C. Atkeson, S. Schwerffeger, C. Finn, and H. Zhao. Robot parkour learning. arXiv preprint arXiv: Arxiv-2309.05663 2023.
[21] Yang, G. Yang, and X. Wang. Neural volumetric memory for visual locomotion control. arXiv preprint arXiv: Arxiv-2304.012000 2023.
[22] H. Benbrahim and A. Franklin. Biped dynamic walking using reinforcement learn- ing. Robotics and Autonomous Systems, 22(3):28--302 1997. ISSN 0921-8890. doi:
http:///boooggll.....co/22222111 com/ cttp://ccacce/arttc...//////////
URL https://wwet httt::://ww..scinccdieeeee Robot Learning: The New Wave.
[23] G. A. Castillo, B. Weng, W. Zhang, and A. Hereid. Reinforcement learning-based cascade motion policy design for robust 3d bipedal locomotion. IEEE Access, (0:20135-201-4) 2022. doi: 1tw1..1009/AC0SSS2202..155111111
10

---

[24] L. Krishna, G. A. Castillo, U. A. Mishra, A. Hereid, and S. Kolathaya. Linear policies are sufficient to realize robust bipedal walking on challenging terrains. arXiv preprint arXiv: Arxiv-2109./2666 2021.
[25] J. Siekmann, K. Green, J. Warila, A. Fern, and J. Hurst. Blind bipedal stair traversal via sim-to-real reinforcement learning. arXiv preprint arXiv: Arxiv-2/05.08328, 2021.
[26] I. Radosavovic, T. Xiao, B. Zhang, T. Darrell, J. Malik, and K. Sreenath. Real-world hu- manoid locomotion with reinforcement learning. arXiv preprint arXiv: 2xxi-233330338// 2023.
[27] Z. Li, X. B. Peng. P. Abbeel, S. Levine, G. Berseth, and K. Sreenath. Reinforcement learning for versatile, dynamic, and robust bipedal locomotion control. arXiv preprint arXiv: Arxiv- 16889, 2024.
[28] E. Kaufmann, I.. Bauersfeld, A. Loquercio, M. Müller, V. Koltun, and D. Scaramuzza. Champion-level drone racing using deep reinforcement learning. Nature, 2023. 6oi:10.1038 14158--03-004419-4 URL htttp:://dot
[29] Y. Song, A. Romero, M. Müller, V. Koltun, and D. Scaramuzza. Reaching the limit in au- tonomous racing: Optimal control versus reinforcement learning. Science Robotics, 8(82): eadg1462, 2023. do1:10.1112/sccrooooiicc doi/abs/11 1111100112200 wtt/////i/ooooiccsssssasssoaa/ae adg1462.
URL https://wwit http::/wwwwwsiieneee org/
[30] V. Lim, H. Huang, L. Chen, J. Wang, J. Ichnowski, Seita, M. Laskey, and K. Goldberg. Planar robot casting with real2sim2real self-supervised learning. arXiv preprint arXiv: Arxiv- 21 1.04844, 2021.
[31] W. Zhou and Held. Learning grasp the ungraspable with emergent extrinsic dexterity, In Liu, D. Kulic, and J. Ichnowski, editors, Conference on Robot Learning, CoRL 2022, 14-18 December 2022, Auckland, New Zealand, volume 205 of Proceedings of Machine Learning Research, pages 150-160. PMLR, 2022. URL URLhttps:/ppooceednnn mlr, press/v205/ zhou23a. html.
[32] M. Kim, J. Han, J. Kim, and B. Kim. Pre- and post-contact policy decomposition for non- prehensile manipulation with zero-shot sim-to-real transfer. arXiv preprint arXiv: Arxiv- 2309.02754, 2023.
[33] X. Zhang, S. Jain, B. Huang, M. Tomizuka, and D. Romeres. Learning generalizable pivoting skills. In IEEE International Conference on Robotics and Automation, ICRA 2023, London, UK, May 29 June 2, 2023, pages 5865-5871 IEEE, 2023. doi: 1oi:10.1109/IIRRA 109//CRA48891 2.22.. 10161271. URL https:://.. /tt///oi.ooooo org/10. 00009/0RA488891..1 2028999...2.......2....1.222 10161271.
[34] S. Kozlovsky, E. Newman, and M. 7acksenhouse. Reinforcement learning of impedance policies for peg-in-hole tasks: Role of asymmetric matrices. IEEE Robotics and Automation Letters, 7(4):10898 -09008 2022.
[35] D. Son, H. Yang, and D. Lee. Sim-to-real transfer of bolting tasks with tight tolerance. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 9056 9063,003,,11 2020. doi:10.1 109//R05555433.0000 9041109222.9944......11..
[36] B. Tang, M. A. Lin, I. Akinola, A. Handa, G. S. Sukhatme, Ramos, D. Fox, and Y. S. Narang. Industreal: Transferring contact-rich assembly tasks from simulation to reality. In E. Bekris, K. Hauser, S. L. Herbert, and J. Yu, editors, Robotics: Science and Systems XIX, Daegu. Republic of Korea, July 10-14, 2023, 2023. doi: 000110.15607/RSS.2233.XX8 https:// //op:://doi..o orgoo//////// 15607 /RSS 2023. XIX. 039.
URL

---

[37] X. Zhang, C. Wang, L. Sun, Z. Wu, X. Zhu, and M. Tomizuka. Efficient sim-to-real trans- fer of contact-rich manipulation skills with online admittance residual learning. In 7th An- nual Conference on Robot Learning, 2023. URL https: /ttppr////ppereiee. net/forum?id gFXVysXh48K.
[38] X. Zhang, M. Tomizuka, and H. Li. Bridging the sim-to-real gap with dynamic compliance tning for industrial insertion. arXiv preprint arXiv: Arxiv-2311.07499, 2023.
[39] Y. Chebotar, A. Handa, V. Makoviychuk, M. Macklin, J. Issac, N. Ratliff, and D. Fox. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. arXiv preprint arXiv: Arxiv-1110.0668,, 2018.
[40] N. Jakobi, P. Husbands, and I. Harvey. Noise and the reality gap: The of simulation in evolutionary robotics. In F. Morán, A. Moreno, J. J. Merelo, and P. Chacón, editors, Advances in Artificial Life, pages 704-720, Berlin, Heidelberg, 1995. Springer Berlin Heidelberg, ISBN 378-3-540-49226-3.
[41] E. Tzeng, C. Devin, J. Hoffman, C. Finn, X. Peng. S. Levine, K. Saenko, and T. Darrell. To- wards adapting deep visuomotor representations from simulated to real environments. ArXiv, R0V1511.07111 2015. URL https: semanticscholes wwtpecholaa..oooooruussss////s/e 1541419.
[42] K. Bousmalis, A. Irpan, P. Wohlhart, Y. Bai, M. Kelcey, M. Kalakrishnan, L. Downs, J. Ibarz, P. Pastor, K. Konolige, S. Levine, and V. Vanhoucke. Using simulation and domain adaptation to improve efficiency of deep robotic grasping. arXiv preprint arXiv: Arxiv-709.00855, 2017.
[43] D. Ho, K. Rao, Z. Xu, E. Jang, M. Khansari, and Y. Bai. Retinagan: An object-aware approach to sim-to-real transfer. arXiv preprint arXiv: 1xviv-2011.031444 2020.
[44] S. Koos, J.-B. Mouret, and S. Doncieux. Crossing the reality gap in evolutionary robotics by promoting transferable controllers. In Proceedings of the 12th Annual Conference Genetic and Evolutionary Computation, GECCO '10, page 119-126, New York, NY, USA, 2010. Association for Computing Machinery. ISBN 978 1573300788 do::10.1145/1830483 1830505. URL https: / /tti./doooooo/oo /ttp:///////o.../////1./ 1111/11804400
1830505.
[45] Z. Xie, G. Berseth, P. Clary, J. Hurst, and M. van de Panne. Feedback control for cassic with deep reinforcement learning. arXiv preprint arXiv: Arxiv-1803.05588 2018.
[46] Martin-Martín, M. Lee, R. Gardner, S. Savarese, J. Bohg, and A. Garg. Variable impedance control in end-effector space: An action space for reinforcement learning in contact-rich tasks. arXiv preprint arXiv: Arxiv-1906.0880 2019.
[47] J. Wong, V. Makoviychuk, A. Anandkumar, and Y. Zhu, Oscar: Data-drived operational space control for adaptive and robust robot manipulation. arXiv preprint arXiv: 1rxiv-2110000700 2021.
[48] Aljalbout, Frank, M. Karl, and van der Smagt. the role of the action space robot manipulation learning and sim-to-real transfer. arXiv preprint arXiv: Arxiv-2312.03673, 2023.
[49] Y. Narang, K. Storey, Akinola, M. Macklin, P. Reist, L. Wawrzyniak, Y. Guo, Á. Moravánszky, State, M. Lu, A. Handa, and D. Fox. Factory: Fast contact for robotic assembly. In K. Hauser, D. A. Shell, and S. Huang, editors, Roboties: Science and Systems XVIII, New York City, NY, USA, June 27 July 1, 2022, 2022. 100..16667/S...22222222 http://io.oorogg/// wrw/10..//10. 10011166777550 /RSS 2022. XVIII.O33
035. URL https: //tp:://ddo.ooo
[50] L. Ljung. System Identificatton,, pages 163-173. Birkhäuser Boston, Boston, MA, 1998. ISBN 978-1-4612-1768-2 doi: 10.11007977/975 10.
1-1-778-----11-1-11111-- 1768-8 11. URL https://doi. org/
12

---

[51] P. Chang and T. Padir. Sim2real2sim: Bridging the gap between simulation and real-world in flexible object manipulation. arXiv preprint arXiv: Arxiv-2002.02534, 2020.
[52] X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. arXiv preprint arXiv: Arxiv-1710.06337, 2017.
[53] Handa, Allshire, V. Makoviychuk, A. Petrenko, R. Singh, J. Liu, D. Makoviichuk, V. Wyk, A. Zhurkevich, B. Sundaralingam, and Y. S. Narang, Dextreme: Transfer of agile in-hand manipulation from simulation to reality. In IEEE International Conference Robotics and Automation, ICRA 2023, London, UK, May 29 June 2, 2023, pages 5977- 5984. IEFE, 2023. 10i: 109/100 0000//CRRA88112222...000 2023 10160216.
URI Btl.httss://do. /tt.::::://iii.//111....... org/10.
[54] J. Wang, Y. Qin, K. Kuang, Y. Korkmaz, A. Gurumoorthy, H. Su, and X. Wang. Cyberdemo: Augmenting simulated human demonstration for real-world dexterous manipulation. arXiv preprint arXiv: Arxiv-2402.14777 14795, 2024.
[55] T. Dai, J. Wong, Y. Jiang, C. Wang, C. Gokmen, Zhang, J. Wu, and Fei-Fei. ACDC: Automated creation of digital cousins for robust policy learning, In 8th Annual Conference Robot Learning, 2024. URL https: http://ppareeeeeeeeeeererrrreeea
[56] G. Schoettler, A. Nair, J. A. Ojea, S. Levine, and E. Solowjow. Meta-reinforrcmmnt learning for robotic industrial insertion tasks. arXiv preprint arXiv: Arxiv-2004. 2020.
[57] Y. Zhang, L. Ke, A. Deshpande, A. Gupta, and S. Srinivasa. Cherry-Picking with Reinforce- ment Learning. In Proceedings of Robolics: Science and Systems, Daegu, Republic of Korea, July 2023. doi: 226677/RSS.2023XXXX....
[58] Y. Chebotar, A. Handa, Makoviychuk, M. Macklin, J. Issac, N. D. Ratliff, and D. Fox. Closing the sim-to-real loop: Adapting simulation randomization with real world experi- ence. In International Conference on Robotics and Automation, ICRA 2019, Montreal, QC. Canada, May 20-24, 2019, pages
IEEE, 2019. 1o0::00.1991ICAA2211995999 URL https: /tw.///oioooo//o org/ http:/0...11////111 2019 8793789.
[59] J. P. Hanna and P. Stone. Grounded action transformation for robot learning in simulation. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI' 17, page 4931 4932. AAAI Press, 2017.
[60] E. Heiden, D. Millard, E. Coumans, and G. S. Sukhatme. Augmenting differentiable simula- tors with neural networks to close the sim2real gap. arXiv preprint urXiv: Arxiv-2007.0600 2020.
[61] A. Cruz and T. Igarashi. survey on interactive reinforcement learning: Design principles and open challenges. arXiv preprint arXiv: Arxiv-2105.129445 2021.
[62] Y. Cui, P. Koppol, H. Admoni, S. Niekum, R. Simmons, A. Steinfeld, and T. Fitzgerald. Understanding the relationship between interactions and outcomes in human-in-the-loop ma- chine learning. In Z.-H. Zhou, editor, Proceedings of the Thirtieth International Joint Con- ference on Artificial Intelligence, IJCAI-21, pages ences on Artificial Intelligence Organization, 2021. doi: 200.22444/aia..222222533 2021/599. URL https://doi. org/ /0/./11..02 24963/ ijcai. 2021/599. Survey Track.
International Joint Confer-
deep reinforcement learning tasks. arXiv preprint arXiv: Arxiv-1909.09900 2019. [63] Zhang, F. Torabi, L. Guan, D. Ballard, and P. Stone. Leveraging human guidance for
[64] S. Javdani, S. S. Srinivasa, and J. A. Bagnell. Shared autonomy via hindsight optimization. arXiv preprint arXiv: Arxiv-1503.06119 2015.
13

---

[73] O. Khatib. A unified approach for motion and force control of robot manipulators: The operational space formulation. JEEE Journal on Robotics and Automation, 3(1):43-55 1987. doi: 109..1109///A.19999 [65] S. Reddy, A. D. Dragan, and S. Levine. Shared autonomy via deep reinforcement learning. arXiv preprint arXiv: Arxiv-1802.01744, 2018.
[66] M. Kelly, C. Sidrane, K. Drigg-Cammbell. and M. J. Kochenderfer. Hg-dagger: Interactive imitation learning with human experts. arXiv preprint arXiv: Arxiv-1810.02890, 2018.
[67] A. Mandlekar, D. Xu, R. Martin-Martin Y. Zhu, L. Fei-Fei, and S. Savarese. Human-in-the loop imitation learning using remote teleoperation. arXiv preprint arXiv: Arxiv-2012.06733, 2020.
[68] A. Mandlekar, D. Xu, J. Wong,
S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Zhu, and R. Martin-Martín. What matters in learning from offline human demonstrations for robot manipulation. arXiv preprint arXiv: Arxiv-2 dexin-2108.0329. 2021.
[69] I. Kostrikov, A. Nair, and S. Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv: Arxiv-2110.06169 2021.
[70] Y. Zhu, Z. Wang, J. Merel, A. Rusu, T. Erez, S. Cabi, S. Tunyasuvunakool, J. Kramár, R. Had- sell, N. de Freitas, and N. Heess. Reinforcement and imitation learning for diverse visuomotor skills. arXiv preprint arXiv: 1rxiv-1802.09. 2018.
[71] Xie, Lee, Xiao, and C. Finn. Decomposing the generalization gap in imitation learning for visual robotic manipulation. arXiv preprint arXiv: Arxi--2307.3365, 2023. 2023.
[72] Y. Qin, B. Huang. Z.-H. Yin, H. Su, and X. Wang, Dexpoint: Generalizable point cloud reinforcement learning for sim-to-real dexterous manipulation. arXiv preprint arXiv: Arxiv- 221 1. ..9423, 2022.
[74] J. Nakanishi, R. Cory, M. Mistry, J. Peters, and S. Schaal. Operational space control: A theoretical and empirical comparison. The International Journal of Robotics Research, 27 (6):737-757, 2008. doi: 10.:1101177/0777880 200000..17770278334998000001 0278364908091463.
URL https: http::://ooooooooo/o /t.gg/10.1177
[75] A. Rusu, S. G. Colmenarejo, C. Gulcehre, Desjardins, Kirkpatrick, R. Pascanu, V. Mnih, K. Kavukeuoglu, and Hadsell. Policy distillation. arXiv preprint arXiv: Arxiv- 15/6295 2015.
[76] T. Chen, M. Tippur, S. Wu, V. Kumar, E. Adelson, and P. Agrawal. Visual dexterity: In-hand dexterous manipulation from depth. arXiv preprint arXiv: 1,44442211.11744,, 2022. 2022.
[77] T. Chen, J. Xu, and P. Agrawal. A system for general in-hand object re-orientation. In Faust, D. Hsu, and Neumann, editors, Conference on Robot Learning, 8-11 November 2021, London, UK, volume 164 of Proceedings of Machine Learning Research, pages 297- 307. PMLR, 2021. URL https: httpa/////cccceoe....e/ mlr arres//v164/che2222 html.
[78] T. Johannink, S. Bahl, A. Nair, J. Luo, A. Kumar, M. Loskyll, J. A. Ojea, E. Solowjow, and S. Levine. Residual reinforcement learning for robot control. arXiv preprint arXiv: Arxiv- 18/2.03201, 2018.
[79] T. Silver, K. Allen, J. Tenenbaum, and L. Kaelbling. Residual policy learning. arXiv preprint arXiv: Arxiv-1812.06298, 2018.
[80] A. Zeng, S. Song, J. Lee, A. Rodriguez, and T. Funkhouser. Tossingbot: Learning to throw arbitrary objects with residual physics. arXiv preprint arXiv: 1xxiv-193.11223 2019.
14

---

[81] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimiza- tion algorithms. arXiv preprint arXiv: Arxiv-1707.06344 2017.
[82] Qi, H. Su, K. Mo, and J. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. arXiv preprint arXiv: Arxiv-1612.00593, 2016.
[83] A. Jaegle F. Gimeno, A. Brock, A. Zisserman, O. Vinyals, and Carreira, Perceiver: General perception with iterative attention. arXiv preprini arXiv: Arxiv-2103.03206, 2021.
[84] J. Lee, Y. Lee, J. Kim, A. R. Kosiorek, Choi, and Y. W. Teh. Set transformer: frame- work for attention-basse baasd permutation-invarian neural networks. arXiv preprint arXiv: Arxiv- 1810.00825, 2018.
[85] M. Hco, Y. Lee, D. Lee, and J. J. Lim. Furniturebench: Reproducible real-world benchmark for long-horizon complex manipulation. In K. E. Bekris, K. Hauser, L. Herbert, and J. Yu, editors, Robotics: Science and Systems XIX, Daegu, Republic of Korea, July 10-14, 2023, 2023. dot:100.1667//SSS.22222 XIX.041. URL attps://doi.org// 1etrg//0..560//RS5 2023.
XIX. 041.
[86] D. A. Pomerleau. Alvinn: An autonomous land vehicle in neural network. In D. Touretzky, editor, Advances in Neural Information Processing Systems, volume 1. Morgan-Kaufmann, 1988. URL
mew.eeiidiinnnnnnnnneeee.eeeeeee hccpppppe..illl/ppppr/////////// pdf.
[87] P. C. Horak and J. C. Trinkle. On the similarities and differences among contact models in robot simulation. IEEE Robotics and Automation Letters, 4(2):493-499, 2019, 10
10. 109/
LRA.2019.2891088
[88] Finn, Abbeel, and S. Levine. Model-agnostie meta-learning for fast adaptation of deep networks, arXiv preprint arXiv: Arxiv-1703.03440 2017.
[89] J. A. Fails and D. R. Olsen. Interactive machine learning. In Proceedings of the 8th In- ternational Conference on Intelligent User Interfaces. IUI '03, page 49-45, New York, NY, USA, 2003. Association for Computing Machinery. ISBN 1581 135866. doi: 1ett1...1455560 1455604005. 604056. URL https: //doi. org/10. 1145/604045...c.m 604404
[90] S. Amershi, M. Cakmak, W. B. Knox, and T. Kulesza. Power the people: The role of humans in interactive machine learning. Al Magazine, 3554):005-120, Dec. 2014. doi: (00.1600/aimaa.335542222 URL https:///jjs aaai ortp/aimagaazin//ndee..r php/
[91] Liu, Nasiriany, Zhang, 7.. Bao, and Zhu. Robot learning on the job: Human-in- thop autonomy and learning during deployment. arXiv preprint arXiv: Arx14-2211.08416, 2022.
[92] R. Hoque, A. Mandlekar, C. R. Garrett, K. Goldberg, and D. Fox. Interventional data gen- eration for robust and data-efficient robot imitation learning. In First Workshop on Out-of- Distribution Generalization in Robotics at CoRL 2023, 2023. URL http::///opeerreiees bet/forrm?iddccKFRee
[93] J. Crandall and M. Goodrich. Characterizing efficiency of human robot interaction: study of shared-control teleoperation. In IEEF/RSJ International Conference Intelligent Robots and Systems, volume 2, pages 1290-1295 vol.2, 2002. doi: 10..110//RDDS.000 2002. 1043932.
[94] A. Dragan and S. Srinivasa. A policy-blending formalism for shared control. The International Journal of Robotics Research. 22(7):90-8055 2013. doi:10.1177/ 0278364913490324, URL https: /wi. org/ /tt0//1111170722222691111993333
15

---

[95] D. Gopinath, S. Jain, and B. D. Argall. Human in-the-loop optimization of shared autonomy in assistive robotics. IEEE Robotics and Automation Letters, 2(1):247-254, 2017. doi: 10. 109/LRAA2016.2593992
[96] A. D. Dragan and S. S. Srinivasa. Formalizing assistive teleoperation, volume 376. MIT Press, July, 2012.
[97] H. J. Jeon, D. P. Losey, and D. Sadigh. Shared autonomy with learned latent actions. arXiv preprint arXiv: Arxiv-2005.03210 2020.
[98] Y. Cui, S. Karamcheti, R. Palleti, N. Shivakumar, P. Liang, and D. Sadigh. "no, to the right" online language corrections for robotic manipulation via shared autonomy. arXiv preprint arXiv: Axxiv-2301.02555, 2023.
[99] P. Wu, Y. Shentu, Z. Yi, X. Lin, and P. Abbeel. Gello: general, low-cost, and intuitive teleoperation framework for robot manipulators. arXiv preprint arXiv: Arxiv-2309. 13037, 2023.
[100] C. Wang, H. Shi, W. Wang. R. Zhang, L. Fei-Fei, and C. K. Liu. Dexcap: Scalable and portable mocap data collection system for dexterous manipulation. arXiv preprint arXiv: Arxiv-2403.07788 2024.
[101] Z. Fu, T. Z. Zhao, and C. Finn. Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation. arXiv preprint arXiv: Arxiv-2401.0211 22201.0211,, 2024.
[102] S. Dass, W. Ai, Y. Jiang. S. Singh, J. Hu, R. Zhang, P. Stone, B. Abbatematteo, and R. Martín- Martín. Telemoma: A modular and versatile teleoperation system for mobile manipulation. 2024.
arXiv preprint arXiv: Arxiv-2403.0743
[103] T. He, Z. Luo, W. Xiao, C. Zhang, K. Kitani, C. Liu, and G. Shi. Learning human-to- humanoid real-time whole-body teleoperation. arXiv preprint arXiv: Arxiv-2403.04336, 2024.
[104] H. Liu, Dass, R. Martín-Martín and Zhu. Model-based runtime monitoring with inter- active imitation learning. arXiv preprint arXiv: Arxiv-2310. 17552, 2023.
[105] Z. Liu, A. Bahety, and S. Song. Reflect: Summarizing robot experiences for failure explana- tion and correction. arXiv preprint arXiv: Arxiv-2306.15724 2023.
2023.
[106] A. Mandlekar, Nasiriany, B. Wen, Akinola, Narang, L. Fan, Y. Zhu, and D. Fox. Mimicgen: A data generation system for scalable robot learning using human demonstrations. arXiv preprint arXiv: Arxiv-2310. 17596, 2023.
[107] L. Ankile, A. Simeonov, I. Shenfeld, and P. Agrawal. Juicer: Data-efficient imitation learning for robotic assembly. arXiv preprint arXiv: SYxY-22404.3329, 2024.
[108] M. N. Mistry and L. Righetti. Operational space control of constrained and underactuated systems. In Robotics: Science and Systems, 2011. URL https:://ppl stw.aiiicssmmatncccoocl..a.
org/CorpusID: 10392712.
[109] D.-A. Clevert, Unterthiner, and Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv: Arxiv-151 2015.
[110] D. Makoviichuk and V. Makoviychuk. rl-games: A high-performance framework for rein- forcement learning. https://github http:///gttuuu...m//eeer//// com./Deenys8/rrlgaaeee. May 2021.
[111] J. Schulman, P. Moritz, Levine, M. Jordan, and Abbeel. High-dimensional continuous
control using generalized advantage estimation. arXiv preprint arXiv: 2015.
16

---

[112] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: Arxiv-14/2.6980, 2014.
[113] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Comput., 9(8): 1735-1780, nov 1997. ISSN 0899-7667. doi: 19w.:/9....9........... URL https: //doi.org/10. 1 1162/necoo 1997.9 8..8..7.
[114] D. Hendrycks and K. Gimpel. Gaussian enror linear units (gelus). arXiv preprint arXiv:
Arxiv-1606.08415, 2016.
[115] Y. Zhu, Joshii, P. Stone, and Y. Zhu. Viola: Imitation learning for vision- manipulation with object proposal priors. 6th Annual Conference Robot Learning, 2022.
[116] C. R. Qi. L. Yi, H. Su, and J. Guibas. Pointnet++: Deep hierarchical feature learning on point sets in metric space. arXiv preprint arXiv: Arxiv-1706.022 02413, 2017.
[117] I. Loshchilov and Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv: Arxiv-1608. 03983, 2016.
[118] A. Fishman, A. Murali, C. Eppner, B. Peele, B. Boots, and D. Fox. Motion policy networks. In Liu, D. Kulic, and J. Ichnowski, editors, Conference on Robot Learning, CoRL. 2022, 14-18 December 2022, Auckland, New Zealand, volume 205 of Proceedings of Machine Learning Research, pages 967-977. PMLR, 2022. URL httph:tpp:/pptt httpps:/prooceedings mlr, press/v205/ Fishman23a.htar html.
[119] Hoque, A. Balakrishna, Novoseller, Wilcox, D. Brown, and K. Goldberg. Thriftydagger: Budget-aware novelty and risk gating for interactive imitation learning. arXiv preprint arXiv: Arxiv-2109.082771 2021.
[120] C. Chi, S. Feng. Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and Song. Diffusion policy: Visuomotor policy learning via action diffusion. In K. F. Bekris, K. Hauser, S. L. Herbert, and J. Yu, editors, Robotics: Science and Systems XIX, Daegu, Republic of Korea, July 10-14, 2023, 2023, 150.10..500//RSS..2000 2023. XIX.026.
URL sotps://doo org/10. 15607/RSS,
[121] A. Abdolmaleki, J. T. Springenberg, Y. Tassa, R. Munos, N. Heess, and M. Riedmiller. Max- imum a posteriori policy optimisation. arXiv preprint arXiv: Arxiv-1806.0690 2018.
[122] R. Tedrake and the Drake Development Team. Drake: Model-based design and verification for robotics, 2019. URL
http:://rraeeeeeeeeeeeeeeeee mit. edu.
[123] F. Xiang. Y. Qin, K. Mo, Y. Xia, H. Zhu, Liu, M. Liu, H. Jiang, Y. Yuan, H. Wang, L. Yi, A. X. Chang, L. J. Guibas, and H. Su. Sapien: A simulated part-based interactive environment. arXiv preprint arXiv: 1xxiv-2003.08513 2020.
[124] M. Mittal, C. Yu, Q. Yu, J. Liu, N. Rudin, D. Hoeller, J. Yuan, P. P. Tehrani, Singh, Y. Guo, H. Mazhar, A. Mandlekar, B. Babich, G. State, M. Hutter, and A. Garg. Orbit: A unified simulation framework for interactive robot learning environments. arXiv preprint arXiv: 200--330/.04999 2023.
[125] A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, A. Wahid, V. Sindhwani, and J. Lee. Transporter networks: Rearranging the visual world for robotic manipulation. arXiv preprint arXiv: Arxiv-2010. 2440110000.,4406 2020.
[126] M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manip- ulation. arXiv preprint arXiv: Arxiv-2109.1220 12098, 2021.
[127] Y. Jiang, A. Gupta, Z. Zhang, G. Wang, Y. Dou, Y. Chen, L. Fei-Fei, A. Anandkumar, Y. Zhu, and L. Fan. Vima: General robot manipulation with multimodal prompts. arXiv preprint arXiv: Arxiv-2210.0300 2022.
17

---

[128] M. Shridhar, L. Manuelli, and D. Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. arXiv preprint arKiv: Sxxiv-2209.0545/1 2022.
[129] D. Batra, A. X. Chang, S. Chernova, A. J. Davison, J. Deng, V. Koltun, S. Levine, J. Malik,
I. Mordatch, R. Mottaghi, M. Savva, and H. Su. Rearrangement: A challenge for embodied ai. arXiv preprint arXiv: Arxiv-20/1.01975, 2020.
[130] J. Gu, D. S. Chaplot, H. Su, and J. Malik. Multi-skill mobile manipulation for object rear- rangement. arXiv preprint arXiv: Arxiv-2209.0277 2022.
[131] S. Yenamandra, A. Ramachandran, K. Yadav, A. Wang, M. Khanna, T. Gervet, T.-Y. Yang, V. Jain, A. W. Clegg, J. Turner, Z. Kira, M. Savva, A. Chang, D. S. Chaplot, D. Batra, R. Mottaghi, Y. Bisk, and C. Paxton. Homerobot: Open-vocabulary mobile manipulation. arXiv preprint arXiv: 23x6v 11565, 2023.
[132] K. Ehsani, T. Gupta, R. Hendrix, J. Salvador, L. Weihs, K.-H. Zeng, K. P. Singh, Y. Kim, W. Han, A. Herrasti, R. Krishna, D. Schwenk, E. VanderBilt, and A. Kembhavi. Imitating shortest paths in simulation enables effective navigation and manipulation in the real world. arXiv preprint arXiv: Arxiv-2312. 02976, 2023.
[133] Y. Wu, W. Yan, T. Kurutach, L. Pinto, and P. Abbeel. Learning to manipulate deformable objects without demonstrations. arXiv preprint arXiv: Arxiv-1910.13439, 2019.
[134] H. Ha and S. Song. Flingbot: The unrcasonable effectiveness of dynamic manipulation for cloth unfolding. arXiv preprint arvi:: Arxiv- Axxiv-2105.03655 2021.
[135] D. Seita, Y. Wang, S. J. Shetty, E. Y. Li, Z. Erickson, and D. Held. Toolflownet: Robotic manipulation with tools via predicting tool flow from point clouds. arXiv preprint arXiv: Arxiv-2211.09006, 2022.
[136] X. Lin, Z. Huang, Y. Li, J. B. Tenenbaum, D. Held, and C. Gan. Diffskill: Skill abstraction from differentiable physics for deformable object manipulations with tools. arXiv preprint arXiv: Arxiv-2203. 17275, 2022.
[137] Y. Ji, Z. Li, Y. Sun, X. B. Peng, S. Levine, G. Berseth, and K. Sreenath. Hierarchical re- inforcement learning for precise soccer shooting skills using a quadrupedal robot. arXiv preprint arXiv: Arxiv-2208.01160) 2022.
[138] Y. J. Ma, W. Liang, G. Wang, D.-.. Huang, O. Bastani, D. Jayaraman, Y. Zhu, L. Fan, and A. Anandkumar. Fureka: Human-level reward design via coding large language models. arXiv preprint arXiv: Arxiv-2310.129311 2023.
[139] A. Boeing and T. Bräunl. Leveraging multiple simulators for crossing the reality gap. In 2012 12th International Conference on Control Automation Robotics & Vision (ICARCV), pages 1113-1119, 2012. 1oi:10.1 100.100100/ICARC..2222..6555
[140] J. Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter. Learning agile and dynamic motor skills for legged robots. Science Robotics, 4a26us7a,5872, 2019. doi: 2wt///iroootiss.aau5888. 1126/ 1126/sciiz /122//ccroootics aau5872.
URL https://www.eet wwwp:/wwww......w......w. org wrw./oi/////aeooooooooooooossooe
[141] J. Luo, Solowjow, C. forcement learning on variable impedance controller for high-precision robotic assembly. In International Conference on Robotics and Automation, ICRA 2019, Montreal, QC, Canada, May 20-24, 2019, pages 3080-3087. IEEE, 2019. doi: 1tt.:0.1100//CA.2119..99. https: /oi. org/ 10. 1109/ IIRA. 2019. 8793506.
Wen, J. A. Ojea, A. M. Agogino, A. Tamar, and P. Abbeel. Rein-
URL
18

---

[142] Y. Yue, J. Broder, R. Kleinberg, and T. Joachims. The k-armed dueling bandits prob- lem. Journal of Computer and System Sciences, 78(3):138-1556, 2012. ISSN 0022-0000. oot.tt////oooogggoo...oosssss...
URL https://www http::::www.scinnddireeee. sciencedirect. com/
ssiimc//Arriicee//////2000011222 JCSS Special Issue: Cloud Computing 2011.
[143] Jain, B. Wojeik, Joachims, and A. Saxena. Learning trajectory preferences for manipu- lators via iterative improvement. arXiv preprint arXiv: Arxiv-1306.6294, 2013.
[144] P. Christiano, J. Leike, B. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. arXiv preprint arXiv: Arxiv-1706.03741, 2017.
[145] E. Bryık, D. P. Losey, M. Palan, N. C. Landolfi, G. Shevchuk, and D. Sadigh. Learning reward functions from diverse sources of human feedback: Optimally integrating demonstrations and preferences. arXiv preprint arXiv: Arxiv-2006. 14091, 2020.
[146] K. Lee, L. Smith, and P. Abbeel. Pebble: fredback-efffcciee interactive reinforcement learn- ing via relabeling experience and unsupervised pre-training. arXiv preprint arXiv: Arxiv- 2106.05091, 2021.
[147] X. Wang, K. Lee, K. Hakhamaneshi, Abbeel, and M. Laskin. Skill preferences: Learning to extract and execute robotic skills from human feedback. arXiv preprint arXiv: Arxiv- 2 2/08.05 05388, 2021.
[148] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agar- wal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, Leike, and R. Lowe. Training language models to follow in- structions with human feedback. arXiv preprint arXiv: Arxiv-2203.021555 2022.
[149] V. Myers, E. Bıyık, and D. Sadigh. Active reward learning from online preferences. arXiv preprint arXiv: Arxiv-2302.13500 2023.
[150] R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn. Direct preference optimization: Your language model is secretly reward model. arXiv preprint arXiv: Arxiv- 23305/82900 2023.
[151] J. Hejna, R. Rafailov, H. Sikchi, C. Finn, S. Niekum, W. B. Knox, and D. Sadigh. Contrastive preference learning: Learning from human feedback without rl. arXiv preprint arXiv: Arxiv- 13139, 2023.
[152] W. B. Knox and P. Stone. Reinforcement learning from human reward: Discounting in episodic tasks. In 2012 IEEE RO-MAN: The 21 IEEE International Symposium on Robot and Human Interactive Communication, pages 878-885, 2012. 16:110.1199/RMMAN.000. 6343862.
[153] B. D. Argall, E. L.. Sauser, and A. G. Billard. Tactile guidance for policy refinement and reuse. In 2010 IEEE 9th International Conference Development and Learning, pages 7-12, 2010. doi: 100.101000ELL...10..05.077
[154] T. Fitzgerald, E. Short, A. Goel, and A. Thomaz. Human-guided trajectory adaptation for tool transfer. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS '19, page 1350-1358, Richland, SC, 2019. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 9781450363099.
[155] A. V. Bajesy, D. P. Losey, M. K. O'Malley, and A. D. Dragan. Learning robot objectives from physical human interaction. In Conference on Robot Learning, 2017. URL https: / //api. semanticscoolar hwtpeccholaaroorooorpaeseeeee (000oppsSDD222402222
19

---

[156] A. Najar, O. Sigaud, and M. Chetouani. Interactively shaping robot behaviour with unlabeled human instructions. arXiv preprint arXiv: Arxiv-1902.01670, 2019.
[157] N. Wilde, E. BIyIk, D. Sadigh, and S. L. Smith. Learning reward functions from scale feed- back. arXiv preprint arXiv: Arxiv-2110.00284 2021.
[158] J. Zhang and K. Cho. Query-efficient imitation learning for end-to-end autonomous driving- arXiv preprint arXiv: 1rxiv-6605.064500 2016.
[159] W. Saunders, G. Sastry, A. Stuhlmueller, and O. Evans. Trial without error: Towards safe reinforcement learning via human intervention. arXiv preprint arXiv: Arxi--1007.05133, 2017.
[160] Z. Wang, X. Xiao, B. Liu, G. Warnell, and P. Stone. Appli: Adaptive planner parameter learning from interventions. arXiv preprint arXiv: Arxiv-2011.00400, 2020.
[161] C. Celemin and J. R. del Solar. An interactive framework for learning continuous actions policies based corrective feedback. Journal of Intelligent & Robotic Systems, 95:77- 97, 2018. 1oi: 01w.:1000007/110000000-----83830
URL http:// http://ink...siigee www.ii..ppiingee..o//rttt....... -13- 1tt.
10.
[162] Z. Peng. W. Mo, C. Duan, Q. Li, and B. Zhou. Learning from active human involvement through proxy value propagation. In Thirty-seventh Conference on Neural Information Pro- cessing Systems, 2023. URL http:///paaaaaeeeeeeeeeeeerrrrrp
[163] Celemin, Pérez-Dattari, Chisari, G. Franzese, L. Souza Rosa, Prakash,
Z. Ajanović, M. Ferraz, A. Valada, and J. Kober. Interactive imitation learning in robotics: A survey. arXiv preprint arXiv: 2eiv-22211.06000 2022.
20

---

A Simulation Training Details
In this section, we provide details about simulation training, including the used simulator backend, task designs, reinforcement learning (RL) training of teacher policy, and student policy distillation.
A.1 The Simulator
We use Isaac Gym Preview 4 [9] as the simulator backend. NVIDIA PhysX' is used as the physics engine to provide realistic and precise simulation. Simulation settings are listed in Table A.I. The robot model is from Franka ROS package". We borrow furniture models from FurnitureBench [85] to create various tasks that require complex and contact-rich manipulation.
Table A.I: Simulation settings.
Hyperparameter
Value
Simulation Frequency
Hz
Control Frequency
60 Hz
Num Substeps
Num Position Iterations
8
Num Velocity Iterations
1
A.2 Task Implementations
We implement four tasks based on the furniture model square table; Stabilize, Reach and Grasp, Insert, and Screw. An overview of simulated tasks is shown in Fig A.1. We elaborate their initial conditions, success criteria, reward functions, and other necessary information.
A.2.1 Stabilize
In this task, the robot needs to push the square tabletop to the right corner of the wall such that supported and remains stable in following assembly steps. The robot initialized such that its gripper locates a neutral position. The tabletop initialized at the coordinate (0.54, 0.00) relative the robot base. We then randomly translate with displacements drawn from 21(00.115, 0.015) along and y directions (the distance unit is meter hereafter). We also apply random Z rotation with values drawn from (4(-111) 15"). Four table legs are initialized in the scene. The task is successful only when the following three conditions are met:
1) The square tabletop contacts the front and right walls;
2) The square tabletop is within a pre-defined region;
3) No table leg in the pre-defined region.
We use the following reward function:
Wsuccess success wwgl/qel Wactson ||
(A.1)
where Wsuccess the success reward, I success indicates the success according to aforementioned conditions, Wa penalizes large joint velocities, qi the joint velocity, Waction penalizes large action commands, and a represents the action command at time step We set Wsuccess 10, Wi 10-5 and Waction 10-5 The episode length is 100. One episode terminates upon success timeout.
A.2.2 Reach and Grasp
In this task, the robot needs to reach and grasp table leg that randomly spawned in the valid workspace region. The task is successful once the robot grasps the table leg and lifts it for a certain
21

---

(a) Stabilize
(b) Reach and Grasp
(c) Insert
(d) Screw
Figure A.1: Visualization of simulated tasks.
height. The object's irregular shape limits certain grasping poses. For example, the end-effector needs to be near orthogonal to the table leg in the xy plane and far away from the screw thread. Therefore, we design a curriculum over the object geometry to warm up the RL learning. gradually adjusts the object geometry from cube, to cuboid, and finally the table leg. all curriculum stages, the reward function is
It Wdistanced + Wiifted liftte + Wsuconss success.
(A.2)
Here, Wdistonce the weight for distance reward, Wistee the reward for the leg being lifted, and Wsuccess the success weight. d the distance to the table leg and is calculated as
10 1- - tann ( (deat + dieft.finger + dright_ + dorthogonal)
(A.3)
where deef is the distance between the end-effector and the table leg, dieft finger the distance between the left gripper tip to the table leg, dright.finger is the distance between the right gripper tip to the table leg, and darthogonal is the difference between the current and the orthogonal grasping orientations. We set Wdistonce = 0.1, Wlifted 1.0, and Wswcoess 200.0. The episode length is 50. One episode terminates upon success or timeout.
A.2.3 Insert
In this task, the robot needs to insert a pre-grasped table leg into the far right assembly hole of the tabletop, while the tabletop is already stabilized. The tabletop is initialized at the coordinate (0.53,0.a 0.05) relative to the robot base. We then randomly translate with displacements sam- pled from (((00.02 0.02) along and directions. We also apply random rotation with values drawn from u(-45", 45%). We further randomize the robot's pose by adding noises sampled from u(-0.25, 0.25) to joint positions. The task is successful when the table leg remains vertical and is close to the correct assembly position within small threshold. We design curricula over the randomization strength to facilitate the learning. The following reward function is used:
T't Wdistanced + L
(A.4)
22

---

where Wdistance is the weight for distance-basse reward, d is the distance between the table leg and target assembly position, W Weuccese is the success weight, and success indicates task success. The distance d consists of an Euclidean distance dposition and an orientation distance dvertical to encourage the robot to keep the table leg vertical.
d = 1 - tanh (22 (dposition + deertical)
(A.5)
We set Wdistance = 1.0 and Wsuccess = 100.0. The episode length is 100. One episode terminates upon success or timeout.
A.2.4 Screw
In this task, the robot is initialized such that its end-effector is close to an inserted table leg. It needs to screw the table leg clockwise into the tabletop. We design curricula over the action space: at the early stage, the robot only controls the end-effector's orientation; at the latter stage, gradually takes full control. We slightly randomize object and robot poses during initialization. The reward function is
Ft = (1- 1l faiture) (Wscrendsrrew Wsuccess success) W. Wedeviation Adeviation
(A.6)
Here, failure indicates the task failure, Wscrew is the screwing reward weight, dscrew measures the screwed angle, Wsuccess is the success weight, and euccess indicates the task success. The task considered as successful when the leg has been screwed 180 into the tabletop.
considered
as failed when the table leg tilts more than 10" from the vertical pose. We Wscrew 0.1, Wsuccess 100.0, and Wdeviationn 10-2 The episode length is 200. One episode terminates upon success, failure, or timeout.
A.3 Teacher Policy Training
A.3.1 Model Details
Observation Space Besides proprioceptive observations, teacher policies also receive privileged observations to facilitate the learning. They include objects' states (poses and velocities), end- effector's 's velocity, contact forces, gripper left and right fingers' positions, gripper center position, and joint velocities. Full observations are summarized in Table A.II.
Table A.II: The observation space for teacher policies.
Name
Dimension
Name
Dimension
Proprioceptive
Privileged
Joint Position
Objects States
Nobjents 13
Cosine Joint Position
End-Effector Velocity
6
Sine Joint Position
Contact Forces
Natjectss
Left and Right
End-Effector Position
Fingers' Positions
End-Effector Rotation
4
Gripper Center Position
3
Gripper Width
Joint Velocity
Controller and Action Space An operational space controller (OSC) [73] is used in teacher policy training to improve sample efficiency. We follow Mistry and Righetti [108] to add nullspace control torques to prevent large changes in joint configuration. The action space is thus the change of end-effector's pose. We further add a binary action to control gripper's opening and closing. For- mally, it can be expressed as Ateasher (ōxr dy, 8z, òr, 8p, ōy, grippper) where (Ōz, dy, 6z) ga is the translation change, (ōr, 8p, 5y) E R is the rotation change, and 1 ggripper {0, 1] is the
gripper action.
23

---

Model Architecture We use feed-forward policies in RL training, It consists of MLP encoders to encode proprioceptive and privileged vector observations, and unimodal Gaussian distributions as the action head. Model hyperparameters are listed in Table A.III.
Table A.III: Model hyperparameters for RL teacher policies.
Hyperparameter
Value
Hyperparameter
Value
Obs. Encoder Hidden Depth
Obs. Encoder Activation
ReLU
Obs. Encoder Hidden Dim
256
Action Head Hidden Layers
[256, 128, 64] ELU [109]
Obs. Encoder Output Dim
256
Action Head Activation
A.3.2 Domain Randomization
We apply domain randomization during RL training to learn more robust teacher policies. Parame- ters are summarized Table A.IV.
Table A.IV: Domain randomization used in RL training.
Parameter
Type
Distribution
Parameter
Type
Distribution
Robot
Objects
Mass Friction
Scaling
U(0.5,
Mass
Scaling
U(0.5, 1.5)
Scaling
1.3.7,1.3)
Friction
Scaling
7.0.5,
Joint Lower Limit
Scaling
log log(4(1.00,
1.01)
Rolling Friction
Scaling
4(0.5,, 1.5)
Joint Upper Limit
Scaling
log(4(1.00,
1.01)
Torsion Friction
Scaling
4(0.5,1.55
Joint Stiffness
Scaling
log24(1.00,
1.01)
Restitution
Additive
4(0.0,
Joint Damping
Scaling
11.00 1.01)
Compliance
Additive
4(0.0,1.0)
Parameter
Type
Distribution
Simulation
Gravity
Additive
41(0.0,0.4
A.3.3 RL Training Details
We use the model-free RL algorithm Proximal Policy Optimization (PPO) [81] to learn teacher policies. Hyperparameters are listed in Table A.V. We customize the framework from Makoviichuk and Makoviychuk [110] use as our training framework.
Table A.V:
V: Hyperparameters used in PPO training.
Hyperparameter
Value
Hyperparameter
Value
Learning Rate
5x 10
Critic Weight
4
Discount Factor
0.99
GAE [111]
0.95
Entropy Weight
PPO
0.2
Optimizer
Adam [112]
Horizon
32
A.4 Student Policy Distillation
A.4.1 Data Generation
We use trained teacher policies as oracles to generate data for student policies training. Concretely, roll out each teacher policy to generate 10, 000 successful trajectories for each task. We exclude trajectories that are shorter than 20 steps.
24

---

A.4.2 Observation Space
Student policies receive observations that can be obtained in the real world. They are point- cloud and proprioceptive observations. We synthesize point clouds from objects' 6D poses im- prove the training throughput. Concretely, given the groundtruth point cloud of the m-th object
p(m) IR Xx3 we transform into the global frame through P) = p(m) (R(m))* (p(m))
Here R (m) E R333 and p(m) E IR3x1 denote the object's orientation and translation in the global frame. Further, the point-cloud representation of a scene S with M objects is aggregated as P -U Pmm For the robot, we only include point clouds for its two fingers and ignore other parts. To facilitate policies to differentiate gripper fingers from the scene, we extend the co- ordinate dimension to include a semantic label {0, 1} that indicates gripper fingers or not. This information can be obtained on real robots through forward kinematics. full point cloud is then downsampled to 768 points. Table A. lists the observation space.
Table AI The observation space for student policies.
Name
Dimension
Point Cloud
768 x 4
Proprioceptive
Joint Position
Cosine Joint Position
Sine Joint Position
End-Effector Position
End-Effector Rotation
Gripper Width
A.4.3 Action Space Distillation
To reduce the controller sim-to-real gap before transfer, train student policies output in the configuration space. To achieve that, we relabel actions à in trajectories generated by teacher policies from end-effector's delta poses to absolute joint positions. This is equivalent to set å, qr-1 for all time steps. Therefore, the action space for student policies is Astudent (q, grippeer), where E T is the joint position within the valid range. In simulation, student policies' actions are deployed with a joint position controller.
A.4.4 Model Architecture
We use feed-forward policies for tasks Reach and Grasp and Insert and recurrent policies for tasks Stabilize and Screw as we find they achieve the best distillation results. PointNets [82] are used to en- code point clouds. Recall that each point in the point cloud also contains a semantic label indicating the gripper or not. We concatenate point coordinates with these semantic labels' vector embeddings before passing into the PointNet encoder. We use Gaussian Mixture Models (GMM) [68] as the action head. Detailed model hyperparameters are listed Table VII.I.
A.4.5 Data Augmentation
We apply strong data augmentation during distillation. For point-cloud observations, random trans- lation and random jitter are independently applied with a probability Paed_ang = 0.4. We also add Gaussian noises to proprioceptive observations. Augmentation parameters are listed in Table A. VIII.
A.4.6 Training Details
To regularize point-cloud features, separately collect dataset containing 59 pairs of matched point clouds in simulation and reality. One pair from them is visualized in Fig A.2. Student policies
25

---

Table VII Model hy hyprparraeeters for student policies.
Hyperparameter
Value
Hyperparameter
Value
Point Cloud
RNN
PointNet Hidden Dim
256
RNN Type
LSTM [113]
PointNet Hidden Depth
2
RNN Num Layers
2
PointNet Output Dim
256
RNN Hidden Dim
512
PointNet Activation
GELU [114]
RNN Horizon
5
Gripper Semantic Embd Dim
128
GMM Action Head
Feature Fusion
Hidden Dim
128
MLP Hidden Dim
512
Hidden Depth
3
MIP Hidden Depth
Num Modes
5
MLP Activation
ReLU
Activation
ReLU
Table V.VIII: Data augmentation used distillation.
Hyperparameter
Value
Hyperparameter
Value
Point Cloud
Proprioception
Augmentation Probability
0.4
Prop. Noise Distribution WO0,
0.1)
Random Translation Distribution
u 4..0.04 0.04)
Prop. Noise Low
-0.3
Random Jittering Ratio
0.1
Prop. Noise High
0.3
Random Jittering Distribution
0.001
Random Jittering Low
-0.015
Random Jittering High
0.015
are trained by minimizing the loss in Sec. 2.2, where we set 8 = 10-3, We use the Adam opti- mizer [112] with learning rate of 10 during training. We periodically roll out student policies in simulation for 000 episodes. We then select the checkpoint that corresponds to the highest success rate use as the base policy in the real-world learning stage.
Figure A.2: Visualization of paired point clouds in simulation (red) and reality (blue).
B Real-World Learning Details
In this section, provide details about real-worle learning, including the hardware setup, human- in-the-loop data collection, and residual policy training.
26

---

B.1 Hardware Setup
shown in Fig. A.3, our system consists of a Franka Emika robot mounted on the tabletop- We use four fixed cameras and one wrist camera for point cloud reconstruction. They are three RealSense D435 and RealSense D415, There is also 3d-printed three-sided wall glued top of the table to provide extemal support. We use a joint position controller from the Deoxys library [115] to control our robot at 1000 Hz.
Left Cameras
Right Cameras
Wrist Camera
Fixed Wall
Figure A.3: System setup. Our system consists of Franka Emika robot mounted on the tabletop, four fixed cameras and one wrist camera (positioned at the rear side of the end-effector) for point cloud reconstruction, and 3d-printed three-sided wall glued onto tabletop to provide external sup- port.
B.2 Obtaining Point Clouds from Multi- Cameras
We use multi-view cameras for point cloud reconstruction to avoid occlusions. Specifically, we first calibrate all cameras to obtain their poses in the robot base frame. We then transform captured point clouds in camera frames to the robot base frame and concatenate them together. We further per- form cropping based on coordinates and remove statistical and radius outliers. To identify points belonging to the gripper so that we can add gripper semantic labels (Sec. A.4.2), we compute poses for two gripper fingers through forward kinematics. We then remove measured points correspond- ing to gripper fingers through K-nearest neighbor, given fingers' poses and synthetic point clouds. Subsequently, we add semantic labels to points belonging to the scene and synthetic gripper's point clouds. Finally, uniformly down-sample without replacement. We opt to not use farthest point sampling [116] due to its slow speed. One example is shown in Fig. A.4.
27

---

Figure A.4: Visualization of real-world point-cloud observations. We obtain them by 1) cropping point clouds fused from multi-view cameras based on coordinates, 2) removing statistical and radius outliers, 3) removing points corresponding to gripper fingers and replacing with synthetic point clouds through forward kinematics, 4) uniformly sampling without replacement, and 5) appending semantic labels to indicate gripper fingers (red) and the scene (blue).
B.3 Human-in-the-LLoop Data Collection
This data collection procedure illustrated in Algorithm As shown in Fig. A.5, we 3Deon- nexion SpaceMouse as the teleoperation device. We design specific UI to facilitate the synchro- nized data collection. Here, the human operator will be asked to intervene or not. The operator answers through keyboard. If the operator does not intervene, the base policy's next action will be deployed. If the operator decides to intervene, the SpaceMouse is then activated to teleoperate the robot. After the correction, the operator can exit the intervention mode by pressing one button on the SpaceMouse. We use this system and interface to collect 20, 100, 90, and 17 trajectories with correction for tasks Stabilize, Reach and Grasp, Insert, and Screw, respectively. We 90% of them training data and the remaining held-out validation We visualize the cumulative distribution function of human correction in Fig. A.6.
SpaceMouse
Figure A.5: Real workspace setup for human-in-the-loone data collection. The human operator provides online correction through 3Dconnexion SpaceMouse while monitoring the robot's 's exe- cution.
B.4
Residual Policy Training
B.4.1 Model Architecture
The residual policy takes the same observations as the base policy (Table A.VI). Furthermore, to effectively predict residual actions, it is also conditioned on base policy's s outputs. Its action head outputs eight-dim vectors, while the first seven dimensions correspond to residual joint positions and the last dimension determines whether negate base policy's gripper action not. Besides, separate intervention head predicts whether the residual action should be applied not (learnd gated residual policy, See. 2.4).
28

---

Algorithm 1: Human Intervention and Online Correction Data Collection
input Base policy x B human policy n real-world environment 3 output Human correction dataset D initialize: D +- 0
& E.rese(() while not E.lerminated do
& deploy the base policy a a (T) nent EEddeploy(a")
human decides intervention or not 1 H _ o
if 1 then
qure - E-.robol_statee
deploy human correction a" ta ~ T (o, ouext) n E q post E.robbotsstats
& update dataset
DH t DH U (qmT quost, 1 o)
end
D update the next observation + o nent
end
For tasks Stabilize and Insert, we use a PointNet [82] as the point-cloud encoder. For tasks Reach and Grasp and Screw, we use a Perceiver [83, 84] as the point-cloud encoder. Residual policies are instantiated as feed-forward policies in all tasks. We use GMM as the action head and simple two-way classifier as the intervention head. Model hyperparameters summarized in Table A.IX.
Table A.IX: Model hyperparameters for residual policies. Hyperparameter
Value
Hyperparameter
Value
PointNet
Feature Fusion
PointNet Hidden Dim
256
MLP Hidden Dim
512
PointNet Hidden Depth
2
MLP Hidden Depth
PointNet Output Dim
256
MLP Activation
ReLU
PointNet Activation
GELU
GMM Action Head
Gripper Semantic Embd Dim
128
Hidden Dim
128
Perceiver
Hidden Depth
Perceiver Hidden Dim Perceiver Number of Heads
256
Num Modes
8
Activation
ReLU
Perceiver Number of Queries
8
Intervention Head
Gripper Semantic Embd Dim
128
Hidden Dim
128
Base Policy Action Conditioning
Hidden Depth
Base Policy Gripper Action Embd Dim
64
Activation
ReLU
B.4.2 Training Details
To train the learned gated residual policy, first only leam the feature encoder and the action head. We then freeze the entire model and only leam the intervention head. We opt for this two-stage

---

Staliline
1.0
Roads and Grapp
CDF of P Porrrction
0.8
Screw
0.6
0.4
0.2
0.0
50
100
Task Progress (%)
Figure A.6: Cumulative distribution function (CDF) of human correction. Shaded regions rep- resent standard deviation. Human correction happens different times across tasks. This fact necessitates TRANSIC's learned gating mechanism.
training since we find that training both action and intervention heads at the same time will result in sub-optimal residual action prediction. We follow the best practice for policy training, including using learning rate warm-up and cosine annealing [117]. Training hyperparameters are listed in Table A.X.
Table A.X: Hyperparameters used in residual policy training.
Hyperparameter
Value
Learning Rate
10 10-4 4
Weight Decay
0
Learning Rate Warm Up Steps
1, 1,000
Learning Rate Cosine Decay Steps
100, 000
Minimal Learning Rate Optimizer
10-6
Adam
C Experiment Settings and Evaluation Details
In this section, we provide details about our experiment settings and evaluation protocols.
C.1 Task Definition
As shown in Fig. 3, we quantitatively benchmark four tasks. They are fundamental skills required assemble square table from FurnitureBench [85]. We randomize objects' initial poses during evaluation.
Stabilize: The robot pushes the square tabletop to the right corner of the wall such that remains stable following assembly steps.
Reach and Grasp: The robot reaches and grasps the table leg. It needs properly adjust the end effector's orientation to avoid infeasible grasping poses.
The robot inserts the pre-grase pre-grasped table leg to the far right assembly hole of the tabletop.
Insert:
The robot's end-effector is initialized close to inserted table leg and it screws the table leg clockwise into the tabletop.
Screw:
30

---

C.2 Main Experiments
We evaluate all methods on four tasks for 20 trials. Each trail starts with different objects and robot poses. We make our best efforts to ensure the same initial settings when evaluating different methods. Specifically, we take pictures for these 20 different initial configurations and refer to them when resetting a new trial. See Figs. A.14, A.15, A. A.16, A.17 for initial configurations of tasks Stabilize, Reach and Grasp, Insert, and Screw, respectively. We follow Liu al. [91] label reward for IQL. Full numerical results are provided in Table A.XI.
Table A.XI: Success rates per tasks. TRANSIC outperforms all baseline methods in all four tasks.
Tasks
TRANSIC
Transfer
Aug. [52]
Fime-Tum
Fine-Tune
HGi-aggerr [66]
IWR [67]
BC [86]
BC-RNN [68]
IQL. [69)
Stabiiize
100%
10%
35%
35%
0%
65%
65%
40%
40%
515
Reach and Group
95%
35%
60%
35%
08
30%
40%
25%
0%
5%
Insert
45%
2546
35%
40%
10%
0%
Screw
C.3 Experiments with Different Sim-to-Real Gaps
C.3.1 Experiment Setup
We explain how different sim-to-real gaps are created.
Perception Error This is done by applying random jitter to 25% points from point clouds with probability p 0.6, as visualized Fig. A.7. We test this sim-to-real gap on the task Reach and Grasp. The jittering noise sampled independently from the distribution (0, 0.03). We clip the noise to be within the + 0.03 range.
(a)
(b)
Figure A.7: Visualization of introduced perception error. a) The original point-cloud observation. b) The erroneous point-cloud observation with random jitter.
Underactuated Controller This is done by making the joint position controller less accurate. We test this gap the task Insert. We emulate an underactuated controller through early stopping. Concretely, at every time new joint position goal Qgoat set, we record the distance to the goal configuration space dg = l|q - 9goai|] || and sample a factor I 2(0080, 0.95). The controller will stop reaching the desired goal once achieves progress, i.e., stop early when ||q |goul < (1 I'dq. Fig. A.8 visualizes the effect.
Embodiment Mismatch This is done by changing the robot gripper be shorter length as demonstrated in Fig. A.9. We test this gap on the task Screw. We notice that the 9 em length difference incurs significant gap.
Dynamics Difference This is done by changing object surfaces and increasing friction. We test this gap on the task Stabilize. Concretely, attach friction tapes the square tabletop's surface to increase friction, hence change the dynamics (Fig. A.10).
31

---

36
0.2
Taieetorry Stop
0.0
18
Elffesser
0.22
0.2
0.0
0.2
Elfiecuor
(m))
Relornee
Nonal
Undrenctusted Contoolle
Figure A.8: Visualization of the trajectory realized by an underactuated controller. The plot displays the end-effector's position in the XY plane. It shows reference circular movement, trajectory tracked by the normal controller, and a trajectory tracked by the underactuated controller.
5 cm
14 cm
Figure A.9: Two different gripper fingers used to create embodiment mismatch. Policies are trained with the longer finger and tested on the shorter finger.
Object Assert Mismatch As shown in Fig. A.11, this is done by replacing the table leg with light bulb. We test this gap on the task Reach and Grasp.
C.3.2 Evaluation
We conduct 20 trails with different initial configurations. Initial conditions for first four experi- ments are the same main experiments (Figs. A.14, A.15, A.16, A.17). Fig. A.18 shows initial configurations for the experiment Object Asset Mismatch.
C.4 Data Scalability Experiments
In Table A.XII, show quantitative results for scalability with human correction dataset size four tasks.
C.5
Ablation Studies
C.5.1 Effects of Different Gating Mechanisms
We introduce the learned gated residual policy in Sec. 2.4 where the gating mechanism controls when to apply residual actions. To assess the quality of learned gating, compare its performance with actual human operator performing gating. Results are shown in Table (row "w/ Human
32

---

(a)
(b)
Figure A.10: Two square tabletops used to create dynamics difference. a) The original surface is smooth. b) We attach friction tapes to change the dynamics.
(a)
(b)
Figure A.11: Two objects used to create asset mismatch. a) Policies are trained with the table leg. b) We test policies with an unseen light bulb.
Gating"). It evident that the learned gating mechanism only incurs negligible performance drops compared to human gating. This suggests that TRANSIC can reliably operate a fully autonomous setting once the gating mechanism is learned.
C.5.2 Policy Robustness
We investigate the policy robustness against 1) point cloud observations with inferior quality by moving two cameras, and 2) suboptimal correction data with noise injection. We remove two cam- eras and only keep three. Note that this is the same number of cameras as in FurnitureBench [85]. For tasks other than Insert, we keep the wrist camera, the right front camera, and the left rear camera. For the task Insert, we keep two front cameras and the left rear camera. We simulate suboptimal correction data by injecting noise into residual actions R This noise of large magnitude, which follows the normal distribution with zero mean and standard deviation corresponding 5% of the largest residual action in the dataset. Results are shown in Table (rows "Reduced Cameras" and "Noisy Correction"). We highlight that TNAISIC is robust to partial point cloud inputs caused by the reduced number of cameras. We attribute this to the heavy point cloud downsampling employed during training. Fishman et al. [118] echos our finding that policies trained with downsampled syn- thetic point cloud inputs can generalize to partial point cloud observations obtained in the real world without the need for shape completion. Meanwhile, when the correction data used to learn residual policies are suboptimal, TRANSIC only shows relative decrease of 6% in the average success rate. We attribute this to the advantage of our integrated deployment-wei -ehe the residual policy behaves suboptimally, the base policy could still compensate for the error in subsequent steps.
33

---

Table A.III: Quantitative results for scalability with human correction dataset size on four
tasks.
Correction Dataset Size (%)
Method
0
25
50
75
100
Stabilize
TRANSIC
80%
80%
100%
100%
35%
IWR [67]
70%
75%
80%
65%
Reach and Grasp
TRANSIC
65% 60%
80% 65%
90%
95%
60%
TWR [67]
40%
40%
Insert
TRANSIC
20%
35%
40%
45%
5%
IWR [67]
5%
15%
30%
40%
Screw
TRANSIC
50% 20%
65% %
75%
85%
35%
IWR [67]
40%
40%
40%
C.5.3 Consistency in Learned Visual Features
To learn consistent visual features between the simulation and reality, propose regularize the point cloud encoder during the distillation stage. As shown in Table (row "w/o Regularization") the performance significantly decreases without such regularization, especially for tasks that require fine-grained visual features. Without it, simulation policies would overfit to synthetic point-cloud observations and hence are not ideal for sim-to-real transfer.
C.6 Qualitative Analysis and Emergent Behaviors
We examine the distribution of the collected human correction dataset. During the human-in-the. loop data collection, the probability of intervening and correcting is reasonably low (Poorection R 0.20). This is consistent with our intuition that, with good base policy, interventions are not neces- sary for most of the time. However, they become critical when the robot tends to behave abnormally due to unaddressed sim-to-real gaps. Moreover, as highlighted in Fig. A.6, interventions happen different times across tasks. This fact renders heuristics-base methods [119] for deciding when intervene difficult, and further necessitates our learned residual policy. Several representative behaviors learned by TRANSIC are demonstrated in Fig. A.12.
Additional Experiment Results and Discussions
D.1 Empirical Justifications for Action Space Distillation
Reasons for the proposed action space distillation are twofold. The first mainly because an OSC is hard sim-to-real transfer, while joint position controller be seamlessly transferred. As sug- gested in Nakanishi et al. [74], OSC requires accurate modeling of robot parameters, such the task-space inertia matrix and gravity compensation. System identification helps but is insufficient. Furthermore, it is often the case that given the same joint torque, the end-effector moves differently in simulation and the real world. Because an OSC uses a task-space error to compute joint torques, this will lead to large joint position deviation.
The second is for better training efficiency. As shown in Fig, A.13, it is almost impossible directly train RL with point-cloud inputs and joint position action space. Even after 7-day training, RL still
34

---

AS
(a) Error Recovery
(b) Unsticking
(c) Safety-aware Actions
(d) Failure Prevention
Figure 12: Emergent behaviors learned by TRANSIC. a) Error recovery. Left: The robot tries to insert the table leg but the direction is wrong; Right: TRANSIC raises the end effector and moves the correct insertion position. b) Unsticking. Left: The robot hovers for while and never reaches the light bulb; Right: TRANSIC helps the robot get unstuck and move to the bulb. c) Safety-aware actions. Left: When pushing the tabletop, the gripper is too low and bends. This might damage the robot; Right: TRANSIC compensates for the command that causes the end effector to move too low. d) Failure prevention. Left: The light bulb will fall and break after gripper opening: Right: TRANSIC adjusts the bulb to stable pose to prevent failure.
shows no sign of improvement. In contrast, TRANSIC takes around days to train on NVIDIA GeForce RTX 3090 GPUs. Therefore, the distillation is important make the training feasible.
16/000 retollree
Beach Grapp
0.03
3.005
(a) Stabilize
(b) Reach and Grasp
Insert
Screw
3.06
3.04
Hoars
(c) Insert
(d) Screw
Figure A.13: Learning curves for RL with point-cloud observations and joint position actions.
35

---

D.2 Distilling Simulation Base Policy with Diffusion Policy
We experiment with learning simulation base policies (Sec. 2.2) with the Diffusion Policy [120]. Concretely, when performing action space distillation to learn student policies, we replace the Gaussian Mixture Model (GMM) action head with the Diffusion Policy. Proper data augmenta- tion (Table A. VIII) also applied robustify learned policies. Hyperparameters are provided in Table A.XIII.
Table X.IIII: Diffusion Policy hyperparameters.
Hyperparameter
Value
Hyperparameter
Value
Architecture
UNet
To
UNet Hidden Dims
[64, 128]
UNet Kernel Size
16
UNet GroupNoem Groups
Denoising Steps (Train)
Diffusion Step Emmdd Dim
128
Num Denoising Steps (Eval)
16
The comparison between GMMs on the real robot is shown in Table. A.XIV. We highlight two findings. First, the significant domain difference between simulation and reality generally cxists re- gardless of different policy parameterizations. Second, since the Diffusion Policy plans and executes future trajectory, is more vulnerable to simulation-to-ree the consequent compounding error. Only executing the first action from the planned trajectory and re-planning at every step may help, but the inference latency renders the real-time execution infea- sible.
gaps due to planning inaccuracy and
Table A.XIV: The real-robot performance difference between GMM and Diffusion Policy. The policy caused by simulation-to-reale gaps will be amplified by the Diffusion Policy because plans and executes a future trajectory.
Reach and Grasp
Average
Stablize
Insert
Screw
GMM
33.7%
35% 35%
60% 50%
5%
35%
Diffusion Policy
22.5%
5%
0%
D.3 Gating Function Justification and Conceptual Comparison
Recall several design choices the proposed gating mechanism: 1) takes inputs of unstructured sensory observations (point cloud); 2) conditioned on base policy's 's outputs for effective prediction; 3) the intervention classifier shares the same feature encoder with the residual policy; and 4) the entire pipeline is learned end-to-end. We contrast against several mechanisms from the literature.
Table A.XV: Gating mechanism conceptual comparison.
How to decide
Condition on base policy's outputs
Shared feature encoder
Input
apply gating not
Point cloud proprioception
Ours
End-to-end learned
Yes
Yes
Residual Policy Learning [79]
No gating
Low-dimensional state
No
No
Residual [78]
gating
Low-dime now-dimensional state
No
Thresholded based
ThriftyDggger [119]
on neural network ensemble
Low-dimensional state
No
Runtime Monitoring [104]
End learned
RGB and propriaception
Yes
Although this gating function and residual policy can be merged into one, we opt not to do so for better empirical performance.
the task Screw, using the gating function with the residual policy achieves a success rate of 85%, while only using the residual policy achieves a success rate 55%. We hypothesize that this is mainly due the data imbalance-because intervention and correction happen with low frequency (Pcorrection leam residual policy from them, it will be biased toward predicting near-zero actions. On the
0.20), most residual actions are zero. If we naively
36

---

other hand, by training the residual policy only on human corrections, we offload learning whether to apply the correction to a gating function and thus reduce the negative effects of unbalanced data.
D.4
Long-Horizon Tasks Statisties
We show statistics about task length from FurnitureBench [85] in Table A.XVI.
Table A.XVI: Statistics about long-horizon tasks from FurnitureBench [85].
Number of Steps
Average Human Demo Length
Lamp
594
Minutes
Square Table
1689
Minutes
E Extended Preliminaries
E.1 Intervention-Basee Policy Learning
We adopt an intervention- based learning framework [66, 67, 91] where human operator inter- vene and take control during the execution of the robot base policy и Denote the human policy as TH. the following combined policy is deployed during data collection:
deployed
If " (1 (1-18 1) n"
(A.7)
where 1 ! a binary function indicating human interventions. Introducing a trajectory distribution q(T) that consists of two abservation-action distributions generated by the robot o and human operator p", the original RL objective leads to the maximization of variational lower bound on logarithmic return [67, 121]:
J( E,)r [log R(T) + log Pw, log q(T),,
(A.8)
where Pra is the induced trajectory distribution. While the human operator optimizes Eq. A.8 through intervention and correction, the robot learner maximizes it through
0 = arg max ((.a)~<())) [log (a]s))
(A.9)
DE0
Various intervention-basee policy learning methods have been derived by weighting observation- action pairs in Eq. A.9 differently. For example, HG-Dagger [66] completely ignores robot data B and only trains on human data D that contain intervention samples. This is equivalent to 4(T) x , Intervention Weighted Regression (IWR) [67] balances the data distribution by emphasizing methods such as traditional behavior cloning (BC) [86] only learn on D with full human demonstrations instead of intervention. This effectively sets (T)) p
human intervention: g(T) ap" pd with a |D |D" Non-interventiion-aae
F Extended Related Work
Robot Learning via Sim-to-Real Transfer Physics-based simulations [6-10, 49, 122-124) have become driving force [1, 2] for developing robotic skills in tabletop manipulation [125-128), mobile manipulation [129-132], fluid and deformable object manipulation [133-136], dexterous in-hand manipulation [13-17], locomotion with various robot morphology [18-26, 137], object tossing [80], acrobatic flight [28, 29]. etc. However, the domain gap between the simulators and the reality is not negligible [10]. Successful sim-to-real transfer includes locomotion [18-27], in-hand re-orientation for dexterous hands where objects are initially placed near the robot [13-17], and noe-preee manipulation limited to simple tasks [30-39]. In this work, we tackle more challenging sim-to-real transfer for complex manipulation tasks and successfully demonstrate that our approach can solve sophisticated contact-rich manipulation tasks. More importantly, requires significantly fewer real-robot data compared the prevalent imitation learning and offline RL approaches [68, 69, 86). This makes solutions that are based simulators and sim-to-ree transfer more appealing roboticists.
37

---

Sim-to-Real Gaps in Manipulation Tasks Despite the complex manipulation skills recently learned with RL in simulation [138), directly deploying learned control policies to physical robots often fails. The sim-to-real gaps [10, 40, 44, 139] that contribute to this performance diserepancy can be coarsely categorized as follows: a) perception gap [18, 41-43J, where synthetic sensory observa- tions differ from those measured in the real world: b) embodiment mismatch [18, 44, 45], where the robot models used in simulation do not match the real-world hardware precisely; controller inac- curacy [46-48], meaning that the results of deploying the same high-level commands (such as in con- figuration space [140] and task space [141]) differ in simulation and real hardware; and d) poor phys- ical realism [49], where physical interactions such as contact and collision are poorly simulated [87].
Although these gaps may not be fully bridged, traditional methods to address them include system identification [18, 30, 50, 51], domain randomization [13, 52-55), real-world adaptation [56], and simulator augmentation [58-60]. However, system identification is mostly engineered a case-by- case basis. Domain randomization suffers from the inability identify and randomize all physical parameters. Methods with real-world adaptation, usually through meta-learning [88], incur potential safety concerns during the adaptation phase. Most of these approaches also rely on explicit and domain-specific knowledge about tasks and the simulator a priori. For instance, to perform system identification for closing the embodiment gap for a quadruped, Tan et al. [18] disassembles the physical robot and carefully calibrates parameters including size, mass, and inertia. Kim et [32] reports that collaborative robots, such as the commonly used Franka Emika robot, have intricate joint friction that is hard to identify and randomized in typical physies simulators. To make simulator more akin to the real world, Chebotar et al. [39] deploys trained virtual robots multiple times to refine the distributions of simulation parameters. This procedure not only introduces a significant real-world sampling effort, but also incurs potential safety concerns due to deploying suboptimal policies. In contrast, our method leverages human intervention data to implicitly overcome the transferring problem in a domain-agnostic way and also leads to safer deployment.
Hlman-in-Thh--Lane Robot Learning Human-in-the-loop machine learning is prevalent frame- work to inject human knowledge into autonomous systems [62, 89, 90]. Various forms of human feedback exist [63], ranging from passive judgement, such as preference [142-151] and evalua- tion [152-157], to active involvement, including intervention [158-160) and correction [161, 162). They are widely adopted in solutions for sequential decision makinn tasks. For instance, interactive imitation learning [66, 67, 91, 163] leverages human intervention and correction to help naive imitators address data mismatch and compounding error. the context of RL, reward functions can be derived to better align agent behaviors with human preferences [145, 148, 149, 152]. Noticeably, recent trend focuses on continually improving robots' capability by iteratively updating and deploying policies with human feedback [91], combining active human involvement with RL [162], and autonomously generating corrective intervention data [92]. Our work further extends this trend by showing that sim-to-real gaps can be effectively eliminated by using human intervention and correction signals.
In shared autonomy, robots and humans share the control authority achieve common goal [64, 65, 93-95]. This control paradigm has been largely studied in assistive robotics and human-robot collaboration [96-98]. In this work, we provide novel perspective by employing it in sim-to-real transfer of robot control policies and demonstrating its importance in attaining effective transfer.
38

---

Figure A.14: Initial settings for evaluating the task Stabilize.
Figure A.15: Initial seftings for evaluating the task Reach and Grasp.
Figure A.16: Initial settings for evaluating the task Insert.
HR
Figure A.17: Initial settings for evaluating the task Screw.
Figure A.18: Initial settings for the experiment Object Asset Mismatch.
39


Reviews:
[{"review_id": "fy2D0Yav7b", "reviewer": "Reviewer_5kGt", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces TRANSIC, an approach for sim-to-real policy transfer by learning from online correction and intervention provided by humans. The concept relies on first learning a base policy from simulation, and then learning a residual policy online from human corrections. In simulation, the base policy is trained using a teacher-student framework, where the teacher policy is trained with RL and Operational Space Control, and this teacher policy is then distilled into a student policy that operates on depth clouds with a joint position action space. During the online rollout, a human provides interventions if the robot execution fails. These interventions are stored within a correction dataset and used to train a residual policy. The final policy is synthesized between the base policy and residual policy using a gating function. Experimental results were conducted on real robot platforms and include four tasks relating to furniture assembly and a lamp assembly task.", "summary_of_recommendation": "Original Recommendation: As written, the paper has many strong points, such as rigorousness of experimental assessment, quality of writing, and thoroughness of literature review. There are a few aspects of the approach that would be helpful to clarify, either in the main paper or in the appendix. I believe addressing the clarifications can further strengthen the paper, so I would be willing to revise my recommendation based on these clarifications. Updated Recommendation: The author response in the rebuttal has addressed all of my concerns, so I have updated my recommendation to strong accept.", "technical_quality": 5, "clarity_of_presentation": 5, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "- Have there been any issues with distribution shifts with point clouds from sim to real? Or, if there were, are such issues correctable with TRANSIC?\n- Can you provide more details for D^pcd? Would it be the case that it is required for objects to have real point cloud measurements prior to training the student policy?\n- Can you clarify the details of the gating function?\n- Can you confirm how often the human is monitoring the executed policy? Is the entire rollout?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "92dPfHFTBc", "reviewer": "Reviewer_pa2D", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper introduces a novel method to tackle sim-to-real problems by introducing humans in the loop to provide correctional action. Human feedback are then used to train a residual policy that will combine with the simulation policy to work well in a set of contact-rich manipulation tasks.", "summary_of_recommendation": "The paper presents an effective method to address sim-to-real transfers using human corrections. The approach is novel, and well-supported by experiments.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "- In 2.1, the POMDP formulation is over-killed. The policy is reactive (mapping from the current observation to action), so it is really just an MDP with a high-dimensional state (i.e., point cloud)\n- In 2.2, the text description should clarify what kind of privileged observation the teacher is allowed to use\n- In Algorithm 1, why the input to \\pi^H include both o and o^{next}?\n- It is stated in the paper that the residual policy is trained to predict q^{post} – q^{pre}; why such residual policy during deployment can be summed with the base policy to have a working control signal? In other words, it is unclear about the relationship between q^{pre} and the output from the base policy.\n- I am curious how easy it is to use the space mouse to generate 6D intervention control inputs.\n- I’m surprised that using OSC action space makes sim-to-real transfers more challenging. I thought that high-level action would make transfers easier as we masked out the kinematics, which might differ between sim and real. Can the authors comment on that?\n- When compared with baselines, the authors should provide the learning curves (e.g., in the appendix) instead of just providing the final performance. It is useful for future research relying on this approach", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4DG4iUsY6a", "reviewer": "Reviewer_LZdQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work proposes to learn a residual policy from human correction to improve sim-to-real transfer of the simulation-based manipulation policies.", "summary_of_recommendation": "Weak reject.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "1. The authors have done a lot of experiments on the real robot. Given the focus of this work is to show effective sim-to-real transfer with the proposed method, what are the success rates of the simulation-trained base policies?\n\n2. In perception error experiments, among all 4 tasks, *insert* and *screw* might be the most sensitive to sensor noise. Would it be possible to show the results on these two tasks?\n\n3. In the limitation section, the authors have mentioned that \"Human operators still manually decide when to intervene during correction data collection.\" After a few trials, can the learned gating function be also used to determine whether human intervention is needed?\n\n4. What are the controller gains used in simulation-based training and real-world deployment? Is there any tuning involved for real-world deployment?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: ### Strengths:

- Re: Clarity: The approach is conceptually easy to understand and well-executed (R-5kGt, R-pa2D); R-pa2D additionally mentions that the method is novel and addresses the critical problem of sim-to-real transfer in a task-agnostic manner. 
- Re: Experiments: R-5kGt and R-pa2D suggest that the experiments are convincing, with a good choice of baselines, where the method is demonstrated to work well in both simulation and hardware, outperforming strong baselines. R-pa2D and R-LZdQ report that the manuscript considers a wide variety of tasks.
- Re: Writing: all reviewers state that the paper is written clearly, with a good literature review (R-5kGt, R-pa2D) and helpful supplementary material (R-5kGt).
- Re: Methodology: R-LZdQ discussed that the paper's approach is more effective and data-efficient compared to traditional sim-to-real methods like domain randomization and data augmentation.
    

### Weaknesses:

- Re: Clarity: R-5kGt and R-LZdQ mentioned concerns related to missing details on the dataset DpcdD^{pcd}Dpcd, the gating function, and the specifics of human intervention frequency; also, the description of the POMDP formulation and the privileged observation for the teacher policy needs clarification (R-pa2D).
- Re: Task / Domain: R-pa2D expressed concerns that the tasks are all in tabletop settings and may be easier due to the green walls used for alignment, which could simplify the tasks. R-LZdQ mentioned concerns that the success rates for some tasks are low, making it difficult to assess if the issue lies with the base policy or the sim-to-real transfer.
    
- Re: Methodology: The need for a separate gating function is questioned, as it should ideally be incorporated into the residual policy (R-LZdQ). R-LZdQ was concerned that the number of robot trajectories with human correction varies across tasks without clear guidelines on determining this number for new tasks.
    
- Re: Evaluation: The paper lacks detailed comparisons of success rates between simulation-trained base policies and real-world performance (R-LZdQ). More analyses on the sensitivity of tasks to sensor noise and the potential for learned gating functions to automate human intervention decisions are needed (R-LZdQ). The overall impact is seen as incremental rather than significantly advancing the field (R-LZdQ).

### Post-rebuttal Meta Review Statement

I applaud the reviewers and authors for engaging in a productive discussion that resulted in several improvements to the paper, especially as it relates to the clarification of the gating function and addition of new experiments (in response to R-5kGt — enabling them to raise their score), reformulation of the problem (in response to R-pa2D), and the addition of further analysis on variable task complexity (in response to R-LZdQ — enabling them to raise their score).

I recommend for Accept as Poster.

I encourage the authors to follow through with all the promised changes and to incorporate all additional reviewer requests, to improve the paper even further.
Decision: Accept