TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correction

Authors: yunfan  jiang
chen  wang
ruohan  zhang
jiajun  wu
li  fei-fei
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-05-21 22:45:40

Abstract: Learning in simulation and transferring the learned policy to the real world has the potential to enable generalist robots. The key challenge of this approach is to address simulation-to-reality (sim-to-real) gaps. Previous methods often require domain-specific knowledge *a priori*. We argue that a straightforward way to obtain such knowledge is by asking humans to observe and assist robot policy execution in the real world. The robots can then learn from humans to close various sim-to-real gaps. We propose TRANSIC, a data-driven approach to enable successful sim-to-real transfer based on a human-in-the-loop framework. TRANSIC allows humans to augment simulation policies to overcome various unmodeled sim-to-real gaps holistically through intervention and online correction. Residual policies can be learned from human corrections and integrated with simulation policies for autonomous execution. We show that our approach can achieve successful sim-to-real transfer in complex and contact-rich manipulation tasks such as furniture assembly. Through synergistic integration of policies learned in simulation and from humans, TRANSIC is effective as a holistic approach to addressing various, often coexisting sim-to-real gaps. It displays attractive properties such as scaling with human effort. Videos and code are available at https://transic-robot.github.io/.

Main paper: 
 introduction: 
 Learning in simulation is a potential approach to the realization of generalist robots capable of solving sophisticated decision-making tasks [1, 2] . Learning to solve these tasks requires a large amount of training data [3] [4] [5] . Providing unlimited training supervision through state-of-the-art simulation [6] [7] [8] [9] [10] could alleviate the burden of collecting data in the real world with physical robots [11, 12] . Therefore, it is crucial to seamlessly transfer and deploy robot control policies acquired in simulation, usually through reinforcement learning (RL), to real-world hardware. Successful demonstrations of this simulation-to-reality (sim-to-real) approach have been shown in dexterous in-hand manipulation [13] [14] [15] [16] [17] , locomotion [18] [19] [20] [21] [22] [23] [24] [25] [26] [27] , and quadrotor flight [28, 29] .Nevertheless, replicating similar success in manipulation tasks with robotic arms remains surprisingly challenging, with only a few cases in simple non-prehensile manipulation [30] [31] [32] [33] , industry assembly under restricted settings [34] [35] [36] [37] [38] , and peg swinging [39] . The difficulty mainly stems from the unavoidable sim-to-real gaps [40] , including but not limited to perception gap [41] [42] [43] , embodiment mismatch [18, 44, 45] , controller inaccuracy [46] [47] [48] , and dynamics realism [49] . Traditionally, researchers tackle them through system identification [18, 30, 50, 51] , domain randomization [13, [52] [53] [54] [55] , real-world adaptation [56, 57] , and simulator augmentation [58] [59] [60] . Many of these approaches require explicit, domain-specific knowledge and expertise in tasks or simulators.Although for a particular simulation-reality pair, there may exist specific inductive biases that can be hand-crafted post hoc to close the sim-to-real gap [18] , this knowledge is often not available a priori. Identifying its effects on task completion is also intractable.We argue that a straightforward and feasible way for humans to obtain such knowledge is to observe and assist policy execution in the real world. If humans can assist the robot to successfully accomplish the tasks in the real world, sim-to-real gaps are effectively addressed. This naturally leads to a generally applicable paradigm that can cover different priors across simulations and realitieshuman-in-the-loop learning [61] [62] [63] and shared-autonomy [64, 65] .Our key insight is that the human-in-the-loop framework is promising for addressing the sim-toreal gaps as a whole, in which humans directly assist the physical robots during policy execution by providing online correction signals. The knowledge required to close sim-to-real gaps can be learned from human signals. We present TRANSIC (transferring policies sim-to-real by learning from online correction, Fig. 1 ), a data-driven approach to enable the successful transfer of robot manipulation policies trained with RL in simulation to the real world. In TRANSIC, once the base robot policies are acquired from simulation training, they are deployed to real robots where human operators monitor the execution. When the robot makes mistakes or gets stuck, humans interrupt and assist robot policies through teleoperation. Such human intervention data are collected to train a residual policy, after which the base policy and the residual policy are combined to solve contactrich manipulation tasks. Unlike previous approaches that heavily rely on domain knowledge, since humans can successfully assist the robot trained in silico to complete real-world tasks, sim-to-real gaps are implicitly handled and addressed by humans in a domain-agnostic manner.To summarize, our key contribution is a novel, holistic human-in-the-loop method called TRANSIC to tackle sim-to-real policy transfer for manipulation tasks. Through extensive evaluation, we show that our method leads to more effective sim-to-real transfer compared to traditional methods [50, 52] and requires less real-robot data compared to the prevalent imitation learning and offline RL algorithms [66] [67] [68] [69] . We demonstrate that successful sim-to-real transfer of short-horizon skills can solve long-horizon, contact-rich manipulation tasks, such as furniture assembly. 
sim-to-real policy transfer by learning from online correction: 
 An overview of TRANSIC is shown in Fig. 2 . This section starts with a brief preliminary review, followed by a description of simulation training. We then introduce residual policies learned from human intervention and online correction and present an integrated framework for deployment during testing. Lastly, we provide implementation details. We formulate a robot manipulation task as an infinite-horizon discrete-time Markov Decision Process (MDP) M := (S, A, T , R, γ, ρ 0 ), where S is the state space, and A is the action space. At time step t, a robot observes s t ∈ S, executes an action a t , and receives a scalar reward r t from the reward function R(s t , a t ). The environment progresses to the next state following the transition function T (s t+1 |s t , a t ). The robot learns a parameterized policy π θ (•|s) to maximize the expected discounted returnJ := E τ ∼pπ θ [ ∞ t=0 γ t r t ]over induced trajectory distribution τ := (s 0 , a 0 , r 0 , ...) ∼ p π θ , where s 0 ∼ ρ 0 is sampled from the initial state distribution and γ ∈ [0, 1) is a discount factor. We consider simulation and real environments as two different MDPs. We adopt an intervention-based learning framework [66, 67] where a human operator can intervene and take control during the execution of the robot policy.Policy Learning with 3D Representation Object geometry matters for contact-rich manipulation. For example, a robot should ideally insert a light bulb into the lamp base with the thread facing down. To retain such 3D information and facilitate sim-to-real transfer, we propose to use point cloud as the main visual modality. Typical RGB observation used in visuomotor policy training [70] suffers from several drawbacks that hinder successful transfer, such as vulnerability to different camera poses [71] and discrepancies between synthetic and real images [42] . Well-calibrated point cloud observation can bypass these issues and has been successfully demonstrated [14, 72] . During the simulation RL training phase, we synthesize point cloud observations for higher throughput. Concretely, given the synthetic point cloud of the m-th object P (m) ∈ R K×3 , we transform it into the global frame throughP (m) g = P (m) R (m) ⊺ + p (m) ⊺ .Here, R (m) ∈ R 3×3 and p (m) ∈ R 3×1 denote the object's orientation and translation in the global frame. Further, the point cloud representation of a scene S with M objects is aggregated as P S = M m=1 P (m) g and subsequently used as policy input.Action Space Distillation A suitable action abstraction is critical for efficient learning [46, 47] as well as sim-to-real transfer [48] . A high-level controller such as the operational space controller (OSC) [73] facilitates RL exploration [46] but may hinder sim-to-real transfer because it requires accurate modeling of robot parameters, such as joint friction, mass, and inertia [74] ; on the other hand, a low-level action space such as the joint position ensures consistent deployment in simulation and real hardware, but renders trial-and-error RL impractical. We draw inspiration from the teacherstudent framework [15, [75] [76] [77] and propose to first train the teacher policy π teacher through RL with OSC and then distill successful trajectories into the student policy π student with joint position control. Specifically, we roll out π teacher and record the robot's joint position at every simulated time interval δt to construct a dataset D teacher = {τ (n) } N n=1 . We then relabel actions from the endeffector's poses to joint positions. Such a relabeled dataset is ready to train student policies through behavior cloning. We name this approach as action space distillation and find it crucial to overcome the sim-to-real controller gap. Furthermore, teacher policies directly receive privileged observations for ease of learning, while student policies learn on synthetic point-cloud inputs to match real-world measurements. The student policy parameterized by θ, π student θ , is trained by minimizing the loss function: L student = -E D teacher log π student θ + βE D pcd ∥ϕ(P real )ϕ(P sim )∥ 2 , where ϕ(•) denotes the point cloud encoder of π student θ and D pcd = {(P real , P sim ) (i) } N i=1 is a separate dataset that contains N pairs of matched point clouds in simulation and reality for regularization purpose. Further justifications of this distillation phase can be found in Appendix D.1.Human-in-the-Loop Data Collection Once the student policy is obtained from simulation, it is used directly as the base policy π B to bootstrap the data collection. Naïvely deploying the base policy on real robots usually results in inferior performance and unsafe motion due to various sim-to-real gaps. In TRANSIC, the base policy is instead deployed in a way that is fully synchronized with the human operator. Concretely, at time step t, once a B t ∼ π B is deployed, a human operator needs to decide whether intervention is necessary, indicated as 1 H t . Intervention is not necessary for most task execution when the robot is approaching objects of interest. However, when the robot tends to behave abnormally, the human operator intervenes and takes full control through teleoperation to correct robot errors. In these cases, the robot's pre-and post-intervention states, as well as intervention indicator 1 H t , are collected to construct the online correction datasetD H ← D H ∪ 1 H t , q pre t , q post t. This procedure is illustrated in Appendix Algorithm 1.Properly modeling human correction can be challenging. This is because humans usually solve tasks not purely based on current observation, hence the non-Markovian decision process [68] . Therefore, directly fine-tuning the base policy π B on human correction dataset D H leads to large motions and even model collapse (Sec. 3). Inspired by prior work on learning residuals to compensate for unknown dynamics and noisy observations [78] [79] [80] , we propose to incorporate human correction behaviors with residual policies. Concretely, at the time of intervention, a residual policy π R ψ parameterized by ψ learns to predict human intervention as the difference between post-and pre-intervention robot states: a R = q post ⊖ q pre , where ⊖ denotes generalized subtraction. For continuous variables such as joint position, it computes the numerical difference; for binary variables such as opening and closing the gripper, it computes exclusive nor. The residual policy is then trained to maximize the likelihood of human correction:L residual = -E D H log π R ψ (a R |•) .In practice, we find that learning a gating function to control whether to apply residual actions or not leads to better empirical performance (Appendix D.3). We call this learned gated residual policy. Denote the gating function as g ψ (•). It shares the same feature encoder with the residual policy π R and is jointly learned through classification on the same correction dataset D H . At the inference time, we effectively use g ψ as an indicator function 1 g to determine whether to apply the predicted residual actions. The policy effectively being deployed to autonomously complete tasks is an integration of base policy π B and residual policy π R , gated by g:π deployed = π B ⊕ 1 g π R . Ajoint position controller is used during deployment.We use the Isaac Gym [9] simulator. Proximal policy optimization (PPO) [81] is used to train teacher policies from scratch with task-specific reward functions and curricula. Student policies 3 ExperimentsWe answer the following research questions through experiments.Q1: Does TRANSIC lead to better transfer performance while requiring less real-world data? Q2: How effectively can TRANSIC address different types of sim-to-real gaps? Q3: How does TRANSIC scale with human effort? Q4: Does TRANSIC exhibit intriguing properties, such as generalization to unseen objects, policy robustness, ability to solve long-horizon tasks, and other emergent behaviors? 
tasks, baselines, and evaluation protocol: 
 As shown in Fig. 3 , we consider complex contact-rich manipulation tasks (Stabilize, Reach and Grasp, Insert, and Screw) that require high precision in FurnitureBench [85] . These tasks are challenging and ideal for testing sim-to-real transfer, since perception, embodiment, controller, and dynamics gaps all need to be addressed. We collect 20, 100, 90, and 17 real-robot trajectories with human correction, respectively. These amount to 62, 434, 489, and 58 corrections for each task. See Appendix B.1 for the detailed system setup. We compare with three groups of baselines. 1) Traditional sim-to-real methods: It includes domain randomization and data augmentation [52] ("DR. & Data Aug."), real-world fine-tuning through BC ("BC Fine-Tune") and implicit Q-learning [69] ("IQL Fine-Tune"). To estimate the performance lower bound, we also include "Direct Transfer" without any data augmentation or real-world fine-tuning. 2) Interactive imitation learning (IL): It includes HG-Dagger [66] and IWR [67] . 3) Learning from real-robot data only: It includes BC [86] , BC-RNN [68] , and IQL [69] that are trained on real-robot demonstrations only. All evaluations are conducted on the real robot and consist of 20 trials starting with different objects and robot poses. See Appendix C for details.TRANSIC is effective for sim-to-real transfer and requires significantly less real-world data (Q1). As shown in Fig. 4 and Table A .XI, TRANSIC achieves the best performance on average and in all four tasks with significant margins. What are the reasons for successful transfer? We observe that adding real-world human correction data does not guarantee improvement. For example, among traditional sim-to-real methods, the best baseline BC Fine-Tune outperforms DR. & Data Aug. by 7%, but IQL Fine-Tune leads to worse performance. In contrast, TRANSIC effectively uses human correction data, which boosts average performance by 1.24×. Not only does it achieveTraditional Sim-to-Real Interactive IL Real Data Only the best transfer performance, but it also improves simulation policies the most among various sim-to-real approaches.Furthermore, TRANSIC outperforms interactive IL methods, including HG-Dagger and IWR, by 0.75× on average. Although both of them weigh the intervention data higher during training, we find that they tend to erase the original policy and lead to catastrophic forgetting. In contrast, by incorporating human correction with a separate residual policy and integrating both base and residual policies through gating, TRANSIC combines the best properties of both policies during deployment. It relies on the simulation policy for robust execution most of the time; when the base policy is likely to fail, it automatically applies the residual policy to prevent failures and correct mistakes.Finally, TRANSIC only requires dozens of real-robot corrections to achieve superior performance. However, methods such as BC-RNN and IQL trained on such a limited number of trajectories suffer from overfitting and model collapse. TRANSIC achieves 3.6× better performance than them. This result highlights the importance of first training in simulation and then leveraging sim-to-real transfer for robot learning practitioners. In summary, we show that in sim-to-real transfer, a good base policy learned from the simulation can be combined with limited real-world data to achieve success. However, effectively utilizing human correction data to address the sim-to-real gap is challenging, especially when we want to prevent catastrophic forgetting of the base policy.Dynamics Controller AssetTRANSIC is effective in addressing different sim-toreal gaps (Q2). We shed light on its ability to close each individual sim-to-real gap by creating five different simulation-reality pairs. For each of them, we intentionally create large gaps between the simulation and the real world. These gaps are applied to real-world settings, including perception error, underactuated controller, embodiment mismatch, dynamics difference, and object asset mismatch. Note that these are artificial settings for a controlled study. See Appendix C.3 for detailed setups.As shown in Fig. 5 , TRANSIC achieves an average success rate of 77% across five different simulation-reality pairs with deliberately exacerbated sim-to-real gaps. This indicates its remarkable ability to close these individual gaps. In contrast, the best baseline method, IWR, only achieves an average success rate of 18%. We attribute this effectiveness in addressing different sim-to-real gaps to the residual policy design. Zeng et al. [80] echos our finding that residual learning is an effective tool to compensate for domain factors that cannot be explicitly modeled. Furthermore, training with data specifically collected from a particular setting generally increases TRANSIC's performance. However, this is not the case for IWR, where fine-tuning on new data can even lead to worse performance. These results show that TRANSIC is better not only in addressing multiple sim-to-real gaps as a whole but also in handling individual gaps of a very different nature. TRANSIC scales with human effort (Q3). We demonstrate that TRANSIC scales better with human data than the best baseline, IWR, as shown in Fig. 6 and Table A .XII. If we increase the dataset size from 25% to 75% of the full size, TRANSIC improves on average by 42%. In contrast, IWR only achieves a 23% relative improvement. Additionally, for tasks other than Insert, IWR performance plateaus at an early stage and even starts to decrease as more human data becomes available. We hypothesize that IWR suffers from catastrophic forgetting and struggles to properly model the behavioral modes of humans and trained robots. On the other hand, TRANSIC bypasses these issues by learning gated residual policies only from human correction. Intriguing properties and emergent behaviors of TRANSIC (Q4). As shown in Fig. 7 , TRANSIC can achieve an average success rate of 75% when zero-shot evaluated on assembling a lamp. However, IWR can only succeed once every three attempts. This evidence suggests that TRANSIC is not overfitting to a particular object; instead, it has learned reusable skills for category-level object generalization. Further, with the ablation results shown in Table 1 , TRANSIC exhibits intriguing properties including effective gating, policy robustness against reduced cameras and suboptimal correction data, and consistency in learned visual features. See Appendix C.5 for detailed setups and discussions. Qualitatively, TRANSIC shows several representative behaviors that resemble humans. For instance, they include error recovery, unsticking, safety-aware actions, and failure prevention, as shown in Fig. A .12. Finally, we demonstrate that successful sim-to-real transfer of individual skills can be effectively chained together to enable long-horizon contact-rich manipulation (Fig. 8 ). See videos at transic-robot.github.io. Robot Learning via Sim-to-Real Transfer Physics-based simulations have become a driving force for developing robotic skills [6] [7] [8] [9] [10] . However, the domain gap between the simulators and reality is not negligible [40] . Successful sim-to-real transfer includes locomotion [18] [19] [20] [21] [22] [23] [24] [25] [26] [27] , dexterous in-hand manipulation [13] [14] [15] [16] [17] , and simple non-prehensile manipulation [30] [31] [32] [33] [34] [35] [36] [37] [38] [39] . In this work, we tackle more challenging sim-to-real transfer for complex whole-arm manipulation tasks and successfully demonstrate that our approach can solve sophisticated contact-rich tasks. More importantly, it requires significantly fewer real-robot data compared to the behavior cloning approach [68] . This makes solutions based on simulators and sim-to-real transfer more appealing to roboticists. The sim-to-real gaps can be coarsely categorized as follows: a) perception gap [41] [42] [43] , where synthetic sensory observations differ from those measured in the real world; b) embodiment mismatch [18, 44, 45] , where the robot models used in simulation do not match the real-world hardware precisely; c) controller inaccuracy [46] [47] [48] , meaning that the results of deploying the same commands differ in simulation and real hardware; and d) poor physical realism [49] , where physical interactions such as contact and collision are poorly simulated [87] . Traditional methods to address them include system identification [18, 30, 50, 51] , domain randomization [13, [52] [53] [54] [55] , real-world adaptation [56] , and simulator augmentation [58] [59] [60] . However, system identification is mostly engineered on a case-by-case basis. Domain randomization suffers from the inability to identify and randomize all physical parameters. Methods with real-world adaptation, usually through meta-learning [88] , incur potential safety concerns during the adaptation phase. In contrast, our method leverages human intervention data to implicitly overcome the gap in a domain-agnostic way, leading to a safer deployment.Human-in-The-Loop Robot Learning Human-in-the-loop machine learning is a prevalent framework to inject human knowledge into autonomous systems [62, 89, 90] . The recent trend focuses on continually improving robots' capability with human feedback [91] and autonomously generating corrective intervention data [92] . Our work further extends this trend by showing that sim-toreal gaps can be effectively eliminated by using human intervention and correction signals. In shared autonomy, robots and humans share the control authority to achieve a common goal [64, 65, [93] [94] [95] . This control paradigm has been largely studied in assistive robotics and human-robot collaboration [96] [97] [98] . In this work, we provide a novel perspective by employing it in the sim-to-real transfer of robot control policies and demonstrating its importance in attaining effective transfer. 
limitations and conclusion: 
 Limitations 1) Current tasks are in a single-arm tabletop scenario. Though TRANSIC can potentially be applied to more complicated robots with recent teleoperation interfaces [99] [100] [101] [102] [103] . 2) Human operators still manually decide when to intervene. This could be automated using failure detection techniques [104, 105] . 3) TRANSIC requires simulation policies with reasonable performance. Nevertheless, it is compatible with recent advances in synthesizing manipulation data [106, 107] .In this work, we present TRANSIC, a human-in-the-loop method for sim-to-real transfer in contactrich manipulation tasks. We show that combining a strong base policy from simulation with limited real-world data can be effective. However, utilizing human correction data without causing catastrophic forgetting of the base policy is challenging. TRANSIC overcomes this by learning a gated residual policy from a small amount of human correction data. We show that TRANSIC effectively addresses various sim-to-real gaps, both collectively and individually, and scales with human effort. height. The object's irregular shape limits certain grasping poses. For example, the end-effector needs to be near orthogonal to the table leg in the xy plane and far away from the screw thread. Therefore, we design a curriculum over the object geometry to warm up the RL learning. It gradually adjusts the object geometry from a cube, to a cuboid, and finally the table leg. In all curriculum stages, the reward function isr t = w distance d + w lif ted 1 lif ted + w success 1 success . (A.2)Here, w distance is the weight for distance reward, w lif ted is the reward for the leg being lifted, and w success is the success weight. d is the distance to the table leg and is calculated asd = 1 -tanh 10 4 (d eef + d lef t f inger + d right f inger + d orthogonal ) , (A.3)where d eef is the distance between the end-effector and the table leg, d lef t f inger is the distance between the left gripper tip to the table leg, d right f inger is the distance between the right gripper tip to the table leg, and d orthogonal is the difference between the current and the orthogonal grasping orientations. We set w distance = 0.1, w lif ted = 1.0, and w success = 200.0. The episode length is 50. One episode terminates upon success or timeout. 
: 
 Here, the robot attempts to first align the light bulb with the base and then insert and screw the light bulb into the base. c) A human operator monitors robot behaviors, intervenes, and provides online correction through teleoperation when necessary. Human data are collected to train a residual policy to tackle various sim-to-real gaps in a holistic manner. d) The simulation and the residual policies are integrated together during test time to achieve a successful sim-to-real transfer for contact-rich tasks, such as screwing a light bulb into the base. 
 
 Appendix: In this task, the robot needs to insert a pre-grasped table leg into the far right assembly hole of the tabletop, while the tabletop is already stabilized. The tabletop is initialized at the coordinate (0.53, 0.05) relative to the robot base. We then randomly translate it with displacements sampled from U(-0.02, 0.02) along x and y directions. We also apply random Z rotation with values drawn from U(-45°, 45°). We further randomize the robot's pose by adding noises sampled from U(-0.25, 0.25) to joint positions. The task is successful when the table leg remains vertical and is close to the correct assembly position within a small threshold. We design curricula over the randomization strength to facilitate the learning. The following reward function is used: , where we set β = 10 -3 . We use the Adam optimizer [112] with a learning rate of 10 -4 during training. We periodically roll out student policies in simulation for 1, 000 episodes. We then select the checkpoint that corresponds to the highest success rate to use as the base policy in the real-world learning stage. r t = w distance d + w success 1 success , (A.4)In this section, we provide details about real-world learning, including the hardware setup, humanin-the-loop data collection, and residual policy training.As shown in Fig. A .3, our system consists of a Franka Emika 3 robot mounted on the tabletop. We use four fixed cameras and one wrist camera for point cloud reconstruction. They are three RealSense D435 and two RealSense D415. There is also a 3d-printed three-sided wall glued on top of the table to provide external support. We use a joint position controller from the Deoxys library [115] to control our robot at 1000 Hz.Wrist Camera Fixed Wall We use multi-view cameras for point cloud reconstruction to avoid occlusions. Specifically, we first calibrate all cameras to obtain their poses in the robot base frame. We then transform captured point clouds in camera frames to the robot base frame and concatenate them together. We further perform cropping based on coordinates and remove statistical and radius outliers. To identify points belonging to the gripper so that we can add gripper semantic labels (Sec. A.4.2), we compute poses for two gripper fingers through forward kinematics. We then remove measured points corresponding to gripper fingers through K-nearest neighbor, given fingers' poses and synthetic point clouds. Subsequently, we add semantic labels to points belonging to the scene and synthetic gripper's point clouds. Finally, we uniformly down-sample without replacement. We opt to not use farthest point sampling [116] due to its slow speed. One example is shown in This data collection procedure is illustrated in Algorithm 1. As shown in Fig. A .5, we use a 3Dconnexion SpaceMouse as the teleoperation device. We design a specific UI to facilitate the synchronized data collection. Here, the human operator will be asked to intervene or not. The operator answers through keyboard. If the operator does not intervene, the base policy's next action will be deployed. If the operator decides to intervene, the SpaceMouse is then activated to teleoperate the robot. After the correction, the operator can exit the intervention mode by pressing one button on the SpaceMouse. We use this system and interface to collect 20, 100, 90, and 17 trajectories with correction for tasks Stabilize, Reach and Grasp, Insert, and Screw, respectively. We use 90% of them as training data and the remaining as held-out validation sets. We visualize the cumulative distribution function of human correction in The residual policy takes the same observations as the base policy (Table A .VI). Furthermore, to effectively predict residual actions, it is also conditioned on base policy's outputs. Its action head outputs eight-dim vectors, while the first seven dimensions correspond to residual joint positions and the last dimension determines whether to negate base policy's gripper action or not. Besides, a separate intervention head predicts whether the residual action should be applied or not (learned gated residual policy, Sec. 2.4). a B ← a B ∼ π B (o) o next ← E.deploy(a B ) ▷ human decides intervention or not 1 H ← π H .intervene(o, o next ) if 1 H then q pre ← E.robot state ▷ deploy human correction a H ← a H ∼ π H (o, o next ) o next ← E.deploy(a H ) q post ← E.robot state ▷ update dataset D H ← D H ∪ q pre , q post , 1 H , o end ▷ update the next observation o ← o next endFor tasks Stabilize and Insert, we use a PointNet [82] as the point-cloud encoder. For tasks Reach and Grasp and Screw, we use a Perceiver [83, 84] as the point-cloud encoder. Residual policies are instantiated as feed-forward policies in all tasks. We use GMM as the action head and a simple two-way classifier as the intervention head. Model hyperparameters are summarized in Table A To train the learned gated residual policy, we first only learn the feature encoder and the action head.We then freeze the entire model and only learn the intervention head. We opt for this two-stage training since we find that training both action and intervention heads at the same time will result in sub-optimal residual action prediction. We follow the best practice for policy training, including using learning rate warm-up and cosine annealing [117] . Training hyperparameters are listed in Table A .X. In this section, we provide details about our experiment settings and evaluation protocols.As shown in Fig. 3 , we quantitatively benchmark four tasks. They are fundamental skills required to assemble a square table from FurnitureBench [85] . We randomize objects' initial poses during evaluation.• Stabilize: The robot pushes the square tabletop to the right corner of the wall such that it remains stable in following assembly steps.• Reach and Grasp: The robot reaches and grasps the table leg. It needs to properly adjust the end effector's orientation to avoid infeasible grasping poses.• Insert: The robot inserts the pre-grasped table leg to the far right assembly hole of the tabletop.• Screw: The robot's end-effector is initialized close to an inserted table leg and it screws the table leg clockwise into the tabletop.We evaluate all methods on four tasks for 20 trials. Each trail starts with different objects and robot poses. We make our best efforts to ensure the same initial settings when evaluating different methods. Specifically, we take pictures for these 20 different initial configurations and refer to them when resetting a trial. See Figs. A.14, A.15, A.16, A.17 for initial configurations of tasks Stabilize, Reach and Grasp, Insert, and Screw, respectively. We follow Liu et al. [91] to label reward for IQL. Full numerical results are provided in Table A .XI. We explain how different sim-to-real gaps are created.Perception Error This is done by applying random jitter to 25% points from point clouds with probability P = 0.6, as visualized in Fig. A.7. We test this sim-to-real gap on the task Reach and Grasp. The jittering noise is sampled independently from the distribution N (0, 0.03). We clip the noise to be within the ± 0.03 range. Underactuated Controller This is done by making the joint position controller less accurate. We test this gap on the task Insert. We emulate an underactuated controller through early stopping.Concretely, at every time a new joint position goal q goal is set, we record the distance to the goal in configuration space d q = ∥qq goal ∥ and sample a factor Γ ∼ U(0.80, 0.95). The controller will stop reaching the desired goal once it achieves Γ progress, i.e., stop early when ∥qq goal ∥ ≤ (1 -Γ)d q . Embodiment Mismatch This is done by changing the robot gripper to be shorter length as demonstrated in Fig. A.9. We test this gap on the task Screw. We notice that the 9 cm length difference incurs a significant gap.Dynamics Difference This is done by changing object surfaces and increasing friction. We test this gap on the task Stabilize. Concretely, we attach friction tapes to the square tabletop's surface to increase friction, hence change the dynamics (Fig In Table A .XII, we show quantitative results for scalability with human correction dataset size on four tasks.We introduce the learned gated residual policy in Sec. 2.4 where the gating mechanism controls when to apply residual actions. To assess the quality of learned gating, we compare its performance with an actual human operator performing gating. Results are shown in Table 1 (row "w/ Human Gating"). It is evident that the learned gating mechanism only incurs negligible performance drops compared to human gating. This suggests that TRANSIC can reliably operate in a fully autonomous setting once the gating mechanism is learned.We investigate the policy robustness against 1) point cloud observations with inferior quality by removing two cameras, and 2) suboptimal correction data with noise injection. We remove two cameras and only keep three. Note that this is the same number of cameras as in FurnitureBench [85] .For tasks other than Insert, we keep the wrist camera, the right front camera, and the left rear camera.For the task Insert, we keep two front cameras and the left rear camera. We simulate suboptimal correction data by injecting noise into residual actions a R . This noise is of large magnitude, which follows the normal distribution with zero mean and standard deviation corresponding to 5% of the largest residual action in the dataset. Results are shown in Table 1 (rows "Reduced Cameras" and "Noisy Correction"). We highlight that TRANSIC is robust to partial point cloud inputs caused by the reduced number of cameras. We attribute this to the heavy point cloud downsampling employed during training. Fishman et al. [118] echos our finding that policies trained with downsampled synthetic point cloud inputs can generalize to partial point cloud observations obtained in the real world without the need for shape completion. Meanwhile, when the correction data used to learn residual policies are suboptimal, TRANSIC only shows a relative decrease of 6% in the average success rate. We attribute this to the advantage of our integrated deployment-when the residual policy behaves suboptimally, the base policy could still compensate for the error in subsequent steps. Stabilize TRANSIC 35% 80% 80% 100% 100% IWR [67] 70% 75% 80% 65%Reach and Grasp TRANSIC 60% 65% 80% 90% 95% IWR [67] 60% 65% 40% 40%Insert TRANSIC 5% 20% 35% 40% 45% IWR [67] 5% 15% 30% 40% Screw TRANSIC 35% 50% 65% 75% 85% IWR [67] 20% 40% 40% 40%To learn consistent visual features between the simulation and reality, we propose to regularize the point cloud encoder during the distillation stage. As shown in Table 1 (row "w/o Regularization"), the performance significantly decreases without such regularization, especially for tasks that require fine-grained visual features. Without it, simulation policies would overfit to synthetic point-cloud observations and hence are not ideal for sim-to-real transfer.We examine the distribution of the collected human correction dataset. During the human-in-theloop data collection, the probability of intervening and correcting is reasonably low (P correction ≈ 0.20). This is consistent with our intuition that, with a good base policy, interventions are not necessary for most of the time. However, they become critical when the robot tends to behave abnormally due to unaddressed sim-to-real gaps. Moreover, as highlighted in Reasons for the proposed action space distillation are twofold. The first is mainly because an OSC is hard to sim-to-real transfer, while a joint position controller can be seamlessly transferred. As suggested in Nakanishi et al. [74] , an OSC requires accurate modeling of robot parameters, such as the task-space inertia matrix and gravity compensation. System identification helps but is insufficient. Furthermore, it is often the case that given the same joint torque, the end-effector moves differently in simulation and the real world. Because an OSC uses a task-space error to compute joint torques, this will lead to large joint position deviation. shows no sign of improvement. In contrast, TRANSIC takes around 3 days to train on NVIDIA GeForce RTX 3090 GPUs. Therefore, the distillation is important to make the training feasible. other hand, by training the residual policy only on human corrections, we offload learning whether to apply the correction to a gating function and thus reduce the negative effects of unbalanced data.We show statistics about task length from FurnitureBench [85] We adopt an intervention-based learning framework [66, 67, 91] where a human operator can intervene and take control during the execution of the robot base policy π B . Denote the human policy as π H , the following combined policy is deployed during data collection:π deployed = 1 H π H + 1 -1 H π B , (A.7)where 1 H is a binary function indicating human interventions. Introducing a trajectory distribution q(τ ) that consists of two observation-action distributions generated by the robot ρ B and human operator ρ H , the original RL objective leads to the maximization of a variational lower bound on logarithmic return [67, 121] :EQUATIONwhere p π θ is the induced trajectory distribution. While the human operator optimizes Eq. A.8 through intervention and correction, the robot learner maximizes it through θ = arg max Various intervention-based policy learning methods have been derived by weighting observationaction pairs in Eq. A.9 differently. For example, HG-Dagger [66] completely ignores robot data D B and only trains on human data D H that contain intervention samples. This is equivalent to q(τ ) ∝ ρ H . Intervention Weighted Regression (IWR) [67] balances the data distribution by emphasizing human intervention: q(τ ) ∝ αρ H + ρ B with α = |D B |/|D H |. Non-intervention-based methods such as traditional behavior cloning (BC) [86] only learn on D H with full human demonstrations instead of intervention. This effectively sets q(τ ) ∝ ρ H .Robot Learning via Sim-to-Real Transfer Physics-based simulations [6-10, 49, 122-124] have become a driving force [1, 2] for developing robotic skills in tabletop manipulation [125] [126] [127] [128] , mobile manipulation [129] [130] [131] [132] , fluid and deformable object manipulation [133] [134] [135] [136] , dexterous in-hand manipulation [13] [14] [15] [16] [17] , locomotion with various robot morphology [18] [19] [20] [21] [22] [23] [24] [25] [26] 137] , object tossing [80] , acrobatic flight [28, 29] , etc. However, the domain gap between the simulators and the reality is not negligible [10] . Successful sim-to-real transfer includes locomotion [18] [19] [20] [21] [22] [23] [24] [25] [26] [27] , in-hand re-orientation for dexterous hands where objects are initially placed near the robot [13] [14] [15] [16] [17] , and non-prehensile manipulation limited to simple tasks [30] [31] [32] [33] [34] [35] [36] [37] [38] [39] . In this work, we tackle more challenging sim-to-real transfer for complex manipulation tasks and successfully demonstrate that our approach can solve sophisticated contact-rich manipulation tasks. More importantly, it requires significantly fewer real-robot data compared to the prevalent imitation learning and offline RL approaches [68, 69, 86] . This makes solutions that are based on simulators and sim-to-real transfer more appealing to roboticists.Sim-to-Real Gaps in Manipulation Tasks Despite the complex manipulation skills recently learned with RL in simulation [138] , directly deploying learned control policies to physical robots often fails. The sim-to-real gaps [10, 40, 44, 139] that contribute to this performance discrepancy can be coarsely categorized as follows: a) perception gap [18, [41] [42] [43] , where synthetic sensory observations differ from those measured in the real world; b) embodiment mismatch [18, 44, 45] , where the robot models used in simulation do not match the real-world hardware precisely; c) controller inaccuracy [46] [47] [48] , meaning that the results of deploying the same high-level commands (such as in configuration space [140] and task space [141] ) differ in simulation and real hardware; and d) poor physical realism [49] , where physical interactions such as contact and collision are poorly simulated [87] .Although these gaps may not be fully bridged, traditional methods to address them include system identification [18, 30, 50, 51] , domain randomization [13, [52] [53] [54] [55] , real-world adaptation [56] , and simulator augmentation [58] [59] [60] . However, system identification is mostly engineered on a case-bycase basis. Domain randomization suffers from the inability to identify and randomize all physical parameters. Methods with real-world adaptation, usually through meta-learning [88] , incur potential safety concerns during the adaptation phase. Most of these approaches also rely on explicit and domain-specific knowledge about tasks and the simulator a priori. For instance, to perform system identification for closing the embodiment gap for a quadruped, Tan et al. [18] disassembles the physical robot and carefully calibrates parameters including size, mass, and inertia. Kim et al. [32] reports that collaborative robots, such as the commonly used Franka Emika robot, have intricate joint friction that is hard to identify and randomized in typical physics simulators. To make a simulator more akin to the real world, Chebotar et al. [39] deploys trained virtual robots multiple times to refine the distributions of simulation parameters. This procedure not only introduces a significant real-world sampling effort, but also incurs potential safety concerns due to deploying suboptimal policies. In contrast, our method leverages human intervention data to implicitly overcome the transferring problem in a domain-agnostic way and also leads to a safer deployment.Human-in-The-Loop Robot Learning Human-in-the-loop machine learning is a prevalent framework to inject human knowledge into autonomous systems [62, 89, 90] . Various forms of human feedback exist [63] , ranging from passive judgement, such as preference [142] [143] [144] [145] [146] [147] [148] [149] [150] [151] and evaluation [152] [153] [154] [155] [156] [157] , to active involvement, including intervention [158] [159] [160] and correction [161, 162] . They are widely adopted in solutions for sequential decision-making tasks. For instance, interactive imitation learning [66, 67, 91, 163] leverages human intervention and correction to help naïve imitators address data mismatch and compounding error. In the context of RL, reward functions can be derived to better align agent behaviors with human preferences [145, 148, 149, 152] . Noticeably, recent trend focuses on continually improving robots' capability by iteratively updating and deploying policies with human feedback [91] , combining active human involvement with RL [162] , and autonomously generating corrective intervention data [92] . Our work further extends this trend by showing that sim-to-real gaps can be effectively eliminated by using human intervention and correction signals.In shared autonomy, robots and humans share the control authority to achieve a common goal [64, 65, [93] [94] [95] . This control paradigm has been largely studied in assistive robotics and human-robot collaboration [96] [97] [98] . In this work, we provide a novel perspective by employing it in sim-to-real transfer of robot control policies and demonstrating its importance in attaining effective transfer. 8th Conference on Robot Learning (CoRL 2024), Munich, Germany.https://developer.nvidia.com/physx-sdkhttps://github.com/frankaemika/franka_ros 


Decision: ### Strengths:

- Re: Clarity: The approach is conceptually easy to understand and well-executed (R-5kGt, R-pa2D); R-pa2D additionally mentions that the method is novel and addresses the critical problem of sim-to-real transfer in a task-agnostic manner. 
- Re: Experiments: R-5kGt and R-pa2D suggest that the experiments are convincing, with a good choice of baselines, where the method is demonstrated to work well in both simulation and hardware, outperforming strong baselines. R-pa2D and R-LZdQ report that the manuscript considers a wide variety of tasks.
- Re: Writing: all reviewers state that the paper is written clearly, with a good literature review (R-5kGt, R-pa2D) and helpful supplementary material (R-5kGt).
- Re: Methodology: R-LZdQ discussed that the paper's approach is more effective and data-efficient compared to traditional sim-to-real methods like domain randomization and data augmentation.
    

### Weaknesses:

- Re: Clarity: R-5kGt and R-LZdQ mentioned concerns related to missing details on the dataset DpcdD^{pcd}Dpcd, the gating function, and the specifics of human intervention frequency; also, the description of the POMDP formulation and the privileged observation for the teacher policy needs clarification (R-pa2D).
- Re: Task / Domain: R-pa2D expressed concerns that the tasks are all in tabletop settings and may be easier due to the green walls used for alignment, which could simplify the tasks. R-LZdQ mentioned concerns that the success rates for some tasks are low, making it difficult to assess if the issue lies with the base policy or the sim-to-real transfer.
    
- Re: Methodology: The need for a separate gating function is questioned, as it should ideally be incorporated into the residual policy (R-LZdQ). R-LZdQ was concerned that the number of robot trajectories with human correction varies across tasks without clear guidelines on determining this number for new tasks.
    
- Re: Evaluation: The paper lacks detailed comparisons of success rates between simulation-trained base policies and real-world performance (R-LZdQ). More analyses on the sensitivity of tasks to sensor noise and the potential for learned gating functions to automate human intervention decisions are needed (R-LZdQ). The overall impact is seen as incremental rather than significantly advancing the field (R-LZdQ).

### Post-rebuttal Meta Review Statement

I applaud the reviewers and authors for engaging in a productive discussion that resulted in several improvements to the paper, especially as it relates to the clarification of the gating function and addition of new experiments (in response to R-5kGt — enabling them to raise their score), reformulation of the problem (in response to R-pa2D), and the addition of further analysis on variable task complexity (in response to R-LZdQ — enabling them to raise their score).

I recommend for Accept as Poster.

I encourage the authors to follow through with all the promised changes and to incorporate all additional reviewer requests, to improve the paper even further.

Reviews: review: title: title: Interesting approach that is well-written and rigorously evaluated, with a few approach details that could be clarified
paper_summary: paper_summary: This paper introduces TRANSIC, an approach for sim-to-real policy transfer by learning from online correction and intervention provided by humans. The concept relies on first learning a base policy from simulation, and then learning a residual policy online from human corrections. In simulation, the base policy is trained using a teacher-student framework, where the teacher policy is trained with RL and Operational Space Control, and this teacher policy is then distilled into a student policy that operates on depth clouds with a joint position action space. During the online rollout, a human provides interventions if the robot execution fails. These interventions are stored within a correction dataset and used to train a residual policy. The final policy is synthesized between the base policy and residual policy using a gating function. Experimental results were conducted on real robot platforms and include four tasks relating to furniture assembly and a lamp assembly task.
main_review: main_review: # Summary
**Strengths:**
- The approach is conceptually easy to understand and appears to be executed well.
- The experiments are rigorous and convincing, with a robust choice of baselines and demonstration on real robot platforms.
- The paper is excellently written. The literature review appears quite thorough.
- The supplemental videos are very helpful.

**Limitations:**
- It would be beneficial and strengthen the paper to provide additional clarifications to certain aspects of the approach, either in the main paper or in the appendix. (See Approach Clarifications for more notes)
- There are a few writing suggestions for improvements. (See Writing for more notes)

The paper is quite rigorous as written, although the paper can potentially continue to be strengthened through clarifying certain aspects of the approach detailed below.

# Detailed Review

**Approach Clarifications:**
- There isn't much explanation for the dataset D^pcd. It would be helpful to provide more details of this dataset in the main paper or appendix.
- The gating function in Sec 2.4: can you provide more details about this in the main paper? Is this an indicator function? It is not very clear how specifically the gating function works, and this would be helpful to clarify in the main paper as it relates to when/how the residual policy is used.
- According to Sec 2.3, is the human is always involved in the deployment for each timestep? It says "at time step t", but it isn't clear how often this happens. I understand from the limitations that the human operators manually decide when to intervene, but it wasn't clear how often the human is queried for corrections.
- How is the success of a particular task evaluated (e.g., as in Figure 4)?
- Regarding the approach being "domain-agnostic" (line 57 and elsewhere): Although it is agnostic from the robot's perspective, would this not be domain-specific from the human standpoint? If so, the wording "...addressed by humans in a domain-agnostic manner" may be misleading, since it does assume the human has domain-specific knowledge to be able to provide the right interventions to the robot. Perhaps this can be rephrased as saying the interventions are domain-agnostic from the robot's perspective.
- For the lamp assembly task, can you clarify what specific aspect makes it unseen (as stated in Figure 6)? Were point clouds of the lamp objects first collected and used in D^pcd?

**Writing:**
- "Appendix Sec E" -> Could also phrase this (and later references to the Appendix) as Appendix E
- Figure 4b may be better as a separate figure, since it addresses a different question (Q2) than Figure 4a (Q1)
- From my understanding, it seems that IWR is used as the only baseline for Q2, Q3, and Q4 because it is the best baseline (line 224). If so, it would be helpful to put this explanation a bit earlier in the text than when it appears.
questions: questions: - Have there been any issues with distribution shifts with point clouds from sim to real? Or, if there were, are such issues correctable with TRANSIC?
- Can you provide more details for D^pcd? Would it be the case that it is required for objects to have real point cloud measurements prior to training the student policy?
- Can you clarify the details of the gating function?
- Can you confirm how often the human is monitoring the executed policy? Is the entire rollout?
limitations: limitations: 3
review_summary: review_summary: Original Recommendation: As written, the paper has many strong points, such as rigorousness of experimental assessment, quality of writing, and thoroughness of literature review. There are a few aspects of the approach that would be helpful to clarify, either in the main paper or in the appendix. I believe addressing the clarifications can further strengthen the paper, so I would be willing to revise my recommendation based on these clarifications. Updated Recommendation: The author response in the rebuttal has addressed all of my concerns, so I have updated my recommendation to strong accept.
title: title: A novel and effective method to address sim-to-real transfers using online human corrections to train a residual policy
paper_summary: paper_summary: The paper introduces a novel method to tackle sim-to-real problems by introducing humans in the loop to provide correctional action. Human feedback are then used to train a residual policy that will combine with the simulation policy to work well in a set of contact-rich manipulation tasks.
main_review: main_review: ### Clarity:
- The paper is well-written, with sufficient details provided in the appendix

### Originality:
- As far as I know, the idea in this paper is novel

### Significance:
- Achieving sim-to-real robustly is a critical problem in robot learning. The paper introduces a very effective method to address the issue in a task-agnostic manner
- The benchmark is nice, challenging enough, and probably will be useful for future research
- The structure of the agent is well-thought, and leverages most state-of-the-art components

### Strengths:
- The set of tasks is challenging, and the method is demonstrated to work well in simulation and hardware, outperforming strong baselines
- The experiments are well organized and very thoughtfully designed. There are great details in the appendix

### Weaknesses:
- Even though the tasks are challenging, they are tabletop and mostly require near top-down actions. Moreover, the existence of the green walls, which can used for alignment, makes the tasks easier to perform
questions: questions: - In 2.1, the POMDP formulation is over-killed. The policy is reactive (mapping from the current observation to action), so it is really just an MDP with a high-dimensional state (i.e., point cloud)
- In 2.2, the text description should clarify what kind of privileged observation the teacher is allowed to use
- In Algorithm 1, why the input to \pi^H include both o and o^{next}?
- It is stated in the paper that the residual policy is trained to predict q^{post} – q^{pre}; why such residual policy during deployment can be summed with the base policy to have a working control signal? In other words, it is unclear about the relationship between q^{pre} and the output from the base policy.
- I am curious how easy it is to use the space mouse to generate 6D intervention control inputs.
- I’m surprised that using OSC action space makes sim-to-real transfers more challenging. I thought that high-level action would make transfers easier as we masked out the kinematics, which might differ between sim and real. Can the authors comment on that?
- When compared with baselines, the authors should provide the learning curves (e.g., in the appendix) instead of just providing the final performance. It is useful for future research relying on this approach
limitations: limitations: 3
review_summary: review_summary: The paper presents an effective method to address sim-to-real transfers using human corrections. The approach is novel, and well-supported by experiments.
title: title: Review for Submission 52
paper_summary: paper_summary: This work proposes to learn a residual policy from human correction to improve sim-to-real transfer of the simulation-based manipulation policies.
main_review: main_review: This work proposes a human-in-the-loop method for improving sim-to-real transfer of simulation-trained manipulation policies. The advantages of the proposed method are: 1. more effective than traditional sim-to-real methods like domain randomization and data augmentation 2. more data-efficient than imitation learning/offline RL-based methods. 

**Strengths:**
1. The problem is well-motivated and the authors not only included a wide variety of tasks but also evaluated different aspects of the sim-to-real gap for robot manipulation.
2. The writing is clear and easy-to-follow. Problems are well-defined and methods are clearly described.
3. The authors compared to a lot of baselines in their evaluations and have done sufficient demonstrations on hardware.

**Weakness:**
1. The overall achieved task success rate is quite low. Without simulation-based policy success rate reported as comparison, it is hard to tell if it is resulting from bad base policy or bad sim-to-real transfer.
2. The number of robot trajectories with human correction seems to differ from task to task, and it is not clear how to determine this number / the upper bound for a new task.
3. It is unclear why a separate gating function is needed. Ideally, the learned residual policy should already incorporate the functionality of the gating function during training (e.g., when correction is not needed, the residual policy predicts close-to-zero action outputs).
questions: questions: 1. The authors have done a lot of experiments on the real robot. Given the focus of this work is to show effective sim-to-real transfer with the proposed method, what are the success rates of the simulation-trained base policies?

2. In perception error experiments, among all 4 tasks, *insert* and *screw* might be the most sensitive to sensor noise. Would it be possible to show the results on these two tasks?

3. In the limitation section, the authors have mentioned that "Human operators still manually decide when to intervene during correction data collection." After a few trials, can the learned gating function be also used to determine whether human intervention is needed?

4. What are the controller gains used in simulation-based training and real-world deployment? Is there any tuning involved for real-world deployment?
limitations: limitations: 2
review_summary: review_summary: Weak reject.