MILES: Making Imitation Learning Easy with Self-Supervision

Authors: georgios  papagiannis
edward  johns
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-05-24 11:07:45

Abstract: Data collection in imitation learning often requires significant, laborious human supervision, such as numerous demonstrations, and/or frequent environment resets for methods that incorporate reinforcement learning. In this work, we propose an alternative approach, MILES: a fully autonomous, self-supervised data collection paradigm, and we show that this enables efficient policy learning from just a single demonstration and a single environment reset. MILES autonomously learns a policy for returning to and then following the single demonstration, whilst being self-guided during data collection, eliminating the need for additional human interventions. We evaluated MILES across several realworld tasks, including tasks that require precise contact-rich manipulation such as locking a lock with a key. We found that, under the constraints of a single demonstration and no repeated environment resetting, MILES significantly outperforms state-of-the-art alternatives like imitation learning methods that leverage reinforcement learning. Videos of our experiments and code can be found on our webpage: www.robot-learning.uk/miles.

Main paper: 
 introduction: 
 Imitation learning is frequently described as a convenient way to teach robots new skills. But is this true in practice? Behavioral cloning (BC) methods leverage supervised learning to train robust policies, but doing so typically requires hundreds or thousands of demonstrations per task [1, 2] to collect a sufficiently diverse training dataset. Imitation learning methods that leverage Reinforcement learning (RL) offer a solution to this as policies can be learned autonomously through random exploration and reward functions can be inferred from a few demonstrations [3, 4] . However, unlike supervised learning methods, policy learning with RL can be unstable [5] , and random exploration makes data collection inefficient. Besides, RL typically requires repeated environment resetting, and so in practice, it is often equally or even more laborious than simply providing numerous demonstrations [6] . Instead, learning from a single demonstration appears to be the most convenient form of imitation learning due to its effortlessness, but policies learned this way suffer from covariate shift [7] . As such, imitation learning today is not as easy as we would like it to be: significant human supervision is still required for data collection either via demonstrations, environment resetting, or both.Motivated by this, we propose MILES, a framework that makes imitation learning easy by enabling humans to teach robots tasks effortlessly with just a single demonstration while requiring no prior task knowledge, and only a single environment reset. MILES learns robot skills by collecting augmentation trajectories in a self-supervised manner, which demonstrate to the robot how to return to, and then follow the single human demonstration, as shown in Figure 1 . MILES leverages these trajectories to train a policy using BC, consequently inheriting the power and ease of supervised learning. However, unlike common BC methods, the self-supervised data collection replaces the need to collect multiple demonstrations, and compared to BC from a single demonstration, MILES does not suffer from covariate shift as its data densely covers the space around the demonstration. Similarly to RL-based imitation learning methods, MILES collects data autonomously, but instead of being guided by random exploration, MILES' data collection is highly efficient as it benefits from self-labeled data that directly guides the robot back to the demonstration. Additionally, exploration can cause disturbances to the environment and thus requires repeated environment resetting. However, as we later explain, MILES' self-supervised data collection procedure strategically shapes policy learning to be independent of such laborious environment resets. Consequently, MILES aims to incorporate the benefits of training BC policies with supervised learning on large demonstration datasets with the benefits of autonomous data collection similar to RL methods in the setting where only a single demonstration is available.Through our real-world experiments, we find that when only a single human demonstration is available, without additional human effort such as repeated environment resetting, self-supervised data collection outperforms recent alternatives in imitation learning that leverage RL and replay-based imitation learning. MILES can learn a suite of versatile skills, shown in Figure 1 , ranging from those involving interactions with movable, articulated objects such as opening the lid of a box, to those involving precise, contact-rich interactions with objects rigidly mounted to a surface, such as inserting and twisting a key in a lock. Each of these is learned from a single demonstration, no further human input, and around 30 minutes of self-supervised data collection. 
related work: 
 As follows, we ground our work relative to methods that can learn manipulation skills from a single demonstration, unlike most approaches that require large demonstration datasets [1, 8, 9] .Imitation learning from prior knowledge. An effective way to compensate for the lack of large demonstration datasets is to leverage prior task knowledge such as access to ground truth object poses [10, 11] or by meta-learning policies by first pretraining on large demonstration datasets [12, 13] . However, precise knowledge of the objects' poses is hard to obtain in practice and meta-learning methods are often limited to tasks similar to the ones seen in the demonstrations. Instead, MILES can learn a new task from just a single demonstration without any prior object or task knowledge.Imitation learning via Reinforcement learning (RL). RL-based imitation learning methods from a single demonstration learn to follow that demonstration by minimizing a similarity metric between the trajectories of the learned policy and the demonstration [3, 4, 14, 15] . Other RL methods that learn from demonstrations infer rewards through alternative means, like goal images [16] . Though effective, these methods are often inefficient as they rely on random exploration and repeated environment resets which require significant human effort. Instead, our self-supervised data collection makes MILES highly efficient and eliminates the need for repeated environment resetting.Imitation learning via pose estimation and demonstration replay. Replay-based imitation learning methods first estimate and move the robot to a similar pose relative to the objects of interest as in the demonstration and then replay the demonstrated robot actions [17, 18, 19, 20, 21] . While these methods are the most efficient in terms of human time, small errors in pose estimation cause errors to compound during demonstration replay, leading to task failures [2] . And even under the assumption of perfect pose estimation, potential environment collisions may prevent the robot from reaching the desired pose or may perturb the objects such that replaying the demonstration fails to complete the task. Instead, MILES' self-supervised data collection procedure retains the human-time efficiency of pose estimation methods, while learning to avoid unnecessary collisions, and minimizing or completely eliminating open-loop replay errors depending on the task.Imitation learning by demonstration augmentation. Demonstration augmentation approaches like DAgger [7] and DART [22] mitigate covariate shift by relying on laborious interactive expert queries to expand the known state distribution of a policy. And methods that do not require an interactive expert still rely on multiple demonstrations or task-specific optimizations [23, 24, 25, 26, 27] which limit their practical application. Instead, MILES is a fully autonomous method that uses self-supervision to augment a single demonstration and can learn a wide range of diverse tasks. 
miles: making imitation learning easy with self-supervision: 
 As follows we describe MILES, a framework that makes imitation learning easy by leveraging a single human demonstration as guidance to collect self-supervised data demonstrating to the robot how to return to, and then follow the demonstration. By training a policy with behavioral cloning on that data, MILES learns to perform a task from a range of initial states and object poses.Assumptions. Our setup assumes access to a wrist camera that is rigidly mounted to the robot's endeffector (EE) and (optionally) a sensor that measures external forces and torques. We follow prior work [19, 2, 17] and assume that each task is object-centric, such that only the task-relevant object is in camera view during data collection and the demonstration can be expressed relative to a single object, where combining several such tasks results in a multi-stage task. Additionally, as we are interested in dealing with all types of tasks, including those that require contact-rich manipulation, we control our robot using an impedance controller. k := {(w τ k m , o τ k m , a τ k m )} M m=1is a robot trajectory whose final, M th state corresponds to a k th state in the demonstration, i.e., (wτ k M , o τ k M ) = (w ζ k , o ζ k ).That is, each augmentation trajectory guides the robot to some k th state in the demonstration from any state (w τ k m , o τ k m ) ∈ τ k . We can fuse each augmentation trajectory with the demonstration segment following eachk th state, {(w ζ n , o ζ n , a ζ n )} N n=k ⊆ ζ,to create a new demonstration that demonstrates to the robot how to return to and then follow the human demonstration as shown in Figure 3 . By collecting augmentation trajectories that densely cover the state space near the demonstration we can create a dataset of new demonstrations which we can leverage to train a policy using standard BC methods. But how do we create such a dataset of augmentation trajectories automatically? Data Collection. We achieve this by collecting data in the simplest possible way. An overview of our data collection procedure is shown in Figure 2 . To generate a τ k , from a demonstration waypoint w ζ k , we first move the robot to some random pose near the demonstration. The robot then attempts to return back to w ζ k in a straight line, while recording RGB and forcefeedback observations {o τ k m } M m=1 , and actions {a τ k m } M m=1 automatically generated by computing the EE's relative movement between consecutive timesteps, as shown in Figure 2 (3) (gripper actions are copied directly from the demonstrated action). In practice, the actual trajectory τ k may not be a straight line, as collisions with external objects can yield a more complex, curved (3) Then, the robot (autonomously) collects self-supervised data. Several augmentation trajectories are collected for each demonstration waypoint until an environment disturbance is detected or sufficient data is collected for all waypoints. Each augmentation trajectory is either a straight line, if the motion occurs in free space, or a more complex, curved path as the augmentation trajectory can be reshaped by collisions with the environment (e.g., with the lock as shown above). ( 3To collect an augmentation trajectory, the robot first moves from a demonstration waypoint to a random pose. (c) Then, it attempts to return back to the waypoint while recording RGB images and force-torque feedback . (d) After completing the trajectory, we check whether the achieved state meets the conditions of reachability and environment disturbance.path due to the robot's compliance; an example of this happening for a locking a lock task is shown in Figure 2 (3). This is particularly useful for contact-rich tasks, as these trajectories contain information on overcoming potential collisions or large friction areas by regulating force when in contact with an object. After executing the potential augmentation trajectory we check that it is valid and that it can be fused with the demonstration by evaluating two key conditions, that of reachability and environment disturbance which we describe in section 3.3. If both conditions are met we store that augmentation trajectory, otherwise, it is discarded. We repeat this process several times for each demonstration state, starting from the first waypoint in the demonstration w ζ 1 and gradually progress through the demonstration, as shown in Figure 2 (3), until: (1) an environment disturbance is detected, in which case we stop the data collection and store the demonstration timestep where the disturbance occurred, denoted R, and the actions ζ remaining := {a ζ n } N n=R for the remaining demonstration states for which no data is collected; or (2) we have collected a prespecified number of Z augmentation trajectories for each of the N demonstration states, in which case R = N .At the end of the data collection process, every augmentation trajectory is fused with ζ to form a new demonstration 3 . And as a result, we obtain a datasetζ k = {(o τ k m , a τ k m )} M m=1 ∪ {(o ζ n , a ζ n )} R n=k as shown in FigureD new = {ζ, ζ 1 1 , ζ 2 1 , .., ζ Z 1 , ..., ζ Z-1 R , ζ Z R }.Every trajectory in D new corresponds to a new demonstration, automatically created, that solves the task up to the R th state in the demonstration. Further details concerning a practical implementation of our data collection procedure and pseudocode can be found in our supplementary material section A.1.As follows, we introduce two conditions that determine whether an augmentation trajectory can be fused with the human demonstration. Consider an augmentation trajectory τ k aimed at returning the robot to the k th demonstration state, (w ζ k , o ζ k ): (1) Condition 1, Reachability: After executing the augmentation trajectory, the EE's pose must equal the pose of the demonstration waypoint w ζ k . This equality can be verified trivially using proprioception. In many scenarios, the environment's dynamics (e.g. collisions) or inevitable systematic inaccuracies in a robot's controller may prevent it from reaching its target waypoint w ζ k . Thus, if w τ k M = w ζ k ,Training. We train a separate policy π for each task as an LSTM network with behavioral cloning that receives as input the RGB and force-torque observations in the dataset D new and regresses the corresponding actions. Note that D new does not contain proprioception data, allowing our policies to generalize to different object poses naturally due to the use of our wrist camera.Inference. We deploy our policy π to solve a task up to the R th demonstrated state. If no environment disturbance occurred during data collection for that task, then the R th state is the final state in the demonstration and π solves the task completely in a closed-loop manner. Otherwise, after π completes the task up to the R th state, the remaining demonstrated action segment ζ remaining is replayed. More details regarding how we deploy our policy, the network architecture, and how we detect that π has reached the R th state can be found in the supplementary material section A.4. 
experiments: 
 We evaluate MILES through several real-world experiments. Through these experiments, we aim to answer the following questions: 1) Can MILES solve a range of everyday tasks and how does it perform against baselines that learn from a single demonstration? 2) How does MILES perform under different component ablations? 3) How important are vision and force modalities to the performance of MILES? and 4) How does MILES perform under different sizes of self-supervised data? Videos of our experiments can be found on our webpage: www.robot-learning.uk/miles Implementation Details. For our experiments, we use a FLIR camera mounted to the wrist of a Franka Emika Robot. We sample Z = 10 augmentation trajectories for each demonstration waypoint (≈ approximately 1 minute of data collection per waypoint). This number is set arbitrarily, but as we show later in our ablations, some tasks may require less data. We collect augmentation trajectories with initial poses near the demonstration in the range of 4cm and 4 degrees around each demonstration waypoint. As commonly done in the literature [3, 16, 17, 20] , we provide our demonstrations starting near each object. At deployment, to reach the object from far away we first estimate the object's pose using pose estimation and approach it before switching to MILES. Finally, we set the environment disturbance threshold θ to 0.94 for all our tasks. Additional details on the pose estimation method we use and how to set each one of MILES' parameters can be found in our supplementary material sections B. 4.1 Can MILES solve a range of everyday tasks and how does it perform against baselines that learn from a single demonstration?In this experiment, we assess the performance of MILES across a diverse set of everyday tasks and compare it to various baseline methods capable of learning from a single demonstration. We select seven distinct tasks, shown in Figure 4 , spanning a range of complexities, each learned from a single demonstration. The tasks are: 1) Lock with key; 2) Insert USB; 3) Plug into socket; 4) Insert power cable; 5) Twist screw; 6) Bread in toaster; 7) Open lid. Tasks 1-4 are contact-rich and and our setup follows prior work on contact-rich manipulation [16, 30, 31, 32, 33] and the NIST benchmark [34] . Similar to prior work [35, 36] , we focus our evaluation on single-task performance, but also report results on generalization, robustness to distractors, and multi-stage tasks in section 4.5. We provide a detailed description of each task in the supplementary material section B.3 including data collection times, information on which tasks involve demonstration replay, which tasks stopped data collection due to an environment disturbance and information on the length of each human demonstration.Baseline Methods. We chose 4 baselines that can learn from a single demonstration without prior task knowledge, similar to MILES. (1) Demo Replay which involves replaying the demonstrated actions.(2) Pose Estimation + Demo Replay follows [17] and leverages MILES' data to perform pose estimation followed by demonstration replay. (3) Reset Free Residual RL replays the demonstration's actions at each timestep and learns corrective actions on top using DDPG [37] . Like MILES, no human intervenes to reset the environment during training, hence we call it "Reset Free". Finally, (4) Reset Free FISH uses the state-of-the-art RL-based imitation learning method FISH [3] but no human intervenes to reset the environment during training. Further, implementation details on the baselines can be found in our supplementary material section B.4.Evaluation. For a fair evaluation, we carefully tuned each method's hyperparameters. Additionally, each learning-based baseline collected the same number of observations as MILES during data collection for each task. We evaluated each method's success rate across 20 trials. For each trial we randomized the relative starting pose of the robot and the task-relevant object equivalently across all methods within a sphere of 20cm around the object as long as the object was visible to the camera. Finally, we emphasize that for all evaluations both MILES and the baselines predict 6-DoF actions.Results. Table 1 presents the success rates of MILES and the 4 baselines across 7 tasks. As shown MILES obtains a high success rate across all tasks. Specifically, MILES obtains an average of 87% success rate across all tasks with the "Open lid" task having the highest success rate of 100%. From the contact-rich tasks, inserting the key and locking the lock achieved the best performance despite the task's low tolerance and complex interaction, with an impressive 90% success rate. The lowest success rate is observed in the USB insertion task, where MILES obtains 70%, a performance dip we attribute to the task's low tolerance of less than 1mm. Despite the USB task, for the remaining tasks that required high precision MILES was able to complete them consistently well.The next best-performing method, Pose Estimation + Demo Replay, obtained an average success rate of 71%. Our experiments showed that small errors in object pose estimation led to failures due to the compounding errors of demonstration replay, as observed in prior work [2, 20] . This is particularly evident in tasks requiring precise manipulation including "Lock with key", "Insert USB" and "Twist screw". Instead, while MILES also replays part of the demonstration for some tasks, the fact that it does so for a much shorter horizon allows it to obtain considerably higher success rates.Reset Free FISH failed to solve the majority of tasks, yielding an average performance of 10% across all evaluated scenarios. During training, for several tasks, the policy caused significant disturbances to the environment that made policy learning very hard without manual resetting. We believe that this is the central reason behind Reset Free FISH's low performance Additionally, compared to the original FISH implementation, as we train policies that predict lower success rate, averaging 7% due to similar challenges with Reset Free FISH. As anticipated, the Demo Replay baseline was the least effective among the baselines. Simply replaying the demonstration without pose estimation or online action correction leads to task failures.While the RL methods would have benefited from manual environment resets during training, our results demonstrate that in our "easy" imitation learning setting, where only a single demonstration is available without additional human interventions, MILES significantly outperformed the baselines.Simulation Evaluation. Finally, to support the reproduction of our results, we conducted additional experiments on our method and the baselines in simulation using 5 tasks from the RLBench benchmark [38] which can be found in the supplementary material section C.1.This section studies MILES' performance by ablating 4 different components of the method: (1)No Environment Disturbance: we ablate the environment disturbance condition by not checking for that condition when collecting augmentation trajectories. ( 2) No reachability: we ablate the reachability condition by relabeling each observation's action (of the existing MILES data), to move the robot to the nearest waypoint in the demonstration based on their Euclidean distance. If the constraint for reachability is not important, then simply moving from each pose to the nearest waypoint in the demonstration in a straight line would be sufficient to solve a task. ( 3) No sequence: we recollect MILES' data but instead of collecting Z augmentation trajectories for the first demonstration state, then progressing to the second state and so on, we collect data without following the demonstration's waypoint sequence and instead follow a random one. (4) No Memory: For this ablation we retrain a network on the existing MILES data that does not account for history.Results. Table 2 shows MILES performance after ablating each component. Collecting augmentation trajectories for each demonstration state in a random order (No Sequence), with an average success rate of 43%. Additionally, not checking for the environment disturbance condition (No Environment Disturbance) appears to cause significant performance degradation for the tasks where an environment disturbance occurred during data collection, corresponding mostly to the non-contact rich tasks. On the other hand, not checking for the reachability condition (No Reachability) also lowers performance, particularly for the precise, contact-rich tasks, indicating that the reachability condition is the most important when learning tasks requiring precise manipulation. Finally, the lower performance obtained by removing the LSTM (No Memory) demonstrates the performance benefits of training memory-based networks on datasets collected using MILES.Lock with key Insert USB Plug into socket Insert power cable In this section, we ablate the use of vision and force feedback as policy inputs for the four contact-rich tasks from our earlier experiments. We retrain and evaluate two policies: one using only vision and one using only force. The results, shown in Figure 5 , indicate that the vision-based policy improves MILES' performance in the "Insert USB" and "Plug into socket" tasks but reduces performance in the other two tasks. This suggests that force feedback might not consistently benefit MILES, possibly due to its noisy signal which makes it hard to distinguish between different environment states. The force-based policy, however, fails almost completely. This is expected as force feedback is zero in free space and can be ambiguous due to symmetries in object surfaces. Overall, while force feedback aids performance in some tasks, it is not always necessary. Vision remains the most crucial modality to MILES' high performance.In this section, we ablate the dataset size used to learn four tasks by splitting their original datasets into chunks containing 75%, 50%, and 25% of the original data. We evaluated the best and worst performing contact-rich tasks ("Lock with key" and "Insert USB") and non-contact-rich tasks ("Open lid" and "Twist screw"). Data collection times for each task can be found in the supplementary material. Figure 6 shows that for high tolerance tasks like "Open lid," MILES achieves a 100% success rate even with 25% of the data, corresponding to only 8 minutes of data collection. However, for precise tasks, success rates decrease as dataset size is reduced. Notably, for "Lock with key" and "Twist screw," reducing the dataset to 50% results in a high failure rate. To summarize, we observe that high tolerance tasks are likely to require less data, and in practice only a few minutes of data collection time. Instead, for high-precision tasks, like inserting a USB, the dataset size appears to impact MILES' performance significantly.Can MILES generalize? We conduct additional experiments on generalization for MILES in the supplementary material section D.1. Can MILES perform multi-stage tasks? We provide additional experimental results on how MILES performs on multi-stage tasks in the supplementary material section D.2. Is MILES robust to distractor objects? We conduct additional experiments studying MILES' performance in the presence of distractors in the supplementary material section D.3. What if MILES stops data collection early due to a detected environment disturbance? We provide a discussion and intuition on MILES' behavior in scenarios where data collection stops early in our supplementary material section D.4. 
discussion: 
 Limitations. We now highlight some important limitations of our method. Firstly, MILES' reliance on a wrist camera enables MILES to obtain spatial generalization, however, simultaneously this limits its field of view and its applicability to larger task spaces. Future work could address this by incorporating an external camera to initially approach an object before switching to the wrist camera, similarly to [20] . Secondly, while MILES is robust to distractors at deployment before data collection begins it requires a human to set up the robot's workspace such that only the task-relevant object is in camera view for the policy to achieve spatial generalization. While this requires only a few seconds of human time, future work could address this by extending MILES to incorporate segmentation methods, similar to [20, 39] , that segment the task-relevant object in the dataset. Similarly, to address any unwanted collisions that MILES could cause in the presence of multiple objects, future work could study incorporating an external camera during self-supervised data collection to plan and collect collision-free augmentation trajectories. Thirdly, our current implementation of MILES trains a separate policy for each task and hence it is unclear how well MILES would generalize to completely new tasks. In future work, we aim to study this by training a single monolithic policy on MILES' self-supervised data combined with replay-trajectory retrieval [40] similarly to our generalization task in section 4.5.We introduced MILES, a framework that makes imitation learning easy. MILES requires only a single demonstration and collects self-supervised data that demonstrate to the robot how to return to and then follow that demonstration. Subsequently, this enabled us to obtain manipulation skills comprising either (1) a single end-to-end policy trained with behavioral cloning, or (2) a combination of an end-to-end policy and demonstration replay. Our real-world experiments showed that when only a single demonstration is available, self-supervised data enable the acquisition of skills that achieve considerably improved performance compared to several state-of-the-art baselines. MILES can learn everyday tasks, ranging from opening a lid, to using a key to lock a lock, to inserting a USB stick into a port, requiring complex and precise contact-rich manipulation. states and as a result how to solve the task. This is because we only know how to solve a task by learning how to follow the demonstration after returning to it. But if an environment disturbance has occurred (e.g., the rectangular object has fallen), following the demonstration's actions no longer leads to task completion. Hence, if data collection continued, all future augmentation trajectories would contain invalid observations and actions, as they would demonstrate behavior that does not solve the task that the human demonstrated. This is why we stop data collection after detecting an environment disturbance.We demonstrate in this section several examples of possible environment disturbances and how we can detect them using the DINO features on the toy screwdriver used in our experiments. We use the screwdriver as an example as during data collection for the "Twist screw" task, data collection was stopped due to an environment disturbance caused at the grasped screwdriver. Additionally, disturbances caused in the grapsed objects are often the most subtle, and as such make for the most interesting cases. shows the screwdriver object (not grasped). All the other figures depict the screwdriver as it appears in the view of the wrist camera when grasped by the robot. Figure A.1 (a) shows a "Demonstration Image" and a "New Image" that depicts the DINO Cosine similarity (higher better) when no environment disturbance has occurred, i.e., the grasp has not changed. The heatmap demonstrates the similarity between each corresponding patch between the "Demonstration Image" and the "New Image" (the cosine similarity reported is the mean of these). As shown, the cosine similarity (0.961) is greater than our universal threshold θ of 0.94 (for more details please see experiments section 4). The reason it is not a perfect 1.0 is due to noise and light changes as the photos were captured at different moments in time. Training: To train our manipulation policy we leverage the dataset D new comprising the fused augmentation trajectories with the demonstration as described in section 3.4. MILES' policy π comprises either (1) an end-to-end network trained with behavioral cloning (BC) or (2) an end-toend network trained with BC combined with demo replay, which is utilized when data collection was interrupted due to a detected environment disturbance. robot to a pose relative to the object of interest as depicted in the first state in the demonstration and replays the complete demonstration. We chose this baseline compared to alternatives, as it leverages task-specific data allowing it to achieve very precise pose estimation.Reset-Free FISH [3] . For Reset-Free FISH we use the implementation provided by the authors as it can be found in: https://github.com/siddhanthaldar/FISH. We only changed the implementation such that the policy always predicts 6-DOF actions instead of constraining the output to specific DOFs, as doing so assumes access to prior task knowledge. To learn residual actions on top of the demonstration we tested both using demo replay as the base policy, as well as VINN [44] but found that demo replay led to better performance.To aid other researchers in reproducing our results, we conducted additional simulation experiments on the RLBench benchmark [38] on 5 tasks, specifically: 1) 'Insert Onto Square Peg', 2) 'Lightbulb In', 3) 'Pick Up Cup', 4) 'Turn Tap' and 5) 'Lamp On'. We performed an identical evaluation to our real-world experiments where we performed 20 evaluation trials for each method. Additionally, we used the images captured only from the wrist camera in RLBench. During training we allowed each method to collect the same amount of data and we did not perform any environment resets during training/data collection for any methods. The results can be seen in Table 4 . As shown, MILES significantly outperforms the baselines, while the relative performance when comparing all methods remained relatively unchanged compared to our real-world results.Similarly to our real-world experiments, the reinforcement learning baselines obtained poor performance for reasons in line with the ones discussed in our experiments section 4.1. Specifically, during training we observed that for the tasks 'Insert Onto Square Peg' and 'Lightbulb In' a random gripper action drops the grasped object during exploration and the policy never manages to grasp it again during training without a reset in the given training time. For the 'Pick Up Cup' task, the reinforcement learning policy knocks the cup off the table during exploration, consequently never learning something useful. For the 'Turn Tap' task the RL policies never learned to properly grasp and rotate the handle and for the 'Lamp On' task, only Reset Free FISH managed to learn a policy that obtains 20% success rate in the given training time. As discussed in our real-world experiments, if instead we had allowed environment resets and more training time that would have resulted in significantly higher success rates for the RL baselines, compared to their current performance.Since MILES uses BC to train policies, existing generalization results for BC [9, 1] also apply to MILES. For tasks that include demonstration replay following the closed-loop policy, MILES can generalize to new objects by retrieving the replay trajectory of the most similar object in the existing demonstrations, similar to prior work [19] . To test this, we tasked MILES with throwing markers of different colors into differently shaped and colored bins, shown in Figure 4 (8). Trained on five bins (marked green) and tested on two new bins (marked red), MILES achieved an 80% success rate on the pink bin and 60% on the gray bin, over 10 trials each starting from poses where simple demonstration replay would fail. The data collection time for this task was on average 34 minutes for each bin and an environment disturbance was detected for each bin. To determine which remaining actions to replay for the previously unseen bins, we selected the remaining actions from the bin in A6the training set whose RGB image in the demonstration has the highest similarity in terms of DINO features with the bin during deployment, inspired by prior work [19] . Videos exhibiting MILES generalization on the two test case bins can be found on our webpage: www.robot-learning.uk/miles.To evaluate MILES' ability to solve multi-stage tasks, we tasked MILES with picking up the plastic bread shown in Figure 4 (6) (as part of the "Bread in Toaster" task) and inserting it into the toaster.To achieve this we broke the task down into two stages: first, we provided a demonstration showing how to pick up the bread and trained MILES. Then, we used the policy already trained on the "Bread in Toaster" task to finish the task. To link the two stages together, first the policy to pick up the bread is deployed. After, the execution ends, the robot returns to its default position. Then, the pose estimation method described in section B.1 is deployed to approach the toaster, and then the policy trained with MILES is deployed to insert the bread into the toaster. Videos exhibiting MILES' multi-stage task performance on picking up and inserting the bread into the toaster can be found on our webpage: www.robot-learning.uk/miles.We found that performing standard image augmentation techniques, including changing the brightness, contrast, noise, cropping random image parts, etc. allowed MILES to be robust to distractor objects, as shown in the videos provided on our webpage: robot-learning.uk/miles.There is no requirement as to how early MILES may stop data collection due to an environment disturbance, as long as it has collected sufficient augmentation trajectories for at least the first demonstration waypoint. During data collection, MILES can effectively learn a policy even if an environment disturbance occurs early. Unlike RL, MILES learns to solve the task closed-loop up to the demonstration waypoint where the disturbance was detected, after which it replays the demonstration. This is because MILES collects data progressively for each demonstration waypoint, rather than rolling out a policy all at once like RL. Consequently, during data collection, if a disturbance occurs as early as (for example) near the 2nd waypoint, MILES will still know how to get to the 1st waypoint during deployment, where it will replay the demonstration.Overall, MILES can handle early environment resets during data collection. While as with the majority of learning-based methods, the more the data the better the performance, as such the later an environment disturbance occurs in the data collection process the better. However, MILES can still learn a robust policy as long as sufficient data has been collected at least for the 1st demonstration waypoint. This is typically trivial as most human demonstrations naturally begin by controlling the robot in free-space far from the object of interest, before interacting with it.E Detailed Pseudocode I τ k M ← Capture RGB wrist-cam image // M is the M th (final) timestep of τ k 16: Disturbance ← CheckEnvDisturbance(o ζ k , I τ k M , θ) (Alg. 5) 17:if Disturbance is True then 8th Conference on Robot Learning (CoRL 2024), Munich, Germany. 
 
 Appendix:  


Decision: The paper propose a new algorithm for imitation learning, MILES, that requires a single demonstration and a single reset. MILES designs a self-supervised data collection process, that generates a large number of augmented demonstration trajectories. Reviews appreciate the simplicity of the method and real-world testing that includes many ablations. Reviews have also noted many shortcomings: the strong assumptions made by the proposed method, the relationships to some past works, and some counterintuitive trends in the experiments.

Authors provided a response that was considered by `dHST`, `3nym`, `onvu`, who all appreciated the response and converged on accept leaning ratings. `zWvW` gave an original rating of weak reject and didn't look through the author response or participate in any discussion. The AC also considered the paper, the reviews, and the author response. 

The AC believes that the proposed method makes strong assumptions and would only work on tasks of a specific form. It seems that MILES is limited to free space motion to a precise pose (e.g. getting the key to the keyhole) and execution of a fixed motion (e.g. turning the key). MILES excels at getting to this precise pose. However, many tasks commonly considered in robot learning literature (e.g. pick an object and place it at a target location, or pick a cup and pour its content into another cup) require careful execution even after the first contact is made with the environment. Other tasks such as tossing a ball require carefully following the precise trajectory to impart sufficient momentum to the ball and not just reaching a precise pose. MILES methodology exploits this specific form of the task and the paper doesn't demonstrate if it will work for more general tasks. The paper's introduction should prominently note these limitations.

Furthermore, even for tasks of this form, the methodology is not robust to accidental changes in the environment, e.g. the screw driver changing pose in the gripper. Author response argues that this can be detected and data collection can be terminated. If this were to happen early on in data collection, then MILES performance wouldn't be much better than just training on the 1 demonstration.

On the experimentation front,
- Generalization to distractor objects in experiments presented in the paper likely comes from the use of wrist camera as the distractor objects are not even visible in the wrist camera image. These claims should be appropriately toned down.
- Pose Estimation + Demo Replay already does quite comparably with MILES only outperforming it on 4/7 tasks. Authors note that Pose Estimation + Demo Replay fails because one can't precisely recover the pose. However, the MILES method itself relies on accurate estimation of whether the environment has changed or not using DINO ViT features, and authors note that it is actually quite sensitive to even slight changes in the environment. If this is so, wouldn't a "pose estimation + local search in a small neighborhood to maximize DINO ViT similarity + demo replay" already be quite strong? Furthermore, this process could be done for multiple way points along the way as opposed to just the first way point.

Reviews: review: title: title: Interesting paper, but raises some questions
paper_summary: paper_summary: The authors propose MILES, which they explain to be an algorithm for performing imitation from a single demonstration without requiring manual environment resets. The crux of MILES appears to be the self-supervised data collection process, which generates a large number of augmented demonstration trajectories. The authors perform an impressive set of on-robot experiments showing that MILES performs well for robot manipulation tasks.
main_review: main_review: (++) The authors take on an important and well-motivated problem in robot learning, ie, how to better perform imitation learning given the practical costs incurred by the paradigm (eg, needing to provide myriad demonstrations and needing to babysit the robot during data collection). 

 

(++) The proposed approach’s reported ability to successfully imitate while relying mainly on self-supervised data collection is attractive. 

 

(++) The authors perform an impressive set of on-robot manipulation experiments in order to demonstrate the efficacy of their algorithm, including a convincing set of ablation experiments for MILES itself. 

 

(--) Even after reading the supplemental material, I find myself questioning the validity of the proposed approach when it comes to the so-called “environment disturbances.” As I understand it, the proposed data collection process will stop when such a disturbance is detected, where these disturbances are when, eg, an object falls over due to suboptimal robot behavior during the self-supervised data collection phase. This seems counter-intuitive since it seems that states in which a disturbance can occur would be some of the more challenging parts of the task, and so it’s not clear how we can hope that MILES would learn to imitate in tasks that run the risk of disturbance in such cases. 

 

(--) I am concerned about the fairness of the experiments given that MILES would appear to have a pretty strong advantage in terms of its ability to reset itself to the demonstration states. What enables the approach to achieve these states? That is, _how_ is the robot moved to a random state near a demonstration waypoint? Apologies if I missed an explanation in the paper, but given that I didn’t find one, I’m assuming some kind of auxilliary controller is used to do this, and I find myself wondering if any of the comparison methods are able to leverage this kind of policy and, if not, whether or not that presents an issue in terms of fair comparison. 

 

(--) While the method is presented as one for imitation learning in general, I can’t help but wonder if the method is really applicable to other settings beyond the manipulation domain studied here. In particular, I am skeptical that MILES would be practical for navigation tasks, where it seems very challenging to be able to place the robot at a random state close to a demonstration waypoint without first having some kind of policy capable of navigation in the first place. 

 

(-) The proposed method seems similar—at least at a high level—to FAIL [Sun et al., ICML 2019]. It would have been nice to have seen this somehow addressed in the paper.


**Post-response update**: Thanks to the authors for the in-depth response to my review here. The responses to address a number of my concerns, and I understand that the authors will add several new discussion points to the paper in light of the questions I raised. As such, I'm happy to bump up my score.
questions: questions: (1) Can the authors provide any intuition as to how MILES can succeed in imitation for tasks in which environment disturbance is possible at points throughout the demonstration trajectory? 

 

(2) Can the authors comment on how MILES is able to guide the robot to a random location near a particular waypoint and how this ability does not cause any issues of fairness when it comes to comparing to the other approaches presented? 

 

(3) How does the propose approach compare to FAIL?
limitations: limitations: 3
review_summary: review_summary: While there’s a lot to like about this paper (well-motivated, novel, and fairly impressive expeirmental results), I have some important reservations that I think need to be addressed before the paper could be accepted.
title: title: Simple but effective idea, but it may benefit from some clarification
paper_summary: paper_summary: This paper proposes a data augmentation technique where a robot learns how to get back to a state in a demonstrated trajectory. This allows the robot to perform the task even with a single demonstration, as the robot can follow the rest of the demonstrated trajectory after it gets back to it.
main_review: main_review: Overall, the idea looks sound and straightforward. It is simple at its core, but it works. It is evaluated well through multiple analyses. But I have two clarity questions. During training, you emphasized there are no resets; my understanding is that this means:

[1] The human demonstrates a trajectory, and this is recorded.

[2] The human or the robot resets the scene, but the scene is the same as the demonstrated one (e.g., the manipulated object is precisely in the same pose).

[3] As the joint/pose trajectory that the robot needs to follow to accomplish the task is the same as the demonstrated one, any augmentation trajectory that converges to it should be able to achieve the task (as long as it does not disturb the environment).

These augmented trajectories will contain observations that the robot may encounter when the object's pose changes, allowing the method to generalize. Alternatively, if the robot had perfect object pose estimation (As in the evaluation “Pose Estimation + Demo Replay”), it could go back to the original demonstration (relative to the object) and follow it. But this may also not be possible as the robot may hit/collide with the environment. Is this correct? If so, this information can be further highlighted in motivation or something like problem formulation.

Regarding resetting, I have one question. In “Validity Condition for Augmentation Trajectories,” condition 2 says that if the environment is disturbed, corresponding augmentation trajectories will not be recorded. However, if the environment is disturbed, this may also cause the original demonstration to be invalid (e.g., the robot accidentally changing the object's position in the scene). Wouldn’t this require a reset? Wouldn't this be a limitation? If not, maybe the authors can clarify this.
questions: questions: I don’t have any questions for rebuttal, but the authors may clarify my questions from my review.
limitations: limitations: 3
review_summary: review_summary: Authors propose a simple but effective idea for data-efficient imitation learning. Paper is well written and method is well demonstrated.
title: title: Review
paper_summary: paper_summary: This paper proposes a method for imitation learning that uses a single demonstration and a single environment reset. It collects data in a self-supervised way by sampling poses around the demonstrated trajectory, moving to the sampled pose, and then storing data that shows how to return from the sampled pose to the demonstration trajectory. The method is shown to outperform alternatives on a set of real-world manipulation tasks given the constraints of a single demonstration and single environment reset.
main_review: main_review: **Strengths**

The problem setting that the paper studies is of great interest and well motivated. Environment resets are costly in the real world, as are collecting demonstrations, so being able to use a single environment reset and single demonstration is compelling. The method is simple and easy to understand, and the paper is well-written and easy to follow. The results showcase the effectiveness of the proposed approach.

**Weaknesses**

The paper could benefit from a more complete and upfront discussion of assumptions and limitations. Some of the items mentioned in the "Setup" section (3.1) seem like they should be made into explicit assumptions, and might also be limitations of the approach. For example, the use of a wrist camera seems necessary for spatial generalization, and the object that must be manipulated must be in the view of the wrist camera at the start of interaction. Furthermore, there are likely additional assumptions -- for example, there shouldn't be other objects in view of the wrist camera (otherwise, this would prevent spatial generalization). One more example is the one in Figure 2 (3.2). When the augmentation trajectory makes contact with objects, the key could move in-hand when making contact with the lock - this would permanently change the state of the environment, and likely require a manual human reset. The only reason it does not is because the key seems rigidly attached to the robot hand. 

Other limitations of the approach are also not clear. For example, some kinds of environment disturbances, might be difficult to detect via DINO features - for example a subtle in-hand orientation change, such as the key in the locking task. It also seems like the value of force-torque data in this framework might be limited. Naively, it seems like data collection would stop at the first state where significant contact occurs, in which case replay would be used at inference. Figure 4 also suggests that force-torque data may not be strictly necessary.

One other concern is the range of evaluations presented in the paper. It would be great to see results on common simulation benchmarks to showcase the generality of the approach, to provide a more thorough comparison to other common imitation learning datasets and tasks, and to enhance the reproducibility of the proposed approach. Some suggestions include the robomimic datasets (https://robomimic.github.io/) or RLBench (https://sites.google.com/view/rlbench).

**Post-Rebuttal**: I have updated my score based on the additional discussion with authors and additional results that have been provided.
questions: questions: In addition to the points raised in the weaknesses above, I have further questions:

- How does the robot attempt to move back from the random pose to the pose in the dataset during self-supervised data collection? More details on the controller used here, and how the linear path is broken up into robot control commands would be useful.

- Why is MILES better than pose estimation + demo replay - that seems like it should be about as good. Would pose estimation + demo replay obtain higher success by using a bottleneck pose closer to the object of interest? What are some examples of MILES' shorter horizon that provide an advantage over the pose estimation + demo replay baseline?

- It would be great to get more details on the LSTM policy architecture and how it consumes observations at train-time and inference. One common LSTM-based policy is BC-RNN (https://arxiv.org/abs/2108.03298) which unrolls an LSTM policy during a rollout sequentially at each timestep (the hidden state is updated while playing each action) and then reset periodically every N steps. Another approach is to use an explicit observation history with frame stacking, and to unroll the LSTM completely over the observation history window to produce a final action. Which approach is being used here?
limitations: limitations: 1
review_summary: review_summary: The paper presents a simple and effective approach, but more evaluation (especially on settings that can be reproduced by others) and discussion on assumptions and limitations would improve the quality of the paper greatly.
title: title: Impressive results, intuitive method,  some unstated assumptions.
paper_summary: paper_summary: This paper proposes MILES, a single-demonstration imitation learning method which collects its own data with just 1 environment reset to train a policy to solve the intended task. The paper demonstrates results on a variety of precise insertion tasks.
main_review: main_review: ## Strengths

**Writing:** The paper is very well written; I was able to thoroughly understand everything with ease.

**Method:** The method is intuitive and combines many techniques to make one algorithm, MILES, which can teach policies given 1 demonstration. 

**Experiments:** Real-robot experiments are quite comprehensive and demonstrate that MILES works well, especially against a SOTA 1-demonstration baseline (FISH).

**Supp**: Videos are nice! Appendix answers some important questions.

## Weaknesses

**Missing/Difficult to find information:** A couple things I wish the paper made much clearer in the main paper/experiments section. I searched through the appendix (skimmed through, not extremely thoroughly) but didn’t find this info:

- What is $R$ for each task?
- What are the demonstration lengths for each task?

**Heavy unstated assumptions:** Related to the above points about $R$ and the demonstration lengths, it seems that the tasks are relatively short horizon: single item pick and place or insertion tasks where the robot starts really close to the insertion location with the object already in hand. Supporting this is how the “Demo Replay” baseline can get 25% success rates on Open lid (which shouldn’t be too hard) but also even 15% on Bread in Toaster, which should be quite difficult. With longer horizons, errors would accumulate and the demo replay should be getting 0% across the board. Meanwhile, the intro and abstract sell MILES as a general method that would work everywhere with just 1 demonstration. This could at least be discussed in the limitations section, or the authors can prove me wrong with additional experiments.

- Additionally, the method would fail in cluttered settings where many type of straight line movement/deviations would cause the robot to knock things over and greatly slow down self-supervised data collection.
- Tasks like opening drawers/cabinets would also be difficult as the gripper would be holding onto the item to be opened while the method tries to pull the arm away to create new trajectories; the final image would look very different from the demonstration image and this would also make self-supervised data collection very difficult.

**Minor Issues:**

- Figure 2 is too long and the text is too small, yet, it has lots of wasted white space. Please make a figure that fits within the margins and uses all the available white space to make the text larger? Perhaps it can be simplified a bit too.
- Table 1 and 2 also extend beyond the margin…. simply wrapping in a `\resizebox{}` can fix this.

---------------
AFTER REBUTTAL: I have now updated to a strong accept.
questions: questions: See above weaknesses.
limitations: limitations: 2
review_summary: review_summary: Weak accept because I believe the paper should fully clarify the assumptions and make some important information regarding the experimental settings easier to find.