MILES: Making Imitation Learning Easy with Self-Supervision
Authors: Georgios Papagiannis
Edward Johns
Venue: CoRL 2024 (2024)
Keywords: Imitation Learning
Robotic Manipulation
Self-Supervised Data Collection
TL;DR: MILES is an imitation learning method that collects data in a self-supervised manner to train policies that can learn complex manipulation skills from a single demonstration.

Abstract: Data collection in imitation learning often requires significant, laborious human supervision, such as numerous demonstrations, and/or frequent environment resets for methods that incorporate reinforcement learning. In this work, we propose an alternative approach, MILES: a fully autonomous, self-supervised data collection paradigm, and we show that this enables efficient policy learning from just a single demonstration and a single environment reset. MILES autonomously learns a policy for returning to and then following the single demonstration, whilst being self-guided during data collection, eliminating the need for additional human interventions. We evaluated MILES across several realworld tasks, including tasks that require precise contact-rich manipulation such as locking a lock with a key. We found that, under the constraints of a single demonstration and no repeated environment resetting, MILES significantly outperforms state-of-the-art alternatives like imitation learning methods that leverage reinforcement learning. Videos of our experiments and code can be found on our webpage: www.robot-learning.uk/miles.

MILES: Making Imitation Learning Easy with Self-Supervision
Georgios Papagiannis and Edward Johns
The Robot Learning Lab Imperial College London, UK papagi annis2l (er21eiingg eimperial..... ac. uk
Abstract: Data collection in imitation learning often requires significant, labori- ous human supervision, such as numerous demonstrations, and/or frequent envi- ronment resets for methods that incorporate reinforcement learning. In this work, we propose an alternative approach, MILES: fully autonomous, self-supervised data collection paradigm, and we show that this enables efficient policy learn- ing from just single demonstration and single environment reset. MILES au- tonomously learns a policy for returning to and then following the single demon- stration, whilst being self-guided during data collection, eliminating the need for additional human interventions. We evaluated MILES across several real- world tasks, including tasks that require precise contact-rich manipulation such as locking lock with a key. We found that, under the constraints of a single demonstration and no repeated environment resetting, MILES significantly out- performs state-of-the-an alternatives like imitation learning methods that leverage reinforcement learning. Videos of our experiments and code can be found on webpage: www.robot-learning.uk uk/miles.
Human Demonstration
(cl Invtation Learning Reinforcement Leaming
Lock the lock with key
Behavioura
Behavisural Cloring
Sell
Dantorsfaators
Pepienation
Dara Collection
Mo Exploctio
No Covartase Shaft
Contras Silt
Covartte Stat.
No Covesaaa
Perposted mnvinmment
Reperte
Repustet
Roperti atd Envorment
will-ay
GlLssocket
power
10ste Screw
Opean
Figure 1: (a) Behavioural cloning from single demonstration to generalize to states outside the demonstration, due covariate shift. (b) Providing multiple demonstrations addresses this, but requires significant human effort. (c) While incorporatnng reinforcement learning addresses the issue of covariate shift and the need for multiple demonstrations, it re- quires frequent environment resetting and highly inefficient due random exploration. self-supervised paradigm that overcomes these issues and can learn range of complex tasks from a single demonstration and no additional human effort, by collecting augmentation trajectories that guide the rohot back the demonstration.
In MILES. propose a
Introduction
Imitation learning is frequently described convenient way to teach robots new skills. But is this true in practice? Behavioral cloning (BC) methods leverage supervised learning to train robust policies, but doing so typically requires hundreds thousands of demonstrations per task [1, 2] to collect a sufficiently diverse training dataset. Imitation learning methods that leverage Reinforce- ment learning (RL) offer solution this policies be learned autonomously through random exploration and reward functions can be inferred from a few demonstrations [3, 4]. However, unlike supervised learning methods, policy learning with RL can be unstable [5], and random exploration
8th Conference on Robot Learning (CoRI. 2024), Munich, Germany.

---

makes data collection inefficient. Besides, RL typically requires repeated environment resetting, and so in practice, is often equally or even more laborious than simply providing numerous demonstra- tions [6]. Instead, learning from a single demonstration appears to be the most convenient form of imitation learning due to its effortlessness, but policies learned this way suffer from covariate shift [7]. As such, imitation learning today is not as easy as we would like it to be: significant human supervision is still required for data collection either via demonstrations, environment resetting, or both.
Motivated by this, we propose MILES, a framework that makes imitation learning easy by enabling humans to teach robots tasks effortlessly with just single demonstration while requiring no prior task knowledge, and only a single environment reset. MILES learns robot skills by collecting aug- mentation trajectories in self-supervised manner, which demonstrate to the robot how to return to, and then follow the single human demonstration, as shown in Figure 1, MILES leverages these trajectories to train a policy using BC, consequently inheriting the power and ease of supervised learning. However, unlike common BC methods, the self-supervised data collection replaces the need to collect multiple demonstrations, and compared to BC from single demonstration, MILES does not suffer from covariate shift as its data densely covers the space around the demonstration. Similarly to RL-based imitation learning methods, MILES collects data autonomously, but instead of being guided by random exploration, MILES' data collection is highly efficient as it benefits from self- data that directly guides the robot back to the demonstration. Additionally, explo- ration can cause disturbances to the environment and thus requires repeated environment resetting. However, we later explain, MILES' self-super data collection procedure strategically shapes policy learning to be independent of such laborious environment resets. Consequently, MILES aims to incorporate the benefits of training BC policies with supervised learning on large demonstration datasets with the benefits of autonomous data collection similar to RL methods in the setting where only a single demonstration is available.
Through our real-world experiments, we find that when only a single human demonstration is avail- able, without additional human effort such as repeated environment resetting, self-supervised data collection outperforms recent alternatives in imitation learning that leverage RL and replay-based imitation learning. MILES can learn suite of versatile skills, shown in Figure 1, ranging from those involving interactions with movable, articulated objects such as opening the lid of box. to those involving precise, contact-rich interactions with objects rigidly mounted inserting and twisting a key in a lock. Each of these is learned from a single demonstration, no further human input, and around 30 minutes of self-supervised data collection.
surface, such as
2 Related Work
As follows, we ground our work relative to methods that can learn manipulation skills from a single demonstration, unlike most approaches that require large demonstration datasets [1, 8, 9].
Imitation learning from prior knowledge. An effective way to compensate for the lack of large demonstration datasets is to leverage prior task knowledge such access to ground truth object poses [10, 11] or by meta-lcarning policies by first pretraining on large demonstration datasets [12, 13]. However, precise knowledge of the objects' poses is hard to obtain in practice and meta-learning methods are often limited to tasks similar to the ones seen in the demonstrations. Instead, MILES can learn new task from just single demonstration without any prior object or task knowledge.
Imitation learning via Reinforcement learning (RL). RL-based imitation learning methods from single demonstration learn to follow that demonstration by minimizing similarity metric between the trajectories of the learned policy and the demonstration [3, 4, 14, 15]. Other RL methods that learn from demonstrations infer rewards through alternative means, like goal images [16]. Though effective, these methods are often inefficient as they rely on random exploration and repeated envi- ronment resets which require significant human effort. Instead, our self-supervised data collection makes MILES highly efficient and climinates the need for repeated environment resetting.
Imitation learning via pose estimation and demonstration replay. deplay-based imitation learn- ing methods first estimate and move the robot to similar pose relative the objects of interest as in the demonstration and then replay the demonstrated robot actions [17, 18, 19, 20, 21]. While these methods are the most efficient terms of human time, small errors in pose estimation cause errors to compound during demonstration replay, leading to task failures [2]. And even under the assumption of perfect pose estimation, potential environment collisions may prevent the robot from reaching the desired pose or may perturb the objects such that replaying the demonstration fails to complete

---

the task. Instead, MILES' self-supervised data collection procedure retains the numan-time effi- ciency of pose estimation methods, while learning avoid unnecessary collisions, and minimizing or completely eliminating open-loop replay errors depending on the task.
Imitation learning by demonstration augmentation. Demonstration augmentation approaches like DAgger [7] and DART [22] mitigate covariate shift by relying on laborious interactive expert queries to expand the known state distribution of policy. And methods that do not require an interactive expert still rely on multiple demonstrations or task-specific optimizations [23, 24, 25, 26, 27] which limit their practical application. Instead, MILES is fully autonomous method that uses self-supervision to augment a single demonstration and can learn wide range of diverse tasks.
3 MILES: Making Imitation Learning Easy with Self-Supervision
As follows we describe MILES, a framework that makes imitation learning easy by leveraging a single human demonstration guidance to collect self-supervised data demonstrating to the robot how to return to, and then follow the demonstration. By training a policy with behavioral cloning on that data, MILES learns to perform a task from range of initial states and object poses.
3.1 Preliminaries
Assumptions. Our setup assumes access to a wrist camera that is rigidly mounted to the robot's end- effector (EE) and (optionally) a sensor that measures external forces and torques. We follow prior work [19, 2, 17] and assume that each task is object-centric, such that only the task-relevant object in camera view during data collection and the demonstration can be expressed relative a single object, where combining several such tasks results in multi-stage task. Additionally, as we are interested in dealing with all types of tasks, including those that require contact-rich manipulation, we control our robot using an impedance controller.
Single Demonstration. For each task, human provides single demonstration S {(wS, of, o comprising a sequence of N waypoints wS, observations of, and actions as, as shown in Figure 2 (I). A waypoint w corresponds to the EE's 6-DoF pose at timestep n cap- tured via proprioception. An observation Sf consists of an RGB image captured from the wrist camera and force-torque measurement. We refer (wh, of)) the state of the environment at timestep An action a, contains the gripper's 's state and the 6-DoF pose tracked by the impedance controller at timestep n. expressed relative to the EE's s pose at timestep n - 1. After providing Ç the human resets the environment only once, such that if the actions in are executed, the robot would successfully perform the task; a trivial process that requires a few seconds of human time.
3.2 Self-Supervised Data Collection
Augmentation Trajectories. Given single demonstration, MILES collects dataset of aug- mentation trajectories == {rk}. where 1 S k < N and each ты : {(w T a ) ]M is robot trajectory whose final, Mn state corresponds to a his state in the demonstration, i.e., M. )Mr (wf. ). That is, each augmentation trajectory guides the robot some kn state in the demonstration from any state (w ofhh ) E Tk- We can fuse each augmentation trajectory with the demonstration segment following each hin state, {(w5, of, ) S Ç. to create new demon- stration that demonstrates to the robot how to return to and then follow the human demonstration as shown in Figure 3. By collecting augmentation trajectories that densely cover the state space near the demonstration we can create a dataset of new demonstrations which we can leverage to train policy using standard BC methods. But how do we create such dataset of augmentation trajectories automatically?
Data Collection. We achieve this by collecting data in the simplest possible way. An overview of our data collection procedure is shown in Figure 2. To generate a Tke from a demonstra- tion waypoint wfe we first move the robot some random pose near the demonstration. The robot then attempts return back to w in a straight line, while recording RGB and force- feedback observations foth JM 1m=1- and actions {ath 1M automatically generated by computing the EE's relative movement between consecutive timesteps, as shown in Figure 2 (3) (gripper actions are copied directly from the demonstrated action). In practice, the actual trajectory тą may not be straight line, as collisions with external objects can yield a more complex, curved
3

---

MILES
Sulf-Superiiedd Collection
each demonstration wayppint nibot collects severd Trojetlries
Shrilght wndectery
Comple surved Inckettrry
provides
slaped
Augrostation Troenirry folloved
Domo wappoint
Domo waypoint2
sntypoänt
intial only
Collecting an Augmentation Tiajectory: Step- haep-by-ttep Ovarview
Augnrraioo Tralectons
Chark Enainement Darutkaree
Pobet In 3
Finished Asgmantattion
Figure 2: MILES Overview: (1) First, the user provides a single demonstration and (2) resets the environment only once. (3) Then, the robot (autonomously) collects sclf-supervised data. Several augmentation trajectories are collected for cach demonstration waypoint until an environment disturbance is detected or sufficient data is collected for all waypoints. Each augmentation trajectory is cither a straight line, if the motion occurs in free space. or more complex, curved path as the the environment (e.g..
augmentation trajectory
reshaped collisions To collect an augmentation trajectory, the robot first moves from demonstration waypoint to a random pose. (c) Then, it atlempls to return back to the waypuint while recording RGB images and force-torque leedback trajectory. we check whether the achieved state meets the conditions of reachability and environment disturbance.
the lock shown above). (3) (a-b)
After completing the
comp
path due to the robot's compliance; an example of this happening for a locking a lock task shown in Figure (3). This is particularly useful for contact-rich tasks, these trajecto-
ries contain information on overcoming potential collisions or large friction areas by regulat- ing force when in contact with an object. After executing the potential augmentation trajec- tory we check that valid and that it can be fused with the demonstration by evaluating two key conditions, that of reachability and environment disturbance which we describe in tion 3.3. both conditions are met we store that augmentation trajectory. otherwise, is discarded.
We repeat this process several times for each demonstra- tion state, starting from the first waypoint in the demon-
Fusing Augmentation Trajectories with the Demonstration
stration wi and gradually progress through the demon- stration, as shown Figure (3), until: (1) an envi- ronment disturbance is detected, in which case we stop the data collection and store the demonstration timestep where the disturbance occurred, denoted R, and the ac- tions Cremaining := J for the remaining demon-
New Demonstrations
Ingrenmttion
stration states for which no data is collected; or (2) we have collected prespecified number of augmentation trajectories for each of the N demonstration states, in which case R = N.
Bogment
Demomazation Trijectry
Augmortation Trajectories
At the end of the data collection process, every augmen- tation trajectory fused with to form a new demonstra- tion &. [ a ah } J.M U {(o,, ) m as shown in Figure 3. And as a result, we obtain a dataset Dnew = Ct. ...... -211, )2). Every trajectory Drew
Target Demonntraaion Waypsams
Figure 3: After finishing the data collection, each augmentation trajectory is fused with the demonstration segment following demonstra- tion waypoint it returns to, to create a datasct of new demonstration trajectories.
corresponds to new demonstration, automatically created, that solves the task up to the Rih state in the demonstration. Further details concerning practical implementation of our data collection procedure and pseudocode can be found in our supplementary material section A.
3.3 Validity Conditions for Augmentation Trajectories
As follows, we introduce two conditions that determine whether an augmentation trajectory can be fused with the human demonstration. Consider an augmentation trajectory тıe aimed at returning the robot to the kig demonstration state, (ws, of):
(1) Condition 1, Reachability: After executing the augmentation trajectory, the EE's pose must equal the pose of the demonstration waypoint w This equality can be verified trivially using

---

Lock-with kny
Cinkect USI
Rlog into socket lnseet power cabli
Twist screwy
Bread in toaster
Open lid
Markaaiiin bin
Figure 4: The tasks used our experiments, The "Markers in Bin" is used evaluate MILES' ability to generalize (the bins marked green denote the training set, while the red denote the test set).
proprioception. In many scenarios, the environment's dynamics (e.g. collisions) or inevitable sys- tematic inaccuracies in robot's controller may prevent it from reaching its target waypoint wf.
Thus, if w * w.. the augmentation trajectory cannot return to demonstration state k, rendering the augmentation trajectory invalid.
(2) Condition 2, Environment Disturbance: While collecting Tke the robot may disturb the envi- ronment, resulting in a final observation ofr that no longer matches that of the demonstration (even if w. is reached). For instance, during data collection the robot's gripper pushes an object to a different pose than it had timestep of the demonstration, the final observation in the augmen- tation trajectory will differ from the demonstration's bis observation. Therefore, if OM + of the augmentation trajectory cannot be combined with the human demonstration to create new, valid demonstration. To detect such disturbances, we compare the cosine similarity of the DINO fea- tures [28, 29] of the RGB image _ from the demonstration's 's observation of and the image 1 after executing the augmentation trajectory. If the similarity falls below a threshold 0, we assume the environment has been disturbed and stop data collection. In the supplementary material sec- tions A.2, A.3, we provide visual explanations of our validity conditions and figures demonstrating the robustness of the DINO features in detecting varying environment disturbances for an object.
3.4 Policy
Training. We train a separate policy for each task as an LSTM network with behavioral cloning that receives as input the RGB and force-torque observations in the dataset Dnew and regresses the corresponding actions. Note that Dnew does not contain proprioception data, allowing policies to generalize to different object poses naturally due to the use of our wrist camera.
Inference. We deploy our policy æ to solve a task up to the Ru: demonstrated state. If no environ- ment disturbance occurred during data collection for that task, then the Rm state the final state in the demonstration and π solves the task completely in closed-loop manner. Otherwise, after r completes the task up to the Rs state, the remaining demonstrated action segment Gremaining is replayed. More details regarding how we deploy our policy, the network architecture, and how we detect that has reached the Ruh state can be found the supplementary material section A.4.
4 Experiments
We evaluate MILES through several real-world experiments. Through these experiments, we aim to answer the following questions: 1) Can MILES solve a range of everyday tasks and how does per- form against baselines that learn from a single demonstration? 2) How does MILES perform under different component ablations? 3) How important are vision and force modalities the performance of MILES? and 4) How does MILES perform under different sizes of self self-supe data? Videos of our experiments can be found on our webpage: www.robot-leariing.o uk/miles
Implementation Details. For our experiments, we use FL.RR camera mounted to the wrist of a Franka Emika Robot. We sample Z 10 augmentation trajectories for each demonstration way- point ( approximately I minute of data collection per waypoint). This number is set arbitrarily, but as we show later our ablations, some tasks may require less data. We collect augmentation trajectories with initial poses near the demonstration in the range of 4cm and 4 degrees around each demonstration waypoint. As commonly done the literature [3, 16, 17, 20], we provide our demonstrations starting near each object. At deployment, to reach the object from far away we first estimate the object's pose using pose estimation and approach before switching to MILES. Finally, we set the environment disturbance threshold 0 to 0.94 for all our tasks. Additional details on the pose estimation method we use and how to set each one of MILESS parameters can be found in our supplementary material sections B.1 and B.2.
5

---

Methods
Lock with key
Insert USB
Plug into socket
Insert power
cable
Twist
screw
Bread in
toaster
Open
lid
Mean
Demo Replay
0
0
15
25
Reset Free Residual RL
0
15
0
0
Reset Free FISH
0
30
25
15
0
0
0
10
Pose Estimation + Demo Replay
50
10
85
80
70
100
100
71
MILES
90
70
85
85
85
95
100
87
Table I: Task success rates (9) for 20 trials reported for each method.
4.1
Can MILES solve range of everyday tasks and how does it perform against baselines that learn from a single demonstration?
In this experiment, we assess the performance of MILES across a diverse set of everyday tasks and compare it to various baseline methods capable of learning from single demonstration. We select seven distinct tasks, shown in Figure 4, spanning a range of complexities, each learned from a single demonstration. The tasks are: 1) Lock with key; 2) Insert USB; 3) Plug into socket; 4) Insert power cable; 5) Twist screw; 6) Bread in toaster; 7) Open lid. Tasks 1-4 are contact-rich and and our setup follows prior work on contact-rich manipulation [16, 30, 31, 32, 33] and the NIST benchmark [34]. Similar to prior work [35, 36), we focus our evaluation on single-task performance, but also report results on generalization, robustness distractors, and multi-stage tasks in section 4.5. We provide detailed description of each task in the supplementary material section B.3 including data collection times, information which tasks involve demonstration replay, which tasks stopped data collection due to an environment disturbance and information on the length of each human demonstration.
Baseline Methods. We chose 4 baselines that learn from single demonstration without prior task knowledge, similar to MILES. (1) Demo Replay which involves replaying the demonstrated actions. (2) Pose Estimation Demo Replay follows [17] and leverages MILES' data to per- form pose estimation followed by demonstration replay. (3) Reset Free Residual RL replays the demonstration's ss actions each timestep and learns corrective actions on top using DDPG [37]. Like MILES, no human intervenes to reset the environment during training, hence we call it "Reset Free". Finally, (4) Reset Free FISH uses the state-of-the-art RL-based imitation learning method FISH [3] but no human intervenes to reset the environment during training. Further, implementation details on the baselines can be found in our supplementary material section B.4.
Evaluation. For a fair evaluation, we carefully tuned each method's s hyperparameters. Additionally, each learning-bast based baseline collected the same number of observations MILES during data col- lection for each task. We evaluated each method's 's success rate across 20 trials. For each trial we randomized the relative starting pose of the robot and the task-relevant object equivalently across all methods within a sphere of 20cm around the object as long as the object was visible to the camera. Finally, we emphasize that for all evaluations both MILES and the baselines predict 6-DoF actions.
Results. Table presents the success rates of MILES and the baselines across tasks. As shown MILES obtains high success rate across all tasks. Specifically, MILES obtains average of 87% success rate across all tasks with the "Open lid" task having the highest success rate of 100%. From the contact-rich tasks, inserting the key and locking the lock achieved the best performance despite the task's 's low tolerance and complex interaction, with an impressive 90% success rate. The lowest success rate is observed in the USB insertion task, where MILES obtains 70%, performance dip we attribute to the task's low tolerance of less than Imm. Despite the USB task, for the remaining tasks that required high precision MILES was able to complete them consistently well.
The next best-perforning method, Pose Estimation + Demo Replay, obtained an average success rate of 71%. Our experiments showed that small errors in object pose estimation led to failures due to the compounding errors of demonstration replay, as observed in prior work [2, 20J. This is particularly evident in tasks requiring precise manipulation including "Lock with key", "Insert USB" and "Twist screw". Instead, while MILES also replays part of the demonstration for some tasks, the fact that it does for a much shorter horizon allows it to obtain considerably higher success rates.
Reset Free FISH failed to solve the majority of tasks, yielding an average performance of 10% across all evaluated scenarios. During training, for several tasks, the policy caused significant distur- bances to the environment that made policy learning very hard without manual resetting. We believe that this is the central reason behind Reset Free FISH's low performance Additionally, compared to the original FISH implementation, as we train policies that predict 6-DoF actions the learning efficiency of Reset Free FISH is negatively affected. Lastly, Reset Free Residual RL obtained

---

Mathod Ablations
Lock with key
Insert USB
Plug into
socket
Insert power
cable
Twist screw
Bread in
toaster
Open lid
Mean
No Sequence
60
20
20
10
85
95
43
No Environment Disturbance
90
70
85
85
0
0
0
47
No Reachability
75
40
95
20
85
95
100
73
No Memory
50
100
90
100
MILES
95
100
Table 2: Task success rates (9) for 20 trials reported for each method ablation.
lower success rate, averaging 7% due to similar challenges with Reset Free FISH. As anticipated, the Demo Replay baseline was the least effective among the baselines. Simply replaying the demon- stration without pose estimation or online action correction leads to task failures.
While the RL methods would have benefited from manual environment resets during training, our results demonstrate that our "easy" imitation learning setting, where only single demonstration is available without additional human interventions, MILES significantly outperformed the baselines.
Simulation Evaluation. Finally, support the reproduction of results, conducted addi- tional experiments on our method and the baselines simulation using 5 tasks from the RLBencc benchmark [38] which can be found in the supplementary material section C.I.
4.2 How does MILES perform under different method ablations?
This section studies MILES' performance by ablating 4 different components of the method: (1) No Environment Disturbance: we ablate the environment disturbance condition by not checking for that condition when collecting augmentation trajectories. (2) No reachability: we ablate the reachability condition by relabeling each observation action (of the existing MILES data), to move the robot to the nearest waypoint in the demonstration based on their Euclidean distance. If the con- straint for reachability is not important, then simply moving from each pose to the nearest waypoint in the demonstration in a straight line would be sufficient to solve task. (3) No sequence: we rec- ollect MILES' data but instead of collecting augmentation trajectories for the first demonstration state, then progressing to the second state and stration's waypoint sequence and instead follow a random one. (4) No Memory: For this ablation retrain network the existing MILES data that does not account for history.
collect data without following the demon-
Results. Table 2 shows MILES performance after ablating each component. Collecting augmenta- tion trajectories for each demonstration state in random order (No Sequence), with an average suc- cess rate of 43%. Additionally, not checking for the environment disturbance condition (No Envi- ronment Disturbance) appears to cause significant performance degradation for the tasks where an environment disturbance occurred during data collection, corresponding mostly to the non-contact rich tasks. On the other hand, not checking for the reachability condition (No Reachability) also lowers performance, particularly for the precise, contact-rich tasks, indicating that the reachability condition is the most important when learning tasks requiring precise manipulation. Finally, the lower performance obtained by removing the LSTM (No Memory) demonstrates the performance benefits of training memory-basee based networks datasets collected using MILES.
4.3 How important are vision and force modalities to the performance of MILES?
In this section, we ablate the use of vision and force feedback policy inputs for the four contact-rich tasks from our earlier exper- iments. We retrain and evaluate two policies: one using only vision and one using only force.
Taak Parfomance under Differant Oesorvation Modalties
20
89
65
Success Rate
The results, shown in Figure 5, indicate that the vbseen-based policy improves MILESS per- formance in the "Insert USB" and "Plug into socket" tasks but reduces performance in the other two tasks. This suggests that force feed- back might not consistently benefit MILES, possibly due to its noisy signal which makes it hard to distinguish between different environment states. The force-based policy, however, fails almost completely. This is expected force feedback is zero in free space and can be ambiguous due to
Vision
Forre
Vaiton
Porre
Figure 5: MILES performance when trained only either vision or force feedback or both.
7

---

symmetries in object surfaces. Overall, while force feedback aids performance in some tasks, it is not always necessary. Vision remains the most crucial modality to MILES' high performance.
4.4 How does MILES perform under different sizes of self-supervised data?
In this section, we ablate the dataset size used to learn four tasks by splitting their origi- nal datasets into chunks containing 75%, 50%, and 25% of the original data. We evalu- ated the best and worst performing contact-rich tasks ("Lock with key" and "Insert USB") and non-contact-rich tasks ("Open lid and "Twist screw"). Data collection times for each task can be found in the supplementary material. Figure 6 shows that for high tolerance tasks like "Open lid," MILES achieves 100% success rate even with 25% of the data, corresponding to only 8 minutes of data collec- tion. However, for precise tasks, success rates decrease as dataset size is reduced. Notably, for "Lock with key" and "Twist screw," reduc- ing the dataset to 50% results in high failure rate. To summarize, we observe that high tol-
lask Performancc under Different Datasst 205
100 100 100 300
55
5S
Go
25
0
erance tasks are likely to require less data, and in practice only a few minutes of data collec- tion time. Instead, for high-precision tasks, like inserting a USB, the dataset size appears to im- pact MILES performance significantly.
25%
50%
75%
Figure 6: MILES' performance when truined on different datuset sizes. 100% corresponds the original dataset. 75%, 50%, and 25% correspond to splits of the original datasct.
4.5 Further questions about MILES
Can MILES generalize? We conduct additional experiments on generalization for MILES in the supplementary material section D.I. Can MILES perform multi-stage tasks? We provide ad- ditional experimental results how MILES performs on multi-stage tasks in the supplementary material section D.2. Is MILES robust to distractor objects? We conduct additional experi- ments studying MILES' performance in the presence of distractors in the supplementary material section D.3. What if MILES stops data collection early due to a detected environment dis- turbance? We provide discussion and intuition on MILES' behavior in scenarios where data collection stops early in our supplementary material section D.4.
Discussion
Limitations. We now highlight some important limitations of our method. Firstly, MILES' reliance on wrist camera enables MILES obtain spatial generalization, however, simultaneously this limits its field of view and its applicability to larger task spaces. Future work could address this by incorporating an external camera initially approach camera, similarly to [20]. Secondly, while MILES is robust to distractors at deployment before data collection begins it requires human set up the robot's workspace such that only the task-relevant object is in camera view for the policy to achieve spatial generalization. While this requires only few seconds of human time, future work could address this by extending MILES incorporate segmentation methods, similar to [20, 39], that segment the task-relevant object in the dataset. Simi- larly, to address any unwanted collisions that MILES could cause in the presence of multiple objects, future work could study incorporating an external camera during self-supervise plan and collect collision-free augmentation trajectories. Thirdly, our current implementation of MILES trains separate policy for each task and hence it is unclear how well MILES would gener- alize to completely new tasks. In future work, we aim to study this by training a single monolithic policy on MILES' self-supervised data combined with replay-trajectory retrieval [40] similarly to our generalization task in section 4.5.
object before switching the wrist
data collection to
Conclusion. We introduced MILES, a framework that makes imitation learning easy. MILES re- quires only a single demonstration and collects self-supervised data that demonstrate the robot how to return to and then follow that demonstration. Subsequently, this enabled us to obtain ma- nipulation skills comprising either (1) single end-to-end policy trained with behavioral cloning, or (2) a combination of an end-to-end policy and demonstration replay. Our real-world experiments showed that when only a single demonstration available, self-supervised data enable the acquisi- tion of skills that achieve considerably improved performance compared to several state-of-the-arr baselines. MILESS can learn everyday tasks, ranging from opening a lid, to using key to lock a lock, to inserting USB stick into port, requiring complex and precise contact-rich manipulation.
8

---

Acknowledgements
We would like to thank Norman Di Palo, Vitalis Vosylius, Pietro Vitiello, Kamil Dreczkowski, Yifei Ren and Ivan Kapelyukh for their insightful discussions the method and feedback in drafting the first versions of this paper.
References
[1] A. Brohan al. Rt-1: Roboties transformer for real-world control at scale. In arXiv preprint arXiv:2212.06817 2022.
A. Mandlekar, S. Nasiriany, B. Wen, L. Akinola, Y. Narang, L. Fan, Y. Zhu, and D. Fox. Mimiegen: data generation system for scalable robot learning using human demonstrations, In Conference on Robot Learning, 2023.
[3] S. Haldar, J. Pari, A. Rai, and L. Pinto. Teach a robot to fish: Versatile imitation from one minute of demonstrations. arXiv preprint urXiv:2303.01999 2023.
[4] J. Ho and S. Ermon. Generative adversarial imitation learning. In Conference on Neural Information Processing Systems, page 4572 4580, 2016.
V. Mnih et al. Human-level control through deep reinforcement learning. Nut., 5187755185544 2015.
[6] E. Johns. Back reality for imitation learning. In Proceedings of the 5th Conference Robot Learning (CoRL), 2021.
[7] S. Ross, Gordon, and Bagnell. reduction of imitation learning and structured prediction to no-regret online learning. International Conference on Artificial Intelligence and Statistics, pages 627-635, 2011.
[8] O. X.-E. Collaboration et al. Open X-Embodiment: Robotic learning datasets and RT-X models, 2023.
[9] E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ehert, C. Lynch, S. Levine, and C. Finn. BC-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning, 2021.
[10] Y. Hu, M. Cui, J. Duan, W. Liu, D. Huang, A. Knoll, and G. Chen. Model predictive optimization for imitation learning from demonstrations. Robotics and Autonomous Systems, 163, 2023.
[11] Y. Huang, Silvério, L.. Rozo, and D. G. Caldwell. Generalized task-parameterizedd skill learning. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 5667-5474, 2018. doi: 10. 10.1199/CRA2201 2.1........1...111777..
[12] C. Finn, T. Yu, T. Zhang, P. Abbeel, and S. Levine. One-shot visual imitation learning via meta-learning. ArXiv, abs/1709, 10w.0/999949 04905,
2017.
[13] Z. Mandi, F. Liu, K. Lee. and P. Abbeel. Towards more generalizable one-shot visual imitation learning. In 2022 International Conference on Robotics and Automation (ICRA), pages 2434-2444, 2022. doi: 10.1 119/CCA44639999 20000399.222299144550
[14] G. Papagiannis and Y. Li. Imitation learning with sinkhorn distances. In European Conference Machine Learning and Knowledge Discovery Datahases, 2022.
[15] Sun, A. Vemula, B. Boots, and Bagnell. Provably efficient imitation Icarning from observation alone. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Confer- ence Machine Learning, volume 97 of Proceedings of Muchine Learning Research, pages 6036-6045. PMLR, 09-15 Jun 2019. URL https: pttp///rocce.....c/oe//// .r.r............ press. /97/ 9b. html.
[16] G. Schoettler, A. Nair, J. Luo, S. Bahl, Aparicio Ojea, E. Solowjow, and S. Levine. Deep reinforcement learning for industrial insertion tasks with visual inputs natural rewards. International Conference on Intelligent Robots and Systems (IROS), 2020.
[17] E. Johns. Coarse-to-fine imitation learning: Robot manipulation from a single demonstration. In JEEE International Conference on Robotics and Automation (ICRA), 2021.
[18] E. Valassakis et al. Demonstrate once, imitate immediately (dome): Learning visual servoing for one-shot imitation learning. 2022.

---

[19] N. D. Palo and E. Johns. On the effectiveness of retrieval, alignment, and replay in manipulation. RA- Letters, 2024.
[20] P. Vitiello, K. Dreczkowski, and E. Johns. One-shot imitation learning: A pose estimation perspective. In Conference on Robot Learning, 2023.
[21] B. Wen, W. Lian, K. E. Bekris, and S. Schaal. You only demonstrate once: Category- manipulation from single visual demonstration. ArXiv, abs/2201.12716. 2022.
[22] Laskey, Lee, R. Fox, A. D. Dragan, and K. Goldberg. Dart: Noise injection for robust imitation learning. In Conference on Robot Learning, 2017.
[23] L. Ke, J. Wang, T. Bhattacharjee, B. Boots, and S. Srinivasa. Grasping with chopsticks: Combating covariate shift in model-free imitation learning for fine manipulation. In International Conference on Robotics and Automation (ICRA), 2021.
[24] A. Zhou, M. J. Kim, L. Wang. P. Florence, and C. Fian. Nerf in the palm of your hand: Corrective augmentation for robotics via novel-view synthesis. 2023.
[25] M. Jia. D. Wang, G. Su, D. Klee, X. Zhu, R. Walters, and R. Platt. Seil: Simulation-augmmentee equivariant imitation learning. In International Conference on Robotics and Automation (ICRA), pages 1845-1851, 2023. doi: 10. 109111999/9999 2023. 20161212
[26] L. Ke, Y. Zhang, A. Deshpande, S, Srinivasa, and A. Gupta. CCIL: Continuity baasnd data augmentation for corrective imitation learning. In First Workshop Out-of- oiststtiibuttin Generalization in Robotics CoRI. 2023, 2023.
[27] G. Cideron, B. Tabanpour, S. Curi, Girgin, I.. Hussenot, G. Dulac-Arnold, M. Geist, O. Pietquin, and R. Dadashi. Get back here: Robust imitation by ecturnttoddistrbatiin planning, 2023.
[28] M. Caron, H. Touvron, I. Misra, H. J'egou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in sulf-supe vision transformers. International Conference on Computer Vision (ICCV), 2021.
[29] S. Amir et al. Deep vit features as dense visual descriptors. ECCVW What is Motion For?, 2022.
[30] J. Luo, O. O. Sushkov, R. Pevceviciute, W. Lian, C. Su, M. Vecerík, N. Ye, S. Schaal, and J. Scholz. Robust multi-modal policies for industrial assembly via reinforcement learning and demonstrations: A large-scale study. ArXiv, abs/2103. 1512, 2021.
[31] Luo, Z. Hu, C. Xu, Y. Tan, J. Berg, A. Sharma, Schaal, C. Finn, A. Gupta, and Levine. Serl: A software suite for sample-efficient robotic reinforcement learning. In International Conference on Robotics and Automation (ICRA), 2024.
[32] T. Z. Zhao, Luo, O. Sushkov, R. Pevceviciute, N. Heess, J. Scholz, S. Schaal, and S. Levine. Of- fline meta--eiinoorcmment learning for industrial insertion. International Conference on Robotics and Automation (ICRA). pages 6386 0066-6333370 2022.
[33] A. Nair, B. Zhu, G. Narayanan, E. Solowjow, and S. Levine. Learning on the job: Self-rewarding offline- to-online finetuning for industrial insertion of novel connectors from vision. In International Conference on Robotics and Automation (ICRA). pages 7154-7161, 2023.
[34] K. Kimble, K. Van Wyk, J. Falco, E. Messina, Y. Sun, M. Shibata, W. Uemura, and Y. Yokokohji. Bench- marking protocols for evaluating small parts robotic assembly systems. IEEE Robotics and Automation Letters, 5(2):883 889, 2020.
[35] C. Chi, Feng, Y. Du. Xu, E. Cousineau, Burchfiel, and Song. Diffusion policy: Visuomotor policy learning via action diffiusion. In Proceedings of Robotics: Science and Systems (RSS), 2023.
[36] T. Zhao, V. Kumar, S. Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware, 2023.
[37] T. Lillicrap, J. J. Hunt, Pritzel, N. M. O. Heess, Erez, Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. ArXiv, 1009/500.000 00009,2971,222 2015.
[38] S. James, Z. Ma. D. R. Arrojo, and A. J. Davison. Rlbench: The robot learning benchmark & learning environment. CoRR, abss/1909.11. 12271, 2019. URL http:hhttp///eeee wwwpiiiooogg/aa.ee.ee abs////////// 1909. 12271.
10

---

[39] D. Seita, Y. Wang. S. J. Shetly, E. Y. Li, Z. Erickson, and D. Held. Toolflownet: Robotic manipulation with tools via predicting tool flow from point clouds. In K. Liu, D. Kulic, and J. Ichnowski, editors, Proceedings of The 6th Conference on Robot Learning. volume 205 of Proceedings of Machine Learning Research, pages 1038-100 1049. PMLR, 14-18 Dee 2023. URL https: htt..tttp..pp.eee....... /ttp///eooo.....e///oo.e/ooo...e www.g.mm.......................m v205/ 2205/seitae seit a22a. html.
press/ mrr.sssss./pp
[40] N. Di Palo and E. Johns. Learning multi-stage tasks with one demonstration via self-replay, In Conference on Robot Learning (CoRL), 2021.
[41] K. He, X. Zhang, S. Ren, and J. Sun. Deep Residual Learning for Image Recognition. In Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR '16, pages 770-778. IEEE, June 2016.
[42] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation, 98), 1997.
[43] A. Mandlekar et al. What matters in learning from offline human demonstrations for robot manipulation.
In Conference on Robot Learning, 2021.
[44] J. Pari, N. M. Shafiullah, S. P. Arunachalam, and L. Pinto, The surprising effectiveness of representa- tion learning for visual imitation. CoRR, 1bs/2112.01511, 2021. URL https:ttp//// /ttp://aaaavvv/aaaeeaieeeeeeawea org/ aabs / 2112. 001511.

---

Supplementary Material
MILES: Making Imitation Learning Easy with Self-Supervision
For videos demonstrating MILES" performance and code implementation please see our webpage: www.robol-lcarinnguuk//m
A
MILES: Additional Details on the Method
A.1
Method Pseudocode
We provide a detailed pseudocode describing MILES, in Algorithms 8.
A.2 Validity Conditions for Augmentation Trajectories
As described in section 3.3, after collecting data for an augmentation trajectory, we check for two conditions: (i) reachability and (ii) environment disturbance, to determine whether an augmenta- tion trajectory is valid and eligible fuse with the demonstration. Figure A.2 shows examples of these two conditions.
A.2.1 How do we check for the Reachability condition?
Reachability. To check for reachability, after executing augmentation trajectory Tk, we verify whether the final achieved pose matches the pose of the kt, demonstration waypoint using propri- oception, described in section 3.3. Pseudocode describing how we check for reachability is also provided in Algorithm 3. is crucial to check for reachability because an augmentation trajectory that does not meet this condition cannot be fused with the demonstration, as it cannot return to the demonstration state. If the waypoint w is unreachable during data collection, we cannot auto- matically determine how to reach w from wāy without collecting observations that do during self-supervised data collection. Consequently, we cannot automatically determine what actions to take to return back to the demonstration from wit, as we can with valid augmentation trajectories. Figure A.2 (a, left) shows an example where the reachability condition is not met due to environ- mental dynamics, such as a key getting "jammed" and failing reach the target waypoint due to collision and friction in the lock. A similar example where the reachability condition is met is shown in Figure A.2 (a, right).
A.2.2 How do we check for the Environment Disturbance condition?
Environment Disturbance. To determine whether an environment disturbance occurred, we com- pare the RGB image captured at the kεΛ demonstration timestep with the RGB image captured at the final timestep of the augmentation trajectory, as described in section 3.3. A detailed pseudocode describing how we determine whether an environment disturbance occurred can be found in Algo- rithm 5, and a visual example can be seen in Figure A.2 (b). The comparison between the two RGB images relies the similarity of their DINO features [28]. Specifically,
pre-trained DINO ViT [28] to obtain the DINO features for different patches of each image similarly to [29]. By com- puting the cosine similarity between the DINO features of each corresponding image patch in IS and Tīr can caleulate the average similarity between the two images [29]. If the similarity is below threshold 0 (to see how we automatically determine 0 please see section B.2.3), we assume the robot has disturbed the environment, and data collection stopped. Our experiments showed that DINO ViT features are necessary because they are robust to lighting changes and noise in the RGB image. Other methods tried, such as template matching or computing the per-pixel Euclidean distance, proved brittle and sensitive to lighting variations or noise in the captured images. Understanding why checking for an environment disturbance is important is straightforward. Consider the rectan- gular object shown in Figure A.2 (b), and assume the task is to learn how to pick up that object. If the robot pushes the rectangular object, causing it to fall over during data collection, the image observed after returning to the demonstration state will no longer match that state's observation from when the demonstration was provided. Consequently, from the point where the disturbance occurred onward, we have no way of knowing how to reach any of the remaining demonstration
A1

---

Examples Medium Enviromment Disturbance
Similarties
Environment Disturbance
Nin leage
Outcrael eay
Decondande 1014
Toy
Figure A.I: The cosine similarity computed using the DINO features for the screwdriver task under varying environment disturbances.
states and result how to solve the task. This is because we only know how to solve task by learning how to follow the demonstration after returning to But an environment disturbance has occurred (e.g., the rectangular object has fallen), following the demonstration's 's actions longer leads to task completion. Hence, if data collection continued, all future augmentation trajectories would contain invalid observations and actions, they would demonstrate behavior that does not solve the task that the human demonstrated. This why we stop data collection after detecting an environment disturbance.
A.3 Additional Results Environment Disturbances and DINO Features
We demonstrate in this section several examples of possible environment disturbances and how we can detect them using the DINO features on the toy screwdriver used in our experiments. We use the screwdriver
example as during data collection for the "Twist screw" task, data collection was stopped due an environment disturbance caused the grasped screwdriver. Additionally, disturbances caused in the grapsed objects are often the most subtle, and as such make for the most interesting cases.
Figure A.1 (e) shows the screwdriver object (not grasped). All the other figures depict the screw- driver as it appears the view of the wrist camera when grasped by the robot. Figure A.1 (a) shows a "Demonstration Image" and a "New Image" that depicts the DINO Cosine similarity (higher bet- ter) when no environment disturbance has occurred, i.e., the grasp has not changed. The heatmap demonstrates the similarity between each corresponding patch between the "Demonstration Image" and the "New Image" (the cosine similarity reported the mean of these). As shown, the cosine similarity (0.961) is greater than our universal threshold f of 0.94 (for more details please see exper- iments section 4). The reason it is not perfect 1.0 is due to noise and light changes as the photos were captured at different moments in time. Figure A. (b), shows detected environment distur- bance based on DINO features. As shown under the "New Image" the screwdriver has moved by small amount in the gripper and the cosine similarity falls slightly below our threshold é. Then, Figure A. (c) shows a slightly bigger detected environment disturbance, and finally Figure A. (d) shows rather large environment disturbance. Generally, shown in Figure A.I, the DINO features are robust in detecting environment disturbances of different scales and as we move from smaller to larger disturbances in the grasped screwdriver the cosine similarity also decreases, expected.
A.4 MILES Policy
Training: To train our manipulation policy we leverage the dataset D comprising the fused
augmentation trajectories with the demonstration as described in section 3.4. MILES policy æ comprises either (1) an end-to-end network trained with behavioral cloning (BC) or (2) end-to- end network trained with BC combined with demo replay, which utilized when data collection was interrupted due to detected environment disturbance.
A2

---

Reachability
Environment Disturbance
Unreachable
Reachable
Observation et Be
Planned Augmentatonn Trijedtrr
Observation cxscuting Augmestation
key pareend bearted
Ercestall perrames andiricton much Degenstraton
State
Inttll
ctoereation of SIND mutih The desecentration stseraation
Yolld Asprentatinn Thepettery
Figure A.2: Reachability: Two examples of possible augmentation trajectories for a locking task are shown; an invalid trajectory (left) that fails reach the target demonstration waypoint due to collisions, friction, and potentially inevitable systematic controller errors and a valid one (right) that successfully reaches the target waypoint. Environment Disturbance: As the robet collects an augmentation trajectory. it perturbs the environment such that after returning to the demonstration s waypoint the live observation and the demonstrated one longer match, indicating that data collection should stop.
A.4.1
How is our policy defined when No Environment Disturbance occurred during data collection?
No Environment Disturbance. When no disturbance occurred our dataset D new contains augmen- tation trajectories that return to and then follow the demonstration from every state. In that case, we leverage Dew to train an end-to-end behavioral cloning policy that comprises a single neural network fv. parameterized by v., that receives as input an RGB image captured from the wrist camera and force-torque feedback predict 6-DoF actions: fa R well as additional binary value indicating the gripper action (IR H W x3 refers to the RGB images where H: height, W: width and R to measured forces and torques). The force-torque feedback is captured directly using Franka Emika Panda's joint force sensors. For policy to generalize spatially, no proprioception input is passed to fu and all actions are predicted relative to the EE's frame. f consists of a ResNet-18 backbone [41] for processing RGB images, and a small MLP embeds force feedback into 100-dimensional space. The output of the force MLP and ResNet- 18 are concatenated and fed into an LSTM [42] network for action prediction. The network trained using standard behavior cloning to maximize the likelihood of Dnew-
R SE(() as
A.4.2 How is our policy defined when an Environment Disturbance occurred during data collection?
Environment Disturbance. When self-supervised data collection was stopped due to an environ- ment disturbance, our dataset Dnew contains augmentation trajectories that can return the robot to any state from the initial demonstration state up to the demonstration state at timestep Rt where R N (see section 3.2). In this scenario, if our policy consists only of fg, then during task execu- tion the robot would be able to solve the task only up to the Ra state, but not complete it. As such, neural network fø identical to the above scenario, but trained up to the Rn state and (2) the second component corre- sponds simply to the sequence of the remaining demonstration actions from the Ri state onwards, for which no self-supervised data was collected, i.e., Gremaining = {a 1.N
we define our policy
consist of two components: (1) the first component
In=R
A.4.3 How do we deploy MILES' policy?
Deployment: Our LSTT-- policy closely follows the implementation of BC-RNN [43]. De- ploying the policy is straightforward and depends on whether data collection was interrupted due to an environment disturbance. If uninterrupted, then only the neural network fo is used complete the task equivalently to policies trained using reinforcement learning or behavioral cloning.
If data collection was interrupted, first fe is deployed to solve the task up to the Ra, state in an identical way as the scenario of "no environment disturbance". After the robot reaches the Reh state then Sremaining is executed. We determine whether the closed-loop policy has completed the task up to the RG, in a very simple way as described in section A.4.4.
During deployment we reset the hidden state of the LSTM at an interval equal to two times the number of timesteps (i.e., waypoints) in the demonstration for which augmentation trajectories were collected. For example, if for task MILES collected augmentation trajectories for 40 demonstration
A3

---

waypoints before stopping due to an environment disturbance, then, during deployment the hidden state of the LSTM is reset every 80 timesteps. We did not find the frequency of resetting the hidden memory to have significant effects on the policy's performance. We would like to note that the only important observation we made was that the number of timesteps should not be very low (e.g., 5) as then the robot would end up progressing towards completing a task very slowly.
Pseudocode describing MILES' policy deployment can be found in Algorithm 8.
A.4.4 How do we determine when to switch from closed-loop control to demonstration replay?
Switching from closed-loop to demonstration replay is straightforward. As the objects and the robot can be at different poses during deployment from the ones during data collection, we cannot,jus use the robot's proprioception to know when the Ra state has been reached. Hence, we deploy fg until it predicts continuously the identity transformation, indicating no robot movement. Then, we switch to demonstration replay, where we replay the rest of the demonstration Sremaining.
B
More details on the Experimental Setup
B.1 Pose Estimation
In practice, as with most methods [3, 16, 17, 20). we naturally provide the demonstrations starting near the task-relevant object to focus self-supervised data collection at the part of the task that is the most important, that is the robot-object interaction part.As such, we need a way to ensure that MILES can still solve any task regardless of how far the robot is from an object. An apparent solution to this is to provide the demonstration starting from a pose far away from the object and deploy MILES data collection. While this is possible as MILES makes no assumptions or restrictions on the length of the demonstration it may be inconvenient. As such, inspired by [2, 18, 20] we use simple pose estimator deployment to estimate the relative pose between the robot at the initial state of the demonstration (for which MILES collected data) and the task-relevant object. As we do not assume any 3D object models, model-free pose estimator can be used. This allows us to first coarsely estimate the pose and move near the task-relevant object from any robot starting pose before deploying MILES. Uncut videos demonstrating this behavior can be found on our webpage: www.robot-leammin woabbb--leamiinguu/iiless uk/miles.
use the method deployed in [17] although any other
B.2
MILES Data Collection Hyperparameters
B.2.1
How do we set the data collection range around each demonstration waypoint?
As discussed in our experiments section 4, collect data in range of 4cm and 4 degrees around each demonstration waypoint. However, this range is not limiting and can be set to any desirable range like any other robot learning method. In our case, we set this range to be the average pose estimation error to reach the initial pose of the demonstration relative to the task-relevant object using the pose estimation method described in section B.1 which we obtained based on [17].
B.2.2
How do we determine the number of augmentation trajectories to collect for each demonstration waypoint?
For all of our experiments, we set the number of augmentation trajectories per demonstration way- point, Z = 10. In our case, we set this arbitrarily, but as we showed in our method's 's data collection ablation in section 4.4 different tasks require diffferent numbers of augmentation trajectories. As such, we provide two guidelines for setting the value for Z. Firstly, high tolerance tasks, like the "Open lid" task reported in our experiments usually require small number of augmentation trajec- tories. On the other hand, precise tasks, like the "USB insertion" task reported in our experiments require more augmentation trajectories. Secondly, as the data collection range around each demon- stration waypoint increases, the number of augmentation trajectories collected should also increase with an approximately linear relationship, i.e., if the range is doubled, then the number of augmenta- tion trajectories should be doubled as well. We recommend as starting point, for a data collection range similar to our experimental setting of 4cm and degrees, to collect 10 augmentation trajecto- ries for precise, low-tolerance tasks, and 4 augmentation trajectories for high-tolerance tasks.
A4

---

Task:
Description
DCT
Task:
Description
DCT
Lock with
Insert into lock and 90
Twist screw
Insert screwdriver into
key
degrees to lock it.
24'
and twist by 90"
22'
Insert USB stick into USB port
Bread in
Insert USB
(< Imm tolerance)
21"
toaster
Put plastic bread inside toaster.
40"
Plug into
Plug a UK plug (3-pin) to socket.
37
Open lid
blue box.
31
socket
Insert
Plug the power cable into the power port a
power
cable
Table 3: Task descriptions of the 7 tasks used our experiments. DCT stands Data Collection Time and corresponds the time spent collecting self-super data.
B.2.3 How do we determine the Environment Disturbance threshold
We determined simply by spawning several random RI.Bench [38] tasks in CoppeliaSim and running MILES. By setting up custom heuristics that determine environment resets in the simulation we found that for the DINO model we use, a similarity of 0 < 0.94 appeared to detect environment disturbances across all tasks successfully. Consequently, used that our real-world experiments too.
B.3 Task Descriptions
detailed description each task along with their Data Collection Times (DCT) can be found in Table 3.
B.3.1 How long is each demonstration?
The demonstration lengths varied across each task. As follows, we list for each task the number of demonstration waypoints comprising each human demonstration (each demonstration waypoint can be interpreted timestep): Lock with key: 32, USB task: 20, Plug into socket: 40, Insert power cable: 29, Twist screw: 47, Bread Toaster: 70, Open lid: 80. All demonstrations were collected using telcoperation. Note that the number of demonstration waypoints is not necessarily equal to the number of waypoints for which MILES collected augmentation trajectories. This is because environment disturbances may have caused the data collection to stop earlier.
B.3.2 For which tasks was an Environment Disturbance detected?
An environment disturbance was detected for the following tasks: Twist screw, Bread in Toaster and Open 1id. As such for these tasks the policies comprise closed-loop and demonstration replay component.
We also note that for the lock with key task, stopped data-collection "half-way" through the 90 degrees twisting rotation for hardware safety. This is because the forces exerted on the robot as was collecting self-supervised data were too high. In this case, we treated this identically to an environment disturbance. At deployment, the learned policy completes most of the task closed-loop, apart from small twisting motion done with demo replay, after the closed-loop policy converges to predicting the identity transformation as discussed in section A.4.4. This is similar to adding force limits to reinforcement learning algorithms and was done to protect our robotic hardware; however, doing so is not requirement.
B.4 Baselines
Here, we provide further implementation details on two of the baselines we used in our paper.
Pose Estimation + Demo Replay. For this baseline, we follow the same problem formulation as [17], but improve upon that baseline two key ways: (1) the data on which it is trained on the same data collected for MILES, as such it contains only valid trajectories that cover a larger part of the task space and (2) instead, of replaying recorded velocities, we also replayed the recorded forees which is particularly important for the contact rich tasks. This baseline estimates and moves the
A5

---

Methoda
Insert Onto Square Peg
Lightbulb
Pick Up
Cup
Turn Tap
Lamp
Demo Replay
5
0
Reset Free Residual RL
0
0
0
Reset Free FISH
0
0
0
0
20
4
Pose Estimation Demo Replay MILES
70
65
90
80
95
80
100
75
100
Table 4: Task success rates (%) each method on RLBench.
robot to a pose relative to the object of interest as depicted in the first state in the demonstration and replays the complete demonstration. We chose this baseline compared to alternatives, as it leverages task-specific data allowing it to achieve very precise pose estimation.
Reset-Free FISH [3]. For Reset-Free FISH can be found in: http:///ithub..om//iit httphh/....sssddaaaaaaaa//ll// such that the policy always predicts 6-DOF actions instead of constraining the output to specific DOFs, as doing so assumes access to prior task knowledge. To learn residual actions on top of the demonstration we tested both using demo replay as the base policy, as well as VINN [44] but found that demo replay led to better performance.
the implementation provided by the authors as We only changed the implementation
C Additional Experiments
C.1 Simulation Results
To aid other researchers in reproducing our results, we conducted additional simulation experiments on the RLBench benchmark [38] on 5 tasks, specifically: 1) 'Insert Onto Square Peg', 2) 'Lightbulb In', 3) 'Pick Up Cup', 4) 'Turn Tap' and 5) 'Lamp On'. We performed an identical evaluation to our real-world experiments where we performed 20 evaluation trials for each method. Additionally, we used the images captured only from the wrist camera in RLBench. During training we allowed each method to collect the same amount of data and we did not perform any environment resets during training'data collection for any methods. The results can be seen in Table 4. As shown, MILES significantly outperforms the basclines, while the relative performance when comparing all methods remained relatively unchanged compared to our real-world results.
real-world experiments, the reinforcement learning baselines obtained poor perfor- mance for reasons line with the ones discussed in our experiments section 4.1. Specifically, during training we observed that for the tasks 'Insert Onto Square Peg' and 'Lightbulb In' a random gripper action drops the grasped object during exploration and the policy never manages to grasp it again during training without a reset in the given training time. For the 'Pick Up Cup' task, the reinforce- ment learning policy knocks the cup off the table during exploration, consequently never learning something useful. For the 'Turn Tap' task the RL policies never learned to properly grasp and rotate the handle and for the 'Lamp On' task, only Reset Free FISH managed to learn a policy that obtains 20% success rate in the given training time. As discussed in our real-world experiments, if instead we had allowed environment resets and more training time that would have resulted in significantly higher success rates for the RL baselines, compared to their current performance.
Similarly
D Further questions about MILES
D.1 Generalization Performance
Since MILES uses BC train policies, existing generalization results for BC [9. 1] also apply to MILES. For tasks that include demonstration replay following the closed-loop policy, MILES can generalize new objects by retrieving the replay trajectory of the most similar object the existing demonstrations, similar to prior work [19]. To test this, we tasked MILES with throwing markers of different colors into differently shaped and colored bins, shown in Figure 4 (8). Trained on five bins (marked green) and tested on two new bins (marked red), MILES achieved an 80% success rate on the pink bin and 60% on the gray bin, over 10 trials each starting from poses where simple demonstration replay would fail. The data collection time for this task was on average 34 minutes for each bin and an environment disturbance was detected for each bin. determine which remaining actions to replay for the previously unseen bins, we selected the remaining actions from the bin in
A6

---

the training set whose RGB image in the demonstration has the highest similarity in terms of DINO features with the bin during deployment, inspired by prior work [19]. Videos exhibiting MILES generalization on the two test case bins can be found on our webpage: www.robot-eammigg.uuniil
D.2 Multi-stage Tasks
To evaluate MILES' ability solve multi-stage tasks, tasked MILES with picking up the plastic bread shown in Figure 4 (6) (as part of the "Bread in Toaster" task) and inserting it into the toaster. To achieve this broke the task down into two stages: first, we provided a demonstration showing how to pick up the bread and trained MILES. Then, we used the policy already trained on the "Bread in Toaster" task finish the task. To link the two stages together, first the policy pick up the bread is deployed. After, the execution ends, the robot returns to its default position. Then, the pose estimation method described in section B.1 is deployed to approach the toaster, and then the policy trained with MILES is deployed to insert the bread into the toaster. Videos exhibiting MILES' multi-stage task performance on picking up and inserting the bread into the toaster can be found on our webpage: www.robot-leaming uk/miles.
D.3 Performance with distractors
We found that performing standard image augmentation techniques. including changing the bright- ness, contrast, noise, cropping random image parts, etc. allowed MILES to be robust to distractor objects, as shown in the videos provided on our webpage: bobot-leamingguukmillss
D.4
What if MILES stops data collection early due a detected environment disturbance?
There is no requirement as to how early MILES may stop data collection due to an environment dis- turbance, long has collected sufficient augmentation trajectories for at least the first demon- stration waypoint. During data collection. MILES can effectively learn policy even if an environ- ment disturbance occurs early. Unlike RL, MILES learns to solve the task closed-loop up to the demonstration waypoint where the disturbance was detected, after which it replays the demonstra- tion. This is because MILES collects data progressively for each demonstration waypoint, rather than rolling out a policy all at once like RL. Consequently, during data collection, if a disturbance occurs as early as (for example) near the 2nd waypoint, MILES will still know how to get to the 1st waypoint during deployment, where it will replay the demonstration.
Overall, MILES handle early environment resets during data collection. While with the majority of learning-basee based methods, the more the data the better the performance, as such the later an environment disturbance occurs the data collection process the better. However, MILES can still learn robust policy as long as sufficient data has been collected at least for the 1st demonstration waypoint. This is typically trivial as most human demonstrations naturally begin by controlling the robot in free-space far from the object of interest, before interacting with it.
A7

---

E Detailed Pseudocode
Algorithm 1: MILES Overview (Simplified)
Input: Single Task Demonstration: <= {wwh, of á ()) Number of augmentation trajectories
per demonstration waypoint Z, environment disturbance threshold (Default: 0.94)
1: D { // init empty dataset of augmentation trajectories
2: Reachable True // init variable that tracks zeachability
3: Disturbance True // init variable that tracks environment
disturbances
4: R=1// init variable that stores the timestep when data collection stops
welf-superviser
5: Move robot to the initial demonstration pose uf
6: for iteration i = 1 to do
7: j - 1// init variable that tracks the number of collected augmentation trajectories per demo waypoint 8: while j < Z do
9:
Tk +SSampleTrajecooo((((() (Alg. 2)
10:
Reachable -ChcckReachaabbiiyyyr (Alg. 3)
11:
if Reachable is False then
12:
ReturnToDemo Waypoonnta () (Alg. 4)
Break // exit while loop
14:
end if
IT
Capture RGB wrist-cam image M the Ma. (final) timestep
of
Disturbance -ChekkEnDDisurraantt( M. 0) (Alg. 5) if Disturbance is True then
17:
R k// store timestep when data collection stops Break // exit while loop
19:
20:
end if
21:
D-DUMA // add augmentation trajectory to dataset j j-j+1
22:
23:
end while
24:
if Disturbance is True then
25:
Break // exit for loop
26:
end if
27:
Proceed the next demonstration state by performing action of // follow the remonstratioon s progression
28: end for
29:
Drew FFuseAnggeenttttionnn WithDemOODD R, ()Alg. 6)
30: TTruinPolicy(DDen neww. R, (Alg..
31: Deploy(T, R, (Alg. 8)
Output:
A8

---

Algorithm 2: SampleTrajecton
Algorithm FuscAuggeenaations With
Input:
Demonstration waypoint -
Input: Dataset of augmentation trajectories D, final data collection time step R, single demonstrations
I: = // init empty augmentation trajectory
2: Sample initial pose w randomly and move there in a straight line. (Optional: record trajectory poses) // Note that the straight line trajectory may altered due the collisions and the robot' compl annee with its environment.
1: Dnew = f init empty dataset to store fused trajectories
2: for Th: in D do
Sxegment
{(w,, og a ) EC
demonstration segment from ki. demo waypoint to Ra.
Move back w // either by tracking recorded trajectory poses backward or by re-planning new, straight-line trajectory (equa1 performance).
4: Tkee = Tk Gsegment
5:
D new D Theew
6:
end for
1// observations, actions incex
Output: Dew
5: while moving to " do
TK Th (wf on,a // add waypoints, observations and actions to augmentation
Algorithm 7: TrainPolicy
Input: Dataset of augmentation trajectories demo 'D Dnew. final data collection timestep Rt single demonstration C
trajectory; actions are automatically inferred as the relative EE poses between consecutive timesteps.
1: Train neural network fax D new using standard behavioral cloning// Discard proprioception waypoints (w and
7: (0%) comprises wrist cam RGB images + force-torque readings)
w only observation inputs are used for fo
if R < length(() then
8: end while
3: = {fe, {a R/ policy consists of an end-to-end neural net + demo replay (if an environment disturbance stopped data collection before the last demo waypoint)
Output: Return augmentation trajectory 7
Algorithm 3: CheckReachability
Input:
Demonstration waypoint w
4: else
I: Reachable +- True // init reachability variable
5:
π == (+) policy consists only of an
2: whr + EE pose // achieved after executimg ing the augmentation trajectory (comprising M timesteps); read from praprioception
end-to-end neural net
6: end
Output:
== 3: Reachable = (w = w // check whether poses are equal (within the controller's feasible precision)
Algorithm 8: Deploy
Input: Policy я, final data collection timestep R, single demonstration
C
Output: Reachable
1: Capture observation
comprising RGB wrist cam
Algorithm 4: RetumnooDemooai Waypoint
image + force-torque feedback
2: Action a = (o)
Input: Demonstration timestep k, single demonstration Ç
3: Perform action
I: Move to initial demonstration waypoint wff E
4: while not the identity transformation do
// replay demonst up the ka timestep
5: Capture observation
2: for iteration / = to / = k do Perform action af Ç
6: Action a = (o)
7: Perform action
4: end for
8: end while
// if environment disturbance stopped data
Algorithm 5: CheckEny Disiuuranne
collection before the last demo waypoint
Input: Demonstration observation of. captured live image 1T)
9: if R < length(()) then
similarity threshold 9 Disturbance <- False // init environment
10: Replay remaining demo {af IN
11: end
disturbance variable
2: I E of // retrieve RGB image I from the demonstration' observations
3: 15 f3: --: + DINO- // compute DINO-ViT features [29, 28] for each image patch fg for
the demo waypoint nt image
4: 11 ,.. -INOO compute DINO-ViT features [29, 28] for each image patch f7 from the current live environment image (captured after executing the augmentation trajectory).
5: sim =AngCosineSiiilanty((f P... [f] (.....))
6: if sim then
A9
7: Disturbance +- True
8: end if
Output: Disturbance


Reviews:
[{"review_id": "6JBU5LtcOr", "reviewer": "Reviewer_zWvW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors propose MILES, which they explain to be an algorithm for performing imitation from a single demonstration without requiring manual environment resets. The crux of MILES appears to be the self-supervised data collection process, which generates a large number of augmented demonstration trajectories. The authors perform an impressive set of on-robot experiments showing that MILES performs well for robot manipulation tasks.", "summary_of_recommendation": "While there’s a lot to like about this paper (well-motivated, novel, and fairly impressive expeirmental results), I have some important reservations that I think need to be addressed before the paper could be accepted.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "(1) Can the authors provide any intuition as to how MILES can succeed in imitation for tasks in which environment disturbance is possible at points throughout the demonstration trajectory? \n\n \n\n(2) Can the authors comment on how MILES is able to guide the robot to a random location near a particular waypoint and how this ability does not cause any issues of fairness when it comes to comparing to the other approaches presented? \n\n \n\n(3) How does the propose approach compare to FAIL?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ih0eQy6OjW", "reviewer": "Reviewer_3nym", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a data augmentation technique where a robot learns how to get back to a state in a demonstrated trajectory. This allows the robot to perform the task even with a single demonstration, as the robot can follow the rest of the demonstrated trajectory after it gets back to it.", "summary_of_recommendation": "Authors propose a simple but effective idea for data-efficient imitation learning. Paper is well written and method is well demonstrated.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "I don’t have any questions for rebuttal, but the authors may clarify my questions from my review.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IRrESshQOX", "reviewer": "Reviewer_onvu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a method for imitation learning that uses a single demonstration and a single environment reset. It collects data in a self-supervised way by sampling poses around the demonstrated trajectory, moving to the sampled pose, and then storing data that shows how to return from the sampled pose to the demonstration trajectory. The method is shown to outperform alternatives on a set of real-world manipulation tasks given the constraints of a single demonstration and single environment reset.", "summary_of_recommendation": "The paper presents a simple and effective approach, but more evaluation (especially on settings that can be reproduced by others) and discussion on assumptions and limitations would improve the quality of the paper greatly.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 1, "questions_for_rebuttal": "In addition to the points raised in the weaknesses above, I have further questions:\n\n- How does the robot attempt to move back from the random pose to the pose in the dataset during self-supervised data collection? More details on the controller used here, and how the linear path is broken up into robot control commands would be useful.\n\n- Why is MILES better than pose estimation + demo replay - that seems like it should be about as good. Would pose estimation + demo replay obtain higher success by using a bottleneck pose closer to the object of interest? What are some examples of MILES' shorter horizon that provide an advantage over the pose estimation + demo replay baseline?\n\n- It would be great to get more details on the LSTM policy architecture and how it consumes observations at train-time and inference. One common LSTM-based policy is BC-RNN (https://arxiv.org/abs/2108.03298) which unrolls an LSTM policy during a rollout sequentially at each timestep (the hidden state is updated while playing each action) and then reset periodically every N steps. Another approach is to use an explicit observation history with frame stacking, and to unroll the LSTM completely over the observation history window to produce a final action. Which approach is being used here?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hykzY0ijH1", "reviewer": "Reviewer_dHST", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes MILES, a single-demonstration imitation learning method which collects its own data with just 1 environment reset to train a policy to solve the intended task. The paper demonstrates results on a variety of precise insertion tasks.", "summary_of_recommendation": "Weak accept because I believe the paper should fully clarify the assumptions and make some important information regarding the experimental settings easier to find.", "technical_quality": 3, "clarity_of_presentation": 5, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "See above weaknesses.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper propose a new algorithm for imitation learning, MILES, that requires a single demonstration and a single reset. MILES designs a self-supervised data collection process, that generates a large number of augmented demonstration trajectories. Reviews appreciate the simplicity of the method and real-world testing that includes many ablations. Reviews have also noted many shortcomings: the strong assumptions made by the proposed method, the relationships to some past works, and some counterintuitive trends in the experiments.

Authors provided a response that was considered by `dHST`, `3nym`, `onvu`, who all appreciated the response and converged on accept leaning ratings. `zWvW` gave an original rating of weak reject and didn't look through the author response or participate in any discussion. The AC also considered the paper, the reviews, and the author response. 

The AC believes that the proposed method makes strong assumptions and would only work on tasks of a specific form. It seems that MILES is limited to free space motion to a precise pose (e.g. getting the key to the keyhole) and execution of a fixed motion (e.g. turning the key). MILES excels at getting to this precise pose. However, many tasks commonly considered in robot learning literature (e.g. pick an object and place it at a target location, or pick a cup and pour its content into another cup) require careful execution even after the first contact is made with the environment. Other tasks such as tossing a ball require carefully following the precise trajectory to impart sufficient momentum to the ball and not just reaching a precise pose. MILES methodology exploits this specific form of the task and the paper doesn't demonstrate if it will work for more general tasks. The paper's introduction should prominently note these limitations.

Furthermore, even for tasks of this form, the methodology is not robust to accidental changes in the environment, e.g. the screw driver changing pose in the gripper. Author response argues that this can be detected and data collection can be terminated. If this were to happen early on in data collection, then MILES performance wouldn't be much better than just training on the 1 demonstration.

On the experimentation front,
- Generalization to distractor objects in experiments presented in the paper likely comes from the use of wrist camera as the distractor objects are not even visible in the wrist camera image. These claims should be appropriately toned down.
- Pose Estimation + Demo Replay already does quite comparably with MILES only outperforming it on 4/7 tasks. Authors note that Pose Estimation + Demo Replay fails because one can't precisely recover the pose. However, the MILES method itself relies on accurate estimation of whether the environment has changed or not using DINO ViT features, and authors note that it is actually quite sensitive to even slight changes in the environment. If this is so, wouldn't a "pose estimation + local search in a small neighborhood to maximize DINO ViT similarity + demo replay" already be quite strong? Furthermore, this process could be done for multiple way points along the way as opposed to just the first way point.
Decision: Accept