Waypoint-Based Imitation Learning for Robotic Manipulation
Authors: Lucy Xiaoyang Shi
Archit Sharma
Tony Z. Zhao
Chelsea Finn
Venue: CoRL 2023 Poster (2023)
Keywords: imitation learning
waypoints
long-horizon

Abstract: While imitation learning methods have seen a resurgent interest for robotic manipulation, the well-known problem of compounding errors continues to afflict behavioral cloning (BC). Waypoints can help address this problem by reducing the horizon of the learning problem for BC, and thus, the errors compounded over time. However, waypoint labeling is underspecified, and requires additional human supervision. Can we generate waypoints automatically without any additional human supervision? Our key insight is that if a trajectory segment can be approximated by linear motion, the endpoints can be used as waypoints. We propose Automatic Waypoint Extraction (AWE) for imitation learning, a preprocessing module to decompose a demonstration into a minimal set of waypoints which when interpolated linearly can approximate the trajectory up to a specified error threshold. AWE can be combined with any BC algorithm, and we find that AWE can increase the success rate of state-of-the-art algorithms by up to 25% in simulation and by 4-28% on real-world bimanual manipulation tasks, reducing the decision making horizon by up to a factor of 10. Videos and code are available at https://lucys0.github.io/awe/.

Waypoint-Based Imitation Learning for Robotic Manipulation
Lucy Xiaoyang Shi*
Archit Sharma'
Tony Z. Zhao
Chelsea Finn
Department of Computer Science
Stanford University
{lucyshi, architsh tonyttsh,oonnyhoo cofinn)estanfoonetcc
While imitation learning methods have seen resurgent interest for robotic manipulation, the well-known problem of compounding errors continues to afflict behavioral cloning (BC). Waypoints can help address this problem by reducing the horizon of the learning problem for BC, and thus, the errors com- pounded over time. However, waypoint labeling underspecified, and requires additional human supervision. Can we generate waypoints automatically without any additional human supervision? Our key insight is that if trajectory segment can be approximated by linear motion, the endpoints can be used as waypoints. We propose Automatic Waypoint Extraction (AWE) for imitation learning, a prepro- cessing module to decompose a demonstration into minimal set of waypoints which when interpolated linearly can approximate the trajectory up to a specified сгror threshold. AWE can be combined with any BC algorithm, and we find that AWE can increase the success rate of state-of-the- algorithms by up 25% simulation and by 4-28% real-world bimanual manipulation tasks, reducing the decision making horizon by up to factor of 10. Videos and code are available at wwthsssi...gttuubb.///eee/e
Abstract:
Keywords: imitation learning, waypoints, long-horizon
Introduction
The simple supervised learning approach of behavioral cloning (BC) has enabled a compelling set of robotic results, from self- driving vehicles [1] to manipulation [2, 3, 4, 5, 6]. However, due to the lack of cor- rective feedback, errors grow quadratically in the length of the episode for behavior cloning (BC) algorithms [7, 8], colloquially known as the compounding errors prob-
lem. Waypoints [9, 10, 4, 11] are rele- vant proposition in this context: breaking the demonstration into subset of states that can reconstruct the trajectory reduces the effective length of the decision-makinn problem, addressing the compounding er- rors problem while still allowing the use simple methods such as BC. Our primary objective is to select set of waypoints to reduce the effective horizon of the demonstration, and not necessarily find key bottleneck states. However, labeling waypoints is both underspecified problem and requires additional human supervision.
waypoint
imitation
extraction
learning
demos
Figure 1: Our approach reduces the horizon of imitation
learning by extracting waypoints from demonstrations.
Equal contribution.
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

Our objective is to develop method for selecting waypoints for given demonstration without any additional human supervision. For a robot arm, if a trajectory segment can be approximated linearly, a low-level controller can reliably imitate the segment without explicitly learning the intermediate states. Thus, we can represent that segment just by the endpoints. Extending this argument to arbitrary trajectories, we can use subsequence of states as waypoints represent the trajectory the trajectory can be approximated well by linearly interpolating between the selected waypoints. The BC prediction problem then transforms from predicting the next action to the next waypoint.
How do we select the waypoint sequence that approximates a given a trajectory? Given a budget for the reconstruction error, where the error is defined as the maximum proprioceptive distance between the actual and reconstructed trajectory. we want to select the shortest subsequence of states for which the reconstruction error is within budget. This can be posed as a standard dynamic programming problem, by iteratively choosing intermediate state the waypoint and recursively selecting waypoints for the two resulting trajectory segments. The recursion terminates whenever the reconstruction error for the linearly interpolated trajectory between the endpoints is already within the budget. Importantly, finding the sequence of waypoints relies only on the robot's proprioceptive information, which is already collected during teleoperation. AWE makes no additional assumption about the extrinsic environment (state estimation, point clouds, etc.) and requires no additional label information from humans.
Overall, our work proposes Automatic Waypoint Extraction (AWE), a preprocessing module that breaks an expert demonstration into sequence of waypoints. AWE requires minimal additional information, and thus, can easily be plugged into current BC pipelines. We combine AWE with two state-of-the-art imitation learning methods, diffusion policy [5] and action-chunking with transformers (ACT) [6]. and study its performance when learning from human- teleppeteeepeerteed demonstrations. On two existing simulated imitation learning benchmarks and multiple real bi-manual manipulation tasks, find that AWE consistently improves performance, with up 25% increase in success rate in simulation and 4-28% increase in success rate on real-world tasks.
2 Related Work
Imitation learning is a long-studied approach to training robotic control policies from demonstra- tions [1. 12, 13]. A central challenge is when compounding errors cause the policy to drift away from states seen in the demonstration data [8], leading to poor performance with small demonstration datasets. Prior approaches have aimed to improve imitation learning performance by developing new policy architectures [14, 11, 15, 16], using policies with expressive distribution classes [17, 18, 5], introducing modified action spaces [19, 20, 21, 22, 4], constructing modified training objectives [2, 6]. utilizing particular visual representations [23, 24], incorporating data augmentation [25, 26], col- lecting online data [8, 27, 28]. We instead aim tackle the challenge of compounding errors by extracting waypoints that shorten the horizon. Our approach is orthogonal to and complementary to many of these prior developments; indeed, our experiments show that AWE can be combined with two recent, representative methods [5, 6] to improve their performance.
Prior works also attempted to reduce the policy horizon. Some use hand-defined high-level primi- tives [19, 22, 29, 30, 31, 32, 33, 34], but they lack flexibility and require extra engineering. Belkhale et al. [35] proposes a hybrid action space that incorporates both sparse waypoints and dense ac- tions. While innovative, it requires humans to label waypoints either during data collection or post-processing, which may limit its scalability. Other recent works extract waypoints using various heuristics [9, 10, 4, 11], such as selecting waypoints at timesteps when the robot at zero-velocity or actuating the gripper [4]. We are inspired by the success of these methods: they provide dramatic per- formance and data-efficiency improvements in some settings. However, we find that the heuristics do not apply in general, leading to low success imitation learning benchmarks and fine manipulation tasks (Sec 5.4). In contrast to these methods, we cireumvent the need for human waypoint labelling or heuristics. Our approach instead automatically extracts waypoints that minimize the trajectory reconstruction cost, which yields improvements on two existing simulated manipulation benchmarks and multiple real robot tasks. We discuss more related works in Appendix D.2.

---

3 Preliminaries
Problem Setup. We assume an expert collected dataset of demonstrations D = { 71 Tn where each trajectory т; = {(oj, +,)) is a sequence of paired raw visual observations o and proprioceptive information æ. The proprioceptive information can either be the end-effector pose or joint pose, and includes the gripper width. In this work, we use pose-control for the action space, i.e, proprioceptives x are the action outputs as well. Next, we briefly review two recent successful methods for BC, which we will use in our experiments.
Diffusion Policy. Diffusion policy [5] models the conditional action distribution as a denoising diffusion probabilisticc model (DDPM) [36]. allowing for better representation the multi-modality in human-collected demonstrations. Specifically, diffusion policy uses DDPM model the action sequence p(A, | Ot, .i),, where A, {a,.... ac+c} represents chunk of next C' actions. The final action is output of the following denoising process [37]:
A--1= a (A -e It, + +N(,,))))
where A is the denoised action sequence at time k. Denoising starts from A sampled from Gaussian noise and repeated till A: = In Eq 1, (ax, 7, o) are the parameters of the denoising process and eœ the score function trained using the MSE loss E(0) = (c+- Eg Eσ(0r, Te. A + <g, k.) The noise at step l of the diffusion process, Ck. is sampled from Gaussian of appropriate variance [36].
Action Chunking with Transformers. Action chunking with transformers (ACT) [6] models the
policy distribution A ot, Xt) as conditional VAE [38, 39), using a transformer based encoder and decoder. The decoder output is a chunk actions of size C. Chunking is particularly important for higg-freeuency fine- grained manipulation tasks, with chunk sizes C' being as high as 100 [6].
4 Automatic Waypoint Extraction for Imitation Learning
The goal of this section to develop our method for Automatic Waypoint Extraction (AWE). First, we define an objective that assesses the quality of the reconstructed trajectory for given sequence of waypoints. Next, we show how simple dynamic programming algorithm be used to select the minimal number of waypoints that have reconstruction error below specified threshold. Finally, discuss how to preprocess demonstration dataset using AWE before plugging into the BC algorithm, along with the some practical considerations when training and evaluating waypoint -base policy.
Reconstruction Loss. For an expert demonstration т, define the sequence of proprioceptives as
Tp (x, 1/11= and let W denote sequence of waypoints such that W = {wo.... wr}, where denotes the proprioceptive information in the waypoint. We reconstruct an approximate trajectory by interpolating between the waypoints, i.e., = f(V) for an interpolation function f. While we restrict linear interpolation in this work, the framework can be extended to incorporate splines.
To measure how well sequence of waypoints approximates the true trajectory, measure how much the interpolated trajectory deviates from the true trajectory. We define the reconstruction loss as the maximum distance of any state in the original trajectory from the reconstructed trajectory, that is,
[(f, = max min f(r, .))
2ET, DEF
where .(:,)) is some distance function (for example, Euclidean & distance). The minser (() denotes shortest distance of proprioceptive state to the interpolated trajectory 4. How do we aggregate projection errors for proprioceptives in the true trajectory? While there are several options, for example, mean projection еттоr over Tp, we define the reconstruction loss as the maximum projection error over all proprioceptives in Tp- The success of a trajectory often re- lies on reaching key states, and the mean error can be low while having a high projection er- ror for those key states. While a low reconstruction loss with maximum projection error also does not guarantee downstream success, it encourages minimizing outlier projection errors po- tentially critical for successful execution. The reconstruction loss L is visualized in Figure 2.
3

---

Waypoint Selection via Dynamic Programming. Given the reconstruction loss, how do we use it to optimize way- points? We consider the following optimization problem:
min [W] s.t. L(f(W),
(3)
x
reconsfruction loss
i.e. minimize the number of selected waypoints such that the reconstruction loss is below the budget 7). As presented, waypoints can be arbitrary points in the proprioceptive space, but we will restrict waypoint selection to the states visited in the expert trajectory The problem simplifies to finding the shortest subsequence of Tp such that the reconstruction loss is less than 7) which can be solved efficiently with dynamic programming (DP). For trajectory segment, either linearly interpolating between the endpoints sufficiently reconstructs the segment (i.e, reconstruction loss less than 1)), in which case the endpoints are returned as waypoints. Or for every intermediate state between the endpoints: (1) break the trajectory into two segments at that intermediate state and, (2) recursively find the shortest subsequence for each segment. Finally, choose the intermediate state resulting the shortest subsequence when the waypoints from its two trajectory segments are merged, and return the merged waypoints. The pseudocode for selecting waypoints with DP is in Algorithm 1.
Demonstration r
Reconstructed f
Waypoint w
Figure 2: Visualizing the loss L.
Preprocessing Demonstrations. For an expert trajectory T = (00,,.0),... (or, [r)), denote the selected waypoints as W = {(wn, to). (w., {t.). where w, denotes the waypoint and t; denotes the time index in T. The training problem for BC algorithm changes from predicting the next proprioceptive state to next waypoint. However, if done naïvely, the training dataset of next use all observations in between two consecutive
waypoints will be much smaller. But, waypoints by labeling them with closest waypoint after the observation. This follows from the intuition that following the waypoints implies the robot tries to reach W/++1 from Wk. and therefore, should target Wk+1 from intermediate states between them as well. The final dataset can be written as D Disapoint = {(t,, It, Weext (t())) )r=o where next. wp()) = argmin (0,
such that i, > t. The process of selecting waypoints and constructing the augmented dataset is repeated for every expert demonstration D, and the resulting datasets are merged to get the final training dataset.
Overview and Practical Considerations. We have proposed AWE, simple method that can preprocess a demonstration into sequence of waypoints without any additional supervision. The training dataset can be relabeled with the next waypoint instead of next propriopceptive state, and plugged into a BC pipeline. The choice of policy distribution class used with AWE is important; waypoints introduce increased multi-modality into the conditional action distribution as different demonstrations may be processed into different waypoints. Using more expressive policy classes capable of representing multi-modal action distributions is critical, as introducing waypoints can make the performance worse for less expressive policy classes (Figure 6).
Why does AWE return meaningful waypoints? An intuitive notion of waypoints relies on registering important events happening in the extrinsic environment (grasping a cup, opening a door, etc.) while AWE uses just the proprioceptive information to select waypoints. AWE relies on the idea that the expert demonstrations will naturally deviate from linear motion during such key events. For simpler parts of the task, such free-space reaching, demonstrations linear motion, resulting in fewer waypoints. Moreover, decreasing the budget 1) allows for selecting more waypoints in general, and thus, better reconstruction as visualized in Figure 4.
more likely to be approximated by
An important consideration at test-time is to allow more time for position-control to reach waypoints, as waypoints are farther apart compared to proprioceptive positions in the original expert demon- strations. The exact instantiation for the low-level controller depends on the whether the robot is operating in the joint space end-effector space, which discuss in Appendix C.I.
5 Experiments
Our experiments seek to answer the following questions: (1) How well does AWE combine with representative behavioral cloning methods? (2) Can it be used to tackle standard imitation learning

---

Table 1: Success rate (% for simulated bimanual tasks. report results on both training with scripted data and training with human data, with seeds and 50 policy evaluations each. Baseline results are obtained from Zhao et al. [6]. Overall, AWE +ACT significantly outperforms previous methods.
Cube Transfer
Bimanual Insertion
scripted data
human data
scripted data
human data
BC-ConvMLP
0
BeT [14]
27
1
3
0
RT-1 [15]
2
0
VINN [24]
3
0
1
0
ACT [6]
86
50
32
20
AWE +ACT (Ours)
99
71
57
30
benchmarks with real human demonstrations? (3) Can it be effective real-robot? (4) How does the parameterization of the policy affect the performance? (5) How do the selected waypoints and downstream performance change we vary the hyperparameters? To answer these questions, we compare the performance of recent state-of-the-art BC methods with and without AWE on 8 tasks and 10 datasets. First, we evaluate AWE on set of simulation environments, specifically two bimanual manipulation tasks from Zhao al. [6] and three manipulation tasks from the RoboMimic benchmark [17]. We evaluate AWE on of three bimanual manipulation tasks on the real robot: coffee making, wiping the table and screwdriver handover. Hyperparameter and implementation details can be found in Appendix B and respectively.
5.1 Bimanual Simulation Suite
The bimanual simulation suite contains two fine- riee-graned long- hong-horizon manipulation tasks MuJoCo [40]. The observation space includes 480 x 640 image and the current joint positions for both robots. The 14-dimensional action space corresponds to the target joint positions. Demonstrations 400 to 500 steps at control frequency of 50Hz. In the Cube Transfer task, the right robot arm needs to pick up the cube from random position on the table, and then hand it to the left arm mid-air. For Bimanual Insertion, both the peg and the socket are placed randomly on the table. The arms need to first pick them respectively, then insert the peg into the socket mid-air. Both tasks require delicate coordination between the two arms and closed-loop visual feedback: error in grasping can directly lead failure of handover or insertion.
Two datasets available for each task: collected with scripted policy, and one collected by human demonstrators, both with 50 demonstrations. As shown in Table 1, AWE outperforms competitive BC baselines on all tasks and datasets in the bimanual simulation suite, where some of the baselines completely fail due to task difficulty. AWE can increase the success rate ACT, the state-of-the-ar method on this benchmark 17% on an average, and up to 25% on the scripted bimanual insertion. The effective length of the training demonstrations reduce by factor of 7x to 10x, even allowing for improvements on human data which is fairly multi-modal to begin with. Notably, the performance improves by 50% when imitating human demonstrations the bimanual insertion task. Overall, this suggests that the benefit from reducing the effective training horizon exceeds any potential downside from the increased multi mudti-modality introduced by AWE.
5.2 RoboMimic Suite
Next, we evaluate on three simulated tasks from the RoboMimic [17] manipulation suite: Lift where the robot has pick up cube from the table, Can where the robots required pick up soda can from large bin and place it into smaller target bin, and Square where robots are tasked to pick up square nut and place it on rod. It is challenging due to the high precision needed to pick up the handle and insert it into a tightly-fitted rod. Episodes start with randomly initialized object configurations. All environments return RGB observations and the action space is the 6DoF end-effector pose, with an additional degree for the gripper.
We combine AWE with the sate-oo-the-an method this benchmark, Diffusion Policy [5]. Since diffusion policy achieves near- cear-perfect success rates Lift, Can, and Square when training
5

---

Table Success rate (%) behavior cloning benchmark, RoboMimic (Visual Policy), AWE + Diffusion is more data-efficient than previous methods. We evaluate the policy every 100 epochs across the training, and report the average of the max performance across 3 training seeds and 30 different environment initial conditions (90 in total), Results on LSTM-GMMM and IBC are obtained from Chi et al. [5] for comparison to more traditional methods. The performance scaling visualized in Figure
Task
# Demos
AWE + Diffusion (Ours)
Diffusion
LSTM-GMM
IBC
100.0 0.0
100.0 0.0
100.0 0.0
100.0 0.0
Lift
100
100.0 + 0.0
100.0 0.0
100.0 0.0
100.0 0.0
96
73
30
69.0 1.4
61.0
50
85.7 + 1.9
82.3+ 3.3
Can
100
95.3 1.7
93.3 0.0
200
96.7
97.3 2.5
88
62.3 = 3.3
44.3 6.1
50
67. = 2.9
57.3 + 4.2
Square
100
91.7 3.9
82.0 + 7.0
200
94.7 = 3.9
95.0 + 4.1
59
dataset with 200 proficient-human demonstrations, we focus our evaluation on how the performance scales with the number of demonstrations, both with and without AWE. The results in Table suggest that AWE consistently improves the performance of diffusion policy as the number of demonstrations is scaled from 30 to 200, while both of them outperform LSTM-GMM and implicit BC [41] with half the demonstration data, or even less. The improvements are larger when the number of demonstrations smaller or the task is longer-horizons for example, an 18% increase in the success rate when using 30 demonstrations on the Square task.
5.3 Real- World Bimanual Tasks
For real-robot evaluations, we use ALOHA [6]. low-cost open-source bimanual hardware setup. The setup consists of two leader arms and two follower arms, where the joint positions are synchronized between the leaders and followers during teleoperation. The observation space consists of RGB images from cameras: two are mounted on the wrist of the follower robots, allowing for close-up views of objects for tie-mmanpulattin, and the other two are mounted on the front and at the top respectively. The demonstration data consists of 4 camera streams and the joint positions for each robot at 50Hz. We refer readers to the original paper for more hardware details.
We experiment with three long-horizon tasks, each requiring precise coordination between the two arms, illustrated Figure 3. For Screwdriver Handover, the right arm needs to pick up the screwdriver that is randomly initialized in 15cm x20cm rectangular region (#1) and hand left arm mid-air (#2), followed by the left arm dropping into the cup (#3), For Wiping the Table, roll of paper towels is randomly placed in 15em x 10em region. The opening of the roll always faces the right side, with naturally occurring variations in length and spacing. The left arm presses the roll to prevent it from moving (#1), while the right arm tears off one segment of the paper towel (#2) and places it on fixed location to absorb the spilled liquid (#3). For Coffee Making, a small coffee pod randomized in a 15cm x x10cm region. The left arm needs to pick it up (#1), followed by the right arm opening the coffee machine (#2). The left arm then carefully inserts the coffee pod into the slot (#3), with the right arm closing the lid (#4). Next, the right arm grasps a transparent cup with upto 2em randomization in the position and places it under the coffee outlet (#5).
the
The three tasks emphasize precision and coordination, and involve deformable or transparent objects that can be hard to perceive or simulate. For example, placing the coffee pod into the machine requires high precision. the small clearance. The screwdriver handover task emphasizes the coordination between two arms. Grasping the paper towel requires accurate perception of the deformable material, which also has low-contrast against itself. The gripper needs to move accurately as only grasp the opening but not collide with the roll and push it away.
easy for the gripper or coffee pod to collide with the machine due to

---

Serewdriver Hundover
Wiping the Table
Coffee Malking
Figure 3: Real-World Bimanual Tasks, We consider three challenging real-world bi-manual tasks: (top) picking up a screw driver, handing it over to the other arm, and placing it in a cup, (middle) tearing off a segment of paper towel and putting it spill, and (bottom) putting coffee pod into coffee machine, closing the coffee machine, and placing a cup undemeath the dispenser. Initial object positions are within the red rectangle.
Table 3: Success rate (%) for real world tasks. AWE improves the success of ACT on all three tasks ranging from 4% 28%. On the longest horizon coffee making task, AWE improves success by 28%.
Screwdriver Handover
Wiping the Table
Coffee Making
ACT
84
92
36
AWE + ACT (Ours)
92
96
64
As shown in Table 3, AWE achieves substantial success on each task. It consistently improves over ACT by 8%, 4%, and 28% on Screwdriver handover, Wiping table, and Coffee Making, respectively, We observe that the most common failure case for ACT is inaccurate action prediction, which results from compounding errors on these long-horizon tasks. For example, the robot may make wrong prediction at the beginning and grasp the coffee pod at an inconvenient position. The subsequent predictions become increasingly incorrect, and thus the robot fails to insert the coffee pod into the machine. On the other hand, AWE can more accurately grasp the coffee pod due to smaller decision horizon, resulting in more successful insertions into the coffee machine. Leveraging the low-level controller to execute linear motions instead of relying on accurate policy predictions can reduce the errors compounded over time. By accurately detecting waypoints for successful handover, for tearing, and for inserting, AWE decreases the policy horizon and consistently improves performance.
5.4 Analysis
Waypoint selection for different error budgets. We visualize ground truth trajectory of end- effector (EE) positions and the EE trajectory reconstructed using AWE for the Can task in Figure 4. As the the error budget is reduced, the reconstructed trajectory tracks the original trajectory better. Importantly, as the budget is decreased, waypoints are added to harder segments of the task, as they are less linear while the number of waypoints for simpler, linear paths stays similar. Smaller error thresholds lead to gradual increases in the number of selected waypoints. We also measure performance with varying error thresholds (the only hyperparameter), for AWE +DiffusionPolicy on the Can task with 50 demonstrations. Figure shows that when the 7) is too high (too few waypoints) or too low (too many waypoints), the agent does take full advantage of AWE.
On the importance of modeling multi-modality for AWE. The usage of waypoints can increase the multimodality of the target conditional action distribution. We compare the performance of AWE when trained with mcan-squared error (MSE) loss (i.e, a unimodal Gaussian with identity covariance) and a more expressive Gaussian mixture model (GMM) with modes. As shown in Figure 6, GMM policies can benefit from AWE, as they can represent multimodal action distributions. However, vanilla BC with MSE loss degrades in performance. BC has mode-covering behavior, and insufficient representative power of unimodal Gaussian can cause the performance to degrade.
7

---

brdgett
encor bodget
?)
0.01
7) 0.005
AWE
added the path & seliiivlly Looa
Success Rate (%)
W/o AWE
80
Whopsista apprsination oftbe path
Ground Truth
Waypoints
Error Threshold
Figure error budget 7/ decreases, our method selects fewer waypoints if linear interpolation aptly approximates the segment. Best viewed on our website.
Figure Success rate vs. error budget threshold 7). Performance drops slightly if the budget is too tight and more significantly if the budget is too permissive.
100
100
Replay Success (9))
90
80-
Velocity pper
Constant
NWE
5.5
6.1
6.8 #Waypoints
7.4
6.1
Uni
Figure AWE requires expressive policy classes. While expressive policies that can represent multimodal distributions benefit from AWE (GMMs on left), the performance degrade for policy classes that sufficiently expressive (right).
Figure 7: Comparing the replay success rate of AWE to common heuristics for waypoint selection. With similar numbers of waypoints, following waypoints from AWE leads more consistent task completion following waypoints from heuristics.
Comparison to heuristics. Prior works [9, 10, 4, 11] have been successful by extracting waypoints using simple hcuristics. Are simple heuristics enough for extracting important waypoints? We experiment with two heuristics. The first one is similar to Shridhar et al. [4], labeling timesteps as waypoints when the end-effector velocity is close to zero, or when the binary gripper state changes. The second heuristic seleets waypoints with fixed intervals. For AWE and both heuristics, we extract waypoints for all 200 trajectories the RoboMimic Lift dataset, and measure the success rate when replaying the demonstration, i.e., following the extracted waypoints starting from the demonstration trajectory's 's initial state. We adjust the selection threshold interval generate similar numbers of waypoints across methods for comparable results. Results in Figure show that these two heuristics do not lead to satisfactory success rates even when simply replaying the trajectories.
6 Conclusion
We presented method for extracting waypoints from demonstrations of robotic manipulation tasks, therefore reducing the horizon of imitation learning problems. We found that AWE can be combined with state-of-the-art imitation learning methods such diffusion policy and ACT to improve performance, especially in data limited settings. AWE also consistently improved performance on three real-world dexterous manipulation tasks. Finally our analysis indicated the importance of the AWE optimization compared to naive or heuristic waypoint selection methods, as well as the effect of the error budget and policy distribution class on performance.
Linitations. AWE leverages proprioceptive information reparameterize demonstration trajectories in end-effector or joint space, an approach that may not be applicable to torque-controlled robot arms, tasks requiring forceful manipulation, or other robotics problems such as purely visual navigation or legged locomotion. Our evaluation only considers quasi-static tasks, and AWE currently does not account for velocities, which might be important for dynamic tasks. Furthermore, for tasks that require extreme precision certain times, we expect that AWE would require tight error budget, diluting the benefit of using waypoints. This limitation might be resolved by identifying when such precision is needed, either automatically by incorporating some human supervision, and subsequently modifying the AWE optimization objective.

---

Acknowledgments
This work was supported by Schmidt Futures and ONR grants N00014-20-1-2675 and N00014-21- 1-2685. We would like to thank Suneel Belkhale and Chen Wang for helpful discussions, and all members of the IRIS lab for constructive feedback.
References
[1] D. A. Pomerleau. Alvinn: An autonomous land vehicle in neural network. Advances in neural information processing systems, 1, 1988.
[2] T. Zhang, Z. McCarthy, O. Jow. D. Lee, X. Chen, K. Goldberg, and P. Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. In 2018 IEEE International Conference on Roboties and Automation (ICRA), pages
IEEE, 2018.
[3] A. Zeng, S. Song, J. Lee, A. Rodriguez, and T. Funkhouser. Tossingbot: Learning to throw arbitrary objects with residual physics. IEEE Transactions Robotics, 36(4):130 2020.
[4] M. Shridhar, L. Manuelli, and D. Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Conference on Robot Learning, pages 785-799. PMLR, 2023.
[5] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprini arXiv:2303.04137, 2023.
[6] Z. Zhao, V. Kumar, S. Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304. 2023.
[7] S. Ross and D. Bagnell, Efficient reductions for imitation learning. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 661-668. JMLR Workshop and Conference Proceedings, 2010.
[8] S. Ross, G. Gordon, and D. Bagnell. reduction imitation learning and structured prediction to no-regret online learning. Proceedings of the fourteenth international conference on artifi- cial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Proceedings, 2011.
[9] K. Hsiao and T. Lozano-Perez. Imitation learning of whole-body grasps. 2006 IEEE/RSJ international conference on intelligent robots and systems, pages 5657-5662. IEEE, 2006.
[10] B. Akgun, M. Cakmak, K. Jiang, and A. Thomaz. Keyframe-based learning from demonstra- tion: Method and evaluation. International Journal of Social Robotics, 4:343-355, 2012.
[11] S. James and A. J. Davison. Q-attention: Enabling efficient learning for vision-based robotic manipulation. IEEE Robotics and Automation Letters, 1(2):1612-1619, 2022.
[12] S. Schaal. Is imitation learning the route to humanoid robots? Trends in cognitive sciences, (6):233-242, 1999.
[13] B. D. Argall, S. Chernova, M. Veloso, and B. Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469-488 2009.
[14] N. M. Shafiullah, 7. Cui, A. A. Altanzaya, and I.. Pinto. Behavior transformers: Cloning modes with one stone. Advances in neural information processing systems, 35:22955-22968, 2022.
[15] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, et al. Rt-l: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06811 2022.
[16] Y. Zhu, A. Joshi, P. Stone, and Y. Zhu. Viola: Imitation learning for vision based manipulation with object proposal priors. arXiv preprint arXiv:2210.1133 2022.

---

[17] A. Mandlekar, D. Xu. J. Wong, S. Nasiriany, C. Wang. R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martin-Martin. What matters in learning from offline human demonstrations for robot manipulation. In arXiv preprint 2XXv:2/08.03299, 2021.
[18] Florence, C. Lynch, A. Zeng, O. A. Ramirez, A. Wahid, L. Downs, A. Wong, Lee. I. Mordatch, and J. Tompson. Implicit behavioral cloning. In Conference on Robot Learning, pages 158-168. PMLR, 2022.
[19] A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani, et al. Transporter networks: Rearranging the visual world for robotic manipulation. Conference on Robot Learning, pages 726-747, PMLR, 2021.
[20] C. Wang, R. Wang, A. Mandlekar, L. Fei-Fei, S. Savarese, and D. Xu. Generalization through hand-eye coordination: An action space for learning ppatiall-iivariin visuomotor control. In 202/ IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 8913-8920. IEEE, 2021.
[21] E. Johns. Coarse-to-fine imitation learning: Robot manipulation from single demonstration. In 2021 IEEE international conference on robotics and automation (ICRA), pages 4613 4619. IEEE, 2021.
[22] M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipula- tion. In Conference on Robot Learning, pages 994-906. PMLR, 2022.
[23] P. Florence, L. Manuelli, and R. Tedrake. Self-supervised correspondence in visuomotor policy learning. IEEE Robotics and Automation Letters, 5(2): 499, 2019.
[24] Pari, N. M. Shafiullah, S. Arunachalam, and L. Pinto. The surprising effectiveness of representation learning for visual imitation. arXiv preprint arXiv:2112.01511, 2021.
[25] M. Jia, D. Wang, G. Su, D. Klee, X. Zhu, R. Walters, and R. Platt. Seil: Simulation-augmentee equivariant imitation learning. arXiv preprint 2rXiv:2211.00114, 2022.
[26] A. Zhou, M. J. Kim, L. Wang, Florence, and C. Finn. Nerf in the palm of your hand: Corrective augmentation for robotics via novel-view synthesis. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 17907-17917, 2023.
[27] M. Laskey, S. Staszak, W. Y.-S. Hsich, J. Mahler, F. T. Pokorny, A. D. Dragan, and K. Goldberg. Shiv: Reducing supervisor burden in dagger using support vectors for efficient learning from demonstrations high dimensional state spaces. In 2016 IEEE International Conference on Robotics and Automation (ICRA). pages 462-469. IEEE, 2016.
[28] E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn. Bc-Z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning. pages 991-1002. PMLR, 2022.
[29] D. Morrison, P. Corke, and J. Leitner. Closing the loop for robotic grasping: A real-time, generative grasp synthesis approach. Robotics: Science And Systems, 2018. 105:70.15 2018.XIV.021.
[30] A. Zeng, S. Song, S. Welker, J. Lee, A. Rodriguez, and T. Funkhouser. Learning synergies between pushing and grasping with self-supervised deep reinforcement learning. leee/rjs International Conference On Intelligent Robots And Systems, 2018. doi: 109/ 8593986.
[31] Wu, X. Sun, A. Zeng, S. Song, Lee, S. Rusinkiewicz, and T. Funkhouser. Spatial action maps for mobile manipulation. Robotics: Science And Systems, 2020. doi:100.1607/RSS..2000 XVI.035.
10

---

[32] K. Zakka, A. Zeng. J. Lee, and S. Song. Form2fit: Learning shape priors for generalizable assembly from disassembly. leee International Conference On Robotics And Automation, 2019.
00010.1109/CRAA444522000 9196733.
[33] A. Zeng, S. Song, J. Lee, A. Rodriguez, and T. Funkhouser. Tossingbot: Learning to throw arbitrary objects with residual physics. leee Transactions On Robotics, 2019. doi: 1e.11...11111 1109/TRO. 2020.2988642.
[34] A. Zeng. S. Song, K.-T. Yu, E. Donlon, F. R. Hogan, M. Bauza, D. Ma, O. Taylor, M. Liu, E. Romo, N. Fazeli, F. Alet, N. C. Dafle, R. Holladay, I. Morona, P. Q. Nair, D. Green, L. Taylor, W. Liu, T. Funkhouser, and A. Rodriguez. Robotic pick-and-place of novel objects in clutter with multi-affordance grasping and cross-domain image matching. arXiv preprint arXiv: 1710.01330, 2017.
[35] S. Belkhale, Y. Cui, and D. Sadigh. Hydra: Hybrid robot actions for imitation learning. arXiv preprint arXiv: 2306. 17237, 2023.
[36] Ho, A. Jain, and Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[37] M. Welling and Y. W. Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11), pages 681-688, 2011.
[38] D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv: 1312.6144 2013.
[39] K. Sohn, H. Lee, and X. Yan. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, 28, 2015.
[40] E. Todorov, T. Erez, and Tassa. Mujoco: physies engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033, 2012.
[41] R. Florence, C. Lynch, A. Zeng, O. Ramirez, A. Wahid, L. Downs, A. S. Wong, J. Lee, Mordatch, and Tompson. Implicit behavioral cloning. Conference On Robot Learning, 2021.
[42] S. Tonneau, A. Del Prete, J. Pettré, C. Park, D. Manocha, and N. Mansard. An efficient acyclic contact planner for multiped robots. IEEE Transactions on Robotics, 34(3):586-601, 2018. doi: 10:10000//TR..00111111 8 2.211968.
[43] B. Ichter, E. Schmerling. T.-W. E. Lee, and A. Faust. Learned critical probabilistic roadmaps for robotic motion planning. arXiv preprint arXiv: 1910.03701, 2019.
[44] Lai. Rapidly-exploring random forest: Adaptively exploits local structure with generalised multi-trees motion planning. IEEE Roboties and Automation Letters. 2021.

---

A AWE Pseudocode
We provide the complete pseudocode for AWE in Algorithm 1.
Algorithm 1 Automatic Waypoint Extraction (AWE)
input: D; // expert demonstrations
input: L. J,
II waypoint selection via dynamic programming
def f get.wayyointss(..y.. M.:
i & M then
I/ check if the endpoints are valid waypoints
if [f T.end)), I 1) then
M [T] (r.start,,
// try all intermediate states as waypoints, and return the smallest set else
// initialize length of current shortest subsequence
m + oo;
// loop over all intermediate states waypoints for to E d.
Whefore get waypoints(x.beecrre()jjj
Waner &- geel.waypints(T.ar (te I/ dedupe w, as it is in both of them
W + (Moefore (1)) U Wateri
if [ < then
m < W|:
M [r] W: return M [r]
I/ construct dataset for next waypoint prediction def preprocess traj(W, r):
Dang <- {}: for (ot, E do
// select the nearest future waypoint in W
+- W.next_waaeeeeeeeeeeeeeeeee (.next_waypoint())
Daug + Dmg U ( Lt, w)};
return Daugi
Dnew : for T E D do
M /1: memoize waypoints for efficient dynamic programming
Dnew +- Dnew U Uprepnoes..rgggeetwwwyyoonss M)) m),
output: D Drew
B Hyperparameters
B.1 Error Budget Threshold
The only hyperparameter we need for waypoint selection is y7. the error threshold (Table 4). The threshold is the same for all data sizes {30, 50, 100, 200} across all tasks on RoboMimic, ie. 7) 0.005. We also use consistent 7 for both scripted data and human data on both tasks in the Bimanual Manipulation suite, i.e. 7) 0.01. Two out of three real-world tasks also the same 7. however, the Coffee Making task, opt for lower 7 to select more waypoints due to the high-precision nature of the task.
12

---

Table 4: Hyperparameter for waypoint selection.
Task
Error thresholod (1)
Lift
0.005
Can
0.005
Square
0.005
Cube Transfer
0.01
Bimanual Insertion
0.01
Screwdriver Handover
0.01
Wiping Table
0.01
Coffee Making
0.008
B.2 ACT in Bimanual Simulation Suite
We use the same hyperparameters as the ACT paper [6], shown in Table 5, except reducing the chunk size from 100 50. Intuitively, the length of trajectories reduces after running AWE, the chunk size also be reduced represent the same wall-clock time.
Hyperparameter
ACT
AWE ACT
learning rate batch size
le-5
le-5
8
8
encoder layers
4
# decoder layers
7
7
feedforward dimension
3200
3200
hidden dimension
512
512
heads
8
8
chunk size
100
50
beta dropout
10
10
0.1
0.1
Table 5: Hyperrpaaaeeters of AWE +ACT and ACT. The only difference is the reduction in chunk size.
B.3
Diffusson Policy in RoboMimic
We use the exact same training hyperparameters as Diffusion Policy [5] (Table 6). The only additional hyperparameter added is the "control multiplier" (bottom row), which allows the low-level controller to take more steps to reach the target position at the inference time. This can be useful when predicted waypoints are far apart.
B.4 A Guide to Hyperparameter Selection
We suggest selecting an error threshold for new tasks based on ratio of the number of waypoints to the average length of the trajectories. Our recommendation is aim for ratio of approximately 1:8, which can be automatically calculated using the waypoint generation script codebase. The ideal ratio may vary depending on the specific task and control frequency. Based a ratio between 1:5 and 1:15 tends to effectively reduce the policy horizon while still maintaining an accurate approximation of the trajectories.
empirical findings,
For tasks involving real robots using ALOHA hardware [6]. we advise turning on temporal ensembling (Sec C.3) ensure smoother actions. Nonetheless, the policy appears overly hesitant, two potential remedies are: (a) disabling temporal ensembling, and (b) increasing DT to emulate blocking controller, where DT refers to the time interval between each update in simulation control loop.
13

---

Hyperparameter
Lift
Can
Square
Ctrl
Pos
Pos
Pos
To
2
2
To
8
Tp
10
10
10
# D-params
9
# V-params
22
22
22
# Layers
8
Emb Dim
256
256
256
Attn Dropout
0.3
0.3
0.3
Lr
1e-4
1e-4
le-4
WDecay
1e-3
1e-3
le-3
D-Iters Train
100
100
100
D-Iters Eval
100
100
100
Control Multiplier
10
10
Table 6: Hyperparameters for diffusion policy. Ctrl: position or velocity control, T;: observation horizon, Ta: action horizon, Tp: action prediction horizon, #D-Parrms: diffusion network number of parameters in millions, #V-Parama vision encoder number of parameters in millions, Emb Dim: transformer token embedding dimension, Attn Dropout: transformer attention dropout probability, Lr: learining rate, WDecay: weight decay (for transformer only), D-Iters, Train: number of training diffusion iterations, D-Iters Eval: number of inference diffusion iterations, Control Multiplier: multiplier for the low-level control steps.
C Implementation and Experiment Details
C.1
Controller
We an Operation Space Controller (OSC) in RoboMimic, which allows position and orientation control of the robot's iss end-effector. It takes in the desired absolute position and orientation of the end-effector, and computes the necessary torques and velocities.
We use the default joint position controller in the Bimanual Manipulation suite. On real-world tasks, made no change the controller except for the Coffee Making task, where increased the step time from 0.02 to 0.1. This allows the controller to operate closer to blocking controller, and execute low-level actions longer until reaching the desired joint position.
C.2 Loss Function
To determine the distance between potential waypoints and the ground truth trajectory, we project the ground truth state onto the linearly interpolated waypoint trajectory and compute the I.2 distance for xyz position. For orientation, convert the axis angles quaternions and slerp two ground truth quaternions to determine the projection. Then sum the position and orientation distances the state loss. For the trajectory loss, we take a max over all states.
C.3 Temporal Ensemble
For the ACT experiments, we adopt a temporal ensemble technique as in the original paper [6]. Temporal ensembling is approach improve the smoothness of action chunking robotic tasks. queries the policy each timestep, creating overlapping chunks and multiple predicted actions for each timestep. These predictions weighting scheme, technique enhances the precision and smoothness of motion without any additional training cost, but requires extra computation during inference. We refer readers to Zhao et al. [6] for more details.
then combined via weighted average using exponential exp(-m x i). that helps in smoothly incorporating new observations. This
C.4 Computation Cost
Computing waypoints is inexpensive, especially compared to the training budget. The wall clock time for labeling one trajectory in Lift is 0.8 seconds on average.
14

---

AWE++DifussonPPlicy (Ours)
DiffusionPolicy
Can
Square
100
100
Success Rate (%)
90
80
80
70
60
30
100
200
30
100
200
Number of Demos
Number Demos
Figure 8: Performance scaling with demonstrations. We compare how the performance scale for diffusion policy [5] with and without AWE. Training on waypoints generated by AWE consistently improves the performance, with improvements being larger on the harder task (Square).
D
Additional Comparisons
D.1 Subsampling
A potential alternative to our proposed AWE method is the straightforward approach of subsampling trajectories. This implicit selection of waypoints can be viewed as a heuristic method. Figure demonstrates that AWE achieves a superior replay success rate, i.e., when one follows the extracted waypoints, starting from the demonstration trajectory's 's initial state, than subsampling. However, how do these methods influence the performance downstream?
To address this, we compare the success rate of a policy learned using waypoints selected by AWE against those from subsampled trajectories. We experiment on two RoboMimic tasks, Can and Square, both using 100 demonstrations. We compare against two subsampling ratios, specifically and ratio of produces a number of waypoints comparable to that of AWE in RoboMimic.
AWE (Ours)
Subsampled by 7
Subsampled by 5
Can (100 demo)
95.3
77.3
72.7
Square (100 demo)
91.7
77.3
86.4
Table 7: Comparison of success rates for policies learned using AWE and subsampling methods Can and Square.
As shown in Table 7, we find that: 1) AWE consistently surpasses the subsampling approach: 2) The most effective subsampling ratio is task-dependent. For instance, in the Can task, subsampling by exceeds subsampling by Conversely, the Square task sees better results with a ratio of This variance suggests that AWE can discern and select waypoints that are more instrumental for downstream learning.
D.2
Keypose-bace based motion planning
Our work conceptually related to keypose-based motion planning, which has seen notable contribu- tions in recent years. Tonneau et al. [42] utilized keyposes for multi-robot planning, emphasizing bottleneck states. Ichter [43] employed prohabilistic roadmaps to identify critical configurations, while Lai [44] R method adaptively targets bottleneck regions. These methods require complete knowledge of the environment plan trajectories. In contrast, method derives waypoints from the robot's proprioceptive data. We assume we only have raw RGB images and no low-level information about the external environment.
15


Reviews:
[{"review_id": "T-nJwpuEEW", "reviewer": "Reviewer_ZvMK", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "\nThe authors propose an algorithm for \"Automatic Waypoint Extraction\", which approximates the demonstrated proprioceptive trajectory through a shorter sequence of \"way points\" to improve the performance of imitation learning algorithms by reducing the \"compounding errors\". \n\nThe points are selected by approximating the original trajectory with continuous piecewise linear / (piecewise constant) functions and added to the dataset as an additional modality.\n\nThe method is tested in a set of experiments in conjunction with two imitation learning techniques: Diffusion Policy (simulated benchmark) and Action Chunking with Transformers (ACT) (experiments with real robots).\n\nThe experiments show that learning methods perform significantly better with the approximated trajectories.", "summary_of_recommendation": "Overall good work, but clarity and quality could be impoved. \nWith minor improvements, this work can provide interesting insights and ground for fruitful discussion at the conference.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "* is the actual interpolation linear or constant? \n* does the augmentation with the closest waypoint serve as an intermediate goal and an outlook into the future (which might contribute to a more stable policy)?\n* how can we derive that the reason for the improved performance is the reduction in \"Compounding Error\" based on the experiments presented in the paper?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Clarity\n\nThe paper is written fairly well and clear. Following remarks might help to improve the clarity of the paper. \n\n## Minor remarks\n- Figure 4: hard to see, especally in a printed version. Maybe a 2d projection would be better for illustration purposes\n- mathematical notations are a bit confusing at times, e.g., line 114 vs line 88. Maybe better someting like $X_\\tau$ instead of $\\tau_p$.\n- line 162: \"additional multi-modality\" => \"additional modality\"\n- line 173: \"test-time\" => \"run time\"\n- line 90: it is unclear which propriceptive data is actually being used in the presented experiments - joint angles or end the effector pose?\n- line 146: \"Algorithm 1\" - referenced, but doesn't exist in the paper (is located in the Appendix).\n- line 267: \"EE trajectory\" was not introduced (\"end-effector trajectory\"?)\n- line 288, 116: \", i.e. \" => \", i.e.,\"\n\n- Algorithm 1 (in the suplimentary material): \n  * $get waypoints(\\tau.before(w),\\eta)$ => $get waypoints(\\tau.before(w),\\eta,\\mathcal{M})$\n  * $get waypoints(\\tau.after(w),\\eta)$ => $get waypoints(\\tau.after(w),\\eta,\\mathcal{M})$\n\n## Form\n- paper makes extensive use of rethorical questions, which remain \"unanswered\" and reduce the clarity of the paper. They can be easyly reduced with concise statements. It would also save some space ;)\n  * Example 1: (line 6-7) \"Can we generate waypoints automatically without any additional human supervision?\" - rhetorically, this needs to be closed with something like \"We belive - yes, it is possible with our proposed method.\"\n  * Example 2: (line 166) a suggestive rethorical question \"Why does AWE return meaningful waypoints?\". Only an intuitive explanation is provided, which is alright in itself, however the question remains open. Replace it with a straigt forward question or title, e.g.,\n\"Does AWE return meaningful waypoints?\"\n- it would be good to include the main algorithm $get_waypoints(...)$ into the main paper. Du reduce its size, the comments can be removed as there is suficient explanation in text already. This would make the paper cearer.\n\n\n# Quality\n\nOverall, the quality of the presented work is fairly good.\n\nThe empirical study consists of primary experiments (in simuation and on the robots) and a number of secondary experiments studying the limits of the proposed approach.\nThe primary experiments are comprehensive and demonstrate an improvement in performance when the proposed pre-processing method is used.\n\n\n## Interpolation\nlines 173-176: \"allow more time\" - is a wague statement.\n\nFor comparison of the performance between waypoint-trajectory and not-approximated trajectories it could be crucial, that the time between the waypoints is calculated based on the real time of the trajectory. \nFrom the paper it is unclear if the overall action was executed with comparable speed.\nFor instance, executing the waypoint-trajectory significantly slower than original trajectory might affect the success of the action, e.g., reduce vibrations and make it more accurate.\n\nHow do other methods for waypoint selection handle interpolation?\nAdding measured times for the execution of the action could also help (in case the learned actions vary in time).\n\n\n## Linear / Constant\n\nAlthough the way points are calculated by means of piece-wise **linear** approximation, the resulting augmented trajectory seems to piece-wise **constant**.\n\nThey waypoints are selected from the trajectory points such that the distance error between the piecewise linear continuous approximation and the trajectory is below a given threashhold.\nThe data is augmented with the newly calculated waypoints.  Thereby, the number of waypoint is smaller than the number of points in the trajectory itself.\nThe augmentation is done by extending each tuple in the trajectory by the closest waypoint. This results, however, in a piecewise constant approximation of the proprioceptive trajectory.\n\nFor instance, consider the following sequence:\n{(o0, x0),     (o1, x1),     (o2, x2),     (o3, x3),     (o4, x4),     (o5, x5),     (o6, x6),     (o7, x7)}\nAssume, that (x_i) is approximated throgh two waypoint (a single line). Then the augmented data could look like this:\n{(o0, x0, w0), (o1, x1, w0), (o2, x2, w0), (o3, x3, w0), (o4, x4, w1), (o5, x5, w1), (o6, x6, w1), (o7, x7, w1)}\nMeaning, that the sequence (x_i) was approximated through two constant functions: f1(i) = w0 and f2(i) = w1.\n\nIn principle it makes sense. However, from the paper it is not clear that the authors are aware of this fact as the algorithm is \nformulated for piece-wise linear functions.\nPerhaps, if the approximation is formulated from the beginning as being piece-wise constant, \nthe norm $l(.,.)$ in (line 124) could chosen to be more simple and easie to calculate.\n\nAs one of practical considerations, (line 173-176), the authors state that the actual iterpolation is done by the motor \ncontroller resulting in a piece-wise linear trajectory.\nHowever, the learning algorithms is presented with a picewise constant trajectory, which could be a signifficant aspect. \n\n\n## Remarks on \"seconday experiments\"\nFollowing the aspects could contribute to a more conclusive and substantiated analysis:\n\n- Figure 7: how many waypoints were calculated in each instance? This would help to assess how the methods relate to each other.\n- Table 2: the performance of \"AWE+Diffusion\" gets worse for \"# Demos\" = 200. This is not adressed - would be important to explore further to understand the lmitations of the algorithm.\n- Figure 5: the effect of the error threashold is illustrated for \"Can task with 50 demonstrations\" (line 273). \nThe performance seems to decrease outside of a specific value. Is this value different for each task? How does it change with the number of demonstrations?\n- how were the heuristic methods selected for comparison?\n\n## Reduction in \"compounding error\"\n\nWith the proposed method, we observe a significant improvement in learning performance in simulated experiments as well as in real robots.\nOne of the arguments explaining this increase, presented in the paper, are the \"compounding errors\". \nHowever, it is not directly clear how this conclusion follows from the observed improvements. \nPerhaps, the origin of the increased performance could be subject of future investigations?\n\n\n# Originality\n\nThe presented work combines known approaches to achieve an increase in learning performance.\n\nFrom my perspective, two main contributions of the paper are:\n- application of piecewise linear (constant) approximation in preprocessing for imitation learning\n- experimental study of the effects of such approximation on learning performace \n\n## Citations\n\nOverall, the related work seems to be well cited and includes a number of recent publications.\n\n## Algorithm: AWE\n\nThe authors propose an algorithm for \"Automatic Waypoint Extraction\" by approximation with continuous, piecewise linear functions.\nThere is an extensive body of work on approximation with piecewise linear (and piecewise constant) function in different disciplines. \nIt seems unlikely that the particular way of computing the approximation is completely novel. \nDiscussion / comparison of related approaches would substantiate the proposed method.\nBesides the purely mathematical, related topics \"Decision tree learning\" and \"Segmented regression\" could be worth considering.\n\n## Compounding Error\n\nThe only paper explicitely cited on \"Compounding Error\" is from 2011. Do other papers cited in section 2 explicitely analyze the reduction in \"Compounding Error\"?\n\n\n\n# Significance\n\nOn the whole, the presented algorithm and results have a somewhat iterative character: the performance of an imitation algorithm is improved by an improved pre-preprocessing algorithm.\nThe observed improvements in learning, however, point to a potential for significant discoveries.\n\nEspecially promissing is the observed performance increase for a low number of training examples leading to a better data efficiency. \nLow data efficiency is often a major hurdle for learning algorithms to be applied in real world scenarios.\n\nMore generally, perhaps, we might think of the way points as intermediate goals or events.\nThis line of research might contribute to a deeper understanding of the interplay between continous and discrete cognition.\nDiscrete actions and events seem to play a crucial role in human cognition.\n\nAn experimental study presented in [*] has shown that $L_0$ regularization for training\nof recurrent neural networks can lead to improvements in learning performance. $L_0$\nregularization enforces piece-wise constant changes in the latent state. Those sparse\nchanges are essentially events.\n\n[*] Christian Gumbsch, Martin V. Butz, and Martius Georg. \"Sparsely Changing\nLatent States for Prediction and Planning in Partially Observable Domains\"\n\n\n# Relevance\n\nThe paper makes an empirical contribution to imitation learning and presents experiments performed on real robots:\n\n- perform experiments in simulation and on real robots\n- demonstrate improved performance in comparison to state of the art methods\n\nAs such, it is highly relevant to in the context of this conference.\n\n\n# Limitations\n\nThe limitations-section discusses technical limitations of the proposed method.\n\nIn some experiments, the observed performance seems to decrease, as described in \"Quality / \" (Table 2, Figure 5). \nThis could point to potential limitation of the approach.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "Rvci3aWLA7l", "reviewer": "Reviewer_cGVi", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work proposes a waypoint based approach for learning manipulation tasks from expert demonstrations. To extract these waypoints a simple dynamic programming approach is proposed. The dynamic programming based approach tries to find waypoints that are temporally distant and also linearly connected. ", "summary_of_recommendation": "The paper aims to tackle the challenging problem of covariate shift. However, in my opinion, the overall novelty is very limited. More importantly, the overall approach is also very arbitrary and not principled or grounded since there is no consistency across demonstrations and I think it's success heavily relies on task domains and kind of demonstrations being used for training. ", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "please see above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Pros:** The paper is easy to read and seems to tackle an interesting and challenging problem. However, the execution leaves many things to be desired. \n\n**Cons:** There is very limited novelty in this paper. The idea of using waypoints is not very novel. \n\nInfact, many approaches in robotics use an implicit form of this approach already. For instance, when we subsample trajectories that have been recorded at a high frequency we (i.e. reduce 100Hz trajectories to 10 Hz), we implicitly select waypoints and use linear interpolation between these waypoints. This is basically what the current paper proposes although they use a dynamic programming formulation to find these waypoints.\n\nThe same approach can also be seen as a slight generalization of the keypose based approaches which identify bottleneck states and then use planners to reach these bottleneck states. Infact, the idea of pre-processing demonstrations to add actions to next waypoint has been already proposed and extensively used in the keypose literature. Thus, overall, the only sort of novelty this paper has is to use a dynamic-programming approach to find waypoints. \n\nAdditionally, I don’t think using dynamic programming (DP) **separately** for each trajectory to find waypoints is optimal. First, since DP is being run separately for each trajectory, the waypoints that the algorithm ends up selecting may not be the consistent across trajectories. This inconsistency is quite sub-optimal since the idea of waypoints (as noted in the paper itself) is to find key bottleneck states, that are shared across demonstrations. But if the preprocessing selects very different waypoints the network will have to learn a much more complex policy which will generalize well. \n\nFinally, for most tasks considered in the paper there is clearly an object-reach stage (e.g. object, pick (lift), object pick-place (can)), so why shouldn’t a waypoint that is close to the object state (this can be discovered directly from contact mode) be directly used. In my view, such a policy can work very well, it would be interesting to see how such a baseline acts on some of tasks considered here. Also, I don’t think the DP based approach will select such an object-centric frame at all (given 1 waypoint to select) since it’s just doing local curve fitting, but clearly we know that using a planner or even classical movement primitives for this object-centric state is sufficient for pick-place tasks.", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}, {"review_id": "3PwNJ84Q11", "reviewer": "Reviewer_3s7U", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper tackle compound errors in Behavioral Cloning (BC) within the Learning from Demonstrations framework. The main idea is to sub-sample the demonstrations, reducing the number of points computed during rollouts, and hence reducing the errors. This creates the problem of selecting the demonstration subset. Selecting few points leads to loosing information, and too many points defeat the purpose of the method.\n\nThe main contribution of the paper is a simple, and yet effective, algorithm for selecting the subset of points based on approximating the demonstrated trajectories with straight lines, respecting a user given reconstruction error threshold, a.k.a, a hyper-parameter which controls the number of points and thus needs to be tuned for avoiding too many of too few points and the problems associated with those cases.\n\nThe method is a preprocessing step, and thus it is complementary to BC methods. The method is sufficiently tested, presenting promising results in most cases.", "summary_of_recommendation": "Please elaborate on the limitation listed in the review.\n\nI cannot only recommend a strong accept because the results are somehow incomplete and the all the claims are not really supported by the results. A better analysis needs to be done, despite the results seem promising.\n\nOn the Impact, the proposed method generates ambiguities and the requirement of multi-modal models limits its applicability, and thus we cannot consider major impacts in robotics.\n\nBeing the problems with the results correctly addressed, I vote in favor of the publication of the paper.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "I have no questions for rebutal, but please address the problems mentioned above. Fell free to rebut comments in case of disagreement.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is really well written. From the abstract you can understand the problem and the idea of the proposed approach. And I have no comments about the presentation of the work, or organization of the manuscript. I only have minor comments on the textual part which follows next\n\n1. in line 88, the subscript j is not defined.\n2. citation [1] in line 64 does not see correct. Please double check.\n3. In the label of table 3 and the claims, it says AWE improves success by 28%, it should be \"from 4%, up to 28% in the tested cases\", otherwise it is miss leading.\n4. Figure 5, missing units in the x-axis, seems to be in meters?\n5. In lines 183 and 184, the authors says that they compare AWE on 8 tasks and 10 datasets, but I do not know where this comparison is. Is that one of the tables in the text? please make it explicit.\n6. line 146, mentions Algorithm 1, which is not to be found in the paper. Where is Algorithm 1? please make it explicit.\n\nThe biggest critic that I had about the approach is that the selection of way-points clearly creates ambiguities in the trajectories, which greatly limits the BC learning capabilities. We can clearly see that this situation would arise in Figure 1. Nevertheless, the authors propose a time-based selection (preprocessing demonstration, line 147) and direct the proposed method to be used with multi-modal models (On the importance of multi-modality for AWE, line 275), which are designed to solve such problems. This is correct and well done.\nHowever, multi-modal models are harder to train, and an evaluation comparing how much data we need to train a: 1) a standard BC method without the way-points selection, 2) a multi-model BC without the way-points selection, and 3) a BC with the way-points selection would allows to evaluate the impacts of the selection and its synergy with multi-modal methods more complete than the current evaluation presented in Figure 6.\n\nStill on the topic of ambiguities created by AWE, we have two mechanisms for handling them, the time-base selection and the multi-modal BC methods. It would be nice to have an evaluation for the contributions of each one of these individually, so we can understand their importance. From figure 6 I imagine that multi-modality is much more important, but we cannot asses the importance of the time-based selection. Thus, do we actually need both? Only multi-modal models are enough?\n\nI think the authors might have missed one of the greatest contribution of Section 4. The proposed loss is actually forms a tighter than the mean projection error, and as such, we can see the later as a relaxation of the former, and the former describes better the actual objective. Maybe the wording \"relaxation\" is misused here, but I hope the parallel with standard optimization problems makes it clear. It would be enriching to elaborate on that discussion, as this seems to be a contribution applicable to several other works as well, and hence a strong contribution.  \n\nNevertheless, the main limitation of the paper seems to be its application for trajectories that can be adequately  approximated by straight lines. This is not the case for quaternions, or joint states which are often in radians, the linear approximation in those cases can be mediocre at its best (this problem is probably mitigated by the learning method but we have no evaluation on that). These limitations can be addressed using higher degree polynomials (not only limited to splines as noted in line 117), or other regression methods, e.g., sine and cosine as basis functions for the regression, or even a NN. Of course, this is a topic for future works/journal extension.\n\nOn the results and analysis there are a few problems, which greatly hurt the claims of the paper:\n1. On table 1, and 3 we only have the success rate in %, whereas we need a more complete statistical analysis, and the variances. Like in Table 2.\n2. In table 2, the results for AWE + diffusion and diffusion lie within the variance of each other. As such we cannot conclude the  adding AWE makes things better as claimed by the authors. Please, restrict the claims to what the data shows. This also can make one doubt about the conclusions of tables 1 and 3.\n3. In figure 5, please present the results for more points in the x axis, only 3 points are not enough for concluding the claims and that the behavior of the curve is what we expect. \n4. Selecting a few points makes as the vector fields generated by the policy to have a \"coarser\" shape, basically what is happening is a non uniform weighting of the the velocity profiles of the model's vector fields. As such, we need to check the impact in the vector fields with and without AWE. This can provide insights or reveal problems which are hidden in the current state of the paper. Please add figures for the vector fields for a couple of cases, and FOR ALL cases in the complementary material.\n5. The link to the source code in the paper's website is broken. Please provide the code, data, and instructions for installing and running, so we can review and understand better what is happening.\n\nA final remark, in section 5.4, line 266, I do believe that to make the proposed approach complete, the authors should elaborate and study a method for defining the error threshold, making the contribution much more solid! In the state we are now, we do not even know how hard is to tune the only hyper-parameter in the method. Again, this is topic for future-works/journal extension.\n\n\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method for imitation learning, which approximates demonstrated trajectories with a sequence of waypoints connected by linear paths. This decomposition is achieved by manually choosing an error budget, and then calculating the shortest subsequence of states that reconstructs the original demonstration within this error budget. Both simulation and real-world results show that, when combined with Diffusion Policy and Action Chunking with Transformers, training on these approximated trajectories significantly improves performance.

Reviewer scores before the rebuttal were: 1 x “weak reject” and 2 x “weak accept”. Authors provided a rebuttal, and an updated paper with some minor clarifications and additions. Following the rebuttal, the reviewer scores remained the same. The AC then initiated a discussion with the reviewers, focussing on the novelty and generality of the proposed method.

All three reviewers engaged in this discussion. There was not a general consensus here. Some reviewers argued that, whilst the idea of decomposing trajectories into waypoints is not novel, the particular way it is done in this method is novel. Other reviewers were concerned that the novelty was not significant compared to other methods proposing decomposition into waypoints. On the issue of generality, some reviewers argued that whilst the method may not work on more dynamic tasks and whilst there are hyperparameters to tune, the performance on static tasks is sufficiently strong. Other reviewers were more concerned that the method may not work well for more complex tasks, particularly those with multi-modal demonstrations.

Overall, despite the lack of consensus, I have a general feeling that the reviewers were more positive than negative. Personally, I appreciate the simplicity of this method and its strong real-world results. I believe that conceptually, it is an easy idea to grasp and discuss, and that the community will therefore benefit from hearing about this work. For the camera-ready paper, please be sure that the clarifications added since the rebuttal remain in the paper, because it is currently over 8 pages.
Decision: Accept (Poster)