HYDRA: Hybrid Robot Actions for Imitation Learning
Authors: Suneel Belkhale
Yuchen Cui
Dorsa Sadigh
Venue: CoRL 2023 Poster (2023)
Keywords: Imitation Learning
Robotics
Manipulation

Abstract: Imitation Learning (IL) is a sample efficient paradigm for robot learning using expert demonstrations. However, policies learned through IL suffer from state distribution shift at test time, due to compounding errors in action prediction which lead to previously unseen states. Choosing an action representation for the policy that minimizes this distribution shift is critical in imitation learning. Prior work propose using temporal action abstractions to reduce compounding errors, but they often sacrifice policy dexterity or require domain-specific knowledge. To address these trade-offs, we introduce HYDRA, a method that leverages a hybrid action space with two levels of action abstractions: sparse high-level waypoints and dense low-level actions. HYDRA dynamically switches between action abstractions at test time to enable both coarse and fine-grained control of a robot. In addition, HYDRA employs action relabeling to increase the consistency of actions in the dataset, further reducing distribution shift. HYDRA outperforms prior imitation learning methods by 30-40% on seven challenging simulation and real world environments, involving long-horizon tasks in the real world like making coffee and toasting bread. Videos are found on our website: https://tinyurl.com/3mc6793z

HYDRA: Hybrid Robot Actions for Imitation Learning
Suneel Belkhale, Yuchen Cui, Dorsa Sadigh Stanford University, Stanford CA (belkhale, yuchenc, dorsa dorsa]]stannf (etasa)ssaafod.cco
Abstract: Imitation Learning (II.) is a sample efficient paradigm for robot learn- ing using expert demonstrations, However, policies learned through IL suffer from state distribution shift at test time, due to compounding errors in action prediction which lead to previously unseen states. Choosing an action representation for the policy that minimizes this distribution shift critical in imitation learning. Prior work propose using temporal action abstractions to reduce compounding errors, but they often sacrifice policy dexterity require domain-speciff knowl- edge. To address these trade-offs, we introduce HYDRA, method that leverages hybrid action space with two levels of action abstractions: sparse high-level waypoints and dense low-level actions. HYDRA dynamically switches between action abstractions at test time to enable both coarse and fine-grained control robot. In addition, HYDRA employs action relabeling to increase the con- sistency of actions in the dataset, further reducing distribution shift. HYDRA outperforms prior imitation learning methods by 30-40% seven challenging simulation and real world environments, involving long-horizon tasks in the real world like making coffee and toasting bread. Videos are found on our website: https: / /t /tt/tiinurrrrr// com 3ammm/3mm777333
1 Introduction
In recent years, supervised learning methods have made remarkable advancements in computer vi- sion (CV), natural language processing (NLP), and human-level game playing [1, 2, 3, 4, 5, 6, 7]. In robotics, imitation learning (IL) has emerged as a data-driven and sample efficient approach for programming robots using expert demonstrations. More specifically, behavioral cloning (BC) meth- ods treat II. as a supervised learning problem and directly train a policy to map states to actions. BC methods are often favored in practice for their simplicity but suffer from the well-known distribu- tion shift problem, where the test time state distribution deviates from the training state distribution, primarily caused by the accumulation of errors in action predictions [8, 9, 10].
Broadly, prior work has explored reducing distribution shift by interactively adding new data [9], incorporating large prior datasets [1 12], choosing better state representations (inputs) [13, 14], or altering model or loss structure [15, 16, 14]. A less explored but critical factor is the action representation (outputs): action prediction error partially stems from how difficult for the policy capture the expert demonstrated actions, action representations are a critical line of defense against distribution shift. Prior work studying action representations generally fall into two camps: (1) methods that use temporal abstractions to treat long action sequences as a single action (i.e., reducing the effective task horizon) and thus reduce the potential for compounding errors, and (2) methods that make the action representation more expressive to minimize the single-step prediction error [17, 18, 16, 19, 15]. However, both approaches come with a number shortcomings.
Methods using temporal abstractions often come at the cost of either the dexterity of the robot or the generality to new settings. One prior approach is for the robot to follow waypoints that cover multiple time steps [17, 14]; however, waypoints alone
reactive enough for dynamic, dexterous action sequences (e.g., inserting coffee pod). Other works use structured movement primitives that can capture more dynamic behaviors like skewering food items or helping a person
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

get dressed [20, 18, 21], but relying on pre-defined primitives often sacrifice generalizability to new settings (e.g,, new primitives beyond skewering for food manipulation). Today, we lack temporal abstractions that reduce distribution shift without losing policy dexterity and generality.
Other methods design each action be more expressive to capture the multi-modality present in human behavior [19, 15, 16]; however, these expressive action spaces often lead to overfitting, high training time, or complex learning objectives. Rather than making the policy more expressive, more robust approach is to make the actions in the dataset more consistent at given state and easier to learn (e.g., showing one consistent way to insert a coffee pod rather than many conflicting ways). Prior work shows that more action consistency (e.g., consistent human demonstrators) with sufficient state coverage lead to better policies [19, 14, 22], likely by reducing online policy errors [23].
To enable both better temporal abstraction and more consistent actions in the dataset, our key insight is to leverage the fact that most robotics tasks are hierarchical in nature - they can be divided into two distinct modes of behaviors: reaching high-level waypoints (e.g.. free-space motion) or fine--raiin -ine-grained manipulation (e.g., object interaction). Then, we can learn a policy that dynamically switches between these modes - this is in fact similar to models of human decision making, where is widely believed that humans can discover action abstractions and switch between them during a task [24, 25]. Capturing both waypoints and fine-grained actions enables us to compress action sequences (i.e., reduce distribution shift) without sacrificing the dynamic parts of the task, thus maintaining dexterity. In practice, this abstraction is general enough to represent most tasks in robot manipulation. Another notable advantage of partitioning tasks into two modes is that, during the waypoint reaching phase, we can relabel our actions with more consistent waypoint-following behaviors, thus increasing action consistency in the dataset.
Leveraging this insight, we propose HYDRA, a method that dynamically switches between two ac- tion representations: sparse waypoint actions for free-space linear motions and dense, single-step delta actions for contact-rich manipulation. HYDRA learns to switch between these action modes with human-labeled modes, which are provided after or during data collection with minimal ad- ditional effort. In addition, HYDRA relabels low-level actions in the dataset during the waypoint periods where the robot is moving in free space (e.g., when reaching a coffee pod) to follow consis- tent paths. These consistent actions simplify policy learning, which reduces action prediction error in the dataset overall and thus reduces distribution shift. HYDRA outperforms baseline imitation learning approaches across seven challenging, long-horizon manipulation tasks spanning both sim- ulation and the real world. In addition, it is able to perform complex coffee making task involving many high precision stages with 80% success, 4x the performance of the best baseline, BC-RNN.
2 Related Work
Data Curation: Several prior works aim to curate data based on some notion of data quality. in order to reduce distribution shift. Most works define quality as the state diversity present in a dataset, To increase state diversity, Ross et al. [9] proposed interactively collect on-policy demonstration data, but this requires experts to label actions for newly visited states. To reduce expert supervision, some methods use interventions to relabel on-policy data, where interventions can be automatically
or human generated [26, 27, 28, 29, 22, 30]. Laskey et al. [31] inject noise during data collection to increase state diversity to achieve similar performance as interactive methods. Recent work has sought to formalize a broader notion of data quality beyond just state diversity [23]. HYDRA takes this broader definition into account, increasing data quality through action consistency.
Model and State Priors: Rather than changing the data, many prior works build in structure to the model itself to address distribution shift. Object-centric state representations have been shown to make policies more generalizable [13]. Similarly, pretrained state representations trained on multi- task data have been shown to improve sample efficiency and robustness [12, 32]. Adding structure into the model itself, for example using implicit representations or diffusion-based policies, has also been shown to improve performance [16, 15]. The changes in HYDRA affect the action space and thus are compatible with many of these prior approaches.
Action Representations: Another approach is change the action representation to reduce com- pounding errors. One category of prior works leverage temporal action abstractions to reduce the

---

number of policy steps. Several works have learned skills from demonstrations, usually requiring lots of data but struggling to generalize [33, 34, 14]. Others use parameterized action primitives or motion primitives, but despite being more sample efficient, these often require privileged state information or are not general enough for scenes [20, 18, 21J. Waypoint action spaces have also been shown be a sample efficient temporal abstraction; however, they fail capture dynamic and dexterous tasks in the environment [35, 16, 36]. Sparse waypoint labeling has also been shown to be easier for people than action labeling [36]. For more dexterity, Johns [37] proposes Coarse-to- Fine Imitation Learning by modeling a single demonstrated trajectory as two parts: an approaching trajectory followed by an interaction trajectory. Di Palo and Johns [38] extend this to allow multi- stage tasks, but this approach still relies on fully known and fixed "stages" of the task along with labeled bottleneck states. Also, due to the use of self-replay during interactions, these approaches would struggle with scene variation or generalization. HYDRA builds on these works, combining waypoints and low-level actions into one fully learned model reduce compounding errors without losing dexterity generality. Another category of works seek to increase the expressivity of single action to reduce action prediction error, for example with Gaussian mixture models or energy mod- els [19, 15, 16]. However, increasing expressivity often leads to overfitting, more complex learning objectives, and increased training and evaluation time. Instead of increasing expressivity, HYDRA takes a more robust approach by increasing action consistency in the data. Prior work shows the importance of consistent actions for minimizing distribution shift [19, 233. HYDRA relabels actions in the dataset after data collection to increase consistency.
3 Preliminaries
Imitation learning (IL) assumes access to dataset D ( Ja of n expert demonstra- tions. Each demonstration тr is sequence of observation-action pairs of length N₁, ti (10,,a))..... (ON,, (ON,,,N;)), aN.)), with observations o E O and actions a E A. O often consists of robot proprioceptive data such as end effector poses and gripper widths, denoted sp E P. as well as envi- ronment observations such as images or object poses, denoted se E E, such that O = P & E. The true state of the environment is s E S. In robotics, the action space usually consists of either torque, velocity, or position commands for the robot. While velocity actions are most common, prior works also use position actions in the form of target waypoints [14, 35). The IL objective is to learn policy тe. O A mapping from observations to actions via the supervised loss:
(0) -E(o,a)-pp [log Th (alo)]
At test time, the learned policy Ty is rolled out under environment dynamics / S x A + S. Per step, we observe og, sample an action â, T--(o)), and obtain the next state sc+1 = f(s, āz).
Distribution Shift in II, A fundamental challenge with imitation learning is state distribution shift between training and test time. Considering training sample (...o,, ae, 0141...) if the learned policy outputs ä, ~ <-1oo)), which has a small action error é, = ãe - de, the next state following this action will also deviate: 5 f(s,, ar + +r), which in turn affects the policy output at the next step. For real world dynamics, this change in next state be highly disproportionate [|</||- For example in the coffee task in Fig. 1, with a slight change in gripper position (small ex) the policy can misgrasp the coffee pod (large change in $1+1 and 0t+1). Furthermore, noise in the dynamics f can lead to even larger changes in 0 As we continue to execute for the next N - t steps, this divergence from the training distribution can compound, often leading to task failure.
Therefore, reducing distribution shift requires reducing é, for all I. E (1,..., N} or increasing the coverage of states st. One approach to reduce policy error is increasing action consistency, which prior work defines as lowering the entropy of the expert policy Fg at each state: He (a)s) [23]. However, there is a trade-off between state coverage and action consistency during data collection, since less consistent actions often lead to more diverse states [23, 19]. HYDRA reduces distribution shift by using temporal abstraction for the action space which shortens the number of policy steps N and thus reduces compounding errors and by improving action consistency in offline data which reduces ||ee|] without reducing state coverage.
3

---

HYDRA
Policy
Hybral
Actions
Sparse Net
st
aypoiin
Mixde Fload
haing T
Denue Net
Action
action
Dunse Action
h2
Minlt-hossdei architectary
Action chosm hssed mesir
Multi hulli-headed architecture HYDRA: During training, we learn predict waypoints, low level actions, and the mode label for each time step. One network (Dense Net) predicts the low level action a, and the mode m.: both the action and mode heads of Dense Net share an intermediate representation network (Sparse Net) predicts the high level waypoint wt, At test time, sample my and either servo to reach a waypoint (m, 0) without requerying the policy, or follow dense action for one time step (m, 1). An example of how sparse and dense modes can be arbitrarily stitched together at test time is shown on the right.
Figure I:
A separate
HYDRA: A Hybrid Action Representation
To reduce distribution shift, our insight that most robot manipulation tasks are a combination of sparse waypoint-reaching, such reaching for an object lifting a mug towards shelf, and dense low-level actions, such grasping object or balancing mug stably on shelf. Waypoints capture free-space motions but struggle to capture dexterous or precise behaviors. Conversely, low-level actions capture these dynamic behaviors but are often redundant during long free-space motions.
Instead of learning from only velocities or waypoints, HYDRA learns a hybrid action representation consisting of both high-level waypoints in the robot's proprioceptive space w E P and low-level ac- tions E A. Additionally, we learn dynamically switch between these modes by predicting which mode m {0, 1}, sparse or dense, should be executed at each demonstrated state. Mode labels annotated with little extra cost by experts either during or after data collection. This flexible ab- straction leads to (I) a compressed action space that reduces compounding errors without sacrificing dexterity or generality, and (2) a more consistent, simple low-level action distribution through action relabeling during the sparse periods. This section presents an overview of the approach, followed by discussions on mode labeling, action relabeling, and training/testing procedures.
Overview: The multi- architecture of HYDRA is outlined in Fig. 1, with heads TT! :0 (0, 1). T 0 + A, T/ O + P. for mode, action, and waypoint respectively. One network, Dense Net, predicts the low-level action a and the mode my at each input Or = s Another network, Sparse Net, separately outputs the desired future waypoint wi for input ot. We assume waypoints can be reached using a known controller and desired waypoint into a low-level action (e.g. linear controller, see the right side of Fig. 1). In practice, Dense Net is recurrent since both the mode and action are highly history-dependent, Sparse Net in contrast only uses the current observation, since waypoints less dependent than actions. Then at test time, HYDRA predicts the mode m, and follows the controller T until reaching the waypoint during predicted sparse periods, and follows low-level actions at each step during predicted dense periods. See Appendix C for more details.
O + A which converts the state
and history
4.1 Data Processing: Mode Labeling and Action Consistency
To dynamically switch action abstractions, we need labeled modes my waypoints wy, and actions a, each time step. We first obtain binary mode labels m, from humans, and then use the mode labels to extract waypoints and to relabel low-level actions. Importantly, modes can be labeled either during demonstration collection (e.g. with a simple button interface), or entirely after demonstra- tion collection (c.g., labeling each frame with its mode). With modes labeled, we can segment each demonstration into sparse waypoint and dense action phases. We provide the details of the labeling and segmentation process in Appendix B. For each sparse phase, we can extract the desired future waypoint w, at o if m, = 0 (sparse), the future waypoint is final proprioceptive state w). = Pe in that sparse segment, where t' t. But if m, = (dense), the waypoint is the next propriocep- tive state wt Pt+1- This yields dataset of D of (o, a, w. m) tuples. Now the policy has full supervision to learn the modes, waypoints, and actions.

---

Mode Labeling Strategy: Since waypoints will be reached online with controller T, the main requirement for laheling modes is that during sparse phases (m, = 0), the labeled waypoint we should be reachable via T starting from or (i.e., without collision): for example, if the demonstrator starts in free space and labels a waypoint close to coffee K-pod, and if the policy uses linear P- controller as 'T, then the K-pod waypoint should be reachable from the initial pose in straight-line path. Otherwise, the learned policy might collide when it tries to reach similar waypoints. We do not assume access to a collision-avoidaaee planner as T in this work, but if one has access to a planner then T can always reach the desired waypoint, so this reachability requirement can be ignored. Other considerations for mode labeling and a discussion of mode sensitivity is provided in Appendix B. We specifically show that our method is not overly sensitive to mode labeling strategies outside of the collision-free requirement above. Furthermore, we show that mode labels can be learned from substantially fewer examples without a major effect on performance Appendix D.3.
Relabeling Low-Level Actions: As discussed in Section 3, action consistency can improve policy performance by simplifying the BC objective in Eq. (1) and thus reducing ||eall. provided the data has enough state coverage. However, making actions consistent during data collection is challenging and can often reduce state coverage [22], so instead HYDRA performs offline action relabeling, i.e., after collection. To relabel human actions ag during the sparse periods, HYDRA uses waypoint controller T recompute a new action each demonstrated robot state s. based on the waypoint wt. We lack consistent relabeling strategy for dense periods, we leave this to future work.
However, a subtle challenge with offline relabeling is that changing the actions in the data can put the policy out of distribution at test time, since new actions can lead to new states online. For example, an are path was demonstrated to get to a waypoint, but linear controller is used for relabeling, the linear action will take us off that path. HYDRA avoids this problem by using a waypoint controller T online during sparse periods, meaning relabeled actions will not be deployed online. Rather, this action relabeling serves primarily to simplify the dense action learning objective of HYDRA and increase action consistency in the overall dataset.
natural question arises: since sparse actions will be executed with T online, could we instead further simplify learning by avoiding training on dense actions during sparse periods? If HYDRA mispredicts a sparse mode dense, then the dense actions will still be executed online, so HYDRA should still be trained on dense actions during sparse periods as back-up. We show that reducing the training weight of dense actions during sparse periods hurts performance in Appendix D.5.
4.2 Training and Evaluation
Training: HYDRA is trained to both imitate low-level actions with policy Tg'. high-level way- points w with T and the mode m with xMM at each time step. To balance the waypoint and action losses, we use a mode-specific loss at each time step that weighs the current mode's 's loss with (1 (1-7),
and the other mode's loss with y. Given a processed dataset DD consisting of tuples of (o, a, w, m). we modify the loss in Eq. (1) with the new heads of HYDRA (mode, action, and waypoint):
(0) = -
[(1 am) log To (alo) + om log W (w]oo)
(2)
L. = B (,o,,,u,))ppp [m log T M
M (m 1o) + (1 m) log mg (m = 0|o)]
(3)
La weighs the BC loss for waypoints and actions by the current mode: am = my (1 - m) (1 7)
the mode-specifee weight for the sparse waypoint part of L. If we are in sparse mode (m 0), then Omm. 7. but in dense mode, On = Thus, low gamma encourages the model to fit the loss for the current mode more than the loss for the other mode, and 7 0.5 will be a mode-agnostic weighting. See Appendix D.5 for results of ablating 7. L. is the mode cross entropy classification loss. Combining these terms with mode loss weight B, we get the full HYDRA objective:
(4) Evaluation: During evaluation, the policy chooses the mode using my. If m. = O, the model will servo in a closed-loop fashion to the predicted waypoint vi using controller T. The policy is queried at every step to continually update the policy hidden state, but importantly its outputs are ignored until we reach the waypoint to avoid action prediction errors. If m 1, the model will execute just the next step using the predicted dense action ä,
[()) = L. (0) + (0)
5

---

ToolHong
KirckonEee
Pegfnscrtion C-
Pickup
Inse Fod
Close
Pickup Mag
Place Map
Press
time
Open
time
Spoen
Sponn
Grasp
Mag
Hong Maj
25
25
time
Figure 2: Simulation Real-world environments, with task stages shown for real world tasks. Simulation: In NutAssemblySquuee we pick up square nut at various positions and orientations and insert it onto vertical square peg. In ToolHang, hanging frame inserted onto fixed stand, followed by placing tool the frame. Both the frame and tool poses randomized. Frame insertion challenging due the small insertion area. KitchenEnv involves turning on a stove, moving pot onto stove, putting an object moving the pot to serving area. Real World: Peglusertion involves inserting peg with hole in the center onto round insertion rod (top right); the peg location and geometry are varied. MakeCoffee is 6-step task (top middle row) involving picking up K-pod, inserting into Keurig machine, closing the lid of the Keurig, positioning a mug. and then pressing start on the Keurig; the K-pod location and mug orientations are varied. Unlike prior work [13], we include a mug component. MakeToast has 7-steps (bottom middle row): a hinged toaster oven is opened, spatula is picked up, bread is placed in the toaster, the toaster is closed, and the dial is turned to start. Bread and spatula initial poses vary. SoriDishes (bottom row) has 6 stages: pick up spoon, place spoon rack, grasp plate and insert rack, and grasp mug and hang the mug. All objects poses vary.
pot, then
5 Experiments
We evaluate the performance of HYDRA in 3 challenging simulation environments and 4 complex real world tasks, shown in Fig. 2. These tasks cover wide range of affordances and levels of precision, from precisely inserting a coffee pod to picking up bread with a spatula. See Appendix C for model hyperparameters, data collection, and training details. Videos can be found on our website.
Data Collection: We leverage proficient human demonstration data for simulated tasks from robomimic [19]. Mode labels and waypoints were annotated offline for simulation datasets as de- scribed in Appendix B. Demonstrations for real world tasks were collected by proficient user using VR teleoperation using an Oculus Quest 2. Mode labels and waypoints were provided during data collection using the side button on the Quest VR controller with no added collection time. All methods share the same underlying dataset.
Simulation: In Fig. 3 (top row), we compare our method to BC and BC-RNN for the NulAssem- blySquare and ToolHang tasks (state-based), as well as the KitchenEnv task (vision-based) from robosuite (see top row in Fig. 2). Our method improves performance on the NutAssemblyyquare task, where basclines are already quite strong. We also ablate the data size from 200 demos to 100 and in Fig. 3, illustrating that HYDRA is more sample efficient than baselines, with the gap grow- ing as data size decreases. HYDRA-NR in Fig. 3 removes action relabeling and drops performance by 8%, which attribute high action multi-modality in non-relabeled sparse periods, but for 50 demos performs similarly, likely due to high action consistency just from having less data.

---

100
NutAssembiySquare
HYDRA HYDRA-NR
100
TootHang
100
KitchenEny
BC-RNN
VIOLA
80
84
82
75
80
80
78
60
66
60
40
50
40
-40
36
20
24
20
29
200 demos
100 demos
50 demos
200 demos
100 demos
Figure 3: Sim Results for HYDRA vs. BC. BC-RNN, and VIOLA: best checkpoint success rate averaged over three seeds. Left to Right: NutAssemblySquar (state), ToolHang (state), and KitchenEnv (vision) tasks. HYDRA beats baselines on all of these tasks, and even beats VIOLA [13] on the kitchen task despite using much smaller and simpler model. We also show comparison for BC-RNN and HYDRA with decreasing data sizes for MutAAssemlyyquue showing that our method is more sample efficient than BC-RNN. HYDRA without action relabeling (HYDRA-NR, NutAssemblySquare and ToolHang) drops performance by 7-8%.
For Tool Hang (top middle in Fig. 3), which is long horizon, consisting of many waypoint dense periods and requiring much higher precision, the gap is even bigger from HYDRA to BC and BC- RNN, While the best baseline gets 29%, our method reaches 63% with the same inputs. Again, no action relabeling (HYDRA-NR) drops performance by 7% but is still much better than baseline.
For KitchenEnv (vision-based)) we also compare to VIOLA [13], an image-based model that uses bounding box features and a large transformer architecture to predict actions. Once again, HYDRA is able to outperform BC-RNN by 35% on this long horizon task. HYDRA also outperforms VIOLA by 9%, despite using a simpler and smaller model.
In Appendix D, we discuss HYDRA-NR results, show waypoint-only baseline, mode labeling ablations, and a relabeling-oni only ablation where action consistency is improved but the waypoint controller is not used online. In Appendix D.3, we show that mode labels can be learned with fewer examples without large performance drops (using 25% of mode labels drops performance by 10%).
Real World: In Fig. 4, we compare our method to BC-RNN (vision-based) for four high precision tasks: Peginsertion, MafeCoffee, Toast, and SortDishes. The latter three are long-horizon, and Fig. 4 shows cumulative success per task stage. In PegInsertion, our method substantially outper- forms BC-RNN at both peg grasping and precise insertion portions of the task, thanks combining precise waypoints with flexible low level actions where necessary.
For MakeCoffee, HYDRA once again beats BC-RNN and VIOLA by substantial margin at all task stages. All methods perform well in grasping the K K-pod, but the performance the basclines declines rapidly in the following phases. While BC-RNN failed to do this task in prior work, we see with a bit of parameter tuning, BC-RNN is strong baseline, achieving 20% performance [13]. VIOLA's reported performance in prior work for pod insertion and closing the lid is 60%, matching what we observe for the corresponding stage of our coffce task. Our task adds two more stages (picking up and placing mug before pressing the button), interestingly causing the final success rate of VIOLA to drop to 20%, the same as BC-RNN. Using the same parameters and model size as BC-RNN, HYDRA achieves 80% final success at this task with the same underlying dataset.
For Toast and SortDishes, HYDRA performs better on all stages of the task as compared to BC-RNN. We omit VIOLA in these tasks since, seen in the coffee task, BC-RNN is competitive baseline (see Appendix A). Both tasks consists of several bottleneck stages where performance drops sharply. In MakeTTass for picking up bread, the spatula must slide underneath a bread slice HYDRA passes this stage 70% of the time, beating BC-RNN by 30%. The last stage (turning the toaster on) is particularly challenging for all methods (precise dial grasp. partial observability), but HYDRA completes it 20% of the time compared to 0% for BC-RNN. SortDishes, the final hang-mug stage similarly requires high precision (small space between rack and mug hang point, partial observability). Not including the challenging last stage, HYDRA beats BC-RNN by 40% on this task. See Appendix D.1 for rollouts of each task for each model.
7

---

Peginsertion
MakeCoffce
BC-RNN VIOLA
HYDRA
100
901
900
710
56
40
40
10
30
20
20
20
pick peg
inseett (full)
pick up pod
Insert pod
classe lid
pickup mag
place mag
prese button (full)
MakeToast
SortDishes
90
100
80
60
60
40
40
20
30
10
10
0
10
open
pick
pick
place
place
dlose towsker
turn cm (full)
pick spoon
place
insert
pick mug hang mug (full)
loaster
spotula
bread
bozad
spotuala
spoon
plate
plate
Figure 4: Real Results for HYDRA vs. BC, BC-RNN, and VIOLA. The x-axis denotes each stage (right-most value is the final success rate). Top Left: HYDRA vs. BC-RNN on the real PegInsertion task for 50 demos under 32 rollouts across 4 different nuts. This task requires very precise grasping and insertion of multiple types of nuts, which 41% success. Top Right: MakeCoffee long-horie ong-horiizon task for 100 demos under rollouts. Our method beats baseline by 60%. Bottom Left: MakeToaat long-horizon task for 100 demos under 10 rollouts. While both methods struggle to turn the toaster on, HYDRA is able to reach 50% success for 6/7 stages compared to 10% for baseline. Bottom Right: SortDishes for 100 demos under 10 rollouts. Waypoints HYDRA precisely capture diverse poses in this task, beating BC-RNN by 40% and 20%
method does with high success. While baseline is unable perform insertion, HYDRA gets
last two stages.
We observe that the performance gain for HYDRA in our real world experiments is notably higher than simulation. We hypothesize this is due to (1) higher variance in action playback on the real robot setup, which HYDRA mitigates during sparse periods using the closed-loop waypoint controller, and (2) increased potential for compounding errors in longer tasks. Overall, HYDRA is well-suited to long horizon tasks even with many high-precision bottleneck stages, due its ability to switch between waypoints and dense actions and its ability increase action consistency offline. We also observed that in our real world tasks, HYDRA exhibits emergent retrying behavior, often re- re-servoing to a consistent and in-distribution waypoint to retry failed dense period.
Discussion
Summary: In this work, we propose HYDRA, which uses a flexible action abstraction to reduce compounding errors, and improves action consistency while maintaining the state diversity present uncurated human demonstrations, HYDRA learns dynamically switch between following way- points and taking low level actions with small amount of added mode label supervision that can be provided either online offline. HYDRA substantially outperforms baselines three simulation tasks and four real world tasks that involve long horizon manipulation with many bottleneck states.
Limitations Future Work: While only minor amount of added supervision, HYDRA relies on having expert-collected mode labels. We show that mode labels can be learned from much less data in Appendix D.3. but future work might consider using unsupervised methods for mode labeling, e.g. skill segmentation [39] automatically extracting "linear" portions of demonstration. We also hypothesize multi-task datasets can help learn general mode-predicter that can be fine-tuned or deployed zero-shot novel tasks. Furthermore, when mode labels are collected online, mode labeling can add a mental load for the demonstrator and might also influence the quality of the data on its own. Future work might conduct more extensive user studies to better understand the effect of providing mode labels both demonstrator and the final learning performance.
Despite these limitations, HYDRA is simple and easy-to-implement method, and it is exciting that it shows substantial improvement over state-of-ther -he-hrt imitation learning techniques and significant promise in solving challenging manipulation tasks in the real world.

---

Acknowledgments
This research in advancing imitation learning supported by ONR, DARPA YFA, Ford, and NSF Awards #1941722, #2218760, and #2125511.
References
[1] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, Zhai, Unterthiner, M. De- hghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transform- ers for image recognition at scale, In International Conference Learning Representations, 2020.
[2] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 16000-16009, 2022.
[3] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, Agarwal, G. Sastry, Askell, Mishkin, J. Clark, al. Learning transferable visual models from natural language supervi- sion. In International conference machine learning, pages 8748-878877 8763. PMLR, 2021.
[4] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[5] R. Thoppilan, De Freitas, Hall, N. Shazeer, A. Kulshreshtha, H.-T. Cheng, Jin, Bos, L. Baker, Du, al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239. 2022.
[6] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 2022.
[7] S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y. Sulsky, Kay, J. T. Springenberg, et al. generalist agent. arXiv preprint arXiv:2205.06/75, 2022.
[8] Ross and D. Bagnell. Efficient reductions for imitation learning. In Proceedings of the thir- teenth international conference artificial intelligence and statisties, pages 661-666 JMLR Workshop and Conference Proceedings, 2010.
[9] S. Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured predic- tion to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Pro- ceedings, 2011.
[10] J. Spencer, S. Choudhury, A. Venkatraman, B. Ziebart, and A. Bagnell. Feedback in imitation learning: The three regimes of covariate shift. arXiv preprint arXiv:2 102.0222 2021.
[11] Driess, F. Xia, M. S. M. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, W. Huang, Y. Chebotar, P. Sermanet, D. Duckworth, S. Levine, V. Vanhoucke, K. Hausman, M. Toussaint, K. Greff, A. Zeng, I. Mordatch, and P. Florence. Palm-e: An embodied multimodal language model. In arXiv preprint arXiv:2303.03378 2023.
[12] Karamcheti, S. Nair, Chen, Kollar, C. Finn, D. Sadigh, and Liang. Language- driven representation learning for robotics. 2023.
[13] Y. Zhu, A. Joshi, P. Stone, and Y. Zhu. Viola: Imitation learning for vision- manipulation with object proposal priors. 6th Annual Conference Robot Learning (CoRL), 2022.

---

[14] S. Belkhale and D. Sadigh. Plato: Predicting latent affordances through object-centric play. In 6th Annual Conference on Robot Learning, 2022.
[15] P. Florence, C. Lynch, A. Zeng, O. A. Ramirez, A. Wahid, L. Downs, A. Wong, J. Lee, I. Mor- datch, and J. Tompson. Implicit behavioral cloning. In Conference on Robot Learning, pages 158-168. PMLR, 2022.
[16] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137, 2023.
[17] M. Shridhar, L. Manuelli, and D. Fox. Perceiver-actor: multi-task transformer for robotic manipulation. In 6th Annual Conference on Robot Learning, 2022.
[18] R. P. Joshi, N. Koganti, and T. Shibata. A framework for robotic clothing assistance by imita- tion learning. Advanced Robotics, 33(22): 156-1174, 2019.
[19] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang. R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martin-Martin. What matters learning from offline human demonstrations for robot manipulation. arXiv preprint 1ix::210880039966 2021.
[20] P. Sundaresan, S. Belkhale, and D. Sadigh. Learning visuo-haptic skewering strategies for robot-assisted feeding. In 6th Annual Conference on Robot Learning, 2022. URL https:
/oppnrevieww net wtttttrrur// forummmii wemmmdddd11g9g@aaa
[21] J. Silvério, Y. Huang, F. J. Abu-Dakka, L. Rozo, and D. G. Caldwell. Uncertainty-aware imitation learning using kernelized movement primitives. In 2019 IEEF/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 90-97. IEEE, 2019.
[22] K. Gandhi, S. Karamcheti, M. Liao, and D. Sadigh. Eliciting compatible demonstrations for multi-human imitation learning. In Proceedings of the 6th Conference Robot Learning (CoRL), 2022.
[23] S. Belkhale, Y. Cui, and D. Sadigh. Data quality in imitation learning, 2023.
[24] L. L. Richmond and M. Zacks. Constructing experience: Event models from perception to action. Trends in Cognitive Sciences, 21(12))962-980, 2017. ISSN 1364-6613, doi:https:
//bb..or/11..1006f...ss...00 2017...............00 URL https: www.wwienced.........w.t
con/ sc
[25] C. Baldassano, J. Chen, A. Zadbood, J. Pillow, U. Hasson, and K. Norman. Discovering event structure in continuous narrative perception and memory, 2016.
[26] Y. Cui, D. Isele, S. Nickum, and K. Fujimura. Uncertainty- data aggregation for deep imitation learning. In 2019 International Conference on Robotics and Automation (ICRA), pages 761-767. IEEE, 2019.
[27] R. Hoque, A. Balakrishna, E. Novoseller, A. Wilcox, D. S. Brown, and K. Goldberg. Thriftydagger: Budget-aware novelty and risk gating for interactive imitation learning. In Conference on Robot Learning, pages 598-608. PMLR, 2022.
[28] M. Kelly, C. Sidrane, K. Criggs-aampbell, and M. J. Kochenderfer. Hg-dagger: Interactive imitation learning with human experts. In 2019 International Conference on Robotics and Automation (ICRA), pages 8077-8083. IEEE, 2019.
[29] A. Mandlekar, D. Xu, R. Martín-Martín, Y. Zhu, L. Fei-Fei, and S. Savarese. Human-in-the- loop imitation learning using remote teleoperation. arXiv preprint arXiv:2012.06733 2020.
[30] M. L. Schrum, E. Hedlund-Botth N. Moorman, and M. C. Gombolay. Mind meld: Personal- ized meta-learning for robot-centric imitation learning. In 2022 17th ACM/IEE International Conference Human-Ron Interaction (HRI), pages 157-165. IEEE, 2022.
10

---

[31] M. Laskey, J. Lee, R. Fox, A. Dragan, and K. Goldberg. Dart: Noise injection for robust imitation learning. In Conference on robot learning, pages 143-156. PMLR, 2017.
[32] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3m: A universal visual repre- sentation for robot manipulation. In 6th Anmual Conference Robot Learning, 2022. URL https: /tttps:/cppeneveeie
net forum? id= het:EoormmiddttCcpppg 6yOrI.
[33] K. Pertsch, Y. Lee, and J. J. Lim, Accelerating reinforcement learning with learned skill priors. In Conference on Robot Learning (CoRL), 2020.
[34] C. Lynch, M. Khansari, T. Xiao, V. Kumar, J. Tompson, S. Levine, and P. Sermanet. Learning latent plans from play. Conference on robot learning, pages 1113-1132. PMLR, 2020.
[35] M. Shridhar, L. Manuelli, and D. Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In 6th Annual Conference on Robot Learning, 2022. URL btttps://opente view. www. forum? demmmidd=pa seiiid_seeC___ _CVV.
[36] B. Akgun, M. Cakmak, K. Jiang, and A. L. Thomaz. Keyframe-based learning from demon- stration: Method and evaluation. International Journal of Social Robotics, 4:343-355, 2012.
[37] E. Johns. Coarse-to-fine imitation learning: Robot manipulation from a single demonstration. In 2021 IEEE international conference on robotics and automation (ICRA), pages 4613-4613 IEEE, 2021.
[38] N. Di Palo and Johns. Learning multi-stage tasks with one demonstration via self-replay. In Conference on Robot Learning, pages 1 1180-1189. PMLR, 2022.
[39] T. Kipf, Y. Li, H. Dai, V. Zambaldi, A. Sanzaez-Gonzalez, E. Grefenstette, P. Kohli, and Battaglia. ComplLE: Compositional imitation learning and execution. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Ma- chine Learning, volume of Proceedings of Machine Learning Research, pages 3418-3428. PMLR, 09-15 Jun 2019. URL https: http::://proceedinnge mlr wttp://s.ww..ppp...pe....m wttppp/eee/s//ellll../ html.
[40] M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. To- bin, O. Pieter Abbeel, and Zaremba. Hindsight experience replay. Advances neural information processing systems, 30, 2017.
[41] L. Zhang and B. C. Stadie. Understanding hindsight goal relabeling requires rethinking diver- gence minimization. In Deep Reinforcement Learning Workshop NeurIPS 2022.
[42] Y. Lin, A. S. Wang, G. Sutanto, A. Rai, and F. Meier. Polymetis. https: facebookre search. search.github. wawaar/poolomaaeaaao 2022.

---

We provide broader discussion of our method in this appendix. In Appendix A we list a set of motivating questions that may arise during reading the main text of this work and provide our re- sponse with links to additional details in corresponding sections in the Appendix. In Appendix B we discuss how to collect mode labels, and considerations for how to define waypoint and dense seg- ments. In Appendix C, we outline training procedures, model architectures, and hyperparameters, In Appendix D, provide ablation experiments for our method, including sensitivity to mode la- bels, learning mode labels from less data, ablations to 7, and robustness of HYDRA to added system noise.
A Motivating Questions
Intuitively, why does HYDRA help improve BC? Humans demonstrate manipulation tasks at an abstraction level that is different from how the robot interprets the data. A BC agent interprets the data literally as taking a specific action at an exact state while the human is noisely reaching for an object. At the high level, HYDRA improves BC by realigning the task abstraction of the robot to the human demonstrator during waypoint mode of the task. Concretely, HYDRA curates the dataset in way that improves action consistency and optimality without reducing state diversity and hence allowing the learned policy to stay closer in distribution at test time.
What's the relationship of HYDRA with works in hindsight relabeling? Hindsight relabel- ing [40] is the idea of relaheling past experiences of goal-reaching trajectories with the final state it reaches to reuse any sub-optimal data (especially for reinforcement learning settings). Recent work of Zhang and Stadie [41] draws the connection between condiiondt imitation learning and hindsight relaheling from a fivergence-minimization perspective. The current implementation of HYDRA operates in single-task imitation learning setting, and therefore only remotely related to the idea in hindsight relabeling. From this perspective, one can think of HYDRA as effectively re- ducing divergence of the dataset's action distribution by relabeling actions for the waypoint periods of the trajectory.
Does mode labeling during collection change demonstrator behavior? We explain the mode labeling process during collection in Appendix B. We acknowledge that asking the demonstrator to provide mode labels during data collection adds additional cognitive load during demonstrating the task, and at the same time may change their demonstration behavior. In practice, asking the demonstrator to provide the two mode labels can communicate the structure the robot leverages to learn tasks and may in turn allow the human to provide better demonstrations (such as consistent waypoints etc.). However, we leave this user study to future work.
How sensitive is HYDRA to mode labeling? In our experiments, we (experts in this task) provided the mode labels for different tasks. We found HYDRA to be robust to the labeling strategies across the two labelers. For simulated environments, we use existing datasets and labeled the modes using an interface that shows the robot view of the task and the human annotator marks whether a frame is waypoint or dense mode. For real robot tasks, the human demonstrator provides the mode labels as they provide the demonstration using button on the telcoperation controller. We provide guidelines for how to perform mode labeling in Appendix B.
How were baselines chosen? BC was not included in Kitchen or the real world tasks since was substantially lower performing than BC-RNN, both in our state-based simulated results and in some initial testing on visual domains. For VIOLA, noted in the text, BC-RNN was shown to be strong baseline in the MakeCoffee task, nearly matching VIOLA's performance despite using a much smaller network architecture. Secondly, VIOLA takes longer to both train and perform inference due to the use of a region proposal network and a large transformer architecture. Thirdly, VIOLA changes the input and encoding structure on top of an approach like BC-RNN, whereas HYDRA changes the output. Thus these methods are actually compatible with each other, and the point of including VIOLA was not as a primary baseline but to show that designing intelligent *output action spaces can be as or more beneficial than large changes in the imput features or encoding structure. This is top of the fact that HYDRA also uses substantially fewer parameters than VIOLA, and thus BC-RNN is more directly comparable approach. Given these limitations,
12

---

and our strong BC-RNN baselines, we argue the addition of VIOLA does not provide any essential signal about HYDRA.
B Labeling Modes in HYDRA
B.1 Providing Mode Labels
The primary assumption made in HYDRA the availability mode labels for sparse and dense periods. Here we provide discussion of how mode labels be collected via simple binary "click" interface, either during demonstration collection after collection. In either case, label dense periods and exact waypoints using single binary "click" variable via an external button: to label waypoint at the end of a sparse period, we provide a single click at the waypoint state; to label a dense period, we sustain the click until the end of the dense period (see left image in Fig. 5). Once clicks are labeled, we demarcate periods in between clicks as sparse modes, and periods with sustained clicks as dense modes (see right image in Fig. 5).
Uncurated demonstration
Relabeled demonstration
Po
P
a
P
P
sparse
click
new a
click-hold
dense
Figure 5: Mode labeling example for peg-insertion task. each human labels binary click signals each time step (labeled during or after collection) to segment trajectories into arbitrary sequences of sparse waypoint phases and dense action phases. Left: Uncurated demo, with single clicks and sustained clicks shown. Right: Relabeled demo, with waypoint and dense segments overlayed in green and orange, respectively. We also relabel actions for the states in sparse segments with the optimal waypoint reaching action shown in white. sparse segments, waypoint head of HYDRA is trained to output the final waypoint at each state along the trajectory.
With the trajectories segmented into sparse and dense modes, we can extract the desired future waypoint We for each 0₁: if m = 0 (sparse), the future waypoint is the next labeled "single click" proprioceptive state wy = Pt where l' > (for example, states o, with ( S l2 in Fig. will use w, = Pt.). But m, (dense), the waypoint is the next proprioceptive state w; Pc+1- Thus construct dataset of D of (ot, de, W1, m²) tuples. Now the policy has full supervision learn both the action and waypoint as well as the mode of operation. In Algorithm 1, we outline this process of turning click-labeled dataset into per-step waypoints and mode labels.
B.2 Waypoint Controller
For all experiments in the main text, a linear controller Tinear for reaching waypoints online. This means that when HYDRA predicts waypoint period (m; 0), will servo closed loop until
13

---

Algorithm Labeling Modes
Given click-labeled dataset: D {or, a2 C....
2: D=0 =0
3: for ao
4: m, = & (ct-1 c(+1)
Sustained click for dense
5: Mark single click waypoint
6:
isolated = -4-1
Ct & G4+1
7:
// Mark start of dense period waypoint
8:
start deese- = -c-1 & my
9:
if isolated start dense then
10:
Wtpit-1
D Set previous waypoints start of next sparse phase
11:
bp
12:
else if m = then
13:
// During dense mode the next state is waypoint
14:
Wi = Pr+1
Ep
16: Add (O,, as, wi m.) to
reaches the predicted w; or times out after seconds. In all of our experiments, the waypoint follower times out after N seconds if it has not reached the waypoint.
For this closed loop servoing during test time, the policy will still be called, but its outputs will be ignored. This is important for recurrent models specifically (e.g., Dense Net), since the hidden state for the policy should be updated similarly to how it was trained (on all states, even during the sparse period). While this mitigates the changes in the hidden state, this might still induce different hidden state than was produced offline, since the human policy followed a non-optimal path to reach waypoint w from state 8t, as compared to the optimal online trajectory generated by T. For example, if the demonstrator follows an arc-like trajectory to pick up coffee pod and marks the waypoint right before picking up the coffee pod, then online the policy with Timear will servo that waypoint directly; the hidden state for these two paths will likely be different. This problem diffieult to observe in practice, and did not empirically show up in practice (as evidenced by the improved performance of our method compared to baselines).
In theory, one could bypass this issue by "skipping" the hidden state of the policy over entire sparse segments during training. Then during test time, if the policy outputs m, O, the policy would not be called again until reaching the output wt However, this requires loading entire sparse segments and more in the training batches, which is computationally expensive and less simple then loading batches of fixed horizon as is commonly done. We leave a broader analysis of the hidden state problem for future work.
Additionally, we experimented with several controller gains and did not notice any effect on perfor- mance. Therefore we choose a fast controller to reach waypoints. These gains are constant for all experiments.
For all experiments, we use Polymetis for control [42], with 10Hz policy frequency and IkHz con- troller frequency. We use a Hybrid Joint Impedance controller to follow end effector waypoint and velocity commands, but our method is not tied to any one underlying controller. End effector pose interpolation follows the shortest position and orientation path SO(3).
B.3 Mode Labeling Sensitivity
In our experiments, we noticed that mode labeling was quite robust to different labeling strategies provided that the labeling strategy satisfies the following guidelines.
Waypoint Following Behaviors: Waypoint following behaviors should be labeled for free-space motions in the environment, when the robot is "in transit" (e.g., reaching). As described in Sec- tion 4.1, key consideration for mode labeling making sure labels for sparse periods are compat- ible with the waypoint controller T. For example, if we are following linear controller, waypoint
14

---

segments should be reproducible with straight line segments from any start state along the waypoint segment. For a given (M, Wt, St), then if m, = 0, we should be able to reach waypoint w; from 8e with T (i.e. without timing out). As mentioned in the main text, if T includes collision avoidance as part of the controller, then we no longer have any requirements on waypoint following behaviors.
An important yet subtle point here is that even though linear controller requires that the path be- tween any two waypoints is collision free, collision. Since SparseNet in our framework is visually aware policy that is learned on each transi- tion in the data (i.e. closed loop policy), it can learn to sequence together multiple waypoints in a row without any additional cost. We show that multiple waypoints can be sequenced in row in Appendix D.2, and several of our experiments involve "pre-grasp" waypoints to avoid collision in the scene (e.g. NutAssemblySquare to avoid collision between the nut and the peg).
always arbitrarily add more waypoints to avoid
Dense Object Interaction: Dense periods should include (but is not limited to) all object interac- tions in the scene where "collision" with the scene is necessary (e.g., grasping a coffee pod, inserting the coffee pod into the coffee machine, picking up toast with a spatula). Humans excel identify- ing these types of interactions, these segments are quite easy to label. The exact amount of time "padded" onto these dense periods did not seem to affect learning in our experiments. Note that if each entire demo is treated as a dense period, our algorithm reduces to BC.
Labeling Strategy Consistency: The final consideration is for the consistency of the mode labeling strategy between different demonstrations. Variation in the exact boundaries choices for waypoints and dense segments inevitable with human labeling. While the effects of certain types of variation can be quite difficult quantify in general, we believe that is important minimize this variation without adding additional burden on the user. In our experiments, for each task and dataset, have only one user provide the mode labels, according to single strategy. For example, in the NutAssemmbySquare task, where to goal is to insert square nut onto a peg, user might define the following strategy:
1. Reach waypoint above the square nut (sparse)
2. Go down, grasp, pick up (dense)
3. Move the nut up (sparse)
4. Move the nut above the insertion point (sparse)
5. precisely insert the nut on the peg (dense)
In general our method is quite robust to variations within a single mode labeling strategy (for single labeler), and we do no additional post-processing on mode labels or waypoints in any of our experiments.
B.4 Training on Mode Labels
With labeled modes and waypoints, HYDRA learns to predict the mode, the waypoint, and the low- level action at every time step according to the loss in Eqn. 4. However, due to training higher dimensional action space (e.g. for robot poses: |A| = 7 + 7 7 1) with a supervised objective, over- fitting can be a key concern during training. For all iision-based experiments, we perform random cropping to 90% the image size. However, there are several interesting mode-specific augmentations that can be done using mode labels and waypoints to mitigate this problem:
Mode Smoothing: While the simple binary cross entropy mode loss in Eqn. 3 suffices for learning to predict modes, sometimes the hard boundary between segments can lead to mode oscillation or cycling when evaluating at test time. For example, model might predict dense mode, then predict sparse mode the next step that brings it back to the previous state, and repeat. In these cases (which are rare in practice) it can be beneficial to smooth the mode labels to extract continuous probabilities for the mode label at each step: p(Tm) convolve(Tm. [1 21), where is the kernel size. This yields the following loss:
Lm (0) Lo.. (p(m) log अı (m 1/0) (1 p(m)) log ne (m 0/0)
(5)
15

---

With this smoothing of the mode labels, we are effectively removing the hard boundary between sparse and dense periods, which can help generalization for the mode prediction head of HYDRA at test time.
Waypoint Period Augmentation: It is common in the literature to add small amounts of proprio- ceptive state noise (increasing state diversity) to demonstrations. However, during object interaction (i.e. dense periods), this noise can make policy learning more difficult since minor variations in the state can have large changes in the action space. However, with knowledge of sparse and dense modes in HYDRA, we could add diverse state augmentations to the proprioceptive state during only the sparse periods. This waypoint period augmentation can help reduce overfitting in SparseNet, since we will learn to reach the same waypoint (action) from many different robot poses (state).
Both mode smoothing and waypoint augmentation, while not utilized in our experiments, illustrate the potential for new augmentation strategies that arise with access to mode labels.
Environment
Method
# Demos
B
H
lr
?
3m
|
DI
[S]
|
GMM
BC
200
256
le4
400
0
NutAsseemblySquaee
BC-RNN
200
256
10
le4
400
0
HYDRA
200
256
10
le-4
0.5
0.01
400
200
200
200
0
BC
200
256
le.4
400
5
ToolHang
BC-RNN
200
256
le44
1000
5
HYDRA
200
256
20
1e.4
0.5
0.1
1000
400
400
400
0
BC-RNN
100
16
10
lc-4
64
1000
5
KitchenEny
HYDRA
100
16
10
le.4
0.5
0.01
64
1000
400
400
400
5
BC-RNN
8
10
lc-4
64
1000
0
Peginsertion
HYDRA
75
8
10
lc-4
0.5 0.01
64
1000
1000
1000
1000
0
BC-RNN
100
8
10
le4
64
1000
0
MakeCoffee
HYDRA
100
8
10
le-4
0.5 0.01
64
1000
1000
1000
1000
BC-RNN
80
8
10
le-4
64
1000
MakeToast
HYDRA
80
8
10 le.4 0.5 0.01
64 1000 1000 1000
1000
0
Table I: Hyperparameters for each environment, from left to right: B is batch size, H is the horizon length for training, Ir is the learning rate, Y is the per time step weighting of the current mode, 8m is the weighting of the mode loss, |i] is the image encoding size (for cach image), [D] is the hidden-size for recurrent dense networks (DenseNet, BC-RNN)
the MLP width (BC), |S] the width of the SparseNet MLP (3 layers), (rg') is the width of the action head (2 layers). |x is the width of the mode head (2 layers), and finally GMM the number of Gaussian mixtures (or 0 if deterministic) used for the dense action space. The top 3 rows are sim environments, where the first two are state only. The bottom three rows are rision-based real-world experiments. Hyperparameters stay mostly constant for HYDRA between experiments, with larger policy sizes for harder tasks. In almost all cases, BC-RNN, BC, and HYDRA share the same hyperparameters.
C Model Architectures & Training
To train HYDRA, we use similar procedure as in prior work [19, 14]. For each input of shape x..... D.. we load sequential batches of size B BxHx x x... DN. where H is the horizon length. Next we outline the network design for HYDRA, and hyperparameters used in each environment.
C.1 Network Design
As described in Section 4, HYDRA consists of SparseNet, which predicts the waypoint trajectory Tw: and DenseNet, which predicts the mode trajectory Tm and low level action trajectory Ta- Both networks condition on the same input observation space (proprioceptive state trajectory TaE and environment state T. For vision based experiments, s consists of both wrist mounted and external camera observations. Each image is encoded via ResNet18 architecture encoder (Iwo encoders, Eg Egrist with separate parameters) which is trained end-to-end Next, the image encodings are concatenated along with the proprioceptive trajectory Tap.
C.2 Model & Training Details
Visual encoders use a ResNet-18 18 architecture trained end-to-end on both external images and end- effector images. We train all methods for 500k training steps over 3 random seeds, and like prior
16

---

Algorithm Training HYDRA
1: Given N (number of training steps)
2: Given mode-labeled dataset: D (Tor Tax Two ..
3: Networks Egg Enguass W T T. "0
4: for in range(NN) do
5: Tos Fex Twns Tm 1
Load (B ....
6: T; Ligrit
(To)
) + Li (T) o TMP
D Encode
7:
= T (Ti)
W
D waypoint (SparseNet) > mode (DenseNet) > action (DenseNet)
8:
Tm = T0 (Te)
9:
Ta Tg (Te)
10:
Compute [(0) in Eqn. 4 and update 0
External
Policy
Hybrid Actions
W g
ResNet-18
MLP
Tw
waypoint
π
ge
Wrist
Te
MLP
T
mode
RNN
ResNet-18
MLP
Ta
action
ht
T
TsP
Figure 6: Specific instantiation of HYDRA for vision based experiments.
work we report the average over the best performing checkpoints per run [19]. We found that BC policy performance fluctuates significantly even for neighboring checkpoints. However, unlike prior work we use a fixed evaluation set of episodes in simulation to choose the best checkpoint. This reduces the likelihood of choosing the checkpoint that was evaluated on favorable environments (i.e., rejection sampling of harder environment initialization).
For all experiments, dense action), and uses a separate MLP with the same inputs for the Sparse Net (predicting sparse waypoints), shown in Fig.
method uses RNN (LSTM) for Dense Net (predicting the mode and the
The input embedding is then passed into SparseNet (MLP) which outputs the waypoint as a robot pose (position and quaternion). DenseNet can be any sequential model (RNN, Transformer, etc) that produces some temporal embedding тe (RNN in our case). This architecture is shown in Fig. 6, and the training cycle shown in Algorithm
C.3 Evaluation Details
During evaluation (see Algorithm 3), the policy chooses the mode using My. If me 0, the model will servo in closed-loop fashion to the predicted waypoint w; (Line 7) using controller (Line 10). The policy is queried at every step to continually update the policy hidden state, but importantly its outputs
ignored until we reach the waypoint avoid action prediction errors. (Line 4). If m, = I, the model will execute one step using the predicted dense action ā (Line 14).
C.4 HYDRA Hyperparameters
The hyperparameters used in the main text for all six environments are shown in Table 1, for BC, BC-RNN, and HYDRA. Hyperparameters stay mostly constant for HYDRA across all of the ex- periments, with larger policy sizes for harder tasks. Additionally, in almost all cases, BC-RNN, BC, and HYDRA share the same hyperparameters where possible. In the real world experiments, hyperparameters are exactly the same both across methods and across environments.
We chose parameters through extensive parameter sweeps (for each column in Table except the number of demos and batch size) and picking the best performing model for all methods. For
17

---

Algorithm 3 Test Time Execution
Given env, (m, a, w/o), initial state oo, controller T
2: = O, w = None
3: while not done do
4: m,, āg. W'11 ~ x(-10=)
Sample policy
5: // Check for new sparse mode
6: if w is not set and m, = 0 then
7:
w = Wi
Set a new waypoint
8:
// Compute the waypoint-optinee action (sparse)
9:
if a is set not reached and not timed-out then 0. T(or, w)
10:
Compute waypoint-optimal action
11:
else
12:
None
> Unset waypoint if reached
13:
// Step the environment
14:
01+1 env.step(ā;)
15:
t=t
ToolHang, we found BC-RNN with a horizon of 20 was overfitting due to the high dimensional input space, while HYDRA seemed more robust that, potentially since more consistent actions leads to a simpler objective (see Section 4.1). For BC, using an MLP size of 1000 for ToolHang also seems to overfit and perform worse than MLP size of 400, while the reverse was true for BC-RNN and HYDRA. We find that all methods are sensitive to these hyperparameters, which an open problem for the community.
D Additional Results & Analysis
In this section we show rollouts of our method and baselines, and then perform ablations of our method and analyze the results, including mode labeling sensitivity, mode label learning from less data, choices in action space design, different loss weightings, and robustness experiments. All ablations are performed on the NutAssemblySquare task unless otherwise stated.
D.1 Rollouts for Real Environments
Fig. 7 shows example rollouts from the uncurated demonstration, the learned BC-RNN policy, and HYDRA. Qualitatively, in the top row of Fig. we see that HYDRA produces more consistent and optimal trajectories at evaluation time that help the policy to stay within the narrow "band" of the successful state distribution at test time, thus improving performance.
For the long horizon MakeToost task, the performance of HYDRA is much better than BC-RNN, but lower overall than in MakeCoffee. We hypothesize that the difference between this task and MakeCoffee is primarily in the consistency of demonstrated actions (see demonstration rollouts in Fig. 7), with significant variation in the behaviors for nearby states especially during dense periods. This leads to BC-RNN having highly noisy and sub-optimal actions, which manifest quite notice- ably in Fig. However, HYDRA yields much more consistent and optimal motions, reducing the distribution-shiftt problem.
D.2
Mode Sensitivity
Next, we consider the sensitivity of HYDRA mode labels, specifically terms of the number of labeled waypoints in each episode. In Table 2, we ablate the number of waypoints by introducing N intermediate waypoints in every sparse segment, for N and N 2. Since there are at least sparse segments labeled in each demo in NulAssemblySSuurr. this corresponds adding at least 3 or more waypoints to each demonstration, respectively. We see that performance drops are relatively minor in both cases, showing that HYDRA is robust to different waypoint choices. We hypothesize that the reason for the minor performance change when adding more waypoints is that SparseNet must learn more complex waypoint space that more multi-modal.
To understand the sensitivity to consistency of mode labels, we also show the results for an aggre- gated dataset consisting of two mode labelers in Table 2. We find that using multiple labelers does not result in huge drop performance, however there is still some sensitivity having different mode labeling strategies amongst the two labelers.
18

---

Uncurated demonstration
BC-RNN rollouts
HYDRA rollouts
Noisy saboptimal actions
Consistent & optimal actions
Notey
MakeCoffee
VERY Noisy actions
Notsy & suboptimai actions
Consistent & optinaal actions
Make ToasT
Figure 7: Cofeee (top) and MakeToast (bottom) rollouts, with the demos (left), HYDRA rollouts (middle), and BC-RNN rollouts (right). method produces more consistent optimal actions compared to both BC-RNN and the demonstrations, and thus is able to stay within the narrow success "band" of the state dis- tribution. BC-RNN has many sub-optimal behaviors, leading to less completed trajectories in middle column. The demonstrations for MakeToast able distribution shift for BC-RNN in the MakeToast task. contrast, HYDRA curates the demonstrations MakeTToat using sparse and dense periods to follow more consistent paths, thus leading to higher success.
noisier than those MakeCoffee, leading to more notice-
Base Add-1 Add-2 Two-Labelen
90.0 86.0 80.0
86.0
Table 2: Success rates for HYDRA when artificially more waypoints are added to sparse periods (left) and for an aggregated dataset with two labelers (right). Adding intermediate waypoints sparse segments has only minor effect performance despite the increase in complexity of the pose action space. Likewise using multiple labelers has minor drop in performance.
D.3 Learning Mode Labels from Less Data
Providing mode labels can be an additional overhead when training HYDRA. To reduce overhead, we might want learn the mode labels from few labeled examples, and use this relabel the rest of the dataset. To show the promise of such an approach, we learn to predict the "click state" at each time step (same in Fig. 5) using a simple RNN architecture with the same parameters as the model used for training. This model outputs two logits, one for the mode itself (m,), and one that represents a switching criteria between segments (st). This allows us to predict not only the sparse or dense label, but also the waypoint label for each sparse segment. We additionally smooth both my and 8 as is commonly done in binary sequence prediction tasks. Table 3, we demonstrate that can learn mode labels from 25% of the data with only 10% drop in performance for the square task, and even less drop when training on 50% or 75% the data.
90%
75%
50%
25%
92
88
86
82
Table Success rates for HYDRA for NutAssemblyyquure when the mode labels are learned (predicting "click state" in Fig. 5).
With this preliminary evidence, we believe the sample efficiency of this mode learning procedure can be improved by incorporating prior data from a wide range of tasks, potentially even using labeled internet data. To address the multi-modality of mode labels that might occur when having multiple
19

---

people provide labels, future work might leverage few-shot or in-context learning approaches to adapt a particular style of mode labeling.
D.4 Variations in the Action Space
Why do we need the dense period at all? In Table 4, we compare HYDRA's hybrid action space to waypoint only ablations, both with and without the test-time controller Timeu-- With Timeur. the model outputs a waypoint and the robot reaches that waypoint using T linear without querying the policy ("open loop"), and without Tinear. the model outputs a new waypoint every step which gets converted to action using T Tinear ("close loop").
First we show results for WP-Next(N] in Table 4, where waypoints are the pose of the robot N steps in the future at each state (hindsight relabeling). Second, we compare to WP-Mode, which uses the same mode labels in HYDRA to get more intelligent future waypoints during sparse segments. No pose-based models see any success, which we hypothesize is due to the mismatch between the human action and the online action T Timmar(o, w). which can lead to out of distribution states. Even in the open loop case, the waypoint only models are unable to perform the task, with failures involving imprecise behaviors during dense periods where exact velocities truly matter.
We additionally compare our method with and without the use of lincar online (first row in Table 4). We see that HYDRA greatly benefits from the online waypoint controller, since Tincarr follows an optimal path while the policy-n--the-loop approach leaves room for compounding errors in both the mode, action, and waypoint prediction. This once again illustrates that HYDRA yields more consistent and optimal actions by employing hybrid action abstraction.
Ours WP-Nextl WP-Next2 WP-Next5 WP-Mode
w/ T
90.0
0.0
0.0
2.0
0.0
w/o T
58.0
0.0
0.0
0.0
0.0
Table 4: Success rates for different action spaces. HYDRA uses hybrid action space, while the the rest use a pose-based action spacc. Top row: waypoints reached using Timer before calling the policy again ("open" loop). Bottom row: waypoint actions are computed at every step and instead of reaching the action, the policy will convert waypoint w to dense action a using Timei ("closed" loop). WP-Next(N) uses the proprioceptive state N steps in the future as the waypoint for each state. WP-Mode uses the same mode labels as in HYDRA to get the waypoints, but does not implement hybrid action space. None of the reasonable performance, showing the importance of both dense actions and waypoint phases.
action spaces get
D.5 Ablating Mode Weighting (7)
We also show the effect of different values of 7, the weight of the current mode loss. If for a given step in training mode my = 0 (sparse), then we weight the sparse waypoint loss for We with 1- --y and the dense action loss for ae with y. Lower y thus corresponds fitting the current mode action loss more than the other mode's 's loss. Therefore, also controls the contribution of the relabeled actions during sparse periods to the overall objective in Eq. (2). We use 7 = 0.5 in most experiments, meaning both action (waypoint and dense action) losses are weighted equally during training. We provide a sweep over y in Table for NutAsseebbySSuare and ToolHang, and we see that choosing y only has a minor effect. Nonetheless, 7 0.5 is consistently the best. This illustrates that (1) HYDRA is fairly robust to y, (2) learning relabeled dense actions during sparse periods and sparse actions during dense periods is beneficial to performance this supports the claim in Section 4.1 that training on relabeled dense actions outperforms uncurated dense actions.
7 0.1 7 0.2 7 0.4 7 0.5
Square
80.0
84.0
88.0
90.0
ToolHang
60.0
62.0
58.0
64.0
Table 5: Success rates for different values of r for both NutAssemblySqurre and ToolHanng For both Nu- lAsvemblySuaae amd Toolflang, 7 does not have large effect. We saw less of change for vision based experiments. Thus for real world experiments, we fix 7 0.5 (no mode-specific weighting).
One issue with our framework the impact of false negative mode predictions, due to the use of the waypoint controller online; however, we mitigate the effect of this by training SparseNet even during dense periods (waypoint is the next state). Thus HYDRA is often able to overcome false negatives and still continue on the correct trajectory, providing another chance at the next step to
20

---

predict the correct mode label. On first glance, it is strange that = 0.5 consistently works the best in practice. We speculate that the choice in gamma actually helps control the issue with false negatives. Specifically, 7 = 0.5 means that SparseNet will be trained equally during dense periods, and vice versa for DenseNet. We believe this helps overcome challenges of both false negatives or false positives.
D.6 Transformer-baser architecture
In Table we show the performance of purely transformer-basee BC implementation on the Kitch- enEny task. We see in this long horizon task that BC-RNN notably outperforms BC-Transformer in this single-task imitation learning setting, and we found similar drops in performance for the state- based simulation experiments. Thus, we did not include BC- BC-Transformer world experiments. We note that VIOLA, which uses a similar underlying transformer but with a object-centric input representation, performs notably better on KitchenEnv than BC-Transformer.
baseline in our real
BC-RNN BC-Transformer VIOLA HYDRA
Square
84
78.0
90.0
Kitchen 52.0
24.0
78.0
87.0
Table 6: Success for different values of BC architectures NutAssemblySquare (state-based) and Kitch- enEny (vision-based). For NutA SquAsyyelyyquurre we see that using BC-Transfommer minorly reduces perfor- mance. In KitchenEnv, we see a larger performance drop for BC-Transformer compared to BC-RNN. VIOLA proves superior transformer based architecture compared to simple OC-Transformer for KitchenEnv. In all cases, HYDRA beats both RNN and Transformer- baselines. All models share the same visual encoder structure and action spaces as described Table
D.7 Robustness of HYDRA to system noise
In Section 3 we noted the fundamental trade-off between consistent actions and state diversity, HY- DRA breaks this tradeoff by relabeling actions in offline data, encouraging action consistency with- out reducing the state coverage of the data. To show that HYDRA still benefits from the state diversity in human data, in Table 7 we analyze the effect of system noise on HYDRA and BC. We find that HYDRA only drops from 90% to 86% (4% drop) under the same system noise as used with BC. This shows that not only does HYDRA capture the state diversity in human data, but it is able to be even more robust to distribution shift than BC. We attribute this boost part to the use of closed loop waypoint controller, which consistently reaches the waypoint under system noise. This also supports the claim made in Section 6 that the gap in performance between HYDRA and baselines in real compared to simulation experiments can in part be attributed to the added system noise found the real world.
Base Noise=0.1 Noise=0.3
BC-RNN
84
76.0
60.0
HYDRA
90
92.0
86.0
Table 7: The effect of increasing system noise (columns left to right) BC-RNN (top row) and HYDRA (bottom row) trained on human data for NutAssemblySquare. While BC-RNN drops 24% under the max system noise, HYDRA only drops 4%, illustrating the ability of HYDRA to capture state diversity and thus be robust to distribution shift.
D.8 HYDRA-NR Analysis
In Section 5 we see that HYDRA without action relabeling (HYDRA-NR) often performs worse than HYDRA, due less action consistency the dataset. However, we see in Fig. that for the 50 episode domain, performance is actually similar for HYDRA and HYDRA-NR. We suspect this is because there so little state coverage that the action variability is artificially low, and thus the benefit to relabeling is minor. In the kitchen environment, we also do not see much change for HYDRA-NR, which we speculate due to higher action consistency in the demonstrations in the dataset. Qualitatively we observed that the kitchen environment demonstrations are much higher quality than other environments, and also involve much smaller action space (only position control, no orientation, consistent with prior work).
21


Reviews:
[{"review_id": "n2LWbwJzbt7", "reviewer": "Reviewer_gKhQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a hybrid action representation for imitation learning. The policy outputs both 1) the action (which is the same as standard behavior cloning) and 2) a waypoint which is given to a lower-level waypoint controller. The policy also produces a “mode”, which determines which of the two representations is used. The mode within demonstrations can be labeled both in an online and offline fashion. In the waypoint mode, the actions in the  demonstrations are relabeled by the waypoint controller. The waypoint mode helps reduce effective horizon and improve data consistency. The simulated and real-robot experiments show that the proposed strategy is effective in learning long-horizon tasks compared with existing baselines.", "summary_of_recommendation": "The paper is well written and technically sound. I like the proposed approach and find it, to my knowledge, novel. My major concerns lie in the experimental results of this paper. I find the evaluation to be not very convincing due to missing or rather, inconsistent choices of baselines. I will consider increasing my score if the authors can either justify their choices of baselines or provide the results for the missing baselines.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- Why are some baselines/ablation results missing from certain experiments (see the first point in **Weaknesses**)?\n- Regarding hyperparameters (Table 1 in Appendices), why does HYDRA use a different horizon (20) than other algorithms for “ToolHang”? - - Why does BC use a much smaller hidden size (400) than other algorithms for “ToolHang”? How are those hyperparameters determined?\n- In the introduction (line 40), can the authors provide an intuitive explanation on why a waypoint-only approach could fail to “insert a coffee pod”?\n- Why not use a recurrent architecture for the sparse net as well?\n- When $m_t=0$, the waypoint controller is used until either the waypoint is reached or a timeout. This strategy seems to be vulnerable to false negatives of mode labeling. What is the consideration behind not just letting the waypoint controller to execute for one step then re-evaluate? \n- Although the authors provide a discussion on the choice of $\\gamma$ in Appendix D.5. I still find it puzzling that $\\gamma=0.5$ (not differentiating between modes) performs the best consistently. Can the authors provide a hypothesis on why this is the case? Is it related to test-time errors of the mode predictor?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Strengths\n- The paper is overall well-organized and well-written. I find the paper enjoyable to read. The motivation of the paper is clear. The equations and notations within the paper are well-explained. The figures are well-made and nicely explains the main idea of the paper.\n- I find the proposed method novel (to my knowledge) and interesting. The idea of using a hybrid action representation is quite neat. The usage waypoint reduces the effective horizon of the task, while being able to predict per-step action greatly improves the flexibility of the proposed approach.\n- The experimental evaluation is on a real robot with visual observation. The tasks considered are non-trivial, some involve long horizons.\n- The limitation discussion is thorough. I appreciate that the authors also show preliminary experimental results on potential future work.\n\n## Weaknesses\n- Sometimes, it is unclear why the authors do not include certain baseline algorithms or ablation studies. For example, why is HYDRA-NR not provided for “NutAssemblySquare” with 100 and 50 demonstrations and “KitchenEnv”? Why is BC not included for KitchenEnv? Why is BC not included for all real robot experiments? Why do the authors only present VIOLA results for “KitchenEnv” and “MakeCoffee”, but not for others? I can see the experimental results much more convincing if all these baseline results are presented in the paper, even if HYDRA does not outperform these baselines.\n- The proposed method requires the labeler to identify the mode. Similar to the fact that BC might not work well with inconsistent actions, it seems that the proposed method may not work well when the mode annotations are not consistent. This could happen especially when the modes are given by different labelers. In Appendix A, the authors mention that HYDRA is robust to “the labeling strategies across the two labelers”. It is unclear if it refers to training HYDRA with each labeling strategy separately or training HYDRA with an aggregated dataset by two labelers.\n- The demonstrations for the real robot experiments are given by a “proficient user”. It would be interesting to see if HYDRA is also effective given demonstrations (including but not limited to mode labeling) from inexperienced users.\n- For long-term tasks, sometimes there is not an absolute ordering for each step. For example, for the “MakeCoffee” task, the robot can achieve the task by either first inserting the lid then placing the mug, as described in the paper, or the other way around. It seems that the demonstrations are intentionally given in a consistent order. It would be interesting to see if HYDRA can also work with demonstrations with inconsistent ordering of intermediate steps.\n\n## Post Rebuttal Updates\nI believe most of my concerns have been adequately addressed during the rebuttal period. As I noted in my original review, I find the paper enjoyable to read and the proposed approach interesting. I also find the results to be more convincing with the additional numbers from HYDRA-NR. I am still not very convinced that comparison with VIOLA is not needed due to VIOLA's complexity. Nonetheless, I find the paper worth publishing.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "dS_AAQ6WFbe", "reviewer": "Reviewer_1bNM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes HYDRA: a hybrid method for robot learning from expert demonstrations. Policies learned through Imitation Learning can suffer from state distribution shift at test time due to compounding errors in action prediction which lead to previously unseen states. The paper contributes to mitigating this issue by defining a two-level action representation: sparse high-level waypoints and dense low-level actions. HYDRA can switch between the two modalities online, leveraging high-level actions during free-space motions and dense low-level actions during contact-rich manipulations, so to maintain policy dexterity and generality at the same time. The framework is a multi-head network that requires additional mode labels to be annotated by the experts during data collection or offline. The authors evaluate their proposal in many simulated and real robotic tasks showing competitive results.", "summary_of_recommendation": "In summary, I recommend accepting the paper, because it is clear and proposes an interesting approach for combining high-level and low-level actions in imitation learning for robot tasks. The evaluation is not very extended, but still promising. The benefits and the limitations are overall well described, but some clarifications on the use of the controller in case of obstacles are needed.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "How do you avoid possible collisions during the use of controller T?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a nice trade-off between learning from predefined temporal abstractions and learning only from low-level expert actions. The problem is clearly explained, along with the relevant related works. The method is quite simple, yet very well described and evaluated. The results seem promising, especially in the real-world domains tested.\nThe main advantages of the method are that it can maintain dexterity of the policy when it is required by the task and, at the same time, avoid cumulating errors in the state during sparse interactions with the environment.\nThe main limitations of the approach regard the amount of additional labels and prior knowledge that it requires. In fact, the method needs both: (i) some extra labels specifying the mode to use at each time step, which must be collected from the experts, and (ii) a controller T to use during preprocessing of the dataset for relabeling the low-level actions and to be used at test time. Regarding the controller, the authors say: \"the main requirement for labeling modes is that during sparse phases (mt = 0), the labeled waypoint wt should be reachable via T starting from ot (i.e., without collision)\". They say that this reachability requirement can be ignored in case a collision-avoidance planner is available, but, however, they do not assume to have such a planner, and they do not provide any explanation on how they manage possible collisions while using the controller. I want the authors to better elaborate on this point, since it is a crucial feature of the approach that could eventually compromise the framework's applicability. This is one reason why I propose only weak acceptance for the paper. Another one regards the comparison with other methods. They only compare their approach with behavior cloning (BC) and BC-RNN. I wonder how their method could compare with other imitation learning algorithms and with Johns 2021, on which they build their approach.\nOverall I found the paper clear, except for the part I underlined, and the approach quite novel and fairly evaluated.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "HgWW3_YmuvA", "reviewer": "Reviewer_QMAq", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper addresses the issue of distribution shift in imitation learning for robot motion generation. They argue the importance for appropriate selection of action representation in combating this effect. Specifically, they propose to use a policy architecture designed with a hybrid action representation, combining sparse waypoint prediction for movement in free-space with dense action prediction for movement near and at contact/interaction points. The network also learns to predict switching between these two modes. The authors also argue for the importance of relabeling actions for improved temporal consistency and waypoint following.", "summary_of_recommendation": "The work is extensive, but the originality and novelty of the approach is unclear. More evaluations are needed to support the claims in the paper.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "Please address the points mentioned above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Remarks and Questions:\n\n- It seems like the data labeling method performs linear interpolation between cartesian end-effector locations:\n- The authors describe this low-level action relabeling as the application of a “linear controller” to “imagine” new actions. This description is unnecessary, and also confusing (the use of “imagining” has typically been used for generative, learning-based methods which predict high-dimensional observations).  \n- How are end-effector poses interpolated during the re-labelling process? This would seem important for learning to align the gripper in the right way when approaching a grasp point.\n- There seems to be an assumption in the linear-interpolation scheme that the linear trajectories are feasible, or can be executed by an underlying controller. If it is the latter,  please describe the controller used. Otherwise, please state how the approach deals with kinematically-feasible locations, singularities, etc.\n- There also seems to be an assumption that the action space must be directly visible by the human annotator for relabelling. How would this approach be applied for cluttered/partially observable workspaces? How about higher-dimensional action spaces (ex. joint-space) that may be required for more complex tasks or constraint satisfaction (ex. obstacle collision avoidance).\n- Sparse data labeling for robot Learning-from-Demonstration is an area that has been studied for a long time in Human-Robot-Interaction community, ex. through the use of key-frame demonstrations (see work done by Andrea Thomaz, for instance). There can be much variability in providing such coarse/sparse demonstrations, and user studies are generally required. How does this work relate to this body of work? How do they deal with consistency of trajectory segmentation across human demonstrators?\n\nPros:\n- Generally well written, with clear presentation and figures.\n- Evaluations are performed on a wide array of tasks on a physical robot system.\n\nCons:\n- Baseline setup is unclear: how were BC-RNN, BC trained and on what data? Was this the same sparse/dense data as HYDRA? It would be more convincing to compare against BC-RNN/BC trained on only dense data from both modes (i.e. no action-relabeling).\n- Unclear whether action relabeling has a large impact, judging by the experimental results. It may be beneficial to include the HYDRA-NR evaluations in more of the environments.\n- IL baselines: given that mode-labeling can be performed online, it would be useful to compare against intervention-based methods, or expert-relabeling (ex. Dagger), at least on a subset of experiments.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "4O2TximdQK", "reviewer": "Reviewer_DQGz", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper addresses the challenge of compounding errors in action prediction for long-horizon tasks when deploying imitation learning policies during test time. They introduce a 2-level policy that outputs sparse high-level waypoints at the higher level and dense low-level actions at the lower level. They argue that the temporal abstraction in the action space reduces the number of steps in the action space for the same horizon length thus reducing compounding errors due to the state-distribution shift during evaluation. Additionally, they argue that relabeling the demonstration dataset to contain consistent actions for the same state make learning dense actions easier. They show experiments in both simulation and real robots to prove that their method beats the baseline methods on long-horizon tasks, and provide explanation why their methodology helps when working with high-variance real-robot demonstrations.", "summary_of_recommendation": "The authors provide novel ideas for mitigating compounding errors when rolling out imitation learning policies, and show compelling evidence in the form of success rates on long-horizon tasks. They were able to articulate the proposed methodology well, making the paper theoretically sound and easy to follow. The contributions are robotics focused and open up some interesting venues for future research towards solving long-horizon tasks.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. Did the authors do any ablation or analysis to prove that the 2-level action space reduces compounding errors due to the state distribution shift when deploying this policy at test time?\n2. The authors argue that making the action space more expressive to capture human multi-modality, coupled with consistent action demonstrations from the expert make the policy learning easier. Wouldn’t reducing diversity in the training data hamper potential usability of the policy when deployed in the real world? If so, can the authors identify tasks where this methodology would hamper performance?\n3. It is unclear why VIOLA was omitted for the MakeToast and SortDishes tasks for experimentation. Can the authors clarify?\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Quality: This paper addresses an important challenge in imitation learning (reducing compounding errors for solving long-horizon tasks,) is theoretically sound, and well-placed in the existing literature.\n\nClarity: The paper is well-written and easy to follow. The experimental setups are verbose enough to make them fairly reproducible, with additional details in the appendix.\n\nStrengths: \n1. The authors were able to clearly lay out the different classes of approaches that can be beneficial for mitigating compounding errors in imitation learning (introducing temporal abstractions in the policy and increasing model expressivity) and place their approach well in the existing literature.\n2. The experimental results show that the proposed method maintains a much higher success rate for the most part of the multi-step tasks compared to baselines. The authors also identify the potential benefits of their method when working with real-world demos with higher variance.\n\nWeaknesses: \n1. For the MakeToast and SortDishes tasks, the proposed method did not perform well on the last few steps of the task. It would have been nice if the authors provided some compelling reasons for the failure.\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a new method for imitation learning, which predicts either coarse, sparse actions (for free space motion) or fine-grained, dense actions (for object interaction). This requires the human operator to specify, for each stage during the demonstration, if that stage is a “coarse” stage or a “fine” stage. The learned policy then predicts which of these two modes should be executed, as well as the action for that mode. Simulation and real-world experiments show that this method gives competitive results.

Reviews before the rebuttal were 1 x “weak reject”, 2 x “weak accept”, and 1 x “strong accept”. Authors addressed the reviewers’ comments, and provided an updated paper. This also included some new experiments that reviewers had asked for. Following the rebuttal, reviewers maintained their pre-rebuttal scores.

The AC and reviewers then had a brief discussion on the paper. Based on this, and the reviews, there is a general consensus that the proposed method is sensible and achieves good performance. Most of the issues raised by the reviewers were answered adequately by the authors and incorporated into the updated paper. The “weak reject” reviewer did not engage with the authors following the rebuttal, but they did not raise any major criticisms in their original review and mainly requested clarifications and new experiments (HYDRA-NR experiments, which the authors then provided). However, some reviewers found that the novelty is somewhat limited with respect to other methods (see below), and that the need for the operator to manually label different phases of the task might make this method impractical if non-experts wish to teach robots new tasks (a limitation noted by the authors). But overall, the reviewers and AC found the method to be sensible, effective, and interesting to the wider robot learning community.

An important point raised during the AC/reviewer discussion was that of novelty and the paper's related work section. Authors state that they build upon [36], but claim that [36] “cannot easily scale to multi-step manipulation tasks with multiple stages of unstructured object interaction.” However, in a CoRL 2021 paper “Learning Multi-Stage Tasks with One Demonstration via Self-Replay”, [36] was indeed extended for multi-step tasks, with a simple modification. Reviewers did not cite this work and so there is less novelty in HYDRA than is claimed by the authors. We encourage the authors to address this in the final paper, and more clearly describe the relationship between HYDRA, [36], and the CoRL 2021 paper.
Decision: Accept (Poster)