Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary_of_paper: This paper addresses the challenge of compounding errors in action prediction for long-horizon tasks when deploying imitation learning policies during test time. They introduce a 2-level policy that outputs sparse high-level waypoints at the higher level and dense low-level actions at the lower level. They argue that the temporal abstraction in the action space reduces the number of steps in the action space for the same horizon length thus reducing compounding errors due to the state-distribution shift during evaluation. Additionally, they argue that relabeling the demonstration dataset to contain consistent actions for the same state make learning dense actions easier. They show experiments in both simulation and real robots to prove that their method beats the baseline methods on long-horizon tasks, and provide explanation why their methodology helps when working with high-variance real-robot demonstrations.
review: Quality: This paper addresses an important challenge in imitation learning (reducing compounding errors for solving long-horizon tasks,) is theoretically sound, and well-placed in the existing literature.

Clarity: The paper is well-written and easy to follow. The experimental setups are verbose enough to make them fairly reproducible, with additional details in the appendix.

Strengths: 
1. The authors were able to clearly lay out the different classes of approaches that can be beneficial for mitigating compounding errors in imitation learning (introducing temporal abstractions in the policy and increasing model expressivity) and place their approach well in the existing literature.
2. The experimental results show that the proposed method maintains a much higher success rate for the most part of the multi-step tasks compared to baselines. The authors also identify the potential benefits of their method when working with real-world demos with higher variance.

Weaknesses: 
1. For the MakeToast and SortDishes tasks, the proposed method did not perform well on the last few steps of the task. It would have been nice if the authors provided some compelling reasons for the failure.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The authors provide novel ideas for mitigating compounding errors when rolling out imitation learning policies, and show compelling evidence in the form of success rates on long-horizon tasks. They were able to articulate the proposed methodology well, making the paper theoretically sound and easy to follow. The contributions are robotics focused and open up some interesting venues for future research towards solving long-horizon tasks.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper addresses the issue of distribution shift in imitation learning for robot motion generation. They argue the importance for appropriate selection of action representation in combating this effect. Specifically, they propose to use a policy architecture designed with a hybrid action representation, combining sparse waypoint prediction for movement in free-space with dense action prediction for movement near and at contact/interaction points. The network also learns to predict switching between these two modes. The authors also argue for the importance of relabeling actions for improved temporal consistency and waypoint following.
review: Remarks and Questions:

- It seems like the data labeling method performs linear interpolation between cartesian end-effector locations:
- The authors describe this low-level action relabeling as the application of a “linear controller” to “imagine” new actions. This description is unnecessary, and also confusing (the use of “imagining” has typically been used for generative, learning-based methods which predict high-dimensional observations).  
- How are end-effector poses interpolated during the re-labelling process? This would seem important for learning to align the gripper in the right way when approaching a grasp point.
- There seems to be an assumption in the linear-interpolation scheme that the linear trajectories are feasible, or can be executed by an underlying controller. If it is the latter,  please describe the controller used. Otherwise, please state how the approach deals with kinematically-feasible locations, singularities, etc.
- There also seems to be an assumption that the action space must be directly visible by the human annotator for relabelling. How would this approach be applied for cluttered/partially observable workspaces? How about higher-dimensional action spaces (ex. joint-space) that may be required for more complex tasks or constraint satisfaction (ex. obstacle collision avoidance).
- Sparse data labeling for robot Learning-from-Demonstration is an area that has been studied for a long time in Human-Robot-Interaction community, ex. through the use of key-frame demonstrations (see work done by Andrea Thomaz, for instance). There can be much variability in providing such coarse/sparse demonstrations, and user studies are generally required. How does this work relate to this body of work? How do they deal with consistency of trajectory segmentation across human demonstrators?

Pros:
- Generally well written, with clear presentation and figures.
- Evaluations are performed on a wide array of tasks on a physical robot system.

Cons:
- Baseline setup is unclear: how were BC-RNN, BC trained and on what data? Was this the same sparse/dense data as HYDRA? It would be more convincing to compare against BC-RNN/BC trained on only dense data from both modes (i.e. no action-relabeling).
- Unclear whether action relabeling has a large impact, judging by the experimental results. It may be beneficial to include the HYDRA-NR evaluations in more of the environments.
- IL baselines: given that mode-labeling can be performed online, it would be useful to compare against intervention-based methods, or expert-relabeling (ex. Dagger), at least on a subset of experiments.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The work is extensive, but the originality and novelty of the approach is unclear. More evaluations are needed to support the claims in the paper.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper proposes HYDRA: a hybrid method for robot learning from expert demonstrations. Policies learned through Imitation Learning can suffer from state distribution shift at test time due to compounding errors in action prediction which lead to previously unseen states. The paper contributes to mitigating this issue by defining a two-level action representation: sparse high-level waypoints and dense low-level actions. HYDRA can switch between the two modalities online, leveraging high-level actions during free-space motions and dense low-level actions during contact-rich manipulations, so to maintain policy dexterity and generality at the same time. The framework is a multi-head network that requires additional mode labels to be annotated by the experts during data collection or offline. The authors evaluate their proposal in many simulated and real robotic tasks showing competitive results.
review: The paper proposes a nice trade-off between learning from predefined temporal abstractions and learning only from low-level expert actions. The problem is clearly explained, along with the relevant related works. The method is quite simple, yet very well described and evaluated. The results seem promising, especially in the real-world domains tested.
The main advantages of the method are that it can maintain dexterity of the policy when it is required by the task and, at the same time, avoid cumulating errors in the state during sparse interactions with the environment.
The main limitations of the approach regard the amount of additional labels and prior knowledge that it requires. In fact, the method needs both: (i) some extra labels specifying the mode to use at each time step, which must be collected from the experts, and (ii) a controller T to use during preprocessing of the dataset for relabeling the low-level actions and to be used at test time. Regarding the controller, the authors say: "the main requirement for labeling modes is that during sparse phases (mt = 0), the labeled waypoint wt should be reachable via T starting from ot (i.e., without collision)". They say that this reachability requirement can be ignored in case a collision-avoidance planner is available, but, however, they do not assume to have such a planner, and they do not provide any explanation on how they manage possible collisions while using the controller. I want the authors to better elaborate on this point, since it is a crucial feature of the approach that could eventually compromise the framework's applicability. This is one reason why I propose only weak acceptance for the paper. Another one regards the comparison with other methods. They only compare their approach with behavior cloning (BC) and BC-RNN. I wonder how their method could compare with other imitation learning algorithms and with Johns 2021, on which they build their approach.
Overall I found the paper clear, except for the part I underlined, and the approach quite novel and fairly evaluated.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: In summary, I recommend accepting the paper, because it is clear and proposes an interesting approach for combining high-level and low-level actions in imitation learning for robot tasks. The evaluation is not very extended, but still promising. The benefits and the limitations are overall well described, but some clarifications on the use of the controller in case of obstacles are needed.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper proposes a hybrid action representation for imitation learning. The policy outputs both 1) the action (which is the same as standard behavior cloning) and 2) a waypoint which is given to a lower-level waypoint controller. The policy also produces a “mode”, which determines which of the two representations is used. The mode within demonstrations can be labeled both in an online and offline fashion. In the waypoint mode, the actions in the  demonstrations are relabeled by the waypoint controller. The waypoint mode helps reduce effective horizon and improve data consistency. The simulated and real-robot experiments show that the proposed strategy is effective in learning long-horizon tasks compared with existing baselines.
review: ## Strengths
- The paper is overall well-organized and well-written. I find the paper enjoyable to read. The motivation of the paper is clear. The equations and notations within the paper are well-explained. The figures are well-made and nicely explains the main idea of the paper.
- I find the proposed method novel (to my knowledge) and interesting. The idea of using a hybrid action representation is quite neat. The usage waypoint reduces the effective horizon of the task, while being able to predict per-step action greatly improves the flexibility of the proposed approach.
- The experimental evaluation is on a real robot with visual observation. The tasks considered are non-trivial, some involve long horizons.
- The limitation discussion is thorough. I appreciate that the authors also show preliminary experimental results on potential future work.

## Weaknesses
- Sometimes, it is unclear why the authors do not include certain baseline algorithms or ablation studies. For example, why is HYDRA-NR not provided for “NutAssemblySquare” with 100 and 50 demonstrations and “KitchenEnv”? Why is BC not included for KitchenEnv? Why is BC not included for all real robot experiments? Why do the authors only present VIOLA results for “KitchenEnv” and “MakeCoffee”, but not for others? I can see the experimental results much more convincing if all these baseline results are presented in the paper, even if HYDRA does not outperform these baselines.
- The proposed method requires the labeler to identify the mode. Similar to the fact that BC might not work well with inconsistent actions, it seems that the proposed method may not work well when the mode annotations are not consistent. This could happen especially when the modes are given by different labelers. In Appendix A, the authors mention that HYDRA is robust to “the labeling strategies across the two labelers”. It is unclear if it refers to training HYDRA with each labeling strategy separately or training HYDRA with an aggregated dataset by two labelers.
- The demonstrations for the real robot experiments are given by a “proficient user”. It would be interesting to see if HYDRA is also effective given demonstrations (including but not limited to mode labeling) from inexperienced users.
- For long-term tasks, sometimes there is not an absolute ordering for each step. For example, for the “MakeCoffee” task, the robot can achieve the task by either first inserting the lid then placing the mug, as described in the paper, or the other way around. It seems that the demonstrations are intentionally given in a consistent order. It would be interesting to see if HYDRA can also work with demonstrations with inconsistent ordering of intermediate steps.

## Post Rebuttal Updates
I believe most of my concerns have been adequately addressed during the rebuttal period. As I noted in my original review, I find the paper enjoyable to read and the proposed approach interesting. I also find the results to be more convincing with the additional numbers from HYDRA-NR. I am still not very convinced that comparison with VIOLA is not needed due to VIOLA's complexity. Nonetheless, I find the paper worth publishing.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper is well written and technically sound. I like the proposed approach and find it, to my knowledge, novel. My major concerns lie in the experimental results of this paper. I find the evaluation to be not very convincing due to missing or rather, inconsistent choices of baselines. I will consider increasing my score if the authors can either justify their choices of baselines or provide the results for the missing baselines.
review_title: null

Meta-review: This paper presents a new method for imitation learning, which predicts either coarse, sparse actions (for free space motion) or fine-grained, dense actions (for object interaction). This requires the human operator to specify, for each stage during the demonstration, if that stage is a “coarse” stage or a “fine” stage. The learned policy then predicts which of these two modes should be executed, as well as the action for that mode. Simulation and real-world experiments show that this method gives competitive results.

Reviews before the rebuttal were 1 x “weak reject”, 2 x “weak accept”, and 1 x “strong accept”. Authors addressed the reviewers’ comments, and provided an updated paper. This also included some new experiments that reviewers had asked for. Following the rebuttal, reviewers maintained their pre-rebuttal scores.

The AC and reviewers then had a brief discussion on the paper. Based on this, and the reviews, there is a general consensus that the proposed method is sensible and achieves good performance. Most of the issues raised by the reviewers were answered adequately by the authors and incorporated into the updated paper. The “weak reject” reviewer did not engage with the authors following the rebuttal, but they did not raise any major criticisms in their original review and mainly requested clarifications and new experiments (HYDRA-NR experiments, which the authors then provided). However, some reviewers found that the novelty is somewhat limited with respect to other methods (see below), and that the need for the operator to manually label different phases of the task might make this method impractical if non-experts wish to teach robots new tasks (a limitation noted by the authors). But overall, the reviewers and AC found the method to be sensible, effective, and interesting to the wider robot learning community.

An important point raised during the AC/reviewer discussion was that of novelty and the paper's related work section. Authors state that they build upon [36], but claim that [36] “cannot easily scale to multi-step manipulation tasks with multiple stages of unstructured object interaction.” However, in a CoRL 2021 paper “Learning Multi-Stage Tasks with One Demonstration via Self-Replay”, [36] was indeed extended for multi-step tasks, with a simple modification. Reviewers did not cite this work and so there is less novelty in HYDRA than is claimed by the authors. We encourage the authors to address this in the final paper, and more clearly describe the relationship between HYDRA, [36], and the CoRL 2021 paper.
Decision: Accept (Poster)