Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes an interactive method to inglobate new demonstrations into an existing dataset and policy in such a way that it is not detrimental to the existing policy. The authors propose to do so by measuring how similar and compatible the new demonstration is with the existing policy, and discarding it in case the algorithm finds it too dissimilar. They also extend this technique to provide online feedback to a human operator on how to better record the demonstration.
The method is based on the assumption that, in a state already visited, the new demonstration should be close to the existing policy. In a novel state, however, new data is always good. The algorithm therefore computes at each step a novelty and consistency measure based on the existing policy and dataset.
strengths_and_weaknesses: I believe the problem this paper tackles is important, and methods to have interactive learning between robots and human may be beneficial. Although in the future we may want to learn complex policies from heterogeneous data, at the moment robot learning can't scale datawise as other fields, and therefore these techniques are still fundamental.
Strengths:
- Interesting problem, tackled with a simple but well designed approach. 
- The experiments section is well written, and indeed shows the effect of an incompatible additional dataset, showing that in this scenarios more data is not always good. Real robot experiments also further prove the effectiveness of the algorithm.
- The interactive algotithm they propose is interesting and potentially helpful especially at larger scale.
Weaknesses:
- Tasks are still relatively simple and do not involve the use of longer horizon skills.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I suggest to accept the paper. The method presented is intuitive but effective and can be beneficial also in larger scale settings. While the tasks do not show particularly complex skills, I believe they still prove the improvements that the algorithm brings, also in the real world.
issues: No particular issue. I would suggest for future extensions to also test it on more challenging, longer horizon tasks.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes an interesting paradigm in which, to improve the consistency among the multiple demonstrations of a task, a measure of compatibility is used to guide the demonstrator. This guidance results in a consistent set of demonstrations, which in turn results in increased success rates. As a result, by eliminating the heterogeneity in the demonstration set, the learning is more efficient since multi-modal models are not required, and at the same time, tasks become more prone to succeed. The method was demonstrated in simulation on a simple task of inserting a nut onto a peg, and in a real robot on a task of placing food on a plate.
strengths_and_weaknesses: The paper is clearly written and easy to follow. It has a straightforward structure. There are certainly merits of the idea of decreasing heterogeneity of the task and the paper motivates it quite well. Regarding the weaknesses, please, refer to the section on issues.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The experiments are too simple and do not seem to justify the complexity of the method. As it is, the narrative of the paper clashes with a body of literature on multi-modal learning but does not convincingly justify the paper's choice of approaching demonstration via loss of heterogeneity. I think there are many ways to improve the paper (see issues section) but this would require significant changes which are more appropriate for a re-submission.
issues: One of the main issues with this paper is that it is easy to counterargue two of the main claims on which the contribution of the paper hinges on. First, that by suppressing heterogeneity the method could be potentially moving itself towards collecting a poor, local minima prone dataset of demonstrations. Many works try to do the opposite, which is to learn the richness of human variability via multi-modal models. Second, if the system is already so informed to the point it is capable of measuring the demonstration adequacy and providing feedback, then what is the actual reason to collect more demonstrations in the first place?

To improve upon the first issue, the paper could leverage insights from the literature on, for example, virtual fixtures and virtual guides [r1, r2]. In virtual guides, experiments motivate well the fact that although users may provide multi-modal demonstrations, it is clear that only a single mode is allowed, for example, in surgical tasks.
The tasks used in the current paper could be reformulated such that it becomes more evident the need for uni-modality. For example, a more convincing story could be written by changing the sentence from "This multimodality is inconsequential to human users, with task variations manifesting as subconscious choices; for example, reaching down, then across to grasp an object, versus reaching across, then down." to something along the lines "Although non-trained users may find natural to demonstrate a task in a variety of ways, there are certain tasks for which a single pattern is more preferable. Such patterns can be captured initially by expert demonstrations which are then used to provide the metric M which will guide subsequent users on how to demonstrate."

To improve upon the second, it would be useful to show why the data used to compute the likelihood in M is not enough to solve the task. For example, would it be possible to show that the current demonstrations to place the egg on the plate are too uncertain and thus more demonstrations are required? I think a look into active learning on robot demonstrations [e.g. r3] can provide some interesting insights to motivate the need for demonstrations.

Minor comment
======================
Under excessive correction, will users become less prone to demonstrate? Wouldn't it be better to let them demonstrate freely, and then use M to give single final feedback on the demonstration over the entire episode, rather than provide high-frequency feedback at each time step?

[r1] https://www.telerobotics.utah.edu/uploads/Main/Abbott_RoboticsResearch07.pdf
[r2] https://hal.archives-ouvertes.fr/hal-02158827/file/main_final.pdf
[r3] https://proceedings.mlr.press/v78/maeda17a.html
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents an approach for measuring and actively soliciting "compatible demonstrations" in interactive imitation learning (IL) from multiple humans, where compatibility is defined relative to the behavior of the initial demonstrator. As opposed to prior work that handles the issue of suboptimal and multimodal humans in IL in a post-hoc manner (after the dataset has been collected), this work proposes to influence the data collection process itself toward the collection of more compatible demonstrations. They define a novel compatibility metric over state-action pairs as a synthesis of (1) action likelihood and (2) state novelty relative to the initial policy, and they apply this metric to both filtering data and actively soliciting compatible data. In user studies in simulation tasks and a real food-plating task, they show that the active method achieves higher success rates over a naive approach in all tasks.
strengths_and_weaknesses: This paper tackles an important problem in imitation learning, i.e., that of multimodal and suboptimal human demonstrations, especially in the multiple human setting. The high-level approach is compelling and novel: instead of the conventional approach of processing human data after it has been collected, the robot can actively influence the human to provide the data in a compatible manner. This is certainly a promising and intuitive strategy for mitigating the compatible demonstration problem. The paper is very well-written, clear, and easy to follow, and the figures are good for providing intuition as well. The experiments are convincing and include both real user studies and real robot experiments.

My main reservations are with the technical aspects of the approach. Firstly, although the overall framework of interactively querying for compatible demonstrations is novel, the actual mechanism for doing so is not particularly novel. Specifically, they propose using the likelihood of actions and the novelty of states under the base policy, neither of which is a novel metric. Secondly, the authors do not provide sufficient technical detail. I was not able to find the details of the likelihood and novelty computations in the main text or supplemental material: I’m assuming novelty is computed as the policy output variance across the ensemble of neural networks, but what does “likelihood” mean? Why does it only take the action as input, when likelihood seems to be a state-dependent measure (a particular action can be more likely at a given state than another one)? Furthermore, how exactly are the thresholds $\lambda$ and $\eta$ calculated, and what are the exact values they take in the experiments? Clarifications here would be appreciated. Thirdly, I have some reservations regarding the overall soundness of the approach. (1) In interactive IL, the policy is iteratively improved by new intervention data. Measuring compatibility against the base policy $\pi_{base}$ seems to prevent future policies from significantly outperforming the base policy, which runs counter to the objective of interactive IL. Is the $\pi_{base}$ here iteratively updated to $\pi_{new}$ when computing compatibility in later rounds of interactive learning? (2) Currently, high novelty demonstrations are deemed compatible, but this seems flawed - if multiple operators are providing demonstrations in novel regions of the state space, aren’t they free to provide demonstrations that are incompatible with each other?

Some more notes:
- Line 95: “these approaches fail in the sequential, interactive imitation learning setting that we study.” Is there justification or a source for this claim? I don’t readily see why an approach like Implicit BC wouldn’t work on the aggregated dataset $D_{new}$.
- Line 212: an incompatible state-action pair is said to have $\mathcal{M} = 0$. Isn’t $\mathcal{M}$ a continuous measure that’s unlikely to be exactly 0?
- Line 216: The feedback phase uses state-similarity to retrieve relevant snippets of demonstrations. How is this done in image space? L2 pixel distance? 
- The active informed algorithm makes several modifications compared to the naive baseline, but is only compared in a composite manner. It would be good to include some ablations or additional baselines for more convincing experimental results. For instance, experiments with only the prompting phase (showing operators the base demonstrations) to see if the demonstration and feedback phases are helpful.
- The real results section is quite short with only two quantitative numbers reported. It would be good to see more detailed metrics here (e.g., standard deviation or other quantitative results). Also, justification or intuition on why the performance gap is so much larger in the real experiment compared to the simulation experiments.

**Post Revisions:** The reviewers have adequately addressed most of my concerns and I have raised my assessment from Weak Reject to Weak Accept. However, some remaining concerns are that the method lacks algorithmic novelty and the real experimental results are a bit lacking.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Originally was a Weak Reject but after revisions I modify my recommendation to a Weak Accept.
issues: 1. Add more technical details to the text or appendix, especially on the computation of the likelihood metric.
2. Add more justification for the chosen compatibility metric, particularly how to address the high novelty yet incompatible case above.
3. Add a “Prompt-Only” baseline or similar baseline for the experiments with active solicitation.
4. Add more quantitative results from the real robot experiment data.
review_title: null

Meta-review: metareview: ### Strengths
- The paper tackles a very interesting problem with a sound method.
- Well written experimental section
- Real robot experiments and user study

### Weaknesses
- Showing that the method also works on more complex tasks would have been nice
- Enforcing consistency (rather than allowing multi-modality) needs to be motivated better
- Why being able to measure demonstration adequacy isn't sufficient by itself needs a better motivation, and should ideally also be shown experimentally
- Some important details need to be added

### Summary
A promising paper that needs some additional explanations and (ideally) experiments.

### After rebuttal and discussion
All reviewers replied and/or participated in the reviewer&AC discussion. The replies managed to clear up quite a few doubts. While not groundbreaking - in terms of novelty of the way of querying for demonstrations and experimental results - this is a solid, well-written paper tackling a relevant problem
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)