Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper introduces another interactive learning method based on data aggregation, within the family of the DAgger algorithms. The method is able to control when there should be a switch between the autonomous control and the human teleoperation based on two different measures: uncertainty about the visited state, and probability of success which is computed similarly to a value function, assuming that there is a sparse reward only in a target state. The approach reduces the load of the teachers, since they don't have to keep full attention to the robot while it is learning, because the active learning scheme requests the new demonstrations according to the intervention criteria.
strengths_and_weaknesses: - The paper is very easy to read, it is a pity not seeing the pseudocode in the main document, however, the text gives a clear explanation of its flow. The experiments cover well most of the questions that could come up about such a method. Below there are some comments with questions that can make the paper more complete.
- All this family of DAgger methods that have been developed incrementally (EnsembleDAgger, SafeDAgger, HG-DAgger, LazyDAgger), work under the strong assumption that the user is perfect and never makes mistakes. What happens when the teacher makes wrong demonstrations often? how is the performance of the learning process affected?
- Could it really be claimed that the novelty property is a contribution?
- How does the method behave at the beginning, when most of the states do not have a high Q value due to the failures of the learning policy, but the budget of human requests is limited, does it go to the extreme cases?
- How does the method deals with situations in which the human wants to intervene in order to change the current policy, but the robot is not switching to the teleoperation mode? It seems that a successful policy would not request feedback from the human, however, the method still should allow customizing the policies to the users' preferences.
What happens in tasks wherein the objective is not easily defined by the convergence to a set of target states? For instance, a biped robot learning to keep balance and walk? 
- What kind of interface was used for the real robot experiment? what kind of considerations should be taken regarding the Human-Robot interface? These methods cannot be completely isolated from the human factors, therefore, these details are as important as the learning method implementation.
- Authors should elaborate the idea in line 236. It is not completely clear why negative results are discarded.
- The word "Berkeley" is really not needed to be mentioned, when describing the participants of the experiment. The review process is double-blind, however, this unnecessary information makes it clear what group the paper comes from. I really disliked this point.


summary_of_recommendation: The presentation of the method is clear, the experiments cover almost a complete evaluation of the proposed approach. There are many questions that should be approached, but I think the authors do not need to reconsider the method or the paper structure in order to address them.
issues: See Strengths And Weaknesses
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper presents ThriftyDAgger, an approach to reduce the burden on human operators during interactive imitation learning. To achieve this, the authors estimate both the novelty and the riskiness of states during execution---the latter being the most significant of their contributions---and only request human intervention when either prediction deems it necessary. The authors demonstrate good performance on a variety of simulated and real world task, achieving state-of-the art performance in both while simultaneously reducing the burden on human operators.
strengths_and_weaknesses: 
Overall, this is a solid, well-written paper and (despite a somewhat narrow theoretical contribution) the experiments are impressive and reinforce the utility and broad applicability of the technique. Among the experiments, perhaps the most impressive are those included in Table 2: Three-Robot Fleet Control User Study Results. These results do an excellent job of showing how human operators can simultaneously train a small fleet of agents while also trying to perform a task of their own. These results show the improvements afforded by ThriftyDAgger and were immediately convincing.

I have a number of comments and suggestions that, if addressed, would help improve the clarity of the work.

First, while it is certainly advantageous that ThriftyDAgger achieves a lower average number of interventions (for this section, I focus on the results shown in Table 1), it seems that the LazyDAgger has a similar "gating mechanism" to reduce the amount of context switching---and therefore the number of interventions. What do the authors hypothesize is the mechanism that allows for a lower number of interventions for ThriftyDAgger? (Is it perhaps what the authors refer to as the "automatic parameter tuning"?) Relatedly, how should we think about direct comparison between the different approaches for this "number of interventions" metric, since I expect it depends quite heavily on the parameter selection? My intuition is that simultaneously achieving lower numbers of interventions and also higher success rates is the key takeaway, but some clarity on the subject from the authors would be helpful.

Second, I understand why the authors use the phrase "Automatic Parameter Tuning" to describe how the risk and novelty are evaluated, yet I am not sure that this is a particularly appropriate name for this process since the parameters for this process are (technically) not changing, and instead the process by which novelty and risk are defined depend on statistics of the data. Instead, it might be more transparent to present the approach in this way, as the risk and novelty online as data is collected. The title of Sec 4.4 could be something like "Computing Risk and Novelty Thresholds from Data": more verbose, but also a better reflection of what the system is doing. In the text, the authors can still claim that this method avoids the need for oft-sensitive parameter tuning, even if one might not regard it as a "parameter tuning" approach. I am open to debate on the subject.

Finally, while I appreciate the definitions provided in Eqs (2) and (3), I am unsure that the definition in equation (3) comes up again, since this does not appear to be the optimization objective used to compute the expected policy. Could the authors clarify the relation between the process used to learn the policy and the objective defined in Eq. (3)? Similarly, while Eq. (2) is used to build intuition, the "burden" does appear to be directly calculated anywhere; this should be mentioned in Sec. 3.

summary_of_recommendation: This paper presents a novel idea that is particularly well-executed. Though the central contribution of the paper is not overwhelmingly transformative, they have put in considerable effort to clearly communicate their contribution in relation to other work and empirically demonstrate its benefits.
issues: I enumerate my suggested changes and provide justification for them in the "Strengths and Weaknesses" section above. In summary, they are as follows:

1. Inclusion of a clearer discussion of what is implied by the lower number of interventions for their technique and how those results should be interpreted to compare the different approaches.
2. Clarifying the discussion of the "Automatic Parameter Tuning" (including potentially changing the name of the relevant section and idea).
3. Some clearer language surrounding the theoretical motivation.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper addresses the problem of robot-gated interactive imitation learning, where a robot actively cedes control to a human operator in situations where it requires action supervision. Compare to prior work, the primary novelty is the inclusion of a "risk" term (in addition to a "novelty" term) to determine when the robot should cede control to the human operator, or take control back from the human operator. The method is shown to outperform baselines in terms of success rate and amount of human effort needed on a peg insertion task in simulation and a cable routing task on a physical robot. A user study is also conducted, where users attempt to focus on a pattern matching game, while also trying to provide interventions (as requested) by a fleet of 3 robot arms that are trying to learn the peg insertion task in simulation.
strengths_and_weaknesses: Strengths

The paper is well-written and tackles an important problem. The proposed method is shown to compare favorably to several baselines on a simulated peg insertion task and a real cable routing task, in terms of the amount of human effort needed to learn the task, and the final performance of the trained model. The included user study across 10 users that each try to assist a fleet of 3 robot arms is also very interesting as a proof-of-concept for the proposed method. The parameters of the switching policy (which are usually manually specified in prior work) are determined from a single hyperparameter that correlates to the budget for the expected amount of effort a human is willing to provide in training the robot -- this is a significant improvement over other methods. The attached video also provides a useful overview.

Weaknesses

The two main weaknesses are the lack of novelty and breadth of the experiments. The main difference from LazyDagger appears to be the inclusion of a risk term, and automatic hyperparameter tuning. That being said, the risk term appears to empirically make a large difference, but there isn't as much intuition or justification provided in the paper for why it makes such a large difference. Furthermore, it's unclear why the proposed risk term works well - the risk term should vary naturally during the episode, by state. At the beginning of the episode, there are several timesteps left to reach the goal, and towards the end, there will be few. How can this term distinguish between states that are far away from the goal, and states that will never reach the goal? Providing some concrete qualitative examples of states along with the value of the risk term at those states might be useful to understand this. 

Also, while the number of baselines is high, the number of tasks used for evaluation is lacking - only one task in simulation and one in real is used for evaluation. It would be useful to demonstrate the effectiveness of the method on at least one more simulated task, and preferably one that is more challenging than the peg insertion task (the fitting appears to be relatively high tolerance).  

Additional comments follow:
- Reporting additional metrics could be beneficial to provide a more complete picture, with respect to the comparison to BC. Table 1 would ideally report the number of episodes / total number of samples used for training each algorithm (both human and robot samples). Including information on the number of successful and unsuccessful trajectories used for each method would be useful as well (perhaps in the supplementary), since the proposed method is able to leverage unsuccessful trajectories to train the risk metric (value function). It might also be useful to report other metrics like wall clock time - there's likely a tradeoff here (for example, perhaps 100 demonstrations could be collected in 1 hour of human time, allowing the robot to match the success rate of the proposed method, which might use 15 minutes of total human time, spread out over 3 hours. The proposed method would take more overall human + robot wall clock time, but less human time). 
- The paper concludes that the proposed method outperforms HG-Dagger, after presenting the results in Table 1 on the peg insertion task. However, as mentioned previously, there should be at least 1 more task in simulation, to make the claim more solid, as the performance gap appears to be much tighter on the real world task. Furthermore, while the proposed algorithm outperforms HG-Dagger in the user study, the setup there was already biased against HG-Dagger, since participants must actively intervent on 3 robots operating at once (which is definitely cognitively demanding). Including further results in the single robot case across multiple users would also substantiate the claim on outperforming HG-Dagger further.
- From the supplementary material, it seems like every action in the peg insertion task is followed by 10 zero actions, in order "to avoid bias due to variable teleoperation speeds". This sounds like an unconventional design choice that could impact the difficulty of the task and could merit further discussion and motivation. This might also explain why the motion of the robot appears stilted in the included video (although that could also be an artifact of the teleoperation interface).
- What happens if the algorithm cedes control to the human, but the robot is in a state in which it is difficult or impossible to recover? Does the human just choose to terminate the episode? How often does this occur in practice? 
- Line 328 - typo - "at which solicit interventions"
summary_of_recommendation: While the proposed method has some favorable characteristics and is shown to outperform baselines, the method is similar to prior work except for what appears to be one critical component (using the risk metric in determining when to switch between robot and human). Further discussion of this critical component would be beneficial. The breadth of the empirical evaluation is also limited.
issues: Discuss the role of the risk term - why it is expected to help significantly, especially in the context of prior work that does not consider it (like LazyDagger and SafeDagger).
review_title: null

Meta-review: This paper proposes an approach for generating queries for supervision from humans in an imitation learning context. The primary idea is to consider novelty of a state and the associated “risk” (i.e., likelihood of hindered task progress) in order to minimize context switching for the human supervisor. The resulting approach (ThriftyDagger) is demonstrated on simulation and hardware experiments involving manipulation tasks. In addition, results of a human user study on using the approach for controlling a group of three simulated robots is presented.

Strengths:
+ The reviewers agree that the inclusion of the risk term in deciding when to make queries to the human supervisor is novel.
+ The reviewers generally agree that the experimental evaluation of the approach is very thorough and convincingly demonstrates improvements in comparison to various baselines approaches.
+ The paper is very clearly written and the technical exposition is crisp. 
+ The approach only requires choosing one hyperparameter (the context switching rate); other parameters are chosen based on this. 

Weaknesses:
- The primary technical novelty of the approach is in the inclusion of the risk term. The reviewers request more justification and intuition for why this term helps the empirical performance (see the comments from Reviewer pnHZ and PRtc for more specific points to discuss).
- Reviewer PRtc noted that demonstrating the approach on a more challenging simulation task could be beneficial (especially since the performance gap between the proposed approach and baselines is smaller on the hardware experiments).

Suggestions:
I urge the authors to consider the reviewers’ detailed feedback in order to further improve the paper. The primary suggestion for improvement is to add more discussion and intuition on the benefits of including the risk term (which is the primary technical novelty in the approach). The reviewers also requested a few additional details and explanations, and suggested changing the phrase “automatic parameter tuning” to something more representative (e.g., “computing risk and novelty thresholds from data”).

----- Post rebuttal -----

The authors have addressed the reviewers' primary concerns during the rebuttal phase. In particular, the authors have (i) added intuition on why the risk term helps,  (ii) added a baseline (picking and stacking), and (iii) provided further details and clarifications. The revised paper is stronger as a result, and the reviewers are in agreement that the paper constitutes a strong contribution and should be accepted. 

Decision: Accept (Oral)