Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this work, the authors propose a new algorithm for data-efficient human-in-the-loop learning, Human-AI Copilot Optimization. The main idea is to have experts intervene during training in cases in which unsafe situations arise. The HACO learned policy utilizes a multi-task objective: doing well relative to a learned value function (based on human interventions), keeping an exploratory policy, and keeping human interventions at a minimum. Experimentally, HACO seems to be able to drastically reduce the amount of number of environment training timesteps required to reach basic competencies to the agent in the test environment, while maintaining good task and safety performance.
main_review: ### Quality

The paper's results are quite interesting, and the experiments – as far as I can tell – seem well executed. I particularly appreciated the additional ablations provided in Table 2. The method seems relatively well motivated. The method proposed seems to do very well relative to the chosen baselines along the performance metrics examined.

### Clarity & Limitations
Stylistically the paper could use more editing. There are various typos (some below) and many phrasings that could be improved. 

There are also various things that could be improved in the results' clarity, or which seem to constitute limitations:
- Who is intervening? 3 experts – how were they selected? While the training is happening, what are the experts doing? "The whole training takes about 50 minutes." Is the expert providing actions for the whole time?
- "The main experiments of HACO repeat 3 times." -> what does this mean? With each human expert, you perform one training run?
- Generally would be helpful to give more intuition as to why IL is being outperformed with much less data?
- "We split the driving scenes into the training set and test set with 50 different scenes in each set. At the beginning of each episode, a scene in the training or test set is randomly selected." -> The latter part of this phrase seems to suggest that you're training on the test set?
- I found it confusing how in Section 3.3, different costs and rewards were defined one by one, without a high-level motivation. Before describing each one, I would state clearly that you are considering _various different metrics of performance_: "Test Return", "Test Cost" (which is different than just negative returns!), and "Test Success Rate". I would potentially propose changing the name from "Cost" to "Safety Violations", and from "Success Rate" to "Goal completion" to make it easier to disambiguate what the motivation between using these multiple metrics is. As currently phrased, they all seem synonymous with "reward" (or the opposite of it) on a first read.
- In Section 3.3, I would switch to introduce cosine similarity first, and then say that in ideal case this is not necessary. I found presenting things in the current order more confusing.
- The sentence "Frequent querying might exhaust the human expert and brings tremendous cognitive costs (Zhang et al., 2021)" is almost verbatim repeated both in section 3 and in section 2. I would remove it from 2.
- "Under the protection from the human expert, HACO yields only 30.14 total training cost in the whole training process, an order of magnitude better than the other baselines," -> if I'm reading Table 1 correctly, it's essentially 2 orders of magnitude better? The next best value seems to be 1840
- Figure 3: it might be useful to have zoomed in (on the x axis) training curves in the appendix. Are the shadings of the training curves standard errors?

A limitation seems to be that of implicit assumptions made by the method: using CQL as in eq. (3) is making relatively strong assumptions about the nature of expert intervention. While it seems to work well in practice, the form of eq. (6) also seems quite arbitrary, and it's unclear how one would extend it to other environments (e.g. with discrete action spaces).

Additionally, the lack of human-in-the-loop baselines seems potentially problematic – after all, the method is presented as a improvement in that line of work. The same could be said about the environment – given how promising this method seems to be in this context, it would be very interesting to see how it performs in other environments. While I realize that this is not something that can be addressed in the timeframe of rebuttals, it would significantly strengthen the paper.

Typos:
- "To encourage the exploration in the area that does not _violating_ human intention,"
- "Human-Gated DAgger (HG-DAgger) (Kelly et al., 2019) utilizes an expert to _intervene exploration_ and"
- "Other forms of human participation _includes_ providing human preferences"
- "rewarding the state-action pairs deduced _by human_ and penalizing those _by agent_."
- "The speed reward _is vehicle’s_ current speed"
- "Since we test different approaches in the driving simulator, no injury would _happens_."

### Originality

Although I'm not an expert in this area, as far as I know, this seems like a novel contribution.
summary_of_the_review: Overall, the novel method presented by this paper, HACO, seems to be able to greatly outperform safe RL, IL, and Offline RL baselines – leading it to being able to achieve relatively similar safety and performance metrics while incurring in many less unsafe situations during training time, and while using 2 orders of magnitude less data than RL approaches. While some portions of the paper and motivations for the method lack clarity and baselining is limited to non human-in-the-loop methods, the strength of the empirical results still suggests that the submission is potentially worth accepting.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes HACO, a human-in-the-loop reinforcement learning method that safely trains an agent to imitate expert behavior while minimizing the number of expert interventions required. The key idea is to have a human watch over the agent (e.g., in a simulated driving environment), and take control whenever the agent enters unsafe states. HACO uses offline RL to train the agent to imitate the actions taken by the human during these interventions. To discourage the agent from intentionally visiting unsafe states in order to trigger human interventions, HACO also assigns a negative reward to the state transitions preceding a human intervention. Experiments with human participants in a simulated driving task show that HACO trains the agent to achieve higher success rates (in a test environment, without a human in the loop) than baseline methods based on imitation learning and offline RL, while requiring less training data and incurring a lower cumulative training cost.
main_review: Overall, I enjoyed reading this paper and found its results convincing. However, there is one missing experimental comparison that I would like to see before I can recommend an accept: prior work on human-in-the-loop imitation learning [2] proposes the Intervention-Weighted Regression (IWR) method, which tackles the same problem under the same assumptions as HACO. While HACO directly penalizes the agent for triggering human interventions, IWR takes a different approach based on dataset balancing. To judge the relative performance and contribution of HACO, it would be helpful to implement and run IWR on the driving task in Table 1 and Figures 3-4.

Missing related work:
 1. [Learning from Interventions](http://www.roboticsproceedings.org/rss16/p055.pdf)
 2. [Human-in-the-Loop Imitation Learning using Remote Teleoperation](https://arxiv.org/abs/2012.06733)

Minor comments:
 - Is there a typo in Equation 3? As written, the objective evaluates to zero. I'm also confused as to why offline RL is necessary to fit the Q-function, since we do not have a reward function in this setting and instead have partial demonstrations. Can't we just use behavioral cloning to fit an imitation policy?
 - Why does the training cost go *up* over time in the left panel of Figure 3?

Update
----
Thank you to the authors for adding the comparison to IWR. I have increased my score.

summary_of_the_review: Missing crucial comparison to prior method
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a method for driving policy learning based on human-AI copilot. The algorithm learns from human interventions and also tries to minimize the total efforts of human intervention. Comprehensive experiments and comparisons with multiple baselines show that the proposed algorithm can achieve high sample efficiency and reduce unsafe events. The contributions are in the design of the copilot learning method.
main_review: Strengths:
1. This paper is well written with clear logic and goals. The experiments are comprehensive. The proposed method is compared with several types of baselines including RL, safe RL, offline RL, and IL. 
2. The method proposed for co-learning and intervention minimization is solid. 

Weakness:
1. The description of the method can be improved. There seem to be some errors in the equations. For example, in equation (3), the two Q(s, \hat{a}) are the same. 
2. I am not sure how to determine the convergence of the algorithm. Why HACO has much fewer steps than the baselines. Especially during the testing phase, why HACO also has fewer steps?
3. The main part of the methodology is based on batch RL. The adding of intervention minimization seems to be a marginal contribution. 
4. The driving tasks are considered to be too easy. 
summary_of_the_review: The paper is solid and clear, with enough experimental support. I think it is higher than the bar.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents HACO, a human-in-the-loop learning algorithm that aims to learn imitative driving policy while minimizing the number of human interventions. HACO builds on CQL and operates under the no-reward assumption. HACO learns a proxy action-value function by penalizing the policy’s action and maximizing during human interventions. It additionally adds an entropy term to encourage exploration. The policy trains by maximizing the proxy action-value, and penalizing an accumulative intervention cost, computed using the cosine difference between the human and the policy actions. HACO is evaluated in a closed-loop driving simulator. HACO outperforms the selected imitation and offline RL baseline and is on par with RL methods which have access to environment rewards. It is also orders of magnitude more sample efficient than standard RL methods.
main_review: === Strengths ===

+ The paper is well written and the presentation is easy to follow.
+ The overall idea and the human-in-the-loop setup are nice and suit very well with real-world applications.
+ The presented approach is simple and effective, and has strong experimental performance compared to IL and is on-par with RL, and does not require extra reward function. It is much more sample efficient than the common RL approaches.
+ The paper also presents a novel driving simulator with procedurally generated maps and active agents. The authors mention that the simulator will be released, which is a big plus.

=== Weaknesses ===

- I like the analysis that justifies the design choice of equation 6, but I think the proposed solution is slightly overfitting to the environment (for example, steering which the environment uses), since in general action space does not necessarily form a metric space.
- There is a typo in equation 3.
- The authors mention the method builds upon CQL, however judging from equation 5 it does not seem to strictly follow CQL, but looks more like a standard Q learning loss. I would appreciate it if the authors can clarify this.

summary_of_the_review: This paper presents a simple yet very effective human-in-the-loop learning algorithm. The idea of minimizing human intervention is nice, and the presented approach has strong experimental performance. The novel simulator on which the presented approach is evaluated will be released, which is another plus. I recommend acceptance for this paper.
review_title: null

Meta-review: The paper presents a new algorithm for augmenting RL training with human examples, and this is applied to learning safe driving policies. This algorithm is properly tested and compared to other relevant algorithms with favorable results. Reviewers agree that this is good work and that it should be published. Reviewers had multiple questions, which were in my opinion answered satisfactorily by the authors. Notably, the authors ran additional tests against other human-in-the-loop RL algorithms with good results. In sum, this seems to be a solid paper, worth accepting.
Decision: Accept (Poster)