Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: In this paper the authors introduce expert guided policy optimization, an algorithm for safe reinforcement learning. In this method, it is assumed that some expert is available at training time which can output a distribution over expert actions given a state. At its core, the algorithm checks at every stage of training whether an action taken by the "student" will be dangerous or not by checking that it is contained within the confidence interval of the expert's distribution for some threshold nu. The authors then provide an upper bound for the training risk (the expected cumulative probability of failure) using this student-expert scheme. In order to train the student, the authors use a modified version of SAC with a modified loss function for the critic to account for the partial demonstrations provided by the expert. In their experience, the algorithm tends to exploit expert behavior, and therefore, they introduce the intervention cost c, which essentially penalizes expert intervention. This is done by introducing an additional critic which effectively keep track of intervention occurrence, and used as part of a soft constraint in the policy's objective. The authors test the framework on a self-driving environment and are able to obtain better cumulative rewards than similar methods whilst also having low training episodic cost.
strengths_and_weaknesses: Strengths:
* The paper is very clearly written.
* It provides a realistic experimental setup, even if it is in simulation. 

Weaknesses:
* It would have been nice to see the algorithm used in more than one driving simulation environment.
summary_of_recommendation: I believe this paper proposes an interesting framework for safe reinforcement learning with access to an expert. While there is only one driving environment being showcased, the experiments do provide significant evidence that this method outperforms other similar safe rl methods.
issues: My only concern is that only one environment was used to showcase the algorithm. It would have been nice to have more driving simulators to test the algorithm. Other than that I do not have any other major concerns I would like to raise at this time.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes the Export-guided Policy Optimization (EGPO) method aiming to achieve safe learning by combining regular learning with expert policy execution activated using a switch when detecting unsafe situation. The assumption is that a stochastic expert policy exists which can generate a safe action with high probability and also evaluate the probability of taking a given action by the expert. The switch function itself is just a check whether a proposed action has high confidence with respect to the expert model. If not, the expert takes over. 

To encourage learning, the frequency of interventions is then limited by adding a term penalizing the excess number of interventions using a Lagrangian multiplier. To improve stability, the multiplier is updated using a PID method. 

To train with off-policy expert data, Conservative Q-Learning (CQL) approach is used to improve the training on trajectories with partial expert demonstrations. 

The authors perform a study on a simulation autonomous driving task with several type of unsafe situation arising from other vehicles and traffic cones. The results show lower overall cost compared to other methods such as SAC-Lag and PPO-Lag. 


other minor comments:

line 52: "the Lagrangian method" is unclear when first introduced
line 151: "negative correlated" is unclear (one needs to read the appendix to understand what this means) 
is the policy equation (2) actually used in practice? it is not referenced later in the paper (except once but without concrete application)
strengths_and_weaknesses: Strengths:
+ a principled way to integrate expert policy in a learning framework
+ reasonably good performance in a simple simulated environment, compared to other existing methods
+ the authors seem to have also applied the method to more realistic mock-up setting (as shown in Appendix)

Weaknesses:
- it is not very clear whether the method works because the expert is really good and is trained on the exact same environment distribution, and so the learning process is cheating when executing the expert; it would have been interesting to see an expert only being activated in unsafe-only situations (as opposed to just any deviation from the expert policy, which the switch function is based on)
- the usage of Lagrangian multipliers and a PID controller is interesting, but also seem very heuristic and its performance is unclear
- the implications of Theorem 1 are not clear, would be great to refer to it in the actual algorithm, or to use in some constructive way; the term K_eta is also a bit of a mystery, can that term potentially be very large and if so, does this theorem upper bound serve any practical purpose?
summary_of_recommendation: While integrating an expert policy for safer learning is important, there are some unclear aspects of the approach (as listed in Weaknesses) which need to be addressed before assessing how successful it will be in practice. 
issues: - please answer the list of Weaknesses above
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents an approach for safe policy optimization in the expert-in-the-loop reinforcement learning setting. By assuming that the failure probability of the expert is bounded, the proposed approach shows that the cumulative probability of failure of the trained policy is bounded. Evaluations on a simulator for autonomous driving demonstrates that the approach can achieve high rewards with minimal costs compared to pure imitation learning baselines. 
strengths_and_weaknesses: Strengths

- The problem of safe policy learning is important and the solution strategy through human-in-the-loop RL is well-motivated. This is a compromise between pure RL methods that do not assume any expert intervention but are unsafe, and pure imitation learning methods that require high quality expert demonstrations and can be much safer in training. 

- The theoretical result of bounded training risk is interesting, and important. This guarantee is important to ensure that the safety of the trained behavioral policy will not diverge too much from the safety level of the expert. 

Weaknesses

- The definition of unsafe actions and states (for example in line 142) is very unclear. In most cases, for example a car that is gradually moving off the road, no single state is safe or unsafe and the trajectory of the car gradually progresses from safe to unsafe regions. So, it is important to clearly define how "unsafety"is quantified in the paper. 

- Theorem 1 has a potentially unbounded term. How large can the value of $K^'$ be in Theorem 1? It is important to upper-bound this term because if the term is large, then the bound on cumulative probability of failure becomes vacuous. 

- The way the "expert policy" is constructed doesn't seem realistic. In line 248-253, it is mentioned that the expert policy is trained with Lagrangian PPO for a long training time and causes a lot of safety violations during training. This is in a sense circumventing the safety problem because the expert is allowed to be unsafe in training. How will this generalize to a real world experiment? In particular, how will the corresponding expert policy be trained when the approach is transferred to a real-robot navigation task?
summary_of_recommendation: My main concern is with the training of the expert policy with a lot of safety violations. As such, the approach doesn't seem scalable to a real world setting, where safety is ultimately important. 
issues: Please address the issues listed under Weaknesses above (copied below for reference) and a few additional clarifications regarding experiments


- The definition of unsafe actions and states (for example in line 142) is very unclear. In most cases, for example a car that is gradually moving off the road, no single state is safe or unsafe and the trajectory of the car gradually progresses from safe to unsafe regions. So, it is important to clearly define how "unsafety"is quantified in the paper. 

- Theorem 1 has a potentially unbounded term. How large can the value of $K^'$ be in Theorem 1? It is important to upper-bound this term because if the term is large, then the bound on cumulative probability of failure becomes vacuous. 

- The way the "expert policy" is constructed doesn't seem realistic. In line 248-253, it is mentioned that the expert policy is trained with Lagrangian PPO for a long training time and causes a lot of safety violations during training. This is in a sense circumventing the safety problem because the expert is allowed to be unsafe in training. How will this generalize to a real world experiment? In particular, how will the corresponding expert policy be trained when the approach is transferred to a real-robot navigation task?

- What is the intuition for why offline RL safety violations are much better than Dagger and GAIL but worse than BC? Is the same expert data used to train the IL methods and the offline RL baseline?

- The method presented in the paper did not seem specific to autonomous driving. What are the challenges for applying this method to obtain safe learning policies in locomotion and manipulation tasks? It will be helpful to list out issues, if any.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper introduces a method for coupling data collected from an agent with the ones provided by an expert, that can override the actions of the agent in case of unsafe behaviour. The method is tested on an autonomous driving task and outperform a significant amount of baselines.
strengths_and_weaknesses: This paper proposes a method for involving an expert agent in the classical RL loop. The method is theoretically sound, and relatively easy to apply. I like the way the authors addresst the inherent issues of their vanilla method, and the way they propose theoretically grounded solutions to address them, e.g., the use of Conservative Q-Learning and the constrained optimization problem.

I think the paper is overall nice to read, but it has some typos that should be fixed:
- line 141: is a function THAT denotes;
- line 151: negativeLY;
- line 186: stabler -> more stable.
summary_of_recommendation: The paper introduces a method that I find overall interesting and practical. I find some limitations that I list below that I’d like to be addressed, possibly in the paper, and that motivate my score.
issues: - how does your method apply to problems with stochastic dynamics and/or partial observability?
- what do you think your method need to safely handle these problems?
- would the expert be considered as an “oracle” also in noisy partially observable problems? I think this would significantly limit the applicability of the method.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper introduces an expert guided reinforcement learning method, called EGPO, to learn safe driving policies. The expert functions as a safety guardian that will take over the control if the agent’s control action is not safe (i.e., the expert is less likely to take that action). Then the expert control will be learned using offline reinforcement learning technique, conservative Q-learning. To mitigate the problem that the learning agent may totally rely on the expert to take safe actions and do not learn how to drive, the paper introduces a method to minimize intervention using a Lagrangian method. The policy parameters and the Lagrangian multipliers are learned iteratively. In particular, the Lagrangian multipliers are updated using a PID controller.
strengths_and_weaknesses: Strengths:
This paper is well written and easy to follow. The video in the supplementary material greatly helped the reviewer in understanding the evaluation scenarios. The idea of using expert action to directly constraint the action of the learning agent is very interesting. Moreover, the paper explored methods to enhance the learning using offline RL and by minimizing the frequency of expert interference. These methods can be apply to general safe RL algorithms that use safe guardian (e.g., RL + CBF), to enhance their exploration efficiency.

Weaknesses:
- The assumption 1 (that requires the amount of unsafe actions for every state to be bounded in the expert policy) seems too strong. For example, in some unsafe states, it is possible that no action can be safe, in particular, when the action space is constrained. Moreover, it is unclear how the authors can verify this assumption for arbitrary expert policies. In the experiments, the expert is learned using Lagrangian PPO. Does this expert satisfy the assumption? Can this assumption be verified using the success rate (since later the paper uses the success rate the differentiate different expert policies, e.g., in figure 6)?

- In line 231, the paper states that the environmental cost is unavailable to EGPO. What is the motivation of making this assumption? It is understandable that the authors may wish to claim that even without the safety reward directly from the environment, the agent can learn to behave safely with the expert guidance. However, in autonomous driving, this is not a realistic assumption since environmental safety reward is “common sense” and easy to obtain. Moreover, the nature of environmental safety reward is difference from the nature of the expert guidance. The former provides a constraint on the state space, while the latter provides a constraint in the state space. It would be more convincing if the paper demonstrates that the imitation of expert still provides strong performance margin when the environmental safety reward is considered and that how the environmental safety reward affects the amount of expert intervention. 

- The authors mention that they plan to extend the method to include human demonstration. How can we obtain a stochastic policy \mathcal{E} in this case?

- In table 1, why is the cumulative cost for the expert higher than that for PPO-Lag for safe RL, offline RL, IL, and the proposed method? For IL and the proposed method, since their safety notions solely comes from the expert demonstration, the results seem counter intuitive. For PPO-Lag in safe RL, is it using the same training method as the expert? Is the difference caused by insufficient training? 

- In table 2 (d) and (e), the performance is comparable to EGPO considering the uncertainty range. Does it mean that the proposed method is less sensitive to CQL loss and environmental reward? Why is it the case?

- This paper claims that the proposed method can make learning more efficient. However, the efficiency of the proposed method should also account for the time spent to obtain the expert policy (whether it is a pretrained policy or from human-demonstration). 
summary_of_recommendation: While the paper presents an interesting idea, the major concerns the reviewer has is:
1. the results need more explanation;
2. the assumptions need further justification;
3. how the method can be applied to real world applications, especially how to obtain the demonstration efficiently and how to ensure the expert policy satisfies the assumptions, needs better explanation. 
issues: Please check the comments above.

There is a typo in (1). It should be if a\in A, choose (a,0); otherwise, choose (a^E,1).
review_title: null

Meta-review: The authors have sufficiently addressed reviewers' concerns; specifically, the new human-in-the-loop experiment was particularly insightful. I recommend an acceptance.
Decision: Accept (Poster)