Safe Driving via Expert Guided Policy Optimization
Authors: Zhenghao Peng
Quanyi Li
Chunxiao Liu
Bolei Zhou
Venue: CoRL2021 Poster (2021)
Keywords: Safe Reinforcement Learning
Human-in-the-loop
Imitation Learning

Abstract: When learning common skills like driving, beginners usually have domain experts standing by to ensure the safety of the learning process. We formulate such learning scheme under the Expert-in-the-loop Reinforcement Learning where a guardian is introduced to safeguard the exploration of the learning agent. While allowing the sufficient exploration in the uncertain environment, the guardian intervenes under dangerous situations and demonstrates the correct actions to avoid potential accidents. Thus ERL enables both exploration and expert's partial demonstration as two training sources. Following such a setting, we develop a novel Expert Guided Policy Optimization (EGPO) method which integrates the guardian in the loop of reinforcement learning. The guardian is composed of an expert policy to generate demonstration and a switch function to decide when to intervene. Particularly, a constrained optimization technique is used to tackle the trivial solution that the agent deliberately behaves dangerously to deceive the expert into taking over. Offline RL technique is further used to learn from the partial demonstration generated by the expert. Safe driving experiments show that our method achieves superior training and test-time safety, outperforms baselines with a substantial margin in sample efficiency, and preserves the generalizabiliy to unseen environments in test-time. Demo video and source code are available at: https://decisionforce.github.io/EGPO/

Safe Driving via Expert Guided Policy Optimization
Zhenghao Peng",, Quanyi Li Chunxiao Liu', Bolei Zhou' The Chinese University of Hong Kong, $ SenseTime Research, Centre for Perceptual and Interactive Intelligence
When learning common skills like driving, beginners usually have do- main experts standing by to ensure the safety of the learning process. We formulate such learning scheme under the pppert-i--the-loyy Reinforcement Learning where a guardian is introduced to safeguard the exploration of the learning agent. While allowing the sufficient exploration in the uncertain environment, the guardian intervenes under dangerous situations and demonstrates the correct actions to avoid potential accidents. Thus ERL enables both exploration and expert's 's partial demonstration as two training sources. Following such a setting, we develop a novel Expert Guided Policy Optimization (EGPO) method which integrates the guardian in the loop of reinforcement learning. The guardian is composed of an expert policy to generate demonstration and a switch function to decide when to intervene. Particularly, a constrained optimization technique is used to tackle the trivial solution that the agent deliberately behaves dangerously to deceive the expert into taking over. Offline RL technique is further used to learn from the partial demonstration generated by the expert. Safe driving experiments show that our method achieves superior training and test-time safety, outperforms baselines with a substantial margin in sample efficiency, and preserves the generalizabiliy to unseen environments in test-time. Demo video and source code available at: https: / /ecci ceneciisiooooreee github. iooii////s /GPO/.
Abstract:
Keywords: Safe Reinforcement Learning, Human cilman-in-taen thot Machine Learning, Autonomous Driving
Introduction
Reinforcement Learning (RL) shows promising results in human-interactive applications ranging from autonomous driving [1]. the power system in smart building [2], to the surgical robotics arm [3]. However, training and test time safety remains as a great concern for the real-world applications of RL. This problem draws significant attention since the agent needs to explore the environment sufficiently in order to optimize its behaviors. It might be inevitable for the agent to experience dangerous situations before it can learn how to avoid them [4], even the training algorithms contain sophisticated techniques to reduce the probability of failures [5, 6, 7].
We humans do not learn purely from trial-and-error exploration, for the sake of safety and efficiency. In daily life, when learning some common skills like driving, we usually ensure the safety by involving domain expert safeguard the learning process. The expert not only demonstrates the correct actions but also acts as a guardian to allow our own safe exploration in the uncertain environment. For example as illustrated in Fig. I, when learning to drive, the student with the learner's permit can directly operate the vehicle in the driver's 's seat while the instructor stands by. When a risky situation happens, the instructor takes over the vehicle to avoid the potential accident. Thus the student can learn how to handle tough situations both from the exploration and the instructor's s demonstrations.
In this work, we formulate such learning scheme with Expert-in-the-loe RL (ERI.. As shown in the right panel of 1ig.1, ERI. incorporates a guardian in the interaction between agent and environment. The guardian contains a switch mechanism and an expert policy. The switch decides to intervene the free exploration of the agent in the situations when the agent is conducting unreasonable behaviors or
"Zhenghao Peng and Quanyi Li contribute equally to this work.
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

Expert
Loarner
Guardian
Expert
Environment
Agent
d
Switch
Modicheed Action &
Dangerous Situation
Expert Intervention
Expert-in-the-lopp Reinforcement Learning
The expert intervenes the learner in dangerous situations. We model it through the Expert- in-the-loop RL scheme on the right panel where a guardian introduced in the loop of the interaction between agent and environment.
Figure 1:
a potential critical failure is happening. In those cases the expert takes over the main operation and starts providing demonstrations on solving the task avoiding dangers. Our setting of ERL extends previous works of Human-in-the-loop RL in two ways: First, the guardian inspects the exploration all the time and actively intervenes if necessary, instead of passively advising which action is good [8] or evaluating the collected trajectories after the agent rolling out [9, 10]. This feature guarantees the safe exploration in training time. Second, the guardian does not merely intervene the exploration and terminate the episode [1 I]. instead, it demonstrates to the agent the correct actions to escape risky states. Those demonstrations become effective training data the agent.
Following the setting of ERL. we develop a novel method called Expert Guided Policy Optimization (EGPO). EGPO addresses two challenges in ERL. First, the learning agent may abuse the guardian and consistently cause intervention so that it can exploit the high performance and safety of the expert. To tackle this issue, we impose the Lagrangian method on the policy optimization limit the intervention frequency. Moreover, we apply the PID controller to update the Lagrangian multiplier, which substantially improves the dual optimization with off-policy RL algorithm. The second issue is the partial demonstration data collected from the guardian. Since those data is highly off off-policy to the learning agent, we introduce offline RL technique into EGPO stabilize the training with the off-polli off-policy partial demonstration. The experiments show that our method can achieve superior training safety while yielding a well- pelforerfon policy with high safety in the test time. Furthermore, our method exhibits better generalization performance compared to previous methods.
As summary, the main contributions of this work
(1) We formulate the Expert-in-the-loe RL (ERL) framework that incorporates the guardian as a demonstrator as well as a safety guardian. (2) We develop novel ERL method called Expert Guided Policy Optimization (EGPO) with practical implementation of guardian mechanism and learning pipeline. (3) Experiments show that our method achieves superior training and test safety, outperforms baselines with a large margin in sample efficiency, and generalizes well to unseen environments in test time.
2 Related Work
Safe RL. Learning RL policy under safety constraints [12, 13, 7] becomes an important topic in the community due the safety concern in real-world applications. Many methods based on constrained optimization have been developed, such as the trust region methods [5], Lagrangian methods [5, 6, 14], barrier methods [15,, 16], Lyapunov methods [4, 17], etc. Another direction based on the safety critic, where an additional value estimator is learned to predict cost, apart from the primal critic estimating the discounted return [7, 18]. Saunders et [11] propose HIRL, scheme for safe RL requiring extra manual efforts to demonstrate and train an imitation learning decider who intervenes the endangered agent. Differently, in our work the guardian does not terminate the exploration but instead continues the trajectory with the expert demonstrating the proper actions to escape risky states. However, majority of the aforementioned methods hold the issue that only the upper bound of failure probability of the learning agent can be guaranteed thcoretically, but there is mechanism to explicitly ensure the occurrence of the critical failures. Dalal et al. [19] assume that cost function is the linear transformation of the action and thus equip the policy network with safety layer that can modulate the output action as an absolutely safe action. The proposed EGPO utilizes the guardian to ensure safe exploration without assuming the structure of the cost function.
Learning from Demonstration. Many works consider leveraging the collected demonstrations to improve policy. Behavior Cloning (BC) [20] and Inverse RI. [21] uses supervised learning to fit the policy function or the reward function respectively to produce the same action as the expert. GAIL [22, 23, 24] and SQIL [25] ask the learning agent to execute in the environment and collect

---

trajectories to evaluate the divergence between the agent and the expert. This exposes the agent to possibly dangerous states. DAgger [26] periodically queries the expert for new demonstrations and is successfully applied to extensive domains [27, 28]. Recently, offline RI. draws wide attention which learns policy from the dataset generated by arbitrary policies [29, 30, 31]. The main challenge (OOD) actions [30]. Conservative Q-Learning (CQL) [32] addresses the impact of OOD actions by learning a conservative Q-function to estimate the lower bounds of true Q values. In this work, we use CQL technique to improve the training the trajectories with partial demonstrations given by the guardian.
of offline RL is the out-of-distributis
RL. An increasing number of works focus on incorporating human into the training loop of RL. The human is responsible for evaluating the trajectories sampled by the learning agent [9, 33, 10], or being a consultant to guide which action to take when the agent requests [8]. Be- sides, the human can also actively monitor the training process, such as deciding whether to terminate the episode if potential danger is happening [34, 11]. Human-Gated DAgger (HG-DAgger) [27] and Expert Intervention Learning (EIL.) [35] utilize experts intervene exploration and carry the agent to safe states before giving back the control. However, it is much less explored in previous works on how to (1) optimize the agent to minimize interventions, (2) efficiently utilize the data generated in free exploration and (3) learn from the takeover trajectories given by the expert. Addressing these aforementioned challenges, our work is derived from the Human-in-the-looe in-the-looop framework where the guardian plays the role of human expert to provide feedback to the learning agent.
Expert Guided Policy Optimization
Extending the setting of Human-in-the- RI., we frame the Expert-in-the-loop RI. (ERI.) that incorporates the guardian to ensure training safety and improve efficiency. We develop novel method called Expert Guided Policy Optimization (EGPO) to implement the guardian mechanism.
3.1 Overview of the Guardian Mechanism
Taking learning to drive as a motivating example, gen- erally speaking, the student driver learns the skills of driving from the instructor through two approaches: (1) Student learns from instructor's 's demonstrations. At the early stage of training, the student observes the demonstrations given by the instructor and learns rapidly by imitating the behaviors. Besides, the stu- dent also learns how the expert tackles dangerous situations; (2) Student in driver's seat operates the vehicle in an exploratory way while the instructor serves as guardian. The student can explore freely until the instructor conducts takeover of the vehicle in dangerous situations. Therefore, the student learns to drive from both the imitation of the expert and the free exploration.
State
Agent Action
Modulated intervention Cost
Expert
T(,, 8)
Expert Action Destritution
Whether (4)2
Yes
3 0--0)e--1
Figure 2: Flowchart of the guardian mecha- nism.
Based on this motivating example, we have the framework of Expert-in-the-lopp RL (ERL). As illustrated in the right panel of Fig. 1, we introduce the component of guardian on top of the conven- tional RL scheme, which resembles the instructor who not only provides high-quality demonstrations to accelcrate the learning, but also safeguards the exploration of agent in the environment. In the proposed EGPO method, the guardian is composed of two parts: an expert and switch function.
The expert policy
_ E-|ss can output safe and reliable actions a in most of the time. Besides, it can provide the probability of taking action produced by the agent: E(als) E [0, 1]. This probability reflects the agreement of the expert on the agent's 's action, which serves as an indicator for intervention in the switch function. We assume the access to such well well-performing expert policy. The switch is another part of the guardian, which decides under what state and timing the expert should intervene and demonstrate the correct actions to the learning agent. As shown in Fig. 2, the switch function T considers the agent action as well as the expert and outputs the modulated action fed to the environment and the intervention occurrence è indicating whether the guardian is taking over the
3

---

control:
(a,0), ( ((a E((ss) otherwise,
a E An
T(s, a, (, a, (a,
(1)
wherein 1) is the confidence level on the expert action probability and (s) {a E A [(a|s) 2 n) the confident action space of the expert. The switch mechanism leads to the formal representation of the behavior policy:
*(ass) Ta s)1.EA,, + F(a)s)((s),
wherein F(s) = - g.,, saa'ssdda' is function denoting the probability of the agent choosing an action that will be rejected by the switch. Emulating how human drivers judge the risky situations, rely on the expert's 's evaluation the safety during training, instead of any external objective criterion.
We derive the guarantee on the training safety from the introduction of guardian. We first have the assumption on the expert:
Assumption (Failure probability of the expert). For all state, the step-wise probability of expert producing unsafe action is bounded by a small value < 1: (aaeE()))) [(s, a) S c, wherein [(s, a) E {0, 1) is a Boolean denotes whether next state s' ~ (sssss, a) is an ground-truth unsafe state.
We use the expected cumulative probability of failure to measure the expected risk encountered by the behavior policy: V Ezg (su) = P=)) Σi=o (s;,, at) wherein P(t) refers to the trajectory distribution deduced by the behavior policy. We propose the main theorem of this work:
Theorem (Upper bound of the training risk). The expected cumulative probability of failure V of the behavior policy ă in EGPO is bounded by the step-wise failure probability of the expert as well as the confidence level n.
V (1
7
KT),
wherein Ky max, J AEA,(s) da has negative correlation 1/-
When is fixed, increasing the confidence level will shrink the upper bound of V. leading to better training safety. The proof is given in the Appendix.
In the implementation, the actions from agent firstly modulated by the guardian and the safe actions will be applied to the environment. We update the learning agent with off-policy RL algorithm. Meanwhile, we also leverage recent offline RL technique to address the partial demonstrations provided by the guardian and further improve the learning stability. The policy learning is presented in Sec. 3.2. Since the intervention from guardian indicates the agent has done something wrong, also optimize the policy to reduce intervention frequency through the constrained optimization in Sec. 3.3.
3.2 Learning Policy from Exploration and Partial Demonstration
The proposed EGPO method can work with most of the RL algorithms to train the safe policy since the guardian mechanism does impose any assumption on the underlying RL methods. In this work, we use an off-policy actor-critic method Soft Actor-Critic (SAC) [36] to train the agent. The method utilizes two neural networks including a Q network estimating the state-action value: Q ò and 0 are the parameters. The training algorithm alternates between
and policy network: the policy evaluation and the policy improvement each iteration. The policy evaluation process updates the estimated Q function by minimizing the 1.2 norm of the entropy regularized TD error:
y(r,, S141) = +
E Rl (1
1Q;(0141, - a log No (Qt-1 1
E Lo(#) (s
(1((),841)) 841) - (0)))
Here B is the replay buffer, Φ is the delayed parameters, is a temperature parameter. On the other hand, the policy improvement objective, which should be minimized, is written as:
T (0) ar-B,arrvto((a)) (ot a4) log mo(ae (arlst))-

---

Since we use a safety- ensured mixed policy म to explore the environment, part of the collected transitions contain the actions from the expert. This part of data comes as partial demonstration denoted as Be, which leads to the distributional shift problem. Many works have been proposed to overcome this problem, such as the V-trace the on-policy algorithm IMPALA [37], the advantage- weighted actor-critic [38] in the off off-policy algorithm, and many other offline RL methods [31, 30, 32]. To train with the off-policy data produced by the guardian, we adopt the recent Conservative Q- Learning (CQL) [32], known an effective offline RL method, in our Learning from Partial Demonstration (LfPD) setting. The objective to update Q function becomes:
1e ()) = =3( E [Qo(s,a)
E
QQ(ss
E (s.a)~
(r Set1) -
S sante aanvie
B aarn
1st Term
2nd Term
3rd Term
(5)
Note that the 1st Term and 2nd Term are expectations over only the partial demonstration 8 instead of the whole batch B. In the partial demonstration data, the 1st Term reduces the values for the actions taken by the agent, while the 2nd Term increases the Q values of expert actions. 'he 3rd Term is the original TD learning objective in Eq. 3. CQL reflects such idea: be conservative the actions sampled by the agent, and be optimistic to the actions sampled by the expert. Minimizing Eq. can lead to better and more stable Q function. In next section, discuss another hurdle in the training and propose a solution for intervention minimization.
3.3 Intervention Minimization via Constrained Optimization
The guardian intervenes the exploration of the agent once it behaves dangerously or inefficiently. However, if no measure is taken to limit intervention frequency, the learning policy is prone to heavily rely on the guardian. It deceives guardian mechanism by always taking dangerous actions so the guardian will take over all the time. In this case, the learning policy receives high reward under the supervision of guardian but fails to finish tasks independently.
In this section, we consider the intervention mininization as a constrained optimization problem and apply the Lagrangian method into the policy improvement process. Concretely, the optimization problem becomes: lr = arg max F.,, (St 7're]. s.t. Er, (Er=o 7'2x] I C wherein C is the intervention frequency limit in one episode. The Lagrangian dual form of the above problem becomes an unconstrained optimization problem with a penalty term:
0- = arg max min E r'r:) AIC ~ -13
(6)
X20 TwUR
where À 2 0 known as the Lagrangian multiplier. The optimization over 0 and can be conducted iteratively between policy gradient ascent and stochastic gradient descent (SGD).
We additionally introduce an intervention critic Q estimate the cumulative intervention occurrence Ex (1= (r.. This network can be optimized following Eq. 3 with the reward replaced by the intervention occurrence. intervention minimization objective L can be written as:
T0) =
E S
Q (8c, az) -),
Now we can update the policy by combining the policy improvement objective Eq. 4 with the intervention minimization objective Eq. 7 to the final objective:
L (0) = (0) + (0)
Conducting SGD on Eq. 8 w.r.t. 0 can improve the return while reduce the intervention.
The SAC with the Lagrangian method has been proposed by Ha et al. [39]. From the attempt to reproduce the result in our task, we find that directly optimizing the Lagrangian dual in the off- off-policy RL algorithm SAC is highly unstable. Stooke et al. [6] analyze that optimizing Lagrangian multiplier brings oscillations and overshoot, which destabilizes the policy learning. This is because the update of the multiplier an integral control from the perspective of control theory. Introducing the extra proportional and derivative control to update the Lagrangian multiplier can reduce the oscillations and
5

---

C. Examples of scenes
Sersory Date
Lider
Varning triangle
Traffic vehicle
Traffic
A. Interface
B. Observation
D. Sources of Dangers
Figure 3: A. The interface of the environment from MetaDrive [40]. B. The observations feeding to the target vehicle. C. The examples of the scenes we use in training and test. D. The three events creating costs: crashing with warning triangle, cone or other vehicles. + cost given once those events occur.
corresponding cost violations. We thus adopt PID controller to update and form the responsive intervention minimization as:
A t- 5 Kg / ddi + Ka 5
wherein 8= E 21 C,
(9)
i=1
di
where we denote the training iteration as i, and Kp. Kú, Ka are the hyper-parametent Optimizing with Eq. 6 reduces the proportional term in Eq. 9, while the integral and derivative terms compensate the accumulated error and overshoot in the intervention occurrence. We apply the PID controller in EGPO, well the baseline SAC-Lagrangian method in the experiments. Empirical results validate that PID control on À brings stabler learning and robustness to hyperparameter.
4 Experiments
4.1 Experimental Settings
Environment. We evaluate the proposed method and baselines in the recent driving simulator MetaDrive [40]. The environment supports generating an unlimited number of scenes via the Procedural Generation. Each of the scenes includes the vehicle agent, the complex road network, the dense traffic flow, and many obstacles such as cones and warning triangles, as shown in Fig. 3D. The task for the agent is to steer the target vehicle with low-level signals, namely acceleration, brake and steering, to reach the predefined destination. Each collision to the traffic vehicles or obstacles yields +1 environmental cost. The episodic cost in test time is the measurement policy, which independent to whether the expert used during training. The reward function only contains dense driving reward and sparse terminal reward. The dense reward is the longitudinal movement toward destination in Frenet coordinates. The sparse reward +20 given when the agent arrives the destination. We build our testing benchmark based MetaDrive rather than other RL environments like the safety gym [41] because we target on the application of autonomous driving and the generalization of the RL methods. Different to those environments, MetaDrive generate an infinite number of driving scenes which allows evaluating the generalization of different methods by splitting the training and test sets in the context of safe RL.
the safety of
Split of training and test sets. Different from the conventional RL setting where the agent is trained and tested in the same fixed environment, we focus on evaluating the generalization through testing performance. We split the scenes into the training set and test set with 100 and 50 different scenes respectively. At the beginning of each episode, scene in the training test set is randomly selected. After each training iteration, we roll out the learning agent without guardian in the test environments and record the percentage of successful episodes over multiple evaluation episodes, called success rate. Besides, we also record the episodic cost given by the environment and present it in following tables and figures.
Training expert policy. In our experiment, the expert policy is stochastic policy trained from the Lagrangian PPO [41] with batch size as large 160,000 and a long training time. To further improve the performance of the expert, we have reward engineering by doubling the cost and adding complex penalty to dangerous actions.

---

Table 1: The test performance of different approaches.
Training Episodic Cost
Episodic Return
Episodic Cost
Success Rate
Category
Method
Expert
PaO-Lag
392.38 499.07
1.26+057
0.86
RL
SAC-RS PPO-RS
346.49
11631
8.68 11.34
0.68
10.10
294. 10 222.38
3.93 24.19
20.09
166
Success Rate
SAC-Lag
333.90
119000
2.21 1.00
0.65 +0.14
0.75
Safe RL
PPO-Lag
288.04
#58.22
1.03 #0.34
0.43
+0.2
CPO
194.06
1 1008.6
1.71 1:00
0.21
1:009
0.25
Offline RL
CQL
373.95
+:.39
0.24 #0.30
0.72 +0.11
0.00
BC
362.18
1:6.9
0.13 2:017
0.57 1:012
1.0 Sampled Steps
IL
Dagger
346. 16 1:2662
0.67
1.023
0.66
1:012
EGPO [Durs]
SAC-Lag
SAC
GAIL
309.66 1:12.77
0.68 1:020
0.60 1:007
PPOHUng
PPO-RS
Expert
Figure 4: Comparison between our method and safe RL baselines.
Ours
EGPO
388.37 10000
0.56 20.35
0.85 1:005
Intervention Frequency
Velocity
Training
Roturn
Exp Sei Capar
420
20
220
20
MajO
1.0
2.0
0.0
1.0
2.0
0.5
1/0
1.5
2.0
0.5
1.0
1.6
2.3
Sampled Staps
Sampled Steps
Figure 5: The learning dynamics of EGPO and baseline methods during training.
Figure 6: The curves of EGPO agents when vary- ing the quality of experts.
Implementation details. We conduct experiments using RLLib [42], distributed learning system which allows large-scale parallel experiments. Generally, we host 8 concurrent trials in an Nvidia GeForce RTX 2080 Ti GPU. Each trial consumes CPUs with 8 parallel rollout workers. Each trial trained over roughly 200, 000 environmental steps, which corresponds to 11 hours of individual driving experience. All experiments are repeated times with different random seeds. Information about other hyper-parameters is given in Appendix.
4.2 Results
Compared to RL and Safe RL baselines. We evaluate two RL baselines PPO [43] and SAC [36] with the reward shaping (RS) method that considers negative cost as auxiliary reward. We also evaluate three safe RL methods, namely the Lagrangian version of PPO and SAC [6. 39] and Constrained Policy Optimization (CPO) [5]. As shown in Fig. 4 and Table 1, EGPO shows superior training and test time safety compared to the baselines. During training, EGPO limits the occurrence of dangers, denoted by the episodic cost, to almost zero. Noticeably, EGPO achieves lower cost compared to the expert policy. EGPO also learns rapidly and results to high test success rate.
Compared to Imitation Learning and Offline RL baselines. We use the expert to generate 250, 000 steps of transitions from training environments and use this dataset to train with Behavior Cloning (BC), GAIL [22], DAgger [26], and offline RL method CQL [32]. As shown in Table 1, EGPO yields better test time success rate compared to the imitation learning baselines. BC outper- forms ours in test time safety, but we find that BC agent learns conservative behaviors resulting in poor success rate and low average velocity to 15.05 km/h, while EGPO runs normally in 27.52 km/h, as shown in Fig. 5.
Learning dynamics. We denote the intervention frequency by the average episodic intervention occurrence Er E c. As illustrated Fig. 5, at the beginning of the training, the guardian is involved more frequently to provide driving demonstrations and prevent agent from entering dangerous states. After acquiring primary driving skills, the agent prone to choosing actions that more acceptable by guardian and thus the takeover frequency decreases.
7

---

Triiiing Cost
Table 2: The test performance when ablating components in EGPO.
7.5
8.0
Episodic Return
Episodic Cost
Success
2.3
Experiment
Rate
8.3
(a) W/ rule-based switch
339. 11:1141
1.00
0.57
0.5
200
(b) W/o intervention min.
=361
1.00
0.00=0000
Twer Stctwess Rode
(c) W/o PID in SAC-Lag.
338.80 11623
0.59
0.67 10.10
1.0
(d) W/o CQL.osss
378.00 26.77
0.43 10.54
0.80 2008
0.5
(e) W/o environmental reward
379.91 4:7.7
+.22
0.79
EGPO
388.37 +10.01
0.56 +0.35
0.85 4.0.5
0.3 1.3
20
Figure 7: Ablation study on r7.
4.3
Ablation Studies
The impact of expert quality. To investigate the impact of the expert if its quality is not as good as the well-performing expert used in the main experiments, we involve two expert policies with 60% and 30% test success rate into the training of EGPO. Those two policies are retrieved from the intermediate checkpoints when training the expert. The result of training EGPO with the inferior experts is shown in Fig. 6. We can see that improving the expert's 's quality can reduce the training cost, This result also empirically justifies the Theorem I where the training safety bounded by the expert safety. Besides, we find better expert leads to better EGPO agent in term of the episodic return. We hypothesize this is because using premature policies as expert will make the switch function produce chaotic intervention signals that mystifies the exploration of the agent.
The impact of confidence level. The confidence level 1) is hyper-parameter. As shown in Fig. we find that when y > 0.05, the performance decreases as v) increases. This is because higher means less freedom of free exploration. In the extreme case where 1) 1.0, all data is collected by the expert. In this case, the intervention minimization multiplier À will goes to large value, which damages the training. When 17 0.0, the whole algorithm reduces to vanilla SAC.
Ablations of the guardian mechanism. (a) We adopt a rule-based switch designed to validate the effectiveness of the statistical switch in Sec. 3.1. The intervention happens when the distance to the nearby vehicles or to the boundary of road is too small. We find that the statistical switch performs better than rules. This is because it is hard to enumerate manual rules that cover all possible dangerous situations. (b) Removing the intervention minimization technique, the takeover frequency becomes extremely high and the agent learns to drive directly toward the boundary of the road. This causes consistent out-of-the-road failures, resulting in the zero success rate and episodic cost. This result shows the importance of the intervention minimization in Sec. 3.3. (c) We find that removing the PID controller on updating À in intervention minimization causes a highly unstable training. It is consistent with the result in [6]. We therefore need to use PID controller to optimize in EGPO and SAC-Lag. (d) Removing CQL loss in Eq. 5 damages the performance. We find this ablation reduces the training stability. (e) We set the environment reward always to zero in EGPO, so that the only supervision signal to train the policy is the intervention occurrence. This method outperforms IL baselines with a large margin, but remains lower than EGPO in the return and success rate. This suggests EGPO can be turned into a practical online Imitation Learning method.
Human-in-the-lowe experiment. To demonstrate the potential of EGPO, we conduct human-in-the- loop experiment, where a human expert supervises the learning progress of the agent. The evaluation result suggests that EGPO can achieve 90% success rate with merely 15,000 environmental steps of training, while SAC-Lag takes 185,000 steps to achieve similar results. EGPO also outperforms Behavior Cloning method in a large margin, while BC even consumes more human data. Please refer to Appendix for more details.
5 Conclusion
We develop an Expert Guided Policy Optimization method for the Expert-in-the-lowe Reinforcement Learning. The method incorporates the guardian mechanism in the interaction of agent and environ- ment to ensure safe and efficient exploration. The experiments on safe driving show that the proposed method can achieve training and test-time safety and outperform previous safe RL. and imitation baselines. In future work we will explore the potential of involving human to provide feedback the learning process.
8

---

Acknowledgments
This project was supported by the Centre for Perceptual and Interactive Intelligence (CPII) Ltd under the Innovation and Technology Fund.
References
[1] A. Kendall, Hawke, D. Janz, Mazur, D. Reda, J.-M. Allen, V.-D. Lam, Bewley, and A. Shah. Learning to drive in a day. In 2019 International Conference on Robotics and Automation (ICRA), pages 2254. IEEE, 2019.
[2] K. Mason and S. Grijalva. A review of reinforcement learning for autonomous building energy management. Computers & Electrical Engineering. 78:300-312 2019.
[3] F. Richter, R. K. Orosco, and M. C. Yip. Open- sourced reinforeement learning environments for surgical 2019.
robotics. arXiv preprint
[4] Y. Chow, O. Nachum, E. Duenez-Guzman, and M. Ghavamzadeh. lyapunov- approach to safe reinforcement learning. In Advances in neural information processing systems, pages 8092-8101, 2018.
[5] Achiam, D. Held, Tamar, and P. Abbeel. Constrained policy optimization. Proceedings of the 34th International Conference on Machine Cearning-Volumne 70, pages 22-31. JMLR. org, 2017.
[6] A. Stooke, Achiam, and Abbeel. Responsive safety in reinforcement learning by pid lagrangian methods. In International Conference on Machine Learning, pages 9133 9143, PMLR, 2020.
H. Bharadhwaj, A. Kumar, N. Rhinchart, S. Levine, Shkurti, and A. Garg. Conservative safety crities for exploration. arXiv preprint arXiv:2010. 144.4907,, 2020.
[8] T. Mandel, Y.-E. Liu, E. Brunskill, and Z. Popović. Where add actions in human-in-the-loog reinforce- ment learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
P. Christiano, J. Leike, T. B. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. arXiv preprint 17X.:::706.077220 2017.
[10] L. Guan, M. Verma, S. Guo, R. Zhang, and S. Kambhampati. Explanation augmented feedback in human-in inttan--o--c--ooop reinforcement learning. arXiv preprint M4.iv:2006.148000 2020.
[11] W. Saunders, G. Sastry, A. Stuhlmueller, and O. Evans. Trial without error: Towards safe reinforcement learning via human intervention. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pages 2067-2069. International Foundation for Autonomous Agents and Multiagent Systems, 2018.
[12] Gareia and Fernández. comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437 1480, 2015.
[13] D. Amodei, C. Olah, Steinhardt, P. Christiano, J. Schulman, and D. Mané. Concrete problems in ai safety, arXiv preprint arXiv: 1606.0655, 2016.
[14] D. A. Calian, D. Mankowitz, T. 7ahavy, 7. Xu, J. Oh, N. Levine, and T. Mann. Balancing constraints and rewards with meta-gradient d4pg. arXiv preprint arXiv:2010.06324, 2020.
[15] A. Taylor, A. Singletary, Y. Yue, and A. Ames. Learning for safety-critical control with control barrier functions. Learning for Dynamics and Control, pages 708-717 PMLR, 2020.
[16] Y. Liu, J. Ding, and X. Liu. Ipo: Interior-point policy optimization under constraints. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 4940-4944 2020.
[17] H. Sikchi, W. Zhou, and D. Held. Lyapunov barrier policy optimization. NeurlPS Deep Reinforcement Learning Workshop 2020, 2020.
[18] K. Srinivasan, B. Eysenbach, S. Ha, J. Tan, and C. Finn. Learning to be safe: Deep rl with safety critic. arXiv preprint arXiv:2010. 14603, 2020.
[19] G. Dalal, K. Dvijotham. M. Vecerik. T. Hester, C. Paduraru, and Y. Tassa. Safe exploration in continuous action spaces. arXiv preprint 1801 0880100777 2018.
[20] B. Widrow. Pattern recognition and adaptive control. IEEE Transactions on Applications and Industry, 83 (74):269 277, 1964.

---

[21] J. Sun, L. Yu, P. Dong. B. L. and B. Zhou. Adversarial inverse reinforcement learning with self-attention dynamics model. IEEE Robotics and Automation Letters, 2021.
[22] Ho and S. Ermon. Generative adversarial imitation learning, arXiv preprint urXiv::600.03447 2016.
[23] J. Sun, H. Sun, T. Han, and B. Zhou. Neuro-symbolic program search for autonomous driving decision module design. In Proceedings of the Conference Robot Learning (CoRL) 2020, 2020.
[24] J. Huang. S. Xie, J. Sun, Q. Ma, C. Liu, D. Lin, and B. Zhou. Learning decision module by imitating driver's 's control behaviors. Proceedings of the Conference on Robot Learning (CoRL) 2020, 2020.
[25] S. Reddy, A. D. Dragan, and S. Levine. Sqil: Imitation learning via reinforcement learning with sparse rewards. arXiv preprint
2019.
[26] S. Ross, G. Gordon, and D. Bagnell, A reduction of imitation learning and structured prediction no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627 635. JMLR Workshop and Conference Proceedings, 2011.
[27] M. Kelly, C. Sidrane, K. learning with human experts. In 2019 International Conference on Robotics and Automation (ICRA), pages 8077-8083. JEEE, 2019.
and M. J. Kochenderfer. Hg dggdagger: Interactive imitation
[28] J. Zhang and K. Cho. Query-efficient imitation learning for end-to-end autonomous driving. arXiv preprint 165.:1600506450, 2016.
[29] S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline reinforcement learning: Tutorial, review, and perspec- tives on open problems. arXiv preprint arXiv:2005.00 01643, 2020.
[30] S. Fujimoto, D. Meger, and D. Precup. Off-policy deep reinforcement learning without exploration. arXiv preprint http::8112029000 2018.
[31] Y. Wu, G. Tucker, and O. Nachum. Behavior regularized offline reinforcement learning, arXiv preprint arXiv:1911.11360 2019.
[32] A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q- g-learning for offline reinforcement learning. arXiv preprint (0XXxv2006.047777 2020.
[33] B. Ibarz, J. Leike, T. Pohlen, G. Irving. S. Legg, and D. Amodei. Reward learning from human preferences and demonstrations in atari. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 8022-8034, 2018
[34] D. Abel, J. Salvatier, A. Stuhlmüller, and O. Evans. Agent-agnostic human-in-the-loop reinforcement learning. arXiv preprint 1xXXv:1701..4077 2017.
[35] J. Speneer, S. Choudhury, M. Barnes, M. Schmittle, M. Chiang, P. Ramadge, and Srinivasa. Learning from interventions. In Robotics: Science and Systems (RSS), 2020.
[36] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, pages 1861-1870, 2018
[37] L. Espeholt, Н. Soyer. R. Munos. K. Simonyan, Mnih. T. Ward, Y. Doron. V. Firoiu, T. Harley, I. Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International Conference Muchine Learning. pages 1407-1416. PMLR, 2018
[38] X. B. Peng, A. Kumar, G. Zhang, and S. Levine. Advantage- regression: Simple and scalable off-policy reinforcement learning. arXiv preprint urXiv:1910.00177, 2019.
[39] S. Ha. P. Xu, Z. Tan, S. Levine, and J. Tan. Learning to walk in the real world with minimal human effort, 2020.
[40] Q. Li, Z. Peng. Z. Xue, Q. Zhang, and B. Zhou. Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning. arXiv preprint (xXiv2210911277 2021.
[41] A. Ray. J. Achiam, and D. Amodei. Benchmarking safe exploration in deep reinforcement learning. openai, 2019.
[42] E. Liang, R. Liaw, R. Nishihara, P. Moritz, R. Fox, K. Goldberg, J. Gonzalez, M. Jordan, and L. Stoica. Rllib: Abstractions for distributed reinforcement learning. In International Conference on Machine Learning. pages 3053-3062, 2018.
[43] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint 177.v:1707.06347 2017.
10


Reviews:
[{"review_id": "jx6SZY8bM1", "reviewer": "Reviewer_RZXA", "summary": "This paper presents an approach for safe policy optimization in the expert-in-the-loop reinforcement learning setting. By assuming that the failure probability of the expert is bounded, the proposed approach shows that the cumulative probability of failure of the trained policy is bounded. Evaluations on a simulator for autonomous driving demonstrates that the approach can achieve high rewards with minimal costs compared to pure imitation learning baselines. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- The problem of safe policy learning is important and the solution strategy through human-in-the-loop RL is well-motivated. This is a compromise between pure RL methods that do not assume any expert intervention but are unsafe, and pure imitation learning methods that require high quality expert demonstrations and can be much safer in training. \n\n- The theoretical result of bounded training risk is interesting, and important. This guarantee is important to ensure that the safety of the trained behavioral policy will not diverge too much from the safety level of the expert. \n\nWeaknesses\n\n- The definition of unsafe actions and states (for example in line 142) is very unclear. In most cases, for example a car that is gradually moving off the road, no single state is safe or unsafe and the trajectory of the car gradually progresses from safe to unsafe regions. So, it is important to clearly define how \"unsafety\"is quantified in the paper. \n\n- Theorem 1 has a potentially unbounded term. How large can the value of $K^'$ be in Theorem 1? It is important to upper-bound this term because if the term is large, then the bound on cumulative probability of failure becomes vacuous. \n\n- The way the \"expert policy\" is constructed doesn't seem realistic. In line 248-253, it is mentioned that the expert policy is trained with Lagrangian PPO for a long training time and causes a lot of safety violations during training. This is in a sense circumventing the safety problem because the expert is allowed to be unsafe in training. How will this generalize to a real world experiment? In particular, how will the corresponding expert policy be trained when the approach is transferred to a real-robot navigation task?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "My main concern is with the training of the expert policy with a lot of safety violations. As such, the approach doesn't seem scalable to a real world setting, where safety is ultimately important. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "Please address the issues listed under Weaknesses above (copied below for reference) and a few additional clarifications regarding experiments\n\n\n- The definition of unsafe actions and states (for example in line 142) is very unclear. In most cases, for example a car that is gradually moving off the road, no single state is safe or unsafe and the trajectory of the car gradually progresses from safe to unsafe regions. So, it is important to clearly define how \"unsafety\"is quantified in the paper. \n\n- Theorem 1 has a potentially unbounded term. How large can the value of $K^'$ be in Theorem 1? It is important to upper-bound this term because if the term is large, then the bound on cumulative probability of failure becomes vacuous. \n\n- The way the \"expert policy\" is constructed doesn't seem realistic. In line 248-253, it is mentioned that the expert policy is trained with Lagrangian PPO for a long training time and causes a lot of safety violations during training. This is in a sense circumventing the safety problem because the expert is allowed to be unsafe in training. How will this generalize to a real world experiment? In particular, how will the corresponding expert policy be trained when the approach is transferred to a real-robot navigation task?\n\n- What is the intuition for why offline RL safety violations are much better than Dagger and GAIL but worse than BC? Is the same expert data used to train the IL methods and the offline RL baseline?\n\n- The method presented in the paper did not seem specific to autonomous driving. What are the challenges for applying this method to obtain safe learning policies in locomotion and manipulation tasks? It will be helpful to list out issues, if any. "}}, {"review_id": "UivX_qbA9eN", "reviewer": "Reviewer_rHZA", "summary": "This paper introduces an expert guided reinforcement learning method, called EGPO, to learn safe driving policies. The expert functions as a safety guardian that will take over the control if the agent’s control action is not safe (i.e., the expert is less likely to take that action). Then the expert control will be learned using offline reinforcement learning technique, conservative Q-learning. To mitigate the problem that the learning agent may totally rely on the expert to take safe actions and do not learn how to drive, the paper introduces a method to minimize intervention using a Lagrangian method. The policy parameters and the Lagrangian multipliers are learned iteratively. In particular, the Lagrangian multipliers are updated using a PID controller.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThis paper is well written and easy to follow. The video in the supplementary material greatly helped the reviewer in understanding the evaluation scenarios. The idea of using expert action to directly constraint the action of the learning agent is very interesting. Moreover, the paper explored methods to enhance the learning using offline RL and by minimizing the frequency of expert interference. These methods can be apply to general safe RL algorithms that use safe guardian (e.g., RL + CBF), to enhance their exploration efficiency.\n\nWeaknesses:\n- The assumption 1 (that requires the amount of unsafe actions for every state to be bounded in the expert policy) seems too strong. For example, in some unsafe states, it is possible that no action can be safe, in particular, when the action space is constrained. Moreover, it is unclear how the authors can verify this assumption for arbitrary expert policies. In the experiments, the expert is learned using Lagrangian PPO. Does this expert satisfy the assumption? Can this assumption be verified using the success rate (since later the paper uses the success rate the differentiate different expert policies, e.g., in figure 6)?\n\n- In line 231, the paper states that the environmental cost is unavailable to EGPO. What is the motivation of making this assumption? It is understandable that the authors may wish to claim that even without the safety reward directly from the environment, the agent can learn to behave safely with the expert guidance. However, in autonomous driving, this is not a realistic assumption since environmental safety reward is “common sense” and easy to obtain. Moreover, the nature of environmental safety reward is difference from the nature of the expert guidance. The former provides a constraint on the state space, while the latter provides a constraint in the state space. It would be more convincing if the paper demonstrates that the imitation of expert still provides strong performance margin when the environmental safety reward is considered and that how the environmental safety reward affects the amount of expert intervention. \n\n- The authors mention that they plan to extend the method to include human demonstration. How can we obtain a stochastic policy \\mathcal{E} in this case?\n\n- In table 1, why is the cumulative cost for the expert higher than that for PPO-Lag for safe RL, offline RL, IL, and the proposed method? For IL and the proposed method, since their safety notions solely comes from the expert demonstration, the results seem counter intuitive. For PPO-Lag in safe RL, is it using the same training method as the expert? Is the difference caused by insufficient training? \n\n- In table 2 (d) and (e), the performance is comparable to EGPO considering the uncertainty range. Does it mean that the proposed method is less sensitive to CQL loss and environmental reward? Why is it the case?\n\n- This paper claims that the proposed method can make learning more efficient. However, the efficiency of the proposed method should also account for the time spent to obtain the expert policy (whether it is a pretrained policy or from human-demonstration). ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "While the paper presents an interesting idea, the major concerns the reviewer has is:\n1. the results need more explanation;\n2. the assumptions need further justification;\n3. how the method can be applied to real world applications, especially how to obtain the demonstration efficiently and how to ensure the expert policy satisfies the assumptions, needs better explanation. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Please check the comments above.\n\nThere is a typo in (1). It should be if a\\in A, choose (a,0); otherwise, choose (a^E,1).", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "KONkNQ_iEiK", "reviewer": "Reviewer_hLxx", "summary": "This paper introduces a method for coupling data collected from an agent with the ones provided by an expert, that can override the actions of the agent in case of unsafe behaviour. The method is tested on an autonomous driving task and outperform a significant amount of baselines.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper proposes a method for involving an expert agent in the classical RL loop. The method is theoretically sound, and relatively easy to apply. I like the way the authors addresst the inherent issues of their vanilla method, and the way they propose theoretically grounded solutions to address them, e.g., the use of Conservative Q-Learning and the constrained optimization problem.\n\nI think the paper is overall nice to read, but it has some typos that should be fixed:\n- line 141: is a function THAT denotes;\n- line 151: negativeLY;\n- line 186: stabler -> more stable.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper introduces a method that I find overall interesting and practical. I find some limitations that I list below that I’d like to be addressed, possibly in the paper, and that motivate my score.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "- how does your method apply to problems with stochastic dynamics and/or partial observability?\n- what do you think your method need to safely handle these problems?\n- would the expert be considered as an “oracle” also in noisy partially observable problems? I think this would significantly limit the applicability of the method."}}, {"review_id": "IWHWjUcOcaX", "reviewer": "Reviewer_o3VH", "summary": "In this paper the authors introduce expert guided policy optimization, an algorithm for safe reinforcement learning. In this method, it is assumed that some expert is available at training time which can output a distribution over expert actions given a state. At its core, the algorithm checks at every stage of training whether an action taken by the \"student\" will be dangerous or not by checking that it is contained within the confidence interval of the expert's distribution for some threshold nu. The authors then provide an upper bound for the training risk (the expected cumulative probability of failure) using this student-expert scheme. In order to train the student, the authors use a modified version of SAC with a modified loss function for the critic to account for the partial demonstrations provided by the expert. In their experience, the algorithm tends to exploit expert behavior, and therefore, they introduce the intervention cost c, which essentially penalizes expert intervention. This is done by introducing an additional critic which effectively keep track of intervention occurrence, and used as part of a soft constraint in the policy's objective. The authors test the framework on a self-driving environment and are able to obtain better cumulative rewards than similar methods whilst also having low training episodic cost.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The paper is very clearly written.\n* It provides a realistic experimental setup, even if it is in simulation. \n\nWeaknesses:\n* It would have been nice to see the algorithm used in more than one driving simulation environment.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I believe this paper proposes an interesting framework for safe reinforcement learning with access to an expert. While there is only one driving environment being showcased, the experiments do provide significant evidence that this method outperforms other similar safe rl methods.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Fair: Some knowledge of the area"}, "extra_text": {"issues": "My only concern is that only one environment was used to showcase the algorithm. It would have been nice to have more driving simulators to test the algorithm. Other than that I do not have any other major concerns I would like to raise at this time. "}}, {"review_id": "5eE2SLe6qv7", "reviewer": "Reviewer_RGdx", "summary": "The paper proposes the Export-guided Policy Optimization (EGPO) method aiming to achieve safe learning by combining regular learning with expert policy execution activated using a switch when detecting unsafe situation. The assumption is that a stochastic expert policy exists which can generate a safe action with high probability and also evaluate the probability of taking a given action by the expert. The switch function itself is just a check whether a proposed action has high confidence with respect to the expert model. If not, the expert takes over. \n\nTo encourage learning, the frequency of interventions is then limited by adding a term penalizing the excess number of interventions using a Lagrangian multiplier. To improve stability, the multiplier is updated using a PID method. \n\nTo train with off-policy expert data, Conservative Q-Learning (CQL) approach is used to improve the training on trajectories with partial expert demonstrations. \n\nThe authors perform a study on a simulation autonomous driving task with several type of unsafe situation arising from other vehicles and traffic cones. The results show lower overall cost compared to other methods such as SAC-Lag and PPO-Lag. \n\n\nother minor comments:\n\nline 52: \"the Lagrangian method\" is unclear when first introduced\nline 151: \"negative correlated\" is unclear (one needs to read the appendix to understand what this means) \nis the policy equation (2) actually used in practice? it is not referenced later in the paper (except once but without concrete application)", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ a principled way to integrate expert policy in a learning framework\n+ reasonably good performance in a simple simulated environment, compared to other existing methods\n+ the authors seem to have also applied the method to more realistic mock-up setting (as shown in Appendix)\n\nWeaknesses:\n- it is not very clear whether the method works because the expert is really good and is trained on the exact same environment distribution, and so the learning process is cheating when executing the expert; it would have been interesting to see an expert only being activated in unsafe-only situations (as opposed to just any deviation from the expert policy, which the switch function is based on)\n- the usage of Lagrangian multipliers and a PID controller is interesting, but also seem very heuristic and its performance is unclear\n- the implications of Theorem 1 are not clear, would be great to refer to it in the actual algorithm, or to use in some constructive way; the term K_eta is also a bit of a mystery, can that term potentially be very large and if so, does this theorem upper bound serve any practical purpose?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "While integrating an expert policy for safer learning is important, there are some unclear aspects of the approach (as listed in Weaknesses) which need to be addressed before assessing how successful it will be in practice. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "- please answer the list of Weaknesses above", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors have sufficiently addressed reviewers' concerns; specifically, the new human-in-the-loop experiment was particularly insightful. I recommend an acceptance.
Decision: Accept (Poster)