Guarded Policy Optimization with Imperfect Online Demonstrations
Authors: Zhenghai Xue
Zhenghao Peng
Quanyi Li
Zhihan Liu
Bolei Zhou
Venue: ICLR 2023 notable top 25% (2023)
Keywords: reinforcement learning
guarded policy optimization
imperfect demonstrations
shared control
metadrive simulator

Abstract: The Teacher-Student Framework (TSF) is a reinforcement learning setting where a teacher agent guards the training of a student agent by intervening and providing online demonstrations. Assuming optimal, the teacher policy has the perfect timing and capability to intervene in the learning process of the student agent, providing safety guarantee and exploration guidance. Nevertheless, in many real-world settings it is expensive or even impossible to obtain a well-performing teacher policy. In this work, we relax the assumption of a well-performing teacher and develop a new method that can incorporate arbitrary teacher policies with modest or inferior performance. We instantiate an Off-Policy Reinforcement Learning algorithm, termed Teacher-Student Shared Control (TS2C), which incorporates teacher intervention based on trajectory-based value estimation. Theoretical analysis validates that the proposed TS2C algorithm attains efficient exploration and substantial safety guarantee without being affected by the teacher's own performance. Experiments on various continuous control tasks show that our method can exploit teacher policies at different performance levels while maintaining a low training cost. Moreover, the student policy surpasses the imperfect teacher policy in terms of higher accumulated reward in held-out testing environments. Code is available at https://metadriverse.github.io/TS2C.

Published as a conference paper at ICLR 2023
GUARDED POLICY OPTIMIZATION WITH IMPERFECT ONLINE DEMONSTRATIONS
Zhenghai Xue', Zhenghao Peng". Quanyi Li Zhihan Liu", Bolei Zhou?
Nanyang Technological University, Singapore, 2 University of California, Los Angeles, 3 The University of Edinburgh, "Northwestern University
ABSTRACT
The Teacher- Stuudet-tt Framework (TSF) is a reinforcement learning setting where teacher agent guards the training of student agent by intervening and providing online demonstrations. Assuming optimal, the teacher policy has the perfect tim- ing and capability to intervene the learning process of the student agent, provid- ing safety guarantee and exploration guidance. Nevertheless, in many real-world even impossible to obtain well-performing teacher
settings it is expensive policy. In this work, we relax the assumption of a well-performing teacher and develop a new method that can incorporate arbitrary teacher policies with mod- est or inferior performance. We instantiate an Off-Policy Reinforcement Learning algorithm, termed Teacher-Student Shared Control (TS2C), which incorporates teacher intervention based on trajectory- based value estimation. Theoretical anal- ysis validates that the proposed TS2C algorithm attains efficient exploration and substantial safety guarantee without being affected by the teacher's own perfor- mance. Experiments on various continuous control tasks show that our method exploit teacher policies different performance levels while maintaining a low training cost. Moreover, the student policy surpasses the imperfect teacher policy in terms of higher accumulated reward in held-out testing environments. Code is available at wuthttss:/// https::/mmmtadiiieeee giteeiiihhhuu.... io/Is2C
INTRODUCTION
In Reinforcement Learning (RL), the Seacher-Student Framework (TSF) Zimmer et al. 2014 Kelly et al. 2019) incorporates well-performing neural controllers or human experts as teacher policies in the learning process of autonomous agents. At each step, the teacher guards the free exploration of the student by intervening when specific intervention criterion holds. Online data collected from both the teacher policy and the student policy will be saved into the replay buffer and exploited with Imitation Learning Off-Policy RL algorithms. Such guarded policy optimization pipeline can either provide safety guarantee (Peng et al., 2021b or facilitate efficient exploration (Torrey & Taylor 2013).
The majority of RL methods in TSF assume the availability of a well-performing teacher pol- icy (Spencer et al.] 2020 Torrey Taylor 2013) so that the student can properly learn from the teacher's demonstration about how to act in the environment. The teacher intervention is triggered when the student acts differently from the teacher (Peng et al. 202T) or when the teacher finds the current state worth exploring (Chisari et al. 2021). This is similar to imitation learning where the training outcome is significantly affected by the quality of demonstrations (Kumar et al., 2020) Fujimoto et al., 2019). Thus with current TSF methods if the teacher is incapable of providing high-quality demonstrations, the student will be misguided and its final performance will be upper- bounded by the performance of the teacher. However, it is time-consuming obtain a well-performing teacher in many real- world applications such as object manipulation with robot arms (Yu et al.) 2020a) and autonomous driving (Li et al. 2022a) As result, current TSF methods will behave poorly with a less capable teacher.
even impossible to
In the real world, the coach of Usain Bolt does not necessarily need to run faster than Usain Bolt. Is it possible to develop a new interactive learning scheme where a student can outperform the teacher while retaining safety guarantee from it? In this work we develop new guarded policy optimization

---

Published as a conference paper at ICLR 2023
method called Teacher-Student Shared Control (TS2C). It follows the setting of teacher policy and a learning student policy, but relaxes the requirement of high-quality demonstrations from the teacher. A new intervention mechanism is designed: Rather than triggering intervention based on the similarity between the actions of teacher and student, the intervention is now determined by a irajectory-based value estimator. The student is allowed conduct an action that deviates from the teacher's, s, as long as its expected return is promising. By relaxing the intervention criterion from step-wise action similarity trajectory-based value estimation, the student has the freedom to act differently when the teacher fails to provide correct demonstration and thus has the potential to outperform the imperfect teacher. We conduct theoretical analysis and show that in previous TSF methods the quality of the online data-collecting policy is upper-bounded by the performance of the teacher policy. In contrast, TS2C is not limited by the imperfect teacher upper-bound performance, while still retaining a lower-bound performance and safety guarantee.
Experiments on various continuous control environments show that under the newly proposed method, the learning student policy can be optimized efficiently and safely under different levels of teachers while other TSF algorithms are largely bounded by the teacher's performance. Further- more, the student policies trained under the proposed TS2C substantially outperform all baseline methods in terms of higher efficiency and lower test-time cost, supporting our theoretical analysis.
BACKGROUND
2.1
RELATED WORK
The Teacher-Student Framework The idea of transferring knowledge from a teacher policy to student policy has been explored in reinforcement learning (Zimmer et al.) 2014). It improves the leamning efficiency of the student policy by leveraging a pretrained teacher policy, usually by adding auxiliary loss to encourage the student policy to be close the teacher policy (Schmitt al.) 2018 Traoré et al. 2019). Though our method follows teacher-student transfer framework, an optimal teacher is not necessity. During training, agents are fully controlled by either the student (Traoré al. 2019 Schmmit et al. 2018) the teacher policy (Rusu et al.) 2016), while our method fol- lows intervention-based RL where a mixed policy controls the agent. Other attempts to relax the need of well-performing teacher models include student-studed transfer (Lin al. 2017 7 Lai al.,
2020 in which heterogeneous agents exchange knowledge through mutual regularisation (Zhao & Hospedales 2021 Peng et al. 2020).
Learning from Demonstrations Another way to exploit the teacher policy is to collect static demonstration data from it. The learning agent will regard the demonstration as optimal transitions to imitate from. If the data is provided without reward signals, agent can learn by imitating the teacher's 's policy distribution (Ly & Akhloufi 2020). matching the trajectory distribution (Ho & Er- mon 2016 Xu et al. 2019)) or learning parameterized reward function with inverse reinforcement learning (Abbeel & Ng 2004 Fu et al., 2017). With additional reward signals, agents can perform Bellman updates pessimistically, as most offline reinforcement learning algorithms do (Levine et al.) 2020). The conservative Bellman update can be performed either by restricting the overestimation of Q-function learning (Fujimoto et al., 2019 Kumar et al.) 2020) by involving model- badded un- certainty estimation (Yu et al.) 2020b Chen et al. 2021b). In contrast to the offline learning from demonstration, in this work we focus the online deployment of teacher policies with teacher- student shared control and show its superiority in reducing the state distributional shift, improving efficiency and ensuring training-time safety.
Intervention-basel Reinforcement Learning Intervention-based RL enables both the expert and the learning agent to generate online samples in the environment. The switch between policies can be random (Ross et al. 201T), rule-based (Parnichkun et al., (022) or determined by the expert, either through the manual intervention of human participators (Abel et al., 2017, Chisari et al. 202T Li et al. 2022b) or by referring to the policy distribution of parameterized expert (Peng et al. 2021
More delicate switching algorithms include RCMP (da Silva et al. 2020) which asks for expert advice when the learner's is action has high estimated uncertainty. RCMP only works for agents with discrete action spaces, while we investigate continuous action space in this paper. Also, Ross & Bagnell (2014) and Sun et al. (2017) query the expert to obtain the optimal value function, which is used to guide the expert intervention. These switching mechanisms assume the expert policy to be optimal, while proposed algorithm can make of a suboptimal expert policy. To exploit
2

---

Published as a conference paper at ICLR 2023
samples collected with different policies, Ross et al. (201] and Kelly al. (0199) compute behavior cloning loss on samples where the expert policy is in control and discard those generated by the learner. Other algorithms (Mandlekar et al., 2020 Chisari et al. 2021) assign positive labels expert samples and compute policy gradient loss based on the pseudo reward. Some other research works focus on provable safety guarantee with shared control (Peng et al. 202T Wagenerr et al.) 2021), while we provide an additional lower-bound guarantee of the accumulated reward for our method.
2.2 NOTATIONS
We consider an infinite-horizon Markov decision process (MDP), defined by the tuple M = (S, A, P. R, y, do) consisting of a finite state space S, finite action space AA, the state transition probability distribution P: S x A x S + [0, 1]. the reward function R S x A + [Rmin Rmax] the discount factor y E (0, 1) and the initial state distribution dg S - [0, 1]. Unless otherwise stated, π denotes stochastic policy π A + [o, 1]. The aiatenaction value and state value func- tions of are defined as (" a) E s.6o ma.a, - pappppppllleeee lp.a))..a)) 1- ? R (s,, a;) and V*((s (1) (s a). The optimal policy is expected maximize the accumulated return [(x)) = 10
The Seacher-Studemt Framework (TSF) models the shared control system as the combination of a teacher policy स. which is pretrained and fixed and student policy r,s be learned. The actual actions applied to the agent are deduced from a mixed policy of and иa. where म. starts gener- ating actions when intervention happens. The details of the intervention mechanism are described in Sec. 3.2 The goal of TSF is to improve the training efficiency and safety of T. with the in- volvement of Te. The discrepancy between ni and Wg on state 8, termed as policy discrepancy, the Lı-norm of output difference: [|<r(-]s) T-(+(s)]l = .A (als) (a|s)) da. We define the discounted state distribution under policy as dz = (1 7)
Pr 8; n, do), where Pr (s, = 8; π. do) the state visitation probability. The state distribution discrepancy de- fined as the difference in -oom of the discounted state distributions deduced from two policies:
I|dx, dx, = Is |d., (s) dx, (s) ds.
GUARDED POLICY OPTIMIZATION WITH ONLINE DEMONSTRATIONS
Fig. [ shows an overview of our proposed method. In addition to the conventional single- agent RL setting.
Warmup Sampling
Value Esimator
include a teacher policy Te in the training loop. The term "teacher" only indicates that the role of this policy is help the student training. No assumption on the op- timality of the teacher is needed. The teacher policy is first used to do warmup rollouts and train a value estimator. During the training of the student policy, both and π receive cur- rent state from the environment. They pro- pose actions a, and az, and then value-based intervention function (s)) determines which action should be taken and applied to the en- vironment. The student policy is then updated with data collected through such intervention.
Control
Teaschr
Action
Intervention Function
Action
Environment
Student
Action
Tranction
Undate
Replay Buffor
Figure 1: Overview of the proposed teacher- student shared control method. Both student and teacher policies are in the training loop and the shared control occurs based on the intervention function.
We first give a theoretical analysis on the general setting of intervention- based RL in Sec. 3.1 We
then discuss the properties of different forms of intervention function 'T in Sec. 3.2 Based these
analyses, we propose new algorithm for teacher-student shared control in Sec. 3.3
All the proofs
in this section are included in Appendix A.1
3.1 ANALYSIS ON INTERVENTION- DINRVENNIION-BSED BASED RI.
In intervention-busee RL, the teacher policy and the student policy act together and become a mixed behavior policy Tb- The intervention function T(s) determines which policy is in charge. Let
3

---

Published as a conference paper at ICLR 2023
((s) denotes the teacher policy n, takes control and T(s) = 0 means otherwise. Then To can be represented as (-[s)) = (s) (s) + (1 (-1))
One issue with the joint control is that the student policy иa is trained with samples collected by the behavior policy Ty. whose action distribution not always aligned with x.. A large state distribution discrepancy between two policies ||dz,, d=, Il cause distributional shift and ruin the training. similar problem exists in behavior cloning (BC), though in BC no intervention
involved and TT learns from samples all collected by the teacher policy Ti.- To analyze the state distribution discrepancy in BC, we first introduce a useful lemma (Achiam et al. 2017).
The state distribution discrepancy between the teacher policy ne and the student policy T, bounded by their expected policy discrepancy:
Lemma 3.1.
|d - dl <
7 Bgdda [ m= 11)1
1 1-7
We apply the lemma to the setting of intervention-buse RL and derive a bound for [|d d 1l1.
Theorem 3.2. For any behavior policy тı deduced by teacher policy To a student policy Te and an intervention function T(s), the state distribution discrepancy between my and n, is bounded by
By 1 --7 Bandan [[Te(. -Ts(: -T((118)12>
[d -d 111
(2)
[o, 1] is the expected intervention rate weighted by the where B Benday (1)))
Bonday -.(l
policy discrepancy.
Both Eq. [ and Eq. 2 bound the state distribution discrepancy by the difference in per-state pol- icy distributions, but the upper bound with intervention is squeezed by the intervention rate B. In practical algorithms, f can be minimized to reduce the state distribution discrepancy and thus lieve the performance drop during test time. Based on Thm. 3.2. we further prove in Appendix A that under the setting of intervention-base and student policy J (T,) can be similarly related. The analysis in this section does not assume certain form of the intervention function T(s). Our analysis provides the insight on the feasibility and efficiency of all previous algorithms in intervention-basee
RL, the accumulated returns of behavior policy J (TT))
RL (Kelly et al.) 2019 Peng et al.) 2021 Chisari et al. (202T). In the following section, we will examine different forms of intervention functions and investigate their properties and performance bounds, especially with imperfect online demonstrations.
3.2 LEARNING FROM IMPERFECT DEMONSTRATIONS
straightforward idea to design the intervention function is to intervene when the student acts differently from the teacher. We model such process with the action-based intervention function Taction (s):
Faction (s) = {1 if Ea-n (s) (n) [log (a s)] <
otherwise,
wherein > is a predefined parameter. similar intervention function is used in EGPO (Peng et al. 2021), where the student's 's action is replaced by the teacher's 's if the student's 's action has low probability under the teacher's 's policy distribution. To measure the effectiveness of certain form of intervention function, we examine the return of the behavior policy (TT)). With Taciion (s) defined in
Eq. 3 we can bound J(m)) with the following theorem.
Theorem 3.3. With the action-based intervention function (action(s), the return of the behavior policy J(T)) is lower and upper bounded by
-3) VH VH-82 ((T)) ((T.) J(m.)+ (1-7)²
(811-BBmaan VH (1 7)2
(4)
where H Exnda, H( is the average entropy of the teacher policy during shared control and B3 is the weighted intervention rate Thm. 3.2
4

---

Published as a conference paper at ICLR 2023
The theorem shows that ((Tb) can be lower bounded by the return of the teacher policy n, and an extra term relating to the entropy of the teacher policy. It implies that action- based intervention function aagion is indeed helpful in providing training data with high return. We discuss the tightness of Thm. 3.3 and give intuitive interpretation of VH in Appendix A.2 A.2
A drawback of the action-based interven- tion function is the strong assumption on the optimal teacher, which is not always feasible. If we turn to employ subopti- mal teacher, the behavior policy would be burdened due to the upper bound in Eq. 4 We illustrate this phenomenon with the ex- ample in Fig. _ where slow vehicle in
Student trajediory
Student action
Teacher action
Teacher trajectory
gray is driving in front of the ego-vehicle in blue. The student policy is aggressive and would like to overtake the gray vehi- cle to reach the destination faster, while the teacher intends to follow the vehicle conservatively. Therefore, T. and Ty will propose different actions in the current state, leading to Taction according to Eq. 3 The mixed policy with shared control will always choose to follow the front vehicle and the agent can never accomplish successful overtake.
In autonomous driving scenario, the ego vehicle is the blue one on the left, following the gray vehicle on the right. The upper trajectory proposed by the student to overtake and the lower trajectory is proposed by the teacher to keep following.
Figure 2:
To empower the student to outperform a suboptimal teacher policy, we investigate a new form of intervention function that encapsulates the long-term term value estimation into the decision of interven- tion, designed as follows:
if VT (s) - Euwn, (-1a) (8. >
Tralue (8)
0 otherwise,
where e 0 is predefined parameter. By using this intervention function, the teacher tolerates student's action if the teacher can not perform sigmificantly better than the student by in return. Tvaluc no longer expects the student to imitate the teacher policy step-by-step. Instead, it makes decision on the basis of long-term return. Taking trajectories in Fig. 2 again as an example, if the overtake behavior has high return, the student will be preferable to Tralue- Then the student control will not be intervened by the conservative teacher. So with the basued intervention function, the agent's s exploration ability will not be limited by a suboptimal teacher. Nevertheless, the lower- bound performance guarantee of the behavior policy Ta still holds, shown as follows.
Theorem 3.4. With the value-base balue intervention function Tratue (5) defined in Eq. the return of the behavior policy mh is lower bounded by
(mb) > J (mT) (1 (1-30 3)E
(6)
In safety-critical scenarios, the step-wise training cost c(s, a). i.e., the penalty on the safety violation during training, can be regarded as a negative reward. We define f(s,a) r(s, a) nc(s, a) the combined reward, where 17 is the weighting hyperparameter. V
and Tralue are similarly defined by substituting with & in the original definition. Then have the following corollary related to expected cumulative training cost, defined by C(T) = (eondo.arra(((l)
(p) EE (st,
Corollary 3.5.
With safety-criticee value-based intervention function (s), the expected cumu- lative training cost of the behavior policy Te is upper bounded by
B1-B)e C(Tb) < C(Ti) + (1117) + - -
In Eq. [ the upper bound of behavior policy's 's training cost consists of three terms: the cost of teacher policy, the threshold in intervention multiplied by coefficients and the superiority of mi in cumulative reward. The first two terms are similar to those in Eq. and the third term means trade- off between training safety and efficient exploration, which can be adjusted by hyperparameter 1,.
5

---

Published as a conference paper at ICLR 2023
Comparing the lower bound performance guarantee of action-based and aalue-based intervention function (Eq. 4 and Eq. ( the performance gap between T6 and Ti can both be bounded with respect to the threshold for intervention and the discount factor y. The difference is that the performance gap when using Taction is in an order of O ((T-)))) while the gap with Tralue is in an order of ( It implies that in theory value-based intervention leads to better lower-bound performance guarantee. In terms of training safety guarantee, value-based intervention function (1--) ) in
Tvalue has better safety guarantee by providing a tighter safety bound with the order of O
of action-based intervention function (see Theorem in (Peng et al. 2021 ). We show in the Sec. 4.3 that the theoretical advances of Teatue in training safety and efficiency can both be verified empirically.
contrast to O (1-
3.3 IMPLEMENTATION
Justified by the aforementioned advantages of the value-based intervention function, we propose practical algorithm called Teacher-Student Shared Control (TS2C). Its workflow listed in Ap- pendix B To obtain the teacher Q-network Q in the value-based intervention function in Eq. S we rollout the teacher policy TH and collect training samples during the warmup period. Gaussian noise is added to the teacher's policy distribution to increase the state coverage during warmup. With limited training data the Q-network may fail to provide accurate estimation when encountering previously unseen states, We propose to teacher Q-ensemble based on the idea of ensembling Q-networks (Chen et al., 2021a). set of ensembled teacher Q-networks Qo with the same archi- tecture and different initialization weights are built and trained with the same data. To learn Q we follow the standard procedure in (Chen et al. (021a) la) and optimize the following loss:
L (Ø) Es [y Mean [Q (s, (s,a)]]²
(8)
where y = replay buffer for storing sequences {(s, a, r, s'))- Teacher will intervene when Tratue returns 1 or the output variance of ensembled Q-networks surpasses the threshold, which means the agent is exploring unknown regions and requires guarding. We also use Qº to compute the state-value functions in Eq. leading to the following practical intervention function:
Ir + Mean [Q (s', a is the Bellman target and D is the
if Mean Eaann (-1s) Qº (8, a) - Ear.. Q* (8,a) a)] > 81
Ts)
or Var (Baws.La Qa (s, a)] > E2.
otherwise.
Eq. 2 shows that the distributional shift and the performance gap to oracle can be reduced with smaller B, i.e., less teacher intervention. Therefore, minimize the amount of teacher intervention via adding negative reward to the transitions one step before the teacher intervention. Incorporating intervention minimization, parameterized by v0:
the following loss function update the student's Q-network
T(bb)= = FA (v' - Q" (s. (s.)
(10)
where y' = E sn) |r (Tss + 7Q (s',a') a log (a'(s'))) is the soft Bellman target with intervention minimization. À is the hyperparameter controlling the intervention mini- mization. a is the coefficient for maximum-entron Critic (SAC) (Haarnoja al. 2018). To update the student's policy network parameterized by 0, apply the objective used in SAC as:
learning updated in the same way as Soft Actor
L(0) E.ND [Bant lalog (TT s)) Q (s,a))
(11)
4 EXPERIMENTS
We conduct experiments to investigate the following questions: (1) Can agents trained with TS2C achieve super-teacher performance with imperfect teacher policies while outperforming other meth- ods in the Teacher-Studed Framework (TSF)? (2) Can TS2C provide safety guarantee and improve training efficiency compared to algorithms without teacher intervention? (3) Is TS2C robust dif- ferent environments and teacher policies trained with different algorithms? To answer questions

---

Published as a conference paper at ICLR 2023
Test Reward with Teacher High
Test Reward with Teacher Medium
Test Reward with Teacher Low
350
350
350
300
200
150
150
150
TS2C
100
Importance
100
100
50
EGPO
50
50
Teacher
Training Cost with Teacher-High
Training Cost with Teacher-Madium
Training Cost with Teacher-Low
10
TS2C
10
EGPO
8
6
2
1.5 Sampled Steps (1e5)
0.5 1.0
2.0
0.5 1.0
1.5
2.0
1.5 Sampled Steps (1e5)
0.5 1.0
2.0
Sampled Steps (1e5)
Figure 3: Comparison between method TS2C and other algorithms with teacher policies pro- viding online demonstrations. "Importance" refers to the Importance Advising algorithm. For each column, the involved teacher policy has high, medium, and low performance respectively.
(2)(2, we conduct preliminary training with the PPO algorithm (Schulman et al. 2017 and save checkpoints on different timesteps. Policies different stages of PPO training are used as teacher policies in TS2C and other algorithms in the TSF. With regard to question (3), we agents trained with PPO (Schulman et al., 2017), SAC (Haarnoja et al. 2018) and Behavior Cloning as the teacher policies from different sources.
4.1 ENVIRONMENT SETUP
The majority the experiments are conducted on the lightweight driving simulator MetaDrive LLi et al. 2022a). One concern with TSF algorithms is that the student may simply record the teacher's actions and overfit the training environment. MetaDrive can test the generalizability of learned agents unseen driving environments with its capability to generate unlimited number of scenes with various road networks and traffic flows. We choose 100 scenes for training and 50 held-out scenes for testing. Examples of the traffic scenes from MetaDrive MetaDrive, the objective is to drive the ego vehicle to the destination without dangerous behaviors such crashing into other vehicles. The reward function consists of the dense reward proportional to the vehicle speed and the driving distance, and the terminal +20 reward when the ego vehicle reaches the destination. Training cost is increased by when the ego vehicle crashes or drives out of the lane. To evaluate TS2C's performance in different environments, we also conduct experiments several environments of the MuloCo simulator (Todorov al. 2012).
shown in Appendix C In
4.2 BASELINES AND IMPLEMENTATION DETAILS
Two sets of algorithms are selected as baselines to compare with. One includes traditional RL and II. algorithms without the TSF. By comparing with these methods we can demonstrate how TS2C improves the efficiency and safety of training. Another set contains previous algorithms with the TSF, including Importance Advising (Torryy & Taylor 2013) and EGPO (Peng et al. [021]. The original Importance Advising uses an intervention function based on the range of the Q-function: [(s) = max.e QD(s.a) min.eA QD(s.a): where QD is the Q-table the teacher policy. Such Q-table is not applicable in the Metadrive simulator with continuous state and action spaces. In practice, we sample N actions from the teacher's 's policy distribution and compute their Q-values on certain state. The intervention happens if the range, i.e., the maximum value minus the minimum

---

Published as a conference paper at ICLR 2023
(a) Test Reward
(b) Training Cost
(c) Average Intervention Rate
300
EGPO
Importance
200
TS2C
0.4
100
PPO
BC
0
1.5 Sampled Steps (1e5)
Sampled Steps (1e5)
Sampled Steps (1e5)
Figure 4: Figures (a) and (b) shows the comparison of efficiency and safety between TS2C and baseline algorithms without teacher policies providing online demonstrations. Figure (c) shows the comparison of the average intervention rate between TS2C and two baseline algorithms in the TSF.
Test Reward on Pendulum Env
Test Reward on Happer Env
Test Reward on Walker2d Env
4000
3000
800
3000
-700
1000
TS2C
1250
1000
EGPO
Teacher
1.0
1.5
1.0
Sampled Steps (164)
Sampled Steps (1e5)
Sampled Steps (1e6)
Figure 5: Performance comparison between method TS2C and baseline algorithms three environments from MuloCo.
value, surpass certain threshold E. The EGPO algorithm uses an intervention function similar to the action-based intervention function introduced in section 3.2 All algorithms are trained with different random seeds. In all figures the solid line computed with the average value across different seeds and the shadow implies the standard deviation. We leave detailed information on the experiments and the result ablation studies on hyperparameters in Appendix
4.3 RESULTS
Super-teacher performance and better safety guarantee The training result with three different levels of teacher policy can be seen in Fig. 3 The first row shows that the performance of D'TS2C is not limited by the imperfect teacher policies. converges within 200k steps, independent of different performances of the teacher. EGPO and Importance Advicing is clearly bounded by teacher- and teacher-low, performing much worse than TS2C with imperfect teachers. The second row of Fig. shows TS2C has lower training cost than both algorithms. Compared to EGPO and Importance Advising, the intervention mechanism TS2C is better-designed and leads to better behaviors.
Better performance with TSF The result of comparing TS2C with baseline methods without the TSF can be seen in Fig. 4 ab). We use the teacher policy with a medium level of performance to train the student in TS2C. achieves better performance and lower training cost than the baseline algorithms SAC, PPO, and BC. The comparative results show the effectiveness of incorporating teacher policies online training. The behavior cloning algorithm does not involve online sampling in the training process, so it has zero training cost.
Extension for different environments and teacher policies The performances of TS2C in dif- ferent MuJoCo environments and different sources of teacher policy are presented in Fig. 5 and respectively. The figures show that TS2C is generalizable to different environments. It can also make use the teacher policies from different sources and achieve super-teacher performance con- sistently. Our TS2C algorithm consideration. On the other hand, though the EGPO algorithm has the best performance in the Pendulum environment, it struggles in the other two environments, namely Hopper and Walker.
outperform SAC in all three MuJoCo environments taken into
8

---

Published as a conference paper at ICLR 2023
Reward Teacher
Teacher
Reward BC Teacher
300
300
300
200
200
200
TS2C
100
Importance EGPO
100
100
Teacher
0.5
1.0
2.0
1.5
2.0
1.5
2.0
Sampled Steps (1e5)
Sampled Steps (1e5)
Sampled Steps (1e5)
Figure 6: Performance comparison between our method TS2C and baseline algorithms with teacher policies providing online demonstrations. The teacher policies are trained by PPO, SAC, and behav- cloning respectively.
4.4 EFFECTS OF INTERVENTION FUNCTIONS
Trajostory with value-based based intervention
We further investigate the intervention behaviors un- der different intervention functions. As shown in Fig. c(c), the average intervention rate Ex-d=, ((s) of TS2C drops quickly as soon as the student pol- icy takes control. The teacher policy only intervenes during a very few states where it can propose actions with higher value than the students. The intervention rate of EGPO remains high due to the action-based intervention function: the teacher intervenes when- ever the student act differently.
We also show different outcomes of action-based and value-based intervention functions with screen- shots in the MetaDrive simulator. In Fig. 7 the ego vehicle happens to drive behind traffic vehicle which is in an orange trajectory. With action-based intervention the teacher takes control and keeps fol- lowing the front vehicle, as shown in the green tra- jectory. In contrast, with the value-based interven- tion the student policy proposes to turn left and over- take the front vehicle as in the blue trajectory. Such action has higher return and therefore tolerable by Trszc, leading to better agent trajectory.
Trajectory with acionnbbased intervention
Figure 7: Visualization of the trajectories resulting from different intervention mecha- nisms. The trajectories of irrelevant traffic vehicles are marked orange. As in the green trajectory, action-based intervention make the car following the front vehicle. Value- based intervention instead can learn overtak- ing behavior as blue trajectory.
CONCLUSION AND DISCUSSION
In this work, we conduct theoretic analysis on intervention-based RL algorithms in the Teacher- Student Framework. It is found that while the intervention mechanism has better properties than some imitation learning methods, using an sation-based intervention function limits the performance of the student policy. We then propose TS2C, a value-based intervention scheme for online policy optimization with imperfect teachers. We provide the theoretic guarantees on its exploration ability and safety. Experiments show that the proposed TS2C method achieves consistent performance independent to the teacher policy being used. Our work brings progress and potential impact to relevant topics such as active learning, numan--n-the-
methods, and safety-critical applications.
Limitations. The proposed algorithm assumes the agent can access environment rewards, and thus defines the intervention function based on value estimations. It may not work in tasks where reward signals are inaccessible. This limitation could be tackled by considering reward-free settings and employing unsupervised skill discovery (Eysenbach et al., | Aubret et al. 2019). These methods provide proxy reward functions that can be used in teacher intervention.

---

Published as a conference paper at ICLR 2023
REFERENCES
Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In
Proceedings the twenty-first international conference on Machine learning, pp. 1, 2004.
David Abel, John Salvatier, Andreas Stuhlmüller, and Owain Evans. Agent-agnostic human-in-the- loop reinforcement learning. arXiv preprint arXiv: 04079, 2017.
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In ICML, volume 70 of Proceedings of Machine Learning Research, pp. 22-31. PMLR, 2017.
Arthur Aubret, Laëtitia Matignon, and Salima Hassas. A survey on intrinsic motivation in reinforce- ment learning. CoRR, abs/1908.0000 069766 2019.
Xinyue Chen, Che Wang, Zijian Zhou, and Keith W. Ross. Randomized ensembled double q- learning: Learning fast without model. In International Conference Learning Representa- tions, 2021a.
Xiong-Hui Chen, Yang Yu, Qingyang Li, Fan-Ming Luo, Zhiwei Qin, Wenjie Shang, and Jieping Ye. Offline model-based adaptable policy learning. Advances in Newral Information Processing Systems, 34, 2021b.
Eugenio Chisari, Tim Welschehold, Joschka Boedecker, Wolfram Burgard, and Abhinav Val- ada. Correct me if am wrong: Interactive learning for robotic manipulation. arXiv preprint arXiv:2110.03316, 2021.
Felipe Leno da Silva, Pablo Hermandez-Leal, Bilal Kartal, and Matthew Taylor. Uncertainty-aware action advising for deep reinforcement learning agents. In AAAI, 2020.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. In ICLR (Poster). OpenReview.net, 2019.
Justin Fu, Katie Luo, and Sergey Levine Learning robust rewards with adversarial inverse rein- forcement learning. arXiv preprint arXiv:1710.11245, 2017.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. International Conference on Machine Learning, pp. 2062-2062. PMLR, 2019.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learming, pp. 1861-1870. PMLR, 2018.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In NIPS, pp. 4565-4557777 2016.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. In NeurlPS, pp. 12498-12509, 2019.
Michael Kelly, Chelsea Sidrane, Katherine
and Mykel Kochenderfer. Hg- dagger: Interactive imitation learning with human experts. In 2019 International Conference
on Robotics and Automation (ICRA), pp. 8077-8083. IEEE, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006.0477 2020.
Kwei-Herng Lai, Daochen Zha, Yuening Li, and Xia Hu. Dual policy distillation. In IJCAI, 2020.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.
Quanyi Li, Zhenghao Peng, Lan Feng, Qihang Zhang, Zhenghai Xue, and Bolei Zhou. Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning. IEEE transactions on pattern analysis and machine intelligence, 2022a.
10

---

Published as a conference paper at ICLR 2023
Quanyi Li, Zhenghao Peng, and Bolei Zhou. Efficient learning of safe driving policy via human-ai copilot optimization. In ICLR. OpenReview.net, 2022b.
Kaixiang Lin, Shu Wang. and Jiayu Zhou. Collaborative deep reinforcement learning. arXiv preprint 1702:0/702. 2017.
Abdoulaye Ly and Moulay Akhloufi. Learning drive by imitation: An overview of deep behav- ior cloning methods. IEEE Transactions on Intelligent Vehicles, 6(2):195-22) 209, 2020.
Ajay Mandlekar, Danfei Xu, Roberto Martin-Martín, Yuke Zhu, Li Fei-Fei, and Silvio Savarese. human-in-the-loop imitation learning using remote teleoperation. arXiv preprint arXiv:2012.06733. 2020.
Rom Parnichkun, Matthew N Dailey, and Atsushi Yamashita. Reil: A framework for reinforeed intervention-based imitation learning. arXiv preprint arXiv:2203. 15390, 2022.
Zhenghao Peng, Hao Sun, and Bolei Zhou. Non-local policy optimization via diversity-regulariied collaborative exploration. arXiv preprint arXiv:2006.0778/ 2020.
Zhenghao Peng, Quanyi Li, Chunxiao Liu, and Bolei Zhou. Safe driving via expert guided policy optimization. In 5th Annual Conference on Robot Learning, 2021.
Stéphane Ross and J. Andrew Bagnell. Reinforcement and imitation learning via interactive no- regret learning. CoRR, abs/1406.5979, 2014.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and struc- tured prediction no-regret online learning. In Proceedings of the fourteenth international con- ference on artificial intelligence and statistics, pp. 627-635. JMLR Workshop and Conference Proceedings, 2011.
Andrei Rusu, Sergio Gomez Colmenarejo, Çaglar Gülçehre. Guillaume Desjardins, James Kirk- patrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy distil- lation. In ICLR. OpenReview.net, 2016.
Simon Schmitt, Jonathan Hudson, Augustin Zidek, Simon Osindero, Carl Doersch, Wojeiech M Czarnecki, Joel Z Leibo, Heinrich Kuttler, Andrew Zisserman, Karen Simonyan, et al. Kickstart- ing deep reinforcement learning. arXiv preprint arXiv:1803.03883 2018
John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jordan, and Philipp Moritz. Trust region policy optimization. In ICML, volume 37 of JMLR Workshop and Conference Proceedings, pp- 1889-1897. JMLR.org. 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1rXiv:1707.06347, 2017.
Jonathan Spencer, Sanjiban Choudhury, Matthew Barnes, Matthew Schmittle, Mung Chiang, Peter Ramadge, and Siddhartha Srinivasa. Learning from interventions: Human-robot interaction as both explicit and implicit feedback. In Robotics: Science and Systems (RSS), 2020.
Wen Sun, Arun Venkatraman, Geoffrey J. Gordon, Byron Boots, and J. Andrew Bagnell. Deeply aggrevated: Differentiable imitation learning for sequential prediction. In ICML, volume 70 of Proceedings of Machine Learning Research, pp. 3309-3318. PMLR, 2017.
Emanuel Todorov. Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In IROS. pp. 5026-5033. IEEE, 2012.
Lisa Torrey and Matthew Taylor. Teaching on a budget: Agents advising agents in reinforcement learning. In Proceedings of the 2013 International Conference on Autonomous Agents and Multi- Agent Systems, AAMAS '13, pp. 1053-1060. International Foundation for Autonomous Agents and Multiagent Systems, 2013.
René Traoré, Hugo Caselles-Dupie Timothée Lesort, Te Sun, Guanghang Cai, David Filliat, and Natalia Díaz- Biaz-Rodríguez Discorl: Continual reinforcement learning via policy distillation. In NeurIPS workshop Deep Reinforcement Learning, 2019.

---

Published as a conference paper at ICLR 2023
Nolan Wagener, Byron Boots, and Ching-Ar An Cheng. Safe reinforeement learning using advantage- based intervention. In ICML, volume 139 of Proceedings of Machine Learning Research, pp- 10630-10640. PMLR, 2021.
Tian Xu, Ziniu Li, and Yang Yu. On value discrepancy of imitation learnin. arXiv preprint arXiv: 191000707 2019.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference Robot Learning, pp. 1094-1100. PMLR, 2020a.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33:14129-1444 2020b.
Chenyang Zhao and Timothy Hospedales. Robust domain randomised reinforcement learning through peer-to-peer distillation. In Proceedings of The 13th Asian Conference on Machine Learning, volume 157 of Proceedings of Machine Learning Research, pp. 1237-1252. PMLR, 2021.
Matthieu Zimmer, Paolo Viappiani, and Paul Weng. Teacher-student framework: reinforcement learning approach. In AAMAS Workshop Autonomous Robots and Multirobot Systems, 2014.
12

---

Published as a conference paper at ICLR 2023
A THEOREMS IN TS2C
A. DETAILED PROOF
We start the proof with the restatement of Lem. 3.1 in Sec. 3.1
Lemma A.1 (Lemma 4.1 in (Xu et al. 2019)).
|| dx 111 & - Esnd | 8) 7' (11
7
(12)
Thm 3.2 can be derived by substituting and I' in Lem A.1 with Th and Te.
Theorem A.2 (Restatement of Thm. 3.2g. For any behavior policy x6 deduced by a teacher policy a student policy n, and intervention function T(s), the state distribution discrepancy between Th and na is bounded by policy discrepancy and intervention rate:
By
|da. du < 1 1-7 Eonda, [oTe( () - TT(-1s)]l1
(13)
Benday (1) is the weighted expected intervention rate.
where B
Fynday
Proof.
dn, 1l1 &
Bandin |m (T((1)) -ma(- (s)]
Banday
|T(() (8) + (1 - (T((s)))T( (8) m.( (r(-18)]l
(14)
? Egndr
[I{(s) [oe(.: - - s)]l
By
Bandan
| (8) -.s( ())1
Based on Thm. 3.2 we further prove that under the setting of shared control, the performance gap of to the optimal policy x² can be bounded by the gap between the teacher policy ni and n², together with the teacher-student policy difference. Therefore, training with the trajectory collected with mixed policy is optimize upper bound of the student's suboptimality. The following lemma is helpful in doing this.
Lemma A.3.
& Rmax
J -
Eands || x'( (1)s
(15)
(1 7)2
Proof. It is direct combination Lemma 4.2 and Lemma 4.3 in (Xu et al. 2019).
Theorem A.4. For any behavior policy consisting of teacher policy we. student policy n and a intervention function T (()), the suboptimality of the student policy is bounded by
3Rmax
(-*)- - ( <
Es-xy [T(e s) - ( s)]h + +1)(=*) - -m))))
(16)
(1 7
13

---

Published as a conference paper at ICLR 2023
Proof.
Rmax Bunda, ||b(- -- [s]]
[J (Tb) J (Ts)) <
7)2
R Rmax
Bondda [T(()T((-1 + (1- ((s)) (11ss T.(. (s]ss]]
(1 7)²
Rmax
7/2 Bondan
[T((s) [Te(. 8) T. |>]]|||
(17)
B BRmax Bowwr [Te 8) m. ||)||
-7)
[ - J (mx)) & J (TT) J (Ta₂) + |J (m*) J (m))
B Blmax Eswa, [ TT(- (]] + + (T)).
<
Theorem A.5 (Restatement of Thm. 3.3
With the action distributional intervention function Taction (s), the return of the behavior policy J (Tb) is lower and upper bounded by
J(m)) 22 B) Rmax VH- J(mb) J(m) v2(1 3) Rmax VH (1 7)
(18)
(1 7)2
where Rmax maxr(s, a) is the maximal possible reward, H = Eondry H((T' is the average
8,0
entropy of the teacher policy during shared control.
Proof.
Rinxx Bendin || 1 8) TT(- (6]6)
[J (TT) - J (T.)) <
(1- 7)2
R Rmax Barndts [|T(() (8) + (1 - (s)))=s() (8) me(-1
11 7)2
Pmmxx Bxndes [| s) n.(- |s]l
(1 -1)²
V2 -3)Rmuxx Bandin
(1 -7)?
(19)
2(1 (3) Pmux Bonda, V Bawne()a)) llogT((alss - log (a|s)]
(1 7))
V2(1 B Bandrr (-1s) -E
(1 77
2( 3) R)Rxmx VH-E. 7)2
Therefore, obtain
22(1 - Rmax
V2(1 -3) H
H - 8 ( (TT)
(1 71-
(1 (1-7)2
(20)
J(m.) 22(1 Rmax 1H-E J(m))
V2(1 B) Rmax
VH E,
71-
(1 7
which concludes the proof.
To prove Thm. 3.4 we introduce useful lemma from (Schulman et al. 2015)
Lemma A.6.
8o
( = J(x') + E. Ea
Σ r2 An (st, at)
(21)
14

---

Published as a conference paper at ICLR 2023
Theorem A.7 (Restatement of Thm. 3.4 With the aauue-based intervention function Tralue (8) de- fined in Eq. 3 the return of the behavior policy Ty is lower bounded by
8
J (mm)) 2 J (mt)
(22)
1-7
Proof.
20
((mb) -[(e) En
7 Ae (sn: an)
En
Σ 7at 12c (8n, (n) Vi((n)]
7" [e (s Vi(sn))
-Eg
" [B (Sns a) + +1 (+ Qi (S a) Ve(sn)]
(1sn))
EEg
n [11 ((s)))
Vi(sw)]]
8
=11 (1-B)LE Bn
nd (1 (sn) -v
Vi((s)]
-(1-- B) ToY CUT
(1 8)E
7
(23)
which concludes the proof.
L
Then we prove corollary related safety-critical scenarios.
value-based intervention function Trauue (3), the expected cumulative training cost of the behavior policy ma is upper bounded by
Corollary A.8 (Restatement of Cor. 3.5). With
(1- 711
+ + (J(o)) J(Te)..
((T)) < C(T.) +
(24)
Proof. We define expected return under policy with combined reward (TT, therefore
j(T) = Eg (( ae)
7/ (st, at)
(25)
Σ 7d [r (sx, ar) + +no ns az)l
= J(()) +T((T)
According to Thm.
A.7 under Tratue(s) (8) we have
j (mL) > (Tt)
(26)
Eq. 24 can be immediately proved by combining Eq. 25 and Eq. 26
n
15

---

Published as a conference paper at ICLR 2023
A.2 DISCUSSIONS ON THE RESULTS
In Thm. 3.3 the average entropy of the teacher policy H and the threshold for action- interven- tion e is included in the bound. We provide intuitive interpretations on the influence of H and here. when (awr)(-)s)) [log (a s)] < For reference, the action-based intervention function action discrepancy between the returns of e. According to Thm 3.3 of our paper, a larger & leads to smaller the behavior and teacher policies. This is because is the threshold for the action-based interven- tion function. If the action likelihood is less than s, the teacher policy will take over the control. A larger means more teacher intervention, constraining the behavior policy to be closer the teacher policy, which leads to a smaller discrepancy in their returns. The influence of H can be similarly analyzed. A larger H leads to larger return discrepancy. Intuitively. this because with higher en- tropy, the teacher policy tends to have a more "averaged" or multi-modal distribution over the action space. So the policy distributions of the student and teacher are more likely to have overlaps, leading to higher action likelihood. In turn, the intervention criterion is less likely to be satisfied, leading to fewer teacher interventions. In general, the intuitive interpretation of Thm. 3.3 indicates that if we would like larger return discrepancy, i.e. larger performance upper bound well smaller lower bound, we should use smaller intervention threshold and teacher policy with higher entropy, and vice versa. Thm. 3.3 has gap with the actual algorithm in that the algorithm uses value-based intervention function which is based on Thm. 3.4. Nevertheless, the intuitive interpretation may enlighten future work how choose proper teacher policy in teacher-student shared control.
With respect to the tightness, Thm. 3.3 has a squared planning horizon (1-7)² in the discrepancy
term. This is in accordance with many previous works (Thm. in (Xu et al. 2019), Thm. 4.1 in (Janner et al. 2019) and Thm. in (Schulman et al. (2055). which include (1 - 1)a in the denominator when it comes to differences of the cumulative return, given the difference in the action distribution. The order of - in Thm. 3.3 is tight, which dominates the gap in accumulated return. Nevertheless, the other constant terms, e.g. Rmax and the average entropy, can be tighter given some additional assumptions. We did not derive tighter bound since the derivation will not be related to the main contribution of this paper, which is the new type of intervention function. Thm. 3.3 and Thm. 3.4 in their current forms enough to demonstrate that the value-based intervention function has the advantage of providing more efficient exploration and better safety guarantee compared with action-basee intervention function.
B THE ALGORITHM
The workflow of TS2C during training is show in Alg.
Algorithm The workflow of TS2C during training
1: Input: Warmup steps W; Scale of warmup noise Training steps N; Teacher policy Te-
2: Initialize student policy n set of parameterized Q-function for teacher policy Qo, parame- terized Q-function for student policy Q" and the replay buffer D.
3: for i 1 to W do
4: Observe state s; and sample a, ~ (-Is) (O,
5: Step the environment with a; and store the tuple (s,, a, Ti 81+1) to D.
6: Update e with Lemporal-Differees loss Eq.
7: end for
8:
for to do
9:
Observe state and sample 0r ~ (-):)))), as x (-(:))))
10:
Compute (s;) with Eq. 9 behavior policy ny (-(-(s;) and ag.
11:
Step the environment with as and store the tuple (s,, as, rt. $1+1, Tvatue(Ss411 D.
12:
Update 1i the student Q-function with the loss in Eq. 10
13:
Update 0 in the student policy with the loss in Eq. 11
14:
end for
16

---

Published as a conference paper at ICLR 2023
C ADDITIONAL EXPERIMENT DEMONSTRATIONS
C.I DEMONSTRATIONS OF DRIVING SCENARIOS
The demonstrations several driving scenarios are shown Fig. We provide demonstrative video showing the agent behavior trained with PPO and our TS2C algorithm in the supplementary materials.
Figure 8: Four examples of the traffic scenes in MetaDrive.
C.2 PARAMETERS
The hyper-pane ppperpparameters used in the experiments are shown in the following tables. In the TS2C al- gorithm, larger values of the intervention threshold and E2 will lead to more strict intervention criterion and the steps with teacher control will be fewer. order to control the policy distribution discrepancy, we choose and 22 to ensure the average intervention rate to be less than 5%. Nev- ertheless, different E1 in the intervention function has little influence on the algorithm performance, shown in Fig. I of our paper. The coefficient for intervention minimization À is simply set to 1. used in other environments, it may need some adjustments to fit the reward scale. The coefficient for maximum entropy learning is updated during training in the SAC algorithm. The number of warmup timesteps is empirically chosen so that the expert value function can be properly trained. Other parameters follow the setting in EGPO (Peng et al. 2021). The hyper-parameters of other algorithms follow their original setting.
Table 2: EGPO (Peng al. 202
Table 1: TS2C (Ours)
Hyper-parameter
Value
Hlyper-parameter
Value
Discount Factor y
0,99
Discount Factor 7
0,99
for target network update Learning Rate
0.005
for target network update Learning Rate
0,005
0.0001
0.0001
Environmental horizon T
2000
Environmental horizon T
2000
Steps before Learning start Intervention Occurrence Limit C
10000
Warmup Timesteps W # of Ensembled Value-Functions N
50000
20
10
Number of Online Evaluation Episode
5
Variance of Gaussian Noise C
0.5
K2
5
Intervention Minimization Ratio À
I
K;
0.01
Value-based Intervention Threshold
1.2
K2
0.1
Value-based Intervention Threshold E2
2.5
CQL Loss Temperature 3
3.0
Activation Function
Relu
Activation Function
Relu
Hidden Layer Sizes
[256, 256]
Hidden Layer Sizes
[256, 256]
17

---

Published as a conference paper at ICLR 2023
Table 3: Importance Advising Taylor 2013)
(Torrey &
Hyper-parameter
Value
Table 4: SAC
(Haarnoja et
2018
Hyper-parameter
Value
Discount Factor Y for target network update Learning Rate
0.99
0.005
Discount Factor
0.99
0.0001
7 for Target Network Update
0.005
Environmental horizon 7 Warmup Timesteps W
2000
Learning Rate
0.0001
50000
Environmental Horizon ''
2000
# of Actions Sampled N
10
Steps before Learning starts
10000
Variance of Gaussian Noise C
0.5
Activation Function
Relu
Range-basee based Intervention Threshold e
2.8
Hidden Layer Sizes
[256, 256]
Activation Function
Relu
Hidden Layer Sizes
[256, 256]
D ADDITIONAL EXPERIMENT RESULTS
D.1 ADDITIONAL. PERFORMANCE COMPARISONS ON METADRIVE
show the results of TS2C trained with various levels of teachers compared with bascline algorithms without shared control. Apart from the Fig. 4 in the main paper presenting the training results of TS2C with the medium level of teacher policy, here present the performance of TS2C trained with the high, medium and low levels of teacher policy. The value-based intervention pro- posed by TS2C can utilize all these teacher policies, leading safer and more efficient training compared to traditional RL algorithms.
In Fig.
Fig. 10 shows the results with different levels of teacher policy. Besides the testing reward and the training cost shown in Fig. 3 of the main paper. we show the training reward and test success rate of TS2C compared with baseline methods with the Maacher-Stdeen Framework (TSF) respectively. Our TS2C algorithm still achieves the best performance among baseline algorithms when evaluated with these two metrics.
Training Cost Teacher-High
Training Cost Teacher-Moe Medium
Training
TS2C
8
SAC
2
0.0
0.5
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
1.0 Sampled Steps
2.0
Sampled Steps
1e5
Sampled Steps
1e5
1e5
Test Reward with Teigh
Test Reward with Teacher-Medium
Test Reward with Teacher-Low
300
300
200
200
200
100
TS2C
100
100
SAC
PPO
0
0.5
2.0
0.5
2.0
0.0
0.5
2,0
Sampled Steps
1e5
Sampled Staps
1e5
Sampled Steps
1e5
Figure 9: Comparison of training cost and test reward between our method TS2C and other algo- rithms without shared control.
18

---

Published as a conference paper at ICLR 2023
Test Reward at 100k Stops
Test Reward at 100k Steps
Tost Reward at 100k Steps
Test Roward at 100k Steps
200
200
200
200
100
100
100
100
2-2.0
C-2.5
L-3.0
WHC
WEE
EVS
Volue Intervention Function
Action
Variance thresholld
With or without Intervention cost
With or without
ensembled value networks (EVN]
Figure 11: Ablation Studies for different variance thresholds, the intervention cost, ensembled value networks and different intervention functions.
Training Reward with Teacher-High
Training Reward with Teacher-Medo Medium
Training Reward with Teacher-Low
300
300
300
200
200
200
100
152C
EGPO
Importance
0.0
0.5
1.5
2.0
2.0
0.5
Sampled Steps
Sampled Steps
Sampled Steps
1.0
Test Success Rate with Teacher-High
1.0
Success Rate with Mediuu,-Medlum,
1.0
Test Success Rate with Teacher-Low
0.8
0.8
EGPO Importance
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
Sampled Steps
1e5
Sampled Steps
1e5
Sampled Steps
1e5
Figure 10: Comparison of training reward and test success rate between our method TS2C and other algorithms with shared control.
D.2 ABLATION STUDIES
We conduct ablation studies and present the results in Fig- 11 We find the intervention cost and ensembled value networks are important to the algorithm's 's performance, while different variance thresholds in the intervention function has little influence. Also, TS2C with action-based interven- tion function behaves poorly in accordance with the thcoretical analysis in Section 3.2
D.3 DISCUSSIONS EXPERIMENT RESULTS
In Fig. 5 our TS2C algorithm can outperform SAC in all three MuJoCo environments taken into consideration. On the other hand, though the EGPO algorithm has the best performance in the Pendulum environment, it struggles in the other two environments, namely Hopper and Walker. This is because the action space of the pendulum environment is only one-dimensional. In this simple environment, the action-basee intervention of the EGPO algorithm is effective. The policy only needs slight adjustments based the imperfeet teacher work properly. In other words, the distance between the optimal action and the teacher action small. However, in more complex environments like Hopper and Walker, the distance between the
is large. As the action-based
19

---

Published as a conference paper at ICLR 2023
intervention is too restrictive, the EGPO algorithm based such intervention fails to achieve good
performance.
In Fig. [ the performance of EGPO with SAC policy the teacher policy is very poor. This is because the employed SAC teacher is less stochastic than the PPO policy. Student's actions have less likelihood in teacher's 's action distribution and are less tolerated by the action-based intervention function in EGPO, leading to large intervention rate and consequently large distributional shift. Our proposed TS2C algorithm does not access teacher internal action distribution and instead intervenes based on the state-action values of teacher policy, so it is robust to the stochasticity of teacher policy.
20


Reviews:
[{"review_id": "yf0Ws90COO", "reviewer": "Reviewer_rkHD", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a novel algorithm to learn from imperfect teacher's policy and achieves super-teacher performance. It provides theoretical justification, performance bounds and experimental results with comparison against strong baselines.", "summary_of_the_review": "I recommend to accept the paper ", "strength_and_weaknesses": "Strengths:\n- The paper addresses a key problem of learning from an imperfect teacher policy\n- It provides performance bounds and thorough empirical evidence\n\nWeaknesses:\n- The results are shown only on one environment\n- Would be nice to have more details and experiments on how to choose epsilon and if it has to be manually chosen every time based on the environment?\n\nTypos:\n- There is a '5' at the beginning of the introduction section\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Xg-DyImwM18", "reviewer": "Reviewer_L3ge", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the Teacher-Student Framework, in the case when the teacher is potentially sub-optimal. Learning here is based on an ensemble off-policy method. The core concept is to develop a teach intervention function that is based on the estimated sub-optimality of student actions with respect to the teacher's value function. The advantage of the proposed method is that it allows for continuous student improvement, to the point that the student takes complete control in the case of sub-optimal teacher. The authors provide theoretical bounds for the proposed method and evaluate it on a driving simulator. The proposed method 1) outperforms baselines across evaluation scenarios and 2) demonstrates the student capability to solve the task, even with a significantly sub-optimal teacher. ", "summary_of_the_review": "Simple and intuitive method to allow student agent to learn and outperform a potentially sub-optimal teacher. Clear writing with good theoretical backing. ", "strength_and_weaknesses": "Strengths:\n\n1. The proposed algorithm is intuitive and straight-forward\n2. It asymptotically outperforms baselines\n3. The proposed method is backed by theoretical analysis\n\nWeaknesses\n\n1. It seems that the algorithm takes some time before it starts improving and is initially substantially outperformed by EGPO. Is this because at the start of the training the switch allows most student actions, while EGPO mostly uses the teacher?\n\nQuestion:\n\nWhat is the effect of the optimization algorithm here? The proposed method uses Q-ensembles, which combined with more often training have demonstrated significant improvement in sample complexity (RedQ). What update frequency was used in this paper and is this directly comparable to the baseline methods? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5k6dstFPgBf", "reviewer": "Reviewer_oviQ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this work, the authors proposed a teacher-student framework (TSF), where a\nteacher agent or human expert guards the training of a student agent by intervening and providing online demonstrations that the teacher policy may not be optimal. This is a more realistic setting than standard TSF in which the teacher policy is always assumed to be near-optimal. Specifically, they develop a new method that can incorporate arbitrary teacher policies with\nmodest performance and utilize an off-policy RL method known as Teacher-Student Shared Control (TS2C) to effectively update the student policy. In this method the main novelty is to incorporate teacher intervention based on trajectory-based value estimation. Theoretical analysis justifies also that the proposed algorithm attains efficient exploration and lower-bound safety guarantee without being affected by the teacher’s\nown performance, and this algorithmic finding is also furthered validated with several benchmark experimentations (especially with autonomous driving simulation), showcasing that TS2C can exploit teacher policies at any performance level, maintain lower\ntraining cost and the corresponding student policy can often outperform the imperfect teacher policy.", "summary_of_the_review": "The work is quite neat in terms of addressing the teacher-student framework in RL problem in which the teacher does not necessarily need to be near-optimal. The TS2C framework is also quite flexible and can be adapted to different off-policy RL algorithms. Experiments show the superiority of this algorithm in terms of imitating the teacher, deciding when the teacher should intervene the student during training and having the learned student. outperforming the teacher.\n\nNovelty wise I think the underlying technique has merits in terms of problem formulation, algorithms and experimentation, albeit the theoretical analysis being a bit straight-forward, which is my main concern of the technicality of this work. ", "strength_and_weaknesses": "Strengths:\nThe paper is well-written with concepts clearly explained. \nThe problems studied in this work is important and the concerns of existing TSF work where the teacher is always optimal is a realistic one with good real-world implications. \nExperiment is thorough, validating the student policy can outperform the teacher in the TS2C algorithm. It is also backed with some theoretical analysis to justify this approach\n\nWeaknesses:\nDiscussions of some experimental parameters are still missing. For example it is still unclear how the epsilon parameter is chosen. \nThe theory part of this work is very straight-forward, the bound developed is pretty much a direct result from existing work.\nespecially lemma 1 and theorem 3.2 are pretty much standard results from occupancy distribution matching and policy-value differences in RL (e.g., TRPO-style proofs).\nHow tight is the bound in Theorem 3.3 (it seems with the (1-\\gamma)^2 in the denominator the bound is rather loose), and how to effectively determine the average entropy in practice to validate the tightness of this bound?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "3YfjRHDoVfM", "reviewer": "Reviewer_wVZY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper considers the setting where one is provided a reference policy and we assume that we could be able to estimate or access the value function of the reference policy. The paper thus proposes the method called TS2C where they construct a mixture policy of the student policy and the reference policy and proposes a heuristic to decide which policy to use at run time. The experiment results show that the student policy indeed can overperform the reference policy in practice. ", "summary_of_the_review": "The writing and the experimental results of the paper are good, but at the current stage, I am not convinced that the paper actually makes enough technical contribution given similar previous works. I would recommend a rejection.\n\n======================\nIncreased the score after the rebuttal.", "strength_and_weaknesses": "# Strength\n* The paper is easy to follow, and the proposed algorithm is simple. \n* The experiment results indeed confirm that the student policy could eventually outperform the reference policy, validating the intuition of the algorithm.\n\n# Weakness\n* The paper should consider a more diverse set of benchmarks.\n* The theory results in the paper are not very illuminating: for example, I am having a hard time understanding the significance of Thm. 3.3: $H$ should also depend on $\\epsilon$ because $\\epsilon$ defines $\\pi_b$. Thus what exactly does the term $\\sqrt{H-\\epsilon}$ mean?\n* The clearest issue of the paper is that the proposed algorithm seems like that the proposed algorithm is highly relevant to AggreVaTe [1] and LOLS [2]. The overall algorithm framework is very similar, and I would say the differences seem rather insignificant. I would assume that works such as [1,2] are overlooked in the literature search, so it would be great to compare and argue in the paper why the contribution of TS2C given the existence these previous works. \n\n[1] Ross, Stephane, and J. Andrew Bagnell. \"Reinforcement and imitation learning via interactive no-regret learning.\"\n\n[2] Chang, Kai-Wei, et al. \"Learning to search better than your teacher.\" ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%