Two-Level Actor-Critic Using Multiple Teachers
Authors: Su Zhang
Srijita Das
Sriram Ganapathi Subramanian
Matthew E. Taylor
Venue: Accepted by TMLR ()

Abstract: Deep reinforcement learning has successfully allowed agents to learn complex behaviors for many tasks. However, a key limitation of current learning approaches is the sample-inefficiency problem, which limits performance of the learning agent. This paper considers how agents can benefit from improved learning via teachers' advice. In particular, we consider the setting with multiple sub-optimal teachers, as opposed to having a single near-optimal teacher. We propose a flexible two-level actor-critic algorithm where the high-level network learns to choose the best teacher in the current situation while the low-level network learns the control policy.

Published in Transactions on Machine Learning Research (11/2023)
Two-Level Actor-Critic Using Multiple Teachers
Su Zhang
su.zhang2 edu
Woshington State University Pullman, United States
Srijita Das
srijita.@ualeert..om
University of Alberta Edmonton, Canada
Sriram Ganapathi Subramanian
Vector Institute
Toronto, Conada
Matthew E. Taylor
matthew. tay.or
University of Alberta
Alberta Machine Intelligence Institute Edmonton, Canada
Reviewed OpenReview: https: https://openrevveaw openreview. net/ forum? id= Ld=
Abstract
Deep reinforcement learning has successfully allowed agents to learn complex behaviors for many lasks. However, a key limitation of current learning approaches the sample- inefficiency problem, which limits performance of the learning agent. This paper considers how agents can benefit from improved learning via teachers' advice. In particular, we consider the setting with multiple sub-optimal teachers, as opposed to having single near- optimal teacher. We propose flexible two-level actor-critic algorithm where the high-level network learns to choose the best teacher in the current situation while the low-level network learns the control policy.
1 Introduction
Reinforcement learning (RL) has been successful in a variety domains ranging from solving difficult games like Go (Silver et al. 2016) and Dota (Berner al. (019)) to more important problems like drug discov- ery (Gottipati et al. 2021). Most of these domains are characterized by high dimensional state and continuous action space. However, sample inefficiency is one of the major challenges of applying these algorithms to real-world tasks like robotics and healthcare (Ibarz et al. 2021) Millions of physical interactions with the environment are typically infeasible due to hardware degradation, safety issues, etc.
To address improved sample efficiency, rather than forcing agents to learn from scratch, domain knowledge from humans or existing agents can be leveraged in various ways (Da Silva & Costa 2019). Reward shap- ing Ngg et al. 1999) and policy shaping (Griflith et al., 2013) are two popular techniques that can bias the learning agent's 's reward policy, respectively, for faster learning. A different method is action advising (or- rey & Taylor 2013), where human agent teacher can tell a learning student what action to take in a state. This approach is particularly useful because the teacher and the student agent can have completely different internal representations (which not. generally true in transfer learning settings (Taylor & Stone ).09
Many existing approaches can leverage advice in RL from a single, near-optimal teacher Hester et al. 2018
Warnell et. al. 2018 Bignold et. al. 2021). Tn this work, consider settings where student receive

---

Published in Transactions on Machine Learning Research (11/2023)
action advice from multiple teachers. This setting can be particularly appropriate when different teachers have different skills (e.g., teachers may perform well in different parts of the state space or perform different sub-tasks). In addition, we also consider that teachers may be suboptimal or even random. This allows us to leverage teachers that perform well while not being hurt (much) by teachers that perform poorly. This paper focuses on the question; "When should the student listen to which leacher?" to effectively use the best teacher's policy for a given state or to decide not to listen to any teacher
As motivating example, let consider the example of an agent learning how to drive. In this scenario, teacher (TT) could know how to effectively do basic driving skills while another teacher (T2) knows how to do night-driving specifically. These teachers could then collectively assist the student agent in the appropriate parts of state-space to achieve the sub-tasks, despite not having knowledge about the entire driving task. Our novel approach would allow a student agent to learn to query T1 more frequently for advice regarding basic driving skills and T2 for night-driving protocols while learning to drive safely. While the above example is similar to hicrarchical RL, in this work, the main task need not be divided into sub-tasks and the teachers could be sub-optimal.
Contributions: Our novel approach:
1. learns from the action advice of multiple teachers by extending the actor-critic framework (Mnih et al. 2016) to a two-level framework where the high level learns how to pick teachers to listen to, and the low level learns the agent's policy
2. a general framework that can leverage commonly available sub-optimal or complimentary teachers
3. robust to the quality of teachers, learning even in the presence of poor teachers
4. succeeds empirically, relative to existing methods in a variety of tasks like Hopper and Pick & Place
Our work is inspired by Two-Level Q-Learning (TLQL) (Li et al., 2019b) but is different in two ways: (1) it can be applicd to variety of tasks having both continuous and discrete action-space and (2) is more resistant to leachers of different qualities ranging from fully optimal to partially sub-optimal teachers. In addition, by using two cetor-critic networks, our approach can be easily incorporated into any existing actor-critic algorithm.
2 Background and Prior Work
2.1 Background
Reinforcement Learning: RL (Sutton & Barto 2018) tasks are often formalized as Markov decision processes (MDPs). The MDP modelled as an infinite horizon system (S,A,r,T.h) where S is the state space, A is the action space, r is the immediate reward, T the transstran probability, and y E [o, 1) is the discount factor. An agent receives state 8, S from the environment at time-step t, takes action a E A, and after interacting with the environment, gets reward rt-1 and transitions to the next state 81+1 S. The agent maximizes the expected return R2 = E ** the discounted sum of rewards from time-step t, and acts according to ा.
Advantage actor-critic: The advantage actor-critic (A2C) method the synchronous variant of the asynchronous advantage actor-critic (A3C) MMiih al. 2016 algorithm. A2C consists of two networks: the policy network 7 (acls:;0) and the value network: V(s;;; The policy and the value network can either have shared parameters or can be entirely different networks with parameters 0 and 0', respectively. The loss function optimized by the policy network is L(0') log (a,]s; (R V(s,; 0')), where R is the n-step return calculated as R = "Dp-1''ra + 7 0'). The state value function V(si; 0') is called baseline [Williams] 1992) and is used to reduce the variance of the policy estimate. The value network minimizes the squared Toss (R (ss:; )² to update the parameters.
This work assumes teachers do learn and not antagonistic, which are instead left. to future work.
2

---

Published in Transactions on Machine Learning Research (11/2023)
2.2 Related Work
Early approaches to incorporating multiple teachers include aggregating multiple teachers into a single teacher by using weighted mixture distributions (Jacobs et al. [991] or aggregating the action distributions for each teacher (Hinton 2002). Recent imitation learning work includes combining multiple suboptimal teachers using online learning (Cheng et al.) 2020), bi-level optimization to learn the confidence of experts from demonstrations (Zhang et al. 2021), and using ad advantage functions to decide and take advice from the best teacher in a state (Li et al.) 2019ah. Our proposed work is a reinforcement learning framework that incorporates action-advice from multiple teachers and differs from imitation learning.
Knowledge reuse from multiple teachers (single-agent): Different measures like similarity and regret bounds (Fernández & Veloso, 2006 Azar et al., 2013 Zhhan et al.) 2016) have been used in prior work to incorporate advice or policies from multiple teachers into single-agent learning. However, the effectiveness of these approaches complex and continuous control tasks has not yet been demonstrated. Gimelfarb ctt al. (2018) use Bayesian model to combine potential based reward/vall functions from multiple teach- ers. Our approach uses action-advice, which is more practical than leveraging value functions from multiple experts. [Seita et al.] (2019) proposed to select the best suited teacher which is at least on par or slightly better than the student agent. However, their work could not accommodate partial tcachers, and they query teachers to seek demonstrations. Most similar to our problem setting is AC-Teach (Kurenkov et al., 2019), which uses an ensemble of suboptimal teachers that could be partial contradictory in a Bayesian actor-critic framework to guide the student agent. AC-Teach uses strategies to be able to listen to the same teacher for longer period, which is unsuitable for tasks where different teachers might give useful advice in different parts of the task.
Knowledge reuse from multiple teachers (multi-agent)) A lot work has been done to reuse knowledge from other agents in multi-agent setting (Da Silva & Costa 2019). Da Silva et al. (2017) propose framework in which the agents learn when to seek or give advice based on heuristics like state-visitation frequency. Omidshafici et al. (2019) provide more general framework where the teacher agent decides when and what to advise, and the student agent learns to utilize the received action-advice. Kim et al. (2011) scale prior work to larger and more complex domains by improving the teacher's policy by considering the effect of advice on the student's 's learning. All the above-mentioned work shares similarities because the role of the teacher and student is not fixed and varies depending on their confidence or uncertainty about a task. Our work focuses on single-agent learning with multiple sub-optimal fixed teachers.
Layered learning: Li al. (2019c) map the problem of selecting multiple source policies as an option- learning framework to accelerate contextual policy reuse in transfer learning. Zhang & Whiteson (2019) uses two-level actor-critic framework for option learning. However, their framework doesn't 't accommodate advice from teachers. Our method is not an option learning framework and both the networks use the same underlying MDP in contrast to the former which uses two different augmented MDP at each layer. Our work is also inspired by Li et al. (019b)), who use a two-level Q-learning algorithm to select the best teacher and its action advice, respectively. However, their evaluations were limited to tabular Q-learning and smaller discrete domains. They also propose a variant with the high-level network modeled as DQN, which still could not be applied to continuous control tasks. Another relevant framework (Xie et al. 2018) uses an external controller as teacher to guide the deep RL agent and uses DQN to select the relevant policy to be used. However, this was limited to policy selection between single teacher and student agent. The effect of the number of leachers and the quality of advice from multiple leachers, including noisy leachers, was not studied systematically in any prior work. A recent work (Subramanian et al., 2022) uses a similar two-level actor-critic framework with sub-optimal teachers in fixed roles. However, the sctup is multi-agent single teacher as opposed to our single-agent multi-teacher setup. Our proposed work can apply to both discrete and continuous control tasks and scales well with a number of noisy teachers and advice quality.
3

---

Published in Transactions on Machine Learning Research (11/2023)
St.7t
St.
High-level
Actor-Critic Ohigh
policy Rhigh
A-C Oww
Low-level policy Nlow
Teacher
Teacher
Teacher N
Make selection and take low-level action
Agent
ar
St+1, Tt+1
Environment
Figure 1:
Two-Level Actor- Critic Structure
3 Two-Level Actor-Critic Using Multiple Teachers
We introduce Two-Level Actor-Critic (TI-AC) method to learn from multiple teachers. This method extends the actor-critic algorithm to two-level network structure, with single critic for both levels, enabling the agent to leverage teachers with different expertise and advice quality.
3.1 Problem Statement
Given: A set of (sub)optimal teachers as denoted by Ep = fei, 62, "., ,e,e] corresponding policies denoted by mg {Tee Tegy "" Tex
[ where Eol = N, with a set of
Objective: Train the learning agent by leveraging action-advice from multiple teacher policy ag when useful to make the agent learn good policy with fewer environment interactions.
Assumptions: We consider a single-agent learning problem with multiple teachers. The teachers are fixed during the agent training process. The teachers can be either human teachers or agents. The teachers can always provide action advice to the student about the current state, and the student need not know the decision mechanics behind the advice. The action advice could be optimal or sub-optimal, depending on the teachers' expertise. We explicitly assume this work that communication between the udunt and teacher is unlimited, free, and uncorrupted. Hence, teachers are always available in this setup.
The multiple teachers used in our approach can be pre-trained agents, classifiers trained from demonstrations, human teachers, etc. At each time-step 1, each individual teacher e provides action advice based the state vector 8t. The student agent will then choose an action to perform according to certain policy, which can either its own policy or the policy any one the several teachers in the set E. The goal of the learning agent is to maximize the reward. Meanwhile, the optimal policy of how to wisely listen to multiple tcachers should also be found.
3.2 Algorithm and Methodology
Figure I shows how the two-level actor-critic structure, detailed in Algorithm The algorithm collect minibatch M with Function collect(Trioh) Tlow, E, m)) (Lines 11 23). The high-level policy is mapping from states to teachers, (s) 5 + E, where E Eg U Agent. The high-level policy also considers the agent itself. The low-level policy is a mapping from states to actions, (s) S - A. During collecting the transactions, at state St, the behavior policy first follows the high-level policy Thigh to choose which teacher listen (Line 14). Once teacher selected, denoted e,, the algorithm seleets an action according this teacher's policy: a₃ < ne, (Line 18). If the agent is selected by the high-level policy, the algorithm selects an action with the agent's 's policy at +- Niow (Line 16). The resulting transition, (St, at St+1, Pt, et); is saved
4

---

Published in Transactions on Machine Learning Research (11/2023)
to the minibatch (Line 19). To update those two networks, in each epoch, the advantage is calculated with
Eq. 2 (Line 7). This advantage calculates the low-level loss with Eq. 3 and updates the low-level network (Line 7), calculating the high-level loss with Eq. and updating the high-level network (Line 8).
Algorithm 1: Two-Level Actor-Critic using Multiple Teachers
Teachers set Eo, union with the RL agent = Eo U Agent, minibatch size m, epoch # Initialize: low-level policy Tlows high-level policy Thigh
Input:
for each episode do repeat
M colleect (Thighh Nlow; E, m)) for epochs in N do
Collect rollouts
6
= { = (st, az, Tt, S1+1, ex) @t)
4 Update with minibatch M
7
Calculate advantage A(s a+) = rt + V Rlo ($4+1) V V... (8t)
8
Update low-level network Otow by optimizing the loss [(Otow) = log Noo (a|s; ) a)
Update high-level network thigh by optimizing the loss [(Ohigh) log Thigh (c s; thigh ( A(s, a)
9
10
until episode end;
11
Function collect (Thigh, Tlaws E, m))
12
Get state s, ~ Env
13
repeat
14
et f (elsh) if is Agent then
Select teacher e with high-level policy Thigh
16
a: + (alss)
Select action with low-level policy Now
17
elsc
18
a: Ta,
>> Select action with teacher policy Te.
19
Exccute action ag and get transaction T++1 = (8t, at Set1.Tt.0 Save transaction T&41 to minibatch M MU {T+11
20
21
81 f St11
22
until M reach size m,
23
return M
Low-level Policy: Select Action The low-level policy network is the first actor-critic network, parameter- ized with flow. The low-level policy Toow (als; (a]s; maps the states to a probability distribution over actions. The discounted value at state under the low-level policy Tiow is
V VatOn (8) = (r(s
(1)
The advantage of laking action d, at state 8t
A(s,, ar) = r(st, Vanin (8c+1) V (St)
(2)
The objective is to maximize the agent's 's value over all states and find the low-level optimal policy with the loss function
[(Orow) = log Timo (als; Tomo
Otowe (ss,a)
(3)
High-level Policy: Select Teacher and Take Advice The high-level policy network is the second actor-critic network, parameterized with 0high- The high-level policy (high(E|s; Ohigh) maps the states to a probability distribution over teachers. The high-level network estimates the expected return of selecting each teacher e E E. The reward for a teacher is the same as the environment reward that the agent received executing this teacher's action advice. Hence, have
r(st, = r(s,, a.le)) r(st,
(4)
2The subscript 't dropped from notation for convenience because the loss-function defined with respect single bani-

---

Published in Transactions on Machine Learning Research (11/2023)
The discounted value at state under the high-level policy Thigh is
V Thish (8) = E
>'r Ex)lsr
t=
E
r' ac)ls
V Now (s).
(5)
The advantage of choosing teacher e, at state at the high-level could be written
A(st, ex) == r(s,, Ex) + Vason (8t+1) V Khhhus (5t) r(ss; at) + Vin ($(+1) - V Vi (8t)
= A(s,, at)
(6)
This indicates that the two level networks are using the same critic for estimating the value function of the states. The objective to maximize the value over all states and find the optimal high-level policy with the loss function
[(0high) log Fhighh(ess: (high) A(s,a)
In this way, we use one single critic for both high-level and low-level policies instead of separate ones. This single critic estimates value function based on the current state. A similar reduction shown by Zhang & Whiteson (019)) in an option-learning framework.
4
Experiments
The experiments are designed to investigate the following two research questions:
R1: Can the two-level actor-critic method handle a mixture of sub-optimal teachers relative to existing methods?
Can the two-level actor-critic method incorporate multiple partial teachers with different arcas of expertise?
R2:
4.1 Experimental Settings:
We present experimental results on a discrete control task (DoorKey) and two continuous control robotics tasks (Hopper and Pick & Place) to demonstrate the effectiveness of our method. The environment details for our experiments are below:
1. Door & Key Environment is grid room environment from the Mininalistic Gridworld Eavironment (MiniGrid) (Chevalier- Boisvert et al.) 2018 The rooms are separated by wall: the agent needs to use the key to open the door and enter the other half to get to the target grid. The state is a 3-tuple vector, and there are discrete actions. (sparse) positive reward given when the goal is reached; otherwise, it is
2. Hopper two-dimensional onc-logged robot, and the task to hop forward as far as possible. We use the BuuperrPyBllllEnn--Ve from the PyBullet Gymperium (Ellenberger) 2018 2019) for the experiments. The 15-dimensional state space and the 3-dimensional action space are both continuous. The dimension of state space is 15, and the dimension of action space is 3. The dense reward is calculated based on its current potential, alive bonus, and power cost.
3. Pick & Place a robotic manipulation task from the Fetch environments (Plappert et al. 2018 The goal is to pick up a box and place it at a (fixed) target location. The observation space (continuous) 25-dimensional vector, and the action space 4-dimensional continuous vector to control the end effector and the gripper. The dense reward is based on the distance between the box and the target.

---

Published in Transactions on Machine Learning Research (11/2023)
(a) MiniGrid: DoorKey
(b) PyBullet: Hopper
(c) MuJoCo: Pick & Place
Figure 2: Experimental Domains
Baselines: We use the A2C algorithm without any advice as the baseline. We also benchmark against the DQN variant Two-level Q-learning (DQN-TI.QI.) (Li et al. 2019b) and AC-Teach (Kurenoov al. 2019). both of which can learn from multiple teachers. DQN-TLQL does not support continuous control. Hence, experiments could not be performed on Hopper and Pick & Place domains for this benchmark. We only consider state-of-the-art algorithms with advice from multiple teachers for fair comparison.
Implementation Details: The A2C baseline uses the default setting of the Stable Baseline3 (Rafin et al. 2019). For DQN-TLQL, since we do not have access to the original codebase, we refer the Stable Baseline3 and implement the DQN version of the Two-level Q-learning algorithm described in (Li al. 2019b). The DQN-TLQL and A2C hyperparameters are listed in Appendix B The implementation and parameter settings of AC-Teach are from the original codebase of Kurenkov el al.'s work (Kurenkov et al. 2019
Teacher Set Details: To construct the teacher ser S we first obtain a near-optimal policy, then introduce different levels of noisd to this policy to get good teacher, noisy teacher, and random teacher. We also construct partial teachers for DoorKey and Pick Place, who could only provide good advice on partial state space. Details of the teacher set for each experiment are Appendix A.1.1 A.2.1 and A.3.1
We consider two widely-used evaluation metrics in RL, which have been proven to be robust in evaluating the performance of learning from teachers approaches: 1) Jumpstart, the improvements of initial performance over the benchmark agent (Taylor & Stone 2009) and 2) Mcan training reward, the plotted learning curve as well as the standard deviation (Sutton & Barlo 2018).
Performance Metric:
To guarantee that all algorithms are compared with the same number of updates, the batch size and gradient steps sctting are adjusted accordingly for both DQN-TIQL and TL-AC. We report the training performance of all the algorithms for fair comparison [ All DQN-TLQL, A2C, and TL-AC results averaged over 5 seeds. Due the limited computing resources and the observation of its relatively steady performance, the AC-Teach results are averaged over seeds.
4.2 Results and Analysis
This scction presents results from the three domains. We address R11 for all three domains. For R2, since it's difficult to obtain partial teachers with pre-trained agents in Hopper, we only conduct experiments with the DoorKey and the Pick & Place domain
R1: Can TL-AC handle a mixture of sub-optimal teachers relative to other existing methods? To answer R1, we provide advice from four sets of three teachers of varying quality to the RL learning agents, including the baseline agents Increasing the number of good teachers will not provide extra help accelerating learning. The results of DoorKey are shown in Figure the A2C baseline does not learn in this domain without any external bias because of sparse reward; hence, intrinsic motivation related approaches
https://github comm/StanforddU/ac-teeee
"Teachers could be humans agents. In this work, use fixed set of trained agents.
WW currently experimented with the fixed percentage of random noise and Gaussian action noise of certain scale.
6In (Kurenkov et al. 2019), the authors report the testing performance of AC-Teach without the teachers. the progress of each step (velocity), the absolute coordinate, it is difficult to find an explicit way to split optimal teacher into two partial teachers.
Becaase the state of Hopper only
8The underlying assumption that single good teacher sufficient solve this task. However, might be difficult. identify them from a pool of teachers.

---

Published in Transactions on Machine Learning Research (11/2023)
TLAC
DON TLOL
AC Toach
A2C baseline
1.0
0.8
Average Neward
0.6
0.6
0.4
0.4
02
0.0
Number Timastops (a) With good teachers
Number (c) With good teacher and naisy teachers
Number Timesteas
(b) With 2 good teachers and 1 noisy teacher
no teachers
Figure 3: DoorKey results show the performance impacts of different teachers. All three algorithms perform well with good teachers. With 3 noisy teachers, our TL-AC method (green line) starts with lower initial performance but gradually catches up and converges better performance.
are used for this domain CChevalier-Boisver et al. 2018). With 2 and 3 good teachers, TL-AC has a similar performance level to DQN-TLQL and performs better than AC-Teach (Figures 3a) and (b)). When there are two noisy and one good teacher (Figure B(c)), TL-AC could effectively outperform both DQN-TLQL and AC-Teach. When only noisy teachers are provided (Figure d)), compared the settings in Figure 3 (c),, the performance of AC-Teach drops slightly but has higher jumpstart, TL-AC starts with lower initial performance but gradually converges to near-optimal performance. DQN-TLQL does not learn from combination of only noisy teachers. For TL-AC, in the beginning, the agent needs to either listen to noisy teachers' advice or act with its own policy, which is random at this stage. When the student's own policy (i.e., the low-level policy) gradually improves as training progresses, it will tend to choose its own policy over the noisy teachers'. Figure 3(d) shows more pronounced learning progress as compared to Figures 3 (c). AC-Teach follows one single selected (teacher) policy for longer time horizon than TL-AC and switches the selection with estimated confidence-basee scheme (Kurenkov et al.) 2019). This may explain why its training performance looks steady but shows fewer improvements over time. This observation holds for all the experiments in this section.
1600
TLAC
ACaThach
AZC baseline
1600
1400
1200
1200
1000
000
800
600
400
200
200
200000 Number of Timesteps
400000
600000
200300
400000
6010c0
200030
400000 600000
Number of Timesteps
XJ0F
Number Timesteps
Number of Timestaps
(b) With good teachers and noisy teacher
(c) With good teacher and noisy teachers
noisy teachers
(e) With good teschers
Figure 4: Hopper results show the performance impacts of providing multiple teachers different qualities. For (b) scenarios, TL-AC could always surpass the A2C bascline. For (d), TL-AC affected by the noisy teachers but gradually learns and improves its policy. Comparatively, AC-Teach performs poorly this task.
We also answer R1 in Hopper (Figure 4 and Pick Place (Figure 5) In scenarios that provide least one good teacher, TL-AC could always learn from it and achieve higher performance. However, when there are only noisy teachers, the performance of TL-AC decreases. In the Hopper task, TL-AC could gradually learn
8

---

Published in Transactions on Machine Learning Research (11/2023)
TLAC
Ac-Fcchh
AZC baseline
-10-
-15-
-20
-25
-3
Number of Timostops
x1.80
Number Timestops
x100
Number of Timestops x1.0
Number Timestops
(a] With 3 good teachers
(b) With 2 good teachers 1 noisy teacher
(c) With 1 good teacher and noisy teachers
(d) With noisy teachers
Pick & Place results show the performance impacts of providing multiple teachers of different qualities. For (a)( TL-AC has similar performance with AC-Teach, while it could be affected by the noisy teachers in (d).
Figure 5:
and improve its policy over time while in the Pick & Place task, it is harder to recover. A possible reason for this performance drop could be that, during the initial learning phase, the noisy teachers' advice could mislead the exploration and the critic (value estimation) update, hence interfering with both the high-level and low-level policy of TL-AC. AC-Teach performs better than TL-AC in Pick & Place with 3 noisy teachers but consistently underperforms A2C and TL-AC in Hopper.
To summarize, the DoorKey experiment showed that TL-AC can effectively learn from multiple teachers of different quality as compared to other baselines in discrete tasks and is especially useful when learning from increasing number of noisy teachers. The Hopper and Pick & Place experiments support the same claim in the innuus tasks, with the exception that it can effectively learn from noisy teachers as long we have at least one good teacher to guide the learning agent.
DON-TLOL
baseline
Averagee feeeard
0.6
0.4
04
0.2
0.0
Number 1 optimal teacher and 4 random teachers
optimal teacher
With 1 optimal teacher and random teacher
optimal teacher and 2 random teachers
Figure DoorKey results show the robustness of providing optimal leacher and increasing the number of random teachers. AC-Teach (orange lines) and TL-AC (green lines) could always maintain good per- formance in all scttings, while DQN-TLQL (blue lincs) is less stcady. Our TL-AC method could quickly converge optimal policy, and it always outperforms other benchmark methods.
To further test the robustness of our method, we experiment with a more extreme setting: providing one optimal teacher and increasing the number of random teachers the teacher set. Figure Seguresshore shows the results of DoorKey domain. TL-AC could outperform the benchmark methods all 4 settings. In Figure 6 d), the jumpstart of our method could be affected by the increasing number of random teachers, but adapts quickly and converge to the optimal policy within 10 training steps (see Table D Meanwhile, the
9In Hopper, AC could catch up with the A2C baseline with longer training time, around 10 training steps, which shown in the figure.

---

Published in Transactions on Machine Learning Research (11/2023)
2000
TLAC
ACYeach
A2C baseline
2000
1750
1250
Average
1000
1000
m
730
0-
200000 400000 800000
200000 900000 6040c0
200000 400000 800000
Number Timesteps x10
Nimber of Timesteps
Number Timesteps
Number of Timestaps
(a) With optimal teacher
(b) With 1 optimal teecher and random teacher
(c) With optimal teacher and rendom Leachers
(d) optimal teacher and random teachers
Figure 7: Hopper results show the robustness of providing optimal teacher and increasing the number of random teachers. TL-AC is robust to this setting and outperforms A2C. AC-Teach performs poorly in this task.
DQN-TLQL is unstable with the increasing number of random teachers, and AC- AC-Teach always maintains a steady performance level much lower than TL-AC. Figure 7 shows the results of the Hopper domain. With increasing numbers of random teachers, TL-AC could still benefit from the optimal teacher and surpass the A2C baseline and AC-Teach. Even with an optimal teacher, the overall performance of AC-Teach after 10 training steps is still less than 200. The results of Pick & Place can be found in Appendix [A.3.3] From this experiment, we conclude that our method is robust to the effect of noisy teachers' advice and could always learn better policy than the benchmark methods in most cases.
R2: Can TL-AC incorporate multiple partial teachers with different. areas of expertise?
1.0
TL-AC
DON-TLOL
AC-Teach
A2C baseline
0.8
Average Neward
0.6
0.4
0.2
2
0
Number Timesteps
Number of Timesteps
Number of Timesteps
x105
Number Timesteps
x10
(a) With 1 fully optimal teacher
(b) With Teacher
(c) With Teacher R
(d) With Teacher L Teacher
Figure 8:
DoorKey results show the capability of incorporating partial teachers. Teacher or Teacher R could only provide good advice in the left or the right room. With Teacher L, our TL-AC (green lines) method will gradually learn a good policy, while DQN-TT.QT, (blue lines) and AC-Teach (orange lines) failed to do so. With Teacher R. none of the algorithms could learn and solve the task with the provided teacher. With Teacher L and Teacher R, our TL-AC method could successfully use the advice from those two teachers better than the other methods.
To answer R2, first experiment with the DoorKey task, allowing Teacher L Teacher R. to provide advice to the agent, and results are shown in Figure The target is located in the right room and the door between rooms is locked. To achieve the larget, the agent needs lo pick up the key, open the door, and then move to the right room. We expected good advice in the left room to more useful than good advice in the right room. In Figure Ft), when only providing Teacher L to the agent, TL-AC (green lines) gradually learns a good policy with this partial teacher, while none of the other methods converges the optimal performance level within the given number of time-steps. When only providing Teacher R, the randomly
10

---

Published in Transactions on Machine Learning Research (11/2023)
suggested actions in the left room will largely disrupt learning. None of the algorithms could eliminate this disturbance and the agent performs poorly in Figure Cc).
In Figure d), when providing both Teacher L and Teacher R to the agent, TL-AC could learn a near- optimal policy within 1 x 10 steps, surpassing the performance of using Teacher L only. This suggests that our TT-AC method could use the useful information from Teacher R and converge to a better policy as compared to the settings of providing those teachers separately. While AC-Teach performs better when using both Teacher L and Teacher R, it could not benefit much from Teacher R. shown in Figure g d). Similarly, DQN-TLQL seems to fail to learn with these settings. It could reuse the advice from optimal teacher trained on the entire task (in Figure (aa) but fails to adopt the advice from the teacher of the left room and the teacher of the right room individually (partial but optimal teacher on the sub-tasks). More details and analyses about the reuse frequency of different teachers' policies of our algorithm can be found in Appendix A.1.3
TL-AC
AC-Teach
A2C baseline
-5-
Average Reward
Number of Timestops x105
Number of Timestops
X10
Number of Timestops x105
Number Timestops
x105
(a) With optimal teacher
(b) With 1 pick teacher
(c) With place teacher
(d) With 1 pick teacher and place teacher
Figure 9: Pick & Place results show the capability of incorporating partial teachers. TL-AC cannot learn a good policy from only a pick or place teacher in (b) but could successfully combine the advice from those two teachers and have a near-optimal performance in (d). AC-Teach(orangg lines) has a similar performance level for all scenarios.
We also experiment with the Pick & Place task. Here, the agent first needs to pick up the box, then move it to the larget location. We use the same hand-coded pick leacher and a place teacher to provide action advice as AC-Teach in Kurenkov et al. (2019). The pick teacher only knows how to move toward the bax and when to grasp it, and the place teacher only knows how to move the box to the goal. The A2C bascline could not learn a good policy for this task within x 10 steps training in accordance with prior work (Nair et al., 2018). Even with lower initial performance as compared to AC-Teach, TL-AC could quickly learn a near-optimal policy with single optimal teacher, or both pick teacher and place teacher, as evident from Figure 9 a) and (d). Teach could learn better policy from only pick teacher
place teacher, while our AL-AC failed to do so, as in Figure g (b) and (c). With only pick teacher a place teacher, the agent still needs to learn a policy from scratch to accomplish the other sub-task (place or pick). Since our low-level policy is based on a simple actor-critic method, which could not learn well in this continuous control task or the sub-task without teacher guidance, therefore our method performs worse than AC-Teach in Figure g b) and (c).
AC-Teach could maintain a steady level of performance in all scenarios but could not learn near-optimal policy with optimal teacher sufficient partial teacher set.
We also analyze the reuse frequency of the pick teacher, place teacher, and agent itself when holding or not holding the box. From Figure TO we can observe that when holding the box, the frequency of listening to the pick teacher (green line) decreases, and the frequency of listening to the place teacher (orange line) increases. The frequency of reusing those two tcachers' policies is about equal between xx 10 x 10 training steps, but training progresses, the agent tends to listen the Place Teacher's advice, and its performance also converges by this time, is evident from Figure dd) When not holding the box, the frequency of
11

---

Published in Transactions on Machine Learning Research (11/2023)
Listen Teacher
Listen Teacher
Listen to Policy
1.0
Frequency
0.6
0.4
0.2
Number of Timesteps
x10
Number of Timesteps
x10
(a) When holding the
(b) When not holding box
Figure 10: Frequency of listening to the partial teachers and agent in Pick & Place.
listening to the pick teacher is higher than the other two policies during the entire training phase. And the frequency of performing the agent's 's own policy (blue line) remains relatively low. In summary, TL-AC could efficiently incorporate the advice from multiple teachers with different areas of expertise, as validated by experiments.
TL-AC
Random
UCB1
-7.
Average Reward
10.0
0.6
0.2
-2.5
Number Timesteps
x10
Number of Timesteps
X10
(8) DoorKey; with Teacher and Teacher
(b) Pick with Teacher Pick and Teacher Place
Figure 11: Using random selection and UCB1 algorithm as high-level policy
To further verify the efficiency and necessity of our state-based high-level policy structure, we also perform ablation studies with partial teachers in DoorKey and Pick & Place. Figure 11 shows the results of comparing our TL-AC method with (1) Randomly selecting the teachers as the high-level policy denoted as Random; (2) Formalizing the high-level teacher selection as a multi-armed bandit problem, and using the UCBI algorithm QAuer 2002) as the high-level policy denoted UCBI. The teacher selected at the beginning of each episode based their estimated return. This structure is similar to the method of Laroche & Feraud (2017). We hypothesize that since UCB1 does not take each state into account, it cannot perform well in both environments. In the DoorKey environment, the random selection has chance of selecting the proper teacher at each state and reaching a good performance, but with high variance. In Pick & Place, the continuous control task is much more difficult and the random baseline cannot perform well.
5 Discussion and Future Work
We introduced Two-Level Actor- Critic algorithm address the task of learning from multiple teachers. With the two-level actor and single critic network, this method can effectively incorporate advice from
12

---

Published in Transactions on Machine Learning Research (11/2023)
multiple teachers with different qualities expertise for both discrete and continuous control tasks for different types of domains.
In discrete tasks like DoorKey, TL-AC could effectively learn from multiple teachers of different qualities, choose the good teachers to listen to, be robust to noise introduced by noisy teachers, and always outperform the baselines. In complex continuous control roboties tasks like Hopper and Pick & Place, TI-AC could still identify good teachers from the pool of teachers. But its capability to recover from bad advice from the sub-optimal teachers depends the teacher set composition, underlying low-level policy learner quality, and task complexity. For instance, in Pick & Place, when the student agent needs to completely ignore the bad advice from noisy teachers and develop its own policy, or when needs to accomplish the "subtask" of moving the picked box to the target location without teachers' advice, TL-AC is not able to outperform AC-Teach. For all the different domains considered in our work, as long as there is full coverage of good advice, TL-AC effectively learns listen to and filter out the noisy guidance from other teachers.
To summarize, advantages of using TL-AC include the following:
1. being lightweight and having a simple structure,
2. casily switching between policies at every time-step to incorporate the best teacher's advice,
3. being adaptable to a wide variety of tasks,
4. simple incorporation into any actor-eritic algorithm, and working with both full and partial teachers.
One limitation of TL-AC includes requiring the teacher to always be available. Second, its performance can be poor when only noisy teachers are available.
As future work, first, conducting experiments with human teachers remains interesting direction. Second, investigating the possibility of incorporating uncertainty and confidence-related schemes into this framework remains another open direction. Third, relaxing the assumption that teachers are always available and introducing a budgeted version of TL-AC where teachers can provide advice a limited number of times are immediate future directions that we aim to pursue.
Acknowledgments
Part of this work has taken place in the Intelligent Robot Learning (IRL) Lab at the University of Alberta, which is supported in part by research grants from the Alberta Machine Intelligence Institute (Amii); a Canada CIFAR AT Chair, Amii; Compute Canada; Huawei; Mitacs; and NSERC.
References
Peter Auer. Using confidence bounds for exploitation trade-ofis. Journal of Machine Learning Research, 300744:3 2002.
Mohammad Gheshlaghi Azar, Alessandro Lazaric, and Emma Brunskill. Regret bounds for reinforcement lcarning with policy advico. In Joint European Conference on Machine Learning and Knowledge Discovery Dulabases, 2013.
Christopher
Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement. learning. arXie preprint arXiu: 1912. 0.66800 2019.
Adam Bignold, Francisco Cruz, Matthew E. Taylor, Tim Brys, Richard Dazeley, Peter Vamplew, and Camcron Foale. A conceptual framework for externall--nffheneed agents: An assisted reinforcement learning review. Journal of Ambient Intelligence and Humanized Computing, pp. 1 24, 2021.
Ching-An Cheng, Andrey Kolobov, and Alckh Agarwal. Policy improvement from multiple experts. NeuRtPSS 2020.
Maxime Chevalier-Boisverr, Lucas Willems, and Suman Pal. Minimalistic gridworld environment for openai gym.
2018.
13

---

Published in Transactions on Machine Learning Research (11/2023)
Felipe Leno Da Silva and Anna Helena Reali Costa. A survey transfer learning for multiagent reinforcement learning systems. Journal of Artificial Intelligence. Research, 2019.
Felipe Leno Da Silva, Ruben Glalt, and Anna Helena Reali Costa. Simultaneously learning and advising in multiagent reinforcement learning, In AAMAS, 2017.
Benjamin Ellenberger. Pybullet gymperium.
gym 2018-2019.
Fernando Fernández and Manuela Veloso. Probabilistic policy reuse in a reinforcement learning agent. In Proceedings of the fifth international joint conference on Autonomous agents and multiagent systems, pp. 720-727, 2006.
Michael Gimelfarb, Scott Sanner, and Chi-Guhn Lee. Reinforcement learning with multiple experts: bayesian model combination approach. NeurIPS, 2018.
Sai Krishna Gottipati, Yashaswi Pathak, Boris Sattarov, Sahir, Rohan Nuttall, Mohammad Amini, Matthew E. Taylor, and Sarath Chandar. Towered actor critic for handling multiple action types in reinforcement learning for drug discovery. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 142-150, 2021.
Shane Grilfith, Kaushik Subramanian, Jonathan Scholz, Charles Isbell, and Andrea L Thomaz. Policy shaping: Integrating human feedback with reinforcement learning. In NTPS, 2013.
Todd Hester, Matej Vecerik, Olivier Pielquin, Mare Lanciot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, lan Osband, et al. Deep q-learming from demonstrations. In AAAI, 2018.
Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Newral computation, 2002.
Julian Ibarz, Jie Tan, Chelsea Finn, Mrinal Kalakrishnan, Peter Pastor, and Sergey Levine. How to train your robot with deep reinforcement learning: lessons have learned. The International Journal of Robotics Research, 2021.
Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey Hinton. Adaptive mixtures of local experts. Newral Computation, 1991.
Dong-Ki Kim, Miao Liu, Shayegan Omidshafiei, Sebastian Lopez-Cot, Matthew Riemer, Golnaz Habibi, Gerald Tesauro, Sami Mourad, Murray Campbell, and Jonathan P How. Learning hierarchical teaching policies for cooperative agents. In AAMAS, 2011.
Andrey Kurenkov, Ajay Mandlekar, Roberto Martin-Martin, Silvio Savarese, and Animesh Garg. Ac-teach: A bayesian actor-critic method for policy learning with an ensemble of suboptimal teachers. CoRL, 2019.
Romain Laroche and Raphael Ferand. Reinforcement learning algorithm selection. arXiv preprint D:XXiv1701.08811 2017.
Guohao Li, Matthias Müller, Vincent Casser, Neil Smith, Dominik L Michels, and Bernard Ghanem. Oil: Observational imitation learning. In Robotics: Science and Systems, 2019a.
Mao Li, Yi Wei, and Daniel Kudenko. Two-level qlearnnig: learning from conflict demonstrations. The Knowledge Engineering Review, 2019b.
Siyuan Li, Fangda Gu, Guangxiang Zhu, and Chongjie Zhang. Context-aware policy reuse. In AAMAS, 2019c.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukeuoglu. Asynchronous methods for deep reinforcement learning. In ICML, 2016.
Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojeiech Zaremba, and Pieter Abbeel. Overcoming exploration in reinforcement learning with demonstrations. In ICRA, 2018.
14

---

Published in Transactions on Machine Learning Research (11/2023)
A Ng, D Harada, and S Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In ICML, 1999.
Shayegan Omidshafiei, Dong-Ki Kim, Miao Liu, Gerald Tesauro, Matthew Riemer, Christopher Amato, Murray Campbell, and Jonathan P How. Learning to teach in cooperative multiagent reinforcement learning. In AAAI, 2019.
Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Macick Chociej, Peter Welinder, et al. Multi-goal reinforeement learning: Chal- lenging robotics environments and request for research. arXiv preprint arXiv:1802 (rX:::102.00464, 2018.
Antonin Raflin, Ashley Hill, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, and Noah Dormann. Stable baselines3. httpe:://github.com wttpr////otn///taaeeeeee-eeeeeee ht//staabl-baaelieess 2019.
Daniel Seita, David Chan, Roshan Rao, Chen Tang, Mandi Zhao, and John Canny. 7pd teaching strategies for deep reinforcement learning from demonstrations. arXiv preprint arXiv:1910.1211 2019.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwiescr, Ioannis Antonoglou, Veda Panneershelvam,, Marc Lanctot, et al. Mastcring the game of go with deep neural networks and tree search. Nalure, 2016.
Sriram Ganapathi Subramanian, Matthew E Taylor, Kate Larson, and Mark Crowley. Multi-agent advisor q-learning. JAIR, 2022.
Richard S Sutton and Andrew G Barlo. Reinforcement learning: An introduction. MIT press, 2018.
Matthew E Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research, 10(7), 2009.
Lisa Torrey and Matthew Taylor. Teaching on a budget: Agents advising agents in reinforcement learning. In Proceedings of the 2013 international conference on Autonomous agents and multi-agent systems, pp. 1053-1060, 2013.
Garrett Warnell, Nicholas Waytowich, Vernon Lawhern, and Peter Stone. Deep tamer: Interactive agent shaping in high-dinensional state spaces. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 1992.
Linhai Xie, Sen Wang, Stefano Rosa, Andrew Markham, and Niki Trigoni. Learning with training wheels: speeding up training with simple controller for deep reinforcement learning. In 2018 IEEE international conference on robotics and automation (ICRA), pp. 2276-6283. IEEE, 2018.
Yusen Zhan, Haitham Bou Ammar, and Matthew E Taylor. Theoretically groundde policy advice from multiple teachers in reinforcement learning settings with applications to negative transfer. IJCAI, 2016.
Shangtong Zhang and Shimon Whiteson. Dac: The double actor-critic architecture for learning options. arXiv preprinl Xin:
Xix:1104,, 1904. 1 12691, 2019.
Songynan Zhang, Zhangjic Cao, Dorsa Sadigh, and Yanan Sui. Confidence-aware imitation lcarning from demonstrations with varying optimality. In NewrIPS, 2021.
15

---

Published in Transactions on Machine Learning Research (11/2023)
A Additional Experiments Details
A.1 DoorKey Experiment
A.1.1 DoorKey: Teacher Set
The MiniGrid- MiniGrid-Doorrey-TSSe DoorKey-7x7-v0 task has sparse reward, the agent could only get reward after successfully reaching the target square. The success requires the agent perform sequences of behaviors: pick up the key, open the door, and reach the larget square. This task is good lestbed to evaluate whether our method could incorporate multiple teachers with different expertise. Since the task difficult to learn for the A2C bascline, we obtain optimal policy with PPO algorithm, which trained for 1 x 10 steps. The training reward of this agent is 0.96, testing mean reward of 1000 episode is 0.77 and standard deviation is 0.38.
With this optimal policy, we construct teacher set as follows:
Optimal Teacher: Always give advice according to the optimal policy.
Good Teacher: Give advice according to the optimal policy with probability of 0.8, and give advice randomly with a probability of 0.2. The average performance is 0.83.
3. Noisy Teacher: Give advice according to the optimal policy with probability of 0.2, and give advice randomly with a probability of 0.8. The average performance is 0.30.
4. Teacher L: Give advice according to the optimal policy when at the left room, and give random advice when at the right room. It knows how to pickup the key and open the door.
Teacher R: Give advice according to the optimal policy when at the left room, and give random advice when at the right room. knows how to navigate to the target square.
6. Random Teacher: Always give advice randomly. The average performance is 0.007.
A.1.2 DoorKey: Jumpstart
Table shows the jumpstarts of benchmark methods in DoorKey.
Table 1: Jumpstart of benchmark methods with providing different teacher set
Teacher Set
TL-AC
AC-Teach
DQN-TLQL
3 Good Teachers
0.85
0.62
0.83
2 Good Teachers +
0.80
0.66
0.76
Noisy Teacher
1 Good Teachers +
0.79
0.69
0.67
2 Noisy Teachers
3 Noisy Teachers
0.13
0.63
0.26
1 Optimal Teacher
0.81
0.72
0.77
Optimal Teacher +
0.73
0.58
0.64
1 Random Teacher
1 Optimal Teacher +
0.52
0.60
0.26
2 Random Teachers
1 Optimal Teacher +
0.31
0.59
0.26
4 Raudom Teachers
Full Optimal Teacher
0.81
0.72
0.77
Teacher L
0.23
0.74
0.05
Teacher R
0.03
0.0
0.002
Teacher L Teache R
0.29
0.57
0.0014
Note: In situations where our TL-AC has a lower jumpstart, it could quickly catch up in the later training stages and surpass the performance of DQN-TLQL and AC-Teach.
16

---

Published in Transactions on Machine Learning Research (11/2023)
Listen to Teacher
Listen to Teacher R
Listen to Agent Policy
Frequency
Number Timesteps
x10
Number Timestaps
x16
Left
(b) Right
Figure 12: Frequency of listening the partial teachers and agent in DoorKey.
A.1.3 DoorKey: Reuse Frequency of Teacher and Teacher R
We also check the reuse frequency the Teacher L, Teacher R, and the agent itself in the left room and right room. From Figure 12 we could observe that the beginning, the frequency of listening to Teacher L (green line) performing agent's own policy (blue line) are about equal (due initial exploration). In the left room, the frequency of reusing Teacher L's policy first increases to reuse the useful information, then decreases as training progresses, and frequency of performing agent's 's own policy first increases and remain at steady frequency level as the agent learns over time. We should focus our attention to the region before x 10 steps for this behavior because (as evident from Figure g the TL-AC agent converges to the optimal policy by this step. The sub-task of the right room is to go to the larget square, which simpler task for agent to learn. In the right room, the frequency of Teacher L's policy drops as training goes, and the agent gradually prefer to perform its own policy (blue line) rather than listen to Teacher L Teacher R.
A.1.4 DoorKey: Performance at the Beginning Stage
AL
DON TLOL
AC Teach
0.8
0.2
0.0
0.0
0.2
0.4
0.6
0.0
1.0
1.0 x10*
0.0
0.2
1.0
0.4
0.6
0.8
1.0
Timesteps
x10*
Timesteps
Timesteps
x10²
Timesteps
x10
(a) With 3 good teachers
(b) With 2 good teachers
(c) With 1 good teacher and noisy teachers
(d) With 3 noisy teachers
and noisy teacher
Figure 13: First 10y steps of the results show in Figure
A.2
Hopper Experiment
A.2.1 Hopper Teacher Set:
This task is a good testbed to evaluate whether our method could handle high-dimensional continuous state and action space. Since the DQN-TLQL could not be applied to this continuous action task, we only compare with the A2C baseline and the AC-Teach method in the following experiments.
17

---

Published in Transactions on Machine Learning Research (11/2023)
For the teacher set construction, first obtain an optimal policy with SAC algorithm, which trained for 1, 1,000,000 steps. The mean reward of this policy over 1000 testing episode is 1928 and standard deviation is 646.
With this optimal policy, we construct teacher set as follows:
Optimal Teacher: Always give advice according to the optimal policy.
Good Teacher: Give advice according to the optimal policy with probability of 0.8, and give advice randomly with a probability of 0.2.
3. Noisy Teacher: Give advice according to the optimal policy with probability of 0.2, and give advice randomly with a probability of 0.8.
4. Random Teacher: Always give advice randomly.
A.2.2 Hopper:
Jumpstart
Table shows the jumpstarts of benchmark methods in Hopper. Our TL-AC method could always have higher jumpstart compared to AC-Teach.
Table 2: Jumpstart of benchmark methods with providing different teacher set
Teacher Set
TL-AC
AC-Teach
3 Good Teachers
60.29
-14.95
2 Good Teachers +
7.82
-26.13
1 Noisy Teacher
1 Good Teachers +
0.9
-33.17
2 Noisy Teachers
3 Noisy Teachers
-25.09
-29.98
Optimal Teacher
36.61
-23.07
1 Optimal Teacher +
19.34
-22.59
1 Random Teacher
1 Optimal Teacher
-7.71
-23.51
2 Random Teachers
1 Optimal Teacher
-11.84
-30.02
4 Random Teachers
A.3 Pick & Place Experiment
A.3.1 Pick & Place Teacher Set:
This task is for evaluating whether method could handle high-dimensional continuous control task. Since the DQN-TIQL could not be applied to this continuous action lask, only compare with the A2C baseline and the AC-Teach method in the following experiments.
For the teacher set construction, we use the same optimal policy as Kurenkov et al.'s work Kurenkov et al. (0199: first directly move to the box and grasp it, then move towards the target location and drop it. The mean reward of this policy over 1000 testing episodes -5.83 and standard deviation is 1.23.
With this optimal policy, we construct. teacher set as follows:
1. Optimal Teacher: Always give advice according to the optimal policy,
2. Good Teacher: Give advice according to the optimal policy, but with addition action noise of 0.01.
The mean reward of this leacher is -5.96 and standard deviation is 1.26 over 1000 lesting episodes.
Noisy Teacher: Give advice according to the optimal policy, but with addition action noise 0.1. The mean reward of this teacher is -10.82 and standard deviation is 5.36 over 1000 testing episodes.
4. Pick Teacher: Give advice according the optimal policy when hold the box, will move towards the box and pick it up.
18

---

Published in Transactions on Machine Learning Research (11/2023)
5. Place Teacher: Give advice according to the optimal policy when hold the box, will move towards the target location.
6. Random Teacher: Always give advice randomly. The mean reward of this teacher is -20.62 and standard deviation is 4.31 over 1000 testing episodes.
A.3.2 Pick & Place: Jumpstart
Table 3 shows the jumpstarts of benchmark methods in Pick & Place. When having teacher set of multiple good teachers, our TL-AC method could have higher jumpstart compared to AC-Teach. When having more noisy or random teachers in the teacher set, the AC-Teach method will have a higher jumpstart.
Table 3: Jumpstart of benchmark methods with providing different teacher set
Teacher Set
TT-AC
AC-Teach
3 Good Teachers
13.59
10.60
2 Good Teachers +
12.80
11.73
1 Noisy Teacher
1 Good Teachers +
10.53
10.94
2 Noisy Teachers
3 Noisy Teachers
7.79
10.72
1 Optimal Teacher
6.59
10.49
1 Optimal Teacher +
-1.15
9.56
1 Random Teacher
1 Optimal Teacher +
4.15
9.17
2 Random Teachers
1 Optimal Teacher +
1.75
9.10
4 Random Teachers
Full Optimal Teacher
6.59
10.49
Teacher Pick
0.21
9.62
Teacher Place
0.36
9.59
Teacher Pick Teach Place
2.72
9.58
A.3.3 Pick & Place: Robustness
While in the Pick & Place task, the performance of our method could be affected with the increasing number of random teachers as in Figure 14. From (a) to (d), the mean reward drops and the standard deviation also fluctuated. Even with such fluctuation, TL-AC could still outperform AC-Teach in all scenarios.
19

---

Published in Transactions on Machine Learning Research (11/2023)
TL-A
AC-Tewch
AZC beseline
-5
-10-
-10
-15
Scweage
w
25
-30
Number ber Tarmestess
Namber Timestests x0.0
Number
imeestes
Namber of Timestesss w0007
(a) With 1 optimal taacher
(b) With optimal teacher random
With optimal teacher and random teachers
(6) With optimal teacher
random toachers
Figure 14: Pick and Place results show robustness of providing optimal teacher and increasing the number random teachers. AC-Teach (orange lines) robust to the increasing number of random teachers. Our TL- AC (green lines) could be affected with increasing number of random teachers, but still performs better than AC-Teach.
B Hyperparameter Settings
B.1 Hyperparameters for DoorKey Experiment
Table 4: Hyperparameters of A2C in DoorKey
Number envs
Numer of steps per update
Buffer size
20
Number steps
5e5
Policy
'MIpPolicy'
Entropy coefficient
0.0
Optimizer
RMSprop
RMSProp epsilon
le-5
Value function coefficient
0.5
Maximum value for gradient clipping
0.5
Gamma
0.99
GAE coefficient
0.9
Normalize advantage
True
Learning rate
7e-4
Hidden Layer
pi= (64, 64], vf= (64, 64]
20

---

Published in Transactions on Machine Learning Research (11/2023)
Table 5: Hyperparameters of TL-AC in DoorKey
Number of envs
Numer of steps
16
per update
Buffer size
16
Number of steps
5c5
Epochs
Target KL
0.02
Clipping parameter
0.2
Policy
'MIpPolicy'
Entropy coefficient
0.0
Optimizer
RLMSpropTFLile
RMSProp epsilon
1c-5
Value function coefficient
0.5
Maximun value
0.5
for gradient clipping
Gamma
0.99
GAE coefficient
0.85
Normalize advantage
True
Learning rate
7e-4
high-lovel: pi=[64, 64], vf=[64, 64
Hidden Layer
low-level: pi= (64, 64), vf= (64, 64]
Table 6: Hyperparameters of DQN-TLQL in DoorKey
Number of envs Numer of steps per update
100
Tau
0.2
Replay memory size
10,000
Target update interval
100
Exploration rate
1-+0.01
Exploration fraction
0.1
Number of steps
5e5
Policy
'MIpPolicy'
Maximum value for gradient clipping
10
Gamma
0.99
Learning rate Optimizer
0.0001
Adam
high-level: pi- 664, 64], vf (64, 64
Hidden Layer
low-level: pi- (64, 64). vf= (64, 64
21

---

Published in Transactions on Machine Learning Research (11/2023)
B.2 Hyperparameters for Hopper Experiment
Table 7: Hyperparameters of A2C in Hopper
Number of cnvs
4
Numer of steps
per update
Buffer size
32
Number of steps
1e6
Policy
'MlPPolicy'
Log sld init
-2
Ortho init
False
Entropy coofficient
0.0
Optimizer
CMMSpropTFLine
RMSProp epsilon
le-5
Use gSDE
True
gSDE sample frequency
Value function
0.4
coelficient
Maximum value for gradient clipping
0.5
Gamma
0.99
GAE coefficient
0.9
Normalize advantage
False
Learning rate
linear schedule (0.00096, 5e-4)
Hidden Layer
14, 64), vf- (64, 64
B.3 Hyperparameters for Pick & Place Experiment
22

---

Published in Transactions on Machine Learning Research (11/2023)
Table 8: Hyperparameters of TL-AC in Hopper
Number of envs
4
Numer of steps
per update
Buffer size
32
Number of steps
1c6
Epochs
10
Target KL
lincar schedulc (0.04, 0.01)
Clipping parameter
linear schedule (0.3, 0.1)
Policy
'MlpPolicy"
Log std init
low:-2, high: 0.0
Ortho init
low: False, high: True 0.0001 TFLke
Entropy coefficient
Optimizer
RMSProp epsilon
le-5
Use gSDE
True
gSDE sample frequency
Value function
0.45
cocfficient
Maximum value for gradient clipping
Gamma
0.99
GAE coefficient
0.9
Normalize advantage
True
Learning rate
linear schedule (5e-4, 3e-5)
high-level: pi= 64), vf= (64, 64
Hidden Layer
low-level: pi-(64, 64), vf-/64, 64
Table 9: Hyperparameters of A2C in Pick & Place
Number envs
Numer of steps
per update
Buffer size
20
Number steps
5e5
Policy
'MIpPolicy'
Entropy coefficient
0.0
Optimizer
RMSpropTFLike
RMSProp epsilon
le-5
Value function
0.5
coefficient
Maximum value for gradient clipping
0.5
Gamma
0.99
GAE coefficient
0.9
Normalize advantage
Truc
Learning rate
7e-4
Hidden Layer
pi=
64, 64,
vf- 64, 64
23

---

Published in Transactions on Machine Learning Research (11/2023)
Table 10: Hyperparameters of TL-AC in Pick & Place
Number of envs
4
Numer of steps
16
per update
Buffer size
96
Number of steps
5e5
Epochs
20
Target KT.
linear schedule(0.03, 0.02)
Clipping parameter
0.2
Policy
'MpPolicy'
Log std init
low:-2, 2, high: 0.0
Ortho init
low: False, high: True
Entropy coefficient
0.0001
Optimizer
RMSProp epsilon
le-5
Use gSDE
True
gSDE sample
Trequency
Value function
0.45
coefficient
Maximum value for gradient clipping
0.5
Gamma
0.99
GAF coefficient
0.9
Normalize advantage
Truc
Learning rate
0.00015
high-level: pi= (64, 64), vf 64, 64 low-level: pi- 64), vf= (64, 64
Hidden Layer
24


Reviews:
[{"review_id": "zFoNRrrJxK", "reviewer": "Reviewer_HEyH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- simple, sound, and novel algorithmic solution\n- the empirical study covers well the various set-ups I could imagine\n\nWeaknesses:\n- the last paragraph of the first page that intends to motivate the setting is not convincing at all. It's unrealistic and not a setting where we would have naturally human experts and even less where there is a reason for needing several complementary experts. I'm sure there could be a better motivating example. (if not, I would suggest to just remove it)\n- the formalization is approximative (details follow).\n- the positioning is fuzzy. I do not know well the competitors and I did not have a clue of how they work by reading the submission. For instance, it seems that the novelty as compared to Li2019b is only the extension to continuous action spaces, but I am not sure. Is it so?\n- some empirical results are poorly analyzed (details follow).\n\nApproximative formalization:\n- equation (4): what is $r$? The reward function is called $R$. I don't understand what this equation is supposed to show/mean. Or maybe do you mean that there exists an MDP where the action space is $E$ instead of $A$ and its reward function is defined as follows?\n- equation (5): you need to show the dependencies here. It does not seem right. If the values of the high and low level policies are equal, what is the use of the teachers?\n- algorithm 1 line 4: why is $m$ passed to collect function? Isn't $\\pi_{low}$ part of $E$, why is it passed to collect?\n- algorithm 1 line 23: what does it mean that $B$ is full? It means that at the next iteration of line 3 loop, $B$ will still be full?\n\nEmpirical analysis:\n- Figure 3: this is weird how none of the algorithm seems to learn anything (most of the curves are constant over time). You need to show how the algorithms reached their respective performances.\n- Figure 3: What are the teachers' performance? It should be visible on the plot.\n- Figure 4: It shows that TL-AC high policy tends to learn to never use $\\pi_{low}$ because it's low-performing at start. You would need a reset (like they do in [Laroche2018].\n- Figure 4: shouldn't AC-teach work okay as long as there is a good teacher to select? Again, it would help the understanding to see the performance of the teachers.\n\nMinor comments and typos:\n- $log$ => $\\log$\n- value network squared loss should start with $L(\\theta')$\n- algorithm 1 is displayed 2 pages too late.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors tackle an online RL setting where several advising policies called teachers are provided. These teachers do not need to be expert, even though their added value depend on it. In order to take advantage of the teachers, the algorithm proposes a bi-level decision making where it first makes the decision of which agent (teacher or low-level policy) to select as control, and second, if the low-level policy has been selected, which action to take. While pretty straightforward, this approach is sound and novel as far as I know. The authors demonstrate the advantage of their approach as compared to a few baselines taking advantage or not of the teachers, in a few settings (mix of good/bad teachers), and in a few environments.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I expect the authors to improve the submission in order to fix the points mentioned above. \nI do not think that there is anything unfixable, but these would require a significant revision of the submission.\n\nAlso, I would be grateful if they could add Reinforcement Learning Algorithm Selection [Laroche2019] in their competitors. The principle is pretty simple too: at the start of every trajectory, a high-level policy (which instantiated as a multi-armed bandit) selects a policy to be in control of the full trajectory (this is therefore a trajectory agent selection and not a timestep agent selection as in the submission). The algorithm portfolio would therefore be the set of teachers plus a learning algorithm (A2C) that learns to solve the task. I think that the addition of [Laroche2018] in the competitors would highlight the need for a state-based expert selection.\n\n[Laroche2018] Laroche, Romain, and Raphael Feraud. \"Reinforcement Learning Algorithm Selection.\" International Conference on Learning Representations 2018.", "broader_impact_concerns": "no concern", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ttkXatn3K0", "reviewer": "Reviewer_n2pd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\n\nThis paper possesses many strengths. The contributions of this paper are quite clear: a new algorithm for a specific setting of RL in which a set of policies are available. This is a general enough setting that I believe the algorithm and experiments will be of interest to the community. Determining usefulness of this algorithm is framed by research questions R1 and R2, which are examined in great detail in the experiments. Experiments are well designed to isolate interesting trends, such as how performance degrades across learning algorithms as the number of noisy expert policies increases. Domains are well chosen, as are baselines. Assumptions are stated plainly.\n\n[Weaknesses]\n\nThe biggest weakness of the work is in its restricted scope. The learning setting studied is stylized, and as such, the proposed algorithm is most useful under specific assumptions (the presence of many reference policies that can be queried freely). I don't find this weakness to be substantial enough to prevent publication. The only other minor weakness is that there is a body of literature that focuses on the RL problem in the presence of reference policies, such as \"Successor features for transfer in reinforcement learning\" by Barreto et al., or \"Policyblocks\" by Pickett and Barto. Both works (and the neighborhood of work surrounding them) consider how to best learn given a set of policies. These policies might be sub-optimal. The most crucial idea I felt was missing from the discussion of the present paper was \"Generalized Policy Improvement\" (or GPI) from the Barreto et al. work---this proposes a way to improve behavior over a set of policies. In follow up work (Barreto et al. 2018), they extend these ideas to Deep RL. I wonder how the authors find this kind of approach fairs in comparison to TL-AC?\n\n\nReferences:\n\nBarreto, André, et al. \"Successor features for transfer in reinforcement learning.\" Advances in neural information processing systems 30 (2017).\n\nBarreto, Andre, et al. \"Transfer in deep reinforcement learning using successor features and generalised policy improvement.\" International Conference on Machine Learning. PMLR, 2018.\n\nPickett, Marc, and Andrew G. Barto. \"Policyblocks: An algorithm for creating useful macro-actions in reinforcement learning.\" ICML. Vol. 19. 2002.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes an approach to reinforcement learning (RL) in the presence of multiple teachers, where a teacher is understood to be a policy. Thus, the RL problem of interest consists of an agent interacting with a Markov decision process (MDP), where the agent also has access to a collection of $N$ policies, $\\{\\pi_{e_1}, \\ldots, \\pi_{e_N}\\}$, describing the teachers behaviors. The key premise of the work is that the teachers may have different sets of expertise: one teacher might be effective in one region of the state space, while ineffective elsewhere (in the given example, one teacher may be good at mopping, while another may be best at cleaning counter tops). Thus, the key question facing the RL agent is whether---and when---to trust each teacher, rather than default to their own behavior. The paper proposes Two-Level Actor-Critic (TL-AC, Algorithm 1), an algorithm that makes use of teacher action advice to learn efficiently. Experiments center around two research questions: (R1) how does TL-AC deal with multiple _sub-optimal_ teachers, and (R2) Can TL-AC make use of teachers with differing expertise? The empirical study is broad and deep, covering a variety domains including a discrete control task (DoorKey), as well as continuous control tasks (Hopper). Baselines are A2C, DQN with Two-level Q-learning, and AC-Teach. The first experiments explore the impact of the presence of more noisy teachers: for instance, Figure 3 and 4 show how performance changes as teachers are changed between three good teachers (subfigures 3a and 4a), and three noisy teachers (subfigures 3d and 4d). The second set of experiments address R2, and explore whether the TL-AC can learn effectively even as teachers have diverse expertise. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Below are some suggested writing changes and other low-level questions:\n- Typo in Algorithm 1: Line 20 \"transcation\" --> \"transaction\". Also, I would call this an \"experience\" rather than a transaction.\n- I recognize previous literature has referred to these policies as \"teachers\", but I find this slightly confusing. The policies are _any_ policy, as the paper states. For this reason I find some conceptual overlap with work on transfer, as in the work by Barreto et al. discussed above.\n- Why are the confidence intervals for A2C so thin, and only appear around specific points in the results? Was the sampling rate different?\n- I found that some of the notation in Algorithm 1 was confusing. At line 15, what is the argmax over? What is being maximized? In lines 7-10, the advantage changes from $s_t$ to $s$. In line 6, you label __M__, but don't use this notation again.\n- It looks as though both $R$ and $r$ are used as the reward function ($R$ appears in the background section, but $r(s,a)$ in Section 4, for example in Equation 1 or 2).\n- The background section is so short I don't believe it needs to be its own section. Consider folding both related work and background into the introduction as subsections?\n- Also, I would prefer the related work section of this paper come _after_ the main methods section (currently Section 4 Two-Level Actor-Critic Using Multiple Teachers). This will allow you to contrast details of your approach.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XTWs5VP8iL", "reviewer": "Reviewer_bExB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "********************Strengths:********************\n\n- ****************Clarity:**************** The paper is very clear and easy to read.\n- ****************Results:**************** Results are somewhat promising and demonstrate robustness to noisy teachers.\n\n**********************Weaknesses:**********************\n\n- ****************Writing:**************** The authors should give an intuition of their approach before then explaining it in detail in Section 4.2. Right now there’s no intuition for why they have set up the two-level hierarchy in the exact manner, perhaps an overview paragraph at the top of 4.2 before then talking about the algorithm would be the best place to put it.\n- **Contribution**: Unfortunately I don’t think this paper presents a significant enough contribution for this conference. They essentially use Li 2019b (Two level q-learning) but modify it to use A2C as the base algorithm so that it can work for continuous control. While they add analysis regarding noisy teachers and such, the paper is mainly framed as a method paper and I don’t think that swapping out the base algorithm of an existing 2-level setup is a significant methodological contribution.\n- **Baselines**: A missing, simple baseline is one that does flat RL with advice from teachers incorporated. That would be, for example, one in which the policy loss is modified by adding a distribution loss term (e.g. KLD loss) that constrains the learned policy to be somewhat close to all of the teacher’s output actions.\n- **************Analysis:************** Why is this TL-AC algorithm better than AC-Teach and DQN-TLQL in certain instances? The analysis could be improved in the experiments to make this more clear. The authors, for example, say “we conclude that our method is robust to the effect of noisy teachers’ advice,” but don’t say why it’s more robust.\n\n**************Minor Issues:**************\n\n- L20 in Alg1: “transcation” → “transaction”", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors present a simple technique to learning an agent under advice from multiple teachers. This two-level technique essentially choses either a teacher’s action to execute or a learned policy’s action to execute. Rewards then propagate through off-policy RL to ensure that the learned agents learns from the teachers’ actions. The authors demonstrate results on MiniGrid, locomtion, and simple pick and place tasks along with analysis of the effects of noisy teachers.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The main drawback of this paper is its lack of methodological contribution. While the authors present some new experiments involving noisy teachers, these experiments are not comprehensive enough to merit acceptance of this paper for its analysis and experiments.\n\nI would recommend framing this paper less as a “new method,” rather an analysis of existing methods (TLQL, AC-Teach, etc.). Then the authors can also experiment with swapping out the underlying algorithms for more modern SOTA algorithms like SAC/PPO/etc. and investigating the effects of these changes on performance on existing tasks (among other design choices), and then the resulting algorithm can be the accumulation of the insights discovered through this analysis. This format would be more valuable for the RL community. \n\n If the authors choose to follow this analysis paper route, I would also recommend experimenting with more teachers and/or with more difficult environments and control tasks.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies an online RL setting where an agent has to select an action under the advice of multiple teachers.
The proposed approach is simple, sound, and original.
The empirical analysis is extensive and well-executed.
However, the paper has some weaknesses (approximative formalization, missing references to related works, better highlighting the contribution, and improving the analysis of the experimental results) that the authors have partially addressed in their rebuttals.
The authors now have to implement these changes in the new revision of their paper.
Decision: Accept (poster)