Collaborative Inter-agent Knowledge Distillation for Reinforcement Learning
Authors: Zhang-Wei Hong
Prabhat Nagarajan
Guilherme Maeda
Venue:  (2020)
Keywords: Reinforcement learning
distillation

Abstract: Reinforcement Learning (RL) has demonstrated promising results across several sequential decision-making tasks. However, reinforcement learning struggles to learn efficiently, thus limiting its pervasive application to several challenging problems. A typical RL agent learns solely from its own trial-and-error experiences, requiring many experiences to learn a successful policy. To alleviate this problem, we propose collaborative inter-agent knowledge distillation (CIKD). CIKD is a learning framework that uses an ensemble of RL agents to execute different policies in the environment while sharing knowledge amongst agents in the ensemble. Our experiments demonstrate that CIKD improves upon state-of-the-art RL methods in sample efficiency and performance on several challenging MuJoCo benchmark tasks. Additionally, we present an in-depth investigation on how CIKD leads to performance improvements.


Under review as a conference paper at ICLR 2020
COLLABORATIVE INTER-AGENT KNOWLEDGE DISTIL- LATION FOR REINFORCEMENT LEARNING
ABSTRACT
Reinforcement Learning (RL) has demonstrated promising results across several sequential decision decision-making tasks. However, reinforcement learning struggles to learn efficiently. thus limiting its pervasive application to several challenging problems. typical RL agent learns solely from its own trial-and-error experiences, requiring many experiences learn successful policy. To alleviate this problem, we propose collaborative inter-agent knowledge distillation (CIKD), CIKD is a learning framework that uses ensemble of RL agents execute different policies in the environment while sharing knowledge amongst agents in the ensemble. Our experiments demonstrate that CIKD improves upon state-of of-te-----e-rrt RL methods in sample efficiency and performance on several challenging MuJoCo benchmark tasks. Additionally, we present an in-depth investigation on how CIKD leads performance improvements.
INTRODUCTION
Reinforcement learning (RL) 47] has demonstrated impressive performance on solving sequen- tial decision- making tasks interactive environments (c.g., video game-playing [32] or robotic control [8)). ). In these tasks, the RL agent's goal is to find the optimal policy, which maximizes the expected return in the task. The agent learns this optimal policy through many trial-and-error interactions with the environment. Through these trial-and-eroor interactions, the agent explores the consequences of different decisions, in the form of a reward or punishment. Unfortunately, millions of interactions required even to solve simple tasks. Such amounts of interaction are infeasible to acquire in real, physical tasks, as is the case for robots.
In order to learn the optimal policy, the agent performs policy improvement, which refers to the incre- mental process refining the agent's policy towards performing nighrrewarding actions. However, the agent has a poor policy, then this incremental updating can be a slow process, requiring large number of interactions before the agent acquires the experiences necessary for learning desirable behaviors. This is because typical RL agent gathers experiences using policy close to its greedy policy, and this greedy policy itself is poor, acquiring high-quality experiences can require a large number of exploratory actions. However, through collaboration amongst multiple agents with different behaviors, perhaps we can reduce the amount of experiences needed for an agent to acquire good policy.
For inspiration, we turn the study of collective animal behavior [46]. Rather than exclusively learn from trial-and-error alone, the optimal behaviors for several tasks (e.g., flocking and foraging) can emerge through collaboration amongst animals. For example, consider ants foraging The ants search divergent paths, resulting in extensive exploration food-gathering policies. Furthermore, the ants that find food share their path with their companions via pheromones. This information-sharin amongst colony members offers each ant beneficial guidance toward food, so that individual ants need not exhaustively search several paths in order to find food. To make analogy to the RL setting, the ants' foraging process can be viewed as an effective way of policy improvement. Motivated by these insights from collective animal behavior, we incorporate the notion of collective knowledge sharing into the RL setting in attempt accelerate and guide the search for the optimal policy.
colony [8] group, across wide range of
Our method emulates the collaborative behaviors of ants via an ensemble of RL agents: a group of agents collectively search for the optimal policy in the same task, while periodically sharing knowledge. Each RL agent resembles an individual in the ant colony. To elicit diverse experiences amongst members of the ensemble, each RL is randomly initialized with different neural network
parameters. Random initialization results in adequate behavioral diversity (22 34] needed for collective exploration. As these agents are diverse nature, at any given time during the course of training, one agent naturally has policy superior its peers. This agent can guide its peers towards

---

Under review as a conference paper at ICLR 2020
senhherperformig policies, just ants share superior paths through pheromones. In order to guide the other agents, our method employs a knowledge distillation framework [24] which is effective at transferring knowledge between neural networks, without assuming identical model architectures. Knowledge distillation encourages other agents to act in a manner similar to the leading agent, allowing agents escape underperforming policies. Moreover, knowledge distillation empowers all agents to continue searching for the optimal policy from better starting points, accelerating knowledge accumulation in the entire ensemble. However, despite the distillation, the agents' knowledge is still preserved, retaining diversity amongst the ensemble.
This paper's primary contribution is collaborative inter-agent knowledge distillation (CIKD), simple yet effective framework for RL that jointly trains an ensemble of RL agents while periodically performing knowledge sharing. We demonstrate empirically that method can improve the state-of- the-art soft-actor critic (SAC) [20) on a suite of challenging MuloCo tasks, exhibiting superior sample efficiency and performance. We further validate the effectiveness of distillation for knowledge sharing by comparing against other methods of sharing knowledge. In addition, we present an in-depth investigation explain the underlying causes of CIKD's performance improvement. Finally, our ablation study shows that small ensemble is sufficient for improving performance.
The remainder of this paper is organized follows. Section 2 discusses the related work. Section introduces the reinforcement learning formulation. Section describes our method. Section experimental findings. Section 6 summarizes contributions and outlines potential avenues for future work.
4
2 RELATED WORK
The idea of jointly training multiple policies through RL has emerged in prior works. The most relevant works [37 (38) train multiple policies for the same task through RI, as our method does. Osband et al. [37 (88 train several agents in an ensemble while storing these agents' experiences in shared buffer. Thus, agents share knowledge by sharing experiences amongst members of the ensemble, which are then used for RL updates. Our method is complementary to Osband et al. in that we can also a shared buffer of experiences, but additionally periodically performing knowledge distillation between members of the ensemble, in particular from the best agent other agents. In our experiments we present supporting evidence to justify the importance of knowledge sharing. Other related methods aggregate multiple policies to select actions (2) 6 1O]
37 38
17 31 40 41
50 [54] Abel al. [. Tosatto al. (52), Wang & Jin [53] sequentially train a series of policies, boosting the learning performance by using the errors of a prior policy. However, rather than perform decision aggregation or sequentially-boosee -boosted training, we focus on improving the performance of each individual agent via knowledge sharing amongst jointly trained agents. Again, our method can be considered to be complementary approach to decision aggregation and sequentially boosted training methods.
45|
We can view the sharing of knowledge from the best policy as exploiting successful behavior patterns. This general notion has been explored in several areas of RL. Rusu et al. [42] and Parisotto et al. 33 can train a single network to perform multiple tasks by transferring multiple pre-trained
RL agents' policies to a single network. Hester et al. [23. Nair et al. (35] accelerate the RL agents' training progress via human experts' guidance. Instead of using experts' policies, Levine & Koltun 300, Nagabandi et al. (33 and Zhang et al. (55) leverage model-based controllers' behaviors (e.g. model predictive controllers (15 or linear-quadratic regulators Dorato et al. 7., facilitating training for RL agents. Additionally, Hong et al. (25) and Oh et al. (36) train agents to imitate the past successful exlfeexperis or policies. Orthogonal to the aforementioned works, our method periodically exploits the current best policy amongst the ensemble, and shares it with the ensemble, enabling a collaborative search for the optimal policy.
Collaborative learning approaches have emerged in other areas of machine learning research. In computer vision, Zhang al. [56) present deep mutual learning. which trains multiple models that mutually imitate each other's outputs on classification tasks. Our distillation is not mutual, and flows in a single direction, from a superior teacher agent to other student agents in the ensemble. In subsequent work by Lan et al. [29], they train ensemble of models to imitate a stronger teacher model. Simultaneously, this teacher model learns to aggregate all of the ensemble models' predictions. They demonstrate that imitating superior teacher leads to better performance than deep mutual learning, which performs mutual imitation amongst members. Our method contrasts from the above methods by periodically electing the teacher for distillation. Since our distillation occurs periodically,
2

---

Under review as a conference paper at ICLR 2020
can collect performance statistics for each agent between periods, which can be used elect the teacher in the distillation phase. In contrast to the distillation of an aggregate policy, as Lan et al. [29]
do, method ensures that the teacher indeed possesses a superior policy.
Collaborative learning has also been explored in RL as well. Teh et al. 49 and Ghosh et al. [6 jointly learn independent policies for multiple tasks or contexts and then distill these policies to a central multi-task policy. Galashov et al. 12 learn a task-specific policy while bounding the divergence between this task-specific policy and some generic policy that can perform basic task- agnostic behaviors. Czarnecki et al. [5] gradually transfer the knowledge of a simple policy to a complex policy during the course of joint training. While method also collaboratively trains policies, we differ from the aforementioned works in several aspects. First, our method periodically elects a leading agent for sharing knowledge rather than either constraining the mutual policy divergence 14, 16
49 56 or imitating aggregated models [29]. The second difference is that our method does not rely training heterogeneous policies (e.g. simple policy and complex policy (5)) which makes our method more generally applicable. Finally, as opposed to Tch et al. [49 and Ghosh al. [16), we consider single-task setting rather than multi-tasking.
similarly employ multiple policies to find the optimal policy. EA repeatedly performs mutation, selection, and reproduction on maintained population. Mutation randomly perturbs the parameters of policies in the population; selection eliminates the underperforming policies by testing the policies' performance in the environment; reproduction produces the next generation of policies from the remaining policies. Unlike EA,, our method does not rely on mutation and reproduction on population of policies. While EA and our method are similar in an abstract sense, in that they both share knowledge amongst agents, they are quite different in practice. EA often eliminates members from its population and performs destructive changes to members of the population. Our method focuses on continuously improving the existing agents, and does not perform destructive changes its population. In fact, our method can be incorporated into EA, serving as a more effective way to optimize each individual policy within the same generation.
Evolutionary algorithms (EA) 13
26
3 BACKGROUND
In this section we describe the general framework of RL. RL formalizes a sequential decision-making task as Markov decision process (MDP) 47). An MDP consists of state space S, a set of actions A. a (potentially stochastic) transition function T S x A + S, a reward function R. S x A + R, and a discount factor y [o, 1]. An RL agent performs episodes of task where an agent starts in random initial state 8o, sampled from the initial state distribution Pag. and performs actions, experiencing new states and rewards. More generally, at timestep f, an agent performs an action ae in state 8t, receives reward Tz-1, and transitions to state 8t+1. according to the transition function T. The discount factor 7 is used to indicate the agent's preference for short-term rewards over long-term rewards.
An RL agent performs actions according to its policy, a conditional probability distribution Te : S x A + [0. 1], where ó denotes the parameters of the policy, which may be the parameters of a neu- ral network. RL methods iteratively update ó via rollouts of experience [(s,, (e, Tr. St+1)) seeking within the parameter space & for the optimal o² that maximizes the expected return rrre |so s| for each t within an episode.
Egepro
4 METHOD
In this section, we formally present the technical details of our method, Collaborative Inter-agent agent Knowledge Distillation (CIKD). We start by providing an overview of CIKD and then its components in detail.
4.1 OVERVIEW
Emulating collaborative foraging behaviors of ants, CIKD employs an ensemble of RL agents to perform wide range of policies on the same task and periodically share knowledge amongst agents. CIKD can be divided into three phases: ensemble initialization, joint training, and inter-agent knowledge distillation. First, in the ensemble initialization phase, we randomly initialize an ensemble of RL agents to achieve behavioral diversity. In the joint training stage, each agent acts in the shared buffer, and these shared experiences are used to environment, adding its experiences Intermittently, we perform inter-agent knowledge distillation, update the parameters of the agent. where we elect leading agent guide the other agents towards superior policies. To this end,
3

---

Under review as a conference paper at ICLR 2020
Ensemble initialization
Joint training
Inter-agent knowledge distillation
experience
Update performance statirica
Uniform(
Policy selection A Luthenr]
It Updatte policiica sill
k = argmax R3
+p)
J Je
The
tra
dh
Tb
d
Collect trajectorios
Knowledgo distilation
Fandomil initalize palicies
Environment
reabaae E..o F.ooo po 1o.gg.oo
Figure I: An overview of collaborative inter-agent knowledge distillation.
distill [24] the best-performing agent's policy the others. Algorithm D and Figure. summarizes our method.
Algorithm Collaborative Inter-agent Knowledge Distillation for Reinforcement Learning
1: Input:
environment E, an ensemble size K, parameter space 4, set of parameterized policies {T 1 fk=o- recent episodic performance statistics {Rx 1 k= n- an episode length T, a joint training length E, distillation interval I, an experience buffer D
2: Output: the optimal policy π
3:
4: i. Ensemble initialization
5: Randomly initialize policy parameters: dy Uniform( (9), Vk [0, K)
6: Initialize the experience buffer: D + Ø
7: Initialize each recent episodic performance statistic Rk, vk E [0, K)
8: Initialize timestep counter: face t- 0
9: while not converged do
10: ii Joint training
for episode [0, F.) do
12:
Policy selection: ke Uniform K))
13:
Perform rollout the environment: T + ROLLOUT(E, Tor.)
14:
Store the experience: 'D 4- D U (r = {(r,, or, Tr, 8241 8t+))))) 1
Update all policies Tox Vk ()) by off-policy RL methods using batches sampled from
D
Update the selected recent- pcent-performance statisties: UPDATESTAT(RR..
17:
Accumulate timestep counter: face <- face + T end for
19:
iii. Inter-agent Knowledge Distillation
20: if Loee mod / = then
21: Elect the teacher agent: h argmax, Rs
22: Minimize L(Ox, Φb, Ak E [o, K) ke using D (Equation. I
23:
end if
24:
end while
4.2
ENSEMBLE INITIALIZATION
We randomly initialize K RL agents in the ensemble. Each RL agent's policy is instantiated with model parameterized by Φx. where stands the agent's index the ensemble, di. is initialized by sampling from the uniform distribution
parameter space & which contains all possible values of óx: OR ~ Uniform @). Despite the simplicity uniform distributions, Osband et al. [37] show that uniformly random initialization can provide adequate behavioral diversity. In this paper, represent each Φx. A4 E (0, K") as a neural network (NN) due to the impressive performance NNs recent RL research 20 32). though other parametric models (e.g. linear models) can be used.
CIKD can be easily applied to off-policy actor-critic methods, which learn both a policy and a critic function that values states or state-action pairs. Off-policy actor-critic methods store replay buffer of past experiences that are used for training. In this paper, we use soft actor-critic (SAC) [20), an off off-policy actor-critic method that has demonstrated ffte-offthe-aa results challenging tasks [19). To apply CIKD to SAC, we create a shared replay buffer for all agents and randomly initialize critic function for each policy Téx-
4

---

Under review as a conference paper at ICLR 2020
4.3 JOINT TRAINING
Each joint training phase consists of E episodes. For each episode, we select an agent in the ensemble to act in the environment (hereinafter, we refer this process as "policy selection") and then update all agents" policies using the experiences from that episode. Below, we describe the policy selection and policy update procedures.
The policy selection strategy is a way to select one policy Tés, from the ensemble, to perform one episode in the environment. This episode is then stored in shared experience buffer D, and the agent's 's recent episodic performance statistic Rk, is updated according to the return achieved in T, where Rs, stores the average episodic return in the most recent M episodes. { {R₂ k= and 'D will later be used in inter-agent distillation (Section (.4). There are two purposes of policy selection. The first purpose is collect diverse experiences from different policies. In this paper, we adopt simple uniform policy selection strategy, whereby at the beginning of each episode we select a policy from the ensemble at random to act in the episode. After selecting a policy Tót, which performs an episode T, we store this in D. Then, we can sample data from D and update all policies using any arbitrary off- aff-policy update method. Since off- off-policy update methods do not require that r necessarily comes from a specific policy, they enable our policies to learn from the trajectories generated by other members of the ensemble.
4.4 AGENN KNOWLEDGE DISTIILLATION
The inter-agent knowledge distillation phase consists of two stages: teacher election and knowledge distillation. The purpose of teacher election is to determine which agent in the ensemble has the best-performing policy, which can then serve as a teacher to the other ensemble members. In our experiments, teacher election is determined from each agent's 's recent episodic performance statistics recorded during the joint training stage. Thus, the selected teacher agent is the agent with the highest recent performance, namely k; arg max; Rx. where k2 is the index of the teacher. Rather than use the agent's 's most recent episodic performance, we use its average return over its previous M episodes, minimize the noise in our estimate of the agent's performance. We additionally considered performing offline evaluations, but this requires additional environmental interactions.
Next, the teacher serves as a guide that leads the other agents towards higher-performin policies. This is done through knowledge distillation (24), which has been shown to be effective at guiding the neural network to behave similarly to another neural network. To distill from the teacher to the students (i.e., the other ensemble members), the teacher samples experiences from the buffer 'D and instructs each student to match its outputs on these samples. The intuition is that after distillation, the students acquire the teacher's knowledge, enabling them correct their sub-optimal behaviors and reinforce their correct behaviors. Specifically, the distillation process can be formalized as minimizing the following loss function:
L(ok ΦK.) = EnDD [D(Tes, (11)), To, (-1s))],
(1)
where D can be any distance metric between two functions. In this paper, D is taken as Kullback- Leibler divergence (DxL) to measure the similarity of policies. We use s to denote historical states sampled from the experience buffer D. DKL is a principled way to measure the similarity between two probability distributions (i.e., policies). As DxL. is an asymmetric metric, we clarify the settings below. Te,, and Ta, are taken as the ideal distribution and approximated distribution, respectively (i.e., P and Q in the standard notation of KullbackLeeible divergence (28).. Note that inter-agent distillation also compatible with actor-critic methods. For actor-critic methods, we additionally distill the critic function from the teacher to the students. In this paper, our distance metric for the critics is the loss function. In practice, the distance metric between critics can be any other metric for regression (i.e., critic training can be viewed as a regression problem).
EXPERIMENTS
The experiments are designed to answer the following questions: (I) Can CIKD improve upon the data efficiency of state-of oh-the-art RL? (2) Is knowledge distillation effective at sharing knowledge? (3) Is is it necessary to choose the best-performing agent be the teacher? (4) Why does CIKD improve the performance of RL methods? (5) What is the impact of the ensemble size? Next, we show our experimental findings for each of the aforementioned questions, and discuss their implications.
5

---

Under review as a conference paper at ICLR 2020
5.1 EXPERIMENTAL SETUP
Implementation. Our goal is to demonstrate how CIKD improves the sample efficiency of an RL. algorithm. Since soft actor-critic (SAC) [20] exhibits state-of-the art sample efficiency across several simulated benchmarks [51] and even in real robots [8] 191 we build on top of SAC. We directly use the hyperparameters for SAC from the original paper 20 in all of our experiments. Unless stated otherwise, the hyperparameters used in our proposed method (Algorithm [ are I = 5000, M 5, and K 3 for all experiments. The value of I and M are tuned via grid search over [1000, 2000, ... 10000), and [1, 2, ..., 10] respectively. We experiment with K E {2, 3, 5} in Section 5.6 For the remainder of our experiments, we term CIKD applied to SAC as SAC-
Benchmarks. We use OpenAl gym 'ss MuJoCo 51 benchmark tasks, as used in the original SAC (0 paper. We choose all tasks selected in the original paper (20] and two additional tasks to evaluate the performance of our method. The description for each task can be found in the source code for OpenAI gym[
Evaluation. We adapt the evaluation approach from the original SAC paper (20). We train each agent for I million timesteps, and run 20 evaluation episodes after every 10000 timesteps (i.e., number of interactions with the environment), where the performance is the mean of these 20 evaluation episodes. We repeat this entire process across 5 different runs, each with different random seeds. We plot the mean value and confidence interval of mean episodic return at each stage of training. The mean value and confidence interval are depicted by the solid line and shaded area, respectively. The confidence interval is estimated by the bootstrapped method [9]. At each evaluation point, we report the highest mean episodic return amongst the agents in the ensemble. In some curves, we additionally report the lowest mean episodic return amongst the agents in the ensemble.
5.2 EFFECTIVENESS OF INTER-AGENT KNOWLEDGE DISTILLATION
In order to evaluate the effectiveness of inter-agent knowledge distillation, we compare our method, SAC-CIKD, with two basclines: Vanilla-SAC and Snsemble-SAC. Vanilla-AAC stands for the original SAC; Ensemble-SAC is the analogous variant of Osband et al. [37]] 's method on SAC. At its core, Oshand's method involves an ensemble of agents that act in the environment and generate trajectories. These trajectories are then used to train all of the agents using an off- off-policy RL algorithm. Our Ensemble-SAC baseline resembles this by having each agent represented as an actor-critic neural network trained with the off-policy algorithm SAC (effectively SAC-CIKD without inter-agent knowledge distillation). Ensemble sizes (K) for Ensemble- SAC and our method are set to 3. Our results are shown in Figure 2 Note that we also plot the worst evaluation in our ensemble at each evaluation phase to provide some insight into the general performance of the ensemble. In all tasks, Moreover, SAC-CIKI has
outperform all baselines, including Vanilla-SAC and Ensemble-SS far better sample efficiency, usually reaching the best baseline's 's convergent performance in half of the environment interactions. We even find that in the majority of tasks, our worst evaluation in the ensemble outperforms the baseline methods. This demonstrates that all members of the ensemble are significantly improving, and our method's superior performance is not simply a consequence of selecting the best agent in the ensemble. Notice that Ensemble-SAn does not significantly improve the performance of Vanilla-SAC. Ensemble-SAC only outperforms Vanilla-SAt in 4 out of 7 tasks and suggests that the diversity of the ensemble is alone insufficient for achieving large performance gains over Vanilla-AAC. In particular, SAC-CIKD superiority over Ensemble-SAL highlights the effectiveness of supplementing shared experiences (Ensemble-SAe SAC with knowledge distillation. In summary, Figure [ demonstrates the effectiveness of inter-agent knowledge distillation on enhancing the performance and data efficiency of state-of -ftth--rrt RL algorithms.
5.3 EFFECTIVENESS OF KNOWLEDGE DISTILLATION FOR KNOWLEDGE SHARINGG
In this section, we investigate the advantage of using knowledge distillation for knowledge sharing. Beyond using knowledge distillation 24], we consider other approaches that may successfully share knowledge amongst agents. First, we consider sharing knowledge by simply providing agents with additional policy updates using the shared experiences. We also consider directly copying the neural network. Below, separately compare these two approaches with knowledge distillation.
Though Section 5.2 has shown that Ensemble-SAC which updates all agents' policies through shared experiences fails to perform as well as SAC-CIKD, SAC-CIKD uses additional gradient updates during
hhap:///gituuhccmoogaaaiggg

---

Under review as a conference paper at ICLR 2020
Reacher-2 x2
HuranodSttrdd.pe 2-12
Humanol542
0000
190000
Naan retm
#300
Moan rdtum
Mean relum
-12
3000
2200
2000
100000
-15
9000
DO00 0.205E 2.00M D.DGW: 3.E0M 1.002
200M 0.20V)
3.C0M DAOM 0.MCM 1.0M
DA0N 0.23M O JOM D.00M ROM 1000
CON 0,23M C.EDM CUOR 0.EOM 1.0M Timedtpps
Timesteps
Timesteps
Timedtpps
Hkopperw22
Arte
12:00
3000
90000
Moan ro0km
4003
SAC-CIKD (best)
Mean 2000
Mourt
SAC-CIKD (worst)
0000
2000
Ensemble-SAC (worsi)
EneembleeSAAe (beat)
2503
VanillaBAC
DO0N 0,201 2,AUN DJOM 0.50M 1.000
300M
0,2011
2.40M DJMM 0.50M 1.00
000M 0.2M 0.COM DUOR 0.5 1/000
Temassapa
Timaniapa
Timastapa
Figure 2: Performance evaluation of inter-agent knowledge distillation. SAC-CIKD represents the imple- mentation of our method upon SAC; Vanilla-SAC stands for the original SAC; Ensemble-SAC is an analogous variant of Osband et al. (37 's method on vanilla-SAC (effectively SAC-CIKI without inter-agent knowledge distillation). See Section 5.2 for details. Notice that in most domains, SAC-CIKD is able to reach the convergent performance of the baselines in less than half the training time.
the inter-agent knowledge distillation phase, whereas Ensemble-SAC only performs joint training. It is unclear whether extra policy updates in lieu of knowledge distillation can achieve the same effects as knowledge distillation. To investigate this, we compare our method with Vanilla-SAe (extra) and Ensemble-SAC (extra), which respectively correspond to Vanilla-SAC and Ensemble-SAC (see Section (.2) agents that are trained with extra policy update steps. Specifically, instead of the knowledge distillation phase, we provide these baseline agents with the same number of policy updates and minibatch sizes as we give the SAC-CIKD agent for knowledge distillation. A policy update here refers to as training step for updating the policy 44 48) and a value function (20) 27). required. by RL algorithms. Figure (3a) shows the performance of the above comparison baselines and SAC-CIKD. It can be seen that SAC-CIKD outperforms all the baselines. This observation shows that knowledge distillation is more effective than policy updates for knowledge sharing.
We additionally study whether the naive method of directly copying parameters from the best- performing agent can also be an effective way to share knowledge between neural networks. We compare a variant of our method, which we denote as SAC-CIKD (hardcopy), against SAC-CIKD. In SAC-CIKD (hardcopy). rather than perform inter-agent knowledge distillation, we simply copy the parameters of the teacher policy and critic into the student policies and critics. Figure (3b) depicts the performance of this variant, the baselines, and SAC-CIKD. We can see that despite the fact that SAC- (hardcopy) can surpass Vanilla-SAC, SAC-CIKD (hardcopy) loses to SAC-CIKD in all tasks. Thus, knowledge distillation is superior to naively copying the best agent's 's parameters. While sharing can be beneficial, being too biased towards the teacher can be counterproductive. Identical behaviors may limit the scope of exploration policies, undermining their quality.
5.4 EFFECTIVENESS OF SELECTING THE BEST PEE--PRRRRRONN AGENT AS THE TEACHER
In the inter-agent knowledge distillation stage, we elect the best-performing agent to be the teacher, and distill this teacher's knowledge to the students (see Section
However, the importance of choosing the best-performing agent remains unclear. Perhaps simply sharing knowledge amongst members is sufficient for good performance, and strictly distilling from gigh-performnn teacher may be unneeded. To investigate the importance of selecting the best- performing agent to be the teacher, we experiment with selecting a random teacher. We denote this variant as SAC-CIKD (random teacher). Figure plots the performance of our method, the baselines. and this variant. It can be seen that SAC-CIKD (random teacher) performs on- an-par with Ensemble- 3 out of 4 tasks, and better on a single task. SAC-CIKD (random teacher) performs worse than SAC-CIKD on 3 tasks, and on-par on a single task. This result suggests that simply sharing knowledge amongst agents can slightly improve performance. Perhaps the reason that random knowledge-sharing can improve performance is because minimizing the KL divergence between two policies (even randomly chosen policy) draws them closer to one another. Doing so then reduces the extrapolation error [1] caused by training the policies with data distributions generated from other policies, thus improving the quality of off-policy updates within the ensemble. Nevertheless, it is still preferable to select the berf-perfo agent when sharing knowledge through distillation.
7

---

Under review as a conference paper at ICLR 2020
Walker2d-v2
Hopger-/2
HailCheetah-22
Ant-12
0000
Meon retum
Moon rotum
Mbser retum
10000
retum
4000
2000
2000
5000
20000
1000
0
0.00M 0.55M 0.50M 0.75M
3.00M 025M
3.00M 0.25M 0.50M 0.75M
0.00M 0.25M 0.50M 0.75M
Timestaps
Timestapa
Timestaps
Timestepa
RAC--C107 (pwed)
Frewentii SSC (wetta) (bwed)
Fineeribee SAA (eatts) (noret)
Viriilee SAC (eairr))
SAC-OKD (wors))
(a)
Walror2d-12
Hosper-12
CChestah-22
A41--2
0000
0000
Moan remm
Maai (OORT)
4003
2300
5000
2000
DOON 0,201 2.40M 0.OM 0.50M 1.000
2.00 0,20M 3.40M DO0M 0.50M 1.00M
0.000 0.21M 0AM 0.0M 3.5 1.00
CON 0,205 C.40M 6.0M 0.E0M 1.00M
Timmodopp
Ennembie--Ge
SAC-CIKD
SAG-CIO (hardoapy:
(b)
Figure 3: (a) Comparison between knowledge distillation and extra policy updates. Vanilla-SA SAC (extra) and Ensemble-SAr (extra) stand for Vanilla-SAC and Ensemble-SAC variants that use extra policy updates, respectively (see Section 5.3 and Section 3 details). (b) Comparison between knowledge distillation and copying parameters. SAC-CIKD (hardcopy) stands the variant of our method which directly copy the neural networks parameters of the best agent to the others. (a)()) Both figures show that knowledge distillation is more effective.
Viil Baerrd---2 H2
Ploper-422
TMMCchestah--22
Am1--12
8000
12300
Moon etam
10000
Meen wkam
2000
7500
Maann
8000
2000
0.00 0.E0M 5:00 D.OM 680M 1.02M
500M 2.20M 040M D.EM O BON
1.0M
0,001 0.30M 0.00 2.50M 0.00M 1.00M Timeehops
DOOM 0.20M 0-40M 0.80M 5,0M 1.0M Timestaps
Timestags
Timeeasee
SALACIO
SALDINN Iandem beachwr)
Figure 4: Comparison between the selecting the best- pest-pprforming teacher random teacher. SAC-CIKD (random teacher) refers the variant of our SAC-CIKD where a randomly chosen teacher used for knowledge distillation. This figure demonstrates that can be more effective to select the best-performing agent as the teacher.
5.5 CAUSES FOR THE PERFORMANCE IMPROVEMENT
In this section, we investigate how inter-agent knowledge distillation improves the performance of each agent in the ensemble. Below, we verify each of our hypotheses in the HalsCheetah-r2 environment. One hypothesis is that knowledge distillation transfers the behaviors of a dominant agent weaker agents. Such a skilled agent may emerge simply from the diversity stemming from random initialization. To investigate this hypothesis, we plot the selected teacher index (i.e. k2) over the entire training process in Figure 5al We find that the selected teacher roughly evenly distributed across all agents. These proportions are presented in Figure. 5bb] It can be observed that there is no strictly dominant agent within the ensemble. Both figures suggest that there is no single dominant agent passing its knowledge to inferior agents. Since we find that there is no single dominant agent. Another hypothesis is that knowledge is accumulated throughout members of the ensemble, with agents surpassing their teachers to then become the next teacher, passing its knowledge in the next period. As such, we are interested in knowing how often agents surpass their teachers immediately subsequent distillation. To measure this, we evaluate (see Section 5.1 for detail) each agent immediately before and after the inter-agent knowledge distillation phase. Figure [5c depicts the percentage of agents that obtain superior performance to their teachers during different stages of training. We see that throughout the whole training process, in at 23% of post-distillation evaluations, the student agent surpasses teacher. Furthermore, in the later stages of training, 33% of distillations result in the student agents outperforming their teachers. From these observations, we conclude that stronger agents likely emerge through continual knowledge distillation.
Since knowledge distillation 24] theoretically aims to match the performance between the students and teachers, a performance increase in the students is counter-intuitive. One possible explanation
8

---

Under review as a conference paper at ICLR 2020
is that the combination of students' inherent skills and teachers' knowledge results in performance improvement. To test this, we randomly reset the parameters of each student agent prior to distillation, to observe the influence of the students' inherent knowledge. Figure 5d plots the performance of this variant and our method. The variant with parameter reset is denoted as SAC-CIKD (reset). It can be observed that SAC-CIKD (reset) is much worse than both SAC-CIKI and Ensemble-SAC. This demonstrates the importance of performing knowledge distillation on an agent that has already learned.
de Come when k-th agennt selected
45 0%
Suufpccodlal www.eeeiiraaaaaaiooooo. the
'
26. 0%
k-1
29. 0%
15
MEDES H- COIEOO SERES
(b)
(c)
WallerGas-222
3000
#300
2302
2300
5000
DOON 5.0M DJOW 0.50M 1.00
2.0M
3:40M DORM 0.50M 1.00
0.00 0.21 FCM DJRM 1.0
C4CM DUOM 0.E0M 100M Timeddpps
Ensemble SAC SWC CIO
SAG ANC.CRO (RROA)
Figure 5: (a) The distribution of selected teacher agent indices. This figure shows that each agent has a chance to become the teacher.
(b) The proportion of time that each agent is selected to be the teacher over the entire training process. This figure shows that there is no strictly dominant agent within the ensemble. (c) The proportion of time that the students surpass the teachers after knowledge distillation. This figure shows that as more experience is collected, inter-agent knowledge distillation enables students to become better than their teachers (see Section 4.4 for the detail of knter-knowledge distillation). inherent knowledge. SAC-CIKD (reset) denotes the variant of our method where we randomly re-initialize students' parameters before knowledge distillation. This figure shows that students' inherent knowledge crucial for the performance of our method.
(d) The importance of students'
5.6 ABLATION STUDY ON ENSEMBLE SIZE
In this section, we study the influence of the ensemble size K as the ensemble size affects the scalability. We test our method with three ensemble sizes K 2, K 3, and K Figure 6 shows the performance of these configurations. We find that same across all three ensemble sizes. Even with an ensemble size of 2, we see better performance than Ensemble-SAC (as K 2 is on-par with K = 3, which outperforms Ensemble-SAC, as shown before). Thus, our method can reap benefits even from small ensembles, and is not extremely sensitive to the ensemble size. However, investigating CIKD large scale ensemble is not the primary focus of this paper, and we leave its investigation for future work.
performs approximately the
Hosper-zz
12500
decin mem
4900
2000
2000
J00J
2900
D00N AUM BROM 0.50M 1.00 Timestops
2000 DIRM 0.BCM 1.001
DMMN U22M AUM BJ93 1ARR
C.RMM EIERS 1X00 Timedeps
Timestaps SAC CMD (6-5)
SAC CHD (6-2)
SAC CID (K-3)
Figure 6: Performance comparison under different ensemble sizes. Three different ensemble configurations with 2, 3, and 5 agents lead to similar performance. This result shows that CIKD does not require a large ensemble size.

---

Under review as a conference paper at ICLR 2020
6 CONCLUSION
In this paper, introduce collaborative inter-agent knowledge distillation (CIKD), method that jointly trains an ensemble of RL. agents while continually sharing information via knowledge distilla- tion. Our experimental results demonstrate that CIKD improves the performance and data efficiency of state-of-the-art RL method on several challenging MuJoCo tasks. Also, we show that know tnowledge distillation is more effective than the other approaches for knowledge sharing. We found that electing the best-pett pesffrerroming agent to serve as the teacher plays significant role in improving performance. Our investigation further showed that the combination of students' and teachers' knowledge is crucial for the performance. Finally, our ablation study showed that a large ensemble is not needed for improving performance.
CIKD open several avenues for future work. First encouraging diversity within the ensemble may lead Additionally, while we used a simple uniform policy selection strategy, a more efficient policy selection strategy may further accelerate learning. Lastly, while our ensemble members used identical architectures, CIKD may benefit from using heterogeneous ensembles. For example, different networks may have different architectures that are conducive to learning different skills, which can then be distilled within the ensemble.
to more efficient exploration 4 25
REFERENCES
[1] David Abel, Alekh Agarwal, Fernando Diaz, Akshay Krishnamurthy, and Robert E Schapire. Exploratory gradient boosting for reinforcement learning in complex domains. arXiv preprint arXiv:1603.041199 2016.
[2] Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option- crrtic architecture. In Thirty-Firs First AAAI Conference on Artificial Intelligence, 2017.
[3] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
[4] Edoardo Conti, Vashisht Madhavan, Felipe Petroski Such, Joel Lehman, Kenneth Stanley, and Jeff Clune. Improving exploration in evolution strategies for deep reinforcement learning via a population of sooeelty-seeking agents. In Advances in Neural Information Processing Systems, pp. 5027-5088 2018.
[5] Wojciech Marian Czamecki, Siddhant M Jayakumar, Max Jaderberg, Leonard Hasenclever, Yee Whye Teh, Simon Osindero, Nicolas Heess, and Razvan Pascanu. Mix&match-agennt curricula for reinforcement learning. arXiv preprint arXiv:1806.01780, 2018
[6] Peter Dayan and Geoffrey E Hinton. Feudal reinforcement learning. In Advances in neural information processing systems, pp. 271-278, 1993.
[7] Peter Dorato, Vito Cerone, and Chaouki Abdallah. Linear quadratic control: introduction. Krieger Publishing Co., Inc., 2000.
[8] Marco Dorigo and Mauro Birattari. Ant colony optimization. Springer, 2010.
[9] Bradley Efron. Better bootstrap confidence intervals. Journal of the American statistical Association, 82(397):11--185, 1987.
[10] Kevin Frans, Jonathan Ho, Xi Chen, Pieter Abbeel. and John Schulman. Meta learning shared hierarchies. arXiv preprint arXiv: 1710009777 2017.
[11] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. arXiv preprint arXiv: 1812,02900 2018.
[12] Alexandre Galashov, Siddhant M Jayakumar, Leonard Hasenclever, Dhruva Tirumala, Jonathan Schwarz, Guillaume Desjardins, Wojeiech M Czamecki, Yee Whye Teh, Razvan Pascanu, and Nicolas Heess. Information asymmetry in kl-regularized rl. arXiv preprint 1rX.v:1905.012444 2019.
[13] Tammay Gangwani and Jian Peng. Policy optimization by genetic distillation. arXiv preprint 171.001711.01012 2017.
10

---

Under review as a conference paper at ICLR 2020
[14] Tanmay Gangwani, Qiang Liu, and Jian Peng. Learning self-imitating diverse policies. arXiv preprint arXiv 10309, 2018.
[15] Carlos E Garcia, David M Prett, and Manfred Morari. Model predictive control: theory and practicea survey. Automatica, 25(3):335-344 1989.
[16] Dibya Ghosh, Avi Singh, Aravind Rajeswaran, Vikash Kumar, and Sergey Levine. Divide-and- conquer reinforcement learning. arXiv preprini arXiv:1711.09874 2017.
[17] Michael Gimelfarb, Scott Sanner, and Chi-Guhn Lee. Reinforcement learning with multiple ex- perts: bayesian model combination approach. In Advances in Neural Information Processing Systems, pp. 9528-9538 2018.
[18] Tuomas Haarnoja, Vitchyr Pong, Aurick Zhou, Murtaza Dalal, Pieter Abbeel, and Sergey Levine. Composable deep reinforcement learning for robotic manipulation. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 6244-6251. IEEE, 2018.
[19] Tuomas Haarnoja, Aurick Zhou, Schoon Ha, Jie Tan, George Tucker, and Sergey Levine. Learning to walk via deep reinforcement learning. arXiv preprint arXiv: 18/2. / / 2018.
[20] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint / 2018.
[21] Matthew Hausknecht, Joel Lehman, Risto Mikkkulainen, and Peter Stone. A neuroevolution approach to general atari game playing. IEEE Transactions on Computational Intelligence and Al in Games, 6(4):355-366, 2014.
[22] Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforeement learning that matters. In Thirty-Second AAAI Conference Artificial Intelligence, 2018.
[23] Todd Hester, Matej Vecerik, Olivier Pietquin, Mare Lanetot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, et al. Deep q-learning from demonstrations. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
[24] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprini arXiv: 02053), 2015.
[25] Zhang-Wei Hong, Tzu-Yun Yun Shann, Shih-Yang Su, Yi-Hsiang Chang, Tsu-Jui Fu, and Chun-Yi Yi Lee. Diversity-driven exploration strategy for deep reinforcement learning. In Advances in Neural Information Processing Systems, pp. 10489-10500, 2018.
[26] Shauharda Khadka and Kagan Tumer. Evolution- guided policy gradient reinforcement learning. In Advances in Neural Information Processing Systems, pp. 1188-1200, 2018.
[27] Vijay R Konda and John N Tsitsiklis. Actor-critic algorithms. Advances in neural information processing systems, pp. 1008-1014, 2000.
[28] Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathematical statistics, 22(1))79-86, 1951.
[29] Xu Lan, Xiatian Zhu, and Shaogang Gong. Knowledge distillation by on-the-fly native ensemble. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 7528-7538, Curran Associates Inc., 2018.
[30] Sergey Levine and Vladlen Koltun. Guided policy search. In International Conference Machine Learning, pp. 1-9, 2013.
[31] Josh Merel, Arum Ahuja, Vu Pham, Saran Tunyasuvunakool, Siqi Liu, Dhruva Tirumala, Nicolas Heess, and Greg Wayne. Hierarchical visuomotor control humanoids. arXiv preprint arXiv: 10l::///1.0099 1.09656, 2018.
11

---

Under review as a conference paper at ICLR 2020
[32] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland. Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
[33] Anusha Nagabandi, Gregory Kahn, Ronald S Fearing, and Sergey Levine. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning, In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 7559-7566. IEEE, 2018.
[34] Prabhat Nagarajan, Garrett Warnell, and Peter Stone. Deterministic implementations for reproducibility in deep reinforcement learning. In AAAI 2019 Workshop on Reproducible AI, January 2019.
[35] Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Over- coming exploration in reinforcement learning with demonstrations. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 6292 6299. IEEE, 2018.
[36] Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee. Self-imitation learning, arXiv preprint arXiv: 1806.0663.. 2018.
[37] Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In Advances in neural information processing systems, pp.
2016.
[38] Ian Osband, Benjamin Van Roy, Daniel Russo, and Zheng Wen. Deep exploration via random- ized value functions. arXiv preprint arXiv: /703. 07608, 2017.
[39] Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Actor-mimic: Deep multitask and transfer reinforcement learning. arXiv preprint arXiv:1511.06344 2015.
[40] Xue Bin Peng, Glen Berseth, and Michiel Van de Panne. Terrain-adaptive locomotion skills using deep reinforcement learning. ACM Transactions on Graphics (TOG), 35())881, 2016.
[41] Xue Bin Peng, Glen Berseth, KangKang Yin, and Michiel Van De Panne. Deeploco: Dy- namic locomotion skills using hierarchical deep reinforcement learning. ACM Transactions on Graphics (TOG), 36(4):41, 2017.
[42] Andrei A Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirkpatrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy distillation. arXiv preprint arXiv: 151,0625,, 2015.
[43] Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies as scalable alternative to reinforcement learning. arXiv preprint 17Xiv:1703.0000 2017.
[44] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897, 2015.
[45] Satinder P Singh. The efficient learning of multiple task sequences. In Advances in neural information processing systems, pp. 251-258 1992.
[46] David JT Sumpter. Collective animal behavior. Princeton University Press, 2010.
[47] Richard Sutton, Andrew G Barto, et al. Introduction to reinforcement learning, volume 135. MIT press Cambridge, 1998.
[48] Richard S Sutton, David McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in neural information processing systems. pp. 1057-1063, 2000.
[49] Yee Teh, Victor Bapst, Wojeiech Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. In Advances in Neural Information Processing Systems, pp. 4496-4506, 2017.
[50] Chen K Tham. Reinforcement learning of multiple tasks using hierarchical emac architecture. Robotics and Autonomous Systems, 1544)224-277, 1995.
12

---

Under review as a conference paper at ICLR 2020
[51] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 0026-5033. IEEE, 2012.
[52] Samuele Tosatto, Matteo Pirotta, Carlo d'Eramo, and Marcello Restelli. Boosted fitted q- iteration. In Proceedings of the 34th International Conference on Machine Learning-Von Volume 70, pp. 3434-3443, JMLR. org. 2017.
[53] Yu Wang and Hongxia Jin. boosting- based deep neural networks algorithm for reinforcement learning. In 2018 Annual American Control Conference (ACC), pp. 1065-1071. IEEE, 2018.
[54] Marco A Wiering and Hado Van Hasselt. Ensemble algorithms in reinforcement learning. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics)) (8(())930--9))))) 2008.
[55] Tianhao Zhang, Gregory Kahn, Sergey Levine, and Pieter Abbeel. Learning deep control policies for autonomous acrial vehicles with mpe-guided policy search. In 2016 IEEE international conference on robotics and automation (ICRA), pp. 528-535. IEEE, 2016.
[56] Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 1320-43288 2018.
13

---

Under review as a conference paper at ICLR 2020
A DIFFERENT DISTILLATION SCHEMES FOR ACTOR-CRITIC METHODS
As actor-critic methods like SAC 19 train a critic in addition to the policy, three straightforward knowledge distillation schemes can be used: distilling both the policy and the critic, distilling the policy only, and distilling the critic only. In this section, we compare the both schemes to justify our choice of distilling both the policy and the critic. We plot the performance of both schemes in Figure [ where SAC-CIKD (policy only) and SAC-CIKD (critic) stand for the variant of SAC-CIKE that distills the policy only and distills the critic only, respectively. We see that both SAC-CIKD (policy) and SAC-CIKI (crilic) sufficiently improve the performance against the vanilla SAC while SAC-CIKD achieves the best performance. This result shows that distilling the critic and the policy simultaneously achieves superior performance.
Hopparood
Amt-112
0930
Meen retam
3000
#000
Moss
2303
2000
1003
Timesteps
Timedtops
SAC FaLottyFaaee c_onyyFalsee)
SAC-CKKO (pi Doyyotyr Faarty=Falie)
SAC CID (PL Falyeeea (_onky-Tue)
Figure 7: Comparison different distillation schemes. SAC-CIKD (policy only) and SAC-CIKD (critic) denote the variants of SAC-CIKD that distill the policy only and distill the critic only, respectively. This figure shows that distilling the critic and the policy simultancously is more effective.
A.1 EXTENDED EXPERIMENTAL RESULTS
We run SAC-CIKD as well as each of our baseline algorithms for longer timesteps in this set of experiments. We ran for 1.5 Million timesteps, which is longer than our 1M timesteps in the body.
Walker2d-i2
Reachor-v2
HumanodStancupp-22 x2
Humano d.12
200000
0000
175000
Mean
ratur
Mean rotur
4000
-10
Mean
125000
-15
2000
100000
2000
-20
75000
000200
1000000
100000
000000
1000300
1600000
500000
1000000
1000000
600000 1000000 1000000 Timastapa
Timestaps
Timeslapa
TImestapa
Hopper-v2
HalChhetah-i2 v/2
8000
0000
Mearn return
SAC CIKD (best)
2000
10000
SAC-CIKD (worat)
Mean
Moan
2000
Ensemble (best)
1000
5000
Ensemble-SAC (worst)
Vanilla SAC
0
G00200 1000000 1500000 Timestops
000000
1000000 15C0000 Timeelaps
0
500D00 1300000 1500000
Timeelsps
Figure 8: Full performance evaluation of inter-agent knowledge distillation. SAC-CIKD represents the implementation of our method upon SAC; Vanilla-SAC stands for the original SAC: Ensemble-SAC is an analogous variant of Osband et al. 37''s 's method on vanilla-SAC (effectivvly SAC-CIKD without inter-agent knowledge distillation). See Section 5.2 for details.
A.2 DOMINANCE IN THE ABSENCE OF KNOWLEDGE DISTILLATION
In this section, we extend our analysis in Section 5.5 with the goal to examine if there a consistently dominated agent in Ensemble-SAC and Vanilla-AAC (see Section 5.2 for details). For Ensemble-SAC, we plot the index of the best teesforeroonnng agent in the ensemble in Figure 9a and the propotions in Figure 9b It can be seen that there is no consistently dominated agent in Ensemble-SAC, which is similar ar to the result of CCKD in Section [5.5 On the other hand, we plot the index of the perf-per agent, among three independently trained Vonilla-SAC agents (with different random
14

---

Under review as a conference paper at ICLR 2020
seeds), in Figure 9g The proportions presented in Figure 9d It can be observed that there is
significantly dominated agent among these agents (i.e. h = 2).
+fd 000
4b heoe choed the deted
Blstt agent Indos
.022.
1000
1
stopa
(b)
JMM
Best agent rdox
PO
10p-00
20.4
of
44
AWAAMP 2007 200P
mm
zlopa
(C)
(d)
Figure (a) The distribution of selected teacher agent indices. (b) The proportion of time that each agent
selected to be the teacher over the entire training process. (c) The distribution of the best-performing agent among three agents Vanilla-SAe trained with different random seeds. (d) The proportion of time that each agent is the best-performin agent over the entire training process.
15


Reviews:
[{"review_id": "HygygT3M9H", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an RL training procedure that maintains an ensemble of k policies and periodically pushes all the policies to be closer to the best performing one. The formulation, experiments and analysis are very clear and show a mild improvement over using the same underlying RL algorithm without the imitation part. The idea is close to many other proposed in the literature, but to my knowledge it is the first time this exact procedure is studied in detail.\n\nThe first piece of their approach is an off-policy RL algorithm. In their case, they use SAC. The second piece is adding an ensemble of policies (3 in their case), and randomly selecting one of them every time a rollout is collected, and using the collected rollout to update all the policies. This effectively implies 3 times more overall gradient updates compared to SAC. They call this ablation SAC-ensemble. Interestingly they only use the most recently collected trajectories to update all policies, and despite storing the rollouts in a replay buffer, they seem to only use the stored transitions for the imitation part described below. Some of their experimental results uses extra gradient steps, although it’s not clear if those gradient steps are also only on the last rollout collected, or on transitions sampled from the replay buffer as it is typical in off-policy RL methods. In general, I think the work could improve with more details about how much the policy training could improve by increasing the number of gradient steps on the full replay buffer.\n\nThe final piece of their method is selecting the best performing policy (or “teacher”) of the ensemble based on the recent experience, and update all other policies by executing some gradient steps on the KL divergence between them and the current “teacher”. They also try an experiment where the “teacher” is selected randomly, and it does surprisingly well in my opinion (specially realizing that the “HalfCheetah” experiments seem to not have all seeds run to convergence, please report the full results). I suspect that most of the benefit of their method comes from randomly perturbing the parameters of the policies in the ensemble. More thorough and careful experimentation needs to be carried out to investigate this direction. This is in fact not very surprising given the results of Evolutionary Strategy methods, or Population-based training (even if usually used for hyper-parameters adaptation).\n\nFurthermore, the authors only run the environments for 1M steps, whereas in previous works some environments are shown to get higher return after more training steps. I would also encourage the authors to report the results in all the standard MuJoCo benchmarks for the ablations (even if it’s in the appendix) to better asses their claims.\n\nOverall, this is a very well presented work, although it lacks some novelty and a few more thorough experiments to fully understand the improvements they show. I think this idea is worth sharing with the community, and I recommend a weak accept.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Hkx1u0nj_H", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper proposed an ensemble method (CIKD) that train multiple agents and\nuse knowledge distillation to transfer knowledge from the current best agent to\nsub-optimal agents periodically.  According to the reported results, CIKD is a\nsimple yet effective approach to improve sample-efficiency and final performance.  \nThe experimental results are sufficient, and the ablation studies are conducted thoroughly. It is shown that both selecting the best agent and using KD to\ntransfer knowledge are effective comparing to other naive alternatives. \n\n\nI recommend the acceptance of this paper. \n\nThe paper proposed a novel approach (CIKD) to improve the sample-efficiency of the state-of-the-art. The proposed ensemble approach is aligned with our intuition, and it is effective. The authors proposed to train several agents at the same time and randomly select one of\nthe agents as a behavior policy during each rollout. Then the collected trajectory is used to update the policy of all agents. Meanwhile,\nthey keep tracking the performance of each agent and use the current best agent to conduct knowledge distillation to other agents periodically. \n\nThis paper first conducts experiments to show when consolidating\nthe SAC with CIKD, both of the final performance and sample-efficiency can be improved. Then a set of ablation studies verified the best agent selection strategy, and the knowledge distillation\nstrategy is necessary for the ensemble method. \n\n\nInvestigation on the reasons for improvement:\nThough extensive ablation studies have shown the effectiveness\nof each component of CIKD. It is still not clear why this approach\ncan be effective. \nIntuitively, it is possible that the exploration from a set of agents would outperform\na single agent. The measure of exploration efficiency could help in explaining the results. Furthermore, better exploration not necessarily\nleads to better performance and sample-efficiency. Does knowledge distillation serve as a better alternative to exploit existing data? \n\nModel/algorithm agnostic\nThe proposed method is more convenient to be applied with off-policy approach when the policy is in the form of softmax. Is it also applicable\nto other approaches? \n\nExperiments:\nHow do you determine when to stop the KD process? As mentioned in section 5.5, if we conduce KD fully, all students would be just imitating\nthe teacher's behavior. It seems the key is to tune a good termination\nthreshold for each task? Are there any guidelines to set up this threshold?\nDo you have some automatic way to terminate the KD procedure?\n\n\nMinor:\nL1, P5, \"how to CIKD improves the sample efficiency\" \n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "Bkg-TbEmor", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a method for using an ensemble of deep reinforcement learning policies, where members of the ensemble are periodically updated to imitate the most promising member of the ensemble. Thus learning proceeds by performing off policy reinforcement learning updates for each individual policy, as well as some supervised learning for inter-policy imitation learning.\n\nI start by what I view as the positive aspects about the paper:\n1- The algorithm is quite simple (to understand and to implement).\n2- Experimental results are performed on a variety of domains, and more importantly, each experiment is motivated by a question.\n\nThat said, I have some concerns about this paper which I list below:\n\n1- Perhaps my biggest concern is that the approach is not motivated from a theory stand point. There has been interesting results in Osband's work [Osband, 2016] (and references therein) for randomized value functions which can serve as a foundation for this work. That said, a) Osband's results, at least immediately, are related to value-function based methods, as opposed to policy gradient b) the KL update which one could argue is the main and only significant contribution of the paper, is not justified by Osband or any other prior work c) there is not anything that this paper adds to the literature to better justify diversity through randomization and/or imitation learning based on the best member of the ensemble.\n\n2- I have found various claims in the paper which are unclear, scientifically not true, or sometimes even contradicting. In Introduction, for example, the authors mention that the agent sometimes gets into a sub-optimal policy and may require a large number of interactions before escaping the sub optimal policy. How does gathering more data help to improve the policy? Either we are in a local maximum, which if we are doing gradient ascent, there is really not much we could do, or that we are in a saddle point, which we can escape by adding some noise to the gradient. [Jin,2017]\n\n3- In section 4.3 the authors talk about on-policy methods requiring importance sampling (IS) ratios. To the best of my knowledge, IS is only used for off-policy learning. Can the authors provide a link to an on-policy method that does IS?\n\n4- Again in section 4.3 authors claim and I quote \"Using off-policy methods, all the policies in the ensemble can easily be updated, since off-policy update methods can perform updates from any \\tau\". But later on in Section 5.3 authors claim that \"off-policy actor-critic methods (e.g. SAC) cannot fully utilize the other agent's or past experience.\" So which statement is true?\n\n5- Again, the KL update is interesting, but is it even surprising that the KL update is necessary for an ensemble of policies updates using policy gradients? In the absence of this KL update, which the authors characterize as the method that Osband proposed, the policies could generally be arbitrarily far from one another. This means that each policy needs to perform policy evaluation using trajectories that are coming from other policies who in principle can be radically different than the policy we want to update. This means that updates will be quite \"off-policy\" which we know can really degrade the quality of the estimated gradient. This is perhaps why even choosing a random policy to update towards is providing \"some\" improvement. I think this is the real insight, but it is not really discussed at all in the paper.\n\n6- On the same note, I do not think that one can say Osband's method is the same as CIKD but only without the KL update. Most notably, Osband's work was presented for value-function-based methods like DQN. These methods work fundamentally different than policy gradient methods, which rely on (near) on-policy updates to perform good policy improvements. In that sense, the presented results make sense, but I disagree with the framing of the results and how they are presented here.\n\n7- In section 5.3, when the authors utilize more policy updates to have a fair comparison, are they retuning hyper parameters? Surely they need to do that, at least for hyper-parameters that are known to be super important such as the step size.\n\n8- Overall I liked section 5.5 that is trying to dissect causes for improvement. However, it seems like that the \"dominant agent\" hypothesis has been rejected hastily, unless I misunderstood the experiment. The authors show that the notion of best is spread across different agents. But of course this will be the case in light of the KL update, since the policies are getting closer to one another. Can you redo the experiment in the absence of the KL update?\n\n9- Have the authors thought about any connection between this and genetic algorithms? In genetic algorithms, the idea is the next set of candidates are chosen based on the most promising candidates in the current iteration. CIKD seems like a soft implementation of this idea.\n\nIn light of the comments above, I am voting for weak rejection, though as I said before, I do see some interesting things in this paper. I encourage the authors to think about CIKD from a theoretical lens in the future.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces an ensemble of RL agents that share knowledge amongst themselves. Because there are no theoretical results, the experiments have to carry the paper.  The reviewers had rather different views on the significance of these experiments and whether they are sufficient to convincingly validate the learning framework introduced. Overall, because of the high bar for ICLR acceptance, this paper falls just below the threshold. 

Decision: Reject