REPAINT: Knowledge Transfer in Deep Actor-Critic Reinforcement Learning
Authors: Yunzhe Tao
Sahika Genc
TAO SUN
Sunil Mallya
Venue:  (2021)
Keywords: reinforcement learning
transfer learning
actor-critic RL
representation transfer
instance transfer
task similarity
MuJoCo
DeepRacer

Abstract: Accelerating the learning processes for complex tasks by leveraging previously learned tasks has been one of the most challenging problems in reinforcement learning, especially when the similarity between source and target tasks is low or unknown. In this work, we propose a REPresentation-And-INstance Transfer algorithm (REPAINT) for deep actor-critic reinforcement learning paradigm. In representation transfer, we adopt a kickstarted training method using a pre-trained teacher policy by introducing an auxiliary cross-entropy loss. In instance transfer, we develop a sampling approach, i.e., advantage-based experience replay, on transitions collected following the teacher policy, where only the samples with high advantage estimates are retained for policy update. We consider both learning an unseen target task by transferring from previously learned teacher tasks and learning a partially unseen task composed of multiple sub-tasks by transferring from a pre-learned teacher sub-task. In several benchmark experiments, REPAINT significantly reduces the total training time and improves the asymptotic performance compared to training with no prior knowledge and other baselines.

Under review as a conference paper at ICLR 2021
REPAINT: KNOWLEDGE TRANSFER IN DEEP ACTOR- CRITIC REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Accelerating the learning processes for complex tasks by leveraging previously learned tasks has been one of the most challenging problems in reinforcement learning, especially when the similarity between source and target tasks is low or unknown. In this work, we propose algorithm (REPAINT) for deep actor-critic reinforcement learning paradigm. In representation transfer, we adopt a kickstarted training method using a pre-trained teacher policy by introducing an auxiliary cross-entropy loss. In instance transfer, we develop a sampling approach, i.e., advantage-bar based experience replay, on tran- sitions collected following the teacher policy, where only the samples with high advantage estimates are retained for policy update. We consider both learning an unseen target task by transferring from previously learned teacher tasks and learn- ing partially unseen task composed of multiple sub-tasks by transferring from a pre-learned teacher sub-task. In several benchmark experiments, REPAINT signif- icantly reduces the total training time and improves the asymptotic performance compared to training with no prior knowledge and other baselines.
Arreseetttiin-AAddNNaaaae
Transfer
INTRODUCTION
Most reinforcement learning methods train agent from scratch, typically requiring huge amount of time and computing resources. Accelerating the learning processes for complex tasks has been one of the most challenging problems in reinforcement learning (Kaelbling al.) [996] |Sutton & Barto 2018). In the past few years, deep reinforcement learning has become more ubiquitous to solve sequential Dacision-making problems in many real-world applications, such as game playing (OpenAl et al.) 2019 Silver et al. 2016), robotics (Kober et al.) 2013) OpenAl et al. 2018)) and autonomous driving (Sallab et al. 2017). The computational cost of learning grows as the task complexity increases in the real-world applications. Therefore, it is desirable for learning algorithm to leverage knowledge acquired in one task to improve performance on other tasks.
Transfer learning has achieved significant success in computer vision, natural language processing, and other knowledge engineering areas (Pan & Yang/ 2209). In transfer learning, the teacher (source) and student (target) tasks not necessarily drawn from the same distribution (Taylor al. 2008a). The unseen student task may be a simple task which is similar to the previously trained tasks, or a complex task with traits borrowed from significantly different teacher tasks. Despite the prevalence of direct weight transfer, knowledge transfer from previously trained agents for reinforcement learning
tasks has been gaining much attention until recently (Barreto al. 2019 Ma et al. 2018 Schmitt
et al. 2018 Lazaric 2012 Taylor & Stone, 2009).
In this work, we propose knowledge transfer algorithm for deep actor-critic reinforcement learn- ing, i.c., REPresentation And INstance Transfer (REPAINT). The algorithm can be categorized as a representation-instne kickstarted training method (Schmitt et al., (018)) using previously trained teacher policy, where the teacher policy is used for computing the auxiliary loss during training. In instance transfer, we develop a new sampling algorithm for the replay buffer collected from the teacher policy, where only keep the transitions that have advantage estimates greater than a threshold. The experimental results across several transfer learning tasks show that, regardless of the similarity between source and target tasks, by introducing knowledge transfer with REPAINT, the number of training iterations needed by the agent achieve some reward target can be significantly reduced when compared to
transfer approach. Specifically, in representation transfer, we adopt

---

Under review as a conference paper at ICLR 2021
training from scratch and training with only representation transfer instance transfer. Additionally, the agent's 'ss asymptotic performance is also improved by REPAINT in comparison with the baselines.
RELATED WORK: TRANSFER REINFORCEMENT LEARNING
Transfer learning algorithms in reinforcement learning can be characterized by the definition of transferred knowledge, which contains the parameters of the reinforcement learning algorithm, the representation of the trained policy, and the instances collected from the environment [Lazaric 2012). When the teacher and student tasks share the same state-action space and they are similar enough (Ferns al. [004] Phillips 2006). parameter transfer the most straightforward approach, namely, one can initialize the policy value network in the student tasks by that from teacher tasks (Mehta) ict al. 2008 Rajendran et al. 2015). Parameter transfer with different state-action variables is more complex, where the crucial aspect the student state-action space (Gupta et al. 2017 Talvitie & Singh 2007; Taylor et al. 2008b).
find a suitable mapping from the teacher state-action space to
Most of the transfer learning algorithms fall into the category of representation transfer, where the reinforcement learning algorithm learns a specific representation of the task or the solution, and the transfer algorithm performs abstraction process to fit it into the student task. [Konddrri et al. (2012) uses the reward shaping approach to learn a portable shaping function for knowledge transfer, while some other works use neural networks for feature abstraction (Duan et al. 2016 Parisotro iet al. 2015 Zhang et al. 2018). Policy distillation (Rusu et al. 2015) or its variants is another popular choice for learning the teacher task representation, where the student policy aims to mimic the behavior of pre-trained teacher policies during its own learning process (Schmitt et al. 22018: Yin
Pan, 2017). Recently, successor representation has been widely used in transfer reinforcement learning. in which the rewards are assumed to share some common features, so that the value function can be simply written as a linear combination of the successor features (SF) (Barreto et al.) 2017] Madarasz & Behrens, 2019). Barreto et al. (2019) extends the method of using SF and generalised policy improvement in Q-learning (Sutton & Barto 2018) to more general environments, [orsa et al.
(2018), Ma et al. 2018), and (Shaall et al. (2015a) learn universal SF approximator for transfer.
The basic idea of instance transfer algorithms is that the transfer of teacher samples may improve the learning on student tasks. Lazaric et al. (2008 and Tirinzoni et al. (2018) selectively transfer samples on the basis of the compliance between tasks in a model-free algorithm, while Taylor et al. (2008a) studies how model-based algorithm benefit from samples coming from the teacher task.
However, most of the aforementioned algorithms either assume specific forms of reward functions or perform well only when the teacher and student tasks are similar. Additionally, very few algorithms are designated to actor-critic reinforcement learning. In this work, we propose representation- instance transfer algorithm to handle the generic cases of task similarity, which is also naturally fitted for actor-critic algorithms and can be easily extended to other policy gradient based algorithms.
3 BACKGROUND: ACTOR-CRITIE REINFORCEMENT LEARNING
general reinforcement learning (RL) agent interacting with environment can be modeled a Markov decision process (MDP), which is defined by tuple M (8, (S,,, where S and A are sets of states and actions, respectively. The state transfer function p(-[s, a) maps a state and action pair to probability distribution over states. S A S IR. denotes the reward function that determines a reward received by the agent for a transition from (s. a) to s'. The discount factor, y E [0, 1], provides means to obtain long-term objective. Specifically, the goal of an RL agent is to learn a policy that maps a state to a probability distribution over actions at each time step f, so that ag- (-1ss) maximizes the accumulated discounted return Σ120 7 "'t, at, 81+1).
To address this problem, a popular choice adopt is the model-free actor-critic architecture, e.g., Konda & Tsitsiklis (2000); [Degri et al. (2012); Mnihh et al.] (2016): (Schulman et al. (2015 (017). where the critic estimates the value function and the actor updates the policy distribution in the direction suggested by the critic. The state value function at time is defined as
VT(s) (s) = EajNT(1As)
'r 91+1) Σ izt
(3.1)
2

---

Under review as a conference paper at ICLR 2021
Similarly, the action value function (also called Q function) is defined by
(s) (
Σ r(s: ai 8141) 81 at
(3.2)
The actor-critic methods usually rely on the advantage function, which is computed as (" Q" (s.a) (*))..
(3.3)
Intuitively, the advantage can be taken the extra reward that could be obtained by taking particular action a. The advantage is usually approximated by generalized advantage estimator (GAE) (Schulman et al. 2015b), which is defined as the arooneniiallywweighee average of the k-step discounted advantage, namely,
2O
A, 4, (GAB(7.A)
Σ (rA)) (rett + ($1+1+1) V" ($++)))
(3.4)
1=0
where the parameter 0 < À E allows a trade-off of the bias and variance.
In deep RL, the critic and actor functions usually parameterized by neural networks. Then the policy gradient methods can be used to update the actor network. For example, in the clipped proximal policy optimization (Clipped PPO) (Schulman et al., (017), the policy's objective function defined to be the minimum between the standard surrogate objective and an e clipped objective:
Letip (f) E. min (re(0) A.. clip (r. c, + A.)]
(3.5)
where the policy is parameterized by and (0) ) is the likelihood ratio that
re (0
(3.6)
The (a, (8x)
Moreover, the clip function truncates re the range of (1 e, +).
REPRESENTATION AND INSTANCE TRANSFER IN ACTOR-CRITIC RL
In this section, describe knowledge transfer algorithm, REPAINT, for actor-critic RI frame- work. There are two core concepts underlying our approach, i.e., representation transfer and instance transfer. In the representation transfer, we employ kickstarted training approach (Schmitt al.) 2018) based on the policy distillation. Then, in the next iteration following the kickstarting, we update the student policy using an advantage- based experience replay. We do not assume that the teacher and student tasks are similar or drawn from the same distribution.
The REPAINT algorithm is provided Algorithm D where the value function and policy function are parameterized by and 0, respectively. Without loss of generality, we demonstrate the policy update using the Clipped PPO loss stated in equation [3.5] and use a single teacher policy in both representation and instance transfers. In practice, it can be directly applied to any advantage-basee policy gradient RL algorithms, and it is straightforward to have different and multiple teacher policies in each transfer step. The more general algorithm and several variants are presented in Section A
4.1 REPRESENTATION TRANSFER: KICKSTARTED TRAINING
In representation transfer, we use kickstarting training pipeline (Schmitt et al., 2018). which can be viewed as combination of policy distillation (Rusu al. 2015) and population based training (Jaderberg et al. 2017). The main idea is to employ an auxiliary loss function which encourages the student policy to be close to the teacher policy on the trajectories sampled by the student. Given teacher policy Treacher, we introduce the auxiliary loss as
Ldisin (0) = H (Tresche ((((ss)((((ss))))
(4.1)
where H--ll-) is the cross-entropy. In order for the agent to maximize its own future rewards, kickstarting adds the above loss the Clipped PPO objective function, i.e., equation 3.5 weighted optimization iteration & by the scaling 3x 2 0:
TL. (0) Tetip (0) .Bg F (0)
(4.2)
3

---

Under review as a conference paper at ICLR 2021
Algorithm 1 REPAINT algorithm with Clipped PPO
for iteration & = 1, 2, do if is odd then Collect trajectories S = (s, r)) following Tewu (s) Fit state value network Vú using to update v/
Compute advantage estimates Aq..... Ay using equation 3.4
Perform gradient optimization on I (6) defined in equation 4.2 else
|/ representation transfer
Collect trajectories S' {(x, a, 8'} following Trencher
I/ instance transfer
Compute for each transition using current reward function and add to S' Compute advantage estimates Ai,.... Aqr using equation 3.4 for t= I=I....,T" do if A
< then
Remove A and the corresponding transition (st, at 8 from S Perform gradient optimization on Tettip (0) defined in equation 3.5
In our experiments, the weighting parameter 3e is relatively large at early iterations, and vanishes as & increases, which is expected to improve the initial performance of the agent while keeping it focused on the current task in later iterations.
4.2 INSTANCE TRANSFER: ADVANTAGE-BASEE EXPERIENCE REPLAY
In the instance transfer iteration, collect training samples following the teacher policy Teachers but compute the rewards using current reward function from the target task. Since the transitions are obtained from different distribution, we do not update the value network in this iteration. When updating policy network, we prioritize the transitions based on the advantage values and only use the samples that have advantages greater than a given threshold Ç. Moreover, since the teacher policy has been used in collecting roll-outs, we compute the loss without the auxiliary cross-entropy, but replace Taae with leacher in equation 3.5
The idea of prioritizing and filtering transitions is simple but intuitively effective. As mentioned in Section 3 the advantage can be viewed as the extra reward that could be obtained by taking particular action. Therefore, by retaining only the "good" samples from the teacher policy, the agent can focus on learning useful behavior under current reward function. We note that our filtering approach is related to, but different from, the prioritized experience replay (Schaul et al. 2015b), which prioritizes the transitions in replay buffer by the temporal-differt utilizes importance sampling for the off- off-poliit policy evaluation. In comparison, replay in actor-critic framework, where the replay buffer is erased after each training iteration. Unlike the stochastic prioritization which imposes the probability of sampling each transition, our method directly filters out most of the transitions from replay buffer and equally prioritizes the remaining data, which reduces the training iteration time. In addition, our method performs policy update without importance sampling. All transitions in a batch have the same weight, since the importance can be reflected by their advantage values. The experimental comparison can be found in Section 5.1
error (TD error), and method uses experience
The concept of advantage-based prioritizing has also been used in offline RL most recently (Peng al. 2019. Siegel al.) 2020) Waan al. 2020), order better leverage the off off-policy data. In offline RL, the "advantage weighting" can be regarded as a regularization to either critic or actor, where the model can focus on learning "good" actions while ignoring poor ones. Again, we want to remark that our proposed advantage-based experience replay has different formulation. To the best of our knowledge, this is the first time that the advantage-based filtering is applied to instance transfer learning. In Figure (b). we will compare the performance of several prioritization rules. The discussion is presented in Section 5.1
We also consider how the sampling experiences from the teacher policy impact the policy gradient update. Our incremental update fits into the framework of the off-policy actor-critic (Degris et al. 2012 Zhang al. 2019). We ignore the clip function in equation 3.5 simplicity and replace mo..
4

---

Under review as a conference paper at ICLR 2021
with Tteachers which leads to
V Velecip(0) - E Bsnduaste Roater p(5,, a) Vo log To [alaa))) Ama
(4.3)
where p(8, a) = (a)a)))/ (Teee) (s)s), and the advantage estimator is from last kickstarted training iteration. With the new term p(s, a), equation 4.3 can be viewed as a weighted sample expectation. Since the samples are drawn from the teacher policy, (alacber(ass) usually has large values. At the early learning stage, our advantage-based filtering results in large step size (due to large A*). Therefore, small p serves as the compensation to the training, which can prevent the system from over-fitting. In later iterations when the student policy is closer to the teacher, the update is close to traditional actor update except the filtering approach, which introduces bias because it changes the distribution in uncontrolled fashion. In practice, can adopt REPAINT in the early training stage, and then reduce to traditional actor-critic algorithms. As a consequence, the agent first learns useful teacher behavior to achieve good initial performance, and focuses on the target task afterwards.
4.3 DISCUSSION
A critical question in knowledge transfer for RL is how to characterize the similarity between source and target tasks, since most of the transfer learning algorithms perform well when the two are similar. If samples from the two MDPs are given, some metrics can be defined to compute or learn the task similarity, e.g., the Kantorovich distance based metric (Song et al. 2016),, the restricted Boltzmann machine distance measure (Ammar et al.) [2014), the policy overlap (Carroll & Seppi (005), and the task compliance defined in Lazaric et al. (2008). However, in general, the similarity is usually unknown before getting any samples, unless other specific information is given. For example, the methods using successor representation (Barreto et al. 2019: Borsa et al. 2018 [Schaul et al. (01 assume that the reward functions among tasks a linear combination of some common features, and they only differ in the feature weights, namely, r(s, 8') = E; 14;0y(s, a, s') with fixed Φ, 's. Then the similarity can be characterized as the distance of the weight vectors.
In this paper, we aim to show that the proposed REPAINT algorithm handles generic cases in task similarity. To demonstrate that, for simplicity, we assume in the experiments that the state and action spaces stay the same between teacher and student tasks, and the reward functions have the form of linear combination of common features. However, we use the cosine distance function to define the task similarity in this paper, namely, the similarity between two tasks with reward functions ri (8, a, 8') Φb (8,00 a, &'TT ui and r2(8, a, s') ))= (8, 0, 8') u2 can be computed as
W1 112
sim(r1,
(4.4)
w w2
We say the two tasks are similar if sim(ri, r2) > 0. Otherwise, they are considered be different.
In regards to the proposed REPAINT algorithm, we also want to remark that the policy distillation weight 8& and advantage filtering threshold are task specific, which are dependent with the one-step rewards. To this end, one can consider to normalize different reward functions in practice, so that the one-step rewards are in the same scale. In general, larger 3% encourages the agent to better match the teacher policy, and larger Ç leads to that fewer samples are kept for policy update, which in result makes current learning concentrate more on the high-advantage state-action transitions. More investigations on the two parameters can be found in the experiments
5 EXPERIMENTS
In this section, we use two platforms across multiple bench-marking tasks for assessing the REPAINT algorithm (see Figure T). We first perform experiments on two continuous control tasks, i.e., Ant and Reacher, in MuJoCo simulator [Todorov] 2016). We compare the performance of REPAINT with training from scratch and training with only kickstarting or instance transfer. Note that since the value network cannot be updated in the instance transfer iteration, we implement training with only instance transfer by alternately applying Clipped PPO and instance transfer (namely, setting 3x = 0 in Algorithm]] [). For discrete control tasks, we use the AWS DeepRacer simulator (Balaji ct al. 2019) to evaluate different algorithms on complex tasks, such as single-car time-trice trial race and multi-car racing against bot cars. The detail of our experimental setup is presented in Section
5

---

Under review as a conference paper at ICLR 2021
MuloCo-Reacher
(b) MuJoCo-Ant
(c) Single-car time trial
(d) Racing with bot
Figure I: The simulation environments used in the experiments. Note that the racing car in (c) only has a monocular camera, while (d) it is equipped by a stereo camera and a Lidar.
mean reveau -15
mean reward
mean rewwe
rewer
Base The Edsaarting
Saneline
10
-15
Siiksta ting
12
15
20%
tranderr REPAINT
transfer
REPAIT
-20
15g
350
evalustion iteration
evaluation iteration
Iteration
Iteration
(a) Comparison against baselines
Comparison of different instance prioritization rules
Figure 2: Evaluation performance for MuJoCo-Reachhs averaged across five runs. In each of (a) and (b), we consider both teacher task similar (left) and different from (right) the target task.
We consider three different types of task similarities in the experiments based on the cosine similarity (equation 4.4). i.c., the teacher (source) task is similar task, different task, sub-task of the target task (some feature weights are zero in the reward function). We also evaluate the effectiveness of using different teacher policies, cross-entropy weights, and advantage thresholds. Moreover, several prioritization (filtering) approaches in the instance transfer are investigated. More extensive results and discussion on the properties of REPAINT are presented in Section C for completeness.
5.1 CONTINUOUS ACTION CONTROL MuJoCo PHYSICS SIMULATOR
MuJoCo-Reacher. In the target task, the agent is rewarded by getting close to the goal point (distance reward) with less movement (control reward). We first compare our REPAINT algorithm against training with only kickstarting or instance transfer and with no prior knowledge (baseline), based on two different teacher tasks. The first teacher policy is trained with similar reward function but a higher weight on control reward, where we set be as an example, so that the cosine similarity is positive and the learning is transferred from similar task. Another teacher policy is trained in a different task, where the agent is penalized when is close to the goal. In this case, the cosine similarity is zero. After each training iteration, evaluate the policy for another 20 episodes. The evaluation performance is presented in Figure[ja) We remark that same trends can be observed from figures of training performance against steps. REPAINT outperforms baseline algorithm and instance transfer in both cases of task similarities, regarding the training convergence time, the asymptotic performance, and the initial performance boosting. Although kickstarted training can improve the initial performance, it has no performance gain in convergence when the teacher behavior opposed to the expected target behavior. On the other hand, while instance transfer does not boost the initial performance, it surpasses the baseline performance asymptotically in both cases.
We also compare the performance of several advantage prioritization (filtering) rules in the instance transfer of REPAINT, which includes keeping transitions with high absolute values of advantages (1-1> a), keeping top 20% of transitions the advantage value ranking (top 20%), our proposed filtering nule (- (). and prioritized experience replay described in Schaul et al. (2015b) (PER). For PER, we used the advantage estimates to compute the prioritization instead of TD errors for fair comparison, and slightly tuned the and / parameters in the probability computation and weights, respectively. From Figure (b.). we can observe that the proposed filtering rule and top 20% rule perform better than others the initial performance, where most of

---

Under review as a conference paper at ICLR 2021
Table 1: Evaluation performance for MuJoCo-Ant, averaged across three runs. In this experiment, teacher task is similar to the target task.
Average return score of i-erations
Model
100
200
300
400
600
800
1000
Baseline
37
202
653
1072
1958
2541
3418
Kickstarting (teacher coef.=
1686
2657
2446
3688
4233
4744
4924
Instance transfer (C = 1.2, teacher coef.= 3)
338
1058
1618
1973
3117
4163
4432
REPAINT (Ç- 1.2, teacher coef.= 3)
1782
2663
3150
3789
4698
4996
5173
REPAINT (Ç 0, teacher coef. = 3)
1747
1932
2665
3245
4250
4561
4809
REPAINT ({ 0.8, teacher coef.= 3)
2004
2593
2987
3638
4299
4847
5074
REPAINT (Ç 1.5, teacher coef.= 3)
2009
2605
2764
3309
4061
4838
5116
REPAINT (Ç 1.2, teacher coef.= 5)
1034
2515
2998
3492
4621
4855
5028
REPAINT (Ç 1.2, teacher cnef.= 10)
267
1423
1785
2019
2504
2914
3420
400
meah progress (95
1.00
mean progress (%)
100
reward
300
80
mhae reward
300
80
80
60
200
Roce ceactrring
80
40
Rosetee
100
100
rotance Guraker
28
Iratance Farata
traticley
20
evaluation Reeation
evaluation iteration
evaluation Reration
evaluation Soonaationnnn iteration
(a) Outer-lane reward task with inner-lane teacher
(b) Inner-lane reward task with outer-lanc teacher
Figure 3: Evaluation performance for DeepRacer single-car time-trial race, including mean accumu-
lated rewards and mean progress (lap completion percentage), averaged across five runs.
the samples are removed for policy update. Moreover, PER does not work well as other approaches when task similarity is low, since it includes all low-advantage teacher samples which have no merits for the student policy to learn. Therefore, suggest the proposed filtering rule with threshold Ç or the ranking-bass based filtering rule with a percentage threshold the instance transfer.
MuJoCo-Ant. Other than the comparison against baselines, we also assess REPAINT algorithm with different filtering thresholds here, i.e., the parameter Ç in Algorithm] [ In this target task, the agent rewarded by survival and moving forward, and penalized by control and contact cost. The teacher task is a similar task, which has the reward function with a higher weight on moving forward. In the experiments, we train each model for 1 1100 iterations, and evaluate each training iteration for another 5 episodes. The evaluation results are shown in Table []] We can again observe that, when task similarity is high, training with REPAINT or kickstarting significantly improves the initial performance, reduces the learning cost of achieving certain performance level, and improves the asymptotic performance. The table also indicates that our algorithm is robust to the threshold parameter. Similar learning progresses we evaluate the transfer performance from the source tasks with different levels of cosine similarities. We set the forward coefficients in the reward function of teacher task be 3, 5, and 10, corresponding to the cosine similarities of 0.87, 0.76, and 0.64. By comparing the performance of REPAINT with 1.2 and different teacher coefficients, we can see that task similarity impacts the overall training, even when they are all related.
observed from training with different Ç values. In addition,
5.2 AUTONOMOUS RACING IN AWS DEEPRACER SIMULATOR
Single-car time trial. In this experiment, we use two different reward functions, one of which encourages the agent to stick to the inner lane (left of the yellow center-line) and the other rewards the agent when it is in the outer lane. When we use one reward function in the student task, we provide the teacher policy that is trained with the other reward. Therefore, the cosine similarity of teacher and target task is zero. We evaluate the policy for 5 episodes after each iteration. The evaluation performance is presented in Figure where both average return and progress (percentage of lap the
7

---

Under review as a conference paper at ICLR 2021
1500
mase Senaaain (86) 60
mean proggrant (16)
80
reward
monu rewer
2000
Baseline
Baseline
1t
Basefine
Basstine
500
20
25
evaluation Iteration
evallation
Iteration
ovaluation
Iteration
evaluation iteratioon
Task with advanced reward
(b) Task with progress based reward
Figure 4: Evaluation performance for DeepRacer multi-car racing against bot cars, averaged across three runs, The value at each iteration is smoothed by the mean value of nearest iterations.
Table 2: Summary of the experimental results.
Teacher type
Target score
Kks (pet. reduced)
Kir (pet. reduced)
KREP (pet. reduced)
Best scores (KS, IT, REP)
Env.
K Baseline
similar different
51 (71%)
(44%)
(76%)
-5.3,-5.9,-5.4
Reacher
-7.4
173
73 (58%)
127 (27%)
51 (71%)
-6.9,-6.4-552
Ant
similar
3685
997
363 (64%)
623 (38%)
334 (66%)
5464,5172,5540
different 394
18
13 (28%)
331,388,
Single-car
different
345
22
15 (32%)
300,319,354
sub-lask
100
34 (66%)
(25%)
(71%)
1542,1610.1623
Multi-car
sub-task 2.7
77
66 (14%)
53 (31%)
25 (68%)
4.9,4.2,6.1
agent accomplished when it went out of track) are given. Although upon convergence, all models can finish a lap without going off-track, REPAINT and kickstarting again significantly boost the initial performance. However, when the teacher task is very different from the target task, training with kickstarting cannot improve the final performance via transfer. In contrast, instance transfer can still reduce the training convergence time with a final performance better than kickstarting.
In Section we also compare the REPAINT algorithm with a commonly used parameter transfer approach, i.e., warm start, when the task similarity is low. Furthermore, the effect of different cross- entropy weights 3g or instance filtering thresholds Ç is studied, which is presented in Section C.3
Racing against bot cars. The REPAINT algorithm is still helpful when the RL agent needs to learn multiple skills task. the environment racing against bot cars, the agent has to keep on the track while avoiding dlw-movvin cars in order to obtain a high return score. Therefore, we first train a teacher policy which is good at object avoidance, namely, the agent is rewarded when it keeps away from all bot cars and gets a penalty when the agent is too close to bot car and heads towards to it. Then in the target tasks, we use two different reward functions to assess the models. First, we
advanced reward where other than keeping on track and object avoidance, it also penalizes the agent when it detects some bot car from the stereo camera and is in the same lane with the bot. The evaluation performance is shown in Figure 4 a). Since the environment has high randomness, such as agent and bot car initial locations and bot car lane changing, we only report average results across three runs but omit the error bars. One can observe that REPAINT outperforms other baselines regarding the training time needed for some certain performance level and the asymptotic performance. Moreover, another target task with a progress- -based reward is also investigated, where the agent is only rewarded based on its completion progress, but gets large penalty when it goes off-track or crashes with bot cars. The evaluation performance is provided in Figure 4b). When the target task is complex and the reward is simple in this case, it is sometimes difficult for the agent to learn a good policy as it lacks guidance from the reward on its actions. However, a teacher policy can provide guidance for the agent achieve high accumulated rewards the early stage of training. From the figure, we can again see that training with REPAINT not only reduces the convergence time, but also largely improves the asymptotic performance compared other models.
8

---

Under review as a conference paper at ICLR 2021
6 SUMMARY AND FUTURE WORK
In this work, we have proposed a knowledge transfer algorithm for deep actor-critic reinforcement learning. The REPAINT algorithm performs representation transfer and instance transfer alternately, and uses simple yet effective method to supplement the kickstarted training, i.e., the advantage-bant based experience replay. The experiments across several tasks with continuous and discrete state-action spaces have demonstrated that REPAINT significantly reduces the training time needed by an agent to reach specified performance level, and also improves the asymptotic performance of the agent, when compared to training with no prior knowledge and only kickstarting or instance transfer.
We provide summary of the transfer performance on the experiments we have conducted. which is given in Table 2 The teacher type indicates whether the teacher task is a sub-task of or similar to the target task, based on the cosine similarity in equation 4.4 The target score is the best return that can be obtained by training from scratch. Then we provide the number of training iterations needed by each model to achieve the target score. The models include training with baseline, kickstarting (KS), instance transfer (IT), and REPAINT (REP). In Section C.5
we also provide the data of wall-clock time. Finally, in the last column, we present the best scores that each knowledge transfer model can achieve in the evaluation. Unlike other baseline algorithms,, the superior performance of REPAINT is observed regardless of the task similarity. In contrast, for example, the kickstarted training provides 71% and 58% reduction The difference of performance is even larger on DeepRacer multi-car racing when different reward functions are used in the target task. For the DeepRacer single-car time trial, where the teacher task is significantly different from the target task, although the improvement of REPAINT is not as notable as in other tasks, it still outperforms the baseline algorithms. In this case, the kickstarting and instance transfer models are not able to reach the performance level of baseline upon convergence.
MuloCo-Reacher when the teacher task is similar versus different.
In future work, we aim to study how our proposed algorithm can automatically learn the task similarity if unknown, and spontancously determine the best 3g and Ç values in the training based on the similarity. Our preliminary results in Section C.3 indicate that when the task similarity is low, larger 3x values may reduce the asymptotic performance of the agent. Moreover, we are also interested in the dependency transfer performance on the neural network architectures. We provide some preliminary experimental results in Section C.4
REFERENCES
Haitham Bou Ammar, Eric Eaton, Matthew E Taylor, Decebal Constantin Mocanu, Kurt Driessens, Gerhard Weiss, and Karl Tuyls. An automated measure of mdp similarity for transfer in reinforce- ment learning. In Workshops at the Ehenty-Eighth AAAI Conference on Artificial Intelligence, 2014.
Bharathan Balaji, Sunil Mallya, Sahika Gene, Saurabh Gupta, Leo Dirac, Vineet Khare, Gourav Roy, Tao Sun, Yunzhe Tao, Brian Townsend, et al. Deepracer: Educational autonomous racing platform for experimentation with sim2real reinforcement learning. arXiv preprint arXiv:1911.00 1.05562, 2019.
André Barreto, Will Dabney, Rémi Munos. Jonathan J Hunt, Tom Schaul, Hado P van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. In Advances in neural information processing systems, pp. 4055-4065, 2017.
André Barreto, Diana Borsa, John Quan, Tom Schaul, David Silver, Matteo Hessel, Daniel Mankowitz, Augustin Zidek, and Remi Munos. Transfer in deep reinforcement learning using successor features and generalised policy improvement. arXiv preprint 1rXiv:1901.1996 2019.
Diana Borsa, André Barreto, John Quan, Daniel Mankowitz, Rémi Munos, Hado van Hasselt, David Silver, and Tom Schaul. Universal successor features approximators. arXiv preprint 11X1v:181. 2018
James L Carroll and Kevin Seppi. Task similarity measures for transfer in reinforcement learning task libraries. In Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005., volume 2, pp. 803-808. IEEE, 2005.
Thomas Degris, Martha White, and Richard S. Sutton. Off-policy actor-critic. CoRR, abs/ 1205.4839, 2012. TtIl:ttt::///
org /tgg/////////// //05 1200. 4833

---

Under review as a conference paper at ICLR 2021
Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, llya Sutskever, and Picter Abbeel. RI Fast reinforcement learning via slow reinforcement learning, arXiv preprint arXiv:1611.0277, 2016.
Norm Ferns, Prakash Panangaden, and Doina Precup. Metrics for finite markov decision processes. In UAI, volume pp. 162-1633 169, 2004.
Ian Goodfellow. Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Abhishek Gupta, Coline Devin, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Learning invariant feature spaces to transfer skills with reinforcement learning. arXiv preprint arXiv: urXiv:1703.02949) 2017.
Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, lain Dunning, Karen Simonyan, et al. Population based training of neural networks. arXiv preprint 0rXiv:1711.09846 2017.
Leslie Pack Kaclbling, Michael L Littman, and Andrew W Moore. Reinforcement learning: A survey. Journal of artificial intelligence research, 4:237-285, 1996.
Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning roboties: A survey. The International Journal of Robotics Research,
1274, 2013.
Vijay R Konda and John N Tsitsiklis. Actor-critic algorithms. In Advances in neural information processing systems, pp. 1014-1014, 2000.
George Konidaris, Ilya Scheidwasser, and Andrew Barto. Transfer in reinforcement learning via shared features. Journal of Machine Learning Research, 13(May):1333-1371, 2012.
Alessandro Lazaric. Transfer in reinforcement learning: a framework and a survey. In Reinforcement Learning, pp. 143-173. Springer, 2012.
Alessandro Lazaric, Marcello Restelli, and Andrea Bonarini. Transfer of samples in batch rein- forcement learning. In Proceedings of the 25th international conference on Machine learning, pp. 544-551, 2008
Chen Ma, Junfeng Wen, and Yoshua Bengio. Universal successor representations for transfer reinforcement learning. arXiv preprint arXiv:1804.0337 2018.
Tamas Madarasz and Tim Behrens. Better transfer learning with inferred successor maps. In Advances in Neural Information Processing Systems, pp. 9026-9037, 2019.
Neville Mehta, Sriraam Natarajan, Prasad Tadepalli, and Alan Fern. Transfer variable-reward hierarchical reinforcement learning. Machine Learning, 73(3):289, 2008.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference machine learning, pp. 1928-1937, 2016.
OpenAI, Marcin Andrychowicz, Bowen Baker, Maciek Chociej. Rafal J6zefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, Jonas Schneider, Szy- mon Sidor, Josh Tobin, Peter Welinder, Lilian Weng, and Wojciech Zaremba. Learning dexterous in-hand manipulation. CoRR, 2018. htt::/ httptttpp//aiiiii.ot/ wwwp:/wwwwwg/////....////// 00177
OpenAI, Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemystaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota with large scale deep reinforcement learning. 2019. URL Stcctss://axiiv.oot org abs / 912. 06680
Sinno Jialin Pan and Qiang Yang. A survey transfer learning. IEEE Transactions on knowledge and data engineering, 22(10)1335511
2009.
Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Actor-mimic: Deep multitask and transfer reinforcement learning. arXiv preprint arXiv: 15110/33/1.. 2015.
10

---

Under review as a conference paper at ICLR 2021
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Edvantage-weightee regression: Simple and scalable ooflpo reinforcement learning. arXiv preprint 1rX00:1910.00117,, 2019.
Caitlin Phillips. Knowledge transfer in markov decision processes. Technical report, Technical report, McGill University, School of Computer Science, 2006. URL.... 2006.
Janarthanan Rajendran, Aravind S Lakshminarayanan, Mitesh M Khapra, P Prasanna, and Balaraman Ravindran. Attend, adapt and transfer: Attentive deep architecture for adaptive transfer from multiple sources in the same domain. arXiv preprint arXiv:1510.0287. 2015.
Andrei A. Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirk- patrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy distillation, 2015.
Ahmad EL Sallab, Mohammed Abdou, Etienne Perot, and Senthil Yogamani. Deep reinforcement learning framework for autonomous driving. Electronic Imaging, 2017(19):70-76, 2017.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In International Conference on Machine Learning, pp. 1312-1320, 2015a.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint 1tt.:::55/1.00000000 2015b.
Simon Schmitt, Jonathan Hudson, Augustin Zidek, Simon Osindero, Carl Doersch, Wojeiech M Czarnecki, Joel Z Leibo, Heinrich Kuttler, Andrew Zisserman, Karen Simonyan, et al. Kickstarting deep reinforcement learning. arXiv preprint 1803.0800 2018.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897, 2015a.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint 2015b.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Noah Y Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, and Martin Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. arXiv preprint arXiv:2002.08396 2020.
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484, 2016.
Jinhua Song, Yang Gao, Hao Wang, and Bo An. Measuring the distance between finite markov decision processes. Proceedings of the 2016 international conference on autonomous agents & multiagent systems, pp. 468-476, 2016.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Erik Talvitie and Satinder Singh. An experts algorithm for transfer learning. In IJCAI, pp- 1065-1070, 2007.
Matthew Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research, 1033 2009.
Matthew E Taylor, Nicholas K Jong, and Peter Stone. Transferring instances for model-based reinforcement learning. In Joint European conference on machine learning and knowledge discovery in databases, pp. 488-505. Springer, 2008a.
Matthew E Taylor, Gregory Kuhlmann, and Peter Stone. Autonomous transfer for reinforcement learning. In AAMAS (1), pp. 283-290. Citeseer, 2008b

---

Under review as a conference paper at ICLR 2021
Andrea Tirinzoni, Andrea Sessa, Matteo Pirotta, and Marcello Restelli. Importance weighted transfer of samples reinforcement learning. arXiv preprint arXiv: 10886, 2018.
Emo Todorov. Mujoco: Modeling, simulation and visualization of multi-joint dynamics with contact.
Seattle WA: Roboti Publishing, 2016.
Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg. Scott
Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, et al. Critic regularized regression. Advances in Neural Information Processing Systems, 33, 2020.
Haiyan Yin and Sinno Jialin Pan. Knowledge transfer for deep reinforcement learning with hierarchi- cal experience replay. In Thirty-First AAAI Conference Artificial Intelligence, 2017.
Amy Zhang, Harsh Satija, and Joelle Pineau. Decoupling dynamics and reward for transfer learning. arXiv preprint arXiv:1804. 1804.9099.18,, 2018.
Shangtong Zhang, Wendelin Boehmer, and Shimon Whiteson. Generalized off-policy actor-critic. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché-Bue, Fox, and R. Garnett (eds.), Advances in Neurul Information Processing Systems 32, pp. 2001-2011. Curran Associates, Inc., 2019. URL. 8474 of -tt.ooooooooooooooooooo.oo.o... ceroorccrtiicppo. pdf
nips. cc/paper/
12

---

Under review as a conference paper at ICLR 2021
A ALGORITHM
In order to directly compare with the baseline algorithms regarding the reduction of number of training iterations, we adopt representation transfer and instance transfer alternately in Algorithm [ so that the REPAINT performs policy update one time per iteration. Indeed, Algorithm | can be easily extended with different alternating ratios other than 1:1 alternating. The corresponding results and discussion can be found in Section C.1 More generically, we can integrate the two transfer steps one iteration, which is shown the following integrated REPAINT algorithm with Clipped PPO.
Algorithm 2 Integrated REPAINT algorithm with Clipped PPO
for iteration & = 1,2, do Collect samples S {(s, a, 8'}} by (s), and {(s, a, s')} } by Teacher Compute for each transition using current reward function and add to S and S', respectively Fit state value network Vi using only S to update V/ Compute advantage estimates AI...., Ar for S and A,,..., Anp for S' by equation 3.4 for t=l..... . do if A; < then
V.....,Ay.
Remove A', and the corresponding transition (st, 0r 8 r'y) from S'
Compute sample gradient of T (?) defined in equation 4.2 using
Compute sample gradient of (lli(() defined in equation 3.5 using S'
Update policy network by 0 + 0 + a1 V L + 0r V cip
Note that in Algorithm 2 representation transfer and instance transfer, respectively.
different learning rates 0r and 02 control the update from
So far, have only used single teacher policy in the knowledge transfer, and adopt the objective function from only Clipped PPO algorithm. However, it is straightforward to using multiple teacher policies in each transfer step, and our algorithm can be directly applied any advantage-based policy gradient RL algorithms. Assume there are m previously trained teacher policies ..,... Tm- In the instance transfer, we can form the replay buffer S' by collecting samples from all teacher policies. Then in the representation transfer, the objective function can be written in a more general way:
LiL (0) Lelip (0)
Σ 3 H (a.(als) (T(((s))
(A.1)
where we can impose different weighting parameters for different teacher policies.
addition, the first term equation A. i.e., (O), can be replaced by the objective of other RL algorithms, e.g., A2C (classical gradient policy):
LAzc(0) Be [log (loss)
(A.2)
and TRPO (Schulman et al. 2015at:
[T Te (a(s) A, |KL ( (-18), To (ss)
TRRPO (0) E,
(A.3)
Toaa (a|s)
for coefficient, 3 of the maximum KI. divergence computed over states.
B EXPERIMENTAL SETUP
B.1 ENVIRONMENTS
We now provide the details of our experimental setup. MuJoCo is well-known physics simulator for evaluating agents on continuous motor control tasks with contact dynamics. In AWS DeepRacer simulator the RL agent, i.e., an autonomous car, learns to drive by interacting with its environment,
rein oeterrrccetttttteaaaaaeoooooooor
13

---

Under review as a conference paper at ICLR 2021
e.g., the track with moving bot cars, by taking action in given state to maximize the expected reward. Figures [ and d) present two environmental settings investigated in this work, i.e., single-car time-trial race, where the goal is finish lap in the shortest time, and racing against moving bot cars, where four bot cars are generated randomly on the track and the RI. agent learns to finish the lap with overtaking bot cars.
In single-car racing, we only install a front-facing camera on the RL agent, which obtains an RGB image with size 120 160 The image is then transformed to gray scale and fed into input embedder. For simplicity, the input embedder is set to be a three-layer convolutional neural network (CNN) (Goodfellow et al., 2016). For the RL agent in racing against bot cars, we use stereo camera and Lidar as sensors. The stereo camera obtains two images simultancously transformed to gray scale, and concatenates the two images as the input, which leads to a 120 x 160 input tensor. The input embedder for stereo camera is also three-layer CNN by default. The stereo camera is used to detect bot cars in the front of learner car, while the Lidar is used to detect any car behind. The backward-faccin Lidar has angle range of 300 degree and 64 dimensional signal. Each laser can detect distance from 12cm to meter. The input embedder for Lidar sensor is set to be a two-layer dense network. In both environments, the output has two heads, V head for state value function output and policy head for the policy function output, each which is set to be a two-layer dense networks but with different output dimensions. The action space consists of combination of five different steering angles and two different throttle degrees, which forms a 10-action discrete space. In the evaluation of DeepRacer experiments, the generalization around nearby states and actions is also considered (Balaji et al. 2019), where we add small noises to the observations and actions.
B.2 HYPERPARAMETERS
We have implemented our algorithms based on Intel Coact The MuloCo environments are from OpenAl GynE If not specified explicitly in the paper, we always use Adam as the optimizer with a learning rate x 10-4, minibatch size as 64, clipping parameter e as 0.2, 30 = 0.2 and 8k+1 0.958 throughout the experiments. The other hyperparameters are presented below.
Table 3: Hyperparameters used in the MuJoCo simulations.
Hyperparameter
Value
Num. of rollout steps
2048
Num. training epochs
10
Discount (?)
0.99
GAE parameter (A)
0.95
Beta entropy
0.0001
Reacher Advantage Threshold (()
0.8
Reacher Num. REPAINT iterations
15
Ant - Num. REPAINT iterations
50
Table Hyperparameters used in the DeepRacer simulations.
Hyperparameter
Value
Num. of rollout episodes
20
Num. of rollout episodes when using Teacher
2
Num. training epochs
8
Discount (7)
0.999
GAE parameter (A)
0.95
Beta entropy
0.001
Advantage Threshold (C)
0.2
Single-car Num. REPAINT iterations
4
Multi-car Num. REPAINT iterations
20
https://gym. www//evs//mmjjeeee /tm/a////oooo///eee/////oooo
14

---

Under review as a conference paper at ICLR 2021
C EXTENSIVE EXPERIMENTAL RESULTS
C.I. MORE RESULTS ON MUJOCO-REACHES
Alternating ratios. In Algorithm 1 we alternate representation transfer and instance transfer after each iteration. Here, we aim to illustrate the effect of using different alternating ratios by the MuJoCo-Reacher environment. We compare the 1:1 alternating with 2:1 ratio, namely, two representation transfer (kickstarting) iterations before and after an instance transfer iteration. The evaluation performance shown Figure 5] When the teacher task similar to the target task, adopting more kickstarted training iterations leads to faster convergence, due to the policy distillation term in the loss function. On the other hand, when the task similarity low, instance transfer contributes more to the knowledge transfer due to the advantage-based experience replay. Therefore, we suggest to set the alternating ratio in Algorithm [I] or the aj and 0g parameters in Algorithm 2 according to the task similarity between source and target tasks. However, the task similarity is usually unknown in most of the real-world applications, or the similarities are mixed when using multiple teacher policies. It is interesting to automatically learn the task similarity and determine the best ratio/parameters before actually starting the transfer learning. We leave the investigation of this topic as future work.
reward
mean reward
-8
-8
-10-
-10
-12
-12
2:1
-14-
2:1
100
150
200
50
100
150
200
evaluation iteration
evaluation iteration
Figure 5: Evaluation performance for MuloCo-Reacher averaged across five runs. Left: Teacher task is similar to the target task. Right: Teacher task is different from the target task.
REPAINT Relevance-Basee Transfer Learning. In order showcase contribution of the advantage-based experience replay in the instance transfer, it is also of interest to compare our proposed instance transfer algorithm with other existing methods. In this experiment, the kickstarting with the selevance-based instance transfer (Lazaric et al., 2008). and compare its performance with REPAINT in the MuJoCo-Reacher environment. In the relevance-based transfer (RBT), the agent first collects samples from both source and target tasks. For each sample from the source task, RBT computes relevance to the target samples. Then the source samples are prioritized by the relevance and RBT transfers the samples according to it. Figure O shows the evaluation performance RBT and REPAINT on MuJoCo- Reacher with either a similar source task or different source task. When similar task is used in knowledge transfer, as most of other transfer learning methods, RBT works well. However, when the source task different from the target task, although RBT attempts to transfer the most relevant samples, it has performance gain over the bascline training. The performance of REPAINT is significantly better than RBT this case.
incorporate
-5.0
misan reward
-75
-10.0
-10.0
15.0
Bassline
+15.
Basolinc
-1..5
RBT
RST
REPNNN
REPANTT
20.0
-20.0
25
50
75
300
125
150
175
260
evaluation iteration
evaluation iteration
Figure Comparison of relevance-basee based transfer (RBT) and REPAINT on MuJoCo-Reacher, averaged across five runs. Left: Teacher task is similar to the target task. Right: Teacher task is different from the target task.
15

---

Under review as a conference paper at ICLR 2021
C.2 REPAINT WARM-START
We also want to compare our hybrid representation-instaaee transfer algorithm with a widely-used parameler transfer approach, i.e., warm-start. In warm-start, the agent initializes with parameters from the teacher policy and conducts the RL algorithm after that. When the target task is similar to the teacher task, it usually works well. But here we compare the two algorithms the DeepRacer single- car experiment, where the two tasks are significantly different. Figure 7 visualizes the trajectories of the agent on the track during evaluations. Each model is trained for two hours and evaluated for another 20 episodes. From both cases, we can see that although the two reward functions encode totally different behaviors, REPAINT can still focus on current task while learning from the teacher policy. This again indicates the effectiveness of the advantage-bass based experience replay in the instance transfer. comparison, training with warm start cannot get rid of the unexpected behavior at convergence due to the reason that it may be stuck some local optimum. Therefore, initialization with previously trained policies can sometimes jump-start the training with good initial performance, but the method contributes to the final performance only when two tasks are highly related.
(a) Outer-lane task: REPAINT vs. warm-start
(b) Inner-lane task: REPAINT vs. warm start
Figure 7: Trajectories of policy evaluations. In each of (a) and (b), evaluation of the models trained from REPAINT is visualized on the left and that trained with warm-start on the right.
C.3 MORE RESULTS ON DEEPRACER SINGLE-CAR CAR TIME TRIAL
the DeepRacer single-car time-trial task, we also study the effect of different cross-entropy weights 3& and instance filtering thresholds Ç, mentioned in the paper. We first present the results of instance transfer learning with different Ç values in Figure g where we can see that our proposed udvantage-based experience replay is robust to the threshold parameter.
400-
mean reward
mean reward
300
300
200
200
Baseline
Baseline
100
100
C-0.2
C=0.2
2=0.4
==0.4
15
6
5
evaluation iteration
evaluation iteration
(a) Outer-lane task with inner-lane teacher
(b) Inner-lane task with outer-lane teacher
Figure 8: Evaluation performance with respect to different Ç's, averaged across five runs.
We then study the performance of training with different cross-entropy loss weights Sq. First. fix the diminishing factor to be 0.95, namely, Bx+1 = 0.953,, and test different B 's. From Figure 9 we can see that training with all Bo values can improve the initial performance compared to baseline. However, when the teacher task is different from the target task, larger fo values, like 0.3, may reduce the agent's s asymptotic performance since the agent overshoots learning from teacher policy. In addition, we then fix 3n 0.2 and test different 8x schedules. The results are shown in Figure 10 We can observe some trade-offs between training convergence time and final performance. By reducing the values faster, that needed to achieve some certain performance level. of interest to automatically determine the best B3 values during training, which needs further investigation. We leave it another future work.
improve the final performance but increase the training time
16

---

Under review as a conference paper at ICLR 2021
400
mean reward
300
mean reward
200
Baseline
Baseline
0-1
100
0,1
100
A-=02
0.2
8o-0.3
5
15
evaluation iteration
evaluation iteration
(a) Outer-lane task with inner-lane teacher
(b) Inner-lane task with outer-lanc teacher
Figure 9: Evaluation performance with respect to different initial Bo ss averaged across five runs.
Here we fix the B update be 5x+1 0.95.
400
300
mean reward
300
mean reward
200
200
Baseline
0.95
100
f 0,
100
8 0.96,
8. 0. 96.
B. 0.86,
8. 0.80.
5
10
15
0
5
10
15
evaluation iteration
evaluation iteration
(a) Outer-lanc task with inner-lane teacher
(b) Inner-lane task with outer-lane teacher
Figure 10: Evaluation performance with respect different 3 schedules, averaged across five runs.
C.4 NEURAL NETWORK ARCHITECTURES
For completeness of the experiments, we also provide some results regarding different neural network architectures in this section. Take the DeepRacer task of multi-car racing against bot cars example, we have used three-layer CNN as the default architecture experiments. Here, we present the comparison of REPAINT against other baselines with the evaluation performance using four-layer CNN (Figure 11 and five-layer CNN (Figure
1000
meaan imnggeed 195)
meen ress 195)
mean reward
800
60
mean reward
600
30
400
200
20
Nooteting
Bulnnca
Mooaning
rellaee vandl
Gdlitaddig
60
80
20
40
60
80
20
40
50
60 evaluation Iteration
evalustion iteratioo
evoluation iteration
evaluation ther attiooo
(a) Task with advanced reward
(b) Task with progress-hased reward
Figure 11: Evaluation performance for DeepRacer multi-car racing against bot cars, using 4-layer CNN. The value at each iteration is smoothed by the mean value of nearest 3 iterations.
C.5 SUMMARY OF WALL-CLOCK TRAINING TIME
In addition to the summary of reduction performance with respect to number of training iterations we also provide the data of wall-clock time in Table
presented Table 2 significant reduction by training with REPAINT, which reaches at least 60% besides the DeepRacer single-car time trial. The kickstarted training performs well when similar teacher policy is used. Although training with only instance transfer cannot boost the initial performance, it still reduces the training cost to achieve some specified performance level.
Again, we can see
17

---

Under review as a conference paper at ICLR 2021
toou Srggre sas (96)
800
Lannn roward Srrun
10
400
Kocoeningg ECTANT
bocin coeetring
Sastine dckita 1ig
20
Bowithn RANHEatTR KaPnIN
200
SENGNT
20
20 evaluation Deration
evaluation Reeaaion
evaluation iteration
evaluation lberatiion
(a) Task with advanced reward
(b) Task with progress- based reward
Figure 12: Evaluation performance for DeepRacer multi-car racing against bot cars, using 5-layer CNN. The value each iteration smoothed by the mean value of nearest iterations.
Table 5: Summary of wall-clock time of experiments.
Training hardware
Teacher type
Target score
TRaseline (hrs)
Tks (pct. reduced)
Tir (pet. reduced)
TREP (pet. reduced)
Env.
similar different
0.6 (71.4%)
1.1 (47.6%
0.4 (11.%)
Reacher
laptop
-7.4
0.9 (37.1%)
1.4 (33.3%)
0.6 (71.4%)
Ant
laptop
similar
3685
19.1
8.0 (58.1%)
12.8 (33.0%)
7.5 (60.7%)
AWS, p2
different
394
2.2
1.5 (31.8%)
Single-car
AWS, p2
different
345
2.3
1.5 (34.8%)
AWS, p2
sub-task
1481
16.4
4.8 (70.7%)
12.6 (23.2%)
4.5 (72.6%)
Multi-car
AWS, p2
sub-task
2.7
9.6
9.3 (3.1%)
8.3 (13.5%)
3.7 (61.5%)
18


Reviews:
[{"review_id": "lAGJrDcGV6S", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper deals with transfer learning in RL. The problem broadly defined is to improve performance of an agent on a new task, given an agent (teacher) trained on a previous (different) task. There are multiple approaches to this from fine-tuning, distillation to instance transfer. A central question in this field is how to transfer learned useful behavior even when task similarity is low. While most methods would work well on similar tasks, naively applying, for example distillation, would hurt when training on a very different target task. \n\nThis paper suggests an approach that builds on two approaches: kickstarting and instance transfer. Kickstarting can be viewed as distillation of a policy with a dynamically tuned coefficient that control how much to weigh the distillation loss against the actor-critic loss. Intuitively this distillation is more helpful at the start of training and the original work uses population based training to automatically anneal this. Applying this as-is to tasks with very different reward functions would lead to a degradation in final performance on the new task. \n\nThe second approach considered here is instance transfer. Briefly, we consider trajectories from the teacher and train the agent on those that are \"good\" as filtered by the advantage under the current reward function. \n\nThe results are encouraging, especially showing that performance is not lost when tasks are dissimilar.  The writing overall is very clear. Below are few specific notes/questions.\n\n* One suggestion if for greater discussion into why this works. It was not clear to me on reading what the fundamental rationale for combining the approaches in this way is, as opposed to say combining kickstarting with some other method like successor features. \n\n* How sensitive is the method to the schedule for kickstarting? Could a well hand-tuned kickstarting schedule alone achieve all of the performance gain if we knew a priori how similar the tasks were? \n\n* The paper mentions automatic identification of task similarity as future work -- do the authors have ideas on how best to do this? \n\nOverall I found this a well-written paper that combines two existing approaches to transfer learning in RL to provide a unified algorithm that can do well in settings with low or high task similarity.  I think the results are promising although given the size of the error bars on some of the plots it's not clear it is completely worth the effort to use this method in all cases. Addressing the main point above on why this is necessarily a principled approach as opposed to just an ensemble of methods would help motivate this. Nonetheless in its current form I would recommend an accept. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "dqJNvnyYz_j", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new transfer learning approach to leverage the previously learned knowledge stored in a pre-trained teacher policy to facilitate the learning in a new task. The proposed method combines an established technique termed kickstarted training with a simple experience filtering method. The kickstarted training approach augments a policy distillation loss to the actor-critic training loss. The experience filtering simply performs thresholding to filter out transitions generated by teacher policy which have reward below some threshold, where the reward for thresholding is taken from the target task. \n\nI lean slightly towards rejection because the novelty of the method is somewhat too limited and the empirical evaluation does not give much new insights to the problem of policy transfer. The only difference between the proposed method and kickstarted training is that the method uses more data, which are the transitions generated by the teacher policy and could receive high reward at the student task domain. Such an add-on does not give much privilege in terms of return or training time.\n\nOther detailed comments :\n- The tasks for MuJoCo-Reacher are too similar. The authors only use an alternative weight on the control reward to create a distinct task. But essentially these can hardly be considered as alternative tasks for policy transfer scenario.\n\n- I wonder how the weights for policy distillation loss are specified for the baseline method kickstarting. Do you use the same value as REPAINT for each evaluation domain?\n\n- In MuJoCo-Reacher-similar, REPAINT and Kickstarting achieves the same standard for mean reward, and training time for the two also seem quite close to each other.\n\n- In MuJoCo-Ant, the improvement of average return for REPAINT over Kickstarting also seem not that significant. I wonder why the results are presented in table rather than plots.\n\n- The proposed method presents a way to quantitatively evaluate the task similarity (Eq 4.4). I'm expecting to see some finer-grained usage of such a formulation, e.g., creating tasks with positive but decreasing levels of similarity and show empirical results. However, the tasks used for the actual evaluation only have vaguely characterized similarity values, which leaves a feeling that the formulation for similarity is not necessary to have.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Rgm_UCAatAW", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The proposed method relies on kickstarting, thus using policy distillation as an auxiliary loss for transferring from a source task to a target task, as a starting point. In addition, the authors add 'instance transfer', i.e., selecting some prioritized data from the source task to be used to train the target task. The combination of these two features produces fairly strong performance on various transfer experiments in four simulation environments. \n\nThe approach is clearly presented. The motivation is clear and related approaches are described. The experiments are described, although the descriptions and conclusions could be more precise. \n\nThe proposed approach has very little novelty, since kickstarting is already published and similar versions of instance transfer have been published. The instance transfer method, which is rather heuristic, requires tuning a threshold which may be a limitation. The results, moreover, do not convincingly show that putting these two features together makes a significant difference to performance. The improvements are minor for all the experiments when compared with the baseline or kickstarting.\n\nPros\n- Clearly described method\n- Well-designed experiments\n\nCons\n- Lack of novelty beyond the combination of 2 known methods\n- Results don't show that the method is substantially better than kickstarting alone", "extra_scores": {}, "extra_text": {}}, {"review_id": "I0GM-sP200R", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Below I list the strengths and weaknesses of the paper in my opinion. Overall I vote to accept the paper for now, but my final decision will depend on the authors' clarifications to my questions below.\n\nStrengths:\n - The algorithm is a simple combination of ideas each of which seems important according to the results.\n - Data efficiency and the ability to use previous knowledge to inform solutions to new tasks is an important problem in RL and I encourage the authors to continue to explore this space. \n - The appendix includes quite a good overview of experimental details and since the work is based on open source implementations it is potentially reproducible.\n - The appendix also includes a number of ablation studies including results on end-to-end wall clock time and additional results. As I describe below though, more can be done in terms of experimental rigor and clarification is needed for some of the presented results.\n\nQuestions/Clarifying points:\n- I am concerned with the strange drops in performance in Figure 2. This instability also holds for the Baseline which is trained from scratch. This indicates it does not have to do with the transfer of the teacher policy where ostensibly the drop corresponds to critic training. Could the authors clarify what is happening here? I think it is important to have a good explanation for this.\n\n- It is mentioned that the Beta_k parameter used in kickstarting is annealed so that it vanishes at later iterations. Could the authors clarify if this schedule is also maintained for the pure ‘kickstarting’ baseline? If it is, I’m uncertain why this baseline performs asymptotically worse on the ‘dissimilar’ tasks. Shouldn’t the influence of the teacher slowly disappear anyway?\n\n- Figure 2b) indicates that using the top 20% metric for advantages works as well (if not better) than the one used in REPAINT. Since this metric also has the property of being scale invariant, would it not be better to use this instead? \n\n- The authors do not really compare against any existing methods for transfer learning. For instance, it would be interesting to see how the proposed method compares to say DAGGER.\n\n - I’m not certain the specific task similarity measure described in the main text is a good one. For instance the ideas introduced by Carroll and Seppi and Lazaric (both referred to in the main text) attempt to quantify similarity in terms of how well an expert on the source task would perform in the target task. For instance Carroll and Seppi mention: ‘The task similarity measure should provide an approximation to the amount of learning improvement that we would get when using the source task to learn the target task under a given transfer technique’. In light of the current work, the ‘similarity’ under the proposed approach is 0 for the ‘dissimilar’ tasks. While kickstarting performs worse than REPAINT on these tasks, it still manages to do reasonably well. It would be interesting to note how dissimilar the tasks are under the other proposed metrics.\n\nApart from these, there are technical details which could help improve the work that did not affect my decision but list below:\n- The main text mentions that the clipped loss L_clip is used even for instance transfer in Algorithm 1. Strictly speaking the importance weight used for this should be the teacher policy. This may be worth mentioning somewhere although it is not too important.\n- For ease of parsing the experiments, it is often useful to have a consistent color schemes for the legend used across figures. Figure 2 and Figure 3 use the same palette but with different names attributed to each method. This isn’t a big issue but fixing it would make it easier to parse the figures and improve the overall presentation.\n- The related work could also include a discussion of recent work on offline reinforcement learning. Specifically recent methods proposed by Peng et al. , Siegel et al. and Wang et al. show advantages of using different forms of the advantage function when learning from offline data. This is similar to the instance transfer prescribed in the paper where the teacher generates the fixed distribution to learn from.\n\n\nReferences:\nSiegel, N., Springenberg, J. T., Berkenkamp, F., Abdolmaleki, A., Neunert, M., Lampe, T., Hafner, R., Heess, N., and Riedmiller, M. Keep doing what worked: Behavior modelling priors for offline reinforcement learning. ICLR, 2020\n\nPeng, X. B., Kumar, A., Zhang, G., and Levine, S. (2019). Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177.\n\nZ. Wang, A. Novikov, K. Żołna, J. T. Springenberg, S. Reed, B. Shahriari, N. Siegel, J. Merel, C. Gulcehre, N. Heess, and N. de Freitas. Critic regularized regression. arXiv preprint arXiv:2006.15134, 2020.\n\nS. Ross, G. Gordon, and A. Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. Journal of Machine Learning Research, 15:627– 635, 2011.\n\n\n--- Edit: After rebuttal ---\nI thank the authors' for their detailed response to my questions and for clarifying so many aspects of the work! I also appreciate the time taken to make the minor corrections that help with the presentation. \n\nMy main concerns have been sufficiently addressed - specifically the references to the dips in performance have eased my concerns with the validity of the experiments. The ablation with the beta parameter is also quite important I think and at least helps clarify that the issue for kickstarting may indeed be one of reaching a local optima.  Given the substantial effort (including implementing a new baseline) that went into the rebuttal I am willing to increase my score and recommend the paper be accepted at this venue.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The is a borderline paper with the reviewers split in their recommendations.  The decision is therefore not easy.

The work is promising, but a key concern is that the contribution appears incremental: the paper proposes to alternate between kickstarting, which is itself not entirely new as an idea, with a simple instance transfer heuristic.  The resulting method is straightforward, which can be considered a strength, but there is no serious technical justification beyond intuitive motivation.  Rather than present technical analysis, the paper focuses more on intuitively delivering the proposal then evaluating it.  This would be acceptable if the empirical outcomes were undeniably impressive, but the outcomes, though positive, are not overwhelming.  The experimental evaluation is limited in scope, considering only the simplest MuJoCo environments and a benchmark racing simulator.

The authors responded to some of the criticisms forcefully, and were comprehensive in their rebuttal, but if the support for the proposed method is to be entirely intuitive and empirical, one would have expected a more comprehensive evaluation where transfer was used to solve more impressively difficult problems.  Overall, I think this work would be better served by adding a technical analysis that validates the significance of the instance transfer heuristic, combined with a broader empirical study that tackles more challenging domains.

Decision: Reject