Beyond Fine-Tuning: Transferring Behavior in Reinforcement Learning
Authors: Víctor Campos
Pablo Sprechmann
Steven Stenberg Hansen
Andre Barreto
Steven Kapturowski
Alex Vitvitskyi
Adria Puigdomenech Badia
Charles Blundell
Venue: NeurIPS 2021 Submitted (2021)
Keywords: deep reinforcement learning
transfer learning
unsupervised learning
exploration

Abstract: Designing agents that acquire knowledge autonomously and use it to solve new tasks efficiently is an important challenge in reinforcement learning. Knowledge acquired during an unsupervised pre-training phase is often transferred by fine-tuning neural network weights once rewards are exposed, as is common practice in supervised domains. Given the nature of the reinforcement learning problem, we argue that standard fine-tuning strategies alone are not enough for efficient transfer in challenging domains. We introduce Behavior Transfer (BT), a technique that leverages pre-trained policies for exploration and that is complementary to transferring neural network weights. Our experiments show that, when combined with large-scale pre-training in the absence of rewards, existing intrinsic motivation objectives can lead to the emergence of complex behaviors. These pre-trained policies can then be leveraged by BT to discover better solutions than without pre-training, and combining BT with standard fine-tuning strategies results in additional benefits. The largest gains are generally observed in domains requiring structured exploration, including settings where the behavior of the pre-trained policies is misaligned with the downstream task.

20 is scarce [60] 16 61 22 [5]. This builds on the intuition that the pre-trained model will map Beyond Fine-Tuning: Transferring Behavior in Reinforcement Learning
Anonymous Author(s)
Affiliation
Address
email
Abstract
Designing agents that acquire knowledge autonomously and use it to solve new tasks efficiently is an important challenge in reinforcement learning. Knowledge acquired during unsupervised pre-training phase is often transferred by fine- tuning neural network weights once rewards are exposed, as is common practice in supervised domains. Given the nature of the reinforcement learning problem, we argue that standard fine-tuning strategies alone are not enough for efficient transfer in challenging domains. We introduce Behavior Transfer (BT), technique that leverages pre-trained policies for exploration and that is complementary to transferring neural network weights. Our experiments show that, when combined with large-scale pre-training in the absence of rewards, existing intrinsic motivation objectives can lead the emergence of complex behaviors. These pre-trained policies can then be leveraged by BT to discover better solutions than without pre-trainin trainingg and combining BT with standard fine-tuning strategies results in additional benefits. The largest gains are generally observed in domains requiring structured exploration, including settings where the behavior of the pre-trained policies is misaligned with the downstream task.
2
4
8
10
11
12
13
14
15
Introduction
18 Transfer in deep learning is often performed through parameter initialization followed by fine-tuning,
19 a technique that allows leverage the power of deep networks in domains where labelled data
21 inputs to a feature space where the downstream task is easy to perform. When combined with methods that can leverage massive amounts of unlabelled data for pre-training, this transfer strategy 30 and natural language 24 processing [15 50). The success of these approaches has led to an ever- growing interest in developing
23 has led to unprecedented results in domains like computer vision
25
techniques for pre-training large scale models on unlabelled data 9 24).
28 In the reinforcement learning (RL) context, unsupervised methods that learn in the absence of reward have also garnered much research attention 23 training are typically evaluated by their ability to enable efficient transfer to previously unseen reward
27
29). The benefits of unsupervised pre-
28
29 functions [28]. spite of their different approaches unsupervised RL, most of the top-performing methods in this setting transfer knowledge through neural network weights. Such approaches deal
30
31 with the data inefficiency associated to training neural networks with gradient descent, similarly to what is done in supervised learning, e.g. by pre-training encoders that extract representations from
33 observations [59]. However, RI. introduces a challenge that is not present in supervised learning: the agent responsible for collecting the right data to learn from. This introduces second source of inefficiency from which transfer approaches can also suffer if they rely on unstructured exploration strategies after pre-training, these lead to exponentially larger data requirements in complex
35
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

Montezuma's Revenge (sparse rewards)
Defender (dense rewards)
12.5k
Without transfer
Roou (reward-free)
126
From scratch
300
56
200
With transfer
2.5k
300
Fine-tuning Buau
BT(nkau)
500
155
28
520
158
28
BT(maau)) Ragu Init
Environment frames
Environment frames
Figure 1: Comparison of transfer strategies on Montezuma's 's Revenge and Defender after pre-training a policy with NGU [48] in the absence of reward. The benefits of our proposed approach to leverage pre-trained behavior for exploration, Behavior Transfer (BT), are complementary the gains provided by pre-trained weight initialization followed by fine-tuning.
37 downstream environments 45 44
To address this problem, one could consider fine-tuning policies
38 that produce meaningful behavior
52 but this approach quickly disregards the pre-trained behavior when learning in the downstream task due to catastrophic forgetting.
43
39
40
In this work, we explicitly separate the transfer of behaviour and weights. We propose to make use of the pre-trained behaviour itself (i.e., the pre-trained policy mapping from observations to actions) in contrast to pre-trained neural network weights for further fine-tuning. While pre-trame pre-trained behavior has been used before for exploitation [56 2 3], our approach employs pre-trained policies to aid with exploration as well to collect experience that can be leveraged via off-policy learning. This strategy accelerates learning. as the agent is exposed to potentially useful experience earlier in training, without compromising the quality of the discovered solution when the pre-trained behavior is not aligned with the downstream task. We expose the pre-trained behaviour to the downstream agent in two ways: firstly, as an extra exploratory strategy that, when randomly activated, persists for number of steps, and secondly as an additional pseudo-action for the learned value function where the agent may elect to defer action selection to the pre-trained policy instead of choosing itself. We call this approach Behavior Transfer (BT).
41
42
43
44
45
46
47
48
50
51
52 Defining unsupervised RL objectives remains an open problem, and solutions are generally influenced by how the acquired knowledge will be used for solving downstream tasks. Instead of proposing yet another objective for unsupervised pre-training we turn to existing techniques for training policies in the absence of reward and make our choice based on two general requirements. First, the objective should scale gracefully with increased compute and data. This has been key for the success of self- superviue approaches in other domains [9 (35), and we argue that it is an important property for unsupervised RL as well. Second, the pre-training stage should return a policy that produces complex behavior that may be leveraged in subsequent transfer stage. The Never Give Up (NGU) [48] intrinsic reward meets both requirements, and our experiments show that large-scale pre-training with this objective leads to state of the art scores in the reward-free Atari benchmark.
53
54
55
56
57
58
60
82
Figure exxeplifies our main findings. We pre-train behaviour using the intrinsic NGU reward during a long unsupervised phase without rewards. This gives rise to exploratory behaviors that seek to visit many different states throughout an episode, and we then compare different strategies for leveraging the acquired knowledge once rewards are reinstated. While fine-tuning the pre-trained weights enables faster learning, the exploratory behavior of the pre-trained policy quickly disregarded as is exposed to rewards. On the other hand, Behavior Transfer (BT) does not modify the pre-trained the new task and is able to achieve higher end scores thanks to better
63
64
85
66
67
68
policy while learning exploration. These two strategies are not mutually exclusive, and BT also benefits from the faster convergence provided by initializing neural networks with pre-traine pre-trained weights when these encode useful information for solving the downstream task.
69
70
71
72 Our contributions can be summarized follows. (1) We propose Behavior Transfer (BT), a technique
73 that leverages pre-trained policies for exploration by treating them as black boxes that are not modified
74 during learning on the downstream task. BT uses the pre-trained policy to collect experience in two ways, namely randomly triggeedd temporally-extema
75
exploration and one-step calls based on 76 value estimates. (2) Our experiments show that large-scale unsupervised pre-training with existing intrinsic rewards can produce meaningful behavior, achieving state of the art results in the reward-free
78 Atari benchmark. These results suggest that scale key for unsupervised RI, akin to what has been
observed in supervised settings. (3) We provide extensive empirical evidence demonstrating the

---

80 benefits of leveraging pre-trained behavior via BT. Our approach obtains the largest gains in hard
81 exploration games, where it almost doubles the median human normalized score achieved by our strongest baseline. Furthermore, we show that BT is able to leverage a single task-agnostic policy
85 to solve multiple tasks in the same environment and to achieve high performance even when the
84 pre-tr policies are misaligned with the task being solved. (4) BT brings benefits to the table
85 that are complementary to those provided by reusing pre-trained neural network weights, and we
86
empirically show that combining these two strategies can result in larger gains.
2
Preliminaries
87
The interaction between the agent and the environment is modelled as a Markov Decission Pro- cess (MDP) 49 An MDP is defined by the tuple AS, P. do, R, 7) where S and A are the state 90 and action spaces, P(s'|s, s. a) is the probability of transitioning from state to after taking action a, ds(s) is the probability distribution over initial states, R: S x AA x S - I is the reward function, and 7 E 10 is the discount factor. The goal is to find policy π (aas) that maximizes the expected oc
86
89
91
92
93 return, G2 = Σ === 7 R,, where R, = r(S,, A., 1,+1). A principled way to address this problem
94 is to use methods that compute action-value functions, Q" (s. a) E. [Gr] St = 8, Ar = aj. where
95 E, []] denotes expectation over transitions induced by [49].
96
We consider a setting where the agent is allowed to first learn within an MDP without rewards, M = (S, A. P. do). for a long period of time. The knowledge acquired during the reward- stage is later leveraged when maximizing reward in new MDPs that share the same underlying dynamics but have different reward functions, M, = (S,A, d, R2 n)) Interactions between the agent and the environment are often assumed incur cost, but we will consider this cost to be relevant only for transitions with reward (28). Even if the cost of unsupervised pre-training becomes non-negligible. can be amortized when the acquired task-agnostic knowledge leveraged solve multiple tasks efficiently [15 9]. Indeed, we would expect this transfer setting to become more relevant as the community moves towards more complex environments, where one may want to train agents to maximize multiple reward functions under constant dynamics. In the limit, one could consider the real world: it has constant or slowly changing dynamics, and humans are able to leverage previously acquired skills to quickly master new tasks.
97
98
89
100
101
102
103
104
105
106
107
3 Behavior Transfer
108
109
Transfer in supervised domains often exploits the fact that related tasks might be solved using similar representations. This practice deals with the data inefficiency of training large neural networks with stochastic gradient descent. However, there is an additional source of data inefficiency when training RL agents: unstructured exploration. Fine-tuning a pre-trained exploratory policy arises as a potential strategy for overcoming this problem, as the agent will observe rich experience much carlier in training than when initializing the policy randomly, but this approach suffers from important limitations. Learning in the downstream task can lead to catastrophically forgetting the pre-trained policy, thus prematurely disregarding its exploratory behavior. Moreover, the same neural network architecture needs to be used for both the pre-trained and the downstream policies, which in practice also imposes a limitation on the type of RL methods that can be employed in the adaptation stage (for instance, if the pre-traie traitri policy was trained using a policy- based method, it might not be possible to fine-tune it using a value-based approach).
110
111
112
113
114
115
116
117
118
119
120
121
Let us assume that we have access pre-trained policy that exhibits exploratory behavior, and defer the discussion on how to train this policy to Section 4 Following such a policy might bring the agent to states that are unlikely to be visited with unstructured exploration techniques such as e-greedy (55]. This property has the potential of accelcrating learning even when the behavior of the pre-trained policy is not aligned with the downstream task, as will effectively shorten the path between otherwise distant states [1). Leveraging pre-trained policies for exploration differs from other approaches in the literature that use such policies directly for exploitation, e.g. via zero-shot transfer [19], methods that define higher-level policy that alternates between the given
122
123
124
125
126
127
128
129
policies 5 (56) or within the framework of generalized policy updates (4. Exploring with pre-trained
policies can accelerate convergence by providing useful experience to the agent, which is possible
131
even when the pre-training and downstream tasks are misaligned. However, strategies that directly
132
use the pre- pre-trained policies for exploitation may result in sub-optimal solutions in such scenario 22]
3

---

133
We propose to leverage the behavior of pre-trained policies during transfer to aid with exploration. An explicit distinction between behavior and representation is made by considering pre- pre-trained policies as black boxes that take observations and return actions. This strategy is agnostic to how the pre-trained behavior encoded and not restricted to learned policies. We rely on off-policy learning methods
134
135
136
137
during transfer to leverage the behavior of a pre-trained policy Tp mp(a]s), We keep Tp fixed during transfer, which prevents catastrophic forgetting of the original behavior when it is parameterized by a neural network (i.e., we instantiate and train new policy with its own set of parameters). We propose Behavior Transfer (BT), which leverages two complementary strategies to achieve this. Since BT is agnostic to the method used to pre-train policies, (TT))) refers to behavior being transferred from policy Tp. We formalize BT in the context of value-basee Q-learning agents, although similar derivations are in principle possible for alternative off off-policy learning methods. Pseudo-code for BT is provided in Algorithm [
138
140
142
144
145
Temporally-extended exploration. We draw inspiration from Lévy flights [57 a class of ecological models for animal foraging, where a fixed direction is followed for duration sampled from a heavy- distribution. This principle was implemented in the context of exploration in RL by ez-greedy [14,, which encodes the notion of direction in the environment via exploration options that repeat the same action throughout the entire flight. Since TTp is more likely to encode a meaningful notion of direction in complex environments than action repeats, we propose a variant of ez- re-greddy where TPp is used the exploration option. An exploratory flight might be started at any step with some probability. The duration for the flight is sampled from a heavy-tailes distribution (Zeta with in all experiments), and control is handed over to Tp during the complete flight. When not in flight, actions are sampled from the behavior policy obtained while maximizing the task reward (e.g. e-greedy derived from the estimated Q values).
146
148
150
152
154
155
156
Extra action. The previous approach switches to during experience collection blindly, and we now consider an alternative strategy for triggering these switches based value. This can be easily implemented through an extra action which samples an action from Tp. which also allows the agent to use the pre-trained policy at test time if deemed beneficial. More formally, this amounts to training policy over an expanded action set A = (a+), }. where at is resolved by sampling an action from Tp. a' Tp(s) (with a' E A). The additional action can be seen as an option that be initiated from any state and always terminates after a single step. Note that selecting the option will lead to
158
160
162
163
the same outcome if the agent had selected a' primitive action, and we take advantage of this observation by using the return of following the option as target to fit both Q(s, Tp (s)) and ((s, d'). Intuitively, this approach induces bias that favours actions selected by Tp, accelerating the collection of rewarding transitions when the pre-trained policy is somewhat aligned with the downstream task. Otherwise, the agent can learn ignore Tp as training progresses by selecting other actions.
184
165
186
167
Algorithm 1: Experience collection pseudo-code for BT
Input:
Action set, A; additional action, a+; extended action set, A+ = AU fa+ }: pre-trained policy, Tp: Q-value estimate for the current policy, Q" a) Va E A+; probability of taking exploratory action, e; probability of starting flight, Clev; flight length distribution, DNN)
while True do
n +0
// flight length
while episode not ended do
Observe state
if n 0 and random() S Clevy then n ~ D(N)
// sample flight length
if n > 0 then
n x n a ~ (s)
else
if random() then a Uniform(At) else <- arg maxx' 1Q=(s, if a = a. then a ~ (s)
end
Take action
end end

---

4 Reward-free pre-training
168
169
It is a common practice to derive objectives for proxy tasks in order drive learning in the absence of reward functions, and there exists a plethora of different approaches in the literature. Model-based approaches can learn world models from unsupervised interaction [26]. However, the diversity of the training data will impact the accuracy of the model [53] and deploying this type of approach in visually complex domains like Atari remains an open problem [27]. Unsupervised RL has also
170
171
172
173
174
been explored through the lens of empowerment [51 42], which studies agents that aim to discover intrinsic options 23 19. While these options can be leveraged by hierarchical agents [21] or integrated within the universal successor features framework [2] 3 8 28), their potential lack of coverage generally limits their applicability to complex downstream tasks [22 An alternative
175
178
177
178
objective is that of exploring the environment by finding policies that induce maximally entropic state distributions 29 (3), although this might become extremely inefficient in high-dimensional state
179
180
spaces without proper priors 40, [59]
181
Recall that our goal is to devise pre-training objective that can help reduce the amount of interaction needed by the agent to collect relevant experience when learning in a downstream task. We argue that such objective needs meet two requirements. First, as suggested by results in other domains ( 35
182
183
184
it should scale gracefully as the amount of compute and experience used for pre-training are increased. This contrasts with the training regimes used in most unsupervised RL approaches, which use a relatively small amount of experience (28 40 55 when compared to distributed agents that do make
185
186
187
use of rewards 33 36). Second, must encourage the emergence of complex behaviors such as navigation or manipulation skills. It has been argued that exploring the environment efficiently will serve as proxy for developing such behaviors (37] and exploration bonuses have been shown to
188
189
190
produce meaningful behavior in the absence of reward [46 [0]. However, many exploration bonuses
191
vanish over the course of training and thus may not be well-suited for long unsupervised pre-training phase. It can be shown that many intrinsic rewards aim at maximizing the entropy of all states visited during training, and so the final policy does not necessarily exhibit exploratory behavior [39].
192
193
194 We propose to use Never Give Up (NGU) 48 as a means for training exploratory policies in an unsupervised setting. The NGU intrinsic reward proposes a curiosity-driven approach for training persistent exploratory policies which combines per-episode and life-long novelty. The per-episode novelty, rt ,pisodic rapidly vanishes over the course of an episode, and it is designed to encourage self- avoiding trajectories. It is computed by comparing a representation of the current observation, ((s).. to those of all the observations visited in the current episode, M (f()) f(1)...... f()- where f S + Rº is an embedding function trained using a self-supervised inverse dynamics model [46). Such a mapping concentrates on the controllable aspects of the environment, ignoring
195
196
196
200
202
all the variability present in the observation that is not affected by the action taken by the agent. The life-long novelty, an slowly vanishes throughout training. and it is computed by using Random With this, the intrinsic reward r ,GU is defined as follows:
203
204
Network Distillation (RND)
ri
NGU
pisoodic min (max fai, L), with T: rt
episodic
(1)
V EVO.EEN. K(f(s)), f(x)) +
205
where I. is fixed maximum reward scaling, N₂ is the set containing the k-nearest neighbors of f(*)) in M, e is a constant and K : RR x R R+ is a kernel function satisfying K(x,,) = (which can be thought of as approximating pseudo-counts (8)). The episodic component of the reward 208 in Equation | is reset by emptying M with each episode, thus the NGU reward does not vanish throughout the training process. This makes it suitable for driving learning in task-agnostic settings. Further details on NGU are reported in the supplementary material.
206
210
Experiments
211
212
Agents are evaluated in the Atari suite 77 benchmark that presents variety of challenges and that
213
a common test ground for RL agents with unsupervised pre-training (28 40 [52. Experiments are run using the distributed R2D2 agent (36] with 256 CPU actors and a single GPU learner. Policies
214
215
use the same N-Network architecture as Agent57 [47], which composed by a convolutional torso
216
followed by an LSTM (32 and dueling head (58) Hyperparameters and a detailed description of
217
the full distributed setting are provided in the supplementary material. All reported results are the
average over three random seeds.
5

---

219
Reward-free learning. The amount of task reward collected by unsupervised policies is often used as a proxy to measure their quality [9). While the actual utility of these policies will not be revealed until they are leveraged for transfer, this proxy lets us evaluate whether the discovered behavior changes as longer pre-training budgets are allowed. We compare unsupervised NGU policies against VISR [28] and APT [40]. which utilize a small amount of supervised interaction to adapt the pre-trained policies. We also consider two additional unsupervised baselines: (i) a constant positive reward at each timestep that favours long episodes, which correlate with high scores in some games [0), and (ii) RND [T, which rewards life-long novelty. Note that the RND reward vanishes, but we include it in our analysis because it was previously used by Burda al. [10] in this setting and implementation choices such as reward normalization may prevent it from fading in practice.
220
222
223
224
226
228
Figure 2 (left) shows how the zero-shot transfer performance of unsupervised policies evolves during a long pre-training phase. NGU reaches the highest scores, but both NGU and RND eventually outperform VISR and APT even though these used supervised interaction. In Table 2 of Appendix
230
232
C we show that unsupervised NGU policies largely outperform several other baselines using the
233
standard pre-training and adaptation setting. These results highlight the importance of large-scale
234
unsupervised pre-training in RL, similarly to the trend observed in supervised domains
Atari-57 07: Median
Mo Montezuma's Revenge
Human narmalood score
Without transfer
NGU e00
RND @o
60
Pos Reward
Pong
With transfer
GPI VISR @400k
APT @400k
23
68 05
15.0
13.0
16.0
3.28
3
G4.B
R202 BT/mkoul @5B
Ursupe sised environmmnt Framnns
Unsupansised unvirorement frammss
Figure 2: Performance a function of the pre-training budget. @N represents the number of frames with reward utilized for transfer. (Left) Median human normalized score across the 57 games in the Atari suite. We observe the emergence of useful behavior when optimizing an intrinsic reward during a long unsupervised pre-trainin trainiiinng of 16B frames, which contrasts with the shorter pre-training of 1B frames in previous works (28 40). (Right) Scores in the games of Montezuma's 's Revenge (sparse rewards) and Pong (dense reward), before and after transfer, as a function of the pre-training budget. A longer pre-training benefits transfer in hard exploration games even if the zero-shot transfer score of the unsupervised policies does not increase.
235
Transfer setting. Transfer approaches are typically evaluated in the Atari benchmark with a budget of 100k RL interactions with reward (400k frames), but we propose to allow longer adaptation phase. Randomly initialized networks tend to overfit in these very low data regimes without strong regularization (38), and we are interested in studying the impact of leveraging behavior both in isolation and combined with transfer via pre-trained weights. Moreover, since the pre-trained policies already competent in the downstream tasks, 100k interactions are exhausted after few episodes and may be insufficient for improving performance. For these reasons, we provide results with up to 1. 1,25B RL steps of supervised interaction (5B frames). This allows evaluating both convergence
237
239
240
241
242
243
speed and asymptotic performance, while still being a relatively small budget for these distributed agents with hundreds of actors 47
244
245
Transfer via behavior.
We start by studying the impact of leveraging behavior in isolation, i.c. with- transferring pre-trained weights, when learning in downstream tasks. We compare BT against two baselines that do not use pre-trained behavior, namely the standard R2D2 agent [36] that uses c-greedy policies for exploration
247
[55], well as a variant of R2D2 with ez-greedy exploration [14. Figure shows that BT is superior to both baselines for any amount of environment interaction with rewards, converging faster early in training and also obtaining higher asymptotic performance. These results also demonstrate the generality of the proposed approach, as it is able to benefit from both RND and NGU policies. Note that BT performs particularly well in the set of six hard exploration game defined by Bellemare et al. (6), which is aligned with our intuition that reusing behavior helps over- coming the inefficiency associated to unstructured exploration. Figure 2 (right) confirms that a long pre-training phase is especially important in hard exploration games such as Montezuma's 's Revenge,
249
250
251
252
253
254
255
256
even it they do not translate into higher zero-shot transfer scores, as it produces more exploratory be- havior. On the other hand, the performance after transfer is independent of the amount of pre-training
257
258
in dense reward games like Pong, where unstructured exploration is enough to reach optimal scores.
ggravitar, ceattezuma_revengee pitfall, private_eye solaris, venture

---

Hard exploration games: median
600
203
Without rewards
500
175
Recu
Human normal zed
132
300
With rewards
203
75
R2D2
53
R202 + ez-greddy
100
25
R202 BT(maa))
18
2.8
38
4.
5B
16
28
3
48
R202 BT((kx)
Environment frames
Environment frames
Figure 3: Median human normalized scores for 22D2-based agents trained from scratch. (Left) Full
Atari suite. (Right) Subset of hard exploration games.
Extra action usage 1
ta
asterix
frestbite
24
0.05
bank bank_heist
pacman
jamesbond
gravitar
64
pong
private _eye
25
2.55
4.58
montezuma_ reveengeeeeeeeeeeeeeeeee
space invaders
2.5
Environment frames
Figure 4: Usage of the extra action in (T(T)) computed as the fraction of steps within an episode in which it is selected by the agent. The usage peaks early in training and slowly decreases afterwards as the new policy becomes stronger at the task.
259
Ablation studies. In order to gain insight on each of the components in BT, we run experiments on a subset of 12 game g requiring different amounts of exploration and featuring both dense and sparse rewards. BT(TNGU) achieves a median score of 368 in this subset, which compares favorably
260
261
262
to the 196 median score of R2D2 with e-greedy exploration. Removing either the extra action or the temporally-exter -extendee exploration reduces the median score of (T(TNGU) to 224. These results suggest that the gains provided by both strategies are complementary, and both are responsible for the strong performance of BT. To provide further insight about the benefits of BT, Figure Fepport the fraction of steps per episode in which the extra action is selected by the greedy policy. hints at the emergence of schedule over the usage of the pre-trained policy, which increases early in training and decays afterwards. We hypothesize that this is due to the fact that the unsupervised policies obtain large episodic returns, but their behavior is suboptimal when maximizing discounted rewards. These policies take many exploratory actions in between rewards, and so the agent eventually figures out more efficient strategies for reaching rewarding states by using primitive actions.
264
286
266
270
272
Transfer to multiple tasks. An appealing property of task-agnostic knowledge is that it can be leveraged to solve multiple tasks. In the RL setting, this can be evaluated by leveraging a single lask-agnostic policy for solving multiple tasks (i.e. reward functions) in the same environment. We evaluate whether the unsupervised NGU policies can be useful beyond the standard Atari tasks by creating two alternative versions of Ms Pacman and Hero with different levels of difficulty. The goal in the modified version of Ms Pacman is to eat vulnerable ghosts, with pac-dots giving 0 (casy version) or -10 (hard version) points. In the modified version of Hero, saving miners gives a fixed return of 1000 points and dynamiting walls gives either 0 (easy version) or -300 (hard version) points. The rest of rewards are removed, e.g. eating fruit in Ms Paeman or the bonus for unused power units in Hero. Note that even in the easy version of the games exploration is harder than in their original counterparts, as there are no small rewards guiding the agent towards its goals. Exploration is even more challenging in the hard version of the games, as the intermediate rewards work deceptive signal that takes the agent away from its actual goal. In this case, finding rewarding behaviors requires stronger commitment an exploration strategy. Unsupervised NGU policies often achieve very low
274
276
278
280
282
283
284
285
286
or even negative rewards in this setting, which contrasts with the strong performance they showed when evaluated under the standard game reward. Figure 5 shows that leveraging the behavior of pre-trained exploration policies provides important gains even in this adversarial scenario. These results suggest that the strong performance observed under the standard game rewards is not due an
287
288
289
Obtained by combining games used to tune hyperparameters in 28 with games where ez-greedy pro- vides clear gains over e-greedy as per [14] asterix, bank_heist, frostbite, gravitar, jamesbond, centezuma_revenge ms _s_pacman, pong, private_eye, space_invadee invaders. tennis, spwn_
7

---

MsPacman
NsPacman: eating ghosts (easy)
MsPacman: eating ghosts (hard)
10
Without rewards Reou (reward-fres)
Hero
Hero: rescuing miners (easy)
Hero: rescuing miners (hard)
With rewards
R202
R202 ce-greedy
R202 BTUnkc.)
20.0
18
2.55
30
18
Figure 5: Scores in Atari games with modified reward functions. We train single task-agnostic policy per environment, and leverage it to solve three different tasks: the standard game reward, a task with sparse rewards (easy), and variant of the same task with deceptive rewards (hard).
alignment between the NGU reward and the game goals, but due to an efficient usage of pre-trained ex exploration policies.
291
292
Combining pre-trained behavior and weights. Our last batch of experiments focuses on studying transfer via pre- pre-trained weights and its compatibility with BT. Policies are composed of convo- lutional torso, an LSTM, and a dueling head. We consider two initialization strategies: a partial initialization approach that loads the torso and the L.ST,, but initializes the head randomly; and a full initialization scheme where all weights are loaded. The former can be understood transferring learned representations [59], but deferring exploration to a random policy. On the other hand, the full initialization approach can be seen as directly transferring the policy and is usually referred to as fune-tuning the pre-trained policy 43 40 52] Note that these approaches only change how weights initialized before training. As in previous experiments, all parameters in the new policy are trained and p is kept fixed when using BT. Figure 6 (top) compares agents with and without BT for different amounts of transfer via weights the Atari benchmark. Loading pre- pre-trained weights results in faster
293
295
296
297
299
301
303
learning early in training. both with and without BT. The largest gains are observed in dense reward games, which translates into higher median scores across the full suite because most games belong
304
305
to this category. Weights alone are not enough in hard exploration games, where leveraging the trained policy via BT provides clear benefits. Perhaps surprisingly, we observe that transferring representations outperforms fine-tuning the pre-trained policy, and we hypothesize that the former more robust to misalignments between the pre-trained policy and the downstream task. This intuition is further supported by the experiments on games with modified reward functions reported in Figure o (middle & bottom), where the faster learning provided by pre-trained weights often comes at the cost of lower end scores. On the other hand, BT is crucial in tasks with sparse and deceptive rewards and also benefits from pre-trained weights in tasks where positive transfer is observed.
306
307
308
309
310
311
312
6
Related work
314
Our work uses the experimental methodology presented by Hansen et al. (28). Whereas that work only considered a fast, simplified adaptation process that limited the final performance on the downstream task, we focus on the more general case of using previously trained policy to aid in solving the full RL problem. Hansen et al. (28 use successor features to identify which of the pre-trained tasks best matches the true reward structure, which has previously been shown to work well for multi-task transfer 3 Bagot et al. 1 augments an agent with the ability to utilize another policy, which is
315
316
317
319
learned in tandem based on an intrinsic reward function. This promising direction is complementary
321 to our work, as it handles the case wherein there is no unsupervised pre-training phase.
322
Gupta et al. (25) provides an alternative method to meta-learn a solver for reinforcement learning prob- lems from unsupervised reward functions. This method utilizes gradient- based meta-learning (0)). which makes the adaptation process standard reinforcement learning updates. This means that even if the downstream reward is far outside of the training distribution, final performance would not neces- sarily be affected. However, these methods are hard to scale to the larger networks considered here, and followup work 34 changed to memory-based meta-learning [7 which relies on information about rewards staying in the recurrent state. This makes it unsuitable to the sort of hard exploration problem our method excels at. Recent work has shown success in transferring representations learned in unsupervised setting to reinforcement learning tasks [54). Our representation transfer experi-
323
324
325
326
327
329
8

---

Atari- 57: median
Hard exploration games: median
600
R2D2
R2D2 + BT(T)au)
From scratch
From scratch
400
Partial musu init
Partial init
Full masu init
Full rucu init
18
10
288
MsPacman
MsPacman: eating ghosts (easy)
MsPacman: eating ghosts (hard)
10c
1.5
8
3.5
4.8
18
28
88
11
18
28
3B
4
Hero
Hero: rescuing miners (easy)
Hero: rescuing miners (hard)
4.c
4
2
0
0
Environment frames
Enwironment frames
Environment frames
Figure 6: Performance of R2D2-based agents with different amounts of transfer via weights. Policies are composed of a CNN encoder followed by an LSTM and dueling head. We compare training from scratch, loading all weights (Full NNgu init) or all weights except those in the dueling head (Partial TNGU init). (Top) Median human normalized scores (HNS) in the full Atari suite (left) and the subset of hard exploration games (right). (Middle & Bottom) Games with modified reward functions as in F
ments suggest that this might handicap final performance, but the possibility also exists that different unsupervised objectives should be used for representation transfer and policy transfer.
332
Discussion
333
334 We studied the problem of transferring pre-trained behavior for exploration in reinforcement learning, an approach that is complementary to the common practice of transferring neural network weights. Our proposed approach. Behavior Transfer (BT), relies on the pre-trained policy for collecting experience in two different ways: (i) through temporally-extendee
336
exploration, which be triggered with some probability at any step, and (ii) via one-step calls to the pre-trained policy based on value estimates. BT results in strong transfer performance when combined with exploratory policies pre- trained in the absence of reward, with the most important gains being observed in hard exploration tasks. These benefits are not due an alignment between our pre-training and downstream tasks, as we also observed positive transfer in games where the pre-trained policy obtained low scores. In order to provide further evidence for this claim, we designed alternative tasks for Atari games involving hard exploration and deceptive rewards. Our transfer strategy outperformed all considered baselines in these settings, even when the pre-trained policy obtained very low or even negative scores, demonstrating the generality of the method. Besides disambiguating the role of the alignment between pre-tree training and downstream tasks, these experiments demonstrate the utility of a single aano-agoo policy for solving multiple tasks in the same environment. Finally, we also demonstrated that BT can be combined with transfer via neural network weights to provide further gains.
338
340
341
342
343
344
345
346
347
348
349
350
Our experimental results highlight the importance of scale when training RL agents in reward-free settings, which is one of the key factors behind the recent success of unsupervised approaches in other domains. This contrasts with the small budgets considered for reward-free RI. in previous works and motivates further research in unsupervised RL approaches that scale with increased data and compute.
351
352
353
354
We argue that scale is one of the missing components in reward-free RL, and it will be necessary condition unfold its full potential. Beyond improving the unsupervised learning phase, we are also
355
356
excited about the possibilities unlocked by BT and that are not possible when transferring knowledge
357
through weights, such as leveraging multiple pre-trained policies and deploying BT in continual learning scenarios where the agent never stops learning and keeps accumulating knowledge and skills.
358
359
Future work should also study improved mechanisms for handing over control to pre-trained policies,
as well as prioritizing the usage of certain behaviors over others when multiple such policies are available to the agent. This could overcome one of the current limitations of BT, which assumes that
361
flights can be started from any state and still produce meaningful behavior.

---

References
363
364
[1] Louis Bagot, Kevin Mets, and Steven Latré. Learning intrinsically motivated options to stimulate policy exploration. In ICML Workshop on LifeLong Learning, 2020.
365
386
[2] André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado P van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. In NeurlPS, 2017.
388
[3] Andre Barreto, Diana Borsa, John Quan, Tom Schaul, David Silver, Matteo Hessel, Daniel Mankowitz, Augustin Zidek, and Remi Munos. Transfer in deep reinforcement learning using successor features and generalised policy improvement. In JCML, 2018.
370
[4] André Barreto, Shaobo Hou, Diana Borsa, David Silver, and Doina Precup. Fast reinforcement learning with generalized policy updates. Proceedings of the National Academy of Sciences, 2020.
372
374
[5] Andrew G Barto and Sridhar Mahadevan. Recent advances in hierarchical reinforcement learning. Discrete event dynamic systems, 2003.
375
378
[6] Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In NeurIPS, 2016.
377
378
Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 2013.
379
380
381
[8] Diana Borsa, André Barreto, John Quan, Daniel Mankowitz, Rémi Munos, Hado van Hasselt, David Silver, and Tom Schaul. Universal successor features approximators. In ICLR, 2019.
382
383
[9] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. /4165, 2020.
385
[10] Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros. Large-scale study of curiosity-driven learning. arXiv preprint
387
2018.
[11] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv: 1810. 12894, 2018
389
390
[12] Víctor Campos. Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giro-i Nieto, and Jordi Torres. Explore, discover and learn: Unsupervised discovery of state-covering skills. In ICML., 2020.
381
382
393
[13] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020.
394
385
[14] Will Dabney, Georg Ostrovski, and André Barreto. Temporally-extender e-greedy exploration. In ICLR, 2021.
396
[15] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACI, 2019.
398
[16] Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In ICML, 2014.
400
[17] Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel. RL2: Fast reinforcement learning via slow reinforcement learning. arXiv preprint 161.07961..0277. 2016.
402
404
[18] Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymyr Mnih Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. IMPALA: Scalable distributed deep-RL with importance weighted actor-learner architectures. In ICML, 2018.
406
10

---

408
[19] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. In ICLR, 2019.
409
410
[20]
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adapta- tion of deep networks. arXiv preprint arXiv: 1703.03400 2017.
412
[21] Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic neural networks for hierarchical reinforcement learning. In ICLR, 2017.
413
414
[22]
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014.
[23] Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint arXiv:1611.07500 2016.
417
418
[24] Jean-Bastiens Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to sulf--upe learning. arXiv preprint urXiv:2006.07733 2020.
420
[25] Abhishek Gupta, Benjamin Eysenbach, Chelsea Finn, and Sergey Levine. Unsupervised meta-lcarning for reinforcement learning. arXiv preprint arXiv:1806.0466 2018.
423
424
[26]
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In NeurIPS, 2018
425
426
[27] Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In ICLR, 2019.
427
428
[28] Steven Hansen, Will Dabney, Andre Barreto, Tom Van de Wiele, David Warde-Farley. and Volodymyr Mnih. Fast task inference with variational intrinsic successor features. In ICLR, 2020.
429
430
[29] Elad Hazan, Sham M Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy exploration. In ICML, 2019.
432
435
[30] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. arXiv preprint
2019.
[31] Olivier J Hénaff, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding. arXiv preprint (xXi::900..0972: 2019.
436
437
[32] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 1997.
[33] Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado Van Hasselt, and David Silver. Distributed prioritized experience replay. arXiv preprint arXiv: 1rXiv:1803.00933 2018
440
[34] Allan Jabri, Kyle Hsu, Abhishek Gupta, Ben Eysenbach, Sergey Levine, and Chelsea Finn. Un- supervised curricula for visual meta-reinforcemmnt learning. In Advances in Neural Information Processing Systems, pages 10519-10531, 2019.
443
444
445
[35] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint 20X1:20001.8336, 2020.
448
447
448
[36] Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney, Recurrent
449
experience replay in distributed reinforcement learning. In ICLR, 2019.
450
[37] Michael Kearns and Satinder Singh. Near-optimal reinforcement learning in polynomial time.
451
Machine learning. 2002.
452
[38] Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation all you need: Regularizing deep reinforeement learning from pixels. In ICLR, 2021.
11

---

454
[39]
Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Rus- lan Salakhutdinow. Efficient exploration via state marginal matching. arXiv preprint arXiv: 1906.00204 2019.
455
[40] Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. arXiv preprint /03:::2033.0551,, 2021.
458
459
[41] Yao Liu and Emma Brunskill. When simple exploration is sample efficient: Identifying sufficient conditions for random exploration to yield pac rl algorithms. arXiv preprint arXiv:1805.0004 2018.
461
462
[42] Shakir Mohamed and Danilo Jimenez Rezende. Variational information maximisation for intrinsically motivated reinforcement learning. In NewrIPS, 2015.
[43] Mirco Mutti, Lorenzo Pratissoli, and Marcello Restelli. Task-agnostic exploration via policy gradient of a non-parametric state entropy estimate. In AAAI, 2021.
465
486
[44] Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn, arXiv preprini arXiv: 602.04621., 2016.
467
[55] Ian Oshand, Benjamin Van Roy, and Zheng Wen. Generalization and exploration via randomized value functions. In ICML, 2016.
489
470
[46] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driver exploration by self-supervised prediction. In ICML, 2017.
471
472
[47] Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human benchmark. In
473
474
ICML, 2020.
475
[48] Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi. Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martín Arjovsky, Alexander Pritzel, Andew Bolt, et al. Never give up: Learning directed exploration strategies. In ICLR, 2020.
476
477
478
[49]
Martin L Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, Inc., 1994.
[50] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 2019.
481
482
[51] Christoph Salge, Cornelius Glackin, and Daniel Polani. Empowerment - an introduction. In Guided Self- Organization: Inception. Springer, 2014.
[52] Max Schwarzer, Nitarshan Rajkumar, Michael Noukhovitch, Ankesh Anand, Laurent Charlin, Devon Hjelm, Philip Bachman, and Aaron Courville. Pretraining reward-free representations for data-efficient reinforcement leaming. In Self-Supervission for Reinforcement Learning Workshop ICLR 2021, 2021. URL https:://ppeneviiieee http:////pppaeeeeeeeeeeet/tt/tta
485
486
487
488
[53] Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning explore via self-supervised world models. In ICML, 2020.
489
490
[54] Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning, arXiv preprint arXiv:2009.08312 2020.
481
492
[55] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
493
494
[56]
Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 1999.
495
496
[57] Gandhimohan M Viswanathan, V Afanasyev, SV Buldyrev, EJ Murphy, PA Prince, and H Eu- gene Stanley. Lévy flight search patterns of wandering albatrosses. Nature, 1996.
497
498
[58] Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Mare Lanctot, and Nando Freitas. Ducling network architectures for deep reinforcement learning. In ICML, 2016.
12

---

500
[59] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement learning with prototypical representations. ICML, 2021.
501
502
[60]
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? arXiv preprint arXiv:1411. 2014.
503
604
[61] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks, In ECCV, 2014.
505
Checklist
507
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] [Yes], [No] or [N/A] You are strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
508
509
510
511
Did you include the license to the code and datasets? [No] The code and the data are proprictary.
512
Please do not modify the questions and only the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
514
515
516
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
518
519
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
521
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
522
523
2. If you are including theoretical results...
524
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either the supplemental material or as a URL)? [No] We did not include source code because it relies on non-public libraries that are specific to our distributed hardware setting. However, we include all the details needed to replicate experiments.
528
530
532
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
533
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] All our experiments were run with three different random seeds. Plots report mean, min and max results. Tables report mean and standard deviation.
535
537
538
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
539
510
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
511
(a) If your work uses existing assets, did you cite the creators? [N/A]
542
(b) Did you mention the license of the assets? [N/A]
543
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
5/4
545
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
13

---

547
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
548
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
551
552
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
554
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "PGsLFEq41v", "reviewer": "Reviewer_fBw3", "summary": "This paper proposes an approach to transferring pretrained models in a deep RL context. Instead of pretraining a neural network and then simply copying the weights and retraining on a new task (as would be the naive approach taken in supervised learning), the paper instead proposes transferring the pretrained policy's behaviour. This is done by modifying the epsilon-greedy exploration strategy to include the ability to use the action suggested by the pretrained policy, which can also be used as a kind of multi-step option for improved exploration.  Pretraining is conducted using the \"Never Give Up\" (NGU) intrinsic reward in a reward-free setting, and experiments are conducted in the full set of Atari games. Results and ablation studies indicate that such a transfer approach can drastically improve the performance of an agent on a new task. \n\nThe main contributions are a) the use of NGU as an unsupervised pretraining objective in RL; b) a method for incorporating a pretrained policy when learning a new policy; c) experimental results and ablation studies across the full suite of Atari tasks that demonstrate the improvement resulting from leveraging the pretrained model when exploring in a new task.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nThere is a lot to like about this paper. I found it extremely polished, well-written and easy to read. The proposed approach is simple, but in a good way, since it can be incorporated into existing work and codebases with very little effort. The experiments are also very extensive, with results reported over all of the Atari domains. However, I have some concerns regarding the novelty, whether the benchmark environments are appropriate, and some missing related work, which I'll discuss below. \n\nMain comments:\n\n1. As mentioned, one of the main positives of the paper is the ease with which the method can be integrated into existing work. Despite lacking theoretical justification, the empirical results across all the Atari games clearly show that, in practice, we can achieve far better performance with this kind of pretraining and transfer approach. I found the gap between the performance of the pretrained policy and the final policy interesting, since it clearly shows that we don't need a particularly strong pretrained policy (or that it doesn't have to be well aligned with the downstream task) for everything to be effective. \n\n2. My main concern is that I'm not sure whether the Atari suite is the correct testbed for this method. One reason why pretraining on say ImageNet in the supervised learning setting is so useful is that the models can be reused on other domains that are not necessarily ImageNet, but which operate in the image domain. In this case, though, a policy is pretrained only on the task that we're trying to solve.  So we need to pretrain a network on Montezuma's Revenge in order to play Montezuma's Revenge, but that's all we can do with it. I think the key here is the importance of amortisation - the cost of pretraining must be paid off by being able to solve downstream tasks quickly. But in the Atari suite, we can't just pretrain one policy and then use it for any Atari game - we need to pretrain for each and every Atari game, which defeats the purpose of pretraining in the first place, since no amortising is taking place. Given this, it is perhaps unsurprising that a neural network pretrained for 16bn timesteps on game A then has subsequently improved performance on game A!  Granted, there are two modified domains that attempt to address this issue, but considering just these two limits the experimental results and there's no display of amortisation.    \nIf the aim is to stick with Atari, then a more convincing approach would be to have a single pretrained model that we can use to solve any subsequent Atari game - the cost of pretraining could then be amortised over 57 subsequent tasks! However, I feel a better domain would have been something like DeepMind lab, where the \"base\" MDP captures the dynamics of the open world, and then subsequent downstream tasks would all be conducted in the same environment (e.g. collecting different objects). This to me would be more analogous to the ImageNet case, since we could then reuse the policy for each task within the DeepMind lab domain, much as we can reuse the ImageNet model in any natural image task. I realise that suggesting a completely different experimental setup is not immediately feasible, so I would also be open to discussion on why the current Atari experiments are appropriate. \n\n\n3. Another issue I'm unsure of is the novelty and contribution of the work, which seems to be combining existing techniques. I thought the discussion of an appropriate pretraining strategy was interesting (since other approaches have attempted to do things like autoencoders etc), and I think this could open many avenues for future work regarding how best to do so in the deep RL setting. However, the actual method of behavioural transfer is a combination and modification of existing techniques, which limits the novelty. For example, using a different policy to solve a new task by modifying the exploration behaviour has been done in prior work (e.g. [1] who call it \\pi-reuse). The difference here is simply that a) the policy can be used as an n-step option, and b) by adding in a virtual action, the agent can learn whether or not to explicitly use the pretrained policy. The contribution then feels like it's mainly about putting together these existing ideas and showing that they work well empirically.\n\n4. While there's a good discussion on prior work, I was very surprised to see no mention of reward-free RL [2], which is exactly the setting considered here (agent learns in a domain with no rewards, and then must quickly learn a downstream task after this pretraining phase). Although much of the work in this space is restricted to tabular [2, 3, 4] or linear function approximation [5], the advantage with these approaches is that they have guarantees. While not directly comparable to this work, I think it is definitely worth mentioning them.   \n\nMinor comments:\n\n1. Future work could potentially focus on other ways of incorporating the pretrained policy, such as using entropy regularised RL or similar.\n\n2. On line 164, should it not read that you update a^+ as well as the action chosen by the pretrained policy? At the moment, you have \\pi(s) and a', which are the same thing. \n\n3. A couple of the references have arxiv versions, whereas published ones exist (e.g. Brown et al. [9] was at NeurIPS 2020)\n\n[1] Fernández, Fernando, and Manuela Veloso. \"Probabilistic policy reuse in a reinforcement learning agent.\" Proceedings of the fifth international joint conference on Autonomous agents and multiagent systems. 2006.  \n[2] Jin, Chi, et al. \"Reward-free exploration for reinforcement learning.\" International Conference on Machine Learning. PMLR, 2020.  \n[3] Zhang, Zihan, Simon Du, and Xiangyang Ji. \"Near Optimal Reward-Free Reinforcement Learning.\" International Conference on Machine Learning. PMLR, 2021.  \n[4] Zanette, Andrea, et al. \"Provably efficient reward-agnostic navigation with linear value iteration.\" NeurIPS, 2020.  \n[5] Wang, Ruosong, et al. \"On reward-free reinforcement learning with linear function approximation.\" NeurIPS, 2020.\n\n\n***POST-REBUTTAL***\n\nThank you for the response, the comparison to prior work and additional experiments that showcase the importance of the *multistep* flight. Despite this, I still have concerns about the use of Atari which result in the following shortcomings (ignoring the Ms PacMan results) :\n\n1. Due to the use of Atari, transfer here is restricted to single target task instances (source task is the game without reward and the target task is the game with reward). However, multitask transfer (for target tasks with different reward functions/dynamics) could have been shown by using a different testbed like Marion, ProcGen, Retro, etc).\n2.  Related to this, the motivation in the introduction is that computer vision benefits from this pretraining. However, that's because that these models can transfer to multiple target tasks, not just one. The fact that the multitask transfer does not occur here is a limitation of  Atari, not the approach, which seems like a waste. \n3. I'm not sure I can agree with the assertion that pretraining is \"free\" because there's no reward function. Why should that be?  Better motivation here would be helpful. But again, if we just had the ability to transfer to multiple tasks, then the pretraining costs could be recouped in the long run. \n\nIf Atari is a must, one final suggestion would be to use the different game \"modes\" offered by the ALE (Machado et al, 2018). Then, at the very least, multitask transfer could be shown on the different version of the same game.   \n  ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There are perhaps two limitations to this work that could also be discussed. \n\n1. Adding a virtual action allows an agent to learn whether or not to reuse the pretrained policy, but it is unclear how we would immediately incorporate this into a continuous-action setting, since we would then have a mixture of continuous and discrete actions (there are, of course, approaches to overcoming this issue, but it wouldn't be straight-forward)\n3. The results here show the importance of an extended pretraining period. However, this may simply not be possible given the domain - how would one collect 10 years worth of data on a single robot?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "Mvt2tmpURAH", "reviewer": "Reviewer_1MjX", "summary": "The authors study the problem of transferring pre-trained behavior for exploration in reinforcement learning. They propose an approached called (BT) which relies on the pre-trained policy for collecting experience through temporally-extended exploration, which can be triggered with some probability at any step, or  one-step calls to the pre-trained policy based on value estimates.\nThe experiments presented in this work show that, when combined with large-scale pre-training in the absence of rewards, existing intrinsic motivation objectives can lead to the emergence of complex behaviors.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Originality:* The authors have proposed a novel approach for transferring pre-trained behaviour for exploration. While they use existing unsupervised pretraining using an existing method (NGU), how pre-trained policy is applied to downstream RL tasks with reward is novel. The authors made it very clear how their work differs from previous contributions and have adequately cited relevant papers.\n\n*Quality*: The submission is technically sound. The claims are supported using the empirical results. \n\n*Clarity*: The manuscript is well written and well organised. My only questions related to the Ablation studies Section. Specifically, in the first few sentences of this section, the authors talk about some results without proper reference to any figures. Please clarify this and refer to any figure which illustrates this. \n\n*Significance*: I think that the results are important and useful for the RL community. The problem setting itself is very well motivated. However, I have the following concerns that I'd like the authors to address:\n\n- The methods are only tested on the Atari benchmark. While I understand that this is what other work in transfer learning might also use for evaluation, I believe the paper can be strengthened if the methods are tested in more interesting environments. Atari tasks are singletons (every episode look the same) and deterministic, whereas for RL methods to generalise to real-world settings, approaches need to handle generalisation and stochasticity. Therefore, I'd suggest the authors try using some procedurally generated environments, such as [OpenAI ProcGen](https://github.com/openai/procgen), [NetHack](https://github.com/facebookresearch/nle), [MiniGrid](https://github.com/maximecb/gym-minigrid) or [MiniHack](https://openreview.net/forum?id=skFwlyefkWJ). In case these approaches don't work well on procedurally-generated tasks, I'd like the authors to explain why is this the case and how this can be improved in future work.\n\n- I'd also be interested in seeing baselines transfer learning results using [Kickstarting](https://arxiv.org/abs/1803.03835) from the pre-trained policy. Since kickstarting is also agnostic to how pre-trained policy is trained, it can be used for transfer learning.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Only one limitation of BT was mentioned in the discussion section (that BT assumes that flights can be started from any state and still produce meaningful behavior.). There are no negative societal impact as far as I'm concerned.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "ImEzICFsLzm", "reviewer": "Reviewer_sTTP", "summary": "The authors present a simple, intuitive method for fine-tuning “behaviors” by pretraining an exploration policy before using it as a temporally extended exploration procedure and a one-step additional action to the downstream policy on a downstream task. They extensively evaluate this on the suite of Atari games and demonstrate strong performance gains. Despite some qualms I have with assumptions and ablations, I think this paper should be accepted to NeurIPS. However, I do hope that the authors address some of my concerns in the rebuttal period. For now, this is a \"weak accept\" but if my concerns are addressed properly the score will be raised appropriately.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Paper Strengths\n**Method and Framing**\nThe method is simple and intuitive, and demonstrates strong performance improvements. Furthermore, the paper questions the traditional train-then-finetune paradigm that is typical in transfer learning for reinforcement learning. This is an interesting point to focus on and is useful to think about more for transfer learning in reinforcement learning in general, irrespective of the exact method.\n\n**Clarity**\nThis paper is well written, with a good flow that integrates background knowledge, some related works, and the method itself within the first four sections.\n\n**Experiments**\nDemonstrating that hard exploration games like Montezuma’s Revenge benefit from more unsupervised environment frames after transfer despite it not having an effect on 0-shot transfer performance is sensible. Furthermore, Figure 4, which highlights the extra action selection distribution over training during behavior transfer, is intriguing and demonstrates that the policy really learns how to take advantage of the pre-trained exploration policies.\n\nOverall, the number and scope of the experiments is impressive, highlighting many strengths of the method under different conditions.\n\n\n## Paper Weaknesses\n**Pretraining Assumption**\n“Interactions between the agent and the environment are often assumed to incur a cost, but we will consider this cost to be relevant only for transitions with reward” - I’m not sure this is a valid assumption. I think the authors should include a comparison/figure which includes the total number of samples including pre-training samples for BT, as sample efficiency is important partially for reasons related to total algorithm running time. Here, having these large numbers of pre-training steps still greatly increases the algorithm running time. \n\n**Technical Contributions**\nNothing here is particularly surprising. However, that is fine given that this method works well and is relatively easy to implement.\n\n**Experiments**\nThe “Transfer to Multiple Tasks” experiment would’ve been more enlightening if there were more tasks and/or more reward functions tested. The authors clearly have the computational resources to run comprehensive experiments on the Atari suite, so it seems strange to only perform this experiment on two tasks with just two reward functions each. Can the authors experiment on more environments?\n\n**Ablations**\nWhy one-step action execution of the pretrained policy? A temporally extended action should still be useful here. Can the authors should include an ablation with different length of action executions of this pretrained policy?\n\n\n-- UPDATE: score revised from 6 to 7 --", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "5BY7WvQ7cOl", "reviewer": "Reviewer_FMRN", "summary": "This paper studies the transfer of unsupervised RL agents in the Atari-57 benchmark suite. \n\nThe authors propose behavior transfer (BT), a simple method for transfer that focuses on using the behavior of the pre-trained policy, separately from using the pre-trained policy as an initialization (a.k.a. fine-tuning). BT augments a downstream off-policy learner in two ways: the downstream learner can use the pre-trained policy to explore (for a temporally extended duration), and the pre-trained policy is also added as a pseudo-action to the action space as an option during exploitation (for a single time step).\n\nIn the experiments, the authors primarily focus on using the never give up (NGU) objective to obtain the pre-trained policy and assess the impact of BT with this pre-trained policy on downstream learning via recurrent replay distributed DQN (R2D2). The protocol differs from prior works in that each learning phase is much longer: 16B frames per game for pre-training (a 64x increase over [28]) and up to 5B frames per game for transfer (a 12500x increase over [35]). This is shown to be important overall. For standard Atari-57, transfer via BT compares favorably against R2D2 baselines that lack unsupervised pre-training, with more pronounced gains for hard exploration games. Ablations show that both BT methods of using the pre-trained policy are necessary for best performance. The authors also assess on custom tasks in the Ms Pacman and Hero games, with BT causing significant improvement over R2D2 baselines despite the zero-shot pre-trained policy used for BT not achieving any success. Finally, the authors also assess BT with fine-tuning from (some subset of) the pre-trained policy's weights. For standard Atari-57, fine-tuning improves over training from scratch for both R2D2 as well as R2D2+BT. However, finetuning without BT from a partial initialization outperforms any variant involving BT, an interesting negative result. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Originality\nThe proposed method, behavior transfer (BT), is, to my knowledge, new. However, it may be helpful to reference and discuss prior works such as [A], which also broadly consider the use of pre-trained behavior in adaptation to new tasks, albeit in a different domain (simulated robotics) and with different problem assumptions (e.g. acquisition of pre-trained behavior via demonstrations). While Atari-57 is a well-established RL benchmark, the authors do bring a new perspective to unsupervised pre-training for Atari-57 by demonstrating the effect of scale in both the unsupervised pre-training and supervised adaptation phases. The authors also generate new reward functions to assess transfer to tasks other than the original games, though this is only done for two games.\n\n## Quality\nGiven that the novelty of BT is the use of pre-trained policies in specific ways during subsequent off-policy RL and not the intrinsic motivation objectives used to train the policies, I was surprised that the R2D2 baseline agents were run without RND/NGU. To me, this seems vital in order to observe the effect of BT in a controlled manner so that we know what to attribute any improvements to. Otherwise, the experiments appear to be thoroughly executed and documented. There are no theoretical claims to assess.\n\n## Clarity\nThe writing and presentation of the method and results are clear. However, I am not sure that the contributed method is appropriately named. Certainly, fine-tuning a pre-trained policy involves \"behavior transfer\" since the behavior of the pre-trained policy collects the data used for transfer learning, but fine-tuning, as the authors point out, is orthogonal to the proposed method. BT also seems too broad for any specific method. I would further recommend updating the title and/or abstract to reflect the scope of the method and experiments, specifically the focus on off-policy RL and Atari-57.\n\n## Significance\nGiven the simplicity of BT, I imagine that future works will use or build on it. Getting transfer to work well in the context of RL is an open question, and the idea that transferring a pre-trained policy can involve mechanisms distinct from fine-tuning is an interesting idea that I expect the field to consider going forward. However, the fact that BT with finetuning is not the best variant for the overall Atari-57 suite should give us pause. The direct impact of this work is also severely limited by the execution and scope of its experiments: aside from not controlling for the use of intrinsic motivation/exploration objectives, it's not clear whether the effective use of pre-trained policies for exploration is specific to pre-training objectives that focus on encouraging exploration. \n\n## References\n[A] Singh et al., Parrot: Data-Driven Behavioral Priors for Reinforcement Learning, ICLR 2021.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are discussed above. The potential societal impact of the work was not discussed by the authors.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper describes a simple but empirically effective strategy for unsupervised pre-training policies and utilizing them for exploration, with a temporally extended approach particularly effective for hard exploration challenges. The reviewers agree that the paper's concepts, while simple, are relevant and have potential to impact the community. Several reviewers raised concerns regarding the placement of the paper with respect to existing work and choices of experimental comparison. During responses, the authors presented additional detail (empirical and motivational) that has satisfied some, but not all reviewers. 

In my opinion, this is a paper is just below the NeurIPS borderline, as the papers brings great performance on important problems, but does borrow elements heavily from existing approaches and gives less theory/analytical justification for its advances. Reviewers and authors have had some consensus about the value in further exploration of the multi-task properties, with potentially non-Atari setups or by finding more ways to run multi-task trials in modified Atari levels. I also encourage such exploration to more firmly establish the impact of the ideas.

A side note that has not continbuted to the decision: the term "transfer" led to some misunderstanding during our discussions. For some reviewers (and myself), transfer can refer to many sub-areas of RL where testing is not done in Atari: changes in robotic dynamics, sim2real, multi-task and more. Here "learning from policies learned in an unsupervised pre-trained fashion" would provide the needed context with certainty, despite being a mouthful.

Decision: Reject