Beyond Fine-Tuning: Transferring Behavior in Reinforcement Learning

Designing agents that acquire knowledge autonomously and use it to solve new1 tasks efficiently is an important challenge in reinforcement learning . Knowledge2 acquired during an unsupervised pre-training phase is often transferred by fine-3 tuning neural network weights once rewards are exposed , as is common practice4 in supervised domains . Given the nature of the reinforcement learning problem,5 we argue that standard fine-tuning strategies alone are not enough for efficient6 transfer in challenging domains . We introduce Behavior Transfer ( BT ) , a technique7 that leverages pre-trained policies for exploration and that is complementary to8 transferring neural network weights . Our experiments show that , when combined9 with large-scale pre-training in the absence of rewards , existing intrinsic motivation10 objectives can lead to the emergence of complex behaviors . These pre-trained11 policies can then be leveraged by BT to discover better solutions than without12 pre-training , and combining BT with standard fine-tuning strategies results in13 additional benefits . The largest gains are generally observed in domains requiring14 structured exploration , including settings where the behavior of the pre-trained15 policies is misaligned with the downstream task.16 1 Introduction17 Transfer in deep learning is often performed through parameter initialization followed by fine-tuning,18 a technique that allows to leverage the power of deep networks in domains where labelled data19 is scarce [ 60 , 16 , 61 , 22 , 15 ] . This builds on the intuition that the pre-trained model will map20 inputs to a feature space where the downstream task is easy to perform . When combined with21 methods that can leverage massive amounts of unlabelled data for pre-training , this transfer strategy22 has led to unprecedented results in domains like computer vision [ 31 , 30 ] and natural language23 processing [ 15 , 50 ] . The success of these approaches has led to an ever-growing interest in developing24 techniques for pre-training large scale models on unlabelled data [ 9 , 13 , 24 ] .25 In the reinforcement learning ( RL ) context , unsupervised methods that learn in the absence of reward26 have also garnered much research attention [ 23 , 21 , 46 , 19 , 29 ] . The benefits of unsupervised pre-27 training are typically evaluated by their ability to enable efficient transfer to previously unseen reward28 functions [ 28 ] . In spite of their different approaches to unsupervised RL , most of the top-performing29 methods in this setting transfer knowledge through neural network weights . Such approaches deal30 with the data inefficiency associated to training neural networks with gradient descent , similarly to31 what is done in supervised learning , e.g . by pre-training encoders that extract representations from32 observations [ 59 ] . However , RL introduces a challenge that is not present in supervised learning : the33 agent is responsible for collecting the right data to learn from . This introduces a second source of34 inefficiency from which transfer approaches can also suffer if they rely on unstructured exploration35 strategies after pre-training , as these can lead to exponentially larger data requirements in complex36 Submitted to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Do not distribute . downstream environments [ 45 , 44 ] . To address this problem , one could consider fine-tuning policies37 that produce meaningful behavior [ 43 , 52 ] , but this approach quickly disregards the pre-trained38 behavior when learning in the downstream task due to catastrophic forgetting.39 In this work , we explicitly separate the transfer of behaviour and weights . We propose to make40 use of the pre-trained behaviour itself ( i.e. , the pre-trained policy mapping from observations to41 actions ) in contrast to pre-trained neural network weights for further fine-tuning . While pre-trained42 behavior has been used before for exploitation [ 5 , 56 , 2 , 3 ] , our approach employs pre-trained policies43 to aid with exploration as well to collect experience that can be leveraged via off-policy learning.44 This strategy accelerates learning , as the agent is exposed to potentially useful experience earlier in45 training , without compromising the quality of the discovered solution when the pre-trained behavior46 is not aligned with the downstream task . We expose the pre-trained behaviour to the downstream47 agent in two ways : firstly , as an extra exploratory strategy that , when randomly activated , persists for48 a number of steps , and secondly as an additional pseudo-action for the learned value function where49 the agent may elect to defer action selection to the pre-trained policy instead of choosing itself . We50 call this approach Behavior Transfer ( BT ) .51 Defining unsupervised RL objectives remains an open problem , and solutions are generally influenced52 by how the acquired knowledge will be used for solving downstream tasks . Instead of proposing yet53 another objective for unsupervised pre-training , we turn to existing techniques for training policies in54 the absence of reward and make our choice based on two general requirements . First , the objective55 should scale gracefully with increased compute and data . This has been key for the success of56 self-supervised approaches in other domains [ 9 , 35 ] , and we argue that it is an important property for57 unsupervised RL as well . Second , the pre-training stage should return a policy that produces complex58 behavior that may be leveraged in a subsequent transfer stage . The Never Give Up ( NGU ) [ 48 ] 59 intrinsic reward meets both requirements , and our experiments show that large-scale pre-training with60 this objective leads to state of the art scores in the reward-free Atari benchmark.61 Figure 1 exemplifies our main findings . We pre-train behaviour using the intrinsic NGU reward during62 a long unsupervised phase without rewards . This gives rise to exploratory behaviors that seek to visit63 many different states throughout an episode , and we then compare different strategies for leveraging64 the acquired knowledge once rewards are reinstated . While fine-tuning the pre-trained weights65 enables faster learning , the exploratory behavior of the pre-trained policy is quickly disregarded as it66 is exposed to rewards . On the other hand , Behavior Transfer ( BT ) does not modify the pre-trained67 policy while learning in the new task and is able to achieve higher end scores thanks to better68 exploration . These two strategies are not mutually exclusive , and BT also benefits from the faster69 convergence provided by initializing neural networks with pre-trained weights when these encode70 useful information for solving the downstream task.71 Our contributions can be summarized as follows . ( 1 ) We propose Behavior Transfer ( BT ) , a technique72 that leverages pre-trained policies for exploration by treating them as black boxes that are not modified73 during learning on the downstream task . BT uses the pre-trained policy to collect experience in74 two ways , namely randomly-triggered temporally-extended exploration and one-step calls based on75 value estimates . ( 2 ) Our experiments show that large-scale unsupervised pre-training with existing76 intrinsic rewards can produce meaningful behavior , achieving state of the art results in the reward-free77 Atari benchmark . These results suggest that scale is key for unsupervised RL , akin to what has been78 observed in supervised settings . ( 3 ) We provide extensive empirical evidence demonstrating the79 benefits of leveraging pre-trained behavior via BT . Our approach obtains the largest gains in hard80 exploration games , where it almost doubles the median human normalized score achieved by our81 strongest baseline . Furthermore , we show that BT is able to leverage a single task-agnostic policy82 to solve multiple tasks in the same environment and to achieve high performance even when the83 pre-trained policies are misaligned with the task being solved . ( 4 ) BT brings benefits to the table84 that are complementary to those provided by reusing pre-trained neural network weights , and we85 empirically show that combining these two strategies can result in larger gains.86 2 Preliminaries87 The interaction between the agent and the environment is modelled as a Markov Decission Pro-88 cess ( MDP ) [ 49 ] . An MDP is defined by the tuple ( S , A , P , d0 , R , γ ) where S and A are the state89 and action spaces , P ( s′|s , a ) is the probability of transitioning from state s to s′ after taking action a,90 d0 ( s ) is the probability distribution over initial states , R : S ×A× S −→ R is the reward function,91 and γ ∈ [ 0 , 1 ) is the discount factor . The goal is to find a policy π ( a|s ) that maximizes the expected92 return , Gt = ∑∞ t=0 γ tRt , where Rt = r ( St , At , St+1 ) . A principled way to address this problem93 is to use methods that compute action-value functions , Qπ ( s , a ) = Eπ [ Gt|St = s , At = a ] , where94 Eπ [ · ] denotes expectation over transitions induced by π [ 49 ] .95 We consider a setting where the agent is allowed to first learn within an MDP without rewards,96 MR = ( S , A , P , d0 ) , for a long period of time . The knowledge acquired during the reward-free97 stage is later leveraged when maximizing reward in new MDPs that share the same underlying98 dynamics but have different reward functions , Mi = ( S , A , P , d0 , Ri , γi ) . Interactions between the99 agent and the environment are often assumed to incur a cost , but we will consider this cost to be100 relevant only for transitions with reward [ 28 ] . Even if the cost of unsupervised pre-training becomes101 non-negligible , it can be amortized when the acquired task-agnostic knowledge is leveraged to solve102 multiple tasks efficiently [ 15 , 9 ] . Indeed , we would expect this transfer setting to become more103 relevant as the community moves towards more complex environments , where one may want to104 train agents to maximize multiple reward functions under constant dynamics . In the limit , one could105 consider the real world : it has constant or slowly changing dynamics , and humans are able to leverage106 previously acquired skills to quickly master new tasks.107 3 Behavior Transfer108 Transfer in supervised domains often exploits the fact that related tasks might be solved using similar109 representations . This practice deals with the data inefficiency of training large neural networks110 with stochastic gradient descent . However , there is an additional source of data inefficiency when111 training RL agents : unstructured exploration . Fine-tuning a pre-trained exploratory policy arises as112 a potential strategy for overcoming this problem , as the agent will observe rich experience much113 earlier in training than when initializing the policy randomly , but this approach suffers from important114 limitations . Learning in the downstream task can lead to catastrophically forgetting the pre-trained115 policy , thus prematurely disregarding its exploratory behavior . Moreover , the same neural network116 architecture needs to be used for both the pre-trained and the downstream policies , which in practice117 also imposes a limitation on the type of RL methods that can be employed in the adaptation stage ( for118 instance , if the pre-trained policy was trained using a policy-based method , it might not be possible119 to fine-tune it using a value-based approach ) .120 Let us assume that we have access to a pre-trained policy that exhibits exploratory behavior , and121 defer the discussion on how to train this policy to Section 4 . Following such a policy might bring122 the agent to states that are unlikely to be visited with unstructured exploration techniques such as123 -greedy [ 55 ] . This property has the potential of accelerating learning even when the behavior of124 the pre-trained policy is not aligned with the downstream task , as it will effectively shorten the125 path between otherwise distant states [ 41 ] . Leveraging pre-trained policies for exploration differs126 from other approaches in the literature that use such policies directly for exploitation , e.g . via127 zero-shot transfer [ 19 ] , methods that define a higher-level policy that alternates between the given128 policies [ 5 , 56 ] , or within the framework of generalized policy updates [ 4 ] . Exploring with pre-trained129 policies can accelerate convergence by providing useful experience to the agent , which is possible130 even when the pre-training and downstream tasks are misaligned . However , strategies that directly131 use the pre-trained policies for exploitation may result in sub-optimal solutions in such scenario [ 2 ] .132 We propose to leverage the behavior of pre-trained policies during transfer to aid with exploration . An133 explicit distinction between behavior and representation is made by considering pre-trained policies as134 black boxes that take observations and return actions . This strategy is agnostic to how the pre-trained135 behavior is encoded and is not restricted to learned policies . We rely on off-policy learning methods136 during transfer to leverage the behavior of a pre-trained policy πp ( a|s ) . We keep πp fixed during137 transfer , which prevents catastrophic forgetting of the original behavior when it is parameterized by a138 neural network ( i.e. , we instantiate and train a new policy with its own set of parameters ) . We propose139 Behavior Transfer ( BT ) , which leverages two complementary strategies to achieve this . Since BT140 is agnostic to the method used to pre-train policies , BT ( πp ) refers to behavior being transferred141 from policy πp . We formalize BT in the context of value-based Q-learning agents , although similar142 derivations are in principle possible for alternative off-policy learning methods . Pseudo-code for BT143 is provided in Algorithm 1.144 Temporally-extended exploration . We draw inspiration from Lévy flights [ 57 ] , a class of ecological145 models for animal foraging , where a fixed direction is followed for a duration sampled from a146 heavy-tailed distribution . This principle was implemented in the context of exploration in RL by147 z-greedy [ 14 ] , which encodes the notion of direction in the environment via exploration options that148 repeat the same action throughout the entire flight . Since πp is more likely to encode a meaningful149 notion of direction in complex environments than action repeats , we propose a variant of z-greedy150 where πp is used as the exploration option . An exploratory flight might be started at any step with151 some probability . The duration for the flight is sampled from a heavy-tailed distribution ( Zeta with152 µ = 2 in all our experiments ) , and control is handed over to πp during the complete flight . When not153 in a flight , actions are sampled from the behavior policy obtained while maximizing the task reward154 ( e.g . an -greedy derived from the estimated Q values ) .155 Extra action . The previous approach switches to πp during experience collection blindly , and we156 now consider an alternative strategy for triggering these switches based on value . This can be easily157 implemented through an extra action which samples an action from πp , which also allows the agent to158 use the pre-trained policy at test time if deemed beneficial . More formally , this amounts to training a159 policy over an expanded action setA+ = A∪ { a+ } , where a+ is resolved by sampling an action from160 πp , a′ ∼ πp ( s ) ( with a′ ∈ A ) . The additional action can be seen as an option that can be initiated161 from any state and always terminates after a single step . Note that selecting the option will lead to162 the same outcome as if the agent had selected a′ as a primitive action , and we take advantage of this163 observation by using the return of following the option as target to fit both Q ( s , πp ( s ) ) and Q ( s , a′ ) .164 Intuitively , this approach induces a bias that favours actions selected by πp , accelerating the collection165 of rewarding transitions when the pre-trained policy is somewhat aligned with the downstream task.166 Otherwise , the agent can learn to ignore πp as training progresses by selecting other actions.167 Algorithm 1 : Experience collection pseudo-code for BT Input : Action set , A ; additional action , a+ ; extended action set , A+ = A ∪ { a+ } ; pre-trained policy , πp ; Q-value estimate for the current policy , Qπ ( s , a ) ∀a ∈ A+ ; probability of taking an exploratory action , ; probability of starting a flight , levy ; flight length distribution , D ( N ) while True do n←− 0 // flight length while episode not ended do Observe state s if n == 0 and random ( ) ≤ levy then n ∼ D ( N ) // sample flight length if n > 0 then n←− n− 1 a ∼ πp ( s ) else if random ( ) ≤ then a ∼ Uniform ( A+ ) else a←− argmaxa′∈A+ [ Qπ ( s , a′ ) ] if a == a+ then a ∼ πp ( s ) end Take action a end end 4 Reward-free pre-training168 It is a common practice to derive objectives for proxy tasks in order to drive learning in the absence169 of reward functions , and there exists a plethora of different approaches in the literature . Model-based170 approaches can learn world models from unsupervised interaction [ 26 ] . However , the diversity of171 the training data will impact the accuracy of the model [ 53 ] and deploying this type of approach172 in visually complex domains like Atari remains an open problem [ 27 ] . Unsupervised RL has also173 been explored through the lens of empowerment [ 51 , 42 ] , which studies agents that aim to discover174 intrinsic options [ 23 , 19 ] . While these options can be leveraged by hierarchical agents [ 21 ] or175 integrated within the universal successor features framework [ 2 , 3 , 8 , 28 ] , their potential lack of176 coverage generally limits their applicability to complex downstream tasks [ 12 ] . An alternative177 objective is that of exploring the environment by finding policies that induce maximally entropic state178 distributions [ 29 , 39 ] , although this might become extremely inefficient in high-dimensional state179 spaces without proper priors [ 40 , 59 ] .180 Recall that our goal is to devise a pre-training objective that can help reduce the amount of interaction181 needed by the agent to collect relevant experience when learning in a downstream task . We argue that182 such objective needs to meet two requirements . First , as suggested by results in other domains [ 9 , 35 ] ,183 it should scale gracefully as the amount of compute and experience used for pre-training are increased.184 This contrasts with the training regimes used in most unsupervised RL approaches , which use a185 relatively small amount of experience [ 28 , 40 , 59 ] when compared to distributed agents that do make186 use of rewards [ 33 , 18 , 36 ] . Second , it must encourage the emergence of complex behaviors such as187 navigation or manipulation skills . It has been argued that exploring the environment efficiently will188 serve as a proxy for developing such behaviors [ 37 ] , and exploration bonuses have been shown to189 produce meaningful behavior in the absence of reward [ 46 , 10 ] . However , many exploration bonuses190 vanish over the course of training and thus may not be well-suited for a long unsupervised pre-training191 phase . It can be shown that many intrinsic rewards aim at maximizing the entropy of all states visited192 during training , and so the final policy does not necessarily exhibit exploratory behavior [ 39 ] .193 We propose to use Never Give Up ( NGU ) [ 48 ] as a means for training exploratory policies in an194 unsupervised setting . The NGU intrinsic reward proposes a curiosity-driven approach for training195 persistent exploratory policies which combines per-episode and life-long novelty . The per-episode196 novelty , repisodict , rapidly vanishes over the course of an episode , and it is designed to encourage self-197 avoiding trajectories . It is computed by comparing a representation of the current observation , f ( st ) ,198 to those of all the observations visited in the current episode , M = { f ( s0 ) , f ( s1 ) , . . . , f ( st−1 ) } ,199 where f : S → Rp is an embedding function trained using a self-supervised inverse dynamics200 model [ 46 ] . Such a mapping concentrates on the controllable aspects of the environment , ignoring201 all the variability present in the observation that is not affected by the action taken by the agent.202 The life-long novelty , αt , slowly vanishes throughout training , and it is computed by using Random203 Network Distillation ( RND ) [ 11 ] . With this , the intrinsic reward rNGUt is defined as follows:204 rNGUt = r episodic t ·min { max { αt , 1 } , L } , with r episodic t = 1√∑ f ( si ) ∈Nk K ( f ( st ) , f ( si ) ) + c ( 1 ) where L is a fixed maximum reward scaling , Nk is the set containing the k-nearest neighbors of f ( st ) 205 in M , c is a constant and K : Rp × Rp → R+ is a kernel function satisfying K ( x , x ) = 1 ( which206 can be thought of as approximating pseudo-counts [ 48 ] ) . The episodic component of the reward207 in Equation 1 is reset by emptying M with each episode , thus the NGU reward does not vanish208 throughout the training process . This makes it suitable for driving learning in task-agnostic settings.209 Further details on NGU are reported in the supplementary material.210 5 Experiments211 Agents are evaluated in the Atari suite [ 7 ] , a benchmark that presents a variety of challenges and that212 is a common test ground for RL agents with unsupervised pre-training [ 28 , 40 , 52 ] . Experiments are213 run using the distributed R2D2 agent [ 36 ] with 256 CPU actors and a single GPU learner . Policies214 use the same Q-Network architecture as Agent57 [ 47 ] , which is composed by a convolutional torso215 followed by an LSTM [ 32 ] and a dueling head [ 58 ] . Hyperparameters and a detailed description of216 the full distributed setting are provided in the supplementary material . All reported results are the217 average over three random seeds.218 Reward-free learning . The amount of task reward collected by unsupervised policies is often219 used as a proxy to measure their quality [ 19 ] . While the actual utility of these policies will not220 be revealed until they are leveraged for transfer , this proxy lets us evaluate whether the discovered221 behavior changes as longer pre-training budgets are allowed . We compare unsupervised NGU policies222 against VISR [ 28 ] and APT [ 40 ] , which utilize a small amount of supervised interaction to adapt223 the pre-trained policies . We also consider two additional unsupervised baselines : ( i ) a constant224 positive reward at each timestep that favours long episodes , which correlate with high scores in some225 games [ 10 ] , and ( ii ) RND [ 11 ] , which rewards life-long novelty . Note that the RND reward vanishes,226 but we include it in our analysis because it was previously used by Burda et al . [ 10 ] in this setting227 and implementation choices such as reward normalization may prevent it from fading in practice.228 Figure 2 ( left ) shows how the zero-shot transfer performance of unsupervised policies evolves during229 a long pre-training phase . NGU reaches the highest scores , but both NGU and RND eventually230 outperform VISR and APT even though these used supervised interaction . In Table 2 of Appendix231 C we show that unsupervised NGU policies largely outperform several other baselines using the232 standard pre-training and adaptation setting . These results highlight the importance of large-scale233 unsupervised pre-training in RL , similarly to the trend observed in supervised domains [ 9 ] .234 Transfer setting . Transfer approaches are typically evaluated in the Atari benchmark with a budget235 of 100k RL interactions with reward ( 400k frames ) , but we propose to allow a longer adaptation236 phase . Randomly initialized networks tend to overfit in these very low data regimes without strong237 regularization [ 38 ] , and we are interested in studying the impact of leveraging behavior both in238 isolation and combined with transfer via pre-trained weights . Moreover , since the pre-trained policies239 are already competent in the downstream tasks , 100k interactions are exhausted after few episodes240 and may be insufficient for improving performance . For these reasons , we provide results with up241 to 1.25B RL steps of supervised interaction ( 5B frames ) . This allows evaluating both convergence242 speed and asymptotic performance , while still being a relatively small budget for these distributed243 agents with hundreds of actors [ 47 ] .244 Transfer via behavior . We start by studying the impact of leveraging behavior in isolation , i.e . with-245 out transferring pre-trained weights , when learning in downstream tasks . We compare BT against two246 baselines that do not use pre-trained behavior , namely the standard R2D2 agent [ 36 ] that uses -greedy247 policies for exploration [ 55 ] , as well as a variant of R2D2 with z-greedy exploration [ 14 ] . Figure 3248 shows that BT is superior to both baselines for any amount of environment interaction with rewards,249 converging faster early in training and also obtaining higher asymptotic performance . These results250 also demonstrate the generality of the proposed approach , as it is able to benefit from both RND251 and NGU policies . Note that BT performs particularly well in the set of six hard exploration games1252 defined by Bellemare et al . [ 6 ] , which is aligned with our intuition that reusing behavior helps over-253 coming the inefficiency associated to unstructured exploration . Figure 2 ( right ) confirms that a long254 pre-training phase is especially important in hard exploration games such as Montezuma ’ s Revenge,255 even it they do not translate into higher zero-shot transfer scores , as it produces more exploratory be-256 havior . On the other hand , the performance after transfer is independent of the amount of pre-training257 in dense reward games like Pong , where unstructured exploration is enough to reach optimal scores.258 1gravitar , montezuma_revenge , pitfall , private_eye , solaris , venture Ablation studies . In order to gain insight on each of the components in BT , we run experiments259 on a subset of 12 games2 requiring different amounts of exploration and featuring both dense and260 sparse rewards . BT ( πNGU ) achieves a median score of 368 in this subset , which compares favorably261 to the 196 median score of R2D2 with -greedy exploration . Removing either the extra action or262 the temporally-extended exploration reduces the median score of BT ( πNGU ) to 224 . These results263 suggest that the gains provided by both strategies are complementary , and both are responsible for the264 strong performance of BT . To provide further insight about the benefits of BT , Figure 4 reports the265 fraction of steps per episode in which the extra action is selected by the greedy policy . It hints at the266 emergence of a schedule over the usage of the pre-trained policy , which increases early in training267 and decays afterwards . We hypothesize that this is due to the fact that the unsupervised policies268 obtain large episodic returns , but their behavior is suboptimal when maximizing discounted rewards.269 These policies take many exploratory actions in between rewards , and so the agent eventually figures270 out more efficient strategies for reaching rewarding states by using primitive actions.271 Transfer to multiple tasks . An appealing property of task-agnostic knowledge is that it can be272 leveraged to solve multiple tasks . In the RL setting , this can be evaluated by leveraging a single273 task-agnostic policy for solving multiple tasks ( i.e . reward functions ) in the same environment . We274 evaluate whether the unsupervised NGU policies can be useful beyond the standard Atari tasks by275 creating two alternative versions of Ms Pacman and Hero with different levels of difficulty . The276 goal in the modified version of Ms Pacman is to eat vulnerable ghosts , with pac-dots giving 0 ( easy277 version ) or −10 ( hard version ) points . In the modified version of Hero , saving miners gives a fixed278 return of 1000 points and dynamiting walls gives either 0 ( easy version ) or−300 ( hard version ) points.279 The rest of rewards are removed , e.g . eating fruit in Ms Pacman or the bonus for unused power units280 in Hero . Note that even in the easy version of the games exploration is harder than in their original281 counterparts , as there are no small rewards guiding the agent towards its goals . Exploration is even282 more challenging in the hard version of the games , as the intermediate rewards work as a deceptive283 signal that takes the agent away from its actual goal . In this case , finding rewarding behaviors requires284 a stronger commitment to an exploration strategy . Unsupervised NGU policies often achieve very low285 or even negative rewards in this setting , which contrasts with the strong performance they showed286 when evaluated under the standard game reward . Figure 5 shows that leveraging the behavior of287 pre-trained exploration policies provides important gains even in this adversarial scenario . These288 results suggest that the strong performance observed under the standard game rewards is not due to an289 2Obtained by combining games used to tune hyperparameters in [ 28 ] with games where z-greedy provides clear gains over -greedy as per [ 14 ] : asterix , bank_heist , frostbite , gravitar , jamesbond , montezuma_revenge , ms_pacman , pong , private_eye , space_invaders , tennis , up_n_down . alignment between the NGU reward and the game goals , but due to an efficient usage of pre-trained290 exploration policies.291 Combining pre-trained behavior and weights . Our last batch of experiments focuses on studying292 transfer via pre-trained weights and its compatibility with BT . Policies are composed of a convo-293 lutional torso , an LSTM , and a dueling head . We consider two initialization strategies : a partial294 initialization approach that loads the torso and the LSTM , but initializes the head randomly ; and a295 full initialization scheme where all weights are loaded . The former can be understood as transferring296 learned representations [ 59 ] , but deferring exploration to a random policy . On the other hand , the297 full initialization approach can be seen as directly transferring the policy and is usually referred to as298 fine-tuning the pre-trained policy [ 43 , 40 , 52 ] . Note that these approaches only change how weights299 are initialized before training . As in previous experiments , all parameters in the new policy are trained300 and πp is kept fixed when using BT . Figure 6 ( top ) compares agents with and without BT for different301 amounts of transfer via weights on the Atari benchmark . Loading pre-trained weights results in faster302 learning early in training , both with and without BT . The largest gains are observed in dense reward303 games , which translates into higher median scores across the full suite because most games belong304 to this category . Weights alone are not enough in hard exploration games , where leveraging the305 pre-trained policy via BT provides clear benefits . Perhaps surprisingly , we observe that transferring306 representations outperforms fine-tuning the pre-trained policy , and we hypothesize that the former307 is more robust to misalignments between the pre-trained policy and the downstream task . This308 intuition is further supported by the experiments on games with modified reward functions reported309 in Figure 6 ( middle & bottom ) , where the faster learning provided by pre-trained weights often comes310 at the cost of lower end scores . On the other hand , BT is crucial in tasks with sparse and deceptive311 rewards and also benefits from pre-trained weights in tasks where positive transfer is observed.312 6 Related work313 Our work uses the experimental methodology presented by Hansen et al . [ 28 ] . Whereas that work only314 considered a fast , simplified adaptation process that limited the final performance on the downstream315 task , we focus on the more general case of using a previously trained policy to aid in solving the316 full RL problem . Hansen et al . [ 28 ] use successor features to identify which of the pre-trained tasks317 best matches the true reward structure , which has previously been shown to work well for multi-task318 transfer [ 3 ] . Bagot et al . [ 1 ] augments an agent with the ability to utilize another policy , which is319 learned in tandem based on an intrinsic reward function . This promising direction is complementary320 to our work , as it handles the case wherein there is no unsupervised pre-training phase.321 Gupta et al . [ 25 ] provides an alternative method to meta-learn a solver for reinforcement learning prob-322 lems from unsupervised reward functions . This method utilizes gradient-based meta-learning [ 20 ] ,323 which makes the adaptation process standard reinforcement learning updates . This means that even if324 the downstream reward is far outside of the training distribution , final performance would not neces-325 sarily be affected . However , these methods are hard to scale to the larger networks considered here,326 and followup work [ 34 ] changed to memory-based meta-learning [ 17 ] which relies on information327 about rewards staying in the recurrent state . This makes it unsuitable to the sort of hard exploration328 problem our method excels at . Recent work has shown success in transferring representations learned329 in an unsupervised setting to reinforcement learning tasks [ 54 ] . Our representation transfer experi-330 ments suggest that this might handicap final performance , but the possibility also exists that different331 unsupervised objectives should be used for representation transfer and policy transfer.332 7 Discussion333 We studied the problem of transferring pre-trained behavior for exploration in reinforcement learning,334 an approach that is complementary to the common practice of transferring neural network weights.335 Our proposed approach , Behavior Transfer ( BT ) , relies on the pre-trained policy for collecting336 experience in two different ways : ( i ) through temporally-extended exploration , which can be triggered337 with some probability at any step , and ( ii ) via one-step calls to the pre-trained policy based on value338 estimates . BT results in strong transfer performance when combined with exploratory policies pre-339 trained in the absence of reward , with the most important gains being observed in hard exploration340 tasks . These benefits are not due to an alignment between our pre-training and downstream tasks,341 as we also observed positive transfer in games where the pre-trained policy obtained low scores.342 In order to provide further evidence for this claim , we designed alternative tasks for Atari games343 involving hard exploration and deceptive rewards . Our transfer strategy outperformed all considered344 baselines in these settings , even when the pre-trained policy obtained very low or even negative scores,345 demonstrating the generality of the method . Besides disambiguating the role of the alignment between346 pre-training and downstream tasks , these experiments demonstrate the utility of a single task-agnostic347 policy for solving multiple tasks in the same environment . Finally , we also demonstrated that BT can348 be combined with transfer via neural network weights to provide further gains.349 Our experimental results highlight the importance of scale when training RL agents in reward-free350 settings , which is one of the key factors behind the recent success of unsupervised approaches in other351 domains . This contrasts with the small budgets considered for reward-free RL in previous works and352 motivates further research in unsupervised RL approaches that scale with increased data and compute.353 We argue that scale is one of the missing components in reward-free RL , and it will be a necessary354 condition to unfold its full potential . Beyond improving the unsupervised learning phase , we are also355 excited about the possibilities unlocked by BT and that are not possible when transferring knowledge356 through weights , such as leveraging multiple pre-trained policies and deploying BT in continual357 learning scenarios where the agent never stops learning and keeps accumulating knowledge and skills.358 Future work should also study improved mechanisms for handing over control to pre-trained policies,359 as well as prioritizing the usage of certain behaviors over others when multiple such policies are360 available to the agent . This could overcome one of the current limitations of BT , which assumes that361 flights can be started from any state and still produce meaningful behavior.362 References363 [ 1 ] Louis Bagot , Kevin Mets , and Steven Latré . Learning intrinsically motivated options to stimulate364 policy exploration . In ICML Workshop on LifeLong Learning , 2020.365 [ 2 ] André Barreto , Will Dabney , Rémi Munos , Jonathan J Hunt , Tom Schaul , Hado P van Hasselt,366 and David Silver . Successor features for transfer in reinforcement learning . In NeurIPS , 2017.367 [ 3 ] Andre Barreto , Diana Borsa , John Quan , Tom Schaul , David Silver , Matteo Hessel , Daniel368 Mankowitz , Augustin Zidek , and Remi Munos . Transfer in deep reinforcement learning using369 successor features and generalised policy improvement . In ICML , 2018.370 [ 4 ] André Barreto , Shaobo Hou , Diana Borsa , David Silver , and Doina Precup . Fast reinforcement371 learning with generalized policy updates . Proceedings of the National Academy of Sciences,372 2020.373 [ 5 ] Andrew G Barto and Sridhar Mahadevan . Recent advances in hierarchical reinforcement374 learning . Discrete event dynamic systems , 2003.375 [ 6 ] Marc Bellemare , Sriram Srinivasan , Georg Ostrovski , Tom Schaul , David Saxton , and Remi376 Munos . Unifying count-based exploration and intrinsic motivation . In NeurIPS , 2016.377 [ 7 ] Marc G Bellemare , Yavar Naddaf , Joel Veness , and Michael Bowling . The arcade learning378 environment : An evaluation platform for general agents . Journal of Artificial Intelligence379 Research , 2013.380 [ 8 ] Diana Borsa , André Barreto , John Quan , Daniel Mankowitz , Rémi Munos , Hado van Hasselt,381 David Silver , and Tom Schaul . Universal successor features approximators . In ICLR , 2019.382 [ 9 ] Tom B Brown , Benjamin Mann , Nick Ryder , Melanie Subbiah , Jared Kaplan , Prafulla Dhariwal,383 Arvind Neelakantan , Pranav Shyam , Girish Sastry , Amanda Askell , et al . Language models are384 few-shot learners . arXiv preprint arXiv:2005.14165 , 2020.385 [ 10 ] Yuri Burda , Harri Edwards , Deepak Pathak , Amos Storkey , Trevor Darrell , and Alexei A Efros.386 Large-scale study of curiosity-driven learning . arXiv preprint arXiv:1808.04355 , 2018.387 [ 11 ] Yuri Burda , Harrison Edwards , Amos Storkey , and Oleg Klimov . Exploration by random388 network distillation . arXiv preprint arXiv:1810.12894 , 2018.389 [ 12 ] Víctor Campos , Alexander Trott , Caiming Xiong , Richard Socher , Xavier Giro-i Nieto , and390 Jordi Torres . Explore , discover and learn : Unsupervised discovery of state-covering skills . In391 ICML , 2020.392 [ 13 ] Ting Chen , Simon Kornblith , Mohammad Norouzi , and Geoffrey Hinton . A simple framework393 for contrastive learning of visual representations . In ICML , 2020.394 [ 14 ] Will Dabney , Georg Ostrovski , and André Barreto . Temporally-extended -greedy exploration.395 In ICLR , 2021.396 [ 15 ] Jacob Devlin , Ming-Wei Chang , Kenton Lee , and Kristina Toutanova . BERT : Pre-training of397 deep bidirectional transformers for language understanding . In NAACL , 2019.398 [ 16 ] Jeff Donahue , Yangqing Jia , Oriol Vinyals , Judy Hoffman , Ning Zhang , Eric Tzeng , and Trevor399 Darrell . Decaf : A deep convolutional activation feature for generic visual recognition . In ICML,400 2014.401 [ 17 ] Yan Duan , John Schulman , Xi Chen , Peter L Bartlett , Ilya Sutskever , and Pieter Abbeel . RL2:402 Fast reinforcement learning via slow reinforcement learning . arXiv preprint arXiv:1611.02779,403 2016.404 [ 18 ] Lasse Espeholt , Hubert Soyer , Remi Munos , Karen Simonyan , Volodymyr Mnih , Tom Ward,405 Yotam Doron , Vlad Firoiu , Tim Harley , Iain Dunning , et al . IMPALA : Scalable distributed406 deep-RL with importance weighted actor-learner architectures . In ICML , 2018.407 [ 19 ] Benjamin Eysenbach , Abhishek Gupta , Julian Ibarz , and Sergey Levine . Diversity is all you408 need : Learning skills without a reward function . In ICLR , 2019.409 [ 20 ] Chelsea Finn , Pieter Abbeel , and Sergey Levine . Model-agnostic meta-learning for fast adapta-410 tion of deep networks . arXiv preprint arXiv:1703.03400 , 2017.411 [ 21 ] Carlos Florensa , Yan Duan , and Pieter Abbeel . Stochastic neural networks for hierarchical412 reinforcement learning . In ICLR , 2017.413 [ 22 ] Ross Girshick , Jeff Donahue , Trevor Darrell , and Jitendra Malik . Rich feature hierarchies for414 accurate object detection and semantic segmentation . In CVPR , 2014.415 [ 23 ] Karol Gregor , Danilo Jimenez Rezende , and Daan Wierstra . Variational intrinsic control . arXiv416 preprint arXiv:1611.07507 , 2016.417 [ 24 ] Jean-Bastien Grill , Florian Strub , Florent Altché , Corentin Tallec , Pierre H Richemond , Elena418 Buchatskaya , Carl Doersch , Bernardo Avila Pires , Zhaohan Daniel Guo , Mohammad Gheshlaghi419 Azar , et al . Bootstrap your own latent : A new approach to self-supervised learning . arXiv420 preprint arXiv:2006.07733 , 2020.421 [ 25 ] Abhishek Gupta , Benjamin Eysenbach , Chelsea Finn , and Sergey Levine . Unsupervised422 meta-learning for reinforcement learning . arXiv preprint arXiv:1806.04640 , 2018.423 [ 26 ] David Ha and Jürgen Schmidhuber . Recurrent world models facilitate policy evolution . In424 NeurIPS , 2018.425 [ 27 ] Danijar Hafner , Timothy Lillicrap , Jimmy Ba , and Mohammad Norouzi . Dream to control:426 Learning behaviors by latent imagination . In ICLR , 2019.427 [ 28 ] Steven Hansen , Will Dabney , Andre Barreto , Tom Van de Wiele , David Warde-Farley , and428 Volodymyr Mnih . Fast task inference with variational intrinsic successor features . In ICLR,429 2020.430 [ 29 ] Elad Hazan , Sham M Kakade , Karan Singh , and Abby Van Soest . Provably efficient maximum431 entropy exploration . In ICML , 2019.432 [ 30 ] Kaiming He , Haoqi Fan , Yuxin Wu , Saining Xie , and Ross Girshick . Momentum contrast for433 unsupervised visual representation learning . arXiv preprint arXiv:1911.05722 , 2019.434 [ 31 ] Olivier J Hénaff , Ali Razavi , Carl Doersch , SM Eslami , and Aaron van den Oord . Data-efficient435 image recognition with contrastive predictive coding . arXiv preprint arXiv:1905.09272 , 2019.436 [ 32 ] Sepp Hochreiter and Jürgen Schmidhuber . Long short-term memory . Neural computation,437 1997.438 [ 33 ] Dan Horgan , John Quan , David Budden , Gabriel Barth-Maron , Matteo Hessel , Hado439 Van Hasselt , and David Silver . Distributed prioritized experience replay . arXiv preprint440 arXiv:1803.00933 , 2018.441 [ 34 ] Allan Jabri , Kyle Hsu , Abhishek Gupta , Ben Eysenbach , Sergey Levine , and Chelsea Finn . Un-442 supervised curricula for visual meta-reinforcement learning . In Advances in Neural Information443 Processing Systems , pages 10519–10531 , 2019.444 [ 35 ] Jared Kaplan , Sam McCandlish , Tom Henighan , Tom B Brown , Benjamin Chess , Rewon Child,445 Scott Gray , Alec Radford , Jeffrey Wu , and Dario Amodei . Scaling laws for neural language446 models . arXiv preprint arXiv:2001.08361 , 2020.447 [ 36 ] Steven Kapturowski , Georg Ostrovski , John Quan , Remi Munos , and Will Dabney . Recurrent448 experience replay in distributed reinforcement learning . In ICLR , 2019.449 [ 37 ] Michael Kearns and Satinder Singh . Near-optimal reinforcement learning in polynomial time.450 Machine learning , 2002.451 [ 38 ] Ilya Kostrikov , Denis Yarats , and Rob Fergus . Image augmentation is all you need : Regularizing452 deep reinforcement learning from pixels . In ICLR , 2021.453 [ 39 ] Lisa Lee , Benjamin Eysenbach , Emilio Parisotto , Eric Xing , Sergey Levine , and Rus-454 lan Salakhutdinov . Efficient exploration via state marginal matching . arXiv preprint455 arXiv:1906.05274 , 2019.456 [ 40 ] Hao Liu and Pieter Abbeel . Behavior from the void : Unsupervised active pre-training . arXiv457 preprint arXiv:2103.04551 , 2021.458 [ 41 ] Yao Liu and Emma Brunskill . When simple exploration is sample efficient : Identifying sufficient459 conditions for random exploration to yield pac rl algorithms . arXiv preprint arXiv:1805.09045,460 2018.461 [ 42 ] Shakir Mohamed and Danilo Jimenez Rezende . Variational information maximisation for462 intrinsically motivated reinforcement learning . In NeurIPS , 2015.463 [ 43 ] Mirco Mutti , Lorenzo Pratissoli , and Marcello Restelli . Task-agnostic exploration via policy464 gradient of a non-parametric state entropy estimate . In AAAI , 2021.465 [ 44 ] Ian Osband , Charles Blundell , Alexander Pritzel , and Benjamin Van Roy . Deep exploration via466 bootstrapped dqn . arXiv preprint arXiv:1602.04621 , 2016.467 [ 45 ] Ian Osband , Benjamin Van Roy , and Zheng Wen . Generalization and exploration via randomized468 value functions . In ICML , 2016.469 [ 46 ] Deepak Pathak , Pulkit Agrawal , Alexei A Efros , and Trevor Darrell . Curiosity-driven exploration470 by self-supervised prediction . In ICML , 2017.471 [ 47 ] Adrià Puigdomènech Badia , Bilal Piot , Steven Kapturowski , Pablo Sprechmann , Alex Vitvitskyi,472 Daniel Guo , and Charles Blundell . Agent57 : Outperforming the atari human benchmark . In473 ICML , 2020.474 [ 48 ] Adrià Puigdomènech Badia , Pablo Sprechmann , Alex Vitvitskyi , Daniel Guo , Bilal Piot , Steven475 Kapturowski , Olivier Tieleman , Martín Arjovsky , Alexander Pritzel , Andew Bolt , et al . Never476 give up : Learning directed exploration strategies . In ICLR , 2020.477 [ 49 ] Martin L Puterman . Markov Decision Processes : Discrete Stochastic Dynamic Programming.478 John Wiley & Sons , Inc. , 1994.479 [ 50 ] Alec Radford , Jeffrey Wu , Rewon Child , David Luan , Dario Amodei , and Ilya Sutskever.480 Language models are unsupervised multitask learners . OpenAI Blog , 2019.481 [ 51 ] Christoph Salge , Cornelius Glackin , and Daniel Polani . Empowerment – an introduction . In482 Guided Self-Organization : Inception . Springer , 2014.483 [ 52 ] Max Schwarzer , Nitarshan Rajkumar , Michael Noukhovitch , Ankesh Anand , Laurent Charlin,484 R Devon Hjelm , Philip Bachman , and Aaron Courville . Pretraining reward-free representations485 for data-efficient reinforcement learning . In Self-Supervision for Reinforcement Learning486 Workshop - ICLR 2021 , 2021 . URL https : //openreview.net/forum ? id=o5z9Le5drua.487 [ 53 ] Ramanan Sekar , Oleh Rybkin , Kostas Daniilidis , Pieter Abbeel , Danijar Hafner , and Deepak488 Pathak . Planning to explore via self-supervised world models . In ICML , 2020.489 [ 54 ] Adam Stooke , Kimin Lee , Pieter Abbeel , and Michael Laskin . Decoupling representation490 learning from reinforcement learning . arXiv preprint arXiv:2009.08319 , 2020.491 [ 55 ] Richard S Sutton and Andrew G Barto . Reinforcement learning : An introduction . MIT press,492 2018.493 [ 56 ] Richard S Sutton , Doina Precup , and Satinder Singh . Between mdps and semi-mdps : A494 framework for temporal abstraction in reinforcement learning . Artificial intelligence , 1999.495 [ 57 ] Gandhimohan M Viswanathan , V Afanasyev , SV Buldyrev , EJ Murphy , PA Prince , and H Eu-496 gene Stanley . Lévy flight search patterns of wandering albatrosses . Nature , 1996.497 [ 58 ] Ziyu Wang , Tom Schaul , Matteo Hessel , Hado Hasselt , Marc Lanctot , and Nando Freitas.498 Dueling network architectures for deep reinforcement learning . In ICML , 2016.499 [ 59 ] Denis Yarats , Rob Fergus , Alessandro Lazaric , and Lerrel Pinto . Reinforcement learning with500 prototypical representations . In ICML , 2021.501 [ 60 ] Jason Yosinski , Jeff Clune , Yoshua Bengio , and Hod Lipson . How transferable are features in502 deep neural networks ? arXiv preprint arXiv:1411.1792 , 2014.503 [ 61 ] Matthew D Zeiler and Rob Fergus . Visualizing and understanding convolutional networks . In504 ECCV , 2014.505 Checklist506 The checklist follows the references . Please read the checklist guidelines carefully for information on507 how to answer these questions . For each question , change the default [ TODO ] to [ Yes ] , [ No ] , or508 [ N/A ] . You are strongly encouraged to include a justification to your answer , either by referencing509 the appropriate section of your paper or providing a brief inline description . For example:510 • Did you include the license to the code and datasets ? [ No ] The code and the data are511 proprietary.512 Please do not modify the questions and only use the provided macros for your answers . Note that the513 Checklist section does not count towards the page limit . In your paper , please delete this instructions514 block and only keep the Checklist section heading above along with the questions/answers below.515 1 . For all authors ... 516 ( a ) Do the main claims made in the abstract and introduction accurately reflect the paper ’ s517 contributions and scope ? [ Yes ] 518 ( b ) Did you describe the limitations of your work ? [ Yes ] 519 ( c ) Did you discuss any potential negative societal impacts of your work ? [ N/A ] 520 ( d ) Have you read the ethics review guidelines and ensured that your paper conforms to521 them ? [ Yes ] 522 2 . If you are including theoretical results ... 523 ( a ) Did you state the full set of assumptions of all theoretical results ? [ N/A ] 524 ( b ) Did you include complete proofs of all theoretical results ? [ N/A ] 525 3 . If you ran experiments ... 526 ( a ) Did you include the code , data , and instructions needed to reproduce the main exper-527 imental results ( either in the supplemental material or as a URL ) ? [ No ] We did not528 include source code because it relies on non-public libraries that are specific to our529 distributed hardware setting . However , we include all the details needed to replicate530 our experiments.531 ( b ) Did you specify all the training details ( e.g. , data splits , hyperparameters , how they532 were chosen ) ? [ Yes ] 533 ( c ) Did you report error bars ( e.g. , with respect to the random seed after running experi-534 ments multiple times ) ? [ Yes ] All our experiments were run with three different random535 seeds . Plots report mean , min and max results . Tables report mean and standard536 deviation.537 ( d ) Did you include the total amount of compute and the type of resources used ( e.g. , type538 of GPUs , internal cluster , or cloud provider ) ? [ Yes ] 539 4 . If you are using existing assets ( e.g. , code , data , models ) or curating/releasing new assets ... 540 ( a ) If your work uses existing assets , did you cite the creators ? [ N/A ] 541 ( b ) Did you mention the license of the assets ? [ N/A ] 542 ( c ) Did you include any new assets either in the supplemental material or as a URL ? [ N/A ] 543 544 ( d ) Did you discuss whether and how consent was obtained from people whose data you ’ re545 using/curating ? [ N/A ] 546 ( e ) Did you discuss whether the data you are using/curating contains personally identifiable547 information or offensive content ? [ N/A ] 548 5 . If you used crowdsourcing or conducted research with human subjects ... 549 ( a ) Did you include the full text of instructions given to participants and screenshots , if550 applicable ? [ N/A ] 551 ( b ) Did you describe any potential participant risks , with links to Institutional Review552 Board ( IRB ) approvals , if applicable ? [ N/A ] 553 ( c ) Did you include the estimated hourly wage paid to participants and the total amount554 spent on participant compensation ? [ N/A ] 555

Summary: This paper studies the transfer of unsupervised RL agents in the Atari-57 benchmark suite.   The authors propose behavior transfer (BT), a simple method for transfer that focuses on using the behavior of the pre-trained policy, separately from using the pre-trained policy as an initialization (a.k.a. fine-tuning). BT augments a downstream off-policy learner in two ways: the downstream learner can use the pre-trained policy to explore (for a temporally extended duration), and the pre-trained policy is also added as a pseudo-action to the action space as an option during exploitation (for a single time step).  In the experiments, the authors primarily focus on using the never give up (NGU) objective to obtain the pre-trained policy and assess the impact of BT with this pre-trained policy on downstream learning via recurrent replay distributed DQN (R2D2). The protocol differs from prior works in that each learning phase is much longer: 16B frames per game for pre-training (a 64x increase over [28]) and up to 5B frames per game for transfer (a 12500x increase over [35]). This is shown to be important overall. For standard Atari-57, transfer via BT compares favorably against R2D2 baselines that lack unsupervised pre-training, with more pronounced gains for hard exploration games. Ablations show that both BT methods of using the pre-trained policy are necessary for best performance. The authors also assess on custom tasks in the Ms Pacman and Hero games, with BT causing significant improvement over R2D2 baselines despite the zero-shot pre-trained policy used for BT not achieving any success. Finally, the authors also assess BT with fine-tuning from (some subset of) the pre-trained policy's weights. For standard Atari-57, fine-tuning improves over training from scratch for both R2D2 as well as R2D2+BT. However, finetuning without BT from a partial initialization outperforms any variant involving BT, an interesting negative result.