Coverage as a Principle for Discovering Transferable Behavior in Reinforcement Learning

1 INTRODUCTION . Unsupervised representation learning techniques have led to unprecedented results in domains like computer vision ( Hénaff et al. , 2019 ; He et al. , 2019 ) and natural language processing ( Devlin et al. , 2019 ; Radford et al. , 2019 ) . These methods are commonly composed of two stages – an initial unsupervised phase , followed by supervised fine-tuning on downstream tasks . The self-supervised nature of the learning objective allows to leverage large collections of unlabelled data in the first stage . This produces models that extract task-agnostic features that are well suited for transfer to downstream tasks . In reinforcement learning ( RL ) , auxiliary representation learning objectives provide denser signals that result in data efficiency gains ( Jaderberg et al. , 2017 ) and even bridge the gap between learning from true state and pixel observations ( Laskin et al. , 2020 ) . However , RL applications have not yet seen the advent of the two-stage setting where task-agnostic pre-training is followed by efficient transfer to downstream tasks . We argue that there are two reasons explaining this lag with respect to their supervised counterparts . First , these methods traditionally focus on transferring representations ( Lesort et al. , 2018 ) . While this is enough in supervised scenarios , we argue that leveraging pre-trained behavior is far more important in RL domains requiring structured exploration . Second , what type of self-supervised objectives enable the acquisition of transferable , task-agnostic knowledge is still an open question . Defining these objectives in the RL setting is complex , as they should account for the fact that the the distribution of the input data will be defined by the behavior of the agent . Transfer in deep learning is often performed through parameter initialization followed by fine-tuning . The most widespread procedure consists in initializing all weights in the neural network using those from the pre-trained model , and then adding an output layer with random parameters ( Girshick et al. , 2014 ; Devlin et al. , 2019 ) . Depending on the amount of available data , pre-trained parameters can either be fine-tuned or kept fixed . This builds on the intuition that the pre-trained model will map inputs to a feature space where the downstream task is easy to perform . In the RL setting , this procedure will completely dismiss the pre-trained policy and fall back to a random one when collecting experience . Given that complex RL problems require structured and temporally-extended behaviors , we argue that representation alone is not enough for efficient transfer in challenging domains . Pre-trained representations do indeed provide data efficiency gains in domains with dense reward signals ( Finn et al. , 2017 ; Yarats et al. , 2019 ; Stooke et al. , 2020a ) , but our experiments show that the standard fine-tuning procedure falls short in hard exploration problems ( c.f . Figure 1 ) . We observe this limitation even when fine-tuning the pre-trained policy , which is aligned with findings from previous works ( Finn et al. , 2017 ) . Learning in the downstream task can lead to catastrophically forgetting the pre-trained policy , something that depends on many difficult-to-measure factors such as the similarity between the tasks . We address the problem of leveraging arbitrary pre-trained policies when solving downstream tasks , a requirement towards enabling efficient transfer in RL . Defining unsupervised RL objectives remains an open problem , and existing solutions are often influenced by how the acquired knowledge will be used for solving downstream tasks . Modelbased approaches can learn world models from unsupervised interaction ( Ha & Schmidhuber , 2018 ) . However , the diversity of the training data will impact the accuracy of the model ( Sekar et al. , 2020 ) and deploying this type of approach in visually complex domains like Atari remains an open problem ( Hafner et al. , 2019 ) . Unsupervised RL has also been explored through the lens of empowerment ( Salge et al. , 2014 ; Mohamed & Rezende , 2015 ) , which studies agents that aim to discover intrinsic options ( Gregor et al. , 2016 ; Eysenbach et al. , 2019 ) . While these options can be leveraged by hierarchical agents ( Florensa et al. , 2017 ) or integrated within the universal successor features framework ( Barreto et al. , 2017 ; 2018 ; Borsa et al. , 2019 ; Hansen et al. , 2020 ) , their lack of coverage generally limits their applicability to complex downstream tasks ( Campos et al. , 2020 ) . We argue that maximizing coverage is a good objective for task-agnostic RL , as agents that succeed at this task will need to develop complex behaviors in order to efficiently explore the environment ( Kearns & Singh , 2002 ) . This problem can be formulated as that of finding policies that induce maximally entropic state distributions , which might become extremely inefficient in high-dimensional state spaces without proper priors ( Hazan et al. , 2019 ; Lee et al. , 2019 ) . In practice , exploration is often encouraged through intrinsic curiosity signals that incorporate priors in order to quantify how different the current state is from those already visited ( Bellemare et al. , 2016 ; Houthooft et al. , 2016 ; Ostrovski et al. , 2017 ; Puigdomènech Badia et al. , 2020b ) . Agents that maximize these novelty-seeking signals have been shown to discover useful behaviors in unsupervised settings ( Pathak et al. , 2017 ; Burda et al. , 2018a ) , but little research has been conducted towards leveraging the acquired knowledge once the agent is exposed to extrinsic reward . We show that coverage-seeking objectives are a good proxy for acquiring knowledge in task-agnostic settings , as leveraging the behaviors discovered in an unsupervised pre-training stage provides important gains when solving downstream tasks . Our contributions can be summarized as follows . ( 1 ) We study how to transfer knowledge in RL through behavior by re-using pre-trained policies , an approach that is complementary to re-using representations . We argue that pre-trained behavior can be used for both exploitation and exploration , and present techniques to achieve both goals . ( 2 ) We propose coverage as a principle for discovering behavior that is suitable for both exploitation and exploration . While coverage is naturally aligned with exploration , we show that this objective will lead to the discovery of behavior that is useful for exploitation as well . ( 3 ) We propose Coverage Pre-training for Transfer ( CPT ) , a method that implements the aforementioned hypotheses , and provide extensive experimental evaluation to support them . Our results show that leveraging the behavior of policies pre-trained to maximize coverage provides important benefits when solving downstream tasks . CPT obtains the largest gains in hard exploration games , where it almost doubles the median human normalized score achieved by our strongest baseline . Importantly , these benefits are observed even when the pre-trained policies are misaligned with the task being solved , confirming that the benefits do not come from a fortuitous alignment between our pre-training objective and the task reward . Furthermore , we show that CPT is able to leverage a single task-agnostic policy to solve multiple tasks in the same environment . 2 REINFORCEMENT LEARNING WITH UNSUPERVISED PRE-TRAINING . We follow a similar setup to that proposed by Hansen et al . ( 2020 ) . In an initial pre-training stage , agents are allowed as many interactions with the environment as needed as long as they are not exposed to task-specific rewards . Rewards are reinstated in a second stage , where the knowledge acquired during unsupervised pre-training should be leveraged in order to enable efficient learning . This is analogous to the evaluation setting for unsupervised learning methods , where pre-training on classification benchmarks with labels removed is evaluated after fine-tuning on small sets of annotated examples . The two-stage setup introduces two main challenges : defining pretext tasks in the absence of reward , and efficiently leveraging knowledge once rewards are reinstated . Our proposed method , Coverage Pre-training for Transfer ( CPT ) , relies on coverage maximization as a pretext task for task-agnostic pre-training in order to produce policies whose behavior can be leveraged for both exploitation and exploration when solving downstream tasks in the same environment . Figure 2 provides intuition about the potential benefits of CPT . 3 LEVERAGING PRE-TRAINED POLICIES . Transfer in supervised domains often exploits the fact that related tasks might be solved using similar representations . This practice deals with the data inefficiency of training large neural networks with stochastic gradient descent . However , there is an additional source of data inefficiency when training RL agents : unstructured exploration . If the agent fails at discovering reward while exploring , it will struggle even when fitting simple function approximators on top of the true state of the MDP . These two strategies are complementary , as they address different sources of inefficiency , which motivates the study of techniques for leveraging pre-trained behavior ( i.e . policies ) . Our approach relies on off-policy learning methods in order to leverage arbitrary pre-trained policies . We make use of the mapping from observations to actions of such policies ( i.e . their behavior ) , and do not transfer knowledge through pre-trained neural network weights . We consider value-based methods with experience replay that estimate action-value functions and derive greedy policies from them . The presented formulation considers a single pre-trained policy , πp , but note that it is straightforward to extend it to multiple such policies . No assumptions are made on how the pre-trained policy is obtained , and it is only used for acting . We propose using the behavior of the pre-trained policy for two complementary purposes : exploitation and exploration . Figure 2 provides intuition about the potential benefits of these two approaches on a simple environment , and pseudo-code for the proposed methods is included in Appendix A . Exploitation . When the behavior of πp is aligned with the downstream task , it can be used for zero-shot transfer . However , we are concerned with the more realistic scenario where only some of the behaviors of πp might be aligned with downstream tasks ( c.f . Figure 2 , right ) . We propose to leverage πp for exploitation by letting the agent combine primitive actions with the behavior of πp . This is achieved by considering an expanded action set A+ = A∪ { πp ( s ) } , so that the agent can fall back to πp for one step when taking the additional action . Intuitively , this new state-dependent action should enable faster convergence when the pre-trained policy discovered behaviors that are useful for the task , while letting the agent ignore it otherwise . The return of taking action a′ ∼ πp ( s ) is used as target to fit both Q ( s , πp ( s ) ) and Q ( s , a′ ) , which implements the observation that they are the same action and thus will lead to the same outcomes . Exploration . Following the pre-trained policy might bring the agent to states that are unlikely to be visited with unstructured exploration techniques such as -greedy . This property has the potential of accelerating learning even when the behavior of the pre-trained policy is not aligned with the downstream task , as it will effectively shorten the path between otherwise distant states ( Liu & Brunskill , 2018 ) . As we rely on off-policy methods that can learn from experience collected by arbitrary policies , we propose to perform temporally-extended exploration with πp , which we will refer to as flights . Inspired by z-greedy and its connection to Lévy flights ( Viswanathan et al. , 1996 ) , a class of ecological models for animal foraging , these flights are started randomly and their duration is sampled from a heavy-tailed distribution . Our proposal can be understood as a variant of z-greedy where pre-trained policies are used as exploration options . An exploratory flight might be started at any step with some probability . The duration for the flight is sampled from a heavy-tailed distribution , and control is handed over to πp during the complete flight . When not in a flight , the exploitative policy that maximizes the extrinsic reward is derived from the estimated Q-values using the -greedy operator . This ensures that all state-action pairs will be visited given enough time , as exploring only with πp does not guarantee such property . Note that this is not needed in z-greedy , which reduces to standard -greedy exploration when sampling a flight duration of one step .

Summary: This paper proposed a transfer approach for reinforcement learning. The proposed approach leverages a policy pre-trained via Never Give Up (NGU) approach, and can facilitate learning challenging RL tasks including the ones with sparse reward. This paper presents many strong pieces of evidence that this approach can be used to tackle challenging RL problems including hard exploration and multi-task learning.