Coverage as a Principle for Discovering Transferable Behavior in Reinforcement Learning

1 INTRODUCTION . Unsupervised representation learning techniques have led to unprecedented results in domains like computer vision ( Hénaff et al. , 2019 ; He et al. , 2019 ) and natural language processing ( Devlin et al. , 2019 ; Radford et al. , 2019 ) . These methods are commonly composed of two stages – an initial unsupervised phase , followed by supervised fine-tuning on downstream tasks . The self-supervised nature of the learning objective allows to leverage large collections of unlabelled data in the first stage . This produces models that extract task-agnostic features that are well suited for transfer to downstream tasks . In reinforcement learning ( RL ) , auxiliary representation learning objectives provide denser signals that result in data efficiency gains ( Jaderberg et al. , 2017 ) and even bridge the gap between learning from true state and pixel observations ( Laskin et al. , 2020 ) . However , RL applications have not yet seen the advent of the two-stage setting where task-agnostic pre-training is followed by efficient transfer to downstream tasks . We argue that there are two reasons explaining this lag with respect to their supervised counterparts . First , these methods traditionally focus on transferring representations ( Lesort et al. , 2018 ) . While this is enough in supervised scenarios , we argue that leveraging pre-trained behavior is far more important in RL domains requiring structured exploration . Second , what type of self-supervised objectives enable the acquisition of transferable , task-agnostic knowledge is still an open question . Defining these objectives in the RL setting is complex , as they should account for the fact that the the distribution of the input data will be defined by the behavior of the agent . Transfer in deep learning is often performed through parameter initialization followed by fine-tuning . The most widespread procedure consists in initializing all weights in the neural network using those from the pre-trained model , and then adding an output layer with random parameters ( Girshick et al. , 2014 ; Devlin et al. , 2019 ) . Depending on the amount of available data , pre-trained parameters can either be fine-tuned or kept fixed . This builds on the intuition that the pre-trained model will map inputs to a feature space where the downstream task is easy to perform . In the RL setting , this procedure will completely dismiss the pre-trained policy and fall back to a random one when collecting experience . Given that complex RL problems require structured and temporally-extended behaviors , we argue that representation alone is not enough for efficient transfer in challenging domains . Pre-trained representations do indeed provide data efficiency gains in domains with dense reward signals ( Finn et al. , 2017 ; Yarats et al. , 2019 ; Stooke et al. , 2020a ) , but our experiments show that the standard fine-tuning procedure falls short in hard exploration problems ( c.f . Figure 1 ) . We observe this limitation even when fine-tuning the pre-trained policy , which is aligned with findings from previous works ( Finn et al. , 2017 ) . Learning in the downstream task can lead to catastrophically forgetting the pre-trained policy , something that depends on many difficult-to-measure factors such as the similarity between the tasks . We address the problem of leveraging arbitrary pre-trained policies when solving downstream tasks , a requirement towards enabling efficient transfer in RL . Defining unsupervised RL objectives remains an open problem , and existing solutions are often influenced by how the acquired knowledge will be used for solving downstream tasks . Modelbased approaches can learn world models from unsupervised interaction ( Ha & Schmidhuber , 2018 ) . However , the diversity of the training data will impact the accuracy of the model ( Sekar et al. , 2020 ) and deploying this type of approach in visually complex domains like Atari remains an open problem ( Hafner et al. , 2019 ) . Unsupervised RL has also been explored through the lens of empowerment ( Salge et al. , 2014 ; Mohamed & Rezende , 2015 ) , which studies agents that aim to discover intrinsic options ( Gregor et al. , 2016 ; Eysenbach et al. , 2019 ) . While these options can be leveraged by hierarchical agents ( Florensa et al. , 2017 ) or integrated within the universal successor features framework ( Barreto et al. , 2017 ; 2018 ; Borsa et al. , 2019 ; Hansen et al. , 2020 ) , their lack of coverage generally limits their applicability to complex downstream tasks ( Campos et al. , 2020 ) . We argue that maximizing coverage is a good objective for task-agnostic RL , as agents that succeed at this task will need to develop complex behaviors in order to efficiently explore the environment ( Kearns & Singh , 2002 ) . This problem can be formulated as that of finding policies that induce maximally entropic state distributions , which might become extremely inefficient in high-dimensional state spaces without proper priors ( Hazan et al. , 2019 ; Lee et al. , 2019 ) . In practice , exploration is often encouraged through intrinsic curiosity signals that incorporate priors in order to quantify how different the current state is from those already visited ( Bellemare et al. , 2016 ; Houthooft et al. , 2016 ; Ostrovski et al. , 2017 ; Puigdomènech Badia et al. , 2020b ) . Agents that maximize these novelty-seeking signals have been shown to discover useful behaviors in unsupervised settings ( Pathak et al. , 2017 ; Burda et al. , 2018a ) , but little research has been conducted towards leveraging the acquired knowledge once the agent is exposed to extrinsic reward . We show that coverage-seeking objectives are a good proxy for acquiring knowledge in task-agnostic settings , as leveraging the behaviors discovered in an unsupervised pre-training stage provides important gains when solving downstream tasks . Our contributions can be summarized as follows . ( 1 ) We study how to transfer knowledge in RL through behavior by re-using pre-trained policies , an approach that is complementary to re-using representations . We argue that pre-trained behavior can be used for both exploitation and exploration , and present techniques to achieve both goals . ( 2 ) We propose coverage as a principle for discovering behavior that is suitable for both exploitation and exploration . While coverage is naturally aligned with exploration , we show that this objective will lead to the discovery of behavior that is useful for exploitation as well . ( 3 ) We propose Coverage Pre-training for Transfer ( CPT ) , a method that implements the aforementioned hypotheses , and provide extensive experimental evaluation to support them . Our results show that leveraging the behavior of policies pre-trained to maximize coverage provides important benefits when solving downstream tasks . CPT obtains the largest gains in hard exploration games , where it almost doubles the median human normalized score achieved by our strongest baseline . Importantly , these benefits are observed even when the pre-trained policies are misaligned with the task being solved , confirming that the benefits do not come from a fortuitous alignment between our pre-training objective and the task reward . Furthermore , we show that CPT is able to leverage a single task-agnostic policy to solve multiple tasks in the same environment . 2 REINFORCEMENT LEARNING WITH UNSUPERVISED PRE-TRAINING . We follow a similar setup to that proposed by Hansen et al . ( 2020 ) . In an initial pre-training stage , agents are allowed as many interactions with the environment as needed as long as they are not exposed to task-specific rewards . Rewards are reinstated in a second stage , where the knowledge acquired during unsupervised pre-training should be leveraged in order to enable efficient learning . This is analogous to the evaluation setting for unsupervised learning methods , where pre-training on classification benchmarks with labels removed is evaluated after fine-tuning on small sets of annotated examples . The two-stage setup introduces two main challenges : defining pretext tasks in the absence of reward , and efficiently leveraging knowledge once rewards are reinstated . Our proposed method , Coverage Pre-training for Transfer ( CPT ) , relies on coverage maximization as a pretext task for task-agnostic pre-training in order to produce policies whose behavior can be leveraged for both exploitation and exploration when solving downstream tasks in the same environment . Figure 2 provides intuition about the potential benefits of CPT . 3 LEVERAGING PRE-TRAINED POLICIES . Transfer in supervised domains often exploits the fact that related tasks might be solved using similar representations . This practice deals with the data inefficiency of training large neural networks with stochastic gradient descent . However , there is an additional source of data inefficiency when training RL agents : unstructured exploration . If the agent fails at discovering reward while exploring , it will struggle even when fitting simple function approximators on top of the true state of the MDP . These two strategies are complementary , as they address different sources of inefficiency , which motivates the study of techniques for leveraging pre-trained behavior ( i.e . policies ) . Our approach relies on off-policy learning methods in order to leverage arbitrary pre-trained policies . We make use of the mapping from observations to actions of such policies ( i.e . their behavior ) , and do not transfer knowledge through pre-trained neural network weights . We consider value-based methods with experience replay that estimate action-value functions and derive greedy policies from them . The presented formulation considers a single pre-trained policy , πp , but note that it is straightforward to extend it to multiple such policies . No assumptions are made on how the pre-trained policy is obtained , and it is only used for acting . We propose using the behavior of the pre-trained policy for two complementary purposes : exploitation and exploration . Figure 2 provides intuition about the potential benefits of these two approaches on a simple environment , and pseudo-code for the proposed methods is included in Appendix A . Exploitation . When the behavior of πp is aligned with the downstream task , it can be used for zero-shot transfer . However , we are concerned with the more realistic scenario where only some of the behaviors of πp might be aligned with downstream tasks ( c.f . Figure 2 , right ) . We propose to leverage πp for exploitation by letting the agent combine primitive actions with the behavior of πp . This is achieved by considering an expanded action set A+ = A∪ { πp ( s ) } , so that the agent can fall back to πp for one step when taking the additional action . Intuitively , this new state-dependent action should enable faster convergence when the pre-trained policy discovered behaviors that are useful for the task , while letting the agent ignore it otherwise . The return of taking action a′ ∼ πp ( s ) is used as target to fit both Q ( s , πp ( s ) ) and Q ( s , a′ ) , which implements the observation that they are the same action and thus will lead to the same outcomes . Exploration . Following the pre-trained policy might bring the agent to states that are unlikely to be visited with unstructured exploration techniques such as -greedy . This property has the potential of accelerating learning even when the behavior of the pre-trained policy is not aligned with the downstream task , as it will effectively shorten the path between otherwise distant states ( Liu & Brunskill , 2018 ) . As we rely on off-policy methods that can learn from experience collected by arbitrary policies , we propose to perform temporally-extended exploration with πp , which we will refer to as flights . Inspired by z-greedy and its connection to Lévy flights ( Viswanathan et al. , 1996 ) , a class of ecological models for animal foraging , these flights are started randomly and their duration is sampled from a heavy-tailed distribution . Our proposal can be understood as a variant of z-greedy where pre-trained policies are used as exploration options . An exploratory flight might be started at any step with some probability . The duration for the flight is sampled from a heavy-tailed distribution , and control is handed over to πp during the complete flight . When not in a flight , the exploitative policy that maximizes the extrinsic reward is derived from the estimated Q-values using the -greedy operator . This ensures that all state-action pairs will be visited given enough time , as exploring only with πp does not guarantee such property . Note that this is not needed in z-greedy , which reduces to standard -greedy exploration when sampling a flight duration of one step .

Summary: In this work the author focus on transfer in RL via proposing to transfer knowledge through behavior instead of representations. They propose using coverage as an objective for the pre-training procedure. They then employ the NGU (Badia 2020) They also propose a method based on coverage pre-training for transfer and provide empirical evidence in support of their method for transfer in RL on the Atari suite.