Coverage as a Principle for Discovering Transferable Behavior in Reinforcement Learning
Authors: Víctor Campos
Pablo Sprechmann
Steven Stenberg Hansen
Andre Barreto
Charles Blundell
Alex Vitvitskyi
Steven Kapturowski
Adria Puigdomenech Badia
Venue:  (2021)
Keywords: deep reinforcement learning
transfer learning
unsupervised learning
exploration

Abstract: Designing agents that acquire knowledge autonomously and use it to solve new tasks efficiently is an important challenge in reinforcement learning. Unsupervised learning provides a useful paradigm for autonomous acquisition of task-agnostic knowledge. In supervised settings, representations discovered through unsupervised pre-training offer important benefits when transferred to downstream tasks. Given the nature of the reinforcement learning problem, we explore how to transfer knowledge through behavior instead of representations. The behavior of pre-trained policies may be used for solving the task at hand (exploitation), as well as for collecting useful data to solve the problem (exploration). We argue that pre-training policies to maximize coverage will result in behavior that is useful for both strategies. When using these policies for both exploitation and exploration, our agents discover solutions that lead to larger returns. The largest gains are generally observed in domains requiring structured exploration, including settings where the behavior of the pre-trained policies is misaligned with the downstream task.

Under review as a conference paper at ICLR 2021
COVERAGE AS A PRINCIPLE FOR DISCOVERING TRANSFERABLE BEHAVIOR IN REMNNTTCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Designing agents that acquire knowledge autonomously and use it to solve new tasks efficiently is an important challenge reinforcement learning. Unsupervised learning provides useful paradigm for autonomous acquisition of task-agnostic knowledge. In supervised settings, representations discovered through unsuper- vised pre-training offer important benefits when transferred to downstream tasks. Given the nature of the reinforcement learning problem, we explore how to transfer knowledge through behavior instead of representations. The behavior of pre-trained policies may be used for solving the task at hand (exploitation), as well as for collecting useful data to solve the problem (exploration). We argue that pre-training policies to maximize coverage will result in behavior that is useful for both strate- gies. When using these policies for both exploitation and exploration, our agents discover solutions that lead to larger returns. The largest gains are generally ob- served domains requiring structured exploration, including settings where the behavior of the pre-trained policies is misaligned with the downstream task.
INTRODUCTION
Unsupervised representation learning techniques have led unprecedented results in domains like computer vision (Hénaff et al. 2019 He et al. 2019) and natural language processing (Devlin et al. 2019: Radford et al. 2019). These methods are commonly composed of two stages an initial unsupervised phase, followed by supervised fine-tuning on downstream tasks. The self-supervised nature of the learning objective allows leverage large collections of unlabelled data in the first stage. This produces models that extract task-agnostic features that are well suited for transfer downstream tasks. reinforcement learning (RL), auxiliary representation learning objectives provide denser signals that result in data efficiency gains (Jaderberg et al.) 2017) and even bridge the gap between learning from true state and pixel observations (Laskin et al. 2020). However, RL applications have yet seen the advent of the two-stage setting where task-agnosti pre-training is followed by efficient transfer to downstream tasks. We argue that there are two reasons explaining this lag with respect their supervised counterparts. First, these methods traditionally foeus transferring representations (Lesort et al.) (018). While this is enough in supervised scenarios, we argue that leveraging pre-trained behavior far more important in RL domains requiring structured exploration. Second, what type of self-supervise objectives enable the acquisition of transferable. task-agnostic knowledge is still an open question. Defining these objectives in the RL setting is complex, as they should account for the fact that the the distribution of the input data will be defined by the behavior of the agent.
Transfer in deep learning is often performed through parameter initialization followed by fine-tuning. The most widespread procedure consists in initializing all weights in the neural network using those from the pre-trained model, and then adding an output layer with random parameters (Girshick) ctt al. 2014 Devlin et al. 2019). Depending on the amount of available data, pre-trained parameters can either be fine-tuned or kept fixed. This builds the intuition that the pre-trained model will map inputs to a feature space where the downstream task is easy to perform. this procedure will completely dismiss the pre-trained policy and fall back to random one when collecting experience. Given that complex RL problems require structured and temporally-extere behaviors, we argue that representation alone is not enough for efficient transfer in challenging
the RL setting,

---

Under review as a conference paper at ICLR 2021
montezuma_ reneeee
space_ jenveeessdess
14
12
50
40 k
30 k
20 k
2.
10
0
18
2B
5
18
3. environment steps
2B
4.8
5
environment steps
VGG
behavior
representation fine-tuning
no transfer
behavior representation
Figure I: Comparison of transfer strategies on Montezuma's 's Revenge (hard exploration) and Space In- vaders (dense reward) from task-agnostic policy pre-trained with NGU (Puigdomenech Badia et al., 2020b). Transferring representations provides significant boost on dense reward games, but does seem help in hard exploration ones. Leveraging the behavior of the pre-trained policy provides important gains in hard exploration problems when compared to standard fine-tuning and is complementary transferring representations. We refer the reader Appendix F for details on the network architecture.
domains. Pre-trained representations do indeed provide data efficiency gains in domains with dense reward signals (Finn al. 2017 Yarats et al., 2019 Stooke al., 2020a), but experiments show that the standard fine-tuning procedure falls short in hard exploration problems (c.f. Figure D. We observe this limitation even when fine-tuning the pre-trate pre-trained policy, which is aligned with findings from previous works (Finn et al.) 2017). Learning in the downstream task can lead to catastrophically forgetting the pre-trained policy, something that depends on many difficult-to-meaace the similarity between the tasks. We address the problem of leveraging arbitrary pre-trained policies when solving downstream tasks, requirement towards enabling efficient transfer in RL.
factors such as
Defining unsupervised RL objectives remains an open problem, and existing solutions are often influenced by how the acquired knowledge will be used for solving downstream tasks. Model- based approaches can learn world models from unsupervised interaction (Ha & Schmidhuber 2018). However, the diversity of the training data will impact the accuracy the model (Sekar al. 2020) and deploying this type of approach in visually complex domains like Atari remains an open problem (Hafner et al. 2019. Unsupervised RL has also been explored through the lens of empowerment (Salge et al.] 2014 [ Mohamed & Rezende (015). which studies agents that aim to discover intrinsic options (Gregor et al. 2016 Eysenbach et al. 2019). While these options can be leveraged by hierarchical agents (Florensa et al., 2017) or integrated within the universal successor features framework (Barreto et al. [007 [018) Borsa et al.) 2019 Hansen et al., 2020), their lack of coverage generally limits their applicability to complex downstream tasks (Campos al. 2020). We argue that maximizing coverage is a good objective for task-agnostic RL, as agents that succeed at this task will need develop complex behaviors order efficiently explore the environment (Kearns) & Singh 2002). This problem can be formulated as that of finding policies that induce maximally entropic state distributions, which might become extremely inefficient in nigh-dimensional state spaces without proper priors (Hazan et al. 2019 Lee et al., 2019). In practice, exploration is often encouraged through intrinsic curiosity signals that incorporate priors in order quantify how different the current state is from those already visited (Bellemare et al. 2016) Houthooft et al. 2016 Ostrovski ct al., 2017 Puigdomenech Badia et al. 2020b). Agents that maximize these novelty-seekin signals have been shown to discover useful behaviors in unsupervised settings (Pathak et al. 2017 Burda et al. 8a), but little research has been conducted towards leveraging the acquired knowledge the agent is exposed to extrinsic reward. We show that coverage-seeking objectives are good proxy for acquiring knowledge in task-agnostic settings, as leveraging the behaviors discovered in an unsupervised pre-training stage provides important gains when solving downstream tasks.
2

---

Under review as a conference paper at ICLR 2021
Our contributions can be summarized follows. (1) We study how transfer knowledge in RL through behavior by re- re-using pre-trained policies, an approach that is complementary to re-using representations. We argue that pre-trained behavior can be used for both exploitation and exploration, and present techniques to achieve both goals. (2) We propose coverage as principle for discovering behavior that is suitable for both exploitation and exploration. While coverage is naturally aligned with exploration, we show that this objective will lead to the discovery of behavior that is useful for exploitation as well. (3) We propose Coverage Pre-training, for Transfer (CPT), method that implements the aforementioned hypotheses, and provide extensive experimental evaluation to support them. Our results show that leveraging the behavior of policies pre-trained to maximize coverage provides important benefits when solving downstream tasks. CPT obtains the largest gains in hard exploration games, where it almost doubles the median human normalized score achieved by our strongest baseline. Importantly, these benefits are observed even when the pre-trained policies are misaligned with the task being solved, confirming that the benefits do not come from a fortuitous alignment between pre-traie vre-trainigg objective and the task reward. Furthermore, we show that CPT is able to leverage a single task-agnostic policy to solve multiple tasks in the same environment.
2 REINFORCEMENT LEARNING WITH UNSUPERVISED PRE- TRAINIG
We follow similar setup that pro- posed by Hansen et al. (020). In an initial pre-training stage, agents are allowed as many interactions with the environment as needed as long as they are not exposed to task-specific rewards. Rewards are reinstated in second stage, where the knowl- edge acquired during unsupervised pre-training should be leveraged in or- der to enable efficient learning. This is analogous to the evaluation setting for unsupervised learning methods, where pre-training on classification benchmarks with labels removed is evaluated after fine-tuning on small sets of annotated examples.
Unsupervised stage
Downstream task
Figure 2: Intuition behind CPT on a simple maze, where the agent needs collect treasure chests (positive reward) while avoiding skulls (negative reward). Trajectories that a policy Tp trained maximize coverage could produce are depicted in orange. Left: while Tp ignores some of the rewarding objects, many learning opportunities appear when following it during training. Right: combining primitive actions (red) with actions from тp (orange) side-steps the need learn behavior that is already available through Tp when solving downstream tasks.
The two-stage setup introduces two main challenges: defining pretext tasks the absence of reward, and efficiently leveraging knowledge once rewards are reinstated. Our proposed method, Coverage Pre-training for Transfer (CPT), relies on coverage maximization as pretext task for task-agnostic pre-training in order to produce policies whose behavior can be leveraged for both exploitation and exploration when solving downstream tasks in the same environment. Figure provides intuition about the potential benefits of CPT.
3 LEVERAGING PRE- TRAINED POLICIES
Transfer in supervised domains often exploits the fact that related tasks might be solved using similar representations. This practice deals with the data inefficiency of training large neural networks with stochastic gradient descent. However, there is an additional source of data inefficiency when training RL agents: unstructured exploration. If the agent fails at discovering reward while exploring. will struggle when fitting simple function approximators two strategies are complementary, as they address different sources of inefficiency, which motivates the study of techniques for leveraging pre-trained behavior (i.e. policies).
top of the true state of the MDP. These
3

---

Under review as a conference paper at ICLR 2021
Our approach relies on off- off-policy learning methods in order to leverage arbitrary pre- pre-trained policies. We make use of the mapping from observations to actions of such policies (i.e. their behavior), and do not transfer knowledge through pre- pre-trained neural network weights. We consider value- methods with experience replay that estimate action-value functions and derive greedy policies from them. The presented formulation considers single pre-trained policy, Tp. but note that it is straightforward to extend it to multiple such policies. No assumptions are made on how the pre-trained policy is obtained, and it is only used for acting. We propose using the behavior of the pre-trained policy for two complementary purposes: exploitation and exploration. Figure 2 provides intuition about the potential benefits of these two approaches on a simple environment, and pseudo-code for the proposed methods is included in Appendix A
When the behavior of Tp is aligned with the downstream task, it can be used for zero-shot transfer. However, we are concerned with the more realistic scenario where only some of the behaviors of x, might be aligned with downstream tasks (c.f. Figure 2 right). We propose to leverage Tpp for exploitation by letting the agent combine primitive actions with the behavior of Ty. This is achieved by considering an expanded action set A = Au (+)) so that the agent can fall back to Tp for one step when taking the additional action. Intuitively, this new state-dependent action should enable faster convergence when the pre-trained policy discovered behaviors that are useful for the task, while letting the agent ignore it otherwise. The return of taking action a' ~ 7p(ss is used as target to fit both Q(, Tp())) and ((+, a'), which implements the observation that they are the same action and thus will lead to the same outcomes.
Exploitation.
Exploration. Following the pre-trained policy might bring the agent to states that are unlikely to be visited with unstructured exploration techniques such as e-greedy This property has the potential of accelerating learning even when the behavior of the pre-trained policy is not aligned with the downstream task, as will effectively shorten the path between otherwise distant states (Liu & Brunskill 2018). As we rely on off-policy methods that can learn from experience collected by arbitrary policies, we propose to perform temporally-extender exploration with Tp. which we will refer to as flights. Inspired by <z-greedy and its connection to Lévy flights (Viswanathan et al., 1996). a class of ecological models for animal foraging, these flights are started randomly and their duration is sampled from a heavy-tailed distribution. Our proposal can be understood as variant of ez-greddy where pre-trained policies are used as exploration options. An exploratory flight might be started at any step with some probability. The duration for the flight is sampled from heavy- distribution, and control is handed over to Tp during the complete flight. When not in a flight, the exploitative policy that maximizes the extrinsic reward is derived from the estimated Q-values using the e-greedy operator. This ensures that all state-action pairs will be visited given enough time, as exploring only with up does not guarantee such property. Note that this is not needed in ez-greedy, which reduces to standard e-greedy exploration when sampling a flight duration of one step.
4 COVERAGE AS A GOAL FOR UNSUPERVISED
So far we considered strategies for leveraging the behavior of arbitrary policies, and we now discuss how to train such policies in initial pre-training stage with rewards removed. In such setting, it is common practice to derive objectives for proxy tasks in order to drive learning. As we proposed to take advantage of pre-trained policies for both exploitation and exploration, it might seem unlikely that a single pre-training objective will produce policies that are useful for both purposes. However, we hypothesize that there exists single criterion that will produce policies that can be used for both exploration and exploitation: coverage. This objective aims at visiting as many states as possible and is naturally aligned with exploration (Kearns & Singh| 2002). Long episodes where the agent visits as many different states as possible result in high returns in some domains such as videogames, locomotion and navigation (Pathak et al.] 2017]] Burda et al. 2018a). We argue that pre-training for coverage will bring benefits beyond these particular domains, as it fosters mastery over the environment. This leads to the discovery of skills and behaviors that can be exploited by the agent when solving downstream tasks even if the pre-trained policy does not obtain high returns.
Policies that maximize coverage should visit as many states as possible within a single episode, which differs from traditional exploration strategies employed when solving single task. The goal of the latter is discovering potentially rewarding states, and the drive for exploration fades as strategies that lead to high returns discovered. The proposed objective is closely related
4

---

Under review as a conference paper at ICLR 2021
to methods for task-agnostic exploration that train policies that induce maximally entropic state visitation distributions (Hazan et al. 2019; Lee et al.) 2019). However, since the problems we are interested in involve large state spaces where states rarely visited more than once, instead propose to consider only the controllable aspects of the state space. This enables disentangling observations from states and gives rise to scalable, and thus more easily covered, notion of the state space.
We choose Never Give Up (NGU) (Puigdomenech Badia al. 2020b means for training policies that maximize coverage. NGU defines an intrinsic reward that combines per-episode and life-long novelty over controllable aspects the state space. It can be derived directly from observations, unlike other approaches that make use of privileged information (Conti et al. 2018)) or require estimating state visitation distributions (Hazan al. 2019), making it suitable for environments that involve high- dimensional observations and partial observability. The intrinsic NGU reward maintains exploration throughout the entire training process, property that makes suitable for driving learning task- agnostic settings. This contrasts with other intrinsic reward signals, that generally vanish training progresses (Ecoffet et al. 2019). NGU was originally designed to solve hard-exploration problems by learning& family of policies with different degrees of exploratory behavior. Thanks to weight sharing, the knowledge discovered by exploratory policies enabled positive transfer to exploitative ones, obtaining impressive results when applied to large-scale domains (Puigdomènech Badia et al. (2020a). We instead propose to use NGU as a pre-training strategy in the absence of reward, transferring knowledge downstream tasks the form of behavior rather than weight sharing.
CPT: COVERAGE PRE-TRAINING FOR TRANSFER
CPT consists of two stages: (1) pre-training a task aask-agnottic policy using the intrinsic NGU reward, and (2) solving downstream tasks in the same environment by leveraging the pre-trained behavior.
Coverage pre-training
The agent interacts with a Markov Decision Process (MDP) defined by the tuple (S,A, INGU. 7), with S being the state space, AA being the action space, P the state- transition distribution, 7 E (0, 1 the discount factor and the reward function TNGU the intrinsic reward used in NGU (Puigdomênech Badia et al. 2020b).
We use a value-based agent with Q-function, QNNGU (s, (s,a): : + E, parameterised
with neural network defined in Appendix IP
We train QGU to maximise the NGU intrinsic reward, obtaining a deterministic policy
given by (s) arg max maax[(NGU(s.... a)).
We use e-greedy as behavioral policy when interacting with the environment.
Transfer
We are given now new MDP given by (S,A, P,r. where the only change with respect to the pre-training stage is new extrinsic reward function S x A - R.
We define Q-function Q" (s, a) S x A' -> I on an extended action set A'' = A U a'.
When the agent selects action a', it executes the action given by the pre-trained policy:
arr max, [QT a)] if arg max, (QT m a)) + a' if arg max (s. a)) =
((s) = (s)
We parameterise Q" using a neural network with random initialization.
We use Lévy flights as behavioral policy when interacting with the environment. See Algorithm for details.
6 EXPERIMENTS
We evaluate CPT in the Atari suite (Bellemare et al. 2013), a benchmark that presents a variety of challenges and is often used to measure the competence of agents. All our experiments are run using the distributed R2D2 agent (Kapturowski et al., 2019). A detailed description of the full distributed setting is provided in Appendix]] We use the same hyperparameters as in Agent57 (Puig domènech Badia et al. (220a),, which are reported in Appendix B All reported results are the average over three random seeds.
5

---

Under review as a conference paper at ICLR 2021
6.1 UNSUPERVISED STAGE
Unsupervised RL methods are often evaluated by measuring the amount of task reward collected by the discovered policies (Burda et al., 2018a Hansen et al. 2020), and we use this metric to evaluate the quality of our unsupervised policies. We pre-train our agents using 16B frames in order to guarantee the discovery of meaningful exploration policies] as it is common to agents interact with the environment for as long needed in this unsupervised stage (Hansen et al.) 2020).
We compare the results of our unsupervised pre- pre-training state against other unsupervised approaches, standard RL algorithms in the low-data regime and methods that perform unsupervised pre-training followed by an adaptation stage. We select some of the top performing methods in the literature, and refer the reader to Appendix for more extensive list of baselines. Since the NGU reward is non-negative, we consider baseline where the agent obtains a constant positive reward at each step in order measure the performance of policies that seek stay alive for as long as possible. Table T shows that unsupervised CPT outperforms all baselines by a large margin, confirming the intuition that coverage is good pre-training objective for the Atari benchmark. These results suggest that there is a strong correlation between exploration and the goals established by game designers (Burda) al. 2018a). In spite of the strong results, it is worth noting that unsupervised CPT achieves lower scores than random policies in some games, and it is quite inefficient at collecting rewards in some environments (e.g. needs long episodes obtain high scores). These observations motivate the development of techniques to leverage these pre-trained policies without compromising performance even when there exists a misalignment between objectives.
Table I: Atari Suite comparisons. @ represents the amount of RL interaction with reward utilized, with four frames observed at each iteration. Mdn and M are median and mean human normalized scores, respectively; > 0 is the number of games with better than random performance; and >> H is the number of games with human-leve level performance as defined in Mnih et al. (2013). Top: unsupervised learning only. Mid: data- lititediedd RL. Bottom: RL with unsupervised pre- training.
26 Game Subset
Full 57 Games
Kaiser et al. (2019)
Mnih et al. (2015
Algorithm
Mdn
M
>0
>
H Mdn
>0
H
Positive Reward R2D2 @0
9.44 59.55
21
4
3.46
45.23
46
5
VISR @0 (Hansen et al. 2020
5.60
81.65
19
5 3.77
49.66
40
7
CPT 60
80.92 494.54
25
12
320.06
52
27
SimPLe @ 110kk (Kaiser et al.
2019)
9.79
36.20
26
4
DQN @:00M (Mih et al. 2015)
100.76 267.51
26
13
80.81 239.29
46
20
GPI VISR @10kk (Hansen et al.
2020
6.59
111.23
22
8.99
109.16
44
12
6.2 LEVERAGING PRE-TRAINED POLICIES
We now evaluate the proposed strategies for leveraging pre-trained policies once the reward function reinstated by training R2D2-based agents (Kapturowski et al. 2019) for 5B frames. This is a relatively small budget for these distributed agents with hundreds of actors (Puigdomènech Badia ct al.) 2020a). We compare the proposed method against e-greedy and ez-greedy (Dabney et al. (2200) exploration strategies. Policies the average return over the last 300 evaluation episodes. Table 2 results in full Atari suite, which confirm the benefits of leveraging the behavior of policy trained to maximize coverage. Our approach is most beneficial in the set of hard exploration games where unstructured exploration generally precludes the discovery of high-performing policies.
evaluated using five parallel evaluator threads, and report
It should be noted that our ez-greedy ablation under-performs relative to Dabney et al. (2020). This due to our hyper-parameters and setting being derived from Puigdomenech Badia al. (2020b). which adopts the standard Atari pre-processing (e.g. gray scale images and frame stacking). In
1 'The pre-training budget was not tuned, but we observe that competitive policies arise early in training. This observation suggests that smaller hudgets are feasible as well. montezzume revenge,, pi fall private_eyee venture, gravi tar, sclaris

---

Under review as a conference paper at ICLR 2021
contrast, Dabney et al. (2020) color images, frame stacking, larger neural network and different hyper- parameterssssss (e.g. smaller replay buffer). Studying if the performance of both NGU and the CPT is preserved in this setting is important direction for future work. We suspect that improving the performance of our ez-greedy ablation will also improve our method, since exploration flights are central to both.
Table 2: Atari Suite comparisons for R2D2-based agents. @.V represents the amount of RL interaction with reward utilized, with four frames observed at each iteration. Mdn, M and CM are median, mean and mean capped human normalized scores, respectively.
Hard Exploration
Full 57 Games
Algorithm
Mdn
M
CM
Mdn
M
CM
e-greeyy explore @5B
32.54
67.18
44.75
487.25
1753.81
90.32
ez-greedy explore @5B
104.08
95.00
67.87
438.81
1263.83
92.53
CPT (unsup)
4.31
22.62
22.62
83.22
318.78
58.50
CPT @5B
191.04
158.05
76.92
561.98
2184.26
93.20
Ablation studies. We run experiments on a sub- set of games in order to gain insight on the in- dividual contribution of each of the proposed ways of leveraging the pre-trained policy. The subset composed by 12 game obtained by combining those used to tune hyperparameters
Median normalized score
e-greedy explore ex-greedy explore
196.98
fine-tuning
266.02
by Hansen et al. (2020) with games where ez- greedy provides clear gains over e-greedy as per [Dabnyy et al. (2020). This results in a set of games that require different amounts of ex- ploration, and fcaturing both dense and sparse rewards. Figure shows that both strategies ob- tain similar median scores across the 12 games, but combining them results in an important per- formance gain. This suggests that the gains they provide are complementary, and both are respon- sible for the strong performance of CPT. Note that CPT also outperforms fine-tuning baseline, where the policy is initialized using the pre-trained weights rather than random ones. We believe that the benefits of both approaches can be combined by training via CPT policy initialized with pre-trained weights.
CPT (unsup)
CPT (exploitation only) CPT (exploration only)
224.57
2224.66
CPT
362.88
50 100 150 200 250 300 350
Figure Ablation results. Using the task-agnostic policy for exploitation and exploration seems to provide complementary benefits, as combining the two techniques results in important gains.
Effect of the pre-trained policy. The behavior of the pre-trained policy will likely have strong impact on the final performance of agents. We consider the amount of pre-training as proxy for the exploration capabilities of the task-agnostic policies. Intuitively, policies trained for longer time spans will develop more complex behaviors that enable visiting larger number of states. Figure 4 reports the end performance of agents after before and after transfer under different lengths of the pre-training phase, and shows how it has a different impact depending on the nature of the task. Montezuma's 's Revenge requires structured exploration for efficient learning, and longer pre-training times provide dramatic improvements in the end performance. Note that these improvements do not correlate with the task performance of the task-agnostic policy, which suggests that gains are due to a more efficient exploration of the state space. On the other hand, the final score in Pong is independent of the amount of pre- tre-training. Simple exploration is enough to discover optimal policies, so the behaviors discovered by the unsupervised policy do not play an important role in this game.
Transfer to multiple tasks. An appealing property task-agnostic knowledge is that it can be leveraged solve multiple tasks. In the RL setting, this be evaluated by leveraging single task-agnostic policy for solving multiple tasks (i.e. reward functions) in the same environment. We evaluate whether the unsupervised NGU policies can be useful beyond the standard Atari tasks by creating two alternative versions of Ms Pacman and Hero with different levels of difficulty. The
Pasterix, bank_heist, frostbite, gravitar, jamesbond, montezuma_revenne
ms_ ms_pacman, pong, private_eye, space_invaders invaders, tennis,
7

---

Under review as a conference paper at ICLR 2021
montezuma revenge
pong
10000
10
-10
8.84
1.88
3.84
6.48
CPT (unsup)
-4- CPT
Figure 4: Effect of the pre-training budget, before and after adaptation, on Montezuma's Re- venge (hard exploration) and Pong (dense reward).
goal in the modified version of Ms Pacman is to eat vulnerable ghosts, with pac-dots giving 0 (easy version) or -10 (hard version) points. In the modified version of Hero, saving miners gives a fixed return of 1000 points and dynamiting walls gives either (easy version) or -300 (hard version) points. The rest of rewards are removed, e.g. eating fruit in Ms Pacman or the bonus for unused power units in Hero. Note that even in the easy version of the games exploration is harder than in the original counterparts, as there are no small rewards guiding the agent towards its goals. In the hard version of the games exploration is even more challenging. as the intermediate rewards work as deceptive signal that takes the agent away from its actual goal. In this case finding rewarding behaviors requires stronger commitment to an exploration strategy. In this setting, the exploratory policies often achieve very low even negative rewards, which contrasts with the strong performance they showed when evaluated under the standard game reward. Even in this adversarial scenario, results in Figure Ssows shows that leveraging pre-trained exploration policies provides important gains. These results suggest that the strong performance observed under the standard game rewards is not due to an alignment between the NGU reward and the game goals, but due to efficient usage of pre-trained exploration policies.
MsPacman
Eating ghosts (easy)
Eating ghosts (hard)
3033
45231
401
8350
8350
144
10983
7060
2000
1030
6000
1000
18000
0
2000
1000
6000
8000
2000
4000
6000
8000
Hero
Saving miners (easy)
Saving miners (hard)
43761
2000
201
1
3000
3000
700
$297
4267/ 4
4.63
10000
20000
30200
40000
1000
2000
3000
4000
-1000
1000
2000
3000
E-greedy explore
ez-greedy explore
Fine-tuning
CPT (unsup)
CPT
Figure 5; Final scores per task in the Atari games of Ms Pacman (top) and Hero (bottom) with modified reward functions. We train a single aank-sge policy per environment, and leverage it to solve three different tasks: the standard game reward, task with sparse rewards (easy), and variant of the same task with deceptive rewards (hard). Despite the pre-trained policy might obtain low or even negative scores in some of the tasks, committing its exploratory behavior eventually lets the agent discover strategies that lead to high returns.
Towards the low-data regime. So far we considered R2D2-based agents tuned for end-performance on massively distributed setups. Some applications might require higher efficiency in the low-data regime, even if this comes at the cost of a drop in end performance. The data efficiency of our method can be boosted by reusing representations from the pre-trained convolutional torso in the NGU policy (c.f. Figure/ofoo for details on the architecture), as shown in Figure We observe that the data efficiency can be boosted further by decreasing the number parallel actors. Figure i0 the appendix showcases the improved data efficiency Montezuma's 's Revenge when using 16 actors (instead of 256 as in previous experiments), obtaining superhuman scores in less than 50M frames. We note that this is around two times faster than the best results in the benchmark by Taiga et al. (2019), even though they consider single-threaded Rainbow- based agents (Hessel et al. 2018 that were designed for data efficiency.
8

---

Under review as a conference paper at ICLR 2021
7 RELATED WORK
Our work uses the experimental methodology presented in Hansen et al. (2020). But whereas that work only considered simplified adaptation process that limited the final performance on the downstream task, the focus here is on the more general case of using a previously trained policy to aid in solving the full reinforcement learning problem. Specifically, VISR uses successor features to identify which of the pre-trained tasks best matches the true reward structure, which has previously been shown to work well for multi-task transfer (Barreto et al., 2018).
Gupta al. (2018) provides an alternative method to meta-learn a solver for reinforcement learn- ing problems from unsupervised reward functions. This method utilizes gradient-based meta- learning (Finn et al. 2017), which makes the adaptation process standard reinforcement learning updates. This means that even if the downstream reward is far outside of the training distribution, final performance would not necessarily be affected. However, these methods are hard to scale to the larger networks considered here, and followup work (Jabri et al. 2019) changed to memory-based meta-learning (Duan et al.) 2016) which relies on information about rewards staying in the recurrent state. This makes it unsuitable to the sort of hard exploration problem our method excels at. Recent work has shown success in transferring representations learned in an unsupervised setting to rein- forcement learning tasks (Stooke et al., 2020b). Our representation transfer experiments suggest that this should handicap final performance. but the possibility also exists that different unsupervised objectives should be used for representation transfer and policy transfer.
Concurrent work by Bagot et al. (2020) also augments an agent with the ability to utilize another policy. However, their work treats the unsupervised policy as an option, only callable for an extended duration. In contrast, we only perform extended calls to the unsupervised policy during exploratory levy flights and augment the action space to allow for single time-step calls. This difference between exploratory and exploitative calls to the unsupervised policy in critical to overall performance, as illustrated in Figure B] In addition, in Bagot al. (2020) the unsupervised policy is learned in tandem based on an intrinsic reward function. This is promising direction which is complementary to our work, it handles the case wherein there is no unsupervised pre-training phase. However. their work only considers tabular domains, so it is unclear how this approach would fair in the high-dimensional state spaces considered here.
8 DISCUSSION
We studied the problem of transferring pre-trained behavior in reinforcement learning, an approach that is complementary to the common practice of transferring representations. Depending on the behavior of the pre-trained policies, we argued that they might be useful for exploitation, exploration, or both. We proposed methods to make use of pre-trained behavior for both purposes: exploiting with the pre-trained policy by making it available to the agent an extra action, and performing temporally-extended exploration with it. While we make no assumption on the nature of the pre- trained policies, this raises the question of how to discover behaviors that are suitable for transfer. We proposed coverage as a principle for pre-training task task-agnostic policies that are suitable for both exploitation and exploration. We chose NGU in experiments for its scalability, but note that our approach could be combined with any other strategy for maximizing coverage. We found that unsupervised training with this objective produces strong performing policies in the Atari suite, likely due to the way in which the goals in some of these tasks were designed (Burda et al. 2018a Sa). Our transfer experiments demonstrate that these pre-trained policies can be used to boost the performance of agents trained to maximize reward, providing the most important gains in hard exploration tasks. These benefits are not due to an alignment between our pre-training and downstream tasks, as we also observed positive transfer in games where the pre-trained policy obtained low scores. In order to provide further evidence for this claim, we designed alternative tasks for Atari games involving hard exploration and deceptive rewards. Our transfer strategy outperformed all considered baselines in these settings, even when the pre-trained policy obtained very low or even negative scores, demonstrating the generality of the method. Besides disambiguating the role of the alignment between pre- training and downstream tasks, these experiments demonstrate the utility of a single task-agnostic policy for solving multiple tasks in the same environment.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Louis Bagot, Kevin Mets, and Steven Latré. Learning intrinsically motivated options to stimulate policy exploration. 2020.
André Barreto, Will Dabney, Rémi Munos. Jonathan J Hunt, Tom Schaul, Hado van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. In NeurIPS, 2017.
Andre Barreto, Diana Borsa, John Quan, Tom Schaul, David Silver. Matteo Hessel, Daniel Mankowitz, Augustin Zidek, and Remi Munos. Transfer in deep reinforcement learning using successor features and generalised policy improvement. In ICML, 2018.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In NeurIPS, 2016.
Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning envi- ronment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 2013.
Diana Borsa, André Barreto, John Quan, Daniel Mankowitz, Rémi Munos, Hado van Hasselt, David Silver, and Tom Schaul. Universal successor features approximators. In ICLR, 2019.
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros. Large-scale study of curiosity-drives learning. arXiv preprint arXiv:1808.04333 2018a.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprini arXiv: 1810::88100/8/// /294, 2018b.
Víctor Campos, Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giro-i Nieto, and Jordi Torres. Explore, discover and learn: Unsupervised discovery of state-covering skills. In ICML, 2020.
Edoardo Conti, Vashisht Madhavan, Felipe Petroski Such, Joel Lehman, Kenneth Stanley, and Jeff Clune. Improving exploration in evolution strategies for deep reinforcement learning via population of novelty-seeking agents. In NeurIPS, 2018.
Will Dabney, Georg Ostrovski, and André Barreto. Temporally e-greedy exploration. arXiv preprint arXiv:2006.0/782, 2020.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep
bidirectional transformers for language understanding. In NAACI, 2019.
Yan Duan, John Schulman, Xi Chen, Peter Bartlett, Ilya Sutskever, and Pieter Abbeel. RL3: Fast reinforcement learning via slow reinforcement learning. arXiv preprint trXiv:1611.02773 2016.
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Go-explore: new approach for hard-exploration problems. arXiv preprint 1XXv:1901.11095, 2019.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. In ICLR, 2019.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. arXiv preprint arXiv: 1703.v/000 2017.
Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic neural networks for hierarchical reinforce- ment learning. In ICLR, 2017.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint urXiv:1611.07000 2016.
Abhishek Gupta, Benjamin Eysenbach, Chelsea Finn, and Sergey Levine. Unsupervised meta-learning for reinforcement learning. arXiv preprint a 04640, 2018.
10

---

Under review as a conference paper at ICLR 2021
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In NewrlPS, 2018.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In ICLR, 2019.
Steven Hansen, Will Dabney, Andre Barreto, Tom Van de Wiele, David Warde-Farley, and Volodymyr Mnih. Fast task inference with variational intrinsic successor features. In ICLR, 2020.
Elad Hazan, Sham M Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy exploration. In ICML, 2019.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. arXiv preprint arXiv:1911.05722, 2019.
Olivier J Hénaff, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding. arXiv preprini arXiv: 1905.0990 2019.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In AAAI, 2018.
Sepp Hochreiter and Jtirgen Schmidhuber. Long short-term memory. Neural computation, 1997.
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. NeurIPS, 2016.
Allan Jabri, Kyle Hsu, Abhishek Gupta, Ben Eysenbach, Sergey Levine, and Chelsea Finn. Unsu- pervised eurricula for visual meta-reinforcemmnt learning. In Advances in Neural Information Processing Systems, pp. 10519-10531, 2019.
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel 7. Leibo, David Silver, and Koray Kavukeuoglu. Reinforcement learning with unsupervised auxiliary tasks. In ICLR, 2017.
Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Camphell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model-based reinforcement learning for atari. arXiv preprint arXiv: (tt:::9903000744, 2019.
Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney. Recurrent experience replay in distributed reinforcement learning. In ICLR, 2019.
Michael Kearns and Satinder Singh. Near-optimal reinforcement learning in polynomial time.
Machine learning, 2002.
Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In ICML, 2020.
Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Ruslan Salakhutdinov. Efficient exploration via state marginal matching. arXiv preprint arXiv: (0Xx::/906605274, 2019.
Timothée Lesort, Natalia Dodr-Rodr,,ee,, Jean-Franois Goudou, and David Filliat, State representa- tion learning for control: An overview. Neural Networks, 2018.
Yao Liu and Emma Brunskill. When simple exploration is sample efficient: Identifying sufficient conditions for random exploration to yield pac algorithms. arXiv preprint a /ati::/80..00000 1805.09045, 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare. Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 2015.
Shakir Mohamed and Danilo Jimenez Rezende. Variational information maximisation for intrinsically motivated reinforcement learning. In NeurlPS, 2015.

---

Under review as a conference paper at ICLR 2021
Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Mare Bellemare. Safe and efficient off- off-policy reinforcement learning. In NeurIPS, 2016.
Georg Ostrovski, Marc G Bellemare, Aaron van den Oord, and Rémi Munos, Count-based exploration with neural density models. arXiv preprint arXiv:1703.013/0 2017.
Deepak Pathak, Pulkit Agrawal, Alexei Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In ICML,
Jing Peng and Ronald J Williams. Incremental multi-step q-learning. In Machine Learning Proceed- ings 1994. Elsevier, 1994.
Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human benchmark. In ICMI., 2020a.
Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martín Arjovsky, Alexander Pritzel, Andew Bolt, al. Never give up: Learning directed exploration strategies. In ICLR, 2020b.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 2019.
Christoph Salge, Cornelius Glackin, and Daniel Polani. Empowerment- - an introduction. In Guided Self-Organization:: Inception. Springer, 2014.
Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. In ICML, 2020.
Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. arXiv preprint arXiv:2009.08319, 2020a.
Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. arXiv preprint 4rXiv:2009.083/9, 2020b.
Adrien Ali Taïga, William Fedus, Marlos C Machado, Aaron Courville, and Mare Bellemare. Benchmarking bonus- exploration methods on the arcade learning environment. arXiv preprint arXiv:1908$ 02388, 2019.
Gandhimohan M Viswanathan, V Afanasyev, SV Buldyrev, EJ Murphy, PA Prince, and Eugene Stanley. Lévy flight search patterns of wandering albatrosses. Nature, 1996.
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In ICML, 2016.
Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Im- proving sample efficiency in model-free reinforcement learning from images. arXiv preprint 11000:191000/741, 2019.
12

---

Under review as a conference paper at ICLR 2021
A PSEUDO-CODE
Algorithm [ provides pseudo-code for the flight logic that controls how the pre-trained policy is used for exploration purposes. each step, a flight started with probability Elevy. The duration for the flight is sampled from heavy-tailed distribution, Rddiat, similarly to ez-greedy (c.f. Appendix [B for more details). When not in a flight, the exploitative policy that maximizes the extrinsic reward derived from the estimated Q-values using the e-greedy operator. This ensures that all state-action pairs will be visited given enough time, as exploring only with тp does not guarantee such property. Note that this is not needed in éz-greedy, which reduces to standard e-greedy exploration when the sampled flight duration equals
Algorithm 2 provides pseudo-code for the actor logic when using the augmented action set, A+ A U {.)) derives an e-greedy policy over |L|| + actions, where the (IAA) + 1)-th action is resolved by sampling from (s).
Finally, Algorithm 3 provides pseudo-code for the actor in the full CPT method that combines Algorithms and
13

---

Under review as a conference paper at ICLR 2021
Algorithm 1: Actor pseudo-code for CPT (exploration only)
Input: Q-value estimate for the current policy, Q* a)
Input: Pre-trained policy, Tp
Input: Probability of starting a flight, Gevy
Input: Flight length distribution, dist while True do
// flight length
while episode not ended do
Observe state
if n 0 and random() S Stovy then
n Nisstt
end
n
> 0 then
12 a - (8())
// explore with "p
else end
Take action
end
end
Algorithm Actor pseudo-code for CPT (exploitation only)
Input: Action set A
Input: Pre-trained policy, Tp
Input: Q-value estimate for the current policy, Q* (*((s AU (mp(s))
Input: Probability taking an exploratory action, c
while True do
while episode not ended do
Observe state s if random() < c then a Uniform(1, |A| 1)
else
a arg max (Q a)] end
if a =
|A| + then 7p (s)
// exploit with 4p
end
Take action
end
end
14

---

Under review as a conference paper at ICLR 2021
Algorithm 3: Actor pseudo-code for CPT
Input: Action AA
Input: Pre-trained policy, "p
Input: Q-value estimate for the current poliey, (* a) Va E AU (Tp(s))
Input: Probability of taking an exploratory action,
Input: Probability starting a flight, Stovy
Input: Flight length distribution, 7ldist while True 0
// flight length
while episode not ended do Observe state
if v = and random() S Elevy then n2 Rdist
end
if 0 then
n n a (s)
// explore with xp
else
if random() S c then Uniform(1. 1, |A| 1)
else
a arg max (2" (s, a)] end if = A + then (s)
// exploit with "
end end
Take action
end
end
15

---

Under review as a conference paper at ICLR 2021
B HYPERPARAMETERS
Table summarizes the main hyperparameters our method. The pre-trained policies were optimized using Retrace (Munos al. 2016). Transfer was performed with Peng's Q(A) (Peng & Williams 1994) instead, which we found to be much more data efficient in our experiments. The reason for this difference is that the benefits of Q(A) were observed once unsupervised policies had been trained on all Atari games, and we suspect that the data efficiency gains will transfer to the pre- vre-training stage as well.
Table 3: Hyperparameter values used R2D2-based agents. The rest of hyperparameters use the values reported by Kapturowski al. (2019).
Hyperparameter
Value
Number of actors
256
Actor parameter update interval
400 environment steps
Sequence length
160 (without burn-in)
Replay buffer size
12.5 10a part-overlapping sequences
Priority exponent
0.9
Importance sampling exponent
0
Learning rule (downstream tasks)
Q(A). = 0.7
Learning rule (NGU pre-training)
Retrace (), À 0.95
Discount (downstream tasks)
0.99
Discount (NGU pre-training)
0.97
Minibatch size
64
Optimizer
Adam
Optimizer settings
10 B 0.9, 82 0.999
Learning rate
2 10-4
Target network update interval
1500 updates
Gevy distribution
Log-Uniform(0. 0.1]
Flight length distribution
Zeta with 2
16

---

Under review as a conference paper at ICLR 2021
C EXTENDED UNSUPERVISED RL RESULTS
Table 4 compares unsupervised CPT with all the methods reported by Hansen al. (2020).
Table 4: Atari Suite comparisons, adapted from Hansen et al. (2020). @ N represents the amount of RL interaction with reward utilized, with four frames observed at each iteration. Mdn and M median and mean human normalized scores, respectively; > 0 is the number of games with better than random performance; and H the number of games with human-level performance as defined in Mnih et al. (2015). Top: unsupervised learning only. Mid: data-limited RL. Bottom: RL with unsupervised pre-training.
26 Game Subset
47 Game Subset
Full 57 Games
Kaiser et al. (2019)
Burda et al.
20188
Mnih et al. 2015
Algorithm
Mdn
M >0
H
Mdn
M
>0
>
H
Mdn
M >0
>> H
IDF Curiosity @0
8.46
24.51
34
5
RF Curiosity @0
7.32
29.03
36
6
Pos Reward NSQ @0
2.18
50.33
14
5
0.69
57.65
26
8
0.29
41.19
28
8
Pos Reward R2D2 @O
9.44
59.55
21
4
14.16
57.53
39
5
3.46
45.23
46
5
Q-DIAYN-5 @O
0.17
-3.60
13
0
0.33
-1.23
25
2
0.34
-2.18
30
2
Q-DIAYN-50 @0
-
-1.65
21.77
4
0
0-6.69
-
16.26
8
0
-3116
20.31
9
0
VISR @o
5.60
81.65
5
4.04
58.47
35
7
3.77
49.66
40
7
CPT @0
80.92
494.54
25
12
96.10
310.27
45
23
81.72
320.06
52
27
SimPLe @100k
9.79
36.20
26
4
DQN @10M
27.80
52.95
25
7
9.91
28.07
41
7
8.61
27
27.55
48
7
DQN @
100.76
267.51
26
13
80.81
239.29
46
20
Rainbow @100k
2.23
10.12
25
1
PPO @500k
20.93
43.74
25
7
NSQ @10M
8.20
33.80
22
3
7.29
29.47
37
4
6.80
28.51
43
5
Q-DIAYN-5 @100k
0.01
16.94
13
2
1.31
19.64
28
6
1.55
16.65
33
6
Q-DIAYN-50 @100k
11.4
27.88
3
0
0-166
16.74
8
0
2.53
244.13
9
0
RF VISR @100k
7.24
58.23
20
6
3.81
42.60
33
9
2.16
35.29
39
9
VISR 110kk
9.50
128.07
21
7
9.42
121.08
35
11
6.81
102.31
40
11
GPI RF VISR @100k
5.55
58.77
20
5
4.24
48.38
34
9
3.60
40.01
40
10
GPI VISR @100k:
6.59
111.23
22
7
11.70
129.76
38
12
8.99
109.16
44
12
D EXTENDED ATARI57 RESULTS
Table 5: Atari Suite comparisons for R2D2-based agents. @NV represents the amount of RL interaction with reward utilized, with four frames observed at each iteration. Mdn, M and CM are median, mean and mean capped human normalized scores, respectively.
Hard Exploration
Full 57 Games
Algorithm
Mdn
M
CM
Mdn
M
CM
e-greedy explore @1B
31.07
39.40
34.75
229.75
864.69
84.56
<z-greedy explore @1
42.55
53.90
46.21
204.52
578.73
85.11
CPT @1B
100.89
94.20
63.95
273.49
1517.13
86.38
e-greedy explore @5B
32.54
67.18
44.75
487.25
1753.81
90.32
cz-greedy explore @5B
104.08
95.00
67.87
438.81
1263.83
92.53
CPT @5B
191.04
158.05
76.92
561.98
2184.26
93.20
17

---

Under review as a conference paper at ICLR 2021
E
ALTERNATIVE REWARD FUNCTIONS
MsPacman: eating ghosts
Pac-dots: points (easy) or 10 points (hard)
Eating vulnerable ghosts:
I in succession: 200 points
#2 in succession: 400 points
#3 in succession: 800 points
#4 in succession: 1600 points
Other actions: 0 points
Hero: rescuing miners
Dynamiting walls: 0 points (easy) or -300 points (hard)
Rescuing miner: 1000 points
Other actions: 0 points
F Q-NETWORK ARCHITECTURE
All policies use the same Q-Network architecture as Agent57 (Puigdomènech Badia et al. 2020a).
which composed by convolutional torso followed by an LSTM (Hochreiter & Schmidhuberi 1997) and dueling head (Wang et al. 2016). When leveraging the behavior of the pre-trained policy solve tasks, increased efficiency (c.f. Figure 6). Shared weights are kept fixed in order to preserve the behavior of the pre-trained policy.
train new policy from scratch share some of the components for
Pre-trained policy
New policy
Pre-trained policy
New policy
(ss,a)
(s,a)
((s.a)
Q (s(a.)
Head,
Head
Head
Head
LSTM
LSTM
LSTM
LSTM
CNN,
CNN.
CNN,
Trainable
Frozen
Figure 6: Q-Network architecture for the reinforcement learning stage. The pre-trained policy can be leveraged without transferring representations (left), but sharing weights generally provides efficiency gains early in training (right).
18

---

Under review as a conference paper at ICLR 2021
G
SCORES PER GAME
Table Results per game at training frames.
Game
é-greedy explore
e2-greedy explore
CPT
alien
10831.17 2114.29
1463-102 1109.15
15657.57 1717.96
amidar
11761.67 1560.86
6784128 718.05
10391.96 891.60
assault
15940.72 3531.69
9177.28 2170.26
15060.31 740.63
asterix
472812:21 222663.81
374966.62
135810.51
630663.91
82753.48
asteroids
45716.28 + 3642.38
147005.85
44313.45
31957.42 13540.09
atlantis
1514724.43 10941.36
1132188.04
43551.36
1491384.233 5978.05
heist
965.63 133.72
1058.75 135.46
13913.32 3529.15
battle zone
292553.41 18196.77
312367.
43554.18
258533.57
22865.64
heam rider
18472.45 1977.78
22403.95
1596.92
16301.02 1853.73
herzerk
12343.83 3331.54
3846.56 1723.24
8359.80 201.10
bowling
141.64 + 4.52
156.32 8.11
174.27 0.10
boxing
99.96 + 0.03
99.94 0.06
100.00 = 0.00
breakout
432.65 + 27.35
393.19 +: 35.12
441.21 + 15.08
centipede
189302.66 66 + 31388.08
358841 20 73578. 20
178635.17 + 17227.
chopper command
61 1111313.11 65206.69
697655
215090.74
573055.88 + 75343.57
crazy climber
229992.57 + 17738.33
212001 .6 1853.07
226821 16 3608.16
defender
547238.15 15 2579.38
216522.06 + 11909.59
540124.74 4488.40
demon attack
143662.42 88.16
141352. 18 + 3848.73
143762.91 + 106. 75
double dunk
23.99 0.02
23.88 0.06
23.85 + 0.15
enduro
2358.37 + 3.32
2359.08 1.03
3361.56-2 = 1.03
fishing derhy
12.80 + 77.79
64.74 + 0.59
52.58 0.32
freeway
33.87 0.08
33.77 0.03
33.79 0.08
frostbite
9287.24 224: + 167.11
8504.41 = 940.72
17692.42 = 2871.83
gopher
117398.58 58 2485.82 82
84140.40 + 12919.83
113716.78 + 3966.91
gravitar
6123.08 08 1: 103.19
5798.68 = 735.59
8373.70
1260.75
hero
46048.07 6970.26
3970X122 4379.84
40825.09 3736..25
hockey
32.43 30.64
30.65 28.17
60.36 4.94
jamesbond
6056.14 1643.52
3843.92 118.35
1484.87 489.66
kangaroo
14672.37 187.16
14730.99 = 114.20
15965. 36.61
krull
10081.04 594.10
10171.52 = 399.81
406596.00
55547.76
kung fu master
200721.644 2265.35
171591.29 8516.87
196638 456.09
montezuma revenge
1478.38 1114.20
1467.77 1104.72
12086. + 1217.76
ms pacman
11212.85 103.23
7511.39 + 406.77
10996. 90 262.74
name this game
32138.12 + 2156.95
37343.04 1917.73
30252. 11 884.84
phoenix
712101.72 + 62738.09
80611.18 + 25316.56
553429.34 + 24278.
pitfall
0.19 1.99:000 0.15
122.34 4.20
0.39 0.39
pong
20.93 0.01
20.49 0.10
20.90 0.01
private eye
23592.22 22 1 11876.55
50770.82 8. : 14984.92
40435.54 + 51.04
qbert
24343. 75 + 1904.89
16975.13 13 + 1332. 44
16057 .3 318.87
riverraid
32325.00 07 + 1185.15
30582.53 = 638.47
28550.32 32 +: 2298.29 03
road runner
423191.00 07 i 53071.15
88890.04 04000.0422 + 24971.18
251261. .0 t: 31741.38
robotank
97.23 +: 1.22
108.92 1: 4.79
98.45 45 1: 2.85
seaquest
188771.88 84 20759.57
175745.09 + 120718.89 82
86605.80 86 +:2 55065.85
skiing
29854.11 +: 85.79
- -30060.81 = 142.32
- 1--30121.95 70.62
solaris
17741.02 + 5340.46
16127.73 T3 + 2975.20
24366.55 59 = 4868.00
space invaders
3621.76 5.81
3547.78 4: 35.31
30609 21 7141.11
gunner
223536 63 i 48548.8. 34
179698.69 + 12194.36
171294.31 + 23185.79
surround
8.24 0.48
1.48 8.12
5.86 1.44
tennis
7,99 22.56
7.98 1: 22.51
23.96 0.01
time pilot
139931.C 67 1: 70521.78
71768.84 84 2933.22
44936. 137.49
tutankham
324.02 + 4.26
311.65 8.62
420.36 30.13
up n down
529363.08
16813.20
394984.70 34313.42
562739.00 = 8527.59
venture
0.00 0.00
1833.85 43.73
2110.64 55.39
video pinball
454023.45 d6 377076.03
107071.98 67142.18
463141.28 1 426927.92
wizard of wor
40833.66 4776.81
38275.31 4177.41
30453.12 2470.20
yars revenge
279765.8
27370.20
250483.70
54593.32
280333. 6970431
zaxxon
56059.14 3217.77
66099.25 8520.19
67611.78 6226.04
19

---

Under review as conference paper at [CLR 2021
Table 7: Final scores per game in our ablation study after 5B frames. We consider versions of CPT where the pre-trained policy is used for exploitation, exploration, both. Game asterix bank heist
e-greedy explore
eZ-greedy explore
CPT (exploration only)
CPT (exploitation only)
CPT
347585.48 + 244661.28
414183.29 + 160442.25
717259.64 0499996422 94011.25
577034.99 + 90735.79
649740.02 56368.04 04
987.94 + 163.14
1050.67 = 143.36
13881.28 + 2693.54
11417.81 - 7006.75
11894.55 = 984.46
frostbite
9312.19 286.04
8228.56 = 1182.64
9132.02 # 372.76
10650.34 3690.80
17895.70 = 2683.33
gravitar
6169.63 83.68
5781.13 757.18
6473.56 : 132.16
7231.54 = 1994.44
8114.29 1027:20
jamesbond
5771.41 + 2084.84
3633.35 = 272.19
1713.43 + 477.47
3898.66 = 1044.03
1567.71 + 472.73
moniezuma revenge
1 1118.65
1465.77 = 1102.87
11216.13 + 837.50
6433.33 372.68
13429.28 413.32
ms pacman
11406.77 121.72
8099.46 + 868.26
10622.99 + 504.86
10611.56 + -21.30
10983.94 = -66.044
pong
20.95 + 0.02
20.49 0.06
20.88 = 0.03
20.94 1.94=00 0.05
20.89 = 0.07
private eye
23589.28 11880.08
50907.99 = 15174.57
40483.83 18.21
37028.70 = 2449.66
40437 440 = 47.86
space invaders
3617.12 + 13.38
3559.49 27.29
31513.66 36.5 + 1093.84
3601.02 + 38.46
27853.60 + 5589.58
tennis
8.02 + 22.58
7.95 22.55
7.95 22.57
-7.62 22.36
23.96
up down
527933.55 + 23035.67
383415.21 + 79529.79
562248.72 + 11617.29
549769.30 = 3015.66
568148 23 = 2861.37
20

---

Under review as a conference paper at ICLR 2021
Table 8: Final scores per task in Atari games with modified reward functions. We report training results for the standard game reward, variant with sparse rewards (casy), and a task with deceptive rewards (hard). Despite the pre-trained policy might obtain low or even negative scores in some of the tasks, committing to its exploratory behavior eventually lets the agent discover strategies that lead high returns.
Game
e-greedy explore
ez-greedy explore
Fine-tuning
(unsup)
Ms Pacman: original
11407 + 122
8099 868
8359 2117
1360
10984
665
Ms Paeman: ghosts (easy)
8375 577
4322
8356 551
146
8789
651
Ms Pacman: ghosts (hard)
2836
4018 1025
1891 1342
-998
7868
1085
Hero: original
43762 4918
39018 3262
46848 1199
42675
3905
Hero: miners (easy)
3000 1: 0
3000 -::
3000 0
1351
4665 1: 470
Hero: miners (hard)
2677 1. 23
2155 1 95
700 0
1473
3547 1 122
21

---

Under review as a conference paper at ICLR 2021
H LEARNING CURVES
CTT (ursup)
E-groedy explore
12-groddy explor
CIT
Figure Training curves in all 57 Atari games after 5B frames. Shading shows maximum and minimum over 3 runs, while dark lines indicate the mean. Leveraging the pre-trained policy pro- vides important gains, particularly hard exploration games such as Montezuma'ss s Revenge, while maintaining performance in games with denser rewards such Pong Asterix.
22

---

Under review as a conference paper at ICLR 2021
asterix
pacman
banic heist
700
15.
099
13N.
600
10
300
2.4
Up down
pong
montezuma revenge
000
164
124
300
200
-10
100
-20
1.0
3.
3.
4.
52
33
2.0
30
4.5
50
10
10
3B
4.
frostbite
gravitar
37.54
15.
135.
28
3.54
privote oye
space Invaders
tonnis
4U1
15.
201
-20
(unsup)
texploration)
texploitation)
Figure 8: Training curves for ablation experiments after SB frames. Shading shows maximum and minimum over 3 runs, while dark lines indicate baselines, but different sets of games. Combining them retains the best of both methods, and boosts performance even further in some games.
the mean. Both methods offer benefits over the
23

---

Under review as a conference paper at ICLR 2021
MsPacman
MsPocman: eating ghosts (easy)
MsPocman: eating ghosts (hard)
21.
31
2k
48
41
4.4
3.4
Herro
Hero: rescuing miners (easy)
Hero: rescuing miners (hard)
51.
31
42.
4.
41
316
31
2.6
28
1.4
3.K
1.5
25
30
5.5
1.
2.8
3B
3.5
2.5
35
CPT (unsup)
explore
E2-greedy
CPT
Figure 9: Alternative reward functions for MsPacman (top) and Hero (bottom). We report training curves for the standard game reward (left), variant with sparse rewards (center), and task with deceptive rewards (right). Despite the pre-trained policy might obtain low or even negative scores in some of the tasks, committing to its exploratory behavior eventually lets the agent discover strategies that lead high returns.
montezuma revenge
5k
10
20
30
40
CPT (unsup)
e-greedy
CPT
Figure 10: Training curves after 50M frames on Montezuma'ss 's Revenge, using 16 actors and the CNN encoder from the pre-trained policy. Pre-trained weights not fine-tuned.
24

---

Under review as a conference paper at ICLR 2021
1
INFLUENCE OF PRE-TRAINED POLICY
montezumaa revenge
12000
10000
6020
4000
-10
-20
1.8
2.8
5.6
100
200
200
400
500
hours
hours
hours
revenge
pong
15000
10000
10
5200
-20
1.288
3:84
6.4
8.56
1.28
3,84
6.48
3.54
- CPT (unsup)
-+- CPT
Figure 11: Influence of the pre-trained policy on the end performance. We evaluate different pre- training times: 8, 24, 40 and 56 hours of pre-training. 8 hours of training correspond roughly to 1.3B environmental steps. We show learning curves (top) and end performance (bottom) for the games of Montezuma'ss 's Revenge and Pong. When comparing end performance, and т. (after 5B environmental steps adaptation). Longer pre-training times lead to policies that cover of the environment. As expected, the performance in Pong is independent of the quality of the pre-trained policy while for Montezuma's 's Revenge, longer pre-training times lead to dramatic improvements after the adaptation stage.
show the performance of m
25

---

observations and Under review as a conference paper at ICLR 2021
J DISTRIBUTED SETTING
All experiments are run using a distributed setting. The evaluation we do is also identical to the one done in R2D2 (Kapturowski et al. 2019): parallel evaluation workers, which share weights with actors and learners, run the Q-network against the environment. This worker and all the actor workers are the two types of workers that draw samples from the environment. For Atari, we apply the standard DQN pre- pe-ppreeesing used in R2D2. The next subsections describe how actors, evaluators, and learner are run in each stage.
J.1 UNSUPERVISED STAGE
The computation of the intrinsic NGU reward, r r,GU, follows the method described in Puig domènech Badia et al. 2020bb Appendix A.1). In particular, the version that combines episodic intrinsic rewards with intrinsic reward from Random Network Distillation (RND) (Burda al. 2018b).
Learner
Sample from the replay buffer sequence of intrinsic rewards N:GU actions a.
Use Q-network to learn from (r (yNGU x a) with Retrace (Munos et al. 2016) using the procedure used by R2D2.
Use last 5 frames of the sampled sequences to train the action prediction network in NGU, This means that, for every batch of sequences, all time steps are used to train the RL loss, whereas only time steps per sequence are used to optimize the action prediction loss.
Use last 5 frames the sampled sequences to train the predictor of RND.
Evaluator and Actor
Obtain 2; and r--1-
With these imputs, compute forward pass of R2D2 to obtain de
With æy, compute NGU using the embedding network in NGU,
(actor) Insert and r:GU in the replay buffer.
Step on the environment with de.
Distributed training
As in R2D2, we train the agent with a single GPU-based learner and fixed discount factor 7. All actors collect experience using the same policy, but with a different value of c. This differs from the original NGU agent, where each actor runs policy with a different degree of exploratory behavior and discount factor.
In the replay buffer, we store fixed-lengti sequences of (æ,, a, r) tuples. These sequences never cross episode boundaries. Given single batch of trajectories we unroll both online and target networks on the same sequence of states to generate value estimates. We use prioritized experience replay. We followed the same prioritization scheme proposed in Kapturowski et al. (2019) using mixture of max and mean of the TD-errors with priority exponent 1.0.
J.2 ADAPTATION STAGE IN CPT
Learner
Sample from the replay buffer sequence of extrinsic rewards observations and actions
a.
(expanded action set) Duplicate transitions collected with T, and relabel the duplicates with the primitive action taken by Tp when acting.
26

---

Under review as a conference paper at ICLR 2021
Use Q-network to learn from (r2, a) with Peng's Q(A) (Peng & Williams 1994) using the procedure used by R2D2.
Actor
(once per episode) Sample Elexy-
Obtain 2t.
If not flight, start with probability Clovy-
If on flight, compute forward pass with Tp to obtain d,. Otherwise, compute forward pass R2D2 to obtain a, If a, = |A| + az <- (z))).
Insert Fe, a, and ra in the replay buffer.
Step the environment with de.
Evaluator
Obtain 24,
Compute forward pass of R2D2 to obtain de. If az = |A + +1, + mp(x).
Step on the environment with de-
Distributed training
As in R2D2, we train the agent with a single GPU-based learner and fixed discount factor y. All actors collect experience using the same policy, but with different value of
In the replay buffer, we store fixed-length sequences of (æ, a, r) tuples. These sequences never cross unroll both online and target networks on the same sequence of states to generate value estimates. We use prioritized experience replay. We followed the same prioritization scheme proposed in Kapturowski et al. (209)) using mixture of max and mean the TD-errors with priority exponent = 1.0.
episode boundaries. Given single batch of trajectories
27


Reviews:
[{"review_id": "odDKQp_erRV", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper studies a pre-training approach to reinforcement learning. The objective is, first to pre-train a model considering that, without reward, interaction with an environment is cheap, and second, to fine-tune a policy given a particular reward function. \n\nAs a first contribution, the paper proposes two strategies to use a pre-trained policy for discovering an efficient task-dependent policy: i) the action set is expanded such that the policy can choose to follow the pre-trained policy and ii) exploration may be done by following the pre-trained policy on t timesteps, t being randomly chosen. As a second contribution, the paper proposes a criterion to pre-train a policy based on a coverage criterion.  The principle is to encourage a policy to generate trajectories that are going through as many states as possible. \n\nExperiments are made on different environments. First, the pre-trained policies are evaluated based on how much reward they are able to collect, and compared to other unsupervised approaches. Second, the final policy is compared to epsilon-greedy and epsilon-z-greedy approaches. In the two cases, the proposed approach outperform the baselines. \n\n== Comments: \nFirst of all, the paper clearly lacks of details, and it is difficult to be sure about what is really done, and what is the final algorithms. As far as I understand, instead of proposing a really new approach, the paper is more stacking two approaches (i.e NGU and R2D2, just changing a little bit the action space) and it does not really provide any justification about what is done. From my point of view, it is more a paper investigating if coverage may be a good unsupervised training criterion than a paper presenting a new model. \n\nConcerning the exploration/exploitation of pre-train policies, the paper does not really describe how the 'flights' are generated (section 3). I would advise the authors to provide more details on this aspect of their algorithm. \n\nConcerning the coverage approach, I am not convinced that the paper allows us to draw any conclusion on the interest of using such a criterion during the pre-training phase. Indeed, the authors are mainly evaluating their approach on Atari games, on which there is a clear relationship between the length of the episodes and the final score achieved by an agent. This is what is shown in Section 5.1: coverage is a good surrogate objective for solving atari games.  The article is thus lacking evaluation on other types of environments, and the performance obtained by the model on Atari games is mainly due to the use of the NGU model which has been developed more specifically for this type of environment.\n\nTo conclude, I think that the paper is failing to provide evidence that the coverage approach is a good approach to unsupervised pre-training of policies. Moreover, I have the feeling that the coverage criterion may be good for particular types of environments (like Atari games), but not for some others, making the proposed approach very specific to particular problems. Combined with the lack of novelty, and the lack of details, I recommend to reject the paper. \n\nConsidering the answers from the authors, I decided to not change my score. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "iQJ9qulvKEK", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a transfer approach for reinforcement learning. The proposed approach leverages a policy pre-trained via Never Give Up (NGU) approach, and can facilitate learning challenging RL tasks including the ones with sparse reward. This paper presents many strong pieces of evidence that this approach can be used to tackle challenging RL problems including hard exploration and multi-task learning. \n\nStrengths\n* Necessity of transferring behavior is well motivated and experiment showed its strength in challenging RL benchmarks.\n* Figure 2 is useful to have intuition on how the proposed transfer method can be useful\n* Table 1 set a strong unsupervised RL performance for Atari Suite\n* Ablation study in Figure 3 is thought-provoking. It is interesting that the significant gain of pretraining come only when both exploitation and exploration method are used jointly.\n* Figure 4 provides a useful intuition that more pretraining is beneficial for transfer to hard exploration task.\n\nWeaknesses\n* The \"flights\" technique is not described in detail in the main text. I managed to find the detail in Appendix A, but the pointer does not exist in the main text.\n* The paper claim \"coverage\" as the desired objective for RL pretraining and tried to support this claim by showing the transfer performance after pretraining via Never Give Up (NGU). I am convinced that NGU is a good pre-training objective but, it is not clear whether a more general claim for \"coverage\" is supported as well. It is not clear whether NGU is optimizing \"coverage\" well, and the relation between \"coverage\" and transfer performance is not studied. \n\nComment / Questions to author\n* \"but little research has been conducted towards leveraging the acquired knowledge once the agent is exposed to extrinsic reward\": I'm not sure whether I agree with this description. My understanding is that (Burda et al., 2018) studied a setting where an intrinsic reward is jointly used with extrinsic rewards. The only difference with this work is that the previous work did not study a setting with a clear separation between \"pre-training\" and \"transfer\".\n* Is there a difference between \"ez-greedy with expended action set A+ (using pre-trained policy)\" vs \"the proposed transfer method (exploitation + exploration)\"?\n* I'm curious about the comparison between CPT vs joint training with extrinsic reward. How authors would compare CPT vs joint training?\n\nRecommendation\nI recommend accepting this paper because this paper presented strong evidence that unsupervised pre-training and transfer may be a powerful approach to solve many challenging RL problems. I believe this observation is likely to catalyze future research of the related approaches, and the proposed method itself may be used for different domains to improve the capability of RL in general.", "extra_scores": {}, "extra_text": {}}, {"review_id": "gTX8xwrr4-V", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes a methodology to use a pre-training phase (latent learning) where the agent tries to maximize the coverage of the state space to then bootstrap task solving where the agent focuses on the task-defined reward.\n\nThe paper doesn’t match conference standards in terms of description of the method. There is no clear explanation (algorithm ? figure?) of the process, how the pre-trained phase and the next phase interact, nor which information is transferred (behaviour = policy ?). No proper definition of coverage is provided. Is the pre-training phase a time when your agent learns based on a different (intrinsic) reward function ? If yes, what is this function? Is it only NGU ?\nIf I understood correctly, the contribution of the work is limited to training NGU without external reward, then using this pretraining to initialize the agent in an exploitation phase. \n\nOne critical question that is not answered: if you gather data in a pre-training phase, why don’t you estimate a (transition) model ? The properties of model-based RL are quite interesting in terms of finding a more optimal policy compared to learning a reactive policy. They might be arguments in favour of keeping a model-free approach but they are not presented.\n\nThe results are in accordance with the claim of the authors that their approach, favouring exploration, improves the final performance. However, you do not evaluate the actual coverage of the agent. Reward itself is a coarse metric, is suitable for the agent performance, but it is a weak claim to say that the differences in performance are due to better coverage. Please provide a measure of the improved coverage to support your claim.\n\nIn conclusion, the paper lacks a proper structure to explain the method, make it reproducible by readers; it is unclear what the contribution is, and it seems quite limited. This flaw makes it difficult to understand the results (that are nevertheless supporting the claims). \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "GXepgS3_scL", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n#### Summary:\n\nIn this work the author focus on transfer in RL via proposing to transfer knowledge through behavior instead of representations. They propose using coverage as an objective for the pre-training procedure. They then employ the NGU (Badia 2020) They also propose a method based on coverage pre-training for transfer and provide empirical evidence in support of their method for transfer in RL on the Atari suite.  \n\n\n#### Strengths:\n\n- The paper has a good experimentation section, including empirical analysis such as ablation studies and effect or pre-training that is insightful.\n\n#### Weakness:\n\n- **Novelty**: The notion of coverage has been explored in the past, for instance in [1, 2, 3]. The authors fail to differentiate what do they do in their method that fixes the limitations of the previous work. \n\n- **Incremental Nature of work**: The proposed method relies heavily on the Never Give Up (Badia et al) for finding the policies that maximize the coverage. It is not clear what this work is proposing other than using NGU in the transfer setting directly.\n \n- **Cost of pretraining**: The pre-training comes at a cost. In their setup, the agents are allowed as many interactions with the environment as needed. This can be extremely futile for high-dimensional state and action spaces. If the dynamics model doesn't change for the downstream tasks, then this choice of pre-training is a valid choice (that has already been explored in the previous works as the authors mention). However, if there is a change in dynamics then there can be scenarios where the pre-training procedure doesn't result in any benefits. The procedure mentioned in Section 3 is already being employed in many different previous works, and it is not clear what is the setting being considered in this work. Also in Section 3, $\\pi_i$ is undefined. \n\n- **Missing references**: The authors make statements and then fail to give the appropriate references for the same. For instance, the authors quote that \"RL techniques have not yet seen the advent of a two-stage setting where task-agnostic pre-training is followed by efficient transfer to downstream tasks\", however, there are multiple works that are based on essentially this approach ([4, 5]).\n \n- **Reproducibility**: There is no mention of code release and makes me skeptical of their results, especially when this an empirical results-driven work.\n\n\n#### References:\n\n- [1] Schmidhuber, Jürgen. \"Formal theory of creativity, fun, and intrinsic motivation (1990–2010).\" IEEE Transactions on Autonomous Mental Development 2.3 (2010): 230-247.\n- [2]  Hazan, Elad, et al. \"Provably efficient maximum entropy exploration.\" International Conference on Machine Learning. 2019.\n- [3] Conti, Edoardo, et al. \"Improving exploration in evolution strategies for deep reinforcement learning via a population of novelty-seeking agents.\" Advances in neural information processing systems. 2018.\n- [4] Lesort, Timothée, et al. \"State representation learning for control: An overview.\" Neural Networks 108 (2018): 379-392.\n- [5] Ha, D. and Schmidhuber, J. (2018). World Models. ArXiv e-prints.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies the unsupervised RL problem, where the agent is allowed to interact with the environment for a certain amount of time without any extrinsic reward. The main idea is that the initial unsupervised training phase can be used to learn a set of "skills" that could help both in exploration and zero-shot transfer for any downstream task.

There is general consensus among the reviewers that the paper is studying an important problem and that the empirical validation is solid. Nonetheless, the technical contribution and the positioning wrt to the relevant literature are relatively weak for proposing acceptance for the paper. Before entering into details, I would like to acknowledge the fact that the rebuttal and the revised version did improve the original submission and clarified some aspects (eg, the structure of the algorithm), yet the contribution does not seems strong enough.

The idea of state space coverage is indeed not novel, either for exploration or for transfer (as properly reviewed in the paper). The authors identified some weaknesses of existing methods (eg, estimating state distributions), but it remains unclear whether the algorithm they propose has any significant technical contribution. In fact, as confirmed by the authors, CPT is rather applying any algorithm that *could* perform a good state space coverage and learn policies at the same time and then use the learned policy during the downstream task combined with a relatively basic "option-level" eps-greedy strategy. In this sense, it seems like CPT overcomes the limitations of previous algorithms, just by applying another existing algorithm (NGU) that is more scalable. While the evidence that this is "enough" to obtain good results is indeed interesting, it doesn't seems like it is pushing the algorithmic state of the art forward.

A more substantial contribution would be to dive deeper into the state coverage problem and provide an algorithm that is more specifically designed for the transfer setting considered in the paper. In fact, there is no clear evidence that NGU is the *right* approach to perform good coverage and return "useful" skills. Since this is the core concern of the paper, the technical contribution should be more significant on this part. The "meta" algorithm in itself seems rather standard otherwise.


Decision: Reject