Fast Task Inference with Variational Intrinsic Successor Features
Authors: Steven Hansen
Will Dabney
Andre Barreto
David Warde-Farley
Tom Van de Wiele
Volodymyr Mnih
Venue:  (2020)
Keywords: Reinforcement Learning
Variational Intrinsic Control
Successor Features

Abstract: It has been established that diverse behaviors spanning the controllable subspace of a Markov decision process can be trained by rewarding a policy for being distinguishable from other policies. However, one limitation of this formulation is the difficulty to generalize beyond the finite set of behaviors being explicitly learned, as may be needed in subsequent tasks. Successor features provide an appealing solution to this generalization problem, but require defining the reward function as linear in some grounded feature space. In this paper, we show that these two techniques can be combined, and that each method solves the other's primary limitation. To do so we introduce Variational Intrinsic Successor FeatuRes (VISR), a novel algorithm which learns controllable features that can be leveraged to provide enhanced generalization and fast task inference through the successor features framework. We empirically validate VISR on the full Atari suite, in a novel setup wherein the rewards are only exposed briefly after a long unsupervised phase. Achieving human-level performance on 12 games and beating all baselines, we believe VISR represents a step towards agents that rapidly learn from limited feedback.

Published as a conference paper at ICLR 2020
FAST TASK INFERENCE WITH VARIATIONAL INTRINSIC SUCCESSOR FEATURES
Steven Hansen
Will Dabney DeepMind
André Barreto DeepMind
DeepMind
stevenhansendgoogle.com com
wdabneydgoogleecom
andrebarreto@google.com
David Warde-Farles
Tom Van de Wiele
Volodymyr Mnih DeepMind vmnih@google.cm com
DeepMind dwf@google. com
DeepMind (Former)
tydviele@gmail.com com
ABSTRACT
has been established that diverse behaviors spanning the controllable subspace of Markov decision process be trained by rewarding policy for being distinguishable from other policics |Gregor et al. 2016 Eysenbach al. 2018). However, one limitation of this formulation is the difficulty to generalize beyond the finite set of behaviors being explicitly learned, as may be needed in subsequent tasks. Successor features (Dayan) 1993: [arreto et al., 2017) provide an appealing solution this generalization problem, but require defining the reward function as lincar in some grounded feature space. In this paper, we show that these two techniques can be combined, and that each method solves the other's primary linitation. To do we introduce Variational Intrinsic Successor FeatuRes (VISR), a novel algorithm which learns cont rollable feat that can be leveraged to provide enhanced generalization and fast task inference through the successor features framework. We empirically validate VISR on the full Atari suite, in novel setup wherein the rewards are only exposed briefly after a long unsupervised phase. Achieving human-level performance on 12 games and beating all baselines, we believe VISR represents a step towards agents that rapidly learn from limited feedback.
let al.j 2018 Warde-Farley
INTRODUCTION
Unsupervised learning has played major role in the recent progress of deep learning. Some of the earliest work of the present deep learning posited unsupervised pre-training method for overcoming optimization difficulties inherent in contemporary supervised deep neural networks (Hinton et al. 2006; Bengio al.) 2007 Since then, modern deep neural networks have enabled a renaissance in generative models, with neural decoders allowing for the training of large scale, highly expressive families of directed models (Goodfellow let al. 2014 Van den Oord et al.] (2016) as well as enabling powerful amortized variational interence over latent variables (Kingma and Welling 2013). We have repeatedly seen how representations from unsupervised learning can be leveraged to dramatically improve sample efficiency in a variety of supervised learning domains (Rasmus et al. 2015 Salimans et al. 2016).
In the reinforcement learning (RI.) setting, the coupling between behavior, state visitation, and the algorithmic processes that give rise to behavior complicate the development of "unsupervised" methods. The generation
behaviors by means other than seeking to maximize an extrinsic reward has long been studied under the psychological auspice of intrinsic motivation (Barto et al. 2004 Barto 2013 Mohamed and Rezende, 2015). often with the goal of improved exploration (Singek and Bartol 2006) Oudeyer and Kaplan 2009 Bellemare et al. 2016). However, while exploration is classically concerned with the discovery of rewarding states, the acquisition of useful state representations and behavioral skills can

---

Published as a conference paper at ICLR 2020
also be cast unsupervised (i.e. extrinsically unrewarded) learning problem for agents interacting with an environment.
In the traditional supervised learning setting, popular classification benchmarks have been employed (with labels removed) as unsupervised representation learning benchmarks, wherein the acquired representations are evaluated based on their usefulness for some downstream task (most commonly the original classification task with only a fraction of the labels reinstated). Analogously, we propose removing the rewards from an RL benchmark environment for unsupervised pre-training of an agent, with their subsequent reinstatement testing for data- efficient adaptation. This setup emulates scenarios where unstructured interaction with the environment, or a closely related environment, is relatively inexpensive to acquire and the agent expected perform one or more tasks defined in this environment in the form of rewards.
The current state-of-the-art for RL with unsupervisod pro-training comes from a class of algorithms which, independent of reward, maximize the mutual information between latent variable policies and their behavior in terms of state visitation, an objective which we refer to as behavioral mutual information (Mohamed and Rezende 2015 Gregor et al. 2016 Eysenbach et al. 2018 Warde-Farley et al.) 2018). These objectives yield policies which exhibit a great deal of diversity in behavior, with variational intrinsic control (Gregor et al., 2016 VTC) and diversity is all you need (Eysenbach et al. 2018 DIAYN) even providing a natural formalism for adapting to the downstream RL problem. However, both methods suffer from poor generalization and a slow inference process when the reward signal is introduced. The fundamental problem faced by these methods is the requirement to effectively interpolate between points in the latent behavior space, as the most task-appropriate latent skill likely lies "between" those learnt during the unsupervised period. The construction of conditional policies which efficiently and effectively generalize to latent codes not encountered during training is an open problem for such methods.
Our main contribution is to address this generalization and slow inference problem by making use of another recent advance in RL, successor features (Barreto et al., (017). Successor features (SF) enable fast transfer learning between tasks that differ only in their reward function, which is assumed to be linear in some features. Prior to this work, the automatic construction of these reward function features was an open research problem (Barreto et al. 2018). We show that, despite being previously cast as learning a policy space, behavioral mutual information (BMT) maximization provides compelling solution to this feature learning problem. Specifically, we show that the BMT objective can be adapted to learn precisely the features required by SF. Together, these methods give rise to an algorithm, Variational Intrinsic Successor FeatuRes (VISR), which significantly improves performance in the RL with unsupervised pre-training scenario. In order to illustrate the efficacy of the proposed method, we augment the popular 57-game Atari suite with such an unsupervised phase. The use of this well-understood collection of tasks allows us to position our contribution more clearly against the current literature. VISR achieves human-level performance on 12 games and purperoosms all baselines, which includes algorithus that operate three regimes: strictly unsupervised, supervised with limited data, and both.
REINFORCEMENT LEARNING WITH UNSUPERVISED PRE-TRAINING
As usual, we assume that the interaction between agent and environment can be modeled as a Markov decision process (MDP, Puterman 1994). An MDP is defined as a tuple M = (S, r,), where S and A are the state and action spaces, p(-|s, a) gives the next- state distribution upon laking action in state 8, and E [0, 1) is a discount factor that gives smaller weights to future rewards. The function r SS x A x S + R specifies the reward received at transition s 3 s'; more generally, we call any signal defined as c: S Ax cumulant (Sutton and Barto 2018).
As previously noted, we consider the scenario where the interaction of the agent with the environment can be split into two stages: an initial unsupervised phase in which the agent does not observe any rewards, and the usual reinforcement learning phase in which rewards are observable.
2

---

Published as a conference paper at ICLR 2020
During the reinforcement learning phase the goal of the agent is to find a policy π T:S :. A that maximizes the expected return G2 = -i= > Ry4s, where RS = r(S,, A,, S(+1). A principled way to address this problem is to use methods derived from dynamic programming, which heavily rely on the concept of a value function (Puterman 1994). The action-value function of a policy is defined as Q (s.a) RM [Ge] SS 8, A, aj where [+] denotes expected value when following policy т. Based on Q" we can compute a greedy policy
π (s) E argmax (T(s
(1)
T' is guaranteed to do at least as well as n, that is: Q (s,aa 2 (T(s,a) a) for all (s,0) E S A. The computation of Q= () and r' are called policy evaluation and policy improcement, respectively; under certain conditions their successive application leads to the optimal value function Q", from which one can derive an optimal policy using ① The alternation ween policy evaluation and policy improvement is at the core of many RL algorithms, which usually carry out these steps only approximately (Sutton and Barto 2018). Clearly, if we replace the reward r(s, a, s') with an arbitrary cumulant cls, a, 8') all the above still holds. In this case we will use Q to refer to the value of π under cumulant and the associated optimal policies will be referred to as Te, where ne(s) is the greody policy ① on (*
Usually it is assumed, either explicitly or implicitly, that during learning there is cost associated with each transition in the environment, and therefore the agent must learn a policy quickly as possible. Here we consider that such a cost. is only significant in the reinforcement learning phase, and therefore during the unsupervised phase the agent is essentially free to interact with the environment as much as desired. The goal in this stage is to collect information about the environment to speed up the reinforcement learning phase as much as possible. In what follows we will make this definition more procise.
UNIVERSAL SUCCESSOR FEATURES AND FAST TASK INFERENCE
Following Barreto et al. 2017 2018
assume that there exist features Φ(s, a, Rd such that the reward function which specifies a task of interest can be written as
r(s,a,8') = a(,,a,"') a,s')T.. w,
(2)
where w Rd are weights that specify how desirable each feature component is, or a 'task vector' for short. Note that, unless we constrain & somehow, ( is not restrictive in any way: for example, by making (8.0, = r(s, a, a.s") for some we can clearly recover the rewards exactly. Barreto et al. (2017) note that (2) allows one to decompose the value of policy π as
(")saa)= = EE [Eient 0011 S s,
= 26" (s) Tw,
(3)
where d, = ((S, A,, 1c+1) and (s, a) are the successor features (STs) of प. STs can be seen as multidimensional value functions in which 0(s,a, play the role of rewards, and as such they can be computed using standard RL algorithms [Szepesvári 2010).
One of the benefits provided by SF' is the possibility of quickly evaluating a policy æ. Suppose that during the unsupervised learning phase we have computed s;;; then, during the supervised phase, we can find a w Rdd by solving a regression problem based on ? and then compute Q" through (3). Once we have Q", we can apply to derive policy that will likely outperform म.
Since was computed without access to the reward, its is not deliberately trying to maximize it. Thus, the solution x' relies on a single step of policy improvement 1 over a policy that is agnostic to the rewards. It turns out that can do better than that by extending the strategy above to multiple policies. Let e: (S + A) + 1² be a enciciin mapping, that is, function that turns policies into vectors in R Borsa et al. (2019) universal successor feature (USFs) are defined as (b(s, a, e(T)) = 36" (s.a). Note that, using USFs, we can evaluate any policy by simply computing
Q" (s. (s,,,,,((T)
(4)
3

---

Published as a conference paper at ICLR 2020
Now that can compute Q" for any should be able to leverage this information to improve our previous solution based on a single policy, This is possible through generalized policy improvement (Barreto et al. 2017 GPI). Let & be USFs, let Ni; n2 .. Tn be arbitrary policies, and let
(s) argmax max (b(8, a, e(m:)) Tw argmax max QT (s,
(5)
It can be shown that 5 is a strict generalization of (1), in the sense that (= a 2 (s,(s,)))
for all Tyy 8, and a. This result can be extended to the case in which (2) holds only approximately and ab is replaced by a universal successor feature approrimator (USFA)
vo (b(s,a) a) (Barreto et al. 2017 2018
Borsa et al. 2019).
The above suggests an approach to leveraging unsupervised pre-training for more data- efficient reinforcement learning. First, during the unsupervised phase, the agent learns USFA 2b. Then, the rewards observed at the carly stages of the RL phase are used to find an approximate solution w for (2). Finally, n policies и; are generated and a policy x derived through (5). If the approximations used in this process are reasonably accurate, will be an improvement over Ti, T2; Tn-
However, in order to actually implement the strategy above have to answer two funda- mental questions: (i) Where do the features & in (2) come from? (ii) How do we define the
policies ग. used in ( It turns out that these questions allow for complementary answers,
as we discuss next.
BEHAVIORAL MUTUAL INFORMATION
Features & should be defined in such a way that the down-stream task reward is likely to be a simple function of them (see (... Since in the RL with unsupervised pre-training regime the task reward not available during the long unsupervised phase, this amounts to utilizing strong inductive bias that is likely to yield features relevant to the rewards of any 'reasonable' task.
One such bias is to only represent the subset of obscrvation space that the agent can control (Grgoo et al. 20166). This can be accomplished by maximizing the mutual information between policy conditioning variable and the agent's 's behavior. There exist many algorithms that. maximiie this quantity through various means and for various definitions of bbehavior'
Eysenbach et al. 2018 Warde- Saard--Farley et al. 2018).
The objective F(O) is to find policy parameters 0 that maximize the mutual information (1) between some policy-conii coniiyyconiiiiinng variable, 2, and some function f of the trajectory induced by the conditioned policy, where H is the entropy of some variable:
F(0)- [22; f(Tx,)) H(2) H(2)f(TTa))
(6)
While in general z will be a function of the state Gregor et al. 2016). it is common to assumee that is drawn from fixed (or at least state-independen distribution for the purposes of stability (Eysenbach et al. 2018). This simplifies the objective to minimizing the conditional entropy of the conditioning variable given the trajectory.
F(0) = (=(f
(7)
When the trajectory is sufficiently long, this corresponds to sampling from the steady state distribution induced by the policy. Commonly f is assumed to return the final state, but for simplicity we will consider that samples single state uniformly over Tra-
F(O) Σ p(s pog = E=,g [(]s)) 8,8
(8)
This intractable conditional distribution can be lower-bounded by a variational approximation (q) which produces the loss function used in practice (sce Section 8.1 for derivation based on Agakov (2004))
T
Ex, {(1s))]
(9)
4

---

Published as a conference paper at ICLR 2020
Agent
We p(w)
Supervised Learning
Policy
Discriminator
arg max eh a)
a( d(a)) exp(e(s)) T
Successor Features
A
g de) + yalf S4+1. de-1)
Functional
1
Training target
Gradient flow
at
St
7
Environment
Figure 1: VISR model diagram. In practice w, is also fed into sb as an input, which also allows for GPI to be used (see Algorithm 1 in Appendix). For the random feature baseline, the discriminator is frozen after initialization, but the same objective is used to train zb.
The variational parameters be optimized by minimizing the negative log likelihood of samples from the true conditional distribution, i.e., q is a discriminator trying to predict the correct from behavior. However, it is not obvious how to optimize the policy parameters as they only affect the loss through the non-differentiable environment. The appropriate intrinsic reward function can be derived (see Section 8.2 for details) through application of the REINFORCE trick, which results in log ((((s) serving this rolc.
Traditionally, the desired product of this optimization was the conditional policy (r). While the discriminator q could be used for imitating demonstrated behaviors (i.e. by inferring the most likely for a given r), for down-stream RL was typically discarded in favor of explicit search over all possible (Eysenbach et al. 2018). In the next section discuss alternative approach to leverage the behaviors learned during the unsupervised phase.
VARIATIONAL INTRINSIC SUCCESSOR FEATURES
The primary motivation behind our proposed approach is combine the rapid task inference mechanism provided by SFs with the ability of BMI methods to learn many diverse behaviors in an unsupervised way.
We begin by observing that both approaches use vectors to parameterize tasks. In the SF formulation tasks correspond to linear weightings w of features ((s). The reward for a task given by w is (sF(s; w) d(s)" w. BMI objectives, on the other hand, define tasks using conditioning vectors 2, with the reward for task 2 given by TBM7 2) = log g(2(s).
We propose restricting conditioning vectors to correspond to task-vectors of the SFs formulation. The restriction that 2 = w, in turn, requires that rsp(s; w) = TBMr(s; which implies that the BMT discriminator q must. have the form
log (w))) (6s)
(10)
One way to satisfy this requirement. is by restricting the task vectors w and features Φ(8) to be unit length and paremeterizing the discriminator & as the Von Mises-Fisher distribution with scale parameter of 1. Note that this form of discriminator differs from the standard choice parameterizing as a multivariate Gaussian, which does not satisfy equation I0
With this variational family for the discriminator, all that is left to complete the base algorithm is to factorize the conditional policy into the conicy-co successor features (1b) and the task vector (w). This is straightforward as any conditional policy can be represented by a UVEA (Schaul et al.) 2015), and any UVEA can be represented by a USFA given appropriate leature basis, such the one have just derived. Figure
5

---

Published as a conference paper at ICLR 2020
shows the resulting model. Training proceeds as in other algorithms maximizing BMI: by randomly sampling a task vector w and then trying to infer it from the state produced by the conditioned policy (in our case w is sampled from uniform distribution over the unit circle). The key difference is that in VISR the structure of the conditional policy (equation 5 enforces the fask/dynand factorization task inference to regression problem derived from equation
in SF (equations 2 and 4h, which in turn reduces
5.1 ADDING GENERALIZED POLICY IMPROVEMENT TO VISR.
Now that SFs have been given feature-learning mechanism, we can return to the second question raised al the end of Section 3 how can oblain diverse set of policies over which to apply GPT?
Recall that are training USFA a,s,,a, whose encoding function is e(T) w (that is, π is the policy that tries to maximize the reward in (10) for a particular value of w). So, the question of which policies use with GPI comes down to the selection of set of vectors w.
One natural w candidate the solution for regression problem derived from 2 Let us call this solution Whase, that is, Φ(s, a, sT) Whase z r(s,, a.s). But what should the other task vectors w's be? Given that task vectors are sampled from uniform distribution over the unit circle during training, there is no single subset that has any privileged status. So, following Borsa et al. (2018), we sample additional w's on the basis of similarity to Whase: Since the discriminator q enforces similarity on the basis of probability under a Von Mises- Fisher distribution, these additional w's are sampled from such a distribution centered on Wbase, with the concentration parameter acting a hyper- prree-trreeeter specifying how diverse the additional w's should be. Calculating the improved policy thus done as follows:
(s, a) *b(8, a, Whase)
Whase
Q (8, a) - ab(s, a, w) W Whose w VMF Whases T(s) argmaxx max (8,
(11)
EXPERIMENTS
Our experiments are divided in four groups corresponding to Scctions 6.1 to 6.4 First, assess how well VISR does in the RL sctup with unsupervised pre-training phase described in Section 2 Since this setup is unique in the literature on the Atari Suite, for the full Lwo-phase process we only compare to ablations on the full VISR model and variant of DIAYN adapted for these tasks (Table boltom section). In order to frame performance relative to prior work, in Section 6.2 we also compare to results for algorithms that operate in a purely unsupervised manner (Table I top section). Next, in Section 6.3 we contrast VISR's performance that of standard RL algorithms in a low data regime (Table 1 middle section). Finally, we assess how well the proposed approach of inferring the task through the solution of a regression derived from does as compared to random search.
6.1 REINFORCEMENT LEARNING WITH UNSUPERVISED TRE-TRAINING
To evaluate VISR, we impose a two-phase setup on the full suite of 57 Atari games (Bellemare et al. 2013). Agents are allowed a long unsupervised training phase (250M steps) without access to rewards, followed by short test phase with rewards (100k steps). The full VISR algorithm includes features learned through the BMI objective and GPI to improve the execution of policies during both the training and test phases (see Algorithm I in the Appendix). The main baseline model, RF VISR, removes the BMI objective, instead learning SFs over features given by a random convolutional network (the same architecture as the o network in the full model). The remaining ablations remove GPI from each of these models. The ablation results shown in Table (bottom) confirm that these components of VISR play complementary roles in the overall functioning of our model (also see Figure 24).

---

Published as a conference paper at ICLR 2020
26 Game Subset
Game Subset
Full 57 Games
Kaiser et al. (2019)
Burda et al.
(0118)
Mnih et al
2015
Algorithm
Mdn
M
>0
>H
Mdn
M
>0
> H
Mdn
M >0 T
IDF Curiosity @0
8.46
24.51
34
RF Curiosity @0
7.32
29.03
36
Pos Reward NSQ @0
2.18
50.33
14
5
0.69
57.65
26
0.29
41.19
28
8
Q-DIAYN-5 @0
0.17
-3.60
13
0
0.33
-1.23
25
2
0.34
-2.18
30
2
Q-DIAYN-50 a0
1.65
21.77
4
0
-1.69
16.26
-3.16
-0031
9
VISR @0
5.60
81.65
19
5
4.04
58.47
35
3.77
49.66
40
7
SimPLe @100k
9.79
36.20
26
4
DQN @10M
27.80
52.95
25
7
9.91
28.07
41
8.61
27.55
48
7
Rainbow @100k
2.23
10.12
25
PPO @500k
20.93
43.74
25
7
NSQ @10M
8.20
33.80
22
3
7.29
29.47
37
6.80
28.51
43
5
Q-DIAYN-5 @100k
0.01
16.94
2
1.31
19.64
28
1.55
16.65
33
6
Q-DIAYN-50 @100k:
1.64
27.88
3
0
1.66
16.744
2.53
-24.13
9
0
RF VISR @100k
7.24
58.23
20
6
3.81
42.60
33
2.16
35.29
39
9
VISR @100A
9.50
128.07
21
7
9.42
121.08
35
11
6.81
102.31
40
11
GPI RF VISR @100k
5.53
58.77
20
5
4.24
48.38
34
3.60
40.01
40
10
GPI VISR @100k
6.59
111.23
22
711.70
129.76
38
12
8.99
109.16
44
12
Table 1: Atari Suite comparisons. @N represents the amount of RL interaction utilized. Mn is median, M is mean, > 0 is the number of games with better than random performance, and > is the number games with human-level performance as defined Mnih al. (2015). Top: unsupervised learning only (Sec. 6.2 Mid: data-limited RL (Sec. 6.3 Bottom: RL
with unsupervised pre-training (Sec. 6.1 Standard deviations given in Table (Appendix).
In addition, DIAYN has been adapted for the Atari domain, using the same training and lesting conditions, base RL algorithm, and network architecture as VISR (Eysenbach et al. 2018 With the standard 50 50-dimensional categorical 2, performance
worse than random. While decreasing the dimensionality to (matching that of VISR) improved this, it was still significantly weaker than even the ablated versions of VISR.
6.2 UNSUPERVISED APPROACHES
Comparing against fully unsupervised approaches, our main external baseline the Intrinsic Curiosity Module PPathak et al. 2017). This uses forward model prediction error in some feature-space to produce an intrinsic reward signal. Two variants have been evaluated a 47 game subset of the Atari suite (Burda et al.) 2018). One uses random features as the basis of their forward model (RF Curiosity), ) and the other uses features learned via inverse-dynamics model (IDF Curiosity). It. important to that, in addition to the ext rewards, these methods did not use the terminal signals provided by the environment, whereas all other methods reported here do use them. The reason for not using the terminal signal was to avoid the possibility of the intrinsic reward reducing to simple "do not die" signal. To rule this out, an explicit "do not die" baseline was run (Pos Reward NSQ), wherein the terminal signal remains and small constant reward is given at every time-step. Finally, the full VISR model was run purely unsupervised. In practice this means not performing the fast-adaptation step (i.e. reward regression), instead switching between random w vectors every 40 time-steps (as done during the training phase). Results shown in Table (top and bottom) make it clear that while VISR is not particularly outstanding in the unsupervised regime, when allowed 100k steps of RL it can vastly outperform these existing unsupervised methods all criteria.

---

Published as a conference paper at ICLR 2020
6.3 LOW-DATA REINFORCEMENT LEARNING
Comparisons to reinforcement learning algorithms in the low-data regime are largely based on similar analysis by Kaiser et al. (2119) on the 26 easiest games in the Atari suite (as judged by above random pertormance for their algorithm). In that work the authors introduce model-based agent (SimPLe) and show that compares favorably to standard RL. algorithms when data is limited. Three canonical RL algorithms are compared against: proximal policy optimization (PPÓ) (Schulman et al. 2017 Rainbow (Hessel et al. 2017], and DQN (Mnih et al. 2015). For each, the results from the lowest data regime reported in the literature are used. In addition, we also compare to a version of N-stop Q-lcarning (NSQ) that uses the same codebase and base network architecture indicate that VISR is highly competil ive with the other RL methods. Note that, while these methods are actually solving the full RL problem, VISR's performance is based exclusively on the solution of a linear regression problem (equation 2 Obviously, this solution can be used to "warm start" an agent which can then refine its policy using any RL algorithm. We expect this version of VISR to have even better performance.
VISR. Results shown in Table [ (middle)
6.4 FAST INFERENCE
In the previous results, it was assumed that solving the linear reward-regression problem is the best way to infer the appropriate task vector. However, EEsenbach et al. (2018) suggest a simpler approach: exhanstive scarch. As there are no guaranteos that extrinsic rewards will be linear in the learned features (ø), it is not obvious which approach is best in practice I
We hypothesize that exploiting the reward-regression task inference mechanism provided by VISR should yield more efficient inference than random search. To show this, 50 episodes (or 100k steps, whichever comes first) are rolled out using trained VISR, each conditioned on a task vector chosen uniformly on a 5-dimensional sphere. From these initial episodes, one can either pick the task vector corresponding to the trajectory with the highest return (random search), or combine the data across all episodes and solve the lincar regression problem. In each condition the VISR policy given by the inferred task vector executed for 30 episodes and the average returns compared.
As shown in Figure D, linear regression substantially improves performance despite using data generated specifically to aid in random search. The mean performance across all 57 games was 109. 16 for reward-regression, compared to random search at 63.57. Even more dramatically, the median score for reward-regression was 8.99 compared to random search at 3.45. Overall, VISR outperformed the random search alternative on 41 of the 57 games, with one tic, using the exact same data for task inference. This corroborates the main hypothesis of this paper, namely, that endowing features derived from BMI with the fast task-inference provided by SFs gives rise a powerful method able to quickly learn competent policies when exposed to a reward signal.
7 CONCLUSIONS
Our results suggest that VISR is the first algorithm to achieve notable performance on the full Atari task suite in a setting of few-step RL with unsupervised pre-training, outperforming all baselines and buying performance equivalent to hundreds of millions of interaction steps compared to DQN on some games (Figure
As a suggestion for future investigations, the somewhat. underwhelming results for the fully unsupervised version of VISR suggest. that there is much room for improvement. While auriosity-based methods are transient (i.c., asymptotically their intrinsic reward vanishes) and lack a fast adaptation mechanism, they do seem to encourage exploratory behavior slightly more than VISR. A possible direction for future work would be to use a curiosity-basel intrinsic reward inside of VISR, to encourage it to better explore the space of controllable policies. Another interesting avenue for future investigation would be to combine the approach recently proposed by Ozair et al. (2019) to enforce the policies computed by VISR to be not only distinguishable but also lar apart in a given metrie space.
SSine VISR utilizes continuous space of possible task vectors, exhaustive search must be
replaced with random search.
8

---

Published as a conference paper at ICLR 2020
(a)
Atari-57: Median
(c)
Full VISR
VISR (no GPI)
Human-normalizer
ACar Carmas (mcresszngg order)
Mr
Sekr
Bark
1000
Unsupervised Training (millions of fromes)
(b)
Human-non norrnnomaaaaai lize score (log)
Regression
Search
SIPPC
-4-
mayen
chazpe cor
Atarl Games (increasing order)
10
10'
10'
Figure 2: (a) Median human-normalized scores over all 57 games, comparing VISR with and without GPI. Averaged over three seeds. (b) Human-normalized performance of VISR across all 57 Atari games after fast task inference. Reward regression in blue, random search in red. Regression outperforms search in all but two games. (c) Number of environment frames required for DQN to match VISR's performance after 100k steps of RL. The green block shows the games in which VISR outperforms DQN using 200 million transitions, the red block shows the games in which VISR is outperforeed by DQN using million transitions, and yellow block shows the games that do not fall in either of the previous categories. Light blue bars denote games in the 26 game set of Pathak al. 20117)
By using SFs features that maximize BMI, we proposed approach, VISR, that solves two open questions in the literature: how to compute features for the former and how to infer tasks in the latter. Beyond the concrete method proposed here, we believe bridging the gap between BMI and SI's is an insightful contribution that may inspire other useful mothods.
ACKNOWLEDGMENTS
We thank Raia Hadsell, Tejas Kulkarni and anonymous reviewers for useful feedback on drafts of the mamuscript. We additionally thank Catalin lonescu and Brendan O'Donoghue for helpful discussions.
REFERENCES
D. F. Agakov. The im algorithm: variational approach information maximization. Advances in Newral Information Processing Systems, 16:201, 2004.
A. Barreto, W. Dabney, R. Munos, J. Hunt, T. Schaul, H. van Hasselt, and D. Silver. Successor features for transfer reinforcement learning. Advances in Newral Information Processing Systems (NIPS), 2017.
A. Barreto, D. Borsa, J. Quan, T. Schaul, D. Silver, M. Hessel, D. Mankowitz, A. Zidek, and R. Munos. Transfer in deep reinforcement learning using successor features and generalised policy improvement. In Proceedings of the International Conference on Machine Learning (ICML), pages 501-510, 2018.
A. G. Barto. Intrinsic motivation and reinforcement learning. In Intrinsically motivated learning in natural and artificial systems, pages 17-47. Springer, 2013.

---

Published as a conference paper at ICLR 2020
A. G. Barto, S. Singh, and N. Chentanez. Intrinsically motivated learning of hicrarchical collections of skills. In Proceedings of the 3rd International Conference on Development and Learning. pages 112-19, 2004.
M. Bellomare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Newral Information Processing Systems, pages 1471-1479, 2016.
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 06 2013.
Y. Bengio, P. Lamblin, D. Popovici, and H. Larochelle. Greedy layer-wise training of deep networks. In Advances in neural information processing systems, pages 153-160, 2007.
D. Borsa, A. Barreto, J. Quan, D. Mankowitz, R. Munos, H. van Hasselt, D. Silver, and T. Schaul. Universal successor features approximators. arXiv preprinl ar 2018.
D. Borsa, A. Barreto, J. Quan, D. Mankowitz, H. van Hasselt, R. Munos, D. Silver, and T. Schaul. Universal successor features approximators. In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
Y. Burda, H. Edwards, D. Pathak, A. Storkey, T. Darrell, and A. A. Efros. Largo-scale study of curiosity-driven learning. aiviv preprint Xiv: 10o:004385,, 04355, 2018.
P. Dayan. Improving generalization for temporal difference learning: The successor represen- tation. Neural Computation, 5(4):613 624, 1993.
L. Espcholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu,
T. Harley, I. Dumning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. arxiv preprinl. arxiv: 802. 01561, 2018.
B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine. Diversity is all you need: Learning skills without reward function, 2018. URL htttp://axxvv.org//a/a abs wttp:/r/aaaaa/....000000
P. W. Glynn. Likelihood ratio gradient estimation: overview. Technical report, STAN- FORD UNIV CA DEPT OF OPERATIONS RESEARCH, 1987.
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley. S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672-2680, 2680, 2014.
K. Gregor, D. J. Rezende, and D. Wierstra. Variational intrinsic control. CoRR, abs/ 1611.07507, 2016. URL http://arxiv wwtp.///////.../////./..
M. Hessel, J. Modayil, H. van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. G. Azar, and D. Silver. Rainbow: Combining improvements in deep reinforcement learning. CoRR, abs/ Nes1100200228 2017.
G. F. Hinton, S. Osindero, and Y.W. Teh. A fast learning algorithm for deep belief nets. Newral computation, 18(7):15271154, 2006.
S. Hochreiter and Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735 1780, 1997.
L. Kaiser, M. Babacizadeh, P. Milos, B. Osinski, R. H. Campbell, K. Czechowski, D. Erhan, C. Finn, P. Kozakowski, S. Levine, et al. Model-based reinforcoment learning for atari. Xiv preprint arXiv:1903.00 N0000093,,,077,, 2019.
S. Kapturowski, G. Ostrovski, J. Quan, R. Munos, and W. Dabney. Recurrent experience replay in distributed reinforcement learning. 2018.
D. Kingma and J. Ba. Adam: method for stochastic optimization. arXiv preprint or Xin:11: 1411 6980, 2014.
10

---

Published as a conference paper at ICLR 2020
D. P. Kingma and M. Welling. (rXit:1112. 6114, 2013.
Auto-encoding variational bayes.
arXiv preprint
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540))529, 2015.
S. Mohamed and D. J. Rezende. Variational information maximisation for intrinsically motivated reinforcement learning. In Advances in neural information processing systems, pages 2125-212 2133, 2015.
P.-Y. Oudeyer and F. Kaplan. What is intrinsic motivation? a typology of computational approaches. Frontiers in neurorobotics, 1:6, 2009.
S. Ozair, C. Lynch, Y. Bengio, A. v. d. Oord, S. Levine, and P. Sermanet. Wasserstein dependency measure for representation learning. arXiv preprint Mii::1903.11760 2019.
D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell. Curiosity-driven exploration by self- supervised prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 16-17, 2017.
M. L. Puterman. Markov Decision Processes Discrete Stochastic Dynamic Programminng John Wiley & Sons, Inc., 1994.
A. Rasimus, M. Berglund, M. Honkala, H. Valpola, and T. Raiko. Semi-supervised learning with ladder networks. In Advances in Neural Information Processing Systems 28, pages 3546 3554. 2015.
T. Salimans, I. Goodfellow, W. Zaremba, V. Choung, A. Radford, and X. Chen. Improved techniques for training gans. In Advances in neural informalion processing systems, pages 2234 2242, 2016.
T. Schaul, D. Horgan, K. Gregor, and D. Silver. Universal Value Function Approximators. In International Conference on Machine Learning (ICML), pages 1312-1320, 2015.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimiza- tion algorithms. orXiv preprint 1xXiv170770634/7 2017.
Ö. Şimşek and A. G. Barto. An intrinsic reward mechanism for efficient exploration. In Procecdings of the 23rd international conference on Machine learning, pages 833-840. ACM, 2006.
R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. MIT Press, 2018. URL https: / //tp//mmmtttssseeeeeeeeeeessssst mit cecca//bookssriiiforrreeteee www.neennneeenneeeeeeeeee
C. Szepesvári. Algorithms for Reinforcement Learning. Synthesis Lectures Artificial Intelligence and Machine Learning. Morgan & Claypool Publishers, 2010.
A. Van den Oord, N. Kalchbrenner, Espeholt, O. Vinyals, A. Graves, al. Conditional image generation with pixclenn decoders. In Advances in neural information processing systems, pages 4790-4798, 2016.
D. Warde-Farley, T. Van de Wiele, T. Kulkarni, C. lonescu, S. Hansen, and V. Muih. Unsupervised control through non-parametric discriminative rewards. arXie preprint or orXie:1811.11959 2018.
P. J. Werbos el al. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE, 78(10):1550 1560, 1990.
R. J. Williams. Simple statistical foldienn-following algorithms for connectionist reinforcemmntt learning. Machine Learning,
1992.
11

---

Published as a conference paper at ICLR 2020
8 APPENDIX
8.1 DERIVATION OF LOWER BOUND
The general form of the derivation of this lower bound on mutual information is due to Agakov (2004), and the specific case of mutual information between a poliey-conditionnng variable and future state is due to Gregor et al. (2016). The lower bound used here is a special case where the entropy of the policy-conditioning variable held constant, as in Eysenbach et al. 2018). For the convenience of the reader, re-derive the bound here.
(2|s))
Σ p(s, 2) logs(z|s)
8,2
Σp(s,
p(s, 2) logp(z|s) + Σ 2(gg Σ p(s, 2) logg(z)s)
6,8
8,8
8,2
Σ p(s, 2) logg (2|() + Σ 8,2
p(s, 2) logp(2|s)
Σ
p(s, 2) log (2|s)
8,2
8,2
(12)
Σ p(s, 2) log (2|s) + Σ p(s)KL(p((s)ll((t))) 8,5
2 Σ p(s, 2) log (2|s)
For convenience, we can refer to maximizing F(0) as minimizing the loss function for parameters 0 = (0x 0g),
Lo (r,.(loogg
(13)
where 0, and 0y refer to the parameters of the policy and variational approximation g. respectively.
8.2 DERIVATION OF INTRINSIC REWARD
We can minimize La with respect to 0,, the parameters of q, using back-propagation. However, properly adjusting the parameters of n, 0x more difficult, as we lack differentiable model of the environment. We now show that we can still derive appropriate score function estimator using the log-likelihood (Glynn 1987) or REINFORCE trick (Williams) 1992). Since in this section we will be talking about Oz only (that. is, will not discuss 0g), we will drop the subscript and refer to the parameters
as simply 0.
Let 7 be a length T trajectory sampled under policy and let pe be the probability of the trajectory under the combination of the policy and environment transition probabilities. We can compute the gradient of Ps with respect to 0 as:
Vopo(T) = po (T) Vo log po)).
(14)
This means that we can adjust pe to make 7 more likely under it. we interpret ps as the distribution induced by the policy æ, then minimizing [3 corresponds to maximizing the following value function:
(s()) Er.2
Σ log q(z st) |s
=E, [log g(2 /)]]
12

---

Published as a conference paper at ICLR 2020
Figure 3: VISR features & learned by variational distribution ((w)s) in a 10-by- 10-by-10 gridworld.
We can then use the policy gradient theorem to calculate the gradient of our loss function with respect to the parameters of the policy, 6, for trajectorics beginning in state s,
voVo(s) / Vopo(T) log =)d+
(15)
T) = E, llog 1 log
Since standard policy gradient (with rewards rt = log (=)(0e)) can expressed as:
T
VoVo(s) = Er,avT Vo Vologi(a
Σ
Th
(16)
we can conclude that log 7(2 s) serves the role the reward function and treat it as such for arbitrary reinforcement lcarning algorithms (n-step Q-learning is used throughout this paper).
8.3 QUALITATIVE GRID-WORLD RESULTS
The complexity and black-box nature of the Atari task suite make any significant analisis of the egpresentations learned by VISR difficult (apart from their indirect effect on fast- inference). Thus, in order analyze the representation learned by VISR we have conducted a much smaller-scale experiment on a standard 10-by-10 grid-world. Here VISR still uses the full 5-sphere for its space of tasks, but it is trained with much smaller network architecture for both the successor fcatures & and variational approximation 6 (both consist fully- connected layers of 100 units with ReLU non-linearities, the latter L2-normalized so as to make mean predictions on the 5-sphere). We train this model for longer than necessary (960,000 trajectories of length 40 for 38,400,000 total steps) as to best capture what representations might look like at convergence.
Figure 3 shows each of the dimension of á across all states of the grid-world. It should be noted that, since these states were observed as one-hot vectors, all of the structure present is the result of the mutual information training objective rather than any corrclations in the input space.
Figure shows randomly sampled reward functions, generated by sampling vector uniformly the 5-sphere and taking the inner product with &. This demonstrates that the space of & contains many different partitionings of the state-space, which lends credence to our claim that externally defined reward functions likely to be not far outside of this space, and thus fast-inference can yield substantial benefits.
Figure shows the 49 value functions corresponding to the reward function sampled in Figure 4 These value functions were computed via generalized policy improvement over the policies from 10 uniformly sampled w's. The clear correspondance between these value functions and their respective reward functions demonstrate that even though VISR tasked with learning infinite space of value functions, it does not significantly suffer from underfitting.
13

---

Published as a conference paper at ICLR 2020
Figure 4: 49 randomly sampled reward functions learned by VISR.
Figure 5: The approximations to the optimal value functions for the reward functions in Figure computed by VISR through GPI on 10 randomly sampled policies.
14

---

Published as a conference paper at ICLR 2020
Algorithm 1: Training VISR
Randomly Initialize é network Randomly Initialize network for 6 :: 1,00 0o do sample from L2 normalized (O, ((dim(W))
1/ L2 normalized output layer
// dim(output) #A x dim(W)
// uniform ball
((,a/w) + v(.a.w) w) Tww,, Vo E A
for 1::11,71 do
Receive observation 8t from environment if using GPT then
Q (,,0) +- Q(:al/)Va E
for i: 10 do
sample w; from VMF(j = w w,K=
(= - v0 a, wx) w, Va E A
end
at f- e-gredy policy based on max; QT (5t.
else
0t - egreedy policy based on Qs+ ((x))-ww)
end
Take action a,, receive observation 8t41 from environment
= argmax, (b a, w) y ((s)) 784 w)
loss& = (1) (1) ae, (t,w)))))))))))) )1
loss& = ((t) Tw
//minimize Von-Mises NLL // minibatch in practice
Gradient descent step on vo and Φ
end
end
These value functions can be thought of as the desired cumulative state-occupancies, and appear represent distinct regions of the state space.
8.4 NETWORK A ARCHITECTURE
A distributed reinforcement learning setup was utilized to accelerate experimentation per Espeholt et al.] (2018). This involved having 100 separate actors, each rumning on its own instance of the environment. After every roll-out of 40 steps, the experiences added to queue. This queue is used by the centralized learner to calculate all of the losses and change the weights of the network, which are then passed back to the actors.
The roll-out length implicitly determines other hyper-parameters out of convenience, namely the amount of backpropagation through time is done before truncation (Werbos et al., 1990), as the sequential structure of the data is lost outside of the roll-out window. The task vector W is also resampled every 40 steps for similar reasons.
The network architecture is the same convolutional residual network as in Espeholt et al. (2018), with the following exceptions. & and s each have their own instance of this network (i.c. there is no parameter sharing). The & network is conditioned on a task vector which is pre-processed as in Borsa et al. (011)). Additionally, we found that individual cumulants in benefited from additional capacity, so each of the 5 cumulants used separate MLP with 256 hidden units to process the output the network trunk. While the trunk the IMPALA network has LSTM (Hochreiter and Schmidhuber 1997p, it is excluded from the @ network, as initial testing found that it destabilized training on some games. A target network used for vb, with update period of 10, 000 updates.
15

---

Published as a conference paper at ICLR 2020
8.5 HYPER-PARAMETERS
Due to the high computational cost, hyper-parameter optimization was minimal. The hyper- parameters were fixed across games and only optimized MsPacman, BankHeist, UpAndDown, and Pong). The Adam oplimizer (Kingma and Ba 2014; was used with learning rate of 10-1 and of 10-3 in Kapturowski et al. (2018). The dimensionality of task vectors was swept-over (with values between 2 and 50 considered),, with eventually chosen. We suspect the optimal value correlates with the amount of data available for reward regression. The discount factor 7 was .99. Standard batch size of 32. A constant e-greody action-scloction strategy with an с of 0.05 for both training and testing.
subset of 5 games (Asterix.
8.6 EXPERIMENTAL METHODS
All experiments were conducted as in Mnih et al. 22055 The frames are scaled to 84 x 84, normalized, and the most recent frames are stacked. At the beginning each episode, between 1 and 30 no-ops time-limit imposed on both training and testing episodes.
executed to provide source of stochasticity. A 5 minute
In all results (modulo some reported from other papers) are the average of random seeds per game per condition. Due to the high computational cost the controlled fast-inference experiments, for the experiments comparing the effect of training steps fast-inference performance (e.g. Figure G an online evaluation scheme was utilized. Rather than actually performing no-reward reinforcement. learning 2 distinct phases, reward information was exposed to 5 of the 100 actors which used the task vector resulting from solving the reward regression via OLS. This regression was continuously solved using the most recent 100, 000 experiences from these actors.
2Snne the default settings of the Atari environment were used, the rewards were clipped, though recent experiments suggest unclipped rewards would be superior
16

---

Published as a conference paper at ICLR 2020
boot Si
Figure 6: Fast-inference performance during unsupervised training. Full VISR in blue, random feature VISR in red. x-axis is training time in millions of frames. y-axis is human- normalized score post-task inference.
17

---

Published as a conference paper at ICLR 2020
0>
8
2
0
7
6
9
11
10
28
30
40
48
43
33
35 29 = 1.44 39
10
20.-2003111 + 0.469
24.13 = 2.28
Full 57 Games
(011
+09+33.0 3.39
-218 + 0.89
= 1.83
27.55 + 1.24
28.51 - 1.87
+ 1.99
102.31 102.31 5.64
40.01 + 0.66
109.16 1 6.65
41.19
49.66
16.65
M
Mnih
-316 0.16
+ 0.06
+ 0.22
0.34 + 0.14
3.77
0.33
6.80 0.28
1.49
2.16 1 0.84
6.81 + 1.04
3.60 1 0.15
1. 0.81
8.61
1.55
-
H Mdn
0.29
8.99
8
Table 2: Results on Atari Suite with standard deviations.
11
12
0>
5
6
8
2
0
7
7
37 4
6
0
9
9
34
36
26
25
8
35
41
28
33
35
34
38
47 Game Subset Burda et al. 2018
57.65 + 4.80
11.2 0.883
+ 0.25
= 2.36
+ 1.05
29.47 1.71
16.74 = 0.61
42. 42.00 - 0.95
1110886: 6.77
48.38 1 0.78
11.70 1 2.12 129.76 7.14
+ 1.01
24.51 +?
29.03 +?
16.26
58.47
47
28.07
29.47
19.64
121.08
M
58.
-1.99 + 0.01
-1.66 + 0.08
0.69 + 0.27
0.33 1 0.12
7.29 0.26
0.43
3.81 + 1.56
1.82
1.21 0.04
0.04
4.04
0.91
8.46 ?
7.32 ?
<>> H Mdn
1.31
9.42
5
0
7
7
2
6
7
5
14
13
21.77 + 1.264
4
19
26
25
25
25
22
13
+ 5.343
20
21
20
22
26 Game Subset Kaiser al. 2019
10.63
+ 1.07
81.65 + 3.06
20.00
1 2.16
- 2.09
+ 35.27
2.89
1.13
07 + 2.69
58.77 2.71
111.23 1: 0.21
+
16.94
27.88
50.33
3.60
36.20
52.95
10.12
13.0843.74444
43.74
33.80
58.23
128
M
-165 1..550000 0.01
-164 0.110
0.57
0.17 0.17 0.11
5.60 0.28
9.79 1 8.12
27.80 77.00266
2.23 0.67
8.20 0.14
0.01 0.36
7.24 + 1.22 1.22
9.50 + 2.11
5.55 1 0.48
6.59 1. 25
2.18
Mdn
2.18
5.55
GPI RF VISR @100k
Pos Reward NSQ 20
Q-DIAYN-50 50 100k
Q-DIAYN-5 @100k
IDF Curiosity @00
RF Curiosity 60
Q-DIAYN-5 @0
Q-DIAYN-50 @0
SimPLe &100k
Rainbow @100k
RF VISR 6110kk
GPI VISR 00100
DQN @10M
PPO @500k
NSQ @10M
VISR @100k:
Algorithm
VISR 80
18


Reviews:
[{"review_id": "rygRKCvOtr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper builds upon the idea of Successor Features; this is a principle used to facilitate generalization beyond the finite set of behaviors being explicitly learned by an MDP. The proposed paper ameliorates the need of defining the reward function as linear in some grounded feature space by resorting to variational autoencoder arguments. The derivations are correct, the motivation adequate, the experiments diverse and convincing. The literature review us up to date and the comparisons proper. This is a valuable contribution to the field. \n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "SyekDcTg9B", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper proposes an algorithm to combine the ideas of unsupervised skill/option discovery (Eysenbach et. al., 2018, Gregor et. al., 2016, referred to as “BMI” in the paper) with successor features “SFs” (Barreto et. al., 2017, 2018). While unsupervised skill/option discovery algorithms employ mutual information maximization of visited states and the latent variables corresponding to options (typically discrete), this paper adds a restriction that this latent variable (now continuous) should be the task vector specified by some learnt successor features.\n\nWith such a restriction, the algorithm can now be used in an unsupervised pre-training stage to learn conditional policies corresponding to several different task vectors and can be used to directly infer (without training or fine-tuning) a good policy for a supervised phase where external reward is present (i.e. via GPI from Barreto et. al., 2018) by simply regressing to the best task vector.\n\nSuch unsupervised pre-training is shown to outperform DIAYN (Eysenbach et. al., 2018) in 3 different Atari suites (including the full 57 game suite) and also ablations to the proposed model where GPI and SFs are excluded individually.\n\nDecision:\nI vote for accept as this paper proposes a novel technique to combine mutual information based intrinsic control objectives with successor features, which allow for combining the benefits of both in a complementary way. An unsupervised phase can now discover good conditional policies with successor features which can be used to infer a good policy to solve an external reward task in a supervised phase, with such a policy capable of attaining human level performance in several Atari games and outperforming several baselines such as DQNs in limited data regimes.\n\nOther comments:\n- The technique for enforcing the restriction in Eq. 10, as well as being able to use it with generalized policy improvement is a good novel contribution in the paper.\n\n- The detailed comparison with baselines on the full Atari suite is sufficient to back the claims in the paper that the strengths of BMI and SFs do complement each other.\n\n- The fact that fast task inference is sufficient to get good performance is impressive i.e. without the need to fine-tune the best inferred policy.\n\n\nMinor typos:\n- In section 5 para 5, “UFVA” -> “UVFA”, “UFSA” -> “USFA”.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJx2exIRFS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors address the problem of finding optimal policies in reinforcement learning problems after an initial unsupervised phase in which the agent can interact with the environment without receiving rewards.  After this initial phase, the agent can again interact with the environment while having access to the reward function. To address this specific setting, the authors propose to use the successor feature representation of policies and combine it with methods that estimate policies in the unsupervised setting (without a reward function) by maximizing the mutual information of a policy-conditioning variable and the agent behaviour. The result is a method called Variational Intrinsic Successor Features (VISF) which obtains significant performance gains on the full Atari task suite in a setting of few-step RL with unsupervised pre-training. The main contribution seems to parameterize the successor features in terms of a variable specifying the policy. This variable will be the same as the linear weights in the linear model for the reward assume by the successor features representation. Finally, a discriminator aims to predict the linear weights of the policy from the observed state-feature representation.\n\nClarity:\n\nI think this is one of the weaknesses of the paper. The writing and clarity could be significantly improved. How do you go from equation 8 to equation 9? How does q lower boun p? In the paragraph after equation 9 the authors mention the score function estimator. But very few details are given. After reading the paper, my impression is that the reproducibility of the results could be very hard, because of the lack of details of the specific implementation. The authors also do not mention that the code will be publicly available after acceptance.\n\nI feel that, to better understand the method, the authors should include experiments in simple and easy to understand synthetic environments which can be more illustrative than ATARI.\n\nWhat is the difference between the policy parameters theta and the conditioning variable z?\n\nNovelty:\n\nThe proposed approach is novel up to my knowledge. I find the idea of parameterizing the successor features in terms of the policy parameters very innovative. \n\nQuality:\n\nThe proposed approach seems well justified and the experiments performed indicate that the method can be useful in practice. However, I think it would be very useful to have results on other environments besides ATARI. For example,  DIAYN contains experiments on a wide range of tasks, including gridworld style tabular experiments to illustrate what their method does. This work would benefit from similar simpler and easier to understand synthetic environments (unlike ATARI).\n\nSignificance:\n\nThe proposed contribution seems significant as illustrated by the experimental results and the novel methodological contributions. However, the lack of clarity and the difficulty in the reproduction of the results limit this.\n\nSome minor comments:\n\nI recommend the authors to remove references in the abstract.\n\nUpdate after the authors' rebuttal:\n\nAfter looking at the response from the authors, I believe that they have successfully addressed my concerns. \nTherefore, I have decided to update my rating and vote for acceptance. I am looking forward to seeing the python notebook with the implementation of the VISR algorithm.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work uses a variational autoencoder-based approach to combine the benefits of recent methods that learn policies with behavioral diversity with the advantages of successor representations, addressing the generalization and slow inference problems of competing methods such as DIAYN.  After discussion of the author rebuttal, the reviewers all agreed on the significant contribution of the paper and that concerns about clarity were sufficiently addressed.  Thus, I recommend this paper for acceptance.
Decision: Accept (Talk)