Entropic Desired Dynamics for Intrinsic Control
Authors: Steven Stenberg Hansen
Guillaume Desjardins
Kate Baumli
David Warde-Farley
Nicolas Heess
Simon Osindero
Volodymyr Mnih
Venue: NeurIPS 2021 Poster (2021)
Keywords: intrinsic control
skill discovery
exploration
unsupervised reinforcement learning

Abstract: An agent might be said, informally, to have mastery of its environment when it has maximised the effective number of states it can reliably reach. In practice, this often means maximizing the number of latent codes that can be discriminated from future states under some short time horizon (e.g. \cite{eysenbach2018diversity}). By situating these latent codes in a globally consistent coordinate system, we show that agents can reliably reach more states in the long term while still optimizing a local objective. A simple instantiation of this idea, \textbf{E}ntropic \textbf{D}esired \textbf{D}ynamics for \textbf{I}ntrinsic \textbf{C}on\textbf{T}rol (EDDICT), assumes fixed additive latent dynamics, which results in tractable learning and an interpretable latent space. 
Compared to prior methods, EDDICT's globally consistent codes allow it to be far more exploratory, as demonstrated by improved state coverage and increased unsupervised performance on hard exploration games such as Montezuma's Revenge.

Entropic Desired Dynamics for Intrinsic Control
Steven Hansen" DeepMind
Guillaume Desjardins DeepMind
Kate Baumli DeepMind
David Warde-Farley DeepMind
Nicolas Heess
Simon Osindero
Volodymyr Mnih DeepMind
DeepMind
DeepMind
Abstract
An agent might be said, informally, have mastery of its environment when it has maximised the effective number of states it can reliably reach. In practice, this often means maximizing the number of latent codes that be discriminated from future states under some short time horizon (e.g. [15)). By situating these latent codes in globally consistent coordinate system, we show that agents can reliably reach more states in the long term while still optimizing local objective. A simple instantiation of this idea, Entropic Desired Dynamics for Intrinsic ConTrol (EDDICT), assumes fixed additive latent dynamics, which results in tractable learning and an interpretable latent space. Compared prior methods, EDDICT's globally consistent codes allow it to be far more exploratory, as demonstrated by improved state coverage and increased unsupervised performance on hard exploration games such as Montezuma's 's Revenge.
Introduction
Endowing reinforcement learning agents with the ability to learn effectively from unsupervised interaction with the environment, i.e. without access extrinsic reward signal, has the potential to make reinforcement learning practical in settings where the tasks the agent will face are initially unknown where task feedback is expensive. The natural question is: what should the agent learn in the absence of extrinsic rewards? One appealing guiding principle is maximizing the number of states the agent can reach and which it can reliably return.
Intrinsic control methods have shown promise in this direction. By maximizing the mutual information between a latent code and future states reached by policy conditioned this code, intrinsic control methods learn to map latent codes to behaviors from which the code can be inferred. One major limitation of such approaches is that the latent codes usually sampled from fixed prior distribution p(z). Using fixed prior means that such approaches are unable to learn codes that correspond states that cannot be reached in the time horizon T. since any code be sampled in any state. Simply increasing the time horizon T' does not solve the problem since it leads to a sparser learning signal. Learning a state-dependent prior has proven be difficult and has been shown lead to fewer learned codes/goal states [15]. This inability to learn how to reach distant states limits the usefulness of such intrinsic control approaches.
We propose to sidestep this limitation by replacing the fixed code distribution p(2) with a fixed dynamics model over codes p(2: 1) Our algorithm, Entropic Desired Dynamies for Intrinsic ConTrol (EDDICT), learns to map sequences latent codes sampled from this dynamics model to behaviors for which the state transition dynamics in the environment match the latent code dynamics. EDDICT learns to map each 2, to a state that reachable from the state corresponding to 2
"Conresoondence to stevenhansen stevenhansen deepmind.com
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

20
21
21
20
21
20
21
TO
F1
TO
TO
TO
(d)
(b)
(c)
(d)
Figure 1: Graphical models for various priors and posteriors of interest. Circles denote random variables which are observed (shaded) or latent (white), with diamonds denoting deterministic quantities. (a) Prior
particular trajectory consisting of two sub-trajectories (7o, TI ), and auxiliary variables {10, (b) Posterior inference with independent codes, as in prior work. (c) Naive posterior inference for the sub-trajectory {21, Ti conditioned on the past. (d) Posterior inference with hindsight. Despite zo being observed, we infer 21 based on the most likely code 20 to have generated To, using the variational reverse predictor (dashed line).
allowing it to reach states much farther than the time horizon I using sequences of codes z. We show that even highly constrained latent dynamics (i.e. additive noise) are sufficient both interpret latent codes in terms of their corresponding locations in state space, and encourage exploratory behavior to far greater extent when compared to prior methods.
2 Notation
Our environment is a special case of a Markov Decision Process (MDP) without rewards or terminal signals: M (S, A, P, Po). P.). S is the state space. A the action space, P(s++ st, az) the conditional distribution representing the state transition dynamics when taking action E A from state s, S and (s(s) the initial state distribution. For simplicity, we present our method in the episodic setting with episodes of length T MK, but relax this assumption in practice. Agents interact with the environment according a policy na (a s) with parameters 0, yielding trajectories
(s. dg, 81, "1. ST], sT]], and distributed as nT* (T) = Po(5o) II=o P( St: To)To(( st).
It will be useful for us to segment given trajectory т into sub-trajectories of length K, with [so: To: To: T.. and T; [aix; Six+1 0xx+1:" 11 (((+1)x-11 8((+1)x) Note that тę defined to include ((++1)xx but not the state Six from which dix was sampled. With slight abuse of notation and denoting T-1 = Sp, we rewrite TT* (r) = (a(s)) M-1 p (-1)). with M the number of sub-trajectories per episode and (T; T:-1) le x P(s 88, ar (m) st).
Hierarchical agents sample a high-level goal or latent variable a ~ p(z) every K steps, and interact with the environment via parametric conditional low-level policy Ta(a s. which can be thought of as a fixed duration option [42]. Composing p(z), Ta(a s,2) and transition dynamics yields an augmented trajectory A [Sp, 20, To, 21, Ti, whose distribution decomposes as p (A) M Po(s)) II - p(z))= (T: Ti-1, 24), with p (Fa 74-1,2) defined analogously to T ( 7-11) with conditional policy n, (a, =t)
To simplify exposition,
index sequences at the timescale of sub-trajectories using index i, e.g. [zi, T4, 2141, 7141,"" and reserve index t for indexing sequences at the granular timescale of actions, e.g. [St, ar, St--1-111111 de.1.-....... Concretely, indexing by should be interpreted ik, as in s; 8ix- For general sequence I I1, - we define Tet == [To ... xt-1] i] and extend this notation augmented trajectories as follows: Aci
[s0, 20, To. Zi- T--
3 Method
We would like to learn goal directed agents, which are capable of reaching any state E S, given a goal or state embedding Z. Extending earlier work empowerment 40 Variational
TThe dependence on 7--1 is thus due to Six E Ti-1-
'e avoid introducing new notation for the corresponding remi-MDP our present notation allows us to reason about sub srbetrajectories for both standard policies a(a s) and conditional policies a(a s,

---

Intrinsic Control (VIC) [9] and related methods (e.g. (15] propose to achieve this by learning conditional policy aaa s, which maximizes I[2; r), the mutual information between the latent code and (possibly subset of) the resulting trajectory obtained by following я. Unfortunately, this objective can be difficult to train in practice as we scale both the number of options and the horizon over which the code is executed [5]. Our method addresses both of these issues in principled manner by introducing temporal dependencies between a sequence of latent codes, evolving under simple linear dynamics, which decomposes the full objective into sum of local mutual information lower-bounds, without loss coherence of the global code.
Entropic Desired Dynamics for Instrinsic Control (EDDICT) can be understood from the perspective of divergence minimization [20] [3 Concretely, we can define a prior policy μ which induces a distribution p" (A) over the space of augmented trajectories, We then learn a (posterior) policy by minimizing the Kl.-divergence between yd (A) and (A).
3.1
VIC as Divergence Minimization
Given prior policy μ.. we construct a prior distribution over an augmented space (T, 2). with auxiliary variables [3 E Z. as p" T2 = p',)) r). The conditional (.. r) is a learnt predictor, with parameters w E f2, which aims to predict from the underlying trajectory.
We can show that an entropy regularized version of VIC obtained by maximizing Oen-vic(O, w) =
-KL (( (2)lp"((,, wrt. the parameters of Ty and qw with p(z) a fixed or learnt distribution over options. Intuitively, we seek code conditioned policy which generates trajectories having high probability under our trajectory prior, and from which 2 can be inferred in hindsight. After some algebra, this simplifies to:
To (at
S2s 2)
Oen-Vic(O,,u w) = Espp(s) (rla)
log qmo [T) logg(=))
log
plat
8e)
Id(=;t)
zegularizer
34 to the mutual information 42; Ellog p (z T) log p(=)] using reverse predictor q trained to approximate the true posterior distribution p (z T) In expectation, the regularization terms correspond to a sum of KL- divergences between our conditional policy and the prior over actions fl The original objective Ovic is obtained by dropping this regularizer and choosing the reverse predictor q == q= 80, 87), which predicts z from the first and last states of the trajectory.
In the above, T,(2;7 7) refers to the variational lower-bound 2
Since we focus on discrete action spaces, we set μ to a uniform distribution over actions, causing b to revert standard entropy rewards [48] In practice, we optimize the above objective using value-based reinforcement learning algorithm and e-greedy policies (in lieu of Boltzmann policy), and thus omit these terms. Note that the auxiliary variable perspective of VIC can also be found in Hausman et al. [23
3.2 Incorporating Temporal Dynamies
Instead of sampling single goal to be reached within the duration of the episode, it may be preferable to sample a sequence of codes either as relative (or local) goals, parameterized relative to the agent's current position, or as way points, a sequence of global goal coordinates which the agent should visit in sequence.
Relative vs Global Codes. Local goals can be implemented for VIC by resampling latent code every K steps and maximizing a sum of local objectives of the type I $:41 8t), with option 2; initiated from state s,. We describe these codes as having local semantics, as an option 2; should only be inferable in the context of the relationship between its initiation state s; and final state 8,41. In essence, each 2; represents local displacement which the low-level policy should execute. In contrast, the strategy of sampling way points in some global frame of reference would require maximizing
'The regularization terms emerge from the fact that the transition dynamics are shared by both p²(r) and (T | z), and thus cancel out in the computation of the Kivdiveree
3

---

Ia(2; 8t41). Unfortunately, this would seem to require learning a state-dependen high-level policy which gives higher probability goals 2; which are reachable (in K' steps) from s;-
Ours a hybrid of these two approaches: by specifying goals relative to previously sampled codes, in the form of Markov chain with simple linear dynamics, EDDICT can recover codes with global semantics while avoiding the need to explicitly train a high-level policy.
EDDICT Prior. As in Section 3.1 we specify joint distribution over the set of sub-trajectories
{ and auxiliary variables { ie( Recall that T-1 = 80. Our prior for an augmented
trajectory A is given by:
M-1
( 00)
II (Tr Ti-1 (w mt),
again with J uniform distribution over actions. As shall see, making the priori assertion that zi is conditionally independent of Ti-1 given т; will ensure that our objective breaks down as sum of local objectives, amenable greedy optimization. This prior is illustrated in Fig. Ta Using the reverse predictor ge = q2, s141), which predicts from 8,+1 alone (the last state of Ty) will then induce codes with global goal semantics.
EDDICT Posterior. We structure our posterior around goal-conditioned policies (a | 8, z), but modified to account for the temporal structure of our prior. We incorporate temporal dependencies between the latent codes in the form of Markov chain p(z: 21-1) with initial distribution p(20). Defining p(z0 2-1) p(zo), we write:
M-1
p (A so) = II
plzz zi-1)p T:
i=0
We now expand the negative KL-divergence corresponding to this choice of prior, posterior and reverse predictor:
KL 1pd (A so)llp so)] = Ex (A|sa)
{
M
Σ
log
gar
(24
81+1)
pd (Tr [T-1))
p( 2
z4-1)
pi (Ti Ti-1,
1=0
The objective then obtained by dropping action entropy terms.
11-1
O00, w)
Σ
To (Acs) (xstlle))))
log d., (..DKTTT (2; 8141)
log p(2s 21-1)
1=
(8,
The objective thus breaks down as a sum of M terms, defined as O0 w) = (iO(+) (f, w; 2;-1, T-1)). is worth pointing out that in expectation, C constitutes a valid lower- bound A2; 8141 24-1) despite the reverse predictor not conditioning 2;-1.
3.3 EDDICT Objective
We obtain EDDICT by incorporating (i) greedy optimization, (ii) hindsight correction, and (iii) linear dynamics into the objective of Equation]
Greedy Optimization. Define the effective entropy as the difference in prgabbbbaailities given by the reverse predictor and the high-level policy over options (cf. O). As written, the objective aims to maximize the long term sum of effective entropies: concretely, each code 2, should seek to be entropic and discemible from 8;41 but also lead states from which future options are themselves
"Note that we use O refer the term Eq. which a function particular value z and T(-1. (+) is reserved for the expected value of (+) under p = (Act).

---

discernible. The variance of any return estimator will thus increase with the number of option periods. To avoid this issue, EDDICT optimizes Eq.
I in greedy- as:
M
OGREEDY w)
E (Ae)))
(0, (1,,;, (011022-1171-11
where we have omitted the policy parameters from the sampling distribution p" (Azi), which is thus considered fixed with respect to the optimization process. Concretely, this can be implemented by treating each option period as pseudo-episode, i.e. using discount factors which are zero on option boundaries as shown in Algorithm [
Hindsight Correction. Unfortunately, the above objective is rather brittle as the distribution over 2 is conditioned solely on 2i-1, and ignores the underlying state in which the code is sampled. We
can improve on this open-loop formulation by reasoning hindsight. From Eq. 2 O is computed
in expectation under p" (Acz) which includes the joint y (2;-1. T4-1 A We rewrite this joint
as p" (T:-1 Açi- (es-1)p"(( (2;-1 Tz-1) R p" (T:-1 (eci-19g000e
8;), since qu a variational
approximation to the true posterior by construction. Incorporating this approximation to Eq. 2 yields
the final objective:
M
OEDDICT(O, w) = Σ Ep (Mez) Eognncc (2, a1ss) [O0) w; 2;-1, 11 1- 1=
(3)
Concretely, when sampling 2; ~ p(z; 1), we thus condition on the code most likely to have yielded state 8t under the reverse predictor. Importantly, this objective induces cross-entropy term between the target distribution q. (2--1 8, s,)p(zz 24-1) and q (z: 8.+1). This ensures that predictions made from 8,+1 are consistent with those from 8;, under our latent state dynamics.
Linear Dynamics The final piece of the puzzle concerns the choice of code distribution. We cannot employ the VIC strategy of fixed entropic distribution, since our codes form a Markov chain. We would further like to avoid the full HRL problem, which would require us to have a parameterized high-level policy over options. Choosing AR(1) process the conditional code distribution satisfies both of these requirements and we thus set p(2: (2-1))= = 2 + A,, with A, sampled from either isotropic Gaussian AR(1) process is that it ensures that the marginal code entropy increases monotonically with each option period (more states visited) while the conditional entropy remains constant (same number of states reachable from any given state), as shown in Fig. 4b] Finally, hard coding the dynamics to be linear, versus learning parametric policy over codes, naturally imposes an interpretable Euclidian topology in code space, as shown in Fig. 4a
uniform distribution
the disc. Another useful property of the
3.4 Algorithm
We now provide more mechanistic view of EDDICT. Algorithm presents online version of the algorithm, with details of the distributed setup used in our experiments presented below.
We optimize our objective using a distributed deep reinforcement learning system [4] based on Peng's Q()) 37 and c-greedy policies. The system consists of a centralized learner, replay buffer (3). and a set of distributed workers each interfacing with their own copy of the environment. Given the latest parameter values and current state the environment 8; (local to each worker), actors sample z; and generate sub-trajen sub-trajectory тą by executing Ia 8, s,2;) for K steps in the environment. The resulting (s,, A., T.) is then fed back to the replay buffer, from which the learner consumes data to perform off-policy updates. Storing the initiation state and offset A,, allows the learner to recompute the code 24 as required using the most up-to-date version of the reverse predictor. Intrinsic rewards derived from the reverse predictor are similarly computed on the learner.
In practice, the learner maximizes OEDDICT by summing two losses. The first implements policy iteration by minimizing the mean-squared error between a target return, computed by Peng's Q(A) under target network (32), and the current Q-value estimates. Our greedy optimization procedure yields a single non-zero reward, log (= si+1), which is received upon option termination. The second loss corresponds to cross-entropy loss of the reverse predictor found in Eq. 3 With (z s) = (s), for some parametric function fw. this amounts to minimizing
5

---

Algorithm 1: EDDICT
Input Environment dynamics P. initial state so- policy me. code predictor gun(z = (N 1).
option period K, discount ", code dimension
[so],
repeat
A; U(D))
e.g. uniform over a disc, isotrppic normal
#: - fu((sk) ) A. for
+ t
iK: 1i+1)K-1 do
(Tlls1, 2::
II parametric epsilon- greedy
81+1 P(ss+11ls,,
If Commpute intrinsic rewards. Note: entropy of code distribution constant under linear dynamics. Ft41 4- log qw 81-1) if (=++)))--110)) else 0
If (optional) add entropy rewards.
7641 + 0 (=(=t)) else y
S141 f- S141
Il augment state with code
Append Cr, Sr+1, F++1, 7++1 T.
Update 0 with any reinforeement learning algorithm on the sub-trajectory т. Il Minimize cross-entropy loss from Eq. [3] for linear dynamics and Gaussion reverse predictor. Update w by gradient descent on A (fw (8(()(()))) - (sux)
( i ++1
|| - (fu( (s.)) This loss is extremely intuitive: train the reverse predictor such that the inferred latent state from 8,, matches the inferred state from 8141 under our latent dynamics. As in 44D we found that uninformative prior performed best in practice (despite our choice of isotropic Gaussian for the predictor), and thus sample A; from a uniform distribution on the disc
Concretely, we parameterize the action-value function Qo (s, a, 2) as an MLP operating on state embeddings, derived from a ResNet [24] and linear action and code embeddings. the reverse predictor qu operates on the same state embeddings as the Q-function, with gradients from both objectives being backpropagated into the ResNet. Complete details of the architecture can be found in the Appendix.
our experiments.
4 Related Work
Intrinsic Control and Empowerment. EDDICT can best be thought as incorporating temporal structure into intrinsic control algorithms 19
15 1 21), which build on empowerment [26] (34.. Relative Variational Intrinsic Control (RVIC) K also extends an intrinsic control objective, but does so by penalizing codes predictable from a single state. leading to codes representing state-agnostie behaviors. In contrast, the parameterization of our reverse predictor, along with fixed high-level policy over options, ensure that EDDICT's codes are reachable from the states in which they are sampled while preserving global state semantics.
is well known that the VIC objective is difficult to train when the code space is large D. At a high-level, EDDICT tackles this issue by breaking down this single goal into sequence of sub-goals This is orthogonal to the approach of Achiam et al. []]. which increases the number of available options over time. HIDIO (51) proposes an objective similar to ours (discriminator rewards over sub-trajectories,
but sample options using a state-dependent high-level policy trained maximize extrinsic rewards over the semi-MDP induced by the low-level policy.
The notion of reusable behavior and hierarchy has long history in the RL literature [e.g. 42 2. In comparison to EDDICT existing work can be broadly categorized with respect to the signal that is used for behavior induction and the nature of the learned representation. EDDICT bears similarity to unsupervised skill discovery methods that induce behavior in the absence of external rewards usually for the in downstream tasks, including [16 36) Other approaches learn skills or behavior representations from demonstrations provided by humans or expert policies [e.g.
Skill Discovery and HRL.
'Our variational bound is looser as a result, since our variational posterior is not matched to the prior. Improving the modeling assumptions of the reverse predictor, e.g. by using a truncated Gaussian, is left for future work.

---

Code Norm Quartiles
Montezuma Room 1 and 2
Figure 2
Figure 3: Montezuma's revenge, (left) Typical observations from the first (left) and second (right) rooms. (right) Observations from a trained EDDICT agent, sorted by L2-norm of corresponding code and aggregated into quartiles. Images are generated by taking the pixel-wise maximum
17 38 while optimizing the reward for one or multiple tasks le.g. 6 or via subgoals that are associated with explicit rewards in a predefined [e.g. space [e.g. |46 477.
25
23
29
18
28
35
or learned
Methodology. Auxiliary variables have long history in variational inference 3 30 as way to obtain more expressive posteriors and serve a similar purpose in the context of EDDICT/VIC. Tirumala et al. [45] incorporates an AR(I) process in the context of HRL and skill transfer, but did so within the prior which served to regularize a set of task-specific high-level policies. In contrast, the AR(1) process in EDDICT ensures that the high-level policy samples goals which are reachable from the current state. AR(1) processes over latent temporal sequences have also been used to prevent "posterior collapse" in VAEs with powerful autoregressive decoders (39), an analogous
phenomenon to "option collapse" in HRL. Hindsight reasoning has seen a multitude of applications reinforcement learning, improving credit assignment [22], training of goal-conditioned policies in sparse reward settings
5 and off-policy learning of options in HRL. [50]
Experiments
Here evaluate EDDICT's learned representations and behavior, and contrast them to prior work in the space of intrinsic control (or skill discovery) methods. We assess the learned representations qualitatively by looking at how well they correspond to privileged information known be relevant to down stream tasks. Namely, the state dimensions given in the DeepMind Control Suite 44 and the avatar coordinates in the Atari Learning Environment (ALE) p We stress that this privileged information is not used during training in any way, with reverse predictors operating on the same input as the Q-function.
The quality of learned behaviors is measured in terms of exploration; we posit that EDDICT explores in the space of controllable outcomes, and that this style of exploration results in reaching many states of interest. To assess this quantitatively, we compare unsupervised behavior policies in terms of reward achievement on the Atari game Montezuma's 's Revenge, which is known require sophisticated exploration order to progress. Additionally, we look at the number of unique states visited per episode using privileged environment information (L.e the underlying RAM states in ALE), this is a proxy for state coverage that is agnostic to the specific reward function of the game [4]. To look specifically at the claim that EDDICT explores the controllable states, also measure estimate of the mutual information between the marginal code distribution and the marginal state distribution.
We consider the following basclines for evaluation: VIC (19, RVIC [8] and ablated version of EDDICT. VIC refers to a scalable variant introduced in [8) that uses a fixed Categorical distribution over 50 outcomes. In the EDDICT ablation (EDDICT-A) A) the code proposal mechanism is simplified by substituting 4 == A for EDDICT's 2:+1 := 2 + A, Note that the reverse predictor remains unchanged, and thus tries to predict A directly from $++1- All algorithms were implemented in the same codebase and thus share the same network architecture and reinforcement learning method.
For the results on Montezuma'ss 's Revenge, we further include results for Q(X) agent trained to maximize the game score (which other methods do not have access to), again matched terms of network architecture.
7

---

1.75
Maggher I.. Onyditornl Emompp
2.75
-3
3.00
0.25
(b)
Figure 4: (a) Point Mass. 2D codes colored by ground truth x (left) and (right) coordinates of the point-mass mass a version of the environment with "U" shaped wall. (top) EDDICT-A (bottom) EDDICT. (b) Montezuma's 's Revenge. Estimated marginal code entropy H[[z.] (solid) and conditional
entropy H[2 81+1] (dashed). Despite codes being less predictable, EDDICT achieves higher mutual information Iy(25; 3141), as measured under the marginal code distribution (shaded).
5.1 Codes as Representations of State
Codes as state representations To illustrate EDDICT's ability to map codes to the controllable aspects of the environment, we have trained EDDICT on a simple continuous control task from the DeepMind Control Suite [43]. This environment comes with set of ground truth state dimensions required to compute the dynamics, a subset of which are under the agent's control. We can thus allow EDDICT to train from raw pixels, and then evaluate the correspondence between the code values and ground truth state dimensions.
Shown in Figure 4a is the state representation learned on version of the Control Suite point mass domain which is modified to include obstacle walls in the shape of "U" to increase the difficulty and exploration requirements of the domain. EDDICT successfully recovers the ground truth coordinates of the point mass position (under the agent's s control), but not the target position (randomly set per episode and not under the agent's control). This property of not representing what can not be controllable effectively solves the well known "noisy TV problem" [10].
Standard intrinsic control methods also have this property of only representing the controllable, but they lack any incentive to represent states unreachable in a single unroll as being distinct. This clearly demonstrated by our ablation's performance, wherein the relationship between nearby states much more tenuous.
Code norm as difficulty Since our desired latent dynamics consist of sum of zero centered IID variables, the marginal code distribution will also centered at zero, with the probability density dissipating as function of the code norm. Assuming EDDICT manages to form mapping between latent codes and states, this implics that less frequently visited states will have higher code norm.
We test this hypothesis empirically, by training EDDICT on Montezuma's 's Revenge. In this game, the agent is represented by an avatar who can move locally around the screen, but who can easily die. When this occurs, the avatar is reborn on platform in the middle of the screen. This means that by construction, states where the avatar is near the center of the screen easier reach than those farther out. Our hypothesis would thus suggest that EDDICT should assign center states with low norm codes and more peripheral states with high norm codes. As Figure Shgurs, happens in practice. One interesting subtlety is that the game actually contains several rooms, each with a different background. These are exceedingly hard to reach and, as expected, this results in EDDICT assigns these states the highest code norms of all.
shows, this exactly what
5.2 Control to Explore
EDDICT's representations only tell half of story. Since codes represent states, and the marginal code entropy increases monotonically, this suggests EDDICT's fixed high-level policy should result in exploratory behavior. But unlike most traditional work on the exploration problem, EDDICT focuses its exploration only what it can control.

---

montezuma, ma_ rereaaeeeeeeeeeee
(10))
2.
Sverage Eptoodic Fecurrn
30
2.0
60
4.0
10
0.0
Faams
16c
Figure 5: Comparing exploratory behavior in Montezuma's Revenge. (right) Game score. All methods except Q()) did not have access to this during training. (left) Number unique avatar positions visited. This is proxy for coverage of the controllable states.
Measuring control Intrinsic control methods measure their control over the environment in terms of the mutual information between a code and its downstream effects of the environment, For episodic resampled but independent codes, this can be estimated straightforwardly the effective code entropy (cf. ( Eq. [ averaged over mini-batch. EDDICT however defines a Markov chain over codes, and thus requires to compute entropy over the marginal code distribution. To do so, we fit an isotropic Gaussian to all of the codes in the batch, yielding mean and variance estimates and g². Our final estimate of I (z; 5;+1) is then H (N, o²)]. plus the average log-prediction reward log (2g 1 over the batch.
Since this lower bound, meaningful when comparing models of the same architecture. This metric thus allows us to answer the question: does EDDICT control the environment to greater degree than its fixed code distribution equivalent? As Figure 4b shows, this is very much the case. Interestingly, this is true despite the codes in general being less predictable; the extra entropy from the sequential sampling more than makes up for it.
not an unbiased estimate, the relative values should still be
Exploring what matters In order evaluate EDDICT's behavior policy, we must designate proxy metric for exploration quality. For Atari 2600, game score is an obvious candidate. But since the methods under consideration learn without sometimes quite noisy. To give a more complete view of exploratory behavior, we also include two 'coverage' metrics which counts the number of unique RAM states the agent visits per episode and over its lifetime. Using the information given in 4. we only count the RAM states corresponding to the controllable avatar. We evaluate EDDICT on these metries across amenable games, and as shown in Table 1 we find that in the majority of cases EDDICT outperforms RVIC, VIC and its ablation on one or more metries. RVIC was the most competitive method, and suggests that global codes are not the only way for intrinsic control methods to yield exploratory behavior.
to score or episode boundaries, this metric is
Of particular interest is Montezuma's 's Revenge, one of the hardest exploration games, attested by the numerous reinforcement learning papers that fail to receive non-zero scores (e.g. 33 14). As shown in Figure EDDICT outperforms other intrinsic control methods by wide margin.
For additional context, we provide a broader set of baseline results on Montezuma's 's Revenge in Table 2 of the Supplemental. These span entropy curiosity-band bassed algorithms which learn single unified policy, instead the code-conditional policies recovered by algorithms in the VIC family, with Never-Give Up 7 and Random Network Distillation [11 greatly outperforming EDDICT on the metric of average episodic reward.
Limitations and Discussion
Endowing agents with the ability to master the environment is an important step towards more general purpose agents, it allows learning in any circumstance without any requirement of a task specific

---

Game
DIAYN
RVIC Cat(16)
EDDICT
EDDICT-A
Berzerk
1,75, 24.8, 365
0.156, 31.3, 138,
0.382, 61.3, 477,
0.562. 10.6, 584
Hero
1.82, 24.1, 1.35k
2.58, 35.2, 805,
2.1, 32.5, 1.34k.
0.856, 20.7, 68.7
Montezuma
4.379.4.611 0
0.577, 5.61, 0.
1.17. 1.07,3099
0,674, 4.54,
Ms. Pacman
0.475, 1.72, 652
0.288, 1.72, 397,
0.349, 1.72, 587,
0.262. 1.72, 360
Private Eye
4.61, 87, 1.54k
5,41, 85.2, 886,
4.86, 86.8, 1.07k.
4.01, 74.5, -43.6
Seaquest
0.346, 10.9, 18.7
1.27, 10.9, 143,
1.91, 10.9, 400,
1.56, 10.9, 238
Table I: Results 6 Atari games 1B frames. Each tuple A,B,C represents mean of: (A) Episodic Coverage (-10%) (B) Lifetime Coverage (-103) (C) Average return. For EDDICT-based agents, we pick the best metric across code sizes. Metries which are best across agents, based on mean performance over 3 seeds, are shown in bold. In cases where there is tie across all methods for metric, none bolded. The full set of results, including VIC, random policy baselines, scores with standard deviations and training curves can be found in the supplemental material.
reward function. EDDICT can both explore and control the environment by learning latent codes that make sense of states in globally consistent coordinate system. But in terms of pure exploration, EDDICT falls short of state of the art methods that learn a single policy (e.g. [I].. Understanding what these advancements mean for learning code-conditional policies is promising future direction.
Additive dynamics can not capture important aspects of some environments that we might wish for our agents represent, such dynamics that irreversible or state-dependent. Ideally, general purpose function approximators (e.g. neural networks) could be used to specify more general dynamics, but how to make such learning tractable while preserving the advantageous properties of EDDICT remains an important open question.
In addition to its inherent merits for environment exploration and manipulation, EDDICT's novel state to code mapping and code transition function could be used aid local planning, or could serve as compact representation on top of which to learn policies, good state similarity metric for goal- RI., or aid many other unlisted tasks. All of these directions are left future work.
7 Societal Impact
Unsupervised reinforcement learning in general, and intrinsic control methods in particular, are far from being commercialized due their insufficient data efficiency and lack of validation in real world environments. However, when this is no longer the case, these methods could significantly reduce the human cost of setting up systems that interact with humans (e.g. robotics), these methods limit the need for handcrafted reward functions and the collection of human preferences. But this benefit comes with cost interpretability and safety. The information theoretic objectives of the methods lead to behavior that can be very hard to predict a priori (e.g. what does 'controlling your environment' look like?). Furthermore, safety constraints might be harder to specify in the absence of a closed-font form reward function. As these methods mature, the emphasis should shift from raw performance to more nuanced approach that addresses these societal concerns head on.
8 Acknowledgements
We would like to thank Stephen Spencer for providing engineering support. We further thank Yury Sulsky and Arturo Bajuelos, who contributed to results found in Appendix B, showing how EDDICT can be incorporated into the exploration policy of standard RL agents.
References
[1] J. Achiam, H. Edwards, D. Amodei, and P. Abbeel. Variational option discovery algorithms. arXiv preprint arXiv: 1xXiv:8877110900 2018
[2] D. B. F. Agakov. The im algorithm: variational approach to information maximization. Advances in Neural Information Processing Systems, 16:201, 2004.
10

---

[3] F. V. Agakov and D. Barber. An auxiliary variational method. In International Conference on Neural Information Processing, pages 561-566. Springer. 2004.
[4] A. Anand, E. Racah, S. Ozair, Y. Bengio, M.-A. Côté, and R. D. Hjelm. Unsupervised state representation learning in atari. arXiv preprint Stti:::99060022200 2019.
[5] M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, O. Pieter Abbeel, and W. Zaremba. Hindsight experience replay. In Advances in Neural Information Processing Systems, volume 30, 2017.
[6] Bacon, J. Harb, and D. Precup. The option-criie critic architecture. In Proceedings of the AAAI Conference Artificial Intelligence, pages 1 1726-1734, 2017.
[7] A. P. Badia, P. Sprechmann, A. Vitvitskyi, D. Guo, B. Piot, S. Kapturowski, O. Tieleman, M. M. Arjovsky, A. Pritzel, A. Bolt, and C. Blundell. Never give up: Learning directed exploration strategies. In International Conference Learning Representations, 2020. URL https
/ //openreview net/ ? 5yd=Sye57
[8] K. Baumli, D. Warde-Farley, S. Hansen, and V. Mnih. Relative variational intrinsic control, 2020.
[9] M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 06 2013.
[10] Y. Burda, H. Edwards, D. Pathak, A. Storkey, T. Darrell, and A. A. Efros. Large-scale study of curiosity- learning. arXiv preprint arXiv: 04355, 2018.
[11] Y. Burda, H. Edwards, Storkey, and O. Klimov. Exploration by random network distillation.
arXiv preprint orXiv:1810. /2894, 2018.
[12] V. Campos, P. Sprechmann, S. Hansen, A. Barreto, S. Kapturowski, A. Vitvitskyi, A. P. Badia, and C. Blundell. Coverage principle for discovering transferable behavior in reinforcement learning. arXiv preprini arXiv:2102. 13515, 2021.
[13] M. Deisenroth, G. Neumann, and J. Peters. A survey on policy search for robotics.
[14] L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, I. Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. arXiv preprint arXiv:/802.0156/ 2018
[15] B. Eysenbach, A. Gupta, Ibarz, and S. Levine. Diversity is all you need: Learning skills without reward function, 2018. UtLppttp://arxi wwwpexiv.org//bbba wwwp/rgggg///////////.//... 1822. 06070
[16] C. Florensa, Y. Duan, and P. Abbeel. Stochastic neural networks for hierarchical reinforcement learning. arXiv preprint arXiv: 104.00100..0301,, 2017.
[17] R. Fox, S. Krishnan, Stoica, and K. Goldberg. Multi-level discovery of deep options, 2017.
[18] K. Frans, J. Ho, X. Chen, P. Abbeel, and Schulman. Meta learning shared hierarchies. CoRR, abs/1710.09767, 2017. http:t httptt::///arattoorrr///////// /1100 .09767
[19] K. Gregor, D. Rezende, and Wierstra. Variational intrinsic control. CoRR, abs/1611.005555
2016. httpttp::// //aaxi org abs/1611.1..1./ 0ttps/111..75001
[20] D. Hafner, P. A. Ortega, J. Ba. T. Parr, K. Friston, and N. Heess. Action and perception as divergence minimization, 2020.
[21] S. Hansen, W. Dabney, Barreto, T. Van de Wiele, D. Warde-Farley, and V. Mnih. Fast task inference with variational intrinsic successor features. arXiv preprint arXiv: 05030, 2019.
[22] A. Harutyunyan, W. Dabney, T. Mesnard, M. Gheshlaghi Azar, B. Piot, N. Heess, H. P. van Hasselt, G. Wayne, S. Singh, D. Precup, and R. Munos. Hindsight credit assignment, In Advances in Neural Information Processing Systems, volume 32, 2019.
[23) K. Hausman, J. T. Springenberg, Z. Wang, N. Heess, and M. Riedmiller. Learning an embedding space for transferable robot skills. In International Conference on Learning Representations, 2018. htt.:/ttts:://pott ewwee net/fformt forum 1d=T0072X2bb
[24] K. Hc, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition, 2015.
[25] N. Heess, G. Wayne, Y. Tassa, T. Lillicrap, M. Riedmiller, and D. Silver. Learning and transfer of modulated locomotor controllers. arXiv preprint arXiv:1610005182. 2016.
11

---

[26] A. S. Klyubin, D. Polani, and C. L. Nehaniv. Empowerment: A universal agent-centric measure of control. In 2005 IEEE Congress on Evolutionary Computation, volume pages 135. IEFE, 2005.
[27] Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review, 2018.
[28] A. Levy, R. Platt, and K. Saenko. Hierarchical reinforcement learning with hindsight. In International Conference on Learning Representations, 2019. URLptttss///ppeneviiee net/ fot.f/rrrmmidxryy2AAAc/1
[29] A. Li, C. Florensa, I. Clavera, and P. Abbeel. Sub-policy adaptation for hierarchical rein- forcement learning. In International Conference on Learning Representations, 2020. URL
https://openreviewwcoo net/ bodoum//id-ByyyooSSSSS
[30] L. Maaloe, C. K. Sønderby, S. K. Sønderby, and O. Winther. Auxiliary deep generative models.
In International conference on machine learning, pages 1445-1453, PMLR, 2016.
[31] J. Merel, L. Hasenclever, A. Galashov, A. Ahuja, V. Pham, G. Wayne, Y. W. Teh, and N. Heess. Neural probabilistic motor primitives for humanoid control. In International
Conference Learning Representations, 2019. URL htttttt://peeeeee//////
wwt. form/f
id=Y
[32] V. Mnnn,,, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing atari with deep reinforcement learning. In Advances in Neural Information Processing Systems Proceddnoss of the deep learning workshop, 2013.
[33] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep rein- forcement learning. Nature, 518(7540):529, 2015.
[34] S. Mohamed and D. J. Rezende, Variational information maximisation for intrinsically motivated reinforcement learning. In Advances in neural information processing systems, pages 2125-
2133, 2015.
[35] O. Nachum, S. S. Gu, H. Lee, and S. Levine. Data-efficient hierarchical reinforcement learning. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018 URL httpst httpps://oroceeeinggs neur ips cc/ppaperr/201///11lee .pdf]
[36] O. Nachum, S. Gu, H. Lee, and S. Levine. Near-optimal representation learning for hierarchical reinforcement learning. In International Conference on Learning Representations, 2019. URL https://openrevieet wwwpviiww.ee//eee
[37] J. Peng and R. Williams. Incremental multi-step Q-learning. Machine Learning, 22:283-290, 1996.
[38] X. B. Peng, M. Chang, G. Zhang, P. Abbeel, and S. Levine. Mep:
Learning composable hier- archical control with multiplicative compositional policies. arXiv preprint arXiv: 1905.0980, 2019.
[39] A. Razavi, A. van den Oord, B. Poole, and O. Vinyals. Preventing posterior collapse with deEsa-VAEs. In International Conference on Learning Representations, 2019. URL https :
/ /openreview
[40] C. Salge, C. Glackin, and D. Polani. Empowerment-an introduction. In Guided Self- Organization: Inception, pages 67-114. Springer, 2014.
[41] A. Sharma, S. Gu, S. Levine, V. Kumar, and K. Hausman. Dynamics-aware unsupervised discovery of skills. In International Conference on Learning Representations, 2020. URL https://openrev/// www.oppeeeeviieeeeoo eview.re.... .ee// Eorrg
[42] R. S. Sutton, D. Precup, and S. Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112(1), 1999.
[43] Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. d. L. Casas, D. Budden, A. Abdolmalcki,
J. Merel, A. Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv: 006900 2018.
[44] Y. W. Teh, V. Bapst, W. M. Czamecki, J. Quan, J. Kirkpatrick, R. Hadsell, N. Heess, and R. Pas- canu. Distral: Robust multitask reinforcement learning. Advances Neural Information Processing Systems (NIPS), pages 4499-4509, 2017.
12

---

[45] D. Tirumala, H. Noh, A. Galashov, L. Hasenclever, A. Ahuja, G. Wayne, R. Pascanu, Y. W. Teh, and N. Heess. Exploiting hierarchy for learning and transfer in kl-regularized rl, 2020.
[46] A. S. Vezhnevets, S. Osindero, T. Schaul, N. Heess, M. Jaderberg. D. Silver, and K. Kavukcuoglu. FeUdal networks for hierarchical reinforcement learning. In Proceedings of the International Conference Machine Learning (ICML), pages 3540-3549, 2017.
[47] D. Warde-Farley T. Van de Wiele, T. Kulkarni, C. lonescu, S. Hansen, and V. Mnih. Unsuper- vised control through non parammemettic discriminative rewards. arXiv preprint arXiv:1811.11359, 2018.
[48] R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcemett learning. Machine Learning, 8:229-256, 1992.
[49] M. Wulfmeier, A. Abdolmaleki, R. Hafner, J. T. Springenberg. M. Neunert, T. Hertweck, T. Lampe, N. Siegel, N. Heess, and M. Riedmiller. Compositional transfer in hierarchical reinforcement learning, 2020.
[50] M. Wulfmeeerr D. Rao, R. Hafner, T. Lampe, A. Abdolmaleki, T. Hertweck, M. Neunert, D. Tirumala, N. Siegel, N. Heess, and M. Riedmiller. Data-efficient hindsight off-policy option learning, 2020.
[51] J. Zhang, H. Yu, and W. Xu. Hierarchical reinforcement learning by discovering intrinsic options. In International Conference on Learning Representations, 2021. URL https:// //
openreview openreview.net//ooee www./neeewoommmeeeeeeee wwt.:etteeooooooooooogPPPPPe//e
13


Reviews:
[{"review_id": "lpEt1uuJ6Tf", "reviewer": "Reviewer_GBWd", "summary": "This paper presents a novel extension of VIC, which incorporates a linear latent dynamics model. This addresses the difficulty of training VIC on large code spaces by latent decomposition of goals into sequences of subgoals. Empirical results show that this leads to a novel intrinsic control algorithm that outperforms alternative VIC variants on challenging exploration problems in terms of state coverage, as well as exhibiting improvements to the interpretability of the learned latent space.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper clearly motivates and constructs EDDICT, making a compelling argument for the usefulness of imposing a latent dynamics model on the latents used for VIC, by showing such a formulation improves over standard VIC in state coverage. The results of the paper provide a useful contribution in demonstrating the benefits of learning relative latent goals in the framework of VIC. \n\nI believe the paper could be strengthened by providing more understanding of how EDDICT performs as an exploration method, as well as how it compares to VIC. Specifically: \n- The authors frame the problem of VIC as a form of empowerment-based exploration, however, in the process ignore one of the original motivations of VIC, namely as a means for unsupervised option discovery. I believe it is fair to focus on the exploration aspect of VIC, but if the aim is to present EDDICT as an exploration method, comparison to at least one or two standard exploration baselines (e.g. RND, RIDE) should be included in the experiments section.\n- It would be useful to see how the state coverage performance compares to DIAYN, which claims to improve over VIC in terms of diversity of sampled goals.\n- Further, as the focus is on EDDICT as an exploration method, it would be valuable to include experiments showing learning curves for EDDICT in combination with extrinsic rewards. \n- On the other hand, while not the focus of the paper's narrative around exploration, it could be interesting to understand how the latent, relative options learned by EDDICT can be used by a manager policy, similar to the HRL experiments performed in DIAYN. This would help us understand the relative merits of the relative goal representation in terms of discovering useful options, which was an original motivation of VIC, which the paper focuses on as the primary baseline.\n\nAdditional feedback \n- The paper claims to have solved the \"noisy TV problem,\" however none of the environments in the experiments clearly feature a noisy TV.  Providing results on an environment with such a noisy TV (e.g. adding a noisy TV to the Point Mass environment) would provide more support for this claim.\n- Presentation of results for \"Code norm as difficulty\" section could be improved. For example, showing a plot of code norm vs distance to the center frame of the starting room in Montezuma's Revenge. The current qualitative results are a bit hard to parse.\n- In regards to the above point, what exactly is being shown in the right side of Figure 2? Is it the final state of several latent codes for each quartile, all overlaid on top of each other? If so, this should be clarified.\n- Results in Table 1 are only over 3 seeds and the number of seeds is not detailed for the other experimental results. Error bars would allow for a more principled comparison of the methods.\n- Related to the previous point on including exploration baselines, the conclusion claims \"But in terms of pure exploration, EDDICT falls short of state of the art methods that learn a single policy\", however, none of the results presented in the experiments section actually seem to directly support this claim, as the performance for these alternative SoTA exploration methods are not included in the reported results.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- The limitations of EDDICT with respect to standard exploration methods could be better showcased in the paper.\n- The extent to which EDDICT's learned options can be used for HRL or transfer to downstream tasks should be compared to that of VIC (or variants like DAIYN) to better understand whether learning relative options induces an additional trade-off in terms of the usefulness of the learned options for downstream HRL or transfer to new environments.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "ZIKj7bHM2Y", "reviewer": "Reviewer_5fT8", "summary": "This paper addressed a valuable problem: what should the agent learn in the absence of extrinsic rewards? Instead of sampling a single goal to be reached within the duration of the episode, The proposed method also samples a sequence of states from global goal coordinates. This paper is well organized and easy to follow. ", "questions": "", "limitations": "", "rating": 5, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengthens:\nCompared to prior methods, the globally consistent codes of the proposed method allow it to be far more exploratory. And the corresponding experimental results also demonstrated that the proposed method could promote the state coverage and increase unsupervised performance\n\nWeakness:\nThe experimental result part is not sufficient. \n\nIt seems that there are some errors in Fig.2 and Fig.3. Inserting problem? \n\nI am not an expert in this research field. Even my current score is 5, I will change my score in the rebuttal stage and refer to other reviews. I will check the response carefully and give my final rating in the rebuttal stage.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "As the author mentioned, general purpose function approximators (e.g. neural networks) could be used to specify more general dynamics, but makes learning tractable remains an important open question.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "D9FpdaGZxhS", "reviewer": "Reviewer_kZgT", "summary": "This paper presents a new pure exploration algorithm, EDDICT, in the thread of the methods that seek to maximize the mutual information between a latent code z and the states reached by a policy conditioned on z. Especially, it considers an additive dynamics on the latent codes as opposed to a fixed sampling distribution in prior works. The method is evaluated on a Point Mass domain and a set of Atari games, which includes Montezuma's Revenge.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Post-rebuttal\n\nHaving read other reviews and after further exchanges with the authors, I am now getting a clearer perspective on the paper and its motivations. The approach might be interesting, and the changes proposed by the authors might strengthen the submission. However, I think that a complete assessment of its value would require a re-evaluation of the updated manuscript. My evaluation on the work as it is remains borderline.\n\n---\n\nTo my understanding, the crucial idea of this paper is to consider a temporal correlation between the latent codes that are sampled along a trajectory, so that to induce a deeper exploration of the environment w.r.t. previous methods that just sample the latent code randomly. However, the general goal of those prior works seem slightly different from the one pursued here: They seek to learn a set of diverse (i.e., discriminable) skills instead of temporally-correlated skills that are combined in a sequence.\n\nBeside this point, two other factors are making me skeptical on the actual value of the paper:\n1) I think the paper fails to convey strong motivations or intuitions behind the algorithmic choices, which are mostly presented as empirical tricks. Of the three ingredients incorporated in the VIC methodology, i.e., greedy optimization, hindsight correction, and latent dynamics, only the latter is clearly motivated in the text;\n2) Even if some results are promising (such as the average return in Montezuma), I found the experimental analysis not sufficiently convincing for an essentially empirical work. On the one hand, the method is only compared against VIC, RVIC and a reduced version of EDDICT without the latent dynamics (which is quite competitive), whereas other methods in the same thread, e.g., DIAYN [14], DISCERN [46], VISR [20], VALOR [1], are not considered. On the other hand, it is not even clear if those methods are the most relevant comparison, as opposed to deep exploration algorithms, such as [10,11] that are mentioned in the text, or hierarchical reinforcement learning.\n\nFor the reasons described above, I am currently providing a negative evaluation to this work. I report below some additional comments.\n\nPRESENTATION\n\nUnfortunately, I found this paper not really easy to read. Several sentences are too convoluted, the terminology and notation are sometimes confusing. The text does not provide much intuition on what is going on with the method.\n\nRELATED WORKS\n\nThe paper by (Campos et al., Explore discover learn: Unsupervised discovery of state-covering skills, 2020) also identified state-covering as the main limitation of mutual information methods. Even if they seem to address the problem from a fairly different perspective, I think it should be mentioned in the paper.\n\nAnother concurrent work (Liu & Abbeel, APS: Active pretraining with successor features, 2021) combines a mutual information method (e.g., VISR) with a specific state-covering incentive, which might also be a relevant comparison to EDDICT.\n\nEXPERIMENTS RESULTS\n\nThe actual significance of the performance reported in Table 1 is unclear without providing confidence intervals around the average of the three runs.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and societal impact are discussed in the paper checklist and through the text.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "9lrwSx05bZK", "reviewer": "Reviewer_CKtR", "summary": "The submission extends variational intrinsic control by adding a temporal dynamics to the prior distribution of latent skills. This allows using multiple skills in the same training trajectory more easily as the skill dynamics ensure that only the skills relevant to the corresponding part of the state space are used for training. Crucially, the temporal skill prior evolves by simply adding Gaussian noise with no learned parameters, which avoids the need to train this prior. Experiments demonstrate improved skill discovery on atari. To my knowledge, the method is novel.\n", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes an interesting and promising method, and appears to fix an important issue with VIC. Further, the method is adequately evaluated and achieves good performance, outperforming some prior work. However, the method appears to have a hidden assumption that the initial state distribution is narrow, as I detail below. This is an important issue since in many practical environments, or in those considered by Eysenbach’18, the initial state distribution is wide, and the method might need to be modified to account for that. Because this issue can be addressed in the rebuttal, I am not providing my final decision at this time.\n\nThe major issue with the proposed approach is the treatment of the initial skill $z_0$. It appears from the text below l114 that its prior distribution is unconditional p(z_0). This distribution does not seem to be specified in the paper. However, it seems to me that any unconditional distribution will be inadequate. Consider an initial state distribution $p(s_0)$ wide enough so that navigating between two states sampled from it is hard. In the extreme case, the initial state distribution can be the entire state space (as commonly done e.g. for some navigation tasks). Since the skills have global meaning and correspond to states, the agent will often observe an initial skill that is too hard to execute, which is exactly the issue that the current submission attempted to address! This is because the prior $p(z_0)$ distribution is too wide, and the reverse predictor distribution will also learn to match it. Perhaps the agent should be evaluated on environments with more diverse initial state distributions to confirm the agent is robust to this.\n\nThere are also a number of more minor issues with clarity\n- What is O in Eq 2? On l133 Eq 2 is stated to be the agent’s objective. However, Eq 2 drops action entropy terms. Why are they dropped? l90 states “In practice, we optimize the above objective using a value-based reinforcement learning algorithm and \u000f-greedy policies (in lieu of a Boltzmann policy), and thus drop these terms altogether.” It is rather unclear to me what this means. Eq 1 is the objective for the policy, it does not change with the chosen exploration strategy. If the proposed method is not derived from control as inference, why is the background section in 3.1 necessary?\n- There is no difference between Eq 2 and Eq 3. The $O^{(i)}$ term is expanded in one and compressed in the other. The text implies that Eq 3 does not consider long-term consequences, but it does, up to M-1.\n- l142, “we thus condition on the code most likely to have yielded state si, under the reverse predictor.” This is not what Eq 4 says. In Eq 4, the code is sampled from the reverse predictor instead of taking the mode. Which one is actually used?\n- l174 is confusing. Is the function $f$ trained through f(s_i+1), or both f(s_i+1) and f(s_i)? This sentence implies that both, but algorithm 1 says that only f(s_i+1). \n- The choice of notation is extremely confusing. Why is prior over actions $\\pi^0$ but prior over latents $q$? Why is the forward distribution denoted as $p^z(\\tau)$ instead of the conditional distributions $p(\\tau|z)$ as used elsewhere in the paper? All 4 distributions in Eq (1) are denoted differently.\n\nFinally, the experimental evaluation omits some important baselines such as DISCERN, which was also evaluated on Atari. Since the proposed method is motivated by learning hierarchical skills, it would also be instructive to provide a comparison on downstream hierarchical RL, as done in DIAYN and DADS.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and societal impact are addressed adequately.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: To reliably reach a large number of different states in the environment requires AI agents to be able to manipulate the environment effectively. 
Practically, this type of exploration reward commonly aims to disambiguate between a set of latent codes using short horizons of future states, which limits the “exploration depth” of the agent.

Instead, this work proposes to generate conditional latent variables which effectively let the agent explore states that cannot be reached from a single unrolling.

The paper presents compelling motivation and theory for the method. However, as pointed out by multiple reviewers, the experimental results are not very compelling. In particular, the number of seeds evaluated is small and the EDDICT results are obtained by maximizing the performance over the size of the codebook, which clearly will bias these numbers. 


One important topic of discussion was the experimental validation of the method. The authors ran additional seeds for some of the experiments in the paper and found that EDDICT still significantly beats a baseline (RVIC) and an ablation (EDDICT-Delta). 

This line in the paper is concerning: “For EDDICT-based agents, we pick the best metric across code sizes.” 
This sounds like the  _best_ code size for EDDICT was reported for each metric. If so, that will both introduce bias and make the results incomparable to the baselines. 

As pointed out by the reviewers, the current results fail to provide convincing experimental evidence that EDDICT works better than the baselines. This issue is compounded by the potentially biased evaluation maximizing over different code-sizes. Given these concerns, I do not believe the paper is ready for acceptance at this point. 

Decision: Accept (Poster)