Entropic Desired Dynamics for Intrinsic Control
Authors: Steven Stenberg Hansen
Guillaume Desjardins
Kate Baumli
David Warde-Farley
Nicolas Heess
Simon Osindero
Volodymyr Mnih
Venue: NeurIPS 2021 Poster (2021)
Keywords: intrinsic control
skill discovery
exploration
unsupervised reinforcement learning

Abstract: An agent might be said, informally, to have mastery of its environment when it has maximised the effective number of states it can reliably reach. In practice, this often means maximizing the number of latent codes that can be discriminated from future states under some short time horizon (e.g. \cite{eysenbach2018diversity}). By situating these latent codes in a globally consistent coordinate system, we show that agents can reliably reach more states in the long term while still optimizing a local objective. A simple instantiation of this idea, \textbf{E}ntropic \textbf{D}esired \textbf{D}ynamics for \textbf{I}ntrinsic \textbf{C}on\textbf{T}rol (EDDICT), assumes fixed additive latent dynamics, which results in tractable learning and an interpretable latent space. 
Compared to prior methods, EDDICT's globally consistent codes allow it to be far more exploratory, as demonstrated by improved state coverage and increased unsupervised performance on hard exploration games such as Montezuma's Revenge.

Entropic Desired Dynamics for Intrinsic Control
Steven Hansen" DeepMind
Guillaume Desjardins DeepMind
Kate Baumli DeepMind
David Warde-Farley DeepMind
Nicolas Heess
Simon Osindero
Volodymyr Mnih DeepMind
DeepMind
DeepMind
Abstract
An agent might be said, informally, have mastery of its environment when it has maximised the effective number of states it can reliably reach. In practice, this often means maximizing the number of latent codes that be discriminated from future states under some short time horizon (e.g. [15)). By situating these latent codes in globally consistent coordinate system, we show that agents can reliably reach more states in the long term while still optimizing local objective. A simple instantiation of this idea, Entropic Desired Dynamics for Intrinsic ConTrol (EDDICT), assumes fixed additive latent dynamics, which results in tractable learning and an interpretable latent space. Compared prior methods, EDDICT's globally consistent codes allow it to be far more exploratory, as demonstrated by improved state coverage and increased unsupervised performance on hard exploration games such as Montezuma's 's Revenge.
Introduction
Endowing reinforcement learning agents with the ability to learn effectively from unsupervised interaction with the environment, i.e. without access extrinsic reward signal, has the potential to make reinforcement learning practical in settings where the tasks the agent will face are initially unknown where task feedback is expensive. The natural question is: what should the agent learn in the absence of extrinsic rewards? One appealing guiding principle is maximizing the number of states the agent can reach and which it can reliably return.
Intrinsic control methods have shown promise in this direction. By maximizing the mutual information between a latent code and future states reached by policy conditioned this code, intrinsic control methods learn to map latent codes to behaviors from which the code can be inferred. One major limitation of such approaches is that the latent codes usually sampled from fixed prior distribution p(z). Using fixed prior means that such approaches are unable to learn codes that correspond states that cannot be reached in the time horizon T. since any code be sampled in any state. Simply increasing the time horizon T' does not solve the problem since it leads to a sparser learning signal. Learning a state-dependent prior has proven be difficult and has been shown lead to fewer learned codes/goal states [15]. This inability to learn how to reach distant states limits the usefulness of such intrinsic control approaches.
We propose to sidestep this limitation by replacing the fixed code distribution p(2) with a fixed dynamics model over codes p(2: 1) Our algorithm, Entropic Desired Dynamies for Intrinsic ConTrol (EDDICT), learns to map sequences latent codes sampled from this dynamics model to behaviors for which the state transition dynamics in the environment match the latent code dynamics. EDDICT learns to map each 2, to a state that reachable from the state corresponding to 2
"Conresoondence to stevenhansen stevenhansen deepmind.com
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

20
21
21
20
21
20
21
TO
F1
TO
TO
TO
(a)
(b)
(c)
(d)
Figure 1: Graphical models for various priors and posteriors of interest. Circles denote random variables which are observed (shaded) or latent (white), with diamonds denoting deterministic quantities. (a) Prior
particular trajectory consisting of two sub-trajectories (7o, TI ), and auxiliary variables {10, (b) Posterior inference with independent codes, as in prior work. (c) Naive posterior inference for the sub-trajectory {21, Ti conditioned on the past. (d) Posterior inference with hindsight. Despite zo being observed, we infer 21 based on the most likely code 20 to have generated To, using the variational reverse predictor (dashed line).
allowing it to reach states much farther than the time horizon I using sequences of codes z. We show that even highly constrained latent dynamics (i.e. additive noise) are sufficient both interpret latent codes in terms of their corresponding locations in state space, and encourage exploratory behavior to far greater extent when compared to prior methods.
2 Notation
Our environment is a special case of a Markov Decision Process (MDP) without rewards or terminal signals: M (S, A, P, Po). P.). S is the state space. A the action space, P(s++ st, az) the conditional distribution representing the state transition dynamics when taking action E A from state s, S and (s(s) the initial state distribution. For simplicity, we present our method in the episodic setting with episodes of length T MK, but relax this assumption in practice. Agents interact with the environment according a policy na (a s) with parameters 0, yielding trajectories
(s. dg, 81, "1. ST], sT]], and distributed as nT* (T) = Po(5o) II=o P( St: To)To(( st).
It will be useful for us to segment given trajectory т into sub-trajectories of length K, with [so: To: To: T.. and T; [aix; Six+1 0xx+1:" 11 (((+1)x-11 8((+1)x) Note that тę defined to include ((++1)xx but not the state Six from which dix was sampled. With slight abuse of notation and denoting T-1 = Sp, we rewrite TT* (r) = (a(s)) M-1 p (-1)). with M the number of sub-trajectories per episode and (T; T:-1) le x P(s 88, ar (m) st).
Hierarchical agents sample a high-level goal or latent variable a ~ p(z) every K steps, and interact with the environment via parametric conditional low-level policy Ta(a s. which can be thought of as a fixed duration option [42]. Composing p(z), Ta(a s,2) and transition dynamics yields an augmented trajectory A [Sp, 20, To, 21, Ti, whose distribution decomposes as p (A) M Po(s)) II - p(z))= (T: Ti-1, 24), with p (Fa 74-1,2) defined analogously to T ( 7-11) with conditional policy n, (a, =t)
To simplify exposition,
index sequences at the timescale of sub-trajectories using index i, e.g. [zi, T4, 2141, 7141,"" and reserve index t for indexing sequences at the granular timescale of actions, e.g. [St, ar, St--1-111111 de.1.-....... Concretely, indexing by should be interpreted ik, as in s; 8ix- For general sequence I I1, - we define Tet == [To ... xt-i] i] and extend this notation augmented trajectories as follows: Aci
[s0, 20, To. Zi- T--
3 Method
We would like to learn goal directed agents, which are capable of reaching any state E S, given goal or state embedding Z. Extending earlier work empowerment 40 Variational
TThe dependence on 7--1 is thus due to Six E Ti-1-
'e avoid introducing new notation for the corresponding remi-MDP our present notation allows us to reason about sub srbetrajectories for both standard policies a(a s) and conditional policies a(a s,

---

Intrinsic Control (VIC) [9] and related methods (e.g. (15] propose to achieve this by learning conditional policy aaa s, which maximizes I[2; r), the mutual information between the latent code and (possibly subset of) the resulting trajectory obtained by following я. Unfortunately, this objective can be difficult to train in practice as we scale both the number of options and the horizon over which the code is executed [5]. Our method addresses both of these issues in principled manner by introducing temporal dependencies between a sequence of latent codes, evolving under simple linear dynamics, which decomposes the full objective into sum of local mutual information lower-bounds, without loss coherence of the global code.
Entropic Desired Dynamics for Instrinsic Control (EDDICT) can be understood from the perspective of divergence minimization [20] [3 Concretely, we can define a prior policy μ which induces a distribution p" (A) over the space of augmented trajectories, We then learn a (posterior) policy by minimizing the Kl.-divergence between yd (A) and (A).
3.1
VIC as Divergence Minimization
Given prior policy μ.. we construct a prior distribution over an augmented space (T, 2). with auxiliary variables [3 E Z. as p" T2 = p',)) r). The conditional (.. r) is a learnt predictor, with parameters w E f2, which aims to predict from the underlying trajectory.
We can show that an entropy regularized version of VIC obtained by maximizing Oen-vic(O, w) =
-KL (( (2)lp"((,, wrt. the parameters of Ty and qw with p(z) a fixed or learnt distribution over options. Intuitively, we seek code conditioned policy which generates trajectories having high probability under our trajectory prior, and from which 2 can be inferred in hindsight. After some algebra, this simplifies to:
To (at
S2s 2)
Oen-Vic(O,,u w) = Espp(s) (rla)
log qmo [T) logg(=))
log
plat
8e)
Id(=;t)
zegularizer
34 to the mutual information 42; Ellog p (z T) log p(=)] using reverse predictor q trained to approximate the true posterior distribution p (z T) In expectation, the regularization terms correspond to a sum of KL- divergences between our conditional policy and the prior over actions fl The original objective Ovic is obtained by dropping this regularizer and choosing the reverse predictor q == q= 80, 87), which predicts z from the first and last states of the trajectory.
In the above, T,(2;7 7) refers to the variational lower-bound 2
Since we focus on discrete action spaces, we set μ to a uniform distribution over actions, causing b to revert standard entropy rewards [48] In practice, we optimize the above objective using value-based reinforcement learning algorithm and e-greedy policies (in lieu of Boltzmann policy), and thus omit these terms. Note that the auxiliary variable perspective of VIC can also be found in Hausman et al. [23
3.2 Incorporating Temporal Dynamies
Instead of sampling single goal to be reached within the duration of the episode, it may be preferable to sample a sequence of codes either as relative (or local) goals, parameterized relative to the agent's current position, or as way points, a sequence of global goal coordinates which the agent should visit in sequence.
Relative vs Global Codes. Local goals can be implemented for VIC by resampling latent code every K steps and maximizing a sum of local objectives of the type I $:41 8t), with option 2; initiated from state s,. We describe these codes as having local semantics, as an option 2; should only be inferable in the context of the relationship between its initiation state s; and final state 8,41. In essence, each 2; represents local displacement which the low-level policy should execute. In contrast, the strategy of sampling way points in some global frame of reference would require maximizing
'The regularization terms emerge from the fact that the transition dynamics are shared by both p²(r) and (T | z), and thus cancel out in the computation of the Kivdiveree
3

---

Ia(2; 8t41). Unfortunately, this would seem to require learning a state-dependen high-level policy which gives higher probability goals 2; which are reachable (in K' steps) from s;-
Ours a hybrid of these two approaches: by specifying goals relative to previously sampled codes, in the form of Markov chain with simple linear dynamics, EDDICT can recover codes with global semantics while avoiding the need to explicitly train a high-level policy.
EDDICT Prior. As in Section 3.1 we specify joint distribution over the set of sub-trajectories
{ and auxiliary variables { ie( Recall that T-1 = 80. Our prior for an augmented
trajectory A is given by:
M-1
( 00)
II (Tr Ti-1 (w mt),
again with J uniform distribution over actions. As shall see, making the priori assertion that zi is conditionally independent of Ti-1 given т; will ensure that our objective breaks down as sum of local objectives, amenable greedy optimization. This prior is illustrated in Fig. Ta Using the reverse predictor ge = q2, s141), which predicts from 8,+1 alone (the last state of Ty) will then induce codes with global goal semantics.
EDDICT Posterior. We structure our posterior around goal-conditioned policies (a | 8, z), but modified to account for the temporal structure of our prior. We incorporate temporal dependencies between the latent codes in the form of Markov chain p(z: 21-1) with initial distribution p(20). Defining p(z0 2-1) p(zo), we write:
M-1
p (A so) = II
plzz zi-1)p T:
i=0
We now expand the negative KL-divergence corresponding to this choice of prior, posterior and reverse predictor:
KL 1pd (A so)llp so)] = Ex (A|sa)
{
M
Σ
log
gar
(24
81+1)
pd (Tr [T-1))
p( 2
z4-1)
pi (Ti Ti-1,
1=0
The objective then obtained by dropping action entropy terms.
11-1
O00, w)
Σ
To (Acs) (xstlle))))
log d., (..DKTTT (2; 8141)
log p(2s 21-1)
1=
(8,
The objective thus breaks down as a sum of M terms, defined as O0 w) = (iO(+) (f, w; 2;-1, T-1)). is worth pointing out that in expectation, C constitutes a valid lower- bound A2; 8141 24-1) despite the reverse predictor not conditioning 2;-1.
3.3 EDDICT Objective
We obtain EDDICT by incorporating (i) greedy optimization, (ii) hindsight correction, and (iii) linear dynamics into the objective of Equation]
Greedy Optimization. Define the effective entropy as the difference in prgabbbbaailities given by the reverse predictor and the high-level policy over options (cf. O). As written, the objective aims to maximize the long term sum of effective entropies: concretely, each code 2, should seek to be entropic and discemible from 8;41 but also lead states from which future options are themselves
"Note that we use O refer the i-th term Eq. which a function particular value z and T(-1. (+) is reserved for the expected value of (+) under p = (Act).

---

discernible. The variance of any return estimator will thus increase with the number of option periods. To avoid this issue, EDDICT optimizes Eq.
I in greedy- as:
M
OGREEDY w)
E (Ae)))
(0, (1,,;, (011022-1171-11
where we have omitted the policy parameters from the sampling distribution p" (Azi), which is thus considered fixed with respect to the optimization process. Concretely, this can be implemented by treating each option period as pseudo-episode, i.e. using discount factors which are zero on option boundaries as shown in Algorithm [
Hindsight Correction. Unfortunately, the above objective is rather brittle as the distribution over 2 is conditioned solely on 2i-1, and ignores the underlying state in which the code is sampled. We
can improve on this open-loop formulation by reasoning hindsight. From Eq. 2 O is computed
in expectation under p" (Acz) which includes the joint y (z;-1. T4-1 A We rewrite this joint
as p" (T:-1 Açi- (es-1)p"(( (2;-1 Tz-1) R p" (T:-1 (eci-19g000e
8;), since qu a variational
approximation to the true posterior by construction. Incorporating this approximation to Eq. 2 yields
the final objective:
M
OEDDICT(O, w) = Σ Ep (Mez) Eognncc a1ss) [O0) w; 2;-1, 11 1- 1=
(3)
Concretely, when sampling 2; ~ p(z; 1), we thus condition on the code most likely to have yielded state 8t under the reverse predictor. Importantly, this objective induces cross-entropy term between the target distribution q. (2--1 8, s,)p(zz 24-1) and q (z: 8.+1). This ensures that predictions made from 8,+1 are consistent with those from 8;, under our latent state dynamics.
Linear Dynamics The final piece of the puzzle concerns the choice of code distribution. We cannot employ the VIC strategy of fixed entropic distribution, since our codes form a Markov chain. We would further like to avoid the full HRL problem, which would require us to have a parameterized high-level policy over options. Choosing AR(1) process the conditional code distribution satisfies both of these requirements and we thus set p(2: (2-1))= = 2 + A,, with A, sampled from either isotropic Gaussian AR(1) process is that it ensures that the marginal code entropy increases monotonically with each option period (more states visited) while the conditional entropy remains constant (same number of states reachable from any given state), as shown in Fig. 4b] Finally, hard coding the dynamics to be linear, versus learning parametric policy over codes, naturally imposes an interpretable Euclidian topology in code space, as shown in Fig. 4a
uniform distribution
the disc. Another useful property of the
3.4 Algorithm
We now provide more mechanistic view of EDDICT. Algorithm presents online version of the algorithm, with details of the distributed setup used in our experiments presented below.
We optimize our objective using a distributed deep reinforcement learning system [4] based on Peng's Q()) 37 and c-greedy policies. The system consists of a centralized learner, replay buffer (3)] and a set of distributed workers each interfacing with their own copy of the environment. Given the latest parameter values and current state the environment 8; (local to each worker), actors sample z; and generate sub-trajen sub-trajectory тą by executing Ia 8, s,2;) for K steps in the environment. The resulting (s,, A., T.) is then fed back to the replay buffer, from which the learner consumes data to perform off-policy updates. Storing the initiation state and offset A,, allows the learner to recompute the code 24 as required using the most up-to-date version of the reverse predictor. Intrinsic rewards derived from the reverse predictor are similarly computed on the learner.
In practice, the learner maximizes OEDDICT by summing two losses. The first implements policy iteration by minimizing the mean-squared error between a target return, computed by Peng's Q(A) under target network (32), and the current Q-value estimates. Our greedy optimization procedure yields a single non-zero reward, log (= si+1), which is received upon option termination. The second loss corresponds to cross-entropy loss of the reverse predictor found in Eq. 3 With (z s) = (s), for some parametric function fw. this amounts to minimizing
5

---

Algorithm 1: EDDICT
Input Environment dynamics P. initial state so- policy me. code predictor gun(z = (N 1).
option period K, discount ", code dimension
[sa],
repeat
A; U(D))
e.g. uniform over a disc, isotrppic normal
#: - fu((sk) ) A. for
+ t
iK: 1i+1)K-1 do
(Tlls1, 2::
II parametric epsilon- greedy
81+1 P(ss+11ls,,
If Commpute intrinsic rewards. Note: entropy of code distribution constant under linear dynamics. Ft41 4- log qw 81-1) if (=++))) else 0
If (optional) add entropy rewards.
7641 + 0 (=(=t)) else y
S141 f- S141
Il augment state with code
Append Cr, Sr+1, F++1, 7++1 T.
Update 0 with any reinforeement learning algorithm on the sub-trajectory т. Il Minimize cross-entropy loss from Eg. [3] for linear dynamics and Gaussion reverse predictor. Update w by gradient descent on A (fw (8(()(()))) - (sux)
( i ++1
|| - (fu( for (s.))l This loss is extremely intuitive: train the reverse predictor such that the inferred latent state from 8,, matches the inferred state from 8141 under our latent dynamics. As in 44D we found that uninformative prior performed best in practice (despite our choice of isotropic Gaussian for the predictor), and thus sample A; from a uniform distribution on the disc
Concretely, we parameterize the action-value function Qo (s, a, 2) as an MLP operating on state embeddings, derived from a ResNet [24] and linear action and code embeddings. the reverse predictor qu operates on the same state embeddings as the Q-function, with gradients from both objectives being backpropagated into the ResNet. Complete details of the architecture can be found in the Appendix.
our experiments.
4 Related Work
Intrinsic Control and Empowerment. EDDICT can best be thought as incorporating temporal structure into intrinsic control algorithms 19
15 1 21), which build on empowerment [26] (34.. Relative Variational Intrinsic Control (RVIC) K also extends an intrinsic control objective, but does so by penalizing codes predictable from a single state. leading to codes representing state-agnostie behaviors. In contrast, the parameterization of our reverse predictor, along with fixed high-level policy over options, ensure that EDDICT's codes are reachable from the states in which they are sampled while preserving global state semantics.
is well known that the VIC objective is difficult to train when the code space is large D. At a high-level, EDDICT tackles this issue by breaking down this single goal into sequence of sub-goals This is orthogonal to the approach of Achiam et al. []]. which increases the number of available options over time. HIDIO (51) proposes an objective similar to ours (discriminator rewards over sub-trajectories,
but sample options using a state-dependent high-level policy trained maximize extrinsic rewards over the semi-MDP induced by the low-level policy.
The notion of reusable behavior and hierarchy has long history in the RL literature [e.g. 42 2. In comparison to EDDICT existing work can be broadly categorized with respect to the signal that is used for behavior induction and the nature of the learned representation. EDDICT bears similarity to unsupervised skill discovery methods that induce behavior in the absence of external rewards usually for the in downstream tasks, including [16 36) Other approaches learn skills or behavior representations from demonstrations provided by humans or expert policies [e.g.
Skill Discovery and HRL.
'Our variational bound is looser as a result, since our variational posterior is not matched to the prior. Improving the modeling assumptions of the reverse predictor, e.g. by using a truncated Gaussian, is left for future work.

---

Code Norm Quartiles
Montezuma Room 1 and 2
Figure 2
Figure 3: Montezuma's revenge, (left) Typical observations from the first (left) and second (right) rooms. (right) Observations from a trained EDDICT agent, sorted by L2-norm of corresponding code and aggregated into quartiles. Images are generated by taking the pixel-wise maximum
17 38 while optimizing the reward for one or multiple tasks le.g. 6 or via subgoals that are associated with explicit rewards in a predefined [e.g. space [e.g. |46 477.
25
23
29
18
28
35
or learned
Methodology. Auxiliary variables have long history in variational inference 3 30 as way to obtain more expressive posteriors and serve a similar purpose in the context of EDDICT/VIC. Tirumala et al. [45] incorporates an AR(I) process in the context of HRL and skill transfer, but did so within the prior which served to regularize a set of task-specific high-level policies. In contrast, the AR(1) process in EDDICT ensures that the high-level policy samples goals which are reachable from the current state. AR(1) processes over latent temporal sequences have also been used to prevent "posterior collapse" in VAEs with powerful autoregressive decoders (39), an analogous
phenomenon to "option collapse" in HRL. Hindsight reasoning has seen a multitude of applications reinforcement learning, improving credit assignment [22], training of goal-conditioned policies in sparse reward settings
5 and off-policy learning of options in HRL. [50]
Experiments
Here evaluate EDDICT's learned representations and behavior, and contrast them to prior work in the space of intrinsic control (or skill discovery) methods. We assess the learned representations qualitatively by looking at how well they correspond to privileged information known be relevant to down stream tasks. Namely, the state dimensions given in the DeepMind Control Suite 44 and the avatar coordinates in the Atari Learning Environment (ALE) p We stress that this privileged information is not used during training in any way, with reverse predictors operating on the same input as the Q-function.
The quality of learned behaviors is measured in terms of exploration; we posit that EDDICT explores in the space of controllable outcomes, and that this style of exploration results in reaching many states of interest. To assess this quantitatively, we compare unsupervised behavior policies in terms of reward achievement on the Atari game Montezuma's 's Revenge, which is known require sophisticated exploration order to progress. Additionally, we look at the number of unique states visited per episode using privileged environment information (L.e the underlying RAM states in ALE), this is a proxy for state coverage that is agnostic to the specific reward function of the game [4]. To look specifically at the claim that EDDICT explores the controllable states, also measure estimate of the mutual information between the marginal code distribution and the marginal state distribution.
We consider the following basclines for evaluation: VIC (19, RVIC [8] and ablated version of EDDICT. VIC refers to a scalable variant introduced in [8) that uses a fixed Categorical distribution over 50 outcomes. In the EDDICT ablation (EDDICT-A) A) the code proposal mechanism is simplified by substituting 4 == A for EDDICT's 2:+1 := 2 + A, Note that the reverse predictor remains unchanged, and thus tries to predict A directly from $++1- All algorithms were implemented in the same codebase and thus share the same network architecture and reinforcement learning method.
For the results on Montezuma'ss 's Revenge, we further include results for Q(X) agent trained to maximize the game score (which other methods do not have access to), again matched terms of network architecture.
7

---

1.75
Maggher I.. Onyditornl Emompp
2.75
-3
3.00
0.25
(b)
Figure 4: (a) Point Mass. 2D codes colored by ground truth x (left) and (right) coordinates of the point-mass mass a version of the environment with "U" shaped wall. (top) EDDICT-A (bottom) EDDICT. (b) Montezuma's 's Revenge. Estimated marginal code entropy H[[z.] (solid) and conditional
entropy H[2 81+1] (dashed). Despite codes being less predictable, EDDICT achieves higher mutual information Iy(25; 3141), as measured under the marginal code distribution (shaded).
5.1 Codes as Representations of State
Codes as state representations To illustrate EDDICT's ability to map codes to the controllable aspects of the environment, we have trained EDDICT on a simple continuous control task from the DeepMind Control Suite [43]. This environment comes with set of ground truth state dimensions required to compute the dynamics, a subset of which are under the agent's control. We can thus allow EDDICT to train from raw pixels, and then evaluate the correspondence between the code values and ground truth state dimensions.
Shown in Figure 4a is the state representation learned on version of the Control Suite point mass domain which is modified to include obstacle walls in the shape of "U" to increase the difficulty and exploration requirements of the domain. EDDICT successfully recovers the ground truth coordinates of the point mass position (under the agent's s control), but not the target position (randomly set per episode and not under the agent's control). This property of not representing what can not be controllable effectively solves the well known "noisy TV problem" [10].
Standard intrinsic control methods also have this property of only representing the controllable, but they lack any incentive to represent states unreachable in a single unroll as being distinct. This clearly demonstrated by our ablation's performance, wherein the relationship between nearby states much more tenuous.
Code norm as difficulty Since our desired latent dynamics consist of sum of zero centered IID variables, the marginal code distribution will also centered at zero, with the probability density dissipating as function of the code norm. Assuming EDDICT manages to form mapping between latent codes and states, this implics that less frequently visited states will have higher code norm.
We test this hypothesis empirically, by training EDDICT on Montezuma's 's Revenge. In this game, the agent is represented by an avatar who can move locally around the screen, but who can easily die. When this occurs, the avatar is reborn on platform in the middle of the screen. This means that by construction, states where the avatar is near the center of the screen easier reach than those farther out. Our hypothesis would thus suggest that EDDICT should assign center states with low norm codes and more peripheral states with high norm codes. As Figure Shgurs, happens in practice. One interesting subtlety is that the game actually contains several rooms, each with a different background. These are exceedingly hard to reach and, as expected, this results in EDDICT assigns these states the highest code norms of all.
shows, this exactly what
5.2 Control to Explore
EDDICT's representations only tell half of story. Since codes represent states, and the marginal code entropy increases monotonically, this suggests EDDICT's fixed high-level policy should result in exploratory behavior. But unlike most traditional work on the exploration problem, EDDICT focuses its exploration only what it can control.

---

montezuma, ma_ rereaaeeeeeeeeeee
(10))
2.
Sverage Eptoodic Fecurrn
30
2.0
60
4.0
10
0.0
Faams
16c
Figure 5: Comparing exploratory behavior in Montezuma's Revenge. (right) Game score. All methods except Q()) did not have access to this during training. (left) Number unique avatar positions visited. This is proxy for coverage of the controllable states.
Measuring control Intrinsic control methods measure their control over the environment in terms of the mutual information between a code and its downstream effects of the environment, For episodic resampled but independent codes, this can be estimated straightforwardly the effective code entropy (cf. ( Eq. [ averaged over mini-batch. EDDICT however defines a Markov chain over codes, and thus requires to compute entropy over the marginal code distribution. To do so, we fit an isotropic Gaussian to all of the codes in the batch, yielding mean and variance estimates and g². Our final estimate of I (z; 5;+1) is then H (N, o²)]. plus the average log-prediction reward log (2g 1 over the batch.
Since this lower bound, meaningful when comparing models of the same architecture. This metric thus allows us to answer the question: does EDDICT control the environment to greater degree than its fixed code distribution equivalent? As Figure 4b shows, this is very much the case. Interestingly, this is true despite the codes in general being less predictable; the extra entropy from the sequential sampling more than makes up for it.
not an unbiased estimate, the relative values should still be
Exploring what matters In order evaluate EDDICT's behavior policy, we must designate proxy metric for exploration quality. For Atari 2600, game score is an obvious candidate. But since the methods under consideration learn without sometimes quite noisy. To give a more complete view of exploratory behavior, we also include two 'coverage' metrics which counts the number of unique RAM states the agent visits per episode and over its lifetime. Using the information given in 4. we only count the RAM states corresponding to the controllable avatar. We evaluate EDDICT on these metries across amenable games, and as shown in Table 1 we find that in the majority of cases EDDICT outperforms RVIC, VIC and its ablation on one or more metries. RVIC was the most competitive method, and suggests that global codes are not the only way for intrinsic control methods to yield exploratory behavior.
to score or episode boundaries, this metric is
Of particular interest is Montezuma's 's Revenge, one of the hardest exploration games, attested by the numerous reinforcement learning papers that fail to receive non-zero scores (e.g. 33 14). As shown in Figure EDDICT outperforms other intrinsic control methods by wide margin.
For additional context, we provide a broader set of baseline results on Montezuma's 's Revenge in Table 2 of the Supplemental. These span entropy curiosity-band bassed algorithms which learn single unified policy, instead the code-conditional policies recovered by algorithms in the VIC family, with Never-Give Up 7 and Random Network Distillation [11 greatly outperforming EDDICT on the metric of average episodic reward.
Limitations and Discussion
Endowing agents with the ability to master the environment is an important step towards more general purpose agents, it allows learning in any circumstance without any requirement of a task specific

---

Game
DIAYN
RVIC Cat(16)
EDDICT
EDDICT-A
Berzerk
1,75, 24.8, 365
0.156, 31.3, 138,
0.382, 61.3, 477,
0.562. 10.6, 584
Hero
1.82, 24.1, 1.35k
2.58, 35.2, 805,
2.1, 32.5, 1.34k.
0.856, 20.7, 68.7
Montezuma
4.379.4.611 0
0.577, 5.61, 0.
1.17. 1.07,3099
0,674, 4.54,
Ms. Pacman
0.475, 1.72, 652
0.288, 1.72, 397,
0.349, 1.72, 587,
0.262. 1.72, 360
Private Eye
4.61, 87, 1.54k
5,41, 85.2, 886,
4.86, 86.8, 1.07k.
4.01, 74.5, -43.6
Seaquest
0.346, 10.9, 18.7
1.27, 10.9, 143,
1.91, 10.9, 400,
1.56, 10.9, 238
Table I: Results 6 Atari games 1B frames. Each tuple A,B,C represents mean of: (A) Episodic Coverage (-10%) (B) Lifetime Coverage (-103) (C) Average return. For EDDICT-based agents, we pick the best metric across code sizes. Metries which are best across agents, based on mean performance over 3 seeds, are shown in bold. In cases where there is tie across all methods for metric, none bolded. The full set of results, including VIC, random policy baselines, scores with standard deviations and training curves can be found in the supplemental material.
reward function. EDDICT can both explore and control the environment by learning latent codes that make sense of states in globally consistent coordinate system. But in terms of pure exploration, EDDICT falls short of state of the art methods that learn a single policy (e.g. [I].. Understanding what these advancements mean for learning code-conditional policies is promising future direction.
Additive dynamics can not capture important aspects of some environments that we might wish for our agents represent, such dynamics that irreversible or state-dependent. Ideally, general purpose function approximators (e.g. neural networks) could be used to specify more general dynamics, but how to make such learning tractable while preserving the advantageous properties of EDDICT remains an important open question.
In addition to its inherent merits for environment exploration and manipulation, EDDICT's novel state to code mapping and code transition function could be used aid local planning, or could serve as compact representation on top of which to learn policies, good state similarity metric for goal- RI., or aid many other unlisted tasks. All of these directions are left future work.
7 Societal Impact
Unsupervised reinforcement learning in general, and intrinsic control methods in particular, are far from being commercialized due their insufficient data efficiency and lack of validation in real world environments. However, when this is no longer the case, these methods could significantly reduce the human cost of setting up systems that interact with humans (e.g. robotics), these methods limit the need for handcrafted reward functions and the collection of human preferences. But this benefit comes with cost interpretability and safety. The information theoretic objectives of the methods lead to behavior that can be very hard to predict a priori (e.g. what does 'controlling your environment' look like?). Furthermore, safety constraints might be harder to specify in the absence of a closed-font form reward function. As these methods mature, the emphasis should shift from raw performance to more nuanced approach that addresses these societal concerns head on.
8 Acknowledgements
We would like to thank Stephen Spencer for providing engineering support. We further thank Yury Sulsky and Arturo Bajuelos, who contributed to results found in Appendix B, showing how EDDICT can be incorporated into the exploration policy of standard RL agents.
References
[1] J. Achiam, H. Edwards, D. Amodei, and P. Abbeel. Variational option discovery algorithms. arXiv preprint arXiv: 1xXiv:8877110200 2018
[2] D. B. F. Agakov. The im algorithm: variational approach to information maximization. Advances in Neural Information Processing Systems, 16:201, 2004.
10

---

[3] F. V. Agakov and D. Barber. An auxiliary variational method. In International Conference on Neural Information Processing, pages 561-566. Springer. 2004.
[4] A. Anand, E. Racah, S. Ozair, Y. Bengio, M.-A. Côté, and R. D. Hjelm. Unsupervised state representation learning in atari. arXiv preprint Stti:::99060022200 2019.
[5] M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, O. Pieter Abbeel, and W. Zaremba. Hindsight experience replay. In Advances in Neural Information Processing Systems, volume 30, 2017.
[6] Bacon, J. Harb, and D. Precup. The option-criie critic architecture. In Proceedings of the AAAI Conference Artificial Intelligence, pages 1 1726-1734, 2017.
[7] A. P. Badia, P. Sprechmann, A. Vitvitskyi, D. Guo, B. Piot, S. Kapturowski, O. Tieleman, M. M. Arjovsky, A. Pritzel, A. Bolt, and C. Blundell. Never give up: Learning directed exploration strategies. In International Conference Learning Representations, 2020. URL https
/ //openreview net/ ? 5yd=Sye57
[8] K. Baumli, D. Warde-Farley, S. Hansen, and V. Mnih. Relative variational intrinsic control, 2020.
[9] M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 06 2013.
[10] Y. Burda, H. Edwards, D. Pathak, A. Storkey, T. Darrell, and A. A. Efros. Large-scale study of curiosity- learning. arXiv preprint arXiv: 04355, 2018.
[11] Y. Burda, H. Edwards, Storkey, and O. Klimov. Exploration by random network distillation.
arXiv preprint orXiv:1810. /2894, 2018.
[12] V. Campos, P. Sprechmann, S. Hansen, A. Barreto, S. Kapturowski, A. Vitvitskyi, A. P. Badia, and C. Blundell. Coverage principle for discovering transferable behavior in reinforcement learning. arXiv preprini arXiv:2102. 13515, 2021.
[13] M. Deisenroth, G. Neumann, and J. Peters. A survey on policy search for robotics.
[14] L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, I. Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. arXiv preprint arXiv:/802.0156/ 2018
[15] B. Eysenbach, A. Gupta, Ibarz, and S. Levine. Diversity is all you need: Learning skills without reward function, 2018. UtLppttp://arxi wwwpexivvorg///bba wwwp/rgggg///////////.//... 1822. 06070
[16] C. Florensa, Y. Duan, and P. Abbeel. Stochastic neural networks for hierarchical reinforcement learning. arXiv preprint arXiv: 104.00100..0301,, 2017.
[17] R. Fox, S. Krishnan, Stoica, and K. Goldberg. Multi-level discovery of deep options, 2017.
[18] K. Frans, J. Ho, X. Chen, P. Abbeel, and Schulman. Meta learning shared hierarchies. CoRR, abs/1710.09767, 2017. http:t httptt::///arattoorrr///////// /1100 .09767
[19] K. Gregor, D. Rezende, and Wierstra. Variational intrinsic control. CoRR, abs/1611.005555
2016. httpttp::// //aaxi org abs/1611.1..1./ 0ttps/111..75001
[20] D. Hafner, P. A. Ortega, J. Ba. T. Parr, K. Friston, and N. Heess. Action and perception as divergence minimization, 2020.
[21] S. Hansen, W. Dabney, Barreto, T. Van de Wiele, D. Warde-Farley, and V. Mnih. Fast task inference with variational intrinsic successor features. arXiv preprint arXiv: 05030, 2019.
[22] A. Harutyunyan, W. Dabney, T. Mesnard, M. Gheshlaghi Azar, B. Piot, N. Heess, H. P. van Hasselt, G. Wayne, S. Singh, D. Precup, and R. Munos. Hindsight credit assignment, In Advances in Neural Information Processing Systems, volume 32, 2019.
[23) K. Hausman, J. T. Springenberg, Z. Wang, N. Heess, and M. Riedmiller. Learning an embedding space for transferable robot skills. In International Conference on Learning Representations, 2018. htt.:/ttts:://pott ewwee net/fformm forum 1d=T0072X2bb
[24] K. Hc, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition, 2015.
[25] N. Heess, G. Wayne, Y. Tassa, T. Lillicrap, M. Riedmiller, and D. Silver. Learning and transfer of modulated locomotor controllers. arXiv preprint arXiv:1610005182. 2016.
11

---

[26] A. S. Klyubin, D. Polani, and C. L. Nehaniv. Empowerment: A universal agent-centric measure of control. In 2005 IEEE Congress on Evolutionary Computation, volume pages 135. IEFE, 2005.
[27] Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review, 2018.
[28] A. Levy, R. Platt, and K. Saenko. Hierarchical reinforcement learning with hindsight. In International Conference on Learning Representations, 2019. URLptttss///ppeneviiee net/ fot.f/rrrmmidxryy2AAAc/1
[29] A. Li, C. Florensa, I. Clavera, and P. Abbeel. Sub-policy adaptation for hierarchical rein- forcement learning. In International Conference on Learning Representations, 2020. URL
https://openreview.coo net/ bodoum/sid-ByyyooSSSSS
[30] L. Maaloe, C. K. Sønderby, S. K. Sønderby, and O. Winther. Auxiliary deep generative models.
In International conference on machine learning, pages 1445-1453, PMLR, 2016.
[31] J. Merel, L. Hasenclever, A. Galashov, A. Ahuja, V. Pham, G. Wayne, Y. W. Teh, and N. Heess. Neural probabilistic motor primitives for humanoid control. In International
Conference Learning Representations, 2019. URL htttttt://peeeeee//////
wwt. form/f
id=Y
[32] V. Mnnn,,, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing atari with deep reinforcement learning. In Advances in Neural Information Processing Systems Proceddnoss of the deep learning workshop, 2013.
[33] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep rein- forcement learning. Nature, 518(7540):529, 2015.
[34] S. Mohamed and D. J. Rezende, Variational information maximisation for intrinsically motivated reinforcement learning. In Advances in neural information processing systems, pages 2125-
2133, 2015.
[35] O. Nachum, S. S. Gu, H. Lee, and S. Levine. Data-efficient hierarchical reinforcement learning. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018 URL httpst httpps://oroceeeinggs neur ips cc/ppaperr/201///11lee .pdf]
[36] O. Nachum, S. Gu, H. Lee, and S. Levine. Near-optimal representation learning for hierarchical reinforcement learning. In International Conference on Learning Representations, 2019. URL https://openrevieet wwwpviiww.ee//eee
[37] J. Peng and R. Williams. Incremental multi-step Q-learning. Machine Learning, 22:283-290, 1996.
[38] X. B. Peng, M. Chang, G. Zhang, P. Abbeel, and S. Levine. Mep:
Learning composable hier- archical control with multiplicative compositional policies. arXiv preprint arXiv: 1905.0980, 2019.
[39] A. Razavi, A. van den Oord, B. Poole, and O. Vinyals. Preventing posterior collapse with deEsa-VAEs. In International Conference on Learning Representations, 2019. URL https :
/ /openreview
[40] C. Salge, C. Glackin, and D. Polani. Empowerment-an introduction. In Guided Self- Organization: Inception, pages 67-114. Springer, 2014.
[41] A. Sharma, S. Gu, S. Levine, V. Kumar, and K. Hausman. Dynamics-aware unsupervised discovery of skills. In International Conference on Learning Representations, 2020. URL https://openrev/// www.oppeeeeviieeeeoooee eview.re.... .ee// Eorrg
[42] R. S. Sutton, D. Precup, and S. Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112(1), 1999.
[43] Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. d. L. Casas, D. Budden, A. Abdolmalcki,
J. Merel, A. Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv: 006900 2018.
[44] Y. W. Teh, V. Bapst, W. M. Czamecki, J. Quan, J. Kirkpatrick, R. Hadsell, N. Heess, and R. Pas- canu. Distral: Robust multitask reinforcement learning. Advances Neural Information Processing Systems (NIPS), pages 4499-4509, 2017.
12

---

[45] D. Tirumala, H. Noh, A. Galashov, L. Hasenclever, A. Ahuja, G. Wayne, R. Pascanu, Y. W. Teh, and N. Heess. Exploiting hierarchy for learning and transfer in kl-regularized rl, 2020.
[46] A. S. Vezhnevets, S. Osindero, T. Schaul, N. Heess, M. Jaderberg. D. Silver, and K. Kavukcuoglu. FeUdal networks for hierarchical reinforcement learning. In Proceedings of the International Conference Machine Learning (ICML), pages 3540-3549, 2017.
[47] D. Warde-Farley T. Van de Wiele, T. Kulkarni, C. lonescu, S. Hansen, and V. Mnih. Unsuper- vised control through non parammemettic discriminative rewards. arXiv preprint arXiv:1811.11359, 2018.
[48] R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcemett learning. Machine Learning, 8:229-256, 1992.
[49] M. Wulfmeier, A. Abdolmaleki, R. Hafner, J. T. Springenberg. M. Neunert, T. Hertweck, T. Lampe, N. Siegel, N. Heess, and M. Riedmiller. Compositional transfer in hierarchical reinforcement learning, 2020.
[50] M. Wulfmeeerr D. Rao, R. Hafner, T. Lampe, A. Abdolmaleki, T. Hertweck, M. Neunert, D. Tirumala, N. Siegel, N. Heess, and M. Riedmiller. Data-efficient hindsight off-policy option learning, 2020.
[51] J. Zhang, H. Yu, and W. Xu. Hierarchical reinforcement learning by discovering intrinsic options. In International Conference on Learning Representations, 2021. URL https:// //
openreview openreview.net//ooee www./neeewoommmeeeeeeee wwt.:etteeooooooooooogPPPPPe//e
13


Reviews:
[{"review_id": "wjo3RY2ecY", "reviewer": "Reviewer_GDC1", "summary": "This paper proposes an intrinsic exploration method, entropic desired dynamics for intrinsic control (EDDICT) to learn globally and temporally consistent latent exploration codes. Compared to variational intrinsic control (VIC), EDDICT proposes a number of changes driven by additionally reasoning over sub-trajectories and adding p(z_i+1|z_i) as a factor to the graphical model. Adding this factor makes the latent code local and therefore potentially easier to learn a policy for, but now the algorithm must reason over multiple latent codes per full trajectory. EDDICT proposes to optimize it greedily (only considering the entropy using the next latent code) but ties the latent structure together globally by choosing linear AR(1) dynamics for z_i and conditioning on the most likely past z (hindsight correction). The experimental results show that EDDICT outperforms prior exploration methods across 6 difficult Atari games including successfully solving Montezuma’s revenge.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "EDDICT is motivated from the viewpoint of learning globally and temporally consistent latent exploration codes. Starting with the divergence-minimization view of VIC, the method is derived by adding a temporal factor p(z_i|z_{i-1}). To deal with the harder optimization with the extra factor, EDDICT proposes three changes: greedy optimization w.r.t. only one option, hindsight correction to use the most likely previous latent, and simple linear dynamics of the latent code. The paper also provides a kind of template for designing better exploration methods by adding constraints and assumptions.\n\nThe experiments show that EDDICT successfully learns globally coherent latent codes, and explores without rewards successfully in difficult Atari environments including Montezuma’s revenge.\n\nFor globally coherent codes, EDDICT and VIC are evaluated on a pointmass environment with a U-shaped wall. EDDICT with a 2D latent code learns a globally coherent mapping of the states in the environment which corresponds directly to x and y position of the pointmass with a hole in representation due to the wall. In comparison, VIC learns a much more entangled embedding.\n\nThen, EDDICT is shown to successfully solve hard exploration Atari games, with an emphasis on Montezuma’s revenge. The results here seem very impressive: EDDICT is the only method that really makes significant progress on Montezuma’s revenge in terms of average return from the ground truth reward (that methods do not have access to), as well as coverage metric (note, see Figure 1 in the appendix *not* the main paper). Interestingly, the only other method that makes some nontrivial progress and is also somewhat competitive in the other 5 Atari games is RVIC. These experiments additionally include EDDICT-delta which if I understood correctly has independent latent codes instead of a temporally connected AR(1) process. EDDICT-delta performs poorly; I inferred this means that connecting z temporally is important, but the conclusion does not seem explained in the text. It would have also been good to include ablations of the greedy optimization and hindsight correction to better understand each contribution individually.\n\nThe coverage of prior seems relatively complete. One thing I would have like to see is comparison to other exploration methods outside latent code methods - for instance, one strong candidate is Go-Explore [1], which also achieves strong performance on Montezuma’s revenge, and also a comparison to curiosity/novelty-based methods which are discussed but not compared.\n\n[1] First return, then explore. Ecoffet 2020.\n\nMinor\n\nLine 150: p(z_i|z_{i+1}) = z_{i-1} + \\delta_i should be z_i = … instead of p( ) = …\n\nTable 1 is a bit hard to parse, maybe turn into something visual?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are discussed in the discussion section.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "qel4J6dXlNV", "reviewer": "Reviewer_t5N6", "summary": "This work proposes a new method, Entropic Desired Dynamics for Intrinsic Control (EDDICT). EDDICT augments the Variational Intrinsic Control (VIC) algorithm with two main novelties: (1) replacing the distribution over latents/codes with a learned Markov chain $p(z_{i} | z_{i-1})$, and (2) using hindsight correction with the learned $q_z(|s)$ to stabilize the proposed objective. The Markov chain on codes enables latent codes to be \"globally aware\" by creating correlation between timesteps. Experiments show EDDICT can achieve better episodic state coverage, lifetime state coverage, and episodic return (of the unsupervised policy) compared to VIC and Relative VIC, as well as an ablated version where (1) is not included. The authors also empirically show the code norm can be inversely proportional to the state frequency, and that the the latent space structure on pointmass captures positional structure.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. Authors propose an interesting new method which achieves better performance on the hard-exploration game Montezeuma's Revenge, particularly over the baseline VIC\n2. Improvements in long-horizon application of skills is an important question in the research community, promising ability to perform longer horizon tasks\n3. Experiments look technically sound\n4. Insightful analyses in Figures 2/3 and 4\n\nCons:\n1. I think the writing/notation was not very clear which led to confusion on my end; I may be wrong about some things but would appreciate if they could be clarified during the rebuttal period or in the text.\n2. I think additional baselines, notably DIAYN [1], would help to strengthen the paper and the motivation\n3. The mathematical derivation seems roundabout, in that the authors motivate the method as optimizing the KL divergence to a uniform distribution\n4. Due to inconsistencies in the text (notably about how $q_w$ is defined and the AR(1) process, see below), and the lack of provided code, I think the paper does not contain enough details to be reproducible.\n\nOverall, I think the paper explores an interesting idea that is interesting to consider. At this time, I would lean towards rejection due to clarity of the paper; however, if these are just misunderstandings on my end, I am open to changing my score:\n1. A main question of mine is the formulation of the objective, i.e. Section 3.1 and Section 3.2 \"EDDICT Posterior\". Here, the objective is formulated as $-KL[p^\\pi(\\Lambda|s_0) || p^{\\pi_{\\emptyset}}(\\Lambda | s_0)]$ (the equation above Equation 2). However, given that $\\pi_{\\emptyset}$ is taken to be uniform, I am not sure why this is interesting, or how it differs from the original VIC objective. I think it makes more sense to start from the VIC objective, $I(z; \\tau | s_0)$ with the modification that now $p(z) = p(z_0) \\prod_i p(z_i | z_{i-1})$, which would yield the same result after some similar assumptions.\n2. In the same equation as above (the equation above Equation 2), there should be a $\\geq$ instead of $=$ as introducing $q_w$ in place of $p$ makes the equation a variational lower bound.\n3. In multiple places $q_w$ is defined differently: as $q_w(z|\\tau)$ on line 79, as $q_w(z|s_0,s_T)$ on line 88, and as $q_w(z_{i-1}|s_i)$ on line 139 and Algorithm 1; I would appreciate if this were made consistent, and possibly would suggest using $q(z_i|\\tau_i)$ if generality is desired before introducing the practical model $q_w(z_{i-1} | s_i)$ (if I am understanding them both correctly).\n4. I would recommend replacing $\\pi$ with $\\pi_\\theta$ when applicable, as otherwise for instance the RHS of Equation 2 seems to have no dependence on $\\theta$.\n5. Why does the intrinsic reward in Algorithm 1 drop the $-\\log p(z_i | z_{i-1})$ term? This seems inconsistent with the text.\n6. In many places in the text, AR(1) (citing Tirumala et al 2020) is cited. Given this is not a well-known idea and its definition is very short, I think it should be explained in the methods/preliminaries. A second thing is that in Algorithm 1, it appears AR(1) is not actually being used, and instead it is the \"Learned AR process\" from Tirumala et al. Also, $\\mu_w(s_0)$ is undefined (and as written seems to share parameters with $q_w$).\n7. Hindsight correction is an idea that is introduced, but I am not sure how it is shown in Algorithm 1.\n\nOtherwise, I think the method is interesting, as inducing a prior/some correlation between latent codes makes sense. I do think, however, that this warrants a comparison to the style of DIAYN where a latent code is sampled at the beginning of the episode and not resampled. This would naturally enable a correlated skill for the entire episode, which EDDICT tries to do as well. I think it makes sense a well-tuned EDDICT could outperform this, as it resembles a middle ground between VIC and DIAYN, and that this experiment could strengthen the paper, but I don't think it can be taken for a given without experimental results. Furthermore, from the DIAYN paper, one strength of this approach is that the \"discriminator looks at every state, which provides additional reward signal\", which is lost in the EDDICT approach as $q_w$ only looks at the final state of each segment. I think this should be fairly easy to run simply by setting $K=1$ and removing the line resampling $z$.\n\nFurthermore, I think the experiments/analysis from Figures 2/3 and 4 are quite interesting. However, I am not sure what the \"target position\" for the Pointmass experiment is (line 252) -- it seems to me we are not analyzing goal-conditioned algorithms. Figure 5/Table 1 is also promising, although I think it would be strengthened by the above proposed comparison to DIAYN, and possibly through contextualization to well-known exploration baselines such as RND [3].\n\nMiscellaneous (no bearing on score)\n1. Typo in \"EDDICT Prior\" section: $i \\in [0, M[$\n2. Figure 2/3 has some numbering issue\n3. For Figure 2/3, I think it would make more clear to provide multiple examples of states in a given code frame, rather than taking the maximum over pixels like this.\n4. I would appreciate if Figure 4 (a) had a legend (does yellow refer to the top of the environment?) -- it seems nontrivial a 2D position is being labeled with a 1D color\n\n=====\n\nReferences\n\n[1] Eysenbach, Benjamin et al 2018. \"Diversity is All You Need: Learning Skills without a Reward Function.\"\n\n[2] Tirumala, Dhruva et al 2020. \"Exploiting Hierarchy for Learning and Transfer in KL-regularized RL.\"\n\n[3] Burda, Yuri et al 2018. \"Exploration by Random Network Distillation.\"\n\n=====\n\nUpdates\n\nI have increased my score from 4 to 6, mainly due to the new comparison to DIAYN as well as other described improvements in clarity.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors have adequately addressed the limitations of their work", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "Z1pl6R8I9E4", "reviewer": "Reviewer_qvkL", "summary": "This paper proposes an approach called Entropic Desired Dynamics for Intrinsic Control (EDDICT) which tries to learn options in an unsupervised manner. Given interaction with the environment, EDDICT aims to learn a latent code which best describes a future trajectory. This work is inspired from Variational Intrinsic Control (VIC). The authors argue that it is too difficult for a single latent code, $z$ to capture, so they propose a set of time-dependent latent codes. These time dependent codes are relative and follow a linear dynamics model. The authors show that the objective function for this setup (minimizing the KL divergence between the prior over trajectories, the distribution of latent codes conditioned on the trajectories and the posterior of observed trajectories) does not nicely break down into the same mutual information objectives as VIC but provide a greedy optimization based approach to solve this. They run toy examples on MuJoCo (pointmass) and on Atari games, and show strong exploratory performance. \n\n\n\n\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: This paper presents, to my knowledge, a novel idea of using time-variant latent codes for intrinsic control. The paper is grounded in theory and provides good explanations and intuition for why they pick this approach of optimizing for latent codes that change over time. EDDICT is well motivated as well, and in my opinion, approaches a problem of signficance. The idea of finding states that are \"controllable\" is definitely interesting and seems useful. The paper is also well written and the figures are informative. I find the pointmass toy experiment quite interesting, as EDDICT performs a lot better than VIC. \n\n\nWeaknesses: While the paper presents a useful idea, I do have a few concerns. Firstly, it would be good for the authors to compare and contrast this approach with others that also try to find \"controllable\" states, such as Bisumulation based approaches [1]. Secondly, I question I have is that if the true dynamics model is highly non-linear, does the fact that latent codes follow a linear dynamics model hurt performance? \n\nAdditionally, the paper shows that EDDICT works well in point-mass regimes as well as in Atari, it would be good to see how this method performs on standard continuous control benchmarks such as DeepMind Control Suite and how well it performs against state-of-the-art unsupervised/self-supervised exploration methods such as [2], [3] etc. \n\n\n[1] Learning Invariant Representations for Reinforcement Learning without Reconstruction. Zhang et al., 2020. \n[2] Large-Scale Study of Curiosity-Driven Learning. Burda et al., 2019\n[3] Go-Explore: a New Approach for Hard-Exploration Problems. Ecoffet et al., 2019. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do discuss limitations but do not discuss any negative societal impacts of their work. Self-supervised/unsupervised learning, especially when applied to robotics has many possible impacts (i.e. safety and interaction with humans, etc). ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "KeGvP-14IdR", "reviewer": "Reviewer_nvNV", "summary": "This paper aims to solve the exploration issue in existing mutual information based intrinsic control methods.\nThe issue is insufficient exploration because using a ﬁxed prior of latent codes means that such approaches are unable to learn codes that correspond to states that cannot be reached in the time horizon T, since any code can be sampled in any state.\n\nThe authors propose to replace the fixed prior distribution of z with a fixed dynamics model over z. \n\nTo make the objective tractable, the authors start from the divergence minimization perspective of VIC and derive a lower bound posterior. They further incorporate two design choices with the optimization of the posterior to ensure learning diverse linear dynamic codes.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While the studied problem is important and the propose method is sensible, experiments are insufficient and not solid enough.\n\nICM [Pathak et al] is a very related baseline since the representation learned there is also about controllable features and similarly it uses this controllable representation for exploration.\n\nFor the coverage/exploration evaluation, the authors should compare it against CPT [Campos et al], APT [Liu et al] and RND [Burda et al]. \n\nIt should also be made clear about the semantic meaning of “average episodic coverage” in Figure 5, e.g., how significantly it is to achieve 6000? One possible way to do this is to show the results from exploration methods, as mentioned above.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "CEPU_tU4L4I", "reviewer": "Reviewer_gr67", "summary": "This paper presents a skill discovery method based on variational intrinsic control. The main idea is to allow the skill codes to represent skills that can be reached at longer time horizons than the skill period through the use of Markovian dynamics of skill variables. The method is derived in terms of a variational inference objective over episodes applied to a prior distribution with a fixed policy and learned skill inference, and a posterior distribution with learned policy and randomized linear skill dynamics. Off-policy RL is used to optimize the resulting intrinsic objective. The paper presents experiments that compare the method to others in terms of coverage metrics and extrinsic reward metrics. The experiments show that the method exhibits generally superior performance in terms of coverage across the investigated set of Atari environments, and qualitatively, that the latent codes exhibit some structure correlated with the spatial structure of the environment.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The motivation of the paper makes sense -- the second paragraph did a good job of making the goals of the paper clear. The resulting method appears to be original. The results demonstrate that the method achieves better coverage than other methods, but its not clear how significant this result is in terms of 'downstream' tasks that rely on large coverage. The main concern I have with this paper is the clarity in the mathematical presentation. This concern stems from a variety of points, ordered from highest to lowest priority:\n\n**Concerns w.r.t. clarity and correctness of the method:**\n- The presentation in Sec 3.1 needs to be improved. L81 says \"we can show that a regularized version of VIC...\", yet fails to present or introduce the unregularized version of VIC.\n- As far as I can tell, Eq (3) is equivalent to Eq (2), yet L133 describes Eq (3) as an alternative way to optimize Eq (3). Relatedly, (3) is objective function, not an optimization problem, so the presentation of (3) doesn't function as a description of an optimization problem, like how the paper text describes it.\n- L117 $\\mathcal O_{\\text{VIC}}$ is undefined.\n- L134 and L138: $p^\\pi(\\Lambda_{< i})$ was never explicitly defined. Thus, it's not clear on L138 that it includes the $p^\\pi(z_{i-1}, \\tau_{i-1} | \\Lambda_{< i - 1})$ (its not clear that the full joint decomposes in this way). I think the definition may be implicit in the posterior definition, even if this is true, this needs to be clarified.\n- Eq 4 doesn't quite make sense -- the final term in the joint needs to be removed from the outer expectation, otherwise, there are two expectations over distributions of $z_{i-1}$.\n- More motivation is needed in L73-75 for why it is useful to view the learning procedure in the context of [19,12,26], rather than simply as variational inference. This doesn't appear to offer the method any clear technical motivation or starting point. This paragraph seems unnecessary -- the main point is the construction of the prior and posterior for VI.   \n\n**Minor concerns:**\n- L256 \"This is clearly demonstrated by our ablation’s performance, wherein the relationship between nearby states is much more tenuous.\" I don't see where the ablation's relationship between nearby states is investigated. E.g. I expected them to appear in Fig 4a, but instead, only the main method and VIC are used (not the ablation, EDDICT-$\\Delta$).\n- Sec 2: The paper should probably use sequence notation for trajectories instead of set notation, because braces commonly denote absence of ordering (whereas parentheses commonly denote presence of ordering).\n- The brace sizes on the first line of Eq. 2 are mismatched.\n- The second bracket in $i \\in [0, M]$ (below L110) is reversed.\n- The Fig1. caption is missing a description of what the diamond denotes in Fig 1d.\n- L73 typo in Intrinsic\n- Below L114 \"between the latent codes in the form of a Markov chain\" -> \"between the latent codes in the form of a Markov chain with dynamics\"\n- Fig 4b should denote some of the MI values numerically (e.g. those at 0 and 2e8).\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper mentions that the method falls short of methods e.g. [10,11], yet these methods are not compared to. The paper needs to include more context for this statement (numbers!) as well as justify why more detailed comparison is not needed, or why the proposed method is still significant.\n\nThe societal impact statement is embedded in the checklist -- instead, I think it should appear either in the main text or in the supplementary material.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the problem of making exploration efficient by learning a global-local structure. The paper generally received positive reviews which tended towards acceptance. However, the reviewers had difficulty understanding some notations and suggested comparing the proposed method to some relevant baselines. The authors provided a rebuttal that addressed many of the reviewers' concerns. The paper was discussed post rebuttal and all the reviewers responded to the rebuttal. Reviewers generally agree that the paper should be accepted but there are still many pending updates that authors have promised to finish. AC agrees with the reviewers and suggests acceptance. The authors are urged to look at reviewers' final feedback and incorporate the new experiments/baselines in the camera-ready.
Decision: Accept (Poster)