Variational Intrinsic Control Revisited
Authors: Taehwan Kwon
Venue: ICLR 2021 Poster (2021)
Keywords: Unsupervised reinforcement learning
Information theory

Abstract: In this paper, we revisit variational intrinsic control (VIC), an unsupervised reinforcement learning method for finding the largest set of intrinsic options available to an agent. In the original work by Gregor et al. (2016), two VIC algorithms were proposed: one that represents the options explicitly, and the other that does it implicitly. We show that the intrinsic reward used in the latter is subject to bias in stochastic environments, causing convergence to suboptimal solutions. To correct this behavior, we propose two methods respectively based on the transitional probability model and Gaussian Mixture Model. We substantiate our claims through rigorous mathematical derivations and experimental analyses. 

Published as a conference paper at ICLR 2021
VARIATIONAL INTRINSIC CONTROL REVISITED
Tachwan Kwon
NC
kth315encsoft.com
ABSTRACT
In this paper, we revisit variational intrinsic control (VIC), an unsupervised rein- forcement learning method for finding the largest set of intrinsic options available to an agent. In the original work by Gregor et al. (2016). two VIC algorithms were proposed: one that represents the options explicitly, and the other that does it implicitly. We show that the intrinsic reward used in the latter is subject to bias in stochastic environments, causing convergence to suboptimal solutions. To cor- rect this behavior and achieve the maximal empowerment, we propose two meth- ods respectively based on the transitional probability model and Gaussian mixture model. We substantiate our claims through rigorous mathematical derivations and experimental analyses.
INTRODUCTION
Variational intrinsic control (VIC) proposed by Gregor et al. (2016) is unsupervised reinforcement learning algorithm that aims to discover as many intrinsic options as possible, i.c., the policies with termination condition that meaningfully affect the world. The main idea of VIC is to maximize the mutual information between the set of options and final states, called empowerment. The maximum empowerment is desirable because it maximizes the information about the final states the agent can achieve with the available options. These options are independent of the extrinsic reward of the environment, they can be considered the agent's universal knowledge about the environment.
The concept of empowerment has been introduced in (Klyubin et al. 2005 Salge et al. 2014) along
with methods for measuring it based on Expectation Maximization (Arimoto 1972, Blahut 1972).
They defined the option as a sequence of fixed number of actions. Yeung (2008) proposed to maximize the empowerment using the Blahut & Arimoto (BA) algorithm, but complexity in- creases exponentially with the sequence length, rendering it impractical for high dimensional and long-horizon options. |Mhamedd & Rezende (2015) adopted techniques from deep learning and vari- ational inference (Barber & Agakov] (2003) and successfully applied empowerment maximization for high dimensional and long-horizon control. However, this method maximizes the empowerment over open-loop options, meaning that the sequence of action is chosen in advance and conducted re- gardless of the (potentially stochastic) environment dynamics. This often impairs the performance, the agent cannot properly react the environment, leading significant underestimation of empowerment (Gregor et al. 2016).
To overcome this limitation, Gregor et al. (2016) proposed to the closed- options, meaning that the sequence of action is chosen considering the transited states. This type of options differ from those in Klyubin et al. 22005), Salge al. (2014) and Mohamed & Rezende (2015) in that they have a termination condition, instead of a fixed number of actions. They presented two algorithms: VIC with explicit and implicit options (we will call them explicit and implicit VIC from here on). Explicit VIC defines a fixed number of options, and each option is sampled at the beginning of the trajectory, conditioning the policies of an agent until termination. In other words, both the state and the sampled option are the input to the policy function of the agent. One clear limitation of explicit VIC is that it requires the preset number of options. This does not only apply to explicit VIC, but also to some recent unsupervised learning algorithms that adopt a discrete option skill with predefined set (Machado et al.) 2017 Eysenbach et al.) 2018). Obviously, presetting the number options limits the number of options that an agent can learn, impeding the maximal level of empowerment. Moreover, choosing a proper number of options is not straightforward, since the maximum of the objective for given number of options depends on several unknown environmental factors such as

---

Published as a conference paper at ICLR 2021
the cardinality of the state space and the transitional model. To overcome this issue, Gregor et al. (2016) proposed implicit VIC which defines the option as the trajectory (i.e., the sequence of states and actions) until termination. There exist multiple trajectories that lead to the same final state and implicit VIC learns to maximize the mutual information between the final state and the trajectory by controlling the latter. As a result, the number of options is longer limited by the preset number, and neither is the empowerment. Despite this advantage, however, these implicit options induce bias in the intrinsic reward and hinder implicit VIC from achieving maximal empowerment. This effect grows with the stochasticity of the environment, and it may cause serious degradation of empowerment, limiting the agent from learning the universal knowledge of the environment. this paper, we aim to solve such empowerment degradation of implicit VIC under stochastic dynamics. To this end, we revisit variational intrinsic control and make the following contributions:
1. We show that the intrinsic reward in implicit VIC suffers from the variational bias in stochastic environments, causing convergence to suboptimal solutions (Section 2).
2. To compensate this bias and achieve the maximal empowerment, we suggest two modifica- tions of implicit VIC: the environment dynamics modeling with the transitional probability (Section and Gaussian mixture model (Section 4
2 VARIATIONAL BIAS OF IMPLICIT VIC IN STOCHASTIC ENVIRONMENTS
In this section, we derive the variational bias of implicit VIC under stochastic env dynamics. First, we adopt the definition of termination action and final state from Gregor al. (0066) and briefly review VIC. The termination action terminates the option and yields the final state 8y St independently of the environmental action space. VIC aims to maximize the empowerment, i.e., the mutual information between option Ω and final state Sy, which can be written as follows:
I(2,, s (s0)) = Σ f
p())ss)) log ((2)ss)) +
Σ 0,55
p (so. l)p((1}ss) log p( sy).
(1)
Since p(2)s0 s0; 8y) is intractable, VIC Grrgoor et al. 2016) derives the variational bound VVB < T and maximizes instead:
I (B,
Σ ((((ss)) logp(((sso)
Σ p(sy|so, ()p(( log (S2))s..s)) 2,,5
(2)
where q(S2)su, 8y) is the inference model to be trained. When JVB is maximized, we have q(S)50, 8y) pS2]s,,337 and achieve the maximum I. As explained in Section [ explicit VIC samples an explicit option at the beginning
trajectory and conditions policy as (a]s, S2) until termination. Due to the randomness of policy, the final state is undetermined for given option until the policy converges to achieve a specific option. Unlike explicit VIC, implicit VIC defines its option as a trajectory until termination, and hence, the final state is determined for given option. This can be expressed as
[1, p(sy 12, (9, so) 1
if 8yj2 otherwise
where 8/10 is the final state of option Ω. This is key characteristic of implicit VIC and is essential for deriving the main results of this paper. We will often use this equation to eliminate 8y for reduction. Interestingly, this makes difference in empowerment maximization between explicit and implicit VIC, which can be explained by rewriting / [92, ss)) as follows:
I(2, s/)) = Hs - H (s so).
Note that ( is 0 for implicit VIC since sy determined for given Ω2 One can notice that to maximize empowerment, the agent needs to learn (1) maximizing H ( (so) and (2) minimizing [1 [2, so). While explicit VIC needs to learn both (I) and (2), implicit VIC needs to learn only (1), since (2) is already achieved by the definition of option. This makes the learning of implicit VIC easier and faster. Moreover, implicit VIC is scalable as explained in Section I Despite these strengths, implicit VIC suffers from variational bias in intrinsic reward under stochastic environ- ments that can seriously degrade the empowerment. We derive this variational bias by decomposing p([(so) and p([2 s, 8o). Since implicit VIC defines the option Ω the trajectory of agent, i.e.,
2

---

Published as a conference paper at ICLR 2021
the sequence of states and actions: Ω (su, ag. 81, d1, ... ST-1, 0r-1 aj, ST 8ʃ 8T 8T-1), 1),
the probability an option can be decomposed as
p((]s0)
II (Te, (tt1))
using Bayes rule with 71 (Su. au, .... St). Similarly, p(2] sy) can be expressed as
p(0 8 sf11)
II
P(a (11)))(ppsssslltpt ar, 8 s11)).
(5)
Note that 8y is replaced by sn since it is determined by the given Q. Using 44. (5) can rewrite the mutual information
and 3
① of implicit VIC as
log ((. + log p(s Te, at 811a)) p(ae(T))
802)))(80) = (9)
Σ
(6)
p(81t1/Te,
(Ti.eL, 1)E2
The intrinsic reward of implicit VIC (Gregor et al.
2016) is given by a Sqa))
va
=
Σ (E1)
log p(a)(T)
where q(a Te, sy) is inference and p(a,!T.) ) policy of an agent. can be shown that Th comes from the first part of (6) (see Appendix [A for details). Under deterministic environment dynamics, the transitional part log (( (+++1 Tr, at, 8 f1 /p(p//((((111//) and denominator and it yields the variational bias i/2 TVT in the intrinsic reward:
ae) is canceled out since both the nominator always 1. However, this is not possible under stochastic environment dynamics
P( ac, 8y1n)
VIC b
Σ
log
p( Te, ac)
From (8). we see that this bias comes from the difference between the transitional probabilities with and without the given final state. This difference large when $ $aj in the nominator plays a crucial role. which then causes a large bias. One extreme case is when 8141 is the necessary transition to reach $yja but not the only transition from ax). Then, p(s Tr, a,, $ f11))
but p(5++ Tr, ar) is not, yielding large bias. In Section we provide the experimental evidence that this variational bias leads a suboptimal training. Even though the original VIC |Gregor et al.] 2016) subtracts b(s0) from ! to reduce the variance of learning, it cannot compensate this bias since it also depends on 52. In the next section, analyze the mutual information (T) in more detail under stochastic environment dynamics and define the variational estimate of (1) 7VE. for training.
3 IMPLICIT VIC WITH TRANSITIONAL PROBABILITY MODEL
In this section, we analyze I(Ω, 8 (s0) under stochastic environment dynamics and propose to ex- plicitly model transitional probabilities to fix variational bias. First, for a given option and final state, we define ps (Q]ss, su), Pp (2]ss so), Px (S2 and par (2(so) as follows:
Pr (R) = II
Pa (arlT. 85), Pp( 80) = II (1
Pp(5 Te, aes sy),
(9)
P- ((((pp)
II (ar(T))
(9[s))
II 1
Note that p([2s/ Su) Px (R)ss Su Su) so) is the true distribution of Ω for given sy where Pr (22)s y su) is a policy-related part and (2]ss) su) is a transitional part of p([2|s,, ([])ssss)) and so do ((((sn), Par (2(so) and (9))s)) It is necessary consider transitional probabilities since they induce variational bias intrinsic reward. Hence we model (9) as follows:
II (7c.0.8441)0
n (a:ler))))
(2 so)
II (re.ae==41))E0
( Te, az, St);
(10)
<(((s)))
<a(a))) (7
II
p² (
II
3

---

Published as a conference paper at ICLR 2021
where TT, p², x and p" are our models to be trained. We know the policy of an agent, have Pr (a)lT)) = TT =aa((a/77 [T). For the other probabilities, they are trained based on our algorithms. Now can rewrite ((22,8l5s as below using 9):
1(22,8150)
Σ C(22, sy(,)))(s)) log so)p (Stss,, so) - log (9)s))
(11)
Using 10 we define I as follows:
B (2, sy(8o) - Ep(,, 150)
50) log lo (Q0)) (12)
This is an estimate of the mutual information between 5 and sy with transitional models. Note that IVE is not a variational lower bound on I unlike I hence the derivation of VIC in Gregor al. (2016) is not applicable in this case. To tackle this problem, we start from absolute difference, [ which can be bounded as
vv LUVE
(0)
#1
(13)
DKL PP
See Appendix B for the derivation. Note that U is the sum of positively weighted KL divergences, which means that it satisfies UVE + and only if (-1s,, - Px (-18); (- so) and p²(+|su) + Po(- for all sy. In other words, our estimate of the mutual informa- tion converges to the true value as our estimates (10) converge to the true distribution (9). It makes sense that we estimate the true value of the mutual information if we know the true distribution. Hence we minimize U UVV with respect to T7, p² and p²_ If TT, pe and pe are parameterized by 0%.
( s0) +
og and Og. we can obtain gradients of UVE using (3) as follows:
V Vg VoEUVE_
=
p(S2|sp) V og log T (Q)
V VozUV== = Σ
((2))p)) V log ( (12) s0);
(14)
2
VOUUV== = Σ
p (((2[=)) log p" (52)=0).
which be estimated from sample mean. Once have (nT (-1s,, so), pa (-1s,, 80), (²(((0)) x (pr(-1s,, so), (0); po(-1sy, 8n). Po(-)))) for all 8y, we can update the policy to maximize If T is parameterized by 0g, the gradients, Vog I and Vog IV can be obtained as follows using
V oz 1
((2))0)) (ogP=(()s)1)) (1) sla, so) log (ol (9)
Ver log (rrs""((0 (2))s))
Vor 1 VE
(O)) ala, su) - log T" ((((ss))))" Ver log 7" (S2
(15)
where p., (52(s)) is replaced by (P((s))) since we know the true value of policy (see Appendix A for details). From 115.. we see that Vor I VE + Voz I as (-1s// sp) + (=(3, (=(( $p Pp(-ls,,s0) so) and pr (-1so)) + pp(((s0) for all sy. It means that of mutual information w.r.t policy as our estimates (10) converge to the true distribution O Note that for deterministic environments, we omit 11 8o) and (*(ss)) since they always oog IVB which means that maximizing IVB equivalent to minimizing UVE for 02 (i.e., is equivalent to the original implicit VIC). Finally, Algorithm [ summarizes the modified implicit VIC with transitional probability model. The addi- tional steps added to the original implicit VIC are marked with (*). Note that Algorithm [ is not always practically applicable since it is hard to model p($ Tr, az) and p(s [Te, at, &,,5f) when the cardinality of the state space is unknown. (We can not set the number of nodes for softmax.) In our experiment of Algorithm I will assume that allows us to model (5 1 F., a.) and (544 1 [r, a, sy) using so softma In the next section, we propose practically applicable method that avoids this intractability of the cardinality.
estimate the correct gradient
1. Then we have V IVB and VegUR
know the cardinality of the state space. This
4

---

Published as a conference paper at ICLR 2021
Algorithm Implicit VIC with transitional probability model
Initialize So, 7), Tirain, 0% 0% 0F and 0%
for derain to Tivain do
Follow T² (at (T)) result in S2 (so, ao, ..., 8y). log T& (aslTe)]
r vvE
+ 3;
x (aslT::
from
15
To rs
Iv P E, [log (0( IT,, ae 0t) log (s
b from
15
Update each parameter:
P t- 0 + mr
V
00 E log T" (ailT))
> from 15
0 8 + VV
(a
from
P + OF + + oz
log p² (T,+
> from
114
0g + 0% vVV
log pa (8c+1(551770 de. 88)
P from
14
end
end for
IMPLICIT VIC WITH GAUSSIAN MIXTURE MODEL
In this section, propose the alternative method to overcome the limitation of Algorithm 1 by modeling the smoothed transitional distributions. This allows us to use the Gaussian Mixture Model (GMM) (Pearson) 1894) and other continuous distributional models for modelling transi- tional distribution. First, we smooth p($++1/T1, az 8f) and p($141/ (T,, into So ( at and fo (II+1/Te, ar) with T++1 S1+1 + Ze+1 and 2++1 o(0,0²1))
fo (I (())))==))))))))======
Σ
p(s
-
3'E8(Tr.0rie))
(16)
Sa [o(
Σ
-,,,0
(ES((TO)))
where S(T,, (x, mension of the state. Then, using Gaussian Mixture Model (GMM) (Pearson 1894). we model (T6) as (I (4,85) and f (Izz (Te az):
S(T,,0,,8)) sy) =
01, 8y) > S
S(T, at) = (s (s''p(s' Te, > 0} and d is the di-
agmm
[3 ar, 84)
E
wilTe, ae Bat
(17)
Mgma
fe fe( =
Σ
wi(Te, aat
at),
Note that if we set Mgmm max 15(T,, at).. 17 can perfectly fit 16 Now using (77 we define 1 the variational estimate with smoothing as follows:
IVE
Σ P(( log so) so) log (l) (8({ss)))(
(18)
12,8
Note that IVE is not a variational lower bound on I, hence we can not also apply derivation of implicit VIC in Gregor et al. (016)). As in Section 3 start from the absolute difference between and 1 The upper bound on -I can be obtained as follows:
|
1 VE
UVE
+ -.V
VE with
Pp(S2 so) (o)
UVF
Σ p(2,, s (so) log
(9 50) [2] so)
(19)
UVE
Σ
p(55l00) DEL. [p= 8, 80 (.15 so)l
(,18,, 80).,,,,))/(ll,,,....
81
See Appendix for the derivation. Note that U.F differs from the first part of UVE in 13 This
upper bound implies UVE 0 f3(:1 s0) f(((()))))))))))))))))))))))))) - po(-1s5,8o) 80) / (-1s0) for all 8y and
5

---

Published as a conference paper at ICLR 2021
U. + 0 if and only if (- (1, ( for all 8y since U.222 is the of positively weighted KL divergences. To estimate the correct value of mutual information from (88), we mini- mize F and U22 The gradient of 2 can be obtained as below using (3):
VOUUT V 02
LEVE
Σ p(2)ss)) Voz log x " su) 0.2
(20)
which can be estimated from the sample mean. As Algorithm 1 satisfies Vog o IVB which means that minimizing U.E equivalent to maximizing VVB with respect to 84. Since U/2
if and only if (-((111,=0) - Px (-18); 86) for all s,, this update will make x (-1s ( 86) converge to Pa (118,, so) for all sy-
Unlike U . it is difficult to directly minimize U.P. due to the absolute value function. How- ever, can be minimized by estimating the correct ratio between transitional distributions, Pal-lsf (0lr.so//pppp <o))pp(-(so). even though each of their true values is unknown. To estimate this ratio, we fit (17 to (16 by minimizing Dxulfo (-IT,, (s.ss))) ff)llf at, sy,8)) sy) and D a (f a.)]- If 12 and fl are parameterized by 8% and 0%. the gradients of KL divergences be obtained as follows:
sy)l=
fo at, 8y) V log (1142lt,,,
-f
V
fo fo at Voy log f ( at).
(21)
which can be estimated from the sample mean. As our estimated smoothed transitional distribution
17 converges to the true smoothed transitional distribution (16). it satisfies f su)/ (- f+(sss)
Po(-lsy, su)/Pp which leads to UYE - for finite T and a < dnnin where T is the average
length of trajectories and dmin is the minimal distance between two different states (see appendix
can minimize U.P to nearly zero. This implies that if we smooth the original transitional distribution with smaller o, the smoothed transition will be sharper and the ratio between the transitional probabilities will
D for details). Hence, for small enough and finite length of trajectories,
be estimated more accurately using them. Once we have (-)s f2(-[sy. sy- So). f2(((so) R
(px (-1s,,,, so): so),
fa(-ss,s5)) (-(so) for all sy after the update by 20 and 21
we have 1 VE ~ 1.
Now, we can update the policy by obtaining Voz I using 33:
Pr (T(888,,80) 60) (1)))))))
pp( 5y, 80)
V =
Σ s())
log
+ log
Vor log (9((00))
(92
Ω
r
x (S] su) (82
+ log ( so) (S2)s))
(22)
Voz VV
Σ Ω
p(()50))
log
Voz T (F2)8)
where Voz I is rewritten from 15 From 22 we see that it satisfies V IV + V I as (9(1,1,,5) so) - Pr (-(s,,,55) and (sls so) Th (s)) + so)ppp for all sy, which can be achieved by (20) and (2T as explained previously. Hence, we can estimate the correct gra- dient of mutual information w.r.t our policy using the estimated smoothed transitional distributions for finite T and σ < dmin- However, choosing an appropriate value of o is not straightforward since dmmn and T are usually unknown and depend on the environment. Besides, too small makes the training of f& and ff unstable due to the extreme gradient of Gaussian function near the mean. Another issue of GMM is the choice of a proper Nmmm of (17). As explained previously, we can perfectly fit (1)) to (16) for Ngmmm maxx..0, SS(T,, a.)l. We may choose a very large 7gmmm for the perfect fit but it makes training hard for its complexity. We leave the proper choice of a and Tgmmm future works and empirically chosen values (o = 0.25 and Ngmm 10) in this paper. Finally, we summarize our method in Algorithm Additional steps added to the original implicit VIC are marked with (+).

---

Published as a conference paper at ICLR 2021
Algorithm 2 Implicit VIC with Gaussian mixture model
Initialize so, 7), Tirain, Tsmooth, 02 0% 0g and 0z
for derain 1 to do
Follow T² (T)) result in ft (so: ao, .... 85).
IVe
-, llog (aslTe, sy) log T" [c)]
from
22
1e
FYE
E. Dlg (2 (8( 85) log (8(t+117))
from
22
Update each parameter:
0 +- A
+ m
V log
>> from
22
02 +- 04
+ VV
62 L, log (a
from
20
AAP 0
A O
for smoothh to T smooth do
D
Sample (=1, 22 2f 2i 1(0,021,
AOP + A + V o0
Σ, log le (8 T441/T 0z)
> from
21
Adg + A + v oz
log (s + 2411 oz: 8f)
> from
end for
OF 0P
AOP IT sooth
0g 09 A09 / smooth
end
end for
EXPERIMENTS
In this section, evaluate implicit VIC Gregor al. 2016), Algorithm 1 and Algorithm We use LSTM (Hochreiter & Schmidhuber 1997 to encode T = (50, ao,...; St) into a vector. We conduct experiments both deterministic and stochastic environments and evaluate results by measuring the mutual information from samples. To measure I, we rewrite
using (3) as follows:
I(,, sylsa)
Σ p(syls)) logp(sy|so) 22,y
Σ E((()s))p(sy122 so) logp(s;12,,0g
Q,sy
= Σ (85) log
which is maximized when 8f is distributed uniformly. We estimate î using the distribution of sy from the samples, i.e., p(sy(so). We apply exponential moving average (0.99 as a smoothing factor) to average of 5 repetitions for estimating p(ss(so). We manually set Tmaz (maximum length of a trajectory) for each experiment such that the termination action is the only available action at Tmoz th action. For training GMM of Algorithm 2 the transitional models are trained to predict the distribution A = It+1 s, instead of 2,+1 since predicting difference usually casier than predicting the whole state. For the training, have warm-up phase which trains the base function b(8a) in Gregor et al. (2016) and the transitional models. After the warm-up phase, we update the base function, policy and transitional models simultaneously. Please see Appendix [.r details on the hyper-paramette settings.
(a) Deterministic ID world.
(b) Deterministic 2D world.
(c) Deterministic tree world.
Figure 1: Deterministic environments. Fig. shows the deterministie ID world. The agent go left right. Fig. [b] shows the deterministic 2D. The agent can go left, up, right and down. Fig Ic shows the deterministic tree world. The agent go left and right.

---

Published as a conference paper at ICLR 2021
We compare the algorithms in deterministic environments in Fig. Note that although is zero for deterministic environments, we still train the transitional models of Algorithm and to show the convergence to the optimum of them. Fig. 2 showss that all three algorithms rapidly achieve the maximal empowerment. We can observer that all the states in environments are visited uniformly after training which is achieved when the mutual information is maximized. Fig. 3 shows the training results of implicit VIC and Algorithm 2 in the 25 x 25 grid world with 4 rooms used in Gregor et al. (2016). Both implicit VIC and our Algorithm 2 successfully learn passing narrow doors between rooms and visit almost the whole reachable states for a given T maa 25. The additional results the Mujoco (Todorov et al. 2012) showing the applicability our Algorithm are in appendix
candom agert
random agent
implich VIC
randomn agent
Implicit VIC
Implict VIC
Impice ViC TM
implisit VIC TM impict MC GMM
implist VIC TM
mpiit VIC GMM
implicit vic GMM
22:
3.0
rondon spenE
randoin
3.9
spent
Imp kct VIC IN
tmp ice VIC
3.2
Inp iot VRC
1.0
14
22
241
Update stop
Update Stop
Updote stop
(a) Deterministic ID world with Tmax 5.
(b)
Deterministic 2D world
(c) Deterministic tree world with Tmax 4.
with T Tmax 5.
Figure 2: Estimated empowerment during the training in deterministic environments. Fig. shows the deterministic ID world and its training results. The agent can go left right. Fig. 2b shows the deterministic 2D world and its training results. The agent can go left, up, right and down. Fig shows the deterministic tree world and its training results. The agent can go left and right. Green shows the distribution of sy.
2a
implicit VIC
implicit VIC+GMM
Figure 3: Deterministic grid world with rooms. The environment a 25 x 25 grid world with rooms. The agent can go left, up, right and down. The agent starts from (4, 4) and (10, 4) with Tmaz = 25. Green shows the distribution of sy.
We compare the algorithms stochastic environments. Please see appendix F22 for the details on the stochasticity of the environments. Fig. 4 shows results in simple stochastic environments. We see that while implicit VIC converges to sub-optimum, our two algorithms achieve the maximal empowerment.
Fig. 5 implicit VIC fails to reach far rooms, whereas our Algorithm 2 reaches every room in the environment. From Fig. and we can notice that implicit VIC fails to reach far states under the stochastic dynamics. It happens because the variational bias implicit VIC is accumulated throughout trajectory, i.e., it gets larger the length of the trajectory increases. Fig. alaso shows the results of training with external rewards. The same mixed reward is used as Gsreorr al. (2016), r + ar with the entropy loss was applied for exploration. While the random agent and implicit VIC converge to sub-optimum, Algorithm achieves the optimal solution.
30. For training the random agent, only ar is used and
8

---

Published as a conference paper at ICLR 2021
implicit
agort
Impict
Implictt Impict
Implict
mmpiot VIC
implicit VIC GMM
2.4
3.3
2.1
maslium nansom agent
3.4
Empgeerment
2.39
Imsicr VCC
1.1
VCC
0.777
ImbiICE VCC CPN
1.0
GNM
0.
1000
2300
2000
4000
3000
15
1000
2000
2000
4000
5000
1000
2030
5000
4003
5000
Update step
Stochastic ID world with T Tmaz 5. Algorithm
(b) Stochastic 2D world with Tmaz Algorithm and achieve ~ 1442% empowerment gain compared to implicit VIC.
Stochastic world with Tmax = 4. Algorithm achieve 4406% empowerment
achieve ~ 1966 empowerment gain compared to implicit VIC.
gain compared to implicit VIC.
Figure 4: Estimated empowerment during the training in stochastic environments. The envi-
ronments are equal to Fig. T Txxcept for their stochasticity. Green shows the distribution of sy-
Implictt
Implictt VIC GMM
Optimal solution
rander agentt
implish (pre- 1eb)
implici (pre-train 1s5)
Update step
1e5
Figure 5: Stochastic gird world with 35 rooms. The environment is 15 15 grid world with 35 rooms (black cells surrounded by gray walls). We set Taaa = 25. The agent can go up, down and right. starts from (0, 6). Once the agent enters a room, the environment returns done and then the final action is available only. Green shows the distribution of sy. The external reward is composed of 1 for every time step. +1 for entering
normal room and + +100 for entering the special room. The sub-optimal solution is reaching the closest room. The optimal solution is trying to reach the special room (the room with the red box) while enters the closest normal room as soon as possible when it is impossible to reach there due the stochastic transition.
CONCLUSION
In this work, we revisited variational intrinsic control (VIC) proposed by Gregor al. (2016). We showed that for VIC with implicit options, the environmental stochasticity induces a variational bias in the intrinsic reward, leading convergence to sub-optima. To reduce this bias and achieve maximal empowerment, we proposed to model the environment dynamics using either the transi- tional probability model the Gaussian mixture model. Evaluations on stochastic environments demonstrated the superiority of our methods over the original VIC algorithm with implicit options.
ACKNOWLEDGMENTS
This research conducted with the support of NC. We thank Scong Hun Lee at University of Zaragoza for his sincere feedback and Seungeun Rho at Scoul National University, Jinyun Chung, Yongchan Park, Hyunsoo Park, Sangbin Moon, Inseok Oh, Seongho Son and Minkyu Yang at NC for their useful comments.
work, Yujeong Lee at KL Partners for her encouragement

---

Published as a conference paper at ICLR 2021
REFERENCES
Suguru Arimoto. An algorithm for computing the capacity of arbitrary discrete memoryless chan- nels. IEEE Transactions Information Theory, 18(1):14-20, 1972.
David Barber and Felix V Agakov. The im algorithm: variational approach to information maxi- mization. In Advances in neural information processing systems, pp. None, 2003.
Richard Blahut. Computation of channel capacity and rate-distortion functions. IEEE transactions Information Theory, 18(4))460-47)) 1972.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without a reward function. In International Conference on Learning Representa- tions, 2018.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint arXiv:161 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Alexander Klyubin, Daniel Polani, and Chrystopher Nehaniv. Empowerment: A universal agent- centric measure of control. In 2005 IEEE Congress Evolutionary Computation, volume pp- 128-135. IEEE, 2005.
Marlos C Machado, Mare Bellemare, and Michael Bowling. A laplacian framework for op- tion discovery in reinforcement learning. In International Conference on Machine Learning, pp. 2295-2304, 2017.
Shakir Mohamed and Danilo Jimenez Rezende. Variational information maximisation for intrinsi- cally motivated reinforcement learning. In Advances in neural information processing systems, pp. 2125-2133, 2015.
Karl Pearson. Iii contributions to the mathematical theory of evolution. Philosophical Transactions of the Royal Society of LA.), (185):71-10,, 1894.
Christoph Salge, Cornelius Glackin, and Daniel Polani. Empowerment-an introduction. In Guided Self- Seff-Organization: Inception, pp. 67-114. Springer, 2014.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Raymond W Yeung. The blahut-arimoto algorithms. In Information Theory and Network Coding, pp. 211-228. Springer, 2008
A DERIVATION OF INTRINSIC REWARD FROM MUTUAL INFORMATION
Here we derive the intrinsic reward of VIC by taking the gradient of (T) with respect to the parameter of policy 0. We omit 50 for simplicity. Note that p(l), p(s)) and p(2 sy) can be parameterized by 0 since they all determined by policy. We start by rewriting (I:
T(2, 84) = >Eoo Po, 88)
log pa (S2, 85) - log
0.29
10

---

Published as a conference paper at ICLR 2021
Then by taking the gradient with respect A, obtain
(51 = Σ (o( sy)
log pe (S2, sy) - log (o))p(()) V log (2(,
0,55
Vepo(S2)
Vopa(ss)
Σ pu(2,
+
po( po(os)
22,y
Σ 8
pa (S2, 88)
po s5) loggo((()p((s1)) V Vologpe
+ Σ
- (00 (a0p))000pp0p00m0mm000/aaame
2,55
Using
Vapo(S2,8)) = E
(s ())oo)(()) = 0, Σ
Po(Sl)s)) = 0, Q,,y
12,5y
and have
Vol(2,88) Σ
Po(S,, sy) logpo(9,,)) sy) log (ogro(()po((s)) Vo log po (9.,
2,55
(23)
(S2) Po
f2
where
2 1og log)
Similarly, obtain
Vol VE (St 84)
Σ (S, 84) (())s)) - log po Vo (8g
2,,y
po pa))r v
V pa)
where
r VE = log ((22)111n) - po())
Using 4 and 5), we can rewrite
r/
log po Tey (11)) 1 ae; 891)
Σ
(ax Tr.
T5.d5.8541EE EQ
Since (a(a ( 8yms is intractable, we may replace it with variational inference go(arlTe, 811)
which result
log (olal (looummssyolooessssla..... a4,
(o(a:17.)p($44111 T.)
For deterministic environment, we have pa 1 (T, a,, 8110) = p( az) = 1 and both rá and can be reduced into
VE
log (a 8
r
Po (a(((TT))
7,01.911100
log Is (allTT,, (T(
VE
Σ
v0
11

---

Published as a conference paper at ICLR 2021
B DERIVATION OF UVE
Here derive UVE from I - F with omitted so for simplicity:
Pr (2lss))),,
Px (S)
1 vV
Σ (522,88)
log pM (P
log
p (S))
D2,y
P- (2)
p(2,, 8y) log (1) (92
log Σ p(2, 8y)
+
pT (2
(12))(((1))
Ps (2)
Pr (1)
Σ p(s))p(O(ss)) log
Σ p(12)p(s)((1) log
p (1)
2,,55
Using
3 and 10 we obtain
Pr (S2ss))
Pa (1)
1 VE
Σ p(s)))((((ss) log
Σ p(2))p(s)(2) log
p ({]s)
(1))
2,a5
Σ p(2) log (=(S)pp(())
Σ p(s)) log
(::(3))
(1))p((())
D,5y
((88) (0nn + DKI.
-UVE
C DERIVATION OF U.E AND 0.2
Here we derive 19). We also omit so for simplicity here. We start from / 1
Pr (9 log
Pr (2)p,
7 VV =
Σ
MΩ, 84)
log
p (f)
p (2)
22,4
log Pr (2]))))
log (22)
Σ
p(2, 84)
(::P?( = P- (S2))
(2((2)
2,59
(9)()))))8((1)) (9)))
Pa (9
Σ R,85
p(2 84)
log
+ log (2(s))))) (1]s))
Pp (f(
Px (OQ]s))
<
Σ
p(2, 84) log
p(2, s1) log
(
2,55
P,ay
Pa (1)s))
Σ
p(2, sy)
Σ p(s))p((Q(s)) log
py (10
p [2 (S
0.3y
12.9
UVE
+ UVE. 6.1
12

---

Published as a conference paper at ICLR 2021
D DERIVATION OF THE BOUNDS ON - I,
Here we derive the bounds on the estimation error of mutual information with smoothing. First, we derive the upper bound on (St+11 4:):
fo (8t+1 Te,
Σ
p(s' a1) o';0,
|| - 8 8112
((( + (2
E
p(,'(Te, exp
20²
dl
V(TT²)) (p(8--177)
E
p(8' expp
20
Amnn
V( (p(Sc_1 + (1 p p(3c41177, exp)
20
da
min
22TT22) ( 0.) + exp (-
20
Obviously, we have V(2m22ya P(s az) - fo (st+1 Te: ae) which results in:
p(S(tITT,,)) S v (2TO2)" fo (s ax) S p(st:(lll..0 az) + exp (- L
fo
(24)
20
Similarly, we can obtain the bounds of fo (8 a,, 85) as follows:
d2
p(s441/72, dz, 85) S "2 Te, a, 85) S p(s at, + exp
min
(25)
20²
Combining (24) and 25
we can obtain
d2
pl (8 Te. Oe, 8))
fo(8c4177, (c,
D($141/T4 ae, 8y) + exp (-
may
(26)
p(st+1/Te: Te, (z) + exp
min 20
fo fo(S441|(,
p(8 Te, az 8t,
Taking log and using log (a + b) < loga for a, > to 26
we have
p(8 d1)
min
log p(s at)
exp
Pmin
20
(e,
log
fo(s4+177,,
p( de, 84)
&
log
exp (
p(sct1|Te,0)) 0z)
Pmin.f
20
which results
d2 d
p( Te, ag, 81) log fo (3r41 1 (e,
d2 Lnai
exp
S log
exp
Pminnf
20
p(st+1/Te, at)
Pmin
20
(27)
Using
9
and
27
we obtain
To
Pp(S[]ss,80) (?
fo (S (~((0))
To exp
unnn 2
exp
<
log
log
f
20²
Pmin
By taking the expectation each side, we have
Tmax exp (-
d2
T
min
exp
Pmin.f
20
Punin.f
20
I Io
(28)
T
dmin 20 Pmin
Tman
d2
exp
exp
Puin
2
with Tmax = maxa To and = En.,, p(0, sy [s))) This implies that the estimation error of
mutual information with smoothing converges to for fine Tiaz or T as - 0. Also, for finite T and dmins it satisfies T, x 0.
13

---

Published as a conference paper at ICLR 2021
E ADDITIONAL EXPERIMENT RESULTS
Here show additional experiment results in HalfCheetah-v3 in the Mujoco environments (Todorov et al. 2012) to show the applicability of our Algorithm 2 We expect that both implicit VIC and our Algorithm 2 will show similar results as Fig. 1 and Fig. 2 since it is deterministic environment. We fixed the length of the trajectory as T = 100 to force the agent to learn long enough trajectories. Each motor's action space quantized to 5 actions. We can observe that excit- ing movements (especially triple backflips) are learned by Algorithm 2. Another exciting fact is that since the number of options can learn unlimited, it shows newer behaviors on and on as learning goes on.
Long backflip (success)
Backflip (fail)
Backflip (fail) & bounce
Backflip (fail) & bounce
Backflip (fail) forward flip (?)
Backflip (fail) bounce
Backflip (fail) recover by bounce
Backflip (success) hind leg up
Backward hop & big backflip (fail)
Double backflips (fail, success)
Double backflips (success, fail)
Double backflips (success, fail)
Forward flip (success) & run
Forward
Forward hop & big backflip (fail)
Forward hop big backflip (fail)
Forward hop & forward flip (?)
Forward run & forward flip (fail)
Hind leg up
Triple backflips (success, success, ?)
Figure 6: Learned behaviors by Algorithm in HalfCheetah-v3, The question mark is written when the result the flip is unknown. The various behaviors such as forward flip, backflip, etc. are learned by Algorithm
F EXPERIMENTAL DETAILS
Here we specify experimental details and environment details.
F.1 HYPER-PARAMETTE
Here we show hyper- ppremmparseeterss used in our experiments. We used learning rate le 3 for Fig.
2 and Fig. and le 4 otherwise.
14

---

Published as a conference paper at ICLR 2021
Table Hyper-paramelers used for experiments
Hyper-parameter
Value
Optimizer
Adam
Learning rate
le-3, le-4
Betas
(0.9, 0.999)
Weight initialization
Gaussian with std. 0.1 and mean 0
Batch size
128
T smooth
128
(GMM)
0.25
Ngmm (GMM)
10
F.2 STOCHASTIC ENVIRONMENTS DETAIL
Here specify the transition tables used in this paper.
Table 2: Transition table of stochastic environments.
Transition
go
go
Action
left
right
go left
0.7
0.3
go right
0.3
0.7
(a) Transition table of the stochastic ID world.
Transition
go
go
Action
left
up
down
right
go left
0.7
0.1
0.1
0.1
go up
0.1
0.7
0.1
0.1
go right
0.1
0.7
0.1
go down
0.1
0.1
0.1
0.7
(b) Transition table of the stochastic 2D world.
Transition
go
go
no
Action
left
right
move
go left
0.8
0.0
0.2
go right
0.6
0.2
0.2
Transition table of the stochastic world.
Transition
go
go
go right
Action
up
down
move
go up
0.7
0.0
0.3
go down
0.0
0.7
0.0
0.3
go right
0.0
0.0
0.7
0.3
(d) Transition table the stochastic grid world with rooms.
15


Reviews:
[{"review_id": "mSmn9VQQ8Bu", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\n\nThe paper points out a limitation of the implicit option version of the Variational Intrinsic Control (VIC) [Gregor et al., 2016] algorithm in the form of a bias in stochastic environments. Two algorithms are proposed that fix the limitation: the first requiring the size of state space to be known and the second which does not make such assumptions. Experiments on simple discrete state environments demonstrate that the original VIC algorithm works well only on deterministic environments whereas the proposed fix works well on the stochastic environments as well.\n\n## Strengths\n- The paper provides a sound theoretical analysis of the limitation of the VIC implicit-option algorithm, the proposed fix and a practical algorithm (*Algorithm 2*).\n- A clear distinction is presented with respect to prior work. The differences between the proposed algorithm and VIC shows that the intrinsic reward now has an added term which depends on an approximate model of the transition probability distribution.\n\n## Weaknesses\n- The paper focuses on a very specific and narrow topic without providing much stand-alone motivation for the same. It implicitly borrows motivation from prior work (VIC, etc) without providing its own. The rigorous mathematical derivations are simply re-deriving the VIC mutual information bounds with a new added term and with some extra details on how to do it with a gaussian mixture model. Overall, the paper seems like a minor extension of prior work.\n- Considering the experiments on partially observed environments presented in the VIC paper, this paper chooses a much simpler set of discrete environments for empirical analysis instead of stepping up to more complicated environments which would have strengthened both the motivation for fixing the bias of VIC and the empirical evidence of the GMM algorithm (*Algorithm 2*).\n- The paper's contributions boil down to Eqn 6 (the bias in VIC) and the two proposed algorithms. However, the difficulty in reading the mathematical notations and expressions severely handicaps the reader's ability to carefully understand these contributions. Some suggestions on improving the notation are provided below.\n\n## Feedback to authors\n- The paper introduces extremely dense notation. The frequent overloading of symbols or use of similar looking symbols (e.g. $p, p^p, \\rho$) makes it quite difficult for the reader to parse each expression. I would recommend usage of longer variable names, e.g.: replace p -> gen, for generative model and replace q -> inf for inference models. Phrases are easier to parse than single character symbols. Also, colorizing certain important symbols can help -- especially for important distinctions such as the true probability distributions vs their estimates.\n\n-------\n## Post-rebuttal update\n\nHaving read through all reviews and the author's response, I am updating my assessment in light of the responses and new experiments. I agree with the authors that the derivation has theoretical value and is not a simple re-derivation of VIC. The new experiments and visualizations have been helpful (I am happy with the author's responses to R3), but the overall clarity of the paper is still lacking due to the dense mathematical notation. In light of this, I am increasing my score from 4 -> 6, slightly leaning towards acceptance.", "extra_scores": {}, "extra_text": {}}, {"review_id": "_FK_tPyTv7_", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary of paper:\nThe authors study a version of VIC that represents options as partial trajectories. They point out that, in stochastic environments, the implicit VIC formulation in the original paper is missing a term in the mutual information (involving log likelihood ratios of state transitions). They introduce two ways of estimating the missing term: one appropriate for discrete state spaces, and another appropriate for continuous state spaces. They then compare the empowerment of implicit VIC with and without their corrections in a few toy domains, showing that their corrections do not hurt in deterministic environments, and provide a small increase in empowerment in stochastic environments.\n\nPros:\nThe missing term highlighted, and the derivations of solutions generally looked correct (at least at the level I followed them). This is a potentially interesting contribution.\n\nCons:\n1) The experiments are a) done only in toy domains and b) even there demonstrate only a 5-10% improvement in empowerment. These experiments are maybe fine as a sanity check, but they are not enough to demonstrate the importance of the authors’ correction term. This is because empowerment is not an important objective in and of itself. Empowerment is used as a unsupervised pre-training step for RL, or as an exploration bonus in conjunction with RL. It is useful to the extent it aids performance in those settings. It is not clear to me whether the correction term is important in those pursuits. It could even be that focussing on the part of empowerment due to stochastic state transitions actually degrades the usefulness of the learnt policy. Additional experiments are needed.\n\n2) In addition to experiments justifying the extra term, it would also be useful for the authors to include more motivation and intuition about why and when it is important. It is not immediately intuitively clear to me why incentivizing an agent to drastically alter its trajectory based on random state transitions is useful.\n\n3) In general, the paper is hard to follow. There are long blocks of equations without enough exposition. New terms are defined without motivating first why they are being introduced. The notation is often too dense (e.g. p^p_p? really?). I mostly felt “in-the-dark” as to where the authors were going while reading the paper. Perhaps they could streamline the derivations, push some of it to the appendix, and spend more time in the main text on motivation and intuition.\n\n4) Given the environments aren’t standard and are very simple, they should definitely be introduced in the main text, and not pushed to the appendix.", "extra_scores": {}, "extra_text": {}}, {"review_id": "P7w6Q4Lw1ec", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the problem of maximizing empowerment in the context of RL, where the aim is to maximize the mutual information between some latent variable and future outcomes (e.g., future states). The paper first observes that a procedure proposed in prior work [Gregor 16] is biased and hence does not recover a (latent-conditioned) policy that maximizes mutual information. The paper then proposes a new method, based on learning the transition dynamics. that does recover the optimal (mutual information maximizing) policy. Experiments on a few simple tasks show that the proposed method outperforms prior methods.\n\n**Significance**: Empowerment remains one of the main methods for autonomous skill discovery. Thus, a better understanding of how to optimize empowerment would be an important contribution to this area. This paper identifies a limitation (a biased objective) in a commmon formulation of empowerment [Gregor 16], and proposes a method to correct for this. I think the significance of this paper hinges on (1) how large this bias is for reasonably complex tasks, and (2) if this type of bias might occur in other RL objectives, besides empowerment. The paper only convincingly shows that removing this bias is useful for maximizing empowerment on small scale problems.\n\n**Novelty**: To the best of my knowledge, this limitation of VIC has not been discussed in prior work.\n\n**Experiments**\n* It would be great to include visualizations or ablation experiments to illustrate why implicit VIC has a lower empowerment than the two proposed methods.\n* It'd also be good to include *explicit* VIC as a baseline, even though it requires pre-specifying the number of skills.\n* The experiments are limited to very simple gridworlds and tree domains.\n\n**Clarity**\n* The derivation of the variational bias in S2 is pretty hard to follow. I'd recommend including a bit more discussion of what implicit VIC is and how it differs from explicit VIC, before continuing with the formal derivation.\n* S3 and especially S4 are also hard to follow. I'd recommend moving most of the derivation to the appendix and just stating the final objective as an equation. Theoretical guarantees can then be stated as Theorems/Lemmas with proper proofs.\n\nOverall, I give this paper a score of 5 / 10, primarily because of (1) a lack of clarity and (2) the limited experiments. I would increase my score if the clarify of writing were (greatly) improved, if experiments on higher dimensional tasks were added (e.g., see those in [Achaim 18, Eysenbach 18]), and if additional visualizations of the (suboptimal) behavior of implicit VIC were added.\n\n**Questions for discussion:**\n* How significant is the bias in implicit VIC [Gregor] in more complex tasks? Is it significant enough to warrant the additional complexity of the proposed approach?\n* Is the GMM approach in S4 just a special case of the model learning approach in S3, where the model is taken to be a GMM?\n* Does *explicit* VIC have the same bias as *implicit* VIC?\n\n\n**Minor comments**\n* \"methods for measuring it\" -- This makes it sound like Arimoto + Blahut proposed methods for measuring empowerment. I'd revise to \"along with methods for measuring it based on Expectation Maximization [Arimoto + Blahut]\"\n* \"This can severely degrade the empowerment\" -- Clarify what this means.\n* \"This type of option differs...\" -- Aren't there two differences? (1) Closed loop options depend on the state at each time step, and (2) these options include a termination condition.\n* \" which hinders the maximal level of learning\" -- Add a citation.\n* \"implicit VIC which defines the option as the trajectory until the termination\" -- This sentence is confusing without knowing about the method apriori.\n* \"becomes possible to learn the maximum number of options for the given environment\" -- Add a citation.\n* \"achieve the maximal empowerment\" -- It'd be good to formally define what the \"maximal empowerment\" means.\n* \"environment dynamics modeling incorporating the transitional probability\" -- Grammar error.\n* \"is the inference to be trained\" -> \"is the inference network/model to be trained\"\n* Eq 4: Where are \\tau_t and s_{g | \\Omega} defined?\n* In S3, it might be clearer to use q(...) instead of p^q(...).\n* In Eq 15, it's unclear how \\log p_\\rho^p depends on \\theta_\\pi^q.\n* \"when the cardinality of the state space is unknown\" -- Where does the *cardinality* of the state space show up as a dependency? Perhaps what is meant is that the method is most readily applicable to *discrete* settings, where the distribution over future states can be approximated exactly, without sampling.\n* S4: I'd recommend providing some intuition for why this alternative method is being derived. Is it going to address a limitation of the method in S3?\n* \"Gaussian Mixture Model (GMM) (Reynolds & Rose, 1995)\" -- I think GMMs existed before 1995. E.g., see early work by Karl Pearson.\n* \"extreme gradient\" -- What is an extreme gradient?\n* \"revisited the variational\" -> \"revisited variational\"\n\n----------------------------------\n**Update after author response**:  Thanks to the authors for answering my questions during the rebuttal paper and for incorporating the feedback into the paper! My original concerns were about clarity, high-dimensional experiments, and visualizations. Since the paper has been revised to include nice visualizations and improve the clarity, I am increasing my score 5 -> 6. \n\nI think the experiments on HalfCheetah are a great proof-of-concept of the method! I'd encourage the authors to include some comparisons against baselines for that task.", "extra_scores": {}, "extra_text": {}}, {"review_id": "AzjpsjHfnOL", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "EDIT: The qualitative results help illustrate what the variational bias entails in practice, and indeed the worse coverage constitutes a problem worth overcoming. The Ant experiment was a good attempt at showing scalability, but the deterministic version isn't terribly informative since then the correction term does nothing. Could add stochasticity by taking some number of random actions between the states the agent sees. I suspect that as you increase stochasticity in this way the uncorrected method would degenerate. Would be a clear accept if you could show that, but as is the paper's contribution is bordering on acceptance. 5-->6  \n\nThe authors show that implicit VIC is biased in stochastic environment due to its blindness to the effect of its 'option' on the state transition dynamics. This is addressed by learning a model of these dynamics to allow for the calculation of the missing terms. Toy experiments are then performed that show the boost in mutual information caused by eliminating this bias.\n\nFirst off, its worth mentioning that this is the first real investigation into what intrinsic VIC actually optimizes. The original VIC paper only really explains things for the explicit case, so even the description of what implicit VIC is trying to do is a novel contribution of this work.\n\nThat said, the primary merit of implicit VIC was its scalability, and the new requirement of a generative model of state dynamics can only hurt this. For this paper's extension to be truly significant, it should show a case with significant state cardinality (e.g. the 3D environment from the VIC paper) where the bias hurts more than the reliance on the learned model. The GMM approximation suggests this could be possible, there aren't any experiments that really require its usage.\n\nIn addition to the scale of the experiments, the breadth of evaluation could be expanded -- showing a gap in the MI is suggestive of a more more interesting gap in behavior. For example, can you use the reverse predictor q(a | s, s_f) to reach all possible states by forcing s_f to equal an arbitrary state? 'Percent of states empirically achievable' should be an easy metric to evaluate in these toy environments and would strengthen the case for your extension.\n\nI have a few minor complaints about the analysis itself. While generally easy to follow, the notation is a bit cumbersome. Once the options are explained in terms of the underlying trajectories, why stick with the option notation? Eliminating the Omegas from the loss terms would be the implications of your extension much for explicit, though perhaps this would just make everything a bit too ugly. I don't follow the significance of equation 5, and would appreciate it being unpacked a bit more. All of the options considered are fixed-length in practice, so I'd omit all of the bits regarding termination actions. What's the difference between showing an upper-bound on the approximation error versus a lower-bound on the true value? If the extension isn't a lower-bound, then the implications of this should be explained.\n\nOverall, I like this paper and wish more papers would be like this. Illuminating a previously neglected algorithm is worthwhile and I want to reward that. But suggesting a model-based approach for an algorithm which cited its model-free nature as a primary motivation requires a more thorough investigation. I'm convinced that you've found a flaw, but I'm not convinced your solution actually improves things. \n\n\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper revisits the under-explored "implicit" variant of Variational Intrinsic Control introduced by Gregor et al. They identify a flaw that biases the original formulation in stochastic environments and propose a fix.

Reviewers agree that there is a [at least a potential, R4] contribution here: "even the description of what implicit VIC is trying to do is a novel contribution of this work", in the words of R2, and "the derivation has theoretical value and is not a simple re-derivation of VIC", in R4's post-rebuttal remarks. Several reviewers raised significant concerns around clarity, which were addressed in an updated manuscript, which also provided new visualizations and new experiments which reviewers found compelling. All reviewers agreed that the revised manuscript was considerably improved.

R4's score stands at the 5, with the other reviewers all standing at 6. R4's main concerns are around whether the missing term in the mutual information identified by the authors is a problem in practice on non-toy tasks (echoing somewhat R3's concerns re: high-dimensional tasks). While this is a valid concern, the function of a conference paper needn't necessarily be to (even attempt) to provide the final word on a matter. Identifying subtle issues such as the one brought forth in this manuscript and re-examining old ideas is a valuable service to the community, and this paper will serve as a beginning to a conversation rather than an end. The AC also considers themselves rather familiar with the original VIC paper, and found the results herein somewhat surprising and noteworthy.

I recommend acceptance, but encourage the authors to incorporate remaining feedback in the camera-ready.
Decision: Accept (Poster)