DS-VIC: Unsupervised Discovery of Decision States for Transfer in RL
Authors: Nirbhay Modhe
Prithvijit Chattopadhyay
Mohit Sharma
Abhishek Das
Devi Parikh
Dhruv Batra
Ramakrishna Vedantam
Venue:  (2020)
Keywords: reinforcement learning
probabilistic inference
variational inference
intrinsic control
transfer learning

Abstract: We learn to identify decision states, namely the parsimonious set of states where decisions meaningfully affect the future states an agent can reach in an environment. We utilize the VIC framework, which maximizes an agent’s `empowerment’, ie the ability to reliably reach a diverse set of states -- and formulate a sandwich bound on the empowerment objective that allows identification of decision states. Unlike previous work, our decision states are discovered without extrinsic rewards -- simply by interacting with the world. Our results show that our decision states are: 1) often interpretable, and 2) lead to better exploration on downstream goal-driven tasks in partially observable environments.

Under review as a conference paper at ICLR 2020
DS-VIC: UNSUPERVISED DISCOVERY OF DECISION STATES FOR TRANSFER IN RL
Anonymous authors Paper under double-blind review
ABSTRACT
We learn to identify 'decision states', namely the parsimonious set of states where decisions meaningfully affect the future states an agent can reach in an environment. We utilize the VIC framework (Gregor et al., 2016), which maximizes an agent's s "empowerment'' i.e. the ability to reliably reach a diverse set of states and formu- late a sandwich bound on the empowerment objective that allows identification of decision states. Unlike previous work (Goyal al. 2019), our decision states are discovered without extrinsic rewards- simply by interacting with the world. Our results show that our decision states are: 1) often interpretable, and 2) lead to better exploration on downstream goal-driven tasks in partially observable environments.
INTRODUCTION
Not all states in decision making process are created equal. Consider the middle illustration in Figure where robot has four 'options', each representing a goal shape it can potentially reach. Given its spawn location, it can initially proceed straight regardless of the option it chooses until the intersection, at which point needs to utilise 1 bit of (Shannon) information from the option variable to inform the choice of whether to turn left or right. However, right after it takes the left turn (say), again does not need the goal information when choosing actions and can follow 'default' or 'option-ind" iiiiinnnndeeeneent' behaviour. Thus, there is a natural difference in the minimum amount of necessary goal information or 'relevant information' (Polani et al. 2006) needed at different states.
Identifying these 'decision states', i.e. states in the environment where a high amount of relevant goal information is needed, leads to better understanding of the environment structure, which has the potential to enable better transfer to novel environments and tasks. This has previously been shown by Goyal et al. (2019) in partially-observabbee goal-driven (i.e. with extrinsic rewards) 2D navigation settings, where using decision states to guide exploration enables faster learning.
Goal-independent Decision States. Our key intuition is that decision states exist in an environment independent of extrinsic goals. For example, one can imagine the (middle) red tile at the intersection in Figure [ being useful decision state even across all possible navigation goals the agent could have. Ofcourse, not all intersections are decision states. For e.g., if the left room was full of lava, where the robot would die, thinking of the middle red tile as decision state would have limited utility, since the relevant goal information (across all reachable goals) in this case would be bits.
Identifying 'unsupervised' (or task-agnostic) decision states is advantageous in scenarios where: 1) rewards are sparse or absent (Pathak et al. 2017), 2) for agent learn meaningful behaviour, proxy goals and rewards need to be hand engineered (making it hard to scale), and 3) the notion of a goal might not even be obvious, e.g. in continuous control tasks (Lillierap al. 2015).
Our DS-VIC method identifies decision states without extrinsic rewards, solely by interacting with partially obbeevlle environments. These decision states are function of the environment well as the states an agent can reliably reach. We demonstrate that our decision states generalise to novel environments and tasks, leading to comparable (or better) exploration and performance on downstream tasks compared to Goyal et al.) (2019) that identifies goal-driven decision states.
Decision States in Empowered Agents. We build on the VIC framework (Gregor et al. 2016) (Figure D. where the core idea is to maximize the empowerment of an agent. In simple terms, an empowered agent is able to reliably reach a diverse set of states in the environment, while avoiding states that are difficult to reach reliably. We use VIC's formulation learn options Ω that maximize (a lower bound Tvrc on) the mutual information T(S), .) where Sy is the final state in a trajectory. We use latent options since operate in setting without external goals. To why this maximizes

---

Under review as a conference paper at ICLR 2020
VIC (Gregor et al., 2016)
DS-VIC
9gg
S4
fg
13g
S3s
So
transfer
decision default behavior
Figure 1: Left: The VIC framework (Gregor al. 2016) in navigation context: agent learns high- ligh-level macro-actions (or options) to reach different states in an environment reliably without any extrinsic reward. Right: DS-VIC identifies decision states (in red) associated with options where the agent empowered to make informed decisions. The rest of the states in a trajectory (dotted lines blue regions) then show default, option-independent hchavior. Identification of decision states leads to improved transfor to novel enviromments.
empowerment, notice that S2) HS))) HS)) where (.) denotes entropy. Thus, empowerment maximizes the diversity in final states S while learning options highly predictive of S (Sage et al. (2013). Using VIC, we identify decision states by attempting to hit the relevant informattonn requird at a particular state by minimizing the mutual information [ A, S.) (for every state St, action Ad) This is similar in spirit to Goyal et al. (2019); Polani et al. (2006), with the key difference that we use latent options Q instead of external goals. Interestingly, we prove that L ((2, A, St)) is an upper bound on empowerment, which we minimize, in addition to maximizing the VIC lower bound JvIC- This shines connection VIC- while they optimize lower bound, our work optimizes a sandwich bound on the empowermen To summarize our contributions,
We propose DS-VIC, novel framework to identify decision states in a task-agnostic manner. These decision states align with our intuitive assessments in various cartially-observed scenarios. We provide an understanding of our proposed objective, proving that our framework optimizes sandwich bound on the empowerment I(22, Sy).
We show that our mechanism to identify decision states is transferable and leads to improved sample-efficiency goal-driven tasks in novel environments. On challenging grid-world navigation task, our method outperforms (a re-implementation of) Goyal et al. (2019).
METHODS
2.1 NOTATION
We consider a Partially Observable Markov Decision Process (POMDP), defined by the tuple AS, P.r), where the (discrete or continuous) state-space S is set of underlying but unobserved denotes an unknown transition
states E and A denotes discrete action space. P P: S x S function, representing p (s141/St, at) St, 81+1 E S, A, E A. Both VIC and DS-VIC train an option (52) conditioned policy (a; (w, st), where w {1,... |(2]]. Following standard practice (Cover & Thomas 1991), we denote random variables in uppercase (2), and items from the sample space of random variables in lowercase (w).
2.2 VARIATIONAL INTRINSIC CONTROL (VIC)
The option Ω in VIC is global latent variable underlying trajectory (Figure 2). To encourage the agent to reach a diverse set of states reliably, VIC proposes maximizing the mutual information (Cover & Thomas 1991) between Q and final state Sy given su, i.e. I(S), Ω So = s0). Informally, this maximizes the empowerment for an agent, í.e. its internal options have high degree of correspondence to the states of the world Sy that it can reach. VIC formulates variational lower bound on this mutual information. Specifically. let p(w 8n) = p(w) be prior on options (we keep the prior fixed as per Eysenbach al. (2018)), p (s) / s. is defined as the (unknown) terminal state distribution achieved when executing the policy π Ta; w, St). and qh 5 so) denote (parameterized) variational approximation to the true posterior options
(So, Ao.
S
'This has potential implications beyond the context of this paper, for example, in training VIC models which target particular amount of "compression" in the option (!2) representation, similar representational choices for supervised (Tishby al. 1999) and unsupervised learning (Alemi et al. 2016).
2

---

Under review as a conference paper at ICLR 2020
VI::
g(w[S} e2, #o)
DS-VIC: Policy Parameterization VIC
plu so)
Ω
St
II IT (oplw,, sc)
So
S1
S
der wl
Z+
x((a ax)
Modil je jetamaal to agenti
Ao
A
A
--.- Inforonce internaa to aggggnt
T
Transitirns
Figure 2: Illustration of VIC for 2 timesteps. L: Given start state So, VIC samples option and follows policy <(a; 1== st) and infers ! from the terminating state (S2), optimizing lower bound on ((S,, So). R: DS-VIC considers particular parameterization of and imposes bottleneck [(A,, (2]S.).
given Sy and So- Then:
log gx Sy, So so) p(2))
1(2, Sy So = so) 2 Eample)) (o
Sy;
2.3 DECISON STATE VIC (DS-VIC)
We identify decision states by employing an information regularizer that finds states with minimum relevant option information. Formally, this means that at every timestep t in the trajectory, we minimize the mutual information [[2,, Ar|S, So s) and the resulting states where this mutual information remains high despite the minimization are identified as decision states. Intuitively, this means that on average (across different options), a decision state state with higher relevant option information than other states (e.g. the red regions in Figure T Overall, our objective is to maximize:
Ivic Syi
Σ 102, St Sq sg)
where B > 0 is trade-off parameter. Thus, this is saying that one wants options Ω which allow the agent to have a high empowerment, while utilizing the least relevant information at each step.
Interestingly, Equation 2 has clear, principled interpretation in terms of the empowerment [ S (S) from the VIC model. We state the following lemma (full proof in Appendix):
Lemma 2.1 Let A₁ be the action random variable at timestep t and state St following an option- conditioned policy (aa])se.w) Then, I(22, At)St, i.e. the conditional mutual information between the option Ω and action A, when summed over all timesteps in the trajectory. upper bounds the condi- tional mutual information I ( SS)) between Q and the final state Sn namely the empowerment as defined by Gregor et al. (2016):
1S,
S Σ 100,A = Upss (2,
(3)
Implications: With this lens, one can view the optimization problem in Equation aas Lagrangian relaxation of the following constrained optimization problem:
max Iv s.t. Ups R
where R > is a constant. Thus, we can essentially target values of R we have in mind by maximizing the lower bound and minimizing the upper bound, by (in practice) sweeping different values of 3 in eq. (2) This is a result potentially of interest beyond the specific context of decision states (Achiam et al. 2018 Gregor et al. 2016). Specifically, while the VIC objective only allows us maximize the empowerment / ($2, Sy H(S,) H H(S, (2), imposing an upper bound can help control how 'abstract' the latent option representation is relative to the states Sy, by finding solutions that have say higher entropy (Sy (22) on states given options. This might lead more abstract options that achieve better generalisation depending on downstream tasks. Note that most approaches currently limit the abstraction by constraining the number of discrete options, which (usually) imposes (tighter) upper bound on (Ω)) since 2 0 in the discrete case. However, this does not hold for the continuous case, where the bound might be more useful. Investigating this is beyond the scope of this current paper, however, as our central aim is to identify decision states, and not scale the VIC framework to continuous options.
3

---

Under review as a conference paper at ICLR 2020
2.4 ALGORITHMIC DETAILS
Upper Bounds for I(2, A, | St, So). Inspired by InfoBot (Goyal et al. 2019). we bottleneck the information in a statistic Z, of the state S, and option F used to parameterize the policy π (A, 12 S.)
(fig. right). This is justified by the the data-pre processin inequality (DPI) (Cover & Thomas
1991)
for the markov chain S2, S, ++ Zi A,, which implies I( A₁ St, So) I 1[2, Zz St, So). We
can then obtain the following upper bound on (S2 Ze St, So) (see appendix for derivation):
p(Z St)
T(22, Ze So =) Spppp()) (re (0, Zepp)))) log
(5)
g(Zx)
where g(=x) is a fixed variational approximation (set to (0, I) as in InfoBot), and pa w, w,& is a parameterized encoder. As explained in section
D the key difference between eq. (5) and InfoBot is that they construct upper bounds on I(G, At St, So). while we bottleneck the option-information
We can compute Monte Carlo estimate of Equation by first sampling an option w at so and then keeping track of all states visited in trajectory т. In addition to the VIC term and our bottleneck regularizer, we also include the entropy of the policy over the actions (maximum-entroppy RL (Ziebart 2010 as a bonus to encourage sufficient exploration. Overall, the DS-VIC objective is:
max J(0, 0) = 0,0.2
log (x (11 So)
Σ 1=0
f-1
(Blog
Po (Z+
S1 ((,))
+ a log (A Si, 2:)]
(6)
((2)
where 0, 6 and v/ are the parameters of the policy, latent variable decoder and the option inference network respectively. Note that we abuse the notation slightly in Equation by writing T ~~ p' w, 5o) to denote a trajectory sampled by following (- w, 80). This is a sequential process where Z, ~ (= St, sr,w). (a(a; St, 24) and S.41 ~ p((++1 St, az) (state transition function). The first term in the objective ensures reliable control in terms of visitation over a diverse set of final states in the environment while learning options; the second term ensures minimality in using the options sampled to make decisions (taking actions) and the third provides an incentive for exploration. See algorithm I in appendix for more details on the entire training pipeline.
Handling Partial Observability. We are generally interested in identifying decision states in partially observable environments (similar to anfooBotGoaa et al. (2019)). To formalise how we handle this, let or be the (partial) observation made by the agent at time i. We set the state 8 = g(og. ... o.) in all the equations above, where is a recurrent neural network (RNN) (Hochreiter & Schmidhuber 1997) encoding partial observations. We found it important to parameterize the policy (Figure 2 right) as: [a; w. st) ] dz p(= st, w)p(p; oz, 2), i.e. where action distribution p(a: oz, 2) are modelled 'reactive' (conditioned only on the current observation) but distributions p(z; w) are 'recurrent" (conditioning on the entire history of observations via 9(-)). This is because the sequence of observations {ol,-..,0 02} might be informative of the option Ω being followed i.e. conditioning p(a, 1-) on S; in addition to could 'leak' information about the underlying option to the actions, making the bottleneck penalty on z potentially ineffective. Finally, when maximizing the likelihood of option Ω given the final state Sj in learning g(w 8y) in VIC (the option inference network), we use the global (x, y) coordinates of the final and initial state of the agent. Note that the agent's 's policy continues to be based only on observations or, and it is only at the end of the trajectory that we use the global coordinates to compute intrinsic reward. Please refer to Sec. Sec.[A.6]for for a discussion of the issues when training the model with partial observations in the option inference network.
Related Objectives. Eysenbach et al. (2018) (DIAYN) attempts to learn skills (similar to options) which can control the states visited by agents while ensuring that states, not actions, are used to distinguish skills. Thus, for an option l and every state S. in a trajectory, they maximize E, [22 S,) I(A,, 22 (t) + H (A[))),, as opposed to ( Sy - 3 E, I(A, S + ((A))S)) in our objective. With the sum over all timesteps for [(22, S.), the bound in theorem 2.1 no longer holds true, which also means that there is no principled reason (unlike our model) to scale the second term with 9 -- which we find to be crucial for identifying decision states (similar to findings in InfoBot). Experimentally, we show that the decision states with DIAYN are no longer as interpretable, since options in this case are more low level and correspond trajectories instead of goals.
4

---

Under review as a conference paper at ICLR 2020
The most closely related work to ours is InfoBot, which maximizes Σ, R(t) (I,,, G[St) for a goal (G) conditioned policy T( G). The key difference is that InfoBot requires extrinsic rewards, while our work is strictly more general and scales even in the absence of extrinsic rewards.
Further, in context of both these works, our work provides a principled connection between action- option information reqularization 7 ((A,))])s) and empowerment of an agent. The tools from Theo- rem 2.T might be useful for analysing these previous objectives which both employ this technique.
EXPERIMENTS
3.1 TRANSFER TO GOAL-DRIVEN TASKS
Do the unsupervised decision states meaningfully transfer to novel tasks? To address this, we replicate the InfoBot experimental setup, replacing decision state identification (pre-training) with DS-VIC.
Exploration-bass Transfer, Goyal et al. (2019) pretrain their model to identify decision states, and then study if decision states improve exploration when training a new policy T.y (a]s, g) from scratch. Given an environment with reward R. (t), goal G, K > 0, and state visitation count c(,)), their reward is:
Rs (t) +
I(G, Z4 SS.)
(d()))
Preirained, Frozen
The count-based reward decays with square root of c(S,) to encourage the model to explore novel states, and the mutual information between goal G and bottleneck variable Z, identifies decision states, and is multiplied with the exploration bonus to encourage visitation of decision states.
We almost identical setup, replacing their decision-state term from supervised pretraining with DIC intrinsic reward pretraining:
Powr Drponemeent ban
-.59
0.62
2:00
R₂ (t()) + V((S))
T(2, Z, S,, St, So)
(8)
Pretrained, Fruzen
I(-) computed with eq. (5) with a parameter- ized p(z, w, 8t) encoder frozen during transfer. See Algorithm [ in appendix for more details.
Crponemeett
2.02 1.39
2.22
Figure 3: [9,, S., at visited states on simple envi-
Decision state heatmaps plotting
Environments. We pre-train and test on grid- worlds from the MiniGrid CheevllierBoiiseert et al. 2018) environments. We first consider of simple environments 4-Roam and Maze (see Fig.33 followed by the Mult tooonNXYY also used by Goyal et al. 2019). The Mul RoomnNKY environments consist of X rooms of size Y. con- nected in random orientations across multiple runs. We also conduct experiments Mountain-Car (Brockman et al. 2016), a continuous state-space environment with discrete actions. In all pre-training environments, options are executed for fixed number of steps before terminating.
ronments 4-Room (top) and maze (bottom). First column depicts environment layout, second and third show results for DS-VIC for á = 1e-3 and B 1 respectively. and the fourth column shows DIAYN. Please refer to Appendix [A.4 for details about how I(2, Zz (S, So) is computed. * denotes lower hound [Eq.L) on empowerment.
We use Advantage Actor-Critic (A2C) for all experiments. Since code for InfoBot (Goyal et al. 2019) was not public, we report numbers based on re- implementation of InfoBot, ensuring consistency with their architectural and hyperparameter choices. We will release our code upon publication.
4 RESULTS
Baselines. We evaluate the following on quality of exploration and transfer to downstream goal-driven tasks with sparse rewards: 1) InfoBot (our implementation) which does goal-driven extraction of decision states, 2) DIAYN whose focus is unsupervised skill acquisition (and not decision
Visitatioo counts have several limitations including the requirement of a discrete state space and state table for maintaining visit counts in POMDPs. However, our exploration bonus need not be restricted by these assumptions. One could alternatively use technique similar to Burda al. (2018) where the incentive is learn to distill trained encoder network, alleviating the need for visitation counts altogether.
5

---

Under review as a conference paper at ICLR 2020
states] but has an (AA,,)))) term which can be used for the bonus in Equation 8 3) count-based exploration which uses visitation counts as exploration incentive (this corresponds to replacing
[[2, 2, with 1 in Equation 8). 4) randomly initialized encoder p(z,, w, 8;)) to check if learning in DS-VIC or InfoBot models improves over random initialization, 5) how different values of & affect performance, and 6) a heuristic baseline that looks for local features like corners and doorways and uses every such occurrence as decision state. This validates the extent to which relevant goal information is useful in identifying decision states, vs. local, feature-based heuristics.
4.1 QUALITATIVE RESULTS
Grid Worlds: Figure shows results on 4-Room and maze. We notice that when 3 = 1. DS-VIC collapses to identifying every state as a decision state and gets poor empowerment: 0.51 (row 1, column 3 in Fig indicating that the regularization strength is too high and hinders empowerment maximization. At lower values of = 1e-3, we get more discernible decision states, Across both environments, we notice that some corners and intersections are identified as decision states. In general, for DS-VIC maze with 3 = 1e-3 (when it gets an empowerment of 2.61), we notice that if an intersection a decision state, and there is a corner next to it, it is often nor a decision state. That is, not every state which looks like corner needs to be decision state, if for example the part of the state space in question is only spanned by one option. Finally, for maze we see that for a similar value of empowerment [ DS-VIC leads to sparser decision state distribution than DIAYN. This makes sense, since intuitively, one expects to have to make more decisions when following a particular trajectory (as in DIAYN), as opposed to terminal state (as in DS-VIC).
0.00
0312
007
0.005
1.0
0.305
C.07
608
6.005
4.000
40000
04
4.00
4,007
40010
02
4016
6.0
4.01E
C50
0.20
0.55
-0.50
(socordeees))
(scooonioatt)
(a) Environment
(b) Trajectories
(c) Final State
(d) Decision States
Figure 4: We visualize identified decision states for 3 options by DSC on the Mountain Car environment. (a) shows the environment layout, (b) the trajectories corresponding to the options in position-velocity space if trajectories corresponding to two different options reach the same position and velocity, then they are said to intersect. (c) the final-state distributions in the position-velocity space, and (d) the identified decision states. In (b), (c), (d), x-axis (position) shows the x-coordinate of the car, while y-axis shows the velocity of the car.
Mountain Car Environment: We also use DS-VIC to identify decision states in the Mountain Car (Brockman al., (016) environment continuous state-space (position and velocity of the car) environment with three discrete actions {+1(right). ). 0, 1 (1efft)) }. which correspond to the direction of the force being applied to the car (see Fig, a)). Fig. bb) shows the trajectories and Fig. ()) shows the final state(s) corresponding to 3 specific options (out of 8) overlaid on a position-velocity plane. Upon observing decision states in the same position-velocity plane (see Fig. 4d), point-radius proportional to [S2, Ze| (s,, 8o), we find that decision states occur at points in the state-space where trajectories associated with different options intersect. Intuitively, these are the states where the agent can switch from one option (mode of behavior) to another and therefore, needs to make a decision. Interestingly, there is a dominance of decision states along the velocity = 0 line. which indicates that the model identifies decision states to be points where the cart has velocity at some position (specific to the option). Finally, decision states often occur, for say, the orange option (12)) in the right half of the position-velocity plane and never in the bottom-left quadrant, where is the only option covering that part of the state-space, and thus default behavior can be practiced.
$ 'Note that to keep things consistent with our method and InfoBot, we use (Z7,2)5.) to impose the mutual information regularization in DIAYN, whereas Eysenbach et al. (2018) use a different construction. "Since DIAYN maximizes the mutual information with every state in trajectory, report the empowerment for the state with maximum mutual information with the option.

---

Under review as a conference paper at ICLR 2020
Method
MultiRoomN334
MultiRoomNSS4
MultiRoomNN6625
(Eucoder pal Zz) (1, (2) pretrained on)
MultiRoomN256
MultRoomNN266
MultiRoomN2S10
InfoBot (Goyal et al. 20199)
90%
85%
N/A
InfoBot (Our Implementation)
99.6%61o.25
80.9%
86.8%.1..38
Count-based Exploration (Our Implementation)
99.8% 99.8%610.2%
71.4 71.4%%226.19
88.45612.55
DIAYN-bused Exploration Bonus (8 = 1)
99. 7 7%
99.7% 7%%
0..1%
Randomly Initialized Encoder
+00%
98.8% +0.7%
67.9%+*1 7%
Heuristic Decision States
N/A
N/A
81.3% 0%
DS-VIC (3- 10-3)
99.9% 19.9%....15
99.1% 1..3%%
90.9%....
DS-VIC (3 1
100%++0%
71.1%==29.19
92.6%.+o.r
Table 1: Success rate (mean + standard error) of the goal-conditioned policy when trained with different exploration bonuses addition to the extrinsic reward (t).). We report results at ~5 x 10 timesteps for MultiRoomN3S4 MultiRoomN5S4 and at ~-8.2 x 10 timesteps for MultiRoomN6625. We also report the performance of InfoBot for completeness. Note that for rooms of size 4 (MuliRoomNNS44 MultiRoomN5S4), incentivizing visit corners and doorways (Heuristic Decision States) is equivalent to count-based exploration.
4.2 QUANTITATIVE RESULTS
Transfer Goal-Driven Tasks, Next, we evaluate Equation g i.e. whether incentivizing the agent to visit decision states in addition to extrinsic reward can aid in transfer to goal-driven tasks in different environments (see Sec. B.1.. We restrict ourselves to the point-navigation task (Goyal et al., 2019) transfer in the always spawned in the first room, and has to go randomly sampled goal location in the last room, and is rewarded only when it reaches the correct cell. Goyal et al. (2019) test the efficacy of different exploration objectives and show that this is hard setting where efficient exploration is necessary. They show that InfoBot outperforms several state-of-the art exploration methods in this environment.
set of partially-observable environments. In this task, the agent is
Concretely, we 1) train DS-VIC to identify de- cision states (Equation 2 on MultiRoomN256 and transfer to goal-driven task Multi and MultiRoomN554 (similar to Goyal et al. (2019)), and 2) train on MultiRoomN2S10 and transfer to MultiRoomN6S25, which a more challenging transfer task i.e. it has larger rooms (25x25) making efficient exploration critical find doors quickly. For DS- VIC and DIAYN, we train for 6 million episodes in the decision state identification phase (pre-training) and pick the checkpoints with highest empowerment values across training.
Quee (8=10
Ours (2-10
Ouee (1-10
Gune (2=105)
Sucgess
06
Rendominn
02-
D0-
Timosteps (
Figure 5: Effect of 3 during VIC pre-training success with exploration bonus for the goal- conditioned policy. Shaded areas represent stan- dard error of the mean over 10 random seeds.
Overall Trends. Table. reports success rate the % of times the agent reaches the goal on validation. First, implementation of InfoBot is competitive with Goyal et al. (2019 For the MultioommN256 to N5S4 transfer (middle column), methods which identily decision states outperform count-based exploration. Interestingly. in this setup, a randomly initialized encoder is competitive with both InfoBot and DS-VIC, and serves as an important baseline. In MultiRoomN25 10 to N6S25 transfer, which has large state space, we find that DS- (at 3 = 10-2) achieves the best performance, followed by count-based exploration and InfoBot (that are both within standard-error of each other). This shows that decision states identified by DS-VIC effectively guide exploration in sparse-reward settings. More importantly, our model beats strong heuristic baseline which treats every corner and doorway as a decision state (more details on this are in the appendix). Finally, the randomly initialized encoder well as DIAYN generalize much worse in this transfer task.
3 sensitivity. We sweep over / in log-scale from {10-1 .... 10 10-6), as shown in Figure 5 (except 10-0-1 which does not converge information regularization baseline. We find that 10-2 works best, followed by 10 and 10 3 In general, performance is robust, i.e. across B, seem reach 90% success for most values.
empowerment) and also report = 0 which recovers no
'While our focus is on identifying and probing how good the decision states from intrinsic training are, more broader comparisons to exploration baselines are in InfoBot (Goyal et al. 2019) (reproduced in the appendix). WW found important to all models (inlcuding InfoBot) order of magnitude more steps compared to Goyal al. 2019) but our models also appear converge to higher success values.
7

---

Under review as a conference paper at ICLR 2020
a.miquu Rooms Reacchod
00-
Average Tosk (anurn
Couriiased
Hurrss Soad
DEVN
64-
62-
60-
00
Timestens (c 117)
Timesteps
16F)
Timesteps 0 10f)
(a) Success Rate
(b) Avg. Task Returns
(c) # Unique Rooms Reached
Figure 6: Transfer results on NultiRoomN6625 after unsupervised pro-training to identify decision states on
MultiRoomN2S6. Shaded regions represent standard errors of the mean over 10 random seeds.
More results on N6S25. To better understand Equation 8 compared to other exploration incentives, in addition to success rate and sample-effciiency (Figure (6, also consider another metric where we count the number of unique rooms reached (other than the starting room 0) as training progresses (Figure 6 (c)). Across metrics, DS-VIC performs the best on the challenging N6S25 transfer. Furthermore, the unique room metric reveals that DIAYN gets stuck after exploring the first room.
5 RELATED WORK
Intrinsic Control and Intrinsic Motivation. Learning how to explore without extrinsic rewards is a foundational problem in Reinforcement Learning (Machado et al. 2017: Pathak et al. 2017 Gregor jtt al.) 2016; Strehl & Littman 2008: Schmidhuber) 1990). Typical curiosity-driven approaches attempt to visit states that maximize the surprise of an agent (Pathak et al. 2017) or improvement in predictions from dynamics model (Stadie et al. 2015 Lopes et al., 2012). While curiosity-driven approaches seek out and explore novel states, they typically do not measure how reliably the agent can reach them. In contrast, approaches for intrinsic control (Eysenbach et al., 2018 Acciam et al.) 2018; Gregor et al., 2016) explore novel states while ensuring those states are reliably reachable. Gregor al. (2016) maximize the number of final states that can be reliably reached by the policy, while Eysenbach et al. 2018) distinguish an option at every state along the trajectory, and Achiam ctt al. (0118) learn options for entire trajectories by encoding the sequence of states at regular intervals. Since decision states are more related to reliably acting in an environment rather than just visiting novel states (without an estimate of reachability),
formulate our regularizer in the intrinsic control framework, specifically building on the work of Gregor et al. (2016).
Default Behavior and Decision States. Recent work in policy compression has focused on learning a default policy when training on a family of tasks, to be able to re-use behavior across tasks. In (Galashov et al., 2018 Teh et a 2017). default behavior is learnt using a set of task-specific policies which then regularizes each policy, while Goyal et al. { (2019) learn a default policy using an information bottleneck on task information and latent variable the policy conditions on. We devise a similar information regularization objective that learns default behavior shared by all intrinsic options without external rewards so as to reduce learning pressure on aption-conditions policies.
Bottleneck states in MDPs. There is rich literature on identification of bottleneck states in MDPs. The core idea is to either identify all the states that are common to multiple goals in an environment (McGovern & Barto 2001 or use diffusion model built using an MDP's transition matrix (Machado) let al.] 2017) 7] Şimsek & Barto 2004 Theocharous & Mahadevan 2002). The key distinction between bottleneck states and decision states that the latter more closely tied to the information available to the agent and what it can act upon, whereas bottleneck states are more tied to the connectivity structure of an MDP and intrinsic to the environment, representing states which when visited allow access to a novel set of states (Goyal et al. (2119). Concretely, while a corner in a room need not be a bottleneck state, since visiting corner does not 'open up" new states (the way a doorway would), it is still a useful state for a goal-driver agent with partial observation to visit (since it a distinct landmark in the state space where meaningful decisions can be made).
Information Bottleneck in Machine Learning. Since the foundational work of Tishby et al. (1999) ; Chechik et al. (2005), there has been a lot of interest in making use of ideas from information
bottleneck (IB) for various tasks such as clustering (Strouse & Schwab] 2017 Still al. 2004). sparse
coding (Chalk et al. 2016),, classification using deep learning (Alemi et al. 2016 Achille & Soatto
2016), cognitive science and language (Zaslavsky al. 2018) and reinforcement learning (Goyal
et al. 2019 Galashov et al., 2018, SStrouse et al. 2018). We apply an information regularizer to an RL. agent that must learn without extrinsic rewards to identify decision states.
8

---

Under review as a conference paper at ICLR 2020
6 CONCLUSION
We devise a principled approach to identify decision states in an environment without any extrinsic reward supervision using a sandwich bound on the empowerment of Gregor et al. (2016). Our approach yields decision states that align with human intuition across environments, and aid directed exploration on external-reward tasks and subsequently lead to better success rate and sample com- plexity in novel environments (competitive to supervised decision states of Goyal et (2019)) All our code and environments will be made publicly available.
REFERENCES
Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational option discovery algorithms. arXiv preprint 18Xiv:1877.10299 2018.
Alessandro Achille and Stefano Soatto. Information dropout: learning optimal representations through noise. 2016.
htttp://arxiv.org/absss.
.08353
Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy, Deep variational information bottleneck. In ICLR, 2016. URL http:://arxiv.orr//bb/////// / 00410
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv: /2894, 2018.
Matthew Chalk, Olivier Marre, and Gasper Tkacik. Relevant sparse codes with variational information bottleneck. 2016.
Gal Chechik, Amir Globerson, Naftali Tishby, and Yair Weiss. Information bottleneck for gaussian variables. J. of Machine Learning Research, ((Jan))165-188) 2005. URL httt:://ww..jiii org/
pdf]
Maxime Chevalier-Boisvert Lucas Willems, and Suman Pal. Minimalistic gridworld environment for openai gym. https://githul. com/ http://waxmmmccbbccm haat///gg-mmiiiiidd.. 2018.
M. Cover and J. A. Thomas. Elements of Information Theory, John Wiley, 1991.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without reward function. CoRR, 10ss1800200070, 2018. URL http:
Alexandre Galashov, Siddhant M Jayakumar, Leonard Hasenclever, Dhruva Tirumala, Jonathan Schwarz, Guillaume Desjardins. Wojeiech M Czarnecki, Yee Whye Teh, Razvan Pascanu, and Nicolas Heess. Information asymmetry in kl-regularized rl. 2018.
Anirudh Goyal, Riashat Islam, Daniel Strouse, Zafarali Ahmed, Matthew Botvinick, Hugo Larochelle, Yoshua Bengio, and Sergey Levine. InfoBot: Transfer and exploration via the information bottleneck. arXiv [stat.ML], January 2019.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint arXiv:1611.07500 2016.
S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation, 133:1733 1997.
Ilya Kostrikov. Pytorch implementations of reinforcement learning algorithms. https://github.ccm Hosstrikko//pytorc--2----- -cktr-ao---aaalll 2018.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint
2015.

---

Under review as a conference paper at ICLR 2020
Manuel Lopes, Tobias Lang, Mare Toussaint, and Pierre- Yves Oudeyer. Exploration in model-based reinforcement learning by empirically estimating learning progress. In Proceedings of the 25th International Conference Neural Information Processing Systems Volume 1, NIPS' 12, pp. 206-214, USA, 2012. Curran Associates Inc. Utthhttp://dl att.///llcccc...//oommm.m//// http:////ccattoo.cc/ii///-///e 5899158
Marios C Machado, Marc G Bellemare, and Michael Bowling. A laplacian framework for option dis- covery in reinforcement learning. In Proceedings the 34th International Conference on Machine Learning Volume 70, ICML'17, pp. 2295-2304, Sydney, NSW, Australia, 2017. JMLR.org.
Amy McGovern and Andrew G. Barto. Automatic discovery of subgoals in reinforcement learning using diverse density. In ICML, 2001.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 16-17, 2017.
Daniel Polani, Chrystopher Nehaniv, Thomas Martinetz, and Jan Kim. Relevant information in optimized persistence vs. progeny strategies. In In: Artificial Life X: Proceedings of the Tenth International Conference on the Simulation and Synthesis of Living Systems. Mit Press, 2006.
Christoph Salge, Cornelius Glackin, and Daniel Polani. Empowerment an introduction. October 2013.
Jürgen Schmidhuber. A possibility for implementing curiosity and boredom in model-building neural controllers. In Proceedings of the First International Conference on Simulation of Adaptive Behavior on From Animals to Animats, pp. 222-227. Cambridge, MA, USA, 1990. MIT Press.
Özgür Şimşek and Andrew G Barto. Using relative novelty to identify useful temporal abstractions in reinforcement learning. In Proceedings of the Twenty-first International Conference on Machine Learning, ICML '04, pp. 95-, New York, NY, USA, 2004. ACM.
Bradly C. Stadie, Sergey Levine, and Pieter Abbeel. Incentivizing exploration in reinforcement learning with deep predictive models. CoRR, abs/1507.008144 2015.
Susanne Still, William Bialek, and Léon Bottou. Geometric clustering using the information bot- tleneck method. In S Thrun, L K Saul, and B Schölkopf (eds.), Advances in Neural Information Processing Systems 16, pp. 1165-1172. MIT Press, 2004.
Alexander I. Strehl and Michael Littman. An analysis of model-based interval estimation for markov decision processes. J. Comput. System Sci., 74(8):1309-133 1331, December 2008.
D J Strouse and David J Schwab. The information bottleneck and geometric clustering. December 2017.
D J Strouse, Max Kleiman- <leimen--Weiee, Josh Tenenbaum, Matt Botvinick, and David Schwab. Learning to share and hide intentions using information regularization. August 2018.
Yee Teh, Victor Bapst, Wojciech M Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. In Advances in Neural Information Processing Systems, pp. 4496 4506, 2017.
Georgios Theocharous and Sridhar Mahadevan. Learning the hierarchical structure of spatial environ- ments using multiresolution statistical models. In JEEE/RSJ Intl. Conf. on Intelligent Robots and Systems, 2002.
N. Tishby, F.C. Pereira, and W. Biale. The information bottleneck method. In The 37th annual Allerton Conf. on Communication, Control, and Computing, pp. 368-377, 1999. URL. http
// www. cs. www..ujjioo.oo. ac 1//pab//eeemiing/apprs///aaa//la
Noga Zaslavsky, Charles Kemp, Terry Regier, and Naftali Tishby. Efficient compression in color naming and its evolution. Proc. Natl. Acad. Sci. U. S. A., 115311)793777992, July 2018.
Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. 2010.
10

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1 ALGORITHM
present the algorithm summarizing our proposed approach DS-VIC- (1) unsupervised discovery of decision states and (2) transfer to goal-driven tasks.
Algorithm DS-VIC
Require: A parameterized encoder Po w, 8t). policy x(ae (.,8t)
Require: A parameterized option inference network qu | 80. 81)
Require: A parameterized goal-conditioned policy T. (a.[().)))
Require: A prior discrete options p(w) and integer H the length of each option trajectory.
Require: A variational approximation of the aption-marginalized encoder ((ze)
Require:
regularization weight / and
coefficient
Require: A set of training environments Perain (E) and transfer environments Peranster
Unsupervised Discovery
Sample training environment Prrain ~ Ptrain (E) for episodes = to max episodes do
Sample spawn location So p( Frrasa) and an option Q p(w)
Unroll state-action trajectory under [al 8t, z1) for H steps with reparametrized Za (=(2:18,,
Infer 92 from (u(w]se So, $1)
Update the parameters 0, v and & based on Eqn.
end for
Transfer to Goal-Driver Tasks
Sample transfer environment Eerenster Peranster for episodes to episodes do
Sample goal G ~ p(g) (rranster)
Unroll state-action trajectory under the goal-cond policy æ, (ax]ss,,
Update policy parameters ry to maximize the reward given by Equ. g end for
A.2 LEMMA PROOF
We state proof of the Lemma
2.1 stating that our proposed regularizer E I(22, A Ar|S,, So) forms an bound on empowerment (S22 S (S) from the main paper. This, combined with the lower-bound presented in VIC (Gregor et al. 2016 forms sandwich bound on ((2, Sy
Lemma 2.1 Let Ar be the action random variable timestep and state S, following an option- S Su) i.e. the conditional mutual information be-
conditioned policy N (a, w), Then, I(2, St tween the option 52 and action A, when summed over all timesteps in the trajectory, upper bounds the conditional mutual information I(Ω, Sy) between ? and the final state Sy namely the empowerment as defined by Gregor al. (2016):
( S (2,AA, So) = (,,
(9)
Proof. To begin, observe that the graphical model presented in Fig. 2 satisfies the markov chain + {s1, a1 M= < Sy (assuming every node conditioned on the intitial state So). Therefore, the pata-processing inequality (DPI) Cover & Thomas (1991) implies:
1(9, Sy1 S I(2, A5-)),)
(10)
Furthermore, using the chain rule of mutual information Cover Thomas (1911) ), we can write:
S(2,5)15%) 7(2, (S, -11 Ay-1)( ((2,55--155) 7(2, 1 (11)
11

---

Under review as a conference paper at ICLR 2020
Repeating the same set of steps recursively gives us: ( Sy) Σ (50(())))44/1)))))
DPI (Cover Thomas 1991
(12a)
-(9,,)-1(56) + A5-
Chain rule of MI
(12b)
[(22,5)-2(5) [(2,,
+ [[2, A Sy11,
Same as 12a- 12b
(12c)
(12d)
= ((2, +
122 A.)
(12e)
(12f)
Note that the graphical model presented in Fig. 2 implies that Q L Sn and hence,
(S)))))) H(2)) (S2) H(2) Solo5)
(13)
= Sy1,)))) - I(2, A,),,,))
(14)
A.3 UPPER BOUND ON (1,, So)
We explain the steps to derive Eqn. (4) in the main paper, as an upper bound on (Ar, SS, So). By the data processing inequality (Cover & Thomas, 1991) / (Ar, S2[S,. So) < [(2, S,S, So) for the graphical model in Fig 2 So we will next derive an upper bound on I(Z,, 52 So).
We write [S22, Se, Su = s), given a start state Sy as:
((,,(12, log
Enr-p(w) (s11 S.
(15)
p(Z)(S))
The key difference here is that our objective here uses the options Q internal to the agent as opposed to Goyal et al. (2019), who use external goal specifications g provided to the agent. Similar to VIC, pd here denotes the (unkown) state distribution at time / from which we draw samples when execute a policy.
We then assume variational approximation q(<, for p(ze(Se). and using the fact that DKL [(al)))))))) 2 O, we get a the following lower bound:
log ((Z) ((Zz)
T(2, Z, S. So = 8) 2
(16)
A.4 DECISION STATE DISCOVERY ON POLICY WITH OPTIONS.
We use Eqn. 16 to discover and visualize the decision states learned in an environment, augmented with random sampling of the start state Sg. Thus, we compute our decision states in an on-policy manner. Mathematically, we can write this as:
E Sompp(s) (4.1 log p(Z, ((2))
(17)
where So is random spawn location uniformly chosen from the set of states in the environment and Q is random option chosen at each of the spawn locations. Thus, for each state S, in the environment, we look at the aggregate of all the trajectories that pass through it and compute the values of log p(a.)e.3)) to identify / visualize decision states. In addition to being principled, this g(=x)
methodology also precisely captures our intuition that it is possible to identify decision states which are common to, or frequented across multiple options. Our results section 4 show that the identified decision states match our assessments of decision states corresponding to some structural regularities in the environment.
7 For experiments, fix g(2x) be unit gaussian, however it could also be learned.
12

---

discuss in the main Under review as a conference paper at ICLR 2020
A.5 IDENTIFICATION OF DECISION STATES DURING TRANSFER
As mentioned in the main paper, we would like to compute [[,, St So) to identify decision states and given (potentially) novel environment and a novel condicondt task, to provide it an exploration bonus. Given a state s', that we would like to compute [S2 Z,|S, = s', So). we can write:
Σ dz p(w)) (S: = 5'1s,, w w)p(= (w) log pla (w,, S = s')
(18)
((xe)
However, this cannot be computed in this form in transfer task, since goal driven agent is not following on-policy actions for an option Ω that would allow us to draw samples from p' (-[So, ?)) above (in order to do monte-carlo estimate of the integral above). Thus instead, we propose to compute the mutual information as follows:
Σ
f dæ, p(so: w|S = d1)p(z(s. s',w) =s w,90
log p(z (la,w, St ss)
(19)
((2))
Now, given state S, s', this requires us to draw samples from p(so, w|S, = s¹), which in general is intractable (since this requires us to know p" (s; (Q), which is not available in closed form). In order to compute the above equation, we make the assumption that p(su, W|S, s') = p(s))p(w). Breaking it down, this makes two assumptions: firstly, p( = s', 80) = p[ ([w])))) This means that all options have equal probability of passing through state s' at time which is not true in general. The second assumption is the same as VIC, namely that p(w)))) = p(w). Instead of making this assumption, could also fit parameterized variational approximation to p(w) SS 8', 80) and train it in conjunction with VIC. However, we found our simple scheme to work well in practice, and thus avoid fitting additional variational approximation.
A.6 DECISION STATES FROM LOCAL OBSERVATIONS
In the main paper, our results assumed that one had access to a good SLAM routine that did mapping and localization from partial observations (wo. -4)- In general, paper, this is not a bad assumption since our model is always trained on a single environment, and thus it is reasonable for to expect it to have sense of the (x. y) coordinates in its internal representation weights as training progresses. In this section, we detail some of the pathologies that we observed when kicking off training, if we took naive approach to SLAM, estimating function f(Luo.... Jz3) }) to directly regress to S2 from partial observations (Algorithm 1). We note that since our particular choice of environment is a discrete gridworld, several partial observations (especially for small agent view sizes like the 3x3 window) look the same to the agent and the agent has a tendency to converge to the most trivial optima for learning options one which learns to end at an easily obtainable partial observation given an option. For instance, while training DS-VIC one particular policy simply learned 4 options corresponding to the four cardinal directions the agent can face, and each can be achieved by left or right turns, without actually requiring any movement by the agent. Since the agent has compass which tells it the direction it is facing, it can ignore the partial observation and just use the direction vector to predict which of the 4 options was sampled.
A.7 BASELINES
We use an exploration bonus coefficient of ς 0.1 for the count-based exploration bonus baseline. The heuristic baseline identifies all occurrences of the following types of states in the Mom enviromments (1) corners of the room, (2) doorways, and gives them a slightly higher coefficient of exploration bonus than the i used for the count-based bonus. We ran sweep for the values of this higher coefficient and found that 0. (i.e. a +5% increase) gave best results.
A.8 IMPLEMENTATION DETAILS
Network Architecture: We use 3 layered convolutional neural network with kernels of size 3x3, 2x2 and 2x2 in the 3 layers respectively to process the agent's 's egocentric observation. We use ReLU the non-linear activation function after each convolutional layer. The output of the CNN is then concatenated with the agent's direction vector (compass) and the option (or goal encoding). The concatenated features are then passed through linear layer with hidden size 64 produce
13

---

Under review as a conference paper at ICLR 2020
the final features used by option encoder (1, w) head and the policy head (rl 2e). We use the (x, y) coordinates of the final state (embedded through a linear layer of hidden size 64) to regress to the option via qu (w| 8z). Furthermore, our parameterized policy is reactive one and the encoder pa (a(=r[st, w) is recurrent over the sequence of states encountered in the episode. The bottleneck random variable Z, is sampled from the parameterized gaussian po(2z) (=(==)),,w) and is made a differentiable stochastic node using the re-parmaterization trick for gaussian random variables.
Training Details: We use Advantage Actor-Critic (A2C) (open-sourced implementation by Kostrikoy (2018) for all our experiments. We use RMSprop as the optimizer for all our experiments. For the partially-obseraale
grid world settings, the agent receives an egocentric view of it's surroundings as input, encoded as an occupancy grid where the channel dimension specifies whether the agent or an obstacle is present at an (I, 3) location. We set the coefficient a in Eqn. to be 10-3 for all our experiments based on sweeps conducted across multiple values. In practice, found it to be difficult to optimize our unsupervised objective in absence of an entropy bonus- the parameterized policy collapses to deterministic one and no options are learned in addition inefficient exploration of the state-space. Morcover, we found that it was hard to obtain reasonable results by optimizing for both the terms in the objective from scratch and therefore, episodes (i.e., we set 3 = 0) after which we turn on the regularization term and let 3 grow linearly for another ~ 8k: episodes to get feasible outcomes at convergence. We experiment with a vocabulary 2, 4 and 32 options (imitating undercomplete and overcomplete settings) for all the environments. For all the exploration incentives presented in Table 1, we first picked a value of x. in Eqn. 7 (decides how to weigh the bonus with respect to the reward) from {10, 1, 0.1, 0.01, 0.001, 0.0001 based on the best sample-complexity on goal-driven transfer.
optimize the intrinsic objective itself for ~ 8k
InfoBot Implementation: Since code to reproduce InfoBot (Goyal al., 2019) was not publicit available, we implemented InfoBot ourselves while making sure that we are consistent with the architectural and hyper-parameter choices adopted by InfoBot (as per the A2C implementation by (Chevalier-Boisse
et al., 2018)). (i) We replace the convolution layers with fully connected layers to process the observations, and (ii) use the layer sizes as mentioned by (Goyal et al. 2019) in the appendix. The checkpoint used to report the final performance for N3S4 and N5S4 were picked by doing validation the transfer task based the success metric. However, this procedure was infeasible for the bigger N6S25 environment and we chose the last checkpoint from the training for this environment.
Convergence Criterion. In practice, we observe that it is hard learn a large number of discrete options using the unsupervised objective. From the entire option vocabulary, the agent only learns a few number of options discriminatively (as identified by the option termination state), with the rest collapsing into the same set of final states. To pick a suitable checkpoint for the transfer experiments, we pick the ones which have learned the maximum number of options reliably as measured by the likelihood of the correct option from the final state log (w) $y)-
Option Curriculum It is standard have fixed-sized discrete option space Ω with a uniform pior (Grregor et al. 2016). However, learning a meaningful option space with larger option vocabulary size [2] = K has been reported to be difficult Achiam al. (2018). We adopt curriculum based approach proposed in Achiam et al. (2018) where the option vocabulary size is gradually increased as the option decoder qu (w]ss)) becomes more confident in mapping back the final state back to the corresponding option sampled at the beginning of the episode. More concretely, whenever qu (w (5)) 0.75 (threshold chosen via hyperparameter tuning), the option vocabulary size increases according to
K min int( 1.5 x K + 1), K Kmaz
For our experiments, we start from K = 2 and terminate the curriculum when K K Kman = 32.
We run each of our experiments on single GTX Titan-X GPU, and use no more than 1 GB of GPU memory per experimental setting.
A.9 ADDITIONAL. RESULTS
we show further results on the MultiRoomN334 and MultiRoomN554 environ- ments to compare DS-VIC with other exploration incentives in terms of sample-complexity and the quality of the exploration incentive. As seen in the figures, DS-VIC is either competitive with or better than the presented baselines. As highlighted before, MultiRoomN334 and MuttiooomN5S4
In Fig. 7 and Fig. S
14

---

Under review as a conference paper at ICLR 2020
Method
MultiRoomNN34
MultiRoomNSS4
MuliiRoomNN66255
(Eucoder pal (t, pretrained on)
MultiRoomN256
MultRoommN266
MultiRoomN2S10
Goal-conditioned A2C
0%
0%
0%
TRPO + VIME
54%
0%
N/A
Curiosity- Exploration
95%
54%
N/A
Count-based Exploration Goyal al. (011
95%
N/A
InfoBor GnfoBorGGyal al. (2019)
90%
85%
N/A
InfoBot (Our Implementation)
99.6 99.6%u...2
08.9%
86.8 81.8%%22..
Count-based Exploration (Our Implementation)
99 0% 2%
71.4%%=22...
88, 4%
DIAYN-based Exploration Bonus (3 = 1)
799.% 2%
99.7%.1..2
0. 1...1%%
Randomly Initialized Encoder
100 100%
08.8% 7%
67-9 67-9%61..n
Heuristic Decision States
N/A
N/A
81.3%%:t..oo 0%
DS-VIC VIC (3 = 10-9)
99.9% 10..
09.1%%.....
90.9 30.9%=+..o 6%
DS-VIC (3 1)
100%
71. 71.1%
92.6%6.o.85
Table 2: Success rate (mean + standard error) of the goal-conditioned policy when trained with different exploration bonuses in addition to the extrinsic reward (l). We report results at ~5 10 timesteps for MultRoomN334, MultRRommN5S4 and at r~8.2 x 10" timesteps for MuldRoomN66255. We also report results from InfoBot (Goyal ct al. (019) for completeness. Note that for rooms of size 4 (MultiooomN354, M5S5))omM5S4), incentivizing to visit corners and doorways (Heuristic Decision States) is equivalent to count-based exploration.
are relatively simpler environments where simpler baselines like count based exploration incentives also perform quite well due to the small size of rooms.
In Table. 2 we report broader comparisons with other exploration baselines InfoBot (Goyal et 2019) compare with.
1.0-
Minnun Reoms Roseleed
Dumbooe
Succers
Datuma
1.5
Hoar ate tared
08-
(607))
0.4
Cara (dx1s")
02-
0.5
08-
00
Timestopa km
Terestppoo
Timestecs 1e
(a) Success Rate
(b) Avg. Task Returns (c) Farthest Room Reached (running best)
Figure 7: (a) Success Rate, (b) Avg. Task Returns, (c) farthest room reached so far, versus time-steps on
MultiRoomN334. Unsupervised pre-training to identify decision states was done on MultRoomm2266, Shaded areas represent standard error of the mean over 3 random seeds.
Uunizu Rooms Reached
Hidl Hoaktetaaod
S,00ets
08-
Detume
0.5
Fandomies
06-
044
Dur (1
02-
07-
Timesteps
Timestaps (1
Tinestegs (.7)
(b) Avg. Task Returns (c) Farthest Room Reached (running best) Figure 8: (a) Success Rate, (b) Avg. Task Returns, (c) farthest room reached so far, versus time-steps on
(a) Success Rate
MultiRoomNSS4. Unsupervised pre-training to identify decision states was done on MultRoommN266. Shaded areas represent standard error of the mean over 3 random seeds.
15


Reviews:
[{"review_id": "rkebgjk0Fr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an algorithm for discovering decision states in MDPs, in a task agnostic manner. The proposed method essentially generalizes the information bottleneck approach used in InfoBot, to an unsupervised setting. Whereas InfoBot recovers decision states in goal-conditioned policies by minimizing mutual information between goals and actions, the proposed approach (DS-VIC) does the same using implicit goals discovered by variational intrinsic control. Concretely, the authors propose adding a regularization term which constrains the mutual information between an episodic latent variable (having high mutual information to a final state) and each action along the (option conditional) trajectory. This objective is shown to be equivalent to a constraint optimization problem, which fits both a lower and upper bound on the VIC mutual information term. On MiniGrid environments, the approach is shown to yield somewhat interpretable decision states. In the footsteps of InfoBot, the authors then show that the resulting regularizer (the “latent-action” mutual information term) can serve as a useful auxiliary reward for transfer tasks with a hard exploration problem (transfer from goal navigation in small to large rooms).\n\nThe paper is interesting and provides some interesting theoretical results which adds to the body of work on variational intrinsic control, and unsupervised reinforcement learning. To the best of my knowledge, the derived upper-bound to the mutual information term used in VIC is novel and would be of interest to the community. The extension of InfoBot to the unsupervised regime, swapping extrinsic goals for inferred options is also intuitive and generalizes published work. \n\nThat being said, I do not think the paper is ready for publication at this point in time.\n\nOn the experimental side, the results are limited and not entirely convincing. Experiments are unfortunately limited to MiniGrid environments and MountainCar, whereas most recent work on empowerment (DIAYN, VALOR, etc.) has focused on more complex continuous control benchmarks. In addition, the experiments themselves are limited in scope and rely mostly on evaluating whether the mutual information term between goal and actions can be used to craft an auxiliary reward for downstream tasks, a task first derived in the InfoBot paper. Unfortunately, the results here are mixed, with the mutual information based reward statistically outperforming count-based bonus (which it builds on) only when the optimal hyper-parameter $\\beta$ (controlling the strength of the regularization term) is known. This somewhat breaks the narrative of unsupervised decision states. A much more compelling use case for the proposed regularization term would be improved data-efficiency on a downstreak task after pre-training with Eq. 6, following in the footsteps of DIAYN. Finally, as shown in the Appendix, the method seems rather brittle, requiring both a schedule on the size of the option layer and the strength of the regularization term, something which would be difficult to do in the absence of a downstream task.\n\nThere is also an important missing baseline which is glossed over. Instead of regularizing the mutual information between goals and actions $I(\\Omega, A_t \\mid S_t, S_0)$ one could simply encourage the low-level goal-conditioned policy to have high entropy. Indeed, one can show that  $KL[\\pi(a_t \\mid s_t, w_t) \\| \\pi_0(a_t)]$, with a learnt or fixed prior $\\pi_0$, is an upper-bound to $I({s_t, w_t} ; a_t)$: hence minimizing this KL (equivalent to maximizing entropy) would naturally prevent high mutual information between options and individual actions. As this term is already present in Eq. 6 it would be interesting to repeat the experiments, dropping the second term (minimality) but instead sweeping over the strength of the entropy regularization term $\\alpha$.\n\nWith respect to clarity, the paper could also be greatly improved. Decision states are never clearly defined to be those having high mutual information $I(\\Omega, A_t \\mid S_t)$. It is also not clear from the main text was is being plotted in Figure 3, requiring the reader to go through Appendix 4 to understand the visualization (without any references to this appendix in the main text). Similarly, I am not quite sure what to make of Figure 4 apart from the fact that different latent options yield different trajectories. See detailed comments below.\n\n\nDetailed Comments:\n(method)\n* It is rather disappointing that the reverse predictor uses privileged information, in the guise of x-y features. This represents quite a lot of prior knowledge about what we wish the options to encode. How does the method perform from the raw state?\n* Although mathematically elegant, I do not believe the sandwich bound explains why Eq. 6 helps uncover decision states. If we had an unbiased estimate of the mutual information between option and last state, then this would imply that an equality constraint on the VIC mutual information term would similarly yield decision states. This seems unlikely. An alternative hypothesis is that Eq. 6 works by injecting a soft prior, both via the temporal decomposition which aims to minimize $I(\\Omega, A_t |...)$ and its upper-bound $I(\\Omega, Z_t | …)$ which bottlenecks state information. Testing this theory could help strengthen the paper.\n* It would be nice to spell-out that the standard “reverse bound” employed by VIC cannot be used to estimate $I(\\omega, A_t \\mid S_t, S_0)$ as this would yield a lower-bound whereas we aim to minimize this term. I was almost tricked into thinking this was a simpler and valid strategy before realizing my mistake.\n* Footnote 7. High variance on $\\beta=1e-4$. Is it possible that too few seeds were used to estimate the standard error on the mean?\n\n(clarity)\n*“Decision states to be points where the cart has velocity=0”: wouldn’t this mostly be restricted to the initial state?\nWhat exactly was done for DIAYN in relation to Eq. 8? The text from S4-Baselines seems at odds with the caption.\n* “Upper bound is too tight”. I don’t think this is what you mean: tight would refer to how good the upper-bound approximation is, which is different from the constraint specifying an upper-bound which is too small.\n* Notation: Section 2.1 states that upper-case denotes random variables and lower-case denotes samples. Following this notation, equation should read e.g. $w \\sim p(\\Omega)$ and not $\\Omega \\sim p(w)$.\n* Notation: $p^J(s_f \\mid w, s_0)$. What does J refer to? J is not defined anywhere.\n* Section 4.1: Did not understand the sentence “we noticed that if an intersection is a decision state [...] having already made the decision.”\n* Section 4.1: Did not understand what is meant by “where trajectories associated with different options intersect”? What does this mean concretely in MountainCar for trajectories to intersect? Furthermore aren’t states with velocity=0 (mostly) restricted to the initial state?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "SkxAp2npYr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors introduce a novel decision point discovery method, wherein the VIC objective is constrained to minimize the amount of information between the option and the actions taken along the trajectory. After relaxing the constraint and introducing an upper bound to I(a; o), a tractable algorithm is produced. An implementation is then tested empirically on several partially observed grid worlds and a simple continuous control task on both qualitative bottleneck identification and quantitative benefits as an exploration bonus in a transfer learning setup.\n\nOverall I think the approach is well motivated and interesting, but the resulting implementation takes too many unmotivated modifications to make work, and the results aren't terribly convincing despite this; as such I currently vote for it's rejection. Specifically, the usage of privileged information (x,y coordinates in what is described as a partially observed domain) and the ad hoc choice of which networks had memory (i.e. an LSTM) don't fit the narrative that motivates the work. Constraining the empowerment should be thing that handles spurious diversity, so the need to use x,y coordinates is concerning.\n\nRegarding the empirical results, do all of the baseline make similar use of domain knowledge / privileged information? For example, does your implementation of DIAYN utilize x,y coordinates in the option predictor? Is the Beta=0 case considered? It isn't mentioned, but perhaps it amounts to one of your other baselines?\n\nThe empirical evidence isn't terribly convincing. On two of the three exploration setups, the random network is as performant, and does need a Beta hyper-parameter to tune. Though, to be fair, the connection between decision state identification and a good count-based exploration bonus is loose. The qualitative results are also a bit lacking. I was expecting the doorways to \"pop out\" more; the relatively muddled decision state activations made me wonder if they were really better than DIAYN's.\n\nThis work would really benefit from a quantitative measure of decision state identification accuracy. Some prior work (e.g. \"Grounding Subgoals in Information Transitions\") were able to do this by choosing environments where the quantities of interest were tractable to calculate exactly. This would at least allow us to see if the discovered decision states correspond to those that are optimal under your metric.\n\nRebuttal EDIT:\nThank you for the thoughtful rebuttal. If this were an option, I'd raise my score to a 5. But as my vote is to 'revise and resubmit' (unfortunately translated to 'reject' as per the conference system), I'll leave it in the 'reject' score bucket.\n\nYour rebuttal lessened my concerns about the using (x,y) and only using an LSTM for the policy. I agree these are largely orthogonal issues, and since they were consistent with their baselines, that is fine.\n\nHowever, the response to the unintuitive nature of the \"decision states\" is less convincing. If all you care about is the downstream task performance, why even show the qualitative results or impose the semantics of \"decision states\" on the learned representations? The sandwich bound is novel in and of itself; I understand the need to relate to prior work, but I actually think dropping the language around \"decision states\" (maybe outside of the algorithm's motivation) and talking purely in information theoretic terms would improve the paper.\n\nYour response to [R3] on the setting of Beta hyper-parameter seems to not be supported by the results. You claim that values work well across multiple tasks, but the best reported value for the \"hard\" task (1e-2) is worse than the random baseline on the \"easy\" tasks.\n\nPerhaps only the \"hard\" task matters and the \"easy\" tasks are only of significance due to their usage in InfoBot. But I'd argue that unless your method is dominating existing methods on both without changing hyper-parameters, switching to a more complex (and commonly used) benchmark would be more convincing.  The Atari Suite or the control tasks used in related work (e.g. DIAYN) would be my suggestion.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJxf_1_9nr", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a mechanism for identifying decision states, even on previously unseen tasks. Decision states are states from which the option taken has high mutual information with the final state of that option, but low mutual information with the action at a time-step, given the current state. An intrinsic reward based on an upper bound of the relevant mutual information speeds up learning in similar environments that the agent has not encountered. \n\nA key contribution of this work is extending the notion of goal-driven decision states to goal-independent decision states. The authors also introduce an interesting upper bound on the mutual information between options and final states.\n\nThe authors provide an empirical evaluation that supports their central claims. \n\nI recommend this paper be accepted because it contributes an interesting theoretical result, a definition of decision state that does not depend on extrinsic rewards, and an algorithm to find such decision states.\n\nFurther suggestions to improve clarity that did not influence the decision:\n* The acronym VIC is used frequently throughout the introduction, but is not explained until section 2. Please introduce variational intrinsic control in the introduction. \n* The partial observability claim is not substantiated by the experiments. From the general response to reviewers, \"Therefore, we make the assumption that the complete state is available, in order to study unsupervised decision states.\" It is not a problem to assume that the complete state is available, but claiming to generalize to partial observability is not entirely correct, even if your method handles the same semi-partially observable case as previous work like VIC or DIAYN.\n* Please explicitly describe the motivation for using a bottleneck variable. \n* In MDPs with fixed episode length, the probability of termination is only non-zero on the last time step. Therefore, information about the time step must be included in a Markov state. The options in this paper terminate based on a time horizon, but there is no mention of whether the intra-option time step is included in the state or bottleneck variables.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJgGuIGTKH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper proposes an unsupervised method for discovering \"decision states\", defined as states where decisions affect the future states an agent can reach in the environment, based on the variational intrinsic control (VIC) framework that maximizes an agent's empowerment. This paper draws connections to many prior works such as VIC, diversity is all you need (DIAYN), and InfoBot and shows results on MiniGrid and MountainCar. \n\nMain Comments:\n\nWhile this is an interesting paper, I did not find the experimental section to be convincing enough for publication at this stage. Moreover, I am concerned by the novelty of the proposed approach, which seems very similar to InfoBot, the main difference between them being the replacement of the goals with options thus moving towards less supervision / use of prior-knowledge. However, if I understand correctly, this method still requires to specify a prior over the options, so it is not clear why DS-VIC would be preferable to InfoBot. If the empirical results showed a more robust and significant gain in performance on more diverse or complex tasks, I would be willing to reconsider my judgement regarding the significance of this work. \n\nMinor Questions / Comments:\n\n1. How do you define the final state S_f? Do you only consider the episodic RL setting? Do you consider S_f to always be after a fixed number of steps or whenever the termination function is triggered?\n\n2. Please include more information about what is represented in Figure 3 and the color scale. I am slightly confused by the interpretation of that plot because (1) it seems like the model does not detect \"all decision states\" (e.g. intersections) . that a human may consider while including others (e.g. corners, for which I do not agree that the agent should be incentivized to go even after learning from the reward structure that there isn't much to gain), (2) why is it that the for example the top-left figure has a rather nonuniform distribution across the rooms (is it influenced by the initial position of the agent?) and (3) the model doesn't seem to be very consistent about what it considers a to be a \"decision state\".\n\n3. Can you show similar plots for the MultiRoom environment? Perhaps those will shed more light into the learned behavior. \n\n4. Including all possible ablations to the objective in equation 6 would be helpful to tease apart the contribution of each term: variational control, information bottleneck, and entropy.\n\n5.  The results in Table 1 and Figures 6 do not show a significant gain in performance. Moreover, I suspect the other methods will converge to similar values soon after 8M steps. For a fair comparison, it would be good to show how the curves look after all (or at least . more of the models) converge. From Table 1, it actually seems to me that InfoBot encounters less penalty across all 3 models, even tho DS-VIC overperforms on the more challenging one. Moreover, in all 3 cases, at least one of the other methods seems to at least be close to the performance of DS-VIC so I am concerned that these may not be very challenging tasks for well-tuned SOTA methods. Plus, the comparison does not seem fair given that the the numbers are reported before the baselines converged. \n \n6. Did you pretrain the baselines for the same number of steps as DS-VIC? Please include more details about this stage and how you ensure the comparison was fair.\n\n7. It might be useful to include other baselines such as the curiosity-based exploration method from Pathak et al, 2017 . or universal value functions (Schaul et al. 2015)\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work is interesting because it's aim is to push the work in intrinsic motivation towards crisp definitions, and thus reads like an algorithmic paper rather than yet another reward heuristic and system building paper. There is some nice theory here, integration with options, and clear connections to existing work.

However, the paper is not ready for publication. There were were several issues that could not be resolved in the reviewers minds (even after the author response and extensive discussion). The primary issues were: (1) There was significant confusion around the beta sensitivity---figs 6,7,8 appear misleading or at least contradictory to the message of the paper. (2) The need for x,y env states. (3) The several reviewers found the decision states unintuitive and confused the quantitative analysis focus if they given the authors primary focus is transfer performance. (4) All reviewers found the experiments lacking. Overall, the results generally don't support the claims of the paper, and there are too many missing details and odd empirical choices.  

Again, there was extensive discussion because all agreed this is an interesting line of work. Taking the reviewers excellent suggestions on board will almost certainly result in an excellent paper. Keep going!
Decision: Reject