Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes a mechanism for identifying decision states, even on previously unseen tasks. Decision states are states from which the option taken has high mutual information with the final state of that option, but low mutual information with the action at a time-step, given the current state. An intrinsic reward based on an upper bound of the relevant mutual information speeds up learning in similar environments that the agent has not encountered. 

A key contribution of this work is extending the notion of goal-driven decision states to goal-independent decision states. The authors also introduce an interesting upper bound on the mutual information between options and final states.

The authors provide an empirical evaluation that supports their central claims. 

I recommend this paper be accepted because it contributes an interesting theoretical result, a definition of decision state that does not depend on extrinsic rewards, and an algorithm to find such decision states.

Further suggestions to improve clarity that did not influence the decision:
* The acronym VIC is used frequently throughout the introduction, but is not explained until section 2. Please introduce variational intrinsic control in the introduction. 
* The partial observability claim is not substantiated by the experiments. From the general response to reviewers, "Therefore, we make the assumption that the complete state is available, in order to study unsupervised decision states." It is not a problem to assume that the complete state is available, but claiming to generalize to partial observability is not entirely correct, even if your method handles the same semi-partially observable case as previous work like VIC or DIAYN.
* Please explicitly describe the motivation for using a bottleneck variable. 
* In MDPs with fixed episode length, the probability of termination is only non-zero on the last time step. Therefore, information about the time step must be included in a Markov state. The options in this paper terminate based on a time horizon, but there is no mention of whether the intra-option time step is included in the state or bottleneck variables.
review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: This paper proposes an algorithm for discovering decision states in MDPs, in a task agnostic manner. The proposed method essentially generalizes the information bottleneck approach used in InfoBot, to an unsupervised setting. Whereas InfoBot recovers decision states in goal-conditioned policies by minimizing mutual information between goals and actions, the proposed approach (DS-VIC) does the same using implicit goals discovered by variational intrinsic control. Concretely, the authors propose adding a regularization term which constrains the mutual information between an episodic latent variable (having high mutual information to a final state) and each action along the (option conditional) trajectory. This objective is shown to be equivalent to a constraint optimization problem, which fits both a lower and upper bound on the VIC mutual information term. On MiniGrid environments, the approach is shown to yield somewhat interpretable decision states. In the footsteps of InfoBot, the authors then show that the resulting regularizer (the “latent-action” mutual information term) can serve as a useful auxiliary reward for transfer tasks with a hard exploration problem (transfer from goal navigation in small to large rooms).

The paper is interesting and provides some interesting theoretical results which adds to the body of work on variational intrinsic control, and unsupervised reinforcement learning. To the best of my knowledge, the derived upper-bound to the mutual information term used in VIC is novel and would be of interest to the community. The extension of InfoBot to the unsupervised regime, swapping extrinsic goals for inferred options is also intuitive and generalizes published work. 

That being said, I do not think the paper is ready for publication at this point in time.

On the experimental side, the results are limited and not entirely convincing. Experiments are unfortunately limited to MiniGrid environments and MountainCar, whereas most recent work on empowerment (DIAYN, VALOR, etc.) has focused on more complex continuous control benchmarks. In addition, the experiments themselves are limited in scope and rely mostly on evaluating whether the mutual information term between goal and actions can be used to craft an auxiliary reward for downstream tasks, a task first derived in the InfoBot paper. Unfortunately, the results here are mixed, with the mutual information based reward statistically outperforming count-based bonus (which it builds on) only when the optimal hyper-parameter $\beta$ (controlling the strength of the regularization term) is known. This somewhat breaks the narrative of unsupervised decision states. A much more compelling use case for the proposed regularization term would be improved data-efficiency on a downstreak task after pre-training with Eq. 6, following in the footsteps of DIAYN. Finally, as shown in the Appendix, the method seems rather brittle, requiring both a schedule on the size of the option layer and the strength of the regularization term, something which would be difficult to do in the absence of a downstream task.

There is also an important missing baseline which is glossed over. Instead of regularizing the mutual information between goals and actions $I(\Omega, A_t \mid S_t, S_0)$ one could simply encourage the low-level goal-conditioned policy to have high entropy. Indeed, one can show that  $KL[\pi(a_t \mid s_t, w_t) \| \pi_0(a_t)]$, with a learnt or fixed prior $\pi_0$, is an upper-bound to $I({s_t, w_t} ; a_t)$: hence minimizing this KL (equivalent to maximizing entropy) would naturally prevent high mutual information between options and individual actions. As this term is already present in Eq. 6 it would be interesting to repeat the experiments, dropping the second term (minimality) but instead sweeping over the strength of the entropy regularization term $\alpha$.

With respect to clarity, the paper could also be greatly improved. Decision states are never clearly defined to be those having high mutual information $I(\Omega, A_t \mid S_t)$. It is also not clear from the main text was is being plotted in Figure 3, requiring the reader to go through Appendix 4 to understand the visualization (without any references to this appendix in the main text). Similarly, I am not quite sure what to make of Figure 4 apart from the fact that different latent options yield different trajectories. See detailed comments below.


Detailed Comments:
(method)
* It is rather disappointing that the reverse predictor uses privileged information, in the guise of x-y features. This represents quite a lot of prior knowledge about what we wish the options to encode. How does the method perform from the raw state?
* Although mathematically elegant, I do not believe the sandwich bound explains why Eq. 6 helps uncover decision states. If we had an unbiased estimate of the mutual information between option and last state, then this would imply that an equality constraint on the VIC mutual information term would similarly yield decision states. This seems unlikely. An alternative hypothesis is that Eq. 6 works by injecting a soft prior, both via the temporal decomposition which aims to minimize $I(\Omega, A_t |...)$ and its upper-bound $I(\Omega, Z_t | …)$ which bottlenecks state information. Testing this theory could help strengthen the paper.
* It would be nice to spell-out that the standard “reverse bound” employed by VIC cannot be used to estimate $I(\omega, A_t \mid S_t, S_0)$ as this would yield a lower-bound whereas we aim to minimize this term. I was almost tricked into thinking this was a simpler and valid strategy before realizing my mistake.
* Footnote 7. High variance on $\beta=1e-4$. Is it possible that too few seeds were used to estimate the standard error on the mean?

(clarity)
*“Decision states to be points where the cart has velocity=0”: wouldn’t this mostly be restricted to the initial state?
What exactly was done for DIAYN in relation to Eq. 8? The text from S4-Baselines seems at odds with the caption.
* “Upper bound is too tight”. I don’t think this is what you mean: tight would refer to how good the upper-bound approximation is, which is different from the constraint specifying an upper-bound which is too small.
* Notation: Section 2.1 states that upper-case denotes random variables and lower-case denotes samples. Following this notation, equation should read e.g. $w \sim p(\Omega)$ and not $\Omega \sim p(w)$.
* Notation: $p^J(s_f \mid w, s_0)$. What does J refer to? J is not defined anywhere.
* Section 4.1: Did not understand the sentence “we noticed that if an intersection is a decision state [...] having already made the decision.”
* Section 4.1: Did not understand what is meant by “where trajectories associated with different options intersect”? What does this mean concretely in MountainCar for trajectories to intersect? Furthermore aren’t states with velocity=0 (mostly) restricted to the initial state?
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The authors introduce a novel decision point discovery method, wherein the VIC objective is constrained to minimize the amount of information between the option and the actions taken along the trajectory. After relaxing the constraint and introducing an upper bound to I(a; o), a tractable algorithm is produced. An implementation is then tested empirically on several partially observed grid worlds and a simple continuous control task on both qualitative bottleneck identification and quantitative benefits as an exploration bonus in a transfer learning setup.

Overall I think the approach is well motivated and interesting, but the resulting implementation takes too many unmotivated modifications to make work, and the results aren't terribly convincing despite this; as such I currently vote for it's rejection. Specifically, the usage of privileged information (x,y coordinates in what is described as a partially observed domain) and the ad hoc choice of which networks had memory (i.e. an LSTM) don't fit the narrative that motivates the work. Constraining the empowerment should be thing that handles spurious diversity, so the need to use x,y coordinates is concerning.

Regarding the empirical results, do all of the baseline make similar use of domain knowledge / privileged information? For example, does your implementation of DIAYN utilize x,y coordinates in the option predictor? Is the Beta=0 case considered? It isn't mentioned, but perhaps it amounts to one of your other baselines?

The empirical evidence isn't terribly convincing. On two of the three exploration setups, the random network is as performant, and does need a Beta hyper-parameter to tune. Though, to be fair, the connection between decision state identification and a good count-based exploration bonus is loose. The qualitative results are also a bit lacking. I was expecting the doorways to "pop out" more; the relatively muddled decision state activations made me wonder if they were really better than DIAYN's.

This work would really benefit from a quantitative measure of decision state identification accuracy. Some prior work (e.g. "Grounding Subgoals in Information Transitions") were able to do this by choosing environments where the quantities of interest were tractable to calculate exactly. This would at least allow us to see if the discovered decision states correspond to those that are optimal under your metric.

Rebuttal EDIT:
Thank you for the thoughtful rebuttal. If this were an option, I'd raise my score to a 5. But as my vote is to 'revise and resubmit' (unfortunately translated to 'reject' as per the conference system), I'll leave it in the 'reject' score bucket.

Your rebuttal lessened my concerns about the using (x,y) and only using an LSTM for the policy. I agree these are largely orthogonal issues, and since they were consistent with their baselines, that is fine.

However, the response to the unintuitive nature of the "decision states" is less convincing. If all you care about is the downstream task performance, why even show the qualitative results or impose the semantics of "decision states" on the learned representations? The sandwich bound is novel in and of itself; I understand the need to relate to prior work, but I actually think dropping the language around "decision states" (maybe outside of the algorithm's motivation) and talking purely in information theoretic terms would improve the paper.

Your response to [R3] on the setting of Beta hyper-parameter seems to not be supported by the results. You claim that values work well across multiple tasks, but the best reported value for the "hard" task (1e-2) is worse than the random baseline on the "easy" tasks.

Perhaps only the "hard" task matters and the "easy" tasks are only of significance due to their usage in InfoBot. But I'd argue that unless your method is dominating existing methods on both without changing hyper-parameters, switching to a more complex (and commonly used) benchmark would be more convincing.  The Atari Suite or the control tasks used in related work (e.g. DIAYN) would be my suggestion.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: Summary:

This paper proposes an unsupervised method for discovering "decision states", defined as states where decisions affect the future states an agent can reach in the environment, based on the variational intrinsic control (VIC) framework that maximizes an agent's empowerment. This paper draws connections to many prior works such as VIC, diversity is all you need (DIAYN), and InfoBot and shows results on MiniGrid and MountainCar. 

Main Comments:

While this is an interesting paper, I did not find the experimental section to be convincing enough for publication at this stage. Moreover, I am concerned by the novelty of the proposed approach, which seems very similar to InfoBot, the main difference between them being the replacement of the goals with options thus moving towards less supervision / use of prior-knowledge. However, if I understand correctly, this method still requires to specify a prior over the options, so it is not clear why DS-VIC would be preferable to InfoBot. If the empirical results showed a more robust and significant gain in performance on more diverse or complex tasks, I would be willing to reconsider my judgement regarding the significance of this work. 

Minor Questions / Comments:

1. How do you define the final state S_f? Do you only consider the episodic RL setting? Do you consider S_f to always be after a fixed number of steps or whenever the termination function is triggered?

2. Please include more information about what is represented in Figure 3 and the color scale. I am slightly confused by the interpretation of that plot because (1) it seems like the model does not detect "all decision states" (e.g. intersections) . that a human may consider while including others (e.g. corners, for which I do not agree that the agent should be incentivized to go even after learning from the reward structure that there isn't much to gain), (2) why is it that the for example the top-left figure has a rather nonuniform distribution across the rooms (is it influenced by the initial position of the agent?) and (3) the model doesn't seem to be very consistent about what it considers a to be a "decision state".

3. Can you show similar plots for the MultiRoom environment? Perhaps those will shed more light into the learned behavior. 

4. Including all possible ablations to the objective in equation 6 would be helpful to tease apart the contribution of each term: variational control, information bottleneck, and entropy.

5.  The results in Table 1 and Figures 6 do not show a significant gain in performance. Moreover, I suspect the other methods will converge to similar values soon after 8M steps. For a fair comparison, it would be good to show how the curves look after all (or at least . more of the models) converge. From Table 1, it actually seems to me that InfoBot encounters less penalty across all 3 models, even tho DS-VIC overperforms on the more challenging one. Moreover, in all 3 cases, at least one of the other methods seems to at least be close to the performance of DS-VIC so I am concerned that these may not be very challenging tasks for well-tuned SOTA methods. Plus, the comparison does not seem fair given that the the numbers are reported before the baselines converged. 
 
6. Did you pretrain the baselines for the same number of steps as DS-VIC? Please include more details about this stage and how you ensure the comparison was fair.

7. It might be useful to include other baselines such as the curiosity-based exploration method from Pathak et al, 2017 . or universal value functions (Schaul et al. 2015)

review_title: Official Blind Review #1

Meta-review: This work is interesting because it's aim is to push the work in intrinsic motivation towards crisp definitions, and thus reads like an algorithmic paper rather than yet another reward heuristic and system building paper. There is some nice theory here, integration with options, and clear connections to existing work.

However, the paper is not ready for publication. There were were several issues that could not be resolved in the reviewers minds (even after the author response and extensive discussion). The primary issues were: (1) There was significant confusion around the beta sensitivity---figs 6,7,8 appear misleading or at least contradictory to the message of the paper. (2) The need for x,y env states. (3) The several reviewers found the decision states unintuitive and confused the quantitative analysis focus if they given the authors primary focus is transfer performance. (4) All reviewers found the experiments lacking. Overall, the results generally don't support the claims of the paper, and there are too many missing details and odd empirical choices.  

Again, there was extensive discussion because all agreed this is an interesting line of work. Taking the reviewers excellent suggestions on board will almost certainly result in an excellent paper. Keep going!
Decision: Reject