Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: 
First, please refrain from making up confusing and obfuscating terminology for concepts that are well defined using standard definitions. For example, equation 1 and the preceding equation need to be discarded. The correct and standard terminology to use here is that R(s1, a1, ....) is known as the discounted sum of rewards, or the return. It is confusingly termed here as the "reward". If R is the reward, what is r(s_t,a_t)? Also, J(\pi) is confusingly NOT the expected reward, but the expected sum of rewards (the little r's, not the big R). The little "r" is also called "reward" and confuses your definition of R and r! Why make up terms on your own when the literature has standard terms, namely the value function, the return, and the discounted sum of reward etc.? (see Puterman, MDP textbook, or Sutton/Barto RL text, or Bertsekas/Tsitsiklis, NDP book). 

Second, please cite and discuss in detail the work of Hung Bui et al., (Journal of AI Research) on the abstract hidden Markov model (or AHMM). It is *exactly* equivalent to your BONN architecture. How does your work differ from AHMM and please provide a detailed comparison, both experimentally and theoretically. There is also prior work on learning AHMMs, see Johns et al., AAAI 2005. 
review_title: Standardize terminology and relate your work to abstract HMMs

final_score: rating: null
review_content: No questions
review_title: No questions

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes an approach to learning hierarchical actions or options in reinforcement learning using the so-called BONN model (for budgeted options with neural networks). The approach is an interesting mixture of the old and the new. Some ideas seem very related to previous work in the literature, such as variants of hidden Markov models proposed by Hung Bui and others (abstract HMM, hierarchical POMDP by Theocharous et al., IROS 2005; Murphy et al., NIPS, ICRA). 

The major difference is that unlike the prior work using a graphical model, this paper uses a gated recurrent network neural model to implement the learning of options from data. The approach is based on minimizing some quantity called the "cognitive effort", but this is confusingly explained, and not very precise. The basic idea here is to define a budget that modified the immediate reward, and so its minimization is viewed as minimizing cognitive effort. The approach seems a bit ad hoc. 

Experiments are reported on a variety of simple discrete and continuous control benchmark domains. 
review_title: Learning options in reinforcement learning 

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper tackles the very important problem of learning options from data. The introduction of the budget constraint is an interesting twist on this problem, which I had not seen before (though other methods apply other constraints.)

I must say I’m not very convinced by the need to introduce the Bi-POMDP framework, where the conventional POMDP framework would do.  In discussions, the authors suggest this makes for simpler comparison with RL models, but I find that it rather obscures the link to POMDP models.

The proposed method makes an interesting contribution, distinct from the existing literature as far as I know.  The extension to discover a discrete set of options is a nice feature for practical applications.

In terms of the algorithm itself, I am actually unclear about lines 4 & 6.  At line 4, I don’t know how \sigma_t is computed. Can you give the precise equation?  At line 6, I don’t know how the new option o_t is generated. Again, can you give the precise procedure?

The paper contains several empirical results, on contrasting simulated domains. For some of these domains, such as CartPole, it’s really not clear that options are necessary. In my mind, the lack of comparison to other options learning methods is a limitation of the current draft.

review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: In this paper, the authors study the problem of discovering options for reinforcement learning. They introduce the Bi-POMDP model, which is a POMDP where the observations are structured as a pair of elements, with the first element only available to the option-choosing component (the "option level") and the second element only to the action-choosing component (the "action level") and the termination component (the "acquisition model"). They also detail the BONN learning model, which consists of three artificial neural network that implement these three components. Finally, they suggest optimizing a tradeoff between the value achieved by the model and the cost of switching between options (the "cognitive effort"), and demonstrate this approach in three simple domains: Cart-Pole, Lunar Lander and two variants of a grid-world maze.

The paper is interesting, and adds considerably to the increasing body of research in hierarchical reinforcement learning (HRL). I found no critical flaws in the paper, but also no high-impact insights or impressive improvements. This paper offers some good ideas that are moderately novel and may advance the field, but has some issues.


The first issue is that it is unclear how much easier it is to compose Bi-POMDPs than hand-crafted options or subgoals. If Bi-POMDPs are to alleviate the design costs of using HRL with human-defined structure, one needs to show that the splitting of observations into two elements (x, y) is easier to do well enough.

For example, in Section 5 the authors are correct in pointing out that sequences of actions ("macro-actions") are open-loop, and therefore not as expressive as closed-loop options. However, by setting x empty in all but one experiment, the authors also restrict themselves to open-loop sub-policies, albeit stochastic ones. Such sub-policies may be sufficiently expressive for the simple domains in this paper, but this is unlikely so in more realistic domains.

In the MAZE_2 domain, x is the agent's position relative to the current room. Designing this domain-specific observation model requires domain knowledge, arguably no less than designing relevant subgoals. It is hard to judge the effectiveness of this approach without design principles for these domain-specific observation models, and more realistic experiments to evaluate their quality.

Finally, it is revealing that the drive for hierarchy is only achieved by limiting x. In Section 3.3 the authors mention that acquisition of y is "crucial for discovering a good policy: an agent only using the observations x_t would be unable to solve the task", which suggests that the design choice of x directly impacts one side of the trade-off between the value and the cognitive effort. Such an important factor should be addressed explicitly.

The authors may be interested in the paper "Learning and Transfer of Modulated Locomotor Controllers" (Heess et al., 2016), which has a similar split observation model, and suffers from the same issue.


The second issue is that the option space seems to be expressive enough to represent y with high fidelity. If this is the case, and if the learned option model indeed maintains a good image of y, then the algorithm is really solving a different problem: reinforcement learning with costly observability of y. This means that it learns how to act given a stale value of y and when to refresh it. It should then be framed accordingly and compared with the relevant literature. It may or may not be as interesting or novel.

In contrast, the standard options framework calls for compression of y into the choice of option o (usually in a small finite space). The agent should learn to extract subtask-relevant information from y, in a way that generalizes to unseen states or subtasks.

That said, the embedding of y in the option space can be interesting in itself, even if it is lossless (1-to-1). Unfortunately, no such analysis was offered by the authors.


Minor issues:

- In 3.3: error in citation, (?) appears instead.

- In 4.1: it is confusing to say that "the environments are more stochastic" when epsilon is increased. It is the agent's policy, not the environment, that becomes more stochastic, which is useful for exploration. If this hurts performance, some discussion is needed of why too much exploration is detrimental to learning. In particular, the paper does not make explicit the number of iterations in the experiments, and it is not clear whether learning with larger epsilon is worse after some fixed number of iterations or asymptotically.

review_title: Interesting paper, some issues

final_score: rating: null
review_content: Why do you state that Bi-POMDP are more general than POMDP?  It seems every problem you describe for the Bi-POMDP case can be equally well modelled in a standard POMDP, introducing an action for choosing the sense y.

Similarly, the augmented reward (Eqn 3) seems like it can be re-written as a standard POMDP reward function, where there is a cost to applying the “observe y” action.

Why is y in Fig.1 modelled as a latent variable?  This seems directly related to \sigma.

Can you add precise specification of lines 6 & 6 in Algorithm 2?

Did you perform empirical comparison to any other option learning method, e.g. Bacon & Precup?  Also, any comparison to other method that doesn’t use option, e.g. standard DQN?

How much data is necessary to fit the parameter \lambda?

What is the year and publisher for Bacon & Precup?
review_title: Questions

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper presents an approach to constructing hierarchical RL representations which relies on assuming agents that need to spend cognitive effort in order to choose their actions.  The paper p[roposes a specific way of formulating option construction via what they call a "Bi-POMDP". This idea is potentially very interesting, plausible form a cognitive science point of view, and definitely deserves attention. However, there are some problems which do not make the paper acceptable in its current form. I am listing them here in order of importance.
1. It is not clear from the description why a Bi-POMDP is not a POMDP. POMDPs allow for vector-based observations.  Suppose the observation vector is (x_t, \sigma_t * y_t). This seems like it would result in a POMDP which is identical to the proposed model. The paper should include an example of a Bi-POMDP which is *not* a POMDP, or be revised to use specific POMDP terminology (see eg the use of augmented MDPs in hierarchical RL, which *are MDPs* but do not work in the original state space)
2. The paper make some specific assumptions about the abstractions (eg determinism in certain places). It is not clear why these are needed at all. Similarly, there are some very specific assumptions regarding the form of the approximations used (Relu, GRUs etc).  Are these necessary? In principle one could implement the ideas in the paper with other, simpler architectures. Was this the first set of choices, or was it arrived at after some experimentation? It is important to understand how much of the performance achieved is due to the specific (fairly powerful) architectures and what one could get through simpler means (eg, feedforward nets)
3. The paper seems quite similar in spirit to Bacon & Precup, 2015b; in fact, it seems that the use of a value function or model that they discuss is a way to provide a y_t. However, there is no direct comparison to that approach. Since it is very related, it would be useful to perform some of those same experiments. Also, their paper works entirely in the MDP, not POMDP framework, so some clarification is needed here regarding the use of POMDPs instead.
4. The choice of domains is somewhat limited to simple tasks, while some of the recent approaches in hierarchical RL use more complex domains (Atari, Minecraft etc). Ideally, the experiments should be extended to some of these more complex tasks.
5. What are the theoretical properties of the proposed approach? Eg, is the proposed algorithm convergent? If Bi-POMDP is a POMDP, then one should be able to leverage POMDP results to build some theory here. If it is not a POMDP, then we need some understanding of how easy/hard a Bi-POMDP is to solve
6. The paper contains many grammar problems and some broken references, and should be proof-read thoroughly 
In summary, while the proposed approach is quite interesting and definitely worth exploring, the paper is not ready for publication in its current form.
review_title: Interesting ideas, but the paper requires more work

Meta-review: Most of the reviewers agreed that the proposed budgeted options framework was interesting, but there were a number of serious concerns raised about the work. Many of the reviewers found the assumptions of the approach to be somewhat odd, and while the particular formulation in the paper was generally assessed as novel, it has connections to a number of previous works that were not explored in detail. Finally, the experimental evaluation is conducted on simple tasks with few comparisons, so it is very difficult to make concrete conclusions about how well the method works.
Decision: Reject