Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: Summary:
The authors propose to learn reusable options to make use of prior information and claim to do so with minimal information from the user (such as # of options needed to solve the task, which options etc). The claim is that the agent is first able to learn a near-optimal policy for a small # of problems and then is able to solve a large # of tasks by such a learned policy.  The authors build on the idea that minimizing the number of decisions made by the agent results in discovering reusable options. The options are learned offline by learning to solve a small number of tasks. Their algorithm introduces one option at a time until introducing a new option doesn’t improve the objective further. The ideas are interesting, However, the paper as it stands is lacking in thorough evaluation.

Detailed comments:
The proposed approach offers two key contributions:
-an objective function to minimize the decision states
-incrementally constructing an option set that can be reused later, without the a priori specification of the # of options needed. 

The introduction is well written, however, given the intuitions behind the objective function; in some sense, the idea here is to minimize the decisions or terminations intuitively relates to terminating only at critical or bottleneck states. It would be useful to provide such motivation in the introduction. 

Intuitively the objective criterion is interesting. With a cursory look at the proofs, they seem fine, although I have to admit I have not looked in detail into the proofs. 

Paper writing could be significantly improved. Several points are not clear and need further clarification:
-The term near-optimal is mentioned several times, but it is not clear the policies are near-optimal with respect to what? The task or a set of tasks? 
-How does the proposed approach ensures that they are near-optimal? Please clarify.
-“We can obtain the estimate for equation 1 by averaging over a set of near-optimal trajectories” The aim as states is to learn options that are capable of generating near-optimal trajectories (by using a small # of terminations). The authors then say that “given a set of options, a policy over options, a near-optimal sample trajectory, we can calculate..” Where does the near-optimal sample trajectory come from? Please provide clarifications.

In experiments: FR rooms experiments are interesting, in the visualization of the option policies, do the figures here show the flattened policy of the options? What do the termination functions look like? 

Atari experiments are limited in nature in that they show only two games. Moreover, It is a bit confusing as to what is multi-task in the ATARI experiments. The authors mention the training of options and then talk about the results in the plots (4) show the training curves. However, they do not mention what are “novel tasks for Breakout/Amidar” in this context. 

Considering the proposed approach is closely related to the idea of selective terminations of options, it is natural to expect a comparison with Harb, 2018 and Hartyuanm 2019. The work could benefit by comparing with the aforementioned baselines. In particular, the visualization in 4b showing options learned in Amidar does not show much improvement from what was observed before in Harb, 2018.  

With the motivation of this paper, I am unable to convince myself about options being “reusable” for multi-task here. It would be very useful for the reader to clarify what “novel tasks” are here to appreciate what is learned. Looking deeper into the appendix, I understand that the authors “first learned a good performing policy with A3C for each game and sample 12 trajectories for training.” This is not at all clear in the main paper. Besides, what does it mean by a "good" policy? If we already have that, it is unclear what gains do we get from the proposed method.  

One obvious limitation here is that they also have a hard imposed constraint here is that the options cannot run for more than 20 time-steps in total, to make the objective function a suitable choice. 

Overall:		
An interesting objective function, Learn not only option set but also the number of options needed and incrementally learn new options. 

Paper writing does not convey clearly what are novel tasks and could be significantly improved.

Since the paper claims multi-task and mentions several lifelong learning works like [1], I was expecting rigorous baselines showing performance over multiple tasks. The experiments are lacking in that evidence except for four rooms domain, which is much simpler a domain. 

Near-optimal property is very much lacking the clarity to the best of my knowledge.

[1]Ammar, Haitham Bou, et al. "Online multi-task learning for policy gradient methods." International Conference on Machine Learning. 2014.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper proposes a new option discovery method for multi-task RL to reuse the option learned in previous tasks for better generalization. The authors utilize demonstrations collected beforehand and train an option learning framework offline by minimizing the expected number of terminations while encouraging diverse options by adding a regularization term. During the offline training, they add one option at a time and move onto the next option when the current loss fails to improve over the previous loss, which enables automatically learning the number of options without manually specifying it. Experiments are conducted on the four rooms environment and Atari 2600 games and demonstrate that the proposed method leads to faster learning on new tasks.

Overall, this paper gives a novel option learning framework that results in some improvement in multi-task learning. While the paper is technically sound and somewhat supported by experimental evidence, the experiments are limited to low-dimensional state space and discrete action space. I do wonder if the method can scale to high-dimensional space with continuous control.

Moreover, the framework requires optimal policies to generate trajectories for offline option learning, which seems to add more supervision signals than prior work such as option-critic. I wonder how the method would perform under sub-optimal demonstrations or even random trajectories generated by some RL policy.

Finally, I wonder how this method can be compared to skill embedding learning methods such as [1], which have been shown to be able to compactly represent skills in a latent space and reuse those skills in high-dimensional robotic manipulation tasks.

[1] Hausman, Karol, Jost Tobias Springenberg, Ziyu Wang, Nicolas Heess, and Martin Riedmiller. "Learning an embedding space for transferable robot skills." (2018).

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The paper proposes a method for learning options that transfer across multiple learning tasks. The method takes a number of demonstration trajectories as input and attempts to create a set of options that can recreate the trajectories with minimal terminations.

I currently recommend rejection. The evaluation of the proposed method is rather weak and does not clearly demonstrate that the author’s goals have been achieved. The paper could also do a better job of situating the approach with regard to existing option learning approaches.

Detailed comments:

- The paper strongly emphasizes reusability of learnt options over multiple tasks as a key goal. This aspect is largely absent from the practical part of the paper, however. The proposed algorithm largely ignores the multi-task aspect beyond requiring demonstrations from different tasks - also see the next remark. In the experiments, the multi-task transfer is not emphasized. In the 4rooms domain options are learnt on training tasks and evaluated on test tasks, but the effect of task distribution or task diversity on generalisation is not investigated. Moreover, the larger scale ATARI experiments do not seem to include any multi-task aspects at all, with options immediately being learnt on the target task.

- The objective in (1) omits any explicit mention of different tasks. It would be good to indicate explicitly how it depends on the distribution of tasks and what the expectation is taken over.

- The authors indicate that their learning objective needs the transition function P for the MDP. This is never further discussed. Do the experiments assume known transition functions? If not, how are these functions estimated? If a model is known, does it still make sense to learn option policies from samples or would it be better to use  planning based options (see e.g.[1])?

- While the paper cites a number of option learning approaches, it could do a better job of situating the research within the literature. There are a number of option induction approaches that explicitly focus on reusability of options - see e.g. [1], [4]. There have also been a large number of approaches that focus on hierarchical learning to represent a distribution over demonstration trajectories: see eg. [3],[5], [6], [7]. Some of these approaches might also be better baselines than OptionCritic which doesn’t explicitly take into account learning from demonstrations or multi-task transfer. 

[1] Mann, T. A., Mannor, S., & Precup, D. (2015). Approximate value iteration with temporally extended actions. JAIR, 53, 375-438.
[2] Konidaris, G., & Barto, A. G. (2007). Building Portable Options: Skill Transfer in Reinforcement Learning. In IJCAI,
[3] Konidaris, G., Kuindersma, S., Grupen, R., & Barto, A. (2012). Robot learning from demonstration by constructing skill trees. IJRR, 31(3), 360-375.
[4] Andreas, J., Klein, D., & Levine, S. (2017). Modular multitask reinforcement learning with policy sketches. ICML
[5] Henderson, P., Chang, W. D., Bacon, P. L., Meger, D., Pineau, J., & Precup, D. (2018). Optiongan: Learning joint reward-policy options using generative adversarial inverse reinforcement learning. AAAI.
[6] Co-Reyes, J. D., Liu, Y., Gupta, A., Eysenbach, B., Abbeel, P., & Levine, S. (2018). Self-consistent trajectory autoencoder: Hierarchical reinforcement learning with trajectory embeddings.ICML
[7] Daniel, C., Van Hoof, H., Peters, J., & Neumann, G. (2016). Probabilistic inference for determining options in reinforcement learning. Machine Learning, 104(2-3), 337-357.


Minor comments:
- The results on ATARI seem to have been ended before reaching final learning performance

- I couldn’t find details for how the transition function in 4-rooms is changed

- Does the optionCritic comparison include the deliberation cost? Since this paper aims to minimise option terminations that seems to be the most logical comparison.

- Why don’t the ATARI results compare against other approaches?

- The influence of the KL penalty isn’t really examined in results beyond looking at performance. How does it influence the trade-off between representing trajectories and diversity?

review_title: Official Blind Review #1

Meta-review: This paper presents a novel option discovery mechanism through incrementally learning reusasble options from a small number of policies that are usable across multiple tasks.

The primary concern with this paper was with a number of issues around the experiments. Specifically, the reviewers took issue with the definition of novel tasks in the Atari context. A more robust discussion and analysis around what tasks are considered novel would be useful. Comparisons to other option discovery papers on the Atari domains is also required.

Additionally, one reviewer had concerns on the hard limit of option execution length which remain unresolved following the discussion.

While this is really promising work, it is not ready to be accepted at this stage.
Decision: Reject