Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Summary:
The authors propose a method for learning hierarchical policies in a multi-task setting built on options and casts the concepts into the Planning as Inference framework. They claim that the method brings several advantages such as:
-Stabilize end-to-end mtl,
-Leads to coordination b/w master policies,
-Allows fine-tuning of options
-Learn termination policies naturally

The proposed approach derives the reward Eq.6 by extending the graph to options for Levine’s tutorial. Eq.6 is simply the extension of the reward of maximum entropy RL to the options framework. The ideas presented in the paper are interesting, but I have concerns about the scalability of such an approach. Please see the detailed comments below.  Additionally, please note that although I have marked a weak reject, I am open to adjusting my score if the rebuttal addresses enough issues.

Detailed Comments:
A primary weakness of this approach is that it seems like there is one network that learns the options and is shared across all task (that would be the prior) and then there is a task-specific network for all options (posterior), wouldn’t this be very difficult to scale if we want to learn reusable options over the lifetime of an agent? If there are n tasks, do you need to use n different networks? 

The authors assume that all options are present everywhere i.e. I ⊆ S. I think the work could benefit from removing this assumption.

The authors mention that unlike (Frans et al., 2018), they learn both intra-option and termination policies: there is definitely more work that aims to learn both the skill and its termination. It would be more complete to cite additional references here that learn both of these or rephrase this sentence. 

It does not seem clear why “term 1 of the regularization is only nonzero whenever we terminate an option and the master policy needs to select another to execute.” This doesn’t seem true as this is a ratio of the two probabilities and not just the instantiation of the random variable. 

The results in moving bandits alone are very convincing. However, in Taxi (2b) distral+action seems to be as good as/even better MSOL. In directional Taxi (2c) Distral(+action) manages to reach the same final performance (if we care about that), can you please comment on this.

Some parts of the experiments section does not seem clear to me, Does the proposed approach use a network per task? if yes, then it is obvious that their method could improve over learning on 12 tasks with one set of network. Please clarify. 

One major concern is that the only high dimensional experiment is a swimmer and it is not immediately clear how much do we gain there. Distral is relatively closer in performance to both MSOL and MSOL frozen. I would recommend evaluation in a variety of high-dimensional domains such as other instances in Mujoco, and visual domains. In particular, the proposed ideas would make a stronger case if the baselines included other multitask hierarchical agents such as [4] for example. A discussion including some of the missing relevant related multi-task literature would also be helpful [1,2,4,5,6].

[1] Mann, T. A., Mannor, S., & Precup, D. (2015). Approximate value iteration with temporally extended actions. JAIR, 53, 375-438.
[2] Konidaris, G., & Barto, A. G. (2007). Building Portable Options: Skill Transfer in Reinforcement Learning. In IJCAI,
[3] Andreas, J., Klein, D., & Levine, S. (2017). Modular multitask reinforcement learning with policy sketches. ICML
[4] Tessler, Chen, et al. "A deep hierarchical approach to lifelong learning in minecraft." Thirty-First AAAI Conference on Artificial Intelligence. 2017.
[5] Ammar, Haitham Bou, et al. "Online multi-task learning for policy gradient methods." International Conference on Machine Learning. 2014.
[6] Mankowitz, Daniel J., Timothy A. Mann, and Shie Mannor. "Adaptive skills adaptive partitions (ASAP)." Advances in Neural Information Processing Systems. 2016.
review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: This paper is about learning hierarchical multitask policies over options.  The hierarchical prior policy is shared amongst all tasks, while the hierarchical posterior policies are allowed to adapt to specific tasks.  Once the prior is learned, it is fixed.   The parameters of the posterior policies are adjusted via an adapted version of A2C.  

I liked the flow and the organization of this paper.

Comments and questions:
* I see that term 1 and term 3 of equation (6) working together to ensure some kind of exploration and exploitation.  Term 2 controls how the option posterior deviates from the prior.  However, when the ratio is 1 or less than 1, the value of (6) would increase, and both cases would have made the posterior more like the prior.  There seems to be no other term that incentivizes the option posterior to deviate, and I do not see how the options are adapting to tasks.  

* The term 1,2,3 in (6) are weighted equally by beta and cannot be fine-tuned to desired trade-offs.  

* Should there be \xi(i) multiplying the last term in (3)?  

* What does delta(z_t - z_{t-1}) mean in section 3.1?  It was not defined anywhere.  

Misspelling and typos:
* page 5: optimized is misspelled in "Details on how we optimiszed..." 

* Appendix C.1: should there not be a superscript dash on A_{\pi_i} since the superscript dash carries the meaning that the term is a constant.  
review_title: Official Blind Review #3

Meta-review: Apologies for only receiving two reviews. R2 gave a WR and R3 gave an A. Given the lack of 3rd review and split nature of the scores, the AC has closely scrutinized the paper/reviews/comments/rebuttal. Thoughts:
 - Paper is on interesting topic.
 - AC agrees with R2's concern about the evaluation not using more complex environments like Mujoco. Without evaluation on a standard benchmark, it is difficult to know objectively if the approach works. 
 - AC agrees with authors that the DISTRAL approach forms a strong baseline. 
 - Nevertheless, the experiments aren't super compelling either.
 - AC has some concerns about scaling issues w.r.t. model size & #tasks. 

The paper is very borderline, but the AC sides with R2's concerns and unfortunately feels the paper cannot be accepted without a stronger evaluation. With this, it would make a compelling paper. 
Decision: Reject