Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper presents a hierarchical reinforcement learning method for coordination of multiple cooperative agents with pre-learned adaptable skills. These skills are learned via a maximum entropy objective where diversity of behaviour given each skill is maximised and controlled via a latent conditioning vector. This allows controllability of the variation in skill execution by the meta-policy via changing the skill-specific latent vectors. The paper presents empirical results in manipulation (pick-push-place and moving a long bar by coordinating two Jaco arms) and locomotion (two Ants pushing a large block to a goal location). The method proposed outperforms the baselines reported. 

Overall, this paper addresses an interesting problem and can be impactful with the caveat for some clarifications and analysis. Given that the authors address my concerns, I would be willing to increase my score.

The main novelty of this work lies in how one can learn sub-skills that can be leveraged and adapted for down-stream tasks. The problem setting used to test the method is a multi-agent setting where it is crucial that skills are adapted to enable cooperation. I found the environments and the problem setting generally interesting and important for testing the proposed method. I have a few concerns that I listed below:


1) I found the notations at times inconsistent and confusing. It would have helped to see some more details on the diagram (Figure 2) to understand how everything fits together. 

2) The set of skills for the two agents are selected by the meta-policy in every T_low steps. It looks like in the Jaco environments T_low = 1. Can you comment on this? This seems slightly concerning since it seems like the meta-controller is treating these skills as primitive actions rather than temporally extended behaviour.

3) Looking at the training curves in Figure 4, there seems to be a really high variance in performance of the method. Can you comment on this as this seems concerning. Could you run more seeds to improve this?

4) It is nice to see in section 4.5 how the hyper-parameters balancing diversity in combination with external reward (equation 4) is tuned and how sensitive that is to achieving adaptability for downstream tasks. The only criticism I have is that it is difficult to understand from "Episode reward" on y-axis what the success rate is (similar to Figure 4)? It would’ve been nice to report results in a consistent way throughout the paper for these environments. 

5) Given that all the tasks in the experiments are cooperative multi-agent settings, I would have liked to see more in depth discussion regarding alternative multi-agent methods. The multi-agent baseline provided (which is using a decentralized policy with a shared critic, inspired by Lowe et al., 2017) seems fair, but I wonder if there has been more recent work in this direction that could have been highlighted? 


review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper provides a specific way of incorporating temporal abstraction into the multi-agent reinforcement learning (MARL) setting. Specifically, this method first discovers diversified skills for every single agent and then train a meta-policy to choose among skills for all agents. 

Overall, this paper is well presented so I can understand it well. Unfortunately, this paper didn't give me too much scientific insight. As maybe this is because I don't know too much about MARL, I would like to ask the author to help me address the following questions. 

My first key question is, should we treat temporal abstraction (TA) under the multi-agent setting different from it under the single-agent setting? If they are the same, why do we bother discussing TA under the multi-agent setting? Why not just discuss it under the simpler single-agent setting? If they are not, what are the differences? 

The second key question is if TA under the multi-agent system is special, then why the DIAYN method, which is proposed under the single-agent setting, could be directly used in the multi-agent setting? Why should we consider the DIAYN method, instead of other skills discovery methods? 

Furthermore, I would also like the author to help me address three more concrete questions. 

1. Section 1, paragraph 2, the author wrote: "However, all these approaches are focused on working with a single end-effector or agent with learned primitive skills, and learning to coordinate has not been addressed." Does the author mean there is no temporal abstraction method for multiple collaborative agents? 

2. Section 3.3, the author wrote, " the prior distribution p(z) is Gaussian." I.e., Z is continuous r.v. I would like to know how the author could learn q(z|s) to approximate p(z|s), which is an arbitrary continuous distribution. Maybe I am wrong, but I don't see a way to do this. 

3. In algorithm 1, a skill, once being chosen, will be executed for T_{low} steps, where T_{low} is fixed and pre-defined by the algorithm designer. I would like to hear to author analyzing the pros and cons of this critical design choice.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper aims to achieve multi-agent coordination by composing diverse skills learned by augmenting individual subtask objectives with DIAYN-style diversity bonuses. Once individual diverse skills are learned for the subtasks, the agents are combined by a meta-agent to coordinate multiple distinct robots to achieve a shared goal.

This is a good application of low-level skill learning to multi-agent coordination. I have settled on a weak acceptance, because the approach is simple and seems scalable, but the acceptance is weak because the method relies on specifying the subtasks in advance.

The approach is well-motivated in that learning individual skills in isolation is generally more tractable than learning their combined application from scratch, and the building blocks of this system are well-chosen. The results demonstrate the importance of the diversity objective, and find a good sweet spot for the diversity weight.

I do have some criticisms, related primarily to the decision to pre-train with both a continuously-parameterized diversity conditioning as well as a discrete set of concrete subtasks. Because these subtasks must be specified in advance, this limits the wide applicability of the resulting approach to those that can be broken down a-priori into components. Did the authors consider using the DIAYN objective on its own to encourage sufficiently diverse behaviors? If this didn't work, would perhaps a larger latent skill vector, or a large discrete set of DIAYN skills, have made it work?

I also don't see the size of the latent skill embedding reported anywhere. How big is this vector; that information should be added to the paper.

However, the approach is generally good. I think the paper would be improved if it included a strong baseline that uses DIAYN only (no a-priori subtasks), so we can evaluate how important that expert knowledge is to the final performance.
review_title: Official Blind Review #2

Meta-review: This paper deals with multi-agent hierarchical reinforcement learning. A discrete set of pre-specified low-level skills are modulated by a conditioning vector and trained in a fashion reminiscent of Diversity Is All You Need, and then combined via a meta-policy which coordinates multiple agents in pursuit of a goal. The idea is that fine control over primitive skills is beneficial for achieving coordinated high-level behaviour.

The paper improved considerably in its completeness and in the addition of baselines, notably DIAYN without discrete, mutually exclusive skills. Reviewers agreed that the problem is interesting and the method, despite involving a degree of hand-crafting, showed promise for informing future directions. 

On the basis that this work addresses an interesting problem setting with a compelling set of experiments, I recommend acceptance.
Decision: Accept (Poster)