Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper considers the multi-agent Reinforcement Learning setting, and proposes a scalable method for the agents to help each other learn, by transferring their policies to each other. The main idea presented of the paper is that an agent i may optimize its policy $\pi_i$ using a combined loss, that not only optimizes the agent returns obtained by the agent, but also imitates the policy of some other agent j. Which agent should imitate what agent is also learned, using an approach inspired from the Options framework, and Option learning algorithms such as the Option-Critic architecture. Several variants of the proposed approach are introduced, depending on how much information the agents can communicate in a particular setting.

This paper is very interesting, as it proposes a method that combines two domains of Reinforcement Learning, namely Multi-Agent systems and Options, that are not often combined. The empirical results are encouraging, and the method seems to scale to a significant amount of agents.

However, the paper is, in my opinion, very difficult to fully understand. I'm an well versed in Options, and knowledgable in multi-agent systems, but the difficulty of understanding the paper does not come from a lack of background information on these two domains (successor features are also very well presented). In my opinion, small but important details are missing. For instance, a precise and explicit description of what the agents exchange as information (policy parameters, actions?), and when, would have helped. Figures 1 and 2, and most of the formulas of the paper, focus on the mathematical aspect of the proposed architecture. They allow to see that all the pieces of information fit together into sound formulas. But it is very difficult to see how to implement the algorithm in practice. I would suggest that the authors add pseudocode sections, for the acting loops of the agents, their training loops, how agents decide which agent to imitate, and how the option and termination functions are learned.

Because the ideas proposed in the paper are interesting, and seem original, I would recommend accepting this paper, but only if the authors manage to make the paper clearer for a first reader.

Author response: discussion with the authors allowed to clarify what the agent see and how they exchange information. I still find the paper a bit difficult to understand, most probably due to its novelty, but I consider that the current version is acceptable. As such, I recommend accepting this paper.
review_title: A combination of distant fields, but difficult to understand

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposed an option-based framework for multiple agents to share knowledge with each other in the same MARL task. For scalability and robustness, two variants of the framework are designed, including 1) a global option advisor, which has the access to the global information of the environment; 2) local option advisor combined with successor representation option to enable more accurate option-value estimation. Experimental results demonstrate the proposed method is able to improve the performance of existing deep RL approaches for multiagent domains.


Pros:

1. This work models the policy transfer among agents as an option learning problem. This is an interesting idea for knowledge transfer in MARL tasks.

2. Comparing to previous teacher-student framework and policy distillation framework, a key feature is that the proposed framework is more adaptive (in what sense?) and applicable to scenarios consisting of more than two agents. 


Cons: 

1. Some details in are not clear (see Questions and Comments)

2. This framework based on centralized information, which may not always be available for multiagent problems, especially for noncooperative settings.

Questions and Comments:
The authors mention that “selecting a joint option means each advice given to each agent begins and ends simultaneously”. Does this mean the options (advice given) all have the same length? Especially for MAOPT-GOA, the termination function is over joint options. But this seems to be a very restricted assumption, given there are a lot of uncertainty in real world and different sub tasks may take different amount of time. 

Advice is given in option level, level action selection are based on the intra-option policy, which is learned through imitation learning, does that mean the student need to collected the demonstration data from an expert (teacher?), how do you obtain the expert policy?

Are the agents homogenous? Meaning do they share the same option sets? How many options are available for each agent for the tasks in the experiment?

It is mentioned that “each option \omega^i contains an intra-option policy corresponding to an agent’s policy \pi^i”. It is unclear why the authors make this statement. Since in general \pi^i may not corresponds to the intra-option policy of \omega^i, unless there is only one option.

=======================After the rebuttal ========================
After rebuttal, I think the author have addressed most my questions. However, I agree with other reviewers’ opinions that the paper need further polishing and clarification, especially regarding the cooperative settings of the problems (homogeneous team, number of options etc as admitted by the authors) and associated theoretical and practical issues. Given the novelty of the idea and the current status of paper, I maintain my score.  
review_title: Interesting idea, but the some description and settings are confusing

final_score: rating: 4: Ok but not good enough - rejection
review_content: 
The authors propose a MARL solution based on the idea of "Local Option Advisor" and "Option-based Policy Transfer". The a


Strenghts

- This is timely area, in which it is good to see progress and alternative solutions to the existing ones.

- The problem of transfer learning might enable to apply MARL in different environments and application scenarios.

Weaknesses

- The significance of the proposed solution is unclear given the fact that the additional complexity of the approach proposed by the authors is not fully justified. Why is basic transfer learning not sufficient?

- The evaluation of the method is based on experiments, but the plots provided by the authors do not show definite evidence of the superior performance (the confidence interval are apparently overalapping).

- The aspects related to transfer learning are not fully evaluated in the paper.

In general, the actual contribution of this work with respect to the state of the art in MARL/transfer learning is not completely clear. The authors propose a rather complex solution, but its actual motivation is not apparent. In particular, it is unclear to see which particular design needs they are trying to address with these additional mechanisms. In other words, the advantages in adopting these mechanisms are not proven by the authors in my opinion.

Moreover, the method is evaluated experimentally, but its actual superiority is unclear. In fact, if you examine Figures 4 and 5, you see that the performance are comparable (the confidence appear to overlap).

Questions

- What are the situations where the proposed solution provides a real advantage? There is also a question about how the proposed solution generalizes to other problems: are there any underlying assumptions of the proposed additional mechanisms to a basic transfer learning model?

- Is there any specific theoretical evidence that shows that the addition of these mechanisms might lead to improved performance?

- What is the tradeoff of the proposed solution with respect to the added complexity? Is this complexity justified? Are there situations/problems that can be tackled only through these added mechanisms?

- Could you addditional explanations to the results in Table 1? How should we interpret them.

- Figure 6 is not completely "readable": how should we interpret the results presented in that figure?


review_title: An interesting extension to classic MARL model, but the design motivation of the mechanisms is not completely justified in my opinion

final_score: rating: 6: Marginally above acceptance threshold
review_content: ##########################################################################

Summary:

The paper proposes a new option-based policy transfer framework for multi-agent reinforcement learning (MARL) called MAOPT. By framing multi-agent transfer as an option learning problem, MAOPT methods are able to learn when to give advice to agents and when to stop it. Authors provide a version of MOAPT for fully cooperative setting based on global state and reward, as well as two versions for mixed settings based on local states and per-agent rewards. The paper presents experimental results on two environments that show performance gains over existing RL methods.

##########################################################################

Reasons for score:

I vote for accepting the paper. I like the idea of utilising the option-based transfer learning approach applied in MARL. My concerns regarding the paper are mainly about the experiments. Hopefully, these would be addressed during the rebuttal period.

##########################################################################

Pros:

1. The proposed option-based method offers a new prospective for policy transfer in MARL. The design of the MAOPT is interesting and reasonable for many MARL problems.
2. The authors propose two different variants of the MAOPT that correspond to different MARL scenarios, namely fully-cooperative and mixed settings. The mixed setting is also studied by two different models, namely MAOPT-LOA and MAOPT-SRO, the latter of which is specifically designed to handle the experience inconsistencies of agents.
2. This paper provides results of several experiments, including both qualitative analysis and quantitative results, to show the effectiveness of the proposed framework.

##########################################################################

Cons:

1. It is interesting to see how the methods scale with the increase of the number of agents. To this end, it would have been great to see experimental results including more than a handful of agents. Even if the methods don't scale outstandingly well, it would be worth adding a discussing regarding this.
2. The authors mention two existing transfer learning methods, namely DVM and LTCR, but compare their method only with the former. It isn't clear to me why the authors haven't used LTCR as a baseline.

##########################################################################

Questions during rebuttal period:

Please address and clarify the cons above.

Also, I'm not really sure about this statement in the introduction: "in a multiagent system (MAS), the exploration strategy of each agent is different...". For instance, there is a MARL method called MAVEN that coordinates the exploration of all agents by conditioning their behaviour on joint variable from a latent space (http://papers.nips.cc/paper/8978-maven-multi-agent-variational-exploration). Maybe you could rephrase your statement to make it a bit more clear?
review_title: Efficient policy transfer method with option-based learning in multi-agent RL

final_score: rating: 6: Marginally above acceptance threshold
review_content: ---------------------
Post-rebuttal
---------------------
I am improving my grade a bit. I recommend the authors to dedicate some time further improving the paper clarity, especially in the matters related to my review and the other reviewers'

---------------------

The authors propose a transfer learning framework for transferring knowledge in multiagent tasks. Their method consists of learning a centralized option-based advisor, that will extract advice to provide to all agents in the system based on the options learned. This advice will be used to compute an auxiliary cost function that will ideally guide all agents towards learning faster.

---------------------
Pros
+ The proposal is novel as far as I can tell
+ The experimental evaluation shows very good performance in all evaluated scenarios
+ Timely and relevant research

----------------------------
Cons
- The manuscript needs polishing. Some English review is needed (paper is understandable but has many small mistakes)
- The assumptions of the method are not very clearly discussed.
- The novel parts of the paper are kind of intertwined with equations and ideas proposed in other works, which makes it harder to quickly see what was proposed in this specific work.
- I feel like LeCTR (Omidshafiei, 2019), cited in the paper, should have been included in the experimental evaluation

----------------------
 Suggestions
----------------------
- I am missing a clear and comprehensive discussion about the assumptions regarding the proposal. Specifically, which communication channels should be available to the method? Does the option advisor need to be able to communicate with all agents all the time? Anything changes regarding the needed infrastructure if we are using GOA, LOA, or SRO?

- Section 3 presents newly proposed concepts together with equations that have been introduced in the option-critic paper, for example. Section 3 should explain only new concepts and equations, making it clear what is a new proposal and what has been proposed before (that ideally should be described in the background section). Also, this section is a little verbose, you can make it more objective to improve readability.

- Why is DVM worse than the regular PPO in Figure 4 (a). It seems to me the algorithm is incorrectly configured or that the algorithm performs poorly in this specific scenario - in which case you should also evaluate another algorithm that performs better in this scenario. In general, I also think that you should add LeCTR at least in the evaluation in the particle environment.

- Add a table at the beginning of section 3 outlining the scenarios in which SRO, LOA, and GOA perform best, and which aspects affect in the algorithm choice.
-----------
minor
----------
- don't let titles without text (3. Approach)
- the paper needs an English review


review_title: Promising work but the manuscript needs polishing

Meta-review: The paper proposes a method on multi-agent options-based policy transfer where agents help each other learn by exchanging policies.

The core idea behind the paper is novel, as it addresses a new and emerging topic of social learning, and of interest to ICLR community. The authors significantly improved the paper with additional experiments and theoretical analysis during the rebuttal process, resulting in a compelling case for the significance of the method. 

Unfortunately, the paper requires addressing the clarity, and a careful proofreading pass, making it unsuitable to ICLR in its current form,

Decision: Reject