Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper proposes a multi-agent algorithm which combines options with a successor representation and imitation learning. My understanding of the algorithm is that it initializes n options (one for each agent), and a meta-policy which selects which option should be used by each agent during each “episode” (see algorithm 1). So agents can use the most relevant option, rather than relying solely on their own option. Individual options are learned with PPO. Successor Representations are used for the meta-policy (I believe) by using a shared feature representation but individual preference vectors for each agent. Finally, the options for all agents are forced to be similar to each other through an imitation learning loss, which is scaled down over the course of training. 
main_review: **Clarity:**
- The paper has significant issues with clarity which make it difficult to assess the proposed algorithm. I had to read the methods section several times to parse what is being done. 
- For example, consider the following: “The option module select an option ω for each agent. During the training phase, the option module uses experience from all agents to update the option-value function and corresponding termination probabilities.” -> this indicates that there is a single option pool being shared among all agents. This would imply that actually each agent is running the identical policy, which is composed of the same set of options. The paper then goes on to say, “Each agent will exploit the knowledge from another agent πω based on the selected option ω, and this is achieved through policy imitation, which serves as a complementary optimization objective”. This is very confusing, because it is not clear how agents actually have their own policies if they are all using the same option pool to act. How can they imitate each others’ policies if they share the same policy? Only after re-reading this section to determine that there are n options, did I realize that the intention is that each option is meant in some sense to correspond to one agent, although this does not appear to be enforced.
- The second problem is that how the SR is used is not made clear by the paper. A careful reading determines it is used for the option-value function, i.e. the meta-policy. This only became apparent to me after reading that the individual agent policies were learned with PPO, but it took several passes. This should be stated more clearly and earlier in the paper (and repeated).
- A further suggestion to help clarify the paper would be to state explicitly which parameters are unique to each agent. The options are actually not unique to the agents, because they are shared freely among the agents. My current understand is that the only agent-specific parameters are the preference vectors w. 
- The motivation given in lines 26-30 is compelling. 
- Using \tau for parameters is not a great use of notation, since \tau is often using to represent trajectories in RL.
- The explanation of successor features (lines 87-98) could be made more clear.
- The transfer loss L_{tr} does not appear to be introduced before it is used in Eq. 4
- There are typos and grammar mistakes throughout e.g. “a regularization term to ensure explorations” in line 176

**Novelty/Originality:**
- The novelty of the proposed algorithm is limited. The idea of using options for multi-agent learning has been explored in [33,12,5] as cited in this paper. The idea of using successor representations with a shared feature representation and individual preference vectors for each agent was proposed in https://arxiv.org/abs/2102.12560, which is not cited by this paper. 
- The idea of using imitation learning to make the agents’ policies more similar appears to be novel, but it is not well motivated. Why use options at all in this case, rather than allowing agents to share parameters for more efficient learning? This should be better motivated. It would also be good to include an ablation study assessing the importance of this imitation learning loss, and to benchmark against multi-agent techniques which instead rely on parameter sharing among agents. 

**Quality:** 
- MADDPG is not a reasonable multi-agent baseline. It is several years old (from 2017), and many more performant algorithms have been proposed since then (e.g. QMIX, etc, or https://arxiv.org/pdf/2106.02195.pdf for a thorough collection of SOTA baselines). It is difficult to assess the contributions of this paper unless it is properly benchmarked against recent techniques. 
- The paper claims that DVM, published in 2019, is “the most recent multiagent transfer method”. This is not true; see e.g. https://arxiv.org/abs/2102.12560
- The ablation studies do not assess the contribution of the options framework, or the imitation learning approach. 
- A positive aspect of the paper is that it uses 10 random seeds per experiment, indicating the results of the experimental evaluations are more reliable and more likely to replicate. Good work!
- The use of the state reconstruction loss to help refine the successor representation is a nice touch.

**Significance:**
- Overall the paper has promise, but it is difficult to assess the significance of the contributions due to the clarity issues and the lack of appropriate benchmarking against recent techniques. 
limitations_and_societal_impact: The paper provides a nice limitations section that analyzes drawbacks of the method. It does not appear to address societal impact. My suggestion would be to include a discussion of possible beneficial applications of effective multi-agent learning as motivation for why the problem is important.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The authors propose a Multiagent Policy Transfer Framework (MAPTF) that learns which agent policy is appropriate for reuse and when to terminate it. Consequently, MAPTF is seen as an option learning approach to policy transfer; it uses the Successor Representation Option (SRO) algorithm proposed by the authors to learn the option-value function under each agent's preference.

The originality of the proposal consists of extending the problem of option learning, integrating it with the problem of policy transfer and reuse. Thus, the idea is to help the learning of a particular agent in the MARL context with the execution of other agents' options. MAPTF learns, for each agent, the option-value function and when to terminate the execution of the option. Experiments performed in Pac-Man and in the Multiagent Particle environment (MPE) showed the effectiveness of the proposal.
main_review: MAPTF belongs to a class of novel combinations of well-known techniques. MAPTF extends the option framework to a MARL context, allowing the reuse of other agents' options in learning. Likewise, the SRO algorithm is an adaptation of previous algorithms – SRO learns the option-value function under the preference of each agent, as well as the termination probability of each option.

The text is a little confusing, thus making it difficult to fully understand the proposal. The evolution of the content given in the Preliminaries section is not perfectly extended to the TL-in-MARL scenario. The notation gets quite confusing.

A question that remains is whether the authors, so as to deal with POSG, simply considered the observation of each agent as state information (i.e, assumed that o = s). Another question is how MARL came about: are they simply agents learning their own policy and completely ignoring the other agents, as if they were completely independent learning? If so, how did one deals with the non-stationarity of the MARL environment? In section 6, the authors comment that they did not address the problem of coordination between agents, but ignoring the non-stationarity of a MARL is more consequential than that. Although the authors claim broad generality for their proposal, I am not convinced that it would work for heterogeneous agents.

The figures could be improved and should be in perfect harmony with both their description in the text and the algorithms presented.

Minor issues:
Algorithm 1, step 2 and 6: How is an option selected? Step 4: How is the new state s’ observed? Step 9: is Eq. 9 used here? How is Dis(.,.) calculated?
Algorithm 2: Is it run for each agent? Make this clear in step 1 of the algorithm.

General question: at the beginning of learning in a MARL, agents still don't know much and exhibit erratic behavior. However, eq. 9 indicates that it uses the policies of other agents the most at the beginning of learning. As everyone is learning at the same time, would the transfer be helpful?

The experimental results presented are very promising. However, as there were many doubts about the modeling carried out, it is not possible to be sure that the contribution is in fact very significant. While the basic ideas are promising, development must be improved.

I would like to reinforce that I have read the authors' rebuttal and thank them for all the clarifications and improvements to the paper.
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose a method for accelerating learning in collaborative multiagent RL domains. The acceleration is achieved by the sharing of information between the agents controlled by a framework that also involves learning multiple options. The authors show clear performance improvements in two different domains against "regular" learning and another transfer learning-related method. However, my main concern with this paper is that it is very difficult to read and understand even at a high level. Therefore, I am not sure I would be able to reproduce what is reported here.
main_review: Although the authors present clear performance improvements in the experimental evaluation, there are multiple points to improve in regard to clarity and the description of the method.

The first point would be the title, which is overly generic. The authors should make sure that the title reflects what is novel in this paper and the main characteristics of the proposed method. While it is true that it is a transfer learning method, dozens of such methods exist, and this title doesn't help a reader screening for papers.

The manuscript itself is also very hard to read. Why are the options a critical point in the knowledge-sharing mechanism? If all options are learned from scratch, how I can expect that they should be particularly useful for other agents? ALso, it is not clear at all exactly which information is communicated to the other agents and how often communication is performed.

Reading your paper, I feel like it could be useful to choose which option to be applied in each step through an attention layer, similarly as it is done in DECAF:

Glatt, Ruben, et al. "DECAF: deep Case-based Policy Inference for knowledge transfer in Reinforcement Learning." Expert Systems with Applications 156 (2020): 113420.

The DVM  algorithm is used for comparisons in the experimental evaluation, but you never explained why this method is most similar to yours, and how they differ (in fact, I think not even the acronym is explained). It would also be very relevant to show your comparisons how often the agents communicate. Communication is a critical aspect in any multiagent application, and it would be good to show the trade-off performance improvement x communication used.

I don't see any reason to show the performance for PPO and a multiagent base method for the same domain. One should be fine.

Finally, I am concerned that both of the domains chosen in your evaluation do not require coordination (i.e., the agents can probably solve the task by simply chasing pac-man/prey independently). It would be good to include a domain where coordination is required.
limitations_and_societal_impact: No short-term societal impact.
review_title: null

Meta-review: The reviewer team agrees that this paper has some clear strengths and weaknesses:
(+) It proposes a new way of combining two known techniques, the option framework and successor representation, for MARL.
(+) The proposed algorithm achieves good performance empirically.
(+) The additional experimental results provided in the author response phase addressed some of the reviewers' concerns, especially the comparison with QMIX and the additional ablation study. These results improve the quality of the paper and the significance of the work.
(-) The applicability of the proposed technique is limited by some level of homogeneity or similarity among the agents. The authors discussed how the technique can be used for heterogeneous agents in the response phase but the benefit of the framework clearly relies on a (sub)set of agents who have similarities in performing the tasks and do not need to carefully coordinate with each other.
(-) While the authors claim that they have made efforts on improving the clarity of the paper since the last submission to an earlier conference, the reviewer team finds that the paper still lacks clarity in a number of places, which makes it hard for the reader to understand the paper. For example, it was not very clear which parameters are shared and which ones are agent-specific. Alg 1 lacks clear explanations in some of the important steps. Some of the related work is mentioned without explaining the differences from this work clearly. The authors' response clarified some of these points but the paper's clarity needs to be improved. In addition, there are typos and grammar mistakes throughout the paper.

Overall, the reviewer team views this paper as a borderline paper mainly due to the lack of clarity. The novel combination of the option framework and successor representation may inspire further research and the significantly improved empirical performance over baselines is promising. In the next version of the paper, I would recommend the authors include the additional experimental results presented in the response phase and improve the clarity of the paper based on reviewers' comments and suggestions.
Decision: Accept (Poster)