Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper tackles the problem of learning asynchronous multi-agent policies with macro-actions. The authors present a set of asynchronous multi-agent Actor-Critic methods in order to solve the problem, which allows agents to `directly' optimize policies that are asynchronous and macro-action based. They apply the framework in 3 standard multi-agent learning paradigms: decentralized learning, centralized learning and centralized learning for decentralized execution. They also empirically show the utility of the methods over the standard individual actor-critic method and centralized actor-critic method on 3 multi-agent cooperative tasks: Box pushing, Overcooked and Warehouse.
main_review: **Pros**:
* Writing: The writing in the paper was largely clear.
* Experimental setup: The experimental setup was exhaustive and the 3 environments used to evaluate the framework was interesting.

**Cons**:
* Presentation of the Mathematical ideas: In Section 3, the presentation of the framework could have been made clearer. Upon reading the update rule for the policy gradient in the new framework, It was non-trivial to get the intuition behind the formulation. Perhaps some more discussion on the differences between IAC/CAC algorithms to the presented framework could bring out the fundamental changes in a formulation more clearly. I think at least the algorithmic block pertaining to MAC-IAC can be brought to the main paper from the appendix, as it is the first time the new framework is being presented. The following subsections of Section 3, where MAC-CAC and MAC-IACC are introduced can include more intuitive discussions on how each technique differs in their update equations. At the moment, this section is a little difficult to parse.

* Novelty: The presented framework is interesting but still requires additional work in its analysis -- such as the limitations of the gradient updates, sample efficiency of the proposed methods, and uniform framework for analysis of the 3 proposed variants. At the moment, the framework shows potential but can be refined further.

* Baselines used: I felt the baselines used for comparison were limited to the presented techniques and primitive action-based techniques. How do the algorithms presented under primitive actions fair when they are also presented with high-level macro actions in their action set? This might be a fairer comparison to evaluate the technique upon. I am also curious to know how the presented framework performs in an adversarial/competitive setting. Is this framework even applicable in such a setting?

**General Comments**
Overall, the presented ideas show promise in scaling up asynchronous learning of macro-action-based multi-agent policies in cooperative environments. However, at the moment the presentation of the paper can be improved and the ideas presented need to be elaborated upon further. The extension seems only incremental at the moment. The experimental framework presented seems exhaustive in the cooperative setting though.

*Originality*: Moderate

*Clarity*: Moderate

*Quality*: Moderate

*Significance*: Moderate to High
summary_of_the_review: Overall the paper showed promise in scaling up the learning of asynchronous macro-action-based multi-agent policies. However, it still requires further work in improving the presentation and performing elaborate analysis of the framework on the three settings discussed -- such as limitations of the gradient updates derived. The work seems to be only incremental in comparison to prior work. Hence, I am inclined to reject it at the moment.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes several extensions of the existing multi-agent independent actor-critic, centralized actor-critic, and independent actor with centralized critic to the MacDec-POMDP for solving multi-agent problems with asynchronous actions. The proposed methods show significant improvements compared with their original forms in several MacDec-POMDP problems.
main_review: Strengths:
The paper focuses on a domain that is common in practice but not well studied, and the proposed methods show clear improvements. 

Weakness:
The major concern I have is the lack of comparison on other macro-action-based methods (like those mentioned in the third paragraph in section 1). The proposed extensions are natural generalizations from their original forms (which could be a strength), and improvements are expected when compared with methods not designed for the Mac-POMDP.  It would be more interesting to see the comparisons with methods designed for the same domain.
summary_of_the_review: While the technical contribution and the experiment results are, in my opinion, not super significant, the paper indeed introduces a practical research direction and brings several intuitive first steps towards this direction. Thus, I recommend acceptance of the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper considers the problem of MacDec-POMDPs where it requires agents to be capable of performing asynchronously without waiting for other agents to terminate. As there is no multi-agent policy gradient method with macro-actions for MacDec-POMDPs, this paper fill this gap and integrates the macro-action-value into multi-agent policy gradient and proposes (i) macro-action-based independent actor-critic (Mac-IAC) method, (ii) macro-action-based centralized actor-critic (Mac-CAC) method, (iii) Naive Independent Actor with Centralized Critic (Naive IACC) as well as Independent Actor with Individual Centralized Critic (Mac-IAICC) via CTDE. Experimental results show that the proposed methods outperforms vanilla baselines.
main_review: •	Strengths:

This paper considers an important problem of asynchronism of action in MARL. The main contribution of this paper is integrating the macro-action-value from the Q-value-based macro-action MARL method into multi-agent policy gradient. 

•	Weaknesses: 

The reviewer has some concerns of this paper:
1.	What is asynchronism in MARL? Can you define it in this paper? The main concern of this paper is that it introduces the asynchronism of actions in MARL, however, with macro-actions which is built on top of the option in hierarchical RL, this paper mainly introduces a MARL method with macro-action. The main concern is it seems the proposed methods are off-topic 
2.	The integration of macro-action-value from previous MARL work is straightforward, the main challenge has been discussed in Xiao et al., 2019. This may make the integration a bit weak. Or can the author highlight the challenge in the paper?
3.	Environmental scenarios evaluated in the experimental section are mainly on hierarchical MARL and multi-task MARL, it hard to find the asynchronism of actions in these scenarios, which is the main contribution of this paper.
4.	There some hierarchical MARL methods [1, 2, 3, 4, 5], and some role-learning [6] methods, can you also compare the proposed methods with some of these methods and discuss these methods?
5.	It seems that related works on asynchronism of actions in MARL is missing.

Reference:

[1] Hierarchical Cooperative Multi-Agent Reinforcement Learning with Skill Discovery

[2] Feudal Multi-Agent Hierarchies for Cooperative Reinforcement Learning

[3] Multi-Agent Common Knowledge Reinforcement Learning

[4] OPtions as REsponses: Grounding Behavioural Hierarchies in Multi-Agent Reinforcement Learning

[5] HAVEN: Hierarchical Cooperative Multi-Agent Reinforcement Learning with Dual Coordination Mechanism

[6] RODE: Learning Roles to Decompose Multi-Agent Tasks


summary_of_the_review: The writing is ok. This is paper is easy to follow. It would be great if the author can make videos to visualize the learned results presented in the appendix. I would like to increase the score to Week Accept if the authors can address the above concerns during the review session.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents a method for learning policy gradient based methods on macro-actions in environments where multiple agents initiate and terminate actions at different timesteps. The paper identifies a suitable framework (MacDec-POMDPs) for these tasks and develops a series of actor-critic mechanisms to allow agents to coordinate asynchronous macros actions. Four main macro-action algorithms are proposed:
* Mac-IAC - that applies a variance reduced version of actor critic directly to the multi-agent domain
* Mac-CAC - which treats all agents as a single global agent
* Mac-IACC - described as naive, which uses a centralised critic but derives independent policy gradients for each agent
* Mac-IAICC - which again uses a centralised critic like mechanism but one where the termination times of the individual agent correct the traces used to train this central critic. Again this derives independent policy gradients for each agent.

The experimental section presents three models: box pushing, overcooked and warehouse. From my reading of the paper, these models provide predefined options to all agents (so all agents have access to all options), where some options are extended and some are single step. Rewards are defined that are broadcast to all agents, and are used to learn the coordination policy (high level) but not the low level options, in the case of the macro action algorithms. Two non-macro action methods are evaluated against Mac-IAC and Mac-CAC on Box-pushing and Overcooked. This finds that macro-action methods are better than the primitive action methods (less interesting - see notes) and that there are situations in which Mac-IAC is better but more typically Mac-CAC outperforms Mac-IAC when there is a significant need for coordination. The next series of experiments compares the four macro-actions methods and show (most pertinantly) that the Mac-IAICC method achieves better coordination than Mac-IAC, nearing the performance of Mac-CAC even in the most coordination intensive conditions. Naive Mac-IACC improves upon Mac-IAC in terms of coordination but suffers in coordination intensive tasks. This supports the authors' argument that the variance in the shared critic for Naive Mac-IACC is damaging and that the proposed individual centralised critic approach for Mac-IAICC significantly improves upon this.
main_review: The paper gives a relatively clear definition of the domain of interest and its value/applicability. The algorithms are mostly clear and seem sensible. There is some significant novelty in the use of central critics with independent gradients particularly when those central critics have to be individualised. The need for centralisation does limit this in the number of agents that can be coordinated though and this could be more clearly commented on. The experiments comparing macro-action based methods are clear and well interpreted with some interesting findings. Finally, for the strengths, I do think that these methods may have some viability for use in industrial applications.

There are a couple of things that I think weaken the paper. The first of these is quite general and relates to the framework in which all this is being done. There seems to be an unusual asymetry to how the two levels of the action hierarchy are treated. There are different policies of course, but the action sets are strictly macro actions, which means that all primitive action decisions must be part of an option. This has implications for generalisation across the same task or from one task to another, as it forces the option to contain both reusable skill information as well as global task information. More surprising to me, observations are split into low level and high level and I cannot see the justification for that. One consequence of this asymmetry emerges in the tested environments. For instance the box pushing environment requires macro actions of length 1 to be defined, which mimic three of the primitive actions.

A second weakness of the paper relates to the ambiguity in some of the descriptions. In particular, the definition of squeezed trajectories and how these are incorporated into each of the proposed algorithms could be much clearer. Another thing that could be clearer is that this paper presents methods for the macro-actions and how to induce coordination with these. It doesn't address how the options might be learned or developed, nor how such things as initiation sets and termination conditions should be defined - a non-trivial thing see (Jong et al., 2008) . As such there is a serious gap between what is proposed here and the use of these methods in practice. See (Konidaris and Barto, 2007; Bacon et al. , 2017) for examples of works which might address this concern.

Finally, there is a weakness in the experiments in terms of fair comparison. In the experiments, there is quite a lot of pre-encoded knowledge in the macro actions. Not least the one-step macro actions, but also examples like: the get-tomato-like actions in the Overcooked domain, or the Move-to-big-box action in the Box Pushing. It seems to me that these options are already predefined as deterministic policies, as no reward structure is given to these nor is there a mechanism for propagating global rewards back in time to relevant option executions (a better approach). Am I right that these macro action methods are given their options at the start? In which case, the comparison between macro-action based methods (Mac-CAC, Mac-IACC and Mac-IAICC) and primitive action based methods (IAC and IACC) would be demonstrably biased towards the macro-action based methods. I would argue that this knowledge injection is at least partly responsible for the big difference between the two types of methods seen in Figure 3. A fairer comparison would be to have options learned as they go for the macro-action methods.

## Notes

Some more detailed notes/comments are as follows:

In Sec 2, what is mathbb H in the definitions? And why does this relate to finite horizon? Ah, is it an integer? This could be clearer.

The authors state that:
> asynchronicity of macro-action execution over agents, where agents may start and complete their own macro-actions at different time-steps

But argue that:
> Previous deep MARL methods for Dec-POMDPs cannot work in this case as they are all based on primitive actions synchronously executed by agents.

However the formulism given in section 2 does model joint actions at each time step which are cross products over all agents in the task. So the primitive actions are taken synchronously (at least as treated by the environment). This should be clarified.

The explanation of squeezed trajectories in Sections 2.5 and 3 are not clear enough. I would recommend the authors include some mathematical equations or definitions to make this clearer.

In Equation (8) it's not clear how the $\vec{h}$ and $\vec{m}$ objects are constructed. As argued already, the macro actions are initiated and terminated asynchronously but the $\vec{m}$ is a joint macro action, so what is that? For instance, at primitive time $t$, does the joint macro action use all the options that agents are in, even if they initiated this some steps ago? How does this asynchronicity influence the convergence properties of the critic and policy gradient optimisation?

It seems odd that you need to explicitly manufacture exploration decay. You say:
> Exploration is deployed by applying a linear decaying $\epsilon$-soft policy.

But the actor-critic paradigm is one which, it is argued, controls the exploration/exploitation trade-off. This should really be justified more clearly in the paper.

## References
(Jong et al., 2008) Nicholas K. Jong Todd Hester Peter Stone (2008). The Utility of Temporal Abstraction in Reinforcement Learning. AAMAS 08.
(Konidaris and Barto, 2007) George Konidaris and Andrew Barto (2007). Building Portable Options: Skill Transfer in Reinforcement Learning.
(Sutton et al., 1999) Richard Sutton, Doina Precup, and Satinder Singh (1999). Between MDPs and Semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning. Artificial Intelligence 112:181–211.
(Bacon et al. , 2017)  Pierre-Luc Bacon, Jean Harb and Doina Precup (2017). The Option-Critic Architecture. AAAI 2017.
summary_of_the_review: The main contribution is relatively interesting and contains some novelty. It is a challenging domain and there is a clear industrial application for those that can get this right. I think there are some weaknesses in explanations and definitions in the main paper which weaken it though. The interpretation of the  experiments as it relates to macro-action methods is clear and insightful and convinces me that there is value in these ideas. However, the comparison with individual action methods does not acknowledge the high level of manual code/knowledge injection that favours the macro action methods. As such, I am falling below the acceptance threshold for this paper. However, I would encourage the authors to continue with the work as it is interesting and promising.
review_title: null

Meta-review: The paper provides an "asynchronous" method for multi-agent actor-critic with macro-actions. A major contribution of this paper is the integration of the macro-action-value from the Q-value-based macro-action MARL method into multi-agent policy gradient. Although it appears an interesting contribution, reviewers found that several parts of the paper were not clear enough and there is a lack of fair comparison with previous works.
Decision: Reject