Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper develops an asynchronous multi-agent actor-critic approach that allows agents to directly optimize asynchronous strategies in three state-of-the-art multi-agent training paradigms: decentralized learning, centralized learning, and decentralized execution and centralized training.This paper also evaluates its methods in both simulation and hardware environments. The experiment results show the method can obtain high-quality and asynchronous solutions. 
strengths_and_weaknesses: Strengths:

+ The experimental design and discussion are very interesting. The experiments are implemented in both simulation and hardware environments, which demonstrates the powerful application ability of this method.
+ The statement is clear and easy-reading.


Weaknesses:

+ The idea of utilizing macro-action in MARL methods is not novel. This paper just transfers the macro-action-based value evaluation in DQN method[1] to AC method. To some extent, the paper lacks originality.
  

[1] Yuchen Xiao, Joshua Hoffman, and Christopher Amato. Macro-action-based deep multi-agent  reinforcement learning. In Proceedings of the Conference on Robot Learning, 2019.
questions: + In the setting of this paper, the number, meaning and complexity of macro-actions are defined in advance. How much influence do different settings of macro-actions have on experiment results?
  
+ In line 343-346, the authors think the value-based approaches have better performance than actor-critic approaches because of their stronger sample efficiency. Why not compare with high sample-efficiency AC methods (such as MAPPO)?
limitations: The authors do not discuss limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a general formulation of macro action-based Multi-Agent Actor Critic framework by extending widely used methods in multi-agent actor critic literature to a Mac-DecPOMDP setting with asynchronous actions. This is desirable since agents can now update themselves asynchronously without having to wait for other agents. Macro-actions are designed such that it can include both temporally extended actions, as well as, primitive actions. 

Existing methods with macro-actions often rely on the assumption that agents behave synchronously and thus cannot be readily applied in the proposed setting. To this end, the authors propose a range of baselines (generalized from their primitive action counterparts) to compare with for evaluation purpose -- Mac-IAC, Mac-CAC, Naive IACC, and Mac-IAICC. The empirical results from three different environments and an additional real-life robotics application show significant performance and desirable behaviors in the agents.
  
strengths_and_weaknesses: Strengths:

1. Although the authors borrow the "trajectory squeezing" technique to achieve asynchronicity (as stated in Sec 2.5), the fact that it is adapted to an on-policy setting is pretty novel. Since, most practical applications in MARL display asynchronous behavior, the set of algorithms presented here are of significance to the MARL and the robotics community.

2. The paper has a clear flow and is well written. I, at least, did not come across any technical or grammatical errors (Kudos to the authors for that !)

3. The authors present their experimental results on three different environments each of which justify the use of one or another proposed macro-action paradigm (by showing clear improvements). Additionally, ablation studies were performed on a real-life robotic warehouse setting, which also clearly highlights desired behavioral aspects of the proposed methods. The authors have also submitted codes for reproduction (although I did not run them).

Weaknesses:

1. The "squeezing" process described in the main paper is a bit ambiguous, however, it is very clearly explained in the Appendix. Since the authors use this as an essential part of their training algorithm, perhaps it would be good to move the explanation from the Appendix to the main paper (just Figure 6 from Appendix would suffice). 

2. Section 4.2 (Advantages of learning with macro-actions): This seems to be an unfair comparison given that macro-actions are pre-specified and it is obviously convenient for the macro-action-based agents to learn faster and better than primitive-action-based agents. As such, the results are not all that appealing. Perhaps this should be pushed to the Appendix to make room for explaining the algorithm (from point 1. above) in more detail.

3. The limitations of the proposed framework should be clearly highlighted. While the authors compare the shortcomings of each module in different environments, I felt that a general discussion of the algorithms performing asynchronous updates is missing. For instance, when you look at the trajectory squeezing process in the Appendix, is there any crucial information that the actor/critic misses out on during gradient updates, and if so, how can they be improved.  

Please note that I am considering the above-mentioned weaknesses as minor issues and authors are requested to make changes to address these issues in the main paper.
questions: Q1. lines 74-75: Can the authors clarify how an agent's "primitive-observation set" is different from it's "macro-observation space". Isn't the latter a subset of the former? How does the environment have two different observation functions?

Q2. lines 302-303: Even after having access to the (full) state information in the Box Pushing task, not only does Mac-CAC outperform Mac-IAICC, it also learns faster. How is this possible given that both models have the same input and CAC has a huge joint macro-action space?

Minor Comments:

1. The authors should try to add more clarity as to why they had to design and compare with their own baselines (and not with existing methods on Mac-DecPOMDP). While I understand that existing methods cannot be readily generalized to the asynchronous setting (which the authors have mentioned in bits and pieces throughout the paper), issuing a more concrete statement upfront would be useful for the readers to place this work in the context of other similar works.

2. Lines 145-147: Please point to the appropriate sections in the Appendix.

limitations: In general, the authors have done well to highlight the limitations of the different proposed algorithms in Section 4.2. However, I felt that a general high-level discussion on the asynchronous updates in the algorithms is missing and the paper could benefit from placing this in the context of other potential approaches.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper extends macro-action-based DQN to actor-critic methods to solve asynchronous hierarchical multi-agent cooperation tasks. Asynchronous learning, where the high-level decisions have different action durations, is an interesting and important topic for complex real-world applications. The motivation is clear, and the paper is well organized. The experiments are performed on both simulation environments and hardware platforms, which is convincing.
strengths_and_weaknesses: Strengths:

Asynchronous learning is meaningful to many real-world applications.

The motivation is clear, and the paper is well organized.

The experiments are extensive and demonstrate that learning with macro-actions is important in tasks requiring high-level planning. And the selected platforms are suitable to verify the effectiveness of asynchronous learning. 

I appreciate the section Mac-IAICC, where the duration issue is clearly discussed and specifically treated. 

Weaknesses:

The novelty is limited since the proposed methods seem to be directly extended from macro-action-based DQN.
questions: The easiest method I can see to solve the duration problem is to set a maximal duration and set an idle action. Thus, the macro-actions have the same duration. What is the limitation of the easiest method?

Since the formulation is similar to the settings of hierarchical RL, I am curious about the performance of existing hierarchical MARL methods although they do not specifically consider the asynchronicity problem. 

The hardware experiments are fancy. Although there is a video in Appendix, it is hard to intuitively get the effectiveness of the proposed methods from Figure 5. 
limitations: The authors have discussed the limitations.
review_title: null

Meta-review: metareview: All reviewers appreciated the clarity of writing, thorough evaluation and effectiveness of the proposed method. Some reviewers had concerns about the novelty of proposed ideas, however I feel these have been satisfactorily addressed in the authors' rebuttal. Thus, I recommend acceptance.
confidence: Certain
Decision: Accept