Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on an interesting and important question: how to perform distributed multi-agent deep reinforcement learning? The author first proposed three challenges to be considered: 1) Demanding data transfer. 2) Inter-process communication. 3) Effective Exploration. Further, the author proposed a container-based distributed marl framework, by placing the part that interacts with the environment in the container relieves the pressure of the cpu, and at the same time encourages different containers to have different policies and uses PER to select high-value samples, thereby improving training efficiency.
main_review: Although this paper focuses on a very interesting issue, there are still some places in this paper that make me confuse. At present, I think the author needs to further clarify my confuse.

Strengths:
1) The field of the paper is very important and interesting
2) The writing of the paper is clear, and the method is intuitive and effective

Weaknesses:
1) The author raised the problem of demanding data transfer at the beginning, especially when the GPU and CPU copy each other. I was very excited about this at the beginning, because this problem is very important. This problem exists in all (MA) RL framework based on cpu simulation, unless the environment is run on the GPU or the transmission effect between the CPU and the GPU is improved from the hardware. Otherwise, there is no particularly good method intuitively. But then I was very disappointed. The author did not solve this problem. The author just placed the environmental interaction part in the container. In fact, once the GPU is used for deep learning training, both the container and the centralizer need to copy the data of the cpu and gpu.
2) For the Inter-process communication problem, the author proposed multi-queue manager and buffer manager to optimize, so as to meet the author's proposal: "A critical design consideration of a successful distributed reinforcement learning framework is the uninterrupted learning of learners. ". However, the author lacks a detailed discussion of this module. Judging from the current version, the author only used "signal" to control read and write conflicts. This seems intuitive, but it does not satisfy the "uninterrupted learning of learners".
3) For the problem of effective exploration, the author adopts an MI-based method to encourage the container to have different policies for efficient exploration. Some methods under this part of single agent RL have already had similar work, such as DIYAN [1], etc., and the use of similar population exploration strategies has also been studied in single agent rl, such as Fig 2 in [2].
4) The author uses KL of different policies to constrain different policies to have different (exploratory) behaviors. From the current version, this part of the policies is in different containers. Intuitively, this will increase the load after parallelization. However, the author does not discuss.
5) Although the author emphasizes that their framework is distributed MARL, the author's distributed approach is to treat the marl problem as a joint single agent, which means that different single agnet distributed training schemes are also a very important comparison  method, but the author did not compare.
6) The author claims to have solved google football. After I checked the author's code, I have a doubt to be clarified: the author uses "scoring" to calculate the reward, which means that the positive reward will be very sparse, so that PER can be efficiently filter out "useful" trajectories, but the comparison method does not have a similar mechanism, which seems very unfair.
7) It is also the experimental part. The current version seems to be that the author’s method uses more resources for training. For better comparison, the control experiment should also have corresponding computing resources, such as using more parallel environments. .

[1] DIVERSITY IS ALL YOU NEED: LEARNING SKILLS WITHOUT A REWARD FUNCTION

[2] Interactive Parallel Exploration for Reinforcement Learning in Continuous Action Spaces

summary_of_the_review: Although this paper focuses on a very interesting issue, there are still some places in this paper that make me confuse. At present, I think the author needs to further clarify my confuse.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper introduces a new distributed value-based multi-agent reinforcement learning framework to solve problems faced by multi-agent tasks. It divides the system into two parts, multiple containers, and one centralizer. Containers are trained with trajectories generated by their own actors interacting with the environment. In contrast, the centralizer is trained with high-priority samples sent by all containers, easing the demanding data transfer. Besides, this paper designs a multi-queue manager between actors and replay buffer to avoid inter-process communication blocking. Furthermore, this paper proposes a new loss function encouraging different containers to be diversified to promote exploration. Finally, it achieves better results on both Google Research Football and StarCraft II micromanagement benchmark.
main_review: Edit: Thanks for providing the clarifications and new experimental results. However, I'm still not convinced of the new experimental results. According to fig.6 and fig.7, the result of QMIX_BETA_v2 without PER is much better than QMIX_BETA with PER, so I recommend all the experiments on both SMAC and GRF taking pymarl-v2 as the benchmark. I believe that the paper could be considerably improved. I keep my score.  

**Strengths**:

- The motivation is clear. The paper clearly argues that a successful framework for distributing acceleration has been missing from prior work on multi-agent RL. 
- The implementation of distributing has some technical novelties. 
- The architecture consisting of container and centralizer can ease the demanding data transfer and accelerate the training process. 
- With lots of training resources, this method can obtain better experimental results. 
- This paper is also well organized and clearly written. 

**Weaknesses**:

- In Subsection 2.3, the approximate calculation of $p(\mathbf{a}_t \mid \mathbf{\tau}_t)$ is wrong. If we approximate $p\left(\mathbf{a}_{t} \mid \mathbf{\tau}_{t}\right)=\frac{1}{N} \sum_{id=1}^{N} p\left(\mathbf{a}_{t} \mid \mathbf{\tau}_{t}, id\right)$, with the Boltzmann SoftMax distribution replacement of $p\left(\mathbf{a}_{t} \mid \mathbf{\tau}_{t}, id\right)$ and $\mathbf{\pi}_{i d}\left(\mathbf{a}_{t} \mid \mathbf{\tau}_{t}\right)=\Pi_{i=1}^{n} \pi_{id}\left(a_{t}^{i} \mid \tau_{t}^{i}\right)$, the approximate equation should be $p(\mathbf{a}_t | \mathbf{\tau}_t) \approx \frac{1}{N} \sum_{j=1}^{N}\left(\Pi_{i=1}^{n} \pi_{j}\left(a_{t}^{i} \mid \tau_{t}^{i}\right)\right)$, not $\Pi_{i=1}^{n}\left(\frac{1}{N} \sum_{j=1}^{N} \pi_{j}\left(a_{t}^{i} \mid \tau_{t}^{i}\right)\right)$. Moreover, there is a minor mistake, that Eqn.(6) uses the symbol $\pi_{id}\left(a_{t}^{i} \mid \tau_{t}^{i}\right)$, while the parameter explanation of Eqn. (5) uses $\pi_{i d}^i\left(a_{t}^{i} \mid \tau_{t}^{i}\right)$, which should be the same.

- The implementation encouraging diversity among containers is similar to CDS[2]. Eqn. (3) is exactly the same, with the different meaning of "id". CDS aims to encourage diversity among agents, while CMARL aims to diversify containers. Meanwhile, based on the analysis in (1), Eqn. (6) and Eqn. (7) need to be corrected.
- The design of a multi-queue manager is equivalent to a two-level replay buffer, which I think is an implemented trick. When the learner samples data, the prepacked batch data are sent to it, making the two processes, leaner sampling data and actor inserting data, complementary interference. However, when prepacking in a multi-queue manager, it also faces the problem of the mutual exclusion of reading and writing to the same storage area, which is inevitable. Therefore, the role of the multi-queue manager is mainly related to the frequency of inserting and sampling experience, i.e., the number of actors, network size, and so on. On the other side, it also brings a new problem: when the learner performs data sampling, the newly generated data cannot be collected in the first time.
- Only trajectories with high priority are sent to the centralizer learner. The computation of the priority is $p_{\tau}=\operatorname{Normalize}\left(\sum_{t} r_{t}\right)+\epsilon$, where $\operatorname{Normalize}\left(X\right) = \frac{X-L}{H-L}$. However, It's unclear that why to use this computation to judge the quality of experience. And I also wonder about the result of QMIX using the same method to screen experience, with multiple actors generating trajectories.
- Finally, I think it's unfair to compare CMARL and other baseline algorithms with the same training time. CMARL uses more resources to train a better model. For example, QMIX only has 1 actor, but CMARL has 39 actors, which means during the same time, the data generated by CMARL is 39 times higher than QMIX. Besides, because of the diversity of different containers, the centralizer and containers have different parameters, so the number of parameters is also higher than QMIX. I only take QMIX as an example, and other baseline algorithms also face the same problems. The paper RIIT[1] recently shows that QMIX could achieve SOTA performance among all SMAC maps with some code level modification only, including the number of actors. So, I recommend the authors reimplement based on pymarl-v2 (I call the optimized version of pymarl by Jian Hu in paper RIIT[1] as pymarl-v2) and show how it is used performs.


[1] Hu J, Jiang S, Harding S A, et al. RIIT: Rethinking the Importance of Implementation Tricks in Multi-Agent Reinforcement Learning[J]. arXiv preprint arXiv:2102.03479, 2021.

[2] Chenghao Li, Chengjie Wu, Tonghan Wang, Jun Yang, Qianchuan Zhao, and Chongjie Zhang. Celebrating diversity in shared multi-agent reinforcement learning. arXiv preprint arXiv:2106.02195, 2021. 
summary_of_the_review: I think this paper introduces a new distributed value-based multi-agent reinforcement learning framework to ease the demanding data transfer and accelerate the training process, which is a benefit of large-scale training. However, this paper has some fatal flaws:
- The approximate equation in subsection 2.3 is wrong.
- The diversity design lacks innovation with a fatal error.
- The design of a multi-queue manager needs to weigh the pros and cons.
- The basis for computing priority is unclear, and another ablation study is necessary.
- The experiment needs a more convincing comparison.

Some other comments: Figure 1 need more clarification and refinement. Too many colors are used and the pipeline is unclear. Its title also has no explanations.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a distributed containerized multi-agent reinforcement learning(CMARL) framework that addresses three challenges in MARL: intra-agent observation data transfer, inter-process communication,and efficient coordinated exploration amongst the agents.  Using a container that collects environment experiences from parallel actors into buffers and learns local policies, CMARL demonstrates notable performance improvements with respect to time as compared to state-of-the-art benchmarks. 

main_review: Strengths:
The presentation of the paper is mostly clear, with each section describing exactly what the corresponding component in CMARL does. Graphical illustrations are also helpful in showcasing the neural network structures. 
The experiment setup is detailed enough for reproduction, and the algorithm has been tested across different environments with respect to the current state-of-the-art algorithmic models. Ablation studies have been carried out on different components of CMARL. 


Weaknesses:
The degree of novelty seems relatively low. On a high level, CMARL resembles a hierarchical stack of two QMIX-like MARL algorithms, with container-level outputs of Q values serving as the input of higher-level centralizer. The multi-queue manager doesn’t seem to be too different from the standard prioritized experience replay mechanism, and more explanations on the novelty of the paper is needed.   
It is unclear how the container achieves ‘diversity’ in Section 2.3. The additional mutual information loss doesn’t provide much intuition on how the experiences will vary. 
While the experiments cover a broad range of environments, it would still be ideal to incorporate additional graphs(e.g. Goal difference vs epoch in addition to goal difference/win rate vs time) to complement existing results. Moreover, the ablation studies seem inadequate - why the specific choices of parameters? How do the ablation scenarios in Table 1 compare to each other, when the total number of agents seem to vary in different cases? More explanations should be provided. 

summary_of_the_review: The paper does lead to some improvement in the field of multi-agent reinforcement learning, although the techniques adopted seem relatively incremental. Furthermore, explanations on experimental parameter choice and novelty should be included.  I’d be happy to change my scores, if the authors address such concerns accordingly.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose a framework for distributed multi-agent reinforcement learning. By grouping actors, replay buffers, learners, and a queue manager into containers CMARL can be used to scale learning to the requirements of the multi-agent setting.
main_review: 
 ~ Positives
 - The paper is well written and conveys its ideas clearly. The method is described in a way that is clear, motivates most of the architectural decisions, and seems to be reproducible. 
 - The authors provide code with their submission which makes the experiments reproducible. The code is built on top of the PyMARL repository which is well-known and easy to use.
 - Encouraging diversity between containers, to my knowledge, is novel. It also seems to offer a significant advantage when compared to the no-diversity baseline in the experiments. I'd be interested in further experiments regarding diversity, e.g.:
 	- is there a number of containers after which diversity doesn't matter? Experiments seem to use two or three containers. If you were to use a much larger number, would that affect the usefulness of the diversity objective?
 	- Some layers are shared across the containers and the centraliser. If they weren't shared, would they more easily develop diverse policies? Since containers use local learners I expect some diversity even without the specific loss.
 	
 
 ~ Negatives
 
- My main issue, and the reason I cannot at this stage recommend acceptance of this work, is the presentation of the experiments section. The presented experiments use time (in minutes) in the x'x axis - and this leads to CMARL looking significantly better than the baseline methods. Specifically, in figure 4, many of the baselines do not appear to learn at all and appear to converge to significantly lower returns than CMARL. However, this is not the case: Papoudakis et al[1] show QMIX (and other methods) to completely solve (close to 100% win-rate) tasks such as 3s5z and MMM2. I understand that it might be the case that those algorithms are significantly slower in wall-clock time, and might just not have enough time to complete their training, but it should still be clear from the figures that they do, in fact, converge to much higher values than shown.
 
This leads me to the second issue of the experiments section: the selected baselines were not supposed to be fast, but rather sample efficient. In the related works section, the authors discuss distributed algorithms that focus on speed. I believe comparisons with these methods would be fair and offer a better comparison with existing literature. For instance, IMPALA implementations in MARL exist and could be applied directly to SMAC and possibly GRF.
 
 That said, my knowledge of the GRF environment is limited, but the results do seem exciting. 
  
- I am unsure how the presented method makes use of the multi-agent nature of the problem. Yes, the underlying algorithm is multi-agent (QMIX), but my understanding is that the containerized solution is agnostic to the algorithm. Is it also possible to apply this as-is in single-agent settings or have I misunderstood this? 
If the multi-agent setting is only used as a motivation to the increased observation sizes/data requirements, then the experiments should justify this. But now, the observation sizes of these environments should be smaller compared to image-based single-agent environments. 
 
 ~ Other 
 
 - page3/first line: It is not typical for MARL to also include the global state. It is indeed very common in the centralised training/decentralised execution paradigm, but there are plenty of multi-agent settings where global state is never observed. I believe the phrasing here should change.
 
 - Sharing (some) parameters between the global and local learners is an interesting idea and is motivated in the paper. I would be interested in seeing how parameter sharing across agents is handled here. In QMIX parameters are always shared across agents [2], is this also the case for CMARL? The authors could discuss how containers can handle parameter sharing across agents.
 
 
 [1] Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, & Stefano V. Albrecht. Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks, Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS), 2021

 [2]Tabish Rashid, Mikayel Samvelyan, Christian Schroeder Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine Learning 2018.

summary_of_the_review: The paper presents clearly some interesting ideas that improve the training time of MARL algorithms. However, the presentation of their results is lacking in both baselines and in metrics (convergence or sample efficiency). At this stage, I cannot recommend this paper for acceptance.
review_title: null

Meta-review: This paper proposes a distributed containerized multi-agent reinforcement learning(CMARL) framework that addresses three challenges in MARL: 1) Demanding data transfer. 2) Inter-process communication. 3) Effective Exploration. Using a container that collects environment experiences from parallel actors into buffers and learns local policies, CMARL demonstrates notable performance improvements with respect to time as compared to state-of-the-art benchmarks.

Although the reviewers acknowledge that the paper addresses a relevant topic, proposes an effective method, and is well written, after reading the authors' feedback and discussing their concerns, the reviewers reached a consensus about rejecting this paper in its current form. They feel that the contribution is too incremental and that the experimental comparisons are somehow unfair.

I suggest the authors take into consideration the reviewers' suggestions while preparing an updated version of their paper for one of the forthcoming machine learning conferences.
Decision: Reject