Containerized Distributed Value-Based Multi-Agent Reinforcement Learning
Authors: Siyang Wu
Tonghan Wang
Chenghao Li
Chongjie Zhang
Venue: ICLR 2022 Submitted (2022)
Keywords: Multi-agent reinforcement learning
Distributed reinforcement learning

Abstract: Multi-agent reinforcement learning tasks put a high demand on the volume of training samples. Different from its single-agent counterpart, distributed value-based multi-agent reinforcement learning faces the unique challenges of demanding data transfer, inter-process communication management, and high requirement of exploration. We propose a containerized learning framework to solve these problems. We pack several environment instances, a local learner and buffer, and a carefully designed multi-queue manager which avoids blocking into a container. Local policies of each container are encouraged to be as diverse as possible, and only trajectories with highest priority are sent to a global learner. In this way, we achieve a scalable, time-efficient, and diverse distributed MARL learning framework with high system throughput. To own knowledge, our method is the first to solve the challenging Google Research Football full game $\mathtt{5\_v\_5}$. On the StarCraft II micromanagement benchmark, our method gets 4-18$\times$ better results compared to state-of-the-art non-distributed MARL algorithms.

Under review as a conference paper at ICLR 2022
CONTAINERIZED DISTRIBUTED MULTI-AGENT REINFORCEMENT LEARNING
VALUE-BASED
Anonymous authors Paper under double-blind review
ABSTRACT
Multi-agent reinforcement learning tasks put high demand the volume of training samples. Different from its single-agent counterpart, distributed value- based multi-agent reinforcement learning faces the unique challenges of demanding data transfer, inter-process communication management, and high requirement exploration. We propose a containerized learning framework to solve these problems. We pack several environment instances, local learner and buffer, and a carefully designed multi-queue manager which avoids blocking into a container. Local policies of each container are encouraged to be as diverse possible, and only trajectories with highest priority are sent to a global learner. In this way, achieve a scalable, time-eflicient, and diverse distributed MARL learning framework with high system throughput. To own knowledge, our method is the first to solve the challenging Google Research Football full game 5 v 5. On the StarCraft II micromanagement benchmark, our method gets 4-18x better results compared to state-of-the-art non-distributed MARL algorithms.
INTRODUCTION
Deep reinforcement learning (DRL) has been proved effective in various complex domains, includ- ing the game of Go (Silver et al. 2017). Dota (OpenAl, 2018), and StarCraft II (Vinyals al.. 2017). However, as problems increase in scale, the training of DRL models is increasingly time- consuming (Mnih et al., (016)) which is partly because complex tasks typically require large amounts of training samples. Distributed learning provides promising direction to significantly reduce training costs by allowing the RI. learner to effectively leverage massive experience collected by large number of actors interacting with different environment instances.
Compared to its single-agent counterparts, multi-agent tasks involve more learning agents and a larger search space consists of joint action-observation history. Therefore, multi-agent reinforcement learning (MARL) typically requires more samples (Kurach et al. (020) and more consuminn,,ng However, most of the impressive progress achieved by distributed DRL focuses the single-agent setting and leaves the efficient training of MARL algorithms largely unstudied. To makeup for this shortage, in this paper, we study distributed value-based multi-agent reinforcement learning. We first pinpoint three unique challenges posed by the increasing number of agents in multi-agent settings listed follows.
(1) Demanding data transfer. Coming with the increased number of agents is larger volume of experience modern MARL algorithms typically require local observation history of all agents apart from global states. If we deploy environments on CPUs like state-of-the-an single-agent distributed DRL algorithms (Espeholt al. 2019), the data transfer between CPUs and GPUs will consume significant CPU computation utility. (2) Inter-process communication. Inter-process communication between environments, buffers, and learners is more likely to block in multi-agent settings. This because the experience volume is large and reading/rewriting becomes Exploration. The search space multi-agent problems grows exponentially with the number of agents. Learning performance of MARL algorithms largely rely on efficient exploration in such large space.
time-consuming. (3)
In this paper, propose containerized distributed value-basse framework (CMARL) to address these problems. We pack several actors interacting with environ- ments, local learner, a local buffer, and carefully designed multi-queue manager into container.
reinforcement learning

---

Under review as a conference paper at ICLR 2022
Container
Container
envy
eR
envy
aetor
actor
actor
Multi queue
Manager
Multi-queue Manager
Initial Priority
Calculator
Buffer Manager
Butfer Manager
Prioritized Replay Buller
Prioritized Replay Buffer
Compute Value:
Computo Value:
PC1
ebs
GRU
obs
Leamer
Leamnr
Centralizer
Container
Figure I: Containerized value-based multi-agent reinforcement learning architecture.
container interacts with its environment instances, collects data, avoids block via its multi-queu manager, and actively updates its local policy. Data with high priority is transferred to global learner that brings together the most talented and valuable experience to train target policy. We further share the parameters of shallow layers for local and global learners to accelerate training while encouraging deep layers of local learners to be as diverse as possible to improve exploration.
The advantages of the CMARL framework are as follows. (1) A container can be deployed either CPUs or GPU, making our method scalable and adaptive with the available computational resources. (2) Largely reduced data transfer. Data collection and prioritization happens within the container, and only highly prioritized data transferred. (3) Multi-queue managers work asynchronously from policy learning, enabling efficient and unblocked inter-process communication when data collection. (4) Diverse behavior enables efficient exploration in the large search space.
In this way, CMARL makes scalable, fime-efficient, and diverse distributed MARL framework with high system throughput. We empirically evaluate the performance of our architecture on both Google Research Football (Kurach al. (020) and StarCraft micromanagement challenges (Samvelyan et aL. 2019). Our method is the only algorithm that can obtain a positive goal difference on the three GRF tasks, and is the first algorithm which can solve the challenging 5_ 5_v_5 full game, the SMAC benchmark, our method obtains 4-18x x better results compared against state-of -ofte--o-th-ar non-distributed MARL algorithms.
METHODS
In this section, we present our novel containerized framework (Fig. D for distributed MARL. We consider fully cooperative multi-agent tasks that can be modelled as Dec-POMDP (Oliehoek) et al. 2016) consisting of a tuple G=(T, S, A, P. R, Ωt O, O,n, 7). where is the finite set of n agents, 0, 1] the discount factor, and S is the true state of the environment. At each timestep. each agent receives an observation 0; E Q drawn according to the observation function O(s, i) and selects action a; A, forming joint action E An, leading
next state s' according the transition function p(s')s |s, a), and observing a reward r = R(s, a) shared by all agents. Each agent (Ω A)"
has local action-observate history T;
Distributed deep reinforcement learning aims to provide a scalable and infecefficient computational framework. In the multi-agent setting, distributed RI encounters unique challenges.
2

---

Under review as a conference paper at ICLR 2022
(1) The experience of MARL agents typically consists of local observations, and actions of all agents. Moreover, the centralized training with decentralized execution paradigm (Foerster et al. 2017] Rashid et al. 2018) requires global states that contain information of all agents. Therefore, the size of experience grows quadratically with the number of agents. Consequently, transferring experience across different devices is very consuming, leading very high CPU usage. Meanwhile, large volume experience may also cause block in pnter-process communication. In order to reduce such overhead, we pack several actors and environments into one container and design a multi-queue manager to avoid blocking. The container only sends portion of experience to the centralized learner, and the centralized learner learns global policy from these experiences. The whole architecture consists of several containers and one centralized learner.
(2) Another challenge of multi-agent reinforcement learning is that the action-observation space grows exponentially with the number of agents, posing great challenge to exploration. Multiple containers allow us to do diverse exploration a way that each container learns an individual policy different from others and uses that policy explore.
Therefore, the proposed containerized framework (Fig. 1 holds the promise to solve these unique challenges of distributed multi-agent reinforcement learning. However, to realize this goal, many structural details need to be designed for each framework component, including the container, the centralized learner, and the training scheme. We now describe them in greater detail.
2.1 CONTAINER
Inside each container, there are & actors interacting with & environment instances collect experience, one container buffer storing experience, and one container learner training the container policy with batches of trajectories sampled from the local buffer.
A critical design consideration of a successful distributed reinforcement learning framework is the uninterrupted learning of learners. To this end, we need constantly sample batches from the container buffer. Meanwhile, the buffer needs to update itself with new experience constantly. In order to well manage these two operations and avoid read/write conflicts, we introduce a buffer manager, which is the only process controlling the buffer.
The buffer manager repeatedly inserts new experience and samples batches. Directly sending new experience from actors to the buffer manager has two shortcomings. (1) The new experience would be stacked the multi-process data-transfer queue when the buffer manager is sampling. In this case, actors have to wait, without collecting Consequently, experience collection slowed down. (2) Receiving trajectories one by one is slower than receiving trajectories in a batch. Moreover, receiving and inserting new experience frequently increases the waiting time between batch sampling. As sampling has to wait, the container learner has wait when there is no batch train. These two shortcomings become particularly problematic in multi-agent settings as trajectories consume more space in the multi-process data-transfer queue.
trajectories until the buffer finishes sampling.
In order tackle these issues, we introduce multi-queue manager. There a shared signal between the multi-queus and the buffer manager. The multi-queue manager constantly gathers new experience together unless the signal indicates that the buffer manager requires new batches. When the signal set, the multi-queue manager compacts all new experience it has gathered to batch and send them to the buffer manager. Since we use prioritized experience replay, this batch goes through an initial priority calculator before the buffer manager where its priority is calculated.
The priority of one trajectory is computed by Pr Normalize rt) + c, where Normalize X-L, and L, H are the lower and upper bound the sum of rewards in a whole trajectory, respectively. c css small constant that avoids zero probability during sampling.
2.2 CENTRALIZER
The centralizer has an architecture similar to the container (Fig. D. except that new experience comes from containers rather than actors. After containers' initial priority calculator receive batch of new experience and compute their priority, containers will send 17% of the experience to the centralizer's experience receiver. Transferred experience is sampled with probability proportional to the priority.
3

---

Under review as a conference paper at ICLR 2022
Here, 7) is real number between 0 and 100, indicating the fraction of experience to be sent to the centralized learner.
Although architecture can be combined with any value-based MARL algorithms, in this paper, we use QMIX (Rashid et al, 2018) as the underlying algorithm. The centralized learner updates a QMIX network. Specifically, agents share three-layer local Q-network, with GRU (Cho et al.. 2014) between two fully-connected layers, and the global Q value Qe a), parameterized by learned as a monotonic combination of local values. The centralized learner is updated by the following TD loss:
ErEB Ete [Qo(T,, ar) - (re + maxa Qe (81+1,,))]**
(0) = BB-
ErEBT,
where Tr is the length of trajectory T, Deen is the centralized buffer, the expectation means that batches sampled from Deen according the priority of trajectories, and 0' is parameters of target network copied from f every C Q-network updates.
2.3 ENCOURAGING DIVERSITY AMONG CONTAINERS
Using multiple containers to interact with the environment leads to more time-efficient experience collection. Training will be boosted in large exploration spaces posed by multi-agent tasks when the collected experience diverse. Such diversity can be achieved by letting containers act and explore differently from each other.
In our architecture, containers maintain a Q-network with the same architecture as the centralized learner. Although the architecture is the same, container Q-networks are learned individually and encouraged to be different. To explicitly encourage diversity among containers, in addition to the local TD loss, every container includes diversity objective in the loss function to maximize the mutual information between container id and its local experience
[T,, id) Er sa iaog p(T|id).
p(T)
We expand the mutual information as follows:
p(oolid)
+ Σ log p(aslTe. id) p(oo)
Σ log pl T ai id) p(o1+1/Tr, a:)
[TT, id) Er,id log
p(a:(T.))
Here, On is the initial observation, and its distribution is independent of container id. Therefore, p(oolid) - p(on). Similarly, p(0r+1|TT a:) is decided by the transition function and is the same for all containers. So we have p(0,+1|Te. O2, id) = p(00+1// [T,, az). It follows that
p(a.lT: id)
I(T, id) = Er.
Σ log
(4)
p(a,
However, p(a, id) typically distribution induced by e-greedy, distinguishing only the action with the highest probability and concealing most information about value functions. Therefore, we use the Boltzmann SoftMax distribution of local Q values to replace p(a, Te, id) and optimize lower bound of Eq.
Σ log
Tea (T)) p(a, Tx)
1(T, id) 2 Er
(5)
where (T) = -I] (a The inequality holds because DKL. [T (aa(aaaTT)))))) (T)) id) is
non-negative. We approximate p( p(a)/T)) I1 (a/l))) by p(a, (T) x II ( Σ ग. (a/l7/)).
where N is the number of containers. Then we get the lower bound of Eq. to optimize:
Tea (atlT=)
I(T, id) 2 Er.id
Σ
log
(6)
N Σ 7
4

---

Under review as a conference paper at ICLR 2022
Σ n
Ergid ,id
DKL.
ma)(1T=))
Tir
Σ
(117))
N
In practice, minimize the following loss for training the learner of the id-th container:
[(0;;)) (ro))))))rr( +
DKL
Aid (
Σ (11))
|
TEB
where 0 is the parameters of the container Q-function, is scaling factor, and X is factor controlling the value of the KI. divergence.
To balance diversity and learning sharing, split agent network in containers into two parts. The lower two layers are shared among the global learner and all containers. Training data is sampled from the global buffer, and weights are periodically copied to the containers every [global_ updote time seconds to avoid frequent transfer and unstable learning in containers. The last layer differs in each container and is updated locally, which enables containers act differently under the same observation.
RELATED WORKS
Distributed single-agent RL algorithms have been a popular research area in recent years, among which the search for distributed value-based methods has been quite fruitful. The first pioncering value-based architecture, GORILA (Nair et al.) 2013), adapts the Deep Q-Network (DQN) for distributing acceleration. It introduces several innovative ideas, such as isolated actors and learners running in parallel, distributed neural networks, and distributed replay buffer storage, but is not widely used for poor time efficiency and sample efficiency caused by asynchronous SGD. Later, APE-X (Horgan et al. 2018) accelerates training by separating data collection from learning, allowing massive scales of samples to be generated and prioritized in distributed manner. R2D2 (Kapturowski et al. 2018) studies distributed training of recurrent policies to fit partially observable environments better. SEED (Espeholt et al.) 2019) features a centralized inference interface within the learner module to reduce the cost of transmitting network weights frequently.
Another popular approach towards distributed single-agent RL is to adapt policy gradient methods to paralleled training. A3C (Mnih et al. (016) first introduces general asynchronous training architecture for actor-critic methods, which workers retrieve network weights from a parameter server, independently interact with the environment to caleulate gradients, and then send the gradients back to the parameter server for centralized policy updates. DPPO (Heess et al. 2017)) is the distributed implementation of PPO using similar method of distributed gradient gathering. On the other hand, IMPALA (Espeholt et al., 2018) introduces an off-policy correction method (called the V-trace method), where the actor-learner architecture is utilized, and experience trajectories are communicated instead of gradients. Compared to A3C, it displays desirable properties of stabler convergence, better scalability and higher robustness.
Reinforcement learning in multi-agent settings is becoming increasingly important for complex real-world decision- maasinnnmakinggg problems. and has been seeing significant progresses recently (Lowe et al.) 2017: Rashid et al. 2018; Wang et al,, 20200] 2021b). Li et al. (2021) encourages diversity for effiient multi-agent reinforcement learning and thus related our work. The difference lies in that we encourage diverse behavior among containers, instead of among agents as in (Li et al. 2021). evident that the training of multi-agent RL generally takes more time and computational resources than that of single-agent RL, and thus distributing acceleration is more urgently demanded. Nevertheless, the demand is largely ignored, as little existing literature has looked into the design of general distributed MARL architectures. In this paper, develop novel architecture to address this challenge, which significantly improves the scalability of distributed systems.
MARL algorithms on
5

---

Under review as a conference paper at ICLR 2022
4 EXPERIMENTS
Our experiment aimed at answering the following problems: (1) Can our containerized distributed framework accelerate the training of MARL agents? How much improvemnnt can it bring compared to non non-distributed state-of oftteoffth algorithms? (2) How does the time efficiency of training improves when given computational devices?
Baselines. We compare our method with various baselines shown in Table [ The baselines include state-of of- the-art multi-agen value decomposition methods (QMIX, Rashid et al. (0088) QPLEX, Wang et al. (2021 aa), identity-aware method (RODE, Wang al. (202Tb)). We show the median and variance of the performance for our method and baselines tested with five random seeds.
method (CDS, et al. (2021), ) and role-based
Table 1: Baseline algorithms and ablations.
Alg.
Description
QMIX
Rashid et al.
2018
Related
QPLEX
Wang et
2021a a
Works
RODE
Wang et al.
202Tbb
CDS
et al. (2021)
CMARL no diversity
actors explore with the centralised policy
CMARL_2 _onttinerrs
containers, each 13 actors
Ablations
CMARL I _onttinnr
container, 13 actors
CMARL 8 actors
containers, each 8 actors
CMARL 2 actors
containers, each actors
Other
QMIX-BEN
Parallel implementation of QMIX, 39 actors.
Distributed
APEX
APE-X (Hogaan et al. 2018) architecture, 10 actors
Medhos
APEX-overload
APE-X (Horgan al. 2018 architecture, 14 actors for corridor. 16 actors for 6h
_h__s vs 8z, 30 actors for GRF
Experiment scheme. We fine-tune hyperparameters corridor from the StarCraft II microman- agement (SMAC) benchmark (Samvelyan et al., (219) and academy_counterate coutterattttttcck saaaackkayyyyaaa from the GRF environment. Then we use the same hyperparameters other super hard SMAC scenarios and the same hyperparameters as to experiments other GRF environments. We did not tune hyperparameters for experiments easy and hard maps of the SMAC benchmark.
corridor to run experiments on
easy
4.1 OVERALL PERFORMANCE ON GOOGLE RESEARCH FOOTBALL AND STARCRAFT II
We benchmark our method 3 Google Research Football (GRF) scenarios (5 (5_vs, (5_vs_5, 5, academy counterattack _esiiee_aaa_____e and academy_ counterattack hard) and 11 scenarios from the SMAC benchmarks (four super hard scenarios: 3s5z_vs 1_62_vs_3362, 6h vs 8z, MMM2, corridor, two hard scenarios: 5m, vs 6m, 3s_vs_ vs _a__2, and five easy scenarios: 2s, vs_ 1sc, _s_vs___ vs_3z, 3s_ _2_, 1c3s5z, 10m vs 11m. To have an overview of our performance, we average performance on differ- ent scenarios for both GRF and SMAC. Because different scenarios needs different training time, we align learning curves by computing the ratio of the elapsed training time to the total training time for each scenario and then take the average. The results are shown in Fig. 2
For scenarios of GRF, we plot the average goal difference, which is the number of goals scored by MARL agents minus the number of goals scored by the other team. The overall performance is shown in Fig. left. After training for more than half of the total training time, our method starts to significantly outperforms baseline algorithms.
The overall performance of SMAC is shown in Fig. 2 right. The overall winning rate our method is at least four times better than any baseline algorithm (4x against QPLEX, 4.5x against QMIX, 10x against RODE, and 18x against CDS). We also notice that RODE and CDS perform better on super hard scenarios, but not easy maps, which lead to the overall unsatisfactory results.

---

Under review as a conference paper at ICLR 2022
CMARL
QMIX
OPLEX
CDS
RCDE
Acerage Over
Avaraee
0.25
100
0.30
03
0.05
0.00
-0.10
20
-0.15
370
62
Figure 2: Left: Average goal difference of our approach and baseline algorithms on 3 Google Research Football scenarios. Right: Average winning rate of our approach and baseline algorithms on 11 SMAC scenarios. The x-axis is the ratio of the elapsed training time to the total training time.
4.2 PERFORMANCE ON GOOGLE RESEARCH FOOTBALL
In this section, we test our method on three challenging Google Research Football (GRF) scenarios: academy_counteratten counterattack _eaattaccckkcckkk_aeeaaa academy senoddmy_counttrtteeee counterattack hard, and 5_vs_5. ___. the first two academy counterattach scenarios, four agents start from the opposition's half and attempt to score against one (easy) or two (hard) defenders and an opposing goalkeeper. In the 5_vs__ scenario, agents play 15-minute regular game, including offense, defense, out-of-bounds, and offside. This task has a very large search space. In GRF scenarios, the purpose of all agents' actions is to coordinate timing and positions sophisticatedly to seize valuable opportunities of scoring.
CMARL
QMIX
QPLEX
CDS
RODE academy counterattack
5 v
academy counterattack rasy
0.4
Goal Dif.
Goal Dif.
-0.2
-0.4
500
1,000
1500
2000
250
500
750
1000
1250
0.0
0
250
500
750
1000
1250
(min)
Time (min)
Time (min)
Figure Comparison of our approach against baseline algorithms on Google Research Football.
We show the performance comparison against baselines in Fig. 3 Our method outperforms in all the scenarios with stable training process. In the crademyccounteaaee
easy scenario and academy_ cacrlyycooutteeee www.ccouuurttte. touuteeattack cirrcckarrrdd_a scenario, the game terminates after our agents scoring or losing the ball, so the goal difference can only be 0 or RODE and CDS overcome the negative effeets of QMIX or QPLEX's monotonicity with roles and identity-aware diversity, respectively. However, these methods need significantly more time demonstrate their advantage. With the benefits of our containerized distributed framework, our method can sample considerably diverse trajectories. Our container priority calculator can further choose critical trajectories for training and sharing, significantly improving learning efficiency. In the $_vs__ scenario, the game ends only when the time step reaches the limit. As demonstrated Fig. 3 left, baselines struggle to learn sophisticated policies and suffer from performance degradation. Our method can stably scale to highly complex policies with the advantages of our containerized distributed framework.
4.3 PERFORMANCE ON STARCRAFT II
In this section, we test our approach on the StarCraft II micromanagement (SMAC) bench- mark (Samvelyan et al. 2019). In this benchmark, maps have been classified as easy, hard and super hard. As shown in Fig we present the results for four super hard scenarios: 3a5z_vs 3362,, 6h_va vs. _2, _a_vs__2, 2s vs 1sc, 2s vs 3z, 3s 52, 1c3s52. 10m 11m. For the four super hard scenarios, approach outperforms all baselines learning efficiency. Especially for 6h. vs 8z and MMM2, our method achieves remarkable performance in less then four hours, while baselines, such RODE and
MMM2, corridor. two hard scenarios: 5m vs 6m, 3s vs 52, and five easy scenarios:
7

---

Under review as a conference paper at ICLR 2022
CDS, achieve comparable performance in more than one day. This significant reduction in learning time indicates the great advantages of our method for practical application, which sheds light on introducing MARL algorithms to real world and for solving challenging problems. For other hard and easy scenarios, our method maintains its advantages of time efficiency and training stability, indicating our method's wide applicable range.
CMARL
QMIX
QPLEX
CDS
RODE
2s, vs, 1sc
253z
3s5z
%
100
Test Win %
100
de
100
Test Win
50
50
50
0
0
0
10
20
30
40
0
20
40
0
25
50
75
100
Time (min)
Time (min)
Time (min)
1c3s5z
10m vs. 11m
5m, vs. 6m
Win 96
100
Test Win %
100
Test Win %
100
50
50
20
100
150
Time (min)
Time (min)
Time (min)
35 ___vs_5z 5z
3s5z_ ___ 3562
6h_vs_ 82
100
100
100
Test Win
Test Win
Test Win
50
100
150
0
200
400
600
800
0
100
200
300
Time (min)
Time (min)
Time (min)
MMM2
corridor
Test %
100
Test Win %
100
50
50
100
200
0
200
400
600
800
Time (min)
Time (min)
Figure 4: Comparison of our approach against baseline algorithms on SMAC scenarios.
4.4 ABLATION STUDIES
Our framework mainly contributes two novelties: (1) The container architecture facilitating efficient data collection, transfer, and training of value functions. (2) Encouraging diversity of experience collected from different containers. In this section, we carry out ablation studies to show the contribution of these novelties to the final performance improvement provided by our method, which can explain why our method can efficiently explore and utilize computation resources. Ablation studies are carried out on one super map from the SMAC benchmark corridor and map from the GRF environment academy academy_counterane wownterattacc hard.
For (I), to test whether the container architecture can help utilize computation resources well, we conduct experiments by running our framework with different numbers of total actors (the sum of the number of actors in each container). There are two possible ways to change the total number of actors by changing the number of containers by changing the number of actors in each container. Specifically, we test with the following configurations: 1) 3 containers with 13 actors inside each, 2) 3 containers with 8 actors inside each, 3) 3 containers with 2 actors inside each, 4) 2 containers with 13 actors inside each and 5) 1 container with 13 actors inside each.
Results are shown in Fig. 5 On corridor, the performance improvement provided by the in- creasing number of actors significant when there 39 actors (CMARL). CMARL_ 8 actors and CMARL_2 2_ containers, with totally 24 and 26 actors, can explore winning strategies while other ablations with fewer actors can not win.
On academy academy_counterateeeet counterattack hard, perhaps interestingly, the performance of our method increases linearly with the total number of actors, regardless of the configuration. CMARL 39 actors can
8

---

Under review as a conference paper at ICLR 2022
win 30% of the games. CMARL actors and CMARL 2 containers, with totally 24 and 26 actors, achieve a win rate of around 20% after 1300 minutes of training.
For (2), to test whether exploring with different policies is efficient, carry out experiments by running our method with actors using centralised learner's policy, which is periodically copied from the centralised learner, to collect experience. Results shown in Fig. [ We can see that, on both corridor and academy_ counterattack hard, the ablation underperforms CMARL, consolidating the function of the diversity-enccourrgeenn learning objective.
CHAALL
CHARL Looraa
CHARL 2uttos CHARL O. actorr
CMARL divenity
CHARL 2.cort mart
conters.conttesstia Jard
60
200
606
333
230
3000
1206
Figure 5: Comparisons of ablations against CMARL.
4.5
COMPARISON WITH OTHER DISTRIBUTED METHODS
CMARL
APEX
APEX_overload
APEX overload
QMIX BETA
academy counterattack hard
vs
40
100
100
Test Win
Test Win
20
500
1000
0
100
200
300
200
400
600
800
(min)
(min)
(min)
Figure 6: Comparisons other distributed methods against CMARL.
We compare method against other distributed methods which use multiple actors to collect experience. Since there not much previous research in distributed MARL, for multi-agent distributed baselines, parallel QMIX (Rashid et al. 2018) with 39 actors collecting experience is tested on the same devices as our method. We call this method OMIX-BETA We also compare our methods with single-agent distributed RI. algorithms, APE-X (Horgan et al.) 2018). by directly applying it to MARL settings. Since the computation cost larger MARL,, when running APE-X with more than 14 actors corridor, 16 actors 6h
6h_vs vs 8z, and 30 actors GRF, the CPU will be overloaded. In the meantime, we also run
We call APE-X running with this amount of actors APE-X with 10 actors, where CPUs operate normally (does not overload). We call this baseline APEX.
Because QMIX-BETA's learner and actors cannot uninterruptedly learn or collect experience, cannot learn winning strategy in the given training time. APEX-overload has a very high CPU usage, so the learning and experience collecting is slow. As result, its performance is worse than APEX. APEX has a lower performance than CMARL on all environments, especially on corridor, where the size a trajectory is the biggest among three maps. This suggests that our method can well deal with large scale tasks.
CONCLUSION
In this paper, propose distributed learning framework for value-based multi-agent reinforcement learning. We use containerized architecture consists of several containers and a centralized learner. Each container interacts with several environment instances and use specially designed multi-queue

---

Under review as a conference paper at ICLR 2022
manager avoid blocking. Experience with high priority sent the centralized learning, that data transfer overhead is significantly reduced. Moreover, trajectories collected from each container encourage to be diverse possible. With these novelties, method achieves impressive performance on both GRF and SMAC benchmark.
The source code for all the experiments along with a README file with instruc- tions on how to run these experiments is attached in the supplementary material.
Reproducibility
REFERENCES
Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 1724-1734, 2014.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International Conference on Machine Learning, pp. 1407-1416. PMLR, 2018.
Lasse Espeholt, Raphaël Marinier, Piotr Stanczyk, Ke Wang, and Marcin Michalski. Seed Scalable and efficient deep-rl with accelerated central inference, In International Conference on Learning Representations, 2019.
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. arXiv preprini arXiv:/705.000 2017.
Nicolas Heess, Dhruva TB, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, SM Eslami, et al. Emergence of locomotion behaviours in rich environments. arXiv preprint arXiv: (007: 707.02286, 2017.
Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Mattco Hessel, Hado van Hasselt, and David Silver. Distributed prioritized experience replay. In International Conference Learning Representations, 2018
Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney. Recurrent experience replay in distributed reinforcement learning. In International conference on learning representations, 2018.
Karol Kurach, Anton Raichuk, Piotr Stanczyk, Michal Zajac. Olivier Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, et al. Google research football: A novel reinforcement learning environment. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 4501-4510, 2020.
Chenghao Li, Chengjie Wu, Tonghan Wang, Jun Yang, Qianchuan Zhao, and Chongjic Zhang. Cele- brating diversity in shared multi-agent reinforcement learning, arXiv preprint arXiv:2106.02111 2021.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperatve-coompettive environments. Advances in Neural Information Processing Systems, pp. 1339-63990 2017.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. International conference on machine learning, pp. 1928-1937. PMLR, 2016.
Arun Nair, Praveen Srinivasan, Sam Blackwell, Cagdas Alcicek, Rory Fearon, Alessandro De Maria, Vedavyas Panneershelvam, Mustafa Suleyman, Charles Beattie, Stig Petersen, et al. Massively parallel methods for deep reinforcement learning. arXiv preprint arXiv:1507.000 2015.
Frans A Olichock, Christopher Amato, et al. A concise introduction to decentralized POMDPs, volume 1. Springer, 2016.
10

---

Under review as a conference paper at ICLR 2022
OpenAl. five.
2018.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine Learning, pp. 4292 4301, 2018.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip H$ Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. arXiv preprint 1.KIvv1902.04044 2019.
David Silver, Thomas Hubert, Julian Schrittwieser, Joannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with general reinforcement learning algorithm. arXiv preprint arXiv:1712.01815. 2017.
Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yea, Alireza Makhzani, Heinrich Küttler, John Agapiou, Julian Schritwieser, al. Starcraft ii: new challenge for reinforcement learning. arXiv preprint 1708. 2017.
Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling multi-agent q-learning. International Conference on Leurning Representations (ICLR), 2021a
Tonghan Wang, Heng Dong, Victor Lesser, and Chongjie Zhang. Roma: Multi-agent reinforcement learning with emergent roles. In Proceedings of the 37th International Conference Machine Learning, 2020.
Tonghan Wang. Tarun Gupta, Anuj Mahajan, Bei Peng, Shimon Whiteson, and Chongjie Zhang. Rode: Learning roles to decompose multi-agent tasks. In Proceedings of the International Conference Learning Representations (ICLR), 2021b.
A COMPARE WITH PARALLELIZED PYMARLV2
We test parallel QMIX with 39 actors, run on the same device ours, and adopt all code level modifications of PyMARL-v2. We call this version academy counterattack hard to compare / We observe that after 1250 minutes training, QMIX QMIX-BETA-V achieves a win rate of about 10%, while ours can achieve 30%.
We run an experiment on
method against SMIX-BETA-v2 The result is
shown in Fig.
CMARL
QMIX_BETA v2 srademmycounterattnn hard
Test Win %
40
20
0
250
500
750
1000
1250
Time (min)
Figure 7: Comparisons of parallel PyMARL-v2 implementation of QMIX against CMARL.
B EXPEND BASELINE RUNNING TIME FOR SMAC
Baselines like RODE and CDS can solve super hard maps of SMAC. We extend the training time of these two baselines four super hard maps SMAC. The results is shown in Fig.
RODE is 1.25 times slower than our method on 3s52_ve 3s6z and 2.67 time slower on MMM2. has trend of learning to win, but the performance is still far below method on 6h 8z and corridor even with time expended. CDS also underperforms our method, being 1.5 times slower than our method on corridor and far below method 3a5z 3s6z, 6h vs 8z and MMM2.
11

---

Under review as a conference paper at ICLR 2022
CMARL
CDS
RODE
3s52 3366
6h v5 82
Test Win
200
400
600
800 (min)
1000
1200
1400
100
200
300
400
500
600
700
800
MMM2
corridar
100
100
80
80
Paryy
Test Win %
60
60
40
40
20
20
100
200
300
400
500
600
700
800
200
400
600
800
1000
1200
1400
Time (min)
Time (min)
Figure Expend running time of baselines.
12


Reviews:
[{"review_id": "uhCmppcy93", "reviewer": "Reviewer_axQN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Edit: Thanks for providing the clarifications and new experimental results. However, I'm still not convinced of the new experimental results. According to fig.6 and fig.7, the result of QMIX_BETA_v2 without PER is much better than QMIX_BETA with PER, so I recommend all the experiments on both SMAC and GRF taking pymarl-v2 as the benchmark. I believe that the paper could be considerably improved. I keep my score.  \n\n**Strengths**:\n\n- The motivation is clear. The paper clearly argues that a successful framework for distributing acceleration has been missing from prior work on multi-agent RL. \n- The implementation of distributing has some technical novelties. \n- The architecture consisting of container and centralizer can ease the demanding data transfer and accelerate the training process. \n- With lots of training resources, this method can obtain better experimental results. \n- This paper is also well organized and clearly written. \n\n**Weaknesses**:\n\n- In Subsection 2.3, the approximate calculation of $p(\\mathbf{a}_t \\mid \\mathbf{\\tau}_t)$ is wrong. If we approximate $p\\left(\\mathbf{a}_{t} \\mid \\mathbf{\\tau}_{t}\\right)=\\frac{1}{N} \\sum_{id=1}^{N} p\\left(\\mathbf{a}_{t} \\mid \\mathbf{\\tau}_{t}, id\\right)$, with the Boltzmann SoftMax distribution replacement of $p\\left(\\mathbf{a}_{t} \\mid \\mathbf{\\tau}_{t}, id\\right)$ and $\\mathbf{\\pi}_{i d}\\left(\\mathbf{a}_{t} \\mid \\mathbf{\\tau}_{t}\\right)=\\Pi_{i=1}^{n} \\pi_{id}\\left(a_{t}^{i} \\mid \\tau_{t}^{i}\\right)$, the approximate equation should be $p(\\mathbf{a}_t | \\mathbf{\\tau}_t) \\approx \\frac{1}{N} \\sum_{j=1}^{N}\\left(\\Pi_{i=1}^{n} \\pi_{j}\\left(a_{t}^{i} \\mid \\tau_{t}^{i}\\right)\\right)$, not $\\Pi_{i=1}^{n}\\left(\\frac{1}{N} \\sum_{j=1}^{N} \\pi_{j}\\left(a_{t}^{i} \\mid \\tau_{t}^{i}\\right)\\right)$. Moreover, there is a minor mistake, that Eqn.(6) uses the symbol $\\pi_{id}\\left(a_{t}^{i} \\mid \\tau_{t}^{i}\\right)$, while the parameter explanation of Eqn. (5) uses $\\pi_{i d}^i\\left(a_{t}^{i} \\mid \\tau_{t}^{i}\\right)$, which should be the same.\n\n- The implementation encouraging diversity among containers is similar to CDS[2]. Eqn. (3) is exactly the same, with the different meaning of \"id\". CDS aims to encourage diversity among agents, while CMARL aims to diversify containers. Meanwhile, based on the analysis in (1), Eqn. (6) and Eqn. (7) need to be corrected.\n- The design of a multi-queue manager is equivalent to a two-level replay buffer, which I think is an implemented trick. When the learner samples data, the prepacked batch data are sent to it, making the two processes, leaner sampling data and actor inserting data, complementary interference. However, when prepacking in a multi-queue manager, it also faces the problem of the mutual exclusion of reading and writing to the same storage area, which is inevitable. Therefore, the role of the multi-queue manager is mainly related to the frequency of inserting and sampling experience, i.e., the number of actors, network size, and so on. On the other side, it also brings a new problem: when the learner performs data sampling, the newly generated data cannot be collected in the first time.\n- Only trajectories with high priority are sent to the centralizer learner. The computation of the priority is $p_{\\tau}=\\operatorname{Normalize}\\left(\\sum_{t} r_{t}\\right)+\\epsilon$, where $\\operatorname{Normalize}\\left(X\\right) = \\frac{X-L}{H-L}$. However, It's unclear that why to use this computation to judge the quality of experience. And I also wonder about the result of QMIX using the same method to screen experience, with multiple actors generating trajectories.\n- Finally, I think it's unfair to compare CMARL and other baseline algorithms with the same training time. CMARL uses more resources to train a better model. For example, QMIX only has 1 actor, but CMARL has 39 actors, which means during the same time, the data generated by CMARL is 39 times higher than QMIX. Besides, because of the diversity of different containers, the centralizer and containers have different parameters, so the number of parameters is also higher than QMIX. I only take QMIX as an example, and other baseline algorithms also face the same problems. The paper RIIT[1] recently shows that QMIX could achieve SOTA performance among all SMAC maps with some code level modification only, including the number of actors. So, I recommend the authors reimplement based on pymarl-v2 (I call the optimized version of pymarl by Jian Hu in paper RIIT[1] as pymarl-v2) and show how it is used performs.\n\n\n[1] Hu J, Jiang S, Harding S A, et al. RIIT: Rethinking the Importance of Implementation Tricks in Multi-Agent Reinforcement Learning[J]. arXiv preprint arXiv:2102.03479, 2021.\n\n[2] Chenghao Li, Chengjie Wu, Tonghan Wang, Jun Yang, Qianchuan Zhao, and Chongjie Zhang. Celebrating diversity in shared multi-agent reinforcement learning. arXiv preprint arXiv:2106.02195, 2021. ", "summary_of_the_paper": "This paper introduces a new distributed value-based multi-agent reinforcement learning framework to solve problems faced by multi-agent tasks. It divides the system into two parts, multiple containers, and one centralizer. Containers are trained with trajectories generated by their own actors interacting with the environment. In contrast, the centralizer is trained with high-priority samples sent by all containers, easing the demanding data transfer. Besides, this paper designs a multi-queue manager between actors and replay buffer to avoid inter-process communication blocking. Furthermore, this paper proposes a new loss function encouraging different containers to be diversified to promote exploration. Finally, it achieves better results on both Google Research Football and StarCraft II micromanagement benchmark.", "summary_of_the_review": "I think this paper introduces a new distributed value-based multi-agent reinforcement learning framework to ease the demanding data transfer and accelerate the training process, which is a benefit of large-scale training. However, this paper has some fatal flaws:\n- The approximate equation in subsection 2.3 is wrong.\n- The diversity design lacks innovation with a fatal error.\n- The design of a multi-queue manager needs to weigh the pros and cons.\n- The basis for computing priority is unclear, and another ablation study is necessary.\n- The experiment needs a more convincing comparison.\n\nSome other comments: Figure 1 need more clarification and refinement. Too many colors are used and the pipeline is unclear. Its title also has no explanations. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "KSWittlgfgK", "reviewer": "Reviewer_HX1e", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Although this paper focuses on a very interesting issue, there are still some places in this paper that make me confuse. At present, I think the author needs to further clarify my confuse.\n\nStrengths:\n1) The field of the paper is very important and interesting\n2) The writing of the paper is clear, and the method is intuitive and effective\n\nWeaknesses:\n1) The author raised the problem of demanding data transfer at the beginning, especially when the GPU and CPU copy each other. I was very excited about this at the beginning, because this problem is very important. This problem exists in all (MA) RL framework based on cpu simulation, unless the environment is run on the GPU or the transmission effect between the CPU and the GPU is improved from the hardware. Otherwise, there is no particularly good method intuitively. But then I was very disappointed. The author did not solve this problem. The author just placed the environmental interaction part in the container. In fact, once the GPU is used for deep learning training, both the container and the centralizer need to copy the data of the cpu and gpu.\n2) For the Inter-process communication problem, the author proposed multi-queue manager and buffer manager to optimize, so as to meet the author's proposal: \"A critical design consideration of a successful distributed reinforcement learning framework is the uninterrupted learning of learners. \". However, the author lacks a detailed discussion of this module. Judging from the current version, the author only used \"signal\" to control read and write conflicts. This seems intuitive, but it does not satisfy the \"uninterrupted learning of learners\".\n3) For the problem of effective exploration, the author adopts an MI-based method to encourage the container to have different policies for efficient exploration. Some methods under this part of single agent RL have already had similar work, such as DIYAN [1], etc., and the use of similar population exploration strategies has also been studied in single agent rl, such as Fig 2 in [2].\n4) The author uses KL of different policies to constrain different policies to have different (exploratory) behaviors. From the current version, this part of the policies is in different containers. Intuitively, this will increase the load after parallelization. However, the author does not discuss.\n5) Although the author emphasizes that their framework is distributed MARL, the author's distributed approach is to treat the marl problem as a joint single agent, which means that different single agnet distributed training schemes are also a very important comparison  method, but the author did not compare.\n6) The author claims to have solved google football. After I checked the author's code, I have a doubt to be clarified: the author uses \"scoring\" to calculate the reward, which means that the positive reward will be very sparse, so that PER can be efficiently filter out \"useful\" trajectories, but the comparison method does not have a similar mechanism, which seems very unfair.\n7) It is also the experimental part. The current version seems to be that the author’s method uses more resources for training. For better comparison, the control experiment should also have corresponding computing resources, such as using more parallel environments. .\n\n[1] DIVERSITY IS ALL YOU NEED: LEARNING SKILLS WITHOUT A REWARD FUNCTION\n\n[2] Interactive Parallel Exploration for Reinforcement Learning in Continuous Action Spaces\n", "summary_of_the_paper": "This paper focuses on an interesting and important question: how to perform distributed multi-agent deep reinforcement learning? The author first proposed three challenges to be considered: 1) Demanding data transfer. 2) Inter-process communication. 3) Effective Exploration. Further, the author proposed a container-based distributed marl framework, by placing the part that interacts with the environment in the container relieves the pressure of the cpu, and at the same time encourages different containers to have different policies and uses PER to select high-value samples, thereby improving training efficiency.", "summary_of_the_review": "Although this paper focuses on a very interesting issue, there are still some places in this paper that make me confuse. At present, I think the author needs to further clarify my confuse.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9JQGNikZu0W", "reviewer": "Reviewer_Qy4m", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n ~ Positives\n - The paper is well written and conveys its ideas clearly. The method is described in a way that is clear, motivates most of the architectural decisions, and seems to be reproducible. \n - The authors provide code with their submission which makes the experiments reproducible. The code is built on top of the PyMARL repository which is well-known and easy to use.\n - Encouraging diversity between containers, to my knowledge, is novel. It also seems to offer a significant advantage when compared to the no-diversity baseline in the experiments. I'd be interested in further experiments regarding diversity, e.g.:\n \t- is there a number of containers after which diversity doesn't matter? Experiments seem to use two or three containers. If you were to use a much larger number, would that affect the usefulness of the diversity objective?\n \t- Some layers are shared across the containers and the centraliser. If they weren't shared, would they more easily develop diverse policies? Since containers use local learners I expect some diversity even without the specific loss.\n \t\n \n ~ Negatives\n \n- My main issue, and the reason I cannot at this stage recommend acceptance of this work, is the presentation of the experiments section. The presented experiments use time (in minutes) in the x'x axis - and this leads to CMARL looking significantly better than the baseline methods. Specifically, in figure 4, many of the baselines do not appear to learn at all and appear to converge to significantly lower returns than CMARL. However, this is not the case: Papoudakis et al[1] show QMIX (and other methods) to completely solve (close to 100% win-rate) tasks such as 3s5z and MMM2. I understand that it might be the case that those algorithms are significantly slower in wall-clock time, and might just not have enough time to complete their training, but it should still be clear from the figures that they do, in fact, converge to much higher values than shown.\n \nThis leads me to the second issue of the experiments section: the selected baselines were not supposed to be fast, but rather sample efficient. In the related works section, the authors discuss distributed algorithms that focus on speed. I believe comparisons with these methods would be fair and offer a better comparison with existing literature. For instance, IMPALA implementations in MARL exist and could be applied directly to SMAC and possibly GRF.\n \n That said, my knowledge of the GRF environment is limited, but the results do seem exciting. \n  \n- I am unsure how the presented method makes use of the multi-agent nature of the problem. Yes, the underlying algorithm is multi-agent (QMIX), but my understanding is that the containerized solution is agnostic to the algorithm. Is it also possible to apply this as-is in single-agent settings or have I misunderstood this? \nIf the multi-agent setting is only used as a motivation to the increased observation sizes/data requirements, then the experiments should justify this. But now, the observation sizes of these environments should be smaller compared to image-based single-agent environments. \n \n ~ Other \n \n - page3/first line: It is not typical for MARL to also include the global state. It is indeed very common in the centralised training/decentralised execution paradigm, but there are plenty of multi-agent settings where global state is never observed. I believe the phrasing here should change.\n \n - Sharing (some) parameters between the global and local learners is an interesting idea and is motivated in the paper. I would be interested in seeing how parameter sharing across agents is handled here. In QMIX parameters are always shared across agents [2], is this also the case for CMARL? The authors could discuss how containers can handle parameter sharing across agents.\n \n \n [1] Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, & Stefano V. Albrecht. Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks, Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS), 2021\n\n [2]Tabish Rashid, Mikayel Samvelyan, Christian Schroeder Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine Learning 2018.\n", "summary_of_the_paper": "The authors propose a framework for distributed multi-agent reinforcement learning. By grouping actors, replay buffers, learners, and a queue manager into containers CMARL can be used to scale learning to the requirements of the multi-agent setting.", "summary_of_the_review": "The paper presents clearly some interesting ideas that improve the training time of MARL algorithms. However, the presentation of their results is lacking in both baselines and in metrics (convergence or sample efficiency). At this stage, I cannot recommend this paper for acceptance. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4BrbPrpuyD7", "reviewer": "Reviewer_uCBF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThe presentation of the paper is mostly clear, with each section describing exactly what the corresponding component in CMARL does. Graphical illustrations are also helpful in showcasing the neural network structures. \nThe experiment setup is detailed enough for reproduction, and the algorithm has been tested across different environments with respect to the current state-of-the-art algorithmic models. Ablation studies have been carried out on different components of CMARL. \n\n\nWeaknesses:\nThe degree of novelty seems relatively low. On a high level, CMARL resembles a hierarchical stack of two QMIX-like MARL algorithms, with container-level outputs of Q values serving as the input of higher-level centralizer. The multi-queue manager doesn’t seem to be too different from the standard prioritized experience replay mechanism, and more explanations on the novelty of the paper is needed.   \nIt is unclear how the container achieves ‘diversity’ in Section 2.3. The additional mutual information loss doesn’t provide much intuition on how the experiences will vary. \nWhile the experiments cover a broad range of environments, it would still be ideal to incorporate additional graphs(e.g. Goal difference vs epoch in addition to goal difference/win rate vs time) to complement existing results. Moreover, the ablation studies seem inadequate - why the specific choices of parameters? How do the ablation scenarios in Table 1 compare to each other, when the total number of agents seem to vary in different cases? More explanations should be provided. \n", "summary_of_the_paper": "This paper proposes a distributed containerized multi-agent reinforcement learning(CMARL) framework that addresses three challenges in MARL: intra-agent observation data transfer, inter-process communication,and efficient coordinated exploration amongst the agents.  Using a container that collects environment experiences from parallel actors into buffers and learns local policies, CMARL demonstrates notable performance improvements with respect to time as compared to state-of-the-art benchmarks. \n", "summary_of_the_review": "The paper does lead to some improvement in the field of multi-agent reinforcement learning, although the techniques adopted seem relatively incremental. Furthermore, explanations on experimental parameter choice and novelty should be included.  I’d be happy to change my scores, if the authors address such concerns accordingly. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a distributed containerized multi-agent reinforcement learning(CMARL) framework that addresses three challenges in MARL: 1) Demanding data transfer. 2) Inter-process communication. 3) Effective Exploration. Using a container that collects environment experiences from parallel actors into buffers and learns local policies, CMARL demonstrates notable performance improvements with respect to time as compared to state-of-the-art benchmarks.

Although the reviewers acknowledge that the paper addresses a relevant topic, proposes an effective method, and is well written, after reading the authors' feedback and discussing their concerns, the reviewers reached a consensus about rejecting this paper in its current form. They feel that the contribution is too incremental and that the experimental comparisons are somehow unfair.

I suggest the authors take into consideration the reviewers' suggestions while preparing an updated version of their paper for one of the forthcoming machine learning conferences.
Decision: Reject