Centralized Training with Hybrid Execution in Multi-Agent Reinforcement Learning
Authors: Pedro Pinto Santos
Diogo S. Carvalho
Miguel Vasco
Francisco S. Melo
Alberto Sardinha
Pedro A. Santos
Ana Paiva
Venue: Submitted to ICLR 2023 (2023)
Keywords: Multi-Agent Reinforcement Learning

Abstract: We introduce hybrid execution in multi-agent reinforcement learning (MARL), a new paradigm in which agents aim to successfully perform cooperative tasks with any communication level at execution time by taking advantage of information-sharing among the agents. Under hybrid execution, the communication level can range from a setting in which no communication is allowed between agents (fully decentralized), to a setting featuring full communication (fully centralized). To formalize our setting, we define a new class of multi-agent partially observable Markov decision processes (POMDPs) that we name hybrid-POMDPs, which explicitly models a communication process between the agents. We contribute MARO, an approach that combines an autoregressive predictive model to estimate missing agents' observations, and a dropout-based RL training scheme that simulates different communication levels during the centralized training phase. We evaluate MARO on standard scenarios and extensions of previous benchmarks tailored to emphasize the negative impact of partial observability in MARL. Experimental results show that our method consistently outperforms baselines, allowing agents to act with faulty communication while successfully exploiting shared information.

Under review as a conference paper at ICLR 2023
CENTRALIZED TRAINING WITH HYBRID EXECUTION IN MULTI-AGENT REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
We introduce hybrid execution in multi- aagett reinforcement learning (MARL), new paradigm in which agents aim to successfully perform cooperative tasks with any communication level execution time by taking advantage of information- sharing among the agents. Under hybrid execution, the communication level can range from a setting in which no communication allowed between agents (fully decentralized), to setting featuring full communication (fully centralized). To for- malize our setting, we define a new class multi-agent partially observable Markov decision processes (POMDPs) that we name POMris-POMDPss which explicitly models a communication process between the agents. We contribute MARO, an approach that combines an autoregressive predictive model to estimate missing agents' observations, and dropout-base brasout-based RL training scheme that simulates dif- ferent communication levels during the centralized training phase. We evaluate MARO on standard scenarios and extensions of previous benchmarks tailored to emphasize the negative impact of partial observability in MARL. Experimental results show that our method consistently outperforms baselines, allowing agents to act with faulty communication while successfully exploiting shared information.
INTRODUCTION
Multi-agent reinforcement learning (MARL) aims to learn maximizin behavior in scenarios involving multiple agents. recent years, deep MARL methods have been successfully applied to multi-agent tasks such as game-playing (Papoudakis et al.) 2020), traffic light control (Wei et al. 2019), or energy management (Fang et al. 2020). Despite recent successes, the multi-agent setting happens to be substantially harder than its single-agent counterpart (Canese al. 2021)) because multiple concurrent learners can create non-stationarity conditions that hinder learning; the curse of dimensionality obstructs centralized approaches MARL due to the exponential growth in state and action spaces with the number of agents; and agents seldom observe the true state of the environment.
As way deal with the exponential growth in the state/action space and with environmental constraints, both in perception and actuation, existing methods aim to learn decentralized policies that allow the agents to act based local perceptions and partial information about other agents' intentions. The paradigm of centralized training with decentralized execution is undoubtedly at the core of recent research in the field OOichock et al. 2011 Rashid et al. 2018 Focrster al. 2016); such paradigm takes advantage of the fact that additional information, available only at training time, can be used learn decentralized policies in a way that the need for communication is alleviated.
While in some settings partial observability and/or communication constraints require learning fully decentralized policies, the assumption that agents cannot communicate at execution time often too restrictive for great number of real-world application domains such robotics, game-playing or autonomous driving (Ho et al. 2019 Yurtsever et al., 2020). In such domains, learning fully decentralized policies should be deemed inappropriate since such policies do not take into account the possibility communication between the agents. Other MARL strategies, which take advantage of additional information shared among the agents, can surely be developed (Zhu et al. 2022).
In this work, our objective to develop agents that are able to exploit the benefits of centralized training while, simultaneously, taking advantage of information-sharin at execution time. We introduce the paradigm of hybrid execution, in which agents act in scenarios with any possible communication level, ranging from communication (fully decentralized) to full communication

---

Under review as a conference paper at ICLR 2023
between the agents (fully centralized). In particular, focus on multi-agent cooperative tasks in which the sharing of local information (observations and actions of the agents) is critical to their successful execution. To formalize our setting, we start by defining hybrid partially observable Markov decision process (H-POMDP), a new class of multi-agent POMDPs that explicitly considers communication process between the agents. Our goal is to find method that allows agents to solve H-POMDPs regardless of the communication process encountered at execution time. To allow for hybrid execution, we propose an autoregressive model that explicitly predicts non-shared information from past observations of the agents. In addition, we propose training scheme for the agents' controllers that simulates communication faults during the centralized training phase. We denote our coupled approach by multi-agent observation sharing with communication dropout (MARO). MARO can be easily integrated with current deep MARL methods.
We evaluate the performance of MARO across different communication levels, in different MARL benchmark environments and using multiple RL. algorithms. Furthermore, we introduce three novel MARL environments that explicitly require communication during execution to successfully perform cooperative tasks, currently missing in literature. Finally, we perform an ablation study that highlights the importance of both the predictive model and the training scheme to the overall performance of MARÓ. The results show that our method consistently outperforms the baselines, allowing agents to exploit shared information during execution and perform tasks under various communication levels.
In summary, our contribution is three-fold: (i) we propose and formalize the setting of hybrid execution in MARL, in which agents must perform partially-observaa possible communication levels; (ii) we propose MARO, an approach that combines an autoregressive predictive model of agents' observations and a novel training scheme; and (iii) we evaluate MARO in different benchmark and novel environments, using different RL algorithms, showing that our approach consistently allows agents to act with different communication levels.
cooperative tasks across all
HYBRID EXECUTION IN MULTI-AGENT REINFORCEMENT LEARNING
A fully cooperative multi-agent system with Markovian dynamics can be modelled as a decentralized partially observable Markov decision process (Dec-POMDP) (Olichoek& & Amato 2016). A Dec- POMDP is tuple ([n).XX,AA A,,, P.r, Y, 2, O), where [n] = {I..... nn is the set of indexes of n agents, X' is the set of states of the environment, A = x;A; is the set of joint actions, where A; is the set of individual actions of agent i, P is the set of probability distributions over next states in X, one for each state and action in X' x A, X x A - R maps states and actions to expected rewards, y E [0, 1[ is a discount factor, = X, is the set of joint observations, where Z, is the set of local observations of agent i, and O is the set of probability distributions over joint observations in Z, one for each state and action in X x A. A decentralized policy for agent is T. : 2, + A, and the joint decentralized policy is π Z + A such that 7 ..,,......
2n.) = (TI (21 (z1)...... ... TT (2n)).
Fully decentralized approaches to MARL directly apply standard single-agent RL algorithms for learning each agents' policy æ; in decentralized manner. In independent Q-learning (IQL. (Tan) 1993), as well as in independent RL. algorithms (Schulman et al. 2017), each agent treats other agents as part of the environment, ignoring the influence of other agents' observations and actions. More recently, under the paradigm of centralized training with decentralized execution, QMIX (Rashid
al. 2018) and other MARL algorithms (Sunehag et al. 2017 Son al.) 2019 Mahajan al.
2019 Wang et al., 2020b Yu et al. 2021) aim at learning decentralized policies with centralization at training time while fostering cooperation among the agents. Finally, if agents can share their
observations, we can use either approach to learn fully centralized policies.
None of the classes of methods aforementioned assumes that agents may sometimes have access to other agents' observations and sometimes not. Therefore, decentralized agents are unable to take advantage of the additional information that they may receive from other agents at execution time, and centralized agents are unable to act when the sharing of information fails. In this work, we introduce hybrid execution in MARL, a setting in which agents act regardless of the communication process while taking advantage of additional information they may receive during execution. To formalize this setting, we define a new class of multi-agent POMDPs that we name (H-POMDPs), which explicitly considers specific communication process among the agents.
2

---

Under review as a conference paper at ICLR 2023
M
Environment
M
Eavironment
of
O
%
0
01
0
+
RL
LSTM
((())
History State
B(p)
171
(a) Predictive Model.
(b) Training Scheme.
Figure I: Our proposed MARO approach for hybrid execution: (a) we introduced an autoregressive predictive model M to estimate the next-step observations p of h1)) from the previous ones of and an history variable hy: (b) we introduce training scheme for RL controllers that randomly drops agent observations following the communication masks mf, sampled accordingly to the communication level p. More details in the main text.
2.1 HYBRID PARTIALLY OBSERVABLE MARKOV DECISION PROCESSES
We define an hybrid-POMDP POMDP (H-POMDP) as tuple ([n] A, 2,O,C) C) where, in addition to the tuple that describes the DeMMPO we consider an n x communication matrix C' such that [C],, Pt.j is the probability that, at certain time step, agent i has access to the local observation of agent j in 2,. H-POMDPs generalize both the notion decentralized execution and of centralized exccution in MARL. Specifically, for given Dec-POMDP: capture fully decentralized execution and matrix ones to capture fully centralized execution.
consider C' the identity matrix to
In setting, assume that at execution time agents will face H-POMDP with unknown communication matrix C (not provided to the agents), sampled from a set C according to an unknown probability distribution μ.. The performance of the agents is measured as (=) = EEcp [J(m; C)], where (T; C') denotes the expected discounted cumulative reward under an H-POMDP with com- munication matrix C. At training time, agents may have access to the fully centralized H-POMDP. Therefore, the setting we consider is one of centralized training with hybrid execution and unknown communication process.
We note here that every H-POMDP has a corresponding Dec-POMDP, which can be obtained by adequately changing the observation space Z and the set of emission probability distributions O. Consequently, any reinforcement learning method can be trained to solve specific H-POMDP, with specific communication matrix C, by solving the corresponding Dec-POMDP. However, we seek to find a method that is able to act on H-POMDPs regardless of the matrix C that models the communication process at execution time. To the best of our knowledge, there exists no documentation of method that solves our problem.
MULTI-AGENT OBSERVATION SHARING WITH COMMUNICATION DROPOUT
While acting H-POMDP, agents may not have access to all perceptual information due to faulty communication process. We propose MARO, a novel approach to exploit shared information and overcome communication issues during task execution. MARO is composed of two elements: an autoregressive predictive model that estimates missing information from previous observations, and a training scheme for the RL controllers that simulates faulty communication training time.
3.1 PREDICTIVE MODEL
The predictive model M, depicted in Fig. la used to estimate the local observations of all agents 0!!7! = of }. where of corresponds to the observation of the i-th agent, in order to overcome missing observations during execution. Thus, we lear transition model p(ot:) o! hh) that given
3

---

Under review as a conference paper at ICLR 2023
the current observations o!!" and some history variable h4 containing information regarding the policy of the agents, is able to predict the step observations 0++1. We instantiate po(o o, !aa he) as an LSTM, parameterized by 8, with:
7
Po po(0 of lan h1) II po polo (o(0) of tn 11)s
where po of
he) hh)) is the Gaussian distribution of the predicted observations of the i-th agent. We train the predictive model and RL controllers simultaneously: we consider single-step observation transitions (o! olm and evaluate the negative log-likelihood of the target next-step observation onn given the estimated next-step observation distribution po(- oo he):
Em(o
log poloo oz hh).
3.2 TRAINING SCHEME
We also introduce an RL training scheme, depicted in Fig. lb which simulates the communication process exccution time and is agnostic the type of algorithm. We setup all RL controllers to receive as input the joint observation o!!T! However, in our setting, some observations may not be shared at execution time. To overcome such issue, employ the predictive model to estimate the non-shared observations o/. with j E [n]] We also setup the controllers to additionally receive as input communication masks m,, binary vectors that indicate the real and predicted components of o3 The sequence of masks the controllers receive allows them implicitly estimate the communication level, as well as the eгrоr that has accumulated due to long periods without receiving information from other agents, and select actions accordingly. For that, the recurrent layers are critical.
During centralized training, methods typically assume that each agent has access to the local observa- tions of all other agents. Instead of using such information train the agents, we instead propose to explicitly simulate the communication conditions of execution time: we randomly dropout agent observations. At the beginning of each episode, we sample communication level p ((, 1) Given p. we build at each time-step a communication mask m {0, 1 for each agent E [m]. with my [i] 1. We sample the communication masks from independent Bernoulli distributions mibi ~ B(p), for j E [n]] \ i. At exccution time, extract the observation masks mi directly from the environment, according to the actual faults in communication.
The communication mask indicates which components of the agent-specific joint- obbervatbeevaioon o dropped. Specifically, use the real observation of if m [k] 1 and use the estimated observa- tion ot otherwise. We provide each agent with an independent instance of the predictive model M', which updates the estimated jointoobservations in the perspective of the agent of and maintains an agent-specific history state h/.
Lin.
alm,, (o) ..... 0 2n,1
4 EVALUATION
In this section, we evaluate our approach for hybrid execution, answering the following questions: (i) What is the performance of MARO in multi-agent cooperative tasks with partial observability, considering unknown levels of communication at execution time?: (ii) What is the importance of the training scheme for hybrid execution, considering different dropout schemes?: (iii) What the importance of the predictive model for hybrid execution?
To address (i), evaluate in Sec. 4.3 our approach against other relevant baselines and considering multiple RL algorithms. The results show that MARO outperforms the baselines, allowing the execution of tasks across multiple communication levels. Regarding (ii), we perform Sec. 1.4. an ablation of the training scheme, highlighting the importance of simulating the communication process at execution time during training. We address (iii) in Sec. 4.4.2 highlighting the benefits of
In the absence of prior information regarding the communication level in the environment, in the training scheme are sampling communication matrices C such that Pi.j P.. p and that Pt, 1.
4

---

Under review as a conference paper at ICLR 2023
the predictive model, both in terms of training sample efficiency and in allowing centralized execution agents to exploit shared information across multiple communication levels.
4.1 SCENARIOS
We focus our evaluation on multi-agent cooperative environments. As discussed by Papoudakis et al. (2021b). the main challenges in current MARL benchmark scenarios, involve coordination, large action space, sparse reward and non-stationarity. As such, in these tasks the sharing of local information is not critical to their successful execution (as we show in Section 4.3 Thus, in order to evaluate the role of passively sharing information amongst agents MARI., we propose three environments adapted from (Lowe et al. 2017):
Two heterogeneous agents cover two designated landmarks in a 2D map while avoiding collisions. In this scenario, one of the agents has access to the X-axis position and velocity of both agents, while the other agent has access to the Y-axis position and velocity of both agents. Both agents observe the landmarks' absolute position;
SpreadXY:
SpreadBlindfold: Three agents cover three designated landmarks in a 2D map while avoiding collisions. Each agent's observation only includes its own position and velocity and the absolute position of all landmarks;
Two heterogeneous agents cover single landmark in 2D map. One of the agents ("Hear" agent) observes the absolute position the landmark, but does not have access to its own position in the environment. The other agent ("See" agent) observes the position and velocities of both agents, yet does not have access to the position of the landmark.
HearSee:
In addition the proposed environments, we evaluate our approach in standard MARL benchmark scenarios, in particular in the SpeakcrListener environment. For complete description of the scenarios, we refer to appendix[B.]]
Finally, we consider H-POMDPs with communication matrices such that each agent can always access its own local observation, i.e., Ps.: = 1, and the communication matrix symmetric between agents and j. i.e., Pt.j Pj.s- To simplify the exposition and the evaluation, we use the same Po.j = p for all pairs of different agents i, j. Therefore, we also use p to unambiguously denote the communication level of given H-POMDP. We perform comparative study between different sampling schemes for the communication matrix Appendix [3..3] highlighting the robustness of MARO under different communication settings. Additionally, note that in all the environments the previous action taken by agent i, af- is included in its local observation of.
4.2 BASELINES AND EXPERIMENTAL METHODOLOGY
We compare MARO against different baselines that correspond to different levels of information- sharing between the agents. We consider two "extreme" cases:
Observation (Obs.): Agents only have access to their own observations and are unable to communicate with other agents, corresponding to standard MARL algorithms designed for decentralized execution;
joint-Observan (J. obs.): Agents always have access to the observations of all agents, corresponding to standard MARL algorithms designed for centralized execution. This baseline is unable to perform when communication fails and can be seen as an upper bound.
To the best of knowledge, there exists method developed specifically for the problem of executing with faulty communication with unknown dynamics to serve as a direct comparison to MARO. As such, adapt the model proposed by Kim al. (2019b) as baseline:
Message-Dropout (MD): Agents train with communication failing half of the times (fixed == 0.5), without communication masks and without the predictive model.
We employ the same controller networks across all evaluations. The networks include recurrent layers to mitigate the effects of partial observability. We consider two different MARL algorithms: QMIX and Independent Q-Learning (IQL.). We follow the training hyperparameters suggested by
5

---

Under review as a conference paper at ICLR 2023
IQL HearSee
QMIX, SpreadXY
IQL, HearSee
QMIX, SpreadXY
1.30
-50
Average neture
115
-75
-200
1100
110
-100
-50
6.30
1,00
0.4
0.
1.0
Training
x1c
(a)
O6s.
ols.
MD
MARO
(b)
Figure 2: (a) Average episodic returns during training with 95% confidence interval for MARO and MD; (b) Average episodic returns during training with 95% confidence interval for different communication levels at execution time for all approaches. Best viewed with zoom.
Table 1:
Average episodic returns and 95% bootstrapped confidence interval over five seeds for all
approaches in all scenarios. Higher is better.
IQL
QMIX
Environment
Obs.
J. obs.
MD
MARO
Obs.
J. obs.
MD
MARO
SpreadXY
-173.2
-140.3
-150.6
-148.7
-166.9
-13..7
-147.1
-144.2
(02-1LD)
(02108)
CLL11
(42.12))
(13.115)
(08,009)
(05.106)
SpreadBlindfold
-432.2
-403.4
.40.77
-05.3
-18.1
-376.0
-00.77
-01.2
(39,-6.)
(3.43.9)
(10.110)
(4.3334)
(83,43.5
(29,033)
6550,500
(-44.6644
HearSee
-61.9
-24.5
-37.1
-25,9
-54.9
-24.1
-25,4
-25,1
(-18-2.0)
(0840.0)
(06498)
(40.,402)
(1.5+13)
(07,499)
(04,44
(4.3,404)
SpeakerListener
-31.7
-25.4
-27.8
-25.6
-26,1
-25.1
-23,2
-23.3
(12214
(1..11))
(43.193)
(45.106))
6133.122)
(11.11
610/122
(1..111)
Papoudakis et al. (2021b); train all models for 4M steps, performing 5 training runs for each experimental setting and 50 evaluation rollouts for each training run. We assume that p = 1 at f = for the MD and MARO algorithms. The performance of the Obs. and J. Obs. agents is evaluated by aggregating evaluation rollouts with p and p = 1, respectively. The other algorithms are evaluated for p sampled from discretized uniform distribution. If the communication level is not explicitly referred, then the values correspond to the average performance across all communication levels. We refer to Appendix B for a complete description of the experimental methodology. including hyperparameters of the predictive model and the RL controllers. Our code is available on Github
4.3 RESULTS
We present the main evaluation results in Table For each environment, RL algorithm and method, present the values of the accumulated rewards obtained. The values that are not significantly different than the highest are presented bold. The results show that MARO consistently performs equal better than the MD baseline across all scenarios and algorithms. MARO is able to exploit the information provided by the other agents, in contrast with the fully decentralized approaches (Obs.). Moreover, MARO often able achieve performances comparable to the fully centralized agent (J. Obs.), which executes with full communication, despite failures in communication. We also note here that, in some MARL environments, information sharing between agents may lead performance gains. For instance, in the SpeakerListener scenario, decentralized QMIX (Obs.) is able to perform competitively in comparison to centralized QMIX (J. Obs.), without requiring information sharing.
In Fig. 2 we highlight the training curves and the performance of the approaches for different com- munication levels p for the HearSee and SpreadXY environments (more in Appendix B). The results show that MARO outperforms the MD baseline in terms of sample efficiency, with bigger jump-start in the initial stages of the training (Fig. 22), and overall performance across all communication levels (Fig. 2b Additionally, MARO sigmificantly outperforms the Obs. baseline in settings with no communication (p- = 0); MD struggles to act in the same setting, performing worse than the fully decentralized baseline. Moreover, the performance of MARO improves as the level of communication in the environment increases, showing that our model is able to efficiently make use of all provided information. Appendix B includes all training curves and performance results.

---

Under review as a conference paper at ICLR 2023
IQ2L, SpeakrLLitenne
IQL. SpreadBlindFold
QMIX, HearSee
QMIX, SpreadKY
T
Recage rotrr
-300
122
-50
-550
-135
-00
0.6
p. p- (0.1)
pm (6.3)
Obs
MARO
Frod Train/
Podl./
Frid
(a)
(b)
Figure 3: (a) Average episodic returns with 95% confidence interval for different communication levels at execution time for the ablated versions of MARO; (b) Average episodic returns with 95% confidence interval for different sampling schemes. Best viewed with zoom.
Table Average episodic returns and 95% bootstrapped confidence interval over five seeds for all versions of MARO in all scenarios. Higher is better.
IQL
QMIX
MARO
Pred. /
Pred. Train
Pred. X
MARO
Pred. /
Pred. X
Pred. X
Environment
Train X
Train X
Train X
Train /
Train X
SpreadXY
-48.77
1588.
-146.4
-197.4
-144.2
-155.3
-144,4
-197.8 6177117)
SpreadBlindfold
.40.33
408.6
402.9
.479.2
.401.2
-39388
407.7
.49.6
(4.334)
(30.430)
618+2.1)
(20+244)
(64464)
652+4
619,669
(38
HearSee
-25,9
-27.7
-6.11
-78.0
-25,1
-27.7
-25.1
-79.4
(0.3:0.3)
(05,105)
(433193)
(4226)
(03,104)
(06,107)
(03,103)
(34.334)
Listner
-25.6
-25.8
-25,8
-4,55
-23,3
-23.5
-23,3
-39.0
(4)-5-00))
(402+47)
(409-00)
(-111+1:)))
(11+)14)
4.4 ABLATION STUDY
We perform an ablation study on the two components of MARO to study their impact on the overall performance of the method. We introduce ablated versions of MARO along two different axes: (i) training with our proposed scheme (Train v), against training with fixed p 1 (Train X); and (ii) using the predictive model to estimate missing observations (Pred V), against naive dropout approach that replaces missing observations with zeros (Pred X). Table 2 shows the results of the ablation study across environments and algorithms. We discuss the results along each axis separately.
4.4.1 TRAINING SCHEME
The results in Table 2 highlight the importance of the training scheme for hybrid execution: introduc- ing our proposed training scheme (Pred X, Train v) over the fully-ablated version of MARO (Pred X, Train X) results significant performance improvement, while removing the training scheme from MARO (Pred V, Train X) reduces the performance of our approach.
In Fig. 3a we evaluate the impact of the training scheme the performance across different communi- cation levels, shown for the SpeakerL.stenng Listener and SpreadBlindfoll enviromments (additional results in Appendix B). The results reveal that MARO is able perform well across the whole communication spectrum. The results also show that while the fully-ablated version of our approach struggles to act with any communication level other than p = 1, adding MARO's training scheme alone results in method that is able to perform close to optimally across the entire spectrum of p. Finally, removing the training scheme from MARO results in a performance drop, especially for p = 0. In summary, the training scheme is beneficial for different values of without sacrificing performance for p
We also evaluate the impact of different sampling strategies of p during training. As previously explained, proposed communication sampling scheme has p ~ 2/ 1) during training. In Fig. 3b we compare our sampling scheme in the HearSee and SpreadXY environments (more in Appendix [ against three other sampling approaches: categorical distribution over the communication level
7

---

Under review as a conference paper at ICLR 2023
extremes, p ~ 04 1; a fixed sampling scheme with p 0.3, which corresponds to the training approach of Kim et al. (2019b) with additional observation masks; and a fixed sampling scheme with p = 1, without observation masks. The results highlight the importance of simulating faulty communication during the training phase, as the fully centralized training scheme (p = 1) is outperformed by all other approaches. In Appendix [B] present the full results of this evaluation, showing the impact of the sampling scheme in other scenarios and RL algorithms.
4.4.2 PREDICTIVE MODEL
The results Table 2 also highlight the significant improvement in the performance of MARO when employing the predictive model to estimate the missing observations. Specifically, adding the predictive model (Pred v, Train X) to the fully ablated version of MARO (Pred X, Train X) results in significant performance increase. Removing the predictive component from MARO (Pred x, Train v) still results in a competitive algorithm without a significant performance drop in the majority of scenarios. However, there are significant advantages to employing the predictive model:
The predictive model improves the sample efficiency of MARO. In Fig. 4c show the training curves for MARO, with and without the predictive model. The results show that employing the predictive model results in significant jump-start in terms of sample efficiency. The predictive model provides the RL controllers with an estimate of the missing agent trajectories, instead of replacing the missing input with zeros, resulting in improved performance during the initial stages of the training. This improvement is consistent across several environments and algorithms, as shown in Appendix B
The predictive model provides robustness to centralized execution methods when performing tasks settings with potential faulty communication, despite never being trained to execute in such conditions. In other words, the predictive model allows for zero-shot multi-agent execution with respect to the communication failures. In Fig. 4d we show that employing the predictive model predict missing information execution time allows standard centralized method to perform the task with minimum performance loss (Pred X, Train X).
The predictive model is able to perform accurate agent modelling with faulty communication. providing an interpretable insight into the decision- makiing process of the agents. In Fig. 4a we show the predicted trajectories of all agents in the perspective of the blue agent, which are close to the real trajectories performed by all agents (more in Appendix B.3.5).
We can also assess the correctness of the predictions made by the predictive model by evaluating the performance MARO against a Switch baseline. In this baseline, the agents choose actions using two controllers, selected accordingly to the level of communication in the environment at each timestep: one that uses the joint observation at the timesteps baseline), and one that uses only the local observation, otherwise (similar to the Observation baseline). We show the results of the comparison in performance between MARO and the Switch baseline in Fig. 4b) The results reveal that MARO is able to exploit the predicted observations for communication levels < 1, outperforming the Switch baseline in different communication settings.
available (similar to the Joint Observation
RELATED WORK
In this section, we connect our work with other lines of research, discussing the similarities and differences between our study and previous works in the field. Due to space limitations, we discuss only the most relevant works and provide an extended discussion of related work Appendix [A]
Closely related to our work are studies that address the problem of partial observability in MARI. As an example, Omidshafiei et al. (2017) propose a decentralized MARL algorithm that uses RNNs to improve the agents' observability, Mao et al. (2020) use RNN to compress the agents' histories, helping to improve agents' observability. The commonly used paradigm of centralized training with decentralized execution also contributes to alleviating partial observability at training time (Oliehoek
et al. 2011 Rashid et al. [2018 Foerster et al.] 2016 2017 Sunehag et al. 2017 Son et al. 2019
Mahajan al. 2019 Wang et al. 2020b] Yu et al. (021).
Other lines of research investigate communication techniques for MARL
Zhu et al. 2022). focusing
on how (Niu et al.
2021 Kim al. 2019a), when (Singh et al. 2018 Hu et al.
2020). and what
8

---

Under review as a conference paper at ICLR 2023
-100
Awonge
Obs
-00
0lo
120
.... Bot sapeteis
Pedicte rueeoon
MARO
MARO
Pied /. Toii
Oluaed agei padions
Too/
0.00
0:35
0.50
0.75
1.00
BGG2
11.03
6.30
038)
1025
1159
0.35
Trass ng step
(a) SpreadBlindF, QMIX.
(b) HearSee, QMIX.
(c) HearSee, IQL.
(d) HearSee, QMIX.
Figure 4: Evaluation of the predictive model of MARO: (a) Trajectory estimation from the perspective of the blue agent, using its agent-specific predictive model; (b) Average episodic returns with 95% confidence interval for different communication levels at execution time of MARO against the Switch baseline; (c) Average episodic returns during training with 95% bootstrapped confidence interval for MARO and an ablated version without the predictive model; (d) Average episodic returns with 95% confidence interval for different communication levels at execution time of ablated versions of MARO, showing the ability of the predictive model for zero-shot execution. Best viewed with zoom.
(Foerster et al. 2016) to communicate to foster cooperation. Previous studies focused on the sharing of (encoded) local observations and actions among agents (Foerster et al., (016) in a proxy-like
manner (Wang et al. 2019). Others learn mechanisms that improve communication efficiency by either limiting the variance of exchanged messages (Zhang et al. 2019) or temporally smoothing information shared between agents (Zhang et al.] 2020). Kim et al. 2019b) propose message-dropout, which aims making learning robust against communication errors. Message-dropout drops the messages received from other agents independently at random during training before inputting them into the RL algorithm. In a similar fashion to message dropout, Wang al. (2020a) propose a recurrent actor-critic algorithm for handling multi-agent coordination with limited communication, showing that recurrence successfully contributes to robust performance under communication failures.
In contrast, we assume that agents have no control over when and with whom to communicate. Hence, they should robustly perform under any type of communication policy/level at execution time. Also, we do not learn the content of the messages and consider rather passive communication setting in which agents share local observations and actions. For this reason, we did not include the works of ZZanng et al. (2019) and Zhang al. (2020) baselines since the comparison between methods would not be meaningful. Instead, following both Kim et al. (2019b) and Wang et al. (2020a), we use sessage-dropou with recurrent learners baseline. Finally, none of the aforementioned works proposed the use of predictive models to account for missing information at execution time as we do in our work, nor mathematically formalized hybrid execution as we present in Sec. 2.1
6 CONCLUSION
In this work, we introduce hybrid execution, a new paradigm in which agents act under any communi- cation level at execution time, while exploiting information-sharing among the agents. To formalize our setting, we define hybrid-POMDDSs a new class of POMDPs that explicitly considers a com- munication process between the agents. To allow for hybrid execution we propose MARO, a novel approach that combines an autoregressive predictive model, to estimate missing observations, and an RL training scheme that randomly drops agent observations, simulating different communication levels during the centralized training phase. We show that MARO allows agents to act across different communication levels, successfully exploiting available shared information.
Future work could comprise: (i) evaluating MARO with actor-critic RL algorithms; (ii) studying other neural network architectures, such as graph neural networks, for better multi-agent trajectory prediction; (iii) studying the performance of MARO under other communication settings.
REFERENCES
Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese. Social Istm: Human trajectory prediction in crowded spaces. In 2016 IEEE Conference
9

---

Under review as a conference paper at ICLR 2023
Computer Viston and Pattern Recognition (CVPR), 2016.
Eugenio Bargiacchi, Timothy Verstraeten, and Diederik M. Roijers. Cooperative prioritized sweeping. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, pp. 160-168, 2021.
Lorenzo Canese, Gian Carlo Cardarilli, Luca Di Nunzio, Rocco Fazzolari, Daniele Giardino, Marco Re, and Sergio Spanò. Multi-agent reinforcement learning: A review of challenges and applications. Applied Sciences, I1(11), 2021.
Xiaohan Fang. Jinkuan Wang, Guanru Song, Yinghua Han, Qiang Zhao, and Zhiao Cao. Multi-agent reinforcement learning approach for residential microgrid energy scheduling. Energies, 13(1), 2020.
Jakob N. Foerster, Yannis M. Assael, Nando de Freitas, and Shimon Whiteson, Learning to commu- nicate with deep multi-agent reinforcement learning. CoRR, abs/1 0006066666, 2016.
Jakob N. Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. CoRR, abs/ Abs//7055.8826, 2017.
Sandro Hauri, Nemanja Djurie, Vladan Radosavljevic, and Slobodan Vucetic. Multi-modal trajectory prediction of NBA players. CoRR, abs/2008.0788 .07870, 2020.
He He, Jordan Goyd-Graber, Kevin Kwok, and Hal Daumé III. Opponent modeling in deep rein- forcement learning. International conference on machine learning, pp. 1804-1813, PMLR, 2016.
Florence Ho, Ana Salta, Ruben Geraldes, Artur Goncalves, Marc Cavazza, and Helmut Prendinger. Multi-agent path finding for uav traffic management. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pp. 131-139, 2019.
Guangzheng Hu, Yuanheng Zhu, Dongbin Zhao, Mengchen Zhao, and Jianye Hao. Event-triggered multi-agent reinforcement learning with communication under limited- bandwidth constraint. CoRR, abs/2010.04978, 2020.
Jiechuan Jiang and Zongqing Lu. Learning attentional communication for multi-agent cooperation. CoRR, abs/1805.07733 2018.
Daewoo Kim, Sangwoo Moon, David Hostallero, Wan Ju Kang, Taeyoung Lee, Kyunghwan Son, and Yung Yi. Learning to schedule communication in multi-agent reinforcement learning. CoRR, abs/1902.01555, 2019a.
Woojun Kim, Myungsik Cho, and Youngchul Sung. Message-dropout; An efficient training method for multi-agent deep reinforcement learning. CoRR, abs/1902.06527,, 2019b.
Woojun Kim, Jongeui Park, and Youngchul Sung. Communication in multi- agent reinforcement learning: Intention sharing. In International Conference on Learning Representations, 2021.
Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed noperative-competiive environments. Advances in neural information processing systems, 30, 2017.
Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. Maven: Multi-agent variational exploration. Advances in Neural Information Processing Systems, 32, 2019.
Weichao Mao, Kaiqing Zhang, Erik Michling, and Tamer Basar. Information state embedding in partially observable cooperative multi-agent reinforcement learning. CoRR, abs/2004.0109% 2020.
Yaru NIu, Rohan Paleja, and Matthew Gombolay. Multi-agent graph-attention communication and teaming. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems. pp. 964-973, 2021.
Frans A. Olichoek and Christopher Amato. A concise introduction to decentralized POMDPs. Springer, 2016.
10

---

Under review as a conference paper at ICLR 2023
Frans A. Olichock, Matthijs T. J. Spaan, and Nikos Vlassis. Optimal and approximate q-value functions for decentralized pomdps. CoRR, abs/1111.0062,, 2011.
Shayegan Omidshafiei, Jason Pazis, Christopher Amato, Jonathan P. How, and John Vian. Deep decen- tralized multi-task multi-agent reinforcement learning under partial observability. In Proceedings of the 34th International Conference on Machine Learning, pp. 2681-26900 2017.
Shayegan Omidshafiei, Daniel Hennes, Marta Garnelo, Eugene Tarassov, Zhe Wang, Romuald Elie, Jerome T. Connor, Paul Muller, Ian Graham, William Spearman, and Karl Tuyls. Time-series imputation of temporally multiagent trajectories. CoRR, abs/2106.04219, 2021.
Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V. Albrecht. Comparative evaluation of multi-agent deep reinforcement learning algorithms. CoRR, abs/2006.07869, 2020.
Georgios Papoudakis, Filippos Christianos, and Stefano Albrecht. Agent modelling under partial observability for deep reinforcement learning. Advances in Neural Information Processing Systems, 2021a.
Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V Albrecht. Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks. Thirty-fifth Conference on Neural Information Processing Systems Dutasets and Benchmarks Track (Round 1), 2021b.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito,, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high beigh-eeffrmance deep learning library. Advances in Neural Information Processing Systems 32, pp. 8024-803 8035. 2019.
Tabish Rashid, Mikayel Samvelyan, Christian Schröder de Witt, Gregory Farquhar, Jakob N. Foerster, and Shimon Whiteson. QMIX: monotonic value function factorisation for deep multi-agent reinforcement learning. CoRR, abs/1803.. 1485, 2018.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Amanpreet Singh, Tushar Jain, and Sainbayar Sukhbaatar. Learning when to communicate at scale in multiagent cooperative and competitive tasks. CoRR, abss/881.0990 2018.
Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International conference on machine learning, pp. 5887-5896. PMLR, 2019.
Sainbayar Sukhbaatar, Arthur Szlam, and Rob Fergus. Learning multiagent communication with backpropagation. CoRR, abs/1605.07736, 2016.
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojeiech Marian Czamnckki Vinicius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Leibo, Karl Tuyls, et al. Value-decomposition networks for cooperative multi-agent learning, arXiv preprint arXiv: 05296, 2017.
Ming Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proceedings of the tenth international conference on machine learning. pp. 330-337, 1993.
Rose E. Wang, Michael Everett, and Jonathan P. How. R-MADDPG for partially observable environments and limited communication. CoRR, abs/2002.06684 2020a.
Rundong Wang, Xu He, Runsheng Yu, Wei Qiu, Bo An, and Zinovi Rabinovich. Learning efficient multi-agent communication: An information bottleneck approach. CoRR, abs/1911 1.0992,, 2019.
Tonghan Wang, Tarun Gupta, Anuj Mabajan, Bei Peng, Shimon Whiteson, and Chongjie Zhang Rode: Learning roles to decompose multi-agent tasks. arXiv preprint arXiv:2010.01523, 2020b
Xihuai Wang, Zhicheng Zhang, and Weinan Zhang. Model-based multi-agent reinforcement learning: Recent progress and prospects, 2022.
11

---

Under review as a conference paper at ICLR 2023
Hua Wei, Guanjie Zheng, Vikash V. Gayah, and Zhenhui Li. A survey traffic signal control methods. CoRR, abs/1904.08 117, 2019.
Daniël Willemsen, Mario Coppola, and Guido C.H.E. de Croon. Mambpo: Sample-efficient multi- robot reinforcement learning using learned world models. In 2021 IEEE/RSJ International Conference on Intelligent Robols and Systems (IROS), pp. 5635-5640, 2021.
Annie Xie, Dylan P Losey, Ryan Tolsma, Chelsea Finn, and Dorsa Sadigh. Learning latent represen- tations influence agget interaction. arXiv preprint arXiv:201 1.06619 2020.
Raymond A. Ych, Alexander G. Schwing, Jonathan Huang, and Kevin Murphy. Diverse generation for multi-agent sports games. In 2019 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 4605 4614,461,, 2019. doi: 10.119//CVP22209.000444
Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness ppo in cooperative, multi-agent games. arXiv preprint arXiv:2103.01955, 2021.
Ekim Yurtsever, Jacob Lambert, Alexander Carballo, and Kazuya Takeda. A survey autonomous driving: Common practices and emerging technologies. IEEE access, 8:58443-58469 2020.
Sai Qian Zhang, Qi Zhang, and Jieyu Lin. Efficient communication in multi-agent reinforcement learning via variance based control. CoRR, abs/1909 2019.
Sai Qian Zhang, Jieyu Lin, and Qi Zhang. Succinct and robust multi-agent communication with temporal message control. CoRR, abs/2010.14391, 2020.
Changxi Zhu, Mehdi Dastani, and Shihan Wang. A survey of multi-agent reinforcement learning with communication, 2022.
12


Reviews:
[{"review_id": "kl-Vw29CCZ", "reviewer": "Reviewer_PDRN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new paradigm for multi-agent reinforcement learning (MARL) whereby the agents are trained in a centralised way but tested in a hybrid fashion. During the hybrid execution, information regarding other agents is hidden during different timesteps, ranging from a fully decentralised mode to a centralized regime. The authors name this problem hybrid-POMDPs and propose a method called MARO to make to most out of the given at times information during execution. The authors provide empirical results that show MARO outperforming other baselines, as well as detailed ablation studies.", "summary_of_the_review": "This is an interesting work that proposes a new MARL problem that has not been studied before. The authors compare their method with existing approaches and perform an ablation study of its components. However, using more complex environments with more agents would strengthen the work. In addition, I encourage the agents to perform more experiments studying different hybrid execution paradigms, as described above.", "strength_and_weaknesses": "## Strengths\n\n- The paper takes on an interesting problem - hybrid execution in MARL. Although the CTDE regime has gained major attention in recent years, no work has previously studied this paradigm which can potentially have real-world use cases. To me the problem itself is novel.\n- The paper includes extensive empirical evaluations and ablation studies that evaluate all aspects of the proposed approach and contrast them with other methods.\n- The paper is clear and easy to follow. The contributions of the work can be easily identified.\n\n## Weaknesses\n\n- The major weakness of this work, in my view, is the environments used for empirical evaluation. The authors only have 2D environments with 2-3 agents. It is very unclear if the proposed method would scale to settings with more agents (e.g. 5, 10, 20) or to more complex domains. I think the SMAC benchmark could be an interesting domain for trying out MARO whilst having a large number of agents alongside complex dynamics\n- While the work is considerably novel (both the problem and the method), the proposed approach is fairly straightforward. It seems obvious that this approach would outperform a fully decentralized case or MD. So what I would want to see here, which is unfortunately missing, is a thorough empirical evaluation of the method for different communication matrices. For example, empirical analysis of the following settings would strengthen the paper - what happens if \n   1. $p{i,j}\\neq p_{j,i}$\n   2. different $p$ value is used for different co-player pairs\n   3. $p$s change throughout the episode\n- (Minor) It would be helpful if the authors include screenshots of the environments in the main text.\n\nNote: My score is not final and can change (in either direction) based on the authors' responses or comments of other reviewers.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "cmymrwNYoN", "reviewer": "Reviewer_ymGd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes multi-agent hybrid-POMDPS which uses a centralized training scheme as well as models a communication process between the agents. Experiments are conducted on standard benchmarks to show the superiority of the proposed method. ", "summary_of_the_review": "The paper introduces hybrid-POMDPs by adding agent communication as an additional parameter to dec-POMDPs. However, the novelty and contribution is limited as the autoregressive prediction model and the masked training procedure does not consider exhaustive multi-agent scenarios. The experiments also show that another baseline performs similar using IQL. The main advantage of the proposed approach over existing works needs to be made clear via experimentation. \n\n", "strength_and_weaknesses": "strengths:\n-the paper proposes an approach called hybrid-POMDPs\n-it adds the notion of agent communications to dec-POMDPs which is fully decentralized \n-agents can have any level of communication (none to complete)\n\n\nweakness:\n-the contribution may be limited because when the communication matrix is known it can be solved as a Dec-POMDP\n-predicting the unknown observations using an LSTM does not consider all possible contextual information to identify dynamic agent behavior \n-the centralized training scheme needs to be explained in more detail\n-how accurately masking during training reflects agent communication during evaluation needs to be discussed\n-the MD approach used for comparison performs equally better in many scenarios\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "M06EhmItpsQ", "reviewer": "Reviewer_eHqC", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In the context of MARL, this work introduces the paradigm of centralised training with hybrid execution, a setting in which the information availability of the other agent's observation is not guaranteed during execution. A novel training approach is also introduced, MARO, in order to deal with this setting, comprising of a predictive model and a training scheme that includes a random drop of agent observations, to simulate the conditions that might be encountered during execution. ", "summary_of_the_review": "All in all this, this work addresses a novel setting, that of faulty observations exchange between agents during execution, and proposes a novel framework to capture this setting and a novel training approach for solving it. I would consider it currently slightly below the acceptance threshold, but if authors can clarify my questions, I will raise my score.", "strength_and_weaknesses": "Strengths:\n- The work formalises the setting of faulty communication of the agent's observations during execution as a hybrid-POMDP (H-POMDP)\n- The work introduces a novel training approach, MARO, that can be combined with many existing MARL methods to improve performance in this faulty communication setting\n\n\nWeaknesses:\n- Lack of motivation for selected benchmarks. In the abstract it is mentioned \"tailored to emphasize the negative impact of partial observability\", but I did not find an explanation for supporting this statement thereafter.\n- Weak comparison against the selected method, can be improved by clarifying more details on the MD approach \n\nQ1. Can you motivate the selection of the benchmark, in contrast for example with the choice of the Kim et al. (2019b) work. Why are they \"tailored to emphasize the negative impact of partial observability\"? \n\nQ2. As far as I can tell, there are never more than three agents in the environment. Have you also tried larger environments? Can you comment on the scalability of the method? Especially in combination with independent learning approaches, I do not see why a larger number of agents might pose an issue. But what about the predictive model?\n\nQ3. Have you tried combining MARO with PG/actor-critic based approaches? It would have been interesting and make for a well-rounded experimental evaluation to also see for example how IPPO behaves.\n\nQ4. Can you clarify if for the adapted MD approach you also used the original training procedures (eg., block-wise/element-wise dropout)? Have you tried also other values for p? \n\nQ5. In 3.2 you mention that the masks enables to measure uncertainty regarding the input. Is this information used anywhere?\n\nQ6. Have you also tested a setting where during execution the p is dynamic?\n\n\nMinor typo:\n- Section 6 'that explicity considers'", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DwsDN8VjAnr", "reviewer": "Reviewer_CnPR", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method for training in MARL setting under hybrid communication setting (decentralized vs fully centralized) by using a autoregressive model for predicting next joint observations. Experiment are provided on some toy domains with 2 decentralized value based baselines.", "summary_of_the_review": "Interesting idea, but the results are not strong. The paper also lacks important related works crucial for better understanding the problem. The authors should focus on the points in the weakness section.", "strength_and_weaknesses": "Strength\n1. The paper is well written in terms of describing the setting used.\n2. The hybrid execution setting is moderately relevant for real applications.\n\nWeakness \n1. The related works is incomplete. The paper lacks discussion about the implications of using the different types of MARL algorithms: decentralized/centralized and policy/value based and their interaction with the hybrid communication setting, which is important (refer to e.g. [1], [2]). Policy based MARL methods like Tesseract [2] and MAPPO [3] can handle this scenario without performance loss in principle. Discussion about the mentioned works should be included for better context and reader understanding.\n\n2. The experimental results are reported only on toy domains, there are more realistic complex domains like Starcraft which can be used. The baselines used are also inadequate, some policy based baselines should be included (for reasons covered in [1, 2]). Even for value based setting, there are stronger baselines available like [4]. \n\n3. The difference with performance of the simple message dropout baseline is not significant, which makes it unclear how effective the proposed approach is.\n\n4. Is the communication matrix C also visible to the agents?\n\n5. In general, the considered setting is significantly easier than decentralized control, so I am unconvinced with the usefulness of the approach.\n\nReferences:\n1. Maven: Multi-agent variational exploration, Mahajan et al, 2019\n2. Tesseract: Tensorised Actors for Multi-Agent Reinforcement Learning, Mahajan et al 2021\n3. The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games, Yu et al 2021\n4. Rode: Learning roles to decompose multi-agent tasks, Wang et al, 2021", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject