CoMIX: A Multi-agent Reinforcement Learning Training Architecture for Efficient Decentralized Coordination and Independent Decision-Making
Authors: Giovanni Minelli
Mirco Musolesi
Venue: Accepted by TMLR ()

Abstract: Robust coordination skills enable agents to operate cohesively in shared environments, together towards a common goal and, ideally, individually without hindering each other's progress. To this end, this paper presents Coordinated QMIX (CoMIX), a novel training framework for decentralized agents that enables emergent coordination through flexible policies, allowing at the same time independent decision-making at individual level. CoMIX models selfish and collaborative behavior as incremental steps in each agent's decision process. This allows agents to dynamically adapt their behavior to different situations balancing independence and collaboration. Experiments using a variety of simulation environments demonstrate that CoMIX outperforms baselines on collaborative tasks. The results validate our incremental approach as effective technique for improving coordination in multi-agent systems.

Published in Transactions on Machine Learning Research (06/2024)
CoMIX: A Multi-agent Reinforcement Learning Training Architecture for Efficient Decentralized Coordination and Independent Dacision-Making
Giovanni Minelli University of Bologna
giovenni.minelll20005 unibo.
Mirco Musolesi
m.muusoeesi@ucll.ccoo uk
University College London
University of Bologna
Reviewed on OpenReview: /ttps::/openreeet net/ forum? id= JoU9hhOOur
Abstract
Robust coordination skills enable agents to operate cohesively in shared environments, to- gether towards a common goal and, ideally, individually without hindering each other's progress. To this end, this paper presents Coordinated QMIX (CoMIX), novel train- ing framework for decentralized agents that enables emergent coordination through flexible policies, allowing at the same time independent decision-making at individual level. CoMIX models selfish and collaborative behavior incremental steps in each agent's 's decision pro- cess. This allows agents to dynamically adapt their behavior to different situations balancing independence and collaboration. Experiments using a variety of simulation environments demonstrate that CoMIX outperforms baselines on collaborative tasks. The results validate our incremental approach as effective technique for improving coordination multi-agent systems.
1 Introduction
Multi-agent systems pose significant challenges due to the complexity of managing interactions (Long Ma ct al.) 2015). Whether agents operate individually or collaboratively, coordination emerges as a pivotal aspect within such systems. Moreover, as the number of agents increases, the lask of ensuring coordination becomes harder and harder. Effective coordination also critical for number of practical application
domains, namely autonomous driving (Prabuchandran al., 2014 Sadigh al., 2018 Falsone al.) 2020
and robotics (Yan et al. 2013 Yu et al. 2023), especially in systems involving mobile robots (Chen & Luh
1994
Sousa et al. 2021 and drone fleets AAleo et al. 2018 Qin & Pournaras
2023
Previous research has focused primarily on information sharing strategies to facilitate coordination among
multiple entities (Groen et al.] 2007 Pesce & Montana, 2020; Kim et al.) 2021). Indeed, agents often find themselves having make decisions based on an incomplete understanding of the world in which they operate, for instance, due to limited perception ability or scarce computational resources these are typical examples of bounded ralionality (Simon) 1996). In this context, effective communication plays a key role. In fact, coordination emerging from indiscriminate communication may not always be the best solution (Cltoion(Cason et al. 2012): an overabundance of shared information can overwhelm agents and slow down decision-making, negatively affecting coordination among the agents.
In addition, there are cases where coordination itself can hinder optimal task execution. For example, popular solutions, such as IC3Net (Singh et al. 20119) BicNet (Peng al. 2017), and ATOC [Jiang Lu 2018), do not explicitly grant agents the autonomy to operate outside the given cooperative scheme by limiting the exploration of the entire action space and the exploitation of individual capabilities. Allowing

---

Published in Transactions on Machine Learning Research (06/2024)
agents to occasionally pursue local strategies could be beneficial to achieve the overall shared collective goal, as agents can help discover better optimized solutions for the team as a whole. For example, in a multi-agent navigation task, a robot periodically exploring new potential paths on its own could provide useful information to others. This raises an important question that we aim to address in this study: can agents acling in a shared environment, through commmunicaaiin and without collaboration constrannts, learn to balance coordinated and individual behaviors order to successfully achieve a given goal?
In order to address this problem, in this paper, we present the design, implementation and evaluation of Coordinated QMIX (CoMIXI a novel framework for decentralized multi-agent reinforcement learning (MARL) combining group and individual decision-making. The conventional approach to addressing the coordination problem in MARL systems is to prioritize group choices over individual ones. Although a solution that gives more importance to group dynamics can be more effective when cooperation is critical for success, it might be detrimental when agents could benefit from acting independently to achieve a given goal, or in complex environments with vast space of possible interactions. For instance, in crowded or sparsely rewarded settings, enforcing top-down coordination on all agents may hinder convergence to an optimal joint policy (Aotani et al., 2018). In contrast, allowing potential independent local choices, rather than mandating group coordination, can lead to emergent collaboration arising naturally where neoded, with benefits for both the individual and the group as a whole (Kar et al. 2013 2020
Hughes et al.
2018 Yang et al.
Yu et al. 2022).
CoMIX embodies this principle by allowing agents to choose whether to co-operate with others or to act independently, first commuicattng their own action intentions derived from previously acquired and current information alone (e.g., sensory data). These are then combined with the intentions of other agents in order to reduce uncertainty and achieve coordination.
By considering variety of challenging MARL environments, we demonstrate CoMIX's ability to efficiently choose between collaboration and independent action, and, consequently, to coordinate with a large number of agents effectively. Moreover, CoMIX based on storing a weighted record of neighbors' previous action intentions for coordination, which makes the method interpretable in terms of agent. choices and resilient to communication disruptions. Overall, our findings highlight the effectiveness of CoMIX in facilitating coordination among agents, by enabling them to selectively coordinate in pursuit of shared goal, allowing at the same time independent decision-making when beneficial.
2 Related Work
In this section, we discuss the relevant related work with a efficient information sharing in multi-agent systems.
focus on mechanisms for group coordination and
Coordination. Multi-Agent Reinforcement. Learning (MARL) frameworks are designed to overcome the added complexity resulting from the presence of multiple interacting entities in an environment (Tuyls & Weiss 2012 Zhang et al.] (021). In particular, we are interested on those that adopt information shar- ing mechanisms (Gronauer & Diepold, 20021; Hernandez-Leal et al.] 2019) to achieve flexible coordination without introducing central entities that could create failure points or bottlenecks. CommNet (Sukhbaatar ett al. 2016) demonstrates remarkable performance by simply collecting messages and averaging them, then using the additional information in each agent's 's policy. IC3Net (Singh et al. 2019) adds mechanism for instructing agents to learn when to communicate, through the adoption of a gating mechanism to allow access to a communication channel. However, poor utilization of the commuicattin channel limit the abil- ity to coordinate and may not lead lo effective policies when many agents participate in communication. BicNet (Peng al. (017)) uses a more complex mechanism leverage incoming information by adopting bidirectional module, and, like others approaches (Jiang & Lui 2018 Das ct al.
2019). is characterized
by an architecture based recurrent modules to maintain consistency in outputs generated in subsequent
steps. However, these solutions are designed only for cooperative tasks (Mao et al. 2017 Foerster et al. 2018 Singh et al., 2019 Zhang et al.] 2020 Li et al.) 2022) and, therefore, lead to policies not considering the independence of agents or allowing them to exploit possibly better solutions by diverging from team
The code available at the following URL: http://hhhuu.cam///aahiilllllloa
2

---

Published in Transactions on Machine Learning Research (06/2024)
executions. Our approach builds on this body of work, with a specific focus on flexible coordination and communication efficiency.
Information Sharing. Communication can be implemented as a direct message sent through a private channel (Niu et al., 2021), or as a broadcast transmission (Sukhbaatar et al. 2016 Lin et al. 2021 How- ever, the latter solution may prevent agents from distinguishing information that valuable for them from irrelevant or even potentially misleading one in case of different action dynamics or goals. In general, the information received can be considered valuable if it enhances agent's understanding and coordina- tion by providing relevant updates on the environment state, goals, intentions, and other factors that aid dccision-making and coordination. ATOC (Jiang & Lu 2018) proposes communication model with a gate mechanism for communication elficiency. Instead, (Das el. al., 2019 Li et al. 2022 Kim et. al.) 2021 act on the listener's 's side, adopting different attention mechanisms to filter out irrelevant messages. However, they gencrally consider each message in isolation rather than in the context of all the others in the communica- tion channel, missing opportunities to coordinate responses among multiple agents as it is often essential for effective teamwork. Some approaches rely on the formation of groups of agents and focus on inter- and cntaa-commnniattio to limit computational efforts and improve coordination performance (Jiang & Lu, Liu et al.) 2020 2021 Niu et al.) 2021), while others reduce the number of messages sent by learning in which situations communication is really necessary and/or when the available information is redundan and therefore communication can be avoided over time (Ding et al. 2020 Liu et al. 2020 Kim et al.) 2021 Kalinowska et al.] 2022). In contrast, CoMIX aims at creating simplified communication channels directly based on environment observations, not requiring agents to make assumptions about the senders or receivers, thus making them potentially deployable even with unseen agents.
2018
Information Representation. The format of a message is crucial aspect of communication, as it must effectively encode information in order to reduce uncertainty and facilitate coordination with the recipient. The information conveyed through the message may encompass various aspects of communication and decision-making, ranging from agent intentions, such as requests, questions, and plans, to contextual information and conversational messages. Some existing work relies on the transmission of internal state vectors (Sukhbaatar al.) 2016 Peng et al., 2017 Singh et al.] 2019 Wang & Sartoretti] 2022), which are typically used by agents to encode their own history, but the reuse of such individual state representations for communication may not be suitable. Tn (Das et al. 2019 Li & Zhang. 2024). targeted messages are used to convey information to individual agents. Simpler communication approaches (Liu et al. 2020 Kim et al, 2021 include current or time-delayed observations plain or encoded as content for the communication message. This allows for more flexible interaction dynamics since the information is transmitted without any additional processing or filtering. CoMIX belongs to this class of solutions, since it relies on sharing observations paired with the agent's 's action decision, in order to guide coordination among agents.
3 Background
In this section, introduce the notation and concepts at the basis of our approach, which will then be presented in detail in the following section.
Deep Reinforcement Learning. The problem is modeled as a decentralized partially observable Markov decision process POMDP) (Oliehoek & Amato 2016), which is defined by the tuple (K, S,A, P.R. 7), where K is the set of n interacting agents, is the set of observable states 8, A is the joint action space consisting of individual action spaces A; of actions a, for each agent E K, P S A + P(S) is the transition probability function that describes the chance of a state transition, R is the joint set of reward functions R, associated with each agent i K, and 7 [0, 1) is the discount factor. At each time step, each agent selects an action based on its individual policy म; : S + P(A:). The system evolves from joint state s = (81,.....n. } to the next state s' under the joint action a = {aI,..... ) with respect to the transition probability function P, while each agent receives an immediate reward r = (ri...., Rn as feedback following a state transition. The objective is to construct an independent policy n, for each agent E K with the goal of maximizing the expected discounted return. The optimal policy n (s) is obtained indirectly during the iterative process by considering state-action function Q; S x A; Ri; associated with each agent, which
3

---

Published in Transactions on Machine Learning Research (06/2024)
estimates the expected return of taking action in state The learning algorithm is based Deep Q- Network (DQN) et al. 2013):: the Q-function is parametrized by neural network presenting recurrent neural network (RNN) modules to keep memory of sequences of choices. The Dec-POMDP setting is then extended with a broadcast communication channel allowing agents to exchange messages to overcome partial observability effects. The individual policies are learned as state-action functions Qi(s,, h2 m, m,a)) evaluating the possible actions 0, in the current local state 8; with the hidden state h1, and the exchanged messages m (mI....,,, Mn.. where my is sent by agent E K.
Centralized Training with Decentralized Execution. Centralized training with decentralized exeeu- tion (CTDE) with Q-value optimization is a popular approach for solving MARL problems (Lowe et al. 2017 Foerster et al.) 2018 Rashid et al. 2020). During training, a central mixer network is used to represent the value of the state of the entire system. We adopt QMIX (Rashid et al. 2020), which learns a non-linear mixing function to map individual Q-values of agents q = (Qi...., Qn to the global Q-value, denoted as QroT. Hypernetworks (Ha et al., 2016) are used to compute a set of mixing weights for each agent, ensuring that global argument of the maxima (argmax) operation produces the same result as individual argmax operations. This is achieved through monotonicity constraint between Qror and individual value Q
aQTOt aQ,
2 0, Vi E [i,m]
(1)
4 The CoMIX Training Architecture
In the following we will describe CoMIX, framework that allows agents to adopt both individual and group strategic behavior by selectively communicating via shared channel, letting coordination arise from continuous interaction and without making prior assumptions about other agents. The overall architecture of CoMLX is depicted in Figure]
4.1
Architecture
a)
b)
CoV
W
MLP
MLP
MLF
Quay Wordd
network
BIGRU
Q+
Communication chaynel
4,, >>
Agent
Agent
Quef
M
GRU
Figure 1: CoMLX training architecture. (a) The system is illustrated by breaking down its components with the information flow from left to right. Each agent observes partial state of the world, processes it, and determines the next action to be taken. After transmitting this information < s,, > through the communication channel, they receive and filter the m messages from other agents using coordination module. The filtered messages m, are then used to compute weights to rescale the original state-action pairs and select the best action. Decisions are made locally. (b) A conceptual demonstration the interpretability of the method showing how two agents, sharing the same portion of the map and with opposite goal directions, change their initial action intentions based on information received from other agents.
4.1.1 Action Policy
The action policy in our approach involves predicting action values as element-wise product of two terms: Q2 Quaf @ W Wooord- This computation occurs within the Q-network through two-step process. In the
4

---

Published in Transactions on Machine Learning Research (06/2024)
first step, Qeels is obtained by considering only the internal state of the agent h; and its current observation This step does not involve communication with other agents. After communication occurs, the second term Wooord is computed within the Q-network to weigh the estimated value for each action. Specifically, W Wcoord(Mi, m,, a;) takes into account the updated hidden state h3 and the messages filtered by the Coordinator module for agent i. We will discuss these aspects in detail in the next section. The messages are averaged in order form a single vector concatenated with the hidden state and processod through a two layer fully connected network to obtain bounded weight value. It worth noting that the raw observation is first passed through an imput processing module, which extracts features and reduces the state space. Following the approach in Lin et al. (2021), shared weights are used among all agents to allow them to reason about data from the same distribution. Finally, each output Q; (for each agent i) of the Q-network is defined as follows:
Qi(s, h4 mi, a;) = Qeels (81, hes as) o W Weoord (h/, m.
(2)
4.1.2 Coordinator
The Coordinator module is responsible for determining the relevance of other agents' communications in relation to the agent's intentions by generating a coordination mask used to filter out incoming messages. A message is defined as the communicated intention of an agent to take a certain action in its current state to achieve an objective, å; = arg max, Q hi, a), and it represented by the tuple mi =< 8,, â; >. Consequently, we define m {m1 {m...... ,,,) as the set of incoming messages sent by n agents. As previously proposed by (Jiang & Lu 2018), the Coordinator module is implemented using a BiGRU layer (Wang al., 2017) and additional Linear layer on top. A two-way softmax operation then extracts the probability of accepting rejecting each message as shown in 4
2: = f< mi m2 > ,.... < my Mn >} <m,,m,>>
(3)
C, = Coord(z.)
(4)
{
if agent i coordinate with agent, j,
where C2
cc. and C.j = (0 otherwise
We then obtain m, the filtered set of messages for agent i, by applying the Hadamard product between incoming messages and the coordination mask c, computed by the Coordinator module:
m, = m @ C
(5)
4.2
Training
4.2.1
Centralized Temporal Difference Supervision
The agents' Q-policy networks are trained end-to-end by adopting CTDE framework that estimates the state-action value of the entire system. We adopt the QMIX framework (Rashid et al. 2020), which despite the monotonicity constraint limiting its representation capacity, has been shown to achieve state-of-the-art the-art performance on various MART. RT. tasks. Additionally, its computational simplicity enables to effectively man- age large joint action spaces. We consider 09 that parameterizes both the components of individual agents responsible for estimating the Q-value and the central mixer network. Following the popular implementation of Rashid et al. 2020), we adopt the time difference error as the update rule:
TQ(09) = TOT_ - TOT
(s, (s.q:0)) (0)
(6)
where TOT = + max TOT (s',q': q':;g';)) and 0 ge¹ are respectively the parameters of the online policy
q' network and target policy network, periodically copied from 0 as in standard DDQN (Van Hasselt et al., 2016). Employing a centralized function that accounts for all agent actions can lead to a decrease of the policy gradient estimate variance compared to using individual Q-networks per agent, as separate Q-networks may inaccurately assess the local Q-function. Then, after training, the central training function is not used anymore, and policy execution is fully decentralized.
5

---

Published in Transactions on Machine Learning Research (06/2024)
4.2.2 Contrastive Optimization
The parameters the Coordinator module are updated using a contrastive optimization scheme. The underlying idea. that providing the individual policy with helpful information could enable it to more accurately predict the expected future rewards associated with each action. Assuming be able achieve approximation of the optimal Q-function, aim to train efficient coordinator. This coordinator acts a filter over the communication channel, selecting the subset. of messages used to calculate the state- action value estimate with maximum cumulative expected reward. We update the Coordinator's parameters ge by extracting the loss signal as the clipped difference between the maximun value of the state-action pairs obtained with filtered messages m. using the predicted coordination mask c, and the estimated value obtained with an alternative subset of messages m; m @ é, where é; (1 (z,; (C)) from Eq. 4 and Eq. 5 Notably, we propagate the error in the probabilities of message selection c; and keep the weights of the Q-networks fixed by introducing stop-gradient operator stop:
Lc(0C) Σ swop(w;
Σ stop(u: ww max(0, max (5, h4 mi, a.) max Qi(s,, hi, m, a.))),; (7)
o,
The additional weighting term my is obtained by means of the QMIX Mixer network, which, by design, projects the individual Q-value estimates from each agent into shared representation space in its first layer. We leverage the monotonicily constraints of the Mixer compute w; as the per-agent averaged projection weights, which scale their estimated Q-values based on the current joint state.
4.2.3 Training Details
We keep the size of the hidden layers fixed at 128 for all submodules of the Q-networks. The two hidden layers of the Mixer network are configured with 32 and 16 nodes, respectively, We use ReLU nonlinearity and apply layer normalization after both recurrent modules - in the Q policy network and in the Coordinator. This helps address scalability issues caused by vanishing gradients when dealing with many agents. To optimize the weights of the Q and Coordinator networks, we use RMSprop. The learning rate is set to 1e for the Q-network and 55 for the Coordinator network. We also use a weight decay value of 16-5 prevent catastrophic forgetting and overfitting. To stabilize Q-lcarning, we update the weights of the target networks every 100 episodes. The batch size and number of recurrent steps during training are chosen based the complexity of the task. Additional details about networks structure, hyperparameters optimization, the simulation environments, and the algorithm at the basis of the CoMIX training loop can be found in the Supplementary Material.
5
Experiments
We evaluate CoMIX in three distinct environments, which allow us to assess different dimensions of the problem, including scalability and robustness to potential disruptions.
5.1 Environments
In this subsection, we introduce the enviromments used in our experimental evaluation, providing an overview of their main characteristics and discussing the criteria for their selection. For a full description of the experimental settings of the environments, please refer to the Supplementary Material.
5.1.1 Switch
Switch is a gridworld environment in which 4 agents spawn in corners and must reach target positions on the opposite side of the map. It consists of a rectangular grid with a long narrow corridor in the center which can only be crossed by a single agent at a time. The vertical position the corridor randomized for each episode to increase unpredietability. The reward given only upon reaching the dostination, and its value is determined by how quickly the task is completed. This environment can specifically be used to evaluate the agents' abilities to pursue long term reward and balance aggressive and coordinated strategies since

---

Published in Transactions on Machine Learning Research (06/2024)
a)
b)
c)
Figure 2: Environments used for experimental evaluation. From left to right (a) Switch; (b) Cooperative Load Transportation, loads (red) and 4 agents (blue) with an equal distance of 15 steps the final position (green), and 10% of randomly selected cells being obstacles (black); (c) Predator-Prey, 4 agents (blue), 16 prey (red) in 12x12 map. The spatial observability of agents is shown in light blue.
laking the init to traverse the corridor earlier leads to better rewards, but risks potential gridlock if agents collide. The agents can only observe their current position and target distance, but not other agents, so they must communicate to coordinate. Therefore, order to complete the task, necessary to balance individual and collaborative behavior. In addition, we would also like to note that partial observability forces agents to communicate and adapt their policies based on the actions of others.
5.1.2 Cooperative Load Transportation
This environment involves pairs of agents synchronizing to transport load goal location while avoiding obstacles. It requires close coordination to move in the same direction and effective communication to prevent interference from other agent pairs. Agents observe their absolute coordinates, distance to the goal, and proximity to obstacles. Small rewards are given for moving the load closer to the goal and a large reward upon completion. We select this environment since it requires very fine-grained coordination and the capability of screening out "distracting" signals from other agents.
5.1.3 Predator-Pre
This is a classic MARL environment (Gupta et al. 2017) adapted to encourage interactions between agents. The environment a grid world in which agents, the predators, chase randomly moving entities, the prey, with the aim of capturing them. The predators' visibility is limited a range of two units in each direction from their positions, and they move at the same speed the prey. The agents sparscly rewarded. The main challenge lies in collaborating receives substantial reward. If the agents act independently, they can receive smaller rewards by attempting to occupy the same cell as the prey. This task is relatively straightforward and serves to distract the agents, as the ultimate measure success the tally of captured prey. Therefore, the agents should prioritize maximizing long-term cumulative rewards through coordinated hunting and accomplishing the challenging group task, rather than focusing on easily attainable short-term individual rewards.
group to surround the prey on all four axes: in this case agent
5.2 Baselines
For the evaluation, comparators, select straightforward "vanilla" implementation of PPO in multi agent setting, IPPO (Schroeder de Witt et al. 2020), and two representative MARL methods that implement attentional communication, namely ICSNet (Singh et al. 2019) and ATOC (Jiang & Lu 2018). In partic- ular, the choice of PPO, is motivated by its demonstrated ability to excel in certain scenarios characterized by specific forms of environmental nonstationarity, thus allowing IPPO to exhibit robustness when dealing with cooperative tasks that can be solved through sequences of independent actions selocted locally. Com- pared with ours, IC3Net utilizes a gating mechanism on incoming communications, wherein it aggregates all additional information by computing the average into single vector that is then used in the internal

---

Published in Transactions on Machine Learning Research (06/2024)
Cokix
Ic3het IFOC
OMMx
CoMix wib comm
wis
Switch
Cooperative Load Transportation
34
Prodator Prey
Compittton
Nome
Prey
02-
100K
X00C
D00K
402K
500C
steps
Agents
Agents
Figure 3: Training results in the three environments used for evaluation. We adopt the following perfor- mance metrics: in Switch, the sum of rewards obtained by all agents normalized to 1; in Cooperative Load Transportation, the distance of the load from the docking area as the percentage of task completion; in Predator-Prey, the number of prey captured.
For a fair comparison, we kept the number of prey constant at 16 by scaling the map size and the number agents, respectively: 12x12 with agents, 14x14 with agents, and 16x16 with 16 agents. The standard deviation is relative to the mean of the results of 5 different seeds.
policy to generate two output heads: one for the actions' probabilities and the other for the individualized probability of communication. On the other hand, ATOC adopts a two-step policy: first reasoning about incoming partial observations, then forming semi- rami-persistent communication groups by considering each other agent individually, and employing a. BiL.TMM (Graves & Schmidhuber 2005) module compute the final action.
5.3 Experimental Results
Figure Ssoows a performance comparison between CoMIX, the baselines and "CoMIX w/o comm', an ablated version without direct communication between agents whose architecture corresponds to the implementation of QMIX. Comparison with the latter and with IPPO, in which agents rely solely on local observations estimate their value function, also highlights the importance of the communication channel. In the Switch environment, the agents do not require complex strategic actions. In this casc, communication is mainly used to signal the presence of an agent. However, different agent behaviors can lead to very rapid task completion or longer episodes due to collisions and mutual blocking of agents, which increase the variance of reported results. In this casc, CoMIX consistently enables all agents to learn behaviors that allow for faster task resolution. Instead, IC3Net, adopting a simpler approach to coordination, learns deterministic action sequences causing agents behave with much variability. In contrast, ATOC's performance affected by its inability handle policies with conflicting goals, which is consequence of weight sharing between agents. Finally, IPPO shows competitive performance but bounded by the partial observability of the agents. We further evaluate CoMIX considering the Cooperative Load Transportation and Predator-Prey tasks at incremental scales reporting the average performance and standard deviation for all methods. In Cooperative Load Transportation. CoMIX outperforms ATOC and IC3Net baselines, demonstrating good abilities in reaching the goal location avoiding obstacles along the way. Even with an increasing number of agents, we demonstrate that able maintain good performance overall by filtering the communication channel from irrelevant messages coming from other agents. IPPO, on the other hand, shows clear superiority in resolution, being unaffected by external communication noise and taking advantage of the lack of marked randomness in the task objective. In the last environment, CoMLX learns how to coordinate both small and large numbers of agents by covering incremental portions of the map, whose size scales accordingly, while the number of preys remains constant. Interestingly, while ATOC does not perform well in coordinating few resources, it becomes competitive as the number agents present in the map increases. Meanwhile, both IC3Net and IPPO underperform as agents act strictly locally. In general, we notice that CoMIX is able to perform consistently across experiments. Additionally, we examine the importance of the communication component in the proposed method with a direct comparison with its ablated version. Despite the Mixer network still
8

---

Published in Transactions on Machine Learning Research (06/2024)
allows for information sharing at global level, notice how tasks like Load Transportation and Predator- Prey are affected by decrease in overall performance. This highlight the importance of local choices in such environments and the potential advantages of incorporating external information into the decision- making process. We observe that collaborative scenarios are infrequently observed. For example, in Cooperative Load Transporlation, avoiding an obstacle leads to numerous instances of movement disagreement, while in Prodator-Prey, agents prioritize local actions instead of forming teams. We also test a scenario which agents are initially deployed at distant locations in map with no prey present. By observing the exploration abilities of the agents, we notice that agents involved in communication often converge into groups compared to those that are not.
Table 1: Effects of communication channel disruption on the performance metric adopted for each envi- ronment. The performance metrics are as follows: in Switch, the sum of rewards obtained by all agents normalized to 1; in Cooperative Load Transportation, the distance of the load from the docking area as the percentage of task completion; in Predator-Prey, the number of prey captured. We show the performance of our method in relation to the reduction in the amount of messages exchanged and the results after op- timization process to handle missing or delayed messages. Depending on the environment, we are able to achieve better performance in terms of number of messages transmitted.
Comm channel usage
Switch
Transport (4 agents)
Predator-Prey (4 agents)
baseline
+ tiine-tuning
baseline +fine-tuning
baseline
+fine tuning
100%
Full comm
0.905
82.07
10.85
50%
0.903
0.907
79.07 4
82.58
10.78
10.90
25%
0.904
0.903
78.64 |
83.45
10.36 $
11.15
10%
0.898
0.905
77.38 1
84.06
10.10 4
10.82
0% No comm
0,900 ~M
0.907
77.78 4
79.35
10.34
10.70 ~
5.4
Analysis
5.4.1
Efficiency
An efficient algorithm should learn to avoid unnecessary communications, thus reducing the effects of in- formation overload. Figure 4 shows how many messages are typically considered when agents are asked to coordinate with others. Average usage varies depending on the task at hand, and we emphasize that we achieve this behavior by fully learning communication without imposing external constraints. In line with the task rules, we observe significant reductions in the number of messages used in Switch and Cooperative Load Transportation. More interestingly, we also notice similar behavior for the Predator-Prey task: al- though all agents share the general goal of capturing prey, the policy restricts each agent to communccating with only a subset of other agents, enabling the formation of smaller, more efficient subgroups to coordinate actions rather than requiring communication across every actor. As further ovaluation, added "noisy" agents to the environments, which send sequence of randomly generated bits, thus increasing the noise in the communication channel in the form of irrelevant messages. We note that their introduction does not substantially affect the average utilization of the communication channel, as expected, since such agents do not contribute to the solution of the task.
5.4.2 Communication
Since packet drops and intermittent availability of communications are not uncommon in the real world, we studied the ability of CoMIX to operate with a disrupted communication channel, in which messages are not delivered at every step. Table [ reports as baseline the results of average performing models trained in normal communication settings and evaluated reducing messages exchanges, up to complete drop of the communication channel. Sequences of messages are dropped randomly the process, providing in their place 'outdated" messages, i.c., the latest information transmitted by the agent. As expected, we notice slight drop in performance that increases communication decreases. On the other hand, fine-tuning the policy
9

---

Published in Transactions on Machine Learning Research (06/2024)
10
Switch Load transport Predator-Prey Noisy agents
0.9
listened messages
0.8
0.7
0.6
0.5
0.4
2
16 Communicating agents
32
Figure 4: Average number of messages accepted by each agent's Coordinator normalized by the number of real agents acting in the environment. By introducing fictitious "noisy agents" transmitting random bits into the commmication channel, we expect the value to remain unaltered.
weights according this new setting, we reverse the trend and even achieve improvements over the initial performance, as evidenced in Table [ To implement this procedure, we freeze the Coordinator weights, and randomly discard sequences of messages during training. Features of the messages communicated in their place (after two linear layers of processing) are then scaled proportionally to the delay times to reflect their temporal relevance. The process of fine-tuning the weights is performed until the performance matches the starting model under normal communication conditions, using a lcarning rate reduced by two order of magnitude. Again, Switch agents demonstrate to not rely particularly on the communication channel, while in the Predator-Pars Prey and Cooperative Load Transportation environments the fine tuning allows improved results with less communication, reaching reduction of -9%% in the latter. We also highlight the comparison of agents trained with communication enabled, but then disabled at run-time (Table last row), versus agents trained with no communication al all (Figure 3): message exchange enables agents to acquire skills, one can take advantage even in situations of no commuicatiinn
5.4.3 Flexibility
Compared with existing approaches that impose strict coordination, CoMIX offers greater flexibility and autonomy at the individual agent level, while still achieving effective coordination when needed. This flexibility is particularly advantageous for tasks where agents can operate independently for periods of time, but need collaborate effectively when opportunities arise. In the Predator-Prey environment, we observe examples of this flexible behavior emerging. Agents would often spread out initially to explore the space independently. However, when prey is detected, the agents demonstrate the ability to dynamically coordinate their actions, surrounding and capturing them through cohesive collaboration. This flexibility allow the agents to effectively explore the large joint action space rather than being constrained to strict coordination protocols. This does not have a negative impact on CoMIX's coordination performance as demonstrated by the experimental results. In Predator-Prey, CoMIX significantly outperforms baselines like IC3Net and IPPO that either enforce rigid communication structures or lack such mechanisms (Figure 3). Notably, CoMIX adjusts its communication and coordination levels based on task demands. It employs sparser communication in the Switch environment, while utilizing higher levels in Load Transport and Predator-Prey (Figure 4. The flcxibility of CoMIX is especially beneficial in sparse rewards settings. Without enforcing over-constrained collaboration, in settings like the one of Predator-Prey, with only 4 agents, CoMIX enables faster convergence, compared to baselines. At the same time, by allowing for independent local choices, it sidesteps the issue of insufficient exploration of individual actions, as evidenced by the performance the Switch environment. In general, the proposed method is suitable for all scenarios in which agents can effectively use a communication channel to coordinate their actions, because of its flexibility and robust to noisy communication channels.
10

---

Published in Transactions on Machine Learning Research (06/2024)
6 Conclusion
In this paper, we have introduced CoMIX, a new training architecture that effectively addresses coordination in multi-agent systems by separating selfish from collaborative behaviors. Through an extensive experimental evaluation by means of a variety of environments, we have shown the applicability of CoMIX to different tasks, and we have highlighted collaborative tasks in decentralized fashion. In addition, we have demonstrated the robustness of our approach to communication disruptions and its effectiveness in filtering unnecessary information before it is transmitted: these are important characteristies for potential real-world deployments.
ability to improve the performance of individual agents' policy in performing
References
Antonio L. Alfeo, Mario G. C. A. Cimino, Nicoletta De Francesco, Alessandro Lazzeri, Massimiliano Lega, and Gigliola Vaglini. Swarm Coordination of Mini-UAVs for Target Search Using Imperfect Sensors. Intelligent Decisions Technologies, 12(2):149-166) January 2018.
Takumi Aotani, Taisuke Kobayashi, and Kenji Sugimoto. Bottom-up multi-agent reinforcement learning for 3590-3595, 3595, 2018. doi: 10. (0.1109/5MC:2018.0000
selective cooperation. In IEEE SMC 2023, pp. 3590
Timothy Cason, Roman Sheremeta, and Jingjing Zhang. Communication and efficiency in competitive coordination games. Games and Economic Behavior, 76:26 09 2012. doi: 10.2139/sshi 1673177....
Qin Chen and J.Y.S. Luh. Coordination and control of a group of small mobile robots. In ICRA 1994, 1994.
Abhishek Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Mike Rabbat, and Joelle Pineau. Tarmac: Targeted multi-agent communication. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), ICML 2019. PMLR, 2019.
Ziluo Ding, Tiejun Huang, and Zongqing Lu. Learning individually inferred communication for multi-agent cooperation. In NewrIPS 2020, 2020.
Alessandro Falsone, Basak Sakeak, and Maria Prandini. Coordinated lane change in autonomous driving: computationally aware solution. IFAC-PAP 2020, 53(2))15211-15211 2020.
Jakob N. Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Coun- terfactual multi-agent policy gradients. In AAAI 2018/IAAL 2018/EAAI 2018. AAAI Press, 2018.
Alex Graves and Jürgen Schmidluber. Framewise phoneme classification with bidiroctional LSTM and other neural network architectures. Neural Nelworks, 18(5):602 610, 2005.
Frans C. A. Groen, Matthijs T. J. Spaan, Jelle R. Kok, and Gregor Pavlin. Real world multi-agent systems: Information sharing, coordination and planning. In Balder D. ten Cate and Henk W. Zeevat (eds.), ILLC 2007, pp. 154-165. Springer Berlin Heidelberg, 2007.
Sven Gronauer and Klaus Diepold. Multi-agent deep reinforcement learning: a survey. Artificial Intelligence Review, 55:895 943, 2021.
Jayesh K Gupta, Maxim Egorov, and Mykel Kochenderfer. Cooperative multi-agent control using deep reinforcement learning. In AAMAS 2017, pp. 66 83. Springer, 2017.
David Ha, Andrew Dai, and Quoe Le. Hypernetworks. In ICLR 2016, 2016.
Pablo Hernandez-Leal, Bilal Kartal, and Malthew E. Taylor. A survey and critique of multiagent deep reinforcement learning. AAMAS 2019, 33:750 797, 2019.
Edward Hughes, Joel Leibo, Malthew Phillips, Karl Tuyls, Edgar Dueñez Guzman, Antonio García Castañeda, Iain Dunning, Tina Zhu, Kevin McKee, Raphael Koster, Heather Roff, and Thore Grae- pel. Incquity aversion improves cooperation intertemporal social dilemmas. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa Bianchi, and R. Garnett (eds.), NeurIPS 2018, volume 31. Curran Associates, Inc., 2018.
11

---

Published in Transactions on Machine Learning Research (06/2024)
Jiechuan Jiang and Zongging Lu. Learning attentional communication for multi-agent cooperation. In NewrIPS 2018, 2018.
Aleksandra Kalinowska, Elnaz Davoodi, Florian Strub, Kory W Mathewson, Ivana Kajic, Michael Bowling, Todd D Murphey, and Patrick M Pilarski. Over-communccaee no more: Situated RL agents learn concise communication protocols. arXiv: 2211.01480, 2022.
Soummyy Kar, José M. Moura, and H. Vincent Poor. Distributed reinforcement learning multi-agent networks. In CAMSAP 2013, pp. 296 299, 2013. doi: 10.1109/ 10.1L09/CAMSAP.2013.. /0A///AAAAA........0000000
Woojun Kim, Jongeui Park, and Youngchul Sung. Communication in multi-agent reinforcement learning: Intention sharing. In ICLR 2021, 2021.
Sheng Li, Yutai Zhou, R. Allen, and Mykel J. Kochenderfer. Learning emergent discrete message communi- cation for cooperative reinforcement learning. In ICRA 2022, 2022.
Xinran Li and Jun Zhang. Context-aware communication for multi-agent reinforcement learning. In AAMAS 2024, 2024.
Toru Lin, Jacob Huh, Christopher Stauffer, Ser Nam Lim, and Phillip Isola. Learning to ground multi- agent communication with autoencoders. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), NewrIPS 2021, 2021.
Weiwei Liu, Shanqi Liu, Junjic Cao, Qi Wang, Xiaolei Lang, and Yong Liu. Learning communication for cooperation in dynamic agent-nunber environment. IEEE/ASME Transactions Mechalronics, 26: 1846-1857, 2021.
Yen-Cheng Liu, Junjiao Tian, Nathan Glaser, and 7solt Kira. When2com: Multi-agent perception via communication graph grouping. In IEEE/CVF CVPR 2020, pp. 4105-4114, 2020.
Haibo Min Long Ma, Shicheng Wang, Yuan Liu, and Shouyi Liao. An overview of research in distributed attitude coordination control. IEEE/CAA JAS 2015, 2015.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-compotiiie environments. In NewrIPS 2017, pp. 3882-6393, 2017.
Hangyu Mao, Zhibo Gong, Yan Ni, and Zhen Xiao. ACCNet: Coto--Coordiinato--Citii Net for 'Learning- to to-Communicate! with Deep Multi-agent Reinforcement Learning. 10X.::1706.3320 2017.
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Alex Graves, Ioannis Antonoglon, Daan Wierstra, and Martin A. Riedmiller. Playing Atari with Deep Reinforcement Learning. In NewrIPS Deep Learning Workshop 2013, 2013.
Yaru Niu, Rohan R. Paleja, and Matthew Craig Gombolay. Multi-agent graph-attention communication and teaming. In AAMAS 2021, 2021.
Frans A. Oliehoek and Christopher Amato. A Concise Introduction to Decentralized POMDPs. Springer, 1st edition, 2016.
Peng Peng, Ying Wen, Yaodong Yang, Quan Yuan, Zhenkun Tang, Haitao Long, and Jun Wang. Multiagent cintrectionalyy-oorddnnaa Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games. In NeurIPS Workshop on Emergent Communication 2017, 2017.
Emanuele Pesce and Giovanni Montana. Improving coordination in small-scale agent- deep reinforce- ment lcarning through memory-driven communication. Machine Learning, 17777777777-177, Sep 2020.
KJ Prabuchandran, Hemanth Kumar AN, and Shalabh Bhatnagar. Multi-agent reinforcement learning for traffic signal control. In ITSC 2014, pp. 2529 2534. IEFE, 2014.
12

---

Published in Transactions on Machine Learning Research (06/2024)
Chuhao Qin and Evangelos Pournaras. Coordination of drones at scale: Decentralized energy swarm intelligence for spatio-temporal sensing. Transportation Reseorch Part C: Emerging Technologies, 157: 104387, 2023.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning. Journal of Machine Learning Research, 21(1), January 2020.
Dorsa Sadigh, Nick Landolfi, Shankar S. Sastry, Sanjit Seshia, and Anca D. Dragan. Planning for cars that coordinate with people: leveraging effects on human actions for planning and active information gathering over human internal stale. Aulonomous Robols, 42(7):1405 1426, October 2018
Christian Schroeder de Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip H. S. Torr, Mingfei Sun, and Shimon Whiteson. Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge? Xiv: 2011.09533, 2020.
Herbert A Simon. The Sciences of the Artificial. MIT Press, 1996.
Amanpreet Singh, Tushar Jain, and Sainbayar Sukhbaatar. Learning when to communicate at scale in multiagent cooperative and competitive tasks. In ICLR 2019, 2019.
Norberto Sousa, Nuno Oliveira, and Isabel Praça. A Mueti-Agent System for Autonomous Mobile Robot Coordination. arXiv:2109. 0121109..2220000 12386, 2021.
Sainbayar Sukhbaatar, Arthur D. Szlam, and Rob Fergus. Learning multiagent communication with back- propagation. In NewrIPS 2016, 2016.
Karl Tuyls and Gerhard Weiss. Multiagent learning: Basics, challenges, and prospects. Al Maguzine, 33: 41-52, 2012.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double Q-lcarning. AAAT 2016, 2016.
Yutong Wang and Guillaume Sartoretti. Femnet: Full communication memory net for team-level cooperation
in multi-agent systems. In AAMAS 2022, 2022.
Yuxuan Wang, R.J. Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J. Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, Quoc Le, Yannis Agiomyrgiannakis. Rob Clark, and Rif A. Saurous. Tacotron: Towards End-to-End Speech Synthesis. In INTERSPEECH 2017, pp. 2017.
Zhi Yan, Nicolas Jouandeau, and Arab Ali Cherif. A survey and analysis of multi-robot coordination. International Journal of Advanced Robotic Systems, 10(12):399, 2013.
Jiachen Yang, Ang Li, Mehrdad Farajtabar, Peter Sunehag, Edward Hughes, and Hongyuan Zha. Learning incentivize other learning agents. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin Curran Associates, Inc., 2020.
(ods.), NeurIPS 2020, volume 33, pp.
Chao Yu, Akash Velu, Fugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and YT WU. The surprising effectiveness of ppo in cooperative multi-agent games. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), NewrIPS 2022, volume 35, pp. 24611-24624. Curran Associates, Inc., 2022.
Chao Yu, Xinyi Yang, Jiaxuan Gao, Jiayu Chen, Yunfei Li, Jijia Liu, Yunfei Xiang, Ruixin Huang, Huazhong Yang, Yi Wu, and Yu Wang. Asynchronous multi-agent reinforcement learning for efficient real-time multi- robot cooperative exploration. AAMAS 2023, 2023.
Kaiqing Zhang, Zhuoran Yang, and Tamer Bagar. Multi-agent reinforcement learning: A selective overview of theories and algorithms. Studies in Systems, Decision and Control, pp. 321 1844,, 2021.
Tianjun Zhang, Huazhe Xu, Xiaolong Wang, Yi Wu, Kurt Keutzer, Joseph E. Gonzalez, and Yuandong Tian. Multi-agent collaboration via reward attribution decomposition. arXim:2010.08559 2020.
13


Reviews:
[{"review_id": "vlyKOZ6aVa", "reviewer": "Reviewer_qAXm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The idea makes intuitive sense, and the authors establish that it does seem to be “actually working” — the agents are making use of the communication, and the coordinator is actually doing something to make sure useful messages get sent.\n\nWeaknesses:\n\n- Many of the ideas and experiments are verbally described but don’t include key details (i also checked the appendix). One example: the authors just refer to “a small MLP” in the appendix without specifying the size.\n\nBoth strengths and weaknesses:\n- The experimental results are mixed. The authors are quite clear and honest about this. It is difficult to compare RL algorithms with very different properties, and just because this approach does not conclusively win out over all competitors, does not mean that there is no contribution here.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors are interested in cooperative multi-agent RL, where agents see states and take actions independently but have a single shared reward. A common paradigm in this type of setting is “centralized training, decentralized execution”. A popular CTDE approach is known as QMIX, where agents’ individual Q-functions (trained according to the standard RL loss) are aggregated in a particular way into a combined Q-function which is used during training (at test time, policies maximizing the individual Q-functions are used). To this approach, the authors add a “coordinator” module, which can accept messages from agents and selectively send them to other agents to improve performance (the coordinator observes all messages to be sent and masks out some of them). The coordinator network is trained using a self-supervised “contrastive” loss to maximize the degree to which its mask helps increase Q-values. The authors show in experiments that their approach can perform competitively, they do some ablation studies on the coordinator, study how efficient the communications are, and consider a case where messages can be disrupted (this last experiment also shows that the message sharing is actually being used).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think it would be helpful for the authors to go through the paper again and look for places where they have been vague or left out technical details -- my guess is these can be corrected very easily.", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VLtDBOI5fe", "reviewer": "Reviewer_Km6W", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Addresses a challenging setting in MARL, namely when should agents coordinate in challenging settings, in potentially sparse interactions\n- Proposes an interesting communication scheme based on observations and action intentions \n- Fairly well written work \n\nWeaknesses:\n- Concerns on clarity and soundness \n\nConceptually, in comparison to QMIX, CoMIX introduces a communication step in the local agent Q networks (based on observation and next action sharing), together with a coordination module. This is claimed to allow agents to filter incoming information and tradeoff between taking independent or collaborative actions. \nMajor concern: In this context, what is the role of the Mixer network, and how does the end-to-end training procedure using the $Q^\\text{TOT}$ signal fit in the CoMIX framework? \n\nSection 4.2.2 was unclear. I failed to understand the motivation behind the inverted probabilities and the averaged pairwise communication losses. Also, how are the $n-1$ masks per agent impacting the training complexity? \n\nI appreciate the ablation, efficiency and resilience studies. However, it is difficult to understand the explanation on the fine-tuning procedure (on the resilience study), as well as the procedure and results of Figure 4 (on the efficiency side).\n\nFinally, I was wondering what is the impact on the exploration behaviour of the communication module in CoMIX, or the other way around (if agents commit to a certain action, but then take an exploratory step?). \n\n- Significance concerns \nFor the baselines, it is correct to state that CoMIX without communication is QMIX? \nUnless I am misunderstanding Figure 3, CoMIX does not seem to present performance improvements in the considered environments and does not scale well with the number of agents. \n\n\nOther remarks:\n- I did not find Fig. 1 b) to be clear, and if I understand correctly, it is a conceptual demonstration and not an actual slice from the inner workings of CoMIX?\n- There are some vague statements that would use more support in the paper presentation:\n    - \"Incremental policy approach\" - what is meant with this and how is this reflected in the technical constitution of CoMIX?\n    - \"balancing independence and collaboration\" - this one a major motivation point for CoMIX, but it is not strongly reflected in the experiments, for example the environment selection. \n    - the introduction and conclusion also discuss 'sparsely rewarded settings', but this is not investigated in the experiments\n- There are also some minor errors present:\n    - 'coordination among them become more...' -> becomes\n    - 'it represents a fundamental aspects...' -> aspect\n    - 'it represents...' and 'It is also critical' -> not clear what the 'it' refers to (perhaps coordination?)\n    - 'more optimal solutions' -> I would consider 'optimal' as non-gradable, so perhaps 'better optimised' is more suitable \n    - 'a solution that give more...' -> gives", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes Coordinated QMIX (CoMIX), a multi-agent reinforcement learning approach within the centralised training with decentralised execution paradigm that incorporates a communication step in the QMIX architecture and allows agents to trade-off collaborative and independent decision-making, when optimising their behaviour.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Overall, I would advise the following:\n- Solidify the targeted setting for CoMIX and make sure to investigate this in the experimental settings (e.g., settings requiring clear independent behaviour and only sparse collaborative moments)\n- The proposed communication scheme is the novel point of the work, however, it is lost inside the QMIX architecture, with the mixer network. In order to highlight the novelty and impact of the communication channel and coordinator, can this module be used without the mixer, in a different architecture?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "oD4Ni6sOJx", "reviewer": "Reviewer_LiJe", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. Extending QMIX by splitting the individual Q function into two parts and adding a multi-agent communication layer makes sense intuitively.\n2. The investigation of communication disruption is novel and is a good step toward practicality. \n\n\nWeaknesses:\n\n0. The key question \"Can agents acting in a shared environment with the ability to communicate autonomously learn coordinated behaviors...\" is already answered in the positive by CTDE and many of the cited works on MARL communication, so it's not clear why the paper treats this as an open question.\n1. The intuition behind the 2-step computation of the individual Q function is not backed up by more rigorous theory or analysis. In principle, having the communication layer at test time means CoMIX agents have more information than QMIX agents. But perhaps the mixer network in QMIX can achieve a similar effect during training, so that communication isn't needed at test time.\n2. The writing in Section 4.1.2 is confusing. The Coordinator module is not yet defined at that point, so it is not clear how the $c_{i,j}$ are computed from the pairs of messages.\n3. The \"self-supervision\" in equation (7) is insufficiently justified. Self supervision is usually understood as perturbing a sample and training a model to reconstruct the original sample based on context (e.g., BERT), thereby providing a context-based representation of individual parts of the sample (e.g. a word). Equation (7) is more like a contrastive loss that tries to increase the gap between the Q value with the actual message and a perturbed Q value from using another choice of message. The effect of this is not clear.\n4. Not clear if the Coordinator parameters $\\theta^C$ are trained purely by minimizing loss (7) or also by the regular TD error loss (6) or by some combination of both.\n5. The method is an extension of QMIX, but experiments do not compare with QMIX.\n6. Performance of CoMIX compared with the chosen baselines (IPPO, ATOC, IC3Net) are ambiguous. Sometimes a baseline is better, sometimes CoMIX is better. Maybe CoMIX is more suitable for some types of environments, but this isn't analyzed in more detail. So it's not clear when one would use CoMIX for a new environment.\n7. Writing $Q_i = Q_{self}Q_{coord}$ is strange if we view $Q$ functions as having \"physical units\" of reward. Reward squared doesn't make sense, just as how writing $F_1 = F_2F_3$ where each $F_i$ is a force with units of Newtons doesn't make physical sense. The authors probably mean that $Q_{coord}$ is a dimensionless coefficient, not an actual action-value function.\n8. Section 5.4.1 is speculation and doesn't add much to the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new method called Coordinated QMIX (CoMIX) for cooperative MARL in the centralized training decentralized execution paradigm with a communication channel. The individual Q function in QMIX is modified to be a product of two functions, one that produces an initial Q value based on the agent's own state, while the other is a coordination/communication model that uses all of the initial Q values of all agents to produce a scalar coefficient that changes the final Q values of each agent. The method is evaluated on three gridworld environments with between 2 to 16 agents, and performs mostly favorably compared with independent PPO and two other MARL communication methods IC3Net and ATOC.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "0. Justify equation (7)\n1. [Critical] Compare to QMIX\n2. [Critical] Clarify how the Coordinator is trained (using loss (7) or loss (6) or combination of both)\n    * [Critical] If Coordinator is trained by both (6) and (7), then the current ablation that removes the coordinator isn't enough to isolate the effect of loss (7). That means authors need to compare to an ablation of CoMIX where the coordinator is still present but the \"self-supervision\" loss equation (7) is removed, i.e. coordinator is trained only by the regular TD loss (6).\n3. More analysis of the kinds of environments where one should or should not use CoMIX, compared to other methods.\n4. [Critical] Some broad claims are not backed up by references, e.g.\n    * intro paragraph 2 \"In fact, coordination emerging from indiscriminate communication may not always be the best solution\"\n    * intro page 2 \"For instance, in crowded or sparsely rewarded settings, enforcing top-down coordination on all agents may hinder convergence to an optimal joint policy.\"\n    * intro page 2 \"In contrast, allowing potential independent local choices, rather than mandating group coordination, can lead to emergent collaboration arising naturally where needed...\", e.g. cite [1,2]\n5. [Critical] Provide evidence for the claim \"CoMIX’s ability to efficiently choose between collaboration and independent action\", e.g. scenarios where the communication layer caused the CoMIX agents to modify their initial Q values to take more cooperative actions, and scenarios where the communication layer can be ignored and agents just use greedy individual actions, and also show how the baselines behave in these cases.\n\n* [1] Inequity aversion improves cooperation in intertemporal social dilemmas. Hughes et al., NeurIPS 2018\n* [2] Learning to Incentivize Other Learning Agents. Yang et al. NeurIPS 2020", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers are supportive about the method and the experimental findings.
Decision: Accept (poster)