DIFFER:Decomposing Individual Reward for Fair Experience Replay in Multi-Agent Reinforcement Learning
Authors: Xunhan Hu
Jian Zhao
Wengang Zhou
Ruili Feng
Houqiang Li
Venue: NeurIPS 2023 poster (2023)
Keywords: Experience Replay; Reinforcement Learning; Multi-Agent System

Abstract: Cooperative multi-agent reinforcement learning (MARL) is a challenging task, as agents must learn complex and diverse individual strategies from a shared team reward. However, existing methods struggle to distinguish and exploit important individual experiences, as they lack an effective way to decompose the team reward into individual rewards. To address this challenge, we propose DIFFER, a powerful theoretical framework for decomposing individual rewards to enable fair experience replay in MARL.
By enforcing the invariance of network gradients, we establish a partial differential equation whose solution yields the underlying individual reward function. The individual TD-error can then be computed from the solved closed-form individual rewards, indicating the importance of each piece of experience in the learning task and guiding the training process. Our method elegantly achieves an equivalence to the original learning framework when individual experiences are homogeneous, while also adapting to achieve more muscular efficiency and fairness when diversity is observed.
Our extensive experiments on popular benchmarks validate the effectiveness of our theory and method, demonstrating significant improvements in learning efficiency and fairness. 
Code is available in supplement material.

DIFFER: Decomposing Individual Reward for Fair Experience Replay in Multi-Agent Reinforcement Learning
Xunhan Jian Zhao', Wengang Zhou Ruili Feng Houqiang Li
University of Science and Technology of China. 2 Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, 3 Allibaaa Group {cathyhxh, 2j 1401 uste. edu. en, zhug@uste. edu. en, ruilifengustc@gman com, lihq@ustc. edu. cn
Abstract
Cooperative multi-agent reinforcement learning (MARL) challenging task, agents must learn complex and diverse individual strategies from a shared team reward. However, existing methods struggle distinguish and exploit important individual experiences, as they lack an effective way to decompose the team reward into individual rewards. To address this challenge, we propose DIFFER, a powerful theoretical framework for decomposing individual rewards to enable fair experience replay in MARL. By enforcing the invariance of network gradients, we establish a partial differential equation whose solution yields the underlying individual reward function. The individual TD-error can then be computed from the solved closed-form individual rewards, indicating the importance of each piece of experience in the learning task and guiding the training process. Our method elegantly achieves an equivalence to the original learning framework when individual experiences are homogeneous, while also adapting achieve more muscular efficiency and fairness when diversity is observed. Our extensive experiments on popular benchmarks validate the effectiveness of our theory and method, demonstrating significant improvements in learning efficiency and fairness. The code available in httpss//giiubbcoomm//ahhhhhhh//f
Introduction
In widely adopted cooperative multi-agent systems 144 team agents typically operates under the constraint of individual local observations and limited communication capabilities. In each time step, all agents collectively interact with the environment by taking team actions, leading to transition the subsequent global state and the provision of team reward. Consequently, the agents must acquire the ability to effectively coordinate their actions and maximize the cumulative team return. Over the past few years, cooperative multi-agent reinforcement learning (MARL) has demonstrated remarkable achievements in addressing such challenges and has exhibited promising potential across diverse domains, including multi multi-player games (9.9) autonomous cars [00
traffic light control 12 13
economy pricing [14] and robot networks 13 16].
In recent years, value factorization methods 17-20 have emerged as leading approaches for address-
ing cooperative multi-agent tasks 23]. These methods are rooted in the centralized training and decentralized execution (CTDE) paradigm (4 Specifically, each agent's network models the
'Corresponding authors: Wengang Zhou and Houqiang Li
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

individual action-value using the agent's local action-observatie history 27 and the last action. These individual action-values are then integrated into a global state-action value through a mixing network. The parameters of both the agent networks and the mixing network are optimized by minimizing the team TD-error. To enhance data utilization, value factorization methods commonly employ experience replay. Here, experience represents the atomic unit of interaction in RI., i.c., a tuple of (observation, action, next observation, reward) [28]. In traditional MARL experience replay algorithms, the replay buffer stores team experiences and allows algorithms to reuse them for updating the current network, which encompasses both the agent networks and the mixing network.
However, the utilization of team experience overlooks the inherent differences among individual experiences, thereby failing to leverage individual experiences worthy of learning. This issue becomes especially pronounced when the agents within the team exhibit heterogeneity in their roles and capabilities. To develop clearer under- standing, let us consider the scenario of MMM2 (as shown in Fig. D within The StarCraft Multi-Agent Challenge (SMAC) environment [29]. In this setup, team com- prises 1 Medivac (responsible for healing teammates), 2 Marauders (focused on offensive actions), and 7 Marines (also dedicated to attacking). Each unit is controlled by a distinct agent, and the team's objective is to eliminate all enemy units. Learning an effective strategy for the Medi- vac unit proves particularly challenging due to its unique sub-task and limited number. Consequently, the individual experiences of the Medivac unit are deemed more likely to provide valuable insights for learning. Unfortunately, extracting these individual experiences from the team experiences proves difficult in the aforementioned MARL setting, as the individual rewards associated with each individual experience are unavailable.
Figure 1: Screenshots of the MMM2 sce- nario in SMAC, highlighting the impor- lance of individual experiences for the Medivac unit, given its unique sub-task and limited number.
In this paper, introduce novel method called Decomposing Individual Reward for Fair Experience Replay (DIFFER), to address the above challenges associated with multi-agent re- inforcement learning (MARL) experience replay. The fairness in MARL experience replay refers to the equitable sampling of individual experiences based on their importance. DIFFER addresses this by decomposing team experiences into individual experiences, facilitating fair experience replay mechanism. We begin by proposing method to calculate the individual reward for each individual experience, ensuring that DIFFER maintains equivalence with the original learning framework when the individual experiences are indistinguishable, To achieve this, we establish a partial differential equation based on the invariance of agent network gradients. By solving this differential equation, we obtain an approximation function for the individual rewards. Consequently. the team experience can be successfully decomposed into individual experiences. These individual experiences are then fed into a fair experience replay, where they are sampled for training the agent networks. The sampling probability is determined proportionally to the temporal difference (TD) error of each individual experience. Moreover, it adapts effectively to enhance both efficiency and fairness in scenarios where diversity is observed. Importantly, DIFFER a generic method that can be readily applied to various existing value factorization techniques, thereby extending its applicability in MARL research.
We evaluate DIFFER on a diverse of challenging multi-agent cooperative tasks, encompassing StarCraft II micromanagement tasks (SMAC) (29). Google Football Research (GRF) (30), and Multi-Agen Mujoco [31] tasks. Our experimental findings demonstrate that DIFFER substantially enhances learning efficiency and improves performance.
2 Preliminaries
Problem Formulation. A fully cooperative multi-agent problem can be described a decentralized
partially observable Markov decision process (Dec-POMDP) (32), which consists of a tuple G =<
S,S, A, P, r, Q2 N,7 >. I is the finite set of N agents. For each time step, each agent e
T= ... N} receives individual observation o; according to the observation function O(s, i) =
Sx Ω. Then, each agent determines its action with individual policy (r(a; (o:) Q -> [0, 1].
Given the team action a {or,.... ,ay). }. the environment transits to next state according to the

---

SxAA + R. which is shared by all agents. Each team policy π oa [T (0) ) T ( |oN]] has team aetion-value function Q (o', s*) E 1s ar. [Return' (s' a a²)] where Return' (s' a²) Ei=o x B is a discount return and 7 is the discount factor. The target a Dec-POMDP problem is to maximize the accumulated return of the team policy, i.e. optimal team policy x arg max Return(T)) = (eso)d(s (Qiot (o', a {²)], where d(so) represents the distribution of initial state.
transition function P(s' s, a) S SxAAxx XS - [0, 1] and gives team reward
Multi-Agent Deep Q-learning. Deep Q-learning [33 uses a neural network parameterized by O to represent the action-value function Q. In the multi-agent Dec-POMDP problem, multi-agent store experience (o, a, o', Rt). Here R is the team reward received after taking team action a with team observation o, and o' represents team observation in the next step. During the training process, the parameter 0 is updated by minimizing the Temporal Difference (TD) error with mini-batch of data sampled from the replay buffer, which is shown as (!) (o Dl(r + (Qo'; 0-) Qo, a; 0))²]. where Q(o'; max,' Q(o', a'; 0)) is the one-step expected future return of the TD target and 0- is the parameter of the target network [35] |r + (Q0: 0 Q(o, a: 0) is known TD-error.
deep Q-learning methods usually adopt the replay buffer 34
Multi-Agent Value Factorization. The value factorization framework comprises N agent networks (one for each agent) and a mixing network. At time step t, the agent network of agent takes the individual observation of thereby computing the individual action-value, denoted Q.. Subse- quently, the individual action-values (Qitie are fed into the mixing network, which calculates the team action-vatt -vauu, denoted as Q The mixing network is constrained by Individual-Glose (IGM) principal[[] [I, ensuring the congruity between the optimal team action deduced from Quot and the optimal individual actions derived from {Qi)ier The N agent networks share parameters. We denote Om and 0p as the parameters of mixing network and agent networks, respectively. During the training phase, the mixing network is updated based on the TD loss Quot, while the agent networks are updated through gradient backward propagation. During the execution phase, each agent takes action with individual policy derived from its agent network in a decentralized manner.
3 Method
In this section, we present DIFFER, a novel experience replay method designed to foster fairness in multi-agent reinforcement learning (MARL) by decomposing individual rewards. DIFFER offers solution to enhance the adaptivity of MARI. algorithms. Instead of relying solely on team experiences, DIFFER trains the agent networks using individual experiences. This approach allows for a more granular and personalized learning process.
At a given time step, agents interact with the environment and acquire team experience team (o, a, o') = ((os)sez (a,)ez,, (o))se, R) = ((o,, a,, of) R). Here, o;, ag and of represent the observation, action, and next observation for each agent E I, respectively. These team experiences are stored in a replay buffer and subsequently sampled to train the mixing network and agent network parameters. We aim to decompose multi-agent team experience into N single-agent individual experiences (x_nd agent networks.
ind Jier = {(o,, ai, 04 rt) )ir. then utilize these individual experiences to update the
The DIFFER framework comprises two stages: (1) Decomposer: Calculating individual rewards to decompose team experiences; (2) Fair Experience Replay: Selecting significant individual experiences to train the agent networks.
3.1 Exploring Individual Rewards through Invariance of Gradients
The DIFFER framework necessitates individual rewards fraker to decompose the team experience x tcam into individual experiences {xind Jez. However, in the context of shared team reward POMDP problems, these individual rewards are not readily available. Therefore, the key challenge for DIFFER is to devise a strategy for calculating individual rewards.
In the traditional experience replay methods, agent networks are updated by the team TD-loss, which is calculated based on team experience and denoted as Liot (R+ a'; (00) Qror (,);
3

---

Ptot
LTD
Mixing Network
Data Flow
Pammetzr-tharing
1
....... Gradicat Flow
Mixing Network
Q1 f
QN
TD Lin
ti
Pttt
L
Q1
QN
Agent
Agent N
Mixing Network
On. 4/
t
Q1
QN
Agent
Agent N
Fair Experience Replay 4) ((()")))
Agent 1
Agent
01. d1
Oy. Aw
Tedisiiaal Ti-ewrkk
Indesiial Diverredy
zi Cnyy muoott
Xi (on:Ru.0hee1))
o
ON Ag
poats
(1 of).
(Op, (.R)
Decomposer
((0, ag of
(o ax %)..R
iuum (i (0p, op.. ).))
f
Replay buffer
Replay buffer
Replay buffer
(a) Training Mixing Network.
(b) Training Agent Networks.
(c) Traditional Method.
Figure 2: (a) In DIFFER, the mixing network is trained using the team TD-loss Lio. while the parameters of the agent networks remain unchanged. (b) The decomposing individual experiences are inputted into Fair Experience Replay, from which they are sampled to calculate the individual TD-loss Lnd for training the agent networks. (c) In traditional MARL experience replay methods, the mixing network and agent networks are updated using the team TD-loss LTD derived from team experiences.
However, in DIFFER, agent networks are updated using the individual TD-loss, which is calculated based individual experience and denoted preserve the overall optimization objective of value factorization methods, it is desirable for an ideal individual reward approximation strategy to maintain invariance between the gradients of Leot (before decomposition) and the sum of L; (after decomposition) with respect to the parameters of the agent networks. The optimization equivalence can be formulated as follows:
(r: + +Qi(d, d'; d;;07) Qi a;; 0p)) To
T Σ 0L. iET 00p²
(Invariance of Gradient)
00p
where 6p represents the parameters of the agent networks (noting that the agent networks of each agent share parameters). It is important emphasize that this discussion focuses solely on the gradient of a single team experience. By solving the above partial differential equation, we are able to obtain an individual reward function that satisfies the invariance of gradients.
Proposition 1. The invariance of gradient in Equ. 7 is satisfied when individual reward of agent i is given by:
QQre
r; = (R+ 7800 Quoi) aQ,
20, + Qi.
for any agent i E
rigorous proof and analysis above proposition can be found in Appendix. Therefore, we approximate the individual rewards r; for individual experiences x while preserving the original optimization objective. In this way, a team experience team is decomposed into individual experiences {xind) J successfully.
N
3.2 Fair Experience Replay Guided by Individual Experiences
15=0 After decomposition, we obtain a set of individual experience E [xj N from team experience
mini-batch. Here, j denotes the index of the individual experience in the set, B represents the mini- batch size, and N indicates experience replay, method that selects significant individual experiences from E to train the agent network. Similar to Priority Experience Replay (PER) method (28), the TD-error of each individual experience is used metric to measure its significance. A larger TD-error for individual
number of agents involved. Our objective
to construct a fair

---

Algorithm Fair Experience Replay.
Require: The agent set I, the maximum steps bmax, the mini-batch and target network update period
M;
1: Initialize replay memory D. set tste 0:
2: Initialize the mixing network, agent network and target network with random parameters;
3: while fstep S max do
4: For each episode, observe initial centralized observation {o, Jer:
5: set t 0;
6:
while Not terminal do
7:
With probability select random action af, otherwise a; arg max Qi(ot, af) for each
agent i;
8:
Take action a', then receive team reward R2 and observation fo heii
9:
t-t +1;
10:
end while
11:
bstep f- kstep + 4;
12:
Insert the current episode data into buffer D;
13:
Sample a random episode mini-batch of team experiences Bteam from D;
14:
Calculate total TD-loss Tott as Equ. (6) and update parameters of mixing network;
15:
Calculate individual reward as Equ. and decompose team experience mini-batch Bteam to individual experience E'
16:
Calculate sample probability as Equ. 3 of each individual experience in mini-batch E;; Calculate the sample ratio Neep and sample significant individual experiences from E;
Calculate individual TD-loss T Lind as Equ. 7 and update parameters of agent networks;
19:
Update target network parameters with period M.
20:
end while
experience indicates that it is more valuable for the current model, and thus more likely to be selected for training. The individual experience x, is sampled with probability:
P
P(x) (ud)
ΣxPE
where P, = 1ō + c. 8, is the TD-error of xind and e a small positive constant in case TD-error is zero. The syper-parameter sample cases if = O. To correct the bias, ind is weighted by peportaacc--smplnngg (IS) weights as follows:
determines the degree of prioritization. It degenerates to uniform
1 N
)8 max wk.
P(x
The hyper-parameter 33 anneals as introduced in PER.
In cases where the agents within team are homogeneous, their individual experiences tend to exhibit similar TD-errors. As a result, these individual experiences are assigned similar probabilities during the fair experience replay process. In such circumstances, the fair experience replay approach employed by DIFFER effectively degenerates into traditional method that utilizes team experiences.
Let E' denote the selected individual experience set, which is subset of the original individual experience E. We define the sample ratio as 17 := #E' # where # #E' and # are the numbers of individual experiences in E' and E, respectively. Since the model is initially at a low training level and Q-value estimation errors are large, only a small portion of the individual experiences is worth training. However, this portion increases as the training progresses. Thus, motivated by the warm-up technique proposed [6), we set 7) < 1.0 at the beginning of training and gradually increase as the number of training steps increases. The sample ratio nt: for given training step t; can be expressed as follows:
Mstart + (Nend Tistart) t plmax
t, ptmax
nc:
(5)
nend:
t, 2 pfmax
Here, the hyper-parameters Vsttrtt and Mend denote the initial and final values of the sample ratio, respectively. The hyper-parameter is the proportion of the time steps at which the sample ratio
5

---

increases, and fmax is the overall number of training steps. The ablation studies regarding the warm-up technique are provided in Appendix.
3.3 Overall Training and Evaluation
During the training phase, we first sample a team experience mini-batch Bleam from the replay buffer. The parameters of mixing network O. are optimized with Liot calculated on the mini-batch by
LTD (0m) Extomm=aa..dd B B (R+ Qior (o', km) Qtor a; 0m)))²
(6)
Next, we approximate the optimization equivalence of individual rewards of each individual expe- rience Equ. (8) and decompose team experience into multiple individual experiences. Among them, we select a significant individual experience set E" with probability in Equ. (3. The parameters of agent networks Pp are optimized by
L L (0p) Σ
Wa(Ts + -1(1;00)-00(0,,40)
where wy is a weight assigned to individual experience x) calculated in Equ.
During the inference phrase, agent i chooses a greedy action according to its individual action- Qi for decentralized execution. Therefore, the DIFFER framework meets centralized training and decentralized execution. The overall training and evaluation are presented Algo.
Experiments
In this section, we conduct several experiments answer the following questions: (1) Are the decomposing individual rewards calculated using Equ. (8) optimization equivalent to the team reward
experimentally? (Sec. 4.2) (2) Can DIFFER improve the performance of existing value factorization
methods compared to the baselines across different environments? (Sec. 4.3) (3) Can DIFFER
determine a more reasonable agent policy compared to the baselines? (Sec. 4.3 (4) Can DIFFER
successfully distinguish and exploit the important individual experiences? (Sec. 4.4
For every graph plot, the solid line shows the mean value and the shaded areas correspond the min-max of the result on random seeds. All the experiments are conducted on Ubuntu 20.04.5 server with Intel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz and GeForce RTX 3090 GPU. The code
is available in
4.1 Experiments Setting
Training Environments. We choose discrete action space environments The StarCraft Multi-Agent Challenge (SMAC) (29] and Google Football Research (GRF) (30), as well as a continuous action space environment Mulli-Agent Mujoco (MAMujoco) 31 to conduct our experiments. SMAC is currently a mainstream cooperative multi-agent environment with partial observability. We use the default environment setting for SMAC with version SC 2.4.10. GFR is a platform that facilitates multi-agent reinforcement learning in the field of football. We conduct the experiments on two academy scenarios in GRF. MAMujoco is a continuous cooperative multi-agent robotic control environment. Each agent represents a part of a robot or single robot. All of the tasks mentioned in this work are configured according to their default configuration.
Base Models. We select QMIX M. COMIX B1, QPLEX (2) and MASER [3 as our base models for comparison. QMIX and QPLEX are renowned value factorization methods that have been widely used in discrete action space environments. COMIX is an extension of QMIX specifically designed for continuous action space environments. These three base models employ team experience train their overall value factorization frameworks, illustrated in Fig. ([.). Furthermore, we include MASER, a MARL experience replay method, our evaluation. Similar to our methods, MASER generates individual rewards and trains agent networks using individual experiences.
Implementation Hyper-parameters. All base models were implemented by faithfully adhering to their respective open-source codes based PyMARL. Regarding the warm up trick for the sample

---

GMIX divids
QNEN civile
QMIX- divid
We rote
Wee Tee Rate
Moun Tol We Rib
TirserpeaM)
(a) 27m__s 1rm_vs_3mm
(b) MMM2
(c) 2c_vs_64z
(d) 3s57_vs__3a0
Figure 3: Performance comparison of QMIX-divide and QMIX on SMAC scenarios, highlighting the
optimization equivalence between team reward and individual reward calculated by DIFFER.
48
QMIX
MAIIX-DUTERRCOM (NED))
ONEN
D0EX--08F121 (oun))
Mear Teal We Rate
QMIX
QMIX--DI CIIIL-DIPEERRCC (Q.2R)
QMIX
QMEX TIETER
MASIR
MASSER
MARIE
MASER
CM
411 Trmustapa000
(a) 27m__v vs_30m
(b) MMM2
(c) 2c__s_____________ vs__47g
(d) 3x52_ vs_3s6z
Figure 4: Performance comparison of DIFX-DI (ours) and QMIX on SMAC scenarios, highlighting the performance improvement of our method towards QMIX.
ratio, we initially set the ratio 0.8, which gradually increases linearly until reaching 1.0 60% of the total training steps. From that point onwards, the sample ratio remains fixed at 1.0 for the remainder of the training process. For comprehensive overview of the hyper-paramette settings employed in our experiments, we refer readers to the appendix.
4.2 Optimization Equivalence of the Individual Experiences
In this section, our objective is to investigate whether the decomposition of individual experiences in DIFFER yields an equivalence to team experiences within the original learning framework. To accomplish this, we focus on the QMIX as the base model, which updates agent networks using the team TD-loss computed from team experiences. We introduce a variant model named -ivix. In QMIX-divide, the agent networks are updated using the individual TD-loss calculated from all decomposed individual experiences. Similar to QMI,, the mixing network of QMIX-divide is updated using the team TD-loss. Compared with DIFFER, QMIX-divide preserves the decomposer of team experiences but omits this selection phase in the fair experience replay. Our aim is to test the hypothesis that the decomposed individual experiences have impact agent network optimization. We compare the performance of QMIX- divide and QMIX on SMAC scenarios, as depicted in Fig. B Our results demonstrate that QiVX-divide and QMIX exhibit nearly identical performance, providing evidence in support of the optimization equivalence between the team reward and the approximated individual reward. Furthermore, experience replay, as the omission of this phase in QMIX-divide leads to comparable performance with the baselines QMIX.
highlight the critical role played by the selection phase in the fair
4.3 Performance Comparison
Performance Improvement towards Value Factorization Methods. We investigated whether DIFFER could enhance the performance of established value factorization methods. To evaluate this, we conducted experiments using QMIX, QPLEX, and COMIX as the base models in various environments. The training curves for each model are depicted in Fig. 4 Fig. 5 Fig. 6 and Fig. / As illustrated in the figures, DIFFER exhibited notable improvements both learning speed and overall performance compared to the baselines during the training process. The magnitude of improvement varied across different scenarios. Notably, DIFFER demonstrated a more significant performance boost in scenarios where agents possessed distinct physical structures (e.g., agents with different
7

---

GTLEN DUTTER F040S QULEX
QULEX DUFTEEIRRRRCCO OULEX
Mane TaenTTeni sertee Rare
Mean Too
IX (oun
www.
Trrestepsh
(a) 27m 30m
(b) MMM2
(c) 2c VS 64zg
(d) 3s52 vs 3s62
Figure 5: Performance comparison of OPLEX-DIFFED (ours) and QPLEX on SMAC scenarios,
highlighting the performance improvement of our method towards QPLEX.
DIFTERIFEERROO QMEX
Moin Teat We Rub.
Map Tokkiie Aue
unin Mare Maraaa
KOOMECCCPPE CHEPER
TinsespoM))
(a) 3 with keeper
(b) counterattack hard
(a) Humanoid-v2_0
maayagent_swimmett
Figure 6: Performance comparison of QMIX- DIFFER (ours) and QMIX on GRF scenarios, highlighting the performance improvement of our method towards QMIX.
Figure 7: Performance comparison of COMIX- DIFFER (ours) and COMIX on MAMujoco sce- narios, highlighting the performance improve- ment our method towards COMIX.
joints in Humanoid v2 0) or subtasks (e.g., agents with different skills in 3s5z vs 3s6z). This observation aligns with our hypothesis that scenarios with greater dissimilarity among agents result in more pronounced improvement through the distinction of individual experiences facilitated by DIFFER. Conversely, scenarios where agents were homogeneous (e.g., 27m_ vs 2s_33mmmm in SMAC), DIFFER exhibited performance similar to that of the base model. In such cases, the lack of significant differences among agents rendered the decomposition of individual experiences less impactful. Overall, our findings demonstrate that DIFFER effectively enhances the performance of established value factorization methods. The degree of improvement depends on the specific characteristics of the scenarios, emphasizing the importance of considering the heterogeneity of agents when applying DIFFER.
Comparison with MARL ER methods. In Fig. 4 we present a comparison of the performance of DIFFER and MASER. While both models implement individual rewards and leverage individual experiences to update agent networks, they differ in their optimization objectives. Specifically, DIFFER retains the original objective of maximizing the team action-value Qtot- In contrast, MASER aims to maximize a mixture of individual action-value Qi and team action-value Qiat. which alters the original optimization objective. As a consequence of this shift in optimization objective, the performance of MASER is observed to be worse than DIFFER in challenging environments.
Nomekkad bnbsitud TDwne
Figure 8: Screenshots QMIX (left) and QMIX-DIFFER (right) on SMAC scenario MMM2. The red team is controlled by trained model, while the blue team is controlled by a built-in AI. The Medivac in the red team is marked by orange square. The red team is controlled by QMIX the left subfigure and by QMIX- DIFFER in the right subfigure.
Timestepo/44
Figure 9: The normalized stan- dard deviation (std) of TD-error of individual experiences.

---

Case Study. The screenshot in Fig. 8 shows the game screen for the QMIX and QMIX AMIX-DIFFER models in the SMAC scenario MMM2. As previously noted, learning an effective strategy for the medivac unit is particularly challenging due to its distinct sub-task and limited number of units. A successful medivac strategy must incorporate a viable movement strategy that avoids enemy attacks and reduces unnecessary damage, well as sound selection of healing objects that maximizes team damage output. In the QMIX controlled agent team (Fig. S left), the medivac failed to learn an appropriate movement strategy, moving too close to the enemy team and dying early in the episode, As result, the other agents bled out quickly and the episode ended in defeat. contrast, effective medivac
the agent team controlled by QMIX-DIFFED strategy, positioning the unit in a safe location to avoid enemy attacks and providing healing support throughout the game. The resulting advantage for the red team led to final victory.
(Fig. 8, right) demonstrated
4.4 Analysis for TD-error of Individual Experiences
In this section, we delve into the differentiation of individual experiences by analyzing their TD- errors. Throughout the training process, we compute the standard deviation (std) of the TD-errors of the decomposing individual experiences and normalize them using the mean TD-error. Figure g showcases the training curve of the normalized std of individual TD-errors. Notably, we observe a substantial increase in the normalized std at the initial stages of training, which remains consistently high until the conclusion of the training process. This observation underscores the substantial distinctions that exist among individual experiences when evaluated based on their TD-errors. Our DIFFER framework, by decomposing individual rewards, effectively captures and exploits these distinctions among individual experiences. Consequently, DIFFER exhibits superior performance compared to the baseline models. These findings highlight the efficacy of DIFFER in leveraging the differentiation among individual experiences to enhance learning outcomes in MARL settings.
4.5 Visualization of Individual Rewards Produced by DIFFER
In order to provide a clear visual repre- sentation of the individual rewards ac- quired through our proposed DIFFER method, we introduce game screen- shots alongside the corresponding in- dividual rewards for each agent, as illustrated in Fig. [0] For the purpose of analysis, we consider two distinct timesteps of an episode within the MMM2 scenario. Our team comprises Marauders (index 0-1), 7 Marines (index 2-8), and Medivac (index 9). The team rewards assigned to agents are proportional to the damage inflicted on enemy units and the num- ber of enemy unit deaths. During the initial stage of the episode (timestep 4, (a)), our units possess sufficient health and do not require assistance from the Medivac. Consequently, the contri- bution of the Medivac to the team's performance is minimal. As a result, the individual reward attributed to the Medivac is relatively lower compared to the Marauders and Marines, who play active roles in offensive opera- tions. Agents 2-5, being in closer proximity to the enemy and inflicting substantial damage, enjoy higher individual rewards due to their effective engagement strategies. At timestep 14 (b), except for agent 7 and 8 who have perished, the individual rewards assigned to each agent are substantially balanced. This indicates a comparable level of contribution from all agents towards achieving team objectives at this particular point in the game. The persistent correlation between individual rewards
Indiviouaa Reward
Agent Index
timestegr-
Individual Reward
Agent Indox
(b) timestep-14 14
Figure 10: The visualization of individual rewards produced by our DIFFER method, along with game screenshots cap- tured at timestep 4 and timestep 14 of the MMM2 scenario. The index of each agent have been marked in orange number. The consistent correlation between individual rewards and the ongoing game situation serves as a testament to the efficacy and rationality of our DIFFER method.

---

and the dynamic in game circumstances not only reaffirms the effectiveness of our DIFFER method but also validates its rationality.
Related Work
Experiment Replay (ER) is widely used mechanism for improving data utilization and accelerating learning efficiency in reinforcement learning (RL.. ER reuses the historical experience data for updating current policy 34 38) In multi-agent RI,, using single-agent ER algorithm directly to obtain individual experience is naive if agents can obtain accurate individual rewards from the environment [39 40). However, in the multi-agent problem setting addressed in this work, team of agents only has access to shared team reward, making it impossible to obtaín an accurate individual reward. Consequently, multi-agent ER algorithms employ joint transitions as minimal training data 11 MASER (37 is a method similar to ours, as it calculates individual rewards from the experience replay buffer and uses individual experience to train the individual value network. However, MASER generates individual rewards to maximize the weighted sum of individual action- value and team action-value, which may violate the original optimization target of multi-agent RL, i.e., maximizing the team action-value. In our work, we propose a novel strategy for approximating individual reward and demonstrate its optimization equivalence. To our knowledge, DIFFER the first work to approximate individual reward in way that ensures optimization equivalence.
6 Conclusion
Limitation. A limitation of the DIFFER method is the introduction of additional computation when calculating the individual reward and TD-error. This increased computational cost is an inherent trade-off for the benefits gained in terms of fairness and improved learning efficiency. The additional computational burden should be considered when applying this method in resource-constrained settings or scenarios with strict real-time requirements.
In conclusion, this paper presents the DIFFER framework. By decomposing the team reward into in- dividual rewards, DIFFER enables agents to effectively distinguish and leverage important individual experiences. Through the invariance of network gradients, we derive partial differential equation that facilitated the computation of the underlying individual rewards function. By incorporating the solved closed-form individual rewards, calculate the individual lemporal-difference error, provid- ing the significance of each experience. Notably, DIFFER showcases its adaptability in handling both homogeneous and diverse individual experiences, demonstrating its versatility in various scenarios. Our extensive experiments on popular benchmarks demonstrate the effectiveness of DIFFER in terms of learning efficiency and fairness, surpassing the performance of existing MARL ER methods. We hope that our work provides a new perspective on experience replay methods in MARL, inspiring further research and advancements in the field.
Acknowledgments
This work is supported by National Key R&D Program of China under Contract 20227D01 19802, and National Natural Science Foundation of China under Contract 61836011. It was also supported by GPU cluster built by MCC Lab of Information Science and Technology Institution, USTC, and the Supercomputing Center of the USTC.
References
[1] T. Rashid, M. Samvelyan, C. Schroeder, G. Farquhar, J. Foerster, and S. Whiteson, "Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning." in Inter- national Conference on Machine Learning (ICML), 2018.
[2] J. Wang, Z. Ren, T. Liu, Y. Yu, and C. Zhang, "Qplex: Duplex dueling multi-agent q-learning," International Conference on Learning Representations (ICLR), 2021.
[3] Foerster, Farquhar, T. Afouras, Nardelli, and Whiteson, "Counterfactual multi-agent policy gradients," in AAAI conference artificial intelligence (AAAI), 2018.
10

---

[4] R. Lowe, Y. I. Wu, A. Tamar, J. Harb, O. Pieter Abbeel, and 1. Mordatch, "Multi-ageni actor-critic for mixed cooperativ--compettive environments, Advances in Neural Information Processing Systems (NeurIPS), 2017.
[5] D. Ye. G. Chen, W. Zhang, S. Chen, B. Yuan, B. Liu, Chen, Z. Liu, F. Qiu, H. Yu, al., "Towards playing full moba games with deep reinforcement learning," Advances in Neural Information Processing Systems (NeurIPS), 2020.
[6] C. Berner, G. Brockman, B. Chan, V. Cheung, P. Debiak, C. Dennison, D. Farhi, Q. Fischer, S. Hashme, C. Hesse, et al., "Dota 2 with large scale deep reinforcement learning," arXiv preprint arXiv: 19/2.06680, 2019.
[7] N. Bulling and V. Goranko, "Combining quantitative and qualitative reasoning in concurrent multi-player games," Autonomous Agents and Multi-Agent Systems (AAMAS), 2022.
[8] Y. Zhao, J. Zhao, X. Hu, W. Zhou, and H. Li, "Douzerot: Improving doudizhu ai by opponent modeling and coach- saachg learning," IEEE Conference on Games (CoG). 2022.
[9] Y. Zhao, J. Zhao, X. Hu, W. Zhou, and H. Li, "Full douzero+: Improving doudizhu ai by opponent modeling, coach- bacch-guded training and bidding learning," IEEE Transactions on Games (CoG), 2023.
[10] Y. Cao, W. Yu, W. Ren, and G. Chen, "An overview of recent progress in the study of distributed multi-agent coordination," IEEE Transactions on Industrial Informatics, 2012.
[11] B. R. Kiran, I. Sobh, V. Talpaert, P. Mannion, A. A. Al Sallab, S. Yogamani, and P. Pérez, "Deep reinforcement learning for autonomous driving: A survey." IEEE Transactions on Intelligent Trunsportation Systems (T-ITS), 2021.
[12] T. Wu, P. Zhou, K. Liu, Yuan, X. Wang, H. Huang, and D. O. Wu, "Multi-agent deep reinforcement learning for urban traffic light control in vehicular networks," IEEE Transactions on Vehicular Technology, 2020.
[13] P. Zhou, X. Chen, Z. Liu, T. Braud, P. Hui, and J. Kangasharju. "Drle: Decentralized reinforce- ment learning the edge for traffic light control in the iov," JEEE Transactions Intelligent Transportation Systems (T-ITS), 2020.
[14] G. Tesauro and J. O. Kephart, "Pricing in agent economies using multi-agent q-learning," Autonomous Agents and Multi-Agent Systems (AAMAS), 2002.
[15] C. Zhang and V. Lesser, "Coordinated multi-agent reinforcement learning in networked dis- tributed pomdps," in AAAI Conference on Artificial Intelligence (AAAI), 2011.
[16] E. Seraj. "Embodied team intelligence multi-robot systems.," " in Autonomous Agents and Multi-Agent Systems (AAMAS), 2022.
[17] D. Koller and R. Parr, "Computing factored value functions for policies in structured mdps," in International Joint Conferences Artificial Intelligence (IJCAI), 1999.
[18] X. Hu, J. Zhao, Y. Zhao, W. Zhou, and H. Li, "Q-sat: Value factorization with self-attention for deep multi-agent reinforcement learning." in International Joint Conference Neural Networks (IJCNN,, 2023.
[19] M. Yang, J. Zhao, X. Hu, W. Zhou, J. Zhu, and H. Li, "Ldsa: Learning dynamic subtask assignment cooperative multi-agent reinforcement learning." " Advances in Neural Information Processing Systems (NeurIPS), 2022.
[20] J. Zhao, M. Yang. Y. Zhao, X. Hu, W. Zhou, and H. Li. "Memarl: Parameterizing value function via mixture of categorical distributions for multi-agent reinforcement learning." IEEE Transactions Games (CoG), 2023.
[21] S. Gronauer and K. Diepold, "Multi-agent deep reinforcement learning: a survey," Artificial Intelligence Review, 2022.

---

[22] A. Oroojlooy and D. Hajinezhad, "A review of cooperative multi-agen deep reinforcement learning," " Applied Intelligence, 2022.
[23] J. Zhao, Y. Zhang, X. Hu, W. Wang, W. Zhou, J. Hao, J. Zhu, and H. Li, "Revisiting qmix: Discriminative credit assignment by gradient entropy regularization"" arXiv preprini arXiv:2202.04427, 2022.
[24] F. A. Olichock, M. T. Spaan, and N. Vlassis, "Optimal and approximate q galue functions for decentralized pomdps." Journal of Artificial Intelligence Research, 2008
[25] L. Kraemer and B. Banerjee. "Multi-agent reinforcement learning as a rehearsal for decentralized planning," Neurocomputtigg. 2016.
[26] J. Zhao, X. Hu, M. Yang, W. Zhou, J. Zhu, and H. Li, "Ctds: Centralized teacher with decentralized student for multi-agent reinforcement learning," IEEE Transactions on Games (CoG), 2022.
[27] M. Hausknecht and Stone, "Deep recurrent q-learning for partially observable mdps," arXiv preprint arXiv:1507.06527, 2015.
[28] T. Schaul, Quan, Antonoglou, and Silver. "Prioritized experience replay," in International Conference on Learning Representations (ICLR), 2016.
[29] M. Samvelyan, T. Rashid, C. S. de Witt, G. Farquhar, N. Nardelli, T. G. J. Rudner, C.-M. Hung. H. S. Torr, J. Foerster, and S. Whiteson, "The StarCraft Multi-Agent Challenge," CoRR, 2019.
[30] K. Kurach, A. Raichuk, P. Stanczyk, M. Zajac, O. Bachem, L. Espeholt, C. Riquelme, D. Vincent, M. Michalski, O. Bousquet, et al., "Google research football: A novel reinforcement learning environment," in AAAI Conference Artificial Intelligence (AAAT), 2020.
[31] B. Peng, T. Rashid, C. Schroeder de Witt, P.-A. Kamienny, P. Torr, W. Böhmer, and Whiteson, "Facmac: Factored multi-agent centralised policy gradients," Advances in Neural Information Processing Systems (NeurIPS), 2021.
[32] F. A. Olichock and C. Amato, A concise introduction to decentralized POMDPs. Springer, 2016.
[33] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, al., "Human-level control through deep reinforcement learning," Nature, 2015.
[34] L.-J. Lin, "Self-improving reactive agents based reinforcement learning, planning and teaching," Machine Learning, 1992.
[35] H. Van Hasselt, A. Guez, and D. Silver, "Deep reinforcement learning with double q-learning." in AAAI Conference on Artificial Intelligence (AAAT), 2016.
[36] S. L. Smith, P.-J. Kindermans, C. Ying, and Q. V. Le, "Don't decay the learning rate, increase the batch size," " in International Conference on Learning Representations (ICLR), 2018.
[37] J. Jeon, W. Kim, W. Jung, and Y. Sung. "Maser: Multi-agent reinforcement learning with subgoals generated from experience replay buffer," in International Conference on Machine Learning (ICML), PMLR, 2022.
[38] S. Fujimoto, D. Meger, and D. Precup, "An equivalence between loss functions and non-uniform sampling in experience replay," Advances in Neural Information Processing Systems (NeurIPS), 2020.
[39] P. Weber, D. Wälchli, M. Zeqiri, and P. Koumoutsakos, "Remember and forget experience replay for multi-agent reinforcement learning," arXiv preprint 22.3::22033333/999 2022.
[40] S. Fan, Song, B. Yang, and X. Jiang, "Prioritized experience replay in multi-actor-atee -cttttttttot critic for reinforcement learning," in Journal of Physics: Conference Series, 2020.
12

---

[41] P. Sunchag, G. Lever, A. Gruslys, W. M. Czarnecki, V. Zambaldi, M. Jaderberg, M. Lanctot, N. Sonnerat, Z. Leibo, K. Tuyls, et al., Valuueddcomposition networks for cooperative multi-agent learning," arXiv preprini arxiv: /1d.:::770009996 2017.
[42] C. de Witt, T. Gupta, D. Makoviichuk, V. Makoviychuk, P. H. Torr, M. Sun, and S. Whiteson, "Is independent learning all you need in the starcraft multi-agent challenge?," arXiv preprint arXiv:2011.09533 2020.
[43] C. Yu, A. Velu, E. Vinitsky, J. Gao, Y. Wang. A. Bayen, and Y. Wu, "The surprising effectiveness of ppo in cooperative multi-agent games," Advances in Neural Information Processing Systems (NeurIPS), 2022.
13

---

A The Proof of Proposition 1
Proposition 1. The invariance of gradient satisfied when individual reward of agent i is given by:
ri 7Ror Qtor) 8Qret
7Q, + Q.-
(8)
aQ.
for any agent i I.
Proof. According to the chain rule,
@bent
iQrot iQ;
8Q iQ;
2( + 7000 - -Qoot)
Σ iEl
(9)
08p
aQuo
aQ; 06p
aQ; 00p
iEI
oLi
0L aQ;
aQ:
Σ iET
Σ
Σ 2(r; + +7 Qi)
(10)
aQ, 00p
00,
Therefore,
acro
0Li
Σ
00,
08,
iE!
eQrot 0Q²
aQ;
b 2(R+ + 7700t - -roc))
Σ
Σ
2(ri + +7 -
iQ;
0
30,
NEl
iEl
aQ
oQi
aQi
3 2 Σ NE!
(R + 1ror -coot)
2
(ri + -
(1I)
aQ,
00p
30,
IE!
UQ
+ (R + 70oot - -Quot)
ri + 7Q; - Vi.iiEl
iQ,
= r; = (R + 7r Quoi)
JQuo
10 + Q..
aQ²
In conclusion, sufficient condition for invariance of gradients is as follow: individual reward of agent is calculated as Equ. ( for any E
B The Hyper-parameters Setting
In DIFFER, the replay buffer stores the most recent 5000 episodes, and mini-batches of size 32 are sampled from it. The target network updated every 200 episodes. RMSProp is utilized as the optimizer for both the mixing network and agent network, with a learning rate of 0.0005, a (decay rate) set to 0.99, and e (small constant) set to 0.00001. Gradients are clipped within the range of [-10, 10]. The discount factor for the expected reward (return) is 0.99.
Regarding the fair experience replay, we apply the "warm-up" technique to the sample ratio 17. linearly increases from 0.8 to 1.0 over 60% of the time steps and remains constant thereafter. The regulation parameter a, which determines the prioritization degree in fair experience replay, set to 0.8. The regulation parameter /3 of the sampling probability anneals linearly from 0.6 to 1.0.
For exploration, we employ e-greedy strategy with linearly annealed from 1.0 to 0.05 over 50K time steps and then held constant for the remainder of the training. The maximum time step during training set to 2005000 for all experiments. Each scenario is run with 5 different random seeds.
The agent network architecture follows a [RQ structure, consisting of a GRU layer for the recurrent layer with a 64-dimensional hidden layer. Before and after the GRU layer, there are fully-comnected layers with 64 dimensions each. The mixing network implemented using open- source code. All experiments settings provided by the benchmark. For the baseline algorithms, we use the authors' code with hyper-parameters fine-tuned specifically for the SMAC benchmark.
the SMAC benchmark adopt the default reward and observation
The supplementary materials contain the code for our DIFFER framework. The main codes of individual reward calculation and fair experience replay are shown in
14

---

DIFFER code/leamers//g_eeaaa Jearner_divideepp and DIFFER CYIFEE__ooe/E//PERXPpiiniii memory.py. re- spectively.
The specific hyper-parameters are listed below.
Table I: The hyper-parameters that used in our DIFFER framework.
Hyper parameters
Meaning
Value
epsilon_start
Start value of anneal
1.0
epsilon_enn end
End value of anneal
0.05
epsilon_ aesilon_anneal_eg aalaaannna_inn time
Duration step of anneal
50000
mempool size
Memory pool size in learner
5000
hatch_size
Batch size per training
Karget_udaate_ineee interval
Interval steps between updates of target network
200
Ir
Learning rate
0.0005
regularization
Weights of I -regulllrree
0.0005
jot_optimizer
Optimizer in training
RMSProp
tot_optim_alpha
Alpha of optimizer
0.99
lot_optim__ey eps
Epsilon of optimizer
0.00001
ind_optimizer
Optimizer training
RMSProp
ind_ mad_optim_alpee
Alpha of optimizer
0.99
ind optim eps
Epsilon of optimizer
0,00001
grad norm_clip
Clip range of gradient normalization
10
Nstart
Start value of warm up
0.8
Mend
End value of y warm up
1.0
p
Duration step ratio y warm up
Regulation parameter sampling probability
0.8
Regulation parameter of sampling probability
0.01
Bstart
Start value of 3 warm
0.6
Send
End value of / warm up
1.0
C Ablation Studies for warm_up Technique of Sample Ratio
To highlight the significance of the morm_up trick for the individual experience sample ratio 1j, we conduct ablation experiments on SMAC maps MMM2. Fig. [T showcases the performance of QMIX- DIFFER with and without the warm_ "pp sample ratio. this context, the model W-warm up (ours) refers to the setting where the sample ratio linearly increases as the training progresses. On the other hand, W/O-warm _ denotes the scenario where the sample ratio remains fixed throughout the training phase. To ensure a fair comparison, we conduct experiments with different fixed values of while keeping the batch-size of all three models in Fig. [T consistent at 32. Let us assume that the overall quantity of training data is K for W/O-warn up-r7=1.0. By employing the setting detailed in Sec. B. we can calculate that the overall training data amounts to 0.94K for W-warm_ _pp. Consequently,
select 1) from the set (0.8, 0.94, 1.0). The results clearly demonstrate that the warm _pp trick leads to performance improvements, even when using a smaller amount of training data. This highlights the importance of incorporating the warm_uy strategy into the training process.
D Analysis of Sampling Percentage
In this section, we present the distribution of individual experience sampling percentages across differ- ent agent classes within the MMM2 scenario from the SMAC environment [9) The team composition consists of three distinct unit classes: Medivac unit responsible for healing teammates, Marauder units specialized in offensive actions, and 7 Marine units dedicated to attacking. Throughout the training process, DIFFER selects important individual experiences from a diverse set of 10 agents to update the agent networks. We calculate the sampling percentages of individual experiences for
15

---

W-warm up (ours)
0.8
W/O-warm W0--warmuup-7-0.3 Up8
W/O-warm -.0pmmmmpp7---44
Mean Test Win Rate
W/O-warm w/O-warm__p-711100 up-77-1.0
0.2
0.0
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
Timesteps(M)
1e6
Figure 11: Ablation studies regarding warm up trick of individual transition sample ratio on MMM2. "W-warm_up (ours)" represents the sample ratio increases the training step increases until 1.0. "W/O-warm_up" represents the sample ratio rj is fixed during training.
MMM2
0.13
mn may
Average Percentage
0.12
Medivac(1)
Marines Marauders(2)
0.10
mmm
0.09
0.8
1.0
1.2
1.4
1.6
1.8
2.0
Timesteps(M)
Average percentage of sampled individual experiences from each unit class on SMAC scenario MMM2. The number the label represents the quantity of unit. The sampling percentage of unit class is high when it poses learning challenge.
Figure 12:
each agent and derive the average sampling percentage each unit class. The corresponding curves, focusing on the later stages of training for improved stability, are depicted in Fig. T2
The graph showcases significant variations in the average sampling percentages among the unit classes, reflecting their distinct learning difficulties. As the learning difficulty of a particular unit class intensifies. larger proportion of individual experiences from that class are sampled during training. This adaptive behavior of the DIFFER method highlights its ability to dynamically adjust the sampling percentages address the unique learning challenges encountered by each unit class. Notably, achieving effective strategies for the Medivac unit proves particularly challenging due to its distinctive sub-task and limited availability. Consequently, the average sampling percentage of the Medivac unit surpasses that of the other two unit classes significantly. Although the number of Marine units exceeds that of Marauder units, the Marines are encouraged adopt specialized positioning strategy. This strategy requires the Marines to spread out in a mikeike formation, aiming
16

---

to minimize the damage inflicted upon them while maximizing their effectiveness in dealing damage the enemy. Consequently, the Marines face higher learning difficulties compared to the Marauders. Reflecting this distinction, the average sampling percentage of the Marine unit class surpasses that of the Marauder unit class.
E Discussion Related to Policy-Based Methods
E.1 Performance Comparison with Policy-Based Method COMA
In this section, we compare DIFFER with the classic policy-basee method COMA on four SMAC scenarios as shown in Fig. T3 We can see that DIFFER performs better than COMA on all four scenarios, which strongly highlight the strengths and advantages of our proposed DIFFER framework.
CIDMA
QMIX DIFFER (Pmn))
Moon Ton/Wn Rub
Mean Ven
QMIX
Mal Fags
Timandace
(a) 2c vs 642g
(b) MMM2
(c) 355z VS 3s62
(d) 27m vs 30m
Figure 13: Performance comparison QMIX-DIFFER (ours) and a classic policy-based method
COMA on SMAC scenarios, highlighting the performance improvement of our method towards COMA.
E.2 Potential Application to Policy-Based Approaches
Our method, DIFFER, is specifically designed address MARL problems where individual rewards are not available, and teams only have access to a shared team reward. In such scenarios, calculating individual action-values becomes challenge. However, DIFFER provides solution by decomposing individual trajectories, enabling the calculation and update of individual action-values. For policy- based methods that approximate individual action-values using individual critics, such IPPO and MADDPG 43), the traditional approach involves treating the team reward as an individual reward and using update individual critics. In this context, the decomposition of team experiences provided by DIFFER may not be directly applicable or necessary since the individual critics can approximate action-valuce using the team reward. On the other hand, for policy-based methods that do not require the approximation of individual action-values, such as MAPPÓ [4), our DIFFER method might not offer significant enhancements. In such cases, where there no need to decompose team trajectories for individual action-vai values training, the contribution of DIFFER may not be as pronounced.
F Environment Introduction
In this section, we provide a brief overview of the anett environment discussed in this paper.
F1
SMAC
The Starcraft Multi-Agent Challenge (SMAC) 29
(as shown in Fig. 14 (a)) is prominent cooperative multi-agent environment known for its partial observability. In this paper, we utilize the default environment setting of SMAC, specifically version SC2.4.10. SMAC offers a variety of scenarios, each featuring a confrontation between two teams of units. One team is controlled by a trained model, while the other team is governed by built-in strategy. The scenarios differ in terms of the initial unit positions, the number and types of units in each team, as well as the characteristics of the map terrain. In the SMAC environment, team is deemed victorious when all units belonging to the
17

---

12-11
(a) SMAC
(b) GRF
(c) MAMujocu
Figure 14: Visualisation three experimental environments.
opposing team are eliminated. The primary objective of the policy is maximize the win rate across all scenarios. To facilitate training, the environment provides shaped reward function that takes into account factors such hit-point damage inflicted and received by agents, the number of units killed, and the outcome of the battle. Learning coordinated micromanagement strategy in diverse maps poses a significant challenge for the agents this environment. Such strategy aims to maximize the damage inflicted upon enemies while minimizing the damage sustained. Overall, the SMAC environment demands agents to acquire sophisticated coordination and decision-making skills in order to excel in diverse scenarios and effectively apply micromanagement strategies.
F.2
GRF
Google Research Football (GRF) environment 30 (shown Fig. 14 (b)) presents challenging multi-agent reinforcement learning (MARL) setting where team of agents must learn to pass the ball amongst themselves and overcome their opponents' defense score goals. Similar to the SMAC environment, the opposing team in GRF is controlled by built-in strategy. In GRF, each agent has 19 different actions at their disposal, including standard move actions in eight directions, as well as various ball-kicking techniques such as short and long passes, shooting, and high passes that are difficult for opponents to intercept. The agents' observations encompass information about their own position, movement direction, the positions of other agents, and the ball. An episode in GRF terminates either after a fixed number steps or when one of the teams successfully scores goal. The winning team receives reward of +100, while the opposing team is rewarded with -1. In this study, we specifically focus on two official scenarios: academy ___ttn____v__ 3_vs with keeper and academy_coure coattr aaarddakaaaard..... hard.
F.3 MAMujoco
The Multi-Agent Mujoco (MAMujoco) environment(3] (shown Fig. [4)c) a highly versatile and realistic simulation platform designed for multi-agent robotic control tasks. It is based on the Mujoco physics engine and offers continuous action spaces, allowing for smooth and precise control of robotic agents. Each agent in the environment represents specific component enabling the simulation of complex multi-robot systems. All of the tasks mentioned in this work are configured according to their default configuration. We set maximum observation distance to k 0 for Humanoid v2 and k = 1 for canyagent_sviimeeee swimmer.
complete robot,
We also provide the description of agent types in each scenario in Table It aims to enhance the reader's 's understanding of the distinct performance improvements brought about by DIFFER.
18

---

Table 2: The agent type distribution in each scenario.
Env.
Scenario Name
# #Agent
#Type
Type Distribution
27m_vs_30m
27
27
2c_vs_ rs_64zg
2
2
SMAC
MMM2
10
3
1-2-7
vs_
3-5
academy 3sss__ 1_
______________ with keeper
3
2
1-2
GRF
hard manyagent_
2-2
Humanoid-v2
2
2
1-1
MAMujoco
2-2-2-2-2-2-2-222222
19


Reviews:
[{"review_id": "IWX5h4nsC7", "reviewer": "Reviewer_Jgve", "summary": "The paper presents a new algorithm DIFFER to help individual agents to learn individual TD-error such that it improves the overall (team) model performance. The paper's proposed method is different from an existing method MASER in that it is maintaining the overall team objective. Overall model performance comparison was made against QMIX which uses the traditional team TD-error only.", "questions": "See the weakness section above.", "limitations": "The authors adequately listed the potential limitation in the paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper adequately points out the limitation of the proposed method in that additional complexity thus more resources is required. The paper also demonstrates how each complexity (i.e. individual TD-error computation and the selection phase) contributes to the final model performance compared to a prior art. This helps the reader make an informed decision based on the tradeoff between complexity and utility demonstrated. The paper seems overall well-written such that it was fairly easy to follow.", "weaknesses": "I would like to see more comparisons of DIFFER's performance and complexity against MASER's, a closely related work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer:\n\nThanks for your valuable suggestion.\n\n1) We conducted experiments to compare the performance of DIFFER and MASER in SMAC environments. According to your suggestion, we have conducted additional experiments to meticulously evaluate the performance of DIFFER in relation to MASER on the Google Football Research (GRF) task, which can be found in the Figure 1 of Rebuttal PDF. The comprehensive results underscore the strengths and advantages of our proposed DIFFER framework.\n\n2) Here we explain the factors leading to the poor performance of MASER. \n\n    a) It is important to acknowledge that MASER introduces a divergence in the optimization objective by seeking to maximize a combination of individual action-value and team action-value, as explained in lines 264-270. This divergence may contribute to its limitations. \n\n    b) It is crucial to note that MASER predominantly focuses on addressing sparse reward multi-agent reinforcement learning (MARL) scenarios, whereas our experiments are exclusively conducted within dense reward environments.\n\nPlease let us know if you have any further concerns, and we are encouraged to have a discussion."}, {"review_id": "jXc4upswlz", "reviewer": "Reviewer_f7XM", "summary": "This paper introduces a method for decomposing shared rewards into individual rewards within value-based multi-agent reinforcement learning (MARL) methods. By decomposing the rewards, the method enables the prioritization of crucial individual experiences. Experimental results demonstrate the effectiveness of the proposed approach across a range of value-based MARL methods.", "questions": "1. Is there any intuitive visualization or illustration available that can help understand the decomposed individual rewards more clearly?\n\n2. How does this method relate to policy-based MARL methods? It would be beneficial to discuss the potential application of individual reward decomposition to policy-based approaches.", "limitations": "The paper primarily focuses on cooperative MARL tasks. It would be intriguing to explore the applicability of reward decomposition in mixed cooperative-competitive settings.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper provides theoretical evidence for the optimization equivalence between value-based MARL methods and those employing decomposed rewards.\n\n2. The concept of decomposing shared rewards into individual rewards is innovative and appears to yield positive results, showcasing the potential for significant impact.\n\n3. The authors have made their code available, ensuring reproducibility of the experiments.", "weaknesses": "1. The paper lacks a discussion of policy-based MARL methods, which is a significant gap in the related work. It would be valuable to include a comprehensive overview and comparison with policy-based approaches.\n\n2. The absence of a comparison with policy-based methods diminishes the completeness of the evaluation. Including such a comparison would further strengthen the paper's contribution. Some related papers are [1][2].", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer:\n\nThank you for the constructive comments.\n\n**Q1: Intuitive visualization of the decomposed individual rewards.**\n\nA1: We have included visualizations displaying the individual rewards for each agent, along with corresponding screenshots, which can be found in the Figure 3 of Rebuttal PDF. The presented visualizations demonstrate that the individual rewards yielded by DIFFER align with the current game situation, further supporting the effectiveness of our approach. To provide greater clarity on the concept and implications of decomposed individual rewards, we will incorporate it in the revised version of our paper.\n\n**Q2: How does this method relate to policy-based MARL methods?**\n\nA2: \n1) *Performance Comparison with Policy-Based Methods:* \n\n     We notice that you may dismiss the related paper details in Weaknesses 2, so that we don’t know which exactly the related papers you mentioned. We have conducted a comprehensive experiment comparing DIFFER with the classic policy-based method COMA[1]. The detailed results of this comparison can be found in the “global” response PDF, which strongly highlight the strengths and advantages of our proposed DIFFER framework.\n\n2) *Potential Application to Policy-Based Approaches:*\n\n     Our method, DIFFER, is specifically designed to address MARL problems where individual rewards are not available, and teams only have access to a shared team reward. In such scenarios, calculating individual action-values becomes a challenge. However, DIFFER provides a solution by decomposing individual trajectories, enabling the calculation and update of individual action-values.\n\n      a) For policy-based methods that approximate individual action-values using individual critics, such as IPPO[2] and MADDPG[3], the traditional approach involves treating the team reward as an individual reward and using it to update individual critics. In this context, the decomposition of team experiences provided by DIFFER may not be directly applicable or necessary since the individual critics can approximate action-values using the team reward.\n\n      b) On the other hand, for policy-based methods that do not require the approximation of individual action-values, such as MAPPO[4], our DIFFER method might not offer significant enhancements. In such cases, where there is no need to decompose team trajectories for individual action-values training, the contribution of DIFFER may not be as pronounced.\n\nIn our forthcoming research, we will embark on an exploration of the application of our innovative method, DIFFER, within the domain of policy-based methods.\n\nPlease share any remaining concerns.\n\n[1] Foerster, Jakob, et al. \"Counterfactual multi-agent policy gradients.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 32. No. 1. 2018.\n\n[2] de Witt, Christian Schroeder, et al. \"Is independent learning all you need in the starcraft multi-agent challenge?.\" arXiv preprint arXiv:2011.09533 (2020).\n\n[3] Yu, Chao, et al. \"The surprising effectiveness of ppo in cooperative multi-agent games.\" Advances in Neural Information Processing Systems 35 (2022): 24611-24624.\n\n[4] Lowe, Ryan, et al. \"Multi-agent actor-critic for mixed cooperative-competitive environments.\" Advances in neural information processing systems 30 (2017)."}, {"review_id": "iQQAPZG5ee", "reviewer": "Reviewer_U3kJ", "summary": " This is a very interesting and strong paper on MARL that seeks the strategy of learning individual policies rather than team policies in order to break symmetry and learn identity and contextual influences on the individual policies. In order to do so, it solves the technical problem of  decomposing a team reward into individual reward, and provides a proof that their method works. They then employ a replay buffer that selects individual experiences to train the agent network akin to prioritized replay. The authors demosntrate their method on a variety of benchmarks, showing improvement. ", "questions": "N/A", "limitations": "The authors have a limitations section that is fair and does not weaken the strengths of the paper. ", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The method is novel; the theory and experiments solid. The writing was very clear and flowed very naturally. \n\nThe authors demonstrate their method not just multi agent Mujoco, but even the difficult StarCraft II micromanagement tasks. ", "weaknesses": "I do not have any concerns. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer:\n\nThanks for your valuable comments. We sincerely appreciate your positive evaluation and recognition of our research. \n\nTo enhance our work’s robustness and comprehensiveness, we plan to incorporate additional experiment results, intuitive visualization, and theoretical analysis in the revised edition of our paper. These enhancements aim to strengthen the validity, clarity, and impact of our findings. Your valuable feedback has been instrumental in shaping the development of our research, and we are grateful for your contribution."}, {"review_id": "jstxenpfF6", "reviewer": "Reviewer_ddQi", "summary": "This paper proposes to train a mixing network and agent networks separately for multiagent reinforcement learning when the system is based on the centralized training and decentralized execution (CTDE) paradigm. The authors derive an individual reward that maintains invariance between the gradients of the mixing network's loss function and the sum of the loss functions of the agent networks. Then, the authors apply Prioritized Experience Replay to select samples for training the agent networks. The proposed method, DIFFER, is compared with QMIX, COMIX, QPLEX, and MASER on several tasks, such as SMAC, GRF, and MAMujoco. The experimental results show that DIFFER outperforms the baseline methods and demonstrate that a fair experience replay mechanism works efficiently. ", "questions": "Major comments:\n- The individual reward (2) is interesting, but I am unsure how the TD error is calculated. I think that the TD error is given by \n  $$\n  (R + \\gamma \\tilde{Q}_{\\mathrm{tot}} - Q_{\\mathrm{tot}}) \\frac{\\partial Q_{\\mathrm{tot}}}{\\partial Q_i}\n  $$ \n  because the term $\\gamma \\tilde{Q}_i – Q_i$ cancels out. Is the target network $\\tilde{Q}_i$ necessary? Is my understanding correct? \n- I would like to know whether the used learning algorithm is off-policy or not. Experience replay buffer technique is typically applied to off-policy reinforcement learning algorithms, but the loss functions (6) and (7) suggest that the action value function is trained in an on-policy manner, like SARSA. Is on-policyness necessary for the proposed method?\n- Is the input of the mixing network consistent with the outputs of the agent networks when the mixing and agent networks are trained separately? \n- Does QMIX-divide adopt uniform sampling instead PER for training the agent networks? If so, the performance difference between QMIX-DIFFER and QMIX-divide can be discussed based on Fujimoto et al. (2020). \nS. Fujimoto et al. (2020). An Equivalence between Loss Functions and Non-Uniform Sampling in Experience Replay. NeurIPS 33. \n\nMinor comments:\n- Line 87: $A^n$ -> $A^N$. \n- Line 96: $r$ -> $R$.\n- Line 154: Since $S$ denotes the state space, it would be better to use a different symbol.\n- Eq.(7) and Line 191: $\\chi^{\\mathrm{indi}}_j$ -> $\\chi^{\\mathrm{ind}}_j$.\n \n", "limitations": "Minor comments:\nThe authors do not discuss this work's potential negative social impacts. \n", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Originality: The idea of fair experience replay based on the constraint between the gradients of the mixing and the agent networks is novel. However, I found the following paper that may be related to this study: D. Hostallero et al. (2020). Inducing Cooperation through Reward Reshaping based on Peer Evaluations in Deep Multi-Agent Reinforcement Learning. In Proc. of AAMAS.\n- Quality: The experimental results support the claims and the proposed method.\n- Clarity: The manuscript is written very well and easy to follow.\n- Significance: While the CTDE approach becomes popular in MARL, many previous studies do not pay attention to the difference between individual experiences. This study sheds light on its importance. \n", "weaknesses": "- Although the experimental results show that the proposed method works well, there is no theoretical analysis. In particular, the gradient of the loss function of the agent network is independent of the target networks as described later. It would be better to discuss the loss function in detail.\n- In addition, the off-policyness of the proposed method should be discussed. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer:\n\nThanks for your valuable comments.\n\n**Q1: Is $\\widetilde{Q_i}$ necessary?**\n\nA1: Your understanding is indeed correct.\n\n1) In the computation of the TD error for the agent networks, there is no need to explicitly determine the value of $\\widetilde{Q_i}$ because it cancels out within the formula. \nThe TD error can be expressed as $(R + \\gamma \\widetilde{Q} _ {\\mathrm{tot}}-Q_{\\mathrm{tot}})\\frac{\\partial Q_{\\mathrm{tot}}}{\\partial Q_i}$, as you have correctly stated.\n\n2) However, it is worth noting that the calculation of $\\widetilde{Q_i}$ remains necessary for the determination of $\\widetilde{Q_{\\mathrm{tot}}}$. The target mixing network receives the $\\widetilde{Q_i}$ values from each individual agent and produces the $\\widetilde{Q_{\\mathrm{tot}}}$ output.\n\n3) While it may not be necessary to explicitly calculate individual rewards using Equation (2) in the code, incorporating this calculation can significantly enhance our understanding and improve interpretability.\n\n**Q2: Is the used learning algorithm off-policy?**\n\nA2: Yes.\n\n1) We would like to clarify that the action value functions in our work are trained in an off-policy manner similar to Q-learning , contrary to an on-policy approach like SARSA. \nThis is evident in Line 100, where we define $\\widetilde{Q}(\\boldsymbol{o’};\\theta^-)$ as the optimal target action value, represented as $\\max_{\\boldsymbol{a’}}Q(\\boldsymbol{o’},\\boldsymbol{a’};\\theta^-)$.\n\n2) As a result, in Equation 6, $\\widetilde{Q_{\\mathrm{tot}}}(\\boldsymbol{o’},\\theta_m^-)=\\max_{\\boldsymbol{a’}} Q_{\\mathrm{tot}}(\\boldsymbol{o’},\\boldsymbol{a’};\\theta_m^-)$ represents the optimal target team action value function, demonstrating the off-policy nature of our learning algorithm.\n\n3) Regarding Equation 7, we apologize for the typo. \nThe part of $\\widetilde{Q_{j}}(o'_j, a_j;\\theta_p^-)$ is incorrect,\n\nwhich should be $\\widetilde{Q_{j}}(o'_j;\\theta_p^-)$.\n\nThe correct expression of Equation 7 should be \n\n$$L^{\\mathrm{ind}} _ {\\mathrm{TD}}(\\theta_p)=\\sum\\nolimits_{\\chi^{\\mathrm{indi}} _ j=(o_j,a_j,o_j',r_j)\\in S'}\\omega_j(r_j +\\gamma \\cdot \\widetilde{Q_{j}}(o'_j;\\theta_p^-)-Q_j(o_j,a_j;\\theta_p))^2$$ \n\nWe regret any confusion stemming from our oversight and appreciate the opportunity to rectify it in the revised version of our paper.\n\n**Q3: Is the input of the mixing network consistent with the outputs of the agent networks?**\n\nA3: Yes. In Algorithm 1 on page 5, specifically in Line 13-19, we provide a detailed description of a network update procedure. \n\n1) After retrieving data from the replay buffer, the mini-batch data is fed into the agent network, resulting in the computation of individual action-values ($Q_i$). These individual action-values are subsequently forwarded to the mixing network, which combines them to obtain the team action-value ($Q_{\\mathrm{tot}}$). \n\n2) We calculate both the individual TD-loss and the team TD-loss independently. Notably, the gradient propagation and network updating steps are carried out independently for both the agent networks and the mixing network.\n\n3) In the revised version of our paper, we intend to emphasize the consistent flow of information between the agent and mixing networks throughout the training process. \n\n**Q4: The reason of the performance difference between QMIX-DIFFER and QMIX-divide.** \n\nA4: \n1) QMIX-divide adopts all samples in the mini-batch instead of PER for training the agent networks.\n\n2) We have thoroughly studied the paper you referred to and have identified that it indeed provides an explanation for our observed performance improvement to a certain degree. Consequently, we have decided to cite this paper in the revised version of our work and carefully discuss the implications it holds for our research. \n\n**Q5: Minor comments.**\n\nA5: We apologize for the presence of these typos and sincerely appreciate your valuable insights. We will promptly rectify the errors to ensure clarity and precision in our paper. \n\nPlease let us know if you have any further concerns, and we are encouraged to have a discussion."}]

Author rebuttal:
We extend our sincere gratitude to all the esteemed reviewers for their invaluable and constructive comments. 
The consistently positive reviews our submission received from all the Reviewers have filled us with delight.
We are greatly encouraged by their recognition of the originality (Review f7XM, U3kJ, ddQi), significant impact (Review U3kJ, ddQi), theoretical foundation (Review f7XM, U3kJ) , reproducibility (Review f7XM), and clarity (Review Jgve, U3kJ, ddQi) showcased in our work. Their meticulous evaluation of our research has further bolstered our confidence in the merits and contributions of our study.

We also appreciate reviewers pointing out our weaknesses. We address their comments point by point and try our best to respond to them. Hope our response addresses the reviewers' concerns.

The additional experiments in the Rebuttal PDF are summarised as follows:

In Rebuttal Figure 1, we show a comparison between  our model DIFFER and MASER[1] on Google Football Research (GRF) tasks, highlighting the performance improvement of our method towards MASER..

In Rebuttal Figure 2, we show a comparison between our model DIFFER and a classic policy-based method COMA[2], highlighting the performance improvement of our method towards COMA..

In Rebuttal Figure 3, we present the visualization of individual rewards produced by our DIFFER method. The consistent correlation between individual rewards and the ongoing game situation serves as a testament to the efficacy and rationality of our DIFFER method.

[1] Jeon, Jeewon, et al. "Maser: Multi-agent reinforcement learning with subgoals generated from experience replay buffer." International Conference on Machine Learning. PMLR, 2022.

[2] Foerster, Jakob, et al. "Counterfactual multi-agent policy gradients." Proceedings of the AAAI conference on artificial intelligence. Vol. 32. No. 1. 2018.




Meta-review: This paper has been positively evaluated by all reviewers. The rebuttal helped to clarify concerns about the theoretical guarantees of the method, the lack of discussion of related works, and the effectiveness of the proposed method. The scores did not change substantially after the rebuttal phase, and the general positive consensus remained.

I encourage the authors to address all the comments and to incorporate the recommended improvements in the final version.
Decision: Accept (poster)