SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xunhan Hu $^{1}$ , Jian Zhao $^{1}$ , Wengang Zhou $^{1,2\dagger}$ , Ruili Feng $^{3,1}$ , Houqiang Li $^{1,2\dagger}$
<sup>1</sup> University of Science and Technology of China, <sup>2</sup> Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, <sup>3</sup> Alibaba Group {cathyhxh,zj140}@mail.ustc.edu.cn, zhwg@ustc.edu.cn, ruilifengustc@gmail.com, lihq@ustc.edu.cn
Cooperative multi-agent reinforcement learning (MARL) is a challenging task, as agents must learn complex and diverse individual strategies from a shared team reward. However, existing methods struggle to distinguish and exploit important individual experiences, as they lack an effective way to decompose the team reward into individual rewards. To address this challenge, we propose DIFFER, a powerful theoretical framework for decomposing individual rewards to enable fair experience replay in MARL. By enforcing the invariance of network gradients, we establish a partial differential equation whose solution yields the underlying individual reward function. The individual TD-error can then be computed from the solved closed-form individual rewards, indicating the importance of each piece of experience in the learning task and guiding the training process. Our method elegantly achieves an equivalence to the original learning framework when individual experiences are homogeneous, while also adapting to achieve more muscular efficiency and fairness when diversity is observed. Our extensive experiments on popular benchmarks validate the effectiveness of our theory and method, demonstrating significant improvements in learning efficiency and fairness. The code is available in https://github.com/cathyhxh/DIFFER.
In widely adopted cooperative multi-agent systems [1-4], a team of agents typically operates under the constraint of individual local observations and limited communication capabilities. In each time step, all agents collectively interact with the environment by taking team actions, leading to a transition to the subsequent global state and the provision of a team reward. Consequently, the agents must acquire the ability to effectively coordinate their actions and maximize the cumulative team return. Over the past few years, cooperative multi-agent reinforcement learning (MARL) has demonstrated remarkable achievements in addressing such challenges and has exhibited promising potential across diverse domains, including multi-player games[5-9], autonomous cars [10, 11], traffic light control [12, 13], economy pricing [14] and robot networks [15, 16].
In recent years, value factorization methods [17-20] have emerged as leading approaches for addressing cooperative multi-agent tasks [21-23]. These methods are rooted in the centralized training and decentralized execution (CTDE) paradigm [24-26]. Specifically, each agent's network models the
individual action-value using the agent's local action-observation history [27] and the last action. These individual action-values are then integrated into a global state-action value through a mixing network. The parameters of both the agent networks and the mixing network are optimized by minimizing the team TD-error. To enhance data utilization, value factorization methods commonly employ experience replay. Here, experience represents the atomic unit of interaction in RL, i.e., a tuple of (observation, action, next observation, reward) [28]. In traditional MARL experience replay algorithms, the replay buffer stores team experiences and allows algorithms to reuse them for updating the current network, which encompasses both the agent networks and the mixing network.
However, the utilization of team experience overlooks the inherent differences among individual experiences, thereby failing to leverage individual experiences worthy of learning. This issue becomes especially pronounced when the agents within the team exhibit heterogeneity in their roles and capabilities. To develop a clearer understanding, let us consider the scenario of MM2 (as shown in Fig. 1) within The StarCraft Multi-Agent Challenge (SMAC) environment [29]. In this setup, a team comprises 1 Medivac (responsible for healing teammates), 2 Marauders (focused on offensive actions), and 7 Marines (also dedicated to attacking). Each unit is controlled by a distinct agent, and the team's objective is to eliminate all enemy units. Learning an effective strategy for the Medivac unit proves particularly challenging due to its unique sub-task and limited number. Consequently, the individual
Figure 1: Screenshots of the MM2 scenario in SMAC, highlighting the importance of individual experiences for the Medivac unit, given its unique sub-task and limited number.
experiences of the Medivac unit are deemed more likely to provide valuable insights for learning. Unfortunately, extracting these individual experiences from the team experiences proves difficult in the aforementioned MARL setting, as the individual rewards associated with each individual experience are unavailable.
In this paper, we introduce a novel method called Decomposing Individual Reward for Fair Experience Replay (DIFFER), to address the above challenges associated with multi-agent reinforcement learning (MARL) experience replay. The fairness in MARL experience replay refers to the equitable sampling of individual experiences based on their importance. DIFFER addresses this by decomposing team experiences into individual experiences, facilitating a fair experience replay mechanism. We begin by proposing a method to calculate the individual reward for each individual experience, ensuring that DIFFER maintains equivalence with the original learning framework when the individual experiences are indistinguishable. To achieve this, we establish a partial differential equation based on the invariance of agent network gradients. By solving this differential equation, we obtain an approximation function for the individual rewards. Consequently, the team experience can be successfully decomposed into individual experiences. These individual experiences are then fed into a fair experience replay, where they are sampled for training the agent networks. The sampling probability is determined proportionally to the temporal difference (TD) error of each individual experience. Moreover, it adapts effectively to enhance both efficiency and fairness in scenarios where diversity is observed. Importantly, DIFFER is a generic method that can be readily applied to various existing value factorization techniques, thereby extending its applicability in MARL research.
We evaluate DIFFER on a diverse set of challenging multi-agent cooperative tasks, encompassing StarCraft II micromanagement tasks (SMAC) [29], Google Football Research (GRF) [30], and Multi-Agent Mujoco [31] tasks. Our experimental findings demonstrate that DIFFER substantially enhances learning efficiency and improves performance.
Problem Formulation. A fully cooperative multi-agent problem can be described as a decentralized partially observable Markov decision process (Dec-POMDP) [32], which consists of a tuple  $G = < I, S, A, P, r, \Omega, O, N, \gamma >$ .  $I$  is the finite set of  $N$  agents. For each time step, each agent  $i \in I := \{1, \dots, N\}$  receives individual observation  $o_i$  according to the observation function  $\bar{O}(s, i) \equiv S \times I \to \Omega$ . Then, each agent determines its action with individual policy  $\pi_i(a_i | o_i) : \Omega \times A \to [0, 1]$ . Given the team action  $\mathbf{a} = \{a_1, \dots, a_N\}$ , the environment transits to a next state according to the
transition function  $P(s'|s, \pmb{a}): S \times A^N \times S \to [0,1]$  and gives a team reward  $R(s, \pmb{a}) \equiv S \times A^N \to \mathbb{R}$ , which is shared by all agents. Each team policy  $\pi(\pmb{a}|o) = [\pi_1(a_1|o_1), \dots, \pi_N(a_N|o_N)]$  has a team action-value function  $Q_{\mathrm{tot}}^t(\pmb{o}^t, \pmb{a}^t, s^t) = \mathbb{E}_{s_{t+1:\infty}, \pmb{a}_{t+1:\infty}}[\mathrm{Return}^t(s^t, \pmb{a}^t)]$ , where  $\mathrm{Return}^t(s^t, \pmb{a}^t) = \sum_{k=0}^\infty \gamma^k R^{t+k}$  is a discount return and  $\gamma$  is the discount factor. The target of a Dec-POMDP problem is to maximize the accumulated return of the team policy, i.e., optimal team policy  $\pi^* = \arg \max_{\pmb{\pi}} \mathrm{Return}(\pmb{\pi}) = \mathbb{E}_{s_0 \sim d(s_0), \pmb{a} \sim \pmb{\pi}}[Q_{\mathrm{tot}}^t(\pmb{o}^t, \pmb{a}^t, s^t)]$ , where  $d(s_0)$  represents the distribution of initial state.
Multi-Agent Deep Q-learning. Deep Q-learning [33] uses a neural network parameterized by  $\theta$  to represent the action-value function  $Q$ . In the multi-agent Dec-POMDP problem, multi-agent deep Q-learning methods usually adopt the replay buffer [34] to store experience  $(o, a, o', R)$ . Here  $R$  is the team reward received after taking team action  $a$  with team observation  $o$ , and  $o'$  represents team observation in the next step. During the training process, the parameter  $\theta$  is updated by minimizing the Temporal Difference (TD) error with a mini-batch of data sampled from the replay buffer, which is shown as  $L_{\mathrm{TD}}(\theta) = \mathbb{E}_{(o, a, r, o') \in D}[(r + \gamma \widetilde{Q}(o'; \theta^-) - Q(o, a; \theta))^2]$ , where  $\widetilde{Q}(o'; \theta^-) = \max_{a'} Q(o', a'; \theta^-)$  is the one-step expected future return of the TD target and  $\theta^-$  is the parameter of the target network [35].  $\delta = |r + \gamma \widetilde{Q}(o'; \theta^-) - Q(o, a; \theta)|$  is known as TD-error.
Multi-Agent Value Factorization. The value factorization framework comprises  $N$  agent networks (one for each agent) and a mixing network. At time step  $t$ , the agent network of agent  $i$  takes the individual observation  $o_i^t$ , thereby computing the individual action-value, denoted as  $Q_i$ . Subsequently, the individual action-values  $\{Q_i\}_{i \in I}$  are fed into the mixing network, which calculates the team action-value, denoted as  $Q_{\mathrm{tot}}$ . The mixing network is constrained by Individual-Global-Max (IGM) principal[1], ensuring the congruity between the optimal team action deduced from  $Q_{\mathrm{tot}}$  and the optimal individual actions derived from  $\{Q_i\}_{i \in I}$ . The  $N$  agent networks share parameters. We denote  $\theta_m$  and  $\theta_p$  as the parameters of mixing network and agent networks, respectively. During the training phase, the mixing network is updated based on the TD loss of  $Q_{\mathrm{tot}}$ , while the agent networks are updated through gradient backward propagation. During the execution phase, each agent takes action with individual policy derived from its agent network in a decentralized manner.
In this section, we present DIFFER, a novel experience replay method designed to foster fairness in multi-agent reinforcement learning (MARL) by decomposing individual rewards. DIFFER offers a solution to enhance the adaptivity of MARL algorithms. Instead of relying solely on team experiences, DIFFER trains the agent networks using individual experiences. This approach allows for a more granular and personalized learning process.
At a given time step, agents interact with the environment and acquire a team experience  $\chi^{\mathrm{team}} = (\pmb{o},\pmb{a},\pmb{o}^{\prime}) = ((o_{i})_{i\in I},(a_{i})_{i\in I},(o_{i}^{\prime})_{i\in I},R) = ((o_{i},a_{i},o_{i}^{\prime})_{i\in I},R)$ . Here,  $o_i$ ,  $a_i$ , and  $o_i^\prime$  represent the observation, action, and next observation for each agent  $i\in I$ , respectively. These team experiences are stored in a replay buffer and subsequently sampled to train the mixing network and agent network parameters. We aim to decompose a multi-agent team experience into  $N$  single-agent individual experiences  $\{\chi_i^{\mathrm{ind}}\}_{i\in I} = \{(o_i,a_i,o_i',r_i)\}_{i\in I}$ , then utilize these individual experiences to update the agent networks.
The DIFFER framework comprises two stages: (1) Decomposer: Calculating individual rewards to decompose team experiences; (2) Fair Experience Replay: Selecting significant individual experiences to train the agent networks.
3.1 Exploring Individual Rewards through Invariance of Gradients
The DIFFER framework necessitates individual rewards  $\{r_i\}_{i\in I}$  to decompose the team experience  $\chi^{\mathrm{team}}$  into individual experiences  $\{\chi_i^{\mathrm{ind}}\}_{i\in I}$ . However, in the context of shared team reward POMDP problems, these individual rewards are not readily available. Therefore, the key challenge for DIFFER is to devise a strategy for calculating individual rewards.
In the traditional experience replay methods, agent networks are updated by the team TD-loss, which is calculated based on team experience and denoted as  $L_{\mathrm{tot}} = (R + \gamma \widetilde{Q}_{\mathrm{tot}}(\pmb{o}', \pmb{a}'; \theta^-) - Q_{\mathrm{tot}}(\pmb{o}, \pmb{a}; \theta))^2$ .
(a) Training Mixing Network.
(b) Training Agent Networks.
(c) Traditional Method.
Figure 2: (a) In DIFFER, the mixing network is trained using the team TD-loss  $L_{\mathrm{tot}}^{\mathrm{TD}}$ , while the parameters of the agent networks remain unchanged. (b) The decomposing individual experiences are inputted into Fair Experience Replay, from which they are sampled to calculate the individual TD-loss  $L_{\mathrm{ind}}^{\mathrm{TD}}$  for training the agent networks. (c) In traditional MARL experience replay methods, the mixing network and agent networks are updated using the team TD-loss  $L_{\mathrm{tot}}^{\mathrm{TD}}$  derived from team experiences.
However, in DIFFER, agent networks are updated using the individual TD-loss, which is calculated based on individual experience and denoted as  $L_{i} = (r_{i} + \gamma \widetilde{Q}_{i}(o_{i}^{\prime},a_{i}^{\prime};\theta_{p}^{-}) - Q_{i}(o_{i},a_{i};\theta_{p}))^{2}$ . To preserve the overall optimization objective of value factorization methods, it is desirable for an ideal individual reward approximation strategy to maintain invariance between the gradients of  $L_{\mathrm{tot}}$  (before decomposition) and the sum of  $L_{i}$  (after decomposition) with respect to the parameters of the agent networks. The optimization equivalence can be formulated as follows:
$$
\text {(I n v a r i a n c e o f G r a d i e n t)} \quad \frac {\partial L _ {\mathrm {t o t}}}{\partial \theta_ {p}} = \sum_ {i \in I} \frac {\partial L _ {i}}{\partial \theta_ {p}}, \tag {1}
$$
where  $\theta_{p}$  represents the parameters of the agent networks (noting that the agent networks of each agent share parameters). It is important to emphasize that this discussion focuses solely on the gradient of a single team experience. By solving the above partial differential equation, we are able to obtain an individual reward function that satisfies the invariance of gradients.
Proposition 1. The invariance of gradient in Equ.(1) is satisfied when individual reward of agent  $i$  is given by:
$$
r _ {i} = \left(R + \gamma \widetilde {Q} _ {\mathrm {t o t}} - Q _ {\mathrm {t o t}}\right) \frac {\partial Q _ {\mathrm {t o t}}}{\partial Q _ {i}} - \gamma \widetilde {Q} _ {i} + Q _ {i}. \tag {2}
$$
for any agent  $i\in I$
A rigorous proof and analysis of above proposition can be found in Appendix. Therefore, we approximate the individual rewards  $r_i$  for individual experiences  $\chi_i^{\mathrm{ind}}$  while preserving the original optimization objective. In this way, a team experience  $\chi^{\mathrm{team}}$  is decomposed into  $N$  individual experiences  $\{\chi_i^{\mathrm{ind}}\}_{i=1}^N$  successfully.
3.2 Fair Experience Replay Guided by Individual Experiences
After decomposition, we obtain a set of individual experience  $E = \{\chi_j^{\mathrm{ind}}\}_{j=0}^{N \cdot B}$  from a team experience mini-batch. Here,  $j$  denotes the index of the individual experience in the set,  $B$  represents the minibatch size, and  $N$  indicates the number of agents involved. Our objective is to construct a fair experience replay, a method that selects significant individual experiences from  $E$  to train the agent network. Similar to Priority Experience Replay (PER) method [28], the TD-error of each individual experience is used as a metric to measure its significance. A larger TD-error for an individual
Algorithm 1 Fair Experience Replay.
Require: The agent set  $I$ , the maximum steps  $t_{\mathrm{max}}$ , the mini-batch and target network update period  $M$ ;
1: Initialize replay memory  $D$ , set  $t_{\mathrm{step}} = 0$
2: Initialize the mixing network, agent network and target network with random parameters;
3: while  $t_{\mathrm{step}} \leq t_{\mathrm{max}}$  do
4: For each episode, observe initial centralized observation  $\{o_i^0\}_{i\in I}$
5: set  $t = 0$
6: while Not terminal do
7: With probability  $\epsilon$  select a random action  $a_i^t$ , otherwise  $a_i = \arg \max_{a_i^t} Q_i(o_i^t, a_i^t)$  for each agent  $i$ ;
8: Take action  $\pmb{a}^t$ , then receive team reward  $R_{t}$  and observation  $\{o_i^{t + 1}\}_{i\in I}$ ;
9:  $t\gets t + 1;$
0: end while
1:  $t_{\mathrm{step}} \gets t_{\mathrm{step}} + t$
2: Insert the current episode data into buffer  $D$
3: Sample a random episode mini-batch of team experiences  $B^{\text{team}}$  from  $D$ ;
4: Calculate total TD-loss  $L_{\mathrm{TD}}^{\mathrm{tot}}$  as Equ. (6) and update parameters of mixing network;
5: Calculate individual reward as Equ. (8) and decompose team experience mini-batch  $B^{\text{team}}$  to individual experience set  $E$ ;
6: Calculate sample probability as Equ. (3) of each individual experience in mini-batch  $E$ ;
7: Calculate the sample ratio  $\eta_{t_{\mathrm{step}}}$  and sample significant individual experiences from  $E$ ;
8: Calculate individual TD-loss  $L_{\mathrm{TD}}^{\mathrm{ind}}$  as Equ. (7) and update parameters of agent networks;
9: Update target network parameters with period  $M$ .
10: end while
experience indicates that it is more valuable for the current model, and thus more likely to be selected for training. The individual experience  $\chi_{j}^{\mathrm{ind}}$  is sampled with probability:
$$
P \left(\chi_ {j} ^ {\text {i n d}}\right) = \frac {p _ {j} ^ {\alpha}}{\sum_ {k} p _ {k} ^ {\alpha}}, \tag {3}
$$
where  $p_j = |\delta_j| + \epsilon$ .  $\delta_j$  is the TD-error of  $\chi_j^{\mathrm{ind}}$  and  $\epsilon$  is a small positive constant in case TD-error is zero. The hyper-parameter  $\alpha$  determines the degree of prioritization. It degenerates to uniform sample cases if  $\alpha = 0$ . To correct the bias,  $\chi_j^{\mathrm{ind}}$  is weighted by importance-sampling (IS) weights as follows:
$$
\omega_ {j} = \left(\frac {1}{N} \cdot \frac {1}{P \left(\chi_ {j} ^ {\text {i n d}}\right)}\right) ^ {\beta} / \max  _ {k} \omega_ {k}. \tag {4}
$$
The hyper-parameter  $\beta$  anneals as introduced in PER.
In cases where the agents within a team are homogeneous, their individual experiences tend to exhibit similar TD-errors. As a result, these individual experiences are assigned similar probabilities during the fair experience replay process. In such circumstances, the fair experience replay approach employed by DIFFER effectively degenerates into a traditional method that utilizes team experiences.
Let  $E'$  denote the selected individual experience set, which is a subset of the original individual experience set  $E$ . We define the sample ratio as  $\eta := \#E'/\#E$ , where  $\#E'$  and  $\#E$  are the numbers of individual experiences in  $E'$  and  $E$ , respectively. Since the model is initially at a low training level and Q-value estimation errors are large, only a small portion of the individual experiences is worth training. However, this portion increases as the training progresses. Thus, motivated by the warm-up technique proposed in [36], we set  $\eta < 1.0$  at the beginning of training and gradually increase it as the number of training steps increases. The sample ratio  $\eta_{t_i}$  for a given training step  $t_i$  can be expressed as follows:
$$
\eta_ {t _ {i}} = \left\{ \begin{array}{l l} \eta_ {\text {s t a r t}} + \left(\eta_ {\text {e n d}} - \eta_ {\text {s t a r t}}\right) \frac {t _ {i}}{p t _ {\max}}, & t _ {i} <   p t _ {\max} \\ \eta_ {\text {e n d}}, & t _ {i} \geq p t _ {\max} \end{array} \right. \tag {5}
$$
Here, the hyper-parameters  $\eta_{\mathrm{start}}$  and  $\eta_{\mathrm{end}}$  denote the initial and final values of the sample ratio, respectively. The hyper-parameter  $p$  is the proportion of the time steps at which the sample ratio
increases, and  $t_{\mathrm{max}}$  is the overall number of training steps. The ablation studies regarding the warm-up technique are provided in Appendix.
3.3 Overall Training and Evaluation
During the training phase, we first sample a team experience mini-batch  $B^{\mathrm{team}}$  from the replay buffer. The parameters of mixing network  $\theta_{m}$  are optimized with  $L_{\mathrm{tot}}$  calculated on the mini-batch by
$$
L _ {\mathrm {T D}} ^ {\mathrm {t o t}} \left(\theta_ {m}\right) = \sum_ {\boldsymbol {\chi} ^ {\mathrm {t e a m}} = \left(\boldsymbol {o}, \boldsymbol {a}, \boldsymbol {o} ^ {\prime}, R\right) \in \boldsymbol {B} ^ {\mathrm {t e a m}}} \left(R + \gamma \cdot \widetilde {\boldsymbol {Q} _ {\mathrm {t o t}}} \left(\boldsymbol {o} ^ {\prime}, \theta_ {m} ^ {-}\right) - Q _ {\mathrm {t o t}} \left(\boldsymbol {o}, \boldsymbol {a}; \theta_ {m}\right)\right) ^ {2}. \tag {6}
$$
Next, we approximate the optimization equivalence of individual rewards of each individual experience as Equ. (8) and decompose a team experience into multiple individual experiences. Among them, we select a significant individual experience set  $E'$  with probability in Equ. (3). The parameters of agent networks  $\theta_p$  are optimized by
$$
L _ {\mathrm {T D}} ^ {\mathrm {i n d}} \left(\theta_ {p}\right) = \sum_ {\chi_ {j} ^ {\mathrm {i n d}} = \left(o _ {j}, a _ {j}, o _ {j} ^ {\prime}, r _ {j}\right) \in E ^ {\prime}} \omega_ {j} \left(r _ {j} + \gamma \cdot \widetilde {Q _ {j}} \left(o _ {j} ^ {\prime}; \theta_ {p} ^ {-}\right) - Q _ {j} \left(o _ {j}, a _ {j}; \theta_ {p}\right)\right) ^ {2}, \tag {7}
$$
where  $\omega_{j}$  is a weight assigned to individual experience  $\chi_{j}^{\mathrm{ind}}$  as calculated in Equ. (4).
During the inference phrase, agent  $i$  chooses a greedy action according to its individual action-value  $Q_{i}$  for decentralized execution. Therefore, the DIFFER framework meets centralized training and decentralized execution. The overall training and evaluation are presented in Algo. 1.
In this section, we conduct several experiments to answer the following questions: (1) Are the decomposing individual rewards calculated using Equ. (8) optimization equivalent to the team reward experimentally? (Sec. 4.2) (2) Can DIFFER improve the performance of existing value factorization methods compared to the baselines across different environments? (Sec. 4.3) (3) Can DIFFER determine a more reasonable agent policy compared to the baselines? (Sec. 4.3) (4) Can DIFFER successfully distinguish and exploit the important individual experiences? (Sec. 4.4)
For every graph we plot, the solid line shows the mean value and the shaded areas correspond to the min-max of the result on 5 random seeds. All the experiments are conducted on a Ubuntu 20.04.5 server with Intel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz and GeForce RTX 3090 GPU. The code is available in https://github.com/cathyhxh/DIFFER.
4.1 Experiments Setting
Training Environments. We choose discrete action space environments The StarCraft Multi-Agent Challenge (SMAC) [29] and Google Football Research (GRF) [30], as well as a continuous action space environment Multi-Agent Mujoco (MAMujoco)[31] to conduct our experiments. SMAC is currently a mainstream cooperative multi-agent environment with partial observability. We use the default environment setting for SMAC with version SC 2.4.10. GFR is a platform that facilitates multi-agent reinforcement learning in the field of football. We conduct the experiments on two academy scenarios in GRF. MAMujoco is a continuous cooperative multi-agent robotic control environment. Each agent represents a part of a robot or a single robot. All of the tasks mentioned in this work are configured according to their default configuration.