SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Base Models. We select QMIX [1], COMIX [31], QPLEX [2] and MASER [37] as our base models for comparison. QMIX and QPLEX are renowned value factorization methods that have been widely used in discrete action space environments. COMIX is an extension of QMIX specifically designed for continuous action space environments. These three base models employ team experience to train their overall value factorization frameworks, as illustrated in Fig. 2(c). Furthermore, we include MASER, a MARL experience replay method, in our evaluation. Similar to our methods, MASER generates individual rewards and trains agent networks using individual experiences.
Implementation Hyper-parameters. All base models were implemented by faithfully adhering to their respective open-source codes based on PyMARL. Regarding the warm_up trick for the sample
(a)  $27\mathrm{m\_vs\_30m}$
(b) MMM2
Figure 3: Performance comparison of QMIX-divide and QMIX on SMAC scenarios, highlighting the optimization equivalence between team reward and individual reward calculated by DIFFER.
(c) 2c_vs_64zg
(d) 3s5z_vs_3s6z
(a)  $27\mathrm{m\_vs\_30m}$
Figure 4: Performance comparison of QMIX-DIFFER (ours) and QMIX on SMAC scenarios, highlighting the performance improvement of our method towards QMIX.
(b) MMM2
ratio, we initially set the ratio to 0.8, which gradually increases linearly until reaching 1.0 at  $60\%$  of the total training steps. From that point onwards, the sample ratio remains fixed at 1.0 for the remainder of the training process. For a comprehensive overview of the hyper-parameter settings employed in our experiments, we refer readers to the appendix.
4.2 Optimization Equivalence of the Individual Experiences
In this section, our objective is to investigate whether the decomposition of individual experiences in DIFFER yields an equivalence to team experiences within the original learning framework. To accomplish this, we focus on the QMIX as the base model, which updates agent networks using the team TD-loss computed from team experiences. We introduce a variant model named QMIX-divide. In QMIX-divide, the agent networks are updated using the individual TD-loss calculated from all decomposed individual experiences. Similar to QMIX, the mixing network of QMIX-divide is updated using the team TD-loss. Compared with DIFFER, QMIX-divide preserves the decomposer of team experiences but omits this selection phase in the fair experience replay. Our aim is to test the hypothesis that the decomposed individual experiences have no impact on agent network optimization. We compare the performance of QMIX-divide and QMIX on SMAC scenarios, as depicted in Fig. 3. Our results demonstrate that QMIX-divide and QMIX exhibit nearly identical performance, providing evidence in support of the optimization equivalence between the team reward and the approximated individual reward. Furthermore, we highlight the critical role played by the selection phase in the fair experience replay, as the omission of this phase in QMIX-divide leads to comparable performance with the baselines QMIX.
Performance Improvement towards Value Factorization Methods. We investigated whether DIFFER could enhance the performance of established value factorization methods. To evaluate this, we conducted experiments using QMIX, QPLEX, and COMIX as the base models in various environments. The training curves for each model are depicted in Fig. 4, Fig. 5, Fig. 6, and Fig. 7. As illustrated in the figures, DIFFER exhibited notable improvements in both learning speed and overall performance compared to the baselines during the training process. The magnitude of improvement varied across different scenarios. Notably, DIFFER demonstrated a more significant performance boost in scenarios where agents possessed distinct physical structures (e.g., agents with different
(b) MMM2
Figure 5: Performance comparison of QPLEX-DIFFER (ours) and QPLEX on SMAC scenarios, highlighting the performance improvement of our method towards QPLEX.
(a) 3_vs_1_with_keeper
Figure 6: Performance comparison of QMIX-DIFFER (ours) and QMIX on GRF scenarios, highlighting the performance improvement of our method towards QMIX.
(b) counterattack-hard
(a) Humanoid-v2_0
Figure 7: Performance comparison of COMIX-DIFFER (ours) and COMIX on MAMujoco scenarios, highlighting the performance improvement of our method towards COMIX.
(b) manyagent_swimmer_1
joints in Humanoid - v2_0) or subtasks (e.g., agents with different skills in 3s5z_vs_3s6z). This observation aligns with our hypothesis that scenarios with greater dissimilarity among agents result in a more pronounced improvement through the distinction of individual experiences facilitated by DIFFER. Conversely, in scenarios where agents were homogeneous (e.g.,  $27\mathrm{m\_vs\_}30\mathrm{m}$  in SMAC), DIFFER exhibited performance similar to that of the base model. In such cases, the lack of significant differences among agents rendered the decomposition of individual experiences less impactful. Overall, our findings demonstrate that DIFFER effectively enhances the performance of established value factorization methods. The degree of improvement depends on the specific characteristics of the scenarios, emphasizing the importance of considering the heterogeneity of agents when applying DIFFER.
Comparison with MARLER methods. In Fig. 4, we present a comparison of the performance of DIFFER and MASER. While both models implement individual rewards and leverage individual experiences to update agent networks, they differ in their optimization objectives. Specifically, DIFFER retains the original objective of maximizing the team action-value  $Q_{tot}$ . In contrast, MASER aims to maximize a mixture of individual action-value  $Q_{i}$  and team action-value  $Q_{tot}$ , which alters the original optimization objective. As a consequence of this shift in optimization objective, the performance of MASER is observed to be worse than DIFFER in challenging environments.
Figure 8: Screenshots of QMIX (left) and QMIX-DIFFER (right) on SMAC scenario MMM2. The red team is controlled by a trained model, while the blue team is controlled by a built-in AI. The Medivac in the red team is marked by an orange square. The red team is controlled by QMIX in the left subfigure and by QMIX-DIFFER in the right subfigure.
Figure 9: The normalized standard deviation (std) of TD-error of individual experiences.
Case Study. The screenshot in Fig. 8 shows the game screen for the QMIX and QMIX-DIFFER models in the SMAC scenario MM2. As previously noted, learning an effective strategy for the medivac unit is particularly challenging due to its distinct sub-task and a limited number of units. A successful medivac strategy must incorporate a viable movement strategy that avoids enemy attacks and reduces unnecessary damage, as well as a sound selection of healing objects that maximizes team damage output. In the QMIX controlled agent team (Fig. 8, left), the medivac failed to learn an appropriate movement strategy, moving too close to the enemy team and dying early in the episode. As a result, the other agents bled out quickly and the episode ended in defeat. In contrast, the agent team controlled by QMIX-DIFFER (Fig. 8, right) demonstrated an effective medivac strategy, positioning the unit in a safe location to avoid enemy attacks and providing healing support throughout the game. The resulting advantage for the red team led to a final victory.
4.4 Analysis for TD-error of Individual Experiences
In this section, we delve into the differentiation of individual experiences by analyzing their TD- errors. Throughout the training process, we compute the standard deviation (std) of the TD- errors of the decomposing individual experiences and normalize them using the mean TD-error. Figure 9 showcases the training curve of the normalized std of individual TD-errors. Notably, we observe a substantial increase in the normalized std at the initial stages of training, which remains consistently high until the conclusion of the training process. This observation underscores the substantial distinctions that exist among individual experiences when evaluated based on their TD- errors. Our DIFFER framework, by decomposing individual rewards, effectively captures and exploits these distinctions among individual experiences. Consequently, DIFFER exhibits superior performance compared to the baseline models. These findings highlight the efficacy of DIFFER in leveraging the differentiation among individual experiences to enhance learning outcomes in MARL settings.
4.5 Visualization of Individual Rewards Produced by DIFFER
In order to provide a clear visual representation of the individual rewards acquired through our proposed DIFFER method, we introduce game screenshots alongside the corresponding individual rewards for each agent, as illustrated in Fig. 10. For the purpose of analysis, we consider two distinct timesteps of an episode within the MMM2 scenario. Our team comprises 2 Marauders (index 0-1), 7 Marines (index 2-8), and a Medivac (index 9). The team rewards assigned to agents are proportional to the damage inflicted on enemy units and the number of enemy unit deaths. During the initial stage of the episode (timestep 4, (a)), our units possess sufficient health and do not require assistance from the Medivac. Consequently, the contribution of the Medivac to the team's performance is minimal. As a result, the individual reward attributed to the Medivac is relatively lower compared to the Marauders and Marines, who play active roles in offensive opera
(b) timestep=14
Figure 10: The visualization of individual rewards produced by our DIFFER method, along with game screenshots captured at timestep 4 and timestep 14 of the MMM2 scenario. The index of each agent have been marked in orange number. The consistent correlation between individual rewards and the ongoing game situation serves as a testament to the efficacy and rationality of our DIFFER method.
tions. Agents 2-5, being in closer proximity to the enemy and inflicting substantial damage, enjoy higher individual rewards due to their effective engagement strategies. At timestep 14 (b), except for agent 7 and 8 who have perished, the individual rewards assigned to each agent are substantially balanced. This indicates a comparable level of contribution from all agents towards achieving team objectives at this particular point in the game. The persistent correlation between individual rewards
and the dynamic in-game circumstances not only reaffirms the effectiveness of our DIFFER method but also validates its rationality.
Experiment Replay (ER) is a widely used mechanism for improving data utilization and accelerating learning efficiency in reinforcement learning (RL). ER reuses the historical experience data for updating current policy [34, 38]. In multi-agent RL, using a single-agent ER algorithm directly to obtain individual experience is naive if agents can obtain accurate individual rewards from the environment [39, 40]. However, in the multi-agent problem setting addressed in this work, a team of agents only has access to a shared team reward, making it impossible to obtain an accurate individual reward. Consequently, multi-agent ER algorithms employ joint transitions as minimal training data [41, 1, 2]. MASER[37] is a method similar to ours, as it calculates individual rewards from the experience replay buffer and uses individual experience to train the individual value network. However, MASER generates individual rewards to maximize the weighted sum of individual action-value and team action-value, which may violate the original optimization target of multi-agent RL, i.e., maximizing the team action-value. In our work, we propose a novel strategy for approximating individual reward and demonstrate its optimization equivalence. To our knowledge, DIFFER is the first work to approximate individual reward in a way that ensures optimization equivalence.
Limitation. A limitation of the DIFFER method is the introduction of additional computation when calculating the individual reward and TD-error. This increased computational cost is an inherent trade-off for the benefits gained in terms of fairness and improved learning efficiency. The additional computational burden should be considered when applying this method in resource-constrained settings or scenarios with strict real-time requirements.
In conclusion, this paper presents the DIFFER framework. By decomposing the team reward into individual rewards, DIFFER enables agents to effectively distinguish and leverage important individual experiences. Through the invariance of network gradients, we derive a partial differential equation that facilitated the computation of the underlying individual rewards function. By incorporating the solved closed-form individual rewards, we calculate the individual temporal-difference error, providing the significance of each experience. Notably, DIFFER showcases its adaptability in handling both homogeneous and diverse individual experiences, demonstrating its versatility in various scenarios. Our extensive experiments on popular benchmarks demonstrate the effectiveness of DIFFER in terms of learning efficiency and fairness, surpassing the performance of existing MARL ER methods. We hope that our work provides a new perspective on experience replay methods in MARL, inspiring further research and advancements in the field.
This work is supported by National Key R&D Program of China under Contract 2022ZD0119802, and National Natural Science Foundation of China under Contract 61836011. It was also supported by GPU cluster built by MCC Lab of Information Science and Technology Institution, USTC, and the Supercomputing Center of the USTC.
[1] T. Rashid, M. Samvelyan, C. Schroeder, G. Farquhar, J. Foerster, and S. Whiteson, “Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning,” in International Conference on Machine Learning (ICML), 2018.
[2] J. Wang, Z. Ren, T. Liu, Y. Yu, and C. Zhang, “Qplex: Duplex dueling multi-agent q-learning,” in International Conference on Learning Representations (ICLR), 2021.
[3] J. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson, "Counterfactual multi-agent policy gradients," in AAAI conference on artificial intelligence (AAAI), 2018.
[41] P. Sunehag, G. Lever, A. Gruslys, W. M. Czarnecki, V. Zambaldi, M. Jaderberg, M. Lanctot, N. Sonnerat, J. Z. Leibo, K. Tuys, et al., "Value-decomposition networks for cooperative multi-agent learning," arXiv preprint arXiv:1706.05296, 2017.
[42] C. S. de Witt, T. Gupta, D. Makoviichuk, V. Makoviichuk, P. H. Torr, M. Sun, and S. Whiteson, "Is independent learning all you need in the starcraft multi-agent challenge?", arXiv preprint arXiv:2011.09533, 2020.
[43] C. Yu, A. Velu, E. Vinitsky, J. Gao, Y. Wang, A. Bayen, and Y. Wu, "The surprising effectiveness of ppo in cooperative multi-agent games," Advances in Neural Information Processing Systems (NeurIPS), 2022.
A The Proof of Proposition 1
Proposition 1. The invariance of gradient is satisfied when individual reward of agent  $i$  is given by:
$$
r _ {i} = \left(R + \gamma \widetilde {Q} _ {\text {t o t}} - Q _ {\text {t o t}}\right) \frac {\partial Q _ {\text {t o t}}}{\partial Q _ {i}} - \gamma \widetilde {Q} _ {i} + Q _ {i}. \tag {8}
$$
Proof. According to the chain rule,
$$
\frac {\partial L _ {\text {t o t}}}{\partial \theta_ {p}} = \frac {\partial L _ {\text {t o t}}}{\partial Q _ {\text {t o t}}} \sum_ {i \in I} \frac {\partial Q _ {\text {t o t}}}{\partial Q _ {i}} \frac {\partial Q _ {i}}{\partial \theta_ {p}} = 2 (R + \gamma \widetilde {Q} _ {\text {t o t}} - Q _ {\text {t o t}}) \sum_ {i \in I} \frac {\partial Q _ {\text {t o t}}}{\partial Q _ {i}} \frac {\partial Q _ {i}}{\partial \theta_ {p}}; \tag {9}
$$
$$
\sum_ {i \in I} \frac {\partial L _ {i}}{\partial \theta_ {p}} = \sum_ {i \in I} \frac {\partial L _ {i}}{\partial Q _ {i}} \frac {\partial Q _ {i}}{\partial \theta_ {p}} = \sum_ {i \in I} 2 \left(r _ {i} + \gamma \widetilde {Q} _ {i} - Q _ {i}\right) \frac {\partial Q _ {i}}{\partial \theta_ {p}}. \tag {10}
$$
$$
\begin{array}{l} \frac {\partial L _ {\mathrm {t o t}}}{\partial \theta_ {p}} = \sum_ {i \in I} \frac {\partial L _ {i}}{\partial \theta_ {p}}, \\ \Longleftrightarrow 2 (R + \gamma \widetilde {Q} _ {\mathrm {t o t}} - Q _ {\mathrm {t o t}}) \sum_ {i \in I} \frac {\partial Q _ {\mathrm {t o t}}}{\partial Q _ {i}} \frac {\partial Q _ {i}}{\partial \theta_ {p}} = \sum_ {i \in I} 2 (r _ {i} + \gamma \widetilde {Q} _ {i} - Q _ {i}) \frac {\partial Q _ {i}}{\partial \theta_ {p}}, \\ \Longleftrightarrow 2 \sum_ {i \in I} \left[ \left(R + \gamma \widetilde {Q} _ {\mathrm {t o t}} - Q _ {\mathrm {t o t}}\right) \frac {\partial Q _ {\mathrm {t o t}}}{\partial Q _ {i}} \right] \frac {\partial Q _ {i}}{\partial \theta_ {p}} = 2 \sum_ {i \in I} \left(r _ {i} + \gamma \widetilde {Q} _ {i} - Q _ {i}\right) \frac {\partial Q _ {i}}{\partial \theta_ {p}}, \tag {11} \\ \Lleftarrow (R + \gamma \widetilde {Q} _ {\mathrm {t o t}} - Q _ {\mathrm {t o t}}) \frac {\partial Q _ {\mathrm {t o t}}}{\partial Q _ {i}} = r _ {i} + \gamma \widetilde {Q} _ {i} - Q _ {i}, \forall i \in I \\ \Longleftrightarrow r _ {i} = \left(R + \gamma \widetilde {Q} _ {\mathrm {t o t}} - Q _ {\mathrm {t o t}}\right) \frac {\partial Q _ {\mathrm {t o t}}}{\partial Q _ {i}} - \gamma \widetilde {Q} _ {i} + Q _ {i}, \forall i \in I. \\ \end{array}
$$
In conclusion, a sufficient condition for invariance of gradients is as follow: individual reward of agent  $i$  is calculated as Equ. (8) for any  $i \in I$ .
B The Hyper-parameters Setting
In DIFFER, the replay buffer stores the most recent 5000 episodes, and mini-batches of size 32 are sampled from it. The target network is updated every 200 episodes. RMSProp is utilized as the optimizer for both the mixing network and agent network, with a learning rate of 0.0005,  $\alpha$  (decay rate) set to 0.99, and  $\epsilon$  (small constant) set to 0.00001. Gradients are clipped within the range of [-10, 10]. The discount factor for the expected reward (return) is 0.99.
Regarding the fair experience replay, we apply the "warm-up" technique to the sample ratio  $\eta$ . It linearly increases from 0.8 to 1.0 over  $60\%$  of the time steps and remains constant thereafter. The regulation parameter  $\alpha$ , which determines the prioritization degree in fair experience replay, is set to 0.8. The regulation parameter  $\beta$  of the sampling probability anneals linearly from 0.6 to 1.0.
For exploration, we employ an  $\epsilon$ -greedy strategy with  $\epsilon$  linearly annealed from 1.0 to 0.05 over 50K time steps and then held constant for the remainder of the training. The maximum time step during training is set to 2005000 for all experiments. Each scenario is run with 5 different random seeds.
The agent network architecture follows a DRQN[27] structure, consisting of a GRU layer for the recurrent layer with a 64-dimensional hidden layer. Before and after the GRU layer, there are fully-connected layers with 64 dimensions each. The mixing network is implemented using open-source code. All experiments on the SMAC benchmark adopt the default reward and observation settings provided by the benchmark. For the baseline algorithms, we use the authors' code with hyper-parameters fine-tuned specifically for the SMAC benchmark.
The supplementary materials contain the code for our DIFFER framework. The main codes of individual reward calculation and fair experience replay are shown in
DIFFER_code/learners/q_learner Divide.py and DIFFER_code/ER/PER/prioritized_memory.py, respectively.
The specific hyper-parameters are listed below.
Table 1: The hyper-parameters that used in our DIFFER framework.
Hyper parameters	Meaning	Value
epsilon_start	Start value of ε anneal	1.0
epsilon_end	End value of ε anneal	0.05
epsilon_annexe_time	Duration step of ε anneal	50000
mempool_size	Memory pool size in learner	5000
batch_size	Batch size per training	32
target_update_interval	Interval steps between 2 updates of target network	200
lr	Learning rate	0.0005
regularization	Weights of L1-regularizer	0.0005
tot_OPTimizer	Optimizer in training	RMSProp
tot_OPT_alpha	Alpha of optimizer	0.99
tot_OPT_EPS	Epsilon of optimizer	0.00001
ind_OPTimizer	Optimizer in training	RMSProp
ind_OPT_alpha	Alpha of optimizer	0.99
ind_OPT_EPS	Epsilon of optimizer	0.00001
grad_norm Clip	Clip range of gradient normalization	10
ηstart	Start value of η warm up	0.8
ηend	End value of η warm up	1.0
p	Duration step ratio of η warm up	0.6
α	Regulation parameter of sampling probability	0.8
ε	Regulation parameter of sampling probability	0.01
βstart	Start value of β warm up	0.6
βend	End value of β warm up	1.0
C Ablation Studies for warm_up Technique of Sample Ratio
To highlight the significance of the warm_up trick for the individual experience sample ratio  $\eta$ , we conduct ablation experiments on SMAC maps MMM2. Fig. 11 showcases the performance of QMIX-DIFFER with and without the warm_up sample ratio. In this context, the model W-warm_up (ours) refers to the setting where the sample ratio linearly increases as the training progresses. On the other hand, W/O-warm_up denotes the scenario where the sample ratio remains fixed throughout the training phase. To ensure a fair comparison, we conduct experiments with different fixed values of  $\eta$ , while keeping the batch-size of all three models in Fig. 11 consistent at 32. Let us assume that the overall quantity of training data is  $K$  for W/O-warm_up- $\eta = 1.0$ . By employing the hyper-parameter setting detailed in Sec. B, we can calculate that the overall training data amounts to  $0.94K$  for W-warm_up. Consequently, we select  $\eta$  from the set  $\{0.8, 0.94, 1.0\}$ . The results clearly demonstrate that the warm_up trick leads to performance improvements, even when using a smaller amount of training data. This highlights the importance of incorporating the warm_up strategy into the training process.
D Analysis of Sampling Percentage
In this section, we present the distribution of individual experience sampling percentages across different agent classes within the MMM2 scenario from the SMAC environment[29]. The team composition consists of three distinct unit classes: 1 Medivac unit responsible for healing teammates, 2 Marauder units specialized in offensive actions, and 7 Marine units dedicated to attacking. Throughout the training process, DIFFER selects important individual experiences from a diverse set of 10 agents to update the agent networks. We calculate the sampling percentages of individual experiences for
Figure 11: Ablation studies regarding the warm up trick of individual transition sample ratio on MMM2. "W-warm_up (ours)" represents the sample ratio increases as the training step increases until 1.0. "W/O-warm_up" represents the sample ratio  $\eta$  is fixed during training.
Figure 12: Average percentage of sampled individual experiences from each unit class on SMAC scenario MMM2. The number in the label represents the quantity of unit. The sampling percentage of a unit class is high when it poses a learning challenge.
each agent and derive the average sampling percentage for each unit class. The corresponding curves, focusing on the later stages of training for improved stability, are depicted in Fig. 12.
The graph showcases significant variations in the average sampling percentages among the unit classes, reflecting their distinct learning difficulties. As the learning difficulty of a particular unit class intensifies, a larger proportion of individual experiences from that class are sampled during training. This adaptive behavior of the DIFFER method highlights its ability to dynamically adjust the sampling percentages to address the unique learning challenges encountered by each unit class. Notably, achieving effective strategies for the Medivac unit proves particularly challenging due to its distinctive sub-task and limited availability. Consequently, the average sampling percentage of the Medivac unit surpasses that of the other two unit classes significantly. Although the number of Marine units exceeds that of Marauder units, the Marines are encouraged to adopt a specialized positioning strategy. This strategy requires the Marines to spread out in a fan-like formation, aiming