SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 1 shows performance of SUPER implemented on dueling DDQN ("SUPER DDQN") compared to the control benchmarks discussed above. In addition, Figure 2 shows final performance in the three PettingZoo environments in a barplot for easier readability. In summary, we see that (quantile and gaussian) SUPER (red bars) clearly outperform baseline DDQN on all domains, as well as both SEAC and MADDPG baselines where we were able to run them. Particularly notable is a jump in performance from DDQN (green) to quantile SUPER (leftmost red bars in Figure 2) in the PettingZoo domains. Performance of SUPER is even comparable to that of parameter sharing, our "best case" fully centralized comparison benchmark. Aside from parameter sharing, SUPER wins or ties all pairwise comparisons against baselines. Results further hold for additional variations of environments with co-evolving opponents, when using SUPER with a plain (non-dueling, non-double) DQN algorithm, and across hyperparameter settings, which we discuss in the appendix.
In more detail, we find that SUPER (red) consistently outperforms the baseline DQN/DDQN algorithm (green), often significantly. For instance in Pursuit SUPER-DDQN achieves over twice the reward of no-sharing DDQN at convergence, increasing from 181.4 (std.dev 4.1) to 454.5 (std.dev 5.9) for quantile SUPER-DDQN measured at 800k training steps. In both Battle and Adversarial-Pursuit, we enabled SUPER-sharing for only one of the two teams each, and see that this significantly improved performance of the sharing team (as measured by sum of rewards of all agents in the team across each episode), especially mid-training. For instance in Battle, the blue team performance increase from -19.0 (std.dev 11.0) for no-sharing DDQN to 5.5 (std.dev 8.7) for quantile SUPER-DDQN at 150k timesteps. In Adversarial-Pursuit, the prey performance increased from -719.8 (std.dev 66.8) to -506.3 (std.dev 35.0) at 300k timesteps.
SUPER performs significantly better than SEAC (blue) and MADDPG (violet). MADDPG performed very poorly on all domains despite extensive attempts at hyperparameter tuning. SEAC shows some learning in Pursuit at 800k timesteps, but remains well below even baseline DDQN performance, again despite extensive hyperparameter tuning. We have also run experiments with SEAC in Pursuit for significantly longer (not shown in the figure), and saw that performance eventually catches up after around 8M timesteps, i.e. a roughly ten-fold difference in speed of convergence. SUPER (red) significantly outperforms parameter-sharing DQN/DDQN (turquoise) in Pursuit and performs similar to it in Battle, whereas parameter-sharing performs significantly better in Adversarial-Pursuit. Note that parameter sharing has often been observed to perform extremely well [40, 39, 11]. We therefore consider this a strong positive result.
Finally, these results and conclusions show remarkable stability across settings, underlying algorithm, and hyperparameters. In Section C.2 in the appendix we show that similar results hold using SUPER
Figure 3: Performance of quantile SUPER vs share-all and uniform random experience sharing in Pursuit at 800k timesteps.
Figure 4: Performance of quantile SUPER with varying bandwidth in Pursuit at 1-2M timesteps.
on a baseline DQN (non-dueling, non-double) algorithm. In Section C.3 in the appendix we show results from SUPER onueling DDQN on domains where the opposing team agents are co-evolving rather than pretrained. In Section C.5 in the appendix we show that these improvements over baseline DDQN are stable across a wide range of hyperparameter choices. Relative performance results and conclusions in all these cases largely mirror the ones presented here.
We ran an ablation study to gain a better understanding of the impact of two key features of our algorithm: selectivity (only sharing a small fraction of experiences) and priority (sharing experiences with the highest td-error). To this end, we compared SUPER against two cut-down versions of the algorithm: One, we share all experiences indiscriminately. This is also equivalent to a single "global" replay buffer shared between all agents. Two, we share (a small fraction of) experiences, but select experiences uniformly at random. Figure 3 shows the performance of these two ablations at end of training in Pursuit and Battle. We see that both ablations perform significantly worse than SUPER, with neither providing a significant performance uplift compared to baseline DDQN. $^4$  This shows that both selectivity as well as priority are necessary to provide a performance uplift, at least in some domains.
5.6 Bandwidth Sensitivity
In the Pursuit domain, we performed an analysis of the performance of SUPER-DDQN for varying target bandwidths ranging from 0.0001 to 1 (sharing all experiences). Figure 4 shows the converged performance of SUPER-DDQN with quantile-based experience selection in Pursuit. A label of "DQN" indicates that no sharing is taking place, i.e. decentralized DDQN; a label of "all" indicates that all experiences are shared, without any prioritized selection. Numerical labels give different target bandwidths. Two things stand out to us: First, sharing all experiences indiscriminately does not result in increased performance. In fact, at convergence, it results in slightly lower performance than no-sharing DDQN. Second, there is a clear peak of performance around a target bandwidth of 0.01 - 0.1, which also holds for gaussian and stochastic experience selection (we refer the reader to the appendix for more details). We conclude that sharing experiences selectively is crucial for learning to benefit from it.
5.7 Experience Selection
Gaussian experience selection performed similarly to the quantile selection we designed it to approximate. Its actual used bandwidth was however much less responsive to target bandwidth than the other two variants. We believe this demonstrates that in principle approximating the actual distribution of
4The situation in Adversarial Pursuit is more nuanced and highly dependent on training time, but overall shows a similar picture especially early in training.
td-errors using mean and standard deviation is feasible, but that more work is needed in determining the optimal value of  $c$  in equation 3. Stochastic experience selection (dotted red) performs similar or worse than both other variants, but generally still comparably or better than baseline DQN/DDQN.
Conclusion We present selective multiagent PER, a selective experience-sharing mechanism that can improve DQN-family algorithms in multiagent settings. Conceptually, our approach is rooted in the same intuition that Prioritized Experience Replay is based on, which is that td-error is a useful approximation of how much an agent could learn from a particular experience. In addition, we introduce the a second key design choice of selectivity, which allows semi-decentralized learning with small bandwidth, and drastically improves performance in some domains.
Experimental evaluation on DQN andueling DDQN shows improved performance compared to fully decentralized training (as measured in sample efficiency and/or converged performance) across a range of hyperparameters of the underlying algorithm, and in multiple benchmark domains. We see most consistent performance improvements with SUPER and a target bandwidth of 0.01-0.1 late in training, more consistent than indiscriminate experience sharing. Given that this effect appeared consistently across a wide range of hyperparameters and multiple environments, as well as on both DQN andueling DDQN, the SUPER approach may be useful as a general-purpose multi-agent RL technique. Equally noteworthy is a significantly improved performance early in training even at very low bandwidths. We consider this to be a potential advantage in future real-world applications of RL where sample efficiency and rapid adaption to new environments are crucial. SUPER consistently and significantly outperforms MADDPG and SEAC, and outperforms parameter sharing in Pursuit (but underperforms in Adversarial-Pursuit, and shows equal performance in Battle).
Discussion Our selective experience approach improves performance of both DQN andueling DDQN baselines, and does so across a range of environments and hyperparameters. It outperforms state-of-the-art multi-agent RL algorithms, in particular MADDPG and SEAC. The only pairwise comparison that SUPER loses is against parameter sharing in Adversarial-Pursuit, in line with a common observation that in practice parameter sharing often outperforms sophisticated multi-agent RL algorithms. However, we note that parameter sharing is an entirely different, fully centralized training paradigm. Furthermore, parameter sharing is limited in its applicability, and does not work well if agents need to take on different roles or behavior to successfully cooperate. We see this in the Pursuit domain, where parameter sharing performs poorly, and SUPER outperforms it by a large margin. The significantly higher performance than MADDPG and SEAC is somewhat expected given that baseline non-sharing DQN algorithms often show state-of-the-art performance in practice, especially with regard to sample efficiency.
It is noteworthy that deterministic (aka "greedy") experience selection seems to perform slightly better than stochastic experience selection, while in PER the opposite is generally the case [30]. We have two hypotheses for why this is the case. One, we note that in PER, the motivation for stochastic prioritization is to avoid low-error experiences never being sampled (nor re-prioritized) in many draws from the buffer. On the other hand, in SUPER we only ever consider each experience once. Thus, if in stochastic experience selection a high-error experience through random chance is not shared on this one opportunity, it will never be seen by other agents. In a sense, we may prefer deterministic experience selection in SUPER for the same reason we prefer stochastic selection in PER, which is to avoid missing out on potentially valuable experiences. Two, in all our current experiments we used (stochastic) PER when sampling training batches from the replay buffer of each agent. When using stochastic SUPER, each experience therefore must pass through two sampling steps before being shared and trained on by another agent. It is possible that this dilutes the probability of a high-error experience being seen too much.
We would also like to point out a slight subtlety in our theoretical motivation for SUPER: We use the sending agent's td-error as a proxy for the usefulness of an experience for the receiving agent. We believe that this is justified in symmetric settings, and our experimental results support this. However, we stress that this is merely a heuristic, and one which we do not expect to work in entirely asymmetric domains. For future work, we would be interested to explore different experience selection heuristics. As an immediate generalization to a more theoretically grounded approach, we wonder if using the td-error of each (potential) receiving agent could extend SUPER to asymmetric settings, and if it
could further improve performance even in symmetric settings. While this would effectively be a centralized-training approach, if it showed similar performance benefits as we have seen in symmetric settings for SUPER, it could nevertheless be a promising avenue for further work. Beyond this, we would be interested to explore other heuristics for experience selection. For instance, we are curious if the sending agent could learn to approximate each receiver's td-error locally, and thus retain the decentralized-with-communication training capability of our current approach. However, given that td-error is intrinsically linked to current policy and thus highly non-stationary, we expect there would be significant practical challenges to this.
In this current work, we focus on the DQN family of algorithms in this paper. In future work, we would like to explore SUPER in conjunction with other off-policy RL algorithms such as SAC [12, 21] and DDPG [32]. The interplay with experience sampling methods other than PER, such as HER [3] would also be interesting. If the improvements we see in this work hold for other algorithms and domains as well, this could improve multi-agent RL performance in many settings.
Finally, our approach is different from the "centralized training, decentralized execution" baselines we compare against in the sense that it does not require fully centralized training. Rather, it can be implemented in a decentralized fashion with a communications channel between agents. We see that performance improvements scale down even to very low bandwidth, making this feasible even with limited bandwidth. We think of this scheme as "decentralized training with communication" and hope this might inspire other semi-decentralized algorithms. In addition to training, we note that such a "decentralized with communication" approach could potentially be deployed during execution, if agents keep learning. While this is beyond the scope of the current paper, in future work we would like to investigate if this could help when transferring agents to new domains, and in particular with adjusting to a sim-to-real gap. We envision that the type of collaborative, decentralized learning introduced here could have impact in future applications of autonomous agents ranging from disaster response to deep space exploration.
The project was sponsored, in part, by a grant from the Cooperative AI Foundation. The content does not necessarily reflect the position or the policy of the Cooperative AI Foundation and no endorsement should be inferred.
[1] Sanjeevan Ahilan and Peter Dayan. Correcting experience replay for multi-agent communication. arXiv preprint arXiv:2010.01192, 2020.
[2] Stefano V Albrecht and Peter Stone. Autonomous agents modelling other agents: A comprehensive survey and open problems. Artificial Intelligence, 258:66-95, 2018.
[3] Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
[4] Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
[5] James Bergstra, Daniel Yamins, and David Cox. Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. In International conference on machine learning, pages 115-123. PMLR, 2013.
[6] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016.
[7] Filippos Christianos, Lukas Schäfer, and Stefano Albrecht. Shared experience actor-critic for multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 33:10707-10717, 2020.
[40] Justin K Terry, Nathaniel Grammel, Sanghyun Son, and Benjamin Black. Parameter sharing for heterogeneous agents in multi-agent reinforcement learning. arXiv e-prints, pages arXiv-2005, 2020.
[41] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
[42] Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. *Dueling network architectures for deep reinforcement learning*. In International conference on machine learning, pages 1995–2003. PMLR, 2016.
[43] Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3):279-292, 1992.
[44] Annie Xie, Dylan P Losey, Ryan Tolsma, Chelsea Finn, and Dorsa Sadigh. Learning latent representations to influence multi-agent interaction. arXiv preprint arXiv:2011.06619, 2020.
[45] Lianmin Zheng, Jiacheng Yang, Han Cai, Weinan Zhang, Jun Wang, and Yong Yu. Magent: A many-agent reinforcement learning platform for artificial collective intelligence. CoRR, abs/1712.00600, 2017.
[46] Changxi Zhu, Mehdi Dastani, and Shihan Wang. A survey of multi-agent reinforcement learning with communication. arXiv preprint arXiv:2203.08975, 2022.
Figure 5: Pursuit Environment
A Algorithm Details
Algorithm 1 SUPER algorithm for DQN
for each training iteration do  
    Collect a batch of experiences  $b$  {DQN}  
    for each agent  $i$  do  
        Insert  $b_i$  into buffer  $_i$  {DQN}  
    end for  
    for each agent  $i$  do  
        Select  $b_i^* \subseteq b_i$  of experiences to share  $^1$  {SUPER}  
        for each agent  $j \neq i$  do  
            Insert  $b*_i$  into buffer  $_j$  {SUPER}  
    end for  
    end for  
    for each agent  $i$  do  
        Sample a train batch  $b_i$  from buffer  $_i$  {DQN}  
        Learn on train batch  $b_i$  {DQN}  
    end for  
end for
Algorithm 1 shows a full pseudocode listing for SUPER on DQN.
B Experiment Domains
B.1 PettingZoo
SISL: Pursuit is a semi-cooperative environment, where a group of pursuers has to capture a group of evaders in a grid-world with an obstacle. The evaders (blue) move randomly, while the pursuers (red) are controlled by RL agents. If a group of two or more agents fully surround an evader, they each receive a reward, and the evader is removed from the environment. The episode ends when all evaders have been captured, or after 500 steps, whichever is earlier. Pursuers also receive a (very small) reward for being adjacent to an evader (even if the evader is not fully surrounded), and a (small) negative reward each timestep, to incentivize them to complete episodes early. We use 8 pursuers and 30 evaders.
MAgent: Battle is a semi-adversarial environment, where two groups of opposing teams are battling against each other. An agent is rewarded 0.2 points for attacking agents in the opposite team, and 5 points if the other agent is killed. All agents start with 10 health points (HP) and lose 2 HP in each attack received, while regaining 0.1 HP in every turn. Once killed, an agent is removed from the environment. An episode ends when all agents from one team are killed. The action space, of size 21 is identical for all agents, with (8) options to attack, (12) to move and one option to do nothing. Since no additional reward is given for collaborating with other agents in the same team, it is considered to be more challenging to form collaboration between agents in this environment. We use a map of size  $18 \times 18$  and 6 agents per team.
MAgent: Adversarial Pursuit is a predator-prey environment, with two types of agents, prey and predator. The predators navigate through obstacles in the map with the purpose of tagging the prey. An agent in the predators team is rewarded 1 point for tagging a prey, while a prey is rewarded  $-1$  when being tagged by a predator. Unlike in the Battle environment, prey agents are not removed from the game when being tagged. Note that prey agents are provided only with a negative or zero reward (when manage to avoid attacks), and their aim is thus to evade predator agents. We use 8 prey agents, and 4 predator agents.
B.2 MeltingPot
We chose a subset of three domains ("substrates") from the MeltingPot package [18]. Cooperative Mining is a collaborative resource harvesting game, where cooperation between players allows them to mine a more valuable type of resource than individual action does. Rationalizable Coordination is a coordination game embedded in a larger rich environment. Similarly Stag Hunt is as stag hunt game payoff matrix embedded in a larger game world.
B.3 Atari
We also ran experiments on a version of the Atari 2600 game Space Invaders [4, 6]. For this, we created two different variants. In the main variant, we ran the game in a native two-player mode. By default, observations from this are not strictly anonymous, as each player is identified using a fixed color. We thus made two changes to make observations strictly anonymous: First we added a colored bar to the bottom of the observation that indicates which player is being controlled. Then, we randomly shuffle each episode which agent controls which player in the game. This way, both policies learn to control each of the two player colors, and are able to make sense of observations from each other.
In a second variant, we took two separate instances of the game running a single-player mode and linked them to create a two-player environment. This provides an additional "sanity check" to ensure that the modifications we made to the primary variant did not influence results in our favor.
C.1 Additional Results on DDQN
In addition to the final performance shown in the main text, we show in Table 1 numerical results from all experiments.
Table 1: Performance in all three environments, taken at 800k timesteps (Pursuit), 300k timesteps (Battle), 300k timesteps (Adv. Pursuit). Numbers in parentheses indicate standard deviation. Highest performance in each environment is printed in bold.
Algorithm | Env	pursuit	battle	adversarial_pursuit
ddqn	181.43 (+-4.16)	9.46 (+-0.86)	-719.78 (+-66.82)
ddqn(paramshare)	-139.03 (+-121.11)	15.15 (+-8.64)	-268.78 (+-60.11)
ddqn_shareall	148.27 (+-9.22)	9.25 (+-6.60)	-568.91 (+-40.82)
maddpg	-342.24 (+-4.12)	-10.31 (+-0.0)	-1071.71 (+-8.56)
seac	32.51 (+-70.27)	-2.73 (+-0.03)	-993.40 (+-120.22)
super_ddqn_gaussian	373.63 (+-9.15)	16.28 (+-3.42)	-480.37 (+-15.79)
super_ddqnquantile	454.56 (+-5.89)	17.05 (+-2.74)	-506.29 (+-35.03)
super_ddqn_stochastic	266.42 (+-85.52)	7.43 (+-5.64)	-582.59 (+-15.20)
C.2 DQN and Dueling DDQN
For all of the DDQN and SUPER-DDQN variants discussed in Section 5, we consider also variants based on standard DQN. Figure 6 shows results from these experiments.
Figure 6: Performance of SUPER-DQN variants with target bandwidth 0.1 on Pursuit and Adversarial-Pursuit. For Pursuit, performance is the total mean episode reward from all agents. For Adversarial-Pursuit, performance is the total mean episode reward from all agents in the prey team. Shaded areas indicate one standard deviation.
Figure 7: Performance of SUPER-dueling-DDQN variants with target bandwidth 0.1 on all Battle and Adversarial=Pursuit, with co-evolving opponents. Performance is the total mean episode reward from all agents in the sharing team (blue team in Battle, grey team in Adversarial-Pursuit). Shaded areas indicate one standard deviation.
C.3 Co-Evolving Teams
In Battle and Adversarial-Pursuit we further show a variant where the opposing team are co-evolving with the blue / prey team. In this variant, all agents start from a randomly initialized policy and train concurrently, using a DDQN algorithm. However, only the blue / prey team share experiences using the SUPER mechanism. We only do this for the DDQN baseline as well as discriminate and share-all SUPER variants. This is in part because some of the other baseline algorithms do not support concurrently training opposing agents with a different algorithm in available implementations; and in part because we consider this variant more relevant to real-world scenarios where fully centralized training may not be feasible. We aim to show here how sharing even a small number of experiences changes the learning dynamics versus to non-sharing opponents. Figure 7 shows this variant.
C.4 Further Ablations
In addition to the ablations presented in the main text, we include here additional results from the Battle environment in Figure 9. Results are broadly similar to the results in the main text, with a notably bad performance for uniform random experience sharing.
C.5 Stability Across Hyperparameters
Figure 8 shows performance of no-sharing DDQN and SUPER-DDQN for different hyperparameters. As we can see, SUPER-DDQN outperforms no-sharing DDQN consistently across all the hyperparameter settings considered.