Selectively Sharing Experiences Improves Multi-Agent Reinforcement Learning
Authors: Matthias Gerstgrasser
Tom Danino
Sarah Keren
Venue: NeurIPS 2023 poster (2023)
Keywords: multi-agent reinforcement learning
reinforcement learning
deep q learning
cooperative ai
TL;DR: We share a small number of relevant experiences among agents to improve performance of multi-agent reinforcement learning.

Abstract: We present a novel multi-agent RL approach, Selective Multi-Agent Prioritized Experience Relay, in which agents share with other agents a limited number of transitions they observe during training. The intuition behind this is that even a small number of relevant experiences from other agents could help each agent learn. Unlike many other multi-agent RL algorithms, this approach allows for largely decentralized training, requiring only a limited communication channel between agents. We show that our approach outperforms baseline no-sharing decentralized training and state-of-the art multi-agent RL algorithms. Further, sharing only a small number of highly relevant experiences outperforms sharing all experiences between agents, and the performance uplift from selective experience sharing is robust across a range of hyperparameters and DQN variants.

Selectively Sharing Experiences Improves Multi-Agent Reinforcement Learning
Matthias Gerstgrasser
Tom Danino Sarah Keren
School of Engineering and Applied Sciences Harvard University
The Taub Faculty of Computer Science Technion Israel Institute of Technology tom. dww.daa damino@campus technion ac il sarahk@cs. technion. ac.il
Computer Science Department
Stanford University
matthias@seas. harvard edu
Abstract
We present novel multi-agent RL approach, Selective Multi-Agent Prioritized Experience Relay, in which agents share with other agents a limited number of transitions they observe during training. The intuition behind this that even a small number of relevant experiences from other agents could help each agent learn. Unlike many other multi-agent RL algorithms, this approach allows for largely decentralized training. requiring only limited communication channel between agents. We show that our approach outperforms baseline no-sharing decentralized training and state-of-the small number of highly relevant experiences outperforms sharing all experiences between agents, and the performance uplift from selective experience sharing is robust across a range of hyperparameters and DQN variants.
multi-agent RL algorithms. Further, sharing only
Introduction
Multi-Agent Reinforcement Learning (RL) is often considered a hard problem: The environment dynamics and returns depend the joint actions of all agents, leading significant variance and non-stationarity in the experiences of each individual agent. Much recent work [29 23] in multi-agent RI. focused mitigating the impact of these. Our work views multi-agent RI. more positively, by treating the presence of other agents can also be an asset that can be leveraged. In particular, we show that groups of agents can collaboratively explore the environment more quickly than individually.
To this end, we present novel multi-agent RL approach that allows agents to share a small number of experiences with other agents. The intuition is that if one agent discovers something important in the environment, then sharing this with the other agents should help them learn faster. However, is crucial that only important experiences indiscriminately will not improve learning. To this end, we make two crucial design choices: Selectivity and priority. Selectivity means only share small fraction of experiences. Priority inspired by well-established technique in single-agent RL., prioritized experience replay (PER) 30) With PER off-policy algorithm such DON 24 will sample experiences not uniformly, but proportionally to "how far off" the current policy's 's predictions are in each state, formally the temporal difference (Id) error. We this same metric prioritize which experiences share with other agents.
shared
will see that sharing all experiences
We dub the resulting multiagent RL approach "Selective Multi-Agent Prioritized Experience Relay" or "SUPER". In this, agents independently use a DQN algorithm to learn, but with a twist: Each agent relays its replay buffer, which they use for learning. This approach has several advantages:
experiences the other agents, who insert them directly into their
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

1. It consistently leads to faster learning and higher eventual performance, across hyperparam eter settings.
2. Unlike many "centralized training, decentralized execution" approaches, the SUPER learning paradigm allows for (semi-) decentralized training, requiring only limited bandwidth communication channel between agents.
3. The paradigm is agnostic to the underlying decentralized training algorithm, and can enhance many existing DQN algorithms. SUPER can be used together with PER, without PER
In addition to the specific algorithm we develop, this work also introduces the paradigm of "decen- tralized training with communication". This is 'middle ground between established approaches of decentralized and centralized training (including "centralized training, decentralized execution").
In the remainder of the paper, we will discuss related literature and technical preliminaries; introduce our novel algorithm in detail; describe experimental evaluation and results; and suggest avenues for future work.
2 Related Work
Multi-Agent RL approaches in the literature can be broadly categorized according to the degree of awareness of each learning agent of the other agents in the system [37]. On one end of this spectrum is independent decentralized learning where multiple learning agents (in the same environment) are unaware or agnostic to the presence of other agents in the system [36 19]. From the perspective of each agent, learning regards the other agents as part of the non-stationary environment. At the opposing end of the spectrum, in centralized control a single policy controls all agents. In between these two, a number of related strands of research have emerged.
Approaches nearer the decentralized end of the spectrum often use communication between the agents [46. 9 [3.. Several forms of cooperative communication have been formulated. by which agents can communicate various types of messages, either to all agents or to specific agent groups through dedicated channels 11 33 26 (7). While communication among agents could help with coordination, training emergent communication protocols also remains a challenging problem; recent empirical results underscore the difficulty of learning meaningful emergent communication protocols, even when relying on centralized training [16]. Related to this, and often used in conjunction, is which equips agent with some model of other agent's 's behavior.
modeling other agents 2 [6] 10 44
This in turn is related to a number recent approaches using centralized training but decentralized execution. A prevailing paradigm within this line of work assumes training stage during which a shared network (such as a critic) can be accessed by all agents to learn decentralised (locally- executable) agent policies 23 I
28 10 29]. These approaches successfully reduce variance during training, e.g. through a shared critic accounting for other agents' behavior, but rely on joint observations and actions of all agents. Within this paradigm, and perhaps closest related to our own work, 77 introduces an approach in which agents share (all of their) experiences with other agents. While this is based on on-policy actor-critic algorithm, it uses importance sampling to incorporate the off off-policy data from other agents, and the authors show that this can lead to improved performance in sparse-reward settings. Our work also relies sharing experiences among agents, but crucially relies on selectively sharing only some experiences.
A separate but related body of work is on transfer learning 88. While this type of work is not directly comparable, conceptually it uses a similar idea of utilizing one agent's 's learning to help another agent.
Off-Policy RL The approach we present in this paper relies intrinsically on off-policy RL algorithms. Most notable in this class is DQN 24), which achieved human-level performance on a wide variety of Atari 2600 games. Various improvements have been made to this algorithm since then, including
Note that while the name "SUPER" pays homage to PER, SUPER not related to PER other than using the same heuristic for relevance of experiences.
"Interestingly, in the appendix to the arXiv version of that paper, the authors state that experience sharing DQN did not improve performance their experiments, stark contrast with the result we will present in this paper. We believe this may be because their attempts shared experiences indiscrininately. compared to our selective sharing approach. We will see that sharing all experiences does not improve performance as much or as consistently as selective sharing.

---

dueling DQN 42 Double DQN (DDQN) 41, Rainbow [1] and Ape-X [1]]. Prioritized Experience
Replay 30 improves the performance of many of these, and is closely related to our own work. A variant for continuous action spaces is DDPG (32 Off-policy actor-critic algorithms [ 21 have
also been developed, in part to extend the paradigm to continuous control domains.
3 Preliminaries
Reinforcement learning (RL) deals with sequential decision-making in unknown environments
34 Multi-Agent Reinforcement Learning extends RL. multi-agent settings. A common model is a Markov game, or stochastic game defined as tuple (S, R. T,7) with states S. joint actions A {A' Ji=1 as collection of action sets A', one for each of the n agents, R = {R' ) as a collection of reward functions R' defining the reward r'(,, 8:) that each agent receives when the joint action a, E A is performed at state 8t, and 7 as the probability distribution over next states when a joint action is performed. In the partially observable case, the definition also includes a joint observation function, defining the observation for each agent at each state. In this framework, at each time step an agent has an experience =< St, At, S >, where the agent performs action A, at state S, after which reward R2+1 is received and the next state is S1+1. We focus here on decentralized execution settings in which each agent follows its own individual policy n; and seeks to maximize its discounted accumulated return. The algorithm we propose in this paper further requires that the Markov game is "anonymous", meaning all agents have the same action and observation spaces and the environment reacts identically to their actions. This is needed so that experiences from one agent are meaningful to another agent as-ís. Notice however that this does not imply that is necessarily optimal for all agents to adopt identical behavior.
Among the variety of RL solution approaches B5| 34]. focus here on malue-based methods that use state and action estimates to find optimal policies. Such methods typically use a value function V=(s) to represent the expected value of following policy from state onward, and Q-function Q= a). to represent for a given policy æ the expected rewards for performing action a in a given state s and following ı thereafter.
Q-learning [43 is temporal difference (td) method that considers the difference in Q values between two consecutive time steps. Formally, the update rule is QS.,, + Q(S,, Ar) + a[Rx+I y max,' Q( a²) Q(S,, A:))], where, a is the learning rate or step size setting the amount by which values are updated during training. The learned oction-value function, Q, directly approximates q², the optimal action-valuu function. During training, in order to guarantee convergence to an optimal policy, Q-learning typically uses e-greedy selection approach, according to which the best action is chosen with probability 1 - e, and a random action, otherwise.
Given an experience the id-error represents the difference between the Q-value estimate and actual reward gained in the transition and the discounted value estimate of the next best actions. Formally,
td(),) | + y max Q(S", Q(S, A)I
(1)
In the vanilla version of Q-learning Q-values stored in table, which is impractical in many real-world problem due to large state space size. In deep Q-Learning (DQN) [25], the Q-value table is replaced by function approximator typically modeled using a neural network such that Q(s, a, 0) R (s,a), where 0 denotes the neural network parameters.
Replay Buffer (RB) and Prioritized RB: An important component in many off-policy RL imple- mentations is a replay buffer [22), which stores past experiences. Evidence shows the replay buffer to stabilize training of the value function for DQN [24 25 and to reduce the amount of experiences required for an RL-agent to complete the learning process and achieve convergence (30.
Initial approaches that used replay buffer, uniformly sampled experiences from the buffer. However, some transitions are more effective for the learning process of an RL agents than others [II] Prioritized Experience Replay (PER) 30 explores the idea that replaying and learning from some transitions, rather than others, can enhance the learning process. PER suggests replacing the standard sampling method, where transitions are replayed according to the frequency they were collected from the environment, with a td-error based method, where transitions are sampled according to the value of their td-error.
3

---

As further extension, stochastic prioritization balances between strictly greedy prioritization and uniform random sampling. Hence, the probability of sampling transition defined as:
P
P(1
(2)
where > is the priority associated to transition and a determines the weight the priority (a is uniform sampling). The value of Pt is determined according the magnitude of the td-error, such that p; = [6:] + 6, de is the td-error and e a small positive constant that guarantees that transitions for which the td-error is zero have non-zero probability of being sampled from the buffer.
SUPER: Selective Multi-Agent Prioritized Experience Relay
Our approach rooted the same intuition as PER: that all experiences are equally relevant. We use this insight to help agents learn by sharing between them only a (small) number of their most relevant experiences. Our approach builds on standard DQN algorithms, and adds this experience sharing mechanism between collecting experiences and performing gradient updates, in each iteration of the algorithm:
1. (DQN) Collect rollout of experiences, and insert each agent's 's experiences into their own replay buffer.
2. (SUPER) Each agent shares their most relevant experiences, which are inserted into all the other agents' replay buffers.
3. (DQN) Each agent samples minibatch experiences from their own replay buffer, and performs gradient descent on it.
Steps and are standard DQN; we merely add an additional step between collecting experiences and learning them. As corollary, this same approach works for any standard variants of steps and 3, such as dueling DQN [42]. DDQN [41]. Rainbow [14] and other DQN improvements. Algorithm in the appendix gives more detailed listing of this algorithm. Notice that the only interaction between the agents training algorithms is the experience sharing step. This means that the algorithm can easily be implemented in decentralized manner with (limited) communications channel.
4.1 Experience Selection
We describe here three variants of the SUPER algorithm, that differ in how they select experiences to be shared.
Deterministic Quantile experience selection The learning algorithm keeps a list I of the (absolute) td-errors of its last & experiences (A 1500 by default). For configured bandwidth 3 and new experience et, the agent shares the experience if its absolute td-error [td(er)] is at least as large as the k 3-largest absolute td-error in l. In other words, the agent aims to share the top --quantile of its experiences, where the quantile is calculated over a sliding window of recent experiences.
[t:(e))) 2 quantile (few)
Deterministic Gaussian experience selection In this, the learning algorithm calculates the mean Ja and variance o² of the (absolute) td-errors of the k most recent experiences (k 1500 by default). then shares an experience e, if
td(e))) 2 μ 0
where e is constant chosen such that 1 - cc) 3. In other words, we use the c-quantile of normal distribution with the (sample) mean and variance most recent experiences. We include and benchmark this variant for two reasons. One, intuitively, we might want to be more sensitive to clusters of outliers, where using a quantile of the actual data might include only part the cluster, while Gaussian model might lead to the entire cluster being included. Two, mean and variance could be computed iteratively without keeping a buffer of recent td-errors, and thereby reducing memory requirements. We aim benchmark if this approximation impacts performance.

---

Stochastic weighted experience selection Finally, and most closely related to classical single- agent PER, this variant shares each experience with a probability that's proportional its absolute td-error. In PFR, given a train batch size b, we sample b transitions from the replay buffer without replacement, weighted by each transition's td-error. In SUPER, we similarly aim to sample a / fraction of experiences, weighted by their td-errors. However, in order to be able to sample transitions online, we calculate for each experience individually probability that approximates sampling- withoutt-eplacemmen in expectation. Formally, taking p; td(e:)), we broadcast experience with probability
p = min 11
P
Σx
similarly equation 2 and taking the sum over sliding window over recent experiences. It is easy to see that if / =1/ //batchsize, this is equivalent to sampling a single experience, weighted by td-error, from the sliding window. For larger bandwidth (3, this approximates sampling multiple E.
experiences without replacement: If none of the 3 En PE terms are greater than 1, this is exact. If
have to truncate any of these terms, slightly undershoot the desired bandwidth.
In our current experiments, we share experiences and update all quantiles, means and variances once per sample batch, for conveniendce and performance reasons; However, we could do both online, i.e. after every sampled transition, in real-world distributed deployments. We not expect this make significant difference.
Experiments and Results
We evaluate the SUPER approach on number of multiagent benchmark domains.
5.1 Algorithm and control benchmarks
Baseline: DDQN As baseline, we use a fully independent dueling DDQN algorithm. The algorithm samples sequence of experiences using joint actions from all the agents' policies, and then inserts each agent's s aatrraati-acciin--ewaatt---ere
transitions into that agent's 's replay buffer. Each agent's policy is periodically trained using a sample from its own replay buffer, sampled using PER. We refer to this baseline simply "DDQN" in the remainder of this section and in figures. In Section Cin in the appendix we also discuss a variant based on vanilla DQN.
SUPER-DDON We implement SUPER on top of the DDQN baseline. Whenever batch of experiences is sampled, each agent shares only its most relevant experiences (according to one of the criteria described in the previous section) which are then inserted into all other agents' replay buffers. As above, we run the SUPER experience sharing on whole sample batches. All DQN hyperparameters are unchanged the only difference from the baseline is the addition of experience sharing between experience collection and learning. This allows for controlled experiments with like-for-like comparisons.
Parameter-Sharing DDQN In parameter-sharing, all agents share the same policy parameters. Note that this is different from joint control, in which single policy controls all agents simultaneously; In parameter sharing, each agent is controlled independently by copy of the policy. Parameter sharing has often been found to perform very well in benchmarks, but also strictly requires centralized learner 40 it for completeness and as "best case" fully centralized comparison.
We therefore do not expect SUPER to outperform parameter sharing, but include
Multi-Agent Baselines We further compare against standard multi-agent RI. algorithms, specifically MADDPG 23 and SEAC [7]. Like parameter-sharing, these are considered "centralized training, decentralized execution" approaches.
5.2 Environments
As discussed in the Preliminaries (Section 3). SUPER applies to environments that are anonymous. We are furthermore particularly interested in environments that have separate reward signal for
5

---

Ennoceant pund.
bulla
-00
200
-000
-00
19809G00000 20000 SOOOO 100003330
1869
10000
20000
20000
30005
19000
230000
X0CI
4X00
50000 1980
250
200
150
190
100
90
2,000 SO0 1585 KEGI 1500 1000 17500000000
250CE 5903 7000 13000 TSICOO 1SC0 17000 DOOKS
2000
5000
75000
TEEKS 1599
19000
17900
Apertrs
5
Figure 1: Performance of SUPER-duche JuUPER-deling-DDOO variants with target bandwidth 0.1 on all domains. For team settings, performance is the total reward of all agents the sharing team; for all other domains performance is the total reward of all agents. Shaded areas indicate one standard deviation.
each agent We therefore run our experiments several domains that are part of well-established benchmark packages. These include three domains from the PettingZoo package [38], three domains from the MeltingPot package [8]] and two-player variant of the Atari 2600 game Space Invaders. We describe these domains in more detail in Section in the appendix. We run particularly large number of baselines and ablations in the PettingZoo domains, and focus on the most salient comparisons in the MeltingPot and Atari domains within available computational resources. Pursuit is particularly well suited to showcase our algorithm, as it strietly requires multiple agents to cooperate order to generate any reward, and each cooperating agent needs to perform slightly different behavior (tagging the target from different direction).
5.3 Experimental Setup
In Pursuit, we train all agents concurrently using the same algorithm, for each of the algorithms listed. Note that only the pursuers agents in this domain, whereas the evaders move randomly. In the standard variants of Battle and Adversarial-Purres dueling DDQN, all agents both teams being trained together and independently. We then take the pre-trained agents of the opposing team (red team in Battle, predator team in Adversarial-Purruiie and use these to control the opposing team during main training of the blue respectively prey team. In this main training phase, only the blue / prey team agents are trained using each of the SUPER and benchmark algorithms, whereas the red predator team are controlled by the pre-trained policies with no further training. Figure shows final performance from the main training phase. In Battle
we first pre-train a set of agents using independent
In principle, SUPER could also used in shared-reward environments, but these present an additional credit assignment problem. While SUPER could potentially be combined with techniques for solving that problem such as QMIX (29). the additional complexity arising from this would make such experiments less meaningful.

---

Brvionment pursall
Ervioone kerdo
Enniormart - -ovoonarrn purs
-20
16
120
-00
800
-300
eome stpe suanter
maree days seardion
saper sdie gtschans
dees peranstoon
mariion
deen
eupee days Loaeatiion
stop you heres
appe po anramaaagg
deen
alor Puaete
Reped ship Jthuorgee
aayt eitrral
skap pettrratterr
edee
dean
Alpi Brn
Algo Frn
Alg Bho
Figure 2: Performance of SUPER SEIEER-dulling-DDD -DQN variants and baselines on all three PettingZoo domains. For Pursuit, performance is the total mean episode reward from all agents. For Battle and Adversarial-Pumes performance is the total mean episode reward from all agents the sharing team (blue team in Battle, prey team in Adversarial- Pursuutt). Shaded areas indicate one standard deviation.
and Adversarial-Puret we further tested a variant where the opposing team are co-evolving with the blue/ / prey team, which discuss in Section Cin the appendix.
5.4 Performance Evaluation
Figure performance of SUPER implemented on dueling DDON ("SUPER DDQN") compared to the control benchmarks discussed above. In addition, Figure 2 shows final performance in the three PettingZoo environments in a barplot for easier readability. In summary, we see that (quantile and gaussian) SUPER (red bars) clearly outperform baseline DDQN on all domains, as well as both SEAC and MADDPG baselines where we were able to run them. Particularly notable is jump in performance from DDQN (green) to quantile SUPER (leftmost red bars in Figure 2 in the PettingZoo domains. Performance of SUPER is even comparable to that of parameter sharing. "best case" fully centralized comparison benchmark. Aside from parameter sharing, SUPER wins or ties all pairwise comparisons against baselines. Results further hold for additional variations of environments with co-evolving opponents, when using SUPER with a plain (non-dueling, non-double) DQN algorithm, and across hyperparameter settings, which discuss in the appendix.
In more detail, we find that SUPER (red) consistently outperforms the baseline DQN/DDQN algorithm (green), often significantly. For instance in Pursuit SUPER-DDQN achieves over twice the reward of no-sharing DDQN at convergence, increasing from 181.4 (std.dev 4.1) to 454.5 (std.dev 5.9) for quantile SUPER-DDQN measured 800k training steps. In both Battle and enabled SUPER- SUaains for only one of the two teams each, and see that this significantly improved performance of the sharing team (as measured by sum of rewards of all agents in the team across each episode), especially mid-training. For instance in Battle, the blue team performance increase from -19.0 (std.dev 1 11.0) for no-sharing DDQN to 5.5 (std.dev 8.7) for quantile SUPER-DDOO at 150k timesteps. In Adversarial-Pursule the prey performance increased from -719.8 (std.dev 66.8) to -5063 dst 35.0) at 300k timesteps.
prrssarial-Pussuui
SUPER performs significantly better than SEAC (blue) and MADDPG (violet). MADDPG performed very poorly on all domains despite extensive attempts at hyperparameter tuning. SEAC shows some learning in Pursuit at 800k timesteps, but remains well below even baseline DDQN performance, again despite extensive hyperparameter tuning. We have also run experiments with SEAC in Pursuit for significantly longer (not shown in the figure), and saw that performance eventually catches up after around 8M timesteps, i.e. roughly ten-fold difference in speed of convergence. SUPER (red) significantly outperforms parameter-sharin to it in Battle, whereas parameter-shane sharing performs significantly better in Sivversarial-uussute Note that parameter sharing has often been observed to perform extremely well 40 39 consider this strong positive result.
DQN/DDQN (turquoise) in Pursuit and performs similar
We therefore
Finally, these results and conclusions show remarkable stability across settings, underlying algorithm, and hyperparameters. In Section C.2 in the appendix we show that similar results hold using SUPER
7

---

400
Patomarce
300
Perfomarce
300
200
200
100
Algorthm /ooiatinn
Bandwich
Figure Performance of quantile SUPER vs share-all and uniform random experience sharing in Pursuit at 800k timesteps.
Figure Performance of quantile SUPER with varying bandwidth in Pursuit at 1-2M timesteps.
on baseline DQN (non-dueling, non-double) algorithm. Section C.3 in the appendix we show results from SUPER on dueling DDQN on domains where the opposing team agents are co-evolving rather than pretrained. Section C.5 in the appendix we show that these improvements over baseline DDQN stable across wide range of hyperparameter choices. Relative performance results and conclusions all these cases largely mirror the ones presented here.
5.5 Ablations
We ran an ablation study to gain better understanding of the impact of two key features of our algorithm: selectivity (only sharing small fraction of experiences) and priority (sharing experiences with the highest td-error). To this end, we compared SUPER against two cut-down versions of the algorithm: One, we share all experiences indiscriminately. This is also equivalent to single "global" replay buffer shared between agents. Two, we share (a small fraction of) experiences, but select experiences uniformly random. Figure 3 shows the performance of these two ablations at end of training Pursuit and Battle. We see that both ablations perform significantly worse than SUPER, with neither providing significant performance uplift compared to baseline DDQN This shows that both selectivity as well as priority are necessary to provide a performance uplift, at least in some domains.
5.6 Bandwidth Sensitivity
In the Pursuit domain, we performed an analysis of the performance of SUPER-DDON for varying target bandwidths ranging from 0.0001 (sharing all experiences). Figure 4 shows the converged performance of SUPER-DDON with quantile-based experience selection in Pursuit. A label of "DQN" indicates that no sharing is taking place, i.c. decentralized DDQN; a label "all" indicates that all experiences are shared, without any prioritized selection. Numerical labels give different target bandwidths. Two things stand out to us: First, sharing all experiences indiscriminately does not result in increased performance. In fact, convergence, it results in slightly lower performance than no-sharing DDQN. Second, there is a clear peak of performance around a target bandwidth of 0.01 0.1, which also holds for gaussian and stochastic experience selection (we refer the reader to the appendix for more details). We conclude that sharing experiences selectively is crucial for learning to benefit from it.
5.7 Experience Selection
Gaussian experience selection performed similarly to the quantile selection we designed it to approxi- mate. Its actual used bandwidth was however much less responsive to target bandwidth than the other two variants. We believe this demonstrates that in principle approximating the actual distribution of
'The situation in Adversarial Pursuit is more nuanced and highly dependent on training time, but overall shows similar picture especially early in training.

---

td-errors using mean and standard deviation is feasible, but that more work is needed in determining the optimal value of in equation [ Stochastic experience selection (dotted red) performs similar or worse than both other variants, but generally still comparably or better than baseline DQN/DDQN.
6 Conclusion & Discussion
Conclusion We present selective multiagent PER, a selective experience- mechanism that can improve DQN-famil algorithms in multiagent settings. Conceptually, our approach is rooted in the same intuition that Prioritized Experience Replay is based on, which is that td-error is a useful approximation of how much an agent could learn from a particular experience. In addition, we introduce the second key design choice of selectivity, which allows semi-decentralized learning with small bandwidth, and drastically improves performance in some domains.
Experimental evaluation on DQN and dueling DDQN shows improved performance compared to fully decentralized training (as measured in sample efficiency and/or converged performance) across a range of hyperparameters of the underlying algorithm, and in multiple benchmark domains. We see most consistent performance improvements with SUPER and a target bandwidth of 0.01-0.1 I late in training, more consistent than indiscriminate experience sharing. Given that this effect appeared consistently across wide range of hyperparameters and multiple environments, as well as on both DQN and dueling DDQN, the SUPER approach may be useful as a general-purpose purpose multi-agen RL technique. Equally noteworthy a significantly improved performance early in training even at very low bandwidths. We consider this to be a potential advantage in future real-world applications of RL where sample efficiency and rapid adaption to new environments are crucial. SUPER consistently and significantly outperforms MADDPG and SEAC, and outperforms parameter sharing in Pursuit (but underperforms in Adversarial PPrsuitt, and shows equal performance in Battle).
Discussion Our selective experience approach improves performance of both DQN and dueling DDQN baselines, and does so across a range of environments and hyperparameters. It outperforms state-of-the-art multi-agent RL algorithms, in particular MADDPG and SEAC. The only pairwise comparison that SUPER loses is against parameter sharing in Adversarial Prersaaial-Pursuitt, in line with a common observation that in practice parameter sharing often outperforms sophisticated multi-agent RL algorithms. However, we note that parameter sharing is an entirely different, fully centralized training paradigm. Furthermore, parameter sharing is limited in its applicability, and does not work well if agents need to take on different roles or behavior to successfully cooperate. We see this in the Pursuit domain, where parameter sharing performs poorly, and SUPER outperforms it by large margin. The significantly higher performance than MADDPG and SEAC is somewhat expected given that baseline non-sharing DQN algorithms often show state-of-the-art performance in practice, especially with regard to sample efficiency.
It noteworthy that deterministic (aka "greedy") experience selection seems to perform slightly better than stochastic experience selection, while in PER the opposite is generally the case [30]. We have two hypotheses for why this is the case. One, we note that in PER, the motivation for stochastic prioritization is to avoid low-error experiences never being sampled (nor re-prioritized) in many draws from the buffer. On the other hand, in SUPER we only ever consider each experience once. Thus, in stochastic experience selection high-error experience through random chance is not shared on this one opportunity, it will never be seen by other agents, In sense, we may prefer deterministic experience selection in SUPER for the same reason we prefer stochastic selection in PER, which is to avoid missing out on potentially valuable experiences. Two, in all our current experiments we used (stochastic) PER when sampling training batches from the replay buffer of each agent. When using stochastic SUPER, each experience therefore must pass through two sampling steps before being shared and trained on by another agent. It is possible that this dilutes the probability of high-error experience being seen too much.
We would also like to point out slight subtlety in our theoretical motivation for SUPER: We use the sending agent's 's td-error as a proxy for the usefulness of an experience for the receiving agent. We believe that this is justified in symmetric settings. and our experimental results support this. However, we stress that this is merely a heuristic, and one which we do not expect to work entirely asymmetric domains. For future work, we would be interested to explore different experience selection heuristics. As an immediate generalization to a more theoretically grounded approach, we wonder if using the td-error of each (potential) receiving agent could extend SUPER to asymmetric settings, and if it
9

---

could further improve performance even in symmetric settings. While this would effectively be contralized-trainiin approach. if showed similar performance benefits as we have seen in symmetric settings for SUPER, it could nevertheless be promising avenue for further work. Beyond this, we would be interested to explore other heuristics for experience selection. For instance, we are curious if the sending agent could learn to approximate each receiver's s td-error locally, and thus retain the eecentralized-with with-communication training capability of our current approach. However, given that td-error is intrinsically linked to current policy and thus highly non-stationary, we expect there would be significant practical challenges to this.
In this current work, we focus on the DQN family of algorithms in this paper. In future work, we would like to explore SUPER in conjunction with other off- off-po RL algorithms such as SAC [22 2 and DDPG 32]. The interplay with experience sampling methods other than PER, such as HER would also be interesting. If the improvements we see in this work hold for other algorithms and domains as well, this could improve multi-agent RI. performance in many settings.
Finally, our approach is different from the "centralized training, decentralized execution" baselines we compare against in the sense that it does not require fully centralized training. Rather, it can be implemented in a decentralized fashion with communications channel between agents. We see that performance improvements scale down even to very low bandwidth, making this feasible even with limited bandwidth. We think of this scheme as "decentralized training with communication" and hope this might inspire other semi-decentralized algorithms. In addition to training, we note that such "decentralized with communication" approach could potentially be deployed during execution, agents keep learning. While this is beyond the scope of the current paper, in future work we would like to investigate if this could help when transferring agents to new domains, and particular with adjusting to sim-to-real gap. We envision that the type of collaborative, decentralized learning introduced here could have impact in future applications of autonomous agents ranging from disaster response to deep space exploration.
Acknowledgements
The project was sponsored, in part, by grant from the Cooperative AI Foundation. The content does not necessarily reflect the position or the policy of the Cooperative AI Foundation and no endorsement should be inferred.
References
[1] Sanjeevan Ahilan and Peter Dayan. Correcting experience replay for multi-agent communication. arXiv preprint arXiv:2010.01192, 2020.
[2] Stefano V Albrecht and Peter Stone. Autonomous agents modelling other agents: A compre- hensive survey and open problems. Artificial Intelligence, 258:66 95, 2018.
[3] Marcin Andrychowicz. Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
[4] Mare Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
[5] James Bergstra, Daniel Yamins, and David Cox. Making science of model search: Hyper- parameter optimization in hundreds of dimensions for vision architectures. In International conference on machine learning, pages 113-123. PMLR, 2013.
[6] Greg Brockman, Vicki Cheung, Ludwig Pettersson. Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint
540, 2016.
[7] Filippos Christianos, Lukas Schäfer, and Stefano Albrecht. Shared experience actor-critic for multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 33:10707-10717, 2020.
10

---

[8] Felipe Leno Da Silva and Anna Helena Reali Costa. A survey on transfer learning for multiagent reinforcement learning systems. Journal of Artificial Intelligence Research, 64:6455703, 2019.
[9] Jakob Foerster, Ioannis Alexandros Assael, Nando De Freitas, and Shimon Whiteson. Learning communicate with deep multi-agent reinforcement learning, Advances in neural information processing systems, 29, 2016.
[10] Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[11] Jayesh K Gupta, Maxim Egorov, and Mykel Kochenderfer. Cooperative multi-agent control using deep reinforcement learning. In International Conference on Autonomous Agents and Multiagent Systems, pages 66-83. Springer, 2017.
[12] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[13] Pablo Hernandez-Leaa, Bilal Kartal, and Matthew E Taylor. A survey and critique of multiagent deep reinforcement learning. Autonomous Agents and Mulli-Agent Systems, 33(6):750-797, 2019.
[14] Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dab- ney. Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Thirty-second AAAI conference on artificial intelligence, 2018.
[15] Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado Van Hasselt, and David Silver. Distributed prioritized experience replay. arXiv preprint arXiv:1803.00933 2018
[16] Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro Ortega, DJ Strouse, Joel Z Leibo, and Nando De Freitas. Social influence as intrinsic motivation for multi-agent deep reinforcement learning. In International conference on machine learning. pages 1040-3049 PMLR, 2019.
[17] Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. Multi-agent cooperation and the emergence of (natural) language. arXiv preprint arXiv:1612.07/88. 2016.
[18] Joel Z. Leibo, Edgar Dué nez Guzmán, Alexander Sasha Vezhnevets, John P. Agapiou, Peter Sunehag, Raphael Koster, Jayd Matyas, Charles Beattie, Igor Mordatch, and Thore Graepel. Scalable evaluation multi-agent reinforcement learning with melting pot. PMLR, 2021.
[19] Joel Z. Leibo. Vinicius Zambaldi, Marc Lanctot, Janusz Marecki, and Thore Graepel. Multi- agent reinforcement learning in sequential social dilemmas. In Proceedings of the 16th Interna- tional Conference Autonomous Agents and Multiagent Systems (AAMAS 2017), AAMAS '17, page 464-473, Richland, SC, 2017. International Foundation for Autonomous Agents and Multiagent Systems.
[20] Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph Gonzalez, Michael I. Jordan, and Ion Stoica. RLlib: Abstractions for distributed reinforcement learning. In International Conference on Machine Learning (ICML), 2018
[21] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Yoshua Bengio and Yann LeCun, editors, 4th International Conference on Learning Repre- sentations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings. 2016.
[22] Long-Ji Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine learning, 8:293-321, 1992.

---

[23] Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb. OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-compettive enviromments. Advances in neural information processing systems, 30, 2017.
[24] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wienstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. In NIPS Deep Learning Workshop. 2013.
[25] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-35 -333, 2015.
[26] Peng Peng, Ying Wen, Yaodong Yang. Quan Yuan, Zhenkun Tang, Haitao Long, and Jun Wang. Multiagent biddreetionally-coordiatee nets: Emergence of human-level coordination in learning play starcraft combat games. arXiv preprint arXiv:1700 10069, 2017.
[27] Emanuele Pesce and Giovanni Montana. Improving coordination in multi-agent deep reinforce- ment learning through memory-driven communication. CoRR, 0bs//19010555 00387,7 2019.
[28] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonie value function factorisation for deep multi agent reinforcement learning. In International conference on machine learning, pages 4295 4304. PMLR, 2018
[29] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder de Witt, Gregory Farquhar, Jakob N Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Learning Research, 21, 2020.
[30] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. In Yoshua Bengio and Yann LeCun, editors, 4th International Conference on Learning Repre- sentations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
[31] Jürgen Schmidhuber, Jieyu Zhao, and Nicol N Schraudolph. Reinforcement learning with self-modifying policies. In Learning to learn, pages 293-309. Springer, 1998.
[32] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference on machine learning, pages 387-395. PMLR, 2014.
[33] Sainbayar Sukhbaatar, Rob Fergus, et al. Learning multiagent communication with backpropa- gation. Advances in neural information processing systems, 29, 2016
[34] Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[35] Csaba Szepesvári. Algorithms for reinforcement learning. Synthesis lectures artificial intelligence and machine learning, 4(1):1-103, 2010.
[36] Ardi Tampuu, Tambet Matiisnn, Dorian Kodelja, Ilya Kuzovkin, Kristjan Korjus, Juhan Aru, Jaan Aru, and Raul Vicente. Multiagent cooperation and competition with deep reinforcement learning. PloS one, 12(4))e0172395, 2017.
[37] Ming Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proceed- ings of the tenth international conference on machine learning, pages 330-337, 1993.
[38] K Terry, Benjamin Black, Nathaniel Grammel. Mario Jayakumar, Ananth Hari, Ryan Sulivan, Luis Santos, Rodrigo Perez, Caroline Horsch, Clemens Dieffendahl, Niall L Williams, Yashas Lokesh, Ryan Sullivan, and Praveen Ravi. Pettingzoo: Gym for multi-agent reinforcement learning. arXiv preprint arXiv:2009. 2020.
[39] Justin K Terry, Nathaniel Grammel, Ananth Hari, Luis Santos, and Benjamin Black. Revisiting parameter sharing in multi-agent deep reinforcement learning. arXiv preprint arXiv:2005. 1.0000,..622. 13625, 2020.
12

---

[40] Justin K Terry, Nathaniel Grammel, Sanghyun Son, and Benjamin Black. Parameter sharing for heterogeneous agents in aggent reinforcement learning. arXiv e-prints, pages arXiv-2005, 2020.
[41] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
[42] Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International conference machine learning, pages 1995-2003, PMLR, 2016.
[43] Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3)279-292.) 1992.
[44] Annie Xie, Dylan P Losey, Ryan Tolsma, Chelsea Finn, and Dorsa Sadigh. Learning latent representations to influence multi-agent interaction. arXiv preprint arXiv:2011.066 2020.
[45] Lianmin Zheng, Jiacheng Yang, Han Cai, Weiman Zhang, Jun Wang, and Yong Yu. Magent: A many-agent reinforcement learning platform for artificial collective intelligence. CoRR, abs/1712.00600,, 2017.
[46] Changxi Zhu, Mehdi Dastani, and Shihan Wang. A survey of multi-agent reinforcement learning with communication. arXiv preprint arXiv:2203.088) 2022.
13

---

Figure 5: Pursuit Environment
A Algorithm Details
Algorithm 1 SUPER algorithm for DQN
for each training iteration do
Collect batch of experiences b (DQN)
for each agent do
Insert b; into buffer, (DQN)
end for
for each agent i do
Select by S b; experiences to share" (SUPER)
for each agent j / do
Insert be; into buffer, (SUPER)
end for end for for each agent do
Sample a train batch by from buffer; (DQN)
Learn train batch b; (DQN)
end for end for
See section "Experience Selection"
Algorithm shows full pseudocode listing for SUPER on DQN.
B Experiment Domains
B.1 PettingZoo
SISL: Pursuit is a semi-cooperative environment, where group of pursuers has to capture group of evaders in grid-world with an obstacle. The evaders (blue) move randomly, while the pursuers (red) are controlled by RL agents. If group of two more agents fully surround an evader, they each receive a reward, and the evader removed from the environment. The episode ends when all evaders have been captured, or after 500 steps, whichever is earlier. Pursuers also receive a (very small) reward for being adjacent to an evader (even the evader is not fully surrounded), and (small) negative reward each timestep, to incentivize them to complete episodes early. We use 8 pursuers and 30 evaders.
MAgent: Battle is a semi-adversarial environment, where two groups of opposing teams are battling against each other. An agent is rewarded 0.2 points for attacking agents the opposite team, and 5 points if the other agent is killed. All agents start with 10 health points (HP) and lose 2 HP in each attack received, while regaining 0.1 HP every turn. Once killed, an agent is removed from the environment. An episode ends when all agents from one team are killed. The action space, of size 21 identical for all agents, with (8) options to attack, (12) to move and one option to nothing. Since no additional reward is given for collaborating with other agents in the same team, it is considered to be more challenging to form collaboration between agents this environment. We use map of size 18 18 and 6 agents per team.
14

---

MAgent: Adversarial Pursuit is predator- prey environment, with two types of agents, prey and predator. The predators navigate through obstacles in the map with the purpose of tagging the prey. An agent in the predators team is rewarded 1 point for tagging a prey, while prey is rewarded -1 when being tagged by predator. Unlike the Battle environment, prey agents are not removed from the game when being tagged. Note that prey agents are provided only with a negative or zero reward (when manage to avoid attacks), and their aim is thus to evade predator agents. We use 8 prey agents, and 4 predator agents.
B.2
MeltingPot
We chose subset of three domains ("substrates") from the MeltingPot package [8]. Cooperative Mining is a collaborative resource harvesting game, where cooperation between players allows them mine more valuable type of resource than individual action does. Rationalizable Coordination is a coordination game embedded in a larger rich environment. Similarly Stag Hunt is as stag hunt game payoff matrix embedded in larger game world.
B.3
Atari
We also ran experiments on a version of the Atari 2600 game Space Invaders [4. For this, we created two different variants. In the main variant, we ran the game in native two-player mode. By default, observations from this are not strictly anonymous, as each player is identified using a fixed color. We thus made two changes make observations strictly anonymous: First added colored bar to the bottom of the observation that indicates which player is being controlled. Then, randomly shuffle each episode which agent controls which player in the game. This way, both policies learn to control each of the two player colors, and are able to make sense of observations from each other.
In a second variant, we took two separate instances of the game running a single-player mode and linked them to create two-player environment. This provides additional "sanity check" to ensure that the modifications we made to the primary variant did not influence results in our favor.
C Further Experimental Results
C.1 Additional Results DDQN
In addition to the final performance shown in the main text, show in Table numerical results from all experiments.
Table I: Performance in all three environments, taken at 800k timesteps (Pursuit), 300k timesteps (Battle), 300k timesteps (Adv. Pursuit). Numbers in parentheses indicate standard deviation. Highest performance in each environment is printed in bold.
Algorithm Env
pursuit
battle
adversarial pursult
ddqn
181.43 (+-4.16)
9.46 (+-0.86)
-719.78 (+-66.82)
ddgn_paramshare
113.00 (+-121,11)
15.15 (+-8.64)
-268.78 (+-60.11)
addnn_shareal
148.27 (+-9.22)
9.25 (+-6.60)
-568.91 (+-40.82)
maddpg
-342.24 (+-4.12)
(+-0.0
-1071.71 (+-8.56)
32.51 (+-70.27)
-2.73 (+-0.03)
-993.00 (+-120.22)
super super_ddqn gaussian
373.63 (+-9.15)
16.28 (+-3.42)
-48.377 (+-15.79)
super_ super_ddqn_on _uqnnqquntill
454.56 (+-5.89)
17.05 (+-2.74)
-50..29 (+-35.03)
super_ dddn_stochasttic stochastic
266.42 (+-85.52)
7.43 (+-5.64)
-582.59 (+-15.20)
C.2 DQN and Dueling DDQN
For all of the DDQN and SUPER-DDON variants discussed in Section we consider also variants based on standard DQN. F shows results from these experiments.
15

---

EnMrrnmmnt
Envitomment
300
-200
Aigarifim
Parfumaros
sspar den quardde
100
siper (on, pin__gminsinnn
-009
siohhwllc
dgn
800
dq,, sshrral
001, pan_paramshore
200
meddpg
-400
-1200
limestepi lotal
timestepe_ lotal
Figure 6: Performance of SUPER-DON variants with target bandwidth 0.1 Pursuit and Adversarial- Pursuit. For Pursuit, performance is the total mean episode reward from all agents. For Adversarial- Pursuit, performance is the total mean episode reward from all agents in the prey team. Shaded areas indicate one standard deviation.
Erivicomenn
Environeent adversarial pursutt pursuil
30
200
20
10
200
Algoriless
quartie
super, chip, geeasion
super. clig stuchastic
ddan
-00
-900
-40
-00
-1200
0
60000 100000 100000 200000 200000 300000
100000 200000 300000 400000 G00000 GO0000 $mestops, total
Uimeslope, 3otal
Figure 7: Performance SUPER SUPER-dueling and Adversarial-Pursuii with co-evolving opponents. Performance is the total mean episode reward from all agents in the sharing team (blue team in Battle, prey team Adversarial-Pursuii Shaded areas indicate one standard deviation.
variants with target bandwidth 0.1 on all Battle
C.3 Co-Evolving Teams
In Battle and Satisarril-Puusuii we further show a variant where the opposing team are co-evolving with the blue prey team. In this variant, all agents start from randomly initialized policy and train concurrently, using DDQN algorithm. However, only the blue / prey team share experiences using the SUPER mechanism. We only do this for the DDQN baseline well discriminate and share-all SUPER variants. This is in part because some of the other baseline algorithms do not support concurrently training opposing agents with different algorithm available implementations; and in part because we consider this variant more relevant to real-world scenarios where fully centralized training may be feasible. We aim to show here how sharing even a small number of experiences changes the learning dynamics versus to non- sharihning opponents. Figure 7 shows this variant.
C.4 Further Ablations
In addition to the ablations presented in the main text, we include here additional results from the Battle environment in Figure 9 Results are broadly similar to the results in the main text, with notably bad performance for uniform random experience sharing.
C.5 Stability Across Hyperparameters
Figure 8 shows performance of no-sharing DDQN and SUPER-DDON for different hyperparam- eters. As we can see, SUPER-DDON outperforms no-sharing DDQN consistently across all the hyperparameter settings considered.
16

---

250
250
X03
205
100
200
100
403
Aggriive
Arw-Puruul jram311 fiaar.00
Anw-Satte 24h-1 frainddd
pwr.kssP.ruttemm11.1 Brand
www.te/Aaattetttt Ena-d
24
300
15
-10
200
-003
102
+83
+70
DON
CINOO
20V NPCROON
WPEE-DO
DCH
Algorttins
Algeniiim
Alpastin
Algertiis
Agortts
Figure 8: Performance of DDQN and SUPER-DDOO DDQN (gaussian experience selection, target band- width 0.1) for differing hyperparameter settings of the underlying DDQN algorithm. Top: Different learning rates and rollout fragment lengths in Pursuit. Bottom: Different exploration settings in Pur- suit and co-evolving variants of Batle and Eiversarial-Pursute Hyperparameters otherwise identical to those used in Figure [ Performance measured at 1M timesteps in Pursuit, 300k timesteps in Battle, 400k timesteps in Adversarial Pursuit..
Environment
400
300
200
100
Algarthmm
Abblation
Figure 9: Performance of quantile SUPER vs share-all and uniform random experience sharing in Pursuit 800k timesteps.
D Additional Analysis of Bandwidth Sensitivity
We present here a more detailed analysis of bandwidth sensitivity of SUPER-DDQV in the three experience selection modes we discuss in the main text. Figure TT shows the mean performance across five seeds for gaussian (left), quantile (middle) and stochastic (right) experience selection, at 1-2M timesteps (top) and 250k timesteps (bottom). We can see that at 1-2M timesteps and target bandwidth of 0.1, all three experience selection criteria perform similary. One thing that stands out is that stochastic selection has much lower performance at other target bandwidths, and also much less performance uplift compared to no-sharing DDQN at 250k timesteps at any bandwidth. Gaussian experience selection appears to be less sensitive to target bandwidth, but upon closer analysis we found that it also was much less responsive in terms of how much actual bandwidth used at different
17

---

SJPER_ gousson
SUPER riUdo Q quardi
SPPER micdo skchasso
Targot Sandwidh
TurssBandreth
Target Bundwedh
KJPER, mode geuscan
KAPER FROO qUAITe
MPEER mode shrdwwrts
208
108
0306 Target Bandwitt
6:011 3.99 Targel Bondawithh
6:999
Targel Bandwith
Figure 10: Performance SUPER with different experience selection and varying bandwidth in Pursuit 1-2M timesteps (top) and at 250k timesteps (bottom).
(left) shows the actual bandwidth used by each selection criterion at different target bandwidths. We can see that quantile and stochastic experience hit their target bandwidth very well in general What stands out, however, is that gaussian selection vastly overshoots the target bandwidth at lower settings, never going significantly below 0.01 actual bandwidth.
settings. Figure II
What is fairer comparison therefore is to look at performance versus actual bandwidth used for each of the approaches, which do in Figure 11 (middle, 1-2M timesteps, and right, 250k timesteps). For these figures, we did the following: First, for each experience selection approach and target band- width, computed the mean performance and mean actual bandwidth across the five seeds. Then, for each experience selection mode, we plotted these (meanactualbandwei) (one for each target bandwidth) in line plot 6 The result gives us a rough estimate of how each approach's 's performance varies with actual bandwidth used. We see again that stochastic selection shows worse performance than quantile low bandwidths, and early in training. We also that gaussian selection very closely approximates quantile selection. Notice that gaussian selection never hits an exact actual bandwidth of 0.1, and so we cannot tell from these data if would match quantile selection's 's performance at its peak. However, we can see that at the actual bandwidths that gaussian selection does hit, it shows very similar performance to quantile selection. As stated the main text, our interpretation of this is that using mean and variance to approximate the exact distribution of absolute td-errors reasonable approximation, but that we might need to be more clever in selecting in equation! 3
meanperformance)
'Quantile selection overshoots at le-4 (0,0001) target bandwidth and is closer to le-3 actual bandwidth these experiments with window size of 1500 (1.5e+3).
usage. which attribute rounding emror, and a quantile less than a single element is not well-defined. "Because each data point now has variance in both z- and y-directions, it is not possible to draw error bars for these.
18

---

2001
0:011
Arbeel Bantwrdh
Bundents
Figure 11: Left: Actual bandwidth used (fraction of experiences shared) different target bandwidths.
Middle, right: Performance compared to actual bandwidth used at 1-2M and 250k timesteps.
E Experiment Hyperparameters & Details
We performed all experiments using the open-source library RLlib 20). Experiments Figure D 6 were ran using RLlib version 2.0.0; experiments in other figures were using version 1.13.0.
Environments used are from PettingZoo [38), including SISL (T] and MAgent [45]. The SUPER algorithm was implemented by modifying RLlib's standard DQN algorithm to perform the SUPER experience sharing between rollout and training. Table 2 lists all the algorithm hyperparameters and environment settings we used for all the experiments. Experiments in the "Stability across hyperparameters" " section had hyperparameters set to those listed in Table exxcept those specified in Figure Any parameters not listed were left at their default values. Hyperparameters were tuned using a grid search; some of the combinations tested are also discussed in the "Stability across hyperparameters" section. For DQN, DDQN and their SUPER variants, we found hyperparameters using a grid search on independent DDQN in each environment, and then used those hyperparameters for all DON/DDQN and SUPER variants in that environment. For all other algorithms performed grid search for each algorithm in each environment. For MADDPG we attempted further optimization using the Python HyperOpt package (5), however yielding no significant improvement over our manual grid search. For SEAC, we performed grid search in each environment, but found no better hyperparameters than the default. We found a CNN network architecture using manual experi- mentation in each environment, and then used this architecture for all algorithms except MADDPG where we used a fully connected net for technical reasons. We tested all other algorithms using both the hand-tuned CNN as well as fully connected network, and found that the latter performed significantly worse, but still reasonable (and in particular significantly better than MADDPG using the same fully connected network, on all domains).
All experiments were repeated with three seeds. All plots show the mean and standard deviation of these seeds at each point in training. For technical reasons, individual experiment runs did not always report data at identical intervals. For instance, one run might report data when it had sampled 5 51000 environment timesteps, and another run might report at 53000 environment timesteps. In order to still be able report a meaningful mean and standard deviation across repeated runs, we rounded down the timesteps reported to the nearest k steps, i.e. taking both the data above to represent each run't performance 50000 steps. We set in Pursuit, 6000 timesteps in the other two domains). Where a run reported more than once in 10000 step interval, we took the mean its reports to represent that run's performance in the interval. Mean and standard deviation were calculated across this mean performance for each of the five seeds. To increase legibility, we applied smoothing to Figures Mand 6
the target reporting interval in each domain (8000 timesteps
using an exponential window with 0.3 for Pursuit, 0.1 for Battle, and 0.25 for Adversarial -attecssaril-Puusuitt and 0.3 for Atari.
19

---

No smoothing was found to be necessary for MeltingPot results. This removes some noise from the reported performance, but does change the relative ordering any two curves.
Implementation & Reproducibility
All source code is included in the appendix and will be made available publication under open-source license. We refer the reader to the included README file, which contains instructions to recreate the experiments discussed in this paper.
Table 2: Hyperparameter Configuration Table SISL: Pursuit
Environment Parameters
HyperParameters
Value
HyperParameters
Value
max cycles
500
x/y sizes
16/16
shared reward
False
num evaders
30
horizon
500
n catch
2
surrounded tag reward constrained window
True
num agents(pursuers)
8
0.01
urgency reward
-0.1
1.0
catch rewards
5
obs range
7
CNN Network
CNN layers Strides
[32.64,64]
Kernel size
[2,2]
SUPER MON/DDQN
learning rate
0.00016
final exploration epsilon
0.001
batch size prioritized replay alpha
nframework
torch
0.6
prioritized replay eps
1c-06
dueling
True
target network update freq
1000
buffer size initial exploration epsilon
120000
rollout fragment length
4
0.1
MADDPG
Actor Ir NN(FC)
0.00025
Critic Ir tau actor feature reg
0.00025
[64,64]
0.015
framework
tensorflow
0.001
SEAC
learning rate batch size framework
3e-4
adam eps
0.001
5
use gae
False
torch
gae lambda
0.95
entropy coef max grad norm recurrent policy
0.01
value loss coef
0.5
0.5
use proper time limits
True
False
use linear Ir decay
False
seac coef num steps
1.0
num processes
5
20

---

Table 3: Hyperparameter Configuration Table- MAgent: Battle
Environment Parameters
HyperParameters
Value
HyperParameters
Value
minimap mode
False
step reward
-0.005
Num blue agents
Num red agents
dead penalty
-0.1
attack penalty
-0.1
attack opponent reward
0.2
max cycles
1000
extra features
False
map size
18
CNN Network
CNN layers
[32.64.64]
Kernel size
[2,2]
Strides
SUPER DQN /DQN
learning rate
le-4
batch size
framework
torch
prioritized replay_alpha
0.6
prioritized replay eps dueling
le-06
horizon
1000
True
target network update_ freq
1200
rollout fragment length
5
buffer size
90000
initial exploration epsilon
0.1
final exploration epsilon
0.001
MADDPG
Actor Ir NN(FC)
0.00025
Critic Ir
0.00025
[64,64]
tau actor feature reg
0.015
framework
tensorflow
0.001
SEAC
learning rate batch size
3e-4
adam eps
0.001
use gae
False
framework
torch
gae lambda
0.95
entropy coef
0.01
value loss coef
0.5
max grad norm recurrent policy
0.5
use proper time limits
True
False
use linear Ir decay
False
seac coef num steps
1.0
num processes
4
5
21

---

Table 4: Hyperparameter Configuration Table MAgent: Adversarial Pursuit
Environment Parameters
HyperParameters Number predators minimap mode
Value
HyperParameters
Value
4
Number preys
8
False
tag penalty
-0.2
max cycles
500
extra features
False
map size
Policy Network
CNN layers Strides
[32,64,64]
Kernel size
[2,2]
SUPER DQN /DDON
learning rate
le-4
batch size
32
framework
torch
prioritized replay alpha
0.6
prioritized replay eps
le-06
horizon
500
dueling
True
target network update freq
1200
buffer size initial exploration epsilon
90000
rollout fragment length
5
0.1
final exploration epsilon
0.001
MADDPG
Actor Ir NN(FC)
0.00025
Critic lr
0.00025
[64,64]
tau actor feature reg
0.015
framework
tensorflow
0.001
SEAC
learning rate batch size framework
3e-4
adam eps
0.001
5
use gae
False
torch
gae lambda
0.95
entropy coef
0.01
value loss coef
0.5
max grad norm recurrent policy
0.5
use proper time limits
True
False
use linear Ir decay
False
seac coef
1.0
num processes
num steps
5
22

---

Table 5: Hyperparameter Configuration Table MeltingPot
Policy Network
CNN layers Strides
[16,128] 8,1
Kernel size
[8,8], [sprite_x, sprite_y]
SUPER DDQN
learning rate
0.00016
batch size
32
framework
prioritized replay alpha torch
0.6
prioritized replay eps dueling
le-06
horizon
500
True
target network update podate_freq
1000
buffer size initial exploration epsilon
30000
rollout fragment length
4
0.1
final exploration epsilon
0.001
Table 6: Hyperparameter Configuration Table Atari
Policy Network
Model Configuration
RLlib Default
SUPER DDQN
learning rate
0.0000625
batch size
32
framework
torch True
adam_ adam__ppilon
0.00015
dueling
target network update_ __rq
8000
buffer size
100000
rollout fragment length
initial exploration epsilon
0.1
final exploration epsilon
0.01
23


Reviews:
[{"review_id": "7U3HKjLQj7", "reviewer": "Reviewer_aJ1o", "summary": "This paper introduces a semi-centralized multi-agent reinforcement learning algorithm named SUPER, by incorporating limited communication in the training phase. \nWith minor modifications, agents are able to achieve better performance than the baseline algorithms by sharing a limited amount of experiences.\n", "questions": "1. Intuitively, it seems that sharing only works for isomorphic agents. What about heterogeneous agents?\n2. What if the agents are not provided with their adequate indivudal rewards?\n3. What if we incorporate importance sampling on the shared experience?", "limitations": "Yes, they have.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper is easy to understand.\n2. The proposed algorithm is easy to implement and demonstrates superior performance compared to the baselines.\n3. The idea of sharing important experiences seems good.", "weaknesses": "1. The difference between SUPER and previous works is not clearly clarified. \nAs mentioned in the paper that one of the previous works use importance sampling to integrate experiences, it might be more persuasive to compare the performance with experiences.\n2. The evaluated environments might be too simple in multi-agent reinforcement learning area.\nAs far as my knowledge, it's more common to observe only a team reward during training, while individual rewards are rare situations.  \n3. The experimental comparison with multi-agent baselines might be unfair.\nIt seems that in DQN and SUPER experiments agents receive their individual rewards, while MADDPG, as far as I know, takes team rewards, which misleads agents.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer aJ1o,\n\nWe would like to thank you for your helpful comments and suggestions, and would like to offer a few clarifications:\n* Difference to previous work: To clarify, while SEAC uses importance sampling, this is only used to “translate” action probabilities for off-policy experiences. However, there is no sampling going on in the sense that only a subset of experiences are shared - all experiences are shared in SEAC. A key innovation in our work is the selectivity and sharing only a small number of important experiences. It is this difference that enables us to develop a decentralized algorithm that needs only limited communication, a training paradigm that is much more broadly applicable than previous centralized algorithms.\n* Training environments: To our knowledge, it is not the case that shared-reward environments are more common than individual-reward environments, both types of environments just provide different challenges (and so are perhaps common in different sub-communities). Shared-reward primarily leads to a credit assignment problem (see e.g. literature on value decomposition networks), whereas individual-reward leads to questions about incentives and cooperation. A large number of seminal works in multi-agent RL has focused on individual-reward environments, e.g. work on game-playing including Poker, the breakthrough AlphaGo AI, and the two recent Science papers on Stratego and Diplomacy; papers on learning game-theoretic equilibria (e.g. V-learning for Nash, recent work on Stackelberg equilibria); work on reward-shaping; sequential social dilemmas; as well as a broad line of literature on opponent shaping (e.g. LOLA, M-FOS, etc). Shared-reward environments are less appropriate for our work, as our algorithm does not aim to solve the credit assignment problem. \n* Comparisons: This is a misunderstanding, MADDPG does take individual rewards. The original MADDPG paper used a mix of cooperative and competitive environments. In the experiments in our paper, MADDPG uses individual rewards for a fair comparison.\n\nTo answer your questions:\n1. This depends on the type of heterogeneity: Heterogeneous behavior is something SUPER can work with, and we already see this in the Pursuit domain. Heterogeneous rewards would also be relatively easy to incorporate, by estimating the recipient’s reward using their own Q-function (this way agents would be able to learn from each other about environment dynamics, even if they have different reward functions). For agents that perceive and act in entirely different spaces, learning from each others' experiences is more difficult, of course. However, it might be possible to do something similar using some form of higher-level semantics.\n2. As above, shared-reward environments are primarily a credit assignment problem, which is not what we are trying to solve. That said, we would like to draw attention to the fact that the Pursuit domain already does partial shared reward: Multiple agents have to surround a prey, and then all those agents share the reward for doing so. Our experiments show that SUPER works well in this setting. Additionally, it might be possible to combine SUPER with credit assignment techniques such as value-decomposition networks; however we feel that trying to do this in the same paper as the base SUPER algorithm would detract from the key messages we would like to convey.\n3. Importance sampling is only needed to translate off-policy experiences for on-policy policy-gradient methods. It is not necessary to do this for DQN algorithms (nor even possible, as the “log-pi” term that needs correcting in the policy gradient theorem is simply not present in the q-learning updates). \n\nWe appreciate your point regarding environments we use in our experiments. Based on this feedback, we have run experiments in several more complex domains from the Melting Pot package, and include results in the PDF shared with all reviewers. We see a significant performance improvement for SUPER compared to baseline DDQN across all the domains we tested, in some cases even greater in magnitude than in our original experiments.\n\nWe thank you again for your constructive feedback, and would welcome further discussion in case you have any remaining concerns.\n"}, {"review_id": "jKM6enXW8s", "reviewer": "Reviewer_8wH6", "summary": "The paper proposes SUPER, a communication-based cooperative multi-agent reinforcement learning method with semi-decentralized training. In SUPER, each agent relays its highest-td-error experiences to the other agents, who insert them directly into their replay buffer, which they use for learning. The author believes that the selective exchange of trajectories between agents can improve the performance of basic off-policy algorithms in cooperative multi-agent tasks.", "questions": "1. Does SUPER perform better than other classical communication methods? And How about their bandwidth costs?\n2. The author claims that SUPER is a method based on semi-decentralized training, so how much less is its bandwidth cost compared to the centralized training algorithm?\n3. What about the performance of the SUPER variant of other off-policy methods such as DDPG? Does it outperform MADDPG in given environments?\n4. Trying to evaluate the performance of SUPER in more domains will greatly improve the convincingness of experiments.", "limitations": "Authors discussed the limitations of their work in detail in the last section.", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well organized and easy to follow.\n2. The motivation is clear, that is, agents selectively relay some important experiences to other agents to help them learn faster. \n3. The basic algorithm of SUPER is independent off-policy methods. Therefore, it relaxes the assumption of centralized training. \n4. In the experimental section, authors carry out many ablation experiments, which show the superior performance of SUPER in given scenarios.\n5. Key details of the implement are sufficiently well-described and key resources are available.", "weaknesses": "1. SUPER is a communication-based method. However, the baselines in the experimental section do not include a communication-related algorithm.\n2. There are too few experimental environments to demonstrate the generalization of SUPER in different domains. In addition, the action spaces of environments in this paper are all discrete .\n3. Authors only compared the performance between the vanilla DQN-like algorithms and their SUPER variants, which means the SUPER framework was not applied to other off-policy methods such as DDPG.\n4. Authors did not actually quantify the bandwidth cost brought by the communication between agents, which is an important measure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 8wH6,\n\nWe would like to thank you for your constructive feedback and suggestions. Regarding your questions:\n\n1. Could you clarify what algorithms you have in mind here? We are not aware of any other work that uses communication as part of its learning algorithm in a similar way; the closest would be SEAC, which we already compare against. Note that what we do is very different from algorithms that learn to communicate (e.g. work on emergent communication, or the “social influence” paper on sequential social dilemmas); we are not sure what a meaningful comparison to such papers could look like.\n\n2. We explore this in Section 5.6 and particularly in Figure 3. To be clear, centralized training would use a bandwidth of (at least) 1. For SUPER, we see highest performance at a bandwidth of 0.1, or a 10-fold reduction; performance at 0.01 is still almost as high, at a 100-fold reduction compared to centralized training. Even at 0.001, or a 1000-fold reduction, we see significantly higher performance than centralized baselines.\n\n3. Regarding DDPG, the single-agent version of DDPG works exclusively for continuous-control applications. For the MADDPG paper, this was modified using a Gumbel-Softmax relaxation to make it compatible with discrete action spaces, but this is not standard and recent research has shown evidence that this might be problematic (see e.g. Tilbury et al, Revisiting the Gumbel-Softmax in MADDPG, arXiv:2302.11793). We therefore do not believe that benchmarks using DDPG on discrete domains would provide a reliable baseline to compare against. On the other hand, for continuous actions, we are not aware of any relevant widely-used multi-agent benchmark domains that use continuous control, nor of any published tuned baselines. Available implementations of MADDPG and SEAC also seem to focus on discrete actions and it is unclear to us how well they support continuous control (if at all). Given this, it would be difficult to perform meaningful comparisons.\n\n   * To clarify, we mention off-policy algorithms as a suggestion for future work; we did not intend to claim that our paper applies SUPER to every off-policy algorithm. We will move the sentence \"Beyond DQN and its variants, SUPER could in principle apply to any off-policy RL algorithm.\" to Discussion and clearly label it as a future direction. Applying a new method to one algorithm is relatively common. For instance, the MADDPG paper makes a general point about actor-critic approaches but focuses on one particular algorithm, and SEAC could be applied to any actor-critic algorithm, but is developed in detail only for A2C.\n\n   * However, note that we do in fact run benchmarks on two different off-policy algorithms, DQN and dueling DDQN, the former in the supplementary material. While for historical reasons DQN-type algorithms tend to be named as variants of DQN (unlike policy-gradient based methods), these are very different algorithms: Double DQN and dueling DQN are as different from DQN as actor-critic PPO and A2C are from plain policy gradient. (Double DQN can be seen as a form of actor-critic architecture, and dueling DQN learns an advantage function like A2C does.) In other words, we already show SUPER on two substantially different off-policy algorithms.\n\n4. Following your feedback, we have run additional experiments on a number of environments from the Melting Pot package. We consistently see a significant performance uplift of SUPER compared to baseline DDQN. In fact, the effect is even larger in some of these additional domains than it was in our original experiments.\n\nWe thank you again for your constructive feedback, and would welcome further discussion in case you have any remaining concerns.\n"}, {"review_id": "pVhMFFLf2D", "reviewer": "Reviewer_mQNR", "summary": "The paper proposes a heuristic for sharing the experience between agents in Multi-Agent Reinforcement Learning (MARL) setting. It is a modification of Prioritised Experience Replay (PER) method, where the agents only share experiences, which td-error exceeds a certain threshold (paper proposes several ways to choose the threshold). The experimental evaluation show that the proposed approach delivers better performance against a set of reasonable baselines.", "questions": "How could the system be upgraded to move beyond \"anonymous\" MARL?", "limitations": "The authors are honest about limitations of their method and it is appreciated.", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper is clear and well positioned within the related work\n- The approach is simple, elegant and easy to reproduce. \n- Experiments are clean and well presented\n- Experimental results are convincing\n\nOverall I believe that the paper can have a moderate, yet significant impact on MARL. Due to it's simplicity the method could be easily deployed and (if the results generalise beyond the 3 domains presented) provide a very decent \"bang for the buck\" in terms of performance vs compute/implementation complexity.", "weaknesses": "- Somewhat limited choice of environments. For example, what happens in environments that require division of labour, say overcooked? I would love to see the performance of the proposed method on a broader benchmark, for example MeltingPot [1]. I understand that compute is a limitation and not all of the environments are appropriate (anonymous), but maybe a subset, which is and requires division of labour would be a great test.\n- Only applicable to anonymous MARL (same reward, actions and observations). This is a big limitation, especially the reward.\n- The contribution 4. (line 43) is confusing and would suggest authors remove it. The \"emergent communication\" is a very different problem and I don't think discussing it here is helpful. \n\nOverall, there are some clear limitations to the approach (listed above). My main concern is whether the improvements generalise beyond the presented domains, especially to the domains where division of labour / diversity of policy is required for the population of agents to be successful. \n\n1. Leibo, J. Z., Dueñez-Guzman, E. A., Vezhnevets, A., Agapiou, J. P., Sunehag, P., Koster, R., ... & Graepel, T. (2021, July). Scalable evaluation of multi-agent reinforcement learning with melting pot. In International conference on machine learning (pp. 6187-6199). PMLR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer mQNR,\n\nWe would like to thank you for your insightful comments and suggestions. We would like to clarify a few details:\n\n* Division of labor: \n   * Regarding the Overcooked domain in particular, while we agree that this is an exciting domain, the sparsity of reward make this an extremely challenging domain. The Melting Pot paper itself shows that baseline RL fails to show any non-zero returns during training, i.e. when not paired with a scenario agent. (Figure 4 in Leibo et al, “collaborative cooking” subfigures, S-P column, where all entries are 0.) In initial experiments we ran, we similarly failed to get any significant learning using DDQN in this domain. Our understanding is that the Overcooked domain is more aimed at testing behavior vis-a-vis different paired partners (e.g. few-shot coordination, generalization between partner behaviors or adapting to human partners), and less suitable for testing agents learning concurrently from scratch. \n   * However, note that we already see division of labor in the Pursuit domain. Here multiple agents have to surround a prey from different directions. Our hypothesis is that part of why SUPER outperforms parameter sharing on this domain is that the SUPER agents are able to learn diversified strategies. \n* Anonymous MARL: Note that reward is not a strict limitation, as one could estimate the recipient’s reward using the recipient’s Q-function, and still benefit from the environment dynamics seen in the sender’s experience. See also our response to reviewer aJ1o’s first question. Furthermore, we would like to suggest that anonymity is not as big a limitation as one would think at first glance. In particular, the real world is anonymous! What we have in mind as a prototypical application of our work is a group of robots exploring a new environment or task in the real world together. For instance, our work could lead to self-driving cars sharing a small subset of the real-world data they collect with one another. We believe such real-world applications are manifold enough to warrant a paper focus on them (and it is our decentralized-with-communication paradigm that makes this approaches especially well-suited to future real-world deployment).\n* Contrib. 4: Thank you for this useful feedback, upon re-reading this we agree. We will remove this. \n\nRegarding your question, as per above, our approach already demonstrates positive results in a domain with some division of labor, and it would be possible to generalize it to domains with heterogeneous rewards. \n\nIn addition, we thank you for your suggestion of the Melting Pot project. We have run experiments on a number of environments from the Melting Pot package, and include results in the PDF shared with all reviewers. We see a significant performance improvement for SUPER compared to baseline DDQN across all the domains we tested, in some cases even greater in magnitude than in our original experiments.\n\nFinally, we would like to thank you for your comment on easy deployment of our method, which we also value highly. We designed our code to drop into the standard RLlib package with minimal modifications, and if accepted plan to submit a pull request to include SUPER in the official RLlib package, so as to make it easily accessible to the wider community.\n\nWe thank you again for your constructive feedback, and would welcome further discussion in case you have any remaining concerns."}, {"review_id": "kyWuktUm4H", "reviewer": "Reviewer_3mbW", "summary": "The paper proposes a modified prioritized experience replay algorithm where agents share a minimal amount of high-quality samples to improve learning.", "questions": "No specific question, feel free to clarify misunderstandings.", "limitations": "no foreseeable negative societal impact.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The proposed method is very simple (therefore easy to implement and reproduce), and widely applicable for many cooperative RL environments, yet the method seems powerful and to significantly improve learning speed.\n-Potential to big impact to the general RL community, and wide applicability such as being added to standard RL frameworks,\n- Paper is clear and easy to understand.", "weaknesses": "- The main weakness of the paper is that the evaluation domain is very simple for an exclusively-empirical paper. I would expect an evaluation in a domain of ~Starcraft complexity to be included.\n\n- I don't understand why the performance of the algorithms was shown just for a couple of arbitrary training points. You should have instead added a line graph where the reader could see the performance of each algorithm/baseline over time.\n\n- The paper touches on it briefly but it would be good to have a deeper discussion of on which multiagent scenarios this idea could be used and how. Is any typpe of \"observation translation\" needed for the agents to share the experiences, and what happens if the agents are heterogenous.\n\n- Ideally, I would like to have seem a comparison against some Transfer learning methods that use approximately the same amount of transferred data. Many applicable algorithms could be found in the survey:\n\n Silva, Felipe Leno, and Anna Helena Reali Costa. \"A survey on transfer learning for multiagent reinforcement learning systems.\" Journal of Artificial Intelligence Research 64 (2019): 645-703.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 3mbW,\n\nWe would like to thank you for your very helpful comments and suggestions. We would like to respond to some of your comments:\n\n* Starcraft: While we agree that Starcraft is an interesting domain, it was designed specifically as a challenge domain for credit assignment in shared reward settings, which is not the problem our work aims to solve. This would therefore not be a useful benchmark domain for our work. However, based on your feedback, we have run additional experiments on several more complex domains from the Melting Pot package, and include results in the PDF shared with all reviewers. We see a strong performance uplift for SUPER compared to baseline DDQN in all tested environments, in some cases even greater than in our earlier experiments.\n* Line vs bar plots: We had line plots in earlier working versions of our paper, but received clear feedback from readers that they were difficult to decipher due to the large number of algorithms we run in each domain, and that a simpler plot or table would communicate our results more effectively. However, we still include the full learning curves in the supplementary material should you wish to have a look. We will also attempt to further improve the formatting of these, or perhaps to split them into different figures for comparison to different baselines. If we find a way to make them more readable, we will move them to the main text.\n* Heterogenous agents: No translation is needed for homogenous agents / anonymous environments. We feel that anonymity is not a very strong limitation, as in particular the real world and the laws of physics are anonymous, i.e. our work could be applied to real-world robotics applications. Note that our idea does work for heterogenous behavior, and we see this in the Pursuit domain. For non-anonymous settings, our idea could extend in a relatively straightforward manner to settings with different reward functions, by estimating the recipient’s reward using their Q-function (while still learning about environment dynamics from the received experience). Settings where agents observe and act in entirely different ways are more difficult; it might be possible to develop an approach similar to ours using more abstract semantics, but this would be outside the scope of our current work. We will clarify this in the paper text.\n* Thank you for your suggestion regarding transfer learning. We will study the suggested paper carefully to see if any direct comparisons are possible, and we will in any case include a discussion of transfer learning and citation of the survey paper in the next revision of our paper. \n\nWe thank you again for your constructive feedback, and would welcome further discussion in case you have any remaining concerns."}]

Author rebuttal:
Dear Reviewers,

In addition to individual responses, we are uploading a PDF with figures from testing SUPER in additional domains, as this was a common point raised in the reviews. Following the suggestion from reviewer mQNR, we used the Melting Pot package for this, and compared SUPER-DDQN against a DDQN baseline on several environments from this package. These are more complex domains (e.g. using image observations) than the environments used in our previous experiments. In all of them, we see a clear performance uplift when using SUPER, in some cases even greater in relative magnitude than in our original experiments. 

Due to the short time, we have only been able to run SUPER and a DDQN baseline. However, note that this is the main comparison of interest, as all other baselines use fully centralized training. We believe the new results demonstrate that the performance uplift from using SUPER holds across a range of domains.


Meta-review: This paper describes a way of sharing experiences between agents using replay buffers. Reviewers appreciated the simplicity of the algorithm and the many different environments where it was evaluated, including the larger environments like Melting Pot and Atari which were added during the rebuttal period. There was some discussion of whether the comparison baselines were appropriate with two reviewers questioning this, but I found the authors' responses to be convincing. I also noted that another reviewer commented positively about the experiments.

Hopefully the authors will be able to add the new results they obtained on melting pot and atari during the rebuttal period to the final version of the paper.
Decision: Accept (poster)