SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Matthias Gerstgrasser
matthias@seas.harvard.edu
Tom Danino
Sarah Keren
The Taub Faculty of Computer Science
tom.danino@campus.technion.ac.il
sarahk@cs.technion.ac.il
We present a novel multi-agent RL approach, Selective Multi-Agent Prioritized Experience Relay, in which agents share with other agents a limited number of transitions they observe during training. The intuition behind this is that even a small number of relevant experiences from other agents could help each agent learn. Unlike many other multi-agent RL algorithms, this approach allows for largely decentralized training, requiring only a limited communication channel between agents. We show that our approach outperforms baseline no-sharing decentralized training and state-of-the-art multi-agent RL algorithms. Further, sharing only a small number of highly relevant experiences outperforms sharing all experiences between agents, and the performance uplift from selective experience sharing is robust across a range of hyperparameters and DQN variants.
Multi-Agent Reinforcement Learning (RL) is often considered a hard problem: The environment dynamics and returns depend on the joint actions of all agents, leading to significant variance and non-stationarity in the experiences of each individual agent. Much recent work [29, 23] in multi-agent RL has focused on mitigating the impact of these. Our work views multi-agent RL more positively, by treating the presence of other agents can also be an asset that can be leveraged. In particular, we show that groups of agents can collaboratively explore the environment more quickly than individually.
To this end, we present a novel multi-agent RL approach that allows agents to share a small number of experiences with other agents. The intuition is that if one agent discovers something important in the environment, then sharing this with the other agents should help them learn faster. However, it is crucial that only important experiences are shared - we will see that sharing all experiences indiscriminately will not improve learning. To this end, we make two crucial design choices: Selectivity and priority. Selectivity means we only share a small fraction of experiences. Priority is inspired by a well-established technique in single-agent RL, prioritized experience replay (PER) [30]. With PER an off-policy algorithm such as DQN [24] will sample experiences not uniformly, but proportionally to "how far off" the current policy's predictions are in each state, formally the temporal difference (td) error. We use this same metric to prioritize which experiences to share with other agents.
We dub the resulting multiagent RL approach "Selective Multi-Agent Prioritized Experience Relay" or "SUPER". In this, agents independently use a DQN algorithm to learn, but with a twist: Each agent relays its highest-td-error experiences to the other agents, who insert them directly into their replay buffer, which they use for learning. This approach has several advantages:
1. It consistently leads to faster learning and higher eventual performance, across hyperparameter settings.
2. Unlike many "centralized training, decentralized execution" approaches, the SUPER learning paradigm allows for (semi-) decentralized training, requiring only a limited bandwidth communication channel between agents.
3. The paradigm is agnostic to the underlying decentralized training algorithm, and can enhance many existing DQN algorithms. SUPER can be used together with PER, or without PER.<sup>1</sup>
In addition to the specific algorithm we develop, this work also introduces the paradigm of "decentralized training with communication". This is a 'middle ground between established approaches of decentralized and centralized training (including "centralized training, decentralized execution").
In the remainder of the paper, we will discuss related literature and technical preliminaries; introduce our novel algorithm in detail; describe experimental evaluation and results; and suggest avenues for future work.
Multi-Agent RL approaches in the literature can be broadly categorized according to the degree of awareness of each learning agent of the other agents in the system [37]. On one end of this spectrum is independent or decentralized learning where multiple learning agents (in the same environment) are unaware or agnostic to the presence of other agents in the system [36, 19]. From the perspective of each agent, learning regards the other agents as part of the non-stationary environment. At the opposing end of the spectrum, in centralized control a single policy controls all agents. In between these two, a number of related strands of research have emerged.
Approaches nearer the decentralized end of the spectrum often use communication between the agents [46, 9, 13]. Several forms of cooperative communication have been formulated, by which agents can communicate various types of messages, either to all agents or to specific agent groups through dedicated channels [17, 33, 26, 27]. While communication among agents could help with coordination, training emergent communication protocols also remains a challenging problem; recent empirical results underscore the difficulty of learning meaningful emergent communication protocols, even when relying on centralized training [16]. Related to this, and often used in conjunction, is modeling other agents [2, 16, 10, 44], which equips agents with some model of other agent's behavior.
This in turn is related to a number of recent approaches using centralized training but decentralized execution. A prevailing paradigm within this line of work assumes a training stage during which a shared network (such as a critic) can be accessed by all agents to learn decentralised (locally-executable) agent policies [23, 1, 28, 10, 29]. These approaches successfully reduce variance during training, e.g. through a shared critic accounting for other agents' behavior, but rely on joint observations and actions of all agents. Within this paradigm, and perhaps closest related to our own work, [7] introduces an approach in which agents share (all of their) experiences with other agents. While this is based on an on-policy actor-critic algorithm, it uses importance sampling to incorporate the off-policy data from other agents, and the authors show that this can lead to improved performance in sparse-reward settings. Our work also relies on sharing experiences among agents, but crucially relies on selectively sharing only some experiences.
A separate but related body of work is on transfer learning [8]. While this type of work is not directly comparable, conceptually it uses a similar idea of utilizing one agent's learning to help another agent.
Off-Policy RL The approach we present in this paper relies intrinsically on off-policy RL algorithms. Most notable in this class is DQN [24], which achieved human-level performance on a wide variety of Atari 2600 games. Various improvements have been made to this algorithm since then, including
<sup>1</sup>Note that while the name "SUPER" pays homage to PER, SUPER is not related to PER other than using the same heuristic for relevance of experiences.
<sup>2</sup>Interestingly, in the appendix to the arXiv version of that paper, the authors state that experience sharing in DQN did not improve performance in their experiments, in stark contrast with the result we will present in this paper. We believe this may be because their attempts shared experiences indiscriminately, compared to our selective sharing approach. We will see that sharing all experiences does not improve performance as much or as consistently as selective sharing.
dueling DQN [42], Double DQN (DDQN) [41], Rainbow [14] and Ape-X [15]. Prioritized Experience Replay [30] improves the performance of many of these, and is closely related to our own work. A variant for continuous action spaces is DDPG [32]. Off-policy actor-critic algorithms [12, 21] have also been developed, in part to extend the paradigm to continuous control domains.
Reinforcement learning (RL) deals with sequential decision-making in unknown environments [34]. Multi-Agent Reinforcement Learning extends RL to multi-agent settings. A common model is a Markov game, or stochastic game defined as a tuple  $\langle S, \mathcal{A}, \mathcal{R}, \mathcal{T}, \gamma \rangle$  with states  $S$ , joint actions  $\mathcal{A} = \{A^i\}_{i=1}^n$  as a collection of action sets  $A^i$ , one for each of the  $n$  agents,  $\mathcal{R} = \{R^i\}_{i=1}^n$  as a collection of reward functions  $R^i$  defining the reward  $r^i(a_t, s_t)$  that each agent receives when the joint action  $a_t \in \mathcal{A}$  is performed at state  $s_t$ , and  $\mathcal{T}$  as the probability distribution over next states when a joint action is performed. In the partially observable case, the definition also includes a joint observation function, defining the observation for each agent at each state. In this framework, at each time step an agent has an experience  $e = < S_t, A_t, R_{t+1}S_{t+1}>$ , where the agent performs action  $A_t$  at state  $S_t$  after which reward  $R_{t+1}$  is received and the next state is  $S_{t+1}$ . We focus here on decentralized execution settings in which each agent follows its own individual policy  $\pi_i$  and seeks to maximize its discounted accumulated return. The algorithm we propose in this paper further requires that the Markov game is "anonymous", meaning all agents have the same action and observation spaces and the environment reacts identically to their actions. This is needed so that experiences from one agent are meaningful to another agent as-is. Notice however that this does not imply that it is necessarily optimal for all agents to adopt identical behavior.
Among the variety of RL solution approaches [35, 34], we focus here on value-based methods that use state and action estimates to find optimal policies. Such methods typically use a value function  $V_{\pi}(s)$  to represent the expected value of following policy  $\pi$  from state  $s$  onward, and a  $Q$ -function  $Q_{\pi}(s,a)$ , to represent for a given policy  $\pi$  the expected rewards for performing action  $a$  in a given state  $s$  and following  $\pi$  thereafter.
Q-learning [43] is a temporal difference (td) method that considers the difference in  $Q$  values between two consecutive time steps. Formally, the update rule is  $Q(S_{t},A_{t})\gets Q(S_{t},A_{t}) + \alpha [R_{t + 1} + \gamma \max_{a^{\prime}}Q(S_{t + 1},a^{\prime}) - Q(S_{t},A_{t}))]$ , where,  $\alpha$  is the learning rate or step size setting the amount by which values are updated during training. The learned action-value function,  $Q$ , directly approximates  $q^{*}$ , the optimal action-value function. During training, in order to guarantee convergence to an optimal policy, Q-learning typically uses an  $\epsilon$ -greedy selection approach, according to which the best action is chosen with probability  $1 - \epsilon$ , and a random action, otherwise.
Given an experience  $e_t$ , the td-error represents the difference between the Q-value estimate and actual reward gained in the transition and the discounted value estimate of the next best actions. Formally,
$$
\operatorname {t d} \left(e _ {t}\right) = \left| R + \gamma \max  _ {a ^ {\prime}} Q \left(S ^ {\prime}, a ^ {\prime}\right) - Q (S, A) \right| \tag {1}
$$
In the vanilla version of Q-learning Q-values are stored in a table, which is impractical in many real-world problem due to large state space size. In deep Q-Learning (DQN) [25], the Q-value table is replaced by a function approximator typically modeled using a neural network such that  $Q(s,a,\theta)\approx Q^{*}(s,a)$ , where  $\theta$  denotes the neural network parameters.
Replay Buffer (RB) and Prioritized RB: An important component in many off-policy RL implementations is a replay buffer [22], which stores past experiences. Evidence shows the replay buffer to stabilize training of the value function for DQN [24, 25] and to reduce the amount of experiences required for an RL-agent to complete the learning process and achieve convergence [30].
Initial approaches that used a replay buffer, uniformly sampled experiences from the buffer. However, some transitions are more effective for the learning process of an RL agents than others [31]. Prioritized Experience Replay (PER) [30] explores the idea that replaying and learning from some transitions, rather than others, can enhance the learning process. PER suggests replacing the standard sampling method, where transitions are replayed according to the frequency they were collected from the environment, with a td-error based method, where transitions are sampled according to the value of their td-error.
As a further extension, stochastic prioritization balances between strictly greedy prioritization and uniform random sampling. Hence, the probability of sampling transition  $i$  is defined as:
$$
P (i) = \frac {p _ {i} ^ {\alpha}}{\sum_ {k} p _ {k} ^ {\alpha}} \tag {2}
$$
where  $p_i > 0$  is the priority associated to transition  $i$  and  $\alpha$  determines the weight of the priority  $(\alpha = 0$  is uniform sampling). The value of  $p_i$  is determined according to the magnitude of the td-error, such that  $p_i = |\delta_i| + \epsilon$ ,  $\delta_i$  is the td-error and  $\epsilon$  is a small positive constant that guarantees that transitions for which the td-error is zero have a non-zero probability of being sampled from the buffer.
4 SUPER: Selective Multi-Agent Prioritized Experience Relay
Our approach is rooted in the same intuition as PER: that not all experiences are equally relevant. We use this insight to help agents learn by sharing between them only a (small) number of their most relevant experiences. Our approach builds on standard DQN algorithms, and adds this experience sharing mechanism between collecting experiences and performing gradient updates, in each iteration of the algorithm:
1. (DQN) Collect a rollout of experiences, and insert each agent's experiences into their own replay buffer.
2. (SUPER) Each agent shares their most relevant experiences, which are inserted into all the other agents' replay buffers.
3. (DQN) Each agent samples a minibatch of experiences from their own replay buffer, and performs gradient descent on it.
Steps 1 and 3 are standard DQN; we merely add an additional step between collecting experiences and learning on them. As a corollary, this same approach works for any standard variants of steps 1 and 3, such asueling DQN [42], DDQN [41], Rainbow [14] and other DQN improvements. Algorithm 1 in the appendix gives a more detailed listing of this algorithm. Notice that the only interaction between the agents training algorithms is in the experience sharing step. This means that the algorithm can easily be implemented in a decentralized manner with a (limited) communications channel.
4.1 Experience Selection
We describe here three variants of the SUPER algorithm, that differ in how they select experiences to be shared.
Deterministic Quantile experience selection The learning algorithm keeps a list  $l$  of the (absolute) td-errors of its last  $k$  experiences ( $k = 1500$  by default). For a configured bandwidth  $\beta$  and a new experience  $e_t$ , the agent shares the experience if its absolute td-error  $|\mathrm{td}(e_t)|$  is at least as large as the  $k*\beta$ -largest absolute td-error in  $l$ . In other words, the agent aims to share the top  $\beta$ -quantile of its experiences, where the quantile is calculated over a sliding window of recent experiences.
$$
| \operatorname {t d} (e _ {t}) | \geq \operatorname {q u a n t i l e} _ {\beta} \left(\left\{e _ {t ^ {\prime}} \right\} _ {t ^ {\prime} = t - k} ^ {t}\right)
$$
Deterministic Gaussian experience selection In this, the learning algorithm calculates the mean  $\mu$  and variance  $\sigma^2$  of the (absolute) td-errors of the  $k$  most recent experiences ( $k = 1500$  by default). It then shares an experience  $e_t$  if
$$
\left| \operatorname {t d} \left(e _ {t}\right) \right| \geq \mu + c \cdot \sigma^ {2} \tag {3}
$$
where  $c$  is a constant chosen such that  $1 - \operatorname{cdf}_{\mathcal{N}}(c) = \beta$ . In other words, we use the  $c$ -quantile of a normal distribution with the (sample) mean and variance of most recent experiences. We include and benchmark this variant for two reasons. One, intuitively, we might want to be more sensitive to clusters of outliers, where using a quantile of the actual data might include only part of the cluster, while a Gaussian model might lead to the entire cluster being included. Two, mean and variance could be computed iteratively without keeping a buffer of recent td-errors, and thereby reducing memory requirements. We aim to benchmark if this approximation impacts performance.
Stochastic weighted experience selection Finally, and most closely related to classical single-agent PER, this variant shares each experience with a probability that's proportional to its absolute td-error. In PER, given a train batch size  $b$ , we sample  $b$  transitions from the replay buffer without replacement, weighted by each transition's td-error. In SUPER, we similarly aim to sample a  $\beta$  fraction of experiences, weighted by their td-errors. However, in order to be able to sample transitions online, we calculate for each experience individually a probability that approximates samplingwithout-replacement in expectation. Formally, taking  $p_i = |\mathrm{td}(e_i)|$ , we broadcast experience  $e_t$  with probability
$$
p = \min  \left(1, \beta \cdot \frac {p _ {i} ^ {\alpha}}{\sum_ {k} p _ {k} ^ {\alpha}}\right)
$$
similarly to equation 2, and taking the sum over a sliding window over recent experiences. It is easy to see that if  $\beta = 1 / \text{batchsize}$ , this is equivalent to sampling a single experience, weighted by td-error, from the sliding window. For larger bandwidth  $\beta$ , this approximates sampling multiple experiences without replacement: If none of the  $\beta \cdot \frac{p_i^\alpha}{\sum_k p_k^\alpha}$  terms are greater than 1, this is exact. If we have to truncate any of these terms, we slightly undershoot the desired bandwidth.
In our current experiments, we share experiences and update all quantiles, means and variances once per sample batch, for convenience and performance reasons; However, we could do both online, i.e. after every sampled transition, in real-world distributed deployments. We do not expect this to make a significant difference.
We evaluate the SUPER approach on a number of multiagent benchmark domains.
5.1 Algorithm and control benchmarks
Baseline: DDQN As a baseline, we use a fully independent dueling DDQN algorithm. The algorithm samples a sequence of experiences using joint actions from all the agents' policies, and then inserts each agent's observation-action-reward-observation transitions into that agent's replay buffer. Each agent's policy is periodically trained using a sample from its own replay buffer, sampled using PER. We refer to this baseline as simply "DDQN" in the remainder of this section and in figures. In Section C in the appendix we also discuss a variant based on vanilla DQN.
SUPER-DDQN We implement SUPER on top of the DDQN baseline. Whenever a batch of experiences is sampled, each agent shares only its most relevant experiences (according to one of the criteria described in the previous section) which are then inserted into all other agents' replay buffers. As above, we run the SUPER experience sharing on whole sample batches. All DQN hyperparameters are unchanged - the only difference from the baseline is the addition of experience sharing between experience collection and learning. This allows for controlled experiments with like-for-like comparisons.
Parameter-Sharing DDQN In parameter-sharing, all agents share the same policy parameters. Note that this is different from joint control, in which a single policy controls all agents simultaneously; In parameter sharing, each agent is controlled independently by a copy of the policy. Parameter sharing has often been found to perform very well in benchmarks, but also strictly requires a centralized learner [40, 39, 11]. We therefore do not expect SUPER to outperform parameter sharing, but include it for completeness and as a "best case" fully centralized comparison.
Multi-Agent Baselines We further compare against standard multi-agent RL algorithms, specifically MADDPG [23] and SEAC [7]. Like parameter-sharing, these are considered "centralized training, decentralized execution" approaches.
5.2 Environments
As discussed in the Preliminaries (Section 3), SUPER applies to environments that are anonymous. We are furthermore particularly interested in environments that have a separate reward signal for
Figure 1: Performance of SUPER-dueling-DDQN variants with target bandwidth 0.1 on all domains. For team settings, performance is the total reward of all agents in the sharing team; for all other domains performance is the total reward of all agents. Shaded areas indicate one standard deviation.
each agent. $^3$  We therefore run our experiments on several domains that are part of well-established benchmark packages. These include three domains from the PettingZoo package [38], three domains from the MeltingPot package [18], and a two-player variant of the Atari 2600 game Space Invaders. We describe these domains in more detail in Section B in the appendix. We run a particularly large number of baselines and ablations in the PettingZoo domains, and focus on the most salient comparisons in the MeltingPot and Atari domains within available computational resources. Pursuit is particularly well suited to showcase our algorithm, as it strictly requires multiple agents to cooperate in order to generate any reward, and each cooperating agent needs to perform slightly different behavior (tagging the target from a different direction).
In Pursuit, we train all agents concurrently using the same algorithm, for each of the algorithms listed. Note that only the pursuers are agents in this domain, whereas the evaders move randomly. In the standard variants of Battle and Adversarial-Pursuit, we first pre-train a set of agents using independent dueling DDQN, all agents on both teams being trained together and independently. We then take the pre-trained agents of the opposing team (red team in Battle, predator team in Adversarial-Pursuit), and use these to control the opposing team during main training of the blue respectively prey team. In this main training phase, only the blue / prey team agents are trained using each of the SUPER and benchmark algorithms, whereas the red / predator team are controlled by the pre-trained policies with no further training. Figure 1 shows final performance from the main training phase. In Battle
In principle, SUPER could also be used in shared-reward environments, but these present an additional credit assignment problem. While SUPER could potentially be combined with techniques for solving that problem such as QMIX [29], the additional complexity arising from this would make such experiments less meaningful.
Figure 2: Performance of SUPER-dueling-DDQN variants and baselines on all three PettingZoo domains. For Pursuit, performance is the total mean episode reward from all agents. For Battle and Adversarial-Pursuit, performance is the total mean episode reward from all agents in the sharing team (blue team in Battle, prey team in Adversarial-Pursuit). Shaded areas indicate one standard deviation.
and Adversarial-Pursuit we further tested a variant where the opposing team are co-evolving with the blue / prey team, which we discuss in Section C in the appendix.