Evolutionary Reinforcement Learning for Sample-Efficient Multiagent Coordination
Authors: Shauharda Khadka
Somdeb Majumdar
Santiago Miret
Stephen McAleer
Kagan Tumer
Venue:  (2020)
Keywords: reinforcement learning
multiagent
neuroevolution

Abstract: Many cooperative multiagent reinforcement learning environments provide agents with a sparse team-based reward as well as a dense agent-specific reward that incentivizes learning basic skills. Training policies solely on the team-based reward is often difficult due to its sparsity. Also, relying solely on the agent-specific reward is sub-optimal because it usually does not capture the team coordination objective. A common approach is to use reward shaping to construct a proxy reward by combining the individual rewards. However, this requires manual tuning for each environment. We introduce Multiagent Evolutionary Reinforcement Learning (MERL), a split-level training platform that handles the two objectives separately through two optimization processes. An evolutionary algorithm maximizes the sparse team-based objective through neuroevolution on a population of teams. Concurrently, a gradient-based optimizer trains policies to only maximize the dense agent-specific rewards. The gradient-based policies are periodically added to the evolutionary population as a way of information transfer between the two optimization processes. This enables the evolutionary algorithm to use skills learned via the agent-specific rewards toward optimizing the global objective. Results demonstrate that MERL significantly outperforms state-of-the-art methods such as MADDPG on a number of difficult coordination benchmarks. 

Under review as a conference paper at ICLR 2020
EVOLUTIONARY REINFORCEMENT LEARNING FOR MULTIAGENT COORDINATION
Anonymous authors Paper under double-blind review
ABSTRACT
Many cooperative multiagent reinforcement learning environments provide agents with a sparse team-based reward, as well as a dense agent-specific reward that incentivizes learning basic skills. Training policies solely on the team-based reward is often difficult due to its sparsity. Also, relying solely on the agent-specific reward is sub-optimal because it usually does not capture the team coordination objective. A common approach is to use reward shaping to construct a proxy reward by combining the individual rewards. However, this requires manual tuning for each environment. We introduce Multiagent Evolutionary Reinforcement Learning (MERL), split-level training platform that handles the two objectives separately through two optimization processes. An evolutionary algorithm maximizes the sparse team-based objective through ncuroevolution on a population of teams. Concurrently, a gradient- based optimizer trains policies to only maximize the dense agent-specific rewards. The gradient-based policies are periodically added to the evolutionary population as way of information transfer between the two optimization processes. This enables the evolutionary algorithm to use skills learned via the agent-specific rewards toward optimizing the global objective. Results demonstrate that MERL significantly outperforms state-of of-the-art methods, such as MADDPG, on number of difficult coordination benchmarks.
INTRODUCTION
Cooperative multiagent reinforcement learning (MARL) studies how multiple agents can learn to coordinate team toward maximizing global objective. Cooperative MARL has been applied to many real world applications such as air traffic control (Tumer and Agogino| (007), multi-robot coordination (Sheng et al. 2006; Yliniemi et al., 2014), communication and language (Lazaridou jet al. 2016 Mordatch and Abbeel 2018), and autonomous driving "Shalev-Shward et al. 2016))
Many such environments endow agents with team reward that reflects the team's 's coordination objective, as well as an agent-specific local reward that rewards basic skills. For instance, in soccer, dense local rewards could capture agent-specific skills such as passing, dribbling and running. The agents must then coordinate when and where to use these skills in order to optimize the team objective, which is winning the game. Usually, the agent-speeific reward is dense and easy learn from, while the team reward is sparse and requires the cooperation of all or most agents.
Having each agent directly optimize the team reward and ignore the agent-specific reward usually fails or is sample-inefin -neffleeeeffiiienttt for complex tasks due to the sparsity of the team reward. Conversely, having each agent directly optimize the agent-specifie reward also fails because does not capture the team's 's objective, even with state of the art multiagent RI. algorithms such as MADDPG (Lowe et al. 2017).
One solution to this problem is to use reward shaping, where extensive domain knowledge about the task used to create a proxy reward function (Rahmattalabi et al. 2016). Constructing this proxy reward function is difficult in complex environments, and is domain-dependenn. Apart from requiring domain knowledge and manual tuning, this approach also poses risks of changing the underlying problem itself (Ng al.] 1999). Simple approaches to creating proxy reward via linear combinations of the two objectives also fail to solve or generalize to complex coordination tasks (Devlin et al.
2011 Williamson al. 2009

---

Under review as a conference paper at ICLR 2020
In this paper, introduce Multiagent Evolutionary Reinforcement Learning (MERL), state-of- tre-art algorithm for cooperative MARL that does not require reward shaping. MERL is a split-level training platform that combines gradient- based and gradient-free optimization. The gradient-free optimizer is an evolutionary algorithm that maximizes the team objective through neuroevolution. The gradient-based optimizer is policy gradient algorithm that maximizes each agent's 's dense, local rewards. These gradient-basee based policies are periodically copied into the evolutionary population. The Iwo processes operate concurrently and share information through a shared replay buffer.
A key strength of MERL is that it is a general method which does not require domain-specific reward shaping. This is because MERL optimizes the team objective directly while simultaneously leveraging agent-specific rewards to learn basic skills. We test MERL in number of multiagent coordination benchmarks. Results demonstrate that MERL significantly outperforms state-of-the-ar methods such as MADDPG, while using the same observations and reward functions. We also demonstrate that MERL scales gracefully to increasing complexity of coordination objectives where MADDPG and its variants fail to learn entirely.
2 BACKGROUND AND RELATED WORK
Markoy Games: A standard reinforcement learning (RL) setting is often formalized as a Markov Decision Process (MDP) and consists of an agent interacting with an environment over a finite number of discrete time steps. This formulation can be extended to multiagent systems in the form of partially observable Markov games (Littman 1994 Lowe al. 2017). An N-agent Markov game is defined by global state of the world, S, and a set of N observations {O:) and N actions {A.} corresponding to the N agents. At each time step 4, each agent observes its corresponding observation Of and maps it to action A using its policy T.
Each agent receives a scalar reward rf based on the global state S, and joint action of the team. The world then transitions to the next state S(+1 which produces a new set of observations {O.]. The process continues until terminal state is reached. R₂ = Σn= r'r is the total return for agent with discount factor 7 E (0, 1]. Each agent aims to maximize its expected return.
TD3: Policy gradient (PG) methods frame the goal of maximizing the expected return as the minimization of loss function. A widely used PG method for continuous, high- digeesiimnsi action spaces is DDPG (Lillicrap et al. 2015). Recently, (Fujimoto et al.) 2018) extended DDPG to Twin Delayed DDPG (TD3), addressing its well-known overestimation problem. TD3 is the state-of-the-art, off-policy algorithm for model-free DRL in continuous action spaces.
TD3 uses an actor-critic architecture (Sutton and Barto, 1998) maintaining deterministic policy (actor) π S + A, and two distinct critics Q S x A + R,, Each critic independently approximates the actor's action-value function Q A separate copy of the actor and critics are kept target networks for stability and are updated periodically. A noisy version of the actor is used to explore the environment during training. The actor is trained using noisy version of the sampled policy gradient computed by backpropagation through the combined actor-critic networks. This mitigates overfitting of the deterministic policy by smoothing the policy gradient updates.
Evolutionary Reinforcement Learning (ERL) is hybrid algorithm that combines Evolutionary Algorithms (EAs) (Floreano et al., 22008; Lüders al. 2017 Fogell [006) SSearr et al. 1993). with policy gradient methods (Khadka and Tumer 2018). Instead of discarding the data generated during standard EA rollout, ERL stores this data central replay buffer shared with the policy gradient's own rollouts- thereby increasing the diversity of the data available for the policy gradient learners. Since the EA directly optimizes for episode-wide return, it biases exploration towards states with higher long-term returns. The policy gradient algorithm which learns using this state distribution inherits this implicit bias towards long-term optimization. Concurrently, the actor trained by the policy gradient algorithm is inserted into the evolutionary population allowing the EA to benefit from the fast gradient-based learning.
Related Work: Lowe al. (0117) introduced MADDPG which tackled the inherent non-stationarity of a multiagent learning environment by leveraging a critic which had full access to the joint state and action during training. Foerster et al. (2018b) utilized a similar setup with centralized critic across agents to tackle StarCraft micromanagemen tasks, An algorithm that could explicitly model other agents' learning investigated in Foerster al. (20T8a). However, all these approaches rely
2

---

Under review as a conference paper at ICLR 2020
on dense agent reward that properly captures the team objective. Methods to solve for these types of agent-specific reward functions were investigated in Li al. (2012) but were limited to tasks with strong simulators where tree-based planning could be used.
closely related work to MERL is Liu et al. 2019) where Population- Based Training (PBT) (Jader berg et al.] 2017) is used optimize the relative importance between collection of dense, shaped rewards automatically during training. This can be interpreted as a singular central reward function constructed by scalarizing a collection of reward signals where the scalarization coefficients are adaptively learned during training. In contrast, MERL optimizes its reward functions independently with information transfer across them facilitated through shared replay buffers and policy migra- tion directly. This form of information transfer through shared replay buffer has been explored extensively in recent literature (Colas al. 2018 Khadka et al. 2019).
MULTIAGENT EVOLUTIONARY REINFORCEMENT LEARNING
MERL leverages both agent-specific and team objectives through hybrid algorithm that combines gradient-free and gradient- optimization. The gradient- free optimizer an evolutionary algo- rithm that maximizes the team objective through neuroevolution. The gradient-based optimizer trains policies to maximize agent-specific rewards. These gradient-based policies are periodically added to the evolutionary population and participate in evolution. This enables the evolutionary algorithm to use agent-specific skills learned by training on the agent-specific rewards toward optimizing the team objective without needing to resort reward shaping.
Algorithm 1 Multiagent Evolutionary Reinforcement Learning
1: Initialize a population of l multi-head teams pop,, each with weights 0t initialized randomly
2: Initialize a shared critic Q with weights @2
3: Initialize an ensemble of N empty cyclic replay buffers R* for each agent
4: Define a white Gaussian noise generator W, random number generator r() E [0,
5: for generation 1, do
6: for team çop, do
7:
9. R Rollout (τ, R, noise=None. {
8:
R = Rollout (æ,, R. noise=W,, &
9:
Assign g as r's fitness
10:
end for
11:
Rank the population pop, based on fitness scores
12:
Select the first teams pop, elites
13:
Select the remaining (k - é) teams æ from pop,, to form Set S using tournament selection while [S] < (k: e) do
14:
15:
Single-point crossover between a randomly sampled π E and æ E S and append to S end while
16:
17:
for Agent k=1.V do
Randomly sample minibatch of transitions (o,, a, 4, 0141) from R
Compute y = + min 2,(0141.a" (82)
19:
j=1,2
20:
where a = = (ks, 10 [action sampled from the h head of T/e tc
21:
Update Q by minimizing the loss: T. E,(y, Q(o;, a 10002
Update The using the sampled policy gradient
V OF J ~ Va ((0)) Vog, # (5) (s|0 ( Moza,
23:
Soft update target networks: (z" + -0 + (1-r)0r' and 0o¹ + -0 + (1 7)0-
24: end for
25:
Migrate the policy gradient team popy for weakest π E popl TT + 8.
26: end for
We represent our multiagent (team) policies using multi-heade neural network π as illustrated in Figure The head π² represents the k-th agent in the team. Given an incoming observation for agent only the output of x is considered as agent k's response. In essence, all
Policy Topology:
3

---

Under review as a conference paper at ICLR 2020
agents act independentll based on their own observations while sharing weights (and by extension. the features) in the lower layers (trunk). This is commonly used to improve learning speed (Silver et al. 2017). Further, each agent & also has its own replay buffer (R*) which stores its experience defined by the tuple (state, action, next state, local reward) for each interaction with the environment (rollout) involving that agent.
Team Reward Optimization: Figure iillstrraess the MERL algo- rithm. population of multi-headed teams, each with the same topology, is initialized with random weights. The replay buffer R shared by the k-th agent across all teams. The population is then evaluated for each rollout. The team reward for each team is dis- bursed at the end of the episode and is considered its fitness score. A selection operator selects portion of the population for survival with probability proportionate their fitness scores. The weights of the teams in the population are probabilistically perturbed through mutation and crossover operators to create the next generation teams. A portion of the teams with the highest relative fitness are preserved as elites. At any given time, the team with the highest fitness, the champion, represents the best solution for the task.
Q2
43
0.0 60b
π
0,9
TI
0.0 6
dob
9,9_0 00000
Trunk
Observation
Figure Team represented as multi-headed policy net æ
The procedure described so far resembles standard EA except that each agent & stores each of its experiences in its associated replay buffer (R*) instead of just discarding it. However, unlike EA,
Policy Gradient:
Evolutionary Population
Team
Trunk
which only learns based on the low- fow-fidelity global reward, MERL also learns from the experiences within episodes of a rollout us-
Pollcy Migration
Champion Trank
Team
Fitness
Replay
ing policy gradients. To enable this kind of "local learning", MERL initializes one multi-headed policy network Tp and one
R
Buffers
Global Reward
critic Q. A noisy version of Tpg is then used to conduct its set of rollouts in the environment, storing each agent k's ex- periences in its corresponding buffer (I²*) similar to the evolutionary rollouts.
Local Policy
Trunk
Gradients
Selection
Figure 2: High level schematic of MERL highlighting the integration of local and global reward functions
Agent-Specific Reward Optimization:
Crucially, each agent's 's replay buffer is kept separate from that of every other agent to ensure diversity amongst the agents. The shared critic samples random mini-batch uniformly from each replay buffer and uses it to update its parameters using gradient descent. Each agent The then draws a mini-batch of experiences from its corresponding buffer ( (R*) and uses sample policy gradient from the shared critic. Unlike the teams in the evolutionary population which directly seek to optimize the team reward, ag seeks to maximize the agent-specific local reward while exploiting the experiences collected via evolution.
Periodically, the Tyg network is copied into the evolving population of teams and can propagate its features by participating in evolution. This is the core mechanism that combines policies learned via agent-specific and team rewards. Regardless of whether the two rewards are aligned, evolution ensures that only the performant derivatives of the migrated network are retained. This mechanism guarantees protection against destructive interference commonly seen when direct scalarization between two reward functions is attempted. Further, the level of information exchange automatically adjusted during the process of learning, contrast to being manually tuned by an expert designer.
Skill Migration:
Algorithm| D provides a detailed pseudo-code of the MERL algorithm. The choice of hyperparameters is explained in the Appendix. Additionally, our source code is available online.
https//tinyart..oo//yrrrts
4

---

Under review as a conference paper at ICLR 2020
4 EXPERIMENTS
Agent
Agene
Prodater
Landmork
Apoil
2
Prodacee 5
Adremary
Prey
Advenary
(a) Predator-Prey
(b) Physical Deception
(c) Keep-Away
(d) Rover Domain
Figure 3: Illustration of environments tested (Lowe et al.
2017 Rahmattalabi et al.
2016)
We adopt environments from (Lowe et al., 2017) and (Rahmattalabi et al. (016) to perform our experiments. Each environment consists multiple agents and landmarks two-dimensional world. Agents take continuous control actions move about the world. Figure 3 illustratess the four environments which are described more detail below.
Predator-Prey: In this environment, N slower cooperating agents (predators) must chase the faster adversary (prey) around an environment with L large landmarks in uundomlygenneraeee locations. The predators get reward when they catch (touch) the prey while the prey is penalized. The team reward for the predators is the cumulative number of prey-touches in an episode. Each predator can also compute the average distance the prey and use it as its agent-specific reward. All agents observe the relative positions and velocities of the other agents as well as the positions of the landmarks. The prey accelerate 33% faster than the predator and has higher top speed. We tests two versions termed simple and hard predator-pre prey where the prey is 30% and 100% faster, respectively. Additionally, the prey itself learns dynamically during training. We use DDPG (Lillicrap al., (0155) as a learning algorithm for training the prey policy. All of our candidate algorithms are tested on their ability to train the team of predators catching this prey.
N agents cooperate to reach a single target Point of Interest (POI) among N POIs. They are rewarded based on the closest distance any agent to the target. lone adversary also desires to reach the target POI. However, the adversary does not know which of the POIs is the correct one. Thus the cooperating agents must learn to spread out and cover all POIs so as to deceive the adversary they are penalized based on the adversary's distance to the target. The team reward for the agents is then the cumulative reward in an episode. We use DDPG (Lillicrap et al. 2015) to train the adversary policy.
Physical Deception:
Keep-Away: In this scenario, a team of cooperating agents must reach a target POI out L total POIs. Each agent is rewarded based on its distance the target. We construct the team reward as simply the sum of the agent-specific rewards in an episode. An adversary also has to occupy the target while keeping the cooperating agents from reaching the target by pushing them away. To incentivize this behavior, the adversary is rewarded based on its distance to the target POI and penalized based on the distance of the target from the nearest cooperating agent. Additionally, it does not know which of the POIs is the target and must infer this from the movement of the agents. DDPG (Lillicrap et al. 2015) is used train the adversary policy.
This environment is adapted from (Rahmattalabi et al. 2016). Here, N agents must cooperate reach set of K POIs. Multiple agents need to simultaneously go the same POI in order to observe it. The number of agents required to observe POI is termed the coupling requirement. Agents do not know and must infer the coupling factor from the rewards obtained. If team with fewer agents than this number go the percentage of POIs observed at the end of an episode.
Rover Domain:
POI, reward is observed. The team's 's reward is
Each agent can also locally compute its distance its closest POI and use it as its agent-specific reward. Its observation comprises two channels to detect POIs and rovers, respectively. Each channel receives intensity information over 10" resolution spanning the 360 around the agent's position loosely based on the characteristic of Pioncer robot (Thrun et al. 2000). This is similar to a LIDAR. Since returns the closest reflector, occlusions make the problem partially-observable. A coupling
5

---

Under review as a conference paper at ICLR 2020
factor of 1 is similar to the cooperative navigation task in Lowe et al. (2017). We test coupling factors from to 7 to capture extremely complex coordination objectives.
Compared Baselines: We compare the performance of MERL with standard neuroevolutionary algorithm (EA) (Fogel 2006), MADDPG (Lowe et al. 2017) and MATD3, a variant of MADDPG that integrates the improvements described within TD3 (Fujimoto et al. 2018) over DDPG. Internally, MERL uses EA and TD3 as its team-reward and agent-specific reward optimizer, respectively. MADDPG was chosen as it is the state-of-the-an multiagent RL algorithm. We implemented MATD3 to ensure that the differences between MADDPG and MERL do not originate from having the more stable TD3 over DDPG.
Methodology for Reported Metries: For MATD3 and MADDPG, the team network was periodi- cally tested on 10 task instances without any exploratory noise. The average score was logged as its performance. For MERL and EA, the team with the highest fitness was chosen as the champion for each generation. The champion was then tested on 10 task instances, and the average score was logged. This protocol shielded the reported metrics from any bias of the population size. We conduct statistically independent runs with random seeds from {2019, 2023] } and report the average with error bars showing a 95% confidence interval. All scores reported are compared against the number of environment steps (frames). step is defined as the multiagent team taking a joint action and receiving a feedback from the environment. To make the comparisons fair across single-tean and population-bast based algorithms, all steps taken by all teams in the population are counted cumulatively.
RESULTS
Pordator-Prey:. Figure shows the comparative performance in control- ling the team of predators in the Predator-Prey Prey environment. Note that this is an adversarial environment where the prey dynamically adapts against the predators. The prey (con- sidered as part of the environment in this analysis) uses DDPG to learn constantly against our team preda- tors. This is why predator perfor- mance (measured as number prey touches) exhibits ebb and flow during learning. MERL outperforms MATD3, EA, and MADDPG across both simple and hard variations of the task. EA seems to be approaching MERL's performance but is significantly slower to learn. This is an expected behavior for neuroevolutionary methods which are known to be sample-ineffeiient. In contrast, MERL, by virtue of its fast policy-gru gradient components, learns significantly faster.
25
#Prey Touches
15
MATED
PERL
MADOPG
20
15
10
#Prey
10
6
Million Steps
Million Steps
Figure 4: Performance on Predator- Prey where the prey is 30% faster (left) and 100% faster (right), respectively.
Physical Deception:
Figure 5 (left)
4.0
shows the comparative performance in controlling the team of agents in the Physical Deception environment. The performance here is largely based on how close the adversary comes to the target POI. Since the adversary starts out untrained, all compared al- gorithms start out with fairly high score. As the adversary gradually learns to infer and move towards the target POI, MATD3 and MADDPG demonstrate gradual decline in performance. However, MERL and EA are able to hold their performance by concocting effective counter-strategies in deceiving the adversary. EA reaches the same performance MERL but is slower learn.
3.5
Performance
-15
-2.0
2.5
2.5
2.0
1.5
-3.0
1.0
-3.5
8
2
Million Steps
Million Steps
Figure 5: Performance on Physical Deception (left) and Keep-Away (right)
Keep-Away: Figure 5 (right) show the comparative performance in Keep-Away. Similar to Physical Deception, MERI and EA are able hold performance by attaining good counter-measures against

---

Under review as a conference paper at ICLR 2020
the adversary while MATD3 and MADDPG fail to do so. However, EA slightly outperforms MERL on this task.
Rover Domain: Figure 6 shows the comparative performance of MERL, MADDPG, MATD3, and EA tested in the rover domain with coupling fac- tors 1.3 and 7. In order to benchmark against the proxy reward functions that use scalarized lin- ear combinations, we test MADDPG and MATD3 with two variations of reward functions. Global represents the scenario where only the sparse team reward is used. Mixed represents the scenario where linear combination of the team-reward and agent-specific reward used. Each reward is normalized before being combined. A weigh- ing coefficient 10 is used to amplify the team- reward's influence in order to counter its sparsity. The weighing coefficient was tuned using a grid search (more details in Fi
goa
Performann
0.4
8.4
0.0
0.5
3.0
1.5
24
0.5
14
1.5
2.0
Million Steps
Million Steps
(a) Coupling
(b) Coupling 3
MEAL
gea
MADDPGG mixed
MADDPG global
MATD3_min mixed
g02
MATD3 global
EA
0.0
0.5
15 2. Le
Million Steps
(c) Coupling 7
(d) Legend
MERL significantly outperforms all baselines across all coupling requirements. The tested base- lines clearly degrade quickly beyond a coupling of The increasing coupling requirement is equiv- alent to increasing difficulty in joint-space exploration and entanglement in the team objective. However, it does not increase the size of the space, complexity of perception, or navigation. This indicates that the degradation in performance is strictly due to the increase in complexity of the team objective.
Figure 6: Performance on the Rover Domain.
Notably, MERL is able to learn on coupling greater than n 6 where methods without explicit reward shaping have been shown fail entirely (Rahmattalabi al., 2016). MERL successfully completes the task using the same set of information and coarse, unshaped reward functions as the other algorithms. The primary mechanism that enables this is MERL's split-level approach that allows it to leverage the agent-specific reward function to solve navigation and perception while concurrently using the team-reward function to learn team formation and effective coordination.
Scalarization Coefficients for Mixed Rewards: Figure 7 shows the performance MATD3 in optimizing mixed rewards computed with different coefficients used am- plify the team-reward relative to the agent-reward. The results demonstrate that finding good balance between these two rewards through linear scalarization is difficult, as all values tested fail make any progress in the task. This is because static scalarization cannot capture the dynamic properties of which reward important when and instead leads to an ineffective proxy. In contrast, MERI. able to leverage both reward functions without the need to explicitly combine them either linearly via more complex mixing functions.
0.10
Performance
0.08
10.0
50.0
0.06
100.0
0.04
0.02
0.00
1
2
Million Steps
Figure 7: MATD3's performance for dif- ferent scalarization coefficients
Team Behaviors: Figure illlusrrates the trajectories gen- erated for the Rover Domain with a coupling of = 3. The trajectories for partially and fully trained MERL are shown in Figure 8 (a) and (b), respectively. During training, when MERL has not discov- ered team success (no POIs are successfully observed), MERL simply optimizes the agent-specific reward for each agent. This allows it reach trajectories such as the ones shown in a) where each agent learns to go towards POI.
Since each agent explicitly aims reach a POI, the probability 3 agents congregating the same POI is higher compared to random undirected exploration by each agent without the dense agent-specific reward. Once this scenario is discovered, the team reward optimizer (EA) within MERL explicitly selects for agent policies that jointly lead to such ream-forming behaviors. Eventually it succeeds
7

---

Under review as a conference paper at ICLR 2020
shown in Figure 8 b). Here, team formation and collaborative pursuit of the POIs is immediately apparent. Two teams of 3 agents each form at the start of the episode. Further, the two teams also coordinate pursue different POIs in order to maximize the team reward. While not perfect (the bottom POI left unobserved), they do succeed in observing 3 out of the 4 POls.
In contrast, MATD3-mixed fails to observe any POL. From the trajectories, it is apparent that the agents have successfully learned perceive and navigate to reach POIs. However, they are unable to use this skill to- wards fulfilling the team objec- tive. Instead each agent rather split on the objective that it is optimizing. Some agents seem to be in sole pursuit of POIs without any regard for team for- mation collaboration while others
(a) MERL: training
(b) MERI: trained
(c) MATD3: trained
Figure Agent trajectories for coupling 3. Red/black squares are observed/umobserved POIs respectively exhibit random movements.
The primary reason for this the mixed reward function that directly combines the agent-specific and team reward functions. Since the reward functions have no guarantees of alignment across the state-space of the task, they invariably lead to learning these sub-optimal joint-behaviors that solve a certain form of scalarized mixed objective. In contrast, MERL by virtue of its bi-level optimization framework is able to leverage both reward functions without the need to explicitly combine them. This enables MERL avoid these sub-optimal policies and solve the task without any reward shaping or manual tuming.
We ran experiments tracking whether the poli- cies migrated from the policy gradient learners to the evolution- ary population were selected or discarded during the subsequent selection process (Figure 9). Note that the expected selection rate if chosen at random 0.1 as policy is migrated into a pop- ulation of 10. In contrast, the selection rate for migrated policies significantly higher across all benchmarks with the exception of Keep-Away. This is consistent with the performance results seen in Kwep-Away where EA initially outperforms MERL.. How- ever, in general, these results indicate that MERL's integrative approach in combining the two optimization processes towards optimizing the team objective crucial.
Selection Rate:
1.2
Rover Icoup ingein
Selection Rate
1.0
Paysical Decestion
0.6
0.4
0.2
0.0
0.2
0.4
0.6
0.8
Normalized Generations
Figure 9: Selection rate for migrat- ing policies
6 CONCLUSION
In this paper, we introduced MERL, a split-level algorithm that leverages both agent-specific and team objectives by combining gradient-based and gradient-free optimization. MERL achieves this by using a fast aelicy-gradien optimizer to exploit dense agent-specific rewards while concurrently leveraging neuroevolution to tackle the ceam-objective.
Results demonstrate that MERL significantly outperforms MADDPG, the state-of-ther multiagent RL method, in wide array of benchmarks. We also tested a modification of MADDPG to integrate TD3 the state-of-th maae-offtheeart single-agent RI. algorithm. These experiments demonstrated that the core improvements of MERL originate from ability to leverage both team and agent-specific reward functions without the need to explicitly combine them. This differentiates MERL from other approaches like reward scalarization and reward shaping that either require extensive manual tuning can detrimentally change the MDP (Ng al. 1999) itself.
Future work will explore MERL for adversarial settings such as Pommerman (Resnick et al. 201 8). StarCraft (Justesen and Risi 2017 Vinyals et al., (207)) and RoboCup (Kitano et al.) 1995 Liu et al.) 2019). Further, extending MERL to general melti- settings such as is the case for multitast Tearning, is another promising for future work.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
C. Colas, O. Sigaud, and P.-Y. Oudeyer. Gep-pg: Decoupling exploration and exploitation in deep reinforcement learning algorithms. arXiv preprint arXiv:1802.05033 2018.
S. Devlin, M. Grzeš, and D. Kudenko. Multi-an Multi-agent, reward shaping for robocup keepaway. In The 10th International Conference on Autonomous Agents and Multiagent Systems- Volume 3, pages 1227-1228. International Foundation for Autonomous Agents and Multiagent Systems, 2011.
D. Floreano, P. Dürr, and Mattiussi. Neuroevolution: from architectures learning. Evolutionary Intelligence, 1(1):47-62, 2008.
J. Foerster, R. Y. Chen, M. Al-Shedivat, S. Whiteson, P. Abbeel, and I. Mordatch. Learning with opponent-learing awareness. Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pages 122-130. International Foundation for Autonomous Agents and Multiagent Systems, 2018a.
J. N. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson. Counterfactual multi-agent policy gradients. In Thirty-Second AAAI Conference Artificial Intelligence, 20.
D. B. Fogel. Evolutionary computation: toward a new philosophy of machine intelligence, volume 1. John Wiley & Sons, 2006.
S. Fujimoto, H. van Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. arXiv preprint arXiv:18022 09477, 2018.
M. Jaderberg, Dalibard, S. Osindero, W. M. Czarnecki, J. Donahue, A. Razavi, O. Vinyals, T. Green, I. Dunning, K. Simonyan, et al. Population based training of neural networks. arXiv preprint 1wtp::111.0998000 2017.
N. Justesen and S. Risi. Learning macromanagement in starcraft from replays using deep learning. In 2017 IEEE Conference on Computational Intelligence and Games (CIG), pages 162-169. IEEE, 2017.
Khadka and K. Tumer. inolution-guided policy gradient in reinforcement learning. In Advances in Neural Information Processing Systems, pages 1196-1208, 2018.
S. Khadka, S. Majumdar, T. Nassar, Z. Dwiel, E. Tumer, S. Miret, Y. Liu, and K. Tumer. Collaborative evolutionary reinforcement learning. arXiv preprint 19X5 0009666,, 2019.
H. Kitano, M. Asada, Y. Kuniyoshi, Noda, and E. Osawa. Robocup: The robot world cup initiative, 1995.
A. Lazaridou, A. Peysakhovich, and M. Baroni. Multi-agent cooperation and the emergence of (natural) language. arXiv preprint arXiv:1612.07182. 2016.
F.D. Li, M. Wu, Y. He, and X. Chen. Optimal control in microgrid using multi-agent reinforcement learning. ISA transactions, 51(6):743-751, 2012.
T. Lillicrap, J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, Silver, and D. Wierstra. Continuous control with deep reinforcement learning. arXiv preprint
2015.
M. L. Littman. Markov games as a framework for multi-agent reinforcement learning. In Machine learning proceedings 1994, pages 157-163, Elsevier, 1994.
Liu, Lever, J. Merel, Tunyasuvunakool, N. Heess, and T. Graepel. Emergent coordination through competition. arXiv preprint arXiv:1902.07151/ 2019.
R. Lowe, Y. Wu, A. Tamar, J. Harb, O. P. Abbeel, and I. Mordatch. Multi-agent actor-critic for mixed cooperative-compete
environments. In Advances in Neural Information Processing Systems, pages 6379-6390, 2017.
B. Lüders. M. Schläger, A. Korach, and S. Risi. Continual and one-shot learning through neu- ral networks with dynamic external memory. In European Conference on the Applications of Exolutionary Computation, pages 886-901. Springer, 2017.

---

Under review as a conference paper at ICLR 2020
1. Mordatch and P. Abbeel. Emergence of grounded compositional language in multi-agent popula- tions. In Thirty-Secone AAAI Conference on Artificial Intelligence, 2018.
A. Y. Ng. D. Harada, and S. Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In ICML, volume 99, pages 278-287, 1999.
A. Rahmattalabi, J. J. Chung, M. Colby, and K. Tumer. D++: Structural credit assignment in tightly coupled multiagent domains. In 20/6 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4424 4429. IEEE, 2016.
C. Resnick, W. Eldridge, D. Ha, D. Britz, Foerster, J. Togelius, K. Cho, and J. Bruna. Pommerman: A multi-agent playground. arXiv preprint arXiv: /tt:::/800.000000 0009/8809000201 07/44, 2018.
S. Shalev-Shwartz, S. Shammah, and A. Shashua. Safe, multi-agent, reinforcement learning for autonomous driving. arXiv preprint arXiv:1610.03295. 2016.
W. Sheng, Q. Yang, J. Tan, and N. Xi. Distributed multi- robot coordination in area exploration. Robolics and Autonomous Systems, 54 (4(1))9945-955 2006.
D. Silver, T. Hubert, Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Ku- maran, T. Graepel, al. Mastering chess and shogi by self-play with general reinforcement learning algorithm. arXiv preprint arXiv: 1712:1712.01815. 2017.
W. M. Spears, K. A. De Jong, T. Bäck, D. B. Fogel, and H. De Garis. An overview of evolutionary computation. In European Conference on Machine Learning, pages 442-459. Springer, 1993.
R. S. Sutton and A. Barto. Reinforcement learning: An introduction, volume Cambridge, 1998.
MIT press
S. Thrun, W. Burgard, and D. Fox. A real-time algorithm for mobile robot mapping with applications to multi-robot and mapping. ICRA, volume I, pages 321-322 221-328, 2000.
K. Tumer and A. Agogino. Distributed agent-based air traffic flow management. In Proceedings of the 6th international joint conference on Autonomous agents and multiagent systems, page 255. ACM, 2007.
O. Vinyals, T. Ewalds, S. Bartunov, P. Georgiev, A. S. Vezhnevets, M. Yeo, A. Makhzani, H. Küttler, Agapiou, J. Schrittwieser, al. Starcraft ii: new challenge for reinforcement learning. arXiv preprint arXiv: 708.04788888 2017.
S. A. Williamson, E. H. Gerding, and N. R. Jennings. Reward shaping for valuing communica- tions during multi-agent coordination. In Proceedings of The 8th International Conference on Autonomous Agents and Multiagent Systems- Volume 1, pages 148. International Foundation for Autonomous Agents and Multiagent Systems, 2009.
L. Yliniemi, A. K. Agogino, and K. Tumer. Multirobot coordination for space exploration. Al Magazine, 35(4):61 2014.
10

---

Under review as a conference paper at ICLR 2020
A
HYPERPARAMETERS DESCRIPTION
Table I: Hyperparameters used for Predator-Prey, Keep-away and Physical Deception
Hyperparameter
MERL
MATD3/MADDPG
Population size k
10
N/A
Rollout size
10
10
Target weight т
0.01
0.01
Actor Learning Rate
0.01
0.01
Critic Learning Rate
0.01
0.01
Discount Learning Rate
0.95
0.95
Replay Buffer Size
le
le
Batch Size
1024
1024
Mutation Probability mudproob
0.9
N/A
Mutation Fraction mut frac
0.1
N/A
Mutation Strength muulstrength
0.1
N/A
Super Mutation Probability supermallppob
0.05
N/A
Reset Mutation Probability resetmutprob
0.05
N/A
Number of elites Exploration Policy
N/A
(0,
(O0, a)
Exploration Noise o Rollouts per fitness & Actor Neural Architecture Critic Neural Architecture TD3 Policy Noise variance
0.4
0.4
10
N/A
[100, 100)
1000, 100)
[100, 100]
300,300
0.2
0.2
TD3 Policy Noise Clip
0.5
0.5
TD3 Policy Update Frequency
Table details the hyperparameters used for MERL, MATD3, and MADDPG in tackling predator- prey and cooperative navigation. The hyperparmacters were inherited from Lowe et al. (2017) to match the original experiments for MADDPG and MATD3. The only exception to this was the use of hyperbolic tangent instead Relu activation functions. Table 2: Hyperparameters used for Rover Domain
Hyperparameter
MERL
MATD3/MADDPG
Population size &
10
N/A
Rollout size
50
Target weight 7
le
le-5
Actor Learning Rate
5e 5
5e 55
Critic Learning Rate
le
1E 5
Discount Learning Rate
0.5
0.97
Replay Buffer Size
1
Le
Batch Size
512
512
Mutation Probability mutproo
0.9
N/A
Mutation Fraction mut frae
0.1
N/A
Mutation Strength mudl nutgttenggh
0.1
N/A
Super Mutation Probability supermeulprod
0.05
N/A
Reset Mutation Probability resetmut
0.05
N/A
Number of clites e Exploration Policy
N/A
((0,
(O0, a)
Exploration Noise o
0.4
0.4
Rollouts per fitness
10
N/A
Actor Neural Architecture Critic Neural Architecture TD3 Policy Noise variance
[100, 100]
1100, 100)
[100, 100]
[300, 300
0.2
0.2
TD3 Policy Noise Clip
0.5
0.5
TD3 Policy Update Frequency
2
2
11

---

Under review as a conference paper at ICLR 2020
Table 2 details the hyperparameters used for MERL, MATD3, and MADDPG in the rover domain. The hyperparameters themselves are defined below:
Optimizer = Adam
Adam optimizer was used update both the actor and critic networks for all learners.
Population size
This parameter controls the number of different actors (policies) that are present in the evolutionary population.
Rollout size
This parameter controls the number of rollout workers (each running episode of the task) per generation.
The two parameters above (population size & and rollout size) collectively modulates the proportion of exploration carried out through noise in the actor's parameter space and its action space.
Note:
Target weight
This parameter controls the magnitude of the soft update between the actors and critic networks, and their target counterparts.
Actor Learning Rate
This parameter controls the learning rate of the actor network.
Critic Learning Rate
This parameter controls the learning rate the critic network.
Discount Rate
This parameter controls the discount rate used compute the return optimized by policy gradient.
Replay Buffer Size
This parameter controls the size of the replay buffer. After the buffer is filled, the oldest experiences are deleted in order to make room for new ones.
Batch Size
This parameters controls the batch size used to compute the gradients.
Actor Activation Function
Hyperbolic tangent was used the activation function.
Critic Activation Function
Hyperbolic tangent was used as the activation function.
Number of Elites
This parameter controls the fraction of the population that are categorized as elites. Since an elite individual (actor) is shielded from the mutation step and preserved as it is, the elite fraction modulates the degree of exploration//exllotation within the evolutionary population.
Mutation Probability
This parameter represents the probability that an actor goes through a mutation operation between generation.
Mutation Fraction
This parameter controls the fraction of the weights in chosen actor (neural network) that are mutated, once the actor is chosen for mutation.
Mutation Strength
This parameter controls the standard deviation of the Gaussian operation that comprises mutation.
Super Mutation Probability
This parameter controls the probability that a super mutation (larger mutation) happens in place of a standard mutation.
Reset Mutation Probability
This parameter controls the probability a neural weight is instead reset between (00, 1) rather than being mutated.
12

---

Under review as a conference paper at ICLR 2020
Exploration Noise
This parameter controls the standard deviation of the Gaussian operation that comprise the noise added to the actor's actions during exploration by the learners (learner roll-outs).
TD3 Policy Noise Variance
This parameter controls the standard deviation of the Gaussian operation that comprise the noise added to the policy output before applying the Bellman backup. This is often referred the magnitude of policy smoothing in TD3.
TD3 Policy Noise Clip
This parameter controls the maximum of the policy noise used to smooth the policy.
TD3 Policy Update Frequency
This parameter controls the number of critic updates per policy update in TD3.
B ROLLOUT METHODOLOGY
Algorithm describes an episode of rollout under MERL detailing the connections between the local reward, global reward, and the associated replay buffer. Algorithm 2 Function Rollout
1: procedure ROLLOUT(T, R, noise, )
2: filness
3: for 1:f do
4:
Reset environment and get initial joint state js while eny is not done do
5:
6:
Initialize an empty list joint action ja = for Each agent (actor head) ~² E and są in js do
7:
ja < ja (*x [0=*) + noise; end for
8:
9:
10:
Execute ja and observe joint local reward jl, global reward g and joint next state js for Each Replay Buffer Rr R and 8k, ags lx, sy, in js, ja, jl, js do
11:
12:
Append transition (Sk, aj. 4,, s6)) to RA end for js j8
13:
14:
15:
if env done: then fitness +
16:
end if
18:
end while
end for
20:
Return fitnese R
21:
end procedure
13

---

Under review as a conference paper at ICLR 2020
C EVOLUTIONARY ALGORITHM POPULATION RUNS
0.6
50
Performance
0.5
100
1000
MERL
0.4
0.3
0.2
0.1
0.0
10-1
100
101
102
Million Steps
Figure 10: Evolutionary Algorithm Population size sweep the rover domain with coupling of 3. MERL was run for 2-million steps while the other EA runs were ran for 100-million steps.
Figure 10 compares EA with varying population sizes in the rover domain with coupling of Among the EA runs, a population size of 100 yields the best results converging to 0.3 in 100-millions frames. MERL (red) on the other hand is ran for 2-million frames and converges 0.48. This is due to MERL's ability to leverage gradient descent from its policy gradient components that lead to significantly faster learning performance.
D EVOLUTIONARY STRATEGIES (ES)
D.1 ES POPULATION SWEEP
0.6
10000
100
0.5
1000
Performance
MERL
0.4
0.3
0.2
0.1
0.0
10-1
100
101
102
Million Steps
Figure 11: Evolutionary Strategies population size sweep on the rover domain with coupling of 3
Figure compares ES with varying population sizes in the rover domain with coupling of 3. Sigma at 0.1. Among the ES runs, population size of 100 yields the best results
for all ES runs converging to 0.1 in 100-millions frames. MERL (red) on the other hand ran for 2-million frames and converges to 0.48.
14

---

Under review as a conference paper at ICLR 2020
D.2 ES SIGMA SWEEP
0.6
0.01
0.5
Performance
0.5
0.2
0.1
MERL
0.4
0.3
0.2
0.1
0.0
10-1
100
10
102
Million Steps
Figure 12: Evolutionary Strategies Noise magnitude (sigma) sweep on the rover domain with coupling 3
Figure 12 compares ES with varying variance of noises (sigma) that control the magnitude each perturbation. The experiments are conducted in the rover domain with coupling of 3 with population size 100. Among the ES runs, a sigma 0.1 yields the best results converging to 0.1 in 100-millions frames. MERL (red) on the other hand is ran for 2-million frames and converges to 0.48.
E PREDATOR-PRET PREY WITH 3 PREY
18
1 prey
16
3 prey
mert prey
Performance
14
12
10
8
6
4
2
0
2
4
Million Steps
Figure 13: Predator-prey with varying numbers of prey. Prey are 30% faster than the predators
Figure 13 shows the results of running MATD3 with varying number of prey in the predator-pre domain. The experiments are ongoing.
15


Reviews:
[{"review_id": "rkg0GWmAYH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to use a two-level optimization process to solve the challenge of optimizing the team reward and the agent's reward simultaneously, which are often not aligned. It applies the evolutionary algorithm to optimize the sparse team reward, while using RL (TD3) to optimize the agent's dense reward. In this way, there is no need to combine these two rewards into a scalar that often requires extensive manual tuning.\n\nI vote for accepting this paper, because it tackles a practical problem in multi-agent learning. The presentation is clear, the algorithm is simple and sensible, the evaluation is thorough, and the results are better than the state-of-the-art.\n\nThe only question that I have is the sharing of replay-buffer for the same agent in different teams (Figure 2). For example, since the second agents in team1 and in team2 might learn to serve different roles in the task and may have completely different policies. I am not sure what is the purpose of this sharing. Considering the two extreme cases of replay buffer sharing, we could share all the data in a single replay buffer, or we could keep a separate replay buffer for each individual agent in each team, the paper chose the compromise between these two extremes. I wonder whether there is any theoretical or practical reasons to make this design choice. Is it important? I hope that the paper could have a deeper discussion if it is an important design decision.\n\n----------------------Update after rebuttal----------------------\n\nThanks for the detailed response and the additional experiments. The response addressed my questions. Thus I will keep my original recommendation of acceptance.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "r1xjRtB1cB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an integration of neuroevolution and gradient-based learning for reinforcement learning applications. The evolutionary algorithm focuses on sparse reward and multiagent/team optimization, while the gradient-based learning is used to inject selectively improved genotypes in the population.\nThis work addresses a very hot topic, i.e. the integration of NE and DRL, and the proposed method offers the positive side of both without introducing major downsides. The presented results come from a relatively simple but useful multiagent benchmark which has broad adoption. The paper is well written, presents several contributions that can be extended and ported to other work, and the results are statistically significant.\n\nThere is one notable piece missing which forces me to bridle my enthusiasm: a discussion of the genotype and of its interpretation into the network phenotype. The form taken by the actual agent is not explicitly stated; following the adoption of TD3 I would expect a policy and two critics, for a grand total of three neural networks, but this remains unverified. And if each agent is composed of three neural networks, and each individual represents a team, does this mean that each genotype is a concatenation of three (flattened) weight matrices per each agent in the team? What is the actual genotype size? It sounds huge, I would expect to be at least several hundred weights; but then this would clash with the proposed minuscule population size of 10 (recent deep neuroevolution work from Uber uses populations THREE orders of magnitude larger). Has the population size been proportionated to the genotype dimensionality? Would it be possible to reference the widely adopted defaults of industry standard CMA-ES? Speaking of algorithms, where is the chosen EA implementation discussed? The overview seems to describe a textbook genetic algorithm, but that has been overtaken as state-of-the-art since decades, constituting a poor match for TD3.\n\nOmitting such a chapter severely limits not only the reproducibility of the work but its full understanding. For example, does the EA have sufficient population size to contribute significantly to the process, or is it just performing as a fancy version of Random Weight Guessing? Could you actually quickly run RWG with direct policy search (rather than random action selection) to establish the effective complexity of the task? My final rating after rebuttal will vary wildly depending on the ability to cover such an important piece of information. \n\nA few minor points, because I think that the paper appearance deserves to match the quality of the content:\n- The images are consistently too small and hard to read. I understand the need to fit in the page limit by the deadline, but for the camera ready version it will be necessary to trim the text and rescale all images.\n- The text is well written but often slowing down the pace for no added value, such as by dedicating a whole page to discussing a series of previously published environments.\n- The hyperparameters of the evolutionary algorithm look completely unoptimized. I would expect a definite improvement in performance with minimal tuning.\n- The \"standard neuroevolutionary algorithm\" from 2006 presented as baseline has not been state-of-the-art for over a decade. I would understand its usage as a baseline if that is indeed the underlying evolutionary setup, but otherwise I see no use for such a baseline.\n\n-----------------------------------------------------------------------------------------------\n# Update following the rebuttal phase\n-----------------------------------------------------------------------------------------------\n\nThank you for your work and for the extended experimentation. I am confident the quality of the work is overall increased.\n\nThe core research question behind my original doubt however remains unaddressed: does the EC part of the algorithm sensibly support the gradient-descent part, or is the algorithm basically behaving as a (noisy) multi-agent TD3?\nSuch a contribution by itself would be undoubtedly important. Submitting it as a principled unification of EC and DL however would be more than a simple misnomer: it could mislead further research in what is an extremely promising area.\n\nThe scientific approach to clarify this point would be to design an experiment showcasing the performance of MARL using a range of sensible population sizes. To understand what \"sensible\" means in this context, I refer to a classic:\nhttp://www.cmap.polytechnique.fr/~nikolaus.hansen/cec2005ipopcmaes.pdf\nA lower bound for the population size with simple / unimodal fitness functions would be $4+floor(3*log(10'000)) = 31$. With such a complex, multimodal fitness though, no contribution from the EA can be expected (based on common practice in the EC field) without at least doubling or tripling that number. The upper bound does not need to be as high as with the recent Uber AI work (10k), but certainly showing the performance with a population of a few hundreds would be the minimum necessary to support your claim. A population size of 10 represents a proper lower bound for a genotype of up to 10 parameters; it is by no means within a reasonable range with your dimensionality of 10'000 parameters, and no researcher with experience in EC would expect anything but noise from such results -- with non-decreasing performance uniquely due to elitism.\nThe new runs in Appendice C only vary the population size for the ES algorithm, proposed as a baseline. No performance of MARL using a sensible population size is presented.\n\nThe fundamental claim is thereby unsustainable by current results. The idea is extremely intriguing and very promising, easily leading to supportive enthusiasm; it is my personal belief however that accepting this work in such a premature stage (and with an incorrect claim) could stunt further research in this direction.\n\n[By the way, the reference Python CMA-ES implementation runs with tens of thousands of parameters and a population size of 60 in a few seconds per generation on a recent laptop: the claim of performance limitations as an excuse for not investigating a core claim suggests that more work would be better invested prior to acceptance.]\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "H1g9lJnpFH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an algorithm to learn coordination strategies for multi-agent reinforcement learning. It combines gradient-based optimization (Actor-critic) with Neuroevolution (genetic algorithms style). Specifically, Actor-critic is used to train an ensemble of agents (referred to as “team”) using a manually designed agent-specific reward. Coordination within a team is then learned with Neuroevolution. The overall design accommodates sharing of data between Actor-critic and Neuroevolution, and migration of policies. Evaluation is done using the multi-particle environments (Lowe et. al. 2017) and a Rover domain task.\n\n\nI have the following questions:\n\n1.\tComparison with PBT-MARL (Liu et al, 2019): PBT-MARL also proposed using gradient-methods (Retrace-SVG0) for learning with dense, shaped (local) rewards, and then using Neuroevolution to optimize the agents for coordination. I don’t think the description in Section 2 characterizes PBT-MARL well enough – reading that gives the impression that it only evolves the scalarization coefficients. PBT-MARL combines rewards with different discount factors (yielding it much more representation power than simple scalarization), and the discount factors are also evolved. Furthermore, the agent network weights are evolved based on team-reward, to learn coordination strategies. \n\nAt a qualitative level, this paper seems to be using a similar approach, albeit the specifics of Neuroevolution and policy-gradients are different. I would like the authors to shed light on the scenarios where they argue their method holds inherent advantage(s) compared to PBT-MARL.\n\n2.\tIf I understand correctly, the individual agents in the team trained with DDPG would converge to similar policies – this is because each individual is trained with the same (agent-specific) reward, all agents in team use the same shared critic Q, and there is no flow of team from Neuroevolution to DDPG phase. This in itself is not a problem, because MADDPG should do the same if all agents are trained with the same team-reward function. The issue is that there are crucial differences in the architectures -- while the MADDPG paper had a separate Q network and policy network for each agent, MERL shares policy parameters (lower layers) between agents and uses a single, shared Q network. Have the authors done ablations with more aligned architectures, so that the improvements due to the main algorithmic contributions can be clearer?  \n\n3.\tThe notation defined in the background section is “s” for completed state (all agents) and “o” for individual observations. In the for-loop in Algorithm 1, are correct variables being used at all places, e.g. for pi, Q? In other words, which functions depend on the complete state of all agents, and which are decentralized? Additionally, should lines 23/24 be inside the for-loop?\n\n4.\tExperiments – In section 4, could the authors provide the values of N, L, K used in different environments, as applicable? The MADDPG paper claims good performance for multi-particle environments, which contradicts Figure 4 and 5. For example, MADDPG claims about 16 touches per episode with 30% faster prey, but Figure 4 has it converging to about 5. This makes me wonder if the parameters (N, L, K) are different in the two evaluations. Also, physical-deception task performance is reported in success% in the MADDPG paper, but the authors use a different distance metric. A standardized way of comparing performance would help the community. \n\n5.\tIn MADDPG paper, an ensemble-based version is reported to perform much better. Since MERL is an ensemble method, is that not a more direct comparison?\n\nMinor points:\n\n1.\tIn environments (expect Rover), did the authors observe any benefits from adding the agent-specific reward to supplement the team-rewards (i.e. mixed reward setting) for the baselines?\n\n2.\tIn Figure 8, why is the right-most POI lit when the red trajectory is far away from it? If the authors could provide a video of the MERL-trained policy for this domain, it’d be really cool.\n\n\n\n----------------------Post-rebuttal update----------------------\n\n\nI am satisfied with the author response and changes to the paper. I have increased my rating accordingly.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work has a lot of promise; however, the author response was not sufficient to address the concerns expressed by reviewer 1, leading to an aggregate rating that is just not sufficient to justify an acceptance recommendation. The AC recommends rejection.
Decision: Reject