Mimicking Evolution with Reinforcement Learning
Authors: João Abrantes
James John Butterworth
Arnaldo J. Abrantes
Frans A Oliehoek
Venue: NeurIPS 2021 Submitted (2021)
Keywords: reinforcement learning
evolution
multi-agent
alife
open-endedness

Abstract: In nature, there are two processes driving the development of the brain: evolution and learning. Evolution acts slowly, across generations, and amongst other things, it defines what agents learn by changing their internal reward function. Learning acts fast, within one’s lifetime, and it quickly updates agents’ policies to maximise the evolved reward function. Although previous work has emulated both of these processes working in tandem, the optimisation of the reward function in order to serve the aims of the evolutionary process is very computationally expensive. This work proposes a fixed reward function, the evolutionary reward, that aims to maximise the number of current (and future) genetically similar agents. Furthermore, we propose a way to approximate the joint action value by averaging the action values of other agents weighted by their genetic similarity. In a finite environment with limited resources this techniques drives improved survival mechanisms and reproductive success. Given that this reward function is fixed, we avoid the computationally intense process of optimising it. We demonstrate the viability of our evolutionary reward by testing it in two bio-inspired, open-ended environments and monitoring a number of metrics such as population size and life expectancy. We compare our technique with the state-of-the-art evolutionary algorithm: CMA-ES, and show the superiority of work at producing agents that maximise the number of its genes across time.

Mimicking Evolution with Reinforcement Learning
Anonymous Author(s)
Affiliation Address email
Abstract
In nature, there two processes driving the development of the brain: evolution and learning. Evolution acts slowly, across generations, and amongst other things, it defines what agents learn by changing their internal reward function. Learning acts fast, within one's lifetime, and it quickly updates agents' policies to maximise the evolved reward function. Although previous work has emulated both of these processes working in tandem, the optimisation of the reward function in order to serve the aims of the evolutionary process is very computationally expensive. This work proposes a fixed reward function, the evolutionary reward, that aims to max- imise the number of current (and future) genetically similar agents. Furthermore, we propose way to approximate the joint action value by averaging the action values of other agents weighted by their genetic similarity. In a finite environment with limited resources this techniques drives improved survival mechanisms and reproductive success. Given that this reward function is fixed, we avoid the com- putationally intense process of optimising it. We demonstrate the viability of our evolutionary reward by testing it in two bio-inspired, open-ended environments and monitoring number of metrics such population size and life expectancy. We compare our technique with the state-of-the-ar evolutionary algorithm: CMA-ES, and show the superiority of work at producing agents that maximise the number of its genes across time.
2
10
11
13
14
15
17
19
Introduction
21 Evolution is the only process we know of today that has given rise to general intelligence (as demon-
22 strated in animals, and specifically in humans). This fact has been inspiring artificial intelligence (AI)
23 researchers run evolution in artificial worlds that mimic key properties of life on Earth. One of these key properties is open-endedness. This means that, as in nature, the fitness function (or any 25 goal function) of the environment is not defined anywhere but it simply emerges from the survival
24
26 and reproduction of genes. For this reason, we call these environments open-ended evolutionary environments (OEEE). They are never-ending environments where adaptable agents are competing 28 for a common limited-resource to survive and replicate their genes. Using them for research is the focus of the field of artificial life (ALife).
27
29
30
Our ability to run evolution efficiently in OEEE will dictate the success of ALife. this work we speed up the way evolution ran OEEE by introducing Evolution via Evolutionary Reward (EvER). In EvER, each agent is born with an evolutionary reward that, when maximised by a learning, also maximises the survival and reproduction of the agent's genes. Due this property we say that
31
32
33
34 this reward is aligned with evolution. This allows learning to search for policies with increasingly
35 evolutionary fitness. Also, by guarantying this alignment we don't need to go through the expensive
36 process of aligning the agents' reward functions through evolution. This reward function was designed
37 to work on any OBEE.
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

51 We say that a reward function is aligned with evolution when the maximisation of the reward leads 38 In the remaining part of this introduction we 1) describe how evolution changes what we learn; 2) introduce our contribution and describe how maximising reward function can lead to the maximisation of evolutionary fitness.
39
41
1.1
Evolving what to learn
In nature, there are two different mechanisms driving the development of the brain. Evolution acts slowly, across generations, and amongst other things, it defines what agents learn by changing their internal reward function. Learning acts fast, within one's lifetime, and it quickly updates agents' policies to maximise pleasure and minimise pain. Combining these two methods has long history in AI research [I 42 8]. This combination (illustrated in Appendix B Figure 3 results in a very computationally expensive algorithm as it requires two loops 1) learning (the inner loop) where 48 agents maximise their innate reward functions across their lifetimes and 2) evolution (the outer loop) where natural selection and mutation defines the reward functions for the next generation (amongst
42
43
44
45
46
50 other things, such as NN topologies and initial weights).
52 to the maximisation of the agent's 's fitness. Through evolution the most aligned reward functions get selected and increase their numbers. Intuitively, one can define the optimally aligned reward function as the reward function that allows learner learn most quickly how to maximise its fitness, assuming the conditions of the world (including other agents) remain the same. However, as agents cvolve and learn, they change their environment and its corresponding fitness function. This change, increases the misalignment between the reward and fitness functions. Therefore, the optimally aligned reward function is always chasing the ever changing fitness function (see Appendix [ for a formal description of this). However, in this paper, we show that in simulation it is possible define fixed reward function which always aligned, although not guaranteed to be optimally aligned, with the essence of fitness: the ability of the individual to survive and reproduce its genes.
53
54
55
56
58
60
62
Our work allows learning to single-handedly drive the search for policies with increasingly evolution- ary fitness by ensuring the alignment of the reward function with the fitness function. This greatly simplifies the two-loop algorithm used to combine evolution and learning that was described earlier in this section. We can do this because our reward is extrinsic to the agent and therefore, only possible within a simulation.
63
64
66
67
1.2 Learning to maximise evolutionary fitness
The distinction between an agent and gene is key to understanding this paper. Formally, evolution is a change in gene frequencies in a population (of agents) over time. The gene is the unit of evolution, and an agent carries one or more genes. Richard Dawkins has famously described our bodies as throwaway survival machines built for replicating immortal genes [6]. His line illustrates well the gene-centered view of evolution
69
71
45 6). view that has been able to explain multiple phenomena 73 such as intragenomic conflict and altruism that are difficult to explain with organism-cenee centered or 77 From the gene's 's perspective, the evolutionary process is constant competition for resources. However, from the agent's perspective, the evolutionary process is mix between cooperative exercise with agents that carry some of its genes (its family) and a competition with unrelated agents. Evolution pressures agents to engage in various degrees of collaboration depending on the degree of kinship between them and the agents they interact with (i.e. depending on the amount of overlap between the genes they carry). This pressure for cooperation amongst relatives was named kin selection
group-centered viewpoints 2 10
75
76
77
78
79
80
81
Evolution acts on the gene level, but RL acts on the agent level. RL can be aligned with the evolutionary process by noting what evolution does to the agents through its selection of genes: evolution generates agents with increasing capabilities to maximise the survival and reproduction success of the genes they carry.
83
84
2 Related work
85
Combining evolution and learning Combining evolution and learning has long history in AI research. The evolutionary reinforcement learning algorithm, introduced 1991 [. makes the evolutionary process determine the initial weights of two neural networks: an action and an evaluation
87

---

89 network. During an agent's lifetime, learning adapts the action network guided by the output of
90 its innate and fixed (during its lifetime) evaluation network. N 442 uses an evolutionary algorithm, NEAT 36), to evolve topologies of NN and their initial weights so that they can better
91
92 learn using RL. In NEAT-Q the reward function remains fixed. However, evolutionary algorithms have also been used to evolve potential- baased shaping rewards and meta-parameters for RI. [8.
93
94
Competing in Arms-race Every time adaptable entities compete against each other an arms-raa racc is created. Each entity's task gets harder every time their competitors learn something useful. This arms race drives the continued emergence of ever new innovative and sophisticated capabilities necessary to out-compete adversaries. Evolutionary Algorithms (EA) have been successfully used to co-evolve multiple competing entities 32 [29]. However, in sequential decision problems EA algorithms discard most of the information by not looking at the whole state-action trajectories the agents encounter throughout their lifetime. This theoretical disadvantage limits their potential efficiency to tackle sequential problems when compared with RL. Empirically. EA algorithms
96
97
98
99
100
101
102
usually have a higher variance when compared with gradient methods [30 23 24]. With regards to gradient methods (deep learning methods in particular), impressive results have been recently achieved by training NN, through mack-propagation, to compete against each other in simulated games
103
104
OOpenFive 4. AlphaZero (3I, GAN (TI). More closely aligned with our proposed methodology. APenAl has recently developed Neural MMO 37 simulated environment that captures some important properties of life on Earth. Neural MMO artificial agents, represented by NN, need to forage for food and water to survive in never-ending simulation. Currently, Neural MMO agents can not reproduce and their goal is to maximise their own survival, instead of maximising the survival and reproduction success of their genes as it happens in nature. We extend this work by introducing genes, the ability for agents to reproduce and we align the agents' reward with evolution. These are key properties of life on Earth that we must have in simulation environments if we hope to have them evolve similar solutions to the ones evolved by nature (in other words, these are key properties to achieve convergent evolution see Appendix ?? for more details on why this important for AI
105
106
107
108
109
110
111
112
113
115
research).
116
Cooperative MARL Cooperative MARL is an active research area within RL that has been 3 9. The setting is usually approached in a binary way 4 41 20).
117
experiencing fast progress 226
118
Agents are grouped into teams and agents within the same team fully cooperate amongst each other whilst agents from different teams don't cooperate at all (cooperation is either one or zero); we define this scenario as the binary cooperative setting. The teams may have a fixed number of members or change dynamically [7
119
120
121
40 [ The most straightforward solution for this setting would be to train independent learners to maximise their team's reward. However, independent learners would face a non- non-stationary learning problem. The MADDPG [22] algorithm tackles this problem by using a multi-agent policy gradient method with a centralised critic and decentralised actors so that training takes into account all the states and actions of the entire team but during execution each agent can
122
123
124
125
126
act independently. More relevant to our work, factored value functions [1] 27 such as Transfer Planning 40) Value Decomposition Networks (VDN) 3S and Q-Mix (28) use different methods to decompose the team's 's central action-value function into the decentralised action-value functions. We
128
build on top of VDN (which is further explained in the Appendix D) to extend the concept of team to the concept of family and introduce continuous degrees of cooperation.
130
3 Background
131
132 Reinforcement Learning We recall the single agent fully-observable RL setting [39]. where the enviromment is typically formulated as Markov decision process (MDP). At every time step, 134 t = .,2,.... the agent observes the environment's 's state 8, E S, and uses it to select an action d, E A. As a consequence, the agent receives reward r, E R C R and the environment transitions to the state 136 8+1-- The tuple ($t+1, re) is sampled from the static probability distribution p S x A + P(S x R) whilst the actions a sampled from the parametric policy function na S + P(A):
St41, p(5t+1, Felst, az), ae - mp(a(s))
(I)
138
The goal of the agent is to find the optimal policy parameters 0" that maximise the expected return R = E 7rr,), where 7 is the discount factor. In the more general framework, the state is
139
only partially observable, meaning that the agent can not directly observe the state but instead it
3

---

141
observes E O which is typically given by a function of the state. In this situation, the environment modelled by partial observable Markov decision process (POMDP) and the policy usually incorporates past history 14- = "..0"",...::0 ".-10-0----
142
148
144
Q-Lennning and Deep Q-Networks The action-value function Q" gives the estimated return when the agent has the state history hi, executes action and follows the policy on the future time steps. can be recursively defined by (h2, oz) Er.,,.r t ~ا 1Q" (he+1. at+1)]] Q- learning and Deep Q-Networks (DQN) [2] are popular methods for obtaining the optimal action value function Q*. Once we have Q", the optimal policy also available as n*' = arg maxa. (he, ae).
145
146
147
148
149
In DQN, the action-value function is approximated by a deep NN with parameters 0. Q is found by minimising the loss function:
150
(0) = ER, Fe I(ve - Q (12, };)) where yt = = + max Q7. (41+1,he11);
(2)
151
where π is the e-greedy policy which takes action arg maxa, (= ((ae;,, h4) with probability 1 1 e, and takes random action with probability 0' the parameters of target network that are periodically copied from 0 and kept constant for a number of iterations.
152
153
Multi- Agent Reinforcement Learning In this work, consider the MARL setting where the underlying environment is modelled by a partially observable stochastic game [3] In this setting, the environment is populated by multiple agents which have individual observations and rewards and act according to individual policies. Their goal is to maximise their own expected return.
155
156
157
158
4
Evolution via Evolutionary Reward
159
In this section, we propose reward function that enables RL algorithms to search for policies with increasingly evolutionary success. We call this reward the evolutionary reward because it is always aligned with the fitness function. We also propose a specific RL algorithm that is particularly suited to maximise the evolutionary reward in open-ended evolutionary environments however other RL algorithms could also be used.
180
161
162
163
184
Evolutionary reward The evolutionary reward of an agent is proportional to the number of copies its genes have in the world's population. Maximising this reward leads the maximisation of the survival and reproduction success of the genes an agent carries. We start by defining the kinship function between pair of agents i and j, who carry N genes represented by the integer vectors g'
165
186
167
168
and gd (we chose to use g for genome, which in biology is the set of genes an agent carries):
N
k:
x
[0, 1].
k(g'.g g') =
N
1
Σ
9
(3)
p=1
where 5 oj. is the Kronecker delta which is one if 9h = oa and zero otherwise. When agent i is alive 170 at time + receives the reward:
189
Σ k(g'.g')
(4)
SEALI
171 where Art1 is the set of agents alive at the instant + 1. Note that since agent i alive at / + 1,
A4t1 includes agent i. 2 1 is the last time step that agent i is alive and so, at this instant, the agent
173 receives its final reward which is proportional to the discounted sum of the number of times its genes
will be present other agents after its death:
Σ k(g', g')).
(5)
JEAL
175 with this reward function, the agents are incentivised to maximise the survival and replication success
176 of the genes they carry. In the agent-centered view, the agents are incentivised to survive and replicate, but also to help their family (kin) survive and replicate; and to make sure that when they die their
177
178 family is in a good position to carry on surviving and replicating. The degree of collaboration with
other family members depends the overlap between their genotype it happens in nature.

---

180
The discount factor, needs to be in the interval [o, 1[ to ensure the final reward remains bounded. Due to the exponential discounting we can compute the final reward up to an error of e by summing over finite period of time denoted by the effective horizon (hr)). To see how to compute the he for
181
182
183
a given environment and 6 see the Appendix [.I] By computing the final reward this way, we can
184
now use RI algorithms like Q-learning to train agents with this evolutionary reward. However, in the
185
next section we introduce a more practical algorithm that allows us to estimate the final reward more efficiently.
186
187
Evolutionary Yalue-Decomposition Networks We propose Evolutionary Value- Deaieeeeooosstiinnnn Networks (E-VDN) as an extension of VDN (38] (explained in the Appendix D) from the binary cooperative setting with static teams to the continuous cooperative setting with dynamic families. E-D helps us reduce the variance of the value estimation and allows us to estimate the final evolutionary reward without having to simulate the environment forward for he iterations.
188
189
190
191
192
Within a team, each agent fully cooperates with all the other members of the team, and it does not cooperate at all with any agent outside of the team. Moreover, if a and b are members of the same team and è is a member of a's team then c and b are also in the same team. Within a family, the degrees of cooperation amongst its members depends on their kinship degree (which can be any real number from 0 to 1). Also, if a and b are members of the same family and e is part of a's 's family, c is not necessarily part of b's family.
193
194
195
196
197
198
Fach agent/ sees the members of its family from an unique perspective, based on the kinship degree it shares with them. In E-VDN, each agent has joint action-value function, Q'. E-VDN assumes Q' can be composed by averaging the actien-value functions across the members i's family weighted by their kinship with agent (this is similar to the VDN's assumption):
199
200
201
(' Re,...., 11440 ), (a) ol of (
1 Σ Tie JEAt
kig' "od'g''V Q (
(6)
where ntf is a normalisation coefficient defined as nł = E)EA, k(g;, ,g)). ) Q is the output of a NN with parameters oy and with the input (he ).). Composing Q' with an average, instead of sum as it happens in VDN, is necessary as E-VDN allows the number of value functions contributing to the composition to vary as the family gets bigger or smaller (agents born and die). This averaging
202
203
204
205
206
allows us to incorporate the local observations of each family member and reduce variance in the
207
value estimation.
208
More importantly, E-VDN allows us to deal with the difficulty of estimating the final reward (5) in a particularly convenient way. As is clear from its definition (5), the final reward is the expected sum (over time) of kinship that agent i has with other agents j after its death. The key idea is to note that this value (r) can be approximated by the Q-value of other agents j that are close to (have high kinship with) agent i:
209
210
211
212
{
nyy ESEAp k(g'. "'''.g')&&p (p..p.....) =0 Pr.(...) if nf >
if n/. = 0
The final reward is zero if, and only if, at the time of its death the agent has surviving family. Each Qi is trained by back-propagatinn gradients, gf. from the Q-learning rule: (2) (1), af d(0,))² = vo,(i! - Q(()).10))),
214
gi = (v
1 =
Σ M(g', "lo'.g'V JEA.
(8)
where 0, is the concatenation of all the parameters θz. used in each Q contributing the estimation
215
216 of Q': i.e. 0: = {0,} } k(gs.))) Note that Q' are neural networks with parameters 0, and Q' is
simply the average stated in 66.
The learning targets vf given by:
218
r maxa4.. (i+1(..))00)
ift T -1 ift ift= T
y
=
rf is the evolutionary reward A/._ is the estimate of the final evolutionary reward 7 and 0 are the parameters of the target network that get periodically copied from 0,. We don't 't use a replay
5

---

50x50 World
5x5 Agent's Vision
Food ovallable
Gonenata nete food caparity
Food
Agent's lifecycle
Figure 1: The binary environment.
221
buffer in our training (which is commonly used in DQN) due to the non-stationary of multi-agent environments (more about this in the Appendix G.2).
222
Since the joint action-value Q' increases monotonically with increasing Q' an agent acting greedily with respect its action-value function will also act greedily in respect to its family action-value function: arg max,; Q.(...) ss arg max,; Q'(ht, af).
223
Experimental Setup
226
We want to test two hypotheses: 1) E-VDN is particularly well suited make agents climb the fitness landscape in open-ended evolutionary environments; 2) E- VDN is able to increase the evolutionary fitness of agents in non-binary cooperative environments. To test the first hypothesis compare E-VDN with another popular evolutionary algorithm. To make it easier to implement the are going to use binary cooperative environment to test the first hypothesis. To test the second hypothesis we will use a non-binary cooperative environment. Note, if an agent carries more than one gene (like happens in nature) we have a non-binary environment.
228
need to
230
competing algorithm
232
233
234
In this section, we give a quick overview of these two multi-agent environments, as well as details of the network architectures and the training regime. For more complete description of the environments, you can refer to the Appendix E. In the binary environment, we compared our algorithm with popular Evolution Strategies algorithm (CMA-ES (14), and describe the training regime used for CMA-ES in the Appendix F
236
238
239
The Binary Environment The binary environment a 2-dimensional grid world, which is ini- tialised with five agents carrying five unique genomes (Figure DT. At each time step, each agent may move one step and produce an attack another agent an adjacent tile. When an agent moves to a tile with food it collects all the food available in it. If an agent chooses to produce an attack, it decreases its victim's health by one point, if the victim's 's health reaches zero it dies and 50% of its collected food is captured by the attacker. The food is used to survive (one unit of food must be consumed every time step to remain alive), and to reproduce. When agents are within their fertile age and they have stored enough food, they reproduce themselves asexually and give birth to an agent carrying an exact copy their genome. Each genome has only a single gene and there are no mutations. These rules make the cooperation between agents binary, agents either fully- cully-cooperate (they have the exact same genome) or they don't 't cooperate at all (their genome has no overlap).
240
241
242
243
245
247
249
250
The Non-binary Environment The non-binary environment has the same rules the binary environment with the difference that the agents now have 32 genes in their genome and they reproduce sexually instead of asexually. When two fertile agents are adjacent, they give birth to an agent who's genome is composed by two halves of the genes of each parent, selected randomly. There are no genders, any agent can reproduce with any other agent. These rules give rise to different levels of collaboration: from 0 to I in steps of 32.
251
252
253
254
255
256
Each agent observes a 5x5 square crop of the surrounding state (Figure D The agent sees six features for every visible tile; i.c the input is 5x5x6 tensor. This includes two features
Policy

---

258
corresponding to tile properties (food available and whether it is occupied or not) and four features corresponding to the occupying agents' properties (age, food stored, kinship and health). Besides these local inputs, each agent also observes its absolute position, family size and the total number of agents in the world. We intend to remove these extra inputs in future work as we provide agents with memory (we're re currently providing our policy with of instead of ht). The NN has ten outputs (five movement actions with no attack and five movement actions with an attack). In this work, we used two different feed forward architectures: one is simply fully connected NN with three hidden layers and 244, 288 parameters in total, the other architecture is composed by convolutional and dense layers and it is much smaller containing only 23, 616 parameters. The smaller NN was used to compare our algorithm with an evolutionary algorithm which doesn't 't scale well to larger networks.
259
261
262
263
265
287
268
Training details In this work, the genome does not directly encode the policy, however, we think it would be interesting to do that in future work. In the binary environment, we train five different policies (with the same architecture but different weights) simultaneously. At each training episode, we sample five policies with replacement and assign each one to one of the five unique genomes. We do this, to force each policy to interact with all other policies (including itself), increasing their robustness in survival and reproduction. During the test episodes, no sampling occurs, each policy is simply assigned to each unique genome. The training episodes had a length between 450 and 550 (note that the reward is computed as if there was no episode end), and the test episodes had length of 500 steps.
269
270
272
274
276
277
In the non-binary environment, due to the large number of unique genomes, it is unfeasible to assign a unique policy to each unique genome. To keep things simple, we chose to use only one policy in this environment. This was not possible to do with CMA-ES, so we did not implement it in this environment (more about CMA-ES on Appendix
278
279
280
Traits encoded by the genes In the non-binary environment, we can think of each of the 32 genes to change some visual feature (e.g. facial feature) of their agent so that it can be better recognised by its family. In the binary environment, besides the gene encoding this visual feature it also encodes which policy, chosen from a set of 5 policies, the agent is going to have. Note that the genes encode fixed traits (they don't i change during an agent's ' lifetime) and their frequency in the population evolve through normal evolution (death and birth). With EvER we don't 't need evolution to create the reward function and continuously align it with the fitness function. The agent's brain is always trying to learn
282
283
284
285
286
287
288
the right things for the survival of its genes, however, the actual genes are evolving at the normal pace of evolution.
289
290
To analyse the impact of our reward function, we deliberately chose to minimise entanglement between genes and other aspects of the agents. However, EvER can be easily used in environments where genes encode more traits like the agent's abilities, visual features, initial weights and the topology of its policy.
281
292
293
Evaluation Metrics In our simple environments, fitter policies can use the environment resources more efficiently and increase their population size to larger numbers. Therefore, to evaluate the performance of the algorithms in generating increasingly fitter species we track the average population size along training time.
295
297
6
Results
299
Training agents with E-VDN generates quite an interesting evolutionary history. Throughout the binary environment history, we found four distinct eras where agents engage in significantly distinct behaviour patterns (1 row of fig. 2) In the first era (the blue line which lasts only a few hundred iterations), the agents learned how to survive, and through their encounters with the other founding
300
301
302
303
agents, they have learnt that it was always (evolutionary) advantageous to attack other agents. In the second era (orange line), the agents' food- good-gathering skills increased to point where they started to reproduce. In this era, the birth-rate and population numbers increased fast. However, with the extra births, fntralfamily encounters became more frequent, and intra-family violence rose to its all-time
304
305
306
maximum driving the average life span down. This intra-family violence quickly decreased in the
308
third era (green line), as agents started to recognize their kin. Kin detection allowed for selective
kindness and selective violence, which took the average life span to its all-time maximum. Finally,
7

---

180
1980
80
50
3003
80
CHA E4
300
25
120
135
75
30 75 100
95
50
125
15 tnactoun)
100
125
Jours)
Wallina (000))
Figure 2: (1st row) Results obtained using E-VDN with the larger NN, each point was obtained by averaging 20 test episodes. The different colours correspond to different eras. This plot was generated with a denser version of the evolutionary reward (more details on the Appendix G.3) (2nd row) Results obtained using CMA-ES and E-VDN algorithms with the smaller NN and the standard evolutionary reward
Both algorithms were trained with 20 CPUs each.
310
in the fourth era (red line), agents learned how sacrifice their lives for the future of their family, Old infertile agents started allowing the younger generation to eat them without retaliation. Through this cannibalism, the families had found system for wealth inheritance. A smart allocation of the family's food resources in the fitter generation led to an increase in the population size with the cost of a shorter life span. This behaviour emerges because the final reward (5) incentivises agents to plan for the success of their genes even after their death. This behaviour is further investigated in the Appendix I. These results show that optimising open-ended evolutionary environments with E-VDN does indeed generate increasingly complex behaviours.
311
312
313
314
315
316
317
The 2nd row of Figure 2 shows the
statistics obtained by training the smaller NN with CMA- ES and E-VDN, From the figure, we observe that E-VDN is able to produce a larger population of agents with longer life-span and a higher birth rate. A small population means that many resources are left unused by the current population, this creates an opportunity for a new and more efficient species to collect the unused resources and multiply its numbers. These opportunities are present in the CMA-ES environment, however the algorithm could not find them, which suggests that E-VDN better at finding the way up the fitness landscape than CMA-ES, Video IJ shows that each family trained with CMA-ES creates a swarm formation in a line that moves around the world diagonally.
319
320
321
322
323
324
325
326
When When there is only one surviving family, this simple strategy allows agents to only step into tiles that have reached their maximum food capacity. However, this is far from an evolutionarily stable strategy [35] (ESS; i.e. strategy that is not easily driven to extinction by a competing strategy), as we verify when we place the best two families trained with CMA-ES on the same environment as the best two E-VDN families and observe the CMA-ES families being consistently driven quickly to extinction by their competition (fig. a of Appendix B
327
328
330
332
Our results, in the non-binary environment, show that in a non-binary cooperative setting E- E-VDN also improves the ability of the trained policy to survive and replicate its genes (Figure b,c and d of Appendix B). This is a key feature that evolutionary algorithms should have in order to take the research in open-ended evolutionary environments further. Note, that the non- -i-biaary environment much harder than the binary one. To replicate, agents need to be adjacent to other agents. In the beginning, all agents are unrelated making it dangerous to get adjacent to another agent as it often
333
334
325
336
337
leads into attacks, but is also dangerous to get too far away from them since with a limited vision is hard to find fertile mate once they lose sight of each other. Video 2) shows simulation of the evolved policy being run on the non-binary environment, it seems that agents found a way to find mates by moving to certain region of the map (the breeding ground) once they are fertile.
338
340
7
Conclusion & Future Work
This paper has introduced an evolutionary reward function that when maximised also maximises the evolutionary fitness of the agent. This allows RI. to be used as a tool for research of open-ended evolutionary systems. To implement this reward function, we extended the concept of team to the
344
8

---

348
concept of family and introduce continuous degrees of cooperation. Future work could explore three directions: 1) Explore different reward function that makes agents maximise the expected geometric
347
growth rate of their genes; 2) Research the minimum set of requirements to emerge natural cognitive
349
abilities in artificial agents such as identity awareness and recognition, friendship and hierarchical
status (by following our proposed methodology for progress in Al (Appendix ??)) 3) Extend the use
351 of genes to encode more fixed traits the agent like its initial weights and the topology of its policy.

---

Broader Impact
352
353
Simulating the key processes that generated life and intelligence in nature promising path to further our understanding in this field and unlock ever more intelligent algorithms able to solve useful problems for the world. However, embodying Al with the goal to survive and self-reproduce can be dangerous, and should never be done outside of a sand-boxed environment.
354
355
357 References
[1]
David Ackley and Michael Littman. Interactions between learning and evolution. Artificial life II, 10:487-509, 1991.
359
360
[2] BURT Austin, Robert Trivers, and Austin Burt. Genes in conflict: the biology of selfish genetic elements. Harvard University Press, 2009.
361
382
[3] Trapit Bansal, Jakub Pachocki, Szymon Sidor, Ilya Sutskever, and Igor Mordatch. Emergent complexity via multi-agent competition. arXiv preprint 170.074710 2017.
363
364
[4] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv: 1912. 06680, 2019.
386
[5] Wendelin Böhmer, Vitaly Kurin, and Shimon Whiteson. Deep coordination graphs. arXiv preprint 1910000910 2019.
368
369
[6] Richard Dawkins. The selfish gene. 1976.
Richard Dawkins. Twelve misunderstandings of kin selection. Zeitschrifi für Terpsychologie, 51(2):184-000 1979.
[8] Stefan Elfwing, Eiji Uchibe, Kenji Doya, and Henrik I Christensen. Co-evolution of shaping rewards and meta- meta--arrmettes in reinforcement learning. Adaptive Behavior, 16(6):400-412, 2008.
373
375
[9] Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Shirot-Second AAAI Conference Artificial Intelligence, 2018
377
[10] Andy Gardner and Francisco Úbeda. The meaning of intragenomic conflict. Nature ecology & evolution, 1(12): 1807-185851 2017.
379
[11] lan Goodfellow, Jean Pouget-Abadic, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672-2680, 2014.
381
383
[12] Carlos Guestrin, Daphne Koller, and Ronald Parr. Multiagent planning with factored mdps. In Advances in neural information processing systems, pages 1523-1530, 2002.
385
[13] Eric Hansen, Daniel S Bernstein, and Shlomo Zilberstein. Dynamic programming for partially observable stochastic games. In AAAI, volume 4, pages 709-715, 2004.
386
[14] Nikolaus Hansen, Sibylle D Müller, and Petros Koumoutsakos. Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (cma-es). Evolutionary computation, 11(1):1-18, 2003.
388
390
[15] Nikolaus Hansen, Youhei Akimoto, and Petr Baudis. CMA-ES/pyema on Github. Zenodo, 1ttt100.5281/2mmooo22555533 February 2019. URL http:::/ttt..orggg/geoooo 5ttp///o.22//eeodo
392
2559634
[16] Verena Heidrich-Meisned and Christian Igel. Evolution strategies for direct policy search. In International Conference on Parallel Problem Solving from Nature, pages 428-437. Springer,
394
2008.
10

---

396
[17] Verena Heidrich-MMisser and Christian Igel. Variable metric reinforcement learning methods applied to the noisy mountain car problem. In European Workshop on Reinforcement Learning. pages 136-150. Springer, 2008.
397
[18] Verena Sealudrich-Meinnee and Christian Igel. Hoeffding and bernstein races for selecting policies evolutionary direct policy search. Proceedings of the 26th Annual International Conference on Machine Learning, pages 401-408. ACM, 2009.
400
[19] Jelle R Kok, Eter Jan Hoen, Bram Bakker, and Nikos Vlassis. Utile coordination: Learning interdependencies among cooperative agents. In EEE Symp. on Computational Intelligence and Games, Colchester, Essex, pages 29-36, 2005.
403
[20] Karol Kurach, Anton Raichuk, Piotr Stanczyk, Michal Zajac, Olivier Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, et al. Google research football: A novel reinforcement learning environment. arXiv preprint arXiv:1907. 11100, 2019.
406
407
408
[21] Richard L Lewis, Satinder Singh, and Andrew G Barto. Where do rewards come from? In Proceedings of the International Symposium on Al-Inspired Biology, pages 2601-2606, 2010.
409
410
[22] Ryan Lowe, Yi Wu, Aviv Tamar. Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi- agent actor-critic for mixed cooperative-compettive environments. In Advances in Neural Information Processing Systems, pages 6399, 2017.
411
412
[23] Simon M Lucas and Thomas P Runarsson. Temporal difference learning versus co-evolution for acquiring othello position evaluation. In 2006 IEEE Symposium on Computational Intelligence and Games, pages 52-59. IEEE, 2006.
414
[24] Simon M Lucas and Julian Togelius. Point-to-por tointtt car racing: initial study of evolution versus temporal difference learning. In 2007 iEEE symposium on computational intelligence and games, pages 260 267. IEEE, 2007.
417
[25] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
420
421
422
[26] Frans A. Oliehoek and Christopher Amato. Concise Introduction to Decentralized POMDPs.
423
SpringerBriefss in Intelligent Systems. Springer. May 2016. doi: 10.10077978-3-319-288 URL btatt:://wwar www.solienoneeeo...oo. http:e/e.....eeeeeeeeeeeeeeaaaaa
424
pdf
425
[27] Frans A Oliehoek, Shimon Whiteson, Matthijs TJ Spaan, et al. Approximate solutions for factored dec-pomdps with many agents. In AAMAS, pages 563-570, 2013.
426
[28] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: monotonic value function factorisation for deep multi-agent reinforcement learning. arXiv preprint arXiv: / 1485, 2018.
428
[29] Craig W Reynolds. Competition, coevolution and the game of tag. In Proceedings of the Fourth International Workshop on the Syuthesis and Simulation of Living Systems, pages 59-69, 1994.
431
432
[30] Thomas Philip Runarsson and Simon M Lucas. Coevolution versus self-play temporal difference learning for acquiring position evaluation in small- hoald-bbrddddd go. IEEE Transactions on Evolutionary Computation, 96):628-640 2005.
438
434
435
[31] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, (22(6419)11100-11 2018.
436
437
438
439
[32] Karl Sims. Evolving 3d morphology and behavior by competition. Artificial life, 1(4):353-372 1994.
440
[33] Satinder Singh, Richard I. Lewis, Andrew G Barto, and Jonathan Sorg. Intrinsically motivated learning: An evolutionary perspective. IEEE Trunsactions on Autonomous Mental Development, 2(2):70-82, 2010.
442
reinforcement reinforeementllamminng
11

---

444
[34] J Maynard Smith. Group selection and kin selection. Nature, 201(44) 114,, 1964.
[35] J Maynard Smith and George R Price. The logic of animal conflict. Nature, 246(5427):15,
446
1973.
447
[36] Kenneth O Stanley and Risto Miikkulainen. Evolving neural networks through augmenting topologies. Evolutionary computation, 10(2):99-127, 2002.
448
449
[37] Joseph Suarez, Yilun Du, Phillip Isola, and Igor Mordatch. Neural mmo: A massively multiagent game environment for training and evaluating intelligent agents. arXiv preprint arXiv:1903.0074. 2019.
450
451
452
[38]
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojeiech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z. Leibo, Karl Tuyls, et al. Value- decomposition networks for cooperative multi-agent learning. arXiv preprint arXiv:1706.05290 2017.
454
456
[39]
Richard S Sutton, Andrew G Barto, et al. Introduction to reinforcement learning, volume 2. MIT press Cambridge, 1998.
457
458
[40]
Elise Van der Pol and Frans A Oliehoek. Coordinated deep reinforcement learners for traffic light control. Proceedings of Learning, Inference and Control of Multi-Agent Systems (at NIPS 2016), 2016.
459
480
[41] Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Jun- young Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350-33) -544, 2019.
462
[42] Shimon Whiteson and Peter Stone. Evolutionary function approximation for reinforcement learning. Journal of Machine Learning Research, 7(May)8877-917, 2006.
485
486
[43] George C Williams. Adaptation and natural selection: a critique of some current evolutionary thought, volume 833082108. Princeton science library OCI.C, 1966.
467
468
1. For all authors...
469
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
470
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
472
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
473
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
474
475
2. If you are including theoretical results...
476
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
477
(b) Did you include complete proofs of all theoretical results? [Yes]
478
3. If you ran experiments..
479
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [No]
481
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
482
483
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No]
484
485
(d) Did you include the total amount of compute and the type of resources used (e.g., type
486
of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
488
(a) If your work uses existing assets, did you cite the creators? [N/A]
489
(b) Did you mention the license of the assets? [N/A]
490
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
12

---

492
(d) Did you discuss whether and how consent was obtained from people whose data you' 'e using/curating? [N/A]
493
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
495
496
5. If you used crowdsourcing or conducted research with human subjects...
497
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
489
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
502
13


Reviews:
[{"review_id": "smCTNIxPL0V", "reviewer": "Reviewer_KvmB", "summary": "** Score revised after reading through initial author comments **\n\nThe papers compares and contrasts evolutionary processes with learning processes and discusses some of the links between the two in reinforcement learning problems. Subsequently, the authors describe some previous work on combining evolution with learning processes and set up a reward function that encourages the maintenance propagation of agents in the kin based on genetic encodings. The authors hypothesize that this reward function is provides a useful learning signal in open-ended environments. According to their review of previous work, this reward function solves the alignment problem between learning and evolution by ensuring that the learning policy learns to take actions that encourage the survival of a set of agents of similar kin. Next, the authors formulate a method to optimize that particular reward function named E-VDN, which is an extension of value decomposition networks with E-VDn taking into account the gene marker make-up of a particular agent. The authors then show experiments in a set of multi-agent reinforcement learning environments and analyze the resulting behavior of their trained agents and compare the results of their methods to agents trained with CMA-ES.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Nits:\n* Line 114 and Line 350 have Appendix ?? in the text\n\nI think that the paper introduces an interesting notion that one can encourage evolution like behavior in reinforcement learning by formulating a reward and framework that encourages the propagation of one's genetic code. In its current form, however, I think that paper still has a couple of major weaknesses relating to the clarity of some of the narrative, a limited set of experiments in only a set of particular environments with comparison to only one other method.\n\n**Originality:** I think that the authors' idea of introducing a novel reward formulation and method to optimize that particular reward with a prescribed encoding of genetic markers into the method is a promising idea that could lead to interesting research directions.\n\n**Quality:** I think that paper in its current form provides a lot of detail in the description of the method and the thought process behind it, but it does fall short on description of some of the RL specific settings pertaining to the experiments. I also think that paper could be significantly strengthened if additional experiments would be conducted to further tests some of the authors' hypothesis. Some specific suggestions:\n* Provide a more thorough description of the MARL setting and its definition and it is particularly suited for your method. The MARL setting is different from the traditional RL setting given multiple agents, so it would be good to have more detail on that in the paper.\n* Conduct more experiments in different environments that test your hypothesis further. The authors specifically mention OpenAI's Neural MMO, which would probably be a good choice for additional experiments. Further experiments comparing to other methods in MARL settings would further strengthen the paper, especially if those methods include evolution+RL algorithms, such as \"Majumdar, Somdeb, et al. \"Evolutionary Reinforcement Learning for Sample-Efficient Multiagent Coordination.\" International Conference on Machine Learning. PMLR, 2020.\" which also claims to use evolution to coordinate in multi-agent settings.\n* Provide a  discussion on the limitations of your work that can address some of the following questions: In what kind of open-ended problems do you think the proposed formulation holds promise and in which ones does it not? It appears that your formulation specifically targets survival and reproduction, which I would argue is not ideal in all open-ended RL settings.\n\n**Clarity:** The technical portion of the paper is relatively clear. Yet, I think that the background section (Section 3) could be significantly improved by providing more clarity and descriptions of the MARL setting. It might also be helpful to have a diagram that compares and contrasts the authors' method to other method, which could be combined with Figure 3 in the appendix.\n\n**Significance:** I think that scope of the authors claims could be substantially improved with additional experiments in other environments and in comparisons with other methods. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The discussion on limitations and societal impacts is pretty limited. Given that the current version still has almost an entire page left, I think it would be good to use the remaining space to address limitations and provide a more thorough discussion on potential impacts. Some things to think about could be:\n* What kind of problems can this formulation solve? What kinds can it not solve?\n* What could some potential drawbacks and mitigations to having more and more intelligent agents interacting with each other and humans in the real world?\n* How expensive is it to train the current framework and how would that scale to larger, more complicated problems?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "kwjWoP8xg1", "reviewer": "Reviewer_2ceV", "summary": "The article proposes an evolutionary simulation where agents use reinforcement learning to optimise fitness (in the sense of natural selection, number of genes in the population). Some of the ideas are interesting, but overall the contribution is limited, the applicability of the results is narrow, and the comparisons against evolutionary algorithms is not particularly fair. Fundamentally, it appears as though the authors have created a simulation where _greenbeard_ genes (see Dawkins 1976) are favoured. Basically, attacking individuals pays off if and only if the cost-benefit ratio exceeds their relatedness. This is Hamilton's rule, and a greenbeard makes it so that the relatedness is observable. This is at least the case of their \"binary environment\" and there are not enough details in the other environment to discern if this is also the case, but some language seems to suggest it is. This is not a novel insight in evolutionary biology, and seeing it work in reinforcement learning agents is not interesting when it is set as the _explicit reward_. Were this to emerge, then it would be much more interesting, particularly without observed relatedness, but also significantly more challenging.", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Originality\n\nThe article provides a very simple 2D environment where agents can move around, gather resources, possibly by attacking other agents, and reproduce. There are many other 2D environments with similar settings, and in the field of ALife that the authors mention, there are many explorations of the dynamics of these. The key difference here is that they are using RL agents, and that they are setting the reward signal as the evolutionary fitness of the individuals. This shows some interesting evolutionary dynamics with distinct phases, but the overall setting is very limited, and not clear what the actual advancement of the state of the art would be. All results are completely predicted by evolutionary theory, and the supposed improvement in computational complexity is simply due to setting the fitness as the reward.\n\n## Quality\n\nThe work should be much better contextualised with the relevant literature in ALife and Evolutionary Biology. both of which have a long history of understanding cooperation.\n\n## Clarity\n\nThe article would benefit greatly from a rewrite, as many claims and concepts spanning evolution and reinforcement learning are thrown together with little context or justification. For instance, halfway through the abstract, the authors claim that optimisation of a reward function to align with evolution is computationally expensive. It is unclear what literature the authors are referring to. There have been a series of articles on using, for instance, population dynamics and evolutionary theory as optimisers (e.g. \nOmidshafiei et al. Neural Replicator Dynamics, arXiv:1906.00190). Another example: the authors claim that learning maximises pleasure and minimises pain... but this is unclear what it refers to. And clearly not true for most living organisms. Perhaps the authors are claiming this is true for all Chordata? Even so, it would be controversial to say any learning is about this.\n\nThe authors go to great lengths to talk about how alignment of reward to evolutionary fitness is hard, and the all one can hope to achieve is some partial alignment. However, the main contribution of the article is literally just computing the evolutionary fitness and turning that into the reward of the agents. At the end of section 1.1, the authors claim that one requires the use of simulation because the reward is extrinsic to the agent. It is unclear what the authors mean by this.\n\n## Significance\n\nThe results are expected by previous theory, which limits the utility of the contribution. The computational improvement claimed requires changing the reward signal to match the desired outcome (evolutionary fitness), which negates the interestingness of the approach.\n\n## Detailed comments\n\nIn Section 1.2, the authors should cite Hamilton's 1964 papers when referring to _kin selection_. Although Maynard-Smith popularised the term, the theory behind it was developed by Hamilton.\n\nIn _Combining evolution and learning_ the authors should probably cite Wang, et al 2018. Evolving intrinsic motivations for altruistic behavior, arXiv:1811.05931.\n\nIn lines 99-100, the authors claim that EAs must discard most of the state-action trajectories. But this is not intrinsically true. It might be that this has been a practical choice, particularly for simple iterated social dilemma scenarios, but definitely not a hard limitation.\n\nIn line 114, I'm not sure what the authors mean by _convergent evolution_ being key properties of life on Earth. The authors seem to suggest that convergent evolution is somehow synonymous with cooperation?\n\nThe intuition behind their reward function comes very late. Only in section 4, already in page 4, does it get introduced. This needs to be put front and centre, as this is the main contribution of the paper. Also, the reward function used is likely to lead to the \"lazy agent\" effect (see Sunehag 2017, and Rashid 2018, cited in the article!), or if not, it should be discussed why.\n\nIn line 168, the authors use the word _genome_ where they probably mean _genotype_.\n\nIn lines 192-197 the authors use the word _team_ but referring to them as _cliques_ would be more appropriate. Also, the distinction between teams and families really ought to come earlier and be clearly explained.\n\nThe caption of the figures should have more information. They are stubs at the moment.\n\nIn section 5, the details need to be more clearly explained. The authors say things like: \"if an agent carries more than one gene [...] we have a non-binary environment.\" What does this mean? Surely the environment shouldn't be dependent on the agents in it. This would be better explained as two different experimental setups, rather than two different environments with some restrictions on the genetics of the agents that can participate in them.\n\nThere are not enough details to know, for instance, what the initial health of the agents is. Nor clear what the distribution of food in the environment is. I presume there must be substantially more than 1 unit per cell, since the agents require 1 unit per step just to survive. Also, it is not clear if the food replenishes, or at what rate. There are also not enough details on the payoffs of interactions. \n\nThe mechanics of movement and attacking are underexplained. The authors claim 5 actions that are pure movement (4 directions, + no-op) and also 5 actions with attack... But what does it mean to attack without moving? How would interactions get resolved if two agents attack simultaneously? Why would an agent not simply attack all the time it is not near a relative?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No, the limitations of the work should be more clearly stated.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "XeNb6n0wvH", "reviewer": "Reviewer_J3e1", "summary": "The paper is concerned with optimisation in an open-ended ALife environment.\nThe authors propose to directly maximise the number of current and future genetically similar agents via an adaptation of Value Decomposition Networks, named Evolutionary Value-Decomposition Networks (E-VDN).\nThis is in contrast to applying a bi-level optimization procedure where evolution optimises the genes for population size and the inner learning algorithm adapts the policy for an objective set by the outer loop.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Originality and Significance\n\nThe idea of directly optimising the number of genetically similar agents directly as the objective of an agent is an interesting and novel approach.\nThis could be an approach to largely reduce computational cost of bi-level evolutionary systems.\nThe authors introduce a value estimation method that allows doing so cheaply called E-VDN.\nIn the context of the mentioned objective, E-VDN allows estimating the future reward of an agent based on the rewards of similar agents.\nThis paper should be interesting to the Meta Learning, Evolutionary, and ALife NeurIPS community.\n\n# Great insights into dynamics of environment\n\nIn terms of their analysis, I particularly like the observed stages in the environment dynamics induced by their method.\nThe stages involve attacking other agents, learning to reproduce, recognising their kin in terms of the attacking behavior, and sacrificing their lives for the future of their genetically similar kinds.\n\n# Environment may not be open-ended\n\nThe authors often argue that their setup is open-ended.\nFrom their analysis, it seems that the generated complexity is not open-ended but converges.\nThis is common for papers that seek open endedness, but it should be further clarified that this goal is not reached.\n\n# The CMA-ES baseline is not used to optimise the genes but the policies\n\nThis is confusing to the reader because previously the word 'evolution' has always been used in the context of only optimising genes.\nIn the case of their CMA-ES baseline, the same 'evolutionary reward' for the inner loop is used with CMA-ES to optimise the policies for one particular gene.\nThe authors should clearly distinguish the use of the word 'evolution' in the two different contexts.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and potential negative societal impact is adequately discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Meta-review of "Mimicking Evolution with Reinforcement Learning"

In nature, and in computational evolution literature, evolution is usually the outer loop of optimization while RL is the inner loop. This bi-level optimization is computationally expensive. This work proposes a way to do both in tandem - the authors propose to directly maximise the number of current and future genetically similar agents. They test the concept on 2 bio-inspired open-ended grid world environments.

Reviews were mixed for this work. Summary of discussions:

2ceV, the most critical reviewer who assigned a score of 3, notes that the comparisons against existing EA methods is not particularly fair (this concern is shared with other reviewers). 2ceV's opinion is that this work is not representative of evolutionary biology since the reward is fixed but not emerged, but on this point the Author's rebuttal is that their intention is to speed up computation, rather than mimic evolutionary biology (precisely because the emergence of reward via bi-level optimization is painfully slow), and I do think their rebuttal is valid here.

J3e1 is most supportive of the work, assigning a score of 6. J3e1 appreciates the proposed approach of providing alternatives to bi-level optimization. Their main concern is that the environments may not be open-ended, and also for the experiments, noted that the "authors pointed out that EvER works only in the MARL setting and we don't have a suitable bi-level baseline to compare to. To resolve this issue, I would suggest to either move to the single agent setting (if possible) or create an appropriate baseline." As the authors noted in the rebuttal, perhaps no baselines exist since the problem is infeasible for MARL systems. I believe this point should be better communicated in the narrative to manage expectations.

KvmB revised the score to 5 after several discussions with authors and other reviewers. Like J3e1, KvmB notes the novelty of the work and appreciates the direction and usefulness of the reward formulation to mimic gene selection to be promising. The main issue they have is the quality of the experiments, which falls short of expectations (several ideas and suggestions are given in the review and back and forth discussion). I also think that if the experiments were stronger, then this work will have a high impact.

In the end, even if we give the authors the benefit of discounting reviewer 2ceV (who does have a few valid points regarding EA comparisons), the work is still borderline (5 and 6), based on the reviews from J3e1 and KvmB which covers the main issues with the paper. I'm recommending a rejection of this work, but I do think it is a promising and interesting direction, especially as a way to avoid bi-level optimization, and I encourage the authors to take the inputs given to improve it further.

Decision: Reject