Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: summary: The article proposes an evolutionary simulation where agents use reinforcement learning to optimise fitness (in the sense of natural selection, number of genes in the population). Some of the ideas are interesting, but overall the contribution is limited, the applicability of the results is narrow, and the comparisons against evolutionary algorithms is not particularly fair. Fundamentally, it appears as though the authors have created a simulation where _greenbeard_ genes (see Dawkins 1976) are favoured. Basically, attacking individuals pays off if and only if the cost-benefit ratio exceeds their relatedness. This is Hamilton's rule, and a greenbeard makes it so that the relatedness is observable. This is at least the case of their "binary environment" and there are not enough details in the other environment to discern if this is also the case, but some language seems to suggest it is. This is not a novel insight in evolutionary biology, and seeing it work in reinforcement learning agents is not interesting when it is set as the _explicit reward_. Were this to emerge, then it would be much more interesting, particularly without observed relatedness, but also significantly more challenging.
main_review: ## Originality

The article provides a very simple 2D environment where agents can move around, gather resources, possibly by attacking other agents, and reproduce. There are many other 2D environments with similar settings, and in the field of ALife that the authors mention, there are many explorations of the dynamics of these. The key difference here is that they are using RL agents, and that they are setting the reward signal as the evolutionary fitness of the individuals. This shows some interesting evolutionary dynamics with distinct phases, but the overall setting is very limited, and not clear what the actual advancement of the state of the art would be. All results are completely predicted by evolutionary theory, and the supposed improvement in computational complexity is simply due to setting the fitness as the reward.

## Quality

The work should be much better contextualised with the relevant literature in ALife and Evolutionary Biology. both of which have a long history of understanding cooperation.

## Clarity

The article would benefit greatly from a rewrite, as many claims and concepts spanning evolution and reinforcement learning are thrown together with little context or justification. For instance, halfway through the abstract, the authors claim that optimisation of a reward function to align with evolution is computationally expensive. It is unclear what literature the authors are referring to. There have been a series of articles on using, for instance, population dynamics and evolutionary theory as optimisers (e.g. 
Omidshafiei et al. Neural Replicator Dynamics, arXiv:1906.00190). Another example: the authors claim that learning maximises pleasure and minimises pain... but this is unclear what it refers to. And clearly not true for most living organisms. Perhaps the authors are claiming this is true for all Chordata? Even so, it would be controversial to say any learning is about this.

The authors go to great lengths to talk about how alignment of reward to evolutionary fitness is hard, and the all one can hope to achieve is some partial alignment. However, the main contribution of the article is literally just computing the evolutionary fitness and turning that into the reward of the agents. At the end of section 1.1, the authors claim that one requires the use of simulation because the reward is extrinsic to the agent. It is unclear what the authors mean by this.

## Significance

The results are expected by previous theory, which limits the utility of the contribution. The computational improvement claimed requires changing the reward signal to match the desired outcome (evolutionary fitness), which negates the interestingness of the approach.

## Detailed comments

In Section 1.2, the authors should cite Hamilton's 1964 papers when referring to _kin selection_. Although Maynard-Smith popularised the term, the theory behind it was developed by Hamilton.

In _Combining evolution and learning_ the authors should probably cite Wang, et al 2018. Evolving intrinsic motivations for altruistic behavior, arXiv:1811.05931.

In lines 99-100, the authors claim that EAs must discard most of the state-action trajectories. But this is not intrinsically true. It might be that this has been a practical choice, particularly for simple iterated social dilemma scenarios, but definitely not a hard limitation.

In line 114, I'm not sure what the authors mean by _convergent evolution_ being key properties of life on Earth. The authors seem to suggest that convergent evolution is somehow synonymous with cooperation?

The intuition behind their reward function comes very late. Only in section 4, already in page 4, does it get introduced. This needs to be put front and centre, as this is the main contribution of the paper. Also, the reward function used is likely to lead to the "lazy agent" effect (see Sunehag 2017, and Rashid 2018, cited in the article!), or if not, it should be discussed why.

In line 168, the authors use the word _genome_ where they probably mean _genotype_.

In lines 192-197 the authors use the word _team_ but referring to them as _cliques_ would be more appropriate. Also, the distinction between teams and families really ought to come earlier and be clearly explained.

The caption of the figures should have more information. They are stubs at the moment.

In section 5, the details need to be more clearly explained. The authors say things like: "if an agent carries more than one gene [...] we have a non-binary environment." What does this mean? Surely the environment shouldn't be dependent on the agents in it. This would be better explained as two different experimental setups, rather than two different environments with some restrictions on the genetics of the agents that can participate in them.

There are not enough details to know, for instance, what the initial health of the agents is. Nor clear what the distribution of food in the environment is. I presume there must be substantially more than 1 unit per cell, since the agents require 1 unit per step just to survive. Also, it is not clear if the food replenishes, or at what rate. There are also not enough details on the payoffs of interactions. 

The mechanics of movement and attacking are underexplained. The authors claim 5 actions that are pure movement (4 directions, + no-op) and also 5 actions with attack... But what does it mean to attack without moving? How would interactions get resolved if two agents attack simultaneously? Why would an agent not simply attack all the time it is not near a relative?
limitations_and_societal_impact: No, the limitations of the work should be more clearly stated.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper is concerned with optimisation in an open-ended ALife environment.
The authors propose to directly maximise the number of current and future genetically similar agents via an adaptation of Value Decomposition Networks, named Evolutionary Value-Decomposition Networks (E-VDN).
This is in contrast to applying a bi-level optimization procedure where evolution optimises the genes for population size and the inner learning algorithm adapts the policy for an objective set by the outer loop.
main_review: # Originality and Significance

The idea of directly optimising the number of genetically similar agents directly as the objective of an agent is an interesting and novel approach.
This could be an approach to largely reduce computational cost of bi-level evolutionary systems.
The authors introduce a value estimation method that allows doing so cheaply called E-VDN.
In the context of the mentioned objective, E-VDN allows estimating the future reward of an agent based on the rewards of similar agents.
This paper should be interesting to the Meta Learning, Evolutionary, and ALife NeurIPS community.

# Great insights into dynamics of environment

In terms of their analysis, I particularly like the observed stages in the environment dynamics induced by their method.
The stages involve attacking other agents, learning to reproduce, recognising their kin in terms of the attacking behavior, and sacrificing their lives for the future of their genetically similar kinds.

# Environment may not be open-ended

The authors often argue that their setup is open-ended.
From their analysis, it seems that the generated complexity is not open-ended but converges.
This is common for papers that seek open endedness, but it should be further clarified that this goal is not reached.

# The CMA-ES baseline is not used to optimise the genes but the policies

This is confusing to the reader because previously the word 'evolution' has always been used in the context of only optimising genes.
In the case of their CMA-ES baseline, the same 'evolutionary reward' for the inner loop is used with CMA-ES to optimise the policies for one particular gene.
The authors should clearly distinguish the use of the word 'evolution' in the two different contexts.
limitations_and_societal_impact: Limitations and potential negative societal impact is adequately discussed.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: ** Score revised after reading through initial author comments **

The papers compares and contrasts evolutionary processes with learning processes and discusses some of the links between the two in reinforcement learning problems. Subsequently, the authors describe some previous work on combining evolution with learning processes and set up a reward function that encourages the maintenance propagation of agents in the kin based on genetic encodings. The authors hypothesize that this reward function is provides a useful learning signal in open-ended environments. According to their review of previous work, this reward function solves the alignment problem between learning and evolution by ensuring that the learning policy learns to take actions that encourage the survival of a set of agents of similar kin. Next, the authors formulate a method to optimize that particular reward function named E-VDN, which is an extension of value decomposition networks with E-VDn taking into account the gene marker make-up of a particular agent. The authors then show experiments in a set of multi-agent reinforcement learning environments and analyze the resulting behavior of their trained agents and compare the results of their methods to agents trained with CMA-ES.
main_review: Nits:
* Line 114 and Line 350 have Appendix ?? in the text

I think that the paper introduces an interesting notion that one can encourage evolution like behavior in reinforcement learning by formulating a reward and framework that encourages the propagation of one's genetic code. In its current form, however, I think that paper still has a couple of major weaknesses relating to the clarity of some of the narrative, a limited set of experiments in only a set of particular environments with comparison to only one other method.

**Originality:** I think that the authors' idea of introducing a novel reward formulation and method to optimize that particular reward with a prescribed encoding of genetic markers into the method is a promising idea that could lead to interesting research directions.

**Quality:** I think that paper in its current form provides a lot of detail in the description of the method and the thought process behind it, but it does fall short on description of some of the RL specific settings pertaining to the experiments. I also think that paper could be significantly strengthened if additional experiments would be conducted to further tests some of the authors' hypothesis. Some specific suggestions:
* Provide a more thorough description of the MARL setting and its definition and it is particularly suited for your method. The MARL setting is different from the traditional RL setting given multiple agents, so it would be good to have more detail on that in the paper.
* Conduct more experiments in different environments that test your hypothesis further. The authors specifically mention OpenAI's Neural MMO, which would probably be a good choice for additional experiments. Further experiments comparing to other methods in MARL settings would further strengthen the paper, especially if those methods include evolution+RL algorithms, such as "Majumdar, Somdeb, et al. "Evolutionary Reinforcement Learning for Sample-Efficient Multiagent Coordination." International Conference on Machine Learning. PMLR, 2020." which also claims to use evolution to coordinate in multi-agent settings.
* Provide a  discussion on the limitations of your work that can address some of the following questions: In what kind of open-ended problems do you think the proposed formulation holds promise and in which ones does it not? It appears that your formulation specifically targets survival and reproduction, which I would argue is not ideal in all open-ended RL settings.

**Clarity:** The technical portion of the paper is relatively clear. Yet, I think that the background section (Section 3) could be significantly improved by providing more clarity and descriptions of the MARL setting. It might also be helpful to have a diagram that compares and contrasts the authors' method to other method, which could be combined with Figure 3 in the appendix.

**Significance:** I think that scope of the authors claims could be substantially improved with additional experiments in other environments and in comparisons with other methods. 
limitations_and_societal_impact: The discussion on limitations and societal impacts is pretty limited. Given that the current version still has almost an entire page left, I think it would be good to use the remaining space to address limitations and provide a more thorough discussion on potential impacts. Some things to think about could be:
* What kind of problems can this formulation solve? What kinds can it not solve?
* What could some potential drawbacks and mitigations to having more and more intelligent agents interacting with each other and humans in the real world?
* How expensive is it to train the current framework and how would that scale to larger, more complicated problems?
review_title: null

Meta-review: Meta-review of "Mimicking Evolution with Reinforcement Learning"

In nature, and in computational evolution literature, evolution is usually the outer loop of optimization while RL is the inner loop. This bi-level optimization is computationally expensive. This work proposes a way to do both in tandem - the authors propose to directly maximise the number of current and future genetically similar agents. They test the concept on 2 bio-inspired open-ended grid world environments.

Reviews were mixed for this work. Summary of discussions:

2ceV, the most critical reviewer who assigned a score of 3, notes that the comparisons against existing EA methods is not particularly fair (this concern is shared with other reviewers). 2ceV's opinion is that this work is not representative of evolutionary biology since the reward is fixed but not emerged, but on this point the Author's rebuttal is that their intention is to speed up computation, rather than mimic evolutionary biology (precisely because the emergence of reward via bi-level optimization is painfully slow), and I do think their rebuttal is valid here.

J3e1 is most supportive of the work, assigning a score of 6. J3e1 appreciates the proposed approach of providing alternatives to bi-level optimization. Their main concern is that the environments may not be open-ended, and also for the experiments, noted that the "authors pointed out that EvER works only in the MARL setting and we don't have a suitable bi-level baseline to compare to. To resolve this issue, I would suggest to either move to the single agent setting (if possible) or create an appropriate baseline." As the authors noted in the rebuttal, perhaps no baselines exist since the problem is infeasible for MARL systems. I believe this point should be better communicated in the narrative to manage expectations.

KvmB revised the score to 5 after several discussions with authors and other reviewers. Like J3e1, KvmB notes the novelty of the work and appreciates the direction and usefulness of the reward formulation to mimic gene selection to be promising. The main issue they have is the quality of the experiments, which falls short of expectations (several ideas and suggestions are given in the review and back and forth discussion). I also think that if the experiments were stronger, then this work will have a high impact.

In the end, even if we give the authors the benefit of discounting reviewer 2ceV (who does have a few valid points regarding EA comparisons), the work is still borderline (5 and 6), based on the reviews from J3e1 and KvmB which covers the main issues with the paper. I'm recommending a rejection of this work, but I do think it is a promising and interesting direction, especially as a way to avoid bi-level optimization, and I encourage the authors to take the inputs given to improve it further.

Decision: Reject