PyGeneses — A Deep Reinforcement Learning Framework to understand complex behaviours
By: Siddhartha Dhar Choudhury
Tags: Deep Learning, Pytorch, Evolution, Reinforcement Learning, Simulation

What is PyGeneses?

PyGeneses is a PyTorch-based Deep Reinforcement Learning (Deep RL) framework that helps users to simulate artificial agents in bio-inspired environments.

Let’s break this line into simpler terms:

Deep Reinforcement Learning: Deep Reinforcement Learning is a technique in Artificial Intelligence that brings together the best of Deep Learning and Reinforcement Learning. Deep learning deals with the concept of neural networks which are capable of learning complex functions from data, and Reinforcement Learning is based on the idea of learning from experience much like animals do in nature.

Deep Reinforcement Learning is a technique in Artificial Intelligence that brings together the best of Deep Learning and Reinforcement Learning. Deep learning deals with the concept of neural networks which are capable of learning complex functions from data, and Reinforcement Learning is based on the idea of learning from experience much like animals do in nature. Artificial agents: Artificial agents are simulated and simplified versions of species found in nature. The rules these agents follow closely mimic that of nature which allows them to display behaviour close to what animals around us (or even human beings) display. Note: These agents are nowhere close to the complexity of living beings in our planet and are just a simplified version of them.

Artificial agents are simulated and simplified versions of species found in nature. The rules these agents follow closely mimic that of nature which allows them to display behaviour close to what animals around us (or even human beings) display. These agents are nowhere close to the complexity of living beings in our planet and are just a simplified version of them. Bio-inspired environments: Bio-inspired environments mimic conditions of survival and rules that we see in our natural environment inside a simulation.

So, in very simple terms — PyGeneses is an attempt to understand complex behaviour (e.g: foraging, murder, courtship) by training artificially generated species using Deep RL algorithms in nature-inspired environments that follow the basic rules of evolution.

Why do we need this framework?

The motivation of this project comes from a famous theory proposed by Charles Darwin called — “Theory of Evolution by Natural Selection”. This is considered one of the most promising theory (to date) which provides explanation for the rise of complex and intelligent organisms in our planet. The theory explains how organisms evolve (or change) in the course of several generations to become a better version of their ancestors thus increasing their chances of survival and thriving.

Since evolution is the best explanation for human intelligence (or any other being’s intelligence in general), it seems apt that we try and simulate evolution (at least at a very high level) to understand the reason behind the complex behaviour and if we are a little too optimistic — intelligence.

PyGeneses provides bio-inspired evolutionary environments containing simple artificial species along with in-built Deep RL algorithms like — REINFORCE and Multi Agent DDPG (Deep Deterministic Policy Gradients) to train these agents towards achieving a certain set of complex goals. It also has an interactive visualization dashboard — VitaBoard to make sense of the agents once they are trained for finite number of generations.

Use Cases

Researching various possible behaviour/trends by artificial agents and drawing parallels with real world.

Tool to simulate a very primitive life.

Understanding the reason for a particular behaviour shown by single agent or group of agents.

Stimulating new circumstances to observe their actions.

Target audience

Due to the simple high level API of PyGeneses we (the developers of PyGeneses) believe that this tool can be helpful for both programmers and non-programmers who want to understand the nature of behaviours in animals or understand concepts of Deep RL.

Most of our APIs require only 3 to 4 lines of code to modify the environment according to your needs, thus allowing users with little to no knowledge of ML or Deep RL to use our tool easily.

We believe that understanding the reason of display of certain behaviours is not only an active area of study but also a fun experiment and through our framework we want to provide a platform for carrying out both of these tasks.

Prima Vita Environment

The Prima Vita environment consists of a single artificial species called Prima vita (from latin, meaning First Life). These agents are capable of performing thirteen different actions — movement in eight directions (north, south, east, west, north-east, north-west, south-east, and south-west), stay at one place, ingest (or consume) a food particle, asexual reproduction, sexual reproduction, fight with another agent.

The agent takes in a state — which is an observation of 200px circular region around the agent. The state includes distance vectors to food particles, other agents in this fixed area followed by the current energy of the agent and its age. The agent uses a neural network for analyzing this state and taking the appropriate action at that time step.

An example of how an agent might perceive the environment

The following diagram provides an overview of a generalized RL system:

Agent and its interaction with environment. Courtesy: Towards Data Science

An Example

We have gone through a lot, now let’s look at a specific training example and the results that we got using REINFORCE algorithm.

The first step is to install pygeneses:-

user@programmer~:$ pip install git+https://github.com/Project-DC/pygeneses

Now that we have PyGeneses installed, let’s get started with an example:

That is it, just 4 lines of code. PyGeneses abstracted minute details and presents a simple and intuitive API for users to interact with. Let us try go through this code:-

The first line from pygeneses.envs.prima_vita import PrimaVita imports the environment called Prima vita (latin for First life) which contains a simple environment along with the actual artificial species (no pun intended 😅) Prima vita.

imports the environment called Prima vita (latin for First life) which contains a simple environment along with the actual artificial species (no pun intended 😅) Prima vita. params_dic = {"initial_population": 30, "state_size": 32, "speed": 10} initializes the various parameters for Prima vita environment. The environment will start with 30 Prima vita agents and their state or observation space is padded to a max length of 32, the speed with which an agent moves is 10 px, finally the model will be updated if the agent has experienced atleast 5 state, action pairs.

initializes the various parameters for Prima vita environment. The environment will start with 30 Prima vita agents and their state or observation space is padded to a max length of 32, the speed with which an agent moves is 10 px, finally the model will be updated if the agent has experienced atleast 5 state, action pairs. model = PrimaVita(mode="human", params_dic=params_dic) , “human” mode allows us to visualize the environment while training, the other mode avaiable “bot” can be used when you do not want to see the agent’s as they are trained, this mode is computationally much less expensive than human mode.

, “human” mode allows us to visualize the environment while training, the other mode avaiable “bot” can be used when you do not want to see the agent’s as they are trained, this mode is computationally much less expensive than human mode. model.run(stop_at=10000) trains the model for 10,000 agents i.e. the environment will run until 10,000 agents die.

We know you might be a bit confused at this point, but adding all the details of the environment and training is out of scope of this blog and requires an entire documentation, for more details do checkout official PyGeneses docs.

Analyze the results

Average age of death v/s Time of birth

Variance in age of death v/s Time of birth

Quality of life v/s Time of birth

A very simple but powerful result that we notice from these graphs is that the average age increases in later generations which shows how the agents have learnt to live for longer than their predecessors and pass this knowledge to the later generations which live even longer to the point that they cannot live for any longer (as maximum allowed age for an agent is 90).

Some other interesting results which we observed while training in human mode are:-

Agents were cluttered in areas where the density of food particles was high as compared to areas with less density of food particles, which makes sense as consuming food increases energy which is required for performing other actions.

Some agents follow other agents throughout their life after a sexual encounter in hope for reproduction in future time steps. This is also observed in many organisms in nature where individuals seek a single mate throughout their life.

Agents which do not have food particles or other agents around it, perform asexual reproduction and die (this is something which we added in the logic for asexual reproduction, techniques like segmentation in nature also kill the parent after asexual reproduction). Since at that time there is no choice for the agent but to propagate its genes and die.

Note: Some of these results are not guaranteed in every run as the process of evolution is highly stochastic (even in real life) and may not lead to these results, but on the positive side, you may see some other interesting results which we did not encounter yet.

Final words

If you are someone who is keen to learn about ML and AI, or a scientist who studies behaviour or even someone who just wants to have some fun experimenting with these artificial agents, then get your local installation of PyGeneses today.

There is so much to PyGeneses, which we could not fit in this blog. To know more or contribute to this project, follow the official docs and the GitHub repository.

The Team

Here is the wonderful team that has worked to develop the tool — PyGeneses.