Gradient descent vs. neuroevolution
By: Lars Hulstaert
Tags: Machine Learning, Deep Learning, Data Science, Neural Networks, Towards Data Science

In March 2017, OpenAI released a blog post on evolution strategies, an optimisation technique that has been around for several decades. The novelty of their paper was that they managed to apply the technique to deep neural networks in the context of reinforcement learning (RL) problems. Before this, the optimisation of deep learning RL models (with typically millions of parameters) was typically achieved with backpropagation. Using evolution strategies for deep neural network (DNN) optimisation seemingly unlocked an exciting new toolbox for deep learning researchers to play with.

This week, Uber AI Research released a set of five papers which are all focussed on 'neuroevolution'. The term neuroevolution refer to the optimisation of neural networks through evolutionary algorithms. The researchers posit that genetic algorithms are an effective method to train deep neural networks for reinforcement learning problems and that they outperform traditional RL approaches in some domains.

Neuroevolution relates to natural evolution as an airplane relates to a bird. Neuroevolution has borrowed some fundamental concepts from nature, similarly as neural networks, planes, etc.

Overview

So what does this mean? Will all DNN's, ranging from supervised, to unsupervised as well as RL applications, be optimised using neuroevolution in the near future? Is neuroevolution the future of deep learning? What exactly is neuroevolution? In this blog post I will attempt to provide an introduction to neuroevolution, and compare it to the traditional backpropagation algorithm. I will also attempt to answer the aforementioned questions and situate neuroevolution techniques in the bigger DL picture.

First I will start by framing the optimisation problem, which is the core problem that backpropagation and neuroevolution try to solve. I will also attempt to make a clear distinction between the difference between supervised learning and reinforcement learning.

Next, I will discuss backpropagation and explain how it relates to neuroevolution. Given that both OpenAI and Uber AI Research just released papers on this technique, I have plenty of topics to tackle. Fortunately, as deep learning neuroevolution is in the early stages of research, the mechanics of neuroevolution still remain fairly comprehensible.

The optimisation problem

As discussed in my previous blog post, machine learning models are in essence function approximators. Whether it is classification, regression or reinforcement learning, the end goal is almost always to find a function that maps input data to output data. You use the training data to infer the parameters and hyperparameters and verify with the test data whether the approximated function performs well on unseen data.

The inputs can be manually defined features or raw data (images, text, etc.) and the outputs are either classes or labels in classification, real values in regression and actions in reinforcement learning. For this blog post we will limit the type of function approximators to deep learning networks, yet the same discussion applies to other models. The parameters that need to be inferred thus correspond to the weights and biases in the network. 'Performing well on train and test data' is expressed through objective measures, such as the log loss for classification, the mean squared error (MSE) for regression and the reward for reinforcement learning.

The core problem is thus finding the parameter settings that results in the lowest loss or the highest reward. Simple! Given an optimisation objective, i.e. the loss or the reward, that needs to be optimised as a function of the networks parameters the goal is thus to tweak the parameters in such a way that the optimisation objective is minimised or maximised.

In order to make this visual, let's use two examples. In both cases, we have plotted an optimisation objective. In the image of the parabola, the x-axis represents the single parameter of the model, and the y-axis represents the optimisation objective (e.g. on test data).

In the image below, the x and y axis represent the two parameters of the model and the z axis represents the optimisation objective (e.g. on the test data).

In reality, it is not possible to plot the 'optimisation surfaces', due to the sheer number of parameters and how they are non-linearly combined in deep learning networks. Nevertheless, the same ideas apply and the optimisation surface is typically high dimensional and complex, with many hills, cradles, valleys, etc.

The goal is now to find an optimisation technique that allows us to crawl in the optimisation surface and find the minimum or maximum. Note that the size and shape of these surfaces is related to the number of parameters, and it is infeasible to explore all options, regardless if we are working with continous or discrete parameters. The problem now has become the following: Given a random starting point in the optimisation surface, find the absolute minimum or maximum.

Deep neural networks are great function approximators (universal function approximators even to a certain extent), but they are also hard to optimise. Hard to optimise in this context means that it is very hard to find the global maximum or minimum on the ‘optimisation surface’. In the next sections I will discuss how gradient descent and neuroevolution can be used to find good solutions.

Gradient descent

The general idea of gradient descent (backpropagation) has been around for several decades. Due to the abundance of data, compute power, and novel innovations, it has become the main technique to optimise the parameters of deep learning models.

The general idea of gradient descent is the following:

-Assume you are in Paris, France, and you need to get Berlin, Germany. Europe in this case is the optimisation surface, Paris is the random starting point and Berlin is the absolute minimum or maximum.

-As you don't have a map, you ask random strangers the direction to Berlin. Some of the random strangers know where Berlin is but others don't, so while most of the times, you walk in the correct direction, it is possible that you also are going into wrong directions. As long as the strangers are more correct than false, it should work out (e.g. stochastic gradient descent, or gradient descent with minibatches).

-You walk 5 miles (the stepsize or learning rate) in the direction the stranger has pointed you to. This process is repeated until you believe you are close enough to Germany. It might turn out you just entered Germany, and you are nowhere close to Berlin (local optima). There is no way to verify whether you reached your end goal and you can only estimate based on your surroundings (the test loss or the reward).

Gradient descent: walking around in Europe without a map.

Going back to the two visual examples, you can imagine how gradient descent works in the case of the parabola and the more complex surface. Essentially, with gradient descent you are walking down the optimisation surface. In the case of the parabola, it is straightforward, as you just have to walk down the slope. If the learning rate is too high though, you might never be able to get to the absolute minimum.

The importance of a proper stepsize (image taken from the Cornell course)

In the second example the situation is more complex. There are several hills and valleys you have to overcome in order to get to the absolute minimum. Several gradient descent variants that attempt to mimic physical behaviour such as a ball rolling down the surface with momentum (e.g. ADAM) in order to avoid getting stuck in local optima.

An example path that gradient descent might take. Notice that depending on the hyperparameters you might get stuck in the initial local minima.

I highly recommend this blog post on gradient descent variants. It clearly explains and illustrates the differences between the different variants and how they solve the different problems that gradient descent faces. There are often several local optima, obstacles and different paths in this optimisation problem, and different variants of gradient descent typically try to tackle some (or all of these) problems. Nowadays, the ADAM optimiser seems to be the most influential one.

Essential to gradient descent is the computation of proper gradients that propel you towards a good solution. In supervised learning, it is possible to obtain 'high quality gradients' with relative ease through the labeled datasets. In reinforcement learning however, you are only given a sparse reward, as the random initial behaviour will not lead to a high reward. In addition this reward only occurs after a couple of actions. While the loss in classification and regression is a relatively good proxy for the function you are trying to approximate, the reward in reinforcement learning is typically not a very good proxy of the behaviour or function you want to learn.

Given that the gradients in reinforcement learning are not always of good quality, evolution algorithms have recently been used by Uber and OpenAI to improve learning.

Neuroevolution

Neuroevolution, genetic algorithms, evolution strategies all revolve around the concept of genetic evolution.

When you are doing genetic optimisation in the context of DNN optimisation, you start from an initial population of models. Typically, a model is randomly initialised, and several offspring are derived based on this initial model. In the case of DNN’s, you initialise a model (as you normally do), and you add small random vectors, sampled from a simple Gaussian distribution, to the parameters. This results in a cloud of models, which all reside somewhere on the optimisation surface. Note that this is the first important distinction with gradient descent. You start (and continue to work) with a population of models, instead of a single (point) model.

Starting from this original population, the genetic optimisation cycles start. In what follows I will describe genetic optimisation in the context of evolution strategies (ES). Evolution strategies, genetic algorithms, etc. all have slightly different approaches as to how genetic optimisation is performed.

Overview of genetic optimisation.

First, a fitness evaluation is performed. Fitness evaluation corresponds with checking where the models are in the optimisation surface and determining which of the models perform best (e.g. are the most fit). Some models will perform better than others, just because of the way they have been initialised.

Next, a selection is performed based on the fitness evaluation. In evolution strategies, the (pseudo) offspring is reduced to a single model, weighted by the fitness evaluation. For DNN's the fitness is defined as the loss or the reward. Essentially, you are thus moving around the optimisation surface and using the offspring to get in the right direction. Notice that this is a second major difference with gradient descent. Instead of computing the gradients, you are setting out multiple 'antenna's' and moving in the direction that looks best. In a way, this is similar to a 'structured random' search. The end result of the selection phase is that you have a single model.

Next, reproduction and combination is performed. Concretely, the same process as in the initial phase is repeated. Based on the newly selected 'prime' model, a new set of offspring is derived. The process then continues with this offspring.

Typically, in genetic optimisation, mutation is also performed in order to improve the variety of the offspring. One example of mutation is to change how the different offspring is created (i.e. different noise levels for the different parameters).

One of the advantages of ES is that the fitness evaluation of the different models in the population can be computed on different cores (out-of-core computation). The only information that needs to be shared after fitness evaluation is their performance (a scalar value) as well as the random seed value that was used to generate the model. As a result, it is not necessary anymore to share the full network with all machines. Both OpenAI and Uber used literally hundreds and thousands of machines in order to do their experiments. With the rise of cloud computing, running these experiments becomes very scalable and is only limited by compute power.

The image below represent two of the main differences between ES and gradient descent. Multiple models are used to move around, and gradients are not calculated, rather the different models are averaged based on their performance. Optimising over a population instead of over a single model results in higher robustness properties, as shown in the Uber research, and shows similarities to Bayesian approaches to DNN optimisation.

Neuroevolution: walking around Europe in a group.

I highly recommend having a look at the great illustrations in the Uber blog post. These illustrate how ES can be used to circumvent some of the problems that gradient descent faces (e.g. getting stuck in local optima). In essence, the evolution strategy performs a gradient approximation. While using the true gradients is better when they are available, this method does show promise when only relatively bad approximations are available for the true gradient and when exploration of the optimisation surface is required, such as in reinforcement learning scenario's.

Conclusion

The researchers at OpenAI and Uber are able to show that the 'gradient approximation' of evolution strategies leads to satisfying (but not necessarily new state-of-the-art) solutions in supervised learning scenario's. On the other hand they are able to show high performance of their methods in reinforcement learning scenario's (comparable to state-of-the-art in some domains).

Will neuroevolution be the future of deep learning? Probably not, but I do believe it shows great promises for hard optimisation problems, such as in reinforcement learning scenario's. In addition, I believe a combination of neuroevolution and gradient descent methods will lead to a significant improvement in RL performance. One downside of neuroevolution is the massive amounts of compute power that are required in order to train these systems, which might limit the democratisation of these techniques.

Given that prominent research groups are focussing on these problems, I am still excited to see what the future holds!