Genetic Algorithm to Optimize Machine Learning Hyperparameters | by Marcos del Cueto | Towards Data Science

Hyperparameter tuning is critical for the correct functioning of Machine Learning models. You can check Timo Böhm’s article to see an overview of hyperparameter tuning.
Genetic algorithms provide a powerful technique for hyperparameter tuning, but they are quite often overlooked.
In this article, I will show an overview of genetic algorithms. I will also offer a detailed step-by-step guide on exploiting available libraries to use genetic algorithms to optimize the hyperparameters of a Machine Learning model.
Genetic algorithms make use of basic concepts from evolution by natural selection to optimize arbitrary functions.
This short article will introduce Differential Evolution and teach how to exploit it to optimize the hyperparameters used in Kernel Ridge Regression.
I provide snippets of code to show how to use a Differential Evolution algorithm in Python. Complete codes and figures are also provided in a GitHub repository, so anyone can dive into the details.
Genetic algorithms, such as Differential Evolution, make use of basic concepts from evolution by natural selection. In our case, we have a population formed by vectors that contain parameters with distinct values.
Parameters are equivalent to genes in biological systems. Their exact value differs between different vectors. Different combinations of parameters cause vectors to display different fitness values.
Random parameter mutations are introduced in the population, and vectors with a larger fitness outlive other vectors.
Using an iterative process, Differential Evolution can minimize a function following steps:
1 — Initialization: create an initial population with NP vectors with random parameter values within boundaries.
2 — Initial evaluation: calculate function value for the NP vectors.
3 — For each vector in the population:
3.1 — Mutation: we build a mutant vector, where each parameter’s value is calculated as a mutation of the parameters of other vectors randomly chosen from the population. A common strategy to calculate this mutant vector is best1bin, where each parameter pi of the mutant vector is calculated as shown in the Equation below. The mutant parameter is a variation of the pi parameter of the best vector (vector with the lowest value) plus a mutation rate (F) times the pi-difference of two vectors randomly chosen, r1 and r2.
3.2 — Recombination: a trial vector is created by selecting each of its parameters as either the value of the current vector or the value of the mutant vector. For each parameter, we generate a random uniform number R in the (0,1) interval. If R is lower than a recombination rate, then we accept the mutant parameter; otherwise, we use the parameter of the current parameter.
3.3 — Replacement: evaluate the function of the trial vector. If it’s more stable than the current one, substitute the current vector by the trial vector.
4 — Repeat Step 3 until population convergence: iteration stops when the standard deviation of the function within the population is smaller than a specific percentage of the mean value of the function. Loop also stops if convergence is not reached after a maximum number of iterations.
All this process is already implemented in several libraries in the main programming languages. Here, we will be using Scipy’s implementation in Python, which lets us do everything we want by using just a few lines of code.
As an example, we will use data that follows the two-dimensional function f(x1,x2)=sin(x1)+cos(x2), plus a small random variation in the interval (-2,2) to spice things up. Therefore, our data will follow the expression:
In the figure below, we show in grayscale the value of sin(x1)+cos(x2) as a reference. Then, we show colored points to indicate our 441 points (21x21 grid), calculated with the expression above in the interval x1:(-10,10) and x2:(-10,10).
We offer below a code snippet with the function used to generate our data:
Our goal is to use the data generated previously to train an ML model that will be able to predict the value of the function f(x1,x2) at different configurations.
In this article, I will be using Kernel Ridge Regression (KRR). We will be using a radial basis function kernel for KRR, which depends on the Gaussian kernel variance γ. With KRR, we also need to optimize the regularization hyperparameter α. If you want more details on KRR and its implementation, feel free to check out my recent tutorial on this topic.
Here we use a 10-fold cross-validation, where our data is split into a training set used to optimize the model, and a test set used to measure the model accuracy. As an accuracy metric, we will use the root-mean-square error (RMSE), which represents the average error between the f(x1,x2) values in the test set and those predicted by our model.
As an optional first step, we can perform a grid-search to see how the RMSE changes with the value of our two hyperparameters: α and γ.
We show below how this grid looks, and the code used to perform the hyperparameter grid search. The Figure below gives us an initial idea of what hyperparameter values return a smaller RMSE.
Finally, we can use the Differential Evolution algorithm provided by Scipy to optimize the hyperparameters by minimizing the RMSE of our model.
Scipy’s differential_evolution function needs as input:
KRR_function. This is the function whose output (RMSE) will be minimized. It needs as input: i) tuple with hyperparameters to optimize (α and γ) and ii) X, y variables containing our data.
Define boundaries of possible values of hyperparameters.
Extra variables used by KRR_function. In our case, this will be given as a tuple containing X and y.
Other relevant options for the Differential Evolution algorithm are :
Strategy. In our case, the default strategy=’best1bin’ is good enough. With this strategy, the value of each parameter of the mutant vectors is obtained as a variation of the best vector’s value for that parameter, proportionally to the difference of two other random vectors.Population size. This selects how many vectors we will consider. A larger number will slow down progress but will make it more likely to discover the global minimum. Here, we use the default value of popsize=15.Mutation constant. This value controls how much the parameters change during the mutation stage. A larger value means a larger search radius but slows down convergence. We use the default value of mutation=0.5.Recombination constant. This constant controls how likely the parameters of trial vectors are to change during the recombination stage. Larger values mean that mutations are more likely to be accepted, which may accelerate convergence at the risk of causing population instability. We use the default value of recombination=0.7.Tolerance. This value controls when the algorithm is considered to converge. We will use tol=0.01, meaning that the algorithm is considered to be converged when the standard deviation of the RMSE of all vectors in the population is smaller than 1% of the average RMSE.
Strategy. In our case, the default strategy=’best1bin’ is good enough. With this strategy, the value of each parameter of the mutant vectors is obtained as a variation of the best vector’s value for that parameter, proportionally to the difference of two other random vectors.
Population size. This selects how many vectors we will consider. A larger number will slow down progress but will make it more likely to discover the global minimum. Here, we use the default value of popsize=15.
Mutation constant. This value controls how much the parameters change during the mutation stage. A larger value means a larger search radius but slows down convergence. We use the default value of mutation=0.5.
Recombination constant. This constant controls how likely the parameters of trial vectors are to change during the recombination stage. Larger values mean that mutations are more likely to be accepted, which may accelerate convergence at the risk of causing population instability. We use the default value of recombination=0.7.
Tolerance. This value controls when the algorithm is considered to converge. We will use tol=0.01, meaning that the algorithm is considered to be converged when the standard deviation of the RMSE of all vectors in the population is smaller than 1% of the average RMSE.
This code returns the converged hyperparameter values that result in the minimized RMSE:
Converged hyperparameters: alpha= 0.347294, gamma= 3.342522Minimum rmse: 1.140462
We can uncomment the last line of the function KRR_function to print all intermediate values:
alpha: 63.925979 . gamma: 19.290688 . rmse: 1.411122alpha: 59.527726 . gamma:  2.228886 . rmse: 1.421191alpha: 24.470318 . gamma:  3.838062 . rmse: 1.379171alpha: 61.944876 . gamma: 15.703799 . rmse: 1.407040alpha: 68.141245 . gamma:  0.847900 . rmse: 1.431469                        [...]alpha:  0.347408 . gamma:  3.342461 . rmse: 1.140462alpha:  0.347294 . gamma:  3.342522 . rmse: 1.140462alpha:  0.347294 . gamma:  3.342522 . rmse: 1.140462alpha:  0.347294 . gamma:  3.342522 . rmse: 1.140462
We can take these configurations explored by the Differential Evolution algorithm, and plot them on top of our previous hyperparameter grid search:
In addition to all the intermediate values explored, we also plotted in red the converged value. This way, we can observe how the algorithm has been exploring different configurations within the set boundaries and it has eventually converged into an (α,γ) combination that minimizes the RMSE.
We have covered how to generate a simple two-dimensional dataset and how to fit it with a Kernel Ridge Regression. We have covered the basics of Differential Evolution and how to use this algorithm to optimize the hyperparameters of a Machine Learning model.
Remember that the dataset presented here, codes, images, etc. are provided in this GitHub repository.
If you are curious about more complex applications of genetic algorithms to optimize Machine Learning hyperparameters, feel free to check our recent work at the University of Liverpool, in collaboration with the Northeast Normal University. In this work, we used a Differential Evolution algorithm to optimize several Machine Learning models to predict the efficiency of organic solar cells.
Was this article helpful to you? Let me know if you were able to successfully use a Differential Evolution algorithm to optimize the hyperparameters of your Machine Learning model!