Making Bayesian Optimization Algorithm Simple for Practical Applications
Source: tai_blog

The Goal of this writing is to show an easy implementation of Bayesian Optimization to solve real-world problems.   Contrary to Machine Learning modeling which the goal is to find a mapping between input and output by utilizing a rather large set of data  in Optimization  defining the exact algorithm inside the Black-Box is not of interest  we do not have the luxury of applying many inputs  Maybe because of the constraints of the process that is too timely or too costly all we are looking for is to find one magical combination of input variables that produces the smallest output and be able to achieve that by examining only a limited number of input values applied to Black-Box.   This problem is prevalent in every discipline  regardless of where you work  you will face this problem where you want to optimize a metric in your process  whether is cost  resources  time to market  quality  reliability  etc.  and in all cases  you have few parameters or knobs you can turn in your process  and you want to find out that magical input values that give you the best-optimized output value with the smallest number of trials.   The situation becomes trickier if black-box output may have some local minimum and maybe one large global minimum  and how can we avoid being trapped in one of those local minimums and missing the largest global minimum?   In this paper  we show how the Bayesian Optimization algorithm  In conjunction with data coming from the field  can work together to discover the optimum point for the process.   You might be sitting at your computer and running a Bayesian Optimization Algorithm  while the physical Black-Box might be sitting in a Lab at some distance. You act as a middleman  talking to both sides. For the Algorithm  we use the SKOPT package of SciKit-learn. You can install this open-source package using the command:   pip install scikit-optimize   Sequential model-based optimization toolbox.pypi.org   The heart of the Algorithm is a Gaussian Process called gp_minimize; for simplicity  Lets call this magical function AI Genie  and You are acting in between this AI-Genie  which is running in your PC  and your physical Black box. The goal of the AI-Genie is to find the minimum output of the black box with as small a number of trials as possible. Also  to make it even simpler  assume that we have only one input in the black box; this process could easily be expanded to a multi-input case. The Picture below shows all the characters in this process:   Here is the actual code:   import numpy as np from skopt import gp_minimize from skopt.space import Real from skopt.utils import use_named_args import matplotlib.pyplot as plt # Define the search space (let's assume we're searching within -100 to 100) search_space = [Real(-100  100  name='X')] # Objective function that interacts with the user @use_named_args(search_space) def objective_function(X): # Print the value of X print(fEnter this value into the black box: X={X}  flush=True) # Ask the user to input the corresponding Y value from the black box Y = float(input(Enter the value returned by the black box (Y): )) # Return the Y value as the result of the objective function return Y # Perform Bayesian Optimization result = gp_minimize(objective_function  search_space  n_calls=15  random_state=0) # Print the result print(fOptimal value found: X = {result.x[0]}) print(fMinimum value of the function: Y = {result.fun}) # Plot the convergence plt.plot(result.func_vals) plt.xlabel('Number of calls') plt.ylabel('Function value') plt.title('Convergence Plot') plt.show()Lets examine the code in more detail:   1- Import required libraries   import numpy as np from skopt import gp_minimize from skopt.space import Real from skopt.utils import use_named_args import matplotlib.pyplot as pltgp_minimize is the main function driving the optimization process   for input parameters to the black box  you can have Integer  Real  and Category. Here  we assume we have just one Real value input   use_name_args is a decorator supplied in SKOPT; its job is to select different values of input parameters and send them to be processed in Black-Box   2- Define search space   # Define the search space (let's assume we're searching within -100 to 100) search_space = [Real(-100  100  name='X')]Offers the system a range of valid values that input can take. For example  here we have one input called X which can take a float value between -100 to 100   3- Black-Box Representation   # Objective function that interacts with the user @use_named_args(search_space) def objective_function(X): # Print the value of X print(fEnter this value into the black box: X={X}  flush=True) # Ask the user to input the corresponding Y value from the black box Y = float(input(Enter the value returned by the black box (Y): )) # Return the Y value as the result of the objective function return YObjective function is the Function representing the Black-Box functionality. The Black-Box is inside the Objective function and receives the input values given to the Objective function from Search Space; the black box accepts that value  it processes the input  and provides the output to objective function  which will then be returned to the optimizing algorithm.   What makes this paper different is that we are acting like a black box inside the Objective function; we get the parameter passed to it  by printing that input value.   Then we pause the program to take that input back to the lab and give it to the physical or virtual black box  get the output of the black box  and then come back to the objective function  which was holding the execution to receive the value we enter as the output coming from the black- box.   and finally  Return the value to the Optimizer and wait for the next input from optimizer.   4- Main Bayesian Optimizer function   # Perform Bayesian Optimization result = gp_minimize(objective_function  search_space  n_calls=15  random_state=0)This is the heart of the algorithm  which we call Ai-Genie; the First parameter for this function is the Objective-function (which holds the black Box inside)  the next parameter is Search_Space  the next parameter is n_calls which the user choose to limit the number of trials  here user is asking the Ai-Genie to provide the minimum value of the output of black box within 15 trials  and last parameter is random_state to initialize the random state.   5- Printing the results   # Print the result print(fOptimal value found: X = {result.x[0]}) print(fMinimum value of the function: Y = {result.fun})This will print the minimum value out of the black box (Y) and the input value (X)  which will get you the minimum output.   Execution   Assume you have set everything and are ready to run the experiment; you have no idea what is inside the black box. You just know for any input you give it  it provides you an out put so lets start the experiment:   1- The First number the optimizer model give you is: 18.568924; the optimizer picks this very first number at random form the range of available input variables.   2- Take this number to the black box  enter it  and wait for the output  The black box returns: 363.373849   3- Take this out  put back to Optimizer  and enter it  wait for Optimizer to provide you with the next number: 68.853150   4- You have finished one round; continue this process till you exhaust the number of trial n_call.   Here X is the number suggested by the Ai-Genie to try on Black Box  and   Y is the output from Black-Box   The final result is given below:   Optimal value found: X = -0.49669415594226507   The minimum value of the function: Y = -0.24998907139506593   Lets plot convergence   # Plot the convergence plt.plot(result.func_vals) plt.xlabel('Number of calls') plt.ylabel('Function value') plt.title('Convergence Plot') plt.show()Notice in a range of -100 to 100  there are an infinite number of float values that Ai-Genie could choose from  but Ai-Genie is so awesome that after testing a few values  it almost knows what the minimum value is after only 10 trials.   Verification   Now that the experiment is concluded  How do I know that the Ai-genie really found the optimum value  and how do I verify it.   In real-world situations  we absolutely do not know what is inside the black box  and we also do not want to know  we are interested just in minimum output  but here just to test the accuracy of the Ai-genie in finding the optimum value  I did not expose this to Ai-genie but I went to black box in the lab  and placed a function that I know inside of it  the function I placed there was :   Y = X**2 + X   We can find the minimum value of this function using Differential equation and set it qual to zero and solve it.   dY/dX = 2X + 1   2X +1 = 0   X = -0.5   Y = -0.25   The values the Bayesian Optimization found without knowing this equation were extremely close  which verifies the power of the algorithm.   This is what makes the Bayesian Optimization algorithm so powerful. We should seriously consider using it more often to find optimal points for any process wherever possible.