Strategies for Global Optimization | by Syed Sadat Nazrul | Towards Data Science

Local and global optimization is usually known and somewhat ignored once we leave high school calculus. For a quick review, take the cover image of this blog. For a given function, there are multiple points where there are dips across space. Each dip is a minimum. However, you can see that only one point is the deepest, known as the global minimum. All other points are local minima. The first section, I will go over calculating this global value for a known function. However, once we leave the realm of this known function of high school math, most optimization problems deal with local optimizations only. Even this, I will go over and provide reasons why. Quick note, there are plenty of other strategies out that I have not mentioned in this blog. However, I am hoping that it gets you interested enough to explore more options.
Let’s review some basic high school calculus. First, going over this simple univariate function:
Graphically, we can see that the global minimum is around -3. However, let’s try to use calculus as we know the function of this plot.
We see that f(-3)=-22.75, the smallest possible value of f(x). We can even go beyond 2D and enter the realm of multivariate calculus for solving this problem.
Most of the time when dealing with data science, we do not have access to the function to perform any calculus. Usually, f(x) is a system, where we can enter the variable, x, and gain some output, y. One possible solution is to perform a stochastic gradient descent, where we iteratively go down the slope till we hit a minimum.
Now let’s implement this problem in python, assuming f to be out black box function.
#Unknown Functionf = lambda x:(1/4*x**4)+(1/3*x**3)-(3*x**2)-7def next_step(x,f,step_size):  y=f(x)  #Left Point  x_left=x-step_size  y_left=f(x_left)  diff_left=y_left-y  #Right Point  x_right=x+step_size  y_right=f(x_right)  diff_right=y_right-y  #Comparison  if diff_right<diff_left:    return x_right, y_right, diff_right  else:    return x_left, y_left, diff_leftdef gradient_descent(f,start,step_size=0.01,tol=0.1, debug=False):  x=start  diff=9999999  while abs(diff)>tol:    x, y, diff=next_step(x,f,step_size)    if debug:      print("Current Point: (%.2f,%.2f), Change: %.2f"%(x,y,diff))  print("Minimum Point: (%.2f,%.2f)"%(x,y))
Now that we have our function, let’s try searching from x=4.
gradient_descent(f, start=4)#Minimum Point: (2.65,-9.54)
This gives us a minimum point of 2.65 . Now, let’s try the same for another starting point, x=-4.
gradient_descent(f, start=-4)#Minimum Point: (-3.51,-20.43)
Using x=-5, we get our global minimum at -3.51 . This is when finding global and local minimum becomes tricky. Our final result is dependent on the starting point.
As it is an unknown function, we do not know:
The ideal start point
The ideal step size
The ideal domain (we cannot traverse an infinitely large space of x)
One possible solution is the use of Simulated Annealing, which gives us a reasonable probability of hitting the global minimum.
The name and inspiration come from annealing in metallurgy, a technique involving heating and controlled cooling of a material to increase the size of its crystals and reduce their defects. Both are attributes of the material that depend on its thermodynamic free energy. Heating and cooling the material affects both the temperature and the thermodynamic free energy. The simulation of annealing can be used to find an approximation of a global minimum for a function with a large number of variables.
This notion of slow cooling implemented in the simulated annealing algorithm is interpreted as a slow decrease in the probability of accepting worse solutions as the solution space is explored. Accepting worse solutions is a fundamental property of metaheuristics because it allows for a more extensive search for the global optimal solution.
In general, the simulated annealing algorithms work as follows:
Let x = x0
For k = 0 through kmax (exclusive):- T decreased at each step- Pick a random neighbour, x_new ← neighbour(x)- If P(E(x), E(x_new), T) ≥ random(0, 1): x ← x_new
Output: the final state x
Here is the Python implementation from our specific function:
import numpy as np#Unknown Functionf = lambda x:(1/4*x**4)+(1/3*x**3)-(3*x**2)-7def acceptance_probability(E, E_new, T):    return np.exp(-(E-E_new)/T)def random_neighbour(x):    return x += np.random.uniform(-1,1)def simulated_annealing(f, steps):    x = np.random.random()    E = f(x)    print("x=%.2f, fmin=%.2f"%(x, E))    for k in range(steps):        T = T*0.9        x = random_neighbour(x)        E_new = f(x)        P = acceptance_probability(E, E_new, T)        if P > np.random.random():            E = E_new        print("x=%.4f, fmin=%.4f, Prob.=%.4f"%(x,E,P))    return Esimulated_annealing(f,20)
Running this equation, we get the following output:
x=0.62, fmin=-8.04x=-0.3446, fmin=-7.3664, Prob.=0.4753x=-0.8717, fmin=-9.3559, Prob.=11.6601x=-0.8329, fmin=-9.1534, Prob.=0.7575x=-1.6213, fmin=-14.5791, Prob.=3903.7178x=-2.3907, fmin=-20.5342, Prob.=23982.6510x=-1.8220, fmin=-20.5342, Prob.=0.0003x=-1.1582, fmin=-20.5342, Prob.=0.0000x=-0.2298, fmin=-20.5342, Prob.=0.0000x=-0.8731, fmin=-20.5342, Prob.=0.0000x=-1.8032, fmin=-20.5342, Prob.=0.0000x=-2.1873, fmin=-20.5342, Prob.=0.0110x=-1.8673, fmin=-20.5342, Prob.=0.0000x=-2.7618, fmin=-22.3598, Prob.=1315.6210x=-2.3266, fmin=-22.3598, Prob.=0.0001x=-2.5017, fmin=-22.3598, Prob.=0.0036x=-2.6164, fmin=-22.3598, Prob.=0.0466x=-1.7016, fmin=-22.3598, Prob.=0.0000x=-1.7248, fmin=-22.3598, Prob.=0.0000x=-1.6569, fmin=-22.3598, Prob.=0.0000x=-1.5051, fmin=-22.3598, Prob.=0.0000
As seen above, the value converges to -22.75. While this method doesn’t guarantee a global optimum, it is reasonably close, given that we are not restricting our domain. Also notice that, although the initial value was much closer to the local minimum, it still managed to find the global minimum.
This section has more to do with time complexity than accuracy. Dynamic Programming is mainly an optimization over plain recursion. Wherever we see a recursive solution that has repeated calls for same inputs, we can optimize it using Dynamic Programming. The idea is to simply store the results of subproblems, so that we do not have to re-compute them when needed later. This simple optimization reduces time complexities from exponential to polynomial. For example, if we write simple recursive solution for Fibonacci Numbers, we get exponential time complexity and if we optimize it by storing solutions of subproblems, time complexity reduces to linear.
Let’s use the Fibonacci sequence as an example:
def f(n):    if n <= 1:        return n    else:        return f(n-1)+f(n-2)
The complexity of this problem is O(2^n). Seems insanely complex if you ask me! However, let’s analyze the problem a bit better.
If you look at a recursion tree for n=6, there are a lot of repetitions. Just count the number of values for f(3). In Dynamic Programming, we will store the value in memory and not re-calculate if called upon again. Here is the Dynamic Programming version of Fibonacci:
store = {}store[0] = 0store[1] = 1def f(n):    store[n] = f(n-1)+f(n-2)    return store[n]
Using the new method, the complexity of each step is O(1). This allows us to explore the entire domain of n to find the optimal solution with minimal time complexity.