Optimization: Loss Function Under the Hood (Part I) | by Shuyu Luo | Towards Data Science

When building a machine learning model, some questions similar like these usually comes into my mind: How does a model being optimized? Why does Model A outperform Model B?
To answer them, I think one of entry points can be understanding loss functions of different models, and furthermore, being able to choose an appropriate loss function or self-define a loss function based on the goal of the project and the tolerance of error type. I will post a series of blogs discussing loss functions and optimization algorithms of a few common supervised learning models. I will try to explain in a way that is friendly to the audience who don’t have a strong mathematical background. Let’s start from Part I, Linear Regression.
For supervised learning, models are optimized by finding optimal coefficients that minimize cost function. Cost function is the sum of losses from each data point calculated with loss function. The model we choose to use is our hypothesis. Here is the hypothesis for Linear Regression model.
The most commonly used loss function for Linear Regression is Least Squared Error, and its cost function is also known as Mean Squared Error(MSE).
As we can see from the formula, cost function is a parabola curve. To minimize it, we need to find its vertex. It can be solved analytically or by using programming algorithms. In this blog, I will focus on one of the most popular programming solutions, Gradient Descent, to walk through the optimization process. Gradient Descent is pervasively applied in different models that takes steps based on a learning rate α, moving towards the vertex of the parabola to find the global minimum for Linear Regression, which is also known as the point with the slope equals to 0. So to get the slope, we take the derivative of cost function at each coefficient θ.
Next, continuously update each θ with a good amount of iterations along with the observation of reduction in cost function until it reaches a horizontal line. In reality, it is impossible for the slope to reach an exact value of 0 due to the chosen step size, but it can be close to 0 as much as possible depending on hyperparameters.
Now keep those concepts in mind, let’s use the real-world data (Boston Housing data from UCI Machine Learning Repository). Since I only demonstrate the optimization part in this blog, I will skip all the processes such as exploratory data analysis, feature engineering, etc., and directly go to data normalization being ready for prediction. Let’s take a look at the first few rows of data.
For explanation simplicity, I selected 2 features(RM: average number of rooms per dwelling, AGE: proportion of owner-occupied units built prior to 1940) to make the prediction, and target variable is MEDV(Median value of owner-occupied homes in $1000's). For calculation efficiency, I converted all the calculations to matrix format. Below are matrices for hypothesis, cost function, derivative of cost function, and the python query for matrix calculation.
# Instantiate a hypothesishypothesis = X@theta - y# Calculate cost functiondef cost(theta, X=X, y=y, m=m):    cost = np.transpose((X@theta - y))@(X@theta - y)    cost = (1/(2*m))*cost    return cost# Calculate derivative of cost functiondef cost_dev(j, theta, X=X, y=y, m=m):    dev = X[:, j]@(X@theta - y)    dev = (1/m)*dev    return dev
All coefficients are set to1 at the beginning. The query below returns 100,000 iterations to update coefficients simultaneously. When the trend starts to flatten out which means gradient descent has converged, and minimum of cost function has been reached.
# Assign a learning ratea = 0.001cost_list = []theta_temp = np.zeros(theta.shape)theta_list = []for i in range(100000):        for j in range(len(theta)):        theta_temp[j] = theta[j] - a*cost_dev(j, theta)        theta = theta_temp                theta_list.append(list(theta))    cost_val = cost(theta)    cost_list.append(cost_val)
Can we make this trend better and be more confident to say the minimum is indeed reached? What would happen if we adjust learning rate to 0.0005 with 1,000,000 iterations.
The trend looks crazy. It’s not really a good demonstration of cost function, but we can be very confident about the result of optimization, assuming the minimum of MSE is reached with current setting. Let’s compare the coefficients just obtained from Gradient Descent algorithm manually to that obtained from Scikit-learn LinearRegression() with the same dataset.
Ta-da! They are almost the same. The goal of this blog is to unveil the basic optimization secret of machine learning model. There are many advanced variations of Gradient Descent and other algorithms have been applied in Scikit-learn packages. Of course, in most scenarios, there’s no need to be bothered by those mathematics. However, being able to understand loss function under the hood help us move forward to the next level as a machine learning engineer!
To be continued......