Loss Function (Part II): Logistic Regression | by Shuyu Luo | Towards Data Science

This series aims to explain loss functions of a few widely-used supervised learning models, and some options of optimization algorithms. In part I, I walked through the optimization process of Linear Regression in details by using Gradient Descent and using Least Squared Error as loss function. In this part, I will move to Logistic Regression.
Remember the hypothesis of linear regression is:
Call this hypothesis of linear regression the raw model output. Logistic regression just has a transformation based on it. For logistic regression, focusing on binary classification here, we have class 0 and class 1. To compare with the target, we want to constrain predictions to some values between 0 and 1. That’s why Sigmoid Function is applied on the raw model output and provides the ability to predict with probability.
What hypothesis function returns is the probability that y = 1, given x, parameterized by θ, written as: h(x) = P(y = 1|x; θ). Decision boundary can be described as: Predict 1, if θTx ≥ 0 → h(x) ≥ 0.5; Predict 0, if θTx < 0 → h(x) < 0.5.
Linear regression uses Least Squared Error as loss function that gives a convex graph and then we can complete the optimization by finding its vertex as global minimum. However, it’s not an option for logistic regression anymore. Since the hypothesis is changed, Least Squared Error will result in a non-convex graph with local minimums by calculating with sigmoid function applied on raw model output.
Intuitively, we want to assign more punishment when predicting 1 while the actual is 0 and when predict 0 while the actual is 1. The loss function of logistic regression is doing this exactly which is called Logistic Loss. See as below. If y = 1, looking at the plot below on left, when prediction = 1, the cost = 0, when prediction = 0, the learning algorithm is punished by a very large cost. Similarly, if y = 0, the plot on right shows, predicting 0 has no punishment but predicting 1 has a large value of cost.
Another advantage of this loss function is that although we are looking at it by y = 1 and y = 0 separately, it can be written as one single formula which brings convenience for calculation:
So the cost function of the model is the summation from all training data samples:
Before fitting the parameters to training data with this cost function, let’s talk about Regularization briefly. There are two commonly used regularization types, L1(Lasso) and L2(Ridge). Instead of optimizing above cost function directly, with regularization, we add a constraint on how big the coefficients can get in order to prevent overfitting. L1 and L2 adapts different ways of setting upper bounds of coefficients, which determines that L1 has the ability to do feature selection by making coefficients 0 for less important features and mitigate the issue of multicollinearity, while L2 also penalizes very large coefficients but does’t make any to 0. There also exists a parameter that controls the weight of the constraint, λ, so that coefficients won’t be punished too hard resulting in underfitting.
In terms of why L1 an L2 have different capabilities due to ‘squared’ and ‘absolute’ value, and how λ effects the weight of regularized term and original fit term, is a very interesting topic. We won’t really dive into it here, but it’s definitely worth your study and research. Following shows how to update original cost function to regularized cost function.
With the right learning algorithm, we can start to fit by minimizing J(θ) as a function of θ to find optimal parameters. We can still apply Gradient Descent as the optimization algorithm. It takes partial derivative of J with respect to θ (the slope of J), and updates θ via each iteration with a selected learning rate α until the Gradient Descent has converged. See the python query below for optimizing L2 regularized logistic regression. There is a good article here that explains vectorized implementation of optimization process in great details.
## Vectorized Implementation of Optimization Using Gradient Descent# Define Cost functiondef cost(t, h, l=l, X=X, y=y, m=m):    cost = np.transpose(-y)@np.log(h) - np.transpose(1-y)@np.log(1-h) + (l/2)*np.transpose(t[1:])@t[1:]    cost = (1/m)*cost    return cost# Define first derivative of cost functiondef cost_dev(j, t, X=X, y=y, m=m):    dev = X[:, j]@(1/(1 + np.exp(-X@theta)) - y)    dev = (1/m)*dev    return dev# Define iterationscost_list = []theta_temp = np.zeros(theta.shape)theta_list = []for i in range(1000000):        for j in range(len(theta)):         if j == 0:            theta_temp[j] = theta[j] - a*cost_dev(j, theta)        else:            theta_temp[j] = theta[j]*(1 - (a*lmbd)/m) - a*cost_dev(j, theta)        theta = theta_temp            hypo = 1/(1 + np.exp(-X@theta))        theta_list.append(list(theta))    cost_val = cost(theta, hypo)    cost_list.append(cost_val)
I would like to talk a little about another popular optimization algorithm, Newton’s Method, that applies different approach to reach the global minimum of cost function. Similar to Gradient Descent, we firstly take the partial derivative of J(θ) that is the slope of J(θ), and note it as f(θ). Instead of decreasing θ by a certain chosen learning rate α multiplied with f(θ) , Newton’s Method gets an updated θ at the point of intersection of the tangent line of f(θ) at previous θ and x axis. After amount of iterations, Newton’s Method will converge at f(θ) = 0.
See the simplified plot above, Starting from the right, the yellow dotted line is the tangent of f(θ) at the θ0. It determines the position of θ1, and the distance from the θ0 to θ1 is Δ. This process repeats until finding the optimal θ that subjects to f(θ) = 0, which is θ3 in this plot. See the equations of updating θ as below.
Obviously Newton’s Method doesn’t need a chosen fixed learning rate, and the step size of each iteration is larger and varying as well, so that it usually converges faster and pretty efficient with a proper size of features and samples.
To be continued...