Gradient Descent Training With Logistic Regression | by Rina Buoy | Towards Data Science

Gradient descent algorithm and its variants ( Adam, SGD etc. ) have become very popular training (optimisation) algorithm in many machine learning applications. Optimisation algorithms can be informally grouped into two categories — gradient-based and gradient-free(ex. particle swarm, genetic algorithm etc.). As you can guess, gradient descent is a gradient-based algorithm. Why gradient is important in training machine learning?
The objective of training a machine learning model is to minimize the loss or error between ground truths and predictions by changing the trainable parameters. And gradient, which is the extension of derivative in multi-dimensional space, tells the direction along which the loss or error is optimally minimized. If you recall from vector calculus class, gradient is defined as the maximum rate of change. Therefore, the formula for gradient descent is simply:
θj is a trainable parameter, j. α is a learning rate. J(θ) is a cost function.
In the below figure, the shortest from the starting point ( the peak) to the optima ( valley) is along the gradient trajectory. The same principle applies the multi-dimensional space which is generally the case for machine learning training.
To demonstrate how gradient descent is applied in machine learning training, we’ll use logistic regression.
To understand how LR works, let’s imagine the following scenario: we want to predict the sex of a person (male = 0, female = 1) based on age (x1), annual income (x2) and education level (x3). If Y is the predicted value, a logistic regression model for this problem would take the form:
Z = b0 + b1(x1) + b2(x2) + b3(x3)
Y = 1.0 / (1.0 + e^-Z)
b0 is often called ‘bias’ and b1, b2 and b3 are called ‘weights’.
Z has the same form as a linear regression while Y is a sigmoid activation function. Y takes a value between 0 and 1. If Y is less than 0.5, we conclude the predicted output is 0 and if Y is greater than 0.5, you conclude the output is 1.
Now, we are ready to look at a more formal form of LR below:
Φn is the augmented transformation of Xn in feature space. tn is the class label. σ is a sigmoid activation. W is a weight vector (including bias term). p(C1|Φ) and p(C2|Φ) is the probability of assigning to C1 and C2 given Φ, respectively.
Given the above formulation, the main goal here is to maximise the likelihood of observing the data given the weight (W). The likelihood function is a joint-distribution of the observed data and is given below:
Π is a product operator.
From the likelihood function, it can be observed that y is Bernoulli distributed.
When working with probability, it is desirable to convert to logarithm since logarithm turns a product into a sum and thus avoid the issue of taking a product with a very small number(typically for probability). Below are the negative log-likelihood (NLL) and its gradient with respect to weights. NLL is used to turn a maximization into a minimization problem. Essentially, minimizing NLL is equivalent to maximizing the likelihood.
The binary case of LR can be extended to the multiclass case with some changes of notation.
Let’s assume there is K class. So, p(Ck) is the probability of assigning to class k given Φ.
Instead of sigmoid activation, softmax activation is used to convert class score (ak) into proper probability.
W is a weight matrix ( DxK) — D is feature space dimension.
The likelihood function and negative likelihood (NLL) are given below.
y is now Multinoulli distributed.
MNIST is a classical dataset, which consists of black-and-white images of hand-drawn digits (between 0 and 9). We will implement a multiclass logistic regression to classify digits in MNIST using PyTorch. Since we want to demonstrate the gradient descent algorithm, we do not use the bulit-in algorithm from torch.optim. Rather than calculating gradients manually, we will use torch.autograd for simplicity. This demo is taken from PyTorch website.
The below codes download the dataset (train and validation set) and also convert into respective numpy arrays.
from pathlib import Pathimport requestsimport pickleimport gzipDATA_PATH = Path("data")PATH = DATA_PATH / "mnist"PATH.mkdir(parents=True, exist_ok=True)URL = "http://deeplearning.net/data/mnist/"FILENAME = "mnist.pkl.gz"if not (PATH / FILENAME).exists():  content = requests.get(URL + FILENAME).content  (PATH / FILENAME).open("wb").write(content)with gzip.open((PATH / FILENAME).as_posix(), "rb") as f:  ((x_train, y_train), (x_valid, y_valid), _) = pickle.load(f,   encoding="latin-1")
When working with PyTorch, we need to convert the above numpy arrays to tensors.
import torchx_train, y_train, x_valid, y_valid = map(torch.tensor, (x_train,  y_train, x_valid, y_valid))
Next, we will create and initialize weights and bias tensors. We use Xavier initialisation for weights while initializing bias with zero value. Since we want torch.autograd to take care of gradient calculations, we need to set requires_grad to True so that PyTorch can keep track of operations which are required for gradient calculations.
import mathweights = torch.randn(784, 10) / math.sqrt(784)weights.requires_grad_()bias = torch.zeros(10, requires_grad=True)
We need to evaluate the likelihood which is yk(Φ). To get yk(Φ), we first need to evaluate ak. Instead of returning yk(Φ), it returns log(yk(Φ)) which is useful for calculating loss function later.
def log_softmax(x):    return x - x.exp().sum(-1).log().unsqueeze(-1)def model(xb):    return log_softmax(xb @ weights + bias)
Now, we can use the likelihood to compute the overall negative log-likelihood which is the loss function of MNIST logistic regression.
def nll(input, target):    return -input[range(target.shape[0]), target].mean()
Until now, we have implemented all the necessary functions needed for training MNIST logistic regression. We will implement mini-batch training.
bs = 64  # batch sizeloss_func = nlllr = 0.5  # learning rateepochs = 2  # how many epochs to train forfor epoch in range(epochs):    for i in range((n - 1) // bs + 1):        start_i = i * bs        end_i = start_i + bs        xb = x_train[start_i:end_i]        yb = y_train[start_i:end_i]        pred = model(xb)        loss = loss_func(pred, yb)        loss.backward()        with torch.no_grad():            weights -= weights.grad * lr            bias -= bias.grad * lr            weights.grad.zero_()            bias.grad.zero_()
.backward() on loss_func does all the gradient calculations which are required for parameter update. Once gradients are computed with .backward(), weights and bias are updated by the product of gradient and learning rate. Learning rate (LR) is used to control the convergence. Large LR can overshoot while small LR can slow down the convergence.
Once weights and bias are updated, their gradients are set to zero; otherwise, gradients are accumulated in the next batches.
The gradient descent implemented above is very basic, yet enough to demonstrate how it works. Modern machine learning frameworks like PyTorch and TensorFlow have far more sophisticated variants of gradient descent like SGD, Adam etc. Still, understanding how gradient descent works is beneficial when we need to train machine learning models.
The equations in this article are taken from ‘ Pattern recognition and machine learning’ by Christopher M. Bishop.