Logistic Regression with PyTorch
Tags: Programming, Pytorch, Logistic Regression, Python, Machine Learning

We learned about linear regression in the last post, and now we move to logistic regression.

As we cannot use linear regression for a classification task, we use logistic, which is an extension to linear regression for a classification task.

Logistic regression is a good approach to problems that require probability as output.

There are two ways you can use output,

1] you can convert them into binary, 2] you can use the probabilities as it is.

let’s look at both the example, 1] how we can use probabilities as it is,

Suppose we create a model to predict a striker's probability when he is playing an away match. We will call that probability :

p(score | Away_Match)

If a model predicts a p(score | Away_Match) of 0.05, then overall away matches, the striker will score approximately 1 goal.

goal = p(score|Away_Match) * Total_away_matches :

0.05 * 19 = 0.95 ~ 1 goal.

2] Let’s now see the second way of binary classification,

Data set

Here, our task is to determine whether a tumor will be benign (harmless) or malignant (harmful) based on leukocyte (white blood cells) count and blood pressure.

import pandas as pd

data = pd.read_csv('tumor.csv')

data.head()

Head of data

Let’s define X and y,

X = data[['leukocyte_count', 'blood_pressure']].values

y = data['tumor_class'].values

Now, let’s plot the data.

import matplotlib.pyplot as plt

colors = {'benign' : 'orange', 'malignant' : 'green'}

plt.scatter(data['leukocyte_count'], data['blood_pressure'], c = [colors[item] for item in y], edgecolors = 'k')

plt.xlabel('leukocyte_count')

plt.ylabel('blood_pressure')

plt.legend(['malignant', 'benign'], loc = 'upper right')

plt.show()

Splitting the data

We will now randomly split our data into 3 sets: Train, Validation, and Test.

Train: used to train our model.

Val: used to validate our model’s performance during training.

Test: used to evaluate our fully trained model.

Here we are going to use the stratify parameter while splitting the data. When we have a classification task, then we must ensure to have the same class distribution for each split (it helps us predict the class well)

For example, Our variable y is a binary categorical variable with values ‘’benign’ and ‘malignant’ and there are 38% of ‘benign’ and 61% of ‘malignant’, stratify = y will make sure that your random split has 38% of ‘benign’ and 61% of ‘malignant’.

# Importing library

from sklearn.model_selection import train_test_split # Defining sets

TRAIN_SIZE = 0.7

VAL_SIZE = 0.15

TEST_SIZE = 0.15

SHUFFLE = True # Creating a function

def split(X, y, val_size, test_size, shuffle):

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = test_size, stratify = y, shuffle = shuffle)

X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size = val_size, stratify = y_train, shuffle = shuffle)

return X_train, X_val, X_test, y_train, y_val, y_test X_train, X_val, X_test, y_train, y_val, y_test = split(X, y, VAL_SIZE, TEST_SIZE, SHUFFLE)

We can see that our class labels are text, and we need to do something about it and hurray! sklearn got us….

Before this, we only worked with numerical data, but now we have labels in the text.

So, we will need to convert this to a numeric form.

For converting labels to numeric form, there is one method called ‘LabelEncoder’ in sklearn. We will apply that.

from sklearn.preprocessing import LabelEncoder

le = LabelEncoder() # fitting on train data

y_le = le.fit(y_train)

classes = y_le.classes_

Now, we will transform the labels into tokens

# Convert labels to tokens

# Before transforming - y_train[0] - 'malignant'

print(f'y_train[0] : {y_train[0]}')

y_train = y_le.transform(y_train)

y_val = y_le.transform(y_val)

y_test = y_le.transform(y_test) # After transforming - y_train[0] - 1

print(f'y_train[0] : {y_train[0]}')



Standardize data

We need to standardize our data (zero mean and unit variance)

z(standardized value) = xi-μ/σ

xi = inputs

μ = mean

σ = standard deviation

from sklearn.preprocessing import StandardScaler

ss = StandardScaler().fit(X_train)

Now, we will apply this fitted data to training and test data

X_train = ss.transform(X_train)

X_val = ss.transform(X_val)

X_test = ss.transform(X_test)

Now, we will define our INPUT DIMENSION and OUTPUT DIMENSION

INPUT_DIM = X_train.shape[1]

NUM_CLASSES = len(classes)

Now, we will use a torch.nn.functional to define a custom layer

import torch.nn.functional as F

from torch import nn

class LogisticRegression(nn.Module):

def __init__(self, input_dim, num_classes):

super(LogisticRegression, self).__init__()

self.fc1 = nn.Linear(input_dim, num_classes)



def forward(self, x_in, apply_softmax = False):

y_pred = self.fc1(x_in)

if apply_softmax:

y_pred = F.softmax(y_pred, dim = 1)

return y_pred

Like we did with Linear regression, we create a linear layer and then apply softmax on it (which we made with nn.functional).

nn.module — We use this when we want to create a custom model. It means we are defining our model in our way. It is initialized with the input and output dimension, which we defined above.

nn.Linear — This applies a linear transformation to our input and output samples.

It’s a fully connected layer, which has input and output. Now, it will have a weight that’s applied to its input; its output will be [XW +b] since it’s just entirely linear, with no activation function.

F.softmax — https://pytorch.org/docs/stable/nn.functional.html?highlight=softm#torch.nn.functional.softmax

Now, we will initialize the model.

model = LogisticRegression(input_dim = INPUT_DIM, num_classes = NUM_CLASSES)

Loss function

The loss function for LR is log-loss :

-ylog(y’)

y’-predicted value

y-actual value

loss_fn = nn.CrossEntropyLoss()

Metrics

Here, I’m using accuracy as my metrics

def accuracy_fn(y_pred, y_true):

n_correct = torch.eq(y_pred, y_true).sum().item()

accuracy = (n_correct / len(y_pred)) * 100

return accuracy

Optimizer

We saw the loss function above (It’s a way of measuring how far you are from your destination). Having this in mind, let us now understand what an optimizer is?

Let’s take an example of pottery. It is a process of forming vessels with clay and other ceramic materials, which are fired at high temperatures to give them a hard and durable form.

Just like this process, while training our model, here the potter is our optimizer, clay and ceramic materials are parameters (weights) and loss function, to make the model more accurate or more errorless.

Now, the pots are of different shapes so, the potter will try to get the pot shape into the desired shape, same as the potter. The optimizer will try to get the correct shape of weights.

learning_rate = 1e-1

optimizer = torch.optim.Adam(model.parameters(), lr = learning_rate)

Here, we are using the ADAM optimizer; it uses adaptive learning rates. We do not have to manually tune the learning rates as we do in the learning rate scheduler.

Sometimes optimizers tend to overshoot destination, and ADAM tries to overcome it by taking bigger steps at first and smaller steps as it reaches the point of our destination.

Now, we need to convert our data into tensors. For that, we are using torch.Tensor.

# Convert data to tensors

X_train = torch.Tensor(X_train)

y_train = torch.LongTensor(y_train)

X_val = torch.Tensor(X_val)

y_val = torch.LongTensor(y_val)

X_test = torch.Tensor(X_test)

y_test = torch.LongTensor(y_test)

Now, let’s train our model

EPOCHS = 100

for epoch in range(EPOCHS):

y_pred = model(X_train)

loss = loss_fn(y_pred, y_train)

optimizer.zero_grad()

loss.backward()

optimizer.step()

if epoch%10==0:

predictions = y_pred.max(dim=1)[1] # class

accuracy = accuracy_fn(y_pred=predictions, y_true=y_train)

print (f"Epoch: {epoch} | loss: {loss:.2f}, accuracy: {accuracy:.1f}")

Accuracy

# Accuracy

train_acc = accuracy_score(y_train, pred_train)

test_acc = accuracy_score(y_test, pred_test)

print (f"train acc: {train_acc:.2f}, test acc: {test_acc:.2f}")

Visualizing the boundary

Classification Report

As we can see, there are so many new terms in the classification report so, let’s understand them one by one.

Performance measures

1] Accuracy: Number of correctly classified points / Number of points in a set

Let’s take an example :

Let’s say we have 100 points, Out of which 60 are positive points and 40 are negative points.

Now, out of these 100 points, our model correctly classified 88 points(53 from positive class and 35 from negative class), and the incorrect classification, i.e., error, is 12 points.

So the accuracy of our model is 88%, and the error is 12 %.

This metric is easy to understand

It does not perform well on unbalanced data.

Confusion matrix

Confusion matrix

Confusion matrix

Confusion matrix

Precision and Recall

Precision and Recall

Inference

X1 = pd.DataFrame([{'leukocyte_count': 13, 'blood_pressure': 16}])

# Standardize

X1 = ss.transform(X1)

Prediction

# Predict

y1 = model(torch.Tensor(X), apply_softmax=True)

prob, _class = y.max(dim=1)

print (f"You have {classes[_class.detach().numpy()[0]]} tumor with {prob.detach().numpy()[0]*100.0:.0f}% of probability")

Result for the inference

Code-https://github.com/pratikraut1/Logistic-regression-with-pytorch

Disclaimer-Images without links in the caption are by the author; otherwise, a link is provided.