PyTorch [Tabular] — Binary Classification | by Akshaj Verma | Towards Data Science

We will use the lower back pain symptoms dataset available on Kaggle. This dataset has 13 columns where the first 12 are the features and the last column is the target column. The data set has 300 rows.
import numpy as npimport pandas as pdimport seaborn as snsimport matplotlib.pyplot as pltimport torchimport torch.nn as nnimport torch.optim as optimfrom torch.utils.data import Dataset, DataLoaderfrom sklearn.preprocessing import StandardScaler    from sklearn.model_selection import train_test_splitfrom sklearn.metrics import confusion_matrix, classification_report
df = pd.read_csv("data/tabular/classification/spine_dataset.csv")df.head()
There is a class imbalance here. While there’s a lot that can be done to combat class imbalance, it outside the scope of this blog post.
sns.countplot(x = 'Class_att', data=df)
PyTorch supports labels starting from 0. That is [0, n]. We need to remap our labels to start from 0.
df['Class_att'] = df['Class_att'].astype('category')encode_map = {    'Abnormal': 1,    'Normal': 0}df['Class_att'].replace(encode_map, inplace=True)
The last column is our output. The input is all the columns but the last one. Here we use .iloc method from the Pandas library to select our input and output columns.
X = df.iloc[:, 0:-1]y = df.iloc[:, -1]
We now split our data into train and test sets. We’ve selected 33% percent of out data to be in the test set.
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=69)
For neural networks to train properly, we need to standardize the input values. We standardize features by removing the mean and scaling to unit variance. The standard score of a sample x where the mean is u and the standard deviation is s is calculated as:
z = (x — u) / s
You can find more about standardization/normalization in neural nets here.
scaler = StandardScaler()X_train = scaler.fit_transform(X_train)X_test = scaler.transform(X_test)
To train our models, we need to set some hyper-parameters. Note that this is a very simple neural network, as a result, we do not tune a lot of hyper-parameters. The goal is to get to know how PyTorch works.
EPOCHS = 50BATCH_SIZE = 64LEARNING_RATE = 0.001
Here we define a Dataloader. If this is new to you, I suggest you read the following blog post on Dataloaders and come back.
towardsdatascience.com
## train dataclass TrainData(Dataset):        def __init__(self, X_data, y_data):        self.X_data = X_data        self.y_data = y_data            def __getitem__(self, index):        return self.X_data[index], self.y_data[index]            def __len__ (self):        return len(self.X_data)train_data = TrainData(torch.FloatTensor(X_train),                        torch.FloatTensor(y_train))## test data    class TestData(Dataset):        def __init__(self, X_data):        self.X_data = X_data            def __getitem__(self, index):        return self.X_data[index]            def __len__ (self):        return len(self.X_data)    test_data = TestData(torch.FloatTensor(X_test))
Let’s initialize our dataloaders. We’ll use a batch_size = 1 for our test dataloader.
train_loader = DataLoader(dataset=train_data, batch_size=BATCH_SIZE, shuffle=True)test_loader = DataLoader(dataset=test_data, batch_size=1)
Here, we define a 2 layer Feed-Forward network with BatchNorm and Dropout.
In our __init__() function, we define the what layers we want to use while in the forward() function we call the defined layers.
Since the number of input features in our dataset is 12, the input to our first nn.Linear layer would be 12. The output could be any number you want.
The only thing you need to ensure is that number of output features of one layer should be equal to the input features of the next layer.
Read more about nn.Linear in the docs. Similarly, we define ReLU, Dropout, and BatchNorm layers.
Once we’ve defined all these layers, it’s time to use them. In the forward() function, we take variable inputs as our input. We pass this input through the different layers we initialized.
The first line of the forward() functions takes the input, passes it through our first linear layer and then applies the ReLU activation on it. Then we apply BatchNorm on the output. Look at the following code to understand it better.
Note that we did not use the Sigmoid activation in our final layer during training. That’s because, we use the nn.BCEWithLogitsLoss() loss function which automatically applies the the Sigmoid activation.
class BinaryClassification(nn.Module):    def __init__(self):        super(BinaryClassification, self).__init__()        # Number of input features is 12.        self.layer_1 = nn.Linear(12, 64)         self.layer_2 = nn.Linear(64, 64)        self.layer_out = nn.Linear(64, 1)                 self.relu = nn.ReLU()        self.dropout = nn.Dropout(p=0.1)        self.batchnorm1 = nn.BatchNorm1d(64)        self.batchnorm2 = nn.BatchNorm1d(64)            def forward(self, inputs):        x = self.relu(self.layer_1(inputs))        x = self.batchnorm1(x)        x = self.relu(self.layer_2(x))        x = self.batchnorm2(x)        x = self.dropout(x)        x = self.layer_out(x)                return x
Once, we’ve defined our architecture, we check if our GPU is active. The amazing thing about PyTorch is that it’s super easy to use the GPU.
The variable device will either say cuda:0 if we have the GPU. If not, it’ll say cpu . You can follow along this tutorial even if you do not have a GPU without any change in code.
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")print(device)###################### OUTPUT ######################cuda:0
Next, we need to initialize our model. After initializing it, we move it to device . Now, this device is a GPU if you have one or it’s CPU if you don’t. The network we’ve used is fairly small. So, it will not take a lot of time to train on a CPU.
After this, we initialize our optimizer and decide on which loss function to use.
model = BinaryClassification()model.to(device)print(model)criterion = nn.BCEWithLogitsLoss()optimizer = optim.Adam(model.parameters(), lr=LEARNING_RATE)###################### OUTPUT ######################BinaryClassification(  (layer_1): Linear(in_features=12, out_features=64, bias=True)  (layer_2): Linear(in_features=64, out_features=64, bias=True)  (layer_out): Linear(in_features=64, out_features=1, bias=True)  (relu): ReLU()  (dropout): Dropout(p=0.1, inplace=False)  (batchnorm1): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)  (batchnorm2): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True))
Before we start the actual training, let’s define a function to calculate accuracy.
In the function below, we take the predicted and actual output as the input. The predicted value(a probability) is rounded off to convert it into either a 0 or a 1.
Once that is done, we simply compare the number of 1/0 we predicted to the number of 1/0 actually present and calculate the accuracy.
Note that the inputs y_pred and y_test are for a batch. Our batch_size was 64. So, this accuracy is being calculated for 64 predictions(tensors) at a time.
def binary_acc(y_pred, y_test):    y_pred_tag = torch.round(torch.sigmoid(y_pred))    correct_results_sum = (y_pred_tag == y_test).sum().float()    acc = correct_results_sum/y_test.shape[0]    acc = torch.round(acc * 100)        return acc
The moment we’ve been waiting for has arrived. Let’s train our model.
You can see we’ve put a model.train() at the before the loop. model.train() tells PyTorch that you’re in training mode.
Well, why do we need to do that? If you’re using layers such as Dropout or BatchNorm which behave differently during training and evaluation, you need to tell PyTorch to act accordingly. While the default mode in PyTorch is the train, so, you don’t explicitly have to write that. But it’s good practice.
Similarly, we’ll call model.eval() when we test our model. We’ll see that below.
Back to training; we start a for-loop. At the top of this for-loop, we initialize our loss and accuracy per epoch to 0. After every epoch, we’ll print out the loss/accuracy and reset it back to 0.
Then we have another for-loop. This for-loop is used to get our data in batches from the train_loader.
We do optimizer.zero_grad() before we make any predictions. Since the backward() function accumulates gradients, we need to set it to 0 manually per mini-batch.
From our defined model, we then obtain a prediction, get the loss(and accuracy) for that mini-batch, perform backpropagation using loss.backward() and optimizer.step() . Finally, we add all the mini-batch losses (and accuracies) to obtain the average loss (and accuracy) for that epoch.
This loss and accuracy is printed out in the outer for loop.
model.train()for e in range(1, EPOCHS+1):    epoch_loss = 0    epoch_acc = 0    for X_batch, y_batch in train_loader:        X_batch, y_batch = X_batch.to(device), y_batch.to(device)        optimizer.zero_grad()                y_pred = model(X_batch)                loss = criterion(y_pred, y_batch.unsqueeze(1))        acc = binary_acc(y_pred, y_batch.unsqueeze(1))                loss.backward()        optimizer.step()                epoch_loss += loss.item()        epoch_acc += acc.item()            print(f'Epoch {e+0:03}: | Loss: {epoch_loss/len(train_loader):.5f} | Acc: {epoch_acc/len(train_loader):.3f}')###################### OUTPUT ######################Epoch 001: | Loss: 0.04027 | Acc: 98.250Epoch 002: | Loss: 0.12023 | Acc: 96.750Epoch 003: | Loss: 0.02067 | Acc: 99.500Epoch 004: | Loss: 0.07329 | Acc: 96.250Epoch 005: | Loss: 0.04676 | Acc: 99.250Epoch 006: | Loss: 0.03005 | Acc: 99.500Epoch 007: | Loss: 0.05777 | Acc: 98.250Epoch 008: | Loss: 0.03446 | Acc: 99.500Epoch 009: | Loss: 0.03443 | Acc: 100.000Epoch 010: | Loss: 0.03368 | Acc: 100.000Epoch 011: | Loss: 0.02395 | Acc: 100.000Epoch 012: | Loss: 0.05094 | Acc: 98.250Epoch 013: | Loss: 0.03618 | Acc: 98.250Epoch 014: | Loss: 0.02143 | Acc: 100.000Epoch 015: | Loss: 0.02730 | Acc: 99.500Epoch 016: | Loss: 0.02323 | Acc: 100.000Epoch 017: | Loss: 0.03395 | Acc: 98.250Epoch 018: | Loss: 0.08600 | Acc: 96.750Epoch 019: | Loss: 0.02394 | Acc: 100.000Epoch 020: | Loss: 0.02363 | Acc: 100.000Epoch 021: | Loss: 0.01660 | Acc: 100.000Epoch 022: | Loss: 0.05766 | Acc: 96.750Epoch 023: | Loss: 0.02115 | Acc: 100.000Epoch 024: | Loss: 0.01331 | Acc: 100.000Epoch 025: | Loss: 0.01504 | Acc: 100.000Epoch 026: | Loss: 0.01727 | Acc: 100.000Epoch 027: | Loss: 0.02128 | Acc: 100.000Epoch 028: | Loss: 0.01106 | Acc: 100.000Epoch 029: | Loss: 0.05802 | Acc: 98.250Epoch 030: | Loss: 0.01275 | Acc: 100.000Epoch 031: | Loss: 0.01272 | Acc: 100.000Epoch 032: | Loss: 0.01949 | Acc: 100.000Epoch 033: | Loss: 0.02848 | Acc: 100.000Epoch 034: | Loss: 0.01514 | Acc: 100.000Epoch 035: | Loss: 0.02949 | Acc: 100.000Epoch 036: | Loss: 0.00895 | Acc: 100.000Epoch 037: | Loss: 0.01692 | Acc: 100.000Epoch 038: | Loss: 0.01678 | Acc: 100.000Epoch 039: | Loss: 0.02755 | Acc: 100.000Epoch 040: | Loss: 0.02021 | Acc: 100.000Epoch 041: | Loss: 0.07972 | Acc: 98.250Epoch 042: | Loss: 0.01421 | Acc: 100.000Epoch 043: | Loss: 0.01558 | Acc: 100.000Epoch 044: | Loss: 0.01185 | Acc: 100.000Epoch 045: | Loss: 0.01830 | Acc: 100.000Epoch 046: | Loss: 0.01367 | Acc: 100.000Epoch 047: | Loss: 0.00880 | Acc: 100.000Epoch 048: | Loss: 0.01046 | Acc: 100.000Epoch 049: | Loss: 0.00933 | Acc: 100.000Epoch 050: | Loss: 0.11034 | Acc: 98.250
After training is done, we need to test how our model fared. Note that we’ve used model.eval() before we run our testing code. To tell PyTorch that we do not want to perform back-propagation during inference, we use torch.no_grad() which reduces memory usage and speeds up computation.
We start by defining a list that will hold our predictions. Then we loop through our batches using the test_loader. For each batch —
We make the predictions using our trained model.
Round off the probabilities to 1 or 0.
Move the batch to the GPU from the CPU.
Convert the tensor to a numpy object and append it to our list.
Flatten out the list so that we can use it as an input to confusion_matrix and classification_report .
y_pred_list = []model.eval()with torch.no_grad():    for X_batch in test_loader:        X_batch = X_batch.to(device)        y_test_pred = model(X_batch)        y_test_pred = torch.sigmoid(y_test_pred)        y_pred_tag = torch.round(y_test_pred)        y_pred_list.append(y_pred_tag.cpu().numpy())y_pred_list = [a.squeeze().tolist() for a in y_pred_list]
Once we have all our predictions, we use the confusion_matrix() function from scikit-learn to calculate the confusion matrix.
confusion_matrix(y_test, y_pred_list)###################### OUTPUT ######################array([[23,  8],       [12, 60]])
To obtain the classification report which has precision, recall, and F1 score, we use the function classification_report .
print(classification_report(y_test, y_pred_list))###################### OUTPUT ######################precision    recall  f1-score   support           0       0.66      0.74      0.70        31           1       0.88      0.83      0.86        72    accuracy                           0.81       103   macro avg       0.77      0.79      0.78       103weighted avg       0.81      0.81      0.81       103
Thank you for reading. Suggestions and constructive criticism are welcome. :)
This blogpost is a part of the series —” How to train you Neural Net”. You can find the series here.
You can find me on LinkedIn and Twitter. If you liked this, check out my other blogposts.