Recreating Keras code in PyTorch- an introductory tutorial | by Dr. Varshita Sher | Towards Data Science

I love Keras, there I said it! However...
As an applied data scientist, nothing gives me more pleasure than quickly whipping up a functional neural network with as little as three lines of code! However, as I have begun to delve deeper into the dark web of neural nets, I would like to accept the fact that Pytorch does allow you a much larger control over the architecture of your network.
Given that most of us are pretty comfortable with Keras (if not, see here for a warm intro to Keras), learning to create a similar network in Pytorch (whilst learning Pytorch basics) isn’t challenging at all. Let’s begin!
Note: We will not be creating a replica of the Keras code as I would like to introduce more features and functionalities of PyTorch in this introductory tutorial!
Here is the code snippet (from my previous post on neural networks in Keras) for creating the model architecture, compiling the model, and finally training the model. It is a loan assessment model that outputs whether a loan should be accepted or rejected.
# Model architecturemodel_m = Sequential([    Dense(units = 8, input_shape= (2,), activation = 'relu'),    Dense(units = 16, activation = 'relu'),    Dense(units = 2, activation = 'softmax') ])# Model compilationmodel_m.compile(optimizer= Adam(learning_rate = 0.0001),               loss = 'sparse_categorical_crossentropy',               metrics = ['accuracy']              )# Model Training and Validationmodel_m.fit(x = scaled_train_samples_mult,           y = train_labels,           batch_size= 10,           epochs = 30,           validation_split= 0.1,           shuffle = True,          verbose = 2          )
To summarize, we have built a network with three hidden layers, all of which are Dense. The activation functions for the three hidden layers are relu,reluand softmax, respectively. input_shape is a tuple (2,0) meaning we have two predictor features. Adam optimizer along with a learning rate lr = 0.0001 has been used to update network weights iteratively based on training data. The loss we will be monitoring on each iteration is sparse_categorical_crossentropy.The accuracy will be used to judge how good the network is. For training, we will be using 30 epochs along with a batch_size of 10. If the meaning of any of the aforementioned terms is unclear, please see here or here.
Note: If you want to follow along this tutorial, check out the notebook on Github.
To maintain some level of consistency with our earlier network built using Keras, I will be using the same dataset, i.e. a loan application dataset. This is how it looks like:
After applying some preprocessing on the dataset, mainly for scaling all the input features (age and area) on a 0–1 scale and label encoding the outcome feature ( application_outcome), this is what we end up with:
We will be using the torch.utils.data.Dataset class in PyTorch to define a Dataset object for the loan application dataset. We will be calling it CVSDataset.
We are going to have four methods in our CSVDataset class — __init__ , __len__ , __getitem__ and get_splits.
class CSVDataset(Dataset):        # reading the csv and defining predictor and output columns    def __init__(self):            # store the input and output features        self.X = df.values[:,:-1]         self.y = df.values[:,-1]             # ensure all data is numerical - type(float)        self.X = self.X.astype('float32')        self.y = self.y.astype('float32')        # number of rows in dataset    def __len__(self):        return len(self.X)        # get a row at an index    def __getitem__(self, index):        return [self.X[index], self.y[index]]    # split into train and testset - using `random_split`    def get_splits(self, split_ratio = 0.2):        test_size = round(split_ratio * len(self.X))        train_size = len(self.X) - test_size                return random_split(self, [train_size, test_size])
Note: At a very basic level, the Dataset class you extend for your own dataset should have __init__,__len__() and __getitem__ methods.
We begin by defining the predictors X and output features y in __init__() and ensuring the type is float since neural networks work only with numerical data.
__len__ return the number of rows in the dataset and __getitem__ returns item at a particular index in the dataset.
Additionally, we have also defined an (optional) method get_splits that lets us split our dataset into train and test set. We could have very well done this using train_test_split in Sklearn, but this is just a way of showing how it can be done within PyTorch using torch.utils.data.random_split.
This is where we are defining what all hidden layers we want in our network and how will the layers interact with each other. To do so, we are going to make use of torch.nn.Module class and extend it. We will be calling our class myNeuralNetwork.
It contains the __init__ method where we define the layers and a forward method that explains how the input is propagated forward through the defined layers.
class myNeuralNetwork(Module):      def __init__(self, n_inputs):        .        .        .      def forward(self,X):        .        .        .
Let’s look at the init method first:
def __init__(self, n_inputs):                # calling constructor of parent class        super().__init__()                # defining the inputs to the first hidden layer        self.hid1 = Linear(n_inputs, 8)         kaiming_uniform_(self.hid1.weight, nonlinearity='relu')        self.act1 = ReLU()                # defining the inputs to the second hidden layer        self.hid2 = Linear(8, 16)        kaiming_uniform_(self.hid2.weight, nonlinearity='relu')        self.act2 = ReLU()                # defining the inputs to the third hidden layer        self.hid3 = Linear(16, 2)        xavier_uniform_(self.hid3.weight)        self.act3 = Softmax(dim=1)
We begin by calling the constructor of the parent class using super().__init__(). Next, we define three hidden layers hid1, hid2 and hid3, along with their weights initialization and activation functions — act1, act2 , and act3 .
If you recall from the summary of the Keras model at the beginning of the article, we had three hidden layers all of which were Dense. Pytorch equivalent of Keras Dense layers is Linear.
The first hidden linear layer hid1 takes n_inputsnumber of inputs and outputs 8 neurons/units.
Note: n_inputs roughly translates to how many predictor columns we have (in our case 2).
The second hidden layer takes 8 neurons as input and outputs 16 units.
The third hidden layer takes 16 neurons as input and produces 2 units as output (which are the probabilities of the loan application being approved or rejected since the softmax activation function is at work in this layer).
The activations functions with respect to each hidden layer have been stored in act1, act2 and act3.
Note: Common examples of activations functions in Pytorch include ReLu , Sigmoid , LogSigmoid , etc.
Additionally, PyTorch lets you initialize the weight tensors for each hidden layer. Common examples include kaiming_uniform, xavier_uniform and orthogonal. You can read about them in more detail on the documentation page.
Now, let’s look at the forward method :
def forward(self, X):                #input and act for layer 1        X = self.hid1(X)        X = self.act1(X)                #input and act for layer 2        X = self.hid2(X)        X = self.act2(X)                #input and act for layer 3        X = self.hid3(X)        X = self.act3(X)                return X
The inputs and activations for each layer are defined in this function. To summarize, the input X goes in to hid1 where the first activation function act1 is applied to it. The output is then passed onto hid2 where the second activation function act2 is applied and so on. Since the last layer has a softmax activation function, the returned X will be a tensor containing n elements where n is the number of output classes (two in our case — approved vs rejected).
A tensor is nothing but the Pytorch’s very own Numpy arrays. It is a generic n-dimensional array to be used for arbitrary numeric computations.
Before we train, it is essential to load the data for training using torch.utils.data.DataLoader. Why is that needed, you ask? It essentially creates batches (for both train and test set) to be sent as input to the model. It takes as input the sample of rows, the batch size, and whether or not to shuffle the rows.
Note: We explicitly set shuffle = False in case of test_dl because we need unshuffled labels at the end to draw confusion matrix.
# Preparing the dataset before training the model# load the datasetdataset = CSVDataset()# get the train and test splittrain, test = dataset.get_splits()# prepare dataloaders - train_dl = DataLoader(train, batch_size = 32, shuffle = True)test_dl = DataLoader(test, batch_size= 32, shuffle= False)
We first define the model by creating myNeuralNetwork object and passing 2 as input. 2 here refers to the number of input features.
We are also going to set epochs = 10 meaning the model is going to train on all of the data 10 times.
For the training process, we are going to use the Stochastic Gradient Descent (SGD)optimizer.
The choices of optimizers include Adam , AdaGrad, SparseAdam, SGD, etc and their use-cases can be found in the documentation.
Finally, we are going to monitor the CrossEntropyLoss while training.
# define the networkmodel = myNeuralNetwork(2) # 2 because we only have 2 input features# define the number of epochsepochs = 10# define the optimizer - SGDoptimizer = SGD(model.parameters(), lr=0.01, momentum=0.9)# define the loss functioncriterion = CrossEntropyLoss()
In order to train the network, we are going to use two for loops. The outer loop in the code below is required for the number of training epochs and the inner loop is used to iterate over the batches of samples intrain_dl . That is, for each batch of samples in train_dl, we are going to clear the gradients, compute the model output, calculate loss and do backpropagation to update the weights.
# iterate through all the epochfor epoch in range(epochs):    # go through all the batches generated by dataloader    for i, (inputs, targets) in enumerate(train_dl):            # clear the gradients            optimizer.zero_grad()            # compute the model output            yhat = model(inputs)            # calculate loss            loss = criterion(yhat, targets.type(torch.LongTensor))            # credit assignment            loss.backward()            # update model weights            optimizer.step()
To evaluate the model, we are again going to use a for loop to go over all the batches in test_dl. For each sample, we are going to pass it to the model and get the output i.e. y_pred . Since y_pred contains two probability values (one for approved and one for rejected), we are going to pick the index of the highest probability as the final output.
Finally, we store the actual output and the predicted output in two lists — actuals and predictions.
# Evaluate the modelpredictions, actuals = list(), list()# loop over all batches in test setfor i, (inputs, targets) in enumerate(test_dl):    # pass input to the model    y_pred = model(inputs)     # retrieve the numpy array    y_pred = y_pred.detach().numpy()    # pick the index of the highest values    res = np.argmax(y_pred, axis = 1)         # actual output    actual = targets.numpy()    actual = actual.reshape(len(actual), 1)        # store the values in respective lists    predictions.append(list(res))    actuals.append(list(actual))    actuals = [val for sublist in vstack(list(chain(*actuals))) for val in sublist]predictions = [val for sublist in vstack(list(chain(*predictions))) for val in sublist]
To print the accuracy score, we will be using accuracy_score fromsklearn.metrics
# evaluating how good the model is!from sklearn.metrics import accuracy_scoreprint("The accuracy is" , accuracy_score(actuals, predictions))*********************The accuracy is 0.7495
Wohooo we did it !
We have created a basic network using PyTorch. Hopefully, it wasn’t too much information overload. If you want to have access to the entire code, check out the notebook on Github. In the next tutorial, we are going to dig a little deeper into some of the advanced capabilities that Pytorch provides.
Until then :)