Deep Learning with PyTorch. First contact with PyTorch for... | by Jordi TORRES.AI | Towards Data Science

We are to going to be using PyTorch in many of the posts in this series, so the reader need to make sure that she/he is familiar with it. This post will introduce the reader to the basics features of PyTorch which enables us to implement Deep Learning models using Python language. The post doesn’t pretend to be a complete manual of PyTorch, it only introduces the minimum knowledge of PyTorch to start coding neural networks in PyTorch and we will be introducing new features as we need them throughout the series. Enjoy it!
Spanish version of this publication
medium.com
The clear leaders in Deep Learning frameworks arena are now the Google-developed TensorFlow and the Facebook-developed PyTorch, and they are pulling away from the rest of the market in usage, share, and momentum.
Three years ago appeared the first version of PyTorch and without question, it is gaining great momentum. Initially incubated by Facebook, PyTorch rapidly developed a reputation from being an ideal flexible framework for rapid experimentation and prototyping gaining thousands of fans within the Deep Learning community. For instance, PhD students in my research team prefer to use PyTorch because it allows to them to write native looking Python code and still get all the benefits of a good framework like auto-differentiation and built-in optimization. This is the reason I decided to use PyTorch in this series.
Though PyTorch has gained momentum in the marketplace thanks to Facebook (and AWS), TensorFlow continues to be ahead in all aspects and is the most used for the industry right now. Yo can read this brief post “TensorFlow vs PyTorch: The battle continues” for a more detail about both environments.
I suggest using the Colaboratory (Colab) offered by Google to execute the code described in this post. It basically consists of a Jupyter notebook environment that requires no configuration and runs completely in the Cloud allowing the use different Deep Learning libraries as PyTorch and TensorFlow. One important feature of Colab is that it provides GPU (and TPU) totally free. Detailed information about the service can be found on the faq page.
By default, Colab notebooks run on CPU. You can switch your notebook to run with GPU (or TPU). In order to obtain access to one GPU you need to choose the tab “Runtime” and then select “Change runtime type” as shown in the following figure:
When a pop-up window appears, select GPU. Ensure “Hardware accelerator” is set to GPU (the default is CPU). Afterwards, ensure that you are connected to the runtime (there is a green check next to “CONNECTED” in the menu ribbon):
Now you are able to run the code presented in this post. I suggest to copy & paste the code of this post in a Colab notebook in order to see the execution meanwhile you are reading this post. Ready?
The entire code of this post can be found on GitHub and can be run as a Colab google notebook using this link.
In this post we will program a neural network model that classifies handwritten digits presented in the previous post. Remember that we created a mathematical model that, given an image, the model identify the number it represents returning a vector with 10 positions indicating the likelihood of each of the ten possible digits.
In order to guide the explanation, we will follow a list of steps to be taken to program a neural network:
Import required librariesLoad and Preprocess the DataDefine the ModelDefine the Optimizer and the Loss FunctionTrain the ModelEvaluate the Model
Import required libraries
Load and Preprocess the Data
Define the Model
Define the Optimizer and the Loss Function
Train the Model
Evaluate the Model
Let’s go for it!
We always need to import torch, the core Python library for PyTorch. For our example we will also import the torchvision package, as well as the usual libraries numpy and matplotlib.
import torch import torchvision
For clarity of the code we could define here some hyperparameters that we will need for training:
import numpy as np import matplotlib.pyplot as plt EPOCH = 10 BATCH_SIZE= 64
Next step is to load data that will be used to train our neural network . We will use the MNIST dataset already introduced in the previous post, which can be downloaded from The MNIST database page using torchvision.dataset. PyTorch Datasets are objects that return a single datapoint on request. Then it is passed on to a Dataloader which handles batching of datapoints and parallelism. This is the code for our example:
xy_trainPT = torchvision.datasets.MNIST(root='./data',              train=True, download=True,transform=             torchvision.transforms.Compose(             [torchvision.transforms.ToTensor()]))xy_trainPT_loader = torch.utils.data.DataLoader                    (xy_trainPT, batch_size=BATCH_SIZE)
Because data is usually too large to fit data into CPU or GPU memory at once, it is split into batches of equal size. Every batch includes data samples and target labels, and both of them have to be tensors (which we will present below). The BATCH_SIZE argument indicates the number of data that we will use for each update of the model parameters.
This dataset contains 60,000 images of hand-made digits to train the model and it is ideal for entering pattern recognition techniques for the first time without having to spend much time preprocessing and formatting data, both very important and expensive steps in the analysis of data and of special complexity when working with images.
We can verify that the previous code have loaded the expected data with the library matplotlib.pyplot :
fig = plt.figure(figsize=(25, 4)) for idx in np.arange(20):   image, label = xy_trainPT [idx]   ax = fig.add_subplot(2, 20/2, idx+1, xticks=[], yticks=[])   ax.imshow(torch.squeeze(image, dim = 0).numpy(),              cmap=plt.cm.binary)   ax.set_title(str(label))
Remember that in the previous post we explained that to facilitate the entry of data into our neural network we make a transformation of the input (image) from 2 dimensions (2D) to a vector of 1 dimension (1D). That is, the matrix of 28×28 numbers can be represented by a vector (array) of 784 numbers (concatenating row by row).
We will apply this transformation when we ingest the data to the neural network using this type of transformation (e.g. applied to the first image):
image, _ = xy_trainPT[0] print(image.size())image_flatten = image.view(image.shape[0], -1)print (image_flatten.size())torch.Size([1, 28, 28]) torch.Size([1, 784])
A Tensor is a multi-dimensional array, fundamental building block of PyTorch, equivalent to NumPy, that stores a collection of numbers:
a = torch.randn(2, 3)print(a)tensor([[ 1.1049, 0.2676, -0.4528],        [ 0.0105, -0.5095, 0.7777]])
And we can know its dimensions and size with:
print(a.size())print(a.dim())torch.Size([2, 3])2
Apart from dimensions, a tensor is characterized by the type of its elements. For this we have the dtype argument that is deliberately similar to the standard NumPy argument type of the same name:
matrix=torch.zeros([2, 4], dtype=torch.int32)print(matrix)tensor([[0, 0, 0, 0],        [0, 0, 0, 0]], dtype=torch.int32)print(matrix.dtype)torch.int32
Torch defines nine types of CPU tensor and nine types of GPU tensor:
As you can see there are a specific types for GPU tensors. PyTorch transparently supports CUDA GPUs, which means that all operations have two versions — CPU and GPU — that are automatically selected. The decision is made based on the type of tensors that you are operating on.
There are different ways to create a tensor in PyTorch: calling a constructor of the required type, converting a NumPy array (or a Python list) into a tensor or asking PyTorch to create a tensor with specific data. For example we can use torch.zeros() function to create a tensor filled with zero values:
b = torch.zeros(2, 3)print(b)tensor([[0., 0., 0.],        [0., 0., 0.]])c = torch.ones(2, 3)print(c)tensor([[1., 1., 1.],        [1., 1., 1.]])
An element of a tensor can be accessed using its index (which starts at 0):
c[0,0]=222 print(c)tensor([[222.,   1.,   1.],                 [  1.,   1.,   1.]]) 
Furthermore, just like in the usual data structures in Python, we can use the range notation in indexing to select and manipulate portions of the tensor with the help of the “ : ” character. Indexes start at 0 and we can use negative values for the indexes, where -1 is the last element and so on. Let’s look at a the following code for examples:
x = torch.Tensor([[1,2,3,4], [5,6,7,8], [9,10,11,12]]) print (x)tensor([[ 1., 2., 3., 4.],        [ 5., 6., 7., 8.],        [ 9., 10., 11., 12.]])print (“x column 1: “, x[:, 1])print (“x row 0: “, x[0, :])print (“x rows 0,1 & cols 1,2: \n”, x[0:2, 1:3])x column 1: tensor([ 2., 6., 10.])x row 0: tensor([1., 2., 3., 4.])x rows 0,1 & cols 1,2:tensor([[2., 3.],        [6., 7.]])
PyTorch tensors can be converted to NumPy matrices and vice versa very efficiently. By doing so, we can take advantage of the tremendous amount of functionality in Python ecosystem that has evolved around the NumPy array type. Let’s see with a simple code how it works:
x = np.array([[1,2], [3,4], [5,6]])print (x)[[1 2] [3 4] [5 6]]
This array x can be easily converted to a tensor as follows:
y=torch.from_numpy(x)print(y)tensor([[1, 2],       [3, 4],       [5, 6]])
We can see that the second print indicates that it is a tensor. Conversely, if we want to transform a tensor into a NumPy array, we can do it as follows:
z = y.numpy()print (z)[[1. 2.] [3. 4.] [5. 6.]]
We will use reshape() function, that returns a tensor with the same data and number of elements as input , but with the specified shape. When possible, the returned tensor will be a view of input. Otherwise, it will be a copy (in memory):
one_d = torch.arange(0,16)print (one_d)two_d= one_d.reshape(4,4)print (two_d)print(two_d.size())tensor([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15])tensor([[ 0, 1, 2, 3],        [ 4, 5, 6, 7],        [ 8, 9, 10, 11],        [12, 13, 14, 15]])torch.Size([4, 4])
In the torch.nn package, you can find many predefined classes providing the basic functionality blocks required for programming neural networks. To define the model presented in the previous post, it can be done with the Sequentialclass from this package:
modelPT= torch.nn.Sequential(                     torch.nn.Linear(784,10),         torch.nn.Sigmoid(),          torch.nn.Linear(10,10),          torch.nn.LogSoftmax(dim=1)          )
The code is defining a neural network composed of a two dense layers (linear layer) of 10 neurons each, one with a Sigmoid activation function and the other with the Softmax activation function. As we advance the series we will introduce other activation functions, as ReLU, that we will use in a next post in this series.
I would like to highlight that the previous code adds a small transformation to the neural network presented in the previous post: additionally, it is applying a logarithm operation to each of the outputs of the last layer. Specifically, the LogSoftmax function which can be seen as:
where Softmax is calculated as defined in the previous post. There are a number of practical and theoretical advantages of LogSoftmax over Softmax that motivate its use in building neural networks that we will discuss in a later section.
In summary, the network that we have defined can be visually represented as shown in the following figure:
The first layer of the neural network receives a tensor of 784 characteristics that represent the pixels that are passed to each and every one of the 10 neurons in the first layer. Once these 10 neurons have processed this information, each of them passes the information to all the neurons in the next layer, that is, all 10 neurons in the first layer are connected to all 10 neurons in the second layer.
The second layer is a layer with a softmax activation function of 10 neurons, which means it will return a tensor of 10 probability values, representing the 10 possible digits. In general, the output layer of a classification network will have as many neurons as classes, except in a binary classification, where it is only needed one neuron. Let’s remember that we have used a LogSoftmax layer, instead of Softmax, so that each value that returns will be the logarithm of the probability that the image of the current digit belongs to each of the classes.
We can use this simple example to analyze the parameters that make up a neural network. For example, in the first layer, for each of the ten neurons, 784 parameters are required for the weights, and therefore 10 × 784 parameters to store the weights of the 10 neurons. Furthermore, 10 additional parameters are required for the 10 biases corresponding to each of the neurons. Therefore, for the first layer 7,850 parameters are required.
In the second layer, being a softmax function, it is required to connect all its 10 neurons with the 10 neurons of the previous layer and, therefore, 10 × 10 parameters are required for the weights; in addition to the 10 biases corresponding to each node. This gives us a total of 110 required parameters in the second layer.
In summary, for our extremely simple neural network we see that 7,960 parameters are required, 7,850 parameters for the first layer and 110 for the second.
By subclassing the nn.Module, the base class for all neural network modules, we can create our own building blocks, which can be stacked together and reused later, which is what is usually done. But given the initiation nature of this post, we can move forward with this basic way of defining our neural networks. The reader can check the oficial documentation for more details on this topic.
As we shown in the previous post these models are trained by solving iteratively an unconstrained optimization problem. In each iteration, a random batch of the training data is fed into the model to compute the Loss function value. Then, the gradient of the Loss function with respect to the weights of the network is computed (backpropagation) and an update of the weights in the negative direction of the gradient is done. These networks are trained until they converge into a Loss function minimum.
Loss functions, around 20 different in PyTorch, reside in the nn package and are implemented as an nn.Module subclass. Some of the common standard loss functions that we will use in this series are:
nn.MSELoss: The mean square error between arguments, which is the standard loss for regression problems.
nn.NLLLoss: It computes the “maximum likelihood” criteria and generaly are used in multi-class classification problems (as our MNIST example).
nn.CrossEntropyLoss: computes the same as nn.NLLLoss, however it expects raw scores for each class and applies LogSoftmax internally (while nn.NLLLoss expects to have log probabilities as the input).
Since we are dealing with a multi-class classification problem, we chose cross-entropy as our Loss function. In this example, we use negative log-likelihood nn.NLLLoss () that combines with the softmax nn.LogSoftmax () function that we have already introduced. As we said we don’t apply Softmax to increase the numerical stability of the training process presenting an alternative way to calculate the Softmax first, which uses exponentiation, and then calculating cross-entropy loss, which uses a logarithm of probabilities. If the reader is interested in more details about this, I recommend having a look at this article.
The downside of defining a neural network with LogSoftmax is that we need to remember to apply softmax every time we need to get probabilities from our neural network output.
In general, the reader will see the Loss function assigned to criterion in PyTorch codes:
criterion = torch.nn.NLLLoss()
Therefore the way to calculate the error made will be as:
loss = criterion(logps, labels)
where we indicate in the arguments the output of our neural network and the correct label.
Remember that the optimizer takes the gradients of model parameters and change these parameters in order to decrease the loss value. We make use of torch.optim which is a module provided by PyTorch to optimize the model, perform gradient descent and update the weights by backpropagation. This package allows us to choose between several algorithms (AdaGrad, RMSProp, Adam, etc.) that are different variants of the gradient descent algorithm, the generic optimization algorithm capable of finding optimal solutions to a wide range of problems. At the moment in this example we will use the basic Stochastic Gradient Descent (SGD):
optimizer = torch.optim.SGD(modelPT.parameters(), lr=0.01)
The arguments are the parameters that the optimizer must adjust and a learning rate that indicates how these adjustments should be. Remember that the optimizer iteratively adjusts the parameters (weights and biases) of the model in the right direction (adding a “little” or subtracting a “little” to its value, where this “little” is defined by the learning rate) so that lead to a decrease in error. In general, the procedure is repeated until the error falls below an acceptable level.
Derivatives are used to calculate the correct direction, and specifically the gradient of the error with respect to the parameters. The autograd package in PyTorch provides exactly this functionality by automatic differentiation to automate the computation of backward passes in neural networks.
Once our model has been defined and the learning method configured, it is ready to be trained. Therefore, we only have to define the training loop that will iterate over all the data so that the optimizer iteratively adjusts the weights. Let’s discuss the common blueprint of a training loop with these lines of code:
1: for e in range(EPOCHS):      running_loss = 02:    for images, labels in xy_trainPT_loader:3:        images = images.view(images.shape[0], -1)4:        output = modelPT(images)5:        loss = criterion(output, labels)6:        loss.backward()7:        optimizer.step()8:        optimizer.zero_grad()          running_loss += loss.item()      print(“Epoch {} — Training loss: {}”.format(e,              running_loss/len(xy_trainPT_loader)))
Line 1: Usually, the training loop iterates over our data over and over again. Remember that one iteration over a full set of examples is called an epoch. The EPOCHS variable indicates the number of iterations over the full set of examples.
Line 2: We already presented that data is usually too large to fit into CPU or GPU memory at once, so it is split into batches of equal size. Every batch includes data samples and target labels, and both of them have to be tensors.
Line 3: To facilitate the entry of data into our neural network we must make a transformation of the input (image) from 2 dimensions (2D) to a vector of 1 dimension (1D).
Line 4: We pass each batch of image tensors into the model which will return a tensor having predictions for that batch, the forward pass.
Line 5: Having got the predictions we pass them into the cross-entropy Loss function ( criterion ) along with their actual labels and calculate the Loss. Usually, Loss functions accept two arguments: output from the network (prediction) and desired output (ground-truth data, which is also called the label of the data sample).
Some PyTorch’s Loss functions take class labels as their targets (e.g. NLLloss), so if we use them (as in our case), we don’t need to convert targets into one-hot vectors as we presented in the previous post (for ease the explanation).
Line 6: We do a backward pass using the Loss value in order to compute gradient of the Loss with respect to model parameters. After loss.backward() call is finished, we have the gradients accumulated.
Line 7: Now it’s time for the optimizer to modify the model parameters with the method step() that takes all gradients from the parameters and applies them.
Line 8: The last, but not least, piece of the training loop is our responsibility to zero gradients of parameters. Calling zero_grad() on our network clear the gradients of all optimized variables.
In order to check how the training process evolve we have added a couple of lines of code in this training loop. First, we get the training loss for the entire epoch by adding all the losses for each batch iteration:
running_loss += loss.item()
And second averaging it over by the iteration count and print it:
print(“Epoch {} — Training loss: {}”.format(e,              running_loss/len(xy_trainPT_loader)))
Looking at this output we see how the training loop adjusts the weights of the network so that in each iteration the loss function produces a smaller loss.
Epoch 0 — Training loss: 2.1822925415882932 Epoch 1 — Training loss: 1.8671700155048736Epoch 2 — Training loss: 1.5379922698809902Epoch 3 — Training loss: 1.287035460029838. . .Epoch 15 — Training loss: 0.5162741374264139Epoch 16 — Training loss: 0.49991638108547815Epoch 17 — Training loss: 0.48541215611800453Epoch 18 — Training loss: 0.4724724407929347Epoch 19 — Training loss: 0.4608637086554631
In PyTorch, there is no a “prefab” data model tuning function as fit() in Keras or Scikit-learn, so the training loop must be specified by the programmer.
Now, that we have finished the training of our model, we will probably want to test how well our model was generalized by applying it on a test dataset.
In PyTorch it is required again that the programmer specifies the evaluation loop:
xy_testPT = torchvision.datasets.MNIST(root='./data',             train=False, download=True,             transform=torchvision.transforms.            Compose([torchvision.transforms.ToTensor()]))xy_test_loaderPT = torch.utils.data.DataLoader(xy_testPT)correct_count, all_count = 0, 0for images,labels in xy_test_loaderPT:  for i in range(len(labels)):    img = images[i].view(1, 784)    logps = modelPT(img)    ps = torch.exp(logps)    probab = list(ps.detach().numpy()[0])    pred_label = probab.index(max(probab))    true_label = labels.numpy()[i]    if(true_label == pred_label):        correct_count += 1    all_count += 1print("\nAccuracy of the model =", (correct_count/all_count))         Accuracy of the model = 0.8657
The reader can see the similarity of the instructions in this loop with those of the previous training loop. But in this case, instead of keeping the Loss calculation, the Accuracy is calculated, that is, the percentage of hits with data that the model has never seen before.
And that’s all! You have already programmed an entire neural network. Congrats!
See you in the next post!
by UPC Barcelona Tech and Barcelona Supercomputing Center
A relaxed introductory series that gradually and with a practical approach introduces the reader to this exciting technology that is the real enabler of the latest disruptive advances in the field of Artificial Intelligence.
I started to write this series in May, during the period of lockdown in Barcelona. Honestly, writing these posts in my spare time helped me to #StayAtHome because of the lockdown. Thank you for reading this publication in those days; it justifies the effort I made.
Disclaimers — These posts were written during this period of lockdown in Barcelona as a personal distraction and dissemination of scientific knowledge, in case it could be of help to someone, but without the purpose of being an academic reference document in the DRL area. If the reader needs a more rigorous document, the last post in the series offers an extensive list of academic resources and books that the reader can consult. The author is aware that this series of posts may contain some errors and suffers from a revision of the English text to improve it if the purpose were an academic document. But although the author would like to improve the content in quantity and quality, his professional commitments do not leave him free time to do so. However, the author agrees to refine all those errors that readers can report as soon as he can.