Image classification on CIFAR 10: A Complete Guide
By: Adam Dhalla
Tags: Cifar 10, Pytorch, Image Classification, Machine Learning, Neural Networks

Now that we can understand a neural network is just replaying logistic regression (and also executing some nonlinear function each layer), it becomes a lot easier to code.

PyTorch for Neural Networks

Again, just like in the logistic regression code, we modify the nn.Module class to accommodate our model.

In PyTorch, nn.Module holds the nn.Linear command, which makes a one layer neural network (= logistic regression) model by passing in the features and output size (in log. regression, output size == number of classes). nn.Linear(features, outputs) would create a weights matrix of size (features, outputs) and a bias of size (outputs).

Passing something through nn.Linear would be equivalent to multiplying the input by the [randomly initialized] weights and adding the bias.

>> model = nn.linear(features, numClasses) # data must have input size of 'features' - be size (x, features) -

x is batch size. >> model(data)

nn.Linear has one method, forward(), which fits (matrix multiplies) data to the model. Since there is only one method, we don’t need to specify nn.Linear.forward() — nn.Linear() will suffice.

Now, we can edit the definition of forward() to run linear regression twice, and to automatically create and randomly initialize our weight and bias matrices for both W1, b1 and W2, b2.

>> import torch

>> import torch.nn.functional as F

>> import torch.nn as nn

We’ll be using ReLU as our activation function. I’d strongly recommend you view the diagrams from earlier and follow along the matrix multiplications as you read through the next few paragraphs.

>> class CIFAR10model(nn.Module): """ feedforward neural network with single hidden layer""" >> def __init__(self, inSize, hiddenSize, outSize): super().__init__() # creates the weights and biases for the first layer. self.linear1 = nn.Linear(inSize, hiddenSize) # creates the weights and biases for the output layer. self.linear2 = nn.Linear(hiddenSize, outSize) >> def forward(self, xb): # flatten the image tensors (xb.size[0] gets the batch size,

-1 adapts to whatever fits all units) xb = xb.reshape(xb.size(0), -1) # get intermediate outputs from hidden layer out = self.linear1(xb) # apply activation function element wise on the outputs from

self.linear out = F.relu(out) # get predictions using the output layer. out = self.linear2(out)



return out

Let’s step through the first part, the instantiation, slowly.

>> class CIFAR10model(nn.Module): """ feedforward neural network with single hidden layer""" >> def __init__(self, inSize, hiddenSize, outSize):

super().__init__()

self.linear1 = nn.Linear(inSize, hiddenSize)

self.linear2 = nn.Linear(hiddenSize, outSize)

Our new class CIFAR10model builds off of the parent module ( nn.module )

First, we use __init__ to instantiate an instance of our new class CIFAR10model. A successful instantiation of CIFAR10 model requires the input size ( inSize ), hidden layer size ( hiddenSize ) and the output size ( outSize ). self is required, and represents the instance of the object itself.

super().__init__ helps inherit the methods from the parent class nn.module.

Then we define self.linear1 and self.linear2 , which are both defined using the nn.Linear method that we gained through the parent class.

Recall that nn.Linear required the inputSize and outputSize to create a logistic regression model (an appropriately sized weight and bias matrix). Just like the mathematical intuition, we can practically run logistic regression twice.

self.linear1 creates W[1] (size: 3072, 32) and b[1] (size: 32, using two inputs — features (3072) and hidden layer (32). Remember that our hidden layer, if viewing this just as a logistic regression, is our output.

creates W[1] (size: 3072, 32) and b[1] (size: 32, using two inputs — features (3072) and hidden layer (32). Remember that our hidden layer, if viewing this just as a logistic regression, is our output. self.linear2 creates W[2] (size: 32, 10) and b[2] (size: 10) using the size of the hidden layer as inputs (32) and the number of classes as outputs (10).

Now, let’s step through the second part, redefining the forward(xb) method, or how your model fits data to the weights and biases created by self.linear1 and self.linear2.

...

>> def forward(self, xb): xb = xb.reshape(xb.size(0), -1)

out = self.linear1(xb)

out = F.relu(out)

out = self.linear2(out) >> return out

forward only requires the batch of training examples (size: batchSize, features) to be passed.

First, xb.reshape (interchangeable with xb.view ), similarly to logistic regression, reshapes the input data to fit the model. Our untouched xb’s will be of size (100, 3, 32, 32), which can’t be multiplied by any of our weights.

What reshape does here is forces the amount of rows in the new xb to = the batchSize (taking the length of our batches, xb.size(0)).

Then, we use -1 to adapt to this — basically, the -1 solves for what amount of columns would hold the same amount of items as our previous matrix of size (100, 3, 32, 32). Since we must have batchSize (100) rows, how many columns would be needed for the matrix to fit (3 x 32 x 32 x100 ) items? This is equivalent to solving for x in 100x = 307200.

out is our running variable of outputs. Our steps are:

out = self.linear1(xb) is to get the values of the 32 intermediate nodes in the neural network, by multiplying our 3072 inputs by W[1] and adding bias b[1].

is to get the values of the 32 intermediate nodes in the neural network, by multiplying our 3072 inputs by W[1] and adding bias b[1]. out = F.relu(out) runs the activation function ReLU element-wise on all values in the intermediate layer — if we’re dealing with batchSize 100, this would be on a matrix of size (100, 32).

What we’re doing with the past two steps.

out = self.linear2(out) gets the values of the 10 output nodes in the neural network, by multiplying our intermediate values by W[2] and adding bias b[2]. These will be a matrix of logits that we will have to use softmax on, where the rows are the training example (100) and the columns contain the logit probabilities for each class (10).

This is what we did in the previous step. Notice I didn’t include an activation function — that’s because we haven’t calculated it yet.

We don’t need to run our activation F.relu since we will be using softmax to convert our logits into concrete probabilities. In binary classification (where the output is just 0 or 1) we would use the sigmoid instead.

since we will be using softmax to convert our logits into concrete probabilities. In binary classification (where the output is just 0 or 1) we would use the sigmoid instead. we then return out, which is a[2], the matrix product of our weights and biases in the second layer. a[2] is a matrix of logits where each row is a training example, and each column is a prediction for the corresponding class.

We can now create an instance of the training model we defined in the previous few blocks of code. We just need to pass three arguments: the features, or inputs ( inSize ), the hidden layer nodes ( hiddenSize ) and the amount of classes, or output size ( outSize )

>> inSize = 3072

>> hiddenSize = 32

>> outSize = 10 >> model = CIFAR10model(inSize, hiddenSize, outSize)

We can now check the size of the weight and bias matrices linear1() and linear2() inside our instance of CIFAR10model, model.

>> for t in model.parameters():

print(t.shape) torch.Size([32, 3072]) # W[1]

torch.Size([32]) # b[1]

torch.Size([10, 32]) # W[2]

torch.Size([10]) # b[2]

These match the diagrams exactly, except for the bias sizes. That’s because, as I explained briefly beforehand, the biases aren’t actually matrices, but vectors (not 100, 32 but 1, 32). They were drawn like that to mostly avoid the additional confusion of Python’s automatic ability to broadcast vectors onto matrices while doing matrix operations.

Let’s do a little check-up on our model so far.

>> for images, labels in trainDL:

print('images.shape: ' + str(images.shape))

outputs = model(images)

loss = F.cross_entropy(outputs, labels)

print('loss: ' + str(loss))

break images.shape: torch.Size([100, 3, 32, 32])

loss: tensor(2.3392, grad_fn=<NllLossBackward>)

Let’s go over the first batch in our training set (trainDL) — we end the loop with a break statement so that it only iterates once.

We can print the shape of our batch, which itself is a four-dimensional matrix of size ([100, 3, 32, 32]) .

We can then fit our model on the 100 images in the first batch, and get our outputs (the 100, 10 logits matrix). We can then calculate our loss using cross entropy.

Printing our loss, we see that it’s 2.3392. This number is pretty useless on it’s own, but speaking that we have not trained our model yet, this is about the highest the loss will be — when we train our model correctly, we should see it decrease from here. We can use some accuracy metric if we’d like to see it in a more tangible form, like accuracy.

>> print('outputs.shape : ', outputs.shape)

>> print('sample outputs :

', F.softmax(outputs)[:2].data) outputs.shape : torch.Size([100, 10]) sample outputs :

tensor([[0.0930, 0.0877, 0.0894, 0.0983, 0.1081, 0.0955, 0.1214, 0.0973, 0.1011, 0.1083], [0.0934, 0.0964, 0.0836, 0.1114, 0.1005, 0.0903, 0.1178, 0.0988, 0.1045, 0.1032]])

We can then check the shape of our outputs to see if it matches our prediction. Indeed it does — with shape [100, 10].

Finally, we can check out some sample result vectors outputted by our simple neural network — let’s print out the results for the first two images in our batch.

They won’t actually be logits since the PyTorch F.cross_entropy automatically computes softmax on the logits vectors before calculating loss.

Now that we’re fitting to a more sophisticated model, our computer’s CPU won’t cut it — it just can’t calculate the matrix operations fast enough. That’s why we need to load to a Graphics Processing Unit (GPU) before training.

Loading to a Graphics Processing Unit (GPU)

What is a GPU?

In 1999, NVidia released the GeForce 256, the world’s first Graphics Processing Unit, for the main purpose of improving frame rates in 3D games for the PC.

GeForce 256.

GPUs aren’t better than the standard Central Processing Unit (CPU) in your computer — in fact, they’re probably a lot worse when it comes to general tasks.

GPUs were built to do one thing. Render graphics for 3D PC games fast, very fast. This meant very fast mathematics. GPUs do calculations in parallel (many calculations at the same time), as opposed to CPU’s, which do calculations serially (one calculation at a time).

GPUs for machine learning

The foundation of machine learning is very fast matrix operations.

It felt natural to draw this connection between GPUs and machine learning. The time came as part of the resurgence in interest in machine learning in the 2010’s, pioneered by the likes of Geoffrey Hinton, Yann LeCun, Yoshia Bengio, Illya Sutskever, Alex Krizhevsky, and Andrew Ng, upon countless others.

In 2009, Rajat Raina, Anand Madhavan, and Andrew Ng from Stanford published Large-scale Deep Unsupervised Learning using Graphics Processors, possibly the first official proposal for using GPUs for the purpose of machine learning.

Many soon followed, climaxing in 2012 with Geoffrey Hinton, Ilya Sutskever, and Alex Krizhevsky’s ImageNet performance, widely thought of as a critical turning point for the feasibility of deep learning. Partially due to the novel use of GPUs, the team blew away any other’s accuracies on the test set.

Loading to a GPU in PyTorch

Utilizing a GPU to speed up your own algorithms, only a decade after their debut into the machine learning community, is shockingly easy.

Google Colab and Kaggle (funnily enough, both owned by Google) both supply their users with free, remotely connected GPUs (and TPUs) to utilize.

in Colab, go to Runtime > Change Runtime Type > GPU

go to Runtime > Change Runtime Type > GPU in Kaggle, go to the three ‘more’ dots in the top righthand corner of the interface, then go Accelerator > GPU.

First, let’s define a helper function that tells us whether we should be using a GPU or a CPU, based on the availability.

>> def getDefaultDevice():

""" Pick GPU if available, else, CPU"""

if torch.cuda.is_available() == True:

return torch.device('cuda')

else:

return torch.device('cpu')

Using torch.cuda.is_avaliable(), we can get a boolean True or False depending on if a GPU is available. CUDA (Compute Unified Device Architecture) is simply the NVIDIA platform for GPUs — CUDA allows operation on both the GPU and the CPU.

If you’re working in Colab/Kaggle and haven’t enabled a GPU, you should get the following:

>> device = getDefaultDevice()

>> device

device(type='cpu')

Now, let’s create a function that moves are data (here, our training set DataLoader) to the GPU if it is present.

>> def toDevice(data, device):

""" move tensors to a chosen device """ if isinstance(data, (list, tuple)):

return [toDevice(x, device) for x in data]



return data.to(device, non_blocking=True)

the if isinstance(data, (list, tuple) line means that if our data passed into the function is in the form (thus, is an instance) of a list or a tuple, loop through the list/tuple and transfer each piece of data into the GPU. It actually refers to itself return [toDevice(x, device) for x in data] , which is fine — all the contents of the list will instead pass through the final return data.to(device, non_blocking=True) statement.

We can then test out this function by transferring the first batch of our images to the GPU and checking if they have moved correctly.

>> for images, labels in trainDL:

print(images.device)

print("transfered to GPU")

images = toDevice(images, device)

print(images.device)

break cpu

transfered to GPU

cuda:0

Just moving our images to the GPU using the toDevice() function we defined, we can see we successfully transferred them.

Finally, let’s create a loader that loads data to the GPU right before they are processed [trained on]. We don’t want to throw all the data to the GPU at the beginning of our program — we only want as limited amount of data on the GPU as possible to speed up runtime.

Let’s create a new class that does just that.

>> class DeviceDataLoader():

""" Wrap a dataloader to move data to a device"""

def __init__(self, dl, device):



# add attributes. self.dl = dl

self.device = device def __iter__(self):

"""Yield a batch of data after moving it to device""" # loop through dataloader attached to self. for b in self.dl:

yield toDevice(b, self.device) def __len__(self):

""" number of batches """ return len(self.dl)

This is a bit complicated. Let’s break it down into smaller parts. First, the __init__ statement.

>> class DeviceDataLoader():

""" Wrap a dataloader to move data to a device"""

def __init__(self, dl, device):



# add attributes. self.dl = dl

self.device = device

...

We don’t actually need to build off of a parent class, so we keep the class DeviceDataLoader(): argument-free. We then instantiate an object of class DeviceDataLoader() , and define it to require two arguments: dl , some DataLoader, and device , the device you would like to load it to.

We then define two attributes of our DeviceDataLoader() object, which are directly obtained from our arguments.

Let us now define our first function for our new type of object, __iter__.

...

def __iter__(self):

"""Yield a batch of data after moving it to device"""



for b in self.dl:

yield toDevice(b, self.device)

...

Since we’re starting from a blank slate (no parent class) we have to define a way to iterate through our DeviceDataLoader() object.

We do this by defining __iter__ , which only requires itself (self) .

We loop through the batches b in the DataLoader dl attribute of our DeviceDataLoader() object — which is just our old training DataLoader.

For each batch, we yield it to the function toDevice(data, device) , which transfers it to the specified device ( self.device ).

Yielding is practically iterative returning — each time this for loop is run, the next item in the yield is returned. In this example,

The first time we run the for loop we put the first item in self.dl to self.device.

to The second time we run the for loop we put the second item in self.dl to self.device

to And so on…

Basically, __iter__ lets you iterate through the DeviceDataLoader() object to transfer the batches to the GPU one by one.

And finally, the final, and simplest function we must define in our new class, len().

...

def __len__(self):

""" number of batches """

return len(self.dl)

This function returns the length of the DataLoader attribute ( dl ) of the object. Since the DataLoader is measured in batches, this returns the number of batches in our data.

We don’t have to take things off the GPU since there is an automatic process in which data that is not being operated on is removed, although there are methods to manually remove data that’s being pesky.

We can now wrap our DataLoaders within our new class, and utilize our new methods that we defined.

device = getDefaultDevice():

>> trainDL = DeviceDataLoader(trainDL, device)

>> validDL = DeviceDataLoader(valDL, device)

Now that we have a way to efficiently compute our calculations, we can begin training our neural network.

training the model

The code implementation of training is nearly identical to that of logistic regression, although the mathematics is a little different. I’ll explain the mathematics of backpropagation in another article I’ll link to at the bottom, but I’ll just give the intuitive explanation and code here.

We still use softmax + cross entropy to calculate loss in a completely identical manner (since we are still dealing with 10 classes) so you can read on that in the previous article.

The main difference is we have essentially a chain of two logistic regression algorithms before we convert our logits into probabilities and our probabilities into a total cost.