Image Classification On CIFAR 10: A Complete Guide
By: Adam Dhalla
Tags: Pytorch, Logistic Regression, AI, Python, Machine Learning

Image Classification On CIFAR 10: A Complete Guide

Picking up a worn mechanical pencil from the floor, a half-used Staedtler eraser, a metal 6-inch ruler, a notebook from Japan, and my computer, at 1am on a school night, October 3rd, 2020, I began to teach myself calculus.

Okay, not teaching myself. There’s no such thing as teaching yourself, these days, with the plethora of content out there. Opening the first lesson in Khan Academy’s Calculus AB playlist, I learned about limits.

Where I started. Thanks, Sal.

Then derivatives. Then calculating them. Then linear algebra. Then multivariable calculus.

All I can say how grateful and fortunate I was to have had the opportunity to spend an upwards of 5 hours a day learning mathematics. I’m incredibly privileged in that respect— to have the time to engage in the beautiful ivory-tower-esque discipline, and to be able to now view machine learning and mathematics in a depth I would have not been able to attain before. So thank you, so, very much, everyone who accommodated, tolerated and communicated with me during that period.

Enter early November. Feeling satisfied with the breadth of mathematics I covered, I began coding in PyTorch. My first project was a simple univariate linear regression problem. With that under my belt, I approached something tougher, with the help of a tutorial.

Image classification with logistic regression.

contents

It’s a rather large tutorial, so I’ll put links here if you’d like to skip:

a few disclaimers

I’ll be going through [most] of the code I used, but I’ll be putting more of an emphasis on the mathematics and inner working the code represents. Most of the code was snatched from Aakash’s fantastic tutorial on PyTorch from freeCodeCamp.org, but slightly changed to accommodate a slightly more complicated dataset (CIFAR instead of MNIST). I’ll include my full adapted code in the form of a Google Colab document at the end of this article.

This tutorial assumes a pretty solid understanding of many machine learning concepts, such as gradient descent, cost functions, linear regression, hyperparameters, and common notation, as well as mathematical concepts like matrix multiplication and partial derivatives.

notation

I generally use standard notation here.

m is the length of the training set.

features are the different inputs into an algorithm.

subscripts generally represent libraries, and if you see a subscript like 23 it means the 3rd feature on the 2nd class.

superscripts generally represent training examples

j as a variable is usually used as a subscript to signify looping.

I tend to say thetas interchangeably with “weights and biases”, since theta is often used instead of W or b in weights or biases.

Throughout the tutorial, I sometimes switch between using column (e.g 10, 1) and row vectors (e.g 1, 10), but they mean the same thing.

Whenever you see log(), think of it as ln().

# necessary libraries import torch

import pandas as pd

import matplotlib.pyplot as plt

import numpy as np

One frog, two frog: The CIFAR-10 Dataset

CIFAR-10, a subset of the now defunct 80 Million Tiny Images Dataset was one of the many possible datasets that was easily accessible within the framework of the PyTorch library (coming ‘plug-and-play’ in the .datasets module). I wish I could say there was a more romantic story behind it, but I chose CIFAR-10 rather arbitrarily.

Each row contains ten examples of the dataset’s ten possible classes. Airplanes, Cars, Birds, Cats, Deer, Dogs, Frogs, Horses, Boats and Trucks.

I decided to use logistic regression to create a model that would attempt to predict which of the 10 classes some image from this dataset would be predicted as.

Made up of 60,000 (50,000 for training/validation and 10,000 for test) 32x32 pixel images, CIFAR-10 has been an extraordinary resource for testing the accuracy of a variety of cutting-edge machine learning algorithms, as well as being a fantastic playground for machine learning experimentation.

Now, we load the dataset.

Loading, wrangling, & splitting — data preparation

It’s an incredible convenience to be able to download the data directly from the PyTorch library, instead of the CIFAR-10 site. De-pickling files was a bit intimidating.

>> import torchvision # import the training set (first 50,000 imgs) by setting train=True. >> trainset = torchvision.datasets.CIFAR10(root='./data', train=True,download=True) # import the test set (10,000 imgs) by setting train=False >> testset = torchvision.datasets.CIFAR10(root='./data', train=False, download=True) # define the ten labels manually. >> classes = ('plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck')

Not much to explain mathematically or theoretically here. By accessing torchvision.datasets, we can access our CIFAR dataset.

indexing into data

Let’s access the first image in our dataset.

>> trainset[0]

(<PIL.Image.Image image mode=RGB size=32x32 at 0x7FF6E08F5908>, 6)

Indexing into trainset with typical notation returns two values. First, an intimidating string of letters and numbers, and 6. The fact that we are returned two values can create the correct image of trainset being a 50000x2 matrix, each example being a pair of image and correct answer, x and y.

That intimidating first value is a PIL.image object — a special code for that image, encoded and saved using the Python Pillow library.

The comparatively simple second term is the ‘right answer’ for that image, the label. Glancing at our predefined label array, we can see that 6 represents an image of a frog. But how do we view our little frog friend?

using matplotlib to view images from dataset

We can use the standard matplotlib.pyplot library to plot our images as colored pixel values.

# you might have to use this magic function depending on your IDE

>> %matplotlib inline

The above code is a magic function. Depending on your IDE, this code will prevent graphs created by matplotlib to open in a different window (a popup) and will instead show them inline, underneath the code.

Now, we can plot our image.

>> example = 0 # index into the 50,000 x 2 trainset vector, and store the label in the variable 'label'. Store the PIL object in variable 'image'. >> image, label = trainset[example] # print the label for the image by indexing label into classes. >> print("this is an image of a " + classes[label] # use imshow to show the PIL object image >> plt.imshow(image)

The image shown by this will show example -th training example from the training set. Executing this code will show us our first image from the training set, a frog, paired with it’s label.

Our 32 x 32 pixel friend in all his glory. Kind of makes you grateful for high definition cameras, doesn’t it?

Transforming images into matrices of numbers (tensors)

There isn’t a way for an algorithm to truly see an image. That’s why we need to find a way to represent an image with numbers, and individually feed each one of those numbers in to our algorithm as a feature.

To understand how we do this, we must take a brief look at simple color theory. More specifically, the Red-Green-Blue (RGB) model.

What every pixel looks like up close.

Every pixel on your screen, and every pixel in an image, is represented by three bars of primary red, green and blue. Each of these bars can be dialed up or down in intensity from 0–256 (why 256? think binary). Combinations of these three colors, each at some intensity between 0 and 256, creates every color we know. Large blocks of color are made up of many pixels with the same RGB values. One pixel’s color can be described by just three numbers.

Sensibly, then, we can represent a whole picture by defining it using it’s R, G, and B values for each pixel.

We can quite easily expand this logic to this entire 32x32 image, and beyond, to larger images.

Formally, we can describe an image using 3 matrices that hold the individual R, G, and B color channels of each pixel. Each of the three matrices will be the same dimensions, the height x width of the image in pixels.

Thus, we can describe each image in our training set using a 3, 32, 32 matrix of pixel intensity values from 0–256. This gives us 3072 (3x32x32) features to plug into our dataset, if we unroll the image matrices into one long vector.

This is how an image sees. When an algorithm tries to predict the content of an image, it isn’t looking the way humans tend to recall and identify the things we see. When an algorithm predicts the content of an image, it’s really just comparing the number matrices with past number matrices and seeing which examples match up best.

Now we can confidently implement (the now trivial-seeming) code to do this image-to-matrix conversion.

>> import torchvision.transforms as transforms # directly access the training set again from .datasets, and set the transform keyword argument to transforms.ToTensor()) >> datasetT = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transforms.ToTensor())

At this stage, it’s most practical to think of tensors as any numerical object that can be operated on mathematically within a machine learning algorithm. Images are not tensors. Matrices, on the other hand, are.

Indexing into our new, transformed training set, we see that the first value on each row (training example) is no longer a PIL.image object, but a tensor, and more specifically, a matrix of dimensions 3, 32, 32. This makes the dimensions of our total training set [50000, 3, 32, 32].

>> example = imgTensor, label = datasetT[example]

>> print("size of image matrix: " + str(imgTensor.shape))

>> print("this is an image of a "+ classes[label] size of image matrix: torch.Size([3, 32, 32])

this is an image of a frog

We can still view the image using matplotlib.imshow even if it’s now in tensor format, requiring no change in syntax. As an added bonus, we can now index into custom values for the first dimension (x, 32, 32) to see the different color channels of the image individually (or :, all, to see the image normally)

# change the 0 (R) to 1 (G) or 2 (B) for different color channels. Additionally, recall that imgTensor represents just our first example. plt.imshow(imgTensor[0, 0:32, 0:32])

print("R channel of image 0 in dataset")

Note that Python defaults to showing graphs in a green colormap. Remember, when we use matplotlib to plot images, we’re treating the images as a heatmap of colored points that form an image. If you want actual red channel values, pass in the tensor to .imshow and use

>> plt.imshow(imgTensor[0, 0:32, 0:32], cmap='Reds') # R

>> plt.imshow(imgTensor[1, 0:32, 0:32], cmap='Greens') # G

>> plt.imshow(imgTensor[2, 0:32, 0:32], cmap='Blues') # B

The R, G, B channels represented separately. Think of each pixel having a value from 256 (deep color) to 0 (light color). White is represented as (R=0, G=0, B=0) and black as (R=256, G=256, B=256). Does this make sense?

training and validation sets

Splitting into training and validation let us plot things like learning curves, which help diagnose common issues in machine learning algorithms. via Machine Learning Mastery

I assume that if you’re reading this article, you have a solid understanding about the significance for splitting your initial X set into training and validation sets. In essence:

the training set is used to train the parameters of a given model, and usually takes up around 60% of a total dataset.

is used to train the parameters of a given model, and usually takes up around 60% of a total dataset. the cross-validation set is used to evaluate the model during training (e.g, to see if gradient descent is working properly), to diagnose over and underfitting, and to adjust hyperparameters like the learning rate. It usually takes up around 20% of the total dataset.

is used to evaluate the model during training (e.g, to see if gradient descent is working properly), to diagnose over and underfitting, and to adjust hyperparameters like the learning rate. It usually takes up around 20% of the total dataset. the test set is used to report the accuracy of the completed model. It usually takes up around 20% of the total dataset.

Since PyTorch already loads the test set independently, let’s split our training set into a 80/20 ratio between training and cross-validation (CV) respectively. There are functions to do this that are built-in to PyTorch, but let’s create a function to shuffle the indexes of the training set and to split the set into whatever ratio we want, from scratch.

#length of training examples matrix (how many training ex.)(m). >> m = 50000 # percentage of training set dedicated to CV. >> pCV = 0.2 # give the amount of cross-validation examples from m. >> mCV = int(m*pCV) # print the lengths >> print("amount of training examples: " + str(m - mCV))

>> print("amount of cross validation examples: " + str(mCV)) amount of training examples: 40000

amount of cross validation examples: 10000

Now, we can define a function that will shuffle the indices of our original training set and allow us to split it up into the amounts described above.

>> def splitIndices(m, pCV): >> """ randomly shuffle a training set's indices, then split

indices into training and cross validation sets. Pass in 'm'

length of training set, and 'pCV', the percentage of the

training set you would like to dedicate to cross validation.""" # determine size of CV set. >> mCV = int(m*pCV) # create random permutation of 0 to m-1 - randomly shuffle all

values from 0 to m. >> indices = np.random.permutation(m) # pick first mCV indices for training, and then validation. >> return indices[mCV:], indices[:mCV]

Basically what this code does is creates a random permutation of the numbers zero through 50000. By taking a random permutation on the number 50,000, all integers between 0 and 49,999 will be randomly shuffled into a list of 50,000 numbers. (e.g a random permutation of 3 might result in list [2, 0, 1, 3] )

Then, we take the first mCV (amount of cross validation examples) items from the list, resulting in a list mCV long containing random indexes of the original training set. We then take the leftover 80% for the randomly shuffled indexes of the new training set.

tl, dr; We randomly shuffled the indexes of the original training set, then took the first 10000 indexes as which examples from the original set will be going to the cross-validation set, and took the last 50000 as which examples will be going to our new training set.

batch loading for mini-batch gradient descent

Since we’re dealing with a large set of features (3072), as well as a large amount of training data (40000), that means regular gradient descent would be extremely costly (explained later in this article when we go into mini-batch gradient descent).

Thus, let us split up both our training and cross validation sets into batches of 100 training examples each.

# import SubsetRandomSampler (for batches) and DataLoader (general data load >> from torch.utils.data.sampler import SubsetRandomSampler

>> from torch.utils.data.dataloader import DataLoader

Now, use these to batch and load the data.

batchSize = 100 # ----------------------------TRAIN SET # tSampler creates a SubsetRandomSampler object that takes random samples of the numbers in trainIndices, or random indices. >> tSampler = SubsetRandomSampler(trainIndices) # training loader - creates batches and pairs with images. >> trainLoader = DataLoader(datasetT, batchSize, sampler=trainSampler) # ----------------------------VALIDATION SET # create SubsetRandomSampler object >> vSampler = SubsetRandomSampler(valIndices) # split into batches of 100 and pair with images. >> valLoader = DataLoader(datasetT, batchSize, sampler=valSampler)

First, set the batch size to 100.

Then, we use SubsetRandomSampler to create an iterable object of randomly shuffled values. This code has little more purpose than to change the object of trainIndices from a list to a SubsetRandomSampler object, and to randomly reshuffle the dataset each time before taking batches.

Then, we can use DataLoader to create the batches. Given the batchSize (100) the sampler (trainSampler), and the transformed image matrix (datasetT) we create a DataLoader object that can be thought of as a list of batches of size (100, 3, 32, 32), or 100 image matrices.

the multiclass logistic regression model

the hypothesis

Our multiclass logistic regression model, for the most part, is nearly identical to linear regression.

As opposed to sigmoid regression for binary classification (classes 0 and 1), we will use softmax regression. Think of softmax regression as identical to sigmoid but for multiclass classification. I’ll go into softmax later. Essentially, Sigmoid takes some single scalar real number and puts it in the range from 0 to 1. Softmax takes in a vector of n scalar real numbers and converts all of them to numbers between 0 and 1, and makes them add up to 1.

The calculations of one batch’s training examples. Matrix drawing sizes are not accurate, use the dimensions provided below them.

Following along with the matrix multiplication, we get a (100, 10) matrix with each row being a training example, and each column being some scalar value associated with the probability that training example is each label.

Where m is the batch size. Each column of this represents the probabilities of the examples belonging to that class.

Therefore, the predicted “probabilities” for each class for one training example is each row. I put probabilities in apostrophes since these aren’t classical 0–1 probabilities, but scalar numbers of any size (they are just the products of a linear expression). Softmax converts these to probabilities, which we’ll go over soon.

>> import torch.nn as nn # this will dictate the rows of the weights matrix >> inputSize = 3*32*32 # this will dictate the columns of the weights matrix >> numClasses = 10 # create our linear regression model (nn.Linear creates bias terms for us) >> model = nn.Linear(inputSize, numClasses)

nn.Linear creates the weights and biases, creating a matrix of shape (imputSize, numClasses). More than just that, nn.Linear creates a function model, which can later be passed a matrix of training examples in order to return predictions. Additionally, nn.Linear also randomly initializes the values of each weight and bias.

Essentially, nn.Linear creates the bias and weights matrix automatically given the amount of classes and amount of features. When we pass something into the model created by nn.Linear, we are just multiplying by the weights matrix and adding the bias.

>> print(model.weight)

Taking a sneak peak at our weights, we can see they are all miniscule values such as -1.0100e-02, all with requires_grad = True. requires_grad = True means that PyTorch allows these values to be differentiated, once we get to gradient descent (the reason why this is an option is that computing derivatives is costly, so PyTorch makes sure to only do this when it is absolutely necessary).

But, if we try to fit our training examples (trainLoader) to this model, we get an error. This is because trainLoader is a matrix of shape (100, 3, 32, 32), which cannot be multiplied by our weights matrix in nn.Linear which is of size (3072, 10).

Matrices can be multiplied if the first matrix’s columns is the same as the second matrix’s rows. So, we need to find a way to turn trainLoader (100, 3, 32, 32) into something compatible with (3072, 10).

The obvious choice would be to reshape trainLoader into size (100, 3072), but let’s instead edit the class nn.Linear itself to change the function to automatically reshape any trainLoader, regardless of size, into a compatible matrix. This can be done with Python classes.

>> class CIFAR10(nn.module):

def __init__(self):

super().__init__()

self.linear = nn.Linear(inputSize, numClasses)

def forward(self, xb):

xb = xb.reshape(-1, 3072)

out = self.linear(xb)

return out >> model = CIFAR10()

What this code does it creates the CIFAR10 class, which builds off the nn.Module class (and thus gains all of it’s methods and functions).

It creates itself using def __init__. Then it defines a new function, forward(self, xb) where xb is a batch of training examples (x-batch).

Forward automatically reshapes the training examples into shape (-1, 3072). Using -1 means that the new matrix will have however many rows it needs to have 3072 columns. Since our original xb (100, 3, 32, 32) would have a total of 307200 items (100 x 3 x 32 x 32), the -1 solves for what amount of rows would hold the same amount of items. (solving for x in 3072x = 307200).

The reason why we do this, and don’t hardcode (100, 3072), is that it adapts to different batch sizes.

Then we set the output of the function as our new xb through the same nn.linear function.

the .forward() function of a model is assumed when we do Model(xb) — there is no need to specify Model.forward(xb). That’s to say, by creating our own class CIFAR10, and redefining the definition of forward, after setting our model to an object of class CIFAR10, to CIFAR10(xb) will fit the model.

Since model = CIFAR10, we now just need to execute model(dataLoader), and not model.forward(dataLoader), to get the program to work, since forward now automatically reshapes the values.

>> print(model.linear.weight.shape)

>> print(model.linear.bias.shape)

We can still check the parameters of our model, even though we modified the overarching class. Since CIFAR10 is a subclass of nn.Module, it gained all of nn.Module’s methods, like model.linear.

Using these codes, we can still check the shapes of the weight and bias matrices.

Now we can finally get our initial predictions, by passing trainLoader into our modified model. Remember, ‘passing trainLoader in our model’ is equivalent to multiplying our training values by our paramaters and adding a bias, just done in a built in function.

>> for images, labels in trainLoader:

outputs = model(images)

break >> print('outputs.shape :', outputs.shape)

outputs.shape : torch.Size([100, 10])

A for loop over trainLoader iterates over the batches. Since we use break in the for loop, we only calculate the initial predictions for the first batch in trainLoader.

Checking the shape of outputs gives a matrix of (100, 10), which matches our initial calculations. There are 100 rows, for each example (image) in the batch, and each of the 10 columns contain the probability that the image (row) is in each class (horse, sheep, etc..). Only one problem — these probabilities are logits, arbitrary real numbers calculated by the model.

Why? When we multiply our inputs by some random weights and adding a bias, nothing’s restricting it to be a probability — they’ll all be random products, like 3.21324, and not probabilities that are between 0 and 1 and add up to 1.

But first, let’s recap. What have we done so far?

1. Import dataset and transform images to RGB matrices of 3x32x32. 2. Split dataset randomly into 50000 training examples and 10000 validation 3. Via SubsetRandomSampler, the dataset is shuffled once again into an iterative object. 4. Via DataLoader, the randomly shuffled indexes paired with corresponding image matrices, in preset batches decided by batchSize. 5. Modify the nn.linear function by editing the class nn.model, and making it so that nn.linear automatically flattens a matrix before entering it in nn.linear 6. Inputting the number of classes (X’s rows) and number of features (X’s columns) into nn.linear gives us a model, and empty set of weights to multiply something by. 7. By iterating through our trainLoader, which contains both the image matrix and the labels, we can input the image matrices through our model to get our outputs.

Let’s print out the predictions for the first two examples.

>> print('sample outputs :

', outputs[:2])

sample outputs : tensor([[-0.2363, 0.4695, -0.0422, -0.3637, -0.0558, -0.2168, -0.0343, -0.0593, -0.2294, 0.8198], [-0.2203, 0.1576, 0.4396, -0.2661, -0.2340, -0.2024, 0.2856, 0.1390, -0.6146, 0.6665]], grad_fn=<SliceBackward>)

Each of the two images have 10 probabilities, which stand for the probabilities each image has for belonging to any of the ten classes.

The numbers seem random (and indeed they are, they’re simply the linear product of the weights and the bias, without any nonlinear function like sigmoid attached).

This is where we need the softmax function.

The softmax function

The numbers that outputs contains are called logits. Here, we can understand logits to be numbers that represent probabilities, but not in the classical sense of adding up to 1. They represent some amount of likelihood, but not concretely. The larger a number is, the larger the probability. How do we convert these logits into standard probabilities (between 0 and 1) that also add into one?

Look familiar? The softmax can be thought of a vectorized implementation of sigmoid. Where Z is a row vector of logits, K is the amount of classes. J indexes through each class in the training sample, and i is the specific class probability we’re looking for. via DeepAi

I stole this fantastic explanation from Ritvik Kharkar’s youtube channel, ritvikmath, so check that video out.

How would we turn one training example’s logit row vector into an equally sized row vector of probabilities?

Whenever we divide a number by a sum of which the number is a part of, we get a result that is less than one. Softmax leverages this idea.

So, why do we need the exponentiation (e’s) in the numerator or denominator? Why not just add up each unit of the outputs vector divided by the sum, and rid of the exponentials?

Well, say if we did do this.