How to Build an Image Classification app using Logistic Regression with a Neural Network mindset
By: Jalal Mansoori
Tags: How To, Image Classification, Towards Data Science, Python, Machine Learning

Now let’s start the coding

Photo by Free-Photos On Pixabay

Recommended Development Environment

Jupyter Notebook

Note: You can find all the necessary files and code in my Github Repository

1. Packages

First, let’s import all the packages that will be very helpful in implementing the building blocks of our learning algorithm.

These packages are like different ingredients that you need during the process of making some kind of food!

numpy is the fundamental package for scientific computing with python and we will gonna use some efficient built-in functions like np.log, np.exp, …

is the fundamental package for scientific computing with python and we will gonna use some efficient built-in functions like np.log, np.exp, … h5py is a common package to interact with a dataset that is stored on an H5 file. The Dataset files are uploaded on Github.

is a common package to interact with a dataset that is stored on an H5 file. The Dataset files are uploaded on Github. matplotlib is a famous library to plot graphs in python.

is a famous library to plot graphs in python. PIL and scipy are used here to test your model with your own picture at the end.

If you are using Jupyter Notebook then import the above packages, if you are facing any issue then make sure that the packages are installed properly.

Now press Shift + Enter Key to Run the cell in Jupyter notebook.

2. Loading and Overview of the dataset

Let’s get more familiar with the dataset!

In the following code, we have loaded our dataset that consists of cat and non-cat images.

Training set

Our Cat classifier model will use this training set to learn the difference between Cat vs Non-Cat features.

train_set_x_orig: This is a matrix storing feature values of a cat and non-cat images. These values are between 0–255 representing different shades of Red, Blue, Green color.

train_set_y: This is a vector that is storing the corresponding output value of image features in train_set_x_orig.

1 means it’s a Cat image.

0 means it’s a Non-Cat image.

Testing set

After our model is trained on the training set, then we will check our model performance using the test set, this is the data our model hasn’t seen!

Overview of the dataset output:

Number of training examples: m_train = 209

Number of testing examples: m_test = 50

Height/Width of each image: num_px = 64

Each image is of size: (64, 64, 3)

train_set_x shape: (209, 64, 64, 3)

train_set_y shape: (1, 209)

test_set_x shape: (50, 64, 64, 3)

test_set_y shape: (1, 50)

So we have 209 training examples and 50 testing examples of Cat vs Non-Cat images. Each color image size is 64 X 64.

Visualize an Example of Image in the training set

Each line of train_set_x_orig and test_set_x_orig is an array representing an image. Feel free to change the index value and re-run to see other images.

Example of Cat and Non-Cat Picture from the training set

Reshaping the training and test set

For convenience, now we need to reshape images of shape (num_px, num_px, 3) in a numpy-array of shape (num_px*num_px*3, 1). After this, our training (and test) dataset is a numpy-array where each column represents a flattened image.

Output:

train_set_x_flatten shape: (12288, 209)

train_set_y shape: (1, 209)

test_set_x_flatten shape: (12288, 50)

test_set_y shape: (1, 50)

Now our training and testing set is in the right dimension for computation purposes. Each column in train_set_x_flatten and test_set_x_flatten is representing features of Images.

Visualization of data stored in the training set after reshaping. (Image by Author)

For example:

train_set_x_flatten[:, 209] will output the feature vector storing 12288 values of the last image in the training dataset.

Same for test_set_x_flatten!

3. Preprocessing Step

One common step in machine learning is to preprocess/standardize the dataset.

“ Having features on a similar scale can help the gradient descent converge more quickly towards the minima. ”

In simple terms, it means that it will be easier for the optimization algorithm to learn those efficient parameter w and b values when features in the dataset are on a similar scale. There are many different kinds of techniques useful for this purpose.

But for picture datasets, it is simpler and more convenient and works almost as well to just divide every row of the dataset by 255 (the maximum value of a pixel channel).

Before Preprocessing, feature values in the training and testing set are between 0 and 255.

After Preprocessing, now feature values in the training and testing set are between 0.0 and 1.0.

what we did up till now 👇

Figure out the dimensions and shapes of the problem (m_train, m_test, num_px, …)

Reshape the datasets such that each example is now a vector of size (num_px * num_px * 3, 1)

“Standardize” the data

Now it’s time to design a simple algorithm to distinguish cat images from non-cat images.

4. Building the parts of our algorithm

Image Design by Author, Photo of Cat by Ty_Swartz On Pixabay

First, we’ll implement the helper functions like sigmoid, initialize the parameters, … and then we’ll merge those functions into our main cat classifier model.

4.1 Sigmoid and Initializing the parameters function

As you’ve seen in the figure above, first we need to compute 𝑠𝑖𝑔𝑚𝑜𝑖𝑑(z) to make predictions.

Just to get started we are initializing the parameters w and b by zeros. You can also initialize these parameters with random values to check the difference between zeros vs random initialization!

4.3 Forward and Backward propagation

Now that we have our sigmoid and initialization functions ready, the next task is to compute forward and backward propagation steps for learning the parameters.

We’ll implement the following equations in a function named propagate() that computes the cost function and its gradient.

Equations to compute Gradients and Cost

The cost function will give us the value representing the amount of error between our model prediction and the actual value.

Our objective is to minimize the cost function and that’s why we need to compute gradients or change of cost with respect to parameters w and b.

4.4 Optimization Algorithm

It’s time to implement a function that will give us the efficient parameters for our model to make predictions with a minimum amount of error.

The goal here is to learn 𝑤 and 𝑏 by minimizing the cost function 𝐽. , using a gradient descent algorithm update rule is:

w= w−𝛼 𝑑w

b= b−𝛼 𝑑b

where 𝛼 is the learning rate, and dw, db is the gradients we computed in propagate() function.

Gradient descent gives us two hyperparameters to play with, learning rate and the number of iterations. We’ll be using these 2 hyperparameters to do some further analysis and improve the performance of our model.

4.5 Predict function

The previous optimization function will give us the w and b parameters. Then we can use those parameters to predict whether the image X contains a cat or not.

5. Merge all the functions into a model

Finally, we will now see how the overall model is structured by putting together all the building blocks (functions implemented in the previous parts) together, in the right order.

Let’s observe the performance of our cat_classifier model using random values in num_iterations and learning_rate:

d = model(train_set_x, train_set_y, test_set_x, test_set_y, num_iterations = 1000, learning_rate = 0.005, print_cost = True)

Output:

Training and Testing Accuracy of model

So our model was able to correctly classify approx 96% of images in the training dataset and 72% of images in the testing dataset. I hope you know that our model was trained on the training dataset and that’s why training set accuracy is much higher than testing accuracy.

But still, 72% of testing accuracy is not bad. Let’s plot the learning curve graph to get a better understanding of the training process!

Learning Curve Graph

Learning Curve Graph

You can see the cost decreasing. It shows that the parameters are being learned. Try to increase the number of iterations in the cell above and rerun the cells. You might see that the training set accuracy goes up, but the test set accuracy goes down. This is called overfitting.

Following are the results of our model by changing the num_iterations value:

# num_iterations = 2000 d = model(train_set_x, train_set_y, test_set_x, test_set_y, num_iterations = 2000, learning_rate = 0.005, print_cost = False) Output: train accuracy: 99.04306220095694 %

test accuracy: 70.0 % # num_iterations = 3000 d = model(train_set_x, train_set_y, test_set_x, test_set_y, num_iterations = 3000, learning_rate = 0.005, print_cost = False) Output: train accuracy: 99.52153110047847 %

test accuracy: 68.0 %

The above performance results show that when we are increasing the num_iterations value, our model training accuracy is increasing but testing accuracy is slightly decreasing!

Here, we see that the model is overfitting the training data, it means that model will only be able to predict correctly on the data it is being trained on. A technique known as regularization is used to reduce overfitting.

There are many possibilities to improve the model’s performance, So to get started in this tutorial I am going to use the cat classifier model with the training accuracy of approx 100% and testing accuracy of 70% which is not bad for this small dataset we are using!

Let’s Play with the learning rate parameter and do some Further analysis

In past, I also wrote an in-depth tutorial on the analysis of learning rate in the Gradient Descent algorithm 👇

Let’s analyze our cat classifier model further, and examine possible choices for the learning rate 𝛼.

Choice of the learning rate

For Gradient Descent to work, you must choose the learning rate wisely. The learning rate 𝛼 determines how rapidly we update the parameters.

If the learning rate is too large we may “overshoot” the optimal value.

Similarly, if it is too small we will need too many iterations to converge to the best values.

That’s why it is crucial to use a well-tuned learning rate.

Intuitive Visualization of Convergence and Divergence. Image by Adam Harley

Let’s compare the learning curve of our model with several choices of learning rates.