Image recognition with Machine Learning on Python, Convolutional Neural Network | by Jonathan Leban | Towards Data Science

This article follows the article I wrote on image processing. After making the data available for image recognition task, it is time to create an algorithm that will perform the task. Among many techniques used to recognize images as multilayer perceptron model, Convolution Neural Network (CNN) appears as a very efficient one. In this article, we will see how to build a CNN and how to apply it on a dataset of images.
When we start building an image recognition model for the first time, it’s usually a good idea to train and evaluate it on a relatively simple dataset.
One of the simplest tasks we can perform is handwritten digit recognition. Given an image of a handwritten digit (i.e., 0, 1, ..., 9), we want our model to be able to correctly classify its numeric value. Though this task seems relatively simple, it is actually used fairly often in real life, such as automatically extracting credit card numbers from a picture. The dataset we will use for digit recognition is the MNIST dataset, which is the dataset used for machine learning-based digit recognition.
The MNIST (Modified National Institute of Standards and Technology) database contains 60,000 training examples and 10,000 testing examples. The database contains grayscale handwritten digits that were resized to fit in a 20x20 pixel box, which was then centered in a 28x28 image (padded with whitespace). The MNIST database is accessible via Python.
In this article, I will show you how to code your Convolutional Neural Network using keras, TensorFlow’s high-level API. I am using tensorflow 2.0 in this article.
Since each grayscale image has dimensions 28x28, there are 784 pixels per image. Therefore, each input image corresponds to a tensor of 784 normalized floating point values between 0.0 and 1.0. The label for an image is a one-hot tensor with 10 classes (each class represents a digit). In terms of our code, we have img_rows = 28, img_cols = 28 and num_classes = 10. Thus the input has shape (number_examples, img_rows, img_cols) hence 60000x28x28. Another important element to set up is the random seed as we want to keep the start point when a computer generates a random number sequence.
We also import the MNIST dataset.
import tensorflow as tf # tensorflow 2.0from keras.datasets import mnistimport numpy as npseed=0np.random.seed(seed) # fix random seedtf.random.set_seed(seed)# input image dimensionsnum_classes = 10 # 10 digitsimg_rows, img_cols = 28, 28 # number of pixels # the data, shuffled and split between train and test sets(X_train, Y_train), (X_test, Y_test) = mnist.load_data()
As mentioned in the previous section, inputs have shape (number_examples, img_rows, img_cols). However, in order to use the data with our convolutional neural network, we need to get it into NHWC format.
NHWC format has a shape with four dimensions:
Number of image data samples (batch size)Height of each imageWidth of each imageChannels per image
Number of image data samples (batch size)
Height of each image
Width of each image
Channels per image
The height and width of each image from the dataset is img_rows and img_cols, while the number of channels is 1 (since the images are grayscale).
Also, each pixel contains a greyscale value quantified by an integer between 0 and 255. So, the database is normalized to have floating point values between 0.0 and 1.0. In this case, 0.0 corresponds to a grayscale pixel value of 255 (pure white), while 1.0 corresponds to a grayscale pixel value of 0 (pure black).
X_train = X_train.reshape(X_train.shape[0], img_rows, img_cols, 1) X_test = X_test.reshape(X_test.shape[0], img_rows, img_cols, 1)input_shape = (img_rows, img_cols, 1)# cast floats to single precisionX_train = X_train.astype('float32')X_test = X_test.astype('float32')# rescale data in interval [0,1]X_train /= 255X_test /= 255
We need to transform our classes into vectors. We do this by tapping the following line:
Y_train = keras.utils.to_categorical(Y_train, num_classes)Y_test = keras.utils.to_categorical(Y_test, num_classes)
To have a better explanation of this step, you should see this article.
Now that we have process our data, we can start to build out model.
As mentioned at the end of the article I wrote on Image processing, filters play a huge role in image recognition. We use filters to transform inputs and extract features that allow our model to recognize certain images. A very high-level example of this would be a curve detecting filter, which allows our model to distinguish between digits with curves and digits without curves.
1- Filters
Like all neural network weights, the filter’s weights are trainable variables. We train our neural network (via the kernel matrix weights) to produce filters that are able to extract the most useful hidden features.
When the input data has multiple channels, a filter will have a separate kernel matrix per channel. The MNIST dataset only has one channel, but for other types of image data (e.g. RGB), we would train the model to obtain optimal weights for each channel’s kernel matrix.
2-Convolution
We’ve now reached the focal point of convolutional neural networks: the convolution. The convolution represents how we apply our filter weights to the input data. The main operation used by a convolution is the matrix dot product, i.e. a summation over the element-wise product of two matrices.
The number of matrix dot products in a convolution depends on the dimensions of the input data and kernel matrix, as well as the stride size. The stride size is the vertical/horizontal offset of the kernel matrix as it moves along the input data.
3-Padding
Sometimes, when we do the dot product operation as seen before, we don’t use a row or a column. To avoid this phenomenon we can use padding.
Thus, if we want to use all the input data in our convolution, we can pad the input data matrix with 0’s. This means we add rows/columns made entirely of 0’s to the edges of the input data matrix. Since 0 multiplied by any number results in 0, the padding doesn’t affect matrix dot products. This is important because we don’t want to add any distortions to our convolution.
4-Convolution layer
A convolution layer in a CNN applies multiple filters to the input tensor. While each filter has a separate kernel matrix for each of the input channels, the overall result of a filter’s convolution is the sum of the convolutions across all the input channels.
Adding more filters to a convolution layer allows the layer to better extract hidden features. However, this comes at the cost of additional training time and computational complexity, since filters add extra weights to the model. The number of channels for the output data is equal to the number of filters the convolution layer uses.
While the convolution layer extracts important hidden features, the number of features can still be pretty large. We can use pooling to reduce the size of the data in the height and width dimensions. This allows the model to perform fewer computations and ultimately train faster. It also prevents overfitting, by extracting only the most salient features and ignoring potential distortions or uncommon features found in only a few examples.
How does pooling work?
Similar to a convolution, we use filter matrices in pooling. However, the pooling filter doesn’t have any weights, nor does it perform matrix dot products. Instead, it applies a reduction operation to subsections of the input data.
The type of pooling that is usually used in CNNs is referred to as max pooling. The filters of max pooling use the max operation to obtain the maximum number in each submatrix of the input data.
1- Adding extra layers
Like all neural networks, CNNs can benefit from additional layers. The additional layers allow a CNN to essentially stack multiple filters together for use on the image data. However, similar to building any neural network, we need to be careful of how many additional layers we add. If we add too many layers to a model, we run the risk of having it overfit to the training data and therefore generalizing very poorly. Furthermore, each additional layer adds computational complexity and increases training time for our model.
2- Increase filters
We usually increase the number of filters in a convolution layer the deeper it is in our model. In this case, our second convolution layer has 64 filters, compared to the 32 filters of the first convolution layer. The deeper the convolution layer, the more detailed the extracted features become. For example, the first convolution layer may have filters that extract features such as lines, edges, and curves. When we get to the second level, the filters of the convolution layer could now extract more distinguishing features, such as the sharp angle of a 77 or the intersecting curves of an 88.
1-Fully-connected layer
We apply a fully-connected layer of size 1024 (i.e. the number of neurons in the layer) to the output data of the second pooling layer. The number of units is somewhat arbitrary. Enough to be powerful, but not so much as to be too resource intensive. The purpose of the fully-connected layer is to aggregate the data features before we convert them to classes. This allows the model to make better predictions than if we had just converted the pooling output directly to classes.
2-Flattening
The data we have been using in our model is of the NHWC format. However, in order to use a fully-connected layer, we need the data to be a matrix, where the number of rows represents the batch size and the columns represent the data features. This time we need to reshape in the opposite direction and converting from NHWC to a 2-D matrix.
1- Co-adaptation
Co-adaptation refers to when multiple neurons in a layer extract the same, or very similar, hidden features from the input data. This can happen when the connection weights for two different neurons are nearly identical.
When a fully-connected layer has a large number of neurons, co-adaptation is more likely to occur. This can be a problem for two reasons. First, it is a waste of computation when we have redundant neurons computing the same output. Second, if many neurons are extracting the same features, it adds more significance to those features for our model. This leads to overfitting if the duplicate extracted features are specific to only the training set.
2-Dropout
The way we minimize co-adaptation for fully-connected layers with many neurons is by applying dropout during training. In dropout, we randomly shut down some fraction of a layer’s neurons at each training step by zeroing out the neuron values.
Since there are 10 possible digits an MNIST image can be, we use a 10 neuron fully-connected layer to obtain the classes for each digit class. The Softmax function is applied to the classes to convert them into per class probabilities.
Now we are ready to build our model. Here is the code:
from keras.models import Sequentialfrom keras.layers import Dense, Conv2D, Flatten from keras.layers import MaxPooling2D, Dropoutmodel = Sequential()#add model layersmodel.add(Conv2D(32, kernel_size=(5, 5),                     activation='relu',                     input_shape=input_shape))model.add(MaxPooling2D(pool_size=(2, 2)))# add second convolutional layer with 20 filtersmodel.add(Conv2D(64, (5, 5), activation='relu'))    # add 2D pooling layermodel.add(MaxPooling2D(pool_size=(2, 2)))    # flatten datamodel.add(Flatten())    # add a dense all-to-all relu layermodel.add(Dense(1024, activation='relu'))    # apply dropout with rate 0.5model.add(Dropout(0.5))    # soft-max layermodel.add(Dense(num_classes, activation='softmax'))
The model type that we will be using is Sequential. Sequential is the easiest way to build a model in Keras. It allows you to build a model layer by layer.
We use the add() method to attach layers to our model. For the purposes of our introductory example, it suffices to focus on Dense layers for simplicity. Every Dense() layer accepts as its first required argument an integer which specifies the number of neurons. The type of activation function for the layer is defined using the activation optional argument, the input of which is the name of the activation function in string format. Examples include relu, tanh, elu, sigmoid, softmax.
In this neural network, we have 2 convolution layers followed each time by a pooling layer. Then we flatten the data to add a dense layer on which we apply dropout with a rate of 0.5. Finally, we add a dense layer to allocate each image with the correct class.
Next, we need to compile our model. Compiling the model takes three parameters: optimizer, loss and metrics.
The optimizer controls the learning rate. We will be using ‘adam’ as our optimizer. Adam is generally a good optimizer to use for many cases. The adam optimizer adjusts the learning rate throughout training.
The learning rate determines how fast the optimal weights for the model are calculated. A smaller learning rate may lead to more accurate weights (up to a certain point), but the downsize is the computing time.
We will use ‘categorical_crossentropy’ for our loss function. This is the most common choice for classification. A lower score indicates that the model is performing better.
To make things even easier to interpret, we will use the ‘accuracy’ metric to see the accuracy score on the validation set when we train the model.
#compile model using accuracy to measure model performancemodel.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
Now we will train our model. To train, we will use the ‘fit()’ function on our model with the following parameters: training data (X_train), target data (Y_train), validation data, and the number of epochs.
For our validation data, we will use the test set provided to us in our dataset, which we have split into X_test and Y_test.
The number of epochs is the number of times the model will cycle through the data. The more epochs we run, the more the model will improve, up to a certain point. After that point, the model will stop improving during each epoch. For our model, we will set the number of epochs to 3.
#train the modelmodel.fit(X_train, Y_train, validation_data=(X_test, Y_test), epochs=3)
Now we have trained our model we can evaluate its performance:
# evaluate the modelscore = model.evaluate(X_test, Y_test, verbose=1)# print performanceprint()print('Test loss:', score[0])print('Test accuracy:', score[1])
Thus, we have an accuracy of 99,3% and a lost of 0.025 on the test set which is very good. We can still improve the model by increasing the number of epoch and by introducing a batch size.
If you want to see the actual predictions that our model has made for the test data, we can use the predict_classes function. We can also to this by using the predict function will give an array with 10 numbers. These numbers are the probabilities that the input image represents each digit (0–9). The array index with the highest number represents the model prediction. The sum of each array equals 1 (since each number is a probability).
To show this, we will show the predictions for the first 4 images in the test set.
Note: If we have new data, we can input our new data into the predict function to see the predictions our model makes on the new data. Since we don’t have any new unseen data, we will show predictions using the test set for now.
#predict first 4 images in the test setmodel.predict_classes(X_test[:4])
#predict first 4 images in the test setmodel.predict(X_test[:4])
We can see that our model predicted 7, 2, 1 and 0 for the first four images.
Let’s compare this with the actual results.
#actual results for first 4 images in test sety_test[:4]
The actual results show that the first four images are also 7, 2,1 and 0. Our model predicted correctly!
import tensorflow as tf # tensorflow 2.0from keras.datasets import mnistimport numpy as npseed=0np.random.seed(seed) # fix random seedtf.random.set_seed(seed)# input image dimensionsnum_classes = 10 # 10 digitsimg_rows, img_cols = 28, 28 # number of pixels# the data, shuffled and split between train and test sets(X_train, Y_train), (X_test, Y_test) = mnist.load_data()X_train = X_train.reshape(X_train.shape[0], img_rows, img_cols, 1) X_test = X_test.reshape(X_test.shape[0], img_rows, img_cols, 1)input_shape = (img_rows, img_cols, 1)# cast floats to single precisionX_train = X_train.astype('float32')X_test = X_test.astype('float32')# rescale data in interval [0,1]X_train /= 255X_test /= 255Y_train = keras.utils.to_categorical(Y_train, num_classes)Y_test = keras.utils.to_categorical(Y_test, num_classes)from keras.models import Sequentialfrom keras.layers import Dense, Conv2D, Flatten from keras.layers import MaxPooling2D, Dropoutmodel = Sequential()#add model layersmodel.add(Conv2D(32, kernel_size=(5, 5),                     activation='relu',                     input_shape=input_shape))model.add(MaxPooling2D(pool_size=(2, 2)))# add second convolutional layer with 20 filtersmodel.add(Conv2D(64, (5, 5), activation='relu'))    # add 2D pooling layermodel.add(MaxPooling2D(pool_size=(2, 2)))    # flatten datamodel.add(Flatten())    # add a dense all-to-all relu layermodel.add(Dense(1024, activation='relu'))    # apply dropout with rate 0.5model.add(Dropout(0.5))    # soft-max layermodel.add(Dense(num_classes, activation='softmax'))#compile model using accuracy to measure model performancemodel.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])#train the modelmodel.fit(X_train, Y_train, validation_data=(X_test, Y_test), epochs=3)# evaluate the modelscore = model.evaluate(X_test, Y_test, verbose=1)# print performanceprint()print('Test loss:', score[0])print('Test accuracy:', score[1])#predict first 4 images in the test setmodel.predict(X_test[:4])model.predict_classes(X_test[:4])#actual results for first 4 images in test setY_test[:4]
In this article, I tackled the second part of image recognition which is building a Convolution Neural Network.
I hope you found what you came here for in this article and stay with me for the next episodes of this image recognition trip!
PS: I am currently a Master of Engineering Student at Berkeley, and if you want to discuss the topic, feel free to reach me. Here is my email.