Convolutional Neural Network Champions —Part 1: LeNet-5 (TensorFlow 2.x)
By: Amir Nejad
Tags: Python, TensorFlow, Deep Learning, Convolutional Network, Data Science

Applying convolutional layers (aka “ConvNet”) on images and extracting key features of the image for analysis is the main premise of ConvNets. Each time “Conv-layers” is applied to an image and divides this image into small slices known as receptive fields, hence extracting important features from the image and neglecting less important features. The kernel convolves with the images using a specific set of weights by multiplying its elements with the corresponding elements of the receptive field. It is common to use “pooling layers” in conjunction with Conv-layers to downsample the convolved features and to reduce the sensitivity of the models to the locations of the features in the input.

Finally, adding dense blocks to the model and formulating the problem (classification and/or regression), one can train such models using the typical gradient descent algorithms such as SGD, ADAM, and RMSprop.

Of course, prior to 1998, the use of convolutional neural networks were limited and typically support vector machine was the preferred method of choice in the field of image classification. However, this narrative changed when LeChun et al. [98] published their work on the use of gradient-based learning for handwritten digits recognition.

Data

LeNet models are developed based on MNIST data. This data-set consists of the hand written digits 0–9; sixty thousand images is used for training/validation of the model and then a thousand images are used to test the model. The images in this data-set have a size of 28×28 pixels. An example can be seen in the following figure. The challenge of using a MNIST data-set is that the digits often have slight changes in shape and appearance (for example, the number 7 is written different way).

Examples of MNIST data sample

Looking at the labels in the MNIST data set, we can see the number of labels are balanced, meaning there is not too much disparity.

Label counts in the MNIST data set

Network Structure

LeCun et al. [98], The proposed structure of LeNet5 network

The proposed model structure of LeNet-5 has 7 layers, excluding input layers. As described in the Data section, images used in this model are MNIST handwritten images. The proposed structure can be seen in the image above, taken from the LeChun et al. [98] paper. The details of each layer are as follows:

Layer C1 is the first Conv-layer with 6 feature maps with strides of 1. Using a formula given in the appendix, one can calculate the output dimension of this layer 28×28 with 156 trainable parameters (refer to appendix 1 for details). Activation function of this layer is tanh (refer to appendix 2 for more details). Layer S2 is an average pooling layer. This layer maps average values from the previous Conv layer to the next Conv layer. The Pooling layer is used to reduce the dependence of the model on the location of the features rather than the shape of the features. The pooling layer in LeNet model has a size of 2 and strides of 2. Layer C3 is the second set of the convolutional layer with 16 feature maps. The output dimension of this layer is 10 with 2,416 parameters. Activation function of this layer is tanh . Layer S4 is another average pooling layer with dimension of 2 and stride size of 2. The next layer is responsible for flattening the output of the previous layer into one dimensional array. The output dimension of this layer is 400 (5×5×16). Layer C5 is a dense block (fully connected layer) with 120 connections and 48,120 parameters (400×120). Activation function of this layer is tanh . Layer F6 is another dense block with 84 parameters and 10,164 parameters (84×120+84). Activation function of this layer is tanh . Output Layer has 10 dimension (equals number of classes in the database) with 850 parameters (10×84+10). Activation function of output layer is sigmoid (refer to appendix 2 for more details).

The following code snippet demonstrates how to build a LeNet model in Python using Tensorflow/Keras library. Keras sequential model is a linear stack of layers. Then, we need to define each layer as seen below. Finally, the model needs to be compiled and the choices of optimizer, loss function, and metrics need to be explicitly defined. The optimizer used in this work is sgd or Stochastic Gradient Descent. The loss function is optimized to train the machine learning model. The loss function used here is cross-entropy, or log loss, and measures the performance of a classification model whose output is a probability value between 0 and 1. An accuracy metric is used to evaluate the performance of training. Loss function is a continuous probability function while accuracy is a discrete function of the number of correctly predicted labels divided by total number of predictions (refer to appendix 3).

LeNet-5 Layers Structure

Note that in the above code snippet, we have not specified anything about how the weight of neural network is initialized. By default, Keras uses a glorot_uniform initializer. Weight values are chosen randomly in a way to make sure that information passed through the network can be processed and extracted. If the weight is too small, the information shrinks as a result. If the weight is too large, the information grows as a result and becomes too big to process. The Glorot uniform algorithm (also known as Xavier algorithm) chooses appropriate random weight values from a multivariate random normal scaled by the size of the neural network [refer Glorot 2010].

The summary of LeNet-5 network constructed with Tensorflow is given below (Using model.summary() ) :

Model: "sequential"

_________________________________________________________________

Layer (type) Output Shape Param #

=================================================================

conv2d (Conv2D) (None, 28, 28, 6) 156

_________________________________________________________________

average_pooling2d (AveragePo (None, 14, 14, 6) 0

_________________________________________________________________

conv2d_1 (Conv2D) (None, 10, 10, 16) 2416

_________________________________________________________________

average_pooling2d_1 (Average (None, 5, 5, 16) 0

_________________________________________________________________

flatten (Flatten) (None, 400) 0

_________________________________________________________________

dense (Dense) (None, 120) 48120

_________________________________________________________________

dense_1 (Dense) (None, 84) 10164

_________________________________________________________________

dense_2 (Dense) (None, 10) 850

=================================================================

Total params: 61,706

Trainable params: 61,706

Non-trainable params: 0

_________________________________________________________________

Now that we have constructed the LeNet model using Tensorflow and Keras, we need to train the model. Using model.fit() and feeding training and validation sets, the model is trained. Additional parameters needed for training are the number of epochs, batch size and verbose:

An epoch is one complete presentation of training data. Samples from the training data set are selected randomly and presented to the model to learn. An epoch, therefore, represents one cycle through the full training data set.

As mentioned before, an epoch refers to the complete presentation of data to model. Training data is selected randomly and fed to the model. The number of samples in the randomly selected data is called batch size. Smaller batch sizes are noisy compared to the larger batch sizes, but they may generalize well. Larger batch sizes are used to avoid memory limitation problems especially when using Graphics Processing Units (GPU).

Verbose specifies the frequency of output logging. If set to 1, each iteration model loss will be printed.

Training code snippet. Note to_categorical command is used to convert a class vector to binary class matrix.

Once the model is trained, we can use a testing set that we have set aside to evaluate the performance of model training using model.evaluate() command:

Testing code snippet

The result of training the model for 10 epochs can be seen in the following figure. Initially, the weights of neural network are chosen randomly, but after 2 epochs of presenting 48,000 pictures to the model, the model loss reduced from 0.38 to 0.1. After 10 epochs of model training, the model accuracy surpassed 95% on testing set. This is a substantially improved accuracy compared to previous models at the time (mainly support vector machines), and as a result LeNet-5 cemented its legacy as one of the earliest champions of computer vision.

LeNet-5 Training results (10 epochs)

Using model.optimizer.get_config() we can interrogate the optimizer parameters. Note that we only specified the type of optimizer, the loss function, and accuracy metrics. As it can be seen from the following snippet, the optimizer used to train LeNet model is a Stochastic Gradient Descent (SGD) optimizer. The learning rate by default is set at 0.01. The learning rate controls the change in the model parameters in response to observed error measured by loss function. Imagine the model training process as traversing from a hill to valley, and the learning rate defines step sizes. The larger step size traverses to the solution faster but it might result in jumping over the solution. On the other hand, smaller step sizes take too long to converge. In the more complex problems, step size decay is commonly used. In this problem, however, no decay is sufficient to get good results. Another parameter using SGD is momentum . Instead of using only the gradient of the current step to guide the search, momentum also accumulates the gradient of the past steps to determine the direction to go. Momentum, therefore, can be used to improve converge the speed of the SGD. Another parameter is nesterov . If set to true (boolean), SGD enables the Nesterov accelerated gradient (NAG) algorithm. NAG is also a closely related algorithm to the momentum in which step sizes are modified using velocity of the learning rate change (refer to Nesterov [1983]).

model.optimizer.get_config():

{'name': 'SGD',

'learning_rate': 0.01,

'decay': 0.0,

'momentum': 0.0,

'nesterov': False}

Model Interrogation

There are multiple ways to assess the performance of a classifier. Accuracy performance on the test data set that held back from any of the model development tasks is the obvious choice. However, the confusion matrix can provide a detailed report of classifier and better assess the classification performance. Furthermore, the Classification accuracy can be misleading if an unequal number of observations in each class are present.

A confusion matrix is a detailed report of the number of samples classified correctly/incorrectly. The number of samples along the diagonal of the confusion matrix are correctly predicted samples. All other samples are miss-classified. The higher the number of samples on the diagonal, the higher the model accuracy. As it can be seen from the confusion matrix of LeNet-5 on MNIST data set, most of the classes are classified correctly. However, there are a few cases the classifier had trouble correctly classifying the label such as label 5,4, and 8. For example, there were 16 cases the classifier incorrectly classified number 2 as number 7. Those aforementioned cases are depicted in the following image.

LeNet-5 Confusion Matrix

Example of miss-classified label

Choice of Optimizer

In the previous section, it is mentioned the SGD optimizer is used for this optimized neural network model. However, because of the slow convergence of SGD and problems of getting stuck at local minima, this method is not popular. Since its introduction, Adaptive Moment Estimation aka Adam (refer to Kingma et al. [2014] for more details) enjoys significant popularity in the field of deep learning. The Adam optimization algorithm is an extension to stochastic gradient descent in which momentum by default is applied to gradient calculation, and separate learning rates for each parameter. Using Adam optimizer and retraining the LeNet-5 from scratch, model accuracy can be increased to 98% as seen in the following learning curves:

LeNet-5 Training results (10 epochs) using Adam optimizer.

Effect of Batch Size

The batch size is one of the most important hyper-parameters in neural network training. As discussed in the previous section, the neural network optimizer during each training epoch randomly selects data and feeds it to the optimizer. The size of the selected data is called the batch size. Setting the batch size to the entire size of the training data may cause the model to be unable to generalize well on data it hasn’t seen before (refer to Takase et al. [2018]). On the other hand, setting batch size to 1 results in higher computational training time. The proper choice of the batch size is particularity important as it leads to model stability and increases in accuracy. The following two bar charts demonstrate testing accuracy and training time of various batch sizes from 4 to 2,048. Testing accuracy of the model for batch sizes 4 to 512 is above 98%. However, the training time of batch size 4 is more than four times the training time of batch size 512. This effect can be more severe on more complex problems with a large number of classes, and large numbers of training samples.

Effect of batch size on model accuracy (upper chart) and training time (lower chart)

Effect of Pooling Layer

As discussed before, the pooling layer is required to down sample the detection of features in feature maps. There are two most commonly used pooling operators: average pooling and max pooling layers. Average pooling layer operates by calculating average values of the selected patch in the feature map, whereas max pooling layer calculates maximum value of the feature map.

Max pooling operation as it can bee seen in the following figure, works by selecting the maximum feature value from the feature map. Max pooling layers discriminate against features with less dominant activation functions and only select the highest values. This way only the most important features are fed through pooling layer. The major drawback of max pooling is that the pooling operator in the regions with features of high magnitude, only the highest value feature is elected and the rest of features are ignored; the discerning features disappeared after performing max pooling operations and results in loss of information (the purple region in the following figure).

Max Pooling operation

Average pooling, on the other hand, works by computing average values of the features in the selected region of the feature map. All parts of the selected region in the feature map are fed through using average pooling. If the magnitude of all the activations is low, the computed mean would also be low and rise due to reduced contrast. The situation will be worst when the most of the activations in the pooling area come with a zero value. In that case, feature map characteristics would reduce by a large amount.

Average Pooling Operation

As indicated earlier, the original LeNet-5 model uses average pooling strategy. Changing average pooling to the max pooling strategy resulted in approximately the same testing accuracy on the MNIST data set. One can argue the point of different pooling layers. However it should be noted that the MNIST data set is rather simple compared to other complex data sets such as CIFAR-10 or Imagenet, hence the performance benefits of max pooling in such data sets can be by far more beneficial.

Comparison between average pooling and max pooling strategies

Effect of Feature Rotation and Flipping

Thus far we have explored different aspects of LeNet-5 model including the choice of optimizer, effect of batch size, and choice of pooling layer. LeNet-5 model is designed based on MNIST data. As we have seen so far, the digits are centered in each image. However, more often than not, the location of the digits in the image in real life is shifted, rotated, and sometimes flipped. In the following few sections we will explore the effects of image augmentation and sensitivity of the LeNet-5 model to image flipping, rotation, and shifting. Image augmentation is done with the help of the Tensorflow image processing module, tf.keras.preprocessing.

Effect of Flipping

In this exercise, images are flipped along the horizontal axis using ImageDataGenerator (horizontal_flip=True) . Applying ImageDataGenerator tests image results in the new data set with the images horizontally flipped as seen in the following image. As it can be seen, it is expected the model have low accuracy on the flipped images data set. As seen from the testing accuracy table, the accuracy of the LeNet-5 model dropped from 98% to 70%.

Test accuracy on flipped images

A closer look at the confusion matrix of the flipped images data set reveals few interesting takeaways. The highest accuracy labels are 0,1,8, and 4. The first three labels are symmetrical (0,1,8) and as a result model has good accuracy of prediction on such classes. But it is interesting that the LeNet-5 model has good classification accuracy on label 4. Another interesting aspect of this test is how the model identifies digits. For example one of the labels that the model suffers from accuracy in the flipped data set is 3. The model almost half the time misclassified it as the number 8. It is very useful to understand how the model identifies each digit in order to build a robust classifier. Packages like SHAP can provide means of understanding input and output mapping of any deep neural network model (look for DeepExplainer module in SHAP library).

Confusion matrix of flipped images prediction

Image Rotation

Image rotation is another possible scenario in real life. Digits can be written in an angle with respect to the image boundaries. With the Tensorflow image augmentation module, one can produce randomly rotated images using the following line of code: ImageDataGenerator(rotation_range=angle) . The testing result of LeNet-5 model with various randomly rotated images can be seen in the following figure. The more rotation, the worse the prediction of the model. It is interesting to note that model predictions are rather satisfactory for up to 20 degrees of rotation, then the predictions degrade rapidly.

Prediction of LeNet-5 on randomly rotated images

Effect of Shifting

One final image augmentation effect is shifting digits along the horizontal or vertical axis within the image. This effect can be easily applied to MNIST test dataset using ImageDataGenerator(width_shift_range=shift) . Note that, in this section I am demonstrating the result of width_shift generator. The sensitivity of LeNet-5 network to width shift is much higher than image flipping and image rotation. As can be seen from the following figure, the accuracy degrades much faster than other discussed image augmentation processes. Only a 10-degree width shift results in accuracy drop from over 95% to about 48%. This effect might be attributed to the filter size and kernel dimensions of the model.

Prediction of LeNet-5 on randomly shifted images

Performance on CIFAR-10 Data set

As we have seen from all the previous sections, LeNet-5 model achieved a significant milestone in the hand written digit recognition. Due to its superior performance on classification problems, LeNet-5 model used in banks and ATM machines for automatic digit classification in mid-1990s. However, the next frontier for this model was addressing image recognition problem to identify various objects in the image.

In this final section, we aim to train the LeNet-5 on CIFAR-10 dataset. CIFAR-10 (Canadian Institute For Advanced Research) is an established computer vision data set with 60,000 color images with the size 32×32 containing 10 object classes as it can be seen from the following picture. The 10 different classes represent airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships, and trucks. As can be seen from the images below, the complexity of images is much higher than that of MNIST.

CIFAR-10 Data Set

Applying LeNet-5 structure to this data set and training the model for 10 epochs, results in an accuracy of 73%. The testing accuracy of this model is 66%. Considering human level accuracy on this data set is about 94% (according to Ho-Phuoc [2018]), LeNet-5 type structure is not efficient enough to achieve high recognition capability.