Convolutional Neural Network Champions — Part 2: AlexNet (TensorFlow 2.x) | by Amir Nejad | Towards Data Science

Convolutional neural networks are special type of neural network that is used for modeling data with strong spatial correlations such as images, multivariate time-series, earth science studies (seismic classification and regression) among many other applications. Convolutional networks have gone under significant changes since 1998 and in this series of articles I aim to reproduce the famous model architecture champions such as LeNet, AlexNet, ResNet etc. My aim is to share my findings and studies with wider audiences and deliver reproducible Python notebooks.
Part 1: Lenet-5 and MNIST classification in Tensorflow:
towardsdatascience.com
Part 3: VGGnet classification on ImageNet and Tensorflow:
towardsdatascience.com
The Python notebook for this study is located in my Github page.
The Tensorflow version used in this study is 2.3.
The LeNet-5 model demonstrated superior ability to read and classify hand written digits. Even though the LeNet-5 network structure performed well on the MNIST dataset, the real test of classifying more complicated images like CIFAR-10 showed the model’s capacity of learning such complex patterns is too low. Therefore, the development of more powerful architectures went to a state of hibernation till 2012 when AlexNet was born. Considered as the first deep CNN model, AlexNet was proposed by Krizhevesky et al. There were several developments in the meantime that throttled improvement in neural network classification accuracy, namely:
Max pooling: Pooling layer is used to reduce sensitivity of neural network models to the location of feature in the image. In the original LeNet-5 model, average pooling layers are used. However, Ranzato et al. [2007] demonstrated good results by learning invariant features using max pooling layers. The max pooling layers discriminate against features with less dominant activation functions and only select the highest values. This way only the most important features are fed through the pooling layer. For more information refer to part 1 of this series (link).GPU/CUDA programming: In the early development of neural networks, one of the main bottlenecks during model training was the computational power of computers as they primarily used the CPU to train models. In 2007, the NVIDIA corporation launched the CUDA (Compute Unified Device Architecture) programming platform to facilitate parallel processing on GPUs (graphic processing units). CUDA enables model training on GPUs and results in much faster training time as compared to CPUs. As a result, training of larger networks became possible.ReLU activation function: Nair and Hinton [2010] demonstrated rectified linear units’ (ReLU) ability to improve classification accuracy of Restricted Boltzmann Machines. ReLU units simply let any value above zero pass through the filter and suppress any value below zero. ReLU function is non-saturating, meaning the limit of the function as the input increases approaches infinity hence it can alleviate the vanishing gradient problem.ImageNet: Another catalyst to the success of the field of deep learning in general is the ImageNet database prepared by the professor Fei-Fei Li’s group at Stanford. ImageNet contains millions of annotated images from thousands of classes. The annual ImageNet Large Scale Visual Recognition Challenge (ILSVRC) oversaw many advancements in convolutional neural network structures including AlexNet. The training data is a subset of ImageNet released in the year 2012 and has 1.2 million images belonging to 1,000 classes. The validation dataset consists of 50,000 images belonging to 1,000 classes (50 image per class). A sample of the ImageNet image can be seen in the example below:
Max pooling: Pooling layer is used to reduce sensitivity of neural network models to the location of feature in the image. In the original LeNet-5 model, average pooling layers are used. However, Ranzato et al. [2007] demonstrated good results by learning invariant features using max pooling layers. The max pooling layers discriminate against features with less dominant activation functions and only select the highest values. This way only the most important features are fed through the pooling layer. For more information refer to part 1 of this series (link).
GPU/CUDA programming: In the early development of neural networks, one of the main bottlenecks during model training was the computational power of computers as they primarily used the CPU to train models. In 2007, the NVIDIA corporation launched the CUDA (Compute Unified Device Architecture) programming platform to facilitate parallel processing on GPUs (graphic processing units). CUDA enables model training on GPUs and results in much faster training time as compared to CPUs. As a result, training of larger networks became possible.
ReLU activation function: Nair and Hinton [2010] demonstrated rectified linear units’ (ReLU) ability to improve classification accuracy of Restricted Boltzmann Machines. ReLU units simply let any value above zero pass through the filter and suppress any value below zero. ReLU function is non-saturating, meaning the limit of the function as the input increases approaches infinity hence it can alleviate the vanishing gradient problem.
ImageNet: Another catalyst to the success of the field of deep learning in general is the ImageNet database prepared by the professor Fei-Fei Li’s group at Stanford. ImageNet contains millions of annotated images from thousands of classes. The annual ImageNet Large Scale Visual Recognition Challenge (ILSVRC) oversaw many advancements in convolutional neural network structures including AlexNet. The training data is a subset of ImageNet released in the year 2012 and has 1.2 million images belonging to 1,000 classes. The validation dataset consists of 50,000 images belonging to 1,000 classes (50 image per class). A sample of the ImageNet image can be seen in the example below:
Note on downloading the data: The official website of ImageNet (link) can provide the images to individuals. However, I received no download link after submitting my request. The easiest way fordownloading images is from ImageNet Object Localization Challenge (link).
AlexNet achieved a winning top-5 test error rate of 15.3% (previous model error 26%) in the ILSVRC-2012 competition. The network architecture is similar to LeNet-5 model (read more on LeNet-5: Link) but with more convolutional layers, hence deeper model.
The main activation function used in the model is the non-saturating rectified linear units (ReLU) functions. The model is mainly comprised of 8 layers: 5 convolutional layers and 3 dense layers. Kernel size is reduced from 11×11 to 3×3. Each convolutional layer is followed by a max pooling layer. The model uses dropout in the first two fully-connected layers to avoid over-fitting. The implementation of AlexNet in Tensorflow is given below.
The model is trained using the Stochastic Gradient Descent (SGD) optimization algorithm. The learning rate was initialized at 0.01 with momentum of 0.9, and weight decay of 0.0005. The code snippet to build AlexNet model in Tensorflow can be seen below:
Note, the optimizer used in the model is gradient descent with momentum. This optimizer is located in a separate package called tensorflow_addons (more info can be seen here).
Training AlexNet on the entire ImageNet dataset is time consuming and requires GPU computing capabilities. Therefore, in this section, I am going to demonstrate training of AlexNet type structure on ImageNet dataset consisting of two classes:
class n03792782: mountain bike, all-terrain bike, off-roader
class n03095699: container ship, container vessel
The training dataset is consists of 2,600 images belonging to two classes. Calling AlexNet function results a network with over 62 million trainable parameters as it can be seen below:
Model: "AlexNet"_________________________________________________________________Layer (type)                 Output Shape              Param #   =================================================================conv2d (Conv2D)              (None, 55, 55, 96)        34944     _________________________________________________________________max_pooling2d (MaxPooling2D) (None, 27, 27, 96)        0         _________________________________________________________________conv2d_1 (Conv2D)            (None, 27, 27, 256)       614656    _________________________________________________________________max_pooling2d_1 (MaxPooling2 (None, 13, 13, 256)       0         _________________________________________________________________conv2d_2 (Conv2D)            (None, 13, 13, 384)       885120    _________________________________________________________________conv2d_3 (Conv2D)            (None, 13, 13, 384)       1327488   _________________________________________________________________conv2d_4 (Conv2D)            (None, 13, 13, 256)       884992    _________________________________________________________________max_pooling2d_2 (MaxPooling2 (None, 6, 6, 256)         0         _________________________________________________________________flatten (Flatten)            (None, 9216)              0         _________________________________________________________________dense (Dense)                (None, 4096)              37752832  _________________________________________________________________dropout (Dropout)            (None, 4096)              0         _________________________________________________________________dense_1 (Dense)              (None, 4096)              16781312  _________________________________________________________________dropout_1 (Dropout)          (None, 4096)              0         _________________________________________________________________dense_2 (Dense)              (None, 1000)              4097000   _________________________________________________________________dense_3 (Dense)              (None, 2)                 2002      =================================================================Total params: 62,380,346Trainable params: 62,380,346Non-trainable params: 0_________________________________________________________________
AlexNet model is trained for 90 epochs on the entire training data and validated on 50K images from the validation dataset. An example of a training model on a CPU can be seen below (to train on GPU, usetf.distribute.MirroredStrategy):
The fully trained AlexNet model on 2 classes can reach the accuracy of 95%. The learning curve and losses of training and validation set can be seen in the following figure. As it can be seen, the training loss on the validation set stays flat after 20 epochs and model learning cannot be improved.
Another way of assessing the model performance is using a method called confusion matrix. A confusion matrix is the table lay-out, comprised of data categories, and the resulting prediction using the trained classifier. An example of a confusion matrix can be seen here. This confusion matrix is obtained by running the trained neural network classifier on 100 validation images (50 images per category). As it can be seen, the model has misclassified only 1 image on the bikes category (denoted by 0). However, the model misclassified 3 images on the ships category (denoted by 1).
The 4 misclassified images are illustrated below. It seems the model cannot fully recognize objects in the image when:
Object is cropped in the image (partially visible object)
Object is in the background or covered by the surroundings
Looking at images above and comparing them to the training images, it is hard to discern the reason behind misclassification on such images. One method to make CNN models more transparent is by visualizing the regions of input that are ‘important’ for predictions. There are many methods built to accomplish this task such as GradCAM, GradCAM++, ScoreCAM etc. I used GradCAM from tf-keras-vis library (more info: https://pypi.org/project/tf-keras-vis/) to inspect model behavior on wrongly classified images. The results can be seen below. As it can be seen, the model has a hard time focusing on the import regions (denoted by red color).
The following images demonstrated examples of correctly classified images. As it can be seen, the model pays attention to important features in the images and predicts the class of image correctly. It is interesting to note that, the trained model seems to be able to identify bicycles and the person riding the bicycle. An objection identification model that not only detects the object but also detects the location of object within an image can properly solve this problem. AlexNet is only capable of detecting an object but not identifying an object within the image. The subject of object identification will be discussed in future episodes of this series.
AlexNet unlocked the new age of computer vision and deep learning. AlexNet introduced (or popularized in some cases) many relevant computer vision methods used today such as the conv+pooling design, dropout, GPU, parallel computing, and ReLU. Therefore as a result of such inventions, AlexNet was capable of reducing classification errors on the ImageNet dataset. However, the AlexNet model needed to improve classification accuracy. As a result many models were built on the success of AlexNet such as VGGNet, which I am going to explore and discuss in the next article. Stay tuned and let me know your opinions below.
Thanks for reading! My name is Amir Nejad,PhD. I’m a data scientist and editor of QuantJam , and I love share my ideas and to collaborate with other fellow data scientists. You can connect with me on Github, Twitter, and LinkedIn.
QuantJam:
medium.com
You can see my other writings at:
amirnejad.medium.com
All images have been produced by the author, except where stated otherwise.
Relu Activation Function:
Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. “Imagenet classification with deep convolutional neural networks.” Advances in neural information processing systems. 2012.
Lecture 9, CNN Architecture, Fei-Fei Li & Serena Yeung, 2017 (Youtube)
Max Pooling:
Ranzato, Marc’Aurelio, et al. “Unsupervised learning of invariant feature hierarchies with applications to object recognition.” 2007 IEEE conference on computer vision and pattern recognition. IEEE, 2007.
ReLU:
Nair, Vinod, and Geoffrey E. Hinton. “Rectified linear units improve restricted boltzmann machines.” ICML. 2010.
CUDA:
E. Lindholm, J. Nickolls, S. Oberman, and J. Montrym, “NVIDIA Tesla: A Unified Graphics and Computing Architecture,” IEEE Micro, vol. 28, no. 2, pp. 39–55, Mar. 2008.
ImageNet:
Russakovsky, Olga, et al. “Imagenet large scale visual recognition challenge.” International journal of computer vision 115.3 (2015): 211–252.