Dog Breed Classifier. I write the blog for Udacity Data…
By: Ma Jianglin
Tags: Udacity, Deep Learning, Machine Learning

How do we evaluate the result?

Let’s evaluate this method for person detection before we are going to use it. The metric we use is the percentage of the first 100 images in the dog and human face datasets with a detected human face. For dogs data set, we expect the detection rate should be 0 while for human face datasets we expect the detection rate is 100.

With the data base we have, for human data set, we achieved 100% accuracy while for dog data set, we achieved 12% accuracy. It seems that OpenCV face detector has relatively large false positive, but the performance is acceptable for the current experiment. However, we have to admit that the method has a limitation: it must require person with clear face. It may happen a person is captured by the camera without clear faces. In this situation, the method we just discussed will fail. A way to solve this problem is to use Deep Learning Neural Network, where we train people’s image with both clear faces and unclear/no faces. After that, we use the trained Neural Network to perform human detection.

Both human data set and dogs data set can be found at the end of the article.

Part 3: Dog Detector

How do we detect dogs?

We use a pre-trained ResNet-50 model to detect dogs in images, and the weights of the model that have been trained on ImageNet, a very large, very popular dataset used for image classification and other vision tasks. ImageNet contains over 10 million URLs, each linking to an image containing an object from one of 1000 categories. Given an image, this pre-trained ResNet-50 model returns a prediction (derived from the available categories in ImageNet) for the object that is contained in the image.

ResNet-50 is a Deep Neural Network architecture, and in total it has 25,636,712 parameters, among which 25,583,592 parameters are trainable. If you want to visualize it, please check netscope.

ResNet-50 outputs a probability vector of size 1000, and we use argmax function to identify the integer that corresponds to the model’s predicted object class. After checking this dictionary we know if the identified integer is between 151 and 268, we are sure that dogs have been detected.

How do we evaluate the result?

The way how we evaluate the dog detection accuracy is similar to what we did with person detection. We use the same human data set and dog data set, and the percentage of the first 100 images in the dog and human data sets with detected dogs is used as the evaluation metric. For human data set, ideally the detection rate should be 0 while the detection rate for dog data set should be 100.

In our experiment, we found the dog detection rate in dog data set reaches 100% percent; the dog detection rate in human data set reaches 1% percent. The result is very good for our application.

Both human data set and dogs data set can be found at the end of the article.

Part 4: Dog Breed Classifier

How do we classify dog breed?

Dog breed is not easy to be classified, and in many cases even human being cannot tell the difference between two breeds. For example:

As Deep Neural Network is becoming more and more popular, and many practitioners in Deep Learning have reported that it can reach much better result than human being. Therefore, it is natural for our current project we are going to use Deep Neural Network for dog breed classification problem.

Before we dive into Deep Neural Network, let’s first check the our dog data set, which is going to train, evaluate and test our dog breed classifier.

How does dog data set look like?

The dog data set contains 8351 images, including 133 dog categories. An overview of the images within this data set is shown below:

The dog data set image thumb nails

We randomly separate the data set into three groups: training (6680), validation (835) and test (836):

Training, validation and test data separation

How do we pre-process data?

Before the data can be used for neural network training, we need process the data so that they can become the inputs of the neural network. In our experiment, the following processing is implemented:

Image size normalization: resize the input image so that its output dimension becomes [224, 224, 3], where 224 refers to the width and height of the input image and 3 refers to the bands of the image (RGB).

Image pixel value normalization: we divide the pixel value by 255.0 so that the pixel value range is between 0.0 and 1.0

How do we design/train/evaluate a Neural Network?

Before we use well-known Neural Network architecture with transfer learning, let’s try our first Neural Network. Our Neural Network architecture is inspired by VGG-16, and it is a shallow Neural Network. We follow the following rules when designing the network:

When the image size reduced by 2, we will increase the feature channel by 2. By doing so, we will make sure there will no information loss during the training.

We also use Dropout layer to regularize the Neural Network.

We use softmax layer as the last layer because this is multi-class classification problem, and we will use categorical cross-entropy as the loss function.

The summary of our Neural Network is as follows:

Model: "sequential"

_________________________________________________________________

Layer (type) Output Shape Param #

=================================================================

conv2d (Conv2D) (None, 224, 224, 16) 448

_________________________________________________________________

max_pooling2d (MaxPooling2D) (None, 112, 112, 16) 0

_________________________________________________________________

conv2d_1 (Conv2D) (None, 112, 112, 32) 4640

_________________________________________________________________

max_pooling2d_1 (MaxPooling2 (None, 56, 56, 32) 0

_________________________________________________________________

conv2d_2 (Conv2D) (None, 56, 56, 64) 18496

_________________________________________________________________

max_pooling2d_2 (MaxPooling2 (None, 28, 28, 64) 0

_________________________________________________________________

flatten (Flatten) (None, 50176) 0

_________________________________________________________________

dense (Dense) (None, 300) 15053100

_________________________________________________________________

dropout (Dropout) (None, 300) 0

_________________________________________________________________

dense_1 (Dense) (None, 133) 40033

=================================================================

Total params: 15,116,717

Trainable params: 15,116,717

Non-trainable params: 0

For training, we use the rmsprop optimizer, and the evaluation metric we use is accuracy, which is defined as:

accuracy = sum(estimated_dog_breed==groundtruth_dog_breed)/test_dataset_number

However, the trained model does not generalize well on test data although we use early-stopping technique to pick up the best one among all the models we saved at different epochs. Our self-defined model reach 7.5359% accuracy.

Transfer learning with VGG16

Our self-designed network does not work well. There are many reasons for it: the architecture is shallow, the training epoch is small, hyper-parameters can be refined…. Among these reasons, we training data set is relatively small compared to the large amount of model parameters to estimated is a contributing factor.

We can use data augmentation to artificially increase the training samples, which is a good direction to go. In our experiment, we decide to use another technique: transfer learning.

The basic idea behind transfer learning is to use well-trained neural network as feature extractor and only retrain the last layer (or a few last layers) for the target application. Transfer learning has physiological connection with our brains, and deep learning practitioners often use it. However, we cannot use our self-defined neural network in the transfer learning context and this is because we cannot find someone has already trained our self-defined network with a lot of images. Instead, we have to use well-designed network in the literature.

Among all the well-know neural network architectures, we select VGG16 due to its simplicity and popularity. You can check its visualization on netscope website. The trained parameters of VGG16 is part of Keras library, and we can use the following code to obtain the feature extractor:

def extract_VGG16(tensor):

from keras.applications.vgg16 import VGG16, preprocess_input

return VGG16(weights='imagenet', include_top=False).predict(preprocess_input(tensor))

In Udacity course the extracted features for training/validation/test data sets have been provided by the tutors, and for VGG16 the extracted feature data base can be download from VGG-16 bottleneck features. By doing so, we can avoid generating data for training the last layer of our network.

The last convolution layer in VGG-16 of size [7, 7, 512], and the extracted VGG-16 bottleneck features have the same shape. Since we want to classify the dog breed into one of the 133 categories, a straightforward way to set up the last layer is to connect with all the bottleneck features (25088) with 133 neural net output units. However, if we do in this way, we will have a lot of parameters to estimate: 25088*133+133=3,336,837, and this number is comparable with our self-defined neural network (15,116,717 parameters). As we discussed before, one purpose of transfer learning is to reduce the number of parameters we have to learn. In order to further reduce the learned model parameters, we will use GlobalAveragePooling2D layer on the bottleneck features. After the bottleneck feature passes this layer, its shape reduced to be [1, 1, 512], and so is the learned parameters: 133*512+133 = 68,229.

The implementation in Keras is as follows:

VGG16_model = Sequential()

VGG16_model.add(GlobalAveragePooling2D(input_shape=train_VGG16.shape[1:]))

VGG16_model.add(Dense(133, activation='softmax'))VGG16_model.summary()

We use the classification accuracy to evaluate our trained model (as we did before), and we found that transfer learning can dramatically improve dog breed classification accuracy, and with VGG-16 pre-trained model, we can have 69.4976% accuracy on test data set.

Transfer learning with Inception 3

Encouraged by our interesting result with transfer learning, we decide to go further, and use Inception 3 neural network as our backbone network for transfer learning.

Inception 3 is a far more advanced neural network architecture compared to VGG-16, and it is deeper with many more parameters to train. We refer to A Review of Popular Deep Learning Architectures: ResNet, InceptionV3, and SqueezeNet to readers who are interested in this network.

Use the same test data set, and use the same accuracy metric, with Inception 3 we can achieve even better classification accuracy: 79.6651%. This is the best accuracy we can have up to now.

Part 4: Dog breed classifier application

Since Inception 3 has obtained the best result, when we create an application using the procedure disclosed in Part 1, the default dog breed classifier comes from Inception 3 model.

We are satisfied with the application, and it can give decent results. Here are some examples: