Transfer Learning-based Road Damage Detection for Multiple Countries
By: Ananda Hange
Tags: Faster R Cnn, Object Detection, Transfer Learning, Deep Learning

Road Damage Detection

Table Of Content:

Introduction Business Problem Data Posing this as a Deep Learning problem Exploratory Data Analysis (EDA) Transfer Learning Final Pipeline of the problem Deployment Of Model Conclusion and Future Scope My Profile Reference

1. Introduction:

Creating accurate Machine Learning Models which are capable of recognizing and localizing multiple objects in a single image remained a core challenge in computer vision. But, with recent advancements in Deep Learning, Object Detection applications are easier to develop than ever before.

Many municipalities and road authorities seek to implement automated evaluation of road damage. However, they often lack technology, know-how, and funds to afford state-of-the art equipment for data collection and analysis of road damages. Although some countries, like Japan, have developed less expensive and readily available Smartphone-based methods for automatic road condition monitoring, other countries still struggle to find efficient solutions.

This work makes the following contributions in this context. Firstly, Analyse a large-scale heterogeneous road damage dataset comprising 26620 images collected from multiple countries (Czech, India and Japan)using smartphones. Secondly, Pre-processing on labelled images. Thirdly, we do transfer learning with state of the art object detection models (3 models).Lastly, build end to end pipeline (deployment) with the best model.

Road Damage Types:

Table : Road Damage Types

As illustrated in above Table , the classification of road damages includes eight types which can be generalized into two categories: cracks and other corruptions. The crack category is either linear or alligator cracks. The linear cracks can be longitudinal and lateral. Meanwhile, the category of the other corruptions include three sub-categories: potholes and rutting, white line blur, and crosswalk blur.

2. Business Problem:

2.1 Business Objective and Constraint :

Given 3 countries road images, train (Fine tune) model to predict damage objects with bounding boxes and label (class). Transfer Learning Concept is used to fine tune existing object detection models on our custom dataset. Detect as many as object with high mAP (mean average precision) metric No strict latency Constraints

3. Data:

The proposed dataset builds upon the recently introduced RDD-2019 dataset.

The dataset is publicly available on the website:https://github.com/sekilab/RoadDamageDetector

Dataset Contains:

Images which contains images from all three countries namely Czech, India and Japan

which contains images from all three countries namely Czech, India and Japan XML files which contain bounding box dimension with label in PASCAL format

which contain bounding box dimension with label in PASCAL format Label file which contains labels.

4. Posing this as a Deep Learning problem:

Object Detection is type of Deep learning (CNN) problem which consists of two tasks such as Classification of labels and Regression for bounding boxes. We have images and their corresponding bounding boxes in the data Given these 2 as input to the model we need to predict the bounding boxes around each detected damage on the road in a given image from the test data. This prediction involves object detection in the images and plotting bounding boxes around it. We use state of the art techniques like Faster RCNN ,SSD MobileNet and Yolov3 to do these tasks. Performance metric that we are going to use to evaluate model is mAP (mean Average Precision)

5. Exploratory Data Analysis (EDA)

First we analyse dataset by using following code:

we get a total number of images as 21041, total number of labels in annotations files are 34702 and we have 10 unique labels(Classes).

Now we see distribution of classes ,

we get,

Table: Labels Distribution

Fig : Count plot for Distribution of labels

We can see from the above distribution that we have imbalance classes . D0w0, D11, D01 and D43 are very rare classes.

Now we will see country wise images count and label count,

we get,

Table : Country wise labels distribution.

We observed:

Czech country have only 4 classes (label) India have all classes (10 classes) Japan has 7 classes. We have imbalance classes.

So we can say that only India contains all classes. Also in India we have class ‘D0w0’ occurs only once. We will remove that or will not include while training.

Japan has more images compared to Czech and India.

Now we will create a dataframe from xml files. Which contain filename, dimension of images, bounding box dimensions and label (class).

we get,

We can see that one image can have multiple labels.And image size is not fixed. So now we check how many images have labels.

So we have a total of 14569 images that have labels. So these images we will use for training other images we will discard.

Now we will visualize some images with bounding boxes:

5.1 Train–Test Split:

After removing ‘D0w0’ labelled image from dataset Finally, we have total images with valid labels are 14568. And total unique labels are 9.

We will split 14568 images in an 80:20 ratio.

So our total images for training are 11654 and for validation 2914.

Since we have imbalance classes ,So we will check distribution of classes after train and test split by using following code snippet:

Train set class distribution:

Fig : Train set labels distribution

Test set class distribution:

Fig : Test set labels distribution

Both train and test Distribution are almost the same. So we can say we sampled data in a stratified manner.

So here we are done with splitting of images.

Now we will do augmentation.

5.2 Augmentation:

Data augmentation is a technique to artificially create new training data from existing training data. This is done by applying domain-specific techniques to examples from the training data that create new and different training examples.

It helps us to increase the size of the dataset and introduce variability in the dataset, without actually collecting new data. The neural network treats these images as distinct images anyway. Also Data Augmentation helps reduce overfitting. Our dataset may have images taken in a limited set of conditions but we might fall short in a variety of conditions that we don’t account for. Here the modified/augmented data helps deal with such scenarios.

So, to get more data, we need to make minor alterations to our existing training data. Here we are specifically talking about Image Data Augmentation. These alterations include flipping the image horizontally, vertically, padding, cropping, rotating, scaling and few other translations.

So first we resize the images as per model we are using for fine tuning and then we will do augmentation technique on top of that. We will apply augmentation techniques on train images only.

Below code snippet shows implementation of augmentation:

So after augmentation we have a total 23306 train images in the train set. Now we are done with data preparation. Let’s go to the modeling step.

6. Transfer Learning:

Transfer Learning

Transfer learning is a machine learning method where a model developed for a task is reused as the starting point for a model on a second task.

It is a popular approach in deep learning where pre-trained models are used as the starting point on computer vision and natural language processing tasks given the vast compute and time resources required to develop neural network models on these problems and from the huge jumps in skill that they provide on related problems.

For our problem we use three pre-trained models such as Faster RCNN ,SSD MobileNet And YoloV3.

6.1 Faster RCNN:

Faster RCNN is composed from 3 parts:

Fig : Faster R-CNN

Convolution layer RPN Fast R-CNN (RoI pooling)

Key steps followed by a Faster R-CNN algorithm to detect objects in an image:

Take an input image and pass it to the ConvNet which returns feature maps for the image. Apply Region Proposal Network (RPN) on these feature maps and get object proposals. Apply ROI pooling layer to bring down all the proposals to the same size. Finally, pass these proposals to a fully connected layer in order to classify any predicted bounding boxes for the image.

Problems with R-CNN network:

Training the data is unwieldy and too long Training happens in multiple phases (e.g. training region proposal vs classifier) Network is too slow at inference time (i.e. when dealing with non-training data)

Training Faster R-CNN on our custom dataset :

For training Faster R-CNN we will use TensorFlow 2 Object Detection API.

So for that we need to install following prerequisite :

Installed TensorFlow Installed TensorFlow Object Detection API

Training Object Detection model comprises following steps:

Preparing Workspace Preparing Dataset Partition Dataset (train — test split) Generate TensorFlow record for dataset Train Model Evaluate Model Export the resulting model and use it to detect objects Inference

All these above Mention steps are described in https://tensorflow-object-detection-api-tutorial.readthedocs.io/en/latest/training.html and all steps with code is available in my GitHub repo link mentioned at the last.

For Faster RCNN I choose pre-trained Faster RCNN Inception resnet 640 x 640 from: https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf2_detection_zoo.md

A. Training Model:

I trained model for 25k steps,

B. Model Evaluation :

We get following result,

Overall, we get mAP @ IOU = 0.5 as 0.244.

C. Inference Model :

6.2 SSD Mobilenet:

To better understand SSD, let’s start by explaining where the name of this architecture comes from

Single Shot : this means that the tasks of object localization and classification are done in a single forward pass of the network

: this means that the tasks of object localization and classification are done in a single forward pass of the network MultiBox: this is the name of a technique for bounding box regression.

this is the name of a technique for bounding box regression. Detector: The network is an object detector that also classifies those detected objects

Many CNN architectures, such as R-CNN (Region-based Convolutional Neural Networks; Fast-CNN, or Faster-CNN have been developed to attain the best accuracy while improving the processing speed. However, the computational load was still too large for processing images on devices with limited computation, power, and space. Therefore, the Single Shot Multi-Box Detector (SSD) framework was developed to improve further the computation speed of the object detection . It uses a single feed-forward convolutional network to detect multiple objects within the image directly and combines predictions from numerous feature maps with different resolutions to handle objects of various sizes.

Source : SSD Single Shot MultiBox Detector

MobileNet is a small low-latency, and low-power convolutional feature extractor that can be built to perform classification, detection, or segmentation similar to popular large-scale models, such as Inception SSD. It is based on depth-wise separable convolution, which factorizes a standard convolution into a depth-wise convolution, and 1 x 1 convolution called a point-wise convolution. Depth-wise convolution maps a single convolution on each input channel separately, and point-wise convolution is a convolution with a kernel size of 1x1 that combines the features created by the depth-wise convolution. In comparison to depth-wise separable convolution, a regular convolution does both filtering and combination steps in the single run. However, SSD MobileNet requires more computational work to accomplish the task, and it needs to learn more weights.

Fig . Comparison of (a) Regular and (b) Depth-wise Separable Convolutions applied on 3-channel (RGB) Image

SSD object detection framework using a MobileNet base network was found to require relatively small CPU load and memory consumption while maintaining high accuracy.

Training of the SSD MobileNet model is the same as the Faster R-CNN model.

Here I choose pre-trained SSD MobileNet V2 fpnlite 640 x 640 model from here.

I trained this model for 60k steps on our custom dataset. And after evaluation of model we get following result:

Overall, We get mAP @IOU=0.5 as 0.375.

Inference Model:

6.3 YOLOv3 :

The “You Only Look Once” (YOLO) method [16] uses a different approach and basically merges the two steps of the RCNN algorithm into one step by developing a neural network which internally divides the image into regions and predicts categories and probabilities for each region. Thus, applying the prediction once makes YOLO achieve a significant speedup compared to R-CNN-based algorithms; hence can be used for real-time prediction

Source : YoloV3 Architecture

If you want to read more about Yolov3 here is good Blog

Training of YOLOv3 model is explained step by step in this blog and this blog.

I followed the same steps and trained the model for 30 epochs. And get mAP@0.5 score as 0.365.

Inference:

Now we train our models on road damage images. Now we summarize some parameters we get from training and inference of models by following chart,

Table : Comparison Chart

As from the above comparison chart we can conclude that SSD MobileNet Model works fairly well compared to other models.

7. Final Pipeline of the problem:

Here we are going to see about model productionization

From the comparison of our model we conclude that the SSD MobileNet model works fairly well. So for the final pipeline we will use the SSD MobileNet model.

Process is as follows:

Select Image from test set. Resize this image to 640 x 640 size Inference

Fig : Inference Image

8. Deployment of model :

Now we Deploy(Develop a simple Web App) our final model using streamlit and ngrok.

The simplicity that streamlit offers for building apps is so cool and useful. But sometimes you may be constrained by lack of a compute or incompatible working environment on your local computer. In most cases Google Colab solves that problem. But what if you want to share your streamlit app with others from Google Colab.

ngrok allows us to create a secure tunnel for accessing our local apps and webhooks. We will be using pyngrok which acts as a python wrapper around ngrok

Steps for building web app using streamlit and ngrok:

Create An Account on Ngrok Get Your Authentication Tokens Write Your App to Your Colab Sandbox Create A Secure Tunnel Using Pyngrok

Checkout demo video of my web app :

9. Conclusion and Future Scope:

We see that Model not able to identify multiple object and small or Blur objects very well with good confidence For this we need to increase training steps i.e. we need to train models for more steps. Also need to try different augmentation techniques This work can be used as a baseline, and experiment can be repeated by collecting more images from different countries, under different seasonal condition for better representing each damage class and improving robustness of overall detection system.

10. My Profile:

GitHub Repo:

All codes related to training models, pipeline and deployment are available in my GitHub repo.

If you are interested in this case study or wants to improve it further, then Jupyter Notebook is available with all codes at my following repo:-

2. LinkedIn Profile:

If you are interested, here is link to my LinkedIn profile

11. Reference :

I hope you learnt something new through this read!

Happy Learning !!!