Automatic License Plate Detection & Recognition using deep learning | by Achraf KHAZRI | Towards Data Science

Introduction
The massive integration of information technologies, under different aspects of the modern world, has led to the treatment of vehicles as conceptual resources in information systems. Since an autonomous information system has no meaning without any data, there is a need to reform vehicle information between reality and the information system.This can be achieved by human agents or by special intelligent equipment that will allow identification of vehicles by their registration plates in real environments. Among intelligent equipment, mention is made of the system of detection and recognition of the number plates of vehicles.The system of vehicle number plate detection and recognition is used to detect the plates then make the recognition of the plate that is to extract the text from an image and all that thanks to the calculation modules that use location algorithms, segmentation plate and character recognition.The detection and reading of license plates is a kind of intelligent system and it is considerable because of the potential applications in several sectors which are quoted:
- Command force: This system is used for the detection of stolen and searched vehicles. The detected plates are compared to those of the reported vehicles.
- Parking management: The management of car entrances and exits.
Road safety: This system is used to detect license plates exceeding a certain speed, coupling the plate reading system with road radar, crossing wildfires ...
Our project will be divised into 3 steps :
Step1 : Licence plate detection
In order to detect licence we will use Yolo ( You Only Look One ) deep learning object detection architecture based on convolution neural networks.
This architecture was introduced by Joseph Redmon , Ali Farhadi, Ross Girshick and Santosh Divvala first version in 2015 and later version 2 and 3.
Yolo v1 : Paper link.
Yolo v2 : Paper link.
Yolo v3 : Paper link.
Yolo is a single network trained end to end to perform a regression task predicting both object bounding box and object class.
This network is extremely fast, it processes images in real-time at 45 frames per second. A smaller version of the network, Fast YOLO, processes an astounding 155 frames per second.
Implementing YOLO V3:
First, we prepared a dataset composed of 700 images of cars that contains Tunisian licence plate, for each image, we make an xml file ( Changed after that to text file that contains coordinates compatible with Darknet config file input. Darknet : project used to retrain YOLO pretrained models) using a desktop application called LabelImg.
# First download Darknet project$ git clone https://github.com/pjreddie/darknet.git# in "darknet/Makefile" put affect 1 to OpenCV, CUDNN and GPU if you # want to train with you GPU then time thos two commands$ cd darknet$ make# Load convert.py to change labels (xml files) into the appropriate # format that darknet understand and past it under darknet/   https://github.com/GuiltyNeuron/ANPR# Unzip the dataset$ unzip dataset.zip# Create two folders, one for the images and the other for labels$ mkdir darknet/images$ mkdir darknet/labels# Convert labels format and create files with location of images# for the test and the training$ python convert.py# Create a folder under darknet/ that will contain your data$ mkdir darknet/custom# Move files train.txt and test.txt that contains data path to# custom folder$ mv train.txt custom/$ mv test.txt custom/# Create file to put licence plate class name "LP"$ touch darknet/custom/classes.names$ echo LP > classes.names# Create Backup folder to save weights$ mkdir custom/weights# Create a file contains information about data and cfg # files locations$ touch darknet/custom/darknet.data# in darknet/custom/darknet.data file paste those informationsclasses = 1train  = custom/train.txtvalid  = custom/test.txtnames = custom/classes.namesbackup = custom/weights/# Copy and paste yolo config file in "darknet/custom"$ cp darknet/cfg/yolov3.cfg darknet/custom# Open yolov3.cfg and change :# " filters=(classes + 5)*3" just the ones before "Yolo"# in our case classes=1, so filters=18# change classes=... to classes=1# Download pretrained model$ wget https://pjreddie.com/media/files/darknet53.conv.74 -O ~/darknet/darknet53.conv.74# Let's train our model !!!!!!!!!!!!!!!!!!!!!$ ./darknet detector train custom/darknet.data custom/yolov3.cfg darknet53.conv.74
After finishing the training, to detectect u liscence plate from an image, choose the latest model from darknet/custom/weights , and put its path or name in file object_detection_yolo.py, also we will use yolov3.cfg file, just in this file put # before training so we desable training then run :
python object-detection_yolo.py --image= image.jpg
and this what we have as a result :
Step2 : Licence plate segmentation
Now we have to segment our plate number. The input is the image of the plate, we will have to be able to extract the unicharacter images. The result of this step, being used as input to the recognition phase, is of great importance. In a system of automatic reading of number plates.
Segmentation is one of the most important processes for the automatic identification of license plates, because any other step is based on it. If the segmentation fails, recognition phase will not be correct.To ensure proper segmentation, preliminary processing will have to be performed.
The histogram of pixel projection consists of finding the upper and lower limits, left and right of each character. We perform a horizontal projection to find the top and bottom positions of the characters. The value of a group of histograms is the sum of the white pixels along a particular line in the horizontal direction. When all the values ​​along all the lines in the horizontal direction are calculated, the horizontal projection histogram is obtained. The average value of the histogram is then used as a threshold to determine the upper and lower limits. The central area whose segment of the histogram is greater than the threshold is recorded as the area delimited by the upper and lower limits. Then in the same way we calculate the vertical projection histogram but by changing the rows by the columns of the image to have the two limits of each character (left and right).
Another approach to extract digits from licence plate is to use open/close morphologye to make some sorte of connected region then use connected component algorith to extract connected regions.
Step3 : Licence plate recognition
The recognition phase is the last step in the development of the automatic license plate reader system. Thus, it closes all the processes passing by the acquisition of the image, followed by the location of the plate until the segmentation. The recognition must make from the images characters obtained at the end of the segmentation phase. The learning model that will be used for this recognition must be able to read an image and to render the corresponding character.
In order to make the most of the data available for learning, we cut each character indivudually by resizing it in a square after applying the same image processing steps used before segmentation of the license plate. As a result, we obtained a set of of data composed of 11 classes and for each class we have 30–40 images of 28X28 pixel dimesion PNG formats; numbers from 0 to 9 and the word in Arabic (Tunisia).
Then, we made some researches based on scientific articles that compare the multilayer perceptron (MLP) and the classifier K nearest neighbors (KNN). And as a result we have found that: performance is increased if the number of hidden layer neurons is also increased when using the MLP classifier and if the nearest neighbor number is also increased when using the KNN. the ability to adjust the performance of the k-NN classifier is very limited here. But an adjustable number of hidden layers and adjustable MLP connection weights provides a greater opportunity to refine the decision regions. So as a result, we will choose the multilayer perceptron for this phase.
Github project repo link