# Smart Driving Alert System Using MT CNN

S. Kavitha, J. V. Anchitaalagammai, S. Murali, M. Soundarya

► To cite this version:

S. Kavitha, J. V. Anchitaalagammai, S. Murali, M. Soundarya. Smart Driving Alert System Using MT CNN. 6th International Conference on Computational Intelligence in Data Science (ICCIDS), Feb 2023, Chennai, India. pp.1-10, 10.1007/978-3-031-38296-3_1. hal-05466041

HAL Id: hal-05466041

https://inria.hal.science/hal-05466041v1

Submitted on 19 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Smart Driving Alert System Using MT CNN

S. Kavitha(✉), J. V. Anchitaalagammai, S. Murali, and M. Soundarya

Velammal College of Engineering and Technology, Viraganoor, Madurai 625009, India

kavitha3101@gmail.com

**Abstract.** Driver's drowsiness plays major cause for road accidents. The proposed system is designed to alert the drivers who are in drowsy state by analyzing human facial emotions. Human face acts as the main indicator for the behavioural state of the individual face. The mean of the system is to detect the Facial features conspire by utilizing the MT CNN Algorithm. Facial feature analysis is used in a different way to detect human behavioural state. Facial Landmark algorithm is used here to get the facial feature points. Here we are obtaining feature points like left and right eyes, left and right ear, mouth. The alert sound and hardware kit alerts the drowsy driver according to the behavioural state of the driver which was analyzed from the captured video.

**Keywords:** Drowsiness Detection · Face Detection · Face Landmark Detection · Multi Task Cascaded Convolution Neural Network · Open CV · Python

## 1 Introduction

Many people took a drive without taking sleep for continuous hours or being tired [1]. A report said that driver fatigue is the main cause for huge number of road accidents [2]. Driver's fatigue detection is a preventive technology which detects driver's drowsiness and prevents accidents [2]. Fatigue may caused due to lack of rest, pill etc., Physical and mental fatigue are small time failures of body's execution [3]. In some cases due to fatigue, driver can't able to pay attention and it followed by yawn, closure of eyes and head movements led to accidents. Facial feature is one of them which show that technology works very differently in today's world. Recognition of facial feature points is used to identify the basic state of features. The feature point we recognized here are left and right eyes, left and right ears, mouth, chin. To recognize these features, we are using Face landmark detection and MT CNN Algorithm and we have the different trained data set of the states of features. The face holds minimal space in every image. In our mechanism, we have a buzzer sound and focus alert corresponding to the state of driver's facial features. In recent times there are so many mechanisms available to alert the driver by detecting their heart beat rate and pressure but those devices are wearable. These features are very useful but sometimes it's quite irritating for the driver's. We can't assure that the driver wear the devices throughout the travel. So, our proposed system locate the facial landmarks and detects the feature points and alert the user according to
the driver's state. To provide the users with the best possible and effortless alert system, Facial Landmark detection based systems have been adopted. It provides more accurate and efficient results faster with less effort. It can develop human centered user interfaces that respond promptly to present, multimodal, human communication. The high performance and therefore the less time demand of the system create it appropriate to an feature detection systems and the rate accuracy is very high.

## 2 Problem Statement

Many road accidents caused due to drowsiness of the driver. The common objective is to create a model that will detect and indicate whether driver is feeling drowsy or not. The developed model takes image for every second and check for eye blinking, mouth opening and head movements of driver. If blink rate is high and simultaneously if eye is closed for certain amount of time or if the mouth is opened for continuous amount of time or head movement is found along with eye closure then it will alert the driver through buzzer.

## 3 Literature Review

The author Maninder Kahlon and Subramaniam Ganesan detects driver drowsiness based on algorithm. The position of driver's eye is determined through his iris visibility. System object has been created by Matlab using Viola Jones algorithm [4, 8] to detect the facial objects such as nose, mouth. Once image has been captured the noise rectangular eyes area were adjusted. Conversion of RGB to Gray scale and then to Binary image is done with suitable threshold value. The drowsy detection is performed based on conditions like pixels ratio, shape of eye and number of pixels in Column greater than the threshold value.

The author Jun-Juh Yan, Hang-Hong Kuo, Ying-Fan Lin and Teh-Lu Liao develops a model based on gray scale image processing and PERCLOS [5] to detect if the driver is fatigued. It is real time drowsiness detection system. It approximately finds the position of driver's face in gray scale images and to analyze the eye position it uses small template. Second, to establish a fatigue model, it uses data from previous step and PERCLOS.

The author RatnaKavya M, Ramya V and Ramya G Franklin proposed a system which uses Raspberry Pi and various sensors which sense Gad and Vibration Sensor are used to detect drowsiness. The car's location is sent using IOT modem which is fixed in the car [6].

The authors Murat Aslan and RahibAbiyev proposed real-time drowsiness detection based on computer vision. Viola Jones algorithm is implemented to detect the faces and to detect drowsiness of driver Convolutional Neural Network has been used [7].

The authors AvigyanSinha, Aneesh R P and Saradha K GopalusesHaar features to recognise landmarks. Haar features are extracted and they are used in various architectures for analysis of performance of fatigue detection. Modified LeNet is used for classification [9].

The authors Najila and Reshma V K developed driver's drowsiness detection system using hybrid deep learning approach. Facial and eye behaviours such as yawning and
eye-blink patterns are analyzed based on the live facial video focused on the driver's face. Facial landmarks present in the detected segments are pointed and the eye aspect ratios [10–12] are computed and detect the drowsiness based on the adaptive thresholding ratio.

The authors Melissa, Brian and Natalia developed a system to detect drowsiness detection using eye state. The system isolates a section of interest from face i.e. eyes and it analyse characters such as moment at which eyes are closing and distance of opening eyes. The alert system warns the driver if it detects drowsiness. Matlab is employed to carry out the process [13].

The author M. Ghazal, Y. Abu Haeyeh, A. Abed and S. Ghazal developed real-time embedded system with low cost for driver's drowsiness detection using CNN. The system uses real-time face detection algorithm to process the video signal. It also establish region of interest and minimize the number of computations. Camera attached on car dashboard provides a video signal. The faces which are detected are then given to CNN binary classifier which is optimized for fatigue recognition to detect whether it is fatigue moment or normal driving. When fatigue is detected the system warns the driver by sending alerts to the driver's smart phone and sends the alert to others to take preventive measures before accidents happen [14].

# 4 Proposed System

Let us see about our proposed system which detects driver drowsiness in this section. The complete flowchart of the developed system is shown in Fig. 1.

## Design Methodology

The system requires only a web camera and a decent computing system to work. Relatively good camera is required to capture the eyes because eye blinking rate of human is very fast. The software that the system is based on is Python, which is an open-source free programming language. It will be cheap and can be easily modified and upgraded at any time with libraries available.

## Feature Points Detection

1) **Video Capturing:** Open CV is the library of programming functions. It captures live video stream using web cam and allows us to make a video capture object. The web cam should be placed at front of the driver. Using CV2 library video are captured by calling cv2.VideoCapture() function.

2) **Pre-Processing:** The captured frame is passed to pre-processing stage. Gaussian blur is applied to the real image to avoid unwanted noise. This is a commonly used technique to provide an image with fine edges and noise free. Extract the region which contains face from blurred image, DLIB's library is used on that image. Histogram equalization is done to avoid unnecessary details and to adjust the contrast of the image.
3) **Face Detection:** The module processes the frontal face direction images so the faces are detected by using dlib based HOG and linear SVM face detector. They have good accuracy and are computationally efficient. Deep learning-based face detection is performed with the help of dlib by using a trained model from disk. Perform conversion of dlib to Open CV bounding boxes. Coordinates which exceed the bound of input are need to be trim. It will return a detector that is a function used to retrieve the face information. It detects facial features like eyes, mouth, nose. The objects which are detected is considered by the feature point location detection phase for further processing.

Features points of driver's face are obtained by using Facial Landmark Detection algorithm. Face landmark detector has pre-trained model and it predicts continuous numbers. The iBUG-300 W dataset is.

<figure><img src="image_3.png" /><figcaption>Fig. 1. Proposed system</figcaption></figure>

used for training the model. Set of images along with corresponding 68 face landmark points are present which is given in Fig. 2. The pre-trained facial landmark feature which was in dlib library of Open CV will plot a 68(x, y) points on the face of any given input image. The landmark marks eyes, nose, mouth and jaw with those 68 points as feature points like left and right eye, Mouth (Upper lips and lower lips), chin to detect driver's fatigue.

MT-CNN is a multitask framework that shows the relation between detection and alignment to speed up the performance. The model consists of three stages which are Deep Convolutional neural network to predict the facial feature points in fine grained manner. The overall architecture of MT-CNN was given in Fig. 3.

**Stage 1:** The first network is Proposed Network (P-Net). Here images in the image pyramid are obtained by full convolution neural network P-Net to get the candidate face frame and its regression. Calibrate the face frame using regression vectors which are estimated earlier. The P-Net weights and biases have been trained. The trained
<figure><img src="image_4.png" /><figcaption>Fig. 2. Face Landmark Points</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Fig. 3. MT-CNN architecture</figcaption></figure>

weights and biases of P-Net gives bounding box with relative accuracy. The network compares few boxes and P-Net is parsed to get bounding box confidence list. Lesser confidence level bounding box gets deleted. Then use Non-Maximum Suppression to merge candidates which have high overlapping rate (see Fig. 4).

<figure><img src="image_6.png" /><figcaption>Fig. 4. P-Net</figcaption></figure>
**Stage 2:** The second stage is Refined Network(R-Net). The candidate frames generated in previous stage is passed to the Refined Network. Bounding boxes coordinates with high accuracy are present here. The input image sent through the convolutional layer. Applies many different filters to it pixel block by pixel block to the input image. R-Net consist of complex Network which remove most of the wrong candidate face frame and then it uses the face frame regression vector to fine-tune the position of the candidate face frame. Then use NMS to reduce further frames (see Fig. 5).

<figure><img src="image_7.png" /><figcaption>Fig. 5. R-Net</figcaption></figure>

**Stage 3:** The third stage is Output Network (O-Net) which is same as the Refined Network but network of O-Net are more complex and it provides accurate results. It provides coordinates of bounding box, coordinates of landmark regions and confidence level of each box. Adjust the position of face frames with wrong candidate frame (see Fig. 6).

<figure><img src="image_8.png" /><figcaption>Fig. 6. O-Net</figcaption></figure>

The VGG-net classifier is used here to classify the frames based on their facial feature points. The alert is played according to the classes of features. The training set used to detect face part is WIDER_Face_train data set. LFW_5590 and NET_7876 data sets are trained for key point detection.
# 5 Result and Analysis

The system was built and implemented on Python 2.7.18. The testing was done on various images for real-time facial recognition. The system is tested in both low and high light conditions. Following are snapshots of working system.

If the eye blink rate is lesser than threshold and it gets closed then the driver is alerted through the buzzer sound which is shown in Fig. 7.

<figure><img src="image_9.png" /><figcaption>Fig. 7. Drowsy Body Posture</figcaption></figure>

If the yawn occurs then the driver is alerted through buzzer sound which is shown in Fig. 8.

<figure><img src="image_10.png" /><figcaption>Fig. 8. Yawn Detection</figcaption></figure>

If the yawn occurs and followed by eye closure then the driver is alerted through focus alert sound which is shown in Fig. 9.

If the head movement along with eye closure occurs then the driver is alerted through the focus alert sound which is shown in Fig. 10.
<figure><img src="image_11.png" /><figcaption>Fig. 9. Drowsy after Yawn</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>Fig. 10. Drowsy detection during head movement</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>Fig. 11: Python Shell during Drowsy detection</figcaption></figure>

Once when the alert is played and driver gets wake up the system gets reset and
repeat the process by checking the facial features (see Figs. 11, 12 and 13).
<figure><img src="image_14.png" /><figcaption>Fig. 12. PyShell after Drowsy detection</figcaption></figure>

<figure><img src="image_15.png" /><figcaption>Fig. 13. Model Accuracy</figcaption></figure>

## 6 Conclusion

The model is proposed for alerting the drowsy driver by classifying the states of features. The model is implemented by using Deep Learning algorithm. MT CNN is employed to classify and to identify the states of features. In this proposed system the states are predicted by the user's facial image. The proposed model is justified by a properly made dataset and deep learning algorithm. The accuracy for facial feature state classification is more appropriate than existing methods. This system would be used to design a mechanism that would be helpful in providing the better alert mechanism to the drivers.

## 7 Future Work

The aim is to develop an alerting system which can detect drowsiness using a webcam and alert the driver in a simulation manner. In future, this system will be extended by incorporating it with a hardware device. The hardware device is attached with a water sprayer kit which is used to alert the driver in efficient way. Water is sprayed according to the conditions and states of detected facial feature points. Different algorithms will be used in the developed model in order to provide better accuracy.
References

1. Dua, H.K., Goel, S., Sharma, V.: Drowsiness detection and alert system In: International Conference on Advances in Computing, Communication Control and Networking (ICACCCN2018) (2018)

2. Bhargava, R., Panchal, R., Verma, P., Sharma, S.: Drowsiness detection while driving using eye tracking. Int. Res. J. Eng. Technol. (IRJET) **07**, 6485–6490 (2020)

3. Kulkarni, S.A., Sathish, Dr., Kumar, S.: Driver drowsiness monitoring system. Int. Res. J. Eng. Technol. (IRJET) 06(June) (2019)

4. Kahlon, M., Ganesan, S.: Driver drowsiness detection system based on binary eyes image data. In: IEEE (2018)

5. Valsan A.V., Mathai, PP.: Monitoring driver's drowsiness state at night based on computer vision. In: International Conference on Computing, Communication, and Intelligent Systems (ICCCIS) (2021)

6. Ratna Kaavya, M., Ramya, V., Franklin, R.G.: Alert system for driver's drowsiness using image processing. In: International Conference on Vision towards Emerging Trends in Communication and Networking (ViTECoN) (2019)

7. Arslan, M., Abiyev, R.: Vision-based drowsiness detection system using convolutional neural networks. In: 2nd International Conference on Electrical, Communication and Computer Engineering (ICECCE), 12–13 June (2020)

8. Fatima, B., Shahid, A.R., Ziauddin, S., Safi, A.A., Ramzan, H.: Driver fatigue detection using viola jones and principal component analysis. Appl. Artif. Intell. **34**(6), 456–483 (2020)

9. Sinha, A., Aneesh, R.P., Gopal, SK.: drowsiness detection system using deep learning. In: Seventh International conference on Bio Signals, Images, and Instrumentation (ICBSII) (2021)

10. Najla1, Reshma, V.K.: Drowsy driving detection via hybrid deep learning. In: International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056 vol. 07 May (2020)

11. Sathasivam, S., Mahamad, A.K., Saon, S., Sidek, A., Som, M.M., Ameen, H.A.: Drowsiness detection system using eye aspect ratio technique. In: IEEE Student Conference on Research and Development (SCOReD) (2020)

12. Maior, C.B.S., Moura, M.J.D.C., Santana, J.M.M., Lins, I.D.: Rea l- time classification for autonomous drowsiness detection using eye aspect ratio. Expert Syst. Appl. **158**, 113505 (2020)

13. Yauri-Machaca, M., Meneses-Claudio, B., Vargas-Cuentas, N.: Design of a vehicle driver drowsiness detection system through image processing using matlab. In: IEEE (2018)

14. Ghazal, M., Abu Haeyeh, Y., Abed, A., Ghazal, S.: Embedded fatigue detection using convolutional neural networks. In: 6th International Conference on Future Internet of Things and Cloud Workshops (FiCloudW) (2018)