[image] [image] Ressource protégée - Inria - Institut national de recherche en sciences et technologies du numérique [Accéder directement au contenu](#)

[image] [image]

* [Documentation](#)

* FR
  Français (FR)

* Anglais (EN)
  Se connecter

* [Portail HAL Inria](#)

* Recherche
  Loading...

## Recherche avancée

Information de documents

Titres

○ Titres

○ Sous-titre

○ Titre de l'ouvrage

○ Titre du volume (Série)

○ Champ de recherche par défaut (multicritères) + texte intégral des PDF

○ Résumé

○ Texte intégral indexé des documents PDF

○ Mots-clés

○ Type de document

○ Sous-type de document

○ Tous les identifiants du document

○ Identifiant HAL du dépôt

○ Langue du document (texte)

○ Pays (Texte)

○ Ville

○ À paraître (true ou false)

Ajouter
# Allowance of Driving Based on Drowsiness Detection Using Audio and Video Processing

S. Sathesh, S. Maheswaran, P. Mohanavenkatesan, M. Mohammed Azarudeen, K. Sowitha, S. Subash

<figure><img src="image_1.png" /></figure>

## To cite this version:

S. Sathesh, S. Maheswaran, P. Mohanavenkatesan, M. Mohammed Azarudeen, K. Sowitha, et al. "Allowance of Driving Based on Drowsiness Detection Using Audio and Video Processing." 5th International Conference on Computational Intelligence in Data Science (ICCIDS), Mar 2022, Virtual, India. pp. 235-250, 10.1007/978-3-031-16364-7_18. hal-04381276

HAL Id: hal-04381276

https://inria.hal.science/hal-04381276v1

Submitted on 9 Jan 2024

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>
## Auteur

### Auteur (multicritères)

○ Auteur (multicritères)

○ Auteur : Nom complet

○ Auteur : Nom de famille

○ Auteur : Prénom

○ Auteur : Complément de nom, deuxième prénom

○ Auteur : Organisme payeur

○ Auteur : IdHal (chaîne de caractères)

○ Auteur : Fonction

○ Auteur : personID (entier)

○ Auteur : Identifiant de l'organisme payeur

○ Auteur : Identifiant de la structure

○ Directeur de thèse

○ Éditeur

○ Éditeur scientifique

○ Éditeur de la série

## Ajouter

### Structure

#### Structure (multicritères)

○ Structure (multicritères)

○ Structure : Acronyme

○ Structure : Nom

○ Structure : Code

○ Structure : Pays

○ Structure : Type

○ Structure : État dans le référentiel

○ Structure : Identifiant HAL de la structure

○ Structure/équipe de recherche : Pays

○ Structure/regroupement d'équipes : Pays

○ Structure/laboratoire : Pays

○ Structure/regroupement de laboratoires : Pays

○ Structure/institution : Pays

○ Structure/regroupement d'institutions : Pays

## Ajouter
<figure><img src="image_3.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
Autres

Champ de recherche par défaut (multicritères)

○ <u>Champ de recherche par défaut (multicritères)</u>

○

○ <u>Revues (multicritères)</u>

○ <u>Revue : Éditeur</u>

○ <u>Revue : Titre abrégé</u>

○ <u>Revue : Titre</u>

○ <u>Revue : Identifiant interne</u>

○ <u>Revue : Date de début de publication</u>

○ <u>Revue : ISSN électronique</u>

○ <u>Revue : ISSN</u>

○ <u>Revue : Éditeur</u>

○ <u>Revue : Couleur dans SHERPA/RoMEO</u>

○ <u>Revue : État dans le référentiel</u>

○

○ <u>Colloque (multicritères)</u>

○ <u>Colloque : Titre</u>

○ <u>Colloque : Organisateur</u>

○ <u>Colloque : date de début (Année)</u>

○ <u>Colloque : date de fin (Année)</u>

○ <u>Conférence invitée (oui/non)</u>

○

○ <u>Projets ANR (multicritères)</u>

○ <u>Projet ANR : Acronyme</u>

○ <u>Projet ANR : Acronyme du programme</u>

○ <u>Projet ANR : Code décision (référence)</u>

○ <u>Projet ANR : Nom</u>

○ <u>Projet ANR : Identifiant interne</u>

○ <u>Projet ANR : État dans le référentiel</u>

○

○ <u>Projets européens (multicritères)</u>

○ <u>Projet européen : Acronyme</u>

○ <u>Projet européen : Identifiant de l'appel à projet</u>

○ <u>Projet européen : Référence</u>

○ <u>Projet européen : Nom</u>

○ <u>Projet européen : Date de fin</u>

○ <u>Projet européen : Financement</u>

○ <u>Projet européen : Date de début</u>
# Allowance of Driving Based on Drowsiness Detection using Audio and Video Processing

Sathesh S¹[0000-0002-4571-0247], Maheswaran S²[0000-0002-1836-1117], Mohanavenkatesan P³, Mohammed Azarudeen M⁴, Sowmitha K⁵ and Subash S⁶

¹ Assistant professor, ² Associate professor, ³,⁴,⁵,⁶ UG Scholar, Department of Electronics and Communication Engineering, Kongu Engineering College, Perundurai, Erode 638060, Tamil Nadu, India.

sathesh808@gmail.com, mmaheswaraneie@gmail.com, mohanpalani2001@gmail.com, mohammedazarudeenm.18ece@kongu.edu, sowmithak7@gmail.com, subramanisubash52@gmail.com

**Abstract.** In the last few years, more accidents are happened mainly due to the drowsiness of the driver. Various accident prevention technologies are developed but still accidents are happening. This is due to that, the technologies which are available in present are all detecting the drowsiness of the driver at the time accidents. So, there is a possibility for happening of accidents. If the drowsiness of the driver, can be predicted before driving, it will be very useful to prevent the accidents. In this method, there is a solution to detect the drowsiness of the driver, before the driver starts to drive. There is system in this method from which the driver got approval for driving. The system has two level verification process for the drivers to detect the drowsiness. The First level of verification process is cap-tcha process using python libraries or audio listening process using gTTS library. The second level verification process is based on detecting the facial expression of the drivers using the haar cascade classifier with OpenCV library in python. This process has the accuracy level of 95% in pre-driving drowsiness detection. The above levels are the two levels which are used to detect the drowsiness of the drivers before start to drive.

**Keywords:** Drowsiness, Accident, gTTS, Haar cascade classifier and OpenCV

## 1 INTRODUCTION

Drowsiness is the state where individual want to rest. Rest is a neurobiological need with unsurprising examples of languor and attentiveness. Security and insurance are assuming significant part in driving. Sluggishness results from the rest part of the circadian pattern of rest and alertness, limitation of rest, or potentially interference or fracture of rest. The deficiency of one night's rest can prompt outrageous momentary tiredness, while constantly confining rest by 1 or 2 hours a night can prompted persistent languor. Dozing is the best way of lessening drowsiness. It is the state where individual wants to rest [1-3]. It has two unequivocal implications, alluding both to the state prior
○ Projet européen : État dans le référentiel

○ Projet européen : Identifiant interne

○

○ Date de production : année

○ Date de mise en ligne : année

○ Date de publication : année

○ Date d'écriture : année

○ Date de modification du dépôt : année

○ Date de dépôt : année

○ Date de publication électronique : année

○

○ Collection HAL (multicritères)

○ Collection HAL : catégorie

○ Collection HAL : Code

○ Collection HAL : Nom

○ Collection HAL : Identifiant interne

○

○ Identifiant interne du contributeur/déposant

○ Nom complet du contributeur/déposant

○

○ Domaines

○ Domaine primaire

○ Domaine racine

○ Sous-domaine niveau 1

○ Sous-domaine niveau 2

○ Sous-domaine niveau 3

○

○ Statut du document

○ Version du document

○ Type de dépôt

○ Type de document

○ ISBN

○ Numéro - référence

○ Identifiant DOI

○

○ Classification

○ Audience

○ Vulgarisation

○ Comité de lecture - texte (oui ou non)

○ Actes de colloque
nodding off and ongoing condition alluding to being in that state free of the day-by-day
mood. Individual can encounter tiredness if they had an adequate weakness and this can
prompt street mishaps. The following Fig. 1 represents the percentage of accidents hap-
pened due to the drowsiness of the drivers.

<figure><img src="image_4.png" /><figcaption>Fig. 1. Accidents percentage due to driver sleepy</figcaption></figure>

The purpose of this project is to develop the simulation of drowsiness detection system.
The focus of the project is to design a system that will approve the transports key for
the drivers. By detecting the drowsiness of the driver before driving the accidents can
be prevented. So, by two monitoring the state of the eyes, it is believed can detect the
early symptom of driver's getting drowsiness, and to avoid the road accidents. The
process of detecting the drowsiness between drivers is to analyze the opening and closet
of the eyes. This process is based on the computer vision or OpenCV to detect image
or video processing [4, 5]. In Facial recognition, this project is focused on the localiza-
tion of the eyes, which involves both eyes and face by applying the existed image pro-
cessing algorithm. The detection of the drowsiness will be determined once the position
of the eyes located.

1.1 Facts About Drowsy Driving

Drowsy driving is responsible for 33% of all fatal accidents. In the previous year, over 168 million people drove their vehicle while drowsy [6]. According to National Sleep Foundation's (NSF) 2015 poll, about thirty seven percent people or more than hundred million people have fallen asleep at the wheel. There are around eleven million drivers who admit to dozing off or being too fatigued to drive, and they admit to having an accident or near-accident.

Mostly the accidents happened due to the drowsiness driving is caused by people
who have their age between 18 and 29. Each year 100,000 accidents are the direct result
○ Référence interne

○ Financement

○ Collaborations

Ajouter

Lancer la recherche

Recherche experte (SolR)

Recherche experte (SolR)

Lancer la recherche
Vers la recherche avancée

• Déposer

Portail INRIA.HAL.SCIENCE
Les publications, logiciels... des scientifiques Inria

• Accueil

• Publications Inria

○ Consulter les derniers dépôts

○ Par équipe de recherche

○ Par auteur

○ Rapports de recherche et techniques

○ Livres blancs

○ Rapports d'activité des équipes

○ Statistiques

• Boite à Outils

○ X2Hal : importer par lot

○ Logiciels : créer le Codemeta.json

○ Créer mon IDHal

○ Créer mon CVHal

○ Haltools : créer ma page de publications

○ Haltools : extraire des données (réservé Inria)

○ Les bonnes pratiques

■ Comment bien s'affilier ?
of driver's fatigue which is estimated by National Highway Traffic Safety Administra-
tion. According to monitory losses in each year is 12.5 billion dollars due to vehicular
crashes results in an estimated counts of 1,550 deaths, 71,000 injuries.

According to other European nations, Australia, Finland and England, drowsiness of
driving represents 10 to 30 percent of all accidents. According to the National Sleep
Foundation's sleep in America poll, more than 40% of drivers become irritated, impat-
tient, and 12% of drivers tend to drive faster when they are drowsy. Most of the vehicle
crashes occurred between at time of 4 to 6 a.m., midnight 2 a.m. and 2 to 4 p.m.

Nearly 23 percent of people agree that crashes of their vehicle is occurred between the above timings. Most of the people in the corporations saw the statistics of the accidents due to drowsiness in the daily newspapers. In United States, driving with drowsiness is the serious problem. The National Highway Safety Traffic Administration has found that determining a precise number of drowsy driving accidents and hospital reports to determine the prevalence of drowsy-driving crashes.

1.2 Causes of Drowsy Driving

There are many causes for drowsiness while driving [6]. The main causes are as follows,

* Lack of adequate sleep.

* The presence of untreated or unrecognized sleep disorders.

* Driving long distances alone.

* Driving at times of the day when you would normally be sleeping.

* Taking medicines, sometimes have drowsiness as a side effect.

The following Fig. 2 represents the accident caused by the drowsiness of the driver.

<figure><img src="image_5.png" /><figcaption>Fig. 2. Accident due to the drowsiness of driver</figcaption></figure>
■ Quelle version puis-je déposer ?

■ Choisir le bon « type de document »

■ Mettre en valeur ses publications

○ Les tutos IES Inria

■ Lier les notices dans HAL

■ Umbrhal Inria pour l'accessibilité

■ Personnaliser son CVHAL

■ Créer et Personnaliser sa collection

• Documentation

○ Aide en ligne HAL

○ Aide en ligne Haltools

○ Aide en ligne X2hal

○ Tutoriels video HAL

○ Documentation API HAL

○ Ajouter des vignettes

○ Identifiant auteur IdHAL et CV

○ Compléter son dossier ORCID avec ses dépôts HAL

○ Inria et la Science Ouverte

X

X

X

Loading...

X

Le document que vous demandez est en access restreint
jusqu'au 2025-01-01 !

Revenir a la notice

Retourner à l'accueil

Contact

•

Ressources
## 2 EXISTING METHOD

The drowsiness is detected using the facial expressions like tiredness face, yawning, eye blinking measurements, etc. In machine learning, various algorithms are used for the detecting the facial expressions. In [1] the driver's fatigue is monitored by his work-shift and climatic conditions the studies shows that the driver fatigue during driving is more in Heavy traffic route (HTR) compared to monotonous route(MR). In [2] they proposed a method of multi model of detecting the drowsiness and distraction of the driver by using various types of sensors and it is separated into different modules which it is independent to each other. It can be enabled or disabled at any time to analyze the driver's fatigue and distraction. In [3] the analyzing of the drowsiness detection system is tested with the four variables namely kernel size, threshold value, lighting condition (morning, noon, afternoon, and night), and eye's characteristic (eyeglasses or not).

<figure><img src="image_6.png" /><figcaption>**Fig. 3.** Workflow of the existing method.</figcaption></figure>
* Documentation

* FAQ

* API

* OAI-PMH

* AuréHAL

## Informations

* À propos

* Données personnelles

* Mentions légales

* Accessibilité

* Conformité RGAA

## Questions juridiques

* Je publie, quels sont mes droits ?

* Loi pour une République numérique

* Stratégie de non-cession des droits

* SHERPA/RoMEO

## Portails

* Portails institutionnels

* HAL

* HAL SHS

* HAL Thèses

* MédiHAL

## CCSD

* CCSD

* Episciences

* Sciencesconf
In [4,5] the approach of detecting the drowsiness of the driver is detected by using Haar cascade and Open CV and the alertness is detected by how much time the eyes are in closed state.

The above process is explained in following Fig. 3 which represents the workflow of the existing method for the detection of drowsiness of the driver.

In [6-9] the windows of the transport are automatically opened or broke to let the air in and help the jammed one to breathe the air. To intercept the sleepiness of the driver in real time, a behavioral based approach is used. In real-time, the drowsiness detection is carried out by way of taking pictures on video. And the studies proposed the surveillance system to monitor the drowsiness which is developed by mobile application on the android operating system.

In [10-15] the studies shows that the drowsiness is detected by using smartphones by capturing live photos and videos and analyzing the drowsiness and make some alarm sounds to alert the driver.In [16, 17] the face spotting is executed with the aid of the local binary sample set of pattern algorithm which is in cascade classifier of OpenCV library. For object detection, Histogram of Oriented Gradients (HOG) that is used in laptop imaginative and prescient. In [18, 19] the picture is split into cells for the manner of encoding of capabilities. Then, instead of requiring a Raspberry Pi, this method is employed as a output which is based on software. This method has three parts of section as follows, Face, Eye and Drowsiness detection.

In [20], there is a slider under the seat position which is away from the crash area and doors are unlocked automatically.In [17, 21] this method's workflow is that the system monitors continuously by camera and it records the inattention of the diver by his fatigue. This system captures the video with the framework at 720p and 30 frames per second.To detect the driver's tiredness of a brief audible alert by the time rest. Until the vehicle is driven, the sluggishness identification cycle will continue. The sound will continue to buzz until the motorist is completely alert. This is affirmed by the framework which continues to take contribution which is completely alert processes which are all available for detecting the drowsiness of the drivers only when the driver have the correct intensity of the lightening for the face and it is also found the drowsiness while in the time of driving.

In [22] the methods used to find the drowsiness are facial expressions, local binary pattern, and EEG low channel peripheral signals. It cannot find the drowsiness of the driver before driving. In [23, 24] the unsupervised machine learning technique is used to find the accident-prone areas like pothole, bumps, sharp turns and normal. Arduino UNO and sensors are used for the detections. GPS is used to coordinates marking of automobiles. If the automobile is at distance of 400m from a danger spot, an alert is passed to the driver to slow down.

## 2.1 Limitations of Existing Method

The main drawback of existing method is that, all the systems are designed to detect the drowsiness only at the time of driving. So, there may be possibilities for the occurring of accidents. This system doesn't work, if there is not a correct lightening effect in the face.
In public transports, if this system detects the drowsiness of the driver, the alarm starts sounds. So, which also disturbs the passengers in those transports.

Sometimes the sound of the alarm may not be heard by the driver. It may also lead to happening of the accidents.

## 3 PROPOSED METHOD

In this proposed system, the drowsiness of the drivers should be identified before the driver starts to drive. So, the major limitation will be overcome by this system. The main objective of this system is to give approval for taking the keys of the transports. If the system finds the drowsiness in face, then the driver should not get approval to take the keys of the transport.

<figure><img src="image_7.png" /><figcaption>Fig. 4. Outline method of proposed system</figcaption></figure>

The above Fig. 4 represents the outline method of this proposed method for allowance of driving for drivers. In this proposed method, there are two levels of verification process should be done.
<figure><img src="image_8.png" /><figcaption>Fig. 5. Block diagram of the proposed method.</figcaption></figure>

The above Fig. 5 represents the block of this proposed system in a step-by-step manner.

## 3.1 First Level Verification Process

The first level verification process is that, captcha entering process for literate people or audio listening process for illiterate people. The captcha process is nothing but, a captcha is displayed on the screen. The driver should enter the captcha correctly which is displayed on the screen. If the driver enters the captcha correctly, then the driver will move on to the second level checking process. The audio listening process is nothing but, an audio with the captcha will be played. The driver should listen the captcha audio correctly and then, the driver should repeat the audio correctly, which is listened by the driver.
<figure><img src="image_9.png" /><figcaption>Fig. 6. Workflow of the proposed method</figcaption></figure>

The above Fig. 6 represents the workflow of the proposed method. If the repeated captcha audio of the driver is correctly matched with system played captcha audio, then the driver will move on to the second level verification process. The packages used for the first level verification process are captcha, gTTS, Speech recognition. The captcha package is used to generate captcha randomly. The gTTS package is used for the creation of captcha audio file [25, 26]. It is also used for the conversion of text to speech process. The speech recognition package is used to recognize the speech by the drivers.

## 3.2 Second Level Verification Process

The second level verification process is that, the camera should capture video of the face of driver for some time before start to drive. After the capturing of the video, the haar cascade classifier in OpenCV python library will detect the face and eyes of the driver in the recorded video. The video should divide into frames. Then each frame is
given as input. Haar cascade classifier is used, because of its computation speed of the features of the haar. This is because of the use of integral image which will also be called as summed area tables. Due to the presence of the adaboosting algorithm, this will be very efficient for the feature selection process.

To detect the face in the image, convert the image into grayscale. Because the OpenCV algorithm for detection of objects takes the input as gray images. Then perform detection using detect Multiscale function, which returns an array of detection with x, y co-ordinates, height, and width of the boundary box of the object. For the detection of eyes, the setting of cascade classifier in left eye and right eye. Then, detect using detect multiscale function. This can be done by extracting the boundary box of the eye. After that, the eye image is taken from frame. The left eye and right eye image data is given to the CNN classifier, which will predict the status of both the eyes. First the colour image is converted into grayscale image and resize into 24 x 24-pixel images. Then, normalization is done for better convergence. After that loading of model is done which will predict the status of both the eyes.

The Haar cascade classifier process involves four steps. Such as Haar features, Integral image, adaboosting, cascading. The following Fig. 6 represents the process of haar cascade classifier for both eyes and face. The second block haar features extraction involves three types of features. Such as, edge features, line features, four rectangle features. The feature Extraction process extracts 1,60,000+ features in the 24 x 24 window. So right now, it is a problem to calculate the huge set of features, i.e., 1,60,000+ features for every 24 x 24 window. This thing looks practically difficult. So, to overcome these difficulties, select only the features with useful information. This can be done by the adaboosting algorithm i.e., it eliminates the redundant features and features which has no useful information. So, using this algorithm, many of the features can be reduced. In the extracted feature's part, it is difficult to sum of all the black region and sum of all white region in each feature. So, there is a trick to solve this problem called integral image. The Fig. 7 represents the processes takes place in haar cascade classifiers. Using this classifier object should be detected or not.

The Fig. 8 represents the process of integral image which shows that how to calculate the integral image value of each pixel.

Sum of the all the pixels in $Z = 1 + 4 - (2 + 3)$

$$Z = W + (W + X + Y + Z) - (W + X + W + Y) \Rightarrow Z \quad (1)$$

As shown in equation (1) is the calculation of the integral image of each pixel of the image. The adaboosting technique selects certain number of features from 1,60,000 features and gives weight to the features. A linear combination of all these features is used to decide it is a face or not. It combines all the weak classifiers and forms a strong classifier. Generally, 2500 features are used to form a strong classifier. The cascading process is nothing but, combining a small group of features i.e., out of 2500 features ten features are kept in one classifier, like that all features are cascaded.
<figure><img src="image_10.png" /><figcaption>Fig. 7. Process of Haar Cascade Classifier</figcaption></figure>
<figure><img src="image_11.png" /><figcaption>Fig. 8. Process of Integral image calculation</figcaption></figure>

The following equation (2) represents the adaboost cascading process of the haar cascade classifier.

$$F(x) = \alpha_1 f_1(x) + \alpha_2 f_2(x) + \alpha_3 f_3(x) + \dots \tag{2}$$

$F(x)$ represents strong classifier and $f(x)$ represents weak classifiers. Now it is easy to detect the face at each classifier. Then, the score is essentially a worth we will use to decide how long the individual has shut his eyes. Thus, if the two eyes are shut, we will continue to expand score and when eyes are open, we decline the score. The maximum score value at which the drowsiness starts is called threshold value. A threshold value is fixed i.e., the closed score of the eyes become more than 15, which indicates that the person's eyes are closed for a long period of time. When the score crosses the threshold value, an alarm sounds and the driver should not allow to drive. Otherwise, drivers will be provided with key of the vehicle to drive.

## 4 RESULTS AND DISCUSSION

The proposed method should detect drowsiness before driving using two levels of verification process. The following figures represent the first level verification process as follows. The following Fig. 9 represents the sample output one with correct captcha for the literate drivers i.e., the captcha verification process for the literate people. As shown in the Fig. 9, if the driver enters the captcha correctly, he will allow for the next level checking process. In this process, the captcha package in python is used for the creation of the captcha and displaying the captcha. There are two functions for the creation of captcha and the creation of the captcha audio file. Also, two functions for the random text captcha for image captcha and random text captcha for the audio captcha. Some of the drivers of public transports are illiterate. For those people, who may not know how to use the computer, the following audio listening process will be done. An audio with
random text value is created and played by the gTTS (Google Text-to-Speech) library
in python.

<figure><img src="image_12.png" /><figcaption>Fig. 9. Sample output with correct captcha for the literate drivers.</figcaption></figure>

After hearing that audio, the driver should repeat the same audio correctly, which was
played by the system. The Fig. 10 represents the sample output with incorrect captcha
for the literate drivers i.e., the captcha verification process for the literate people.

As shown in the Fig. 10, if the driver enters the captcha incorrectly, he will not allow
for the next level checking process. As said as above in this process, the captcha pack-
age in python is used for the creation of the captcha and displaying the captcha.

There are two functions for the creation of captcha and the creation of the captcha
audio file. Also, two functions for therandom text captcha for image captcha and ran-
dom text captcha for the audio captcha. Some of the drivers of public transports are
illiterate. For those people, who may not know how to use the computer, the following
audio listening process will be done.

An audio with random text value is created and played by the gTTS library in python.

<figure><img src="image_13.png" /><figcaption>Fig. 10. Sample output with incorrect captcha for the literate drivers.</figcaption></figure>
```
if literate enter 'y' else enter 'N' : n
7 3 5 4 8
73548
Recognizing...
rec_text--> 73548
```

LEVEL 1 CHECKING IS COMPLETED

MOVE ON TO LEVEL 2 CHECKING

<caption>Fig. 11. Correctly matched output for illiterate drivers.</caption>

The above Fig. 11 represents the correctly matched output for the illiterate drivers. As shown in the above picture the audio captcha which is repeated by the driver is recognizing with the shown message recognizing. An audio with random text value is created and played by the gTTS library in python. The driver should repeat the same voice generated by the system. If the recognized voice of the driver is not matched with the system generated voice of the captcha, then, its through the message as invalid captcha and the driver of the vehicle should not be allowed to drive the vehicle.

```
if literate enter 'y' else enter 'N' : n
8 6 6 0 8
86608
Recognizing...
rec_text--> 86608
```

INVALID CAPTCHA

<caption>Fig. 12. Incorrectly matched output for illiterate drivers.</caption>

The above Fig. 12 represents the incorrectly matched output for the illiterate drivers. As shown in the above picture the audio captcha which is repeated by the driver is recognizing with the shown message recognizing. Then the recognized text is displayed and check whether both audio is matched or not. If the audio got matched, the driver allows to start the second level checking. If the driver's audio captcha is not matched with the system audio captcha, then the driver will not allow to drive the vehicle without checking with the second level. The haar cascade classifier detects the face of the driver by extract the features. The status of eyes is calculated by the CNN loaded model.

The Fig. 13 represents the drowsiness of the eyes with closed score status given by the CNN model and haar cascade classifier. The haar cascade classifier detects the face of the driver by extract the features. The status of eyes is calculated by the CNN loaded model.
model. The algorithmic representation of the second level verification is given as follows

i) A video is captured using the cv2 videocapture( ) method in openCV.

ii) The captured video is converted to frames.

iii) Convert the frames into grayscale using cv2.cvtcolor(frame, cv2.color_BGR2GRAY).

iv) Prediction of both right eye and left eye, which is to be cleary open. In which the resizing and the reshaping process takes place.

v) After the prediction, breaking of the loop will be done.

vi) If both predicted values of the eyes are strictly zero, then closed score value is printed on the frame, else open score value is printed on the frame.

vii) The closed score is above the value of 15, then the alarm sounds. The driver is not allowed to drive the vehicle, otherwise driver will be allowed for the driving of the vehicle.

<figure><img src="image_14.png" /><figcaption>Fig. 13. Detection of drowsiness eyes with closed score.</figcaption></figure>

In the Fig. 13 the drowsiness detected with the closed score of 25. So, the alarm sounds and the driver will not allow to drive i.e., the keys of the transport will not be given. Then the following Fig. 14 represents that, the extracted video does not have the drowsiness in the face of the diver, then second level verification is completed successfully i.e., the keys will be provided to the drivers.

<figure><img src="image_15.png" /><figcaption>Fig. 14. Output of second level verification for video without drowsiness.</figcaption></figure>
## 4.1 Efficiency of the system

The feedback of the fifty drivers is collected. The feedback of the driver before this system is that they should feel unsafe due to their drowsiness. But, after the arrival of this system, they feel very safe to drive at day and night times because this system finds the drowsiness of the driver, before the driving starts. So, it is very safe to drive the vehicles.

<figure><img src="image_16.png" /><figcaption>Fig. 15. Accuracy of the proposed system</figcaption></figure>

Fig. 15 Shows the percentage of accuracy of the system. Here, D1 to D10 refers drivers, P and E refers proposed and existing system.

## 5 CONCLUSION

The safety and protection are plays important role in driving. Sluggishness results from the rest part of the circadian pattern of rest and alertness, limitation of rest, or potentially interference or discontinuity of rest. Sleeping is the most effective way to reduce sleepiness. To reduce the accidents, drowsiness should detect before driving. So, this system prevents the accidents by not allowing the drowsy drivers. Using this system, there is some possibility of reduction in some percentage of accidents, which will be better. The above system successfully detected the drowsiness by using the two levels of verification process. First level verification process successfully verified and the second level of verification using facial features is also successfully verified. Drowsiness results from the rest part of the circadian pattern of rest and alertness, limitation of rest, as well as interference or discontinuity of rest.
References

1. Makowiec-Dąbrowska, T., Gadzicka, E., Siedlecka, J., Szyjkowska, A., Viebig, P., Kozak, P., Bortkiewicz, A.: Climate conditions and work-related fatigue among professional drivers. International journal of biometeorology. 63(2), pp. 121-8 (2019).

2. Craye, C., Rashwan, A., Kamel, M.S., Karray, F.: A multi-modal driver fatigue and distract ion assessment system. International Journal of Intelligent Transportation Systems Re search. 14(3), pp. 173-94 (2016).

3. Purnamasari, P.D., Kriswoyo, A., Ratna, A.A.P., Sudiana, D.: Eye Based Drowsiness Detection System for Driver. Journal of Electrical Engineering & Technology. 17(1), pp. 697-705 (2022).

4. Ravi Teja, P., Anjana Gowri, G., Preethi Lalithya, G., Ajay, R., Anuradha, T., Kumar, P.: Driver Drowsiness Detection Using Convolution Neural Networks. Smart Computing Tech niques and Applications: Springer. p. 617-(2021)

5. Sathesh, S., Pradheep, V.A., Maheswaran, S., Premkumar, P., Gokul, N.S., Sriram, P.: Com puter Vision based Real Time Tracking System to Identify Overtaking Vehicles for Safety Precaution Using Single Board Computer. Journal of Advanced Research in Dynamical and Control Systems. 12(07-Special Issue), pp. 1551-61 (2020).

6. Drowsy Driving, https://www.sleepfoundation.org/drowsy-driving, last accessed 2021/07/15

7. Weng, C.-H., Lai, Y.-H., Lai, S.-H.: Driver drowsiness detection via a hierarchical temporal deep belief network. Asian Conference on Computer Vision; 2016: Springer.

8. Adochiei, I.-R., Ştirbu, O.-I., Adochiei, N.-I., Pericle-Gabriel, M., Larco, C.-M., Mustata, S.-M., Costin, D.: Drivers' Drowsiness Detection and Warning Systems for Critical Infra-structures. 2020 International Conference on e-Health and Bioengineering (EHB); 2020: IEEE.

9. Awasthi, A., Nand, P., Verma, M., Astya, R.: Drowsiness detection using behavioral-cen- tered technique-A Review. 2021 11th International Conference on Cloud Computing, Data Science & Engineering (Confluence); 2021: IEEE.

10. Fazeen, M., Gozick, B., Dantu, R., Bhukhiya, M., González, M.C.: Safe driving using mo bile phones. IEEE Transactions on Intelligent Transportation Systems. 13(3), pp. 1462-8 (2012).

11. Galarza, E.E., Egas, F.D., Silva, F.M., Velasco, P.M., Galarza, E.D.: Real time driver drowsi-ness detection based on driver's face image behavior using a system of human computer interaction implemented in a smartphone. International Conference on Information Tech nology & Systems; 2018: Springer.

12. Chaudhary, V., Dalwai, Z., Kulkarni, V.: Intelligent Distraction and Drowsiness Detection System for Automobiles. 2021 International Conference on Intelligent Technologies (CONIT); 2021: IEEE.

13. Chhabra, R., Verma, S., Krishna, C.R.: Detecting aggressive driving behavior using mobile smartphone. Proceedings of 2nd International Conference on Communication, Computing and Networking; 2019: Springer.

14. Fino, E., Mazzetti, M.: Monitoring healthy and disturbed sleep through smartphone applica tions: a review of experimental evidence. Sleep and Breathing. 23(1), pp. 13-24 (2019).

15. Lemola, S., Perkinson-Gloor, N., Brand, S., Dewald-Kaufmann, J.F., Grob, A.: Adolescents' electronic media use at night, sleep disturbance, and depressive symptoms in the smartphone age. Journal of youth and adolescence. 44(2), pp. 405-18 (2015).
16. Khan, S., Akram, A., Usman, N.: Real Time Automatic Attendance System for Face Recognition Using Face API and OpenCV. Wireless Personal Communications. 113(1), pp. 469-80 (2020).

17. Bruce, B.R., Aitken, J.M., Petke, J.: Deep parameter optimisation for face detection using the Viola-Jones algorithm in OpenCV. International Symposium on Search Based Software Engineering; 2016: Springer.

18. Lashkov, I., Kashevnik, A., Shilov, N., Parfenov, V., Shabaev, A.: Driver dangerous state detection based on OpenCV & dlib libraries using mobile video processing. 2019 IEEE International Conference on Computational Science and Engineering (CSE) and IEEE International Conference on Embedded and Ubiquitous Computing (EUC); 2019: IEEE.

19. Parthasaradhy, P., Manjunathachari, K.: Accident avoidance and prediction system using adaptive probabilistic threshold monitoring technique. Microprocessors and Microsystems. 71, pp. 102869 (2019).

20. Chaudhary, U., Patel, A., Patel, A., Soni, M.: Survey paper on automatic vehicle accident detection and rescue system. Data Science and Intelligent Applications: Springer. p. 319- (2021)

21. Li, L., Chen, Y., Li, Z.: Yawning detection for monitoring driver fatigue based on two cameras. 2009 12th International IEEE Conference on Intelligent Transportation Systems; 2009: IEEE.

22. Pimplaskar, D., Nagmode, M., Borkar, A.: Real time eye blinking detection and tracking using opencv. technology. 13(14), pp. 15 (2015).

23. Han, W., Yang, Y., Huang, G.-B., Sourina, O., Klanner, F., Denk, C.: Driver drowsiness detection based on novel eye openness recognition method and unsupervised feature learning. 2015 IEEE International Conference on Systems, Man, and Cybernetics; 2015: IEEE.

24. Yang, C., Wang, X., Mao, S.: Unsupervised drowsy driving detection with RFID. IEEE transactions on vehicular technology. 69(8), pp. 8151-63 (2020).

25. Virtue, S., Vidal-Puig, A.: GTTs and ITTs in mice: simple tests, complex answers. Nature Metabolism. pp. 1-4 (2021).

26. Rodríguez-Fuentes, L.J., Varona, A., Penagarikano, M., Bordel, G., Diez, M.: GTTS Systems for the SWS Task at MediaEval 20MediaEval; 2013: Citeseer.