Don’t Sleep: Building your first Drowsiness Detection System | by Vincent Tatan | Towards Data Science

It was a very important day, the test and project deadline were due next week but you hadn’t prepared much because of the new Halo release. Out of anxiety, you rushed to library and opened your laptop. However, as you lost your sleep leveling up your character, your mind quickly took you to dreamland. Coffee did no avail as you slept soundly in front of your laptop. Time was running short and You were desperate to stay awake.
What if I tell you... Your laptop could help you stay awake.
Imagine you were that pitiful guy, you could have activated DDS app installed in your laptop. This would trigger your laptop webcam. Every time you fell drowsy, your laptop would notice and ring an alarm to your headphone. After you woke up,your laptop would notice and turned off the alarm. You could then resume your work.
Cool use case..?
Now, let us figure out how to develop the DDS model. Hope you are ready ☺
Imagine your friend’s or loved one’s sleepy face, how did you know they were sleepy? Well, the most obvious signs are:
Eyes: eyes are open (awake), eyes are closed (sleepy)Mouth: Mouth is closing (awake), mouth is split/droopy (sleepy)Head Position: Head is at the same position(awake), head is bobbing (sleepy)
Eyes: eyes are open (awake), eyes are closed (sleepy)
Mouth: Mouth is closing (awake), mouth is split/droopy (sleepy)
Head Position: Head is at the same position(awake), head is bobbing (sleepy)
How do we teach our computer to notice these signs?
Ratio of black pixels to white pixels (BW): to detect whether your eyes are open or closed. Because of your dark pupil, the larger your eyes open, the more black pixels would appear.
However, the problem is, there might be other confound variables which would interrupt the BW ratio:
Camera to user distance: users might be far away from the camera depending on their optimum study position. Obviously the further away they are, the less is the BW ratio and vice versa.Eyes and pupil size: Eye size for different races might be different. Caucasians and Latinos will have different eye size. Don’t even mention a Chinese like me who owns exceptionally small BW ratio. Cheers.
Camera to user distance: users might be far away from the camera depending on their optimum study position. Obviously the further away they are, the less is the BW ratio and vice versa.
Eyes and pupil size: Eye size for different races might be different. Caucasians and Latinos will have different eye size. Don’t even mention a Chinese like me who owns exceptionally small BW ratio. Cheers.
To address these confound variables, we need to calibrate the BW ratio upon activation. This is our baseline to determine if users are opening or closing their eyes.
Objects Detection and Labelling: to detect if the mouths are split or not. If the mouth is closing then there would be one connected lips. However, if the mouth is split, then your lips are not connected and will return multiple objects/lips detected.
Face Detection with Accelerometer: to graph the movement of the face. The more your face moves vertically, the more likely you are sleepy.
In this publication, we will focus on the eyes and mouths as our identifiers to classify sleepiness. Now, let us get started :).
DrowsinessDetection.m: Capture a live camera tracking and run classifier methods for each VideoFrame takenEyeDetection.m : Capture and crop eyes for each VideoFrame and run classification variable (ratios)MouthDetection.m : Capture and crop mouth for each VideoFrame and run classification variable (lips detected)Scoring.m : Classify sleepiness based on the measurements returned by EyeDetection.m and MouthDetection.mMeasureTracking.m : Loop through different pictures collected and print different measurements of EyeDetection.m and MeasureDetection.m for analysis.
DrowsinessDetection.m: Capture a live camera tracking and run classifier methods for each VideoFrame taken
EyeDetection.m : Capture and crop eyes for each VideoFrame and run classification variable (ratios)
MouthDetection.m : Capture and crop mouth for each VideoFrame and run classification variable (lips detected)
Scoring.m : Classify sleepiness based on the measurements returned by EyeDetection.m and MouthDetection.m
MeasureTracking.m : Loop through different pictures collected and print different measurements of EyeDetection.m and MeasureDetection.m for analysis.
Please take a look at the following chart for better understanding.
The job of the DDS Controller is to activate your laptop’s webcam and use the classifiers to evaluate your features.
First, let us create a face detector object with the point tracker to keep track of the face detector object
faceDetector = vision.CascadeObjectDetector();pointTracker = vision.PointTracker(‘MaxBidirectionalError’, 2);
Once we are done, you could start the webcam, then add on the loop to make sure that it captures in frames. The system will keep taking a picture until a person is detected.
Once a face is detected, we will calibrate the features. This will produce the following initial ratio calculations.
Then we will activate EyeDetection.m to detect the eye inside the image . The first image of successful detection will be used to initiate threshold ratio for the subsequent classifications.
cam = webcam();while truedisp(‘Open your eyes wide before picture is taken’)pause(3)videoFrame = snapshot(cam);frameSize = size(videoFrame);try   EyeDetection   initialratio=ratio;   thresholdratio = initialratio*0.95;   break
Once we detect the person’s face and eye, we will move on to prepare the tiktok alarm to be sounded. We will run this sound only if the Scoring.m returns ‘sleepy’ as the verdict.
If the face is not detected, the application will keep taking snapshots and playing the buzzer until a face is detected or the user force stop it.
The outputs from both EyeDetection and MouthDetection will be passed into Scoring.m for scoring and coming out with the verdict.
EyeDetectiondisp([‘ratio is ‘,num2str(ratio,3),’ threshold is ‘,num2str(thresholdratio,3)]) ;MouthDetectiondisp(‘Lips detected: ‘+lipsdetected);
Once you stop the application, the webcam and video objects will be released.
clear cam;release(videoPlayer);release(pointTracker);release(faceDetector);
Given the image taken by the camera, we will initiate EyeDetector Object with CascadeObjectDetector.
The cascade object detector uses the Viola-Jones algorithm to detect people’s faces, noses, eyes, mouth, or upper body. You can also use the Image Labeler to train a custom classifier to use with this System object. — Mathworks
Please refer to this amazing publication for your understanding regarding Viola-Jones Algorithm used to detect faces.
ieeexplore.ieee.org
We would use the CascadeObjectDetector to get the bounding box of your face and isolate your eyes.
im1=videoFrame;% Get EyeDetector ObjectEyeDetector = vision.CascadeObjectDetector(‘LeftEyeCART’);% only one eyes are used% Use EyeDetector on A and get the facesEyeBBOX =step(EyeDetector,im1);% Annotate these eyes on the top of the imageimannotateeye = insertObjectAnnotation(im1,’rectangle’,EyeBBOX,’Eye’);% Getting the last box and cropEyeBBOX=EyeBBOX(1,:);imeye3 = imcrop(im1,EyeBBOX);
We will then process the image to black and white then create a strel (morphological element/shape).
A strel object represents a flat morphological structuring element, which is an essential part of morphological dilation and erosion operations. We use the strel function (described below) to create a flat structuring element. — Mathworks
www.mathworks.com
% Process the image to bw, complement and strelimeye4=im2bw(imeye3,0.1);imeye5=imclose(imeye4, strel(‘sphere’,4));
We will then find the number of white pixels by summing all the image pixel with each value 1 to represent the white pixels. This will allow us to find the BW ratio.
numberOfWhitePixels = sum(imeye5);numberOfBlackPixels = numel(imeye5) — numberOfWhitePixels ;% Now calculate the ratio.ratio = numberOfBlackPixels / numberOfWhitePixels;
We will characterize sleepy mouth when your lips are half parted (separated). By reading the image derived from the camera, we will extract the mouth object with CascadeObjectDetector.
mouthDetector = vision.CascadeObjectDetector(‘Mouth’,’MergeThreshold’,90); % Use Mouth Detector on A and get the faces MouthBBOX =step(mouthDetector,im1); % Annotate these mouth on the top of the image imannotatemouth = insertObjectAnnotation(im1,’rectangle’,MouthBBOX,’Mouth’); % Getting the last box and crop MouthBBOX=MouthBBOX(1,:);
Interestingly, There are some funny cases when the application classify 3 mouths with somebody who has unibrow and thick eyebrows. For some reasons, these objects were classified as mouths. Deep apologies to my participants for the harassment. But no worries, fixes are coming.
% If mouth are above the eyes. Then please discard itif MouthBBOX(2)<=EyeBBOX(2)+EyeBBOX(4)MouthBBOX= [0 0 1 1];binaryImage = imcrop(im1,MouthBBOX);
We will then process to detect its objects so that they would connect the nearby blobs into one object. Then we store all of the labels and objects inside LabeledIm and ConnectedObj to see if the lips parted.
% Process the image to bw, complement and strelbinaryImage= im2bw(binaryImage,0.4);binaryImage = imclose(binaryImage, true(8)); % Close gaps and connect nearby blobs.binaryImage = imcomplement(binaryImage);% bwlabel should contain bunch of objects which are separated from othersstrel[LabeledIm,ConnectedObj] = bwlabel(binaryImage);
From here we count the number of objects to see if the mouth is split or not.
lipsdetected = ConnectedObj;if lipsdetected>=2   mouthstatus = ‘split’;else   mouthstatus=’closing’;end
The idea of naive bayes is to construct relevant classifiers which represents the feature values. A naive Bayes classifier will consider each of the known features to independently contribute the probability of certain classes.
In our case, given the facts that the users’ mouth and eyes are sleepy (closed eyes, split mouth), what is the probability the user is sleepy? Obviously the probability would be higher than when we classify the users’ mouth and eyes as awake (open eyes, closed mouth).
The assumption is that the features (mouth and eyes) are independent. This is fair as both features are affected by your sleepiness and are not dependent towards each other.
Please use this publication for your understanding of naive bayes
towardsdatascience.com
Using the naive bayes theorem, we derive the probability percentage.
Therefore, every time the camera sends image, the scoring.m will check the image and decide — probability wise whether a person is sleepy or not.
First, We will extract the results from eye and mouth detection. Then, we will roll a random number to compare with the probability classification and declare whether you are awake or sleepy. This simulates the probability of your falling asleep or awake.
awakeprob = [100 85 50 15]Randomnumber = rand(1)*100if strcmp(eyestatus ,’open’)&&strcmp(mouthstatus,’closing’)   if randomnumber <= awakeprob(1)      verdict = ‘awake’;   else      verdict = ‘sleepy’;endelseif strcmp(eyestatus ,’open’)&&strcmp(mouthstatus,’split’)   if randomnumber <= awakeprob(2)      verdict = ‘awake’;   else      verdict = ‘sleepy’;   endelseif strcmp(eyestatus ,’closed’)&&strcmp(mouthstatus,’closing’)   if randomnumber <= awakeprob(3)      verdict = ‘awake’;   else      verdict = ‘sleepy’;endelse   if randomnumber <= awakeprob(4)      verdict = ‘awake’;   else      verdict = ‘sleepy’;
With that, you have successfully made your first Drowsiness Detection System. Congratulations!!!
The purpose of MeasureTracking.m is to hold samples data and evaluate the general threshold which differentiate awake or sleepiness classification from both eye and mouth detection.
Supposedly, if my opened eyes generate 256.67 BW ratio and closed eyes generate 211.58 BW ratio, how does DDS know that I am closing my eyes?
Statistical inference comes to the rescue. What this means is that we will start inferring the population thresholds by collecting and analyzing the sample thresholds.
medium.com
Simply put, we assume that there is the general percentage threshold of BW ratio to differentiate open and closed eyes. The population threshold could be generalized from a certain sample size.
In this case, we retrieved 19 participants and collected pairs of opened and closed eyes pictures. Then, we measured the BW ratio for each which will be translated as the optimum threshold value. As a side note, we make sure that the samples are diverse with different races and nationalities.
These ratios are taken from the pictures of open and closed eyes
From here, we analysed that the optimum threshold is 8%. A very simple calculation, but it brings 86.67% of accuracy.
From here we use the optimum threshold as 2 objects. This simple threshold has already produced 84.21% of accuracy.
A video explains billions pictures — Vincent Tatan. :)
The biggest challenge of this application is the bias. Unfortunately, for some different facial features, the system will go crazy in detecting the lips and the eyes. It was embarrassing, but I applaud some of my participants who have patiently accepted this bias and still would like to keep in touch with me. Thanks!
To alleviate this issue, we calibrated the readings upon application initiation to take into accounts the person’s eyes or lips size. This allowed us to greatly reduce the gap of facial features across races. However, this is far from perfect. Therefore, please feel free to solve and contribute.
As for your further reference, I post this TED video which best described how my machine struggled.
The purpose for this Proof Of Concepts (POC) was created as a part of a class project at Vrije Universiteit of Amsterdam. The goal of this application is to use computer vision and face detection algorithm to help students or professionals stay awake in front of their laptops during critical projects.
In this POC, I used Matlab as the scripting language, vision.CascadeObjectDetector library to get the facial feature bounding boxes, and finally the supports from my fellow classmates as my helpful participants. Thank you guys :)
The Github Python Notebook Code is located below.
github.com
Feel free to clone the repository and contribute whenever you have time.
I would like to thank you my fellow classmates from VU Universiteit Amsterdam who gave me the chance to use you in my bias experiments. I apologize if for some reasons my machine detected your eyebrows as lips, in which I cringed and muttered “Bad machine! Bad!”. I really enjoyed learning Computer Vision and Matlab to create this useful product of mine.
Whew... That’s it, about my idea which I formulated into writings. I really hope this has been a great read for you guys. With that, I hope my idea could be a source of inspiration for you to develop and innovate.
Please Comment out below to suggest and feedback.
Happy coding :)
Vincent Tatan is a Data and Technology enthusiast with relevant working experiences from Visa Inc. and Lazada to implement microservice architectures, data engineering, and analytics pipeline projects.
Vincent is a native Indonesian with a record of accomplishments in problem solving with strengths in Full Stack Development, Data Analytics, and Strategic Planning.
He has been actively consulting SMU BI & Analytics Club, guiding aspiring data scientists and engineers from various backgrounds, and opening up his expertise for businesses to develop their products .
Please reach out to Vincent via LinkedIn , Medium or Youtube Channel
Disclaimer
This disclaimer informs readers that the views, thoughts, and opinions expressed in the text belong solely to the author, and not necessarily to the author’s employer, organization, committee or other group or individual. References are picked up from the list and any similarities with other works are purely coincidental
This article was made purely as the author’s side project and in no way driven by any other hidden agenda.