Driving distraction detection based on gaze activity

Electronics LettersVolume 57, Issue 22 p. 857-859 Image and vision processing and display technologyOpen Access Driving distraction detection based on gaze activity Yingji Zhang, Yingji Zhang Shandong Provincial Key Laboratory of Network Based Intelligent Computing, University of Jinan, Jinan, ChinaSearch for more papers by this authorXiaohui Yang, Corresponding Author Xiaohui Yang ise_xhyang@ujn.edu.cn orcid.org/0000-0001-9677-979X Shandong Provincial Key Laboratory of Network Based Intelligent Computing, University of Jinan, Jinan, China Email: ise_xhyang@ujn.edu.cnSearch for more papers by this authorZhiquan Feng, Zhiquan Feng Shandong Provincial Key Laboratory of Network Based Intelligent Computing, University of Jinan, Jinan, ChinaSearch for more papers by this author Yingji Zhang, Yingji Zhang Shandong Provincial Key Laboratory of Network Based Intelligent Computing, University of Jinan, Jinan, ChinaSearch for more papers by this authorXiaohui Yang, Corresponding Author Xiaohui Yang ise_xhyang@ujn.edu.cn orcid.org/0000-0001-9677-979X Shandong Provincial Key Laboratory of Network Based Intelligent Computing, University of Jinan, Jinan, China Email: ise_xhyang@ujn.edu.cnSearch for more papers by this authorZhiquan Feng, Zhiquan Feng Shandong Provincial Key Laboratory of Network Based Intelligent Computing, University of Jinan, Jinan, ChinaSearch for more papers by this author First published: 04 August 2021 https://doi.org/10.1049/ell2.12286AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract Driving distraction detection can effectively prevent the occurrence of traffic accidents. Thus, monitoring a driver's state is very important for road safety. At present, most driving distraction detection methods focus on singular aspects, such as gaze distraction or hand distraction, and rarely focus on cognitive distraction as it is difficult to detect. This study proposed a driving distraction detection method based on gaze activity. A multi-channel convolutional neural network was used to classify the driver's gaze area and calculate the gaze activity. In cognitive distraction, the driver's gaze activity is significantly lower than that in the normal driving state. The activity thresholds via experiments have been obtained and used to determine whether drivers were in a cognitively distracted state. Through experiments, the accuracy of this method reached 92.36%. To identify driving distractions more comprehensively, a gaze distraction algorithm based on the two-second rule has been added to the method. The experiment demonstrated that our method in combination with this algorithm improved all of the indicators (when compared to only the gaze activity algorithm), and the accuracy rate increased to 95%. Introduction With the rapid development of the automobile industry, road safety can now be ensured with the help of hardware. However, a significant number of traffic accidents result from unsafe driving behaviour or distracted driving. According to National Highway Traffic Safety Administration (NHTSA) data statistics, approximately 25% of collision accidents are caused by drivers' inattention, resulting from, for example, distraction or fatigue; therefore, it is necessary to detect driving distractions to reduce the risk of traffic accidents [1, 2]. Driving distraction means that the driver takes his attention away from the driving task and shifts it to secondary tasks. Although distraction is a cognitive state, researchers still divide driving distraction into several categories, such as manual distraction, gaze distraction, and cognitive distraction [3-5]. Among them, cognitive distraction is the most difficult to evaluate, and it also occurs most frequently. Cognitive distraction detection methods with the driver as the main body have typically been divided into two types according to the input: physiological-based and visual-based. Physiological-based approaches mainly use drivers' physiological information, such as heart rate and electroencephalogram (EEG), as input data for distraction detection. They typically use simulators to extract physiological indicators with intrusive devices. Lanata et al. [6] collected the driver's heart rate, respiration, and skin electrical signals in an indoor simulation experiment and evaluated the driver's cognitive state in combination with vehicle operating parameters. Wang et al. [7] used EEG to analyse the driver's brain activity, and this method exhibited an accuracy of 81% in predicting the start time of the distraction state, and the accuracy of the end time was 70%. Physiological-based approaches can more intuitively reflect the changes in the driver's attention state; however, because natural data extraction requires complex and expensive intrusive equipment which has a greater impact on drivers in actual situations, a simulated environment is usually used. Visual-based approaches generally use eye trackers or other equipment to extract gaze information, or use cameras placed in the cab for driving data collection to extract the features of the driver's face, eyes, body, and various joint points. In 2019, Anh et al. [8] established a vestibular eye reflex model and an optokinetic response model to obtain the driver's eye movement parameters, and used these to conduct experiments in simulated and actual scenes. The final experiment showed that eye movement data could be used as a parameter of cognitive distraction. Cheng et al. [9] proposed a cognitive distraction detection method based on head-eye behaviour characteristics. The results of natural driving experiments show that for drivers without glasses, eye activity is the index with the highest rate of cognitive distraction detection. The driving distraction detection method proposed in this study is a visual-based approach, and the main research directions are gaze distraction and cognitive distraction. The specific contributions are as follows: (1) The driver gaze dataset was collected in a natural driving scene to train the model and evaluate the distraction detection algorithm. (2) We proposed an algorithm for gaze distraction detection based on the two-second criterion. (3) We proposed a multi-area driver cognitive distraction detection algorithm based on gaze activity, and the corresponding threshold was obtained through experiments. By identifying the driver's gaze area in a time window, the gaze activity was calculated to determine whether the driver was cognitively distracted. In the following section, the framework and process of our method are explained in detail. Proposed methods Gaze and cognitive distractions are the two main forms of driving distractions. Gaze distraction means that the driver's gaze has moved away from the road, and can be detected based on the length of time the gaze remains away from the road. In cognitive distraction, the driver is thinking about something unrelated to the driving task, but the gaze remains fixated on the road. We can then use the corresponding algorithm to analyse and detect distraction. The overall framework of the proposed system is shown in Figure 1. Fig. 1Open in figure viewerPowerPoint Overall framework of the system The method can be divided into three main parts, the first being data processing. We divided a cab into nine areas from the perspective of the driver. The specific areas are shown in Figure 2, and mainly include the left and right mirrors, rear-view mirrors, front windshield, instrument panel, and console area. We set the camera near the rear-view mirror, which can capture 30 frames per second of a colour video stream with a resolution of 1920 × 1080 pixels. Twenty-three drivers were required to gaze at nine areas successively in the same vehicle in the daytime to obtain static datasets for model training. In addition, on the premise of ensuring safety, ten drivers were selected to collect distracted and undistracted video data during the normal driving process for subsequent threshold experiments and model tests. We converted the colour video stream into video frame data and annotated them using OpenCV. Next, we extracted and saved four sub-images (left eye, right eye, face, and head). Owing to the large amount of data, the saved image data was finally converted into the TFRecords format for use. Fig. 2Open in figure viewerPowerPoint Driver's gaze area distribution The second part was the driver's gaze area identification. This module uses a four-channel convolutional neural network [10], where the basic model uses a four-layer convolutional neural network, to identify the driver's gaze area. During the training, the four parts of the static data were input into the model for learning. We set the batch size = 32, step = 20,000 in order to train 43,200 images and obtain the weights. During testing, we decomposed the driving distraction video into continuous 5 s groups of video frames and used trained weights to identify the gaze areas and save them in time windows, which were used for calculation and analysis of the distraction detection algorithm. The third part was distraction detection. In our proposed system, there were three types of driving states: normal driving, gaze distraction, and cognitive distraction. The specific process is illustrated in Figure 3. Fig. 3Open in figure viewerPowerPoint Distraction detection process Gaze distraction is more common than the other distraction types. Studies have shown that when the gaze remained in the cab area for more than 2 s, the probability of accidents increased by more than three times [11]. Therefore, we considered 2 s of continuous gaze in the cab area (dashboard and console, excluding all rear-view mirrors) as the standard. Because we extracted three frames per second, when the driver's gaze was detected to be in the cab area for six consecutive frames, it was considered as gaze distraction. Cognitive distraction is a brain distraction that is difficult to detect because the gaze remains fixated on the road. Therefore, we proposed that cognitive distraction can be detected based on the driver's gaze activity. When a driver is cognitively distracted, his gaze activity will be reduced due to the shift in the brain's attention. And we found that different gaze concentration areas (such as left lane change, right lane change, and straight ahead) caused a significant change in gaze activity. Accordingly, we used K-means to divide the gaze concentration area into 'Left', 'Right', and 'Middle'. The gaze area information in the current time window was used to calculate the specific concentration area. In the clustering experiment, the three-dimensional cab area was transformed into two-dimensional coordinates. To make the gaze information more intuitive, we redefined nine gaze areas and expressed them in plane coordinates, where 0 = (0, 0), 1 = (1, 2), 2 = (1, 1), 3 = (1, 0), 4 = (2, 2), 5 = (2, –1), 6 = (3, 2), 7 = (3, 1), and 8 = (4, 0). Figure 4 shows the K-means distribution of the driver's gaze in a single time window. In this figure, the dots represent the driver's gaze points and the crosses represent the cluster centre points. We define A as the cluster centre point of a single time window; its value was obtained by clustering the gaze points as coordinate input, and its ranges in different regions were obtained after several experiments. Table 1 lists the classification range of A. Fig. 4Open in figure viewerPowerPoint Cluster distribution of gaze in left area, middle area, and right area Table 1. Value range of the cluster centre point of each gaze concentration area Area Left Middle Right Range A < 1.1 1.1 < A < 2 A > 2 After obtaining the gaze concentration area, we calculated the driver's gaze activity. By comparing the data of each part, we found that when the driver's gaze was concentrated in the left area, the gaze activity range was smaller in the other two areas. Therefore, it was easy to distinguish distracted and undistracted states. The gaze activity in the left area was calculated as follows: V L = 1 f p s × t ∑ i f p s × t g i − A 2 (1)where fps is the number of frames extracted per second, t is the length of a time window, and gi is the gaze area of each frame. In our system, we set fps = 3 and t = 5. When the driver's gaze was concentrated in the middle or right areas, there was a large overlap between them at the front windshield, and the range of gaze movement was too wide. Thus, a slight change in gaze caused a significant change in gaze activity. To avoid unnecessary errors due to the small threshold difference, we adjusted the gaze activity formula: V M R = 1 f p s × t ∑ i f p s × t g i − A 2 2 (2) Using the above algorithm to calculate the driver's gaze activity in the time window, the corresponding threshold was obtained from the experiment to determine the driver's state. If less than the threshold, the driver's state was considered cognitively distracted. Experimental results and discussion To obtain the threshold of gaze activity and to verify the effectiveness of our proposed method, we conducted a series of experiments. Because of the particularity of the multi-channel convolutional neural network, the input data followed the one-to-one correspondence rule; hence, the four inputs all originated from the same frame of the image. Threshold experiments were conducted on the visual activity in the left, middle, and right areas. First, we calculated the gaze activity using all distracted and undistracted data in each area to obtain the approximate range of their respective thresholds. We then selected a value from this range to test the effect. Five experiments were performed for each threshold parameter to obtain the average values of the test results. Finally, the distracted and undistracted data were combined, and the best one was selected as the threshold. As shown in Tables 2, 3, and 4, the threshold parameters of the left, middle, and right areas are 0.34, 9.6, and 25, respectively. Figure 4 shows the performance of the selected threshold on the distraction dataset. In contrast to our prediction before the experiment, the best performance among the three areas was in the left area, and the worst performance was in the middle area. We assumed that this was because the gaze range of the left area was concentrated, and the gaze activity was significantly different compared to that in the other areas. It can be seen from Figure 5 that for the middle and right areas, the gaze movement range is significantly large, and these two areas show a large overlap at the front windshield; thus, the middle area can be mistakenly identified as the right area, which leads to misjudgements. Table 2. Threshold experiment results for the left area Threshold Left distraction (LD) Left normal (LN) Total 0.3 0.9375 0.995 0.9663 0.34 0.9525 0.986 0.9693 0.38 0.96 0.9764 0.9682 0.4 0.9775 0.9592 0.9684 0.42 0.9688 0.9571 0.9629 Table 3. Threshold experiment results for the middle area Threshold Middle distraction (MD) Middle normal (MN) Total 5.3 0.7705 0.9502 0.8604 7.3 0.8901 0.9126 0.9014 9.6 0.9192 0.8883 0.9037 12.2 0.9429 0.8109 0.8769 15.2 0.9796 0.76 0.8698 Table 4. Threshold experiment results for the right area Threshold Right distraction (RD) Right normal (RN) Total 16.8 0.828 0.986 0.907 19.4 0.9008 0.982 0.9414 22 0.9203 0.9605 0.9404 25 0.9599 0.9633 0.9616 28 0.9707 0.9407 0.9557 Fig. 5Open in figure viewerPowerPoint Threshold performance in distraction data (left area, middle area, and right area) After obtaining the thresholds for each part, the distracted and undistracted data were integrated to evaluate the method's performance. The evaluation index selected the prediction accuracy rate (P), recall rate (R), and comprehensive evaluation index (F1), all of which can be obtained from the confusion matrix shown in Figure 6. Fig. 6Open in figure viewerPowerPoint Confusion matrix using only gaze activity algorithm Table 5 shows the final experimental results, which indicates that through the gaze activity algorithm, the accuracy can reach 92.36%. In addition, when both the gaze distraction algorithm and gaze activity algorithm were used, the accuracy reached 95%. We have proved that the proposed algorithm can be implemented in real-time for distraction detection. And it is shown that our method meets the requirements of engineering applications in the aspect of computational complexity. Table 5. Evaluation results of the proposed method Index P R F1 Cognitive distraction 0.924 0.95 1.80 Cognitive + Gaze distraction 0.95 0.96 1.845 Conclusion In this study, we proposed a distracted driving detection method based on gaze activity to correctly identify the distracted state of a driver. This method used the drivers' gaze area to calculate their gaze activity and the duration of time when their gaze was away from the road. Subsequently, this information was used to identify gaze and cognitive distractions. After a series of experiments, we obtained an effective gaze activity threshold and the best configuration; however, there were still some unresolved problems. For example, because of the overlap region between areas, the system repeatedly identified the middle area as the right area, leading to errors. In the future, we recommend improvements in this area, combined with the hand distraction detection algorithm, to be able to identify driving distractions more comprehensively. Acknowledgments This work was supported by the Shandong Provincial Natural Science Foundation (No. ZR2020LZH004), the National Natural Science Foundation of China (No. 61603151), and cooperated with State Key Laboratory of High-end Server & Storage Technology. Open Research Data Availability Statement The data that support the findings of this study are available from the corresponding author upon reasonable request. References 1Stutts, J.C., Reinfurt, D.W., Rodgman, E.A.: The role of driver distraction in crashes: an analysis of 1995-1999 Crashworthiness Data System Data. Annu. Proc. Assoc. Adv. Automot. Med. 45, 287– 301 (2001) CASPubMedGoogle Scholar 2Hatoyama, K., et al.: Perception of time in traffic congestion and drivers' stress. In: International Conference on Transportation and Development 2019: Smarter and Safer Mobility and Cities. Reston, VA: American Society of Civil Engineers. 165– 174 (2019) CrossrefGoogle Scholar 3Née, M., et al.: Road safety and distraction, results from a responsibility case-control study among a sample of road users interviewed at the emergency room. Accid. Anal. Prev. 122, 19– 24 (2019) CrossrefPubMedWeb of Science®Google Scholar 4Foley, J.P., et al.: Towards operationalizing driver distraction. In: Proceedings of the Seventh International Driving Symposium on Human Factors in Driver Assessment, Training and Vehicle Design, Bolton Landing, New York, Iowa City, 17–20 June 2013 Google Scholar 5Strayer, D.L., et al.: Assessing cognitive distraction in the automobile. Hum. factors 57(8), 1300– 1324 (2015) CrossrefPubMedWeb of Science®Google Scholar 6Lanatà, A., et al.: How the autonomic nervous system and driving style change with incremental stressing conditions during simulated driving. IEEE Trans. Intell. Transp. Syst. 16(3), 1505– 1517 (2014) CrossrefWeb of Science®Google Scholar 7Wang, S., et al.: Online prediction of driver distraction based on brain activity patterns. IEEE Trans. Intell. Transp. Syst. 16(1), 136– 150 (2014) CrossrefWeb of Science®Google Scholar 8Le, A.S., Suzuki, T., Aoki, H.: Evaluating driver cognitive distraction by eye tracking: From simulator to driving. Transp. Res. Interdiscip. Perspect. 4, 100087 (2020) Google Scholar 9Cheng, W.D. et al.: Research on driver's cognitive distraction in mobile phone call behaviour based on image recognition. China J. Highw. Transp. 34(5), 168– 181 (2021) Google Scholar 10Zhang, Y., Yang, X., Ma, Z.. Driver's gaze zone estimation method: a four-channel convolutional neural network model. In: 2020 2nd International Conference on Big-data Service and Intelligent Computation, Xiamen, China, 3–5 December, 2020Google Scholar 11Klauer, S.G., et al.: The impact of driver inattention on near-crash/crash risk: An analysis using the 100-car naturalistic driving study data. Nat. Highway Traffic Safety Admin., Washington, DC, Tech. Rep., 2006 Google Scholar Volume57, Issue22October 2021Pages 857-859 FiguresReferencesRelatedInformation