# An Intelligent Real-Time Driver Activity Recognition System Using Spatio-Temporal Features
## 1 Introduction
### Related Works  
In the last few decades, detecting driver behaviors has become an active research theme. Various machine learning (ML) and deep learning (DL) techniques have evolved to recognize driver behavior. In more recent decades, many studies have focused on extracting driver body features such as eye gaze direction [5] and eye closure duration, head pose [6] or posture [7,8], hand cues [9], and gaze fixation time [10] to classify drivers' distraction actions. These features have been widely used to identify driver sleepiness [11,12], driver fatigue [13,14], and driver drowsiness [15,16]. However, extracting gaze direction, the positions of hand and body joints, and head pose angles is not always easy to achieve and requires specific hardware devices, which leads to high computational costs [17]. Works utilizing machine learning models include [PERSON] et al. (2018), who used vehicle data such as acceleration, speed, and pedal positions to identify drivers' engagement in secondary tasks, including cellphone calls and texting. The author reported that Decision Tree and Random Forest models performed best in detecting and classifying these tasks. Similarly, in 2019, [PERSON] et al. (2019) used an Extreme Gradient Boosting (XGB) algorithm to identify factors leading to crashes and near-crash events. The XGB classifier was compared to three other ML algorithms, and driver behavior and intersections were identified as major crash contributors. Additionally, [PERSON] et al. (2019) implemented driver visual analysis using the driver eye glance frequency, glance duration, and glance eccentricity to distinguish crash and/or near-crash events from non-crash events. In addition, three subsequent studies used the naturalistic driving dataset (SHRP2) to investigate the causes of the crash and near-crash events associated with driver secondary tasks. The studies revealed driver actions including reaching for objects, reading, exceeding the speed limit, violations of stop signs, and improper turns as significant contributors to crash and near-crashes events ([PERSON] et al., 2019; [PERSON] et al., 2020; [PERSON] et al., 2020). In another study, [PERSON] et al. (2020) proposed the support vector machine (SVM) algorithm, which extracts the characteristic features of driver images to detect the use of cell phones while driving. On the other hand, some of the works on deep learning include [PERSON] et al. (2020), who adopted a deep learning model, CNN, to analyze four driver activities based on the Southeast University (SEU) driving posture datasets. This dataset contains video clips covering four driving postures such as eating, safe driving, smoking, and answering mobile phones. [PERSON] et al. (2020) presented a context-dependent distracted driver detection algorithm, AttenD2.0, which was an extension of their original AttenD ([PERSON] et al., 2020) algorithm that uses eye-tracking data. The authors used multiple time buffers to effectively keep track of how frequently and how long the driver's eye is off the road ahead, thus identifying a distracted driver. In addition, a region convolutional neural network (RCNN) was proposed to classify driver activities such as using a cell phone, eating, braking, and operating the steering wheel ([PERSON] et al., 2020). During the experiment, skin-like regions of the images were first  
Figure 1: Overview of the proposed framework: (**a**) RGBDT sensor module composed of RGB, depth, and thermal IR sensors; (**b**) the experimental setup to capture the driver activity inside a real vehicle cabin; (**c**) NIVIDIA Jetson Xavier embedded board used to load the trained model for real-time prediction; and (**d**) main architecture of the proposed model. The CNN layer extracts the primary spatial features of the sequential input, and LSTM further explores the temporal information to perform the final prediction of the driver state. extracted using a Gaussian mixture model (GMM) method and then forwarded to RCNN to classify the final driver state. [PERSON] et al.