# An Intelligent Real-Time Driver Activity Recognition System Using Spatio-Temporal Features
## 1 Introduction  
### Motivation  
The dramatic worldwide increases in road traffic deaths and injuries have become a serious health and development concern around the world. Over a million people lose their lives as a result of traffic accidents, and as many as 50 million people are injured each year worldwide [1]. Currently, road traffic accidents are the leading cause of death for children and young adults. Researchers predict that between now and 2030, road traffic accidents will increase from being the ninth leading cause of death globally to the seventh [1]. According to the National Highway Traffic Safety Administration (NHTSA), approximately 60% of road traffic incidents occurred during the day, while more than 40% of all accidents occurred at night, despite less traffic. And driver distraction was a factor in 8% of fatal accidents and 15% of accidents resulting in injuries [2]. Several factors have been identified which affect the likelihood of road traffic accidents. Among them, driverdistraction is one of the most prominent factors in road accidents. Driver distraction is any type of activity that draws driver attention away from safe driving during the day- and nighttime conditions. Dialing or talking on a cell phone, operating the navigation system, smoking, eating, and drinking while driving are a few examples of distracted driving [2]. In addition, nodding off to sleep, fainting, and mind wandering while driving lead to a relative disengagement from safe driving and are considered to be major contributors to fatal traffic accidents [3]. In general, driver distractions are categorized into three major classes, including visual distractions (taking eyes off the road), manual distractions (taking hands off the wheel), and cognitive distractions (taking the mind off of safe driving) [4]. Therefore, limiting the contributors to these risk factors is important to the success of efforts to reduce the severity of road accidents and injuries. In this paper, a real-time driver behavior recognition system based on LRCN is proposed, which effectively detects driver activities in both day- and nighttime conditions. Figure 1 depicts the overview of the proposed system. A series of five consecutive images is used as an input to the LRCN model. The more generic name for LRCN is CNN-LSTM, corresponding to LSTM that uses a convolutional layer in the front end. A pre-trained CNN named Visual Geometry Group (VGG), a neural network with 16 layers, is applied to facilitate fast training by benefiting from transfer learning, where a trained model architecture and weights can be reused for a similar purpose. Then, each layer of the CNN is wrapped with a TimeDistributed (TimeDis) layer to maintain the temporal structure of the input data. Next, LSTM is integrated for end-to-end fine tuning as well as to learn the sequential input to decide the final driver state. The main motivation for selecting LRCN as a modeling approach is that driver actions appear in a sequence of images comprising spatial and temporal information which CNN alone fails to capture. In fact, a single CNN can detect these correlations, but it requires deep feature engineering such as background subtraction and other related segmentation techniques, which can be computationally expensive. Instead, LSTM is explicitly designed for sequence data to learn long-term dependencies. Thus, it is considered to capture and synthesize both the spatial and temporal features without any background subtraction efforts. Unlike existing studies, our approach works against nine different driver behaviors including non-distracted driving, drinking, controlling, smoking, texting, looking outside, talking with a passenger, head nodding, and fainting during both day- and nighttime in real-time conditions. During nighttime, due to the low intensity lighting conditions, RGB sensors alone are inadequate for accurately capturing drivers' actions.