# An Intelligent Real-Time Driver Activity Recognition System Using Spatio-Temporal Features
## 2 Experiment Setup and Data Collection  
### Data Collection  
As shown in Figure 2, the data collection was carried out in a real vehicle cabin in collaboration with the Korea Automotive Technology Institute (KATECH) by using a well-equipped Hyundai Genesis G80 model car. A total of 31 participants were recruited (19 males, ages ranging between 25 and 44 years; and 12 females, ages ranging between 28 and 41 years). Among the participants, 10 drivers wore eyeglasses during the collection process. This allows the proposed model to recognize driver activities without any limitation of wearing eyeglasses. We particularly focused on nine driver behaviors that typically occur while driving and operating a vehicle.  
* Non-distracted driving: driving with both hands on the wheel.
* Drinking: the participant drinking a plastic bottle while driving.
* Talking: the driver engages in conversation with the passenger by raising their right arm hand from the wheel.
* Controlling: when a driver is interacting with a navigation system, radio, and dashboard.
* Looking outside: looking completely outside in the left direction while driving.
* Texting: the participant manually dials or writes a text while driving.
* Smoking: smoking while driving.
* Head nodding: when the driver is nodding off to sleep.
* Fainting: corresponds to when the driver's head and body are completely down.  
The collection process was conducted in two subsequent phases. In the first phase, 14 drivers were involved in real driving conditions and requested to perform the pre-defined actions. However, head nodding and fainting actions in a real driving scenario may cause an unexpected accident. Thus, it was excluded from the first collection process. In this collection process, only RGB color images were obtained for all of the participants at several different times. In contrast, in the second collection phase, another 17 drivers were recruited to perform all of the pre-defined actions including the head nodding and fainting activities in non-driving conditions, and both RGB color images and IR images were collected at the same time. IR images only sense differences of heat within a temperature range, so instead of color or geometry, it includes the temperature information and eliminates the typical illumination problems which occur in RGB images [37].  
Figure 2: Data collection process. The RGBDT sensor is mounted on the front side to capture the driver maneuvers.  
Moreover, the IR images enable the proposed system to study the driver's activity during nighttime conditions, which is disregarded in most similar existing works. Around 60 thousand and 20 thousand image frames were collected for RGB and IR images, respectively. Figures 3 and 4 show overviews of the data samples collected for RGB and IR images, respectively. For each driver, a video dataset was obtained using a RGBDT sensor mounted on the front to capture the whole head and body movement with a resolution of 1280 \(\times\) 720 and 640 \(\times\) 480 for the RGB and IR images, respectively. The video recorded using the sensor was then routinely sectioned into a sequence of image frames. Then, data labeling was performed by first creating an action category for each driver and manually labeling each image according to the pre-defined categories.