# A Multimodal Recurrent Model for Driver Distraction Detection
## 2 Materials and Methods
### Model Evaluation and Training  
The models were trained using a university-provided GPU rack equipped with _NVIDIA GeForce RTX 3090_ GPUs. _TensorFlow_ (with _CUDA_ and _cuDNN_ support) as well as _Keras_ were used for model creation and training. This section describes the evaluation and training procedures, as well as the techniques used to improve model performance.  
A combination of hold-out and modified cross-validation was used to evaluate the models. Data from four drivers were randomly selected for training, while data from the remaining two drivers were reserved for validation, resulting in a 68%/32% split between  
Figure 5: CNN-LSTM model used for recurrent vision-based driver distraction detection.  
Figure 6: CNN-LSTM-S model used for multimodal recurrent vision-based driver distraction detection.  
training and validation. To ensure model generalization to new drivers, leave-one-driver-out cross-validation was performed, where each model was trained six times using data from five drivers for training and the remaining driver's data for testing. This approach provided an accurate estimate of the generalization performance of each model.  
For the vision-based _CNN_ model, due to memory constraints, the _TensorFlow Data API_ was used to generate _TensorFlow_ datasets for training and validation. Images were rescaled to a range of [0, 1] and a batch size of 16 was used. The _Adam Optimizer_ with a learning rate of \(10^{-4}\) and the _SparseCategoricalCrossEntropy_ loss function were chosen. A early stopping mechanism was implemented to prevent overfitting by stopping training if the validation loss did not improve for three consecutive epochs.  
Input sequences of absolute image trajectories and sensor data were used for the recurrent networks. Initially, a sequence length of two seconds with a one-second overlap was chosen, but this configuration resulted in poor generalization. A new configuration of sequence length of 3 seconds and a 0.5-second step was adopted, which significantly improved performance. A custom data generator was developed using the _TensorFlow Sequence API_ to load and fill image sequences and handle sensor data. The training configuration, including optimizer, learning rate, and loss function, remained the same as the _CNN_, but the batch size was adjusted to 8 sequences.  
During initial training, overfitting due to unbalanced class distribution was observed. The models tended to predict the dominant class, _normal_driving_, resulting in a validation accuracy that reflected the class distribution. To address this, the loss function was weighted using the inverse class distribution, forcing the model to pay more attention to the minority classes and achieving a more balanced learning process.  
Another observation was that the models, especially the _CNN_, performed best in the first epoch, but declined in subsequent epochs, likely due to high redundancy in the data. To mitigate this, temporal downsampling was applied, reducing the sampling rate from 30 frames per second to 5 frames per second. This adjustment allowed for training over multiple epochs, allowing for better monitoring of validation loss trends.