# Applying Neural Networks with Time-Frequency Features for the Detection of Mental Fatigue
## 3 Results  
### Performance  
The overall AI model performance results are presented below (Table 1, Figure 2), as measured with four standard machine learning metrics: accuracy, precision, recall and F-score. As such, the attained classification accuracy was 97%, with an additional very high F-score (reaching 97%), thus indicating that mental fatigue was identified effectively (the confusion matrix of the AI model results is presented in Figure 3a). For the 150 epochs of our model, training was completed in 44 min. Figure 3b presents the validation history in the form of validation accuracy over the course of epochs. Moreover, the estimated processing time for the approximated prediction time of a single instance was approximately 4 ms (following the suggested methodology and the computing resources setup described above). To evaluate whether the resulting performance could be further improved, an additional 300 epoch model was evaluated. However, despite the fact that in a separate training session the overall accuracy increased to 97.6 (\(\approx\)0.1% increase) the test dataset required double the amount of training time in the same computational setup. As such, the extra \begin{table}
\begin{tabular}{c c c c c} \hline  & **Precision** & **Recall** & **F-Score** & **Support (No. of Samples)** \\ \hline
**Fatigued** & 98\% & 97\% & 97\% & 124 \\
**Rested** & 97\% & 97\% & 97\% & 117 \\
**Overall Accuracy** & - & - & **97\%** & **241** \\ \hline \end{tabular}
\end{table} 
Table 1: The performance of the proposed model. Figure 3: (**a**) Proposed model confusion matrix; (**b**) validation curve for 150 epochs (blue) and 300 epochs (magenta). Darker colors represent a smoother curve. Figure 2: The proposed model performance regarding: (**a**) train loss over the epochs; (**b**) validation loss over the epochs; (**c**) receiver operating characteristic (ROC) curve. time could be considered unnecessary since the validation accuracy reaches a plateau and the model converges (Figure 3b). In AI-applications, the computational cost is an important factor in order to balance the performance and the time needed for the model solution. In this regard, the requirements of the proposed model can be considered minimal, both in terms of computational power and in the time required for training and evaluating the AI model. Specifically, as described above, the model was trained in 44 min with no CUDA cores, while each consequent prediction was performed in 4 ms. On this premise, to provide an indication of the computational and the temporal efficiency of the proposed configuration, a thorough comparison in the same computation settings was conducted. As such, various state-of-the art AI models that examine mental fatigue classification were recreated and trained with the same computational setup, with the algorithmic cost-efficiency evaluation including the time required for training and testing (Table 2). Specifically, the recreated AI models were fed with the same input data as the proposed approach and the duration of training and prediction time were measured. The models included: (a) a dual convolutional neural network (CNN) [11]; (b) a 1-dimensional u-net combined with a long short term memory (LSTM) network [12]; and (c) a modified principal component analysis network (PCANet) with a support vector machine (SVM) classifier [26]. The input of each of these models was a modification of the dataset described in Section 2.4 in order to comply with the parameters and designs provided by the studies' authors. In detail the dual CNN and UNET(CNN) + LSTM models require single channel time-series and, thus, a single channel time-series was provided. Similarly, the input for the PCANet was a \(C\times r\) matrix, where \(C\) is the number of channels and \(r\) is the number of the PCA components of the reduced time-series data. The aforementioned AI designs were recreated using PyTorch to minimize any implementation differences.