# Applying Neural Networks with Time-Frequency Features for the Detection of Mental Fatigue
## 2 Materials and Methods
### Artificial Intelligence Modeling  
#### 2.5.1 Deep Learning Model  
The AI architecture consisted of three 2-dimensional convolutional layers in a sequential order (Figure 1). In the first layer, a 3 \(\times\) 3 kernel channel-wise (63 channels) convolution is conducted, with 32 output filters (each filter is producing one output matrix; thus, resulting in 32 first layer matrices). No padding was performed and a standard stride of 1 was used. The kernel size was specifically chosen to incorporate a 3 Hz frequency band and 0.75 sec time band features. The 3 Hz frequency window was selected as it is a wide bandwidth that can correlate and therefore be explained (as it is a good divisor) with the most commonly used EEG bands (delta, theta, alpha, beta, and gamma). As far as the time window duration is concerned, a time window of 750 ms was selected based on similar experimental paradigms (for 2-back working memory tasks) that present the mean reaction time of approximately 700 ms [22]. The second layer is comprised of 16 output filters with a kernel size of 3 \(\times\) 3, also computing channel-wise convolution, capturing higher-level features of the 32 first-layer input matrices. The final convolution layer consists of 4 output filters with a kernel size of 4 \(\times\) 4. Usually in DL architectures, a pooling layer is used after the convolutional layers to down-sample the feature space [23]. However, the reduction of the dimension of the outputs was not required in the proposed approach, and thus no pooling layer was utilized. Since the scope of this study includes the application of AI methods with minimal computational resources, the hyperparameters of the created shallow network were selected for fast execution time per time step during training. Thereafter, the selected number of layers was 3, taking into account the medium size of the dataset. Furthermore, the filter sizes follow a logarithmic pattern, reducing the number of features after each layer to half, thus decreasing the training and the inference time. The optimizer, the learning rate, the batch size, and the activation function were selected based on common DL practices and testing with a variety of values. The optimizer that was selected was Adam [24], the initial learning rate was selected to be 0.001, the batch size was 32 samples, and the sigmoid function was chosen as the activation function.  
Finally, a fully connected (FC) layer with 792 input nodes and 2 output nodes for each of the classes was utilized. In order to introduce non-linearity to our model, a bounded logistic function (sigmoid function) was employed as an activation function for the output probabilities. Following each convolutional layer, a batch normalization layer was utilized to increase the stability and robustness of the AI network during the training processes. Furthermore, a dropout configuration with a probability of a zero-element set to 0.25 was used to reduce overfitting and handle uncertainty. Dropout is a regularization technique that can reduce uncertainty in deep learning models by randomly dropping out neurons during training. All algorithms were implemented with custom code in the Python programming language and the PyTorch, version 1.13 [25] deep learning framework.