# InfNRS-QC: Crowd-Sourced Creation of a Dataset and Machine Learning Model for fNIRS Quality Control
## 2 Materials and Methods
### Deep Learning Experiments  
The architecture of the Deep Neural Network (DNN) here employed is based on the architecture described by [PERSON] et al. (2018), and consists of three sequential components: (i) a Convolutional Branch; (ii) a Long Short-Term Memory (LSTM) module; a Fully Connected Head (FCH). The Convolutional Branch consists of four convolutional blocks, each one consisting of a convolutional layer with kernel size set to 3, a batch normalization layer ([PERSON] et al., 2018), a Rectified Linear Unit ([PERSON] et al., 2018), and a pooling layer based on maximum, with kernel size 2. Additionally, in the second and third blocks a dropout layer was added to reduce overfitting. In each block, the convolutional layer expands the channels' number. The first layer expands from 2 to 32 channels, while in the subsequent layers the number of channels is duplicated iteratively reaching 256 channels. A pooling layer is then used to compute the average of the convoluted signal at 10-time points, followed by an additional dropout layer. Following the Convolutional Branch is an LSTM Module ([PERSON] et al., 2018; [PERSON] et al., 2018), a recursive layer used to leverage the specific properties of sequential data. The Network here employed contains a single-layer LSTM module, with a number of features in the hidden state set to 100. The DNN was implemented in Python (v. 3.8.10), using the Numpy ([PERSON] et al., 2018), Pandas ([PERSON], 2018), and Scikit-learn ([PERSON] et al., 2014), and Torch ([PERSON] et al., 2016) packages (Numpy v. 1.19.4, Scikit-learn v. 0.23.2, Pandas v. 1.1.4, Torch v. 1.9.0 + cu102). The network was trained for 1000 epochs, with a batch size of 128. The learning rate is initially fixed to 1, and divided by \(\sqrt{10}\) every two epochs. Network's performances are evaluated in terms of accuracy, precision, and recall scores, as well as of the F1 score and Matthew Correlation Coefficient (MCC). While the accuracy of a model --the ratio between correctly classified segments and the total number of segments-- is commonly used as the main metric to assess the performances of a model, it has been reported to be biased for dataset with an unbalanced number of labels per class, as in the case of the dataset here presented. In such cases, the accuracy score has been \begin{table}
\begin{tabular}{l c c c} \hline \hline
**Rating** & \(Q_{m}\) & \(Q_{e}\) & \(Q_{c}\) \\ \hline Keep & 19 & 53 & 47 \\ Keep after Correction & 324 & 213 & 212 \\ Reject & 167 & 244 & 251 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Breakdown of the number of ratings (N = 510) per aggregation and class type. proved to overestimate the performances of a classifier [10; 32; 33]. To take into account such a bias, different metrics have been introduced to assess the performances of binary and multiclass classifiers, such as the F1 score and the Matthew Correlation Coefficients. While the first has obtained a higher adoption in the field, the score is not class-independent, thus indicating different scores for binary classifiers accordingly to which class is labeled as positive and which is labeled as negative. Additionally, the F1 score does not take into account segments correctly classified as negative recall scores, therefore not providing a complete and objective evaluation of a model's performance. To overcome the limitations of the F1 score, a new metric named Matthew Correlation Coefficient (MCC), based on a special case of the \(\phi\) correlation has been introduced [34; 35]. As compared to the F1 score, MCC has two main advantages. First, all the four categories of the confusion matrix (true positives, false negatives, true negatives, and false positives) are considered in the metric, as well as the ratio between elements of the different classes in the dataset, therefore providing a more balanced performance indicator [36]. Secondly, the metric is class-independent, thus not influenced by the assignment of the positive and negative labels to the different classes.