# InfNRS-QC: Crowd-Sourced Creation of a Dataset and Machine Learning Model for fNIRS Quality Control
## 4 Discussion  
In this work, we tested the possibility of using a Deep Neural Network model to support researchers in identifying usable and unusable fNIRS signals. First, a web platform for the collection of human labels for fNIRS signals has been designed and implemented, then ratings for 510 segments of fNIRS signals were obtained by raters of different expertise levels. Collected labels were then fed to train a DNN model. The model's accuracy performances in train and test partitions indicate that the model is learning well on the test partition, but the performance drops on the test partition, suggesting a possible overfitting during the training phase. While the recall score (0.95) indicates that the majority of the relevant elements--which are usable segments--are correctly identified as usable, the precision score (0.61) suggests that a significant amount of signals labeled from the users as unusable are mislabeled by the model, as shown in the Confusion Matrix of the test partition (Table 6). Imagining a possible implementation in a real research setting, the results here reported suggest that the model can successfully help researchers identifying usable fNIRS segments from a pool of segments that contains both usable or unusable segments, which is the typical case of fNIRS experimental studies. In fact, segments may contain a different type of artifacts, some of which can be corrected, while some are so extensively tied to the signals that require a portion of signals to be discarded. However, the current implementation fails at excluding completely unusable segments, which are labeled by the model so as to include them in further analysis. Presently, the model can still support researchers by reducing the number of segments requiring a manual inspection. The ability of the model to match users' labels for usable signals is also reflected by the F1 score (0.74), while the inability to correctly label non-usable signals with a high degree of precision is highlighted by the MCC score (0.25). The limits in our model's classification accuracy can be explained by different factors. First, the limited number of segments that have been included in the dataset for this work (N = 510). In fact, the small number of segments may not have been sufficient to cover all the possible combinations of artifacts and noises that can affect the fNIRS signals. However, while a higher number of segments may have been helpful to reduce the bias of the model \begin{table}
\begin{tabular}{c c c c} \hline \hline  & & \multicolumn{3}{c}{**Predictions**} \\ \cline{3-4}  & & **Reject** & **Keep** \\ \hline User labels & Reject & 10 & 35 \\  & Keep & 3 & 54 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Confusion matrix for the test partition. \begin{table}
\begin{tabular}{c c c} \hline \hline \multirow{2}{*}{**Metric**} & \multicolumn{2}{c}{**Partition**} \\ \cline{2-3}  & **Train** & **Test** \\ \hline Accuracy & 0.70 & 0.63 \\ MCC & 0.18 & 0.25 \\ Precision & & 0.61 \\ Recall & & 0.95 \\ F1 & & 0.74 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Summary of the results by metrics and partition. \begin{table}
\begin{tabular}{c c c c} \hline \hline  & & \multicolumn{3}{c}{**Predictions**} \\ \cline{3-4}  & & **Reject** & **Keep** \\ \hline User labels & Reject & 29 & 92 \\ Keep & 30 & 257 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Confusion matrix for the train partition. toward the segments used for training--with a possible reduction in the accuracy on the train partition and the simultaneous increase of the precision on the test partition--the labeling stage of the segments would have required more extensive resources in terms of time and users. By limiting the number of segments, we have been able to obtain a higher number of ratings per each signal, therefore reducing the impact of a single rater on the overall label used for training, which is evaluated as described in Section 2.3.