# InfNRS-QC: Crowd-Sourced Creation of a Dataset and Machine Learning Model for fNIRS Quality Control
## 2 Materials and Methods
### Collected Data and Processing  
The subset of 510 segments used for the proof-of-concept were selected from the Mother-Child Synchrony study. A total of two thousand four hundred and one (N = 2401) ratings were collected: a breakdown of the ratings by user is reported in Table 2.  
Each rating consists of three pieces of information: the label of the signal quality, the self-reported level of expertise of the rater, and the time required for the rater to assign a label to the signal (reaction time). No identifiable or demographic data of the raters are collected. Overall, the Percent Agreement between self-described Expert fNIRS users is 62.4%, between Intermediate users is 14.4%, and between Beginner users is 39.4%, while the average percent agreement between expert and beginner users is 30.4%, and between expert and intermediate users is 29.3%. The average percent agreement between intermediate and beginner users is 33.9%. \begin{table}
\begin{tabular}{c c} \hline \hline
**Study** & **Number of Signals** \\ \hline Mother-Child Synchrony & 836 \\ Father-Child Synchrony & 2034 \\
3-Love Study & 1035 \\ Mother-Father Synchrony & 480 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Breakdown of the number of signals per study of origin that have been included in the current dataset. The time required for the rating was used to compute the confidence weight (\(w_{c}\)) for each rating. All rating times of each user were assigned to four confidence levels, based on thresholds corresponding to the 25 th, 50 th, and 75 th percentile of the distribution of the rating times. Ratings below the 25 th percentile were associated with a high confidence and assigned a \(w_{c}=1\); similarly, other levels were associated with lower confidence and assigned a \(w_{c}=0.75\) (25 th to 50 th percentile), \(w_{c}=0.5\) (50 th to 75 th percentile), and \(w_{c}=0.25\) (greater than 75 th percentile). The self-reported expertise was also used to assign and experience weight (\(w_{e}\)) to each user. Users self-reported as \"Expert\" were assigned a \(w_{e}=1\), \"Intermediate\" users were assigned a \(w_{e}=0.66\) and \"Beginner\" users were assigned a \(w_{e}=0.33\)  
The ratings and the weight were used to compute the class of each segment in the dataset. The three labels correspond to three different quality levels (\(q\)) of the signals:  
* Accept. The presented segment of a signal has a good quality, that is deemed acceptable by the user. This class was assigned a \(q=3\). * Keep after correction. The portion is affected by noise or artifacts (e.g., spikes), but after applying appropriate signal processing methods to increase the signal to noise ratio and remove artifacts, the portion can likely be used for further analysis. This class was assigned a \(q=2\). * Reject. The portion is very noisy or affected by artifacts that cannot be corrected using standard signal processing techniques. This class was assigned a \(q=1\). Figure 1: Screenshot of the fNIRS-QC web interface. \begin{table}
\begin{tabular}{c l c c c c} \hline \hline
**User ID** & **Expertise** & **Reject** & **Keep after Correction** & **Keep** & **Total** \\ \hline
1 & Intermediate & 153 & 249 & 100 & 502 \\
2 & Intermediate & 5 & 7 & 18 & 30 \\
3 & Expert & 52 & 51 & 33 & 136 \\
4 & Intermediate & 8 & 81 & 11 & 100 \\
5 & Intermediate & 5 & 95 & 0 & 100 \\
6 & Expert & 18 & 6 & 8 & 32 \\
7 & Intermediate & 188 & 281 & 33 & 502 \\
8 & Beginner & 123 & 251 & 125 & 499 \\
9 & Beginner & 319 & 57 & 124 & 500 \\ \hline  & & 871 & 1078 & 452 & 2401 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Breakdown of the ratings by user (N = 9). Three different methods were adopted to aggregate the ratings from different users and compute a unique quality level \(Q\) for each segment. The first (\(Q_{m}\)) method was simply the majority vote, in which the most voted quality level \(q\) was assigned. Other methods were based on weights.