dataesr/scientific-paragraphs-categorization huggingface dataset:
A RNN is added after the CNN to model the dependency between contiguous sleep epochs. [11]  proposed a lighter CNN which can deal efficiently with multimodal data while having fewer parameters than previous methods. [12]  [13] [14]  [15]  [16]  [17]  have all reported state-of-theart performances on various sleep staging datasets with CNN. These models (excluding  [11] ) have millions of parameters which increases computational cost and the risk of overfitting while lowering data efficiency. Most of these models are applied on a single signal from the PSG which may limit the accuracy of the estimated sleep stages. [18]  [19]  [20] introduce a different approach, the raw PSG signals of a sleep epoch are transformed into a short term Fourier transform and processed either by a 1D CNN or by a RNN followed by an attention layer  [21].
To model temporal dependencies  [18]  feeds the succession of encoded sleep epochs into a second RNN. State-of-the-art performance are reached on the publicly available MASS dataset  [22]. Most automated approaches are trained and evaluated on a single manual sleep scoring making it difficult to evaluate how they actually perform considering the low inter-rater agreement. One notable exception,  [17]  deals with the issue of interrater variability using annotations from 6 sleep technologists on a subset of training records. However the multiple sleep staging annotations are not currently publicly available. Another challenge in the evaluation and comparison of automated approaches is that no shared dataset has made a consensus for benchmarking different approaches when it has been shown that performance can greatly vary across datasets  [23].
In this study we introduce two publicly available datasets; DOD-H (Dreem Open Dataset -Healthy) and DOD-O (Dreem Open Dataset -Obstructive). DOD-H is built from recordings from 25 healthy adult volunteers. DOD-O is built from recordings from 55 patients suffering from obstructive sleep apnea (OSA). Both datasets were scored by 5 experienced sleep technologists across 3 different sleep centers. Using these datasets we propose a methodology inspired from  [17]  and  [24]  to evaluate a sleep stage algorithm against multiple sleep technologists, in order to simulate a real-life setting. This evaluation framework is available at http://github. com/Dreem-Organization/dreemlearning-evaluation together with the scores from the various sleep technologists and the PSG data for both DOD-O and DOD-H.
Using this framework we benchmark and compare several approaches from the literature  [10] [18] [11]  [25]  [9]. We also introduce and benchmark a new deep learning method, SimpleSleepNet, inspired by SeqSleepNet  [18] , DeepSleepNet  [10]  and  [11]. First, we compare the performance of human scorers and recent literature models (including SimpleSleep-Net) on DOD-H and DOD-O. Then, SimpleSleepNet is used to study the impact on sleep staging performance of the following factors: temporal context, dataset size, number of input signals, size and complexity of the model. The benchmark code is publicly available at https://github. com/Dreem-Organization/dreem-learning-open.
II. MATERIALS AND METHODS A. Datasets 1) Dataset 1: healthy patients: Dataset 1 was collected at the French Armed Forces Biomedical Research Institute's (IRBA) Fatigue and Vigilance Unit (Bretigny-Sur-Orge, France) from 25 volunteers. Volunteers were recruited without regard to gender or ethnicity from the local community via flyers. Volunteers were healthy sleepers without sleep complaints between the ages of 18 and 65, their PSG results confirmed the absence of a sleep disorder. More details and exclusion criteria can be found in  [24]. Demographics are summarized in Table  I   [2]. All scorers are registered Polysomnography Technologists, with over 5 years of clinical / scoring experience. 2) Dataset 2: patients with OSA: The dataset 2 was collected at the Stanford Sleep Medicine Center and consists of PSG recordings from 55 patients (Clinical trial number NCT03657329).
This differs from our previous work  [24]  where we used the scorer with the highest softagreement over all the records of the dataset, hence inducing a dependency to the dataset. We also compute a weight between 0 and 1 for each epoch based on how many scorers voted for the consensus sleep staging epoch. These weights are used to balance the importance of each epoch in the computation of each of the following metrics. To measure agreement between two sleep stagings on a specific record, we measure F1 -score = 2 * P r * Re P r+Re with P r = T P T P +F P and Re = T P T P +F N , and TP, FP, and FN are the number of true positives, false positives, and false negatives, respectively. The score is computed per-class, one class against the others, and averaged taking the proportion of each class into account.
C. SimpleSleepNet SimpleSleepNet is a new automated sleep staging model based on recent advances in the field. The initial stage in SimpleSleepNet is inspired by  [18]. Where it differs from the latter is in its use of a channel-wise dropout. Moreover, it replaces the filter bank with a linear layer, recombines the EEG derivations using a linear layer (an approach inspired by  [11] ), and omits the norm from the GRU. The second stage, which models epoch dependencies, is inspired by  [10]  but differs from it in that it uses positional embedding. SimpleSleepNet uses fewer parameters than the other models by reducing the size of the hidden layers. In this section, a comprehensive description of each module of SimpleSleepNet is presented. Figure  1  summarizes the overall architecture of the network.
D. Evaluation At evaluation time, the multiple available predictions for an epoch are aggregated following  [18] : given an epoch t and a temporal context k, the aggregated sleep stage probabilities is the geometric mean π(t) = exp 1 2k+1 i=k i=-k log(π (t+i) t ) and the predicted sleep stage used for evaluation is ỹ(t) = argmax j∈[[0,5]] π(t) j III. EXPERIMENTS A. Baselines To benchmark the current state-of-the-art in automated sleep staging on both DOD-O and DOD-H, we selected recent approaches from the literature reporting good performances on publicly available datasets. These approaches were reimplemented in Pytorch  [29] , for reproducibility the code is publicly available in the following repository: https://github. com/Dreem-Organization/dreem-learningopen. The presented approach SimpleSleepNet is also included in the benchmark.
The features are computed on the F4-M1 derivation on DOD-H dataset and on F4-O2 on DOD-O. 2) Tsinalis et al. 5) SeqSleepNet  [18] : SeqSleepNet takes the spectrogram of the signal as the input, the number of Fourier bins is reduced with a learned frequency filter-bank which projects the original bins on a smaller frequency space. The reduced STFT is then fed to a bidirectional LSTM with recurrent batch-normalization  [30]  followed by an attention layer. The attention layer reduces the temporal dimension and encodes the 30-second sleep epoch into a single vector. The encoded representations of consecutive sleep epochs are then fed to a bidirectional GRU, the output of the GRU is used by the classification layer to output the final sleep stage estimate. 6) SimpleSleepNet: The Fourier bins are projected on n = 30 filters and the original number of channels is kept (c = C).
C. Benchmark on DODO and DODH The overall, best and worst performances of the five scorers are reported in Table  III  for both datasets, all the metrics are computed subject-wise and not epoch-wise to be more representative of a clinical setting. On DOD-H, the average scorer F1 is 86. 8 ± 7. 6 %. The average scorer accuracy is above the one reported in  [4]. F1 is higher for REM (90. 8 ± 10. 3 %), followed by N2 (88. 9 ± 7. 6 %), Wake (84. 3 ± 13. 6 %) and lower for N3 (78. 5 ± 23. 9 %) which also shows the highest variability. N1 has the lowest F1 (50. 3 ± 14. 7 %). On DOD-O, the performances of the scorers are slightly lower than on DOD-H with an overall scorer F1 of 84. 8 ± 8. 6 %. F1 is higher for Wake epochs (90. 8 ± 8. 2 %). For all the other stages it is slightly lower with 85. 6 ± 23.
3 % for REM, 85. 6 ± 10. 7 % for N2 and 44. 6 ± 16. 8 % for N1. N3 is notably lower with an F1 of 56. 9 ± 33. 1 %. Standard deviation (SD) sensibly increases for all the stages compared to DOD-H. Figure  2  shows the scorers confusion matrices on both dataset, most of the errors involve N1 being mistaken for WAKE or N2 and N3 being mistaken for N2. The performances of the automated approaches are also given in Table  III. SimpleSleepNet shows the best performance on both datasets for the considered metrics when compared to both humans and other approaches. On DOD-H, SimpleSleep-Net is better than the best scorer and shows a lower SD with an F1 of 89. 9 ± 4. 1 %. On DOD-O, it also performs better but with a slightly higher SD than the best scorer with an F1 of 88. 3 ± 9. 0 %. With the exception of  [11]  and  [9] , every model performs better with a much lower variability on DOD-H than on DOD-O.
D. SimpleSleepNet ablation study To assess the importance of each of the modules of the architecture of SimpleSleepNet, ablated models were trained on both datasets. While technically not being an ablation of the model itself, the influence of the preprocessing step is assessed in No filtering where the filtering is removed. In No channel dropout the channel dropout is removed (p kill = 0). Then, we evaluate the effects of the blocks of the epoch encoder. In No frequency reductions the linear frequency reduction is removed, in Filter bank it is replaced by a filter-bank  [18] , in No channel recombination the linear channel recombination is removed and in No attention the attention layer is replaced by an average-pooling layer. The architecture of the sequence encoder is analyzed by removing the positional embedding in No positional embedding, by using a single layer in the GRU encoder in Single GRU layer, and by removing the skipconnection in No skip connection.
F. Direct transfer learning In a real-life, clinical setting, one may wish to train a staging model of a source dataset and to use it on another unlabelled dataset. To assess the transferability of SimpleSleepNet, we train and validate it on DOD-H (resp. DOD-O) and test it on DOD-O (resp. DOD-H). The experiment is repeated 20 times, for each repetition, 70 % of the records from the source dataset are randomly selected for training and the remaining 30 % for validation. All the records of the target dataset are used to test the model performance. The results of the experiment are shown in Table  VII. When SimpleSleepNet is trained on DOD-O and evaluated on DOD-H, the F1 drops from 89. 9 % to 84. 8 % compared to a model trained from scratch on DOD-H. The standard deviation of the performance metrics almost doubles.
G. Benchmark on external dataset 1) MASS SS3  [22] : The MASS SS3 cohort is composed of 62 nights from healthy subjects, done with a full PSG montage (20 scalp EEG,2 EOG, 3 EMG and 1 ECG) and manually scored by a sleep expert according to the AASM standard. The models were trained on the C4-O1, F4-EOG Left, F8-Cz, on the average of the two EOGs and on the average of EMG-Chin1 and EMG-Chin2 which are available for all records and frequently used by the models evaluated on MASS. We used the same preprocessing and training parameters as in the previous section III-B. The models are evaluated in a 31-folds validation way (as in  [10] ). 2) Sleep EDF  [31] : The Sleep EDF database contains 197 nights from 106 subjects, amongst these nights, 153 are from 82 subjects without any sleep-related medications (SC study) and 44 are from subjects with trouble falling asleep (ST study).
22 of the 44 nights are done after a Temazepam intake. We consider two splits, S-EDF-20 with the subjects 0 to 19 from the SC study and S-EDF-Extended will all the subjects from the database. Similarly to  [10]  [32], we only considered the epochs in-between 30 minutes before the first non-wake epoch epoch and 30 minutes after the last non-wake epoch. The models are trained and evaluated using a 20-folds CV on S-EDF-20 and 10-folds CV on S-EDF-Extended. Records from a subject are in the same fold. The models are trained on the FPZ-Cz, Pz-Oz and the EOG derivation without further processing. 3) Results: The results are presented Table  VIII. Our implementation of the literature models reaches equal or improved performance when compared to the original publications. This improvement can be explained by three different reasons.
There are also more ties on    1 ) is trained on F4-EOG and (2) on Fpz-Cz to limit the number of parameters. DOD-O than DOD-H. This is in agreement with  [17]  where models accuracy drops by 9 % on narcoleptic subjects vs healthy subjects and with  [33]  where the scorers reliability was much higher on healthy subjects than on those with OSA. Besides, the training requires more recordings to reach human performance on DOD-O than on DOD-H. All those elements suggest that the inter-subjects variability is higher within DOD-O than within DOD-H. Yet, interestingly, transfer learning from DOD-O to DOD-H is much more effective than the other way around. This implies that data acquired from patients suffering from OSA contains information related to healthy sleep as well as information specific to OSA.
This also shows that although SimpleSleepNet reaches a better F1 on DOD-H than on DOD-O, the model trained on DOD-O is much better in its generalization capacity than the one trained on DOD-H. These analyses could be extended to datasets with other sleep-related issues to see how much they impact the performance of human and automated sleep staging. This also suggests that a dataset containing high inter-subject variability, for instance with a mix of both abnormal and normal sleep, would probably lead to better models in terms of their ability to generalize. This is also highlighted in  [17]. The transfer learning experiment also highlights a practical limitation regarding the usability of such automatic method outside of the scope of the same population of patients or/and device than the one on which it has been trained on.
Given a few annotated records, automated sleep staging could reach similar performances to human scorers in a clinical setting if the data are acquired with a consistent PSG montage and patient typology. This is often the case in a typical sleep clinic setting. That being said, an interesting direction of research would be to create a model able to adapt to various PSG montage without fine-tuning or weight modifications. On external datasets, SimpleSleepNet, DeepSleepNet, and SeqSleepNet also show the best performances. However, these datasets were scored by a single expert. Inter-rater variability prevents us from drawing strong conclusions regarding the absolute performance of the various models on these datasets. Specifically, the models could be overfitting on human expert scoring. We observe that most benchmarked methods using datadriven feature extraction perform better than the expert feature extraction approach.
This is especially true on DOD-O and SleepEDF-Extended which present a higher level of variability, suggesting a better ability for such deep learning models to capture relevant information in complex data like abnormal sleep. SimpleSleepNet outperforms the best human scorer and all other sleep staging models on DOD-O and DOD-H. It is also among the best-ranked models on external datasets. It uses significantly fewer parameters than other approaches. The presented ablation study shows that the various building blocks of SimpleSleepNet allow reaching the best performance on DOD-O. SimpleSleepNet reaches close-to-human performance with only a few (∼10) recordings, suggesting that sleep stage classification is a relatively simple problem in terms of data quantity needed to reach satisfactory performance.
The temporal context and number of signals also seem to play a minor role in improving performance. The results provided in this study are available with both data and code for reproducibility. It should be noted that the benchmarked automated approaches were all reimplemented. The performances of our implementation were validated on the MASS SS3 and SleepEDF datasets with performance similar or above the original implementations. Furthermore, to ensure the fairness of the benchmark, every method was tuned to provide good results on the datasets of this study. All reported results are from a single run, rerunning the experiments might result in slightly different results due to randomness and variability.
V. CONCLUSION In this work, we introduced two open multi-scored sleep staging datasets with 25 from healthy subjects and 55 nights patients suffering from OSA. We proposed a methodology for evaluation against multiple human scorers. We showed the relevance of a multi-scored sleep dataset to assess how automated sleep staging performs in a clinical setting. We demonstrated that recent automated sleep staging performances are often onpar with the average human scorer, and that the best automated sleep staging are better than the best human scorer. We also introduced a new efficient sleep staging model, SimpleSleepNet, which outperforms previous state-of-the-art models and human scorers on both datasets and on two frequently benchmarked datasets. Better understanding and quantification of the performance of such automated approaches could be a step toward a broader use of these approaches in sleep clinics.
ACKNOWLEDGMENT We would like to thank the  Fatigue and Vigilance team including Drogou C. , Erblang M. , Dorey R. , Quiquempoix M. ,  Gomez-Merino D. and  Rabat A. for their help in the clinical trial at the  Institut de Recherche Biologique des Armées (IRBA). We would like to thanks  Dr. Michael E. Ballard ,  Hugo Jourde ,  Polina Davidenko ,  Sarah deLanda  and  Stephanie Lettieri  for their help in realizing the clinical trial at the  Stanford Sleep Medicine Center. We also would like to thank the Dreem team, including  Mason Harris  for his help gathering both datasets together with the sleep technologists' scorings, and  Pierrick Arnal ,  Théo Moutakanni ,  Clémence Pinaud  and  Olivier Tranzer  for their help with the manuscript.
, 2018; Parisot et al. , 2018; Kashyap et al. , 2019; Yoo et al. , 2019). In this work, we compared kernel regression with three DNN architectures in RSFCbased behavioral prediction. Kernel regression is a non-parametric classical machine learning algorithm  (Murphy, 2012)  that has previously been utilized in various neuroimaging prediction problems, including RSFC-based behavioral prediction  (Raz et al. , 2017; Zhu et al. , 2017; Kong et al. , 2019; Li et al. , 2019). Our three DNN implementations included a generic, fully-connected feedforward neural network, and two state-of-the-art DNNs specifically developed for RSFC-based prediction  (Kawahara et al. , 2017; Parisot et al. , 2017 Parisot et al. , , 2018)). An initial version of this study utilizing only the fluid intelligence measure in the HCP dataset has been previously presented at a workshop  (He et al.
Datasets Two datasets were considered: the Human Connectome Project (HCP) S1200 release  (Van Essen et al. , 2013)  and the UK Biobank  (Sudlow et al. , 2015; Miller et al. , 2016). Both datasets contained multiple types of neuroimaging data, including structural MRI, rs-fMRI, and multiple behavioral and demographic measures for each subject. HCP S1200 release comprised 1206 healthy young adults (age 22-35). There were 1,094 subjects with both structural MRI and rs-fMRI. Both structural MRI and rs-fMRI were acquired on a customized Siemens 3T "Connectome Skyra" scanner at Washington University at St. Louis. The structural MRI was 0. 7mm isotropic. The rs-fMRI was 2mm isotropic with TR of 0. 72s and 1200 frames per run (14. 4 minutes). Each subject had two sessions of rs-fMRI, and each session contained two rs-fMRI runs.
FC-based prediction setup We considered 58 behavioral measures across cognition, emotion and personality from the HCP (Table  S1 ;  Kong et al. , 2019). By restricting the dataset to participants with at least one run (that survived censoring) and all 58 behavioral measures, we were left with 953 subjects. 23, 67, 62 and 801 subjects had 1, 2, 3 and 4 runs respectively. In the case of the UK Biobank, we considered four behavioral and demographic measures: age, sex, fluid intelligence and pairs matching 1 (number of incorrect matches). By restricting the dataset to participants with 55 x 55 RSFC matrices and all four measures, we were left with 8868 subjects. For both datasets, kernel regression and three DNNs were applied to predict the behavioral and demographic measures of individual subjects based on individuals' RSFC matrices.
Instead, each node represents a subject and edges are determined by the similarity between subjects. This similarity is problem dependent. For example, in the case of autism spectrum disorder (ASD) classification, similarity between two subjects is defined based on sex, sites and RSFC, i. e. , two subjects are more similar if they have the same sex, from the same site and have similar RSFC patterns  (Parisot et al. , 2017 (Parisot et al. , , 2018)). The use of sex and sites in the graph definition were particularly important for this specific application, since ASD is characterized by strong sex-specific effects and the database included data from multiple unharmonized sites (Di  Martino et al. , 2014). Similar to the original studies  (Parisot et al. , 2017 (Parisot et al. , , 2018)) , we utilized vectorized RSFC (lower triangular entries of the RSFC matrix) of all subjects as inputs to the GCNN.
HCP training, validation and testing For the HCP dataset, 20-fold cross-validation was performed. The 953 subjects were divided into 20 folds, such that family members were not split across folds. Inner-loop crossvalidation was performed for hyperparameter tuning. More specifically, for a given test fold, cross-validation was performed on the remaining 19 folds with different hyperparameters. The best hyperparameters were then used to train on the 19 folds. The trained model was then applied to the test fold. This was repeated for all 20 test folds. In the case of kernel regression, there was only one single hyperparameter λ (that controls the regularization; see Appendix A. 1). A separate hyperparameter was tuned for each fold and each behavioral measure separately based on a grid search over the hyperparameter.
Such a bias is avoided in the UK Biobank dataset (see below). Further details about DNN hyperparameters are found in Appendix A2. As is common in the FC-based prediction literature  (Finn et al. , 2015) , model performance was evaluated based on the Pearson's correlation between predicted and actual behavioral measures across subjects within each test fold. Furthermore, since certain behavioral measures were correlated with motion  (Siegel et al. , 2017) , age, sex, and motion (FD) were regressed from the behavioral measures from the training and test folds  (Kong et al. , 2019; Li et al. , 2019). Regression coefficients were estimated from the training folds and applied to the test folds. Mean absolute error (MAE) and coefficient of determination (COD) will also be reported.
UK Biobank training, validation and testing The to predict all four measures simultaneously (like in the HCP dataset) did not appear to improve performance and so was not further pursued. In the case of kernel regression, the hyperparameter λ was tuned using the validation set based on a grid search over the hyperparameter. Like before, prediction accuracies for age, fluid intelligence and pairs matching were evaluated based on the Pearson's correlation between predicted and actual measures across subjects within the test set. Since the age prediction literature often used mean absolute error (MAE) as an evaluation metric  (Liem et al. , 2017; Cole et al. , 2018; Varikuti et al. , 2018) , we included MAE as an evaluation metric. For completeness, we also computed MAE for pairs matching and fluid intelligence.