dataesr/scientific-paragraphs-categorization huggingface dataset:
The PSG montage contains Fz-Cz, Cz-Oz, C4-M1, EMG, EOG, and ECG derivations. Sleep Health Hearth Study (SHHS) :  [32]  [33] The Sleep Health Hearth Study is composed of two multi-center polysomnographic studies of subjects aged 40 years and older. The nights are scored according to the R&K standard  [25]  by different scorers from multiple sleep centers. We merge stages S3 and S4 into N3. We consider the second study SHHS-2 composed of 2651 nights with two EEG, one EOG, and one EMG derivation. In the literature most publications use 70% of the dataset for training and the remaining 30% for evaluation without further details. The Cyclic Alternating Pattern dataset (CAP): The CAP dataset is composed of 108 nights, with 92 pathological records from subjects with a REM behavior disorder, periodic leg movements, nocturnal frontal lobe epilepsy, sleep apnea, narcolepsy, and bruxism.
The dataset is scored according to the R&K standard  [25]. We merge Stages S3 and S4 into N3. The nights are recorded and scored in a single sleep center and each PSG contains at least 3 EEG channels, EOG, EMG, and ECG derivations. The records nfle25 and nfle33 are excluded because of missing channels. 2) Preprocessing: We preprocess the data record-wise for every dataset. First, we apply an IIR band-pass filter between 0. 2 and 30Hz to ensure the frequency content consistency between the datasets. This frequency range contains the information needed for sleep scoring. We then resample the signals to 60Hz using polyphase filtering from the scipy library. We scale each record to have a unit-interquartile range and zeromedian to ensure similar scale across datasets as in  [9]. To ensure the stability of the training, we clip values that are greater than 20 times the interquartile range.
B. Benchmark In this first experiment, we benchmark RobustSleepNet using the three training settings described in section II-A on each dataset. We are interested in comparing the DT performance and FT performance of RobustSleepNet with its LFS performance. The latter being the one commonly used in the literature. We also provide values from the literature as references. In the DT setting, we train and validate the model on seven datasets out of the eight datasets. The seven datasets are the source datasets. We evaluate the model on the remaining target dataset. In the DT setting, we use 70% of the training records from the source datasets to train the model, and we use the 30% remaining for validation. We select a subset of 250 records for the larger datasets (MESA, MrOS, SHHS) to have a similar proportion of records from each of the datasets during training.
We select the 250 records with the highest macro F1 in the LFS setting. This selection discards records with poor quality or scoring. In the LFS and FT settings, we use k-folds to evaluate the model; hence, we can predict a hypnogram for each record of the dataset correctly, i. e. , without using it during training. We indicate the number of folds in III; as there are several nights from the same subject in Sleep-EDF, we group all the nights of one subject in the same fold to avoid subject bias. In the FT setting, we use the final weights of RobustSleepNet obtained in the DT setting to initialize the model. We use fewer folds than in previous works due to computational complexity; as a result, the model is trained using fewer training examples. This setup could disadvantage LFS and FT setting models, especially for the smallest datasets, because fewer data are available during training than in the literature.
As described in II-B, we report epoch-wise F1 for consistency with the literature. For RobustSleepNet, the F1 provided is an average over three independents training runs. This average mitigates training randomness. For other approaches, we report results from the original papers. To our knowledge, they are computed on a single run and do not provide uncertainty bounds. We report the results of RobustSleepNet and from the literature for the three settings in Tab. II. In the LFS setting, RobustSleepNet performs well on most datasets. It performs on par with recent models from the literature  [8] [5][10]  [7]. The FT setting provides an average 2. 5% relative increase of macro F1 when compared to the LFS setting. In this setting, RobustSleepNet outperforms other methods from the literature on all the datasets except on MASS SS3 and SHHS.
On SS3, DeepSleepNet has a slightly higher F1 while XSleepNet has a higher F1 on SHHS (but is only evaluated on 30% of the subjects). The FT setting also provides a 5. 1% relative F1 increase compared to DT. The improvement is more significant on CAP, MrOS, MESA, and SHHS and more minor on DOD-H, SC39, and ST. In the DT setting, RobustSleepNet performs much better than previous attempts from the literature. Interestingly, it outperforms the LFS F1 on DOD-H, SC39, and ST. It also reaches the same level of performance as in the LFS setting   We assess how the performance of RobustSleepNet in the DT setting evolves depending on the number of records used for training. We incrementally increase the number of records used for training from two records per dataset (14 records) to 128 records per dataset (896 records).
We sample the same number of records per dataset when possible. When a dataset contains fewer records than required, we use all the records from the dataset. The remaining required records are sampled from the other datasets. Figure  3  shows the performances on the complete target dataset. We report the performance as the percentage of the LFS performance achieved by RobustSleepNet. The performance sensibly increases within the first few records. The results variability also decreases with the number of training records, and the F1 increases slightly with every additional record. Interestingly, we reach the LFS baseline within the first few records on DOD-H. 2) Number of attention heads: : To assess the importance of having multiple attention heads on performance, we train and evaluate a RobustSleepNet model with up to 16 attention heads.
We report the average F1 for over the eight datasets in IV 3) Influence of the channels in the target dataset: : To assess the impact of removing specific channels on performance, we evaluate RobustSleepNet in the DT setting a using montage with different channels: Single EEG (Best performing EEG channel), EEG (all the available EEG channels), EOG (all the available EOG channels), EEG+EOG (all the EEG and EOG channels). The baseline montage uses all the available derivations described in Tab. I. Its results are reported in Tab. II. Results are presented in Figure  4. The EEG combined with the EOG performs along with the baseline on every dataset. The average F1 drop is 1%. When using a single EEG channel, the drop is around 5% on average. It is around 10% on MESA and MrOS and around 3% on the other datasets.
D. Influence of Interscorer variability on transfer learning A study by the AASM on 2500 nights showed an average inter-rater agreement of 82. 6%  [4]  for sleep stage classification. We assess the impact of the inter-rater variability when training and evaluating RobustSleepNet using DOD-O and DOD-H. These two datasets include scoring from five scorers from different sleep centers. We rank the scorers based on their agreement with the other scorers  [6]   [5]. The scorer with the highest (resp. worse) agreement is ranked first (resp fifth). 1) Influence on training: We train RobustSleepNet based on the annotations of each scorer from DOD-O or DOD-H. We also train one model on the consensus of the four best scorers  [5]. We then evaluate each model using the DT setting     [5]. The reported F1 is the average F1 obtained when evaluating the other seven datasets of this study.
on the other datasets. We report the average F1 over all the datasets in Table  V  for each model. The choice of scorer implies a large variability in the final performance. Training the model on the consensus of scorers instead of an individual scorer increases the average F1 by 4% when DOD-O is the source dataset and 2. 5% when DOD-H is the source dataset. The reported F1 values are correlated to the scorer reliability. For instance, the models trained on the annotations from the worst scorer on DOD-O and DOD-H also have the lowest F1. 2) Influence on evaluation: We perform a reverse experiment to assess the influence of inter-scorer variability on the evaluation of a model. We use the model trained in DT for DOD-O and DOD-H. We evaluate by comparing each of the individual scorers and to the consensus.
E. Datasets generalization We assess the generalization and easiness defined in part II-C) for each dataset. RobustSleepNet is trained on each of the individual datasets and evaluated on all the other datasets in DT. To limit the influence of the size of the dataset, we use the 250 records for MESA, MrOS, and SHHS selected in II. We report the F 1 norm in-between all datasets. Using F 1 norm instead of F1, in that case, enables relative comparison between datasets. We also report easiness and generalization. The results are presented in Table  5. We observe three groups among the datasets. The first group is composed of the larger datasets MESA, MrOS, and SHHS. This group has the highest generalization, the lowest easiness, and the oldest subjects. They also present a large variety of pathologies.
F. Number of finetuning records We explore the evolution of the performance depending on the number of records used for FT. Starting from the DT model, we gradually increase the number k of training records used for FT. As in part III-B, 3 or 5 folds are used for the evaluation. We randomly select the k training records in each fold and use them for training and validation. We provide the results in Figure  6  as a percentage of the LFS performance from part III-B. We observe that the F1 decreases within the first few records before increasing for DOD-H, MrOS, and MESA. FT sensibly increases F1 when we use more than ten records. LFS F1 is reached within 10 to 40 records except for the two most extensive datasets, MESA and SHHS.
IV. DISCUSSION RobustSleepNet can handle a variety of input signals using multi-head attentive recombination. This ability allowed for training the model on a corpus of eight datasets presenting a large variety of montages. Doing so, the model learned a large variety of demographics and input signals. In practice, it reached on average 97% of the best LFS performance on each dataset, without using any information from the dataset. To our knowledge, this outperforms previous approaches from the literature  [34] [12]  [11]  [16]. Considering this DT performance, one can expect a pre-trained version of RobustSleepNet to perform well on unseen data. The only exception to this is the performance achieved on CAP in DT, which still presents a significant accuracy drop. However, we will discuss possible explanations as to why RobustSleepNet might underperform on that dataset.
We have provided the code to use a pre-trained version of RobustSleepNet out of the box on an arbitrary PSG file. We trained this model on the eight datasets used in this study. Increasing the number of records used for training in DT leads to a performance close to LFS performance. The    II. Hence, the LFS F1 is represented by the horizontal dashed line at 100. The performance when no finetuning records are used corresponds to DT performances. performance is relatively high even with a few records, with on average 84% of the supervised learning performance using only 14 records (2 records per source dataset) and 88% using 28 records (4 records per source dataset). The fact that we use seven datasets with highly variable demographics to train the model might explain why we quickly reach that level of F1.
One other clue to support this is that RobustSleepNet exceeds the LFS performance on DOD-H with only 56 records from other datasets. DOD-H contains only healthy young individuals; hence, it is likely to present low variability as suggested in  [5]. The use of multiple records with high variability might be why it performs so well on this "easy" dataset. While we demonstrated how the attention layers tackle the input-shape issue, the parametrization of that layer does not impact the performance much. All the attention heads use the same input, so they might tend to be similar to each other. Using several attention heads between 2 and 4 seems optimal on the considered datasets. The channel ablation experiment highlights the ability of our model to perform well on various setups. In particular, the use of only a single EEG channel leads to exploitable results.
The high variations in the performance of individual scorers highlight the need to use such a consensus for a proper evaluation. When evaluating or training a model on a single scorer, one might be over-fitting on the scorer's way of classifying. We assessed the individual contribution of each dataset of the corpus in DT by computing the normalized F1, the easiness, and the generalization metrics. The first observation one can glean is that easiness and generalization seem to be correlated and inversely correlated respectively with age. We interpret this observation in terms of variability: younger subjects present a lower level of variability than older subjects. Data from older subjects include information about young and old populations, whereas data from young subjects does not generalize to an older population.
However, age is not the sole explanatory variable. If we consider DOD-O and CAP, we observe very different easiness and generalization even though subjects from both datasets are in the same range of age. DOD-O presents a lot more channels than CAP; hence it is likely to be more informative. Moreover, DOD-O was labeled by a consensus of several sleep experts, whereas a single scorer scored CAP. The consensus is probably boosting the generalization capacity of DOD-O, as discussed with the inter-rater variability experiment. Finally, DOD-O's subjects suffer from sleep apnea, whereas CAP's subjects suffer from various conditions that are less prevalent in the population. These factors could explain the low generalization value and easiness of CAP versus the relatively high for DOD-O. They also raise one limitation in the computation of generalization and easiness: the values are relative to the corpus of datasets selected.
Eventually, the variability in the training datasets in terms of age, montage, PSG device, scoring, or pathology could all be explanatory variables to explain performance in DT. These conditions are met for this study's three most extensive datasets and have the highest generalization values. An interesting experiment would be, for each dataset, to train on an increasing number of datasets and assess the performance each time. However, this experiment is hard to perform and hard to plot due to the combinatory complexity it requires. The performance of RobustSleepNet in LFS is on par with the literature models on each dataset. The proposed approach works well on small and medium databases but might be under-performing on the three largest. We link this to the relatively small number of parameters used on RobustSleepNet.
However, using the fact that RobustSleepNet can be pre-trained on any dataset, we used FT and reached high performance on every dataset. In practice, there is no actual overhead cost to finetune a model instead of learning from scratch because one needs to retrain in every case. However, finetuning using only a few records leads to a drop in performance compared to the DT model for some datasets. That is why FT should be performed with sufficient records to improve performance and reach the LFS performance. Using around 10 to 40 records was enough for the small datasets of this study. Further increasing the number of records keeps improving performance slightly. Again, for the larger datasets of this study, more records were needed. However, in absolute, FT might be diminishing the generalization capacity of the model by overfitting on the specificity of the target dataset.
One clue for that is that the largest datasets scored by several sleep experts in various sleep centers benefit from FT only with many records. This particularity suggests that they are hard to overfit on. CAP also benefits a lot from FT, supporting the previous observations that CAP is quite distinct from the other datasets in terms of variability. While not being the primary goal of this work, RobustSleepNet showed promising results in LFS and especially in FT. Future work should investigate its performance compared to state-of-the-art models more thoroughly in a comparative study such as the one realized in  [10]. One interesting axis of improvement for RobustSleepNet would be integrating the information related to the sensor positions, particularly the location of each EEG channel on the head.
CONCLUSION This work introduced an automated sleep staging model called RobustSleepNet, designed for real-life use. Its archi-tecture is close to previous deep learning methods such as SeqSleepNet and SimpleSleepNet; however, it differs from these by using attention layers to handle any arbitrary input shape. Thus, it can be used out of the box on any PSG montage. Using this property, we were able to train and evaluate the model in a leave one out dataset fashion on a corpus of 8 datasets. The model achieved a direct transfer performance close to the supervised one on most datasets. This performance is beyond any previous literature attempts; hence, it is likely to provide good results as is on unseen data. We highlight the need to train on populations with variability in age, montage and pathology to get the best performance on unseen data.
An important parameter is using sleep scoring from multiple sleep experts to reduce the noise due to interrater variability. We also demonstrated that using finetuning improves performances compared to learning from scratch especially on the smallest datasets. One could use it to target specific populations. Further improvements could be made to RobustSleepNet to handle the specificity of the targeted record's montage. Finally, one could extend the model to providing metrics on the estimated confidence of the automated sleep staging. ACKNOWLEDGMENT.
To compare with Dataset Nights Channels used Male (%) Age BMI AHI Other disorders TST (min) DOD-O 55 8xEEG, 2xEOG, EMG 63. 6 45. 6 ± 16. 5 29. 6 ± 6. 4 18. 5 ± 16. 2 - 387 DOD-H 25 12xEEG, 2xEOG, EMG 76. 0 35. 3 ± 7. 5 23. 8 ± 3. 4 ≤ 5 - 427 MASS 115 3xEEG, 2xEOG, EMG 53. 9 52. 2 ± 17. 8 - ≤ 20 - 426 Sleep-EDF 197 2xEEG, EOG 56 54. 7 ± 22. 6 - - INS(44), TEM(22) 426 MrOS 586 5xEEG, EOG, 2xEMG 100 81. 0 ± 4. 4 26. 9 ± 3. 9 17. 1 ± 15. 6 PD(16), AD(15), Narcolepsy(1), RLS(25), PLM(11) 341 MESA 2053 3xEEG, 2xEOG, EMG 53. 5 69. 6 ± 9. 2 - 16. 1 ± 13. 7 - 359 SHHS 2651 2xEEG, 2xEOG, EMG 45. 6 68. 3 ± 10. 5 28. 3 ± 5. 1 14. 6 ± 12. 0 NARC(9), INS(46), PLM(56) 371 CAP 106 2xEEG, EOG 61 45. 2 ± 19. 6 - - NFLE(40), PLM(10), RBD(22), BRUX(2), NARC(5), INS(9) 410.
We consider eight sleep staging datasets with variable demographics and PSG montages. Demographics of the datasets are reported in Table I. Dreem Open Datasets Obstructive (DOD-O): [5] Public dataset scored by five sleep-experts according to the AASM standards. DOD-O is composed of 55 subjects diagnosed with sleep apnea. EEG, EOG, EMG, and ECG derivations are available. 5]] πj III. EXPERIMENTS AND EVALUATION A. Experimental Setup 1) Datasets:.
section 2. 3). Several scenarios (different temporal dynamics, connectivity networks, additive trends) are investigated to assess the interest and relevance of the proposed Complex Wavelet indices (W-ICOH and W-wPLI) compared to Fourier-based ones in terms of estimation performance and robustness to trends. The proposed Complex Wavelet indices assessing functional connectivity from fractal dynamics are extensively tested on MEG data, collected on 36 individuals, both at rest and during a visual discrimination learning task. The experimental data are described in section 3 (see also  Zilber et al. , 2014). Analyses of functional connectivity assessed from fractal dynamics within infraslow arrhythmic cross temporal dynamics regime, ranging from 0. 1 to 1. 5 Hz for this data set  (La Rocca et al.
The proposed Complex Wavelet tools constitute, to the best of our knowledge, the first operational tools for a relevant assessment of functional connectivity from fractal dynamics, i. e. , functional connectivity in scale-free cross-temporal dynamics for the large-band infraslow arrhythmic brain activity recorded in M/EEG. MATLAB codes, designed and implemented by ourselves, for the synthesis of multivariate scale-free synthetic data and for the computation of Complex Wavelet-based indices to assess functional connectivity from fractal dynamics, will be made publicly available at the time of publication.
Sample size is n = 2 14 , chosen to match the size of the infraslow regime of the MEG data (cf. sections 3 and 4). To model MEG data as those analyzed in section 4 and as commonly indicated in the literature  (He et al. , 2010) , one needs to use both fractional Gaussian noise (fGn), the increments of fractional Brownian motion (fBm), with parameter H ranging from say 0. 6 to 1 and fractional Brownian motion itself with parameters ranging from 0 to 1. Therefore, the numerical simulations conducted here were based on bivariate processes, each component being either fGn or fBm, with 0 < H < 1. For the Fourier-based spectral estimation, the classical averaged windowed periodogram estimate of the power spectral density was computed, with Hanning windows of a width corresponding to the frequency bands of the complex wavelet filters, to enable relevant comparisons of the tools.
For the Complex-Wavelet based estimation, q-shift complex wavelets were used, as described in  Selesnick et al. (2005)  and references therein (see, e. g. ,  Lina and Mayrand, 1995  for an alternative choice). Indices assessing functional connectivity from fractal dynamics (both Fourier and wavelet-based) were computed as average over a range of frequencies and scales that match those of the infraslow scale-free range observed on the MEG data described and analyzed hereafter. Performances are reported as means (and confidence intervals) computed from N = 1, 000 independent realizations of bivariate fractional Gaussian noise.
EXPERIMENTAL MEG DATA The proposed complex wavelet-based assessment of functional connectivity in infraslow arrhythmic brain activity was tested on  MEG measurements, consisting of non-invasive recordings of simultaneous time-series reflecting the whole brain activity, both at rest and during the completion of a task. All details about the experimental paradigm and the task can be found in  Zilber et al. (2014). In short, the task was designed from a short-term learning paradigm and consisted of visual coherence discrimination. Two sets of colored (green and red) dots were mixed and shown on a screen, each dot with random and independent movement. After a variable duration interval (0. 3-0. 6 s) of incoherent motion, a fraction of randomly chosen dots belonging to either of the two sets (also randomly chosen at each trial) followed a coherent motion during 1 s.
Functional Connectivity From Fractal Dynamics Assessment At the methodological level, the results presented in section 4 clearly showed that W-COH fails to characterized correctly functional connectivity, which is in clear agreement with the numerical simulations reported in section 2. 3 on synthetic data fGn/fBm and with results reported in the literature (cf. Stam et al. , 2007; Vinck et al. , 2011). More interestingly, compared to W-ICOH, W-wPLI was observed to more accurately quantify functional connectivity assessment from fractal dynamics, both at rest and during a task in MEG data, as well as to better highlight relevant changes in functional connectivity assessed from fractal dynamics between rest and task. This is in agreement with previously reported results, showing that for band-limited oscillatory activities, F-wPLI was a better index to assess functional connectivity than F-ICOH.