PedSleepMAE: Generative Model for Multimodal Pediatric Sleep Signals

Authors: st  saurav
raj  pandey
Venue: IEEE_org_EMBS_BHI_2024_Conference
Published: 2024-06-14 09:58:39

Abstract: Pediatric sleep is an important but often overlooked area in health informatics. We present PedSleepMAE, a generative model that fully leverages multimodal pediatric sleep signals including multichannel EEGs, respiratory signals, EOGs and EMG. This masked autoencoder-based model performs comparably to supervised learning models in sleep scoring and in the detection of apnea, hypopnea, EEG arousal and oxygen desaturation. Its embeddings are also shown to capture subtle differences in sleep signals coming from a rare genetic disorder. Furthermore, PedSleepMAE generates realistic signals that can be used for sleep segment retrieval, outlier detection, and missing channel imputation. This is the first general-purpose generative model trained on  multiple types of pediatric sleep signals.

Main paper: 
 i. introduction: 
 Sleep is vital to the health and well-being of infants, children, and adolescents. Sleep disorders and disturbances can affect childhood neurobehavioral development and cognitive functions and even cause morbidity in severe cases of obstructive sleep apnea (OSA) [1] . However, pediatric sleep has been often overlooked in health informatics, partially due to underreporting, underdiagnosis [1] and lack of public datasets. Most importantly, there is a common misconception that pediatric sleep is not distinct from adult sleep, when in fact children have unique physiology and therefore must be studied separately from adults, including for computational models. For example, [2] found that popular sleep stage classification models based on adult data struggle to generalize well on patients younger than 10 years old.The clinical gold standard for sleep medicine is currently based on polysomnography (PSG) or an overnight sleep study. Since sleep is a complex process that involves multiple organ systems, PSG data encompass many channels and modalities, such as electroencephalography (EEG), respiratory signals and sometimes videos. During PSG, a sleep technician monitors the signals and manually annotates events such as movement, coughing, breathing changes and sleep stages (e.g. REM). Compared to adult counterparts, software for automated detection of such sleep events is not as widespread (or verified and trusted) in pediatric sleep clinics, making this a laborintensive process. Combined with the rich PSG data, this strongly motivates the use of machine learning in pediatric sleep research.In the past few years, deep learning-based generative models and self-supervised learning (SSL) have come to the forefront of machine learning. Unlike supervised learning, SSL trains the model to do auxiliary tasks on unlabeled data (e.g. predict whether an image is rotated). Once informative embeddings or features are learned, SSL only then uses labeled data for downstream classification or regression tasks of interest. This approach can fully leverage large unlabeled datasets and have been shown to generalize well to unseen data, beating supervised models in some cases [3] . Considering the cost of acquiring gold standard labels in medicine, SSL is a particularly promising approach in health informatics [4] .This work presents PedSleepMAE (Pediatric Sleep Masked Autoencoder), a generative model for multichannel pediatric PSGs. To our best knowledge, PedSleepMAE (Fig. 1 ) is the first to satisfy all of the following characteristics:• Explicitly focuses on pediatric sleep and leverages large public data collected in real clinical setting. • Handles multiple types and channels of PSG signals, e.g.beyond EEGs [5] , [6] and can be flexible with missing channels. • Trains a transformer-based model via SSL such that the model can be used for multiple tasks, e.g. beyond sleep scoring [5] or apnea detection [7] . Our experiments are carefully designed to demonstrate interesting potential use cases of the PedSleepMAE framework:• Automated sleep event detection and sleep scoring.• Generate hypotheses for sleep biomarkers in conjunction with the electronic health record (EHR). We utilize the Nationwide Children's Hospital (NCH) Sleep DataBank [8] , which is a large, public, and fairly recent pediatric PSG dataset collected in a real clinical setting. Within this dataset, we analyzed 2,379 PSGs that had 16 of the most common channels [8, Table 2] . This includes 7 channels of EEG (C3-M2, O1-M2, O2-M1, CZ-O1, C4-M1, F4-M1, F3-M2), CO 2 level (CAPNO), O 2 level (SpO2), breathing effort (RESP Thoracic, RESP Abdominal), snoring (SNORE), CPAP airflow (C-FLOW), electrooculogram (EOG; LOC-M2, ROC-M1) and electromyogram (EMG; CHIN1-CHIN2). These PSGs are also accompanied by patients' electronic health records (EHR) and annotations such as sleep stages (Wake, Non-REM 1, Non-REM 2, Non-REM 3, REM), EEG arousal, apnea, and hypopnea for every 30 seconds of sleep. PSG is considered in units of 30 seconds, recorded (or downsampled) at 128Hz across 16 channels. We normalize all channels to 0 mean and 1 standard deviation for stable model training.Masked autoencoder (MAE) [9] is a state-of-the-art deeplearning-based generative model. It is based on the transformer neural network architecture [10] and has a encoderdecoder structure. During SSL, input data is divided into small patches and then randomly masked, and the MAE is trained to reconstruct the masked patches. This is similar to masked language modeling in natural language processing, e.g. BERT [11] . During this reconstruction process, the encoder learns informative non-linear features or embeddings, and the decoder learns how to generate data back from the embeddings. MAEs are distinct from traditional autoencoders, which focus on data compression, and are much easier to train than variational autoencoders (VAEs) or generative adversarial networks (GANs) since there are no regularization terms.Although MAE is originally proposed for images and videos, we easily adapted it to a multichannel signal setting for PSGs. Both encoder and decoder module consist of three layers of Vision Transformer [12] attention blocks with four attention heads. The attention mechanism allows the model to capture both intra-and inter-channel relationships in PSG. We used a masking ratio of 50% for SSL, and the dimension of the embeddings used in our experiments is 7,680. Appendix A describes the model and pretraining strategy in greater detail.We evaluate how much diagnostic information is in the 7680-dimensional embeddings from the PedSleepMAE encoder. The PSGs in NCH Sleep DataBank are accompanied by rich EHR and annotated with clinician-verified sleep events. We fully utilize these clinical labels to qualitatively and quantitatively measure how well these groups are separated in the embedding space.We first employ Uniform Manifold Approximation and Projection (UMAP) [13] to reduce the embeddings into 2 dimensions and visualize them in Fig. 2 . Each plot corresponds to one PSG, and each point represents 30 seconds of sleep, colored by different sleep events. PSGs are selected as follows: one PSG chosen randomly from all patients (Fig. 2a ), one PSG from the top 5 with the highest apnea occurrences (Fig. 2b), and one PSG from those with 5 to 30 cases of hypopnea (Fig. 2c ). This was to avoid cherry picking while still ensuring that there are enough sleep events to be plotted and also showing some breadth in our results. For readability, we defer UMAP plots of samples colored by oxygen desaturation and EEG arousal to Figs. 10 and 11 in Appendix B. Figs. 2, 10 and 11 reveal clusters in most cases. We emphasize that PedSleepMAE was able to capture characteristics of various sleep events and distill it into the embeddings despite not having seen any of these labels during pretraining. We also applied UMAP to visualize randomly selected embeddings from multiple PSGs (Fig. 12 in Appendix B). Unlike with earlier per-PSG plots, this process did not yield distinct clusters as anticipated. We hypothesize that due to high variability and confounding variables among patients, such as differences in age, sex and underlying health conditions, clusters are not distinct enough to be seen in 2-dimensional UMAP space.To more quantitatively measure the separation of embeddings per sleep event (which might not be visible in 2dimensional UMAP space), we designed 6 downstream classification tasks. They are 5-stage sleep scoring and 5 binary detection problems: oxygen desaturation, EEG arousal, apnea, hypopnea, and a combined case of apnea and hypopnea. We froze PedSleepMAE, then trained linear classifiers on top of the embeddings. The classifiers are intentionally kept as simple as possible so that we can measure the diagnostic information in the embeddings and not the classifier. See Appendix C for more details on experiment setup.Table I and Fig. 3 present the overall accuracy, F-1 score, AUC score and confusion matrix on the test sets. For sleep scoring, we report the weighted F-1 score and weighted AUC using a one-vs-rest (OvR) approach. Our F-1 scores for all binary classification tasks are consistently higher than the corresponding percentage of positive cases, which represents the random baseline F-1 score. Considering that there was no fine-tuning of PedSleepMAE, and the classifiers are simply fitting a hyperplane in the embedding space, the classification is successful in all cases. This is comparable to 76% sleep scoring accuracy reported by [5] and 66.8% oxygen desaturation accuracy from [6] , even though they used supervised learning. However, it should be noted that those works only used 7 EEG channels, and we cannot solely rely on accuracy due to the high class imbalance. Also while they used the same dataset as us, there is variability in which subset of PSGs ended up being included in the analysis. Still, these results strongly suggest that PedSleepMAE embeddings contain information relevant to sleep health.We now move beyond 30-second sleep events and demonstrate that PedSleepMAE embeddings can capture more nuanced sleep characteristics that may not be even fully elucidated yet by science. Following the case study in [8] , we focus on Prader-Willi syndrome (PWS) using the EHR. For context, PWS is a rare genetic disorder that approximately affects 1 out of 10,000 to 30,000 people, and there has been observations of breathing abnormalities and sleep disorders of PWS patients [14] . Note that compared to earlier labels, For this experiment, we decide against training a linear classifier as in Sec. III-B. Automatic detection of apnea, for example, is a meaningful undertaking on its own beyond the fact that it can demonstrate the usefulness of our embeddings. But predicting PWS diagnosis has no clinical use at the end of the day. So instead we perform a cluster analysis on PWS and non-PWS cohorts using silhouette score [15] . Originally used to evaluate clustering algorithms, silhouette score measures how similar a point is to its own cluster compared to other clusters. In general, higher silhouette score indicates better clustering with +1 being perfect, and 0 being that all samples lie on the decision boundary. Silhouette scores are known to suffer from the curse of dimensionality [16] , but we sidestep this issue by interpreting scores only relative to each other. We borrow this metric to measure the separation between PWS and non-PWS sleep in PedSleepMAE embedding space.Cohort 1 has 9,600 PedSleepMAE embeddings from PWS patients, and Cohort 2 has 279,236 embeddings from obese but non-PWS patients. We control for the effect of OSA as in [8] . For computational feasibility, we randomly select 2,500 embeddings from each cohort to compute a silhouette score [17] . This sampling process is repeated 100 times, which is then used to calculate the 95% confidence interval (CI) of the true silhouette score. Same analysis is performed 20 more times with randomly shuffled clusters. The 21 CIs are presented in Fig. 4 .We initially hypothesized that the embeddings would have captured small differences in sleep characteristics that are relevant to PWS. If the features are not related to the PWS labels at all (i.e. under the null hypothesis), the silhouette score calculated from PWS/non-PWS clusters should be indistinguishable from the score calculated from any random two clusters. Therefore, the non-overlapping CIs in Fig. 4 provide evidence supporting our hypothesis. We also conducted a Welch's t-test comparing the true PWS/non-PWS cluster with While the effect size is small, the statistical significance is robust. Furthermore, since the exact effect of PWS on the sleep signals are not fully understood yet, using generative models to identify potential biomarkers could be an exciting avenue of future research.Now that we have established the usefulness of the embeddings, we bring the decoder back into the picture. In this section, we demonstrate how the decoder can be used to retrieve or generate representative examples for a patient, as well as impute missing channels.Seminal works in computer vision (e.g. [18] ) and natural language processing (e.g. [19] ) have shown that with good generative models, arithmetic operations in the embedding space could lead to analogous operations in the original data space. For example, one could take the Euclidean average of face embedding vectors and push it through the decoder to create an "average face image." In comparison, taking the average in the pixel space will not result in a valid face image. Borrowing this idea, we use PedSleepMAE to retrieve and generate representative examples of a patient's sleep.First, we investigate whether pairwise relationships in the embedding space are maintained in the generated signal space. We check this by measuring pairwise Euclidean distances in both spaces and calculating Pearson's correlation coefficient (ρ). ρ ranges from -1 to 1, where 1 indicates a perfect positive linear relationship. Fig. 5 shows a strong positive correlation of 0.93 based on 1,000 samples from a randomly selected patient. While ρ is calculated using all pairwise distances, only 2,000 pairwise distances are plotted to ensure readability. To make sure this was not an outlier patient, we repeated this experiment on 1,000 samples randomly selected from all patients, which yielded ρ = 0.88 (Fig. 14 in Appendix D ). We Given this promising quantitative result, we next tackled representative signal generation and retrieval. Let's consider a single sleep stage (REM). We begin by acquiring the embeddings for all REM sleep from a randomly selected PSG, then take the Euclidean average to generate an "average embedding." This vector is then pushed through the decoder to generate an "average 30 seconds of REM sleep" for this PSG (Fig. 6 ). Albeit noisy, they seem to be valid signals with the right shape, which is not the case if we take the Euclidean average in the original signal space. These synthetic signals can circumvent privacy concerns and be useful toy data in classrooms or be used to develop code quickly in fast-paced projects while waiting for the real medical data to arrive.Of course, a clinician is unlikely to accept this set of AIgenerated signals to make clinical decisions. But now that we have a notion of mean and distance, we can perform a nearest neighbors search to retrieve an actual data sample that is closest to the mean (Fig. 7 ). On the flip side, we can also find outliers and bring them to the doctor's attention.Lastly, we showcase a PedSleepMAE-generated "average 30 seconds of sleep with apnea" in Fig. 8 . We used a random PSG with at least 100 apnea occurrences to calculate the average embedding. A notable finding is the decreasing SpO2 values and increasing CAPNO levels over time, as well as the fluctuations in C-Flow and RESP abdominal. In patients with apnea, there are frequent interruptions in breathing during sleep, which can lead to periods of reduced blood oxygen levels and increased CO 2 levels. During this time, the amount of air flowing from the CPAP machine (if the patient has one) and the effort their body puts into breathing (as measured by elastic bands) may also fluctuate. This event is clearly reflected in Fig. 8 . Moreover, the timings are coordinated across channels, which suggest that PedSleepMAE has learned accurate inter-channel information about apnea. Our final experiment is on using PedSleepMAE for missing channel imputation. Recall that during SSL, the model is trained to reconstruct small patches of size 8 (≈ 0.0625 seconds). Generating 30 seconds is a much more difficult task. We systematically remove one channel at a time and let PedSleepMAE reconstruct the missing signal based on the other 15 channels. Table II reports the reconstruction error across 5,000 samples as measured by MSE and DTW. For comparison, a simple mean imputation should theoretically give an MSE of 1 in expectation, because we normalized each channel to 0 mean and 1 standard deviation (SD) in the beginning. It is difficult to make a definite statement due to high SD, but it is promising that the MSE values are much lower than that baseline. We also show an example of the reconstruction for channel EEG F3-M2 in Fig. 9 . While there are differences in the amplitude of the signals, the reconstructed signal maintains a similar shape.We designed PedSleepMAE, a masked autoencoder (MAE) trained on a large set of pediatric sleep signals. Our transformer-based model learns informative representations of the multimodal data, which includes EEGs, respiratory signals, EOGs and EMG. Our extensive evaluations were both qualitative and quantitative, and we demonstrated a wide range of interesting use cases from apnea detection to information retrieval to data imputation.While writing this paper, we came across a concurrent work [20] that built a foundation model on an impressively whether we use Euclidean or dynamic time warping (DTW) distance, or whether we consider a single random patient or random samples from across all patients, pairwise similarities in embedding space are preserved in the generated signal space. We used fast implementation of DTW by [23] and calculated Pearson's correlation coefficient with SciPy [24] . These results provide justification for generating "average" or representative signals using PedSleepMAE. Additionally, strong linear correlation suggest that a nearest neighbor search done in either the embedding space or the generated signal space would give similar results. Therefore our nearest neighbor search (Fig. 7 ) is done in the generated signal space. 
 
 Appendix:  



Reviews: review: title: title: Review: PedSleepMAE: Generative Model for Multimodal Pediatric Sleep Signals
paper_summary: paper_summary: This manuscript presents an autoencoder based generative model for the following objectives:

- Self-supervised sleep event classification tasks, slightly poorer performance compared to supervised model
- Capture characteristics Prader-Willi Syndrome via subtle differences in sleep signals
- Generate sleep signals for segment retrieval, outlier detection, and data imputation.
strength_weakness: strength_weakness: - For sleep event classification tasks, this manuscript present a result with high accuracy and AUROC but ***low F-1 score (6 out of 7 binary tasks have F-1 score lower than 55%)***, indicating the model is barely outperforming random baseline in terms of correctly classifying ‘positive’ cases. In this case of highly imbalanced dataset, good performance in accuracy and AUROC does not align with the objective of classifying positive cases.
- In *Fig. 4*, the silhouette scores from PWS/non-PWS clusters shows a CI around 0.006. This improvement is insignificant in a scale of 0-1 and should not be considered as strong evidence supporting the hypothesis.
    
    > Therefore, the non-overlapping CIs in Fig. 4 provide strong evidence supporting our hypothesis.
    >
questions: questions: - It is difficult to assess the quality of prediction via MSE and DTW given in Table II, without normalizing the signal, provide a percentage error, or provide a comparison against other state-of-art studies, especially when the signals in different channels are not of the same scale. Could the authors provide a MSE metric with normalized signals?
title: title: PedSleepMAE: Generative Model for Multimodal Pediatric Sleep Signals
paper_summary: paper_summary: PedSleepMAE is a generative model that effectively integrates multimodal pediatric sleep signals, including EEGs, respiratory signals, EOGs, and EMG, using a masked autoencoder framework. It matches supervised learning models in accuracy for sleep scoring and detecting various sleep abnormalities. The model’s embeddings identify subtle differences associated with rare genetic disorders and it generates realistic signals for applications such as sleep segment retrieval and missing data imputation. This is the first general-purpose generative model for pediatric sleep signals.
strength_weakness: strength_weakness: Overall, the manuscript is well-written and presents an important work. The model is well-described but lacks a comparison with some more recent models and methods. A comprehensive comparison would further strengthen the manuscript. The description for Figure 2 should be revised a little to improve user understanding.
questions: questions: In the statement, "Each point represents 30 seconds of sleep, and each plot corresponds to one PSG," what is the definition of 'plot'? It appears that the plot may represent data from all PSGs (since the subsequent description mentions that Fig. 2b shows the top 5 PSGs with the highest occurrences of apnea).

The authors note that while writing this paper, they encountered a concurrent study [20] that developed a foundation model using a notably large private PSG dataset with similar modalities. Including a comparison of the methodologies from this concurrent work in the introduction section would provide valuable context and enhance the study's positioning within the field.
title: title: PedSleepMAE: Generative Model for Multimodal Pediatric Sleep Signals
paper_summary: paper_summary: The paper introduces PedSleepMAE, a generative model using a masked autoencoder to analyze pediatric sleep data from multichannel EEGs, respiratory signals, EOGs, and EMG. The model performs comparably to supervised learning models in tasks like sleep scoring and apnea detection. Moreover, it captures subtle differences in sleep signals, even identifying characteristics associated with rare genetic disorders and can generate realistic signals for applications such as sleep segment retrieval and missing channel imputation.
strength_weakness: strength_weakness: Despite its strengths, PedSleepMAE faces several challenges. The variability and inconsistency in EHR data, stemming from differences in recording and annotation practices across clinical settings, can impact the model's generalizability to other datasets or environments. Additionally, the model's reliance on a specific dataset, the NCH Sleep DataBank, raises concerns about its ability to generalize findings to broader pediatric populations. Privacy and security issues related to the sensitive patient information contained in EHR data are also significant concerns. Furthermore, the high dimensionality of the model's embeddings complicates visualization and interpretation, with techniques like UMAP not always producing clear clusters. The richness of EHR data may introduce confounding variables unrelated to sleep disorders, potentially hindering the model's ability to accurately identify relevant patterns and biomarkers.
questions: questions: 1. Could you explain how the Masked Autoencoder was adapted for one-dimensional data? 
2. How did you handle the input channels? Were the embeddings extracted by processing each channel independently?
3. Have you considered evaluating your method on external and unseen datasets?
4. Could you provide a more detailed explanation of the detection process within your model?
5. Given the imbalanced data, would it be possible to include the Cohen's Kappa metric in your evaluation?
6. Did you apply any post-processing techniques to the generated signals to mitigate noise?
7. Which aspect of the experimental design posed the greatest challenges for you?
8. Is it possible for the proposed tasks (sleep scoring and events detection) to be performed simultaneously? Or do these tasks need to be pre-specified and performed one-by-one?