SleepMG: Multimodal Generalizable Sleep Staging with Inter-modal Balance of Classification and Domain Discrimination

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-19 09:31:25

Abstract: Sleep staging is crucial for sleep tracking and health assessment. Polysomnography (PSG), containing multiple modalities such as electroencephalography, electrooculography, electromyography, and electrocardiography, is the fundamental means of sleep staging. 
However, due to performance differences in both classification and domain discrimination across modalities in PSG, existing domain generalization methods face a dilemma of modal imbalance. To balance inter-modal differences and achieve highly accurate cross-domain sleep staging, we propose SleepMG, a Multimodal Generalizable Sleep staging method. SleepMG assesses the classification and domain discrimination performances of each modality and further defines the modal performance metrics by calculating the variance between the performance score and the average performance of each modality. Guided by these metrics, the gradients of the classifier and domain discriminator are adaptively adjusted, placing greater emphasis on poorly-balanced modalities while reducing emphasis on well-balanced modalities. Experimental results on public sleep staging datasets demonstrate that SleepMG outperforms state-of-the-art sleep staging methods, effectively balancing multiple modalities as evidenced by the visual experiment of modal imbalance degree. Our code will be released after formal publication.

Main paper: 
 introduction: 
 Sleep staging [54] is critical for health assessment and intervention, providing critical information on sleep quality of subjects and assisting in screening for brain and neurological health [5, 62, 12, 45] . Polysomnography (PSG) is multimodal physiological signals collected synchronously from different positions of the subject during sleep. PSG-based sleep staging [51, 25] and epilepsy. To ensure consistency, researchers mostly followed the American Academy of Sleep Medicine (AASM) [4] standard, classifying sleep into five stages for conducting studies. Initially, researchers manually staged sleep from the perspective of signal processing and medical knowledge based on the AASM standard. With the rise of deep learning, researchers began to employ automatic feature learning [51, 13, 46] to solve the time-consuming and labor-intensive problem in sleep staging. In sleep research, Convolutional Neural Networks (CNNs) were employed to automatically capture the spatial patterns in different sleep stages [48, 15] and distinctive features in different frequency bands [53] , Recurrent Neural Networks (RNNs) [7, 49] and Long Short-Term Memory Networks (LSTMs) [63, 40] were employed to learn the dynamic patterns of signals [7, 49, 63] and capture the discriminative features at different time points [40] . Graph Convolutional Networks (GCNs) [20] were employed to model the association patterns between EEG channels. Sequence-to-sequence models [36, 38] were employed to capture the temporal dependencies between sequences and then learn distinguishing features in sleep staging tasks. Furthermore, the attention mechanism [42, 65, 10] was introduced to capture the differentiating features based on the relative importance of different sleep epochs.However, as depicted in Figure 1 , modal differences exist when multiple modalities of PSG jointly represent the sleep state. To explore the differences, modality-by-modality feature learning and fusion methods [17, 19, 60] have been designed to capture the differentiated features across modalities. Moreover, researchers have developed attention-based fusion techniques [60, 8] to enable multimodal models to focus on more important modalities. However, the attention mechanism amplifies the dominance of stronger modalities, exacerbating the inherent imbalance [35, 11] among them. This imbalance further hinders the model from fully utilizing the potential of all modalities. To balance modalities and make each modality as optimal as possible, Peng et al. [35] balanced two modality-specific feature extractors according to the classification performance of each modality. However, it is regrettable that they compared the relative performance of two modalities, making it suitable only for balancing between two modalities. Additionally, balancing modality-specific feature extractors may interfere with the model to decouple and learn modality-related information, a more pronounced phenomenon in multimodal balancing scenarios. Furthermore, PSG signals exhibit severe subject-dependency and existing generalizable sleep staging methods aim to enhance the cross-subject generalization capability of the model. The pretraining and fine-tuning method [37, 50] in transfer learning was introduced to do the subject personalized calibration. Unfortunately, the method requires significant data and some labeled data from the target subjects, posing inconvenience. As a result, the domain generalization method [57] was integrated. For example, domain adversarial methods [18, 29] were introduced to confuse the discrimination of the domain (i.e., the subject) by backpropagating the negative gradient of the domain discrimination loss, thereby allowing the model to learn domain-invariant information that is beneficial to unseen-domain generalization. Additionally, explicit feature alignment methods [39, 50] were introduced to learn shared feature representations across domains, improving the performance of the model on new fields or unseen data. However, as depicted in Figure 1 , modal differences are not only reflected in classification performance, [6, 27] , but also in domain discrimination performance [33, 24, 59 ]. The classification performance varies across different modalities [41] , and it differs even more within different subject domains. The above methods enhance the cross-domain generalization of the model. However, none consider the inter-modal difference of domain generalization capability [33, 59] , which is also crucial for sleep staging.To tackle these challenges, we introduce SleepMG, a novel multimodal generalizable sleep staging method through classification and domain discrimination balancing. SleepMG first quantifies and balances the modal differences in classification and domain discrimination. SleepMG integrates domain adversarial learning into the multimodal feature learning framework and explicitly distinguishes the classification and domain discrimination performances of each modality through a globally shared classifier and discriminator separately. Guided by modal performance metrics, the gradients of model of the classifier and domain discriminator are adaptively adjusted, balancing the emphasis of the model on well and poorlybalanced modalities. SleepMG achieves modality-balanced and improves the multimodal generalizable sleep staging.To summarize, our contributions are as follows:• We construct a multimodal generalizable sleep staging model and first introduce metrics to assess the classification and domain discrimination performances of multiple modalities. • By leveraging modal performance metrics, the gradients of the classifier and domain discriminator are adjusted adaptively, increasing greater emphasis on poorly-balanced modalities while reducing emphasis on well-balanced ones. This adaptive adjustment achieves balance across multiple modalities in two critical aspects. • Extensive experiments on public datasets show that our proposed SleepMG achieves state-of-the-art sleep staging results, demonstrating its effectiveness in achieving multimodal balance and cross-domain generalization. 
related work: 
 This section reviews the related work from two perspectives: sleep staging and generalizable sleep staging.Sleep staging is crucial in assessing sleep quality and identifying potential neurological disorders [45] . Various methods have been developed to classify sleep stages based on PSG. Early studies relied on manual sleep staging, where experts manually classified sleep stages based on signal processing techniques and medical knowledge, following the AASM standard [3] . The AASM standard categorizes sleep into three main stages: the Wake stage, the Non-Rapid Eye Movement (NREM) stage, and the Rapid Eye Movement (REM) stage. The NREM stage is further divided into three sub-stages: N1, N2, and N3. These stages correspond to different degrees of sleep, resulting in five distinct categories. This manual approach is time-consuming and labor-intensive, prompting researchers to turn to deep learning techniques for automated sleep staging. Researchers utilized CNN [48, 15] to capture local spatial information in PSG data. Additionally, researchers have utilized RNNs [7, 63] and LSTM [48, 44] models to capture the sequential dependencies and temporal dynamics in PSG data. To model the correlations between different channels in PSG data, Jia et al. [20] introduced GCN-based GraphSleepNet, which represents PSG data as a graph to capture complex relationships. To handle the multi-level nature of PSG data, Phan et al. [36, 38] proposed sequence-to-sequence models to capture temporal dependencies across sleep epochs. PSG includes multiple modalities collected from subjects, such as electroencephalography (EEG) [9, 14, 28, 22] , electrooculography (EOG) [21] , electromyography (EMG) [58] , and electrocardiography (ECG) [32] . To explore the difference and consistency across modalities, researchers have designed modality-specific feature learning [52] modules and corresponding fusion modules [61] . Xiang et al. [58] designed a spatial encoder to capture the high-level shared semantic information between EEG and EMG. Considering the differences of importance across modalities, attention mechanisms have been incorporated into sleep staging models to help the model focus on more important modalities or parts. Zheng et al. [60] also used the attention mechanism to help learn the interdependencies across modalities. However, there are differences across modalities, and attention-based fusion can exacerbate the intermodal imbalance, resulting in insufficient learning of each modal and limiting the classification capability of multimodal models.PSG, as a physiological signal, is highly subject-dependent [34] , and the data distribution across subjects varies significantly. In this paper, we explore methods to enhance the out-of-distribution generalization of sleep staging models, aiming to construct a general automated sleep staging model that can be applied across subjects (i.e., domains). We refer to this model as a generalizable sleep staging model [30, 26, 43] . To achieve this, researchers have introduced transfer learning methods. A common approach is the pretraining-finetuning method [26, 56] . Wang et al. [56] trained on a large sleep dataset, MASS, to learn general sleep-related representations, and then fine-tuned on a smaller sleep dataset, sleep-edf sub-dataset. While this method has demonstrated effectiveness, it is not without its drawbacks. The pretraining phase requires a large dataset and substantial computational resources. The finetuning phase requires a small amount of labeled data, impractical in crossdomain application scenarios such as healthcare. Banluesombatkul et al. [1] used meta-learning for pretraining, which improved data dependency and task generalization to some extent. However, the method increased the computational cost during the pretraining phase. The finetuning phase still requires a small amount of labeled data, making cross-domain generalization unachievable. Tang et al. [50] addressed domain adaptation by explicitly aligning the feature distributions between the source and target domains, removing the reliance on labels in the target domain. However, it still requires data from the target domain and does not solve the cross-domain generalization problem.To overcome these limitations, researchers integrated the domain generalization method. For example, some researchers incorporated domain adversarial learning [18, 29] into their model. The domain discriminator in domain adversarial played a crucial role in enhancing the cross-domain generalization performance of the model by intentionally confusing the capability of the model to identify different domains. The domain adversarial method has achieved considerable generalization performance. However, there is no specific domain adversarial technique for multimodal sleep staging. Moreover, differences exist across modalities, not only in terms of classification capability but also in domain generalization capability. Quantifying and balancing these two types of inter-modal differences is necessary. 
preliminary: 
 The training set is D 𝑡𝑟𝑎𝑖𝑛 = {(𝑥 𝑖 , 𝑦 𝑖 , 𝑑 𝑖 )| 𝑖 ∈ {1, . . . , 
the proposed sleepmg: 
 The pipeline of SleepMG is shown in Figure 2 . We introduce SleepMG from three aspects: 1) Naive multimodal generalizable sleep staging (NaiveMG), the backbone of SleepMG. 2) Modal performance metrics of classification and domain discrimination. 3) Metric-guided adaptive modality balancing and generalizable sleep staging.To realize SleepMG, we first conduct the naive integration of modalityspecific feature learning and domain adversarial methods, named the Naive Multimodal Generalizable method (NaiveMG). Specifically, NaiveMG employs FeatureNet [18] , a dual-scale CNN model to establish each modality-specific feature extractor. These feature extractors capture the unique characteristics of each modality. In addition, NaiveMG fuses extracted features and classifies sleep stages using the simple fully connected classifier 𝐺 𝑦 . To further enhance cross-domain generalization, we set domain discriminator 𝐺 𝑑 with the same simple fully connected to confuse the identification of the model about domains by back-propagating negative gradients.By incorporating the above techniques, NaiveMG significantly enhances the cross-domain generalization capability of multimodal sleep staging models, enabling more accurate and reliable sleep stage classification across domains. The classification loss L 𝑦 and domain discrimination loss L 𝑑 are both calculated using the crossentropy loss function. Details are as follows:EQUATIONwhere 𝐺 𝑎 (𝑓 𝑖 ) = [𝐺 𝑎 (𝑓 𝑖 ) [1] , 𝐺 𝑎 (𝑓 𝑖 ) [2] , . . . ,EQUATIONUtilizing NaiveMG as the backbone, SleepMG further quantifies inter-modal differences in classification and domain discrimination. where𝐺 𝑎 ( ∼ 𝑓 𝑢 𝑖 ) = 𝐺 𝑎 ( ∼ 𝑓 𝑢 𝑖 ) [1] , 𝐺 𝑎 ( ∼ 𝑓 𝑢 𝑖 ) [2] , . . . , 𝐺 𝑎 ( ∼ 𝑓 𝑢 𝑖 ) [𝐾 𝑎 ]represents the list of softmax predicted output of the modality 𝑢, ∼ 𝑓 𝑢 𝑖 represents the feature of single modality 𝑢 that has been zeropadded to match the length of multimodal fusion feature 𝑓 𝑖 .Values of Each Modality. Following [35] , we calculate the sum of correctly predicted category probabilities from the softmax output as two performance values at the 𝑡 time step of each modality. The classification and domain discrimination performance values of each modality are as follows:EQUATION)where 𝑡 is the time step, and 1 is the equality function that returns 1 if the inputs are equal and returns 0 otherwise. (𝑠 𝑎,𝑢 𝑡 -1 𝑈 ) 2 𝑈 𝑢=1 (𝑠 𝑎,𝑢 𝑡 -1 𝑈 ) 2 (6)where 𝑡𝑎𝑛ℎ is the active function for smooth, and the sum scaling method is employed for normalization.We backpropagate the gradients 𝑔 EQUATIONEQUATIONwhere 𝑔 𝑎,𝑢 𝑡 denote the gradient of modality-specific parts of 𝜃 where 𝑡 represents the time step, 𝜂 represents the learning rate, 𝜖 1 represents a small constant to prevent division by zero. By setting a larger 𝜌 𝑎,𝑢 𝑡 for the gradient 𝑔 𝑎,𝑢 𝑡 of models 𝜃 𝑎,𝑢 𝑡 with poorly-balanced modality in classification and domain discrimination performance and a smaller 𝜌 𝑎,𝑢 𝑡 for the gradient 𝑔 𝑎,𝑢 𝑡 with well-balanced modality, we achieve a multimodal generalizable sleep staging model that is modality-balanced. Notably, Since the gradients backpropagated through the domain discriminator undergo the same reversal, the balance degree of modalities remains unaffected.The core of SleepMG is elucidated in Algorithm 1. SleepMG aims to calculate the performance of multiple modalities' classification and domain discrimination, ensuring the inter-modal balance in both classification and domain discrimination and achieving multimodal generalizable sleep staging. The backbone is naiveMG, which includes multimodal feature learning and domain adversarial methods. Multimodal PSG is first processed through modality-specific feature extractors 𝐺 𝑢 𝑓 to extract features and fuse. Subsequently, domain adversarial methods are integrated to improve cross-domain generalization. The model is optimized by minimizing classification loss L 𝑦 and maximizing domain discrimination loss L 𝑑 . By the early convergence stage, we assess the classification and domain discrimination capabilities of each modality by zero-padding their features and utilizing the shared classifier and discriminator. Additionally, the parameters 𝜃 𝑓 ,𝑢 𝑡 of 𝐺 𝑢 𝑓 are updated with minimizing L as usual. Furthermore, we calculate modal performance metrics of classification and domain discrimination, leveraging these metrics to influence the gradient update of parameters 𝜃 𝑎 𝑡 of 𝐺 𝑎 with minimizing L. To achieve modality balance, the gradients backpropagated by modality-specific parts 𝜃 𝑎,𝑢 𝑡 that correspond to the poorly-balanced modalities are relatively enhanced with large 𝜌 𝑎,𝑢 𝑡 , while those that correspond to the well-balanced modalities are relatively reduced with small 𝜌 𝑎,𝑢 𝑡 . The balanced multimodal generalizable sleep staging model is obtained when all the models converge.  
experiments: 
 All experiments are implemented with Python 3.8.5 and Pytorch 1.7.1. We conduct them on a computer server with 640GB RAM and two NVIDIA RTX A5000 GPUs with 24GB VRAM each.We evaluate SleepMG on two public datasets: ISRUC-S3 [23] , and MASS-SS3 [31] . For a fair comparison, we remove the Movement and Unknown stages and merge the N3 and N4 stages into a single N3 stage, resulting in five categories of Wake, N1, N2, N3, and REM sleep stages according to the AASM standard. Each PSG recording is downsampled to 100 Hz and divided into 30-second epochs.ISRUC-S3 collects PSG from 10 subjects (one male and nine female) over 8 hours in a single night, containing 8589 sleep epochs. We select ten of the 12 channels and remove the leg EMG far from the brain. It comprises four modalities: six-channel EEG, twochannel EOG, one-channel chin EMG, and ECG.MASS-SS3 collects PSG from 62 subjects (28 male and 34 female) over 8 hours in a single night, containing 59304 sleep epochs. We select the same 10 channels with four modalities as ISRUC-S3.In the experiment, we utilize FeatureNet [16, 18] as the backbone of NaiveMG and SleepMG. We employ the leave-one-subject-out method to divide the dataset into five domains with different subjects and perform five-fold cross-validation to split the data into training and testing sets. From the training set, we further allocate 20% as a validation set. The best model based on the validation set is saved and evaluated on the cross-domain testing set. The final reported results represent the overall evaluation of the entire dataset. We employ the Adam optimizer for model optimization with a learning rate of 0.001, attenuation factors 𝛽 1 of 0.5 and 𝛽 2 of 0.999. The batch size for data processing is 256, and the training process is conducted over 50 epochs. The gradient reversal ratio employed in domain-adversarial training for the ISRUC-S3 and MASS-SS3 datasets is 0.1 and 1, respectively.We employ 𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦, 𝑀𝑎𝑐𝑟𝑜 𝐹 1, 𝐾𝑎𝑝𝑝𝑎, and 𝐹 1 for each category as the evaluation metrics of the experimental results. 𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦 measures the proportion of correctly classified sleep epochs. The 𝐹 1 score for each category evaluates precision and recall, comprehensively evaluating the performance of the model. 𝑀𝑎𝑐𝑟𝑜 𝐹 1 score calculates the average F1 score across all sleep stage categories, providing insight into the capability of the model to perform well across categories without favoring a specific category. 𝐾𝑎𝑝𝑝𝑎 measures the agreement between the prediction of the model and the ground truth, accounting for chance. It assesses inter-rater agreement by considering both observed and expected accuracy.We select eight classical and state-of-the-art sleep staging methods as baseline methods from four perspectives: four conventional sleep staging [18, 48, 64, 10] method, one generalizable sleep staging [50] method is based on transfer learning to improve cross-domain sleep staging, two multimodal sleep staging [17, 60] methods involve modality-specific feature extractors, and one balance method [35] is special-designed for multimodal fusion and balance.• FeatureNet [18] : The conventional method uses a dual-scale CNN structure with varying kernel sizes and channel configurations. The first set of kernels consists of sizes {50, 8, 8, 8} and corresponding channels {32, 64, 64, 64}. The second set of kernels consists of sizes {64, 8, 6, 6, 4} and corresponding channels {64, 64, 64, 64}. • DeepSleepNet [48] : The conventional method employs the CNN-BiLSTM model. CNNs captures local temporal patterns and spatial relationships, while the BiLSTM component captures long-term temporal dependencies. • MaskSleepNet [64] : The conventional method combines dualscale CNNs for feature extraction, squeeze and excitation block for optimizing feature weights, and multi-head attention for capturing temporal information. • AttnSleep [10] : The conventional method integrates a multiresolution CNN model, adaptive feature recalibration for learning interdependencies among features, and multi-head attention for capturing temporal dependencies.• DAN [50] : The generalizable method is based on CNN-BiGRU and leverages the MMD alignment to learn domain-invariant representation.• SleepPrintNet [17] : The multimodal method involves modalityspecific feature learning based on the 1D convolutional model. In addition to learning temporal features across all PSG modalities, the EEG modality specifically focuses on spectralspatial feature learning. • MMASleepNet [60] : The multimodal method incorporates modality-specific feature learning through 1D convolutional models and squeeze-and-excitation, along with feature fusion utilizing transformer encoder models. • OGM-GE [35] : The balance method was originally an audiovisual dual-modality balancing method, which improved the model by balancing the classification performance of two modalities and modulating the gradient of the feature extractor. To be fair, we employ FeatureNet as its backbone.To demonstrate the advantage of SleepMG in sleep staging, we compared SleepMG with a total of eight methods, including four conventional methods, one generalizable method, two multimodal methods, and one multimodal balance method. The experimental results in Table 1 show that SleepMG consistently outperforms other methods on both datasets. On the ISRUC-S3 dataset, the second-best 𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦 method is the NaiveGS method, DAN, and SleepMG achieves an 𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦 improvement of nearly two percentage points compared to it and approximately 3.6% 𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦 improvement compared to the backbone method, FeatureNet. These results indicate the effectiveness of transfer learning methods in enhancing cross-domain sleep staging performance, and SleepMG demonstrates superior performance by balancing modalities in classification and domain discrimination. On the MASS-SS3 dataset, the second-best method is the modal balancing method (OGM-GE), and SleepMG further improves by approximately 1.0% compared to it, highlighting the importance of modality balancing and the greater role of SleepMG in balancing the two performances of modalities.As illustrated in Figure 4 , to further validate our progress in achieving modality balance, we visualized the changes of modal imbalance degree during training on the ISRUC-S3 dataset. The modal , the sum of squared mean relative differences [2, 47] . The lower the modal imbalance degree, the more balanced each modality will be. It can be observed that the SleepMG method exhibits lower modal imbalance in both classification and domain discrimination, indicating a better balance. Additionally, we visualized the changes in test accuracy in Figure 5 , revealing a significant improvement in cross-domain sleep staging on the test set when the modalities are more balanced. As is shown in Table 2 , to further demonstrate the contribution of each module in SleepMG, we compared the effects of multimodal and generalizable modules of NaiveMG (✓✓ in Table 2 ). We employ FeatureNet [18] as the backbone (×× in show that the generalization method provides a little improvement, while the multimodal module exhibits a more noticeable effect compared to the generalizable module alone. Furthermore, the combination of multimodal and generalizable modules leads to a significant improvement.To explore effective feature fusion methods for multimodal sleep staging, we employ FeatureNet to extract single-modal features and fuse them with direct concatenation, self-attention weighting, and modality balancing approaches, respectively. From the experimental results in Table 3 , it can be seen that self-attention performed worse than direct concatenation. Conversely, the modality balancing method performed well on both datasets. This suggests self-attention focuses more on well-performing modalities, exacerbating modality imbalances and harming classification. Furthermore, contrary to the attention-based approach, the balancing method assigns greater weight to poorly performing modalities, positively contributing to classification accuracy. As shown in Table 4 , we compared the effects of balancing different components of NaiveMG (× × × in Table 4 ). The results indicate that balancing the modality-specific feature extractor has minimal or no effect, while the fine-grained inter-modal balance of the modality-specific parts of classifier 𝐺 𝑦 and domain discriminator 𝐺 𝑑 has positive effects. In multimodal feature learning, focusing solely on achieving modality balance may significantly weaken the model's ability to extract modality-related information. In domain adversarial learning, classification capability and domain discrimination capability are often seen as opposing objectives. However, both imbalances across modalities can negatively affect the performance of the model. For example, if there is a significant difference in domain discrimination capability between different modalities, the model may learn irrelevant information about the participants that is modality-dependent or modality-specific, which can limit the capability of the model to generalize across domains. This intermodal imbalance can be detrimental to performance. In summary, these two inter-modal imbalances can negatively affect model performance. The fine-grained inter-modal balance of the classification and domain discrimination capabilities is crucial to mitigate these negative effects and improve overall performance in sleep staging.For more interpretable analysis, we exploit the t-SNE [55] to visualize the feature embeddings of Feature and SleepMG methods. As shown in Figure 6 , SleepMG exhibits more distinct classification boundaries than the FeatureNet method. For example, in SleepMG, the cluster representing the deep sleep stage N3 is fewer in number and more tightly grouped, while the cluster representing the N2 stage (indicated by green) clearly separates the blue N3 stage from other stages. The light sleep stage N1, which is the most challenging to differentiate, exhibits a tightly grouped yellow cluster in SleepMG, displaying clearer boundaries.  
conclusion: 
 This paper introduces SleepMG, a novel Multimodal Generalizable Sleep Staging method. Besides the naive integration of multimodal feature learning models and domain generalization methods, SleepMG quantitatively assesses and adaptively balances the classification and domain discrimination capabilities of multiple modalities, addressing the issues of inter-modal imbalances. Experimental results demonstrate that SleepMG outperforms state-of-the-art performance in cross-domain sleep staging and achieves inter-modal balances.  
 
 Appendix:  


Decision: The paper introduces a novel method for sleep staging called SleepMG, which is designed to balance inter-modal differences and achieve highly accurate cross-domain sleep staging.

After considering the paper, the reviewer's comments, and the rebuttal I recommend 'accept' for the paper.

The reviewers highlight the following strengths and limitations:

Strengths:
1.  Experimental results show that SleepMG outperforms other eight state-of-the-art sleep staging methods, effectively balancing multiple modalities across two PSG datasets.
2. Intensive experiments have been conducted to show the effectiveness of SleepMG.
3. This paper is well-organized and written.

Limitations:
1. One reviewer considers the proposed method impractical
2. Some reviewers find particularly Figure 1 does not seem useful.

Reviews: review: paper_summary: paper_summary: This paper presents a multi-modal based sleep staging method, SleepMG. It aims to address the inter-modal balance by defining the performance metrics of individual modalities and utilizing such metrics to adaptively adjust the gradients of classifiers and domain discriminator. Experiments were conducted on two public datasets: ISRUC-S3 and MASS-SS3.
strength_weakness: strength_weakness: 1) The proposed method is reasonable with a clear motivation
2) Experimental results are comprehensive with rich comparisons, good ablation studies, and various discussions.
limitations: limitations: 1) While the research motivation is clear, the proposed solution lacks clear contributions, as the gradient adjustment looks quite empirical. I view this study as a multi-modal fusion task and would like to see strong argument along this point. 
2) While the authors argue for generalization, there is no clear emphasis at the end of Introduction for summarizing contributions. 
3) Comparisons with other (larger) datasets such as SleepEDF with nearly 80 subjects are needed. In addition, the current improvement seems marginal (e.g., less than 0.01). 
4) I feel that Figure 1 is too simple to provide much useful context. 
5) Language improvement: “modal differences” may be “modality differences”, “to do the subject personalized calibration”, “none consider(s)”, “gradients of model of the classifier”, “modality-balanced”, more explanations on Figure 2, …
paper_summary: paper_summary: This paper presents SleepMG that detects the sleep stage of a subject using multimodal signals (i.e., EEG, EOG, EMG and ECG). Especially, SleepMG focuses on balancing the modalities in terms of their classification and domain discrimination performances. For this purpose, the modal performance metrics are devised and used to weight gradients of the modality-specific parts. The experimental results on two public datasets ISRUC-S3 and MASS-SS3 show the effectiveness of SleepMG.
strength_weakness: strength_weakness: - The idea to balancing modalities seems interesting.
- Especially, balancing modalities is considered on not only the classification performance of each modality but also the discrimination performance, in order to strengthen the domain-invariant power (cross-subject generality capability) of SleepMG.
- Intensive experiments have been conducted to show the effectiveness of SleepMG.
- This paper is well-written.
- Although it is not a contribution of this paper, domain adversarial learning looks useful for extracting domain-invariance (i.e., subject-invariance) features.
limitations: limitations: - I'm not completely convinced with the importance of balancing modalities. In particular, I'm wondering if it may weaken the effect of a very useful modality so that the final sleep staging performance may be degraded. Is it possible to theoretically proof that balancing modalities is better than biasing certain modalities?
- I'm not sure if the proposed modal performance metrics really lead to balancing modalities. This is because $\rho_t^{a,u}$ by Eq. (6) $ seems to be a normalised $v_t^{a,u}$ in Eq. (4) representing the sum of correct predictions of modality $u$. I'm wondering why using such $\rho_t^{a,u}$ to weight gradients is useful for balancing modalities.
- There seems no explanation about the "well-balanced" or "poorly-balanced" situations. In addition, how is the "well-balanced" situation different from "well-performing" situtation?
- In Fig. 5, the final performance of FeatureNet seems to be better than that of SleepMG. Does this figure really presents the effectiveness of balancing modalities? Also, considering Fig. 4, I'm wondering whether or not the modality imbalance is really a problem after a sufficient number of training epochs.
paper_summary: paper_summary: This paper proposes SleepMG, which introduces two features into the modeling of PSG signals, (1) inter-model balance and (2) domain generalization. Extensive experiments validate their models' effectiveness and the variants of SleepMG.
strength_weakness: strength_weakness: 1. The motivation of this paper is well-defined and sufficiently supported.
2. The method of introducing an imbalance score into the modeling of different modalities is novel.
3. The experiments and analysis are sufficient.
4. The presentation of this paper is good.
limitations: limitations: 1. The author proposes a cross-modality method but only tests in one setting: i.e., six-channel EEG, two-channel EOG. one-channel chin EMG, and ECG. More experimental setups of combinations of modalities should be considered. For example, using all the channels in MASS-SS3 (I think it is not necessary to select the same 10 channels in MASS-SS3) or using different datasets such as SHHS and CFS which contain more data samples.

2. The author adopts a domain classifier to force the model to learn domain-invariant representation for sleep staging. The design is reasonable but only if the number of participants is small. If the number of subjects is large, e.g., 1737 subjects in the SHHS dataset, and 320 subjects in the CFS dataset, it will be very difficult for the classifier to make good predictions. On the other hand, the model has to change its structure when the number of subjects increases in a continual training setting. One solution is using a loss function to guide the model to distinguish from different modalities (e.g., contrastive learning).

3. Some small issues regarding the paper presentation, e.g., the text in Figure 4 is too small, lacks a notation table in the method section (there are too many notations in this paper), and Figure 2 is difficult to understand before we carefully read the method section.
paper_summary: paper_summary: This paper introduces a novel method for sleep staging called SleepMG, which is designed to balance inter-modal differences and achieve highly accurate cross-domain sleep staging.  Experimental results show that SleepMG outperforms other eight state-of-the-art sleep staging methods, effectively balancing multiple modalities across two PSG datasets.
strength_weakness: strength_weakness: 1. This paper is well constructed and written, with a high level of clarity. The ablation study is sufficient and fully convincing.
2. SleepMG leverages multiple modalities (EEG, EOG, EMG, ECG) to improve the accuracy of sleep staging, providing a more comprehensive approach compared to using a single modality.
3. The mathematical modeling and algorithmic development are both sound and robust. The use of adaptive gradients based on performance metrics is particularly noteworthy for its potential to optimize learning dynamically.
4. SleepMG enhances cross-domain generalization, which is crucial for applying the model to new, unseen subjects.
5. The development of "SleepMG," which assesses and adjusts for modal imbalances in real-time, is a significant contribution. This approach not only advances the state-of-the-art in sleep staging but also offers a potential framework for other multimodal challenges.
limitations: limitations: 1. Displaying the confusion matrix would be useful for further explaining the results.
2. Figure 1 does not seem particularly useful.
3. Highlighting future work on multi-modal and inter-modal balance is favorable.