SleepSMC: Ubiquitous Sleep Staging via Supervised Multimodal Coordination
Authors: Shuo Ma
Yingwei Zhang
Yiqiang Chen
Hualei Wang
Yuan Jin
Wei Zhang
Ziyu Jia
Venue: ICLR 2025 Poster (2025)
Keywords: Sleep staging
Multimodal coordination
Ubiquitous computing
Primary area: applications to neuroscience & cognitive science

Abstract: Sleep staging is critical for assessing sleep quality and tracking health. Polysomnography (PSG) provides comprehensive multimodal sleep-related information, but its complexity and impracticality limit its practical use in daily and ubiquitous monitoring. Conversely, unimodal devices offer more convenience but less accuracy. Existing multimodal learning paradigms typically assume that the data types remain consistent between the training and testing phases. This makes it challenging to leverage information from other modalities in ubiquitous scenarios (e.g., at home) where only one modality is available. To address this issue, we introduce a novel framework for ubiquitous Sleep staging via Supervised Multimodal Coordination, called SleepSMC. To capture category-related consistency and complementarity across modality-level instances, we propose supervised modality-level instance contrastive coordination. Specifically, modality-level instances within the same category are considered positive pairs, while those from different categories are considered negative pairs. To explore the varying reliability of auxiliary modalities, we calculate uncertainty estimates based on the variance in confidence scores for correct predictions during multiple rounds of random masks. These uncertainty estimates are employed to assign adaptive weights to multiple auxiliary modalities during contrastive learning, ensuring that the primary modality learns from high-quality, category-related features. Experimental results on four public datasets, ISRUC-S3, MASS-SS3, Sleep-EDF-78, and ISRUC-S1, show that SleepSMC achieves state-of-the-art cross-subject performance. SleepSMC significantly improves performance when only one modality is present during testing, making it suitable for ubiquitous sleep monitoring.

Published as a conference paper at ICLR 2025
SLEEPSMC: UBIQUITOUS SLEEP STAGING VIA SU- PERVISED MULTIMODAL COORDINATION
Shuo Ma!.2 Yingwei Zhang!? Yiqiang Chen 12,33 Hualei Wang Yuan Jin',
Wei Zhang', Ziyu Jia
Beijing Key Lab. of Mobile Computing and Pervasive Device, Institute of Computing Technology,
Chinese Academy of Sciences' University Chinese Academy Sciences Pengcheng Laboratory
Al Dream (Zhuhai) Intelligent Technology Co. Ltd. 5 Beijing Key Laboratory of Brainnetome and Brain-Computer Interface, Institute of Automation, Chinese Academy of Sciences
6 Brainnetome Center, Institute of Automation, Chinese Academy of Sciences
{mashuc20g, thangyingween yęchen, wanghualei233 (0te:ttssseeeeeee0eeeee @ict. ac. en, {john, david. zhang f waadrram.ccom jia. www.@outoook..cooom com
ABSTRACT
Sleep staging is critical for assessing sleep quality and tracking health. Polysomnography (PSG) provides comprehensive multimodal sleep-related infor- mation, but its complexity and impracticality limit its practical use in daily and ubiquitous monitoring. Conversely, unimodal devices offer more convenience but less accuracy. Existing multimodal learning paradigms typically assume that the data types remain consistent between the training and testing phases. This makes it challenging to leverage information from other modalities in ubiquitous sce- narios (e.g., at home) where only one modality available. To address this is- sue, we introduce a novel framework for ubiquitous Sleep staging via Supervised Multimodal Coordination, called SleepSMC. To capture category-related consis- tency and complementarity across modality-level instances, we propose super- vised modality-level instance contrastive coordination. Specifically, modality- level instances within the same category are considered positive pairs, while those from different categories are considered negative pairs. To explore the varying reliability of auxiliary modalities, we calculate uncertainty estimates based on the variance in confidence scores for correct predictions during multiple rounds of random masks. These uncertainty estimates employed to assign adaptive weights to multiple auxiliary modalities during contrastive learning, ensuring that the primary modality Iearns from high-quality, category-related features. Exper- imental results on four public datasets, ISRUC-S3, MASS-SS3, Sleep-EDF-78, and ISRUC-S1, show that SCeepSMC achieves ofte-offthhe-rt cross-subject per- formance. SleepSMC significantly improves performance when only one modal- ity is present during testing, making it suitable for ubiquitous sleep monitoring.
INTRODUCTION
Sleep staging(Lig Liu & Jia 2023 Tu et 2016) is crucial for health monitoring, sleep quality as- sessment, and the diagnosis of neurological disorders (Zhang et al.) 2019 Tia et al.] 2022
Scott
al. (2022). According the American Academy of Sleep Medicine (AASM) (Berry et al. 2012) standards, sleep is categorized into five stages. Sleep staging relies on the integrated representation of various modalities within multimodal polysomnography (PSG) (Tautan et al. 2020 K Kwon al.) 2021). each containing shared and unique sleep-related information. Researchers usually analyze the performance of these signals under the guidance the AASM to determine sleep stages.
In addition to PSG, sleep staging can also be achieved using multi-channel electroencephalogra- phy (EEG) (Liu & Jia 2023), single-channel EEG (Phan et al. 2022), wearable devices like ear-
EEG (Mikkelsen et al.] 2019) and smartwatch (Chang et al.] 2018),
more comfortable
contactless devices like Ballisooardiograahy (BCG) pressure sensors (Mengxing et al. 2019) and
"Corresponding author: Yiqiang Chen

---

Published as a conference paper at ICLR 2025
(a) Uncomfortable multimodal PSG
(b) Comfortable unimodal device
Figure 1: Multimodal devices are accurate but uncomfortable, while unimodal devices are less ac- curate but more comfortable, making them ideal for use in ubiquitous scenarios (e.g.. at home).
Radar-based systems (Walid et al., 2021). However, PSG and multi-channel EEG, due to their rich brain-related information, provide the highest accuracy for sleep staging. Single-channel EEG and wearable devices, which only capture partial or indirect brain signals, generally offer lower accuracy, while contactless devices tend to have the lowest precision. As shown in Figure while complex multimodal devices enhance accuracy, they also significantly impact sleep comfort, making them less suitable for ubiquitous scenarios (e.g., at home) (Ma et af.] (2024a). Leveraging multimodal data improve not only multimodal testing performance but also unimodal performance remains key challenge.
However, most multimodal learning approaches (Jia et al.
2020
2021c
Yubo et al. 2022 Cao
et al. 2024]] Ma et al. 2023 2024b) assume that consistent modalitis available during training and testing are consistent, leading difficult to maintain accuracy when only one modality is available during testing. The multimodal coordination is dedicated to addressing this issue. It employs methods like feature alignment to explore modality-level consistency and complementarity dur- ing coordinated training, enhancing the performance of multimodal systems and even improving the effectiveness of the single primary modality during testing. For example, Liu et al. (2023b) aligned multimodal representations with EEG representations, distilling the knowledge from mul- tiple modalities into EEG-based models. Liu et al. (2024) employed contrastive learning to align representations both within the ECG modality and between it and the medical text modality, thereby enhancing the performance during testing when only ECG is available. However, existing methods lack targeted supervision (Khosla et al. 2220 Mai al. 2023) during knowledge transfer, making it difficult to learn category-specific features effectively, and the information from auxiliary modalities effectively filtered.
To address these challenges, we propose SleepSMC, a ubiquitous sleep staging method that com- bines supervised modality-level instance contrastive coordination and uncertainty-basee based feature weighting. SIeepSMC leverages contrastive coordination to effectively align category- in- formation across modality- instances. By bringing same-category instances closer (including alignment within the same modality) and pushing different-category instances apart, SleepSMC en- sures the model learns discriminative features that incorporate both consistency and complementar- ity. To enhance this learning process, SleepSMC introduces uncertainty-brit based feature weighting for auxiliary modalities. It estimates the robustness each auxiliary modality under random masks and adjusts their contribution accordingly in contrastive learning, ensuring that it emphasizes more liable and category eelagor--related features. During testing, SleepSMC demonstrates superior performance only when multiple modalities available, but also in unimodal scenarios, bridging the gap between multimodal training and real-world deployment.
To summarize, contributions are follows:
To the best of our knowledge, we first introduce multimodal collaboration in sleep staging, leveraging multiple auxiliary modalities to improve the performance of primary modality- based sleep staging in an end-to-end manner.
We utilize supervised modality-level instance contrastive coordination to capture category- related consistency and complementarity across intra-modality and inter-modality.
We utilize uncertainty estimates to adaptively weight auxiliary modality features during training. This approach ensures that more reliable auxiliary modality features contribute more significantly to the contrastive learning process, thereby improving the performance of the primary modality.
2

---

Published as a conference paper at ICLR 2025
Extensive experimental results four public datasets demonstrate that SleepSMC not only enhances multimodal across-subject sleep staging performance but also significantly boosts unimodal across-subject sleep staging performance.
2 RELATED WORK
Sleep Staging, Sleep staging is crucial for monitoring sleep quality and diagnosing neurological conditions. According to the AASM standard, sleep comprises five stages: Wake, NREM stages NI, N2, N3, and REM. Sleep staging is typically based on deep learning models, like CNN (Tsi-
nalis et al. 2016, Supratak et al. 2017). RNN (Bresch et al. 2018,, Supratak Guo 2020). and
LSTM (Liang et al.) 2023a Phyo et al. 2022). which enables the automatic extraction of spatial and
temporal features from PSG signals. To exploit complementarity and consistency across modalities in PSG, researchers have designed modality-speiffi feature extraction and fusion modules Xiang et al. 2023 Yubo et al,, 2022). However, the reliance on multiple modalities is impractical in ubiquitous settings where only modality is available. This limitation underscores the need for models to leverage multimodal data during training but maintain high performance with only one modality during testing. To solve this issue, Mikkelssnn et al. (2019) developed unimodal portable wearable devices, ear EEG to enhance the comfort of sleep monitoring in ubiquitous scenarios. Dia ct al. (2024) and Liang et al. (2023a) transferred knowledge from large-scale models lightweight models through multi-level alignment for wearable device use. However, due to the limited local in- formation that unimodal devices can perceive, the accuracy of unimodal sleep staging is constrained.
Multimodal Coordination. The limitations of low accuracy in unimodal sleep staging highlight the need for multimodal coordination methods (Zadeh et al., 22200; Liang et al. 20236] Rahate et al.] 2022). Multimodal coordination methods (Jia et al. [202 la Jiang et al. 22024] Lin & 2u24| Liu
primary modality during training. while relying single modality during testing, making them better suited for real-world applications. For instance. Liu et al. (024) employed unsupervised contrastive learning to align ECG data with medical text, improving ECG-only scenarios. Despite its successes, this approach is complex, involving pre- training and fine-tuning. Liu et al. (2023b) leveraged the dual- modal features EEG and skin response, along with unimodal EEG, for knowledge distillation to boost GEG-based emotion recognition. The above methods all rely solely on the consistency of multimodal data itself and lack the learning of category-related information.
al. 2023a) leverage multimodal data to enhance the performance
Most uncertainty estimation methods focus solely on input with only a single modality, exploring robustness evaluation (Salman et al.) 2019 Rosenfeld et 2020 Xu 2020; Lyu et al. 2021) and decision boundary adjustments (Weng et al.) 2018]] Leino el al. for unimodal models. Existing Multimodal Uncertainty Estimation methods Yang et al.
Uncertainty Estimation.
ct al.
2021
2024
Tellamekala et al. (2023) typically play a role in multimodal fusion. For instance, Tellamekala
ct al. (2033) quantifted uncertainty in modalities and employed calibration and ordinal constraints to multimodal models, improving emotion recognition performance and robustness. These works are tailored for effective multimodal fusion, including retaining useful information from each modality while removing redundant or irrelevant data. They are always requiring all the modalities present during training to be available during testing. If only a single modality is used during testing, performance suffers severely, failing unimodal applications.
3 PRELIMINARY
The training set is Derain = (xx;,:)) i E {1,..., }}} and the testing set is Dres: = {ا e {I...., J}}. The training data Devain and testing data Drest are from different subjects. I; lef, 2F, xx""] and x, [x. =m.] are PSG signals containing synchronized EEG, EOG, EMG, and ECG, and each epoch lasts 30 seconds. 2y and x are the i-th and j-th samples of training and testing set, respectively. yi 2,, 2,.,,.. K) is the corresponding sleep category and K is the total number of categories. 1 and J are the training and testing set sample numbers, respectively. U is the number of modalities, and U / is the number of all modality-level instances.
Sleep staging problem is defined as ÿ argmax, (Gy((())))) where i denotes the predicted category and 2,.,, K} denotes the index of category. Gj = [G G?. G.y) denote modality- specific feature extractors for EEG, EOG, chin EMG, respectively, and the parameter of Gy denoted
3

---

Published as a conference paper at ICLR 2025
Modality-speciffe Feature Extraction and Classification
Wiles
Supervised
Modality-level Instance
Gy
N2
Contraxtive Coordination
EEG
N21
Gj
XBM
xf
EGG
Training Set
Gy
EOG mmy x1
Gf
BOG
Wale
cy
Chin EMG
m
CCOOOCO
Uncertainty-buue Feature Weighting
x
T
Tesken Mask
Gf
pf
W/
Trainable
Frozeen
G
P
The pipeline SIeepSMC begins with modality-specific feature extraction and classifi- cation for each input modality. Subsequently, uncertainty- feature weighting is employed by estimating the reliability of auxiliary modalities through multiple rounds of random masks. These weights are employed to refine the extracted features during supervised modality-level instance con- trastive coordination, which aligns category-relate
Figure 2:
information across modality-level instances. In the figure, EEG is shown example of the primary modality. Shapes indicate modalities, colors indicate categories, and bidirectional arrows show the alignment of some positive instance pairs.
as or = [ofe, 0h.0 0..m) G G and G denote modality-speciffc label classifiers (only one classifier employed for testing in unimodal testing scenario). f f and f." represent the features extracted from synchronous modality-level instances x;, 2ς and a:"" respectively.
To simplify the symbol expression,
u E {e, o, m) denote the modality-related symbol. In this paper, ubiquitous sleep staging refers to using only single primary modality during test- ing (e.g., EEG), which more convenient for real-world applications. The primary modality is denoted as Up [e, o, m). and auxiliary modalities are denoted as {e, o.m. \ Up- The aux- iliary modalities assist during training but are not used during testing. p, {it, Ph, p denote
the predicted logits for different modalities from the same 24. while y E {0% Df. 9 } denote the
predicted category labels. f denotes the modality-Ievel feature. The main mathematical notations used in this paper are summarized in Table 4
THE PROPOSED SLEEPSMC
The pipeline of SIcepSMC is shown Figure 2 and the corresponding algorithm pseudocode is shown in Appendix A.2 SleepSMC is built upon three key components: Modality-specifie fea- ture extraction and classification: This serves as the backbone of SleepSMC. preserving the unique characteristics of each modality. 2) Uncertainty-basee feature weighting: This component ensures that more reliable and valuable features contribute more during training. 3) Supervised modality- level instance contrastive coordination: This enables the learning of category-relatee shared and complementary features across modality-level instances, ensuring effective multimodal coordina- tion for unimodal testing scenarios.
4.1 MODALITY-SPECIFE FEATURE EXTRACTION AND CLASSIHICATION
In SleepSMC, we utilize the FeatureNet (Jia et al. 2021b) (a dual-scale CNN model) as the backbone for modality-specifie feature extraction and classification. For given input sample 2;, the modality-
4

---

Published as a conference paper at ICLR 2025
specific feature extractor Gÿ is employed extract feature f, for the instance a/. These features are subsequently employed to predict the sleep stage classification for each modality.
The modality-speciin classification loss La for modality u is computed as follows:
S
- Cer(G CTn(C)))))).)) 2=1
(1)
where Leg denotes the cross-entropy loss for the classification task, (f.") denotes the predicted category logits for ft", Yı denotes the true sleep stage label for sample i.
This modality-specific architecture enables the model to effectively preserve the unique information contributed by EEG, EOG, and chin EMG signals.
4.2
ENCERTAINTY-BASER FEATURE WEIGHTING
To handle the uncertainty differences between auxiliary modalities, we introduce an uncertainty estimation mechanism that assesses and adaptively weights auxiliary modality features during trastive learning. This approach allows the model to emphasize reliable auxiliary modality features more while reducing the impact of those with higher uncertainty. Notably, the primary modality is excluded from the uncertainty estimation process and remains unweighted.
Uncertainty Estimation with Frozen Gradients. For each auxiliary modality ua multiple rounds of random masks are employed during training. Specifically, random intervals selected based on normal distribution are replaced with zero in the imput signal x for each auxiliary modality. This process is formalized as:
cp(x ,))
where yo represents random mask transformation, and is the percentage of the signal to be replaced with zeros. This transformation is employed T times, generating T different perturbed versions of the signal:
X
(3)
For each perturbed instance dhe we compute the confidence score pi for the correct category from modality-specific classifier Gya::
G* (G (4
P.P
where ys denotes the ground truth label, and denotes the transformation index for the same instance.
Then, the uncertainty variance for auxiliary modality ua computed as:
T
T
2
Piçt
(5)
Notably, during the entire calculation of uncertainty metrics, all gradients are frozen to ensure that the uncertainty estimation does not interfere with the training process.
Feature Weighting Based on Uncertainty Estimates. After calculating the uncertainty variance ri with frozen gradients, this variance is utilized to weight the auxiliary modality features. This weighting employed solely for the auxiliary modalities during the contrastive learning process, leaving the primary modality unaffected by the uncertainty estimation.
The feature weighting process involves two key stages. First, exponential function is employed to the negative value of -r for the auxiliary modalities. The transformation ensures that modalities with high uncertainty (large r!!)) are assigned exponentially smaller weights, while those with low
5

---

Published as a conference paper at ICLR 2025
uncertainty retain larger weights (small "." The exponential function introduces smooth, contin- uous inverse scaling, where weights decrease more sharply for high uncertainty and more gradually for low uncertainty, reflecting the varying tolerance of the model. This process is formalized as:
exp((xr."))
where r. is the uncertainty variance for instance xa (i.e., sample v; with modality ua).
Next, we apply softmax function to normalize these transformed values between auxiliary modal- ities and multiply the normalized weights by the number of auxiliary modalities (U - 1) to ensure that their combined contribution is properly scaled. The feature weights for each auxiliary modality Ua are computed as:
cxp(2"" /T)
W thu
① (U -)
Σv exp(22/71
where T denotes the temperature scaling parameter, v E {1, 2, U - -1} indexes all auxiliary modal- ities. @ denotes element-wise product.
These weighted modality features are then fed into the contrastive learning process, where gradients are backpropagated to optimize the model. By prioritizing the more reliable features, this method enhances the model's s overall robustness and improves its ability to align and coordinate features across modality-level instances. Notably, this weighting mechanism only directly affects contrastive learning loss and does not directly impact classification loss. The weighted features for the auxiliary modalities are computed as follows:
=
u f
(8)
where ft represents the features from of auxiliary modality ua f.dh is the weighted feature for auxiliary modality ua, O denotes element-wing product.
Proofs in Appendix We quantify the information transfer using perturbation radius and provide upper and lower bounds to explain how our uncertainty- based weighting works. Additionally, we employ minimax theory to prove that even in the worst-case scenario, where auxiliary modalities have high uncertainty (i.e., low weights), effective information transfer can still be achieved.
4.3 SUPERVISED MODALITY-LEVEI INSTANCE CONTRASTIVE COORDINATION
After modality-specinff features are extracted and weighted based on uncertainty, SleepSMC lever- ages supervised modality-leva instance contrastive learning to align modality-level instances of the same sleep stage, while simultaneously separating those of different sleep stages. Modality-Ievel alignment effectively encourages multimodal coordination, particularly in unimodal testing scenar- ios.
Contrastive Loss Function.
Firstly, the features from the modality-specific feature extractor be Lg-normalized as f== 1t
Then, for input data, the supervised contrastive loss
115""
computed by comparing each anchor instance with positive pairs (i.e., instances of the same category) and negative pairs (i.c., instances from different categories), excluding the anchor itself. The supervised contrastive loss defined as follows:
exp( f//T) EE' exp( f2/T)
Lcon U Σ
Σ (EQ(P)
log -XI-1
(9)
Q())
where i' 21, U x /} indexes of all modality-level instances, fov and f are the normalized weighted features for the anchor instance x and the positive instance x" respectively. The dot product measures the similarity between features. denotes the temperature scaling parameter, controlling the sharpness of the similarity scores. Q(t') denotes the set of indices of all positive in- stances (i.e., those belonging the same category) relative to the anchor instance. The denominator

---

Published as a conference paper at ICLR 2025
sums over all instances, excluding the anchor instance itself, resulting in (U I 1) comparisons, where E (1, U x - 1} denotes the corresponding index.
This loss function encourages the model to draw instances of the same category closer together within the feature space, while simultancously pushing instances from different categorics further apart. This modality-lvee alignment captures both inter-modal and intra-modal consistency and complementarity, enabling the model to focus more effectively on category-related information.
Final Loss and Optimization. The final objective function combines the supervised contrastive loss with modality-specifte classification loss:
Listll
Σ ME fefe,,m)
L + foon
(10)
where Las is the classification loss for all instances with modality
Finally, the total loss is backpropagated to jointly optimize the model parameters. This joint opti- mization process enhances the model's performance in cross- sleep staging, particularly in unimodal testing scenarios.
EXPERIMENTS
All experiments are implemented with Python 3.8.5 and Pytorch 1.7.1. We conduct them on computer server with 640GB RAM and two NVIDIA RTX A5000 GPUs with 24GB VRAM each. Detailed experiment settings are shown in Appendix A.3 A.3
5.1 DATASET AND DATA PROCESSING
We evaluate SleepSMC on four public datasets: ISRUC-S3 (Khalighi et al. 2016), MASS- SS3 (O'reilly al.) 2014), Sleep-EDF-75 (Kemp et al. | 20000) and ISRUC-SI (Khalighi al. 2016). According to the AASM standard, we remove the Movement and Unknown stages and merge the S3 and $4 stages into a N3 stage, resulting in five categories of Wake, N1, N2, N3, and REM sleep stages. Each PSG recording is downsampled to 100 Hz. and divided into 30-second sleep epochs.
ISRUC-S3 collects PSG from subjects (one male and nine females) over 10 nights, containing 8589 sleep epochs. We select nine of the 12 channels and remove the leg EMG and ECG channels that are far from the brain. We select three modalities with nine channels (six-channel EEG, two- channel EOG, and one-channel chin EMG). MASS-SS3 collects PSG from 62 subjects (28 males and 34 females) over 62 nights, containing 59304 sleep epochs. We select three similar modalities with nine channels as ISRUC-S3. Sleep-E.DF-78 collects PSG from 78 subjects (41 males and 37 females) over 153 nights, containing 195479 sleep epochs. We selected three similar modalities with full channels (two-channel EEG, one-channel EOG, and one-channel chin EMG) ISRUC-S3. ISRUC-SI collects PSG from 100 subjects (55 males and 45 females) over 100 nights, containing 87187 sleep epochs. We select the same channels and modalities ISRUC-S3.
5.2 COMPARATIVE EXPERIMENT RESULTS
To demonstrate the advantages of SleepSMC in sleep staging, we compare SleepSMC with a total of methods on both multimodal and unimodal testing scenarios on four datasets.
Multimodal testing scenario. We assign uncertainty-base weights to all modalities for contrastive learning and concatenate features from all modalities for joint testing during testing. Table[[present the results of the multimodal testing scenario, which indicate that SleepSMC consistently outper- forms the other methods across all four datasets. Each method based on multimodal data achieves relatively high performance in sleep staging. However, SIeepSMC still improves the Accuracy acy of the ISRUC-S3 dataset by 1.7% compared to the second-best method, BSTT, with 2.5% increase in Macro F1 and 2.3% increase in K Kappa. Furthermore, compared to the baseline method, Fea- tureNet, SleepSMC outperforms by over 3% on all three metrics. The improvements made to the other datasets are also significant.
Unimodal testing scenario. We designate one modality as the primary modality and the other two modalities as auxiliary modalities. During training, train using all modalities collectively,
7

---

Published as a conference paper at ICLR 2025
assigning uncertainty-bart -ased weights the auxiliary modalities. During testing, only the single (5
primary modality is employed for evaluation. Tables 2
6 and 7 in Appendix (A.4) present the results of the unimodal testing scenario, which indicate that SleepSMC consistently outperforms the other methods across all four datasets. Notably, traditional multimodal methods show a sharp accuracy drop with single-modality testing, while SleepSMC remains effective. On the ISRUC- S3 dataset, SIeepSMC improves the Accuracy in each primary modality experiment by over 1% compared to the second-best method. For the poor-perr noor-performing primary modality, EMG, it increases by 2.1% in Accuracy, with improvements of 1.7% in Macro F1 and 2.6% in K Kappa. On the other datasets, some methods fail to classify NI and N3 stages, especially when using only EMG from ISRUC-S1, with BSTT suffering from severe prediction imbalance, identifying only one category. In contrast, SIe consistently performs well, with improvements
aaggle-m performance.
Table 1:
The performance comparison of state-of thrr methods and SleepSMC for multimodal testing scenario on four datasets. The bold and underline items denote the best and second-bess results, respectively.
Overall results Macro
FI for each category
Dataset
Method
Accuracy
Kappa
Wake
N3
REM
NeatureNet Dia et al. 20216
0,7628
0,7495
0.6975
0.8706
0.5430
0.7152
0.8470
0.7715
DeepSleepNet (Supratak et ai 2017
0.7426
0.7135
0.6682
0.8788
0.4330
0,7248
0.8262
0.7050
AttnSleep Braele ar 2021
0,7656
0,7480
0.6993
0.8620
0.5166
0.7659
0.8754
0,7202
DAN Bang crai 2022
0.7720
0.7431
0.7058
0.8302
0,4441
0.7808
0.8794
0.7808
BSTT & Jia 2014
0,7756
0.7568
0.7114
0.8325
0.4901
0.7698
0.8722
0.8193
ISRUC-S3
SleepPrintNar Uma drale
XSleepNar Paapp dl 2021
0,7702
0.6705
0.7573
0.6440
0.5771
0.7043
0.7888
0.8287
0.4283
0.5311
0.7725
0,6909
0.8638
0,8045
0,7903
0,5077
MMASlaspNet of
2012
0.7732
0.7343
0,7066
0.8778
0,3950
0,7708
0,8703
0.7576
SimCI R Chen er Ai
201201
0,7738
0.7597
0.7110
0.8771
0.5600
0.7288
0. 0.8488
0.7839
DrFusee
0.7741
0.7469
0.7091
0.8748
0.4612
0.7625
0. 0,8658
0,7702
MERL
LiU Rl
0.7559
0.7458
0.68766
0.8350
0,5275
0.7432
0.8554
0.7678
Ours
0.7930
0.7815
0.7344
0.8765
0.5715
0,7753
0.8724
0.8117
NeatureNet Lia 202215
0.8530
0.8042
0.7835
0.8808
0.5118
0.8950
0.8695
DeepSteepNet (Supralin, et mi 2017
0.8561
0.8012
0.7862
0.8917
0,5056
0.8997
0.8401
0.8690
AutSSleep [Adieee el al
0.8592
0,8036
0.7914
0.8958
0.4989
0.9007
0.8526
0.8700
DAN Tang el al. 2012
0.8034
0.6801
0.7002
0.8050
0,1385
0,8708
0. 0.7999
0.7862
BSTT Liu
2014
0,8114
0,7492
0.7190
0.8292
0.4226
0,8637
0. 8023
0.8283
MASS-SS3
XSicepNar Pran oe di
2012
0.8066
0.7464
0.7158
0. 8.052
0. 0,4044
0. 0.8660
0. 0.7995
0.8116
SieepPrintNa 0
2020
0,8457
0.7855
0.7656
0.8628
0,8918
0.8223
0.8612
MMASleepNet
2u22
0.8627
0.7983
0. 0.7940
0. 8873
0. 0,4668
0.9023
0.8457
0.8892
SimCLR R Chen
0.8607
0.8101
0.7931
0.8944
0.5335
0.9008
0.8453
0,8765
DrFuse etal 704
0.8628
0.8086
0.7964
0. 8848
0,5186
0.9028
0.8542
0.8827
MERI
0.8605
0.8055
0.7915
0.8907
0.5200
0.8996
0.8324
0.8851
Ours
0.8686
0.8193
0.8058
0.9047
0.5472
0,9049
0.8463
0.8932
FeatureNet (Ja et al 20216
0.8012
0.7331
0.7257
0.9195
0,3849
0.8220
0.7807
0.7586
DeepSleepNer (Suprinak
2017
0.7973
0.7392
0.7205
0.9100
0.4271
0.8267
0,7843
0,7477
AttSSleep (Rieee el all 2012
0.7492
0.6912
0.6559
0.8280
0.3815
0.8158
0.7137
0.7170
DAN (Tang
0,7239
0.5915
0.6057
0.8519
0.0695
0.7629
0.7048
0.5682
BSTT Lau & Jil
20163
0.7321
0.6335
0.6245
0.8627
0.1963
0.7587
0.7323
0.6178
Sleep.EDD Beep-EDF-7
XSleepNat
0,7577
0.6855
0.6631
0.8739
0.3589
0.7977
0.7261
0.6709
SleepPriotNer 0id 2000
0.7849
0.7149
0.7009
0.9060
0,3711
0,8084
0.7541
0.7353
MMASleepNel Yl
0.7914
0.7012
0.7078
0.9126
0.4088
0.8186
0.6055
0.7611
SimCLR Chen 1
2020
0,8027
0.7352
0.7267
0.9170
0.3957
0,8281
0.7837
0.7532
DrPuse Yoo 2014
0,8009
0,7411
0.7235
0.9082
0.4189
0.8264
0.7846
0,7674
MERL Liu oC
2024
0.7990
0.7267
0.7196
0.9135
0,3681
0.8204
0.7841
0.7472
Oars
0,8158
0,7558
0.7450
0.9243
0,4285
0,8359
0,8000
0.7905
BSTT
Liu & Jia 2023
0.7247
0.6890
0.6423
0.8359
0,3622
0.7115
0.8000
0.7356
XSleepNadi SaleepNadPiaaa Piean et 2021
0,7444
0.7226
0.6707
0. 168688
0.4510
0.7325
0.8083
0.7526
ISRUC-SI
Driiuse Yao al MERL LU
0.7441
0.7215
0.6669
0.8143
0,4791
0.7310
0.7978
0.7852
0.7245
0.7042
0.6417
0.7954
0.4714
0.7086
0.7869
0.7588
Onrs
0.7710
0.7462
0.7018
0.8862
0.4795
0.7510
0.8138
0.8004
5.3 ABLATION EXPERIMENT RESULTS
To further illustrate the contributions of each module in SleepSMC, we performed ablation exper- iments on the Uncertaint--asse -bsed Feature Weighting and Supervised Modality-level Instance Con- trastive Coordination modules. As shown in Table 3 the results of multimodal and unimodal testing scenarios three datasets demonstrate the effecttieness of each module. The Supervised Modality- level Contrastive Coordination module plays a more significant role in both scenarios. Meanwhile, the Uncertainty-based Feature Weighting module demonstrates relatively enhanced performance in
8

---

Published as a conference paper at ICLR 2025
Table 2: The performance comparison of state-of-the-art methods and SIeepSMC on the ISRUC-S3
dataset for unimodal testing scenario. The bold and underline items denote the best and second- best results, respectively.
Modality
Method
Overall results Macro F1
for each category
Accuracy
Kappa
Wake
N1
N2
N3
REM
NeatureNet Jia al.
20216
0.7277
0.7104
0.6510
0.8806
0,5064
0.6726
0.8006
0.6917
DeepSleepNer
(Spprana e ai
201
0.6904
0.6507
0.6057
0.8641 1
0.3296
0.6349
0,7959
0.6289
AttnSSoop (Eldge oral 2021
0.7338
0.7105
0.6592
0.8581
0,4636
0.7320
0.8524
0.6463
DAN Lang die 2025
0.7212
0.6791
0.6400
0,8077
0.3511 I
0.7352
0.8686
0.6328
BSTT Jil
A1241
0.7191
0.6921
0.6371
0.8061
0,4312
0.6989
0,8502
0.6742
BBG
XSleepNer Phan hl
702
0.6555
0.6322
0,5614
0.8525
0,4562
0.6225
0,8015
0.428
SleepPrin/Ner Dilre
2040
0.5459
0.4862.
0.3924
0.3404
0.6161
0.6669
0.2968
MMASleepNed ( Tabo e Fin 20/2
0.63113
0.5975
0,51.50
0.7815
0,3486
0.6771
0.6471
0.5333
SimCI
212
0.7338
0.7163
0.6598
0,4978
0.6883
0,8260
0.6915
DrFFuse Tab el 0.
0.7532
0.7138
0.6818
0.8780
0.3872
0.7794
0,8609
0.6636
MERL
0.7467
0.7295
0.6758
0.5212
0.7328
0.8603
0.6808
Ours
0.7646
0.7397
0.6969
0.8882
0.5069
0.7467
0.8636
0.6932
FeatureNet Jia al. 102
0.7210
0.6932
0.6399
0. 0,7995
0,4640
0.7196
0,8570
0.6259
DeepSleepNet
Stpprala mim
201
0.7234
0.6902
0.6388
0.8142
0.4145
0.7100
0.8541
0.6583
AttnSleep
(Eldele 2l
0.7226
0.6416
0,4608
0.7115
0,8591
0.6399
DAN lang clai 20/4
0.7136
0.6647
0.6288
0.7733
0.2902
0.7406
0.8652
0.6542
BSTT
0,4700
0,2790
0.5895
0.6400
0.0020
XSleepNatMMann al 2021
0.6288
0.6071
0.5233
0.6958
0,3684
0.6572
0
0.5260
BOG
SleepPrintNer a.
0.3745
0.2531
0.1788
0.3553
0.0239
0,5600
0.0000
0,3183
MMASleepNet Yubo et 2022
0.20916
0.1745
0.0619
0.2750
0.2712
0.0000
0.0300
0.3264
SimCLR (Chei
0.7246
0.7007
0.6458
0.8097
0,4788
0.7096
0,8523
0.6529
DrFuss Yab el min
A024
0.6947
0.6799
0.6078
0.7522
0.4579
0.7115
0.8317
0.6460
MERL
0.6976
0.6741
0.6132
0,7996
0,3912
0.688
0,8351
0.6640
OBrs
0.7444
0.7168
0.6697
0.8386
0.4765
0.7360
0.8722
0.6607
FeatureNet Jia 2021b
0.4040
0.3731
0.2214
0.5326
0.1181
0,4282
0,3794
0,4075
DeepSleepNet (Sppraa crai 201
0.4166
0.3704
0,2404
0.5597
0.0403
0.4548
0.3978
0.3992
AttnSkeop
(Elogle
0.3915
0.3814
0.2191
0.5096
0,2067
0,3804
0,4152
0,3950
DAN Tang e ain 2016
0.4048
0.3381
0.2267
0.5541
0.0065
0.4670
0.2262
0.4365
BSTT
0.3046
0.0934
0.0000
0.0000
0.0000
0.4669
0.0000
0.0000
EMG
XSleepNar (hhan at
201
0.3660
0.3484
0.1935
0.4519
0.1654
0. 3665
0,3833
0.3748
SleepPrintNer
20:0
0.3319
0.2313
0.0939
0.4214
0.0359
0.4327
0.0000
0.2667
MMASleepNet (Tubo mi 2ure
0.2517
0.1969
0.1062
0.4155
0.1450
0.0000
0.0300
0.4240
SimCLR (Chen
20K0
0.4177
0.3906
0.2435
0.5605
0.1397
0.4303
0.4258
0.3968
DrFuxe Yab cl S.. MERL
A1744
0.3857
0.3789
0.2318
0.6026
0.1923
0.3549
0.3272
0.4176
al
0.3981
0.3907
0,2348
0.4875
0,2077
0.3879
0.4008
0.4696
OBrs
0.4384
0.4075
0.2693
0.5868
0.1281
0.4404
0.4301
0.4523
unimodal than multimodal scenarios. This suggests that when only the primary modality is avail- able during testing, the quality of auxiliary modalities during training becomes more critical, making feature weighting essential.
In addition, in Table 8 we conduct experiments on the uncertainty variance metric, where we com- pare the direct application of the negative value of 5g -r"0 with the effect of using Eq/6 which incorporates an exponential function for smoothing and scaling, thereby demonstrating the necessity of the latter.
Table 3: Ablation study of Uncertainty-based Feature Weighting and Supervised Modality-level instance Contrastive Coordination modules of SleepSMC( vn on three public datasets. The bold and underline items denote the best and second-best results, respectively.
Accuracy/ / Macro Fl Kappa
Dataset
U
multimodal
EEG
EOG
EMG
0.7628 / 0.7495 / 0.6975
0.7277 0.7104 / 0.6510
0.7210/
0.6932
0.6399
0.404D 0,3731 0,2214
ISRUC-S3
x
0.7918 / 0.7767 / 00.7229
0.7609 0.7358 0.6932
0,7343/
0,7087
0.6570
0.4209 0,3974 0.2490
0,7930 / 0,7815/ / 0.7344
0.7646 0.7397
0.6969
0.7444/ 0.7168
0,6697
0,4384 / 0,4075 0,2693
0.8530/ / 0.8042 / 0.7835
0.8366
0.7571 / 0.7575
0,7970/
0.7102 0.6939
0.5327
0.3664 /0,2439
MASS-SS3
0.8680 / 0.8185 / 0.8045
0,8455 0.2725 0.7703
0.8148/
/0.4499 / /0.2249
0.5339 / 00,3679/ /0,2463
0.8686/ / 0,8193/ / 0.8058
0.8517 0.7871 0..799
0.8227
0.7534
0.7359
0.5408 0.3770 / 0,2613
0. 8.102// / 0.7331 / 7077
0.7772 / 0.6944 / 0.6928
0.7238
0.6375/ 006155
0,5180 3,70076/ / 0.28499
Sleep-EDD-77
0.8146/ / 0.7552/ / 0.7434
0.7894/
0.7 / 0.7088
0.7250/
0.6399
D.6172
0.5211 00.155 /0.2908
0.8158/ / 0.7558 / 0.7450
0.7959: 0.7217 / 0..1169
0.7311
0.6494 0.6258
0.5293 0.3167 0.3017

---

Published as a conference paper at ICLR 2025
5.4 INTERPRETABLE ANALYSIS RESULTS
Weight Visualization Analysis. As shown in Figure 3 we visualize the uncertainty-based weight variations of auxiliary modalities under different primary modalities during training. From Fig- ure 3 (a) and (b), it can be observed that despite the lower accuracy of EMG-based sleep stag- ing, SIeepSMC effectively adjusts the weight of EMG during contrastive learning due its robust- ness (lower uncertainty), resulting in significant improvements. This demonstrates that prioritizing robustness over accuracy in supervised contrastive learning is crucial for performance enhancement.
wightt change builary mobiiees
The chenge
Coro
(a) EEG
(b) EOG
(c) EMG
Figure 3: Visualization of the uncertainty-based weight changes for the two auxiliary modalities with the primary modalities: (a) EEG, (b) EOG, and (c) EMG.
Feature Visualization Analysis. We exploit t-SNE (Van der Maaten & Hinton 2008) to visual- ize the feature embeddings of each primary modality for both FeatureNet and SIeepSMC methods. As shown in Figure 4 regardless of which primary modality is employed for testing. SleepSMC consistently displays more explicit and compact classification boundaries. Notably, for the most challenging NI stage, the distribution improves from being completely scattered to having a clear boundary. Although EMG-based sleep staging generally performs poorly, SleepSMC still signifi- cantly improves over FeatureNet and achieves a clear separation for categories like Wake and REM.
(a) EEG-base
(b) EOG-base
EMG-base
(d) BEG-ours
(c) BOG-ours
(f) EMG-ours
Figure 4: Visualization the t-SNE emheddings for each modality of the ISRUC-S3 dataset using
(a)(b)c) FeatureNet and (f) SleepSMC.
CONCLUSION
We proposed SleepSMC, a framework that combines supervised modality-level instance contrastive coordination with uncertainty-based feature weighting for ubiquitous sleep staging. SleepSMC en- hances the performance of the primary modality by leveraging supervised modality-level instance contrastive coordination during training and dynamically adjusting the contributions of auxiliary modality features based on uncertainty. Experiments on four public datasets show that SleepSMC achieves superior cross-subject performance, even when only single modality is employed dur- ing testing. The framework effectively captures category-related relationships across and within modalities, demonstrating its robustness, interpretability, and potential for real-worke applications.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This work is supported by the Natural Science Foundation of China (No.6232487), Improvement Project of Chinese Academy of Sciences NN.CGSZXXKY2222250007, the Science and Technol- ogy Innovation Program of Hunan Province (No.2022RC4066, No.2024J19031, and Al Dream Project (No. N2M.22222024
REFERENCES
Richard B Berry, Rohit Budhiraja, Daniel J Gottlieb, David Gozal, Conrad Iber, Vishesh Kapur, Carole I. Marcus, Reena Mehra, Sairam Parthasarathy, Stuart F Quan, al. Rules for scoring respiratory events in sleep: update of the 2007 aasm manual for the scoring of sleep and associated events: deliberations of the sleep apnea definitions task force of the american academy of sleep medicine. Journal of clinical sleep medicine, 865):577-11,, 2012.
Erik Bresch, Ulf Großekathöfer, and Gary Carcia-Moling Recurrent deep neural networks for real- time sleep stage classification from single channel eeg. Frontiers in computational neuroxcience, 12:1-12, 2018.
Lei Cao, Wenrong Wang, Yilin Dong, and Chunjiang Fan. Advancing classroom fatigue recognition: A multimodal fusion approach using self-attention mechanism. Biomedical Signal Processing and Control (BSPC), 89: 195566,6, 2024.
Liqiong Chang. Jiaqi Lu. Ju Wang. Xiaojiang Chen, Dingyi Fang, Zhanyong Tang, Petteri Nurmi, and Zheng Wang. Sleepguard: Capturing rich sleep information using smartwatch sensing data. Proceedings of the ACM Interactive, Mobile, Wearable and Ubiquitous Technolo- gies (IMWUT), 2(3):1334, 2018.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learn- ing (ICML), pp. 1597-1607. PMLR, 2020.
Emadeldeen Eldele, Zhenghua Chen, Chengyu Liu, Min Wu, Chee-Keong Kwoh, Xiaoli Li, and Cuntai Guan. An attention-baset deep learning approach for sleep stage classification with single- channel eeg. IEEE Transactions on Neural Systems and Rehabilitation Engineering (TNSRE), 29: 809- 010.818,222 2021.
Chao Jia Yinfei Yang, Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung. Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning (ICML). pp. 4904- 4916. PMLR, 2021a.
Ziyu Jia, Xiyang Cai, Gaoxing Zheng, Jing Wang, and Youfang Lin. Sleepprintnet: A multivariate multimodal neural network based on physiological time-series for automatic sleep staging. IEEE Transactions on Artificial Intelligence (TAI), 1(3):248-255 257, 2020.
Ziyu Jia, Youfang Lin, Jing Wang, Xiaojun Ning, Yuanlai He, Ronghao Zhou, Yuhan Zhou, and H Lehman Li-wei. Multi-view eralization for sleep stage classification. IEEE Transactions on Neural Systems and Rehabilitation Enginsample (TNSRE)). 29:1977-1986. 2021b.
graph convolutional networks with domain gen-
Ziyu Jia, Youfang Lin, Jing Wang, Xuehui Wang, Peiyi Xie, and Yingbin Zhang. Salientsleepnet: Multimodal salient wave detection network for sleep staging. In Twenty-nine international joint conference on artificial intelligence (IJCAI). pp. 2614-2620, 2021c.
Ziyu Jia, Junyu Ji, Xinliang Zhou, and Yuhan Zhou. Hybrid spiking neural network for sleep elec- troencephalogram signals. Science China Information Sciences, 140403,, 2022.
Ziyu Jia, Heng Liang, Yucheng Liu. Haichao Wang, and Tianzi Jiang. Distillsleepnet: Heteroge- neous luvei-level knowledge distillation via teacher assistant for sleep staging. IEEE Transactions Big Data (TBD), pp. 1-13, 2024.
11

---

Published as a conference paper at ICLR 2025
Jun-Peng Jiang, Han-Jia Ye, Leye Wang, Yang Yang, Yuan Jiang, and De-Chuan Zhan. Tabular insights, visual impacts: Transferring expertise from tables to images. In Forty-firs: International Conference Machine Learning (ICML), pp. 1-12, 2024.
Bob Kemp, Aeilko H Zwinderman, Bert Tuk, Hilbert AC Kamphuisen, and Josefien JL Oberye. Analysis of sleep-dependent neuronal feedback loop: the slow-wave microcontinuity of the eeg. IEEE Transactions on Biomedical Engineering (TBME), 47(9): 185-1194, 2000.
Sirvan Khalighi, Teresa Sousa, José Moutinho Santos, and Urbano Nunes. lsruc-sleep: A compre- hensive public dataset for sleep researchers. Computer methods and programs biomedicine, 124:180-199 2016.
Prannay Khosla, Piotr Teterwak, Chen Wang. Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishman. Supervised contrastive learning. Advances in neural information processing systems (NeurIPS), 33:18661-186773 2020.
Shinjae Kwon, Hojoong Kim, and Woon-Houe Yeo. Recent advances in wearable sensors and portable electronics for sleep monitoring. Iscience, 24(5), 2021.
Klas Leino, Zifan Wang, and Matt Fredrikson. Globally-robust neural networks. In International Conference on Machine Learning (ICML), pp. 6211 6222. PMLR, 2021.
Heng Liang, Yucheng Liu, Haichao Wang, Ziyu Jia, and Brainnetome Center. Teacher assistant- based knowledge distillation extracting multi-level features on single channel sleep eeg. In Thirty- two international joint conference artificial intelligence (ICAI), pp. 3948-3956, 2023a.
Paul Pu Liang, Amir Zadch, and Louis-Philippe Morency. Foundations trends in multimodal machine learning: Principles, challenges, and open questions. ACM Computing Surveys, 2023b.
Ronghao Lin and Haifeng Hu. Adapt and explore: Multimodal mixup for representation learning. Information Fusion, 105: 105:10 2024.
Che Liu, Zhongwei Wan, Cheng Ouyang, Anand Shah, Wenjia Bai, and Rossella Arcucci. Zero- shot eog classification with multimodal learning and test-time clinical knowledge enhancement. In Forty-first International Conference on Machine Learning (ICML), pp. 1-10, 2024.
Dongjun Liu, Weichen Dai, Hangkui Zhang, Xuanyu Jin, Jianting Cao, and Wanzeng Kong. Brain- machine coupled learning method for facial emotion recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2023a.
Yuchen Liu and Ziyu Jia. Bstt: A bayesian spatial- temporll transformer for sleep staging. In The Eleventh International Conference Learning Representations (ICLR), 2023.
Yucheng Liu, Ziyu Jia, and Haichao Wang. Emotionkd: A cross-modal knowledge distillation framework for emotion recognition based on physiological signals. In Proceedings of the 31st ACM International Conference on Multimedia (ACM MM), pp. 6122-6131, 2023b.
Zhaoyang Lyu, Minghao Guo, Tong Wu, Guodong Xu, Kehuan Zhang, and Dahua Lin. Towards evaluating and training verifiably robust neural networks. In Proceedings the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), pp. 4308 4317, 2021.
Shuo Ma, Yingwei Zhang, Yiqiang Chen, Tao Xie, Shuchao Song, and Ziyu Jia. Exploring structure incentive domain adversarial learning for generalizable sleep stage classification. ACM Transac- fions on Intelligent Systems and Technology (ACM TIST), 15(1):1-30, 2023.
Shuo Ma, Yingwei Zhang, Yunqi Liu, Yiqiang Chen, Weiwen Yang, Jianrong Yang, Mingming Zhao, and Ziyu Jia. Ccam: Cross-channel association mining for ubiquitous sleep staging. IEEE Internet of Things Journal (IoTJ), 12(3):2713-277 2024a.
Shuo Ma. Yingwei Zhang, Zhang Qiqi, Yiqiang Chen, Wang Haoran, and Ziyu Jia. Sleepmg: Multimodal generalizable sleep staging with inter-modal balance of classification and domain discrimination. In Proceedings of the 32th ACM international conference on Multimedia (ACM MM), pp. 1-10, 2024b.
12

---

Published as a conference paper at ICLR 2025
Sijie Mai, Ying Zeng, and Haifeng Hu. Learning from the global view: Supervised contrastive learning of multimodal representation. Information Fusion, 100:1-10, 2023.
LIU Mengxing, QIN Liping, and YE Shuming. A mattress system of recognizing sleep postures based on beg signal. Chinese journal of medical instrumentation, 43(4):243 2222244222222222 2019.
Kaare B Mikkelsen, Yousef R Tabar, Simon L Kappel, Christian Christensen, Hans O Toft, Mar- tin C Hemmsen, Mike L Rank, Marit Otto, and Preben Kidmose. Accurate whole-night sleep monitoring with dry- dry-contact ear-eeg. Scientific reports, 91):16824, 2019.
Christian O'reilly, Nadia Gosselin, Julie Carrier, and Tore Nielsen. Montreal archive of sleep studies: an open-access resource for instrument benchmarking and exploratory research. Journal of sleep research, 23(6):628-00) 623))666666666,-, 2014.
Huy Phan, Oliver Y Chén, Minh C Tran, Philipp Koch, Alfred Mertins, and Maarten De Vos. Xsleep- net: Multi-view sequential model for automatic sleep staging. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 44(9):5903-59) 2021.
Huy Phan, Kaare Mikkelsen, Oliver Y Chén, Philipp Koch, Alfred Mertins, and Maarten De Vos. Sleeptransformer: Automatic sleep staging with interpretability and uncertainty quantification. IEEE Transactions on Biomedical Engineering (TBME), 69(8):2456-244 2467, 2022.
Jacun Phyo, Wonjun Ko, Eunjin Jeon, and Heung-Il Suk. Transsleep: Transitioning-awwe awaree attention- based deep neural network for sleep staging. IEEE Transactions on Cybernetics (TCYB), 2022.
Anil Rahate, Rahee Walambe, Sheela Ramanna, and Ketan Kotecha. Multimodal co-learning: Chal- lenges, applications with datasets, recent advances and future directions. Information Fusion, 81: 203-239. 2022.
Elan Rosenfeld, Ezra Winston, Pradeep Ravikumar, and Zico Kolter. Certified robustness to label- flipping attacks via randomized smoothing. In International Conference Machine Learn- ing (ICML), pp. 8230-8241. PMLR, 2020.
Hadi Salman, Jerry Li, Ilya Razenshteyn, Pengehuan Zhang, Huan Zhang, Sebastien Bubeck, and Greg Yang. Provably robust deep learning via adversarially trained smoothed classifiers. Ad- vances in neural information processing systems (NeurIPS), 32, 2019.
Hannah Scott, Bastien Lechat, Jack Manners, Nicole Lovato, Andrew Vakulin, Peter Catcheside, Danny J Eckert, and Amy C Reynolds. Emerging applications of objective sleep assessments towards the improved management of insomnia. Sleep Medicine, 2022.
Akara Supratak and Yike Guo. Tinysleepnet: An efficient deep learning model for sleep stage scoring based on raw single-channel eeg. In 2020 42nd Annual International Conference of the IEEE Engineering Medicine Biology Society (EMBC), pp. 641 641-644. IEEE, 2020.
Akara Supratak, Hao Dong, Chao Wu, and Yike Guo. Deepsleepnet: model for automatic sleep stage scoring based on raw single-channel eeg. IEEE Transactions on Neural Systems and Reha- bilitation Engineering (TNSRE), 25(1 1998828-0008 2017.
Minfang Tang, Zhiwei Zhang, Zhengling He, Weisong Li, Xiuying Mou, Lidong Du, Peng Wang, Zhan Zhao, Xianxiang Chen, Xiaoran Li, et al. Deep adaptation network for subject-specific sleep stage classification based on single-lead ecg. Biomedicul Signal Processing and Control (BSPC), 75:1-13, 2022.
Alexandra-Maria Täutan, Alessandro C Rossi, Ruben De Francisco, and Bogdan Ionescu. Auto- matic sleep stage detection: study the influence of various psg input signals. In 2020 42nd annual international conference of the ieee engineering in medicine & biology society (EMBC), pp. 5330-5334. IEEE, 2020.
Mani Kumar Tellamekala, Shahin Amiriparian, Bjöm Schuller, Elisabeth André, Timo Gies- brecht, and Michel Valstar. Cold fusion: Calibrated and ordinal latent distribution fusion for uncertainty- aware multimodal emotion recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), pp. 805-822, 2023.
13

---

Published as a conference paper at ICLR 2025
Orestis Tsinalis, Paul M Matthews, and Yike Guo. Automatic sleep stage scoring using time- frequency analysis and stacked sparse autoencoders. Annals of biomedical engineering, 44(5): 1587-1597, 2016.
Enmei Tu, Nikola Kasabov, and Jie Yang. Mapping temporal variables into the neucube for improved pattern recognition, predictive modeling, and understanding of stream data. IEEE transactions on neural networks and learning systems (TNNLS), 28(6):1305-1317, 2016.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research (JMLR), 9(1)), 2008.
Brahim Walid, Jianhua Ma, Muxin Ma, Alex Qi, Yunlong Luo, and Yihong Qi. Recent ad- vances in radar-based sleep monitoring-4 review. In 2021 IEEE Intl Conf on Dependable, Autonomic and Secure Computing, Intl Conf Pervasive Intelligence and Computing, Intl Conf on Cloud and Big Data Computing, Intl Conf on Cyber Science and Technology Congress pp. 759-766. IEEE, 2021.
Lily Weng. Huan Zhang, Hongge Chen, Zhao Song, Cho-Jui Hsieh, Luca Daniel, Duane Boning, and Inderjit Dhillon. Towards fast computation of certified robustness for relu networks. In International Conference on Machine Learning (ICML), pp. 5276-5285, PMLR, 2018.
Tian-Y Yu Xiang, Xiao- Zhou, Xiao-Liang Xie, Shi-Qi Liu, Hong-Jun Yang. Zhen- Qiu Feng. Mei- Jiang Gui, Hao Li, De-Xing Huang, and Lleng-Guang Hou. Learning shared semantic information from multimodal bio-signals for brain-muscle modulation analysis. In Proceedings of the 31st ACM International Conference on Multimedia (ACM MM), pp. 6016-6024, 2023.
Kaidi Xu, Zhouxing Shi, Huan Zhang, Yihan Wang, Kai-Wei Chang, Minlie Huang, Bhavya Kailkhura, Xue Lin, and Cho-Jui Hsieh. Automatic perturbation analysis for scalable certified robustness and beyond. Advances Neural Information Processing Systems (NeurIPS), 33:1129-
2020.
Zequn Yang, Yake Wei, Ce Liang, and Di Hu. Quantifying and enhancing multi-modal robustness with modality preference. In The Twelfth International Conference on Learning Representa- tions (ICLR), pp. 1-23, 2024.
Wenfang Yao, Kejing Yin, William K Cheung, Jia Liu, and Jing Qin. Drfuse: Learning disentangled representation for clinical multi-modal fusion with missing modality and modal inconsistency. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 38, pp. 16416- 16424, 2024.
Zheng Yubo, Luo Yingying, Zou Bing, Zhang Lin, and Li Lei. Mmasleepnet: A multimodal at- tention network based on clectrophysiological signals for automatic sleep staging. Frontiers in Neuroscience, 16:973761, 2022.
Amir Zadeh, Paul Pu Liang, and Louis-Philipen Morency. Foundations of multimodal co-learning. Information Fusion, 64:188-15 193, 2020.
Linda Zhang, Daniel Fabbri, Raghu Upender, and David Kent. Automated sleep stage scoring of the sleep heart health study using deep neural networks. Sleep. 42(11):zsz 59, 2019.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 SYMBOL DEFINITION
Table 4: List of mathematical notations used in the SleepSMC method.
Notation
Definition
Dirain: Drest
Training set and cross cross-subjeect testing
Fi, Bj
Input PSG signal of the i-th and j-th sample, respectively. Ground truth sleep stage label for the i-th and j-th sample. Total number of modalities.
Ye Mi
J.
Number samples in the training and testing sets, respectively. Total number of sleep stage categories.
K
u E {e, o,
Modality index: e for EEG, o for EOG, m for EMG. Modality-spechhe input signal of the i-th sample for modality u. Features extracted for modality u from the i-th sample.
ft
Gy, Gy
Feature extractor and classifier for modality Parameters of the feature extractor for modality u. Primary modality (used during testing).
Up
ua
Auxiliary modality (used only during training). Number of random masks for uncertainty estimation. Random mask transformation applied input with ratio 7). Confidence score for the predicted category from modality u. Uncertainty variance of the i-th sample for modality u. Weight assigned to the features of modality for the i-th sample.
T
((x,))
wi
fp
L2- n.2-oormalized feature of modality u for the i-th sample. -specic classification loss for modality u. Supervised contrastive loss.
L
Lcon
Liotal
Total loss combining classification and contrastive losses.
A.2 METHOB IMPLEMENT
The pipeline of SleepSMC shown in Algorithm I During training, the model performs modality- specific feature extraction for each instance and applies uncertainty-baset feature weighting for auxiliary modalities. The model parameters are then updated by optimizing the modality-specifii classification loss and supervised contrastive loss. During testing, only the primary modality (e.g., EEG) is employed to extract features from cross-subject test data, and the trained classifier is em- ployed to predict the sleep stages. The final output is the predicted results for the testing set.
15

---

Published as a conference paper at ICLR 2025
Algorithm SleepSMC: End-to-End Training and Testing
Input:
Training data Dirain = {z,, Wo) testing data Drest = {ag1z=1
Output:
Predicted results Yj for Drest
1: Initialization: Initialize model parameters 0f for feature extractors and 0* for classifiers Gy
2: Training:
3: For not converged de
4: for i = 1 to do
b Modality-Specific Feature Extraction
5: Extract modality-specific Teatures:
6: ft = (zf); f G) (a:{); ft"
7:
for cach auxiliary modality ua fe, o.
m Up do
Uncertainty- Based Feature Weighting
8:
Estimate uncertainty rit using Eq
Compute feature weight wi" using Eq
10:
Weight auxiliary features: f" end for
w.
11:
Normalize features 7
13:
end for
14:
Compute modality codaaitt-speccffic classification loss Lig using Eq.
15:
Compute supervised contrastive loss Leee using Eq.
b Supervised Contrastive Learning
Compute total loss: Luaul = Σn. Luis + Leve
17:
Update model parameters 0¹, 6t using backpropagation
18:
end For
19:
20: Testing:
21: for j = to do
>. Predicting on test data after training > Take EEG as an example
22: Extract features from cross-subject test data: 1j (4))
23:
Predict sleep stage ú argmax, [G](f)]&
24:
end for
25: Return: Predicted results ûn for Drest
A.3
EXPERIMENT SETTINGS AND IMPLEMENTATION
A.3.1 PARAMATER SETTINGS AND METRICS
In the experiment, we use FeatureNet (Jia et al. 2021b) as the backbone of SleepSMC to extract modality features. The subjects divided into five domains for five-fold cross-validation, with each domain serving as the testing set and the remaining four as the training set. From each training set, 20% is set aside validation set for model selection. The best model saved and evaluated on the corresponding cross-subject testing set, and the final results are averaged across all five folds for overall evaluation. The model is trained for 100 epochs using the Adam optimizer with a learning rate 0.001. The temperature - for contrastive learning is set to 0.1. The number of random masks T is set 9, with a random masking ratio 17 of 15% on data.
The evaluation metrics include Accuracy, Macro F1, Kappa, and the F1 score for each category. Accur acy represents the proportion of correctly classified instances. The Fl score combines preci- sion and recall for each category, while M Macro F1 averages the Fl scores across all sleep stages. K Kappa measures the agreement between predictions and ground truth, considering random chance.
A.3.2 BASELINE METHODS AND SETTINGS
We select 11 classical and state-of-the-ard sleep staging methods as baselines: six conventional (Fea- tureNet, DeepSleepNet, AttnSleep. DAN, BSTT, XSleepNet), two multimodal (SleepPrintNet, MMASleepNet) and three multimodal coordination methods((imCLLR, DrFuse, MERL). In mul- timodal and multimodal coordination methods, multiple modalities are employed as inputs during training, but only single modality is employed during testing, with missing modalities filled with zeros. Among these, FeatureNet (Jia et al. 2021b) utilizes a dual-scale CNN architecture with varying kernel sizes to perform modality-specific feature extraction, followed by fully connected layers to classify fused features. DeepSleepNet (Supratak et al., 2017) combines CNNs for local pattern and spatial relationship extraction with BiLSTM for modeling long-term dependencies. At- inSleep (Eldele et al., 202T) integrates a multi-ress restuutosoll CNN, adaptive feature recalibration for learning feature interdependencies, and multi-head attention for temporal modeling. DAN (Tang
16

---

Published as a conference paper at ICLR 2025
et al. 2022 is CNN- BiGRU model that employs MMD alignment to learn
fea- tures. BSTT (Liu & Jia 2023) uses Bayesian inference to learn spatial and temporal relationships, combining Transformer and fully connected networks, with KL divergence enforcing consistency between prior and posterior distributions. XSleepNet (Phan et al.
2021 combines CNN, GRU, and attention mechanisms to learn spatial and temporal features from EEG signals, using multi-branch architectures for joint classification. SleepPrintNet (Jia et al. 2020 is a multimodal method using ID CNN for modality-specific feature extraction, focusing on temporal features across all modal- ities and epectral-spation features for EEG. MMASleepNet (Yubo et al., (022) combines 1D CNN and squeeze-ande gqueeze-and-excitalie and-excitation for modality-speciffe learning, with transformer encoders for feature fu- sion. SimCLR (Chen et al. 2020) uses FeatureNet for modality-ppecific learning and classification, treating pairs of modalities from the same synchronized samples as positive pairs and others as negative pairs. DrFuse (Yao et al. 2024) combines shared and modality-specine
models to learn joint representations from multimodal inputs. MERL et al. (2024) achieves intra- modaltty and mnter-modality alignment through contrastive learning, effectively learning and sharing modality rep- resentations.
A.4 SUPPLEMENTARY EXPERIMENT RESULTS
Table 5: The performance comparison of state-of-the-art methods and SleepSMC MASS-SS3 dataset for the unimodal testing scenario. The bold and underline items denote the best and second- best results, respectively.
Overall results
for each category
Modality
Method
Accuracy
Macro FT
Kappa
Wake
NI
N2
N3
REM
FeatureNet (Ja al
202
0.8366
0.7571
0.7575
0.8813
0.3558
0.8943
0.8302
0.8238
DeepSleepNet (Supratak et
2017
0.8348
0.7456
0.7513
0.8811
0.3028
0.8911
0.8367
0.8160
AttSSleep [suale el at 2021
0.8320
0.7512
0.7489
0.8801
0.3483
0.8918
0.8231
0.8129
DAN
Tang al.
0.7976
0.6556
0.8301
0.0318
0.8734
BSTT & Jin 2174
0,7927
0.6855
0.6856
0.8183
0.1978
0.8650
0.8141
0.7321
EBG
XSleepNat Phan
202)
0.7973
0.7175
0,7000
0.8514
0.3134
0.8619
0,7935
0.7674
SleepPrintNee
0,7203
0.5159
0.5591
0.3086
0.0000
0.8518
0.8267
0.5925
MMASleepNel Tubo el
2012
0.7997
0.6587
0.6885
0.7988
0.0449
0.8706
0.8376
0.7415
SimCLk Chen
2020
0,8350
0.7586
0.7577
0.8913
0,3653
0.8893
0.8267
0.8204
DeFuse You el st.
2034
0.8375
0.7513
0.7573
0.8684
0.3278
0.8995
0.8455
0.8153
MERI
0,8330
0.7659
0.7508
0.8768
0.4255
0.8888
0,81
0.8276
Ours
0.8517
0.7871
0.7798
0.8955
0,4478
0.9046
0.8502
0.8376
FeatureNer
202
0.7970
0.7102
0.6939
0.8056
0.3223
0.8647
0.8022
0.7562
DeepSleepNet (Supratak et 2017
0,8025
0.7125
0.7036
0.7940
0.31
0,8689
0.8053
0.7833
AttSSleep Bladle al 2021
0.8006
0,7044
0,6966
0.8037
0,2878
0.8673
0.7919
0.7715
DAN
Tang 20223
0.7478
0,5804
0.6137
0.6088
0.0000
0.8600
0.7672
0.6662
BSTT & Jun 2175
0.7740
0.6956
0.6608
0.7291
0.3846
0.8627
0.7733
0.7281
BOG
XSleepNa
0,7804
0.6984
0.6721
0.7754
0.3494
0.8516
0.7587
0.7570
SleepPrintNer eul 2040
0.1891
0.0922
0.0239
0.1285
0.0008
0.0000
0.0000
0.3317
MMASleepNet Tubo al
0.2988
0.1235
0,1887
0.0000
0.3139
SimCLR Chen a
2020
0,8000
0.7098
0.6993
0.7840
0.3262
0.8754
0,8084
0.7548
DeFuse
0.7805
0.7659
0,3973
0.8653
MERI
I 2 ml.
2024
0.7941
0.7142
0.6948
0,8025
0.3433
0.8622
0.7792
0.7835
Oars
0.8227
0.7534
0.7359
0.8358
0.4329
0,8854
0,8099
0.8031
FeatureNet [Jia et 2021b
0.5327
0,3664
0,2439
0.4860
0.1469
0.6539
0.0680
0.4773
DecpSleepNet (Supratak al 2017
0.5232
0.2176
0.0815
0.2376
0.0097
0.6790
0.0000
0.1619
AutSleep Brocle al.
0.5267
0,2708
0.3196
0.0435
0.6751
0.3048
DAN Lang el sill Z0z23
0.5126
0,2651
0,1294
0.4137
0.0233
0.6606
0.0000
0.2276
0.5187
0.2137
0.1157
0.0040
0.0000
0.3150
EMG
XSleepNet Phan et (
0.5252
0,3062
0.1812
0.4361
0.0642
0.6628
0.0003
0.3676
SleepPrintNc Dal
2UAU
0,2280
0.1840
0,1133
0.2552
0.1762
0.0000
0,0000
0,4887
MMASleepNel Tubo
0.3062
0.2293
0.1296
0.3235
0.0928
0.2765
0.0000
0.4536
SimCLR
Clen af
2020
0.5287
0,3664
0,2431
0.4929
0.1389
0.6484
0,0783
0,4734
DrFuse
0.5059
0.3597
0.2216
0.4999
0,1808
0.6374
0,1443
0.4760
MERI. T.ll C1 al
2024
0.5084
0.3761
0.2346
0.3470
0.2237
0.6296
0.1545
0.5257
0.5408
0.3770
0.2613
0.4991
0.1220
0.6578
0,0988
0. 0,5069
17

---

Published as a conference paper at ICLR 2025
Table 6: The performance comparison of state-of-the-art methods and SleepSMC on Sleep-EDF- 78 dataset for the unimodal testing scenario. The bold and underline items denote the best and second-best results, respectively.
Overall results
F1 for each category
Modality
Method
Accuracy
Macro Fl
Kappa
Wake
NI
N2
N3
REM
Nea
al.
202
0.7772
0.6928
0.9123
0.3039
0.8154
0.7860
0.6542
DeepSleepNet
(Suprata A
2017
0,7588
0.6740
0.6655
0.8829
0.2554
0.8187
0.8011
0.6118
Steep [imeee araie
0.7789
0.7069
0.6947
0.8996
0.3425
0.8289
0.7830
0.6803
DAN Thnn er al. 2024
0,7111
0.5638
0.5876
0.8788
0.0584
0.7685
0,7029
0,4104
BSTI Liu Jil 2016
0.7320
0.6503
0,6266
0.8555
0.3101
0.7632
0.6426
0.6801
EBG
XSleepNat Plan [222
0,7685
0.6812
0.6782
0,9038
0.3049
0.8141
0.7420
0.6409
SleepPrintNar Uia et al 2020
0.6343
0,5500
0.4975
0.7610
0.1300
0.7233
0.7093
0.4266
MMASleepNer Yubo
0.6811
0.5124
0,5416
0.8356
0.0531
0.7488
0,4435
0,4810
SimCLR Chen al 2020
0.7824
0.6941
0.6974
0.9170
0.2931
0.8187
0,7919
0.6496
DrFuse
0.7753
0.6880
0.6853
0,9018
0,2947
0.8200
0,7948
0.6289
MERL Lau cl al.
20243
0,7660
0.6848
0,6770
0,9082
0.3168
0.7956
0,7495
0.6538
Ours
0.7959
0.7217
0.7169
0.9197
0.3581
0.8280
0.7991
0.7034
FeatureNet (Ji al. 202
0.7238
0.6375
0.6159
0.8468
0.2920
0.7587
0.6259
0.6641
DeepSleepNet (Supratak aral 2017
0.7192
0.6240
0.6041
0.8474
0.2472
0.7509
0.6667
0.6075
AttnSleep [Pinete
0,7231
0.6395
0.6139
0.8561
0.29339
0.7529
0.6475
0.6470
DAN Tangerrt Züzea
0.6538
0.4130
0,4890
0.8298
0.0010
0.7020
0.0836
0.4484
BSTT
0,7110
0.6314
0,5995
0.8348
0,3112
0.7482
0.6203
0.6422
BOG
XSleepNat Plan el 202
0,7053
0.6149
0,5888
0.8215
0,2576
0.7564
0.6327
0.6061
SleepPrintNat Una et al. 2020
0,4182
0,3979
0.6571
0,2702
0.6854
MMASleepNet Yubo er
2072
0,6076
0.4518
0,4463
0,7038
0,2826
0.6815
0,0332
0,5577
SimCLR Chen al
2020
0.7239
0.6334
0,6156
0.8467
0,2713
0.7582
0.6250
0.6656
DrFuse Yau
0.6807
0.5692
0.5570
0.8328
0,3548
0.7280
0,2902
0.6404
MERI Tau cl al:
2124
0.6855
0.6034
0.5678
0.8179
0.2831
0.7323
0.5706
0.6130
0.7311
0.6494
0.6258
0.8561
0,3140
0.7618
0.6409
0.6744
FeatureNet (Ja et al.
2021b
0.5180
0.3076
0,2849
0.6710
0.0241
0.5407
0.0198
0.2825
DeepSleepNet (Supratak d 1017
0.5157
0.2873
0,2735
0.6714
0.0175
0.5419
0.0000
0.2056
Attnplepp [ireete er alr 20241
0.5180
0,2833
0,2749
0.6760
0.0205
0.5528
0.0156
0.1514
DAN Tang crain 20222
0.4913
0,2615
0,2353
0.6429
0.0000
0.5035
0,0000
0.1613
BSTT &iu 2035
0,5147
0.3026
0.2786
0.6661
0.0210
0,5389
0,0173
0.2696
EMG
XSleepNat Phan al 2022
0.5242
0,3095
0,2930
0.6781
0.0234
0.5392
0.0011
0.3059
SleepPrintNat
2020
0.3563
0,2096
0.0497
0,4970
0.0237
0.2921
0,0000
0.2350
MMASleppNet Yabo al
2022
0,3779
0,2254
0.0736
0.4740
0.0242
0.4067
0.0000
0.2219
SimCLR Chen
2020
0,5182
0.2935
0,2805
0.6753
0.0219
0,5412
0,0000
0.2288
DeFuse 3a el 20
2024
0.4058
0,2899
0,1573
0,5439
0.1177
0.4309
0.0687
0,2881
MERI
0.4208
0,3080
0,2025
0.5734
0.1459
0.3884
0,0042
Ours
0,5293
0.3167
0.3017
0.6814
0.0219
0,5529
0,0204
0.3071
Table The performance comparison of state-of-tte tfrt---rtt methods and SleepSMC on the ISRUC-S1 dataset for unimodal testing scenario. The bold and underline items denote the best and second- best results, respectively.
Overall results
for each category
ISRUC-S1
Method
Accuracy
Macro F1
Kuppa
Wake
NI
N2
N3
REM
BSTT Lio Jia
2023
0.6840
0.6367
0.5878
0.8060
0.2983
0.6794
0.7881
0.61
XSleepNe Paan
2021
0.7092
0.6735
0.6233
0.8407
0.4051
0.7122
0.8078
0.6019
EBG
DrFuse Yao et at 2022
0.6978
0.6620
0.6087
0.7317
0.3883
0.7208
0.7893
0.6797
MFRI
0.7096
0.6690
0.6223
0.8034
0.3618
0.7042
0.7917
0.6837
Ours
0.7328
0.6959
0.6536
0.8664
0.4035
0.7145
0.8085
0.6865
BSTT Liu
Jin
2023
0.3123
0.1115
0.0019
0.0807
0.0000
0.4767
0.0001
0,0000
XSleepNa Paam Piim o 21
2021
0.6320
0.6047
0.5229
0.7025
0.3460
0.6446
0.7203
0.6100
BOG
DrPuse Yao ét 2022
0.6539
0.6223
0.5466
0.7306
0,3632
0.6610
0,7528
0.6042
MERL Liu of al, Ours
2024
0.6579
0.6261
0.5573
0.7005
0,3639
0.6482
0.8027
0.6152
0.7066
0.6753
0.6156
0,8264
0,3825
0.6901
0,7989
0.6784
BSTT Liu & Jia 2025
0.3155
0.0959
0.0000
0.0000
0.0000
0.4797
0.0000
0.0000
XSkeepNe Pain DrFuse You el al
2021
0.3883
0.3636
0.2083
0.4413
0.1315
0,3854
0.3553
0,5044
EMG
2024
0.3395
0.2528
0.1418
0.4906
0.0697
0.3662
0.0196
0.3181
MERL Liu 20249
0.3786
0.3458
0.2012
0 50506
0.1779
0.3902
0.2308
0.4235
0.4190
0.3705
0.2382
0.5601
0.0748
0.4190
0.3685
0.4325
18

---

Published as a conference paper at ICLR 2025
Table 8: Ablation study of feature weights on public datasets, The bold items denote the best results.
Overall Results
Dataset
Weight
Modality
Accuracy
Macro FI
Kappa
0.7910
0.7755
0.7315
Multimodal
Cap(
0.7930
0.7815
0.7344
0.7501
0.7259
0.6794
ISRUC-S3
earpi -
EEG
0.7646
0.7397
0.6969
0.7317
0.7087
0.6541
BOG
expi
0.7444
0.7168
0.6697
0.4194
0.4018
0.2523
expi (
EMG
0.4384
0.4075
0.2693
0.8683
0.8193
0.8050
Multimodal
eapi P
0.8686
0.8193
0.8058
0.3474
0.7779
0.7731
EEG
eapp
0,8517
0.7871
0.7798
MASS-SS3
0.8195
0.7478
0.7305
EOG
expl
0.8227
0.7534
0.7359
0.5350
0.3734
0,2514
0.5408
0.3770
0.2613
19

---

Published as a conference paper at ICLR 2025
A.5 DETAILED ANALYSIS AND PROOFS IN SECTION 4
We seek to establish a robustness bound that quantifies the amount of useful information transferred from auxiliary modalities to the primary modality under perturbations. This robustness is crucial for ensuring reliable learning and decision-making in scenarios where the auxiliary modalities might be noisy or uncertain. Details are follows:
Definition. We introduce the notations and mathematical definitions used in this section:
1) The primary modality is denoted as x; and the auxiliary modality as x;",, where up and u. rep- resent the primary and auxiliary modalities, respectively. These modalities provide complementary sources of information for classification tasks.
2) Feature maps extracted from the primary and auxiliary modalities are denoted as f,"" and f respectively. These features are assumed encapsulate modality- representations of the input data.
3) The uncertainty weight whth for the auxiliary modality is defined as:
- exp(-rr"")
(1I)
where r is the uncertainty estimate of the auxiliary modality. A lower uncertainty weight (w²") corresponds to higher uncertainty (r(-) in the auxiliary modality, thereby reducing its contribution to the overall learning process.
4) The perturbation between the auxiliary and primary modalities given by:
8,
(12)
This represents the feature-level discrepancy between the two modalities, which plays a key role in quantifying the robustness information transfer.
5) The classification margin M, for the primary modality is defined as:
M4 = 0." max
(0. "2) kk
(13)
where y/ is the true category label, and [] modality. The margin M, measures the separation between the score of the true category and the highest score among incorrect categories. A larger margin implies a more confident decision by the classifier.
la is the logit of the k-th category for the primary
6) The total information transfer defined as:
Ua ua- from auxiliary modalities to the primary modality up is total
Nu -1p
11. -
(14)
total
This term quantifies the cumulative contribution of auxiliary modalities to the primary modality. The
term whe scales the contribution based on uncertainty, while the L2-norm [| ff" captures feature-level similarity between modalities.
7) The perturbation radius is the minimum L2-norm of ó, that satisfies the margin regularization constraint:
R= min || subject to M, 2 max 0. - The c);
(15)
where is small margin parameter. The perturbation radius R provides quantitative measure of the model's robustness under feature perturbations.
Theorem 1. Suppose the auxiliary modality u, is associated with uncertainty weight w:""
exp(-r."* ), where r: 2 0 quantifies the uncertainty of uo. Assume the classification margin M; satisfies the regularization condition:
M Z max 0."" -. -),
(16)
where > is small positive margin parameter, y."h and ú¹"" are the logits of the true category predicted by the primary and auxiliary modalities, respectively.
Under this condition, the following results hold:
20

---

Published as a conference paper at ICLR 2025
1) The perturbation radius R satisfies:
min ||1:ll2 with ||6:1l2 2 w/- Il5 f1 112.
(17)
62
The perturbation radius quantifies the smallest discrepancy between features of the primary and auxiliary modalities that still maintain the margin regularization condition. The scaling by w."" ensures that modalities with higher uncertainty contribute less to the effective perturbation.
2) The total information transfer / + is bounded by:
Σ w 1Ag
11 - -st 1 ll2
Ma total
Σ
(18)
This bound ensures that the total contribution of auxiliary modalities the primary modality is appropriately constrained by the uncertainty weights we and the feature similarities.
3) A minimax optimization ensures the robustness of the primary modality:
min max |61.12.
(19)
subject to Mammooth 2 e. The minimax formulation addresses the worst-case scenario by minimizing the maximum possible perturbation under the uncertainty constraints. This strategy ensures that the model remains robust even when auxiliary modalities have high levels of uncertainty.
Proofs of Theorem 1:
1) Perturbation Radius and Margin Regularization: The perturbation radius R is defined as:
= min || subject
M, 2 max(0, 0." - e).
(20)
Incorporating uncertainty weights n the scaled perturbation satisfies:
|01112 w. I1 f.2 112-
(21)
2) Smoothing Process: To account for random fluctuations, the smoothed margin M;mooohh is defined as:
T
Mamooth Σ M.(),
(22)
where T is the number of perturbations. This smoothing process ensures the model's robustness across multiple perturbation instances.
3) Bounds on Information Transfer: Using the definition of total information transfer:
total
Me Hip
115,"*
(23)
The bounds are established by considering the extreme cases of maximum and minimum uncertainty weights:
Σ Bkm
w th 115
1 112 S
M
Mam
1n - 1 112
(24)
4) Minimax Optimization: To ensure robustness under worst-case conditions:
min max | w
(25)
subject to M. 2 This ensures the robustness of the primary modality by minimizing the impact of high-uncertainty auxiliary modalities on the optimization objective.
This concludes the proof.
21

---

Published as a conference paper at ICLR 2025
A.6 DISCUSSION AND LIMITATIONS
Our limitations include that our method simulates missing signals to assess auxiliary modality qual- ity, but it does address cases where primary or auxiliary modalities are actually missing in real-world scenarios. Additionally, we did not conduct experiments on datasets with ubiquitous modalities (c.g., ear-EEG). We plan address these limitations in future work.
In addition, we sincerely appreciate reviewer 'Dg5B' for their valuable suggestions. Indeed, our method has the potential be applied other multimodal fields and achieve meaningful benefits, and we plan to explore this further in the future. We also extend our gratitude to reviewer "A2VX' for their insightful feedback. In this study, we primarily focus on single-modalig sleep staging experiments to ensure more comfortable and practical approach for ubiquitous scenarios. Moving forward, we aim to investigate additional combinations of modalities to enhance our methodology further.
22


Reviews:
[{"review_id": "BMqGsmb7L0", "reviewer": "Reviewer_Dg5B", "summary": "This paper presents SleepSMC, a sleep stage classification framework that integrates uncertainty-based feature reweighting with modality-level contrastive learning to handle multimodal physiological data (EEG, EOG, EMG) effectively. The reweighting mechanism assigns weights to auxiliary modalities based on their uncertainty. Then, they use contrastive learning to align representations of the same sleep stage across different modalities. SleepSMC achieves modality-invariant embeddings that allow for robust performance even when only a single primary modality is available at test time. SleepSMC is evaluated on three public datasets, consistently outperforming SOTA baselines in both multimodal and unimodal settings.  They present ablation studies demonstrating that reweighting and contrastive learning are both effective individually and moreso when combined. Visualization analyses further demonstrate the model’s interpretability, showing clear, well-separated embeddings for each sleep stage.", "questions": "An important aspect of multimodal classification models is cross-subject generalization. Given that real-world applications often involve new subjects with varying characteristics, an evaluation on unseen subjects would provide valuable insight into the model’s robustness. If cross-subject generalization was analyzed, could the authors include these results? Otherwise, adding this analysis would strengthen the paper’s practical relevance.\n\nAdditionally, while the reported improvements are statistically significant, they appear relatively modest, with gains in the range of 2-3%. Could the authors elaborate on why these incremental gains are meaningful in the context of sleep staging? Providing additional context on the impact of these gains for real-world applications would clarify the value of these results.\n\nAlthough the reweighting technique is effective within sleep staging, it would be useful to understand its potential applicability beyond this domain. Do the authors see possible applications of uncertainty-based reweighting in other multimodal learning tasks? Expanding on this would highlight the versatility and broader impact of the proposed method.\n\nIn terms of practical applications, it would be helpful to know how this model performs in real-world scenarios where data quality and availability are less controlled. Could the model handle dynamic shifts in modality reliability, and do the authors envision practical deployments for health monitoring or similar applications?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This paper presents SleepSMC, a sleep stage classification framework that integrates uncertainty-based feature reweighting with modality-level contrastive learning to handle multimodal physiological data (EEG, EOG, EMG) effectively. The reweighting mechanism assigns weights to auxiliary modalities based on their uncertainty. Then, they use contrastive learning to align representations of the same sleep stage across different modalities. SleepSMC achieves modality-invariant embeddings that allow for robust performance even when only a single primary modality is available at test time. SleepSMC is evaluated on three public datasets, consistently outperforming SOTA baselines in both multimodal and unimodal settings.  They present ablation studies demonstrating that reweighting and contrastive learning are both effective individually and moreso when combined. Visualization analyses further demonstrate the model’s interpretability, showing clear, well-separated embeddings for each sleep stage. \n\nOriginality\nThe uncertainty reweighting technique is clever and novel.  Most importantly, it is both empirically and theoretically effective. The contrastive learning component, combined with reweighting, enables the model to create modality-invariant embeddings, allowing SleepSMC to generalize effectively across different primary modalities.\n\n\nQuality\nThe quality of the work is high. The evaluation is thorough, providing empirical, theoretical, and qualitative support for their approach. Reweighting consistently outperformed non-reweighted setups in both multimodal and unimodal testing, with clear gains in accuracy, Macro F1, and Kappa scores, particularly under noisy conditions. Combining reweighting with contrastive learning further boosted performance compared to contrastive learning alone, showing that these methods work well together, especially with unreliable auxiliary modalities. SleepSMC also generalized effectively across target modalities, performing robustly whether EEG, EOG, or EMG was used as the primary modality, demonstrating flexibility across configurations.\nIn both multimodal and unimodal testing, SleepSMC maintained high accuracy, with unimodal performance benefiting notably from reweighting. t-SNE visualizations showed clear, well-separated clusters for each sleep stage, supporting the model’s modality-invariant embedding space and interpretability. These results confirm that SleepSMC’s reweighting and contrastive learning mechanisms enhance robustness and adaptability, making it a practical solution for real-world sleep staging.\n\n\nClarity\nThe work is mostly clearly presented. The methodology and results are well-structured, and the theoretical analysis is solid, providing strong support for the approach. However, the introduction of mathematical notation in Section 3 could be organized more effectively; starting with a table or list of symbols and definitions before diving into the equations would help clarify the notation and reduce cognitive load for the reader. The paper is otherwise transparent in discussing its limitations, and the overall clarity of the empirical findings is strong.\n\n\nSignificance\nThe work makes a meaningful contribution, adding to the literature on multimodal integration, particularly in scenarios where only unimodal data is available at test time—an approach that aligns well with many real-world applications.", "weaknesses": "1. The introduction of mathematical notation in Section 3 is somewhat disorganized and could be clarified better. Starting this section with a concise table or list of symbols and definitions would provide readers with a quick reference point, making it easier to follow the subsequent equations. This adjustment would enhance readability, especially for readers less familiar with the specific notation conventions used.\n\n2. The work makes a solid contribution to the field, with gains in the range of 2-3%, which are statistically significant but don’t drastically improve performance over baseline methods. While these improvements are valuable they're somewhat incremental.  Perhaps the gains are more significant in other multimodal learning problems.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5WsaqlQSHJ", "reviewer": "Reviewer_iRq4", "summary": "This paper introduced multimodal collaboration in sleep staging, leveraging multiple auxiliary modalities to improve the performance of primary modality-based sleep staging in an end-to-end manner. The authors utilized supervised modality-level instance contrastive coordination to capture category-related consistency and complementarity across intra-modality and inter-modality.", "questions": "1. How the proposed method is different from SOTA?\n2. What is the novel contribution that makes this paper unique?\n3. What is the computational complexity?\n4. How equation 10 become optimized?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Experiments are performed using 3 different datasets. The organization of the paper is good.", "weaknesses": "The limitations are as follows:\n\nThe motivation is not clear.\nHow this contribution is different from the following contributions? a)CoRe-Sleep: A Multimodal Fusion Framework for Time Series Robust to Imperfect Modalities,\" in IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 32, pp. 840-849, 2024, doi: 10.1109/TNSRE.2024.3354388. b) Multi-Modal Sleep Stage Classification With Two-Stream Encoder-Decoder,\" in IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 32, pp. 2096-2105, 2024, doi: 10.1109/TNSRE.2024.3394738.\nThe motivation behind the use of Uncertainty Estimation with Frozen Gradients is not clear.\nThe methods mentioned in Table 2 are from before 2022. It is necessary to compare with some recent SOTA methods", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JZg0VvcbDA", "reviewer": "Reviewer_HHyg", "summary": "To address the issue that only one modality is available in ubiquitous scenarios, this paper introduce multimodal collaboration in sleep staging, leveraging multiple auxiliary modalities to improve the performance of primary modality-based sleep staging in an end-to-end manner. The paper utilize supervised modality-level instance contrastive coordination and uncertainty estimates to learn coordinated features. The experiment results show that the proposed method achieves SOTA performance in multimodal scenarios and unimodal scenarios.", "questions": "Please See Weakness", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper aims to address the issue that only one modality is available in ubiquitous scenarios, which has certain practical significance.\n\nThe paper utilizes uncertainty estimates to adaptively weight auxiliary modality features during\ntraining, which ensures that more reliable auxiliary modality features contribute more significantly to the contrastive learning process.\n\nThe presentation of the paper is generally quite clear.", "weaknesses": "(1) This paper does not introduce a new concept; rather, it applies existing multimodal coordination and uncertainty estimates to the ubiquitous scenarios of sleep staging. Moreover, the proposed method for improving sleep staging does not seem to differ significantly from the multimodal coordination methods already used in areas such as vision. Some design details in the method, such as certain aspects related to uncertainty estimates, do not demonstrate a specific focus on the sleep staging task. \n\n(2) The authors seem to use a single-epoch sleep staging paradigm instead of a sequence-to-sequence sleep staging approach. Why is this the case? In fact, there is a strong correlation between adjacent epochs in sleep staging, and the single-epoch paradigm has already fallen behind in performance and been largely abandoned. As a paper aimed at improving sleep staging performance, it is difficult to understand the choice of using a single-epoch paradigm. Could the authors explain why this paradigm was chosen and whether the proposed method can be adapted to the sequence-to-sequence paradigm?\n\n(3) The reviewer has some concerns about the experimental design and results. The baselines chosen by the authors appear to be relatively weak and outdated, such as the selection of SimCLR, which is a very old self-supervised learning method. The baselines in the paper perform very weakly in multimodal scenarios (shown in Table 1), making it difficult to demonstrate the advantages of the proposed method. Some very strong baselines, such as SalientSleepNet [1], BSTT [2] and XSleepNet [3] were not compared. Meanwhile, the reviewer cannot understand why the authors chose ISRUC-S3 instead of ISRUC-S1 as the primary evaluation dataset. ISRUC-S1 has more subjects and a larger data volume, making it more convincing compared to ISRUC-S3.\n\n(4) Although the presentation of the paper is relatively clear, the content in \"DETAILED ANALYSIS AND PROOFS IN SECTION 4\" is difficult for readers to understand. As a supplement and proof of certain concepts in the main text, this section does not sufficiently clarify the formal definitions of each concept and the complete proof process. For example, what is the complete proposition that needs to be proven in this section? What are the assumptions underlying the proof process? What is the formal definition of \"margin\"? What is the formal definition of \"information transfer\"? The authors should provide clear explanations and complete definitions of these concepts in the appendix to ensure the readability of the paper.\n\n[1] Jia Z, Lin Y, Wang J, et al. SalientSleepNet: Multimodal salient wave detection network for sleep staging[J]. arXiv preprint arXiv:2105.13864, 2021.\n\n[2] Liu Y, Jia Z. Bstt: A bayesian spatial-temporal transformer for sleep staging[C]//The Eleventh International Conference on Learning Representations. 2023.\n\n[3] Phan H, Chén O Y, Tran M C, et al. XSleepNet: Multi-view sequential model for automatic sleep staging[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 44(9): 5903-5915.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4uM3e05J4x", "reviewer": "Reviewer_A2VX", "summary": "The manuscript proposes SleepSMC, a method for detecting sleep stages leveraging contrastive learning and feature weighting based on uncertainty. The approach is specifically designed to handle scenarios where multiple data modalities are available during training, but only a single modality is accessible during testing (multimodal scenario has also been evaluated). Using three publicly available datasets, the authors demonstrate that SleepSMC achieves performance improvements over existing baselines.", "questions": "[Problem setting & method] \n1. Could the authors clarify the rationale for assuming that only one modality is available during inference? While it is understandable that not all modalities used in training may be accessible at test time, it seems uncommon to assume the presence of only a single modality. A more practical scenario might involve each user having a unique set of available modalities at test time, rather than relying on one fixed modality. Could the authors elaborate on the motivation behind this specific setting?\n\n2. Would it be expected that the auxiliary modality classifiers exhibit higher uncertainty at the beginning of training? If so, could this uncertainty indicate an opportunity for the model to learn more effectively? Did the authors experiment with dynamic tuning of weights over the course of training? Additionally, numerous metrics (such as entropy, model confidence, loss) could potentially assess sample importance. What was the rationale for selecting model confidence? Were other metrics considered, and if so, what were the comparative results?\n\n[Related work & Baselines]  \n1. In Section 2, Liu et al. (2023b) and Liu et al. (2024) are referenced as leveraging multimodal data to enhance performance, with the former employing contrastive learning to align modalities. Could the authors clarify why these methods were not considered as baselines? Additionally, it is noted that these works focus on multimodal consistency without capturing class-specific information. However, the design choices of related works should not necessarily be considered limitations. Could the authors discuss the impact of excluding class-specific information? How can we assess the relative performance without a comparison to the proposed method?\n\n2.  In Section 2, several works related to uncertainty estimation are discussed, but the conclusion remains somewhat ambiguous. Why are these methods not applicable to the target scenario? What challenges prevent their direct application to the proposed task, especially given that multimodal data are still used during training?\n\n[Results]\n1. In most scenarios, including unimodal cases, the primary improvements appear to stem from supervised contrastive learning. However, in Section 5.5, it is stated that “uncertainty-based feature weighting has a greater impact in the unimodal testing scenario,” which may be an overstatement. Could the authors provide comparative improvement results (e.g., average ± standard deviation) for both components? If the enhancement from uncertainty-based feature weighting is minor, what justifies its inclusion?\n\n2. Could the authors explain the observed fluctuations in EEG and EOG uncertainty weights as training progresses (as seen in Figure 3c)?\n\n[Minor comments]:\n- Font size in Figure 3 appears small.\n- Class labels in Figure 4 are difficult to read; increasing the legend font size could improve readability.\n- Typo in Section 4.2, first sentence: “adaptively weighted” should be “adaptively weights.”", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-structured and clear, and easy to follow.\n- Includes extensive experiments comparing the proposed method with various baselines.", "weaknesses": "- Limited technical novelty.\n- More recent & missing-modality-specific baselines could be considered as stronger baselines.\n\n[Details]\n1. Could the authors explain why \"Supervised Contrastive Learning\" by Khosla et al. (2020) was not considered as one of the baselines? What specific motivations underlie the use of contrastive learning here, and in what ways does the proposed technique differ (in terms of technical novelty) from the original approach in Khosla et al.? \n        - Khosla, Prannay, et al. \"Supervised contrastive learning.\" Advances in Neural Information Processing Systems 33 (2020): 18661-18673.\n\n2. Is there any method or technique in this work that is specifically tailored for sleep stage detection? The proposed approach appears applicable to most multimodal data; If that's the case, other recent works targeting learning/inference with missing modalities could be considered as baselines as well:\n    - Wang, Hu, et al. \"Multi-modal learning with missing modality via shared-specific feature modelling.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n   - Yao, Wenfang, et al. \"DrFuse: Learning Disentangled Representation for Clinical Multi-Modal Fusion with Missing Modality and Modal Inconsistency.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 15. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)