Robust Sleep Staging over Incomplete Multimodal Physiological Signals via Contrastive Imagination
Authors: Qi Shen
Junchang Xin
Bing Tian Dai
Shudi Zhang
Zhiqiong Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: Automated sleep staging; Incomplete modality; Multimodal learning; Contrastive learning; Temporal context learning
Primary area: machine_learning_for_healthcare

Abstract: Multimodal physiological signals, such as EEG, EOG and EMG, provide rich and reliable physiological information for automated sleep staging (ASS). However, in the real world, the completeness of various modalities is difficult to guarantee, which seriously affects the performance of ASS based on multimodal learning. Furthermore, the exploration of temporal context information within PTSs is also a serious challenge. To this end, we propose a robust multimodal sleep staging framework named contrastive imagination modality sleep network (CIMSleepNet). Specifically, CIMSleepNet handles the issue of arbitrary modal missing through the combination of modal awareness imagination module (MAIM) and semantic & modal calibration contrastive learning (SMCCL). Among them, MAIM can capture the interaction among modalities by learning the shared representation distribution of all modalities. Meanwhile, SMCCL introduces prior information of semantics and modalities to check semantic consistency while maintaining the uniqueness of each modality. Utilizing the calibration of SMCCL, the data distribution recovered by MAIM is aligned with the real data distribution. We further design a multi-level cross-branch temporal attention mechanism, which can facilitate the mining of interactive temporal context representations at both the intra-epoch and inter-epoch levels. Extensive experiments on five multimodal sleep datasets demonstrate that CIMSleepNet remarkably outperforms other competitive methods under various missing modality patterns. The source code is available at: https://github.com/SQAIYY/CIMSleepNet.

Robust Sleep Staging over Incomplete Multimodal Physiological Signals via Contrastive Imagination
Qi Shen', Junchang Xin', Bing Tian Dai', Shudi Zhang', Zhiqiong Wang'*
Coolleee of Medicine and Biological Information Engineering, Northeastern University, China College of Computer Science and Engineering, Northeastern University, China Schooo of Information Systems, Singapore Management University, Singapore 2210521@stu, neu. edu. cn, xinjume janjjnnhhnnmmill.... neu. edu. en, btdai @smu edu. sg, 23105350stu tu neu. edu. cn, wangzq@bmie new. edu. cn
Abstract
Multimodal physiological signals, such as EEG, EOG and EMG, provide rich and reliable physiological information for automated sleep staging (ASS). However, in the real world, the completeness of various modalities is difficult to guarantee, which seriously affects the performance of ASS based on multimodal learning. Furthermore, the exploration of temporal context information within PSs is also serious challenge. To this end, we propose a robust multimodal sleep staging framework named contrastive imagination modality sleep network (CIMSleepNet), Specifically, CIMSleepNet handles the issue arbitrary modal missing through the combination of modal awareness imagination module (MAIM) and semantic & modal calibration contrastive learning (SMCCL... Among them, MAIM can capture the interaction among modalities by learning the shared representation distribution of all modalities. Meanwhile, SMCCL introduces prior information of semantics and modalities to check semantic consistency while maintaining the uniqueness of each modality. Utilizing the calibration of SMCCL, the data distribution recovered by MAIM is aligned with the real data distribution. We further design a multi-level cross cross-branch temporal attention mechanism, which can facilitate the mining of cross-scale temporal context representations at both the intra-epoch and inter-epoch levels. Extensive experiments on five multimodal sleep datasets demonstrate that CIMSleepNet remarkably outperforms other competitive methods under various missing modality patterns. The source code is available at:
Introduction
Automated sleep staging (ASS) is essential promote sleep quality assessment and sleep disorder diagnosis, providing convenience for the public the daily monitoring sleep within their home environment. Many machine learning algorithms, including feature engineering and deep learning, have been proposed for ASS [1, 2, 3, 4, 5]. In particular, deep learning methods represented by convolutional neural network (CNN) have achieved remarkable results in the field of ASS [6]. Compared with feature engineering, deep learning does not require the guidance of prior knowledge and has the advantage of automatically extracting physiological signals (PSs) features.
In clinical applications, due to the complexity of human physiological states, subjects usually need to wear multiple sensors obtain more comprehensive and integrated physiological information from multimodal PSs collected from different sources [7]. Hence, several multimodal fusion algorithms
CCoresspodiigg author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Theoretical scenario
Real-world scenario
Complete modality
Incomplete: modality
Sleep staging
Sleep staging
(a)
(b)
Figure The distribution of multimodal data in different scenarios. (a) exhibits the complete modality, and (b) exhibits the incomplete modality.
[8, 9, 10, 11] based deep learning have been developed to cope with the challenges of multimodal ASS. Although various multimodal fusion algorithms provide guarantees for automated processing and analysis of these multimodal PSs, they still have some limitations. As illustrated in the Fig. 1 (a), existing methods are almost all conducted under the assumption that all modal data are complete. However, in real scenarios, the modal data will be incomplete due sensor malfunctions or detachment, as shown in the Fig. 1 (b). Unfortunately, the second scenario will seriously affect the reasoning process of algorithms, resulting
sharp decline in performance [12].
Further, how to mine dynamic temporal changes and complex sage-transitioning patterns in PSs is another challenge for ASS. Most sleep staging works [13, 14, 15, 16, 17] utilize recurrent neural network (RNN) and its variants to model temporal dependencies within learnable hidden states. Recently, due to its efficient parallel computing abilitty and powerful global context modeling ability, Transformer has gradually become the preferred alternative to RNN in the ASS field [18, 19, 20]. However, Transformer lacks the recurrent modeling abilities of RNN, which is crucial for mining the structural representations and positional embedding of input sequences [21, 22]. Meanwhile, most methods are limited to mining temporal correlations at single level in PSs, i.e., intra-epoch level inter-epoch level. These issues make it difficult for existing temporal models to fully understand the complex variability patterns in PTS, thereby affecting the performance of sleep staging.
Considering the above challenges, we propose a robust multimodal sleep staging framework named contrastive imagination modality sleep network (CIMSleepNet),, suitable for scenarios with incom- plete modalities. The core contributions of CIMSlcepNeet are summarized as follows.
We first design modal awareness imagination module (MAIM), which can realize the imputation of missing modalities to restore the completeness of the various modalities. MAIM leverages the distribution of available modalities as prior conditions learn multi- modal shared representations and enhance the inter-modal correlation, thereby improving the recovery process of missing modalities.
We provide a novel insight into the impact of the intrinsic connection between semantic and modality on data distribution. Hence, a semantic & modal calibration contrastive learning (SMCCL) is presented modify the restored data distribution. It can utilize bidirectional guidance of semantic and modality to align the restored data with the real distribution.
We further explore a multi-level cross-branch temporal attention (MCTA) mechanism that enables interactive modeling of recurrent features and self-attention weights from the intra- epoch and inter-epoch levels to yield more comprehensive temporal representations.
Extensive experiments on five multimodal sleep datasets exhibit that CIMSleepNet can significantly improve multimodal ASS performance under various missing modality patterns.
2
Related Work
Multimodal Learning for Sleep Staging: In the ASS field, several pioneering studies have been devoted to exploring how to utilize multimodal PSs acquired from various sensors to improve ASS performance. Andreotti et [8] selected three polysomnography (PSG) signals related to sleep, electroencephallgram (EEG), electrooculogram (EOG) and electromyogram (EMG), as input to CNN to improve ASS accuracy. Similarly, Jia al. [23] effectively mined salient waves form multimodal sleep PSs with multimodal salient wave detection network. Lin et al. [11] designed a cross-link fusion module to eliminate redundant information in multimodal PSs. Huy al. [8] focused on the training mode of the deep model, and proposed an adaptive gradient blending strategy, which

---

improves the joint learning representation ability of multimodal PSs in different views. Furthermore, multimodal PSs collected by some consumer electronic devices have gradually been applied in ASS field. For instance, Walch et al. [24] utilized feature engineering methods to analyze human motion signals and heart rate (HR) signals collected by Apple Watch, and verified their relevance to the sleep stage. Then, Zhai et al. [9] and Mads et al. [25] further improved multimodal sleep staging performance based on consumer electronic devices by constructing a feature fusion method based deep learning. However, these studies have largely neglected the impact of incomplete modalities scenarios, which are more representative of real-world data distributions. Kontras et al. [26] ingeniously combined self-attention and aosssntttntionn mechanisms to extract coordinating representations for multimodal PSs, thereby mitigating the interference caused by missing modalities on neural network. Nevertheless, this method was developed to handle the complete absence of one or more modalities, whereas it is impractical in real-life clinical applications.
Contrastive Learning Under Missing Modalities: Invariant contrastive learning (ICL. and semantic contrastive learning (SCL.) are eurrently promising choices for solving the modality missing issue. For instance, Lin et al. [27] proposed a cross-modal ICL, aiming to utilize available modalities to achieve prediction of missing modalities. Similarly, Liu et al. [28] narrowed the gap between heterogeneous modalities through ICL for reconstructing missing modalities. SCL introduces category information on the basis of the former to achieve semantic structure preservation in missing modal cases [29, 30]. These studies focus on learning multimodal consistency representations, i.e., only recovering the multimodal shared information to deal with multimodal missing issues. However, this strategy leads to the loss of specific information unique to each modality, thereby failing to exploit inter-modal complementarity.
Temporal Context Learning in sequence modeling: It has achieved rapid development driven by the sequence-to-sequenc models. For instance, Supratak et al. [13] introduced bidirectional long short-term memory (Bi-LSTM) to learn transition rules during sleep stages. Phan et al. [14] applied bidirectional gated recurrent unit (Bi-GRU) to model contextual information of sequence representations. Phyo et al. [16] provided a Bi-LSTM equipped with two auxiliary tasks to explicitly learn periodic transition patterns. Besides, Qu et al. [18] employed Transformer to improve the ability to mine context information in parallel optimization manner. Eldele et al. [19] deployed temporal CNN to Transformer, further improving its ability to capture temporal features. Although Transformer has advantages over RNN and its variants in terms of computational efficiency and context learning, it lacks recurrent modeling ability, resulting in the omission of some important temporal attribute information [21, 22]. Furthermore, studies [22, 31, 32, 33] have proved that the features learned by RNN and Transformer are complementary. The above optimization perspective provides valuable inspiration for to design novel temporal context architectures.
3 Methodology
3.1 Problem Formulation
We first define a complete multimodal PSs dataset D = ((,,,,))) where X; is the ith multimodal epoch (sample), y, is the sleep stage label of the ith epoch and N is total number of epochs. Suppose X, contains M modities, i.c., X, = {x( )=]>>... x/ E RC where Cj and Lj are the number of channels and sampling points of the jth modality, respectively. Furthermore, yi (0, 10. K - 1}. where K is the number of sleep stage categories. Different from the complete modality missing issue of Kontras et al. [26]. we mainly focus the chunk-based missing pattern, ie., random missing in units of multiple epochs, which is a common situation in biomedical research [34]. This is mainly due to the fact that subjects tend to be interrupted for an extended period of time during the data collection. To construct incomplete modal dataset, we define a mask matrix Z - ([ J Ji=1 1=1 E NxM at the epoch level to track the missing status of modalities. If x/ is observed, 2 1; otherwise, Z =0 0. Note that, Z,J A 2 A, .ZMM-1 + O, i.e., each X; must have at least one available modality. According to the mask matrix, the missing rate of the dataset can be defined as = 1- 1 Σ. ΣM z Then, we define the incomplete multimodal PSs dataset N-M i = ((X ))) where X, and X, have the same shape, i.e., X, = ( x E CRC, After that, we reorganize the dataset DD with new shape, i.e., x² E R NxTxC; to perform temporal context modeling. Among them, N [N/T] and T is the length of contextual information. Finally,
3

---

MAIM
Missing Medality Imsurattion
SMCC
Distribution Aligniient
Modality
Modality 2
Aftract
X
Modilit--peilll Encoder
Mediiity-peeffee
Cine
Secondh Similanty
First Sinilarty
Decoder
Thinkkoad similarit
Dan Flou
Madk
Inommplae Vicw
Complete
Loss Mark
Peature Extraction and Temperat Contest Modeling
Modality-ppoits
Mndaiity-ppccffe
Temponi
X
X
CNN
Tempural
Sleep Stigita
Figure 2:
The overall framework of CMISleepNet. It consists of three main components: MAIM, SMCCL and MCTA mechanism. Two incomplete modalities, X and X are taken examples for illustration. the missing modality imputation phase, MAIM learns multimodal shared representa- tions from the available modal distribution to recover complete modalities X and X Mcanwhile, X and X² are fed into SMCCL to perform distribution alignment, making the recovered modal data closer to the real data distribution. Furthermore, temporal CNN is utilized to performer feature extraction of X and X² and obtain the multimodal fusion representation F. After that, is fed into Transformer containing MCTA for temporal context modeling to obtain the temporal representation F, which is then used for prediction of sleep stage scores. CMISleepNet also includes three objective functions: ( for missing modality imputation, (e)) for distribution alignment, goe) for sleep staging.
we also define modality matrix S = 1 5 i=1 j=l &RHxM to provide information about the modalities involved in each epoch, where sy E (0,1, M - 1} is the modal label of the jth modality of the ith epoch and H = .T.
M
As schematized in Fig. 2, we present CIMSleepNet, which aims to cope with the issues of modality missing and temporal context modeling multimodal ASS. Given incomplete multimodal PSs, we first employ MAIM to impute the missing modal data (Sec. 3.2). Meanwhile, SMCCL is utilized to modify the distribution of the recovered data (Sec. 3.3). Then, we leverage temporal CNN and MCTA embedded in the Transformer structure to perform feature extraction and temporal context modeling on the recovered complete multimodal data, respectively (Sec. 3.4). Finally, the model parameters are optimized by combining various objective functions to achieve sleep staging (Sec. 3.5).
3.2 Missing Modality Imputation
To impute missing modalities, we design MAIM, which mainly consists of M (M= in Fig. 2) modality-specific encoders E( ))- [E M and decoders D -(-)= (D)())) Each encoder
and decoder is implemented via separable temporal CNN [16] to reduce the parameter redundancy.
Modality encoder. As as depicted in Fig. 2, incomplete multimodal PSs X and mask matrix Z are transmitted in MAIM through multimodal data flow and mask matrix flow respectively. Firstly, multiple encoders are utilized project multimodal PSs into the latent space, and the latent representations of all modalities are fused in a multiply add operation, formulated as
f ΣS Z
Σ
N
E x
z
(I)
5=
where f; denotes the multimodal shared representation obtained from X, Since the modalities in the training set are also incomplete, the best choice for guiding the missing data in the reconstruction process is other available data of the same modality [30]. However, the data recovered by this way loses the diversity of the original data and cannot retain the original semantic structure. To improve the data diversity, we drew inspiration from multimodal variational autoencoder (MVAE) [35] to learn not the shared representations of multimodal PSs but their distributions. Learning diverse data ensures that the generated data is not limited to the data that guide it, making it easier for SMCCL to perform calibration. We first utilize multilayer perceptron (MIP) to obtain two vectors, 14 and as which are used to describe the mean and variance in the distribution from the f. Then, f, is subjected

---

to reparameteriizttin to obtain the latent representation ft. Formally, f, = #4 + exp ( OE where C is element-wise multiplication and E; is a random variable sampled from the distribution of f, After that, f; is mapped back into the input space of f; to get multimodal shared representation f..
Modality-speciffe decoder. In the decoding stage, f. is fed into each decoder for reconstructing {D, (f)))= Similar to MVAE, the parameters of MAIM
modality-specific data, i.c.. [x] }
guidded are optimized guided by the joint of mean square error (MSE) L' and Kullback-Leibie (KL)
divergence e'*²) ( We refer to the overall loss function as the modal imagination loss function (())
Suppose the batchsize is h, ((t) can be denoted as
Σ M
+ ne ne
M
(2)
B
7 > 2B 1= k== (1+ (o+) - (4t) (ot)²)
D
M B j=1
where B = h. T, D is the dimension of f;, is the loss weight, X is the real sample (if x/ is missing, X the random sampling of the available data in the same modality). We found that the value of n) is not sensitive, but removing e exx2) results in significant decrease in performance of CMISleepNet. Hence, we set 17 to In particular, pxx²) is used to constrain how close the latent variable distribution is to the prior distribution, prompting the decoder to generate more diverse samples. Then, the mask matrix is utilized to judge whether all recovered data is a missing state before. if x is missing, x will be used as the recovered modality; otherwise, x itself will be used. It can be expressed by mask matrix as x = x + (1 - xx
3.3 Distribution Alignment
Different from contrastive learning based on modality consistency [27, 28, 30, 29], our SMCCL introduces semantic and modal information, which not only preserves the semantic structure but also restores the specific modality information to a great extent. As illustrated in Fig. 2, SMCCL covers three similarity levels. The first-level similarity is applied to narrow the distance between different samples with two identical patterns, i.e., the same category and the same modality. Second-level and third-level similarities are utilized to correct the distribution between samples with any of the same single patterns. Note that, the constraint strength of the first-level similarity should be higher than that of the other two levels of similarity because it can be dual-guided in semantics and modality. The meaningful, and samples that meet these similarity criteria should
latter two levels of similarity not be repelled. Because these data still have semantic similarity or modal similarity. Furthermore, contrastive learning is performed within batch, and the original complete data that meets the first-level similarity standard with the restored data may not necessarily exist in a batch, which further reflects the necessity of the latter two levels of similarity.
Supposing that a batch contains B epochs, we divide the above similarity levels by constructing similarity weight matrix W = {{[ BxMM BxM To divide the similarity levels of all sample pairs, 1==1 we use the label set {ys )== and modality matrix S to introduce both semantic and modal information for each sample. We first replicate the label set, increasing its modality dimension, to obtain label weight Y = {1 ) IM Flatten two matrices and replicate in the row and column dimension to expand to R = B- M. We redefine two matrices Y = {[, 1==1 دز= and S = (13))// Then, calculate the contrastive mask matrices of Y and S, U and V, formulated as:
B
!??
U = ([14 R
1. デ
R V = (60)
L. 777
(3)
0.
where "I" is positive pair and "0" is negative pair. Besides, ū/ and Sy are the elements in Y and 5T respectively. Further, the similarity weight matrix W can be constructed by
W U (I- O(V U V)
- Uo
the 1th level
2th level
the level
where @ denotes element- multiplication and θ = ((0))) J==1 1-=1 is used to set the weights for the second-level and third-level similarity. We refer to as the modality consistency matrix and
R R
@ as the modality consistency score. In θ, ((0)) ((0)) () I is the modality consistency score of the kth modality and other modalities, which contain all the same O values. We rename
R j=1 to 02 and calculate it by the modal mutual information under @ in {{O)) (i=(k-)) --1))
k-B
5

---

information theory [36]. Taking the kth modality as an example, we use the projector & (-) composed of MLP to map the reconstructed complete modality data into a low-dimensional feature space and activate it by the Softmax function dιr( i.e., o (B&x(xx)) Formally,
1 Σ t= Ligk 8x = M
1(0";0") H( .6f)
(5)
where 1g is an indicator, when æ is true, the result is "I", otherwise it is "0", ("")"))))) is the mutual information of of and Φ, H H(0" ) is the joint entropy of &² and d'. The value range of 0g is between 0 and 1, and it can automatically adjust the ratio of the second-level and third-level similarity according to the modal consistency. For instance, if the value of 0g is larger, it means that the inter- modal consistency is higher, but the amount of specific modal information is lower. Hence, is necessary to increase the introduction of modal information, i.e., to increase the weight of the third-level similarity of formula (4). Vice versa. To more intuitively represent the construction process of the similarity weight W, we provide an example in Appendix C. To formulate Hidk 45. we define discrete joint probability distribution P(m, and two discrete marginal probability distributions P(m) and P(n). Since o² and ø² are activated by Softmax function, &f and o² can be regarded as the distribution of two discrete cluster assignment variables m and n D categories like [29, 37]. Among them, D is the feature dimensson of &² and ø'. Hence, we redefine P(m, n), P(m) and (' E C Dx RxxD P Expand( + 2d,= and P,, = Expand( D 2dm=1 P.... Po.d..)) BRDxD respectively. As a result, the discrete form of 11ah; o-) can be expressed Hio" 45)
P(n) P = (6"
P. o.. E R Bxxxx
1 H x²)
P PmP
log
(6)
The theoretical result of formula (6) are demonstrated in Appendix D. To match the dimensions of the two redefined matrices Y and S, we perform flatten operation on each batch of reconstructed data to obtain X = {xx B-M Then, we fed X into another projector g( ) for the computation of contrastive loss, i.e., = g(X) (()) { According W. we propose novel contrastive learning, SMCCL, can be defined as
((0)
B-M
B.M
Ligj 1 wu/>o wa log B.M
explip: (p//T)
EEEY Liss exp(1p; spt/T)
where ((3) named distribution alignment loss, N N is the number of w! > 0 batch and is a temperature coefficient, which is set to 0.07 like [38]. In SMCCL, g(+) adjusts the attention given different sample pairs based on W, achieving more fine-grained distribution calibration.
3.4 Feature Extraction and Temporal Context Modeling
As illustrated in Fig. 2, the recovered complete modal dataset X= [1x])_111 1M is also fed into the temporal CNN for feature extraction and concatenation obtain multimodal fusion temporal representation F E BxCC B = ht during the distribution calibration process. Among them, Cs the number of channels, D the feature dimension, h is the batch size and T is the context length. Then, we utilize Transformer composed of layer normalization (LN), MCTA, and MLP for temporal context modeling, thereby obtaining temporal representation F E BxDC We focus on introducing MCTA, with its single-head structure depicted in Fig. 3. Firstly, the fusion representation after the first LN is divided into S heads, i.e., F = {f, where f, E R BxD((())) After that, f, is fed into MCTA. It has two branches and includes intra-epoch and inter-epoch levels, which can fully mine the temporal context information of latent features.
Intra-cpoch level: In the 1th branch, temporal CNN is adopted to generate the query Q, and the key K and value V! Related study [39] have proven that temporal CNN exhibits efficiency beyond linear operations, while also eliminating the requirement for positional encoding, In the 2th branch, we use Bi-GRU learn the recurrent representation of f.. Similarly, key K. and value V from (. are obtained via temporal CNN. To achieve cross- boss-b interaction, we splice !!?)) and ( with K! (B) and V. As a result, the intra-epoch cross-branch attention can be calculated as
((T) = Intra CA. Softmax Q. KT V., K. VCIS
[KS .v..
[
(8)

---

Inter-epoch level: As shown in Fig. 3, the Ith branch and the 2th branch of the inter-epoch level exhibit reversed pattern compared to the intra-epoch level. This design enables MCTA to not only realize the interaction of cross-branch in parallel manner, but also capture rich temporal representations layer by layer. In this level, f(2) E R4xTx(D-CI3) and r E R x(D-CIS) serve as the input of the two branches respectively. In the 1th branch, similar to the 2th branch at the intra- epoch level, f!T) is mapped to f! to obtain 0 and V! In the 2th branch, f( is mapped to Q.. K, and V, by temporal CNN. Likewise, the inter-epoch cross-branch attention can be calculated as Intrs-qurch Leve
Inse-spoch Level
Tenpora CNN
(P) k 30
Commooo S
Branci
Softman
OD~B-aR
Tumoveal
ODT3"OK
Tempraal CAA
Rac MRDDee
GELL
Tomjer
Salmas
Figure 3: Design of the multi-level cross- branch temporal attention (MCTA) mechanism. D and the number of channels temporal CNN at different levels; the values of D/2 and T/2 are rounded down; k is the kernel size; st is the stride. M and N are the neuron counts of Bi-GRU at different levels, where = C/S and D C/S.
In the interactive process, MCTA can effectively integrate recurrent bias into self-attention weights to improve the shortcomings of traditional Transformer recurrent modeling ability.
T Inter CA, = Softmax( Q. KT V. K. [R 1 V. VCIS
[V.
After that, we concatenate B()) E: [4XTX(D-CIS)) and f. (DX-X(D)CIS) and perform dimensional- a
ity reduction via temporal CNN to obtain the fused representation i. R (x((() Finally, extending
single-head MCTA to multiple heads can be expressed as F f f2
[is] BR
3.5 Optimization Objective
We utilize temporal representation I BxCC to perform sleep staging, Mcanwhile, cross entropy loss ((() is regarded good choice guide the learning of model parameters, i.c..
el0)
Σ i=1=1
W, (1.,) + (1 Mis) In (1 - (i.;))
(10)
where B is the batch size, K is the number of categories, W is the category weight, is the real label and ÿ the predicted label. After that, construct the total objective loss for CIMSleepNet. Formally, = p(e) + ae(() + BB(*), where a and ,3 are the weight of the loss term.
EXPERIMENTS
4.1 Datasets and Implementation Details
Datasets: Five multimodal sleep datasets, Sleep-EDF-20 [40, 41], Sleep-EDF-78 [40, 41], SVUH- UCD [40], Motion and heart rate (MHR) [24] and SHHS [42, 43] are used for the effectiveness of CIMSSeepNet. The first four datasets are used to verify the performance of CIMSleepNet when the modality randomly partially missing, and the last dataset used to verify its performance when the modality is completely missing. We choose EEG and EOG, for SDeep-EDF-20, Sleep-EDF-78
7

---

Table 1: Performance comparison for complete and incomplete modalities in randomly partially missing case. Here "incomplete" means the maximum missing rate.
Complete
Incomplete
Datasets
Methods
Acc
MF1
K
Acc
MF1
K
FealConcat
0.825
0.761
0.771
0.497
0.429
0.285
MultitaskCNN [8]
0.835
0.753
0.775
0.589
0.506
0.449
SelientSeepPNet [23]
0.872
0.827
0.827
0.634
0.565
0.485
Sleep-RDF-20
MM-Net [1]]
0,867
0.817
0.822
0.570
0.493
0.432
TransSleep [16]
0,864
0.819
0.821
0.594
0.521
0.457
XSleepNet [10]
0,864
0.809
0.819
0.623
0.560
0.478
CIMSleepNet
0,867
0.821
0.824
0.853
0.801
0.805
FeatConcat
0.788
0.726
0.717
0.526
0.471
0.392
MultitaskCNNN [8]
0.795
0.727
0.722
0.613
0.535
0.453
SalientSleepNet [23]
0.843
0.794
0.791
0.722
0.643
0.625
Sleep-RDF-78
MM-Net [1]]
0.845
0.796
0.794
0.706
0.628
0.597
TransSleep [16]
0.846
0.797
0.795
0.738
0.654
0.637
XSleepNet [10]
0,838
0.776
0.779
0.697
0.622
0.583
CIMSleepNet
0,849
0.799
0.797
0.830
0.772
0.775
FeatConcat
0.745
0.731
0.672
0.502
0.445
0.336
MultituskCNN [8]
0.774
0.763
0.705
0.643
0.630
0,533
SVUIL-UCD
TransSleep [16]
0.794
0.782
0.732
0.725
0.698
0.636
XSleepNet [10]
0.783
0.761
0.725
0.708
0.689
0.615
CIMSleepNet
0.801
0.794
0.751
0.788
0.777
0.726
FealConcat
0.700
0,464
0.237
0.477
0.011
MLP [24]
0.723
0.306
0.610
0.348
0.035
MHR
DeepCNN [9]
0.759
0.615
0.421
0.616
0.354
0.039
CIMSlcepNet
0.729
0.553
0.348
0.701
0.466
0.240
and SHHS; EEG, EOG and EMG, for SVUH-UCD; motion signal and HR, for MHR. We provide detailed introduction and preprocessing methods of all datasets Appendix
Implementation Details: In the first four datasets, CIMSleepNet is trained and tested using k-fold coos-validatiin, with total of five repetitions of this procedure. Each result is the average of five results. In the last dataset, the training strategy refers to [26]. The detailed experimental settings and important hyperparameter settings in Appendix F.
4.2 Comparison with the state-of-the-arts
In randomly partially missing case, we compare our CIMSleepNet with ASS methods that can support multimodal learning: FeatConcat, MultitasKCNN [8], SalientSleepNet
Table 2: Performance comparison in completely missing case.
Test Modalities
Methods
Acc
MF1
K
[23]. MM-Net [11], TransSleep [16]. XSleepNet [10], MLP [24] and Deep-
CoRe-Slecp [26]
0.882
0.808
0.834
EEG
CIMSlccpNet
0.891
0.817
0.845
CNN [9]. We leverage mask matrix Z and the public code of these methods to simulate the incomplete modality case. Then, we compare CIMSleep- Net with them under different missing rate According to the calculation
CoRe-Sleep [26]
0.853
0.753
0.792
EOG
CIMSleepNet
0.858
0.760
0.798
CoRe-Sleep [26)
0.895
0.853
EEG+EOG
CIMSleepNet
0.903
0.862
formula of p. for modalities, the missing rate ranges from [0.0, 0.1, 0.2, 0.3, 0.4, 0.5]; for three
modalities, the missing rate ranges from [0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7], where 0.7 is an approx- imate value of 2/3. In the completely missing case, compare CIMSleepNet with CoRe-Sleep [26]. the only existing ASS method that can handle complete missing of one or more modalities. We employ accuracy (Acc), macro Fl-score (MF1) and Cohen Kappa (K) [44] quantitatively analyze all methods. We also compare the data recovery performance of SMCCL with ICL [28] and SCL [30]. All methods are described in Appendix G.

---

08
0.7
0.7
FeaCbear
Accunay
Fast
Accuneyy
0.7
0.7
SchesShoppiia
0.6
Net
KM4
FedClent
0.6
TeeeShep
ToesShop
Xloppha
0.5
MSapli
DepCMN
CiMSlephie
(MMSkophie
(MSlophi
0.5
0.2
0.5
0.0
0.1
0,3
04
0.1 0.2 0,3 0.4 0.5 0.6 0.7
0.0
0.2
0.3
04
Mising
Missing
Missing
0.5-
0.5
0.6
Miero FTssore
0.7
Maxro Floanee
0.5
SelmaShepphe
0.6
0.5
TeesSoop
Xileshi
0.5
Kleppha
DeyCNN
CiSSeephi
(IMSkophie
0.0
0.2
0.3
0.5
0.0
0.1
0.3
0.4
0.0
0.2 0.3 0.4 0.5 0.6 0.7
0.0
0.2
0,3 0.4
0.5
Missing mie
Miscing cale
Missing rale
Missing sate
0.8
0.5
Kappo
0.7
Colun Colen Kagpe
0.6
Kappe
0.3
SchenSSeephi
Colon
0.6
0.5
02
0.5
0.1
XSleyhie
0.3
0.0
0.2
0.5
0.0
0.1
0.3
0.4
0.5
0.0 0.1 0.2 0.3 0.4 0.5 066 0.7
0.0
0.1
0.2
0,3
0.4
0.5
Missing mid
Missing rale
Missing rate
Missing cale
Sleq--EDF-20
Sleep-EDP-78
SVUH-UCD
MHR
Figure 4: Impact of various missing rates. The shaded area represents the range of upper and lower standard deviations.
Quantitative results: As shown in Tab 1, CIMSlcepNet achieves performance comparable to the state-of-the-arts in the complete modality. In the incomplete modalities, com- pared to the performance on complete modalities, all mod- els exhibit a decrease in performance on the four datasets. Fortumately, CIMSleepNet has the least performance degra- dation and performs the best. As schematized in Fig. 4, further evaluate the performance of CIMSleepNet and other methods under different missing rates. We observe that CIMSlcepNet outperforms other methods in almost all datasets and missing rates. As the missing rate increases, the performance of other methods begins to decline sig- nificantly. Relatively speaking, CIMSleepNet exhibits more stable trend. Further, Tab 2 exhibits the performance of CIMSleepNet trained with م 0.5 (maximizing the model's s robustness to missing modalities) and tested under complete modality absence. We observe that CIMSleepNet outperforms CoRe-Sleep terms of performance across different testing modalities.
EEG
EOG
Original
EEG
EOG
Recovered
SMCCL
Figure 5: Visualization of the recovered modalities by ICI, SCL and SMCCL.
Qualitative results: We substitute ICL. and SCL with SMCCL on CIMSleepNet to compare the
performance these three contrastive learning methods data recovery (when p 0.5). As depicted in Fig. 5, we randomly selected 500 recovered missing samples (500 EEG epochs and 500 EOG epochs) in Sleep-EDF-20 and projected them into 2D space via t-SNE [45]. ICL only focuses on the inter-modal consistency and ignores the recovery of semantic information. SCL retains semantic information based on ICL, thereby improving data matching. However, ICL and SCL tend learn the inter-modal consistency, i.e., utilize multimodal shared information to guide the recovery of missing data. This strategy easily leads to the loss of modality-specifie information, thus failing to exploit the moter-modal complementarity. Different from ICL. and SCL., our SMCCL explores the intrinsic connection between semantic and modal information under mutual information theory. Hence, compared to ICL and SCL, the data recovered by SMCCL exhibits more consistent distribution

---

with the original data, further demonstrating its effectiveness in handling missing modalities. We also visualize the features extracted by each method. Specifically, we randomly select the data of one subject (9th) among the Sleep-EDF-20. As illustrated in Fig. 6, we use t-SNE [45] to visualize the distribution of features generated by all methods at p = 0.5, which are extracted before the final decision head. Compared with other methods, our CIMSleepNet can extract more discriminative representations in incomplete modalities, further demonstrating its robustness.
FeatConnaa
MultitasKCNNN
SalientSleepNet
MM-Net
NI
N2
REM
TransSleep
XSloepNet
CIMSloepNet
Figure 6: Visualization of latent features of different methods Slcep-EDF-22
We conduct ablation studies for CIMSleepNet Sleep-EDF-25 under the con- dition of missing rate 0.5. It can be observed from Tab 3 that no matter which component is deleted, each evaluation metric of the results will decrease. It is particularly noteworthy that in the absence of both MAIM and SMCCL, the performance drops significantly, further demonstrating their importance in dealing with the missing modality issue. Furthermore, we find that although the two components designed to mitigate modality missing issue (MAIM and SMCCL) introduce additional parameters, the increase much less than that introduced by the sequence modeling component (MCTA), However, sequence modeling is crucial for capturing the temporal information of PSs and improving model performance [10, 46]. The ablation experiment of MCTA, parameter sensitivity analysis and training process analysis are detailed in Appendix H, and J, respectively.
Ablation studies:
Table 3: Ablation study CIMSleepNet on Sleep-EDF-2 "V" indicates the use this component. MCTA indicates the Transformer equipped with MCTA. The context length of single inference is 25.
MAIM
SMCCI
MCTA
Acc
MFI
Model Size (MB)
GFL.OPs
0.497
0.429
0.285
2.344
0.069
0.771
0.704
0.672
5.767
0,096
0.786
0.726
0.699
8,458
0.071
0.694
0.629
0.536
30.272
2.206
0.810
0.756
0.759
4,412
0.097
0.829
0.778
0.777
33.696
2.876
0.834
0.786
0.784
36,386
2,246
/
0.853
0.801
0.805
37.678
2.902
Conclusion
We try to challenge multimodal ASS under incomplete modalities by proposing CIMSleepNet. In CIMSleepNet, MAIM reconstructs missing modality data by establishing interactions among modali- ties, which allows for the provision of complete modality data support for subsequent components. Meanwhile, SMCCL ingeniously leverages semantic information and modal information to subdivide similarity into three levels, thereby simulating real data distribution. Then, MCTA mechanism accom- plishes comprehensive temporal context modeling, further improving the expressive ability of latent temporal representations. Extensive experiments demonstrate that the effectiveness of CIMSleepNet in various incomplete modalities.
10

---

Acknowledgments
This work was supported by the National Natural Science Foundation of China (62072089) and the Fundamental Research Funds for the Central Universities of China (N2116016, N2224001-10).
References
[1] S. Güneş, K. Polat, and S. Yosunkaya, "Efficient sleep stage recognition system based on ceg signal using k-means clustering based feature weighting," Expert Systems with Applications, vol. 37, no. 12, pp- 7922-7928, 2010.
[2] M. Perslev, Jensen, S. Darkner, P. Jennum, and C. Igel, "U-time: A fully convolutional network for time series segmentation applied to sleep staging," Advances in Neural Information Processing Systems, vol. 32, 2019.
[3] Z. Jia, Y. Lin, J. Wang, R. Zhou, X. Ning, Y. He, and Y. Zhao, "Graphsleepnet: Adaptive spatial-temporal graph convolutional networks for sleep stage classification." " in IJCAI, vol. 2021, 2020, pp. 1324-1330.
[4] J. Wang, S. Zhao, H. Jiang, S. Li, T. Li, and G. Pan, "Generalizable sleep staging via multi-level domain aligmment," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 1, 2024, pp. 265-273.
[5] S. Ma, Y. Zhang, Y. Chen, T. Xic, S. Song, and Z. Jia, "Exploring structure incentive domain adversar- ial learning for generalizable sleep stage classification," " ACM Transactions on Intelligent Systems and Technology, vol. 15, no. I, pp. 1-30, 2024.
[6] T. R. Sri, Madala, S. L. Duddukuru, R. Reddipalli, K. Polasi et al., "A systematic review on deep learning models for sleep stage classification," in 2022 6th International Conference on Trends in Electronics and Informatics (ICOET). IEEE, 2022, pp. 1505-1511.
(7] G. Muhammad, F. Alshehri, F. Karray, A. El Saddik, M. Alsulaiman, and T. H. Falk, "A comprehensive survey multimodal medical signals fusion for smart healthcare systems," Information Fusion, vol. 76, pp. 355-375, 2021.
[8] H. Phan, F. Andrcotti, N. Cooray, O. Y. Chén, and M. De Vos, "Joint classification and prediction enn framework for automatic sleep stage classification," IEEE Transactions on Biomedical Engineering, vol. 66, no. 5, pp. 1285-1296, 2018
[9] B. Zhai, Y. Guan, M. Catt, and T. Plötz, "Ubi-sleepnet:
Advanced multimodal fusion techniques for shrce-stage sleep classification using ubiquitous sensing," Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, vol. 5, no. 4, pp. 1-33, 2021.
[10] H. Phan, O. Chén, M. C. Tran, P. Koch, A. Mertins, and De Vos, "Xsleepnet: Multi-view sequential model for automatic sleep staging," IEEF Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 9, pp. 5903-5915, 2021.
[11] Y. Lin, M. Wang, F. Hu, X. Cheng, and J. Xu, "Multimodal polysomnography based automatic sleep stage classification via multiview fusion network," IEEE Transactions on Instrumentation and Measurement, 2023.
[12] D. Ramachandram and G. W. Taylor, "Deep multimodal learning: A survey recent advances and trends," IEEE signal processing magazine, vol. 34, no. 6, pp. 96-108, 2017.
[13] A. Supratak, H. Dong, C. Wu, and Y. Guo, "Deepsleepnet: A model for automatic sleep stage scoring based raw single-channel eeg," JEFF Transactions on Neural Systems and Rehabilitation Fingineering, vol. 25, no. 1, pp. 1998-2008, 2017.
[14] H. Phan, F. Andrcotti, N. Cooray, O. Chén, M. De Vos, "'eqsleepnet: end-to-end hicrarchical recurrent neural network for peuuence-to-eqqeeee automatic sleep staging," IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 27, no. 3, pp. 400-4 410, 2019.
[15] H. Sco, Back, S. Lee, D. Park, T. Kim, and K. Lee, "Intra-and inter-epoch temporal context network (iitnet) using sub-epoch features for automatic sleep scoring on raw single-channel cog." Biomedical signal processing and control, vol. 61, p. 102037, 2020.
[16] J. Phyo, W. Ko, E. Jeon, and H.-I. Suk. "Transsleep: Transitioning aware antention-based deep neural network for sleep staging," IEEE Transactions on Cybernetics, 2022.
11

---

[17] H. W. Loh, P. Ooi, J. Vicnesh, S. L. Oh, O. Faust. A. Gertych, and U. R. Acharya, "Automated detection of sleep stages using deep learning techniques: systematic review of the last decade (2010-2020)," Applied Sciences, vol. 10, no. 24, p. 8963, 2020.
[18] W. Qu, Z. Wang, H. Hong, Z. Chi, D. D. Feng, R. Grunstein, and C. Gordon, "A residual based attention model for eeg based sleep staging," IEEE journal of biomedical and health informatics, vol. 24, no. 10, pp. 2833-2843, 2020.
[19] E. Eldele, Z. Chen, C. Liu, M. Wu, C.-K. Kwoh, X. Li, and C. Guan, "An attention- based deep learning approach for sleep stage classification with single-chan Ainle-channel ceg," IEEE Transactions on Newral Systems and Rehabilitation Engineering, vol. 29, pp. 809 818, 2021.
[20] Y. Dai, X. X.L., S. Liang, I.. Wang. Q. Duan, H. Yang, C. Zhang, X. Chen, I. Li, X. Li et al., "Multichannel- sleepnet: A transformer-basse based model for automatic sleep stage classification with psg." IEEE Journal of Biomedical and Health Informatics, 2023.
[21] A. Bulatov, Y. Kuratov, and Burtsev, "Recurrent memory transformer," Advances Neural Information Processing Systems, vol. 35. pp. 11 079-11 091, 2022.
[22] Zhou, S.-I. Kamata, Wang. and X. Xue. "Multiscanning--aan based ran cun-transformer for hyperspectral image classification," IEEE Trunsactions on Geoxcience and Remote Sensing, 2023.
[23] Z. Jia, Y. Lin, J. Wang. X. Wang. P. Xie, and Y. Zhang, "Sallentsseeppnet Multimodal salient wave detection network for sleep staging," arXiv preprint arXiv:2 13864, 2021.
[24] O. Walch, Y. Huang, D. Forger, and C. Goldstein, "Sleep stage prediction with raw acceleration and photoplethysmograge smography heart rate data derived from a consumer wearable device," Sleep, vol. 42, no. 12, p. 282 2019.
[25] M. Olsen, J. M. 7eitzer, R. N. Richardson, P. Davidenko, P. J. Jennum, H. B. Sørensen, and E. Mignot, "A flexible deep learning architecture for temporal sleep stage classification using accelerometry and photoplehyysoorrahyy" JEEE Tronsactions on Biomedical Engineering, vol. 70, no. I, pp. 228-237, 2022.
[26] K. Kontras, C. Chatzichristos, H. Phan, J. Suykens, and M. De Vos, "Core-sleep: A multimodal fusion framework for time series robust to imperfect modalities." IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2024.
[27] Y. Lin, Y. Gou, Z. Liu, B. Li, Lv, and X. Peng, "Completer: Incomplete multi-view clustering via contrastive prediction," in Proceedings of the IEEE/CVF conference computer vision and puttern recognition, 2021, pp. 11 174-11 183.
[28] R. Liu, H. Zuo. Z. Lian, B. W. Schuller, and H. Li, "Contrastive learning based modality- -iveaiiannnnariantt feature acquisition for robust multimodal emotion recognition with missing modalities." IEEF Transactions on Affective Computing, 2024.
[29] Y. Lin, Y. Gou, X. Liu, J. Bai, J. Lv, and Peng. "Dual contrastive prediction for incomplete multi-view representation learning," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 4, pp. 4447 4461,4411,,1000 2022.
[30] S. Qian and C. Wang, "Com: Contrastive masked-attention model for incomplete multimodal learning." Neural Networks, vol. 162, pp. 443-455, 2023.
[31] D. Chen, S. Yongchareon, E. M.-K. Lai, J. Yo, Q. Z. Sheng, and Y. Li, "Transformer with bidirectional gru for nonintrusive, based activity recognition in multiresident environment," IEEE Internet of Things Journal, vol. 9, no. 23, pp. 23 716 23777, 2022.
[32] R. Pramanik, Sikdar, and R. Sarkar, "Transformer-basee based deep reverse attention network for multi-sensory human activity recognition." Engineering Applications of Artificial Intelligence, vol. 122, p. 106150. 2023.
[33] Q Luo, He, X. Han, Y. Wang, and H. Li, "Lsttm: long-short term transformer basedd spatiotemporal neural network for traffic flow forecasting." Knowledge-Baad Based Systems, vol. 293, p. 111637, 2024.
[34] K. T. Ahmed, S. Baul, Y. Fu, and W. Zhang, "Attention-basee multi-modal missing value imputation for time series data with high missing rate," in Proceedings of the 2023 SIAM International Conference on Data Mining (SDM). SIAM, 2023, pp. 477.
[35] D. Khattar, J. S. Goud, M. Gupta, and V. Varma, "Mvae: Multimodal variational autoencoder for fake news detection." in The world wide web conference, 2019, pp. 2915-2921.
12

---

[36] M. Cover, Elements of information theory. John Wiley & Sons, 1999.
[37] J. Huang, S. Gong, and X. Zhu, "Deep semantic clustering by partition confidence maximisation," in Proceedings of the IEEE/CVF conference computer vision and pattern recognition, 2020, pp. 8849- 8858.
[38] P. Khosla, P. Teterwak, C. Wang, A. Sarna, Y. Tian, P. Isola, A. Maschinot, C. Liu, and D. Krishnan, "Superviied contrastive learning," Advances in neural information processing systems, vol. 33, pp. 18 661- 18 673, 2020.
[39] S. Li. X. Jin, Y. Xuan, X. Zhou, W. Chen, Y.-.. Wang, and X. Yan, "Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting," Advances in neural information prucessing systems, vol. 32, 2019.
[40] A. Goldberger, A. Amaral, L. Glass, M. Hausdorff, C. Ivanov, G. Mark, J. E. Mietus, G. Moody, C.-K. Peng. and H. E. Stanley, "Physiobank, physiotoolkit, and physionet: components of new research resource for complex physiologic signals." circulation, vol. 101, no. 23, pp. e215 e220, 2000.
[41] B. Kemp, A. H. Zwinderman, B. Tuk, H. A. Kamphuisen, and J. Oberye, "Analysis of sleep-dependent neuronal feedback loop: the slow-wave microcontinuity of the ceg," IEEE Transactions on Biomedical Engineering, vol. 47, 9, pp. 1185 1194, 2000.
[42] S. F. Quan, B. V. Howard. C. Iber, J. P. Kiley. F. J. Nieto, G. T. O'Conoor, D. M. Rapoport, S. Redline. J. Robbins, J. M. Samet al., "The sleep heart health study: design, rationale, and methods," Sleep, vol. 20, no. 12, pp. 1077-1085, 1997.
[43] G.-Q. Zhang. L. Cui, R. Mueller, S. Tao, M. Kim, M. Rueschman, S. Mariani, D. Mobley, and S. Redline, "The national sleep research resource: towards sleep data commons." Journal of the American Medical Informatics Association, vol. 25, no. 10, pp. 1351-1358, 2018
[44] Cohen, "A coefficient of agreement for nominal scales," " Educational and psychotogical measurement, vol. 20, no. 1, pp. 37 46, 1960.
[45] L. Maaten vol. 9, no. 11, 2008.
Hinton, "Visualizing data using t-sne." Journal of machine learning research.
[46] Phan, K. Lorenzen, E. Heremans, Chén, C. Tran, Koch, A. Mertins, Baumert, K. Mikkelsen, and M. De Vos, "L-seqsleepnet Whole-cycle long sequence modeling for automatic sleep staging," IEFF Journal of Biomedical and Health Informatics, vol. 27, no. 10, pp. 4748-4757, 2023.
[47] B. Yang. W. Wu, Y. Liu, and H. Liu, "A novel sleep stage contextual refinement algorithm leveraging conditional random fields," IEEE Transactions on Instrumentation and Measurement, vol. 71, pp. 1-13, 2022.
13

---

A Limitations
Our work also has some limitations. To deal with the missing modality issue and temporal dependency, we introduce additional architecture, which will incur additional computational overhead. It is worth mentioning that the lack of labeling information is also a common phenomenon in real-world applications. Hence, expect develop an unsupervised semi-supervised learning multimodal approach our forthcoming study, which can simultaneously address the challenges of modality missing and label missing.
B Broader Impacts
Our work facilitates the daily sleep quality assessment and sleep disorder diagnosis for the public, and lays the foundation for promoting personalized treatment of sleep disorders. However, the multimodal physiological data required for model training may involve sensitive personal health data, which may bring potential social privacy and security issues.
C Similarity Weight Matrix Construction Example
Supposing that a batch contains multimodal epochs and the number modal types is 3, the calculation process and results of the similarity weight matrix W are illustrated in Fig. 7. Specificallyy the redefinition of matrices and is realized by using flatten and replicate operations. Moreover, calculate Y and S separately to obtain the mask matrices U and V by
U ((
R Uf
{
1, 0, T.
= o V
(fo])
of
=
{
1, 0,
(1I)
where "1" is positive pair and "0" is negative pair. Besides, if and &f are the elements in YT and ST respectively. Further, combining modality consistency matrix O the similarity weight matrix W can be constructed by
W = --OV +
(12)
the 1th level
the 2th level
the 3th level
where @ denotes element-wiss multiplication and θ in Fig. 7 is composed of the set consisting of 0,. 8g and 0g.
D Theoretical Proof
We simplify Roti d') High )*)
in continuous random variables. It can be expressed as follows
H( ,0') = H( ++ HH()) o
T(th: Φ²) dx + p(y))n p(y) dy - JS p(x.2)ln piz.y) drdy
H(0², Φ¹)
[(())ln plol
IJ p(x,))nn pa..)) dody
[f p(a, y)ln pao drdy + JJJ p(r, 1) ply) dody IS playy)) ple.))
drdy
S. p(x.g)) p(a..) dædy
(13)
[f p(r.y))n plrmnt drdy ff p(r.y))nn pं dady
JS p(x,))) (z..) drdy
plaa)) drdy [f p(z,y)in playpaa)
[S p(r,z)ln p(upp drdy
= =Iʃ log pl
p(a.z)
dxdy
p(r)p(y))
14

---

0
0
2
Label matrx
Modality mainx
Flatte Replicate
Flatten Replicate
0
0
0
0
0
0
0
0
0
0
0
2
2
0
0
0
0
0
Y
1
1
1
IS
2
2
2
2
2
2
1
0
0
2
2
2
2
2
2
2
2
2
2
2
2
Contrastive maskk matix calculation
Formala (LI)
Contrastive mask calenlation
Formilaa (T))
9
0.
0.
9
9
0
0
0
6
0
0
0
0
0
0
6
0
0
0
0
0
0
0
0
0,
0,
0,
0,
0,
0,
0,
0,
0
0
0
0
IV
0
0
1
9
0.
0
8
6
0
1
0
0
8
0
0
0
0
0
0
0
0
0,
0
0,
0
0
0
0,
0
0
0
0
0
0,
0
0
0,
0
0
0
Similarity weight matrix calculation
Formola (12)
0
0
0
1- 0
0
6
0.
0
0
1-
0
0
0
1
0g
0
W
e
1
0
0
0g
0
1-
0g
0
0
1-0,
1-6,
0
1- 0
0
0
0
0
1-0
0
Figure 7: Example of similarity weight matrix W construction.
where p(r) and p(y) are the marginal probability distributions of d and d', respectively, i.e., the continuous form of P(m) and P(n) in formula (6). p(z,y) is their joint probability distribution, i.e., the continuous form P(m, n) in formula (6). Therefore, the expression formula (6) is obtained.
E Data and Preprocessing
1) Sleep-EDF-22 [40, 41]: The dataset has been widely applied in sleep study, comprising 39 nights of PSG recordings from 20 subjects. The subjects are aged between 25 and 34 years old, with 10 males and 10 females. Each recording are divided into epochs in units of 30 seconds. The data
preprocessing method draws on the previous work [10]. After preprocessing, the context length T is set to 25, and the redundant segments at the front end are discarded.. These epochs are classified into five different categories, including wake (W), rapid eye movement (REM), and three types of non-REM (N1, N2, and N3). Then, two modalities, electroencephalogran (EEG) (Fpz-Cz channel) and electrooculogram (EOG) (ROC-LOC channel), are utilized to evaluate CIMSleepNet. Among them, the sampling frequency of EEG and EOG is both 100Hz.
15

---

2) Sleep-EDF-78 [40, 41]: The dataset includes 153 recordings from 78 subjects. The subjects have a wide age range, from 25 to 101 years old, and included 41 males and 37 females. Similarly, the length of each epoch is 30 seconds, and the method [10] is utilized to preprocess Sleep-EDF-78 data. We set the context length T 25 for each modality. The choice of modalities, categories and sampling frequency are the same as for Sleep-EDF-20.
3) SVUH-UCD [40]: The dataset focuses on sleep staging study with sleep disorders. It includes 25 PSG recordings from 25 sleep apnea patients. Their ages ranged from 28 to 68, including 21 males and 4 females. Following previous study [47]. we choose EEG (C3-A2 channel, 128 Hz), EOG (horizontal channel, 64 Hz) and EMG (64 Hz), and resample these recordings to 100Hz. Furthermore, we also set the context length 7 to 25. This dataset are also divided into five sleep stages.
4) MHR [24]: This is a public sleep dataset based on wearable devices, which contains overnight sleep recordings from 31 subjects. Subjects are allowed 8 hours of sleep monitoring opportunities and each epoch is 30 seconds in length. We preprocess this dataset using the method described in previous work [24]. After the preprocessing is completed, we set the context length T to 20 and exclude redundant epochs. Following the usage rules of this dataset, we employ two modalities, the motion signals composed of three-axis accelerometry and the heart rate signals, to perform classification tasks for the three categories: W, non-REM, and REM. In these two modalities, the sampling frequency of motion signal and heart rate signal is 50Hz and IHz respectively.
5) SHHS [42, 43]: This dataset is a large sleep dataset collected from multiple sleep centers, which contains two sub-datasets, namely SHHS-1 and SHHS-2. Following previous studies [10, 26), we choose SHHS-1 for our experiment. SHHS-1 consists of 5,791 subjects aged hetween 39 and 90 years old. We employ the EEG (C4-A1 channel, 125Hz) and EOG (L-R channel, 50Hz), and resample them to 100 Hz. Furthermore, the context length T is set to 25, We also divide it into five sleep stages.
F Implementation Details
We choose the programming language based Python 3.8 and the deep learning framework based on PyTorch 1.13 to build and train the model. All experiments are conducted on a server containing an RTX 4090 GPU (24GB) and Intel(R) Xeon(R) Gold 6430 processor (120GB) equipped with 16 virtual CPUs. The total objective loss is mainly optimized through the Adam optimizer. In the first four dataset, CMISleepNet is trained (Held-out validation set 4 subjects for Sleep-EDF-20, SVUH- UCD and MHR; 7 subjects for (tcp--EDF778) and tested by k-fold cross-validation Specifically, CMISleepNet performs five random samplings and five k-fold volssvvaisssionss for each missing rate in every dataset. After each k-fold vross-validation, the prediction results from the test sets of all folds are combined as one time result. Each missing rate result is the average of five results. In the last dataset, the training strategy refers to [26], i.e., using a random split of 0.7 and 0.3 for the train (Held-out 100 subjects for validation) and test set.
The important hyperparameters on different datasets can be described as: We set the learning rate of all datasets 0.001 and 0.0001 before and after the 10th iteration, respectively. The weight decay is set to 0.0001 for all datasets. The maximum number of iterations is set to 100 for all datasets. The number of intra-epoch heads S₁ = 4 and the number of inter-epoch heads S2 8 for all datasets. The number of cross-validation folds, k, is set to 20 for Sleep-EDF-20. 20 for Sleep-EDF-78; 25 for SVUH-UCD and 15 for MHR. The coefficient set W utilized to adjust category weights are set to [1.5, 2.5, 1.5,1.0, 1.5] for Sleep-EDF-20; [1.5, 2.2, 1.5,1.0, 1.5] for Sleep-EDF-78. [1.5, 2.0, 1 1.5,1.0, 1.5) for SVUH-UCD; 2.0, 1.0, 2.0] for MHR; [2.0, 3.0, 1.5, 1.5] for SHHS. Further, we set the coefficients a and B8 of the total objective function to 0.001 and 0.01, respectively for all datasets.
G Compared Methods
In randomly partially missing case, we compare our CIMSleepNet with 8 ASS methods that can support multimodal learning: FeatConcat, MultitasKCNN [8], SalientSleepNet [23], MM-Net [11], TransSleep [16), XSleepNet [10], MLP [24] and DeepCNN [9]. In the completely missing case, we compare CIMSleepNet with CoRe-Sleet [26], the only existing ASS method that can handle complete missing of one or more modalities. We also compare the data recovery performance of
SMCCL with ICL [28] and SCL [30]. All methods are described as follows.
16

---

1) FeatConcat: The temporal CNNs are utilized as the feature extractor of each modality data, and the features of different modalities are directly fused. Moreover, the fused features are sent MLP for classification.
2) MultitaskCNN [8]: Based the sleep staging task, the task of predicting adjacent epochs is added to improve the multimodal sleep staging performance.
3) SalientSleepNet [23]: A dual-branch U2-Net structure is proposed to improve the feature extraction of salient waves of multimodal physiological signals.
4) MM-Net [[]]: A cross-link fusion module is exploited to reduce redundant information of multi-modality and multi-view.
5) TransSleep [16]: Two auxiliary tasks, segment confusion stage estimation and stage-transition detection, designed to address stage transitions during sleep. We make it capable of handling multimodal data by increasing the number of channels at the input head of TransSleep.
6) XSleepNet [10]: An adaptive gradient blending strategy is designed to improve the joint represen- tation ability of the original signal and the corresponding time-frequency image.
7) MLP [24]: Combining motion features and heart rate features, and using MLP to implement sleep staging (Wake/ NREM/ REM) based on wearable devices.
8) DeepCNN [9]: A deep CNN is constructed to explore the impact of early-stage fusion, late-stage fusion and hybrid fusion. We chose the late-stage fusion solution because it has the best performance.
9) CoRe-Sleep [26]: The ingenious combination of self-attention and cross-attention improves the robustness of the model under imperfect data.
10) ICL [28]: Improving modal consistency by bringing different modalities of the same instance closer together.
SCL [30]: The introduction of semantic information improves the ability to recover the semantic structure information of data.
H Ablation studies of MCTA
We also explore the internal details of Transformer equipped with MCTA. The six baseline models in Tab are substructures of Transformer equipped with MCTA. Among them, Intra-X denotes using two X layers for intra-epoch temporal dependency modeling. Inter-X denotes using two X layers for inter-epoch context modeling. Intra Inter-X denotes using four layers for temporal context modeling, with the first two layers capturing intra-epoch temporal dependency and the last two layers capturing inter-epoch context. refers to GRU or Transformer. can be observed from the results that both Intra & Inter-GRU and Intra & Inter-Treet Transformer outperform their respective single-level models. Further, CIMSleepNet performs the best when using Transformer equipped with MCTA, which proves the effectiveness of multi-level cross-branch representation fusion.
Table 4: Ablation study of Transformer equipped with MCTA Sleep-EDF-20.
Methods
Acc
MF1
Intra-GRU
0.827
0.775
0.772
Inter-GRU
0.835
0.780
0.787
Intra & Inter-GRU
0.839
0.788
0.791
Fransforrns
0.813
0.770
0.765
Inter- Transfrranfr
0.837
0.789
0.793
Intra & Inter- Transformer
0.845
0.795
0.797
Transformer with MCTA
0.853
0.801
0.805
17

---

I Parameter Sensitivity Analysis
We explore the impact of and 8 on the performance of CIMSlcepNet on the Sleep-EDF-20. As shown in Fig. 8, as the values of and / change, the performance of CIMSleepNet fluctuates to varying degrees. We also observe that | has greater sensitivity to CIMSleepNet compared a.
08
0.8
Accuma
0,7
0,7
0.5
0.
0,0001
0000
a1
0.01
0.00|
h0001
0.1
0.01
0.01
0
0.01
0.00
0.0001
0.1
0.01
0,001
0.1
0.01
0.00
0.1
0.01
a.01
a
Figure Hyperparameters, and 8, analysis Sleep-EDF-20.
Training Process Analysis
As schematized in Fig. 9, we provide visualizations of different validation loss curves to explore their real-time changes during training. We can observed: During the entire training process, modal imagination loss and distribution alignment loss will decrease as the number of iterations increases, which shows that the data imputation ability and distribution fitting ability the model are gradually improving. It is worth mentioning that, in the early stages of training, the rate of decrease in distribution alignment loss is greater than that of modal imagination loss. This phenomenon occurs because the data generated during the initial training stage deviates significantly from the real distribution, requiring substantial adjustments through distribution alignment loss function. When the modal imagination loss and distribution alignment loss are in stationary state, the classification loss continues to decrease, which indicates that the data generated in the stationary state will further improve the classification performance of the model.
Modal imputation loss (EEG)
6
Modal imputation loss (EOG)
Distribution alignment loss
Classification loss
Loss
4-
2
20
40
60
80
100
Iteration
Figure 9: Training dynamics of modal imagination loss, distribution alignment loss and classification loss on Sleep-EDF-20. Among them, modal imagination loss is presented in two modalities: EEG and EOG respectively.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims contained in the abstract and introduction accurately reflect the contribution and scope of our study,
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer:[Yes)
Justification:
We create a separate section in Appendix A to discuss the limitations of this study.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiiatinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
19

---

Answer: [Yes]
Justification: The manuscript contains proofs of theoretical results, and all theorems, formu- las and proofs are supported by corresponding theories and references. Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: This manuscript fully presents all the information necessary to reproduce the main experimental results of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
20

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
The datasets used in this study are public datasets, and we provide their citations. In addition, we open the corresponding codes.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm co/
momiiissioooicyyy. for more details. Sttplc/ggiidde/oeeeeeeeeeee
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips. //nips.cc/pulie wttpccccccpppppp..ccccc /egguiddee/ooeemmiiii CooeeSubmi ottce/eeuuniiiiiiooooooooo.o
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All experimental details are described in detail in Appendix F.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Our experimental results are the average of training under multiple random seeds. The error margins are included in the results for various missing modalities.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
21

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide detailed information about the computational resources in the Appendix F. We also provide experiment computational efficiency in Tab 3.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https : http://wwrrss.cccceeeee cc/public// Beticc///hiitttteeeee csQuidelines!
Answer: [Yes]
Justification: Our manuscript complies with the NeurlPS ethical guidelines both in terms of research area as well as methods and datasets.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We create separate section in Appendix B discuss the broader impacts of this study.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
22

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our study poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The creators original owners of all assets used in this research duly ac- knowledged, and the licenses and terms of use are clearly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
23

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode, cww/cooo/aaaa /atasetss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
Justification: The new assets introduced in the paper are well documented, and documenta- tion is provided along with the assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/codeemmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: We use public datasets and do not involve crowdsourced experiments and research on humans.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Our study does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
24

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "ag97PmwtV6", "reviewer": "Reviewer_PgT4", "summary": "The paper introduces a framework designed for sleep staging that addresses challenges associated with missing modalities in multimodal physiological signal datasets. The proposed model incorporates a modal awareness module and a semantic & modal calibration contrastive learning mechanism to handle missing data and ensure semantic consistency across modalities.", "questions": "The main issue with the paper is the idea of trying to prove the concept of missing modality is very important. The idea could simply be circumvented by training a model for the modalities available and when all modalities are available, this approach is too complex compared to other approaches that are simpler. There are better sleep staging approaches which have not been compared against.", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- the integration of MAIM and SMCCL to address the missing modality challenge could be useful for real-world scenarios where data incompleteness is common.\n- the utilization of a multi-level cross-branch temporal attention mechanism enables the model to capture both intra-epoch and inter-epoch temporal dependencies effectively.\n- the model is thoroughly evaluated on five sleep datasets.\n- providing the code in supplementary materials fosters transparency and allows for community-driven improvements and testing in diverse scenarios.", "weaknesses": "- the paper introduces a potentially high computational overhead due to the complexity of the proposed model, especially with the integration of components like MAIM, SMCCL, and the multi-level attention mechanism.\n- while the model performs well on controlled datasets, the real-world effectiveness and adaptability of the model in handling various types and degrees of missing data in uncontrolled environments are not extensively discussed.\n- the paper does not extensively compare the proposed model with other state-of-the-art approaches that use less computationally intensive methods to handle missing modalities, which could provide a better balance between performance and efficiency.\n- the method's reliance on existing multimodal data and potentially biased toward the modalities and specific characteristics of the datasets used for training could limit its effectiveness across broader populations or different types of sleep-related conditions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weaknesses**:\n\nR1: Please see **Author Rebuttal** for details.\n\nR2: In response to this issue, we would like to clarify and add the following: Firstly, we conduct a comprehensive validation of the CIMSleepNet's effectiveness under random partial missing and complete missing conditions across five different multimodal sleep datasets. Meanwhile, we also explore the impact of different missing rates on the CIMSleepNet and set five different missing situations at each missing rate. These preset missing conditions cover a variety of possible practical application scenarios, thereby effectively simulating data missing conditions in natural environments. Furthermore, these experimental strategies are consistent with previous work on incomplete multimodal learning [1][2][3][4], which is the only way to evaluate the performance difference of the model under the same dataset with complete and incomplete modalities.\n\nR3: Thank you for your valuable comment. In the field of sleep staging, there is currently only one work on incomplete multimodal learning [1]. We have compared with this state-of-the-art method and indicated that our CIMSleepNet performs better in handling missing modalities. Although there are works [3][4] in other fields to deal with missing modalities, most of them require the modality to be complete in the training set and are not applicable to the field of sleep staging. In our study, the multimodal data in the training set is also incomplete, which makes our study closer to practical applications. Moreover, we also compare two non-sleep staging methods related to our SMCCL to deal with missing modalities, i.e., invariant contrastive learning [5] and semantic contrastive learning [2]. The results further exhibit the effectiveness of our core components. These comparative experiments demonstrate the advantages of our method in dealing with missing modal data. As for the issue of computational efficiency, as described in the Author Rebuttal, the model size and computational time of our method are within an acceptable range.\n\nR4: We understand your concerns regarding the model's generalization ability for sleep staging across different modalities. However, current multimodal sleep staging methods mainly rely on EEG, EOG, and EMG signals, which are also the data used by most previous sleep staging studies and our study. Moreover, we apply our method to heart rate and motion signals collected from real wearable devices and verify the generalization ability of the model across different modalities. In fact, the multimodal dataset used in this study already covers the main physiological signals commonly used in current sleep staging study [6]. These signals have been widely verified and recognized, which can be effectively used for sleep staging.\n\n**Questions**:\n\nR1: Thank you for your valuable comment. We actually deal with the issue of incomplete modalities in both the training and testing sets, which cannot be solved simply by training a model on the available modalities. On the contrary, our method is more suitable for real application scenarios. Because in practical applications, the signals collected by devices, especially portable wearable devices, often encounter incomplete modal data. Hence, each component we designed is necessary in this context. Furthermore, we demonstrate in Author Rebuttal that the complexity of our model is within an acceptable range. Currently, in the field of sleep staging, there is only one work that involves the problem of multimodal missingness, which we have compared in our paper. In addition, we also compare some advanced multimodal sleep staging methods with open source code and test the performance of these methods in the case of incomplete multimodality. Some advanced sleep staging methods are not compared because some of them only support single-modality sleep staging. There are several new multimodal sleep staging methods [7][8], but since their codes are not open source, we cannot reproduce these methods to test their performance in the absence of modality. Subsequently, we will present the results of these methods as reported in their original papers. Their performance is similar to that of our method on the complete multimodaliies.\n\n\n\n[1] Kontras K, Chatzichristos C, Phan H, et. al. CoRe-Sleep: A Multimodal Fusion Framework for Time Series Robust to Imperfect Modalities. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2024, 32: 840-849.\n\n[2] Qian S, Wang C. COM: Contrastive Masked-attention model for incomplete multimodal learning. Neural Networks. 2023, 162: 443-455.\n\n[3] Wang Y, Li Y, Cui Z. Incomplete multimodality-diffused emotion recognition. In Proceedings of the 37th International Conference on Neural Information Processing Systems. 2023: 17117-17128.\n\n[4] Lian Z, Chen L, Sun L, Liu B, Tao J. GCNet: Graph completion network for incomplete multimodal learning in conversation. IEEE Transactions on pattern analysis and machine intelligence. 2023, 45(7): 8419-8432.\n\n[5] Liu R, Zuo H, Lian Z, et. al. Contrastive Learning based Modality-Invariant Feature Acquisition for Robust Multimodal Emotion Recognition with Missing Modalities. IEEE Transactions on Affective Computing. 2024: 1-18.\n\n[6] Faust O, Razaghi H, Barika R, et. al. A review of automated sleep stage scoring based on physiological signals for the new millennia. Computer methods and programs in biomedicine. 2019, 176: 81-91.\n\n[7] Zhu H, Zhou W, Fu C, et. al. Masksleepnet: A cross-modality adaptation neural network for heterogeneous signals processing in sleep staging. IEEE Journal of Biomedical and Health Informatics. 2023, 27(5): 2353-2364.\n\n[8] Li T, Gong Y, Lv Y, et. al. GAC-SleepNet: A dual-structured sleep staging method based on graph structure and Euclidean structure. Computers in Biology and Medicine. 2023, 165: 107477.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AdqyMAbVWE", "reviewer": "Reviewer_RxY3", "summary": "This paper proposes a robust multimodal sleep staging framework named Contrastive Imagination Modality Sleep Network (CIMSleepNet) to address the issues of missing modalities and temporal context modeling in automated sleep staging (ASS). CIMSleepNet combines a Modal Awareness Imagination Module (MAIM) for imputing missing data and a Semantic & Modal Calibration Contrastive Learning (SMCCL) approach to align the recovered data with real data distributions. Additionally, a multi-level cross-branch temporal attention mechanism is embedded to enhance the extraction of cross-scale temporal context representations. Extensive experiments on five multimodal sleep datasets demonstrate that CIMSleepNet significantly outperforms existing methods under various missing modality scenarios.", "questions": "The introduction of the Modal Awareness Imagination Module (MAIM) and the Semantic & Modal Calibration Contrastive Learning (SMCCL) adds considerable complexity to the CIMSleepNet framework. Could the authors provide a detailed analysis of the computational efficiency and resource requirements of CIMSleepNet compared to other state-of-the-art methods? Specifically, how does the increased computational overhead impact the scalability and real-time applicability of the framework in practical sleep monitoring systems?", "limitations": "see weakness", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. CIMSleepNet’s implementation of the Modal Awareness Imagination Module (MAIM) and Semantic & Modal Calibration Contrastive Learning (SMCCL) effectively addresses the issue of missing modalities. This robust recovery process aligns the recovered data distribution closely with the real data, significantly enhancing the performance and reliability of multimodal sleep staging under various incomplete modality conditions.\n2. The framework’s multi-level cross-branch temporal attention mechanism (MCTA) excels in mining cross-scale temporal context representations at both intra-epoch and inter-epoch levels. This feature ensures a thorough and precise extraction of temporal features, which is crucial for accurate sleep staging.", "weaknesses": "Despite its robustness to missing modalities during inference, CIMSleepNet still requires a substantial amount of complete multimodal data for effective training. This dependence on complete data might pose challenges in real-world scenarios where obtaining comprehensive multimodal datasets is often difficult.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weaknesses**:\n\nR1: Thank you for your valuable comment. In fact, our CIMSleepNet supports incomplete multimodal data in both the training and testing phases. Our proposed MAIM can recover the missing modality data from other available modality data of the same sample in incomplete multimodality data. Then, the semantic and modality information of the recovered data are corrected by the proposed SMCCL to generate data that matches the true distribution. As reported in Table 1 from the original manuscript, we take the Sleep-EDF20 dataset (containing two modalities) as an example. When the missing rate reaches 0.5, all multimodal samples in the training set are incomplete, but CIMSleepNet still achieves a high degree of robustness.\n\n**Questions**:\n\nR1: Thank you for your suggestion! We further analyze the computational efficiency and resource requirements of CIMSleepNet. Please refer to the Author Rebuttal for details.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uXCin7kcd1", "reviewer": "Reviewer_QG76", "summary": "This paper presents CIMSleepNet, a framework designed to address the challenges of automated sleep staging when faced with incomplete multimodal physiological data. This framework is particularly relevant in real-world applications where sensor malfunctions or detachment often results in incomplete data, thereby affecting the performance of sleep staging systems.", "questions": "1. Scalability Issue: It would be interesting to see how does CIMSleepNet scale with even larger datasets or more varied modal distributions? Is its performance consistent across other non-sleep related tasks?\n2. Potential Extension: Can the method be extended to handle missing labels in addition to missing modalities, addressing semi-supervised learning scenarios?\n3. Time Cost Issue: What is the impact on inference time when using the proposed framework?", "limitations": "1. Computational Cost: Increased computational overhead from additional architecture components.\n2. Modality and Data Limitation: Tested on a limited set of modalities and datasets.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Thanks for your work! Here are my comments:\n1. CIMSleepNet effectively addresses the common issue of missing modalities in sleep stage classification through its Modal Awareness Imagination Module (MAIM) and the proposed Semantic & Modal Calibration Contrastive Learning (SMCCL). This innovative combination helps in approximating the missing modal data and aligning it closer to real data distributions better while comparing to other methods.\n2. The framework is validated across five different multimodal sleep datasets, showing its robustness and effectiveness in improving sleep staging performance.\n3. The multi-level cross-branch temporal attention mechanism allows the framework to capture comprehensive temporal context information, which can enhance the model's ability to interpret complex stage-transitioning patterns using PSs.", "weaknesses": "1. The architecture integrates multiple sophisticated components, which could potentially increase the computational overhead, making it less efficient for real-time applications, also, the training cost of time for this model is not given out.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weaknesses:**\n\nR1: Thank you for your valuable suggestions! We have further analyze the computational efficiency and resource requirements of CIMSleepNet. Please refer to the Author Rebuttal for details.\n\n**Questions:**\n\nR1: As schematized in Table 2 of the original manuscript, we have compared the currently only state-of-the-art incomplete multimodal sleep staging method on the largest publicly available multimodal sleep staging dataset, i.e., the SHHS dataset. CIMSleepNet exhibits significant advantages in both complete and incomplete multimodal  cases. Moreover, as reported in Table 3 from the PDF, we further demonstrate the superiority of CIMSleepNet over other state-of-the-art multimodal sleep staging methods. These advantages mainly stem from the excellent data recovery ability of MAIM combined with SMCCL and the powerful sequence modeling ability of MCTA. Current multimodal sleep staging methods mainly rely on EEG, EOG, and EMG signals, which are also the data used by most previous sleep staging studies and our study. Bseides, we apply our method to heart rate and motion signals collected from real wearable devices and verify the generalization ability of the model across different modalities. In fact, the multimodal dataset used in this study already covers the main physiological signals commonly used in current sleep staging study [1]. These signals have been widely verified and recognized and can be effectively used for sleep staging.\n\nAs reported in Table 4 of the PDF, we conduct experiments on CIMSLeepNet using two non-sleep-related datasets, UCI-HAR [2] and WESAD [3]. The introduction and preprocessing of the two datasets are as follows:\n\nUCI-HAR: This is a public database applied to human activity recognition, which contains the daily activity recordings of 30 subjects. Subjects are instructed to wear a smartphone with embedded inertial sensors to perform signal acquisition for six physical activities. We set the context length $T$ to 20 and removed extra segments. The database includes three types of motion signals: three-axis total acceleration, three-axis body acceleration, and three-axis angular velocity. The sampling frequency of each motion signal is 50Hz, and the length of each segment is 2.56 s. We employ these three motion signals to conduct experiments on CIMSLeepNet.\n\nWESAD: The database, which focuses on study in the field of stress detection, includes physiological signals from 15 subjects. Many studies [4][5][6] have successively confirmed that  electrodermal activity (EDA)  andelectrocardiogram (ECG) have significant advantages in stress detection and emotion recognition. Therefore, we adopt these two modalities to achieve the validity verification of CIMSLeepNet. Refer to previous methods [5][6] to preprocess EDA and ECG. In the process, we also resampled the data from both modalities to 300Hz. In addition, the method of study [5] is adopted to divide the segments, which makes the length of each segment 10 s. After that, the context length $T$ to 20. Ultimately, CIMSLeepNet is utilized to perform discrimination between normal emotional and stressful states.\n\nDuring training, the number of cross-validation folds, $k$, is set to 10 for UCI-HAR and 15 for WESAD. The coefficient set $\\smash{{{{\\mathbf{\\vec{W}}}}}}$ utilized to adjust category weights are set to [1.0, 1.0, 1.0, 1.0, 1.0, 1.0] for UCI-HAR and [1.0, 1.5] for WESAD.\n\nAs reported in Table 4 from the PDF, our CIMSleepNet also performs well on two datasets in non-sleep related fields. Especially under incomplete modalities, CIMSleepNet still maintains good robustness. This further proves the effectiveness of CIMSleepNet in other tasks and multimodal signals.\n\nR2: We believe this is a very meaningful research direction. Since our method also encounters modality missing in the training set, it requires supervised information to guide both the modality recovery and sleep staging processes. Although our current work mainly focuses on handling missing modalities, in the future we will explore how to utilize partially labeled data and incomplete modalities to extend CIMSleepNet to semi-supervised learning scenarios.\n\nR3: Thank you for your valuable suggestions! We further analyze the computational efficiency, resource requirements, and inference time of CIMSleepNet. Please refer to the Author Rebuttal for details.\n\n**Limitations:**\n\nR1: Please refer to the **Author Rebuttal** for details.\n\nR2: Please refer to R1 in **Questions**.\n\n[1] Faust O, Razaghi H, Barika R, et. al. A review of automated sleep stage scoring based on physiological signals for the new millennia. Computer methods and programs in biomedicine. 2019, 176: 81-91.\n\n[2] Ignatov A. Real-time human activity recognition from accelerometer data using convolutional neural networks. Applied Soft Computing. 2018, 62: 915-922.\n\n[3] Schmidt P, Reiss A, Duerichen R, et. al. Introducing wesad, a multimodal dataset for wearable stress and affect detection. In Proceedings of the 20th ACM international conference on multimodal interaction. 2018: 400-408.\n\n[4] Xu Y, Liu GY. A method of emotion recognition based on ECG signal. In 2009 international conference on computational intelligence and natural computing. 2009, 1: 202-205.\n\n[5] Sarkar P, Etemad A. Self-supervised ECG representation learning for emotion recognition. IEEE Transactions on Affective Computing. 2020, 13(3): 1541-1554.\n\n[6] Zhu L, Spachos P, Ng PC, et. al. Stress detection through wrist-based electrodermal activity monitoring and machine learning. IEEE Journal of Biomedical and Health Informatics. 2023, 27(5): 2155-2165.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lts4y3LCny", "reviewer": "Reviewer_HATA", "summary": "This manuscript proposes an end-to-end framework that incorporates many procedures into an automatic sleep staging system that is specifically designed in the context of missing data.  This includes a module for missing modality imputation, a module for distribution alignment, and a module for feature extraction and temporal context modeling.  The net result of these modifications is that the approach appears to work much better than competing algorithms in the context of missingness.", "questions": "As above, how does a more standard two-step procedure (infer missingness, then apply algorithm) perform compared to these current methods?\n\nWhy do ICL and SCL not infer the same distribution on the recovered samples (Figure 5), as the recovered samples should largely match the data distribution?\n\nWhy are the uncertainties in Figure 4 so small?  For example, in Sleep-EDF-20, there are only 20 subjects and I would have expected a larger standard deviation.", "limitations": "See first question above.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This manuscript is fairly clear on its overall concepts.  The scientific motivation is well-justified, as the missing data is a problem in practice and has received significantly less attention in the field.  There are several novel contributions in the different stages of modeling.", "weaknesses": "There are several minor weaknesses.\n\nFirst, it is unclear how the different comparison methods are used in the presence of missing data.  It sounds like the missing data is just removed from the data and sent through the comparison methods without modification.  However, that would be a departure from typical practice and represent an unrealistic scenario.  Note that EEG is routinely screened and bad channels are replaced by statistical inference (such as in HAPPE, MNE, FIELDTRIP, etc.).  As such, the fair comparison would be to a 2 stage procedure where the data is first imputed and then put through the framework.   In my experience, you see very little impact on performance for tasks like this up to a reasonable amount of missingness, certainly much less than the impact reported in Figure 4.\n\nWhile the proposed techniques are mathematically nuanced, it is not always clear why they work.  For example, in the qualitative results, SMCCL is matching better than ICL or SCL, but it is not clear why that is happening to me.  Given that the data is recovered, I would have expected the distributions to match better.  I would appreciate a bigger commentary here.  I have the same issue with MTCA in the appendix, where there is a comparison table but not a clear description of why the performance differences are happening.\n\nI found the discussion of existing work on multi-modality with missing data to be a bit sparse.  This is important context for this work, as there is a fair amount of work on missingness in multi-modality previously in different scientific domains and it is not compared to here.  At a minimum, I would appreciate an increased discussion of these methods and a discussion of why they were not appropriate as comparison methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weaknesses:**\n\nR1: Thank you for your valuable comments. Similar to mainstream sleep staging methods [1][2], the EEG, EOG, and EMG signals we use are all single-channel. Taking EEG as an example, multi-channel EEG requires expensive equipment such as PSG with strict environmental requirements for collection. In contrast, single-channel EEG is easier to collect with portable devices and reduces non-invasiveness to subjects. Single-channel signals are not limited to a specific collection environment and are more conducive to the promotion of sleep stages. This means that in actual applications, we will not collect multi-channel data. Hence, the method of using other channels to perform statistical inference on the missing channel is not applicable to our study. In addition, our experimental scheme for the missing modality is consistent with previous work [2], which are all single-step.\n\nHowever, interpolation methods in the temporal dimension are applicable. As depicted in Figure 1 from the PDF, we explore the performance of five models at various missing rates. Among them, BaseSleepNet is the baseline model after CIMSleepNet removes MAIM and SMCCL. We perform nearest neighbor interpolation (NNI), 3rd order polynomial interpolation (3PI) and cubic spline interpolation (CSI) on the subjects' nightly recordings. Then, the three interpolated data are fed into BaseSleepNet. We observe that the three interpolation methods only slightly improve the performance of BaseSleepNet.  Especially as the missing rate increases, they can even negatively impact BaseSleepNet's performance. Temporal imputation of missing values can lead to semantic information confusion, thus interfering with sleep staging. CIMSleepNet achieves an integration of 'missing imputation-sleep staging'. The end-to-end method reduces the complexity and potential errors associated with multi-step processing, avoiding inconsistencies between different steps.\n\nR2: Thank you for your valuable suggestions. Since the multimodal data in the training set is also incomplete, MAIM learns from the available modalities in the incomplete multimodal data and utilizes other available data within the same modality for guidance to recover the missing data. This is an appropriate strategy, but it can lead to modality and semantic biases [3]. As shown in Figure 5 from the manuscript, ICL only focuses on the inter-modal consistency and ignores the recovery of semantic information. SCL retains semantic information based on ICL, thereby improving data matching. However, ICL and SCL tend to learn the inter-modal consistency, i.e., utilize multimodal shared information to guide the recovery of missing data. This strategy easily leads to the loss of modality-specific information, thus failing to exploit the inter-modal complementarity. Different from ICL and SCL, our SMCCL explores the intrinsic connection between semantic and modal information under mutual information theory. The designed three-level similarity adaptively introduces semantic and modal information into the restored data, so that it not only retains the semantic structure but also restores the specific modal information to a large extent. Hence, as indicated in Figure 5, the data restored by SMCCL is more inclined to the real data distribution.\n\nThere are two reasons why MCTA achieves optimal performance. Firstly, MCTA not only realizes the mining of intra-epoch (local) temporal information, but also realizes inter-epoch (global) contextual learning. Compared with single-level methods, MCTA shows a more comprehensive sequence modeling ability. Secondly, MCTA realizes the interaction between self-attention weights and recurrent bias, which not only enables efficient context learning, but also improves the shortcomings of the Transformer's insufficient recurrent modeling ability.\n\nR3: We discuss five methods related to our study in the related work section. One is the only existing incomplete multimodal sleep staging method. The other four methods use ICL or SCL to solve the missing modality, which is related to our SMCCL. Although we cannot directly apply these four methods to sleep staging, ICL and SCL can be embedded in our method. This is why we start with contrastive learning to introduce the work related to missing modality. We will further add other related work on missing modality to clarify why they cannot be directly applied to sleep staging. Moreover, they require complete training sets, which contradicts the practical applications of sleep staging.\n\n**Questions:**\n\nR1: Please refer to R1 in **Weaknesses**.\n\nR2: Please refer to R2 in **Weaknesses**.\n\nR3: We use $k$-fold cross-validation for each training, which allows us to fully utilize the entire dataset. Moreover, it reduces bias that may arise from a small data size and single dataset partitioning. Take Sleep-EDF-20 as an instance, it contains 39 nights of recordings from 20 subjects. Each sample is 30 s, so the sample size is relatively sufficient to ensure the stable performance in each training. For sleep-EDF-20, all subjects are healthy and within a small age range (25 and 34 years old). Hence, on Sleep-EDF-20, the small individual differences are also an important reason for the smallest uncertainty and the best performance. Thanks to the support of MAIM and SMCCL, the high robustness of CIMSleepNet is also one of the reasons for its small performance uncertainty.\n\n\n\n\n\n\n\n\n[1] Phan H, Chén OY, Tran MC, et. al. XSleepNet: Multi-view sequential model for automatic sleep staging. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2021, 4(9): 5903-5915.\n\n[2] Kontras K, Chatzichristos C, Phan H, et. al. CoRe-Sleep: A Multimodal Fusion Framework for Time Series Robust to Imperfect Modalities. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2024, 32: 840-849.\n\n[3] Qian S, Wang C. COM: Contrastive Masked-attention model for incomplete multimodal learning. Neural Networks. 2023, 162: 443-455.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thank you for your decision and constructive comments on our manuscript. We have tried our best to modify some of the content and uploaded a PDF consisting of figures and tables. In the subsequent rebuttal per review, we will refer to it as "PDF". The blue font part in the PDF is the newly added content. Furthermore, we have also made detailed clarifications for some of the questions and misunderstandings that may exist in our manuscript in the comments. We have analyzed the computational efficiency issues that most reviewers are concerned about in this section. For other responses, please see the rebuttal per review for detailed responses.

**Analysis of the Computational Efficiency and Resource Requirements**

As indicated in Table 1 from the PDF, we calculate the parameters, FLOPs (single inference), and average training time per iteration of CIMSleepNet and other state-of-the-art methods on the Sleep-EDF-20 dataset. Moreover, we also deploy the weights of each model on the Raspberry Pi 4B platform to test its inference time, thereby simulating its running efficiency on the mobile device. From Table 1, we observe that the sequence modeling models (i.e., TransSleep, XSleepNet, and our CIMSleepNet) are still within acceptable ranges despite having higher model size and computation time than non-sequence modeling models. Note that, sequence modeling refers to learning temporal context information using recurrent neural networks and Transformers, etc. Because all models can be successfully deployed on the Raspberry Pi 4B platform, and the average inference time of these methods for a single data epoch (duration of 30 s) is between 0.3 s and 1.1 s, which is much lower than 30 s. This means that all models exhibit good real-time performance, and after processing the current data epoch, there is enough time to process the data of the next epoch.

As shown in Table 2 from the PDF, we further analyze the memory consumption and computational complexity of each component in CIMSleepNet. We find that although the two components designed to mitigate modality missing issue (MAIM and SMCCL) introduce additional parameters and FLOPs, the increase is much less than that introduced by the sequence modeling component (MCTA). This phenomenon is similar to what is observed in TransSleep and XSleepNet, where the use of sequence modeling methods also leads to a significant increase in the parameters. However, compared to non-sequence modeling methods, sequence modeling methods can demonstrate more powerful potential on large-scale sleep datasets [1][2]. As reported in Table 3 from the PDF, we train and test all the methods in Table 1 on a large-scale dataset, i.e., the SHHS dataset. The results show that the performance of non-sequence modeling methods on large-scale datasets is far inferior to that of sequence modeling methods. Although SalientSleepNet and MM-Net improve performance on small-scale datasets by enhancing the structure of convolutional neural networks (CNNs), these CNN improvements are far less effective than sequence modeling methods on large-scale datasets. Hence, we abandon the improvement of CNN structure and focus on optimizing sequence modeling methods to achieve excellent performance on both small-scale and large-scale datasets.

[1] Phan H, Lorenzen KP, Heremans E, et. al. L-SeqSleepNet: Whole-cycle long sequence modelling for automatic sleep staging. IEEE Journal of Biomedical and Health Informatics. 2023, 27(10): 4748-4757.

[2] Phan H, Mikkelsen K, Chén OY, et. al. Sleeptransformer: Automatic sleep staging with interpretability and uncertainty quantification. IEEE Transactions on Biomedical Engineering. 2022, 69(8): 2456-2467.

Meta-review: This decision was perhaps the most difficult one, not only because of the larger spread of scores but also because the overall recommendation would be approval but I can see myself more inclined with the lower score. Moreover, both the reviews and the discussion were appalling despite my encouragement to the reviewers to engage. Hence, applying the in dubio pro reo principle and respecting the majority I opted for recommending acceptance as poster.
Regarding the paper itself, the reviews (weak as they look to me) showed praises on the preliminaries i.e. problem statement and justification (R-HATA), and the extensive experimentation and thorough evaluation (R-RxY3 and R-PgT4) across many datasets (R-QG76). Counterbalancing this strengths, reviewers mainly showed consensus on two things; the computational complexity and cost (R-QG76 and R-RxY3, as well as R-HATA which uses the related term “mathematically nuanced”) and its difficult in translating to the real world (R-RxY3). During the rebuttal, the authors made an attempt at addressing the former i.e. “we further analyze the memory consumption and computational complexity of each component”, but did not appear to address the later.
Anyhow, in its current status, I can see the potential of the work but I can clearly see why the scores were not unanimously enthusiastic.
No issues with ethics were highlighted.
Decision: Accept (poster)