Multimodal Physiological Signals Representation Learning via Multiscale Contrasting for Depression Recognition

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 18:00:42

Abstract: Depression recognition based on physiological signals such as functional near-infrared spectroscopy (fNIRS) and electroencephalogram (EEG) has made considerable progress. However, most existing studies ignore the complementarity and semantic consistency of multimodal physiological signals under the same stimulation task in complex spatio-temporal patterns. In this paper, we introduce a multimodal physiological signals representation learning framework using Siamese architecture via multiscale contrasting for depression recognition (MRLMC). First, fNIRS and EEG are transformed into different but correlated data based on a time-domain data augmentation strategy. Then, we design a spatio-temporal contrasting module to learn the representation of fNIRS and EEG through weight-sharing multiscale spatio-temporal convolution. Furthermore, to enhance the learning of semantic representation associated with stimulation tasks, a semantic consistency contrast module is proposed, aiming to maximize the semantic similarity of fNIRS and EEG. Extensive experiments on publicly available and self-collected multimodal physiological signals datasets indicate that MRLMC outperforms the state-of-the-art models. Moreover, our proposed framework is capable of transferring to multimodal time series downstream tasks. We will release the code and weights after review.

Main paper: 
 introduction: 
 Depression is a common mental disorder, which is different from regular mood changes and feelings about everyday life. Characterized by persistent feelings of sadness, lack of interest, social withdrawal, diminished social skills, and even physical symptoms such as dizziness and nausea, depression significantly affects various aspects of life, including relationships with family, friends, and the community, as well as work and study efficiency [20, 28, 49] . It is estimated that about 3.8% of the population are experiencing depression [33] and more than 700 thousand people die due to suicide every year [32] .The first recurrence rate of depression reaches 50% and repeated attacks significantly increase the disability rate. The significant factors affecting diagnosis and treatment are lack of resources and trained healthcare personnel. In addition, the inability to make an accurate assessment is another factor affecting effective treatment. Thus, it is urgent to enhance the accuracy of depression recognition and assessment at early stages, aiming to diminish both recurrence and disability rates. Currently, the recognition and assessment of depression mainly depend on the experienced doctors to perform clinical diagnosis based on professional scales such as the Patient Health Questionnaire (PHQ-9) [22] and Beck Depression Inventory (BDI-II) [13] , as well as biomarker data. However, With the increasing number of patients, early detection is often limited and time-consuming, and subject to individual subjective observation and lack of real-time measurement. Recent strides in brain science have provided critical insights for depression diagnosis [18, 19, 36, 51, 55] , with techniques like electroencephalogram (EEG) [2, 9, 31, 44] and functional near-infrared spectroscopy (fNIRS) [3, 40, 62, 63] becoming increasingly prominent due to their safety, portability, affordability, temporal precision, and minimal environmental demands. Therefore, it is necessary to explore an automatic depression recognition method based on physiological signals to assist the clinical diagnosis of doctors and accelerate the treatment for patients [15, 16, 30] .The wide collection and analysis of multimodal physiological signals such as fNIRS and EEG provide more potential to combine them to perform mental disease recognition. The distinct sampling mechanisms of fNIRS and EEG pose challenges for direct fusion at the data level, leading to a predominant focus on feature-level fusion strategies in recent research. For example, Pietro et al. employed EEG and fNIRS to classify the four symptoms of Alzheimer's disease, which achieved higher accuracy by integrating its complementary characteristics compared with single-modal experiments [5] . Shin et al. utilized typical eigenvalue scores and a common spatial pattern method to fuse the fNIRS and EEG feature [46] . Similarly, Qiu et al. proposed a multimodal feature-level fusion method, achieving good results in the classification of brain activity induced by preference music and neutral music [38] . Furthermore, Zhang et al. designed a feature fusion method based on spatio-temporal alignment strategy to obtain a significantly improved classification level in the motor imagery paradigm compared to the non-aligned method [60] . However, focusing only on feature-level fusion for EEG and fNIRS with time series property makes it easy to ignore the spatio-temporal representation and multimodal complementary features. Moreover, the existing studies have not considered the deep semantic information reflected by physiological signals under specific stimulation tasks, such as the activation status of brain regions.To address the above issues, we propose a Multimodal physiological signals Representation Learning framework via Multiscale Contrasting for depression recognition (MRLMC). This framework employs the Siamese network architecture, which utilizes two encoders with the same structure and shared weights to process different modalities. Specifically, first, fNIRS and EEG are fed into a time-domain data augmentation module to generate different but correlated data. This ensures that MRLMC learns the two types of augmented feature representation of the data. Then, we design a multiscale spatio-temporal convolution (MSC) module to learn the spatio-temporal representation and dynamic characteristics of multimodal physiological signals. The spatio-temporal contrasting module aims to minimize the differences in fNIRS and EEG feature representations while enhancing their complementary nature. Furthermore, we propose a semantic consistency module to further mine the deep semantic information such as the activation status of brain regions. It aims to maximize the semantic similarity of multimodal physiological signals. In summary, the main contributions of this paper include: • 
related work: 
 For EEG-based depression recognition research, Rajendra et al. proposed a convolutional network for EEG data with 15 normal controls and 15 depression patients to perform depression classification and found that the signal in the right hemisphere is more active than the signal in the left hemisphere [1] . classifier based on LSTM to perform classification tasks [26] . fNIRS data has reliably reflect cognitive profiles on the brain in different stimulation tasks [29, 39] , and presents signal differences under different stimulation task time points [57] . Wang et al. proposed a transformer-based fNIRS classification network to explore spatiallevel and channel-level representations of fNIRS signals to improve data utilization and feature representation [54] . Similarly, Zhang et al. achieved mild cognitive impairment recognition by exploiting the multidimensional features of fNIRS data including channel, temporal, and spatial features [59] . Wang et al. transformed fNIRS signals into 2-D wavelet feature maps by using wavelet transform and parallel-CNN feature fusion to diagnose depressive disorder [52] . However, these works mentioned above ignore the nonlinear and segment characteristics of EEG and fNIRS. In addition, ignoring the dynamic characteristics and semantic representation of neural activity under stimulation tasks results in weak classification performance.There are many brain-computer studies on multimodal recognition tasks based on fNIRS and EEG but less research in the area of multimodal depression recognition. He et al. proposed a multimodal multitask neural network model to fuse the EEG and fNIRS signals to achieve motor imagery classification [14] . Gao et al. utilized an EEG-informed fNIRS general linear model to extract common spatial pattern features and the support vector machine was used as the classifier [8] . Differently, we establish a multimodal contrastive learning framework based on the Siamese network architecture. fNIRS and EEG are fed into the spatio-temporal contrasting module and semantic consistency module to extract complementary features, dynamic features and semantic consistency representations to realize multimodal depression recognition. 
methodology: 
 In this section, we describe the components of MRLMC framework in detail. As shown in Figure 1 , the MRLMC framework adopts the Siamese network architecture to learn the feature representations of fNIRS and EEG signals. Specifically, we first utilize the time-domain data augmentation method to generate different but correlated data. Then, we design a spatio-temporal contrasting module to extract the feature representation and dynamic characteristics of the physiological signals. Finally, a deep semantic representation of fNIRS and EEG signals is achieved through the semantic consistency module. This multimodal semantic representation is then fused and fed into the classification layer to realize depression recognition.The collection of fNIRS and EEG data involves stringent conditions, which present challenges due to limited medical resources and the prevalent stigma associated with patients. Therefore, in scenarios with limited data, the data augmentation method plays an important role, and it is also a key part of realizing single-modal contrasting learning. As shown in Figure 2 , when only singlemodal (either fNIRS or EEG) is available, both the raw and augmented data are utilized as pairs. When the input is fNIRS and EEG, they are shaped as a pair of data, with the data augmentation strategy randomly applied to part of the data. The commonly used jitterand-scale strategy and permutation-and-jitter strategy data augmentation methods do not consider both the collection paradigm and the process of physiological data. Since the physiological data for depression patients are mostly collected with specific stimulation tasks, the time-domain augmentation methods including time warping and time masking [42] are utilized to generate different but correlated data.Given a sample 𝑥, the time step of the time masking method is [𝑡 0 , 𝑡 0 + 𝑡 𝑡𝑚 ], where 𝑡 0 ∈ [0, 𝑡 𝑞 ), and the masking parameter 𝑡 𝑡𝑚 ∈ (0, 𝜆], 𝜆 ≤ 𝑡 𝑞 , introducing an upper bound that the width of the time masking cannot be larger than the response time of each question of stimulation task. Similarly, the time step of the time warping method is [𝑡 0 , 𝑡 0 + 𝑡 𝑡 𝑤 ], where 𝑡 0 ∈ [0, 𝑡 𝑞 ) and the warping parameter 𝑡 𝑡 𝑤 ∈ (0, 𝜆], 𝜆 ≤ 𝑡 𝑞 . The augmented data is denoted as 𝑥 ′ , which has the same time scale as 𝑥. Formally, let 𝐷 𝑀𝑢𝑙𝑡𝑖 = {𝐹 𝑖 , 𝐸 𝑖 } 𝑁 be a dataset of fNIRS and EEG, such that each fNIRS sample 𝐹 𝑖 corresponds to EEG sample 𝐸 𝑖 . For each input sample 𝐹 𝑖 and 𝐸 𝑖 , we denote the augmented data as 𝑥 𝑖 and 𝑦 𝑖 , 𝐷 = {𝑥 𝑖 , 𝑦 𝑖 } 𝑁 as the input data. In the case of single modal inputs, 𝑥 𝑖 and 𝑦 𝑖 denote raw data and augmented data respectively and the 𝑁 is the number of samples. Multimodal data is fed into the spatio-temporal contrasting module to extract latent representation.Physiological signals, as a kind of multichannel time series data, are characterized by spatio-temporal features that are the most important kind of representation. Specific stimulation tasks are usually performed to collect physiological signals. When the participants are handling stimulation tasks, the status of the brain is transformed from a resting state to an activated state. Regarding Given an input signal 𝑥, its dimension is 𝑁 𝐶ℎ𝑎𝑛𝑛𝑒𝑙 × 𝑇 , where 𝑁 𝐶ℎ𝑎𝑛𝑛𝑒𝑙 is the number of channels of data and 𝑇 is the collection duration, which is determined by the collection device and the data type. Then, the 𝑥 is fed into the encoder to get the latent representation. The encoder based on the convolution layer maps 𝑥 into a latent representation 𝐶 = 𝑓 𝑒𝑛𝑐 (𝑥), 𝐶 ∈ R 𝑑 , where 𝑑 is the dimension of the feature. Thus, we get 𝐶 for the feature representation of a physiological signal, which is then fed into the multiscale convolution layers. The 𝐶 is passed to the 𝑁 𝑆𝑐𝑎𝑙𝑒 layer multiscale convolution to extract high-dimensional representations 𝐶 𝑒𝑛𝑐 . Then, the representations are fed into 𝑁 𝑆𝑐𝑎𝑙𝑒 spatio-temporal feature extraction blocks 𝑓 𝑏𝑙𝑜𝑐𝑘 (•) to extract spatio-temporal representation of physiological signals. Finally, we get spatio-temporal representation 𝑣 of a physiological signal,EQUATIONwhere𝜑 𝑖 = 𝑚𝑎𝑥 (𝛼 * 𝑁𝑜𝑟𝑚(𝑓 𝑏𝑙𝑜𝑐𝑘 (𝐶 𝑒𝑛𝑐 𝑖 )), 𝑁𝑜𝑟𝑚(𝑓 𝑏𝑙𝑜𝑐𝑘 (𝐶 𝑒𝑛𝑐 𝑖 ))), (2) which simplified to 𝑣 = [𝜑 1 , 𝜑 2 , • • • , 𝜑 𝑁 𝑆𝑐𝑎𝑙𝑒 ], 𝑣 ∈ R 𝑚, where 𝑚 = 𝑁 𝑆𝑐𝑎𝑙𝑒 × 𝑁 𝑂𝑢𝑡 is the dimension of feature, 𝑁 𝑂𝑢𝑡 is the output dimension of the spatio-temporal feature extraction blocks and 𝛼 is the control weight.Through the spatio-temporal contrasting module, the multimodal data generate spatio-temporal representations 𝑣 and 𝑢, where 𝑢 is generated from another modal or augmented data. Given a batch of input samples denoted as 𝑁 = 𝑏𝑎𝑡𝑐ℎ_𝑠𝑖𝑧𝑒, we get 2𝑁 items from fNIRS and EEG. For a 𝑢 item, we denote 𝑢 + as the positive sample for 𝑣, and thus (𝑣, 𝑢 + ) are considered as the positive pair. The other (2𝑁 -2) items in the same batch are considered negative samples for 𝑣, then 𝑣 forms negative pairs with (2𝑁 -2) negative samples. Therefore, we can define the spatio-temporal contrasting loss to maximize the similarity between positive pairs and the difference between negative pairs. Given the 𝑣 and 𝑢 items, we compare the similarity of positive pair (𝑣, 𝑛 + ) with the similarity of (2𝑁 -2) negative pairs, the spatiotemporal contrasting loss L 𝑀𝑆𝐶 is defined as follows:EQUATIONwhere 𝑠𝑖𝑚(•) denotes cosine similarity,EQUATION)where 𝜏 is a temperature parameter. Through the spatio-temporal contrasting loss L 𝑀𝑆𝐶 , the differences of feature representations between fNIRS and EEG could be minimized, which also maximizes the complementarity of these two representations. And then the spatio-temporal representations are fed into the semantic consistency module to further learn deep semantic information.The depression patients are characterized by persistent low mood, pleasure deficit, and cognitive impairment, which presents the difference with the control group on the brain activity level and activation state when performing the stimulation task [42] . fNIRS and EEG reflect brain activation state by detecting slight changes in brain activity, so it is necessary to mine deeper semantic information that can reflect brain activation state. We propose a semantic consistency module to maximize the semantic similarity of multimodal physiological signals and further mine deep semantic information such as brain activation state.We utilize the transformer unit as the semantic feature extraction model because of its context-awareness. The architecture of the transformer unit is shown in Figure 4 , which mainly consists of successive blocks of multi-head attention (MHAttn) and MLP. The MLP block consists of two fully connected layers and a non-linear ReLU. The transformer unit is defined by the following equations:𝑀𝐻𝐴𝑡𝑡𝑛(𝑄, 𝐾, 𝑉 ) = 𝐶𝑜𝑛𝑐𝑎𝑡 (𝐻𝑒𝑎𝑑 1 , 𝐻𝑒𝑎𝑑 2 , • • • , 𝐻𝑒𝑎𝑑 𝑁 𝐻 𝑒𝑎𝑑 )W 𝑂(5) where 𝑄 represents the input feature vector, 𝐾 represents the key vector, 𝑉 denotes the value vector, 𝑁 𝐻𝑒𝑎𝑑 represents the number of heads, and W 𝑂 denotes the final output weights. 𝐻𝑒𝑎𝑑 𝑖 is defined as follows:EQUATIONwhere W 𝑄 𝑖 , W 𝐾 𝑖 , W 𝑉 𝑖 denote the weight matrics of 𝑄, 𝐾, 𝑉 , respectively. 𝐴𝑡𝑡𝑒𝑛𝑡𝑖𝑜𝑛(•) is define asEQUATIONwhere 𝑑 𝐾 denotes the dimentional size of vector 𝐾. Given the spatiotemporal representations 𝑣, we pass it through the transformer unit as follows:EQUATIONand then the 𝜓 𝑖 is input to the MLP block:EQUATIONwhere 𝑁 𝑇 𝑟𝑎𝑛𝑠 denotes the number layers stacked to generate the final feature 𝑧.Given the multimodal spatio-temporal representations 𝑣 and 𝑢, a multilayer stacked transformer unit is utilized to extract the semantic feature 𝑧 𝑓 and 𝑧 𝑒 . The dimension size of 𝑧 𝑓 and 𝑧 𝑒 are the same as 𝑣 and 𝑢. We utilize cosine similarity as the semantic consistency loss to maximize the semantic similarity of multimodal physiological signals. The semantic consistency loss can be denoted as follows:L 𝑆𝐶 = 𝑠𝑖𝑚(𝑧 𝑓 , 𝑧 𝑒 ).(10)Ultimately, 𝑧 𝑓 and 𝑧 𝑒 are concatenated and fed into the classification layer for depression recognition, which includes two fully connected layers and the ReLU layer. In real healthcare scenarios, the collected dataset exists the class imbalance problem, so the focal loss function is utilized to perform depression recognition, which is defined as follows:EQUATIONwhere 𝑃 denotes the predictive probability of the model, 𝛼 is the weighting factor to balance the positive and negative samples, and 𝛾 is the adjustable parameter. The adjustment factor (1 -𝑃) 𝛾 can be adjusted adaptively according to the difficulty of the sample.In instances where samples are inherently easier to classify, the parameter 𝑃 is larger, causing the adjustment factor to tend to zero. Consequently, this results in a reduced impact on the loss function, prompting the model to focus more on samples that are difficult to classify. The overall loss is the combination of the spatio-temporal contrasting loss, semantic consistency loss, and classification loss as follows:EQUATIONwhere 𝜆 1 and 𝜆 2 are fixed scalar hyperparameters denoting the relative weight of each loss. 
experiments: 
 The datasets and implementation details are first presented in this section. We then conducted extensive experiments to validate the effectiveness of the MRLMC framework.To evaluate the performance of our proposed method, we conduct a series of experiments on two datasets.MODMA dataset [25] is a publicly available dataset, and we only use event-related EEG data, including 53 participants (24 outpatients diagnosed with depression and 29 healthy controls). It uses a Dot-probe stimulation task to record EEG signals. The Dot-probe is composed of facial pictures from the standardized native Chinese Facial Affective Picture System [27] . The facial pictures are classified into four sets as fear, sad, happy, and neutral emotions based on their valence. Any two facial images of different valences appear on the screen. During the experiment, participants were asked to focus on the screen and watch freely with their eyes. When the dot appeared, they were asked to press the button quickly and accurately without making any body movements, including head or legs, and as much as possible without making unnecessary eye movements, glances and blinks. Continuous EEG signals were recorded using a 128-channel device. The sampling frequency was 250 Hz.fNIRS-EEG dataset is a self-collected multimodal physiological signals dataset, including fNIRS and EEG signals. We utilize a verbal fluency stimulation task to record data, including 96 participants (79 depression patients and 17 healthy controls) for only fNIRS, and 64 participants (52 depression patients and 12 healthy controls) for both fNIRS and EEG. During the data collection process, doctors helped participants wear the device to ensure the probe was tightly attached to the scalp until the channel pass rate reached 80%. The entire stimulation task includes a pre-task silence period, a task period and a post-task silence period. The silent period required participants to sit up straight in front of the computer, remain calm, and not shake their bodies. During the task period, three questions appear on the computer screen, and participants are asked to name the fruits, appliances and vegetables they can associate with the questions. As shown in Figure 5 , the near-infrared device used in this study has 16 near-infrared (NIR) emission probes and receiving probes, and a total of 53 channels are connected. The detector emits near-infrared light at 690nm and 830nm. Throughout the test period, the NIR device collected the intensity of the emitted light at two wavelengths at a sampling frequency of 100hz. Through the test, each participant had data of 150×100×53×2, where 150 is the duration of the test, 100 is the data collection frequency, 53 is the number of channels and 2 is the number of wavelengths. The EEG device used in this study has 16 channels, and the electrode-wearing method follows the 10-20 lead system standard. The EEG device collects electrical signals at a sampling frequency of 1000hz, with data for each participant of 150×1000×16, where 150 is the duration of the test, 1000 is the data collection frequency, 16 is the number of channels. Multiple experiments were conducted to take the average value of the evaluation indicators.For the MODMA dataset, we utilized the EEGLAB toolkit [6] within the MATLAB platform for EEG denoising. We applied a bandpass filter with a frequency range of 1-40 Hz to the raw EEG signal. Then, we utilized the extended ICA algorithm to obtain multiple independent EEG components to eliminate artifacts and noise components, such as electrooculogram (EOG), ECG, EMG, and eye movement. Finally, the ICLabel plugin was used to remove the identified artifacts and noise components. Additionally, we only selected part of the channel data from the prefrontal brain area, which processes emotional expression.For the fNIRS-EEG dataset, we first utilized the near-infrared data analysis tools for fNIRS data preprocessing. The preprocessing steps begin with the elimination of motion artifacts unrelated to the raw data using the temporal derivative distribution repair method. Subsequently, the light intensity signal was converted into an optical density profile, which was then filtered using the finite impulse response band-pass filter with 0.01-0.08Hz to eliminate noise caused by physiological fluctuations such as pulse and respiration and baseline drift caused by environmental and temperature changes. Finally, the optical density data were converted to concentration change of oxygenated hemoglobin (HbO) and deoxyhemoglobin (HbR) using a modified Beer-Lambert method. Based on fNIRS-based research [3, 10, 63] , this study also deliberately focused on the HbO concentration change data in subsequent method design. Additionally, we only selected part of the channels, which are the red font channels shown in Figure 5 . For the EEG signals, the same preprocessing method as the MODMA dataset was implemented. Specifically, resampling was implemented for both fNIRS and EEG data.The model is constructed using the Pytorch framework and optimized using the RMSprop optimizer. The learning rate, batch size and other parameters are shown in Table 6 .To demonstrate the effectiveness of the MRLMC model and its applicability on single modal modes, we first conducted sufficient experiments on the MODMA dataset. The benchmark algorithms include EEGNet [23] , STGCN [56] , DGCNN [48] , HGP-SL [61] , SAGE [24] , SST-Emotionnet [17] , SGP-SL [4] , CGIPool [34] , SGP-SL [4] , TSception [7] , CLG [43] , dFL [45] and 1DEEG-Transformer [37] for comparison. All models utilize EEGNet [23] 0.568 -0.668 0.600 STGCN [56] 0.588 -0.577 0.596 DGCNN [48] 0.597 -0.459 0.552 HGP-SL [61] 0.585 0.536 0.625 0.577 SAGE [24] 0.679 0.640 0.667 0.653 SST-Emotionnet [17] 0.736 0.692 0.750 0.720 CGIPool [34] 0.736 0.692 0.750 0.720 SGP-SL [4] 0.849 0.808 0.875 0.840 TSception [7] 0.544 -0.445 0.486 CLG [43] 0.765 -0.757 0.759 dFL [45] 0.750 -0.614 -1DEEG-Transformer [37] 0.782 0.784 0.692 0.749 MRLMC 0.867 0.875 0.875 0.864 the raw EEG signals. In addition, the DGCNN and CGIPool models construct the extracted features into a graph structure and mine the relationships between the channels of data. Based on existing research, it has been shown that the prefrontal lobe area of the brain performs emotional expression, which is gradually activated when a stimulation task is performed. Therefore, we implemented a channel selection process before feature extraction. Compared to the DGCNN and CGIPool networks, the MRLMC model improves by 18% in accuracy since the proposed MSC module can also extract channel features. Especially, we also mine the deep semantic information of the data, aiming to mine semantic features such as brain activation levels, and maximize the semantic representation of multimodal data based on consistency loss.Table 3 shows the performance of the MRLMC model on fNIRS data in the fNIRS-EEG dataset. To evaluate the superiority of our method, the baseline methods selected are Logistic Regression (LR), K-Nearest Neighbor (KNN), Support Vector Machine (SVM) [47] , AlexNet [21] , Residual [47] 0.823 0.000 0.000 0.000 AlexNet [21] 0.830 0.790 0.830 0.800 ResNet [12] 0.720 0.670 0.720 0.700 RF [63] 0.833 0.625 0.175 0.267 XGB [63] 0.833 0.525 0.413 0.446 Corr-AlexNet [53] 0.900 0.910 0.900 0.880 GCN [58] 0.854 0.700 0.488 0.563 Diffpool [58] 0.875 0.750 0.475 0.571 MRLMC 0.913 0.827 0.908 0.834 Network (ResNet) [12] , Random Forest (RF) [63] , XGB [63] , Corr-AlexNet [53] , GCN [58] and Diffpool [58] . Our proposed method achieved 0.913, 0.827, 0.908 and 0.834 in accuracy, precision, recall and F1-score, respectively, which are satisfactory results. The accuracy of traditional machine learning methods such as LR, KNN and SVM is not satisfactory, while the accuracy of deep learning algorithms such as AlexNet is relatively improved, which highlights the superior performance of deep learning algorithms in depression recognition based on physiological signals. The Corr-AlexNet, GCN and Diffpool networks compared to traditional machine learning improve the accuracy by about 8%. These methods rely on manually extracted features for learning and lack deep exploration of spatio-temporal representation, dynamic features, and semantic representation. The MRLMC model extracts the spatio-temporal representation and dynamic features of the data through the spatiotemporal contrasting module. Additionally, the main symptoms of patients with depression include low mood and slow thinking, which causes their brains to be activated differently when performing stimulating tasks. The MRLMC model utilizes the semantic consistency module to dig deep into the semantic representation to reflect brain activation states. Compared with traditional machine learning, the accuracy is improved by about 11%, and compared with the method of manually extracting features for recognition, the accuracy is improved by about 1.5%. Overall, based on task-state physiological data, extracting spatio-temporal representation and semantic representation can achieve higher recognition accuracy. The results indicate that satisfactory performance is obtained when utilizing all losses. The performance of using L 𝑀𝑆𝐶 or L 𝑆𝐶 alone is better than using only recognition loss. This proves that our proposed L 𝑀𝑆𝐶 and L 𝑆𝐶 can help the model obtain useful spatiotemporal representation and semantic information. This means that the spatio-temporal contrasting and semantic consistency modules are effective for multi-modal physiological signals for depression recognition.To further investigate the MRLMC model, we analyze in detail the impact of several important parameters of the model on performance in this section. Table 6 exhibits the performance of the MRLMC model with different parameters on the fNIRS-EEG dataset. The first three rows of indicators verify the effects of the number of spatio-temporal convolution blocks on model performance, the middle two rows verify the effects of the number of transformer units, and the last two rows verify the effects of multi-head attention. The results show that different parameters have different effects on the model. When the number of convolution blocks is 5 or 6, the recognition accuracy reaches excellent results. The number of transformer encoder units has a slightly greater impact on the performance of depression recognition. As the number of units increases, the network complexity increases, which causes overfitting of the model. In addition, information may be lost or confused during the transmission process, making it difficult for the network to learn useful semantic information. When the number of multihead attention is 8 or 16, the model achieves superior performance. Figure 7 shows the performance of the MRLMC model with different number of spatio-temporal convolution blocks on the fNIRS-EEG dataset. As the number of convolution blocks increases, the recognition accuracy decreases, which proves that spatiotemporal representation has a great impact on the recognition performance for small-scale datasets. The increased number of blocks means that the complexity of the model increases. The main characteristics of multimodal physiological signals are their spatio-temporal representation and dynamic variability, and their key information is often hidden in local sequence patterns and global temporal dependence. Networks with high complexity may fail to capture these key information, making it difficult to learn effective spatiotemporal representation. Therefore, for the small-scale fNIRS-EEG dataset, the results of spatio-temporal convolution blocks of 5 or 6 are most excellent. If the MRLMC model is to be transferred to other downstream tasks of multimodal time series, the number of spatio-temporal convolution blocks needs to be determined based on the characteristics of the data. 
conclusion: 
 In this paper, we propose a multimodal physiological signals representation learning framework via multiscale contrasting for depression recognition. The Siamese network architecture is utilized to maximize the complementarity between multimodal data. We design multiscale spatio-temporal convolution to obtain more discriminative spatio-temporal representations and dynamic features. The spatio-temporal contrasting module aims to minimize the feature representation and maximize the complementarity of fNIRS and EEG. Meanwhile, the semantic consistency module captures contextual information and the deep semantic information of the data to maximize the semantic representation of multimodal data based on semantic consistency loss. Extensive experiments are implemented on MODMA and fNIRS-EEG datasets, and our proposed model achieves state-of-the-art performance on both singlemodal and multimodal data. Moreover, the analysis of the feature distribution and key parameters of each module shows that each module plays an important role in mining spatio-temporal representations and semantic features. Notably, the proposed model is a generalized architecture based on multichannel physiological signals, which can be extended to other mental disorders and cognitive ability recognition in the future.  
 
 Appendix:  


Decision: Final decision - accept

Congrats! All reviewers saw merit in the work

Please ensure that all comments from the reviewers are addressed as per the rebuttal. In addition, the following discussion arose from the reviewers:
-A clearer explanation of how the convolution layer is used to extract temporal features from the input data.
-According to Table 3, the proposed MRLMC brought about 9% improvement compared to SVM. How did the authors get the number "11%"?

Reviews: review: paper_summary: paper_summary: This paper introduces a multimodal physiological signals learning framework, using a siamese architecture for depression recognition. This uses near-infrared spectroscopy (fNIRS) and EEG. Extensive evaluation is performed on publicly available datasets and comparisons performed with existing state of the art. Results show that this framework outperforms the state of the art.
strength_weakness: strength_weakness: This paper adopts a novel approach with the use of siamese architecture. Extensive evaluation is performed showed very promising results which outperform the current SOTA. Paper is very well written. Strong analysis of results is also performed.
limitations: limitations: Tables 4, 5, and 6 could be better explained in the text and are a little difficult to follow.
Please correct the following typos:
line 80 - lower case 'w'
208/9 - Clean up / re-word this sentence, it is a little difficult to follow.
447- - define(d)
475 - the collected dataset exists the class imbalance problem - Please re-word this sentence, it is a little difficult to follow
paper_summary: paper_summary: This paper proposes a multi-modal framework called MRLMC, which combines fNIRS and EEG signals to realize depression prediction. The authors design a spatio-temporal contrasting module and a semantic consistency module to get the feature vector from the two input signals. At last, the features are integrated to predict the depression status.
strength_weakness: strength_weakness: 1. Exhaustive experiments on both public and self-collected datasets were conducted to prove the effectiveness of the proposed MRLMC, and the results showed that this method was superior to other methods with both single-modal and multi-modal inputs. 
2. It is innovative to combine two kinds of physiological signals, EEG and fNIRS, to achieve better prediction results for depression.
limitations: limitations: 1. How did the proposed MSC module extract “spatial-temporal representations” as it only consists of several common convolution layers? It seems that only because the input data contains spatial-temporal information that the module can get the result.
2. In section 4.3.2, some statements like “Compared with traditional machine learning, the accuracy is improved by about 11%”. is not precise. How did the authors get the number? Based on the average of the results of traditional machine learning methods or the result of one of the methods?
3. A concern is that MRLMC seems to be the combination of several convolution layers and transformer blocks, so the novelty of the proposed method is insufficient.
paper_summary: paper_summary: The research presents a multimodal physiological signals framework, MRLMC, for depression recognition. It enhances the complementarity and semantic consistency of fNIRS and EEG data during identical stimulation tasks, overcoming the limitations of previous approaches. By applying time-domain data augmentation, the framework generates diverse but related datasets. A spatio-temporal contrasting module uses multiscale convolution to learn representations, while a semantic consistency contrast module maximizes the similarity in meaning between fNIRS and EEG signals. Experiments show MRLMC surpasses current models in depression detection accuracy and is extendable to other multimodal time series applications.
strength_weakness: strength_weakness: 1. The research issue is meaningful.
2. The writing and story-telling is good.
3. Figures are clear and ablation study is sufficient.
limitations: limitations: 1. In Table 3 and Table 4, why are the results of MRLMC in Table 3 not the same as those of the first line in Table 4? I think these two are both results with single fNIRS only, and they should be the same.
2. I am a bit confused about the semantic consistency loss (equation 10), which is aimed at maximizing the cosine similarity between fNIRS and EEG representations. Shoud it be 1-sim(z^f,z^e)? Moreover, it is not very intuitive to simply push the two modalities togather as this may lead to the loss of complementary information. Although the ablation study in Table 5 seems to 
3. In  Section 4.2.1, the authors claimed that they conducted mulitple experiments and reported the average value. I think standard deviation should also be reproted to show the stability of each method.
4. How did the training, testing, and validation set splitted? Did the training, testing, and validation set contain samples from the same subject? Each set should exclude subjects from each other to obtain a convincing result.