A Multimodal BiMamba Network with Test-Time Adaptation for Emotion Recognition Based on Physiological Signals
Authors: Ziyu Jia
Tingyu Du
Zhengyu Tian
Hongkai Li
Yong Zhang
Chenyu Liu
Venue: NeurIPS 2025 poster ()
Keywords: Emotion Recognition
Mamba Network
Test-Time Adaptation
Multimodal Physiological Signals
Primary area: neuroscience_and_cognitive_science

Abstract: Emotion recognition based on physiological signals plays a vital role in psychological health and human–computer interaction, particularly with the substantial advances in multimodal emotion recognition techniques. However, two key challenges remain unresolved: 1) how to effectively model the intra-modal long-range dependencies and inter-modal correlations in multimodal physiological emotion signals, and 2) how to address the performance limitations resulting from missing multimodal data. In this paper, we propose a multimodal bidirectional Mamba (BiMamba) network with test-time adaptation (TTA) for emotion recognition named BiM-TTA. Specifically, BiM-TTA consists of a multimodal BiMamba network and a multimodal TTA. The former includes intra-modal and inter-modal BiMamba modules, which model long-range dependencies along the time dimension and capture cross-modal correlations along the channel dimension, respectively. The latter (TTA) mitigates the amplified distribution shifts caused by missing multimodal data through two-level entropy-based sample filtering and mutual information sharing across modalities. By addressing these challenges, BiM-TTA achieves state-of-the-art results on two multimodal emotion datasets.

A Multimodal BiMamba Network with Test-Time Adaptation for Emotion Recognition Based on Physiological Signals
Ziyu Jia
Tingyu 3.
Zhengyu Tian Hongkai Li Yong Zhang
Chenyu Liu?*
Instittte of Automation, Chinese Academy Sciences Shanghai Key Laboratory of Data Science Beijing Key Laboratory of Mobile Computing and Pervasive Devices,
Institute of Computing Technology, Chinese Academy of Sciences 'University of Chinese Academy of Sciences
"School of Computer Science and Technology. Beijing Jiaotong University "School of Information Engineering, Huzhou University "College of Computing and Data Science, Nanyang Technological University
jia. wiw.ooutoooooo...oooo com.
(tingyu_du, zhengyu_tian, hong 0sina. com, zhyong@zjhu. edu. cn, chenyu003@e ntu, edu.
Abstract
Emotion recognition based on physiological signals plays vital role in psycho- logical health and human-computer interaction, particularly with the substantial advances in multimodal emotion recognition techniques. However, two key chal- lenges remain unresolved: 1) how to effectively model the intra-modal long-range dependencies and inter-modal correlations in multimodal physiological emotion signals, and 2) how to address the performance limitations resulting from missing multimodal data. In this paper, we propose multimodal bidirectional Mamba (Bi- Mamba) network with test-time adaptation (TTA) for emotion recognition named BiM-TTA. Specifically, BiM-TTA consists of multimodal BiMamba network and a multimodal TTA. The former includes intra-modal and inter-modal BiMamba modules, which model long-range dependencies along the time dimension and cap- ture cross-modal correlations along the channel dimension, respectively. The latter (TTA) mitigates the amplified distribution shifts caused by missing multimodal data through two-level entropy-based sample filtering and mutual information sharing across modalities. By addressing these challenges, BiM-TTA achieves state-of site--of-hh-aas tet-hh-ar art results on two multimodal emotion datasets.
Introduction
The mining and analysis of emotional data contribute to the diagnosis of mental disorders and psychological health assessment. In recent years, data sources for emotion recognition research have mainly focused on two aspects: non-physiological signals and physiological signals[]. Non- physiological signals, such audio and video, are affected by subjective bias and intentional concealment[2]. This makes difficult for non- poyyyisiiollaia signals to represent the emotional state of the body reliably. In contrast, physiological signals, such glectroencephalograahy (EEG) and electrooculography (EOG), reflect the true emotional state the human body objectively[3
*Coreeponniig author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

61. Therefore, emotion recognition based on physiological signals demonstrates great application potential in fields related mental health 7 g
Moreover, compared to unimodal physiological signals, emotion recognition based on multimodal physiological signals combines information from multiple sources to provide more comprehensive emotional assessment. Although multimodal physiological methods for emotion recognition have achieved significant progress [1I there are still two challenges:
1) How to effectively model both the intra-modal long-range dependencies and inter-modal corre- lations of multimodal motton--elattee physiological signals. For the former, as typical form of time-series data, emotion-ree reeattee-eeeted physiological signals exhibit long-range dependencies reflecting the gradual accumulation of emotional changes. Emotions, such anxicty, develop progressively and require a duration for their physiological manifestations to accumulate. Such accumulation underscores that emotional changes are not just state that occurs instantaneously but a process that evolves. For the latter, the inter-modal correlations are evident in the way different modalities respond. For instance, increases in EEG activity during emotional arousal are often accompanied by galvanic skin response (GSR) conductance peaks and decreases in electrocardiogram (ECG) heart-rate variability 112.
Traditional backbone networks have limitations in modeling intra-modal long-range dependencies and inter-modal correlations. For motralmodal modeling, ANN-based networks [13] [4] excel at extracting local features but struggle to capture essential long-range temporal information. Although stacking more convolutional layers together with pooling operations expand the receptive field can theoretically increase context 15 17), it often degrades fine lineggraneed local details 18]. What's able to capture global emotional patterns, but their attention
more, Transformer- based networks mechanism lacks explicit temporal filtering capability and tends to distribute weight uniformly across all time steps. This makes it difficult for these methods to construct practical long-range dependencies based on key nodes of emotion fluctuations. For inter-modal modeling, the token-level attention in Transformer-based networks only computes pairwise relationships between channels and maintains only instantaneous interaction states. However, the complex dependencies inherent in physiological signals cannot be fully captured through pairwise channel-wise interactions, making difficult to model high-order correlations. This limitation essentially stems from the lack of explicit global state variables in the attention mechanism 20). which hinders the effective integration of cross- modal information. Therefore, how to effectively model the intra-modal long-range dependencies and inter-modal correlations of multimodal physiological signals remains a significant challenge.
2) How to mitigate the impact of missing multimodal omotion-related physiological data on model performance. When acquiring emotion-related physiological signals, subjects must remain seated for extended periods while exposed to high-intensity stimuli to generate emotional responses. However, uncontrollable factors such as perspiration, changes in posture, or the drying of the conductive gel may cause sensors to slip or experience poor contact, resulting in incomplete multimodal signal acquisition to varying degrees [1] (2].. Such incompleteness amplifies the distribution shifts in emotion-relatedd physiological data, ultimately leading to degraded model performance. First, emotion- related physiological data inherently exhibit distribution shifts between the training and test sets 23) These shifts arise from within-subject variations in emotional state, cognitive load, and environmental conditions across sessions. Furthermore, the unavoidable presence of missing data disrupts the original data patterns, skews feature distributions, and introduces bias, which collectively amplify the existing distribution shifts in physiological data. This amplification makes it increasingly challenging for pre-trained models to capture emotional patterns accurately.
Existing methods focus on training phase measures deal with the missing data problem. For example, Salazar et al. 24 propose feature- and decision-level fusion methods to address the data missingness issue in multimodal emotion recognition. However, these methods typically require retraining to adapt to new missing data patterns, limiting their application flexibility. Furthermore, a promising approach to mitigate the effects of missing data by fine-tuning the model during the testing phase. For instance, Yang et al. (25]. Lei and Pernkopf [66 address unimodal corruption in multimodal data by adjusting self-attention modules to assign lower weights to the corrupted modality. However, these methods cannot handle cases of multimodal corruption where multiple physiological signals are simultaneously missing. Therefore, mitigating the negative impact of missing multimodal data on emotion recognition models remains significant challenge in practical applications.
2

---

In light of these challenges, Mamba's selective input mechanism efficiently models long-range range dependencies, and its explicit global state variables as part of its state space model (SSM) capture inter-modal correlations simultaneously, thereby effectively addressing the first challenge. Addition- ally, test-time adaptation (TTA) requires only minimal parameter fine-tuning to mitigate distribution shifts, thus resolving the second challenge. Therefore, we propose multimodal bidirectional Mamba (BiMamba) network with TTA for emotion recognition named BiM-TTA. BiM-TTA consists of a multimodal BiMamba network and a multimodal TTA. The multimodal BiMamba network includes an intra-modal BiMamba module and an inter-modal BiMamba module, and the multimodal TTA includes two-level entropy- based sample filtering and mutual information sharing across modalities.
Overall, the key contributions of this study can be summarised as follows:
We design multimodal BiMamba network, where the intra-modal BiMamba module models long-range dependencies within modalities, and the inter-modal BiMamba module captures modrl correlations.
We propose multimodal TTA method that alleviates the negative impact of amplified distribution shifts caused by missing multimodal data on model performance.
The evaluation of BiM-TTA on two multimodal emotion datasets confirms its state-of-the-art performance and effectiveness.
Related Work
In recent years, physiological signals have become a key focus in emotion recognition due to their ability to accurately and objectively reflect the genuine emotions of subjects. For instance, Liu et al.
27 used maximum relevance and minimum redundancy to extract emotional information for feature selection. Similarly, Bazgir et al. [28] enhanced the accuracy by applying a cross-validated SVM with radial basis function kernel for classification. However, traditional machine learning methods, which often rely heavily on expert knowledge, are limited by feature design and selection [29] 30).
To overcome these limitations, researchers have applied deep learning methods to emotion recognition tasks. For example, in unimodal emotion recognition, Jia et al. [31] proposed SST-EmotionNNtt, an attention-bart based 3D dense network that simultaneously integrates spatial spectral-tempeer features within a unified framework. Ding et al. [14] designed TSception, multi-scale convolutional neural network that extract temporal dynamics and spatial asymmetry features from EEG signals. These deep learning methods achieve remarkable results in the unimodal domain.
Moreover, studies have shown that multimodal methods better capture the diversity and com- plexity of emotions, resulting in superior performance in emotion recognition. For instance, Ma et al. 32 developed multimodal residual LSTM network, sharing weights across modalities. Hssayeni and Ghoraani [33] explored the use of deep convolutional neural networks for two multimodal data fusion to evaluate positive and negative emotions. Koorathota et al. 34 proposed the Multimodal Neurophysiological Transformer, which employs cross-modee attention to capture mnterlmodal cor- relations. Chen al. (35) designed an attention-bia -ased recurrent graph convolutional network that integrates multimodal physiological features with a convolutional block attention module. Huang et al. 36 introduced GJFusion to address modality heterogeneity through channel-level inter-modality correlations based on graph joints.
Despite achieving promising performance, existing methods in multimodal emotion recognition primarily rely on traditional backbone networks, posing limitations in modeling long-range dependen- cies and inter-modal correlations of multimodal physiological signals related to emotion. addition, these methods rarely address the issue of missing multimodal data. To address these challenges, we propose BiM-TTA, which consists of the multimodal BiMamba backbone network and TTA. Further discussion of related works on Mamba and TTA is presented in Appendix A.1
3 Methodology
As shown in Figure I we propose BiM-TTA, comprising the multimodal BiMamba network and the multimodal TTA. The multimodal BiMamba network consists of intra- and inter-modal BiMamba modules, which extract modality-specifff features and perform feature fusion across modalities.
3

---

The multimodal TTA includes two steps: two-level cntropy-based sample filtering and mutual information sharing across modalities. These steps effectively mitigate the impact of distribution shifts amplification caused by missing multimodal data on model performance.
BiMamba
Inidencoder
BiMamba,
Us
Concat
BiMamba
Classifier
Lineara
Inter-modal BiMamba
Intoerr
BiMamba,
Ug
Classifier,
Pans
SSM
SSM
Classifior
E2
train
Par
Xat
Encodery
BiMambay
Up
Sunv18
CORVEID
Classifiers
Ey
Source Domain
Intra-modal
BiMamba
Lineaty
Multimodal BiMamba Network
Sample
Information Sharing
Ent(x)
p'O)(x)) p'(O2kaz)
Ent(x;) :
Samples
Filtering
P(alka))
Modalitt Encoder
Fasion Layer
Classiffor
Eais
Ent(itu)
PCPn (oee)
p' Ira))
Weakshhif samples
Multimodal BiMamba Networs
Prest
Ent(x)
Strong
Ent(x)
p(P)))
samplex
Target
Domain
Next tteration
Multimodal TTA
Figure 1: The overall structure of BiM-TTA. It contains the multimodal BiMamba network and the multimodal TTA. For the former, the intra- and inter-modal BiMamba modules capture and fuse the features of the different modalities, respectively. For the latter, samples with weak distribution shifts and rich multimodal information are selected for adaptation, while the remaining samples are retained until the next iteration. Then, mutual information sharing across modalities is performed match the information between different modalities effectively. "Concat" represents the concatenation of multiple feature vectors U1 U2 ..., UM. "T" denotes matrix transposition.
3.1 Preliminary
In this paper, define emotion recognition based physiological signals multi-class classifica- tion task for time-series data. The input data consists physiological signals x = (21, a }. where E RC.xL,i E 21,2,... M) represents the i-th-modality data, C2 is the number of acqui- sition channels for the i-th modality, and represents the number of time steps of the input signal. The ground truth label is discrete value corresponding to category, which is then converted into a one-hot encoding p(y x) E -R-V where N is the number of categories. The model outputs the predicted category probability p(ū xx) N
3.2 Multimodal BiMamba Network
As shown in the upper part of Figure the multimodal BiMamba network consists of two modules: the intra-modal BiMamba module and the inter-modal BiMamba module. The former is designed to model each modality independently, and the latter effectively fuses multimodal features.
3.2.1 Intra-modal BiMamba Module
Mamba is designed for natural language processing tasks, where its output at each time step depends only on the current input and hidden state, without involving information from future time steps. This unidirectional modeling approach is well suited for generative autoregressive tasks, as these tasks rely primarily on previous information for inference (37]. However, in emotion classification tasks based on physiological signals, it is necessary to capture contextual information from physiological time series simultaneously, and a unidirectional approach cannot fully capture the complex temporal patterns. To address this limitation, we design the intra-modal BiMamba module. This module com- prehensively integrates the temporal features of each modality through a selective input mechanism and bidirectional modeling.

---

Initially, we design the initial encoder InitEncoder,() to extract shallow features from specific modality:
h; InitEncoder; ( E (1, M)
(1)
where 22 E RC is the ipput of the i-th modality, in which C, denotes the number of input channels for the i-th modality and L denotes the number of time steps. h; E _ xc; represents the initial feature representation output of the i-th modality, Cý represents the number of output channels of the i-th modality, and L' represents the feature length of the output signal. M represents the total number of modalities.
Next, we introduce the BiMamba to model the temporal dimension further, and its structure is shown in Figure T Its core consists of the following three steps:
1) First, BiMamba employs gating mechanism that adaptively weights features to highlight emotion- relevant information while suppressing noise and redundancy:
9: = a (WPh; bg)
(2)
where 9i represents the output of the gating mechanism. a(-) represents the SiLU activation function. W represents the weight matrix and by represents the bias.
2) Second, BiMamba models temporal dependencies from both forward and backward perspectives using state-space modeling, thereby capturing richer contextual dynamics:
h g: & SSM, (a (Conv IDo, (Whh,; + b())
b 9t & SSM (o (Conv [ConvID,- (Rev, (W/h, + 69.) )))
_ and h E RL'xC' are the feature representations of the i-th modality in forward and backward modeling. Revr(.) denotes the flipping on the time dimension L'. SSM represents Selective State Space Model. W.th represents the weight matrix and 6y represents the bias.
where h
3) Finally, BiMamba applies linear projection and residual connection to integrate bidirectional features and stabilize training:
h1 + Rev, (h):) +
2 = h4
W
bf
(5)
where u, E 4'x is the output of the intra-modal BiMamba module of the i-th modality. WF is the weight matrix, and 69 is the bias.
3.2.2 Inter-modal BiMamba Module
In the intra-modal BiMamba module, focus state modeling at the unimodal time scale. However, relying solely on intra-modal modeling neglects inter-modal correlations that are crucial for comprehensive multimodal understanding. Hence, we propose the inter-modal BiMamba module for modeling correlations across modalities. Specifically, we concatenate the features of each modality along the channel dimension C₁, (1.2...., M). and swap the time and channel dimensions. This process is defined as:
m Transpose (w1l//2l1 ..... (uM))
(6)
where m E REE x' represents the concatenated multimodal feature matrix. || represents concatenation along the channel dimension. Transpose(-) represents the swapping of the time and channel dimensions. This operation integrates the features of each modality into the unified feature matrix m. m, each set of continuous channels represents the features of modality.
Subsequently, we input the multimodal feature matrix m into the BiMamba structure to perform bidirectional state modeling of the features from different modalities along the channel dimension >M C² This process is defined as:
H BiMamba(m)
(7)
where H E REM, C represents the inter-modal feature fusion representation. BiMamba()) defined 3.2.T Through the inter-modal BiMamba module, the features of different modali- ties can interact with each other. In the forward process, the hidden states built from modalities
5

---

1} assist in constructing the feature representation of the i-th modality. In the backward
process, the hidden states derived from modalities {M, M ......& i. 1} facilitate the construction of supplementary feature representations for the i-th modality. The bidirectional state modeling mechanism can capture inter-modal correlations from multiple perspectives, thereby enhancing the representational ability of multimodal features.
3.2.3 Auxiliary task
To effectively learn the optimal states of each modality encoder during training, while balancing the training progress across modalities and preventing overfitting in any single modality, we further design an auxiliary task. After independently modeling the unique feature representations of each modality through the intra-modal BiMamba module, an additional classifier is introduced to output unimodal prediction probabilities:
p (3: mx) p (je uu) softmax (W,u; bx)
where p(u <;;) E RR represents the prediction probability of the i-th modality. N is the number of classes. u, represents the feature matrix output by the sirra-modal BiMamba module of the i-th modality. W. is the weight matrix and b, is the bias.
To optimize the auxiliary classifier for each modality, the cross-entropy loss Li is computed as:
C. = n2 Σ
ply x)00 log xx)())
where ply (z)(0) represents the one-hot encoding the label for the j-th sample classification task. p(h 2)) represents the prediction probability the i-th modality for the j-th sample. n denotes the number of samples in batch.
To jointly optimize the model across modalities, the overall training objective Lirein is defined the sum of the main classification loss Linck and the auxiliary losses from all modalities:
M
Ctrasn Leask
CqCi
(10)
where Lrask denotes the cross-entropy loss for the final classification output. a; represents the auxiliary task weight of the i-th modality.
3.3
Multimodal TTA
As shown the bottom half of Figure 1 the multimodal TTA method consists of two key steps: two-level entropy-based sample filtering and mutual information sharing across modalities.
3.3.1 Two-level Entropy-based Sample Filtering
Missing multimodal data affects samples differently, with a small subset suffering severe degradation of multimodal information due to the loss of key emotional cues. Directly using such samples for fine-tuning often harms model performance. To address this, we propose a two-level entropy-based sample filtering method:
(a) Multimodal entropy reflects the model's certainty about its prediction [38]. A lower value suggests that the prediction is more reliable, which often corresponds to distribution in the target domain closer to that in the source domain. Using such samples for fine-tuning leads to more stable and reliable model adaptation. Conversely, higher value indicates greater uncertainty, which often corresponds to strong distribution shifts from the source domain, making these samples unsuitable for fine-tuning.
(b) Unimodal entropy measures the extent to which a sample relies on multimodal information 26
39 A lower value indicates that the prediction can made primarily based on a single dominant modality, suggesting that the sample contains limited multimodal information. In contrast, higher value implies that the prediction is more likely to integrate multiple modalities, indicating that the sample possesses richer multimodal information and is more suitable for fine-tuning.

---

Therefore, we design method that selectively retains samples with low multimodal entropy and high unimodal entropy. These samples are more conducive to stable model adaptation and contain richer multimodal information. First, we compute the multimodal entropy Ent(x)) and unimodal entropy Ent (I:)) of each sample:
N
Ent (z) =
Σ
p =
logp (ú = c x))
(11)
N
Ent (I:) = Σ
p (1 - c | xi) logp (5i - c xi)
(12)
c=
where p(g æ)) is the multimodal prediction probability, p(u xi) is the prediction probability of the i-th modality, and N is the number of categories. Next, we employ iterative entropy-basee sample selection strategy to progressively expand the range target-domain samples. During this process, the model first adapts to samples with weak distribution shifts and rich multimodal information. As the iterations progress, it gradually incorporates samples with strong distribution shifts to achieve smooth adaptation from the source domain to the target domain. Specifically, gradually increasing threshold is used to ensure a smooth adaptation. The thresholds Ym and 7/u defined as:
mm
Ent(x) +mmm Be.
(13)
M nr Ent((x)))) (0) - % * 3/
7/
(14)
where Y/. and T/e are related to the variances of multimodal and unimodal entropy, respectively. 3 represents the smoothing factor. B, = s (1 - 33) is the current iteration number, iter the total number of iterations, 3 is hyperparameter. M is the total number of modalities. represents the number of samples in a batch. Samples used in previous iterations are excluded from further ones. Details about the rationale for threshold selection provided in Appendix A.2 Finally, the filtering employs the following identification criteria:
M
S(x)
x
(x) S 7m and Σ Ent,)t 2 Tu
(15)
where 14 is the hyperparameter for the unimodal entropy weight of the i-th modality.
3.3.2 Mutual Information Sharing Across Modalities
Mutual information sharing across modalities uses the more informative modalities to guide the learning of modalities with significant missing information. This alleviates the impact of amplified distribution shifts between modalities, thereby mitigating the negative effects of amplified overall distribution shifts in the target domain. Specifically, we define the complementary probability of the prediction probability p( x;) of the i-th modality as p' (Mi æ;)):
Σ p(ús m.y) - p(ñe wx)
p' x.)
(16)
M
1
where M represents the total number of modalities.
To improve the consistency of predictions across different modalities, we can minimize the KL divergence between the probability P(ú, (zy)) and its complementary probability p' (2,). However, if modality is severely corrupted, minimizing the KL divergence may negatively impact the informative modalities. Therefore, we also include the multimodal probability p(ū x)) improve stability and reliability. The mutual information sharing loss across modalities Lmis (xr) is defined as:
M
(x)
DEL (1) xx)) 1 (p' (st x;)) + (ū x))
i=1
(17)
M
N
(Mi x2)(() (1) log 2p (vi 12:)
// (ι 9) + p x))0
7

---

where the factor 2/2 represents the average weight. N represents the number of prediction categories. Through mutual information sharing across modalities, the model can effectively align information between different modalities. When some modalities perform poorly, other modalities can provide valuable information, enhancing the overall prediction performance.
3.3.3 TTA optimization
Regarding the TTA optimization, we focus on two aspects: loss computation and fine-tuning details.
Loss Computation We use weighted terms to emphasize the contribution of samples during the adaptation process. The weight term a (x) is defined as:
a
(18)
exp (Ent (x) Ento)
where Ento represents the predefined normalization factor [4]. Ent(()) is the multimodal entropy of the sample. Based on the weighted term, the final loss ftest formulated as:
Listt (z) = a (z) [(xES(a)) (Ent (z) + ALmis (z))
(19)
where (.)(() (-) represents the indicator function. S(z) is the retained sample set obtained through two-level entropy-based sample filtering. À represents the hyperparameter.
TineiTuning Details Inspired by surgical [4]] only the parameters of the first convolu- tional layer in the initial encoder, the first fully connected layer of the inter-modal BiMamba module, and all batch normalization layers are fine-tuned. This selective fine-tuning strategy effectively reduces computational overhead while maintaining the stability of the model's essential component. Finally, the objective function for multimodal TTA is defined as:
min Liest (2)
(20)
④ represents the tunable parameters. è represents all the parameters of the model. This means that only the tunable parameters are updated during test based on Lrest. while the other parameters remain fixed. The details of the multimodal TTA algorithm are presented Appendix A.3
where Ô
4 Experiments
4.1 Datasets
BiM-TTA is evaluated on two publicly available multimodal datasets: DEAP and MAHNOB- The detailed description DEAP and MAHNOB-HCI is presented in Appendix A.4
43
4.2 Experiment Settings and Implementation
We employ trial-wise 10-fold cross coss-validation experimental setups under two conditions: 1) without missing multimodal data, and 2) with missing multimodal data. We split each trial into 4s non- overlapping segments, also known as cropped experiments. In each round of the 10-fold cross- validation, 9 folds used for training and the remaining fold is used for testing. Within the training folds, 80% of the data are used for model training and the remaining 20% for validation. To ensure the representativeness of the evaluation, both the training and test sets, thereby avoiding potential data leakage risks [14]. In the experimental evaluation of missing multimodal data, we emulate varying degrees of data absence by applying random masking with predefined ratios. Specifically, masking ratios of 0.2, 0.4, 0.6, and 0.8 are used to represent increasing levels of data incompleteness,, from mild severe.
make that the same trial does not appear in
For modality selection, EEG, EOG, and clcctromyography (EMG) are used for DEAP, whereas EEG, ECG, and GSR are used for MAHNOB-HCC The proposed BiM-TTA model is implemented on the Py Torch framework and optimized using Adam with a learning rate of 0.001. The batch size is configured to 64 for DEAP and 32 for MAHNOB-HO HCI. For fair comparison, modify all baseline methods to their multimodal versions. Detailed values of the hyperparameters mentioned in the paper provided in Appendix A.5
Our baseline includes two categories: the emotion recognition baseline models and the TTA baseline methods. Detailed descriptions of the baselines are in Appendix A.6
8

---

4.3 Experiment Analysis
The proposed method is first evaluated against representative emotion recognition models two datasets without missing data, and subsequently compared with state-of-the-art TTA methods with missing data. achieves superior performance both comparisons.
Table I: Comparison of emotion recognition baselines DEAP and MAHNOB-HCI in terms of valence and arousal accuracy.
DEAP
MAHNOB-HCI
Valence
Arousal
Valence
Arousal
SVM
0.552
0.584
0.564
0.573
EEGNet
0.566
0.593
0.609
0.612
ACRNN
0.609
0.638
0.610
0.613
HetEmotionNet
0.625
0.633
0.607
0.601
TSception
0.613
0.635
0.633
0.599
LGGNet
0.618
0.636
0.632
0.609
BEG-Deformer
0.609
0.630
0.587
0.595
MambaFormer
0.621
0.587
0.588
0.619
SST
0.613
0.623
0.606
0.616
VSGT
0.631
0.628
0.613
0.599
(OMMS
0.673
0.641
0.650
0.635
For experiments without missing data, shown in Table 1 deep learning methods generally outper- form SVM. EEGNet and TSception primarily use CNNs to extract intra-modal temporal features. ACRNN combines CNNs and unidirectional LSTM model temporal features, and HetEmotionNet uses unidirectional GRU to model the time dimension. However, these CNN- and unidirectional RNN-based models have limitations in capturing long-range dependencies within the modality. Fur- thermore, GEGG-Dformmr combines CNNs and Transformers to extract complex temporal features. ManbaFormer embeds Mamba modules into Transformer feed-forward layers to unify long- and short-range modeling, and SST couples a Mamba-ber based global expert with windowed Transformer local expert and fuses their outputs. Although these hybrid models strengthen temporal dependency modeling, they still overlook inter-channel interactions, as they rely on simplified fusion strategies such as single-layer convolutions or linear projections, which constrain their capacity to capture complex inter-modal relationships. VSGT and LGGNet model the inter-channel relationships using GNNs, yet their temporal modeling relies solely on CNNs and fully connected layers, which hampers the capture of complex long-range dependencies. In contrast, BiM-TTA models long-range dependen- cies within modalities and correlations between modalities through the intra-modal BiMamba module and inter-modal BiMamba module. It achieves the best performance on both datasets, demonstrating superiority the multimodal BiMamba backbone network.
Table 2: Comparative analysis of accuracy for different TTA methods DEAP with missing data, relative to the bascline method "No Adapt". The No Adapt baseline corresponds to a model pretrained on the source domain and directly evaluated on the target domain without any adaptation. Results report the improvement rate of each TTA method over "No Adapt" at mask ratios of 0.2, 0.4, 0.6, and 0.8, along with the average improvement.
Valence %))
Arousal %
Mask ratio
0.2
0.4
0.6
0.8
Avg
0.2
0.4
0.6
0.8
Avg
Tent
-0.162
0.000
-0.314
-0.162
-0.162
0.471
-0.469
0.167
0.232
0.101
EATA
-0.315
0.157
-0.154
0.076
-0.061
0.701
-0.705
0.305
0.234
0.134
READ
1.562
0.625
0.234
0.703
0.781
0.312
-0.070
0.391
0.859
0.372
2LTTA
0.937
0.937
0.546
1.010
0.858
0.390
0.156
0.937
0.234
0.429
BiM-TTAAours)
1.406
1.250
0.859
1.172
1.172
1.016
1.719
1.094
1.406
1.309

---

Table 3: Comparative analysis of accuracy of different TTA methods MAHNOB-HH HCI with missing data, relative to the baseline method No Adapt. Results show the improvement rate of each TTA method over "No Adapt" at the four mask ratios and the average improvement.
Valence %)
Arousal %)
Mask ratio
0.2
0.4
0.6
0.8
Avg
0.2
0.4
0.6
0.8
Avg
Tent
-0.185
0.120
-0.291
0.183
-0.043
0.529
0.046
-0.370
0.139
0.086
EATA
0,556
-0.185
0.046
0.265
0.171
0.635
0.185
-0.139
0.185
0.217
READ 0.523
0.370
0.079
-0.741
0.058
0.529
-0.741
-0.556
0.079
-0.146
2LTTA
0.450
-0.218
0.079
0.185
0.124
0.575
-0.324
-0.185
0.000
0.017
BiM-TTA(ours)
2.413
0.787
0.787
0.370
1.089
0.866
0.417
0.556
0.185
0.506
For experiments with missing data, Table and Table present the performance improvements of BiM-TTA and other TTA methods relative to the "No Adapt" method. "No Adapt" denotes a
model trained on the source domain and subsequently evaluated on the target domain containing missing data, without any adaptation. Tent improves model performance on the target domain by minimizing predictive entropy. READ fine-tunes the self-attention fusion layer to adjust inter-modal weights, alleviating the impact of information disparity across different modalities. 2LTTA employs two-level objective function that includes entropy-based sample reweighting and hiversity-promotie loss. However, they indiscriminately adapt the model to all samples with missing data at once, which may result significant parameter adjustments to fit the target domain, thereby reducing overall performance. Moreover, EATA selectively adapts to reliable samples. However, it does not account for the varying levels of degradation across different modalities and thus does not fully integrate the useful complementary information between modalities. In contrast, method uses two-level entropy-base -baed sample filtering to avoid the model directly adapting to samples with strong distribution shifts and limited multimodal information. Furthermore. utilize inter-modal information sharing to aligo the information between different modalities, facilitating the full adaptation of the model.
In summary, compared to all baselines, BiM-TTA achieves the best results in both cases for two datasets. To further support these results, we provide additional analyzes in the appendices: Ap- pendix A.7 reports ablation and component analyzes, Appendix A.8 presents visualization experi- ments that illustrate the key challenges addressed by our method, Appendix A.9 presents hyperpa- rameter studies, and Appendix A.10 evaluates computational efficiency.
5 Conclusion
In this paper, we propose a multimodal BiMamba network with TTA for emotion recognition, In the training phase, the multimodal BiMamba network effectively captures intra-modal dependencies and inter-modal correlations of multimodal physiological signals. The two-level entropy-based sample filtering and mutual information sharing across modalities achieve smooth adaptation to the target domain and reduce distribution shifts across modalities, thereby alleviating the negative impact of amplified distribution shifts caused by missing multimodal data. Experiment results demonstrate that our model achieves state-of-the- performance. Moreover, the ablation studies further confirm the contribution of each component within BiM-TTA. We also discuss the limitations of BiM-TTA in the Appendix A. In summary, we design a general multimodal backbone that incorporates a multimodal TTA mechanism. We will further extend BiM-TTA to broader physiological analysis tasks, including sleep stage classification and motor imagery.
Acknowledgments
This work is supported by the Youth Science Fund Project of National Natural Science Foundation of China (No.62306317), and sponsored by Beijing Nova Program (Grant No. 20250484804).
Contribution Statement
Tingyu Du, Zhengyu Tian, and Hongkai Li have equal contributions to this paper.
10

---

References
[1] Jian Shen, Xiaowei Zhang, Gang Wang, Zhijie Ding, and Bin Hu. An improved empirical mode decomposition of electroenccehaalgraam signals for depression detection. IEEE transactions affective computing, 13(1):262-271, 2019.
[2] Jun Deng, Xinzhou Xu, Zixing Zhang, Sascha Frühholz, and Bjöm Schuller. Semisupervised autoencoders for speech emotion recognition. JEEE/ACM Transactions Audio, Speech, and Language Processing, 26(1):31-43, 2017.
[3] Ziyu Jia, Yucheng Liu, Haichao Wang, and Tianzi Jiang. Cross- knowledge distillation for enhanced unimodal emotion recognition. IEEE Transactions on Affective Computing, 2025.
[4] Chenyu Liu, Xinliang Zhou, Yihao Wu, Yi Ding, Liming Zhai, Kun Wang. Ziyu Jia, and Yang Liu. A comprehensive survey on eeg-based emotion recognition: A graph-based perspective. arXiv preprini arXiv:2408. 06087, 2024.
[5] Chenyu Liu, Xinliang Zhou, Zhengri Zhu, Liming Zhai, Ziyu Jia, and Yang Liu. Vbh-gnn: Variational bayesian heterogeneous graph neural networks for cross-subject emotion recognition. In The Twelfth International Conference on Learning Representations, 2024.
[6] Chenyu Liu, Xinliang Zhou, Jiaping Xiao, Zhengri Zhu, Liming Zhai. Ziyu Jia, and Yang Liu. Vsgt: variational spatial and gaussian temporal graph models for eeg-based emotion recognition. In Proceedings of the Thirty-Thin International Joint Conference on Artificial Intelligence, pages 3078-3086, 2024.
[7] Xiaojun Ning, Jing Wang, Youfang Lin, Xiyang Cai, Haobin Chen, Haijun Gou, Xiaoli Li, and Ziyu Jia. Metaemotionnet: peatial-ppetral-temmrre based attention 3-d dense network with meta-learning for eeg emotion recognition. IEEE Transactions on Instrumentation and Measurement, 73:1-13, 2023.
[8] Cheng Cheng, Wenzhe Liu, Xinying Wang, Lin Feng, and Ziyu Jia. Disd-net: A dynamic interactive network with self-distillation for cross-subject multi-modal emotion recognition. IEEE Transactions on Multimedia, 2025.
[9] Ziyu Jia, Youfang Lin, Jing Wang, Zhiyang Feng, Xiangheng Xie, and Caijic Chen. Hetemo- tionnet: two-stream heterogencous graph recurrent neural network for multi-modal emotion recogmition. In Proceedings of the 29th ACM international conference on multimedia, pages 1047-1056, 2021.
[10] Yong Zhang, Cheng Cheng, and YiDie Zhang. Multimodal emotion recognition based manifold learning and convolution neural network. Multimedia Tools and Applications, 81(2)): 33253-33268. 2022.
[11] Jing Wang. Zhiyang Feng, Xiaojun Ning, Youfang Lin, Badong Chen, and Ziyu Jia. Two-stream dynamic heterogencous graph recurrent neural network for multi-label multi-modal emotion recognition. IEEE Transactions on Affective Computing, 2025.
[12] Juan Abdon Srranda-Corren. Mojtaba Khomami Abadi, Nicu Sebe, and Ioannis Patras. Amigos: A dataset for affect, personality and mood research on individuals and groups. IEEE transactions on affective computing, 12(2):479-493, 2018.
[13] Vernon J Lawhern, Amelia Solon, Nicholas R Waytowich, Stephen M Gordon, Chou Hung, and Brent Lance. Eegnet: a compact convolutional neural network for eeg-based brain compute interfaces. Journal of neural engineering, 15(5):056013, 2018.
[14] Yi Ding, Neethu Robinson, Su Zhang, Qiuhao Zeng, and Cuntai Guan. Tsception: Capturing temporal dynamics and spatial asymmetry from eeg for emotion recognition. IEEE Transactions on Affective Computing, 14(3):2238-222) 2022.
[15] Robin Tibor Schirrmeister, Jost Tobias Springenberg, Lukas Dominique Josef Fiederer, Martin Glasstetter, Katharina Eggensperger, Michael Tangermann, Frank Hutter, Wolfram Burgard, and Tonio Ball. Deep learning with convolutional neural networks for eeg decoding and visualization. Human brain mapping, 38( 38(11):5311-540, 2017.

---

[16] Siavash Sakhavi, Cuntai Guan, and Shuicheng Yan. Learning temporal information for brain- computer interface using convolutional neural networks. IEEE transactions on neural networks and learning systems, 29(1 1)55199-622, 2018.
[17] Punnawish Thuwajit, Phurin Rangpong, Phattarapong Sawangjai, Phairot Autthasan, Rat- tanaphon Chaisaen, Nannapas Banluesombatkul, Puttaranun Boonchit, Nattasate Tatsar- ingkansakul, Thapanun Sudhawiyangkul, and Theerawit Wilaiprasitporn. Eegwavenet: Multi- scale enn-based spatiotemporal feature extraction for eeg seizure detection. IEEE transactions on industrial informatics, 18(8):5547-5557 2021.
[18] Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel. Understanding the effective receptive field in deep convolutional neural networks. Advances in neural information processing systems, 29, 2016.
[19] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[20] Wenbing Li, Hang Zhou, Junqing Yu, Zikai Song, and Wei Yang. Coupled mamba: Enhanced multimodal fusion with coupled state space model. Advances in Neural Information Processing 2024.
Systems,
[21] Stephanie Balters and Martin Steinert. Capturing emotion reactivity through physiology mea- surement as a foundation for affective engineering in engineering design science and engineering practices. Journal of Intelligent Manufacturing, 28(7):1585-16077 2017.
[22] Jajack Heikenfeld, Andrew Jajack, Jim Rogers, Philipp Gutruf, Lei Tian, Tingrui Pan, Ruya Li, Michelle Khine, Jintae Kim, and Juanhong Wang. Wearable sensors: modalities, challenges, and prospects. Lab on Chip, 18(2):217-248, 2018
[23] Jiashuo Liu, Zheyan Shen, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. Towards out-of-distribution generalization: A survey. arXiv preprint arXiv:2 13624, 2021.
[24] Camilo Salazar, Edwin Montoya-Múnera, and Jose Aguilar. Analysis of different affective state multimodal recognition approaches with missing data-oriented to virtual learning environments. Heliyon, 7(6), 2021.
[25] Mouxing Yang, Yunfan Li, Changqing Zhang, Peng Hu, and Xi Peng. Test-time adaptation against multi-modal reliability bias. In The twelfth international conference on learning representations, 2024.
[26] Jixiang Lei and Franz Pernkopf. Two-level test-time adaptation in multimodal learning. In ICML 2024 Workshop on Foundation Models in the Wild, 2024.
[27] Jingxin Liu, Hongying Meng, Asoke Nandi, and Maozhen Li. Emotion detection from eeg recordings. In 2016 12th international conference on natural computation, fuzzy systems and knowledge discovery (ICNC-FSKD), pages 1722-1727. IEEE, 2016
[28] Omid Bazgir, Zeynab Mohammadi, and Seyed Amir Hassan Habibi. Emotion recognition with machine learning using eeg signals. In 2018 25th national and 3rd international iranian conference on biomedical engineering (ICBME), pages 1-5. IEEE, 2018
[29] Ziyu Jia, Youfang Lin, Jing Wang, Ronghao Zhou, Xiaojun Ning, Yuanlai He, and Yaoshuai Zhao. Graphsleepnet: Adaptive spatial-temporal graph convolutional networks for sleep stage classification. In Ijcai, volume 2021, pages 1324 1330, 2020.
[30] Ziyu Jia, Youfang Lin, Jing Wang. Xiaojun Ning, Yuanlai He, Ronghao Zhou, Yuhan Zhou, and Li-wei H I Lehman. Multi-view spatial-temporal graph convolutional networks with do- main generalization for sleep stage classification. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 19:1977-1986, 2021.
[31] Ziyu Jia, Youfang Lin, Xiyang Cai, Haobin Chen, Haijun Gou, and Jing Wang. Sst-emotionnet: ppatial-spectral--tempooo based attention 3d dense network for eeg emotion recognition. In Proceedings of the 28th ACM international conference multimedia, pages 2909-2917, 2020.
12

---

[32] Jiaxin Ma. Hao Tang, Wei-Long Zheng, and Bao-Liang Lu. Emotion recognition using mul- timodal residual Istm network. In Proceedings of the 27th ACM international conference on multimedia, pages 176-183, 2019.
[33] Murtadha D Hssayeni and Behnaz Ghoraani. Multi-modal physiological data fusion for affect estimation using deep learning. IEEE Access, 1221642-21133 2021.
[34] Sharath Koorathota, Zain Khan, Pawan Lapborisuth, and Paul Sajda. Multimodal neurophysio- logical transformer for emotion recognition. In 2022 44th annual international conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pages 3563-3567. IEEE, 2022.
[35] Jingxia Chen, Yang Liu, Wen Xue, Kailei Hu, and Wentao Lin. Multimodal eeg emotion recognition based on the attention recurrent graph convolutional network. information, 13(11): 550, 2022.
[36] Wuliang Huang, Yiqiang Chen, Xinlong Jiang, Teng Zhang, and Qian Chen. Gifusion: A channel-level correlation construction method for multimodal physiological signal fusion. ACM Transactions on Multimedia Computing, Communications and Applications, 20(2):1-23, 2023.
[37] Yiyu Gui, MingZhi Chen, Yuqi Su, Guibo Luo, and Yuchao Yang. Eegmamba: Bidirectional state space models with mixture of experts for eeg classification. arXiv e-prints, pages aiviv- 2407, 2024.
[38] Jean-Philippe Thiran, Ferran Marques, and Hervé Bourlard. Multimodal Signal Processing: Theory and applications for human-comee comannccmmpteer interaction. Academic Press, 2009.
[39] Zirun Guo and Tao Jin. Smoothing the shift: Towards stable test-time adaptation under complex multimodal noises. arXiv preprint arXiv:2503.02611 026611, 2025.
[40] Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan. Efficient test-time model adaptation without forgetting. In International confer- ence machine learning, pages 16888-16905. PMLR, 2022.
[41] Yoonho Lee, Annie S Chen, Fahim Tajwar, Ananya Kumar, Huaxiu Yao, Percy Liang, and Chelsea Finn. Surgical fine-tuning improves adaptation distribution shifts. arXiv preprint arXiv:2210.1 1466, 2022.
[42] Sander Koelstra, Christian Muhl, Mohammad Soleymani, Jong-Scok Lee, Ashkan Yazdani, Touradj Ebrahimi, Thierry Pun, Anton Nijholt, and Ioannis Patras. Deap: A database for emotion analysis; using physiological signals. IEEE transactions affective computing, 3(1):
18-31, 2011.
[43] Mohammad Soleymani, Jeroen Lichtenauer. Thierry Pun, and Maja Pantic. A multimodal database for affect recogmition and implicit tagging. IEEE transactions on affective computing. 3(1):42-55, 2011.
[44] Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Combining recurrent, convolutional, and contimuous-time models with linear state space layers. Advances neural information processing systems, 34:572-585, 2021.
[45] Albert Gu, Karan Goel, and Christopher Ré, Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.0033 2021.
[46] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. In First Conference on Language Modeling, 2024.
[47] Xiongxiao Xu, Yueqing Liang, Baixiang Huang, Zhiling Lan, and Kai Shu. Integrating mamba and transformer for long-short range time series forecasting. CoRR, 2024.
[48] Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexci Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In International conference on machine learning, pages 9229-9248. PMLR, 2020.
[49] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell, Tent: Fully test-time adaptation by entropy minimization. arXiv preprint arXiv:2006. 10726, 2020.
13

---

[50] Rajdeep Chatterjee and Tathagata Bandyopadhyay. Eeg based motor imagery classification using svm and mlp. In 2016 2nd International Conference on Computational Intelligence and Networks (CINE), pages 84 89. IEEE, 2016.
[51] Wei Tao, Chang Li, Rencheng Song, Juan Cheng, Yu Liu, Feng Wan, and Xun Chen. Eeg- based emotion recogmition via channel-wise attention and self attention. IEEE Transactions on Affective Computing, (4(11:3822-3)) -9338233--222 2020.
[52] Yi Ding, Neethu Robinson, Chengxuan Tong, Qiuhao Zeng, and Cuntai Guan. Lggnet: Learning from
representations for brain-computer interface. IEEE Transactions on Neural Networks and Learning Systems, 35(7):9773-9786, 2023.
[53] Yi Ding, Yong Li, Hao Sun, Rui Liu, Chengxuan Tong, Chenyu Liu, Xinliang Zhou, and Cuntai Guan. Eeg- deggddefrmmr: A dense convolutional transformer for brain-computer interfaces. IEEE Journal of Biomedical and Health Informatics, 2024.
[54] Otavio G Lins, Terence Picton, Patrick Berg, and Michael Scherg. Ocular artifacts in eeg and event-related potentials i: Scalp topography. Brain topography, 6(1):51-63, 1993.
[55] David A Robinson. Behavior of the saccadic system: Metries of timing and accuracy. Progress in brain research, 267(():329-355 2022.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the contributions of our model, and
these claims align with our empirical results presented in Section and 4 of the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations in Section A. We propose an emotion recognition method and present its methodological description and experimental validation in this paper. Our approach holds under specific assumptions explicitly stated the main text. A limitation of this method is that its performance remains uncertain when these assumptions are violated-a common challenge shared by all methodological frameworks. Furthermore, while the method's s applicability needs to be verified across diverse real-world datasets, such validation falls beyond the scope of this study. As these limitations are standard in research practice, we have omitted them from the main manuscript. Should the reviewers deem it necessary to include these clarifications in the main text, we would pleased to incorporate them accordingly.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
Our paper purely experimental and does not introduce new theoretical results or formal theorems. Therefore, there are no assumptions or proofs to state, making this question not applicable.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We fully describe the model architecture, hyperparameter settings, and experi- mental procedures in Sections 3. and Appendices A.3 A.5 so that others can reproduce our results. We release the source code and pretrained models to enable researchers replicate our experiments.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The code and instructions for our model are available via an anonymous link: pettts::/annonyovvve 4open acience////BM-MMTAA--B0
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
15

---

Justification: The datasets used in this study are all open-source, and the experimental settings are clearly described. In Section A Appendix [A.5] we provide comprehensive details on dataset splits, feature preprocessing. hyperparameter configurations, and training epochs to ensure that the experimental design is clear, understandable, and reproducible. Moreover, both the baseline models and the models proposed in this paper employ identical experimental setups, thereby ensuring fair comparison.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We reported the best results within the adjustable parameter range for each method.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Appendix A. notes that experiments were run on NVIDIA A4000 (16G) GPU and reports approximate training time.
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We have read the NeurIPS Code of Ethics and confirm that our research adheres to its guidelines. In particular, work avoids deceptive practices and respects user privacy and data rights.
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We construct a model for emotion recognition. As discussed in Section | Emotion recognition plays crucial role in the medical field, with important applications in mental health disorders, physiological health assessment, and clinical medicine. All our experiments produced no other negative social impacts.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We use open-source datasets and adhere general data privacy and security guidelines. Our work does not involve releasing high-risk models or datasets, so no specific usage restrictions safeguards needed beyond standard best practices.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All external code, open-source datasets, and models used in our work are properly cited and their licenses are respected. We cite the original sources for each dataset and library. See Appendix A.4
16

---

13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We open the code and model weights of our method for reproducible results |attps://anonymous. 4open. science/r/ Baaiceerr/BMMTTTA -TA---TA--- T-A--604-444
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: No crowdsourcing or human-subject studies were conducted in this research. 15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not involve any human-subject research.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: We did not use any large language models in developing our core methodology.
17

---

Appendix
A.1 Further Discussion of Related Work
A.1.1
Mamba
have emerged as promising class of architectures for sequence modeling. For example, Gu et al. 45 developed the Structured State Space Sequence Model (S4), which significantly improves computational efficiency by combining the HIPPO matrix with efficient computation methods, Building upon S4, Gu and Dao [46] proposed Mamba, which designs dynamic selection mechanism to filter out irrelevant information and extract global features more effectively. Mamba also introduces a bardware-award algorithm for efficient computation, showing great potential in long-sequenee modeling. Mamba has demonstrated excellent performance across various tasks. Following Mamba, Xu et al. [47] proposed SST, a model that integrates the strengths of Transformer and Mamba to capture both global and local sequence dependencies for general time-series prediction tasks. In the emotion recognition task using physiological signals, Gui al. [37 proposed HEG-Mamba, which utilizes BiMamba to better encode EEG physiological signals. It also introduces task-aware model equipped with general experts to learn task-specific representations from EEG data.
Structured state space sequence models (SSMs)
44
A.1.2 Test-Time Adaptation
Test-time adaptation (TTA) focuses on adapting source models to target domains without access to source domain data or target domain labels. Traditional domain adaptation requires training with both source and target domain data. In contrast, test-time training[48] enhances test-time adaptability by training source models with supervised and self-supervised objectives in the training stage. However, these methods rely on proxy tasks and assume the training process is accessible, which limits their application scope. TTA addresses this by adjusting models only during testing without intervening in the training stage. For instance, Wang et al. [49] proposed Tent, which updates model normalization layers by minimizing entropy during test. Niu et al. [40] introduced EATA, employing sample selection criterion based on entropy minimization. Recently, Yang et al. [25] developed READ, which addresses reliability bias by modulating the attention-baa based fusion layers in a self-adaptive manner and designing a novel objective function for robust multimodal adaptation. Lei and Pernkopf (66 proposed 2I,TA, using Shannon entropy as the objective for the Transformer encoder of the loss as objective for the modality fusion block.
corrupted modality and a
A.2 Selection of Entropy Threshold
Entropy Threshold Design We adopt a poft-thresholdng mechanism which the entropy threshold adaptively adjusted for each batch. The threshold is defined as the batch entropy expectation plus or minus times its variance. By Chebyshev's inequality, this formulation provides probabilistic bound on sample retention while mitigating the effect of outliers. Empirical results show that setting = 2 achieves optimal performance, with at least 80% of samples retained across both datasets, ensuring effective utilization of test data. To further improve flexibility, we introduce smoothing factor that gradually relaxes the threshold during inference until reaches the precomputed batch- aicch-ad value, allowing the model to incorporate more representative test samples.
Impact of Threshold Strictness Both overly strict and overly loose thresholds degrade perfor- mance, and thus an appropriate threshold needs to be determined empirically. When the threshold too strict, only small number of samples are retained, which restricts the model's ability to learn the target-domain distribution. contrast, when the threshold too loose, many low-quality samples are included, increasing the risk of adapting to low-quality data and compromising model stability.
18

---

A.3 Algorithm for Multimodal TTA
Algorithm Multimodal TTA
Input: Target samples X.
1: Initialize remain = X and S(x) = Ø.
2: while = 1 to iter and Xymmain * do
3: Calculate entropy of multimodal outputs and unimodal outputs for Xyemain using Eq 11 and
[2
4:
Calculate multimodal and unimodal entropy bounds using Eq and Eq
5:
Select S(x) from X based on entropy criteria using Eq
6:
Calculate mutual information sharing loss Lmis (z) for S (z) using Eq 17
7:
Calculate total loss Les for (x) using Eq 19
8:
Update tunable parameters of the model using Lisst-
9:
Update samples: X X (()).
10:
end while
11:
return Adapted prediction probabilities and loss values.
A.4 Description of Datasets
1) The DEAP dataset: involves 32 participants, each completing 40 one-minute trials based on music videos. In the experiment, EEG is recorded with BioSemi 32-channel cap following the international 10-20 system, while additional channels capture peripheral physiological signals including respiration rate, EOG, and EMG. After each video, participants provide self-assessments of arousal, valence, liking, dominance, and familiarity. During preprocessing, all signals are resampled 128 Hz. The EEG is bandpass filtered between 4 and 45 Hz, and ocular artifacts are removed. Arousal and valence are rated on a scale from to A threshold of 5 used to distinguish low and high emotional classes.
2) The MAHNOB-IHI dataset: It includes 30 subjects, each completing 20 video-based trials with self-reported ratings from I to 9. In the experiment, EEG signals are recorded using the BioSemi Active system equipped with 32 Ag/AgCI electrodes, while six peripheral physiological signals are simultancously recorded. These peripheral physiological signals include ECG, GSR, respiration, and body temperature. To maintain consistency across modalities, all recordings are kept at their original sampling rate of 256 Hz. A fixed threshold of 5 is applied to discretize the continuous arousal and valence ratings into binary categories. As the recordings of subjects 12, 15, and 26 are unavailable, the final analysis is conducted on 27 subjects, consistent with previous studies such as TSception and HetEmmttioNNei 9
A.5
The Other Hyperparameters Settings
Table 4: The values the hyperparameters described in the paper
Parameter
DEAP
MAHNOB-HCI
The auxiliary task training weights
[0.8, 0.3, 0.2]
[0.5, 0.05, 0.03]
The TTA parameter X
0.3
The unimodal entropy weights μ4
[1, 0.3, 0.2]
The total number of iterations iler The TTA parameter | The learning rate of TTA
7
0.2
0.0007
A.6 Detailed Introduction to the Baseline Methods
We compare our model approach with ten state-of-the-art emotion recognition models in the same domain, including:
19

---

SVM[50]] A traditional machine learning method that utilizes statistical features, wavelet- based energy-entrop RMS, and other techniques construct feature vectors for emotion recognition.
EEGNet A lightweight convolutional neural network designed for EEG-based brain brain-computer interface applications.
ACRNN(5I]: A model that combines channel-wist attention mechanism, convolutional neural network (CNN), a recurrent neural network (RNN), and an extended self-attention mechanism.
HetEmotionNet( [ A model that fuses multimodal physiological signals for emotion recognition using two-stream heterogeneous graph recurrent neural network, capturing sparial-spectral-eempooz features, heterogeneity, correlation, and dependencies.
TSception 14] A multi-scale convolutional neural network combining dynamic temporal layer, asymmetric spatial layer, and high-level fusion layer.
LGGNet[52] model that models local- local-global- global-grapp representations of EEG through multiscale temporal convolutions and local- and global-graph traup--iltering layers.
EEG-Deformer [53] A model that incorporates a Hierarchical Coarse-to-Fine Transformer (HCT) block and Dense Information Purification (DIP) module into a ONN-Transformer.
MambaFormer 47 A hybrid sequence model that embeds Mamba modules into Trans- former feed-forward layers, combining state-space efficiency with attention expressiveness for modeling both long- and short-range temporal dependencies.
S4T A hybrid model that couples Mamba-based global expert with a windowed Transformer local expert, fusing their outputs through a routing mechanism strengthen long-range temporal modeling.
VSGT A graph-based model for EEG emotion recognition that leverages a variational spatial encoder and a Gaussian temporal encoder to incorporate prior knowledge and model spatial and cross-temporal dependencies across brain regions.
We then compare our TTA method against the following baseline methods:
No Adapt: A baseline that evaluates a model pretrained on source domain data directly on target domain data without performing adaptation.
Tent[49] A method that adapts models at test time by minimizing prediction entropy through updating batch normalization parameters.
EATA :0): A method that filters reliable samples based on prediction entropy and assigns adaptive weights both samples and important parameters.
READ[25: A multimodal TTA method that mitigates reliability bias by employing new paradigm that modulates the attention between modalities aelf-adaetive way and adopting novel objective function for robust multimodal adaptation.
2LTTA (6) : A multimodal TTA method that addresses intra-modal distribution shifts and cross-modal reliability bias in multimodal learning by modulating normalization layers and self-attention modules and employing two-level objective function with entropy-base bbsed sample reweighting and piversity-promoting loss.
A.7 Ablation Studies and Component Analysis
To examine the overall contribution of major components, we first conduct ablation experiments on the multimodal BiMamba network and multimodal TTA. As shown in Figure 2 the results indicate that the removal of both the multimodal BiMamba network and multimodal TTA leads to a decrease in model performance, with the absence of multimodal TTA causing more significant performance drop. This demonstrates that multimodal TTA is more effective on datasets with missing multimodal data. Overall, both the multimodal BiMamba network and multimodal TTA are effective for emotion recognition in the proposed model.
Beyond this coarse-level analysis, we further design fine-grained experiments to evaluate the effec- tiveness of subcomponents: (1) comparative experiments for the two BiMamba modules without missing multimodal data, and (2) ablation studies of Multimodal TTA with missing multimodal data.
20

---

61.50%
63.50%
61.00%
63.00%
80.50%
62.50%
60.00%
62.00%
59.50%
61. 61.50%
59.00%
61.00%
Valence
Arousal
BiM-TTA(w/o TTA)
BiM-TTA(w/o Mamba)
BiM-TTA(ours)
Figure 2: Ablation study DEAP with missing data. "w/o Mamba" indicates the absence of the BiMamba in our network, and "w/o TTA" indicates the absence of the multimodal TTA. The results report the average performance across the four mask ratios.
LSTM
MLP
68.00%
Transformer
68.00%
Attention
66.00%
Mamba intra-modal BiMamba
66.00%
inter-modal BiMamba
64.00%
64.00%
62.00%
60.00%
60.00%
Arousal
Valence
Arousal
Valence
Figure 3: Comparison of intra-modal BiMamba and inter-modal BiMamba with other methods
For the intra-modal BiMamba module, compare its performance against Mamba, Transformer, and LSTM architectures to evaluate its intra-modes sequence modeling capabilities. For the inter-modal BiMamba module, we compare it with iitention-basee fusion and MLP fusion methods to assess its ability to integrate multimodal features. The corresponding experimental results are shown in Figure which demonstrate that both intra- and inter-modal BiMamba consistently outperform the alternatives, highlighting the superior modeling capacity of our BiMamba modules.
64.00%
BIM-TTA/w/O Information
Unimodal Entropy
BIM TTAw/o Sample Filtoring)
64.00%
Multimodal Entropy
BIM TTAour)
G2.00%
Two-level Entropy
60.00%
60.00%
58.00%
58.00%
56 56.00%
56.00%
Volence
Arousal
Valence
Arousal
Figure 4: Ablation study TTA and Two-level Entropy-based Sample Filtering
For the multimodal TTA, we perform ablation studies from two perspectives: the impact of removing individual components and the effectiveness of single-level entropy- filtering. The results are summarized in Figure demonstrating that each component has a beneficial impact on the overall performance.
A.8 Visualization Experiments
To better illustrate how BiM-TTA addresses the two key challenges of multimodal emotion recogni- tion, we provide two types of visualization experiments.
21

---

Signal
0.35
0.30
M M
0.05
0.00
delay
EDG Signal [colored activation
0.6
Reatie Importanze
0.1
-02-
EMG gnal Icolored by activation)
Amptzad
mamm
100
200
320
400
300
Time Duample Indes)
Figure 5: Grad-CAM visualization of BiM-TTA
First, Figure 5 shows Grad-CAM visualizations of BiM-TTA, which illustrate how the model captures intra-modal long-range dependencies as well as modal correlations. The EMG signal demonstrates that the model effectively captures long-range intra-modal features. Moreover, the following inter-modal correlations emerge: 1) when the EOG shows pronounced activation between time steps and 120, resulting high-frequency motion artifacts (ocular artifacts) contaminate EEG s54) and the model mitigates this interference by reducing the attention weights assigned to those EEG segments. 2) activation in the EEG between time steps 260 and 300 provokes a delayed eye-movement response in the EOG, which appears between time steps 290 and 350[55). These dynamics align closely with the physiological patterns documented emotion-recooniiton research.
Train Test Defore TTA Test After TTA
Train Before TTA
Test Before TTA
Test Before TTA
Figure 6: t-SNE visualization of TTA
22

---

Second. Figure 6 presents t-SNE visualization of feature distributions, clearly demonstrating that BiM-TTA can mitigate the distribution shifts induced by missing multimodal data. The green points represent the feature distribution extracted by the model from the original test set before applying TTA, while the blue points correspond to the training distribution. After applying TTA, the test distribution progressively aligns with the training distribution, eventually appearing as the red points. This progressive alignment demonstrates that TTA is effective in mitigating the distribution gap between the test and training sets, thereby enabling the model to perform emotion recognition more effectively under testing conditions.
A.9 Hyperparameter Studies
To further analyze the effect of hyperparameter settings, we examine five key hyperparameters: the loss weight of unimodal auxiliary tasks og, the loss-balance coefficient X, the weight of unimodal entropy p4, the total number of iterations iter, and the initial value of the smoothing factor B.
Table 5: Studies on the Auxiliary task weights of multimodal BiMamba on DEAP with missing data. The weights correspond the parameters for EEG, EOG, and EMG. The results report the average performance across the four mask ratios.
Valence
Arousal
0.7
0.608
0.627
EEG
0.8
0.610
0.631
0.9
0.609
0.624
0.2
0.609
0.626
EOG
0.3
0.610
0,631
0.4
0.607
0.625
0.1
0.604
0.623
EMG
0.2
0.610
0.631
0.3
0.605
0.620
The results for the auxiliary-task weight ag are reported in Table Based on experimental observa- tions, the EEG branch plays more critical role than EOG and EMG in classification performance. Therefore, its auxiliary task is assigned larger weight.
The effects of the remaining four hyperparameters (), Mús iter, and (3) are illustrated in Figure 7 For À and μ4, the model remains stable across wide range of values, indicating that performance is not sensitive to fine-tuning these coefficients. For iler, performance peaks within a moderate range, where the model adapts effectively without overfitting. Too few iterations lead to insufficient adaptation, while too many cause mild over-adaptation and slightly degrade performance. For B, relatively small values achieve the best balance between reliability and adaptability. smaller 3 enforces a stricter initial threshold, retaining only samples with rich multimodal information and high prediction confidence. In contrast, a larger 3 relaxes the threshold too early, potentially admitting low-quality samples and disrupting learned representations, which degrades performance. Overall, although different hyperparameter choices lead slight variations performance, the model consistently outperforms the No Adapt setting, indicating that the proposed method remains robust across a broad range of hyperparameter configurations.
To gain deeper insight into the adaptation dynamics, we further analyze the performance trend of TTA during the iterative process, as shown in Figure With the iteration parameter iler to 7, the model performance progressively improves over iterations and reaches its optimal average level at the final step.
A.10 Computational Efficiency Analysis
We compare the computational cost required to perform complete inference for single subject on the DEAP dataset. To ensure faimess and accuracy, we use the official open-source implementations of the latest best-performing baseline models for comparison. As shown in Table our model lightweight, maintains real-time performance, and achieves an optimal trade-off between performance and efficiency. All experiments are conducted on an NVIDIA RTX A4000 (16G). A single training run (30 epochs) takes approximately 60 minutes on a single GPU.
23

---

The value t
2000%
1 1000%
0000%
500%
The
The value of the
2.00%
1.50%
1.50%
0.50%
0.50%
0.00%
0.503
0.50%
+1.0%%
-1.00%
0.0
0.2 0.3 0.4
0.6 0.7 0.8 1.0
0. 0.1
05 0.6 DJ
The value of the iter
The value of the
3.30%
1.50%
0.50%
0.50%
0.0%%
0..0%
0.50%
-1.00%
-1.00%
0.0
05
10
Valence
Arousal
-+-Avg
Figure 7: Hyperparameter studies of BiM-TTA DEAP with missing data. The parameters include X. P2, P3, iler, and B3, with set to by default. The results report the average improvement across the four mask ratios.
1.20%
Arousal
0.70%
Valence
Avg
0.20%
Figure 8: Model Performance across TTA Iterations
Table 6: Comparison of computational overhead parameters with two recent state-of-the-an models
Model
Parameter Count Inference Time (s)
FLOPs (M)
GPU Memory (MB)
LGGNet
721,319
0.73
10.75
95.40
BEG-Deformer
915,394
0.43
5.39
8.21
BiM-TTA(ours)
22,431
0.47
1.18
2.29
A.11 Limitations
would like to discuss some of the limitations identified during this study. Firstly, although BiM-TTA demonstrates outstanding performance in this study, its latency and resource consumption in real-time online systems (c.g.. wearable devices) have not yet been evaluated. In future work, we plan to apply pruming and quantization, perform distillation, design lightweight variants to meet the demands of real-time embedded scenarios. Secondly, although we have proposed a general model architecture, its applicability in multi-task settings remains insufficiently validated. Going forward, we will assess the effectiveness of BiM-TTA on tasks such as sleep stage classification and motor imagery.
24


Reviews:
[{"review_id": "ZA1ilxP8Ka", "reviewer": "Reviewer_krBP", "summary": "In this paper, the authors propose a multimodal bidirectional Mamba (BiMamba) network with test-time adaptation (TTA) for emotion recognition, named BiM-TTA. Specifically, BiM-TTA consists of a multimodal BiMamba network and a multimodal TTA module. The multimodal BiMamba network includes intra-modal and inter-modal BiMamba modules, which model long-range dependencies along the time dimension and capture cross-modal correlations along the channel dimension, respectively. The TTA module effectively mitigates the negative impact of distribution shifts amplified by missing multimodal data through two-level entropy-based sample filtering and mutual information sharing across modalities.", "questions": "We believe it is of great need to give the thoughts of the BiMamba design", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The paper is well written and easy to read. And the effectiveness of the proposed method has been validated by the experiments. \n\nThe BiMamba seems the one of the main contributions of this work. However, we only see the description of the data processing flow in Bi Mamba instead of the intuition behind the module. Thus, we believe it is of great need to give the thoughts of the BiMamba design.", "was_revised": true, "final_justification": "We have recieved the feedback from the authors. After considering the rebuttal and discussions with authors, other reviewers and AC, we agree to upgrade the rating score from 3 to 4.", "rebuttal": "# Response\n## W1&Q1:\n\n　　Thank you for your suggestions on the methodology section of our paper. To clarify the design thoughts of the BiMamba module, we have added the corresponding explanation in \"3.2.1 Intra-modal BiMamba Module\" and \"3.2.2 Inter-modal BiMamba Module\" of the main text. The module's workflow comprises the following three steps: **(1) Gating mechanism $\\to$ (2) Bidirectional state-space modelling $\\to$ (3) Linear projection and residual connection.** Specifically, the intuition behind them is:\n\n\n1.  **Gating mechanism: This mechanism assigns learnable weights to each channel or time step, adaptively emphasizing emotion-relevant representations while suppressing irrelevant information.**\n\n    >In particular, physiological signals not only contain key features such as abrupt emotional change points and dynamic regions that faithfully reflect emotional fluctuations, but also often include environmental noise and repetitive resting state potentials that constitute redundant, task-irrelevant information [1][2]. The key features provide discriminative cues for emotion recognition, while the redundant information may obscure or interfere with the model's perception of sudden emotional changes. Therefore, the gating mechanism introduces learnable weight coefficients during feature extraction to dynamically adjust the response strength of each channel and time step, allocating higher weights at emotional change points to amplify discriminative signals and allocating lower weights in resting state or noisy segments to suppress interference. In this way, the model focuses on the most emotionally informative dynamic features, thereby improving recognition accuracy.\n\n2.  **Bidirectional state-space modelling: Forward and backward features are constructed in parallel to capture richer contextual relationships.**\n\n    >I. **Bidirectional State-Space Modeling Along the Temporal Dimension:** Emotion recognition is inherently a sequential modeling task, since emotional states emerge from temporally dependent dynamics rather than isolated points[3]. Unidirectional modeling often struggles to capture strong temporal correlations, such as lagged or precursory signals, leading to overlooked emotional cues[4]. To address this, BiMamba first applies linear mapping, convolution, and activation for initial local feature extraction, then deploys forward and backward state-space model SSM branches in parallel along the time dimension. This bidirectional modeling perceives complete contextual dependencies from both temporal paths, enhancing the model's understanding and discrimination of cross-time emotional dynamics.\n\n    >II. **Bidirectional Cross-Modal Interaction Along the Channel Dimension:** Emotional states commonly exhibit correlations across multiple physiological modalities, which are reflected at the channel level. For instance, increases in EEG $\\theta$- and $\\delta$-band power often exhibit positive Pearson correlations with galvanic skin response (GSR) conductance peaks and negative correlations with electrocardiogram (ECG) heart-rate variability during emotional arousal[5]. In the forward branch, the representation of the $i$-th modality references the hidden states of preceding modalities from 1 through $i-1$. In the backward branch, it is supplemented by the hidden states of subsequent modalities from $M$ down to $i+1$, where $M$ is the total number of modalities. This bidirectional information flow enables each modality to leverage context from earlier modalities and complete its features using later ones, generating complementary expressions for the same modality and strengthening inter-modal associations and fusion.\n\n3.  **Linear projection and residual connections: Integrate bidirectional features and alleviate the vanishing gradient problem.**\n\n    >We first fuse the forward feature $h_i^\\rightarrow$ and backward feature $h_i^\\leftarrow$ by averaging, then apply a linear mapping to merge features and perform dimensionality transformation. Finally, a residual connection is introduced to alleviate gradient vanishing.\n\n*Manuscript revision:* We have incorporated the above design thoughts into \"3.2.1 Intra-modal BiMamba Module\" and \"3.2.2 Inter-modal BiMamba Module\" to aid reader comprehension.\n\n---\n---\n\n# References\n[1] Barradas, I. et al. *Dynamic emotion intensity estimation from physiological signals*. PLOS ONE, 2025.\n\n[2] Brake, N., Duc, F., Rokos, A., Arseneau, F., Shahiri, S., Khadra, A., & Plourde, G., *A neurophysiological basis for aperiodic EEG and the background spectral trend*, Nature Communications, 2024.\n\n[3] Hofmann, S.M., Klotzsche, F., Mariola, A., Nikulin, V., Villringer, A., & Gaebler, M.. *Decoding subjective emotional arousal from EEG during an immersive virtual reality experience*. eLife, 2021.\n\n[4] Zhang X, Zhang Q, Liu H, et al. *Mamba in speech: Towards an alternative to self-attention*. IEEE Transactions on Audio, Speech and Language Processing, 2025.\n\n[5] Miranda-Correa JA, Abadi MK, Sebe N, Patras I. *AMIGOS: A dataset for affect, personality and mood research on individuals and groups*.  IEEE Transactions on Affective Computing, 2018."}, {"review_id": "0zZ3Vq015B", "reviewer": "Reviewer_TeXe", "summary": "This paper proposes BiM-TTA, a multimodal emotion recognition method based on physiological signals that integrates a bidirectional Mamba (BiMamba) network with a dedicated test-time adaptation (TTA) strategy. The BiMamba network consists of two modules: (1) an intra-modal BiMamba that models long-range temporal dependencies within each physiological modality and (2) an inter-modal Bimodal Bi-Mamba which captures cross-modal correlations by operating along the channel dimension. To address the challenge of missing or corrupted multimodal data at test time, the authors introduce a two-level entropy-based sample filtering mechanism combined with mutual information sharing across modalities during TTA. This adaptation selectively fine-tunes a small subset of model parameters on target domain data, mitigating distribution shifts caused by missing signals.\n\nThe method is evaluated on two multimodal emotion corpora (DEAP and MAHNOB-HCI). It is tested under both complete and missing data scenarios, performing better than several strong baselines and recent TTA methods.", "questions": "Please clarify why no comparison was made with recent hybrid models.", "limitations": "The authors briefly mention limitations in Appendix A.13, acknowledging that BiM-TTA has not yet been validated on tasks beyond emotion recognition, such as sleep stage classification or motor imagery, and that future work is planned in this direction.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1) The paper addresses two substantial and well-motivated challenges in multimodal emotion recognition from physiological signals: (1) modeling long-range intra-modal dependencies and inter-modal correlations, and (2) handling distribution shifts and missing data at test time.\n2) The proposed BiMamba architecture is a technically sound and well-justified application of structured state space models (SSMs) in a multimodal context.\n3) The test-time adaptation (TTA) mechanism is designed, combining two-level entropy-based sample filtering and mutual information sharing across modalities.\n4) The empirical evaluation is thorough, with comparisons against strong baselines on two public corpora  and both complete and missing data conditions.\n5) The ablation studies, iterative TTA analysis, Grad-CAM, and t-SNE visualizations convincingly validate the technical components and adaptation process.\n\nWeaknesses:\n1) While the TTA mechanism has been engineered, its conceptual novelty is moderate, as there are similar entropy-based selective adaptation strategies. The proposed two-level filtering and gradual threshold relaxation are incremental, rather than fundamentally new.\n2) A lack of comparison with Mamba-Transformer hybrid baselines or recent Transformer-based SSM models creates a gap in benchmarking against the closest architectural alternatives.", "was_revised": false, "final_justification": "", "rebuttal": "# Response\n\n## W1\n\n　　Thank you very much for your attention to and detailed review of our conceptual novelty.\nIn response to your questions, we are here to further clarify and supplement two aspects to more clearly demonstrate our innovations in the two-level entropy-based sample filtering strategy and its synergy with the overall architecture:\n\n1.  **Our work represents an extension and expansion of TTA strategy within the field.**\n\n    >Although entropy-based selective adaptation strategies have been extensively validated in computer vision and related domains, the specific downstream task of “multimodal physiological signals + emotion recognition + data missing” remains largely unexplored. Compared to existing EEG‐based TTA methods such as ATTA[1] and OTTA[2], our method uses entropy in a more distinctive and targeted manner. ATTA updates the adaptive model via unimodal cross-entropy, and OTTA minimizes only classification entropy. In contrast, we introduce both unimodal entropy and multimodal entropy, where the former captures the richness of emotional information present in each sample’s multimodal physiological signals and the latter gauges the model’s confidence in predicting the emotional state. Based on these, we construct a soft-threshold filtering strategy that dynamically removes samples whose emotional features degrade due to missing key information, more finely distinguishing sample quality across emotions. This ensures stability during adaptation while more effectively improving cross‐domain emotion recognition accuracy.\n\n2.  **The filtering mechanism does not function in isolation. Instead, it serves as an integral part of the overall architecture,working in concert with other modules to jointly enhance the model's adaptive capability during the testing phase.**\n\n    >I. **Synergistic Effect between Two-level Entropy-based Sample Filtering and Mutual Information Sharing Across Modalities.** Our multimodal TTA method comprises two complementary components: a global two-level entropy-based sample filtering strategy and a local cross modalities strategy.\n    >- The global strategy uses a soft threshold jointly constructed from unimodal and multimodal entropy to select a subset of information‐rich, high‐confidence samples, thereby ensuring data quality during adaptation.\n    >- The local strategy, based on these reliable samples, further promotes cross‐modal alignment by minimizing the KL divergence among prediction distributions of each modality.    \n    Together, they achieve progressive optimization from global sample filtering to local modal alignment.\n\n    >II. **Synergistic Effect between the Multimodal BiMamba Backbone and TTA Method with Two-level Entropy-based Sample Filtering and Mutual Information Sharing Across Modalities.**\n    >\n    >**The multimodal BiMamba backbone network** first captures emotional features along temporal and channel dimensions, mining long-range temporal dependencies and cross-modal correlations to generate strong base representations.Building on these high-quality features, **the TTA method** applies the two-level entropy-based sample filtering and mutual information sharing across modalities: the former ensures adaptation only on reliable samples, and the latter dynamically adjusts inter-modal information flow within each sample, mitigating interference from imbalanced modality information. The network and TTA collaborate across feature construction and test‐time adaptation to enhance emotion recognition performance.\n\n---\n\n## W2 & Q1\n\n　　As no existing work has explored Mamba-Transformer hybrids in physiological-signal emotion recognition, this direction remains unexplored and was not included in our original comparisons. Following your suggestion, we have added three recent SSM-Transformer approaches for time-series modelling as supplementary baselines, including MambaFormer and SST as Mamba-Transformer hybrid models and Heracles as a Transformer-based SSM method. And we have incorporated the results and analysis of these baselines into \"4.3 Experiment Analysis.\" Thank you for your valuable suggestion.\n\n\n\n*Table: Comparative analysis of accuracy (Valence, Arousal) for different emotion baselines on DEAP and MAHNOB-HCI datasets. * indicates p<0.05.*\n\n| Method             | DEAP          |              | MAHNOB-HCI     |               |\n|--------------------|---------------|--------------|----------------|----------------|\n|                    | Valence       | Arousal      | Valence        | Arousal        |\n| MambaFormer [3]    | 0.621         | 0.587*       | 0.588*         | 0.619          |\n| SST [3]            | 0.613*        | 0.623        | 0.606*         | 0.616*         |\n| Heracles [4]       | 0.633         | 0.611*       | 0.621          | 0.603*         |\n| **BiM‑TTA (ours)** | **0.673**     | **0.641**    | **0.650**      | **0.635**      |\n\n\n\n**Experimental analysis:**\n MambaFormer embeds the Mamba module into each Transformer encoder’s feed‑forward layer to create a unified long‑and‑short‑range architecture. SST introduces a Mamba‑based global pattern expert alongside a windowed Transformer local variation expert and dynamically fuses their outputs via a long‑short‑range router. Heracles leverages a real‑valued Hartley‑kernel SSM to capture global sequence dependencies, a localized convolutional network to extract fine‑grained details and attention mechanisms in its deeper layers to enable rich temporal token interactions. These models exhibit strong long‑short‑range modeling capabilities. However, their designs focus solely on temporal dynamics and do not explicitly model cross‑channel interactions, which prevents them from capturing inter‑modal relationships and limits their effectiveness in multimodal emotion recognition tasks. Moreover, the complexity of these hybrid architectures can introduce a greater risk of overfitting on relatively small physiological‑signal datasets, further affecting their performance.\n\n*Manuscript revision:* We have updated the results in Table 1 and revised Section 4.3 \"Experiment Analysis\" accordingly.\n\n---\n---\n\n# References\n\n[1] Jia Z, Yang X, Zhou C, et al.  *Atta: Adaptive Test-Time Adaptation for Multi-Modal Sleep Stage Classification*.  Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), 2024.\n\n[2] Guo H, Li C, Peng H, Han Z, Chen X.  *Test Time Adaptation for Cross-Domain Sleep Stage Classification*.  Biomedical Signal Processing and Control, 2025.\n\n[3] Xu X, Liang Y, Huang B, Lan Z, Shu K.  *Integrating Mamba and Transformer for Long-Short Range Time Series Forecasting*.  CoRR, 2024. \n\n[4] Badri N. Patro, Suhas Ranganath, Vinay P. Namboodiri, Vijay S. Agneeswaran. *Heracles: A Hybrid SSM-Transformer Model for High-Resolution Image and Time-Series Analysis*. arXiv, 2024."}, {"review_id": "PMpRBRSvUq", "reviewer": "Reviewer_RRS6", "summary": "The paper presents BiM-TTA for emotion recognition. The contribution is to handle temporal dependencies, modal correlations, and missing data problems.", "questions": "1)Why is there no consideration of the case where a modality is completely missing? \n2)The contribution and importance of different modality do not seem to be explicitly considered? \n3)How is the Entropy threshold selected? \n4)What is the impact of different initial values or smoothing factors on the model?\n5)This paper emphasizes that Mamba is an architecture that is good at solving long-distance dependencies, so why is Mamba still used when learning the relationship between modalities? The number of modalities in this paper is not large.", "limitations": "YES", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "# Strengths: \n- The motivation is clear and reasonable. \n- The entropy-based TTA is novel and mitigates missing data issues. \n- Performance on two datasets is good. Also this paper provides a lot of extra experiments.\n# Weaknesses: \n- The structure of the paper is not appropriate. A large number of experiment are placed in the appendix, and although many of them provide important impacts, these are not mentioned and explained in the main text.\n- There is insufficient explanation of some details, such as the basis for the selection of some hyperparameters. The main text does not summarize some the hyperparameter experiments, making it difficult to establish a connection between the supplementary experiments in the appendix and the arguments in the main text.", "was_revised": true, "final_justification": "The authors have addressed most of my concerns. The proposed method shows novelty, and the performance on two datasets is promising. Therefore, I have given a positive score.", "rebuttal": "# Response\n\n## W1\n\n　　Thank you for your valuable suggestions. We have optimized the manuscript structure as follows:\n\n1.  **We have relocated several key experiments from the appendix to the main experimental section.** \n\n    >Considering that certain key experiments, such as ablation studies and hyperparameter analyses, are crucial for validating the model's effectiveness, we have moved these specific experiments from the appendix to the main experimental section. We have also supplemented them with more detailed design descriptions and result analyses to improve the completeness and readability of the main text.\n\n2.  **We have added an \"Experimental Overview\" subsection.**\n\n    >To improve the overall readability of the main text, we add a new subsection titled \"Experimental Overview\" to briefly summarize the design and key findings of each experiment. At the same time, we provide clear guidance directing readers to the more detailed figures and analyses in the appendix, enabling a more efficient understanding of the core contributions.\n\n*Manuscript revision:* We have added subsections \"4.4 Experimental Overview\", \"4.5 Ablation Experiments\", and \"4.6 Hyperparameters Effection\" in the revised manuscript.\n\n---\n\n## W2\n\n　　Thank you for your valuable feedback regarding the basis for the selection of some hyperparameters and the summary of experiments. We have added relevant content in the main text, as outlined below:\n\n1.  **Explanation of hyperparameter selection.**\n\n    >In \"4.6 Hyperparameter Effects\" of the main text, We have supplemented the basis for choosing key hyperparameters, such as the entropy threshold and smoothing factor, and analyzed their performance curves under varying values. The basis for selecting the entropy threshold and the smoothing factor can be found in our responses to Q3 and Q4.\n\n2.  **Summary and connection of hyperparameter experiments.**\n\n    >We have added a new subsection \"4.4 Experimental Overview,\" which briefly summarizes the impact of each hyperparameter on model performance and directs readers to the corresponding experimental details in the manuscript.\n\n*Manuscript revision:* We have added subsections \"4.4 Experimental Overview\", \"4.5 Ablation Experiments\", and \"4.6 Hyperparameters Effection\" in the revised manuscript.\n\n---\n\n## Q1\n\n**The case where a modality is completely missing rarely occurs in physiological signal acquisition experiments.**\n>In actual physiological signal collection, complete modality missing is usually detected in time and fixed by inspecting or replacing the device. Therefore, our study mainly focuses on the more common problem of partial modality data missing.\n\n---\n\n## Q2\n\n**We regulate the contribution of different modalities in the model by adjusting the loss weight of the auxiliary task and the hyperparameter of Mutual Information Sharing Across Modalities.** \n>These parameters reflect the relative importance and influence of each modality. To further enhance interpretability, we plan to add a weight-visualization analysis for the inter-modal BiMamba in a future version, which will intuitively illustrate each modality's specific contribution.\n\n---\n\n## Q3\n\n**The entropy threshold is determined jointly by batch-level statistics and hyperparameter tuning.** \n>Inspired by Chebyshev's inequality, we set the threshold to the batch's mean entropy plus $n$ times its variance, where $n$ is a hyperparameter controlling the strictness of sample filtering. This design aims to retain high‑quality samples while suppressing interference from outlier samples. The detailed methodology for selecting the entropy threshold is provided in \"A.3 Selection of Entropy Threshold\", and we will emphasize this discussion more prominently in the main text to clarify its role within the overall method.\n\n---\n\n## Q4\n\n**The initial value $\\beta$ and the smoothing factor $\\beta_t$ influence both the number and quality of samples entering the adaptation process at each iteration.** The smoothing factor $\\beta_t$ is jointly determined by the initial value $\\beta$ and the growth strategy, with the specific effects described as follows:\n\n>I.  **Initial $\\beta$:** A smaller $\\beta$ yields a stricter initial threshold, retaining only samples with rich multimodal information and high prediction confidence, thereby enhancing stability at early testing phase. A larger $\\beta$ relaxes the threshold, potentially admitting low‐quality samples too soon and disrupting learned representations, which may degrade performance.\n>   \nThe optimal value of $\\beta$ is determined experimentally and set to 0.2, with relevant results shown at \"The value of the $\\beta$\" in the lower right corner of Figure 2.\n\n>II.  **Growth strategy:** After determining a suitable $\\beta$, we adopt the following growth strategy:\n    $$\\beta_t = \\beta + (1 - \\beta)\\cdot\\frac{t}{\\mathrm{iter}}$$\n    where $\\beta_t$ increases linearly with iterations, gradually relaxing the entropy threshold. This process allows more samples to be sequentially incorporated into the adaptation process and enables the model to more effectively cover the target domain.\n\nIn summary, by setting an appropriate initial $\\beta$ and growth rate, this strategy balances early robustness with later adaptability, facilitating a smooth, high-quality model adaptation.\n\n---\n\n## Q5\n**In our paper, we emphasize both Mamba’s strength in modeling long‑range dependencies and its ability to capture inter‑modal correlations.**\n>Specifically, besides effectively capturing long-range temporal dependencies, its explicit global state variables, as part of the state space model (SSM), also effectively characterize correlations between different modalities. Based on these considerations, we ultimately select Mamba as the core component of the backbone network. As briefly mentioned in line 82 of the introduction, this topic is initially discussed. In subsequent versions, we will further elaborate to aid readers' deeper understanding."}, {"review_id": "BTwdq1VW4i", "reviewer": "Reviewer_Z1w5", "summary": "This paper proposes a multimodal network BiMamba with test time adaptation TTA for emotion recognition. It addresses long-range dependence and inter-modal correlation in multimodal physiological signals, and it mitigates performance degradation of missing data.", "questions": "1. For the first questions, please refer to Weaknesses 2.\n2. How to define the number of iterations? According to the experimental results in Figure 6 in the appendix, it can still be improved.\n3. What is the basis for setting the two thresholds? What is the impact of different thresholds on the model results?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. Introducing TTA to sold missing data problem is a good and innovative method. The reason of using mamba is clear and reasonable.\n2. This paper is well written with a clear structure, such as the correspondence between formulas and figures. The experiment section and appendix provide solid details.\n3. The problem solved by this paper is meaningful and not often mentioned. Therefore, the motivation is reasonable.\n\nWeaknesses:\n1. This paper puts most of the experiment in the appendix, but the guidance is not clear enough.\n2. The explanation of two-level entropy filtering is not clear enough. This paper mentions that these two filters seem to use different filter criteria and seem to have opposite threshold selections (one tends to be high and the other tends to be low). The large amount of text in this part makes it more difficult to sort out the relationship between them.\n3. Figure 1 needs to be improved. The author seems to want to use color matching to illustrate the correlation between the components. But there are too many colors in the figure, and the colors of components and the background are relatively close and difficult to identify.", "was_revised": false, "final_justification": "", "rebuttal": "# Response\n\n## W1\n\n　　Thank you for your valuable suggestions. We have optimised the manuscript structure as follows:\n\n1.  **We have relocated several key experiments from the appendix to the main experimental section.** \n>Considering that certain key experiments, such as ablation studies and hyperparameter analyses, are crucial for validating the model's effectiveness, we have moved these specific experiments from the appendix to the main experimental section. We have also supplemented them with more detailed design descriptions and result analyses to improve the completeness and readability of the main text.\n\n2.  **We have added an \"Experimental Overview\" subsection.** \n>To improve the overall readability of the main text, we add a new subsection titled \"Experimental Overview\" to briefly summarize the design and key findings of each experiment. At the same time, we provide clear guidance directing readers to the more detailed figures and analyses in the appendix, enabling a more efficient understanding of the core contributions.\n\n*Manuscript revision:* We have added subsections \"4.4 Experimental Overview\", \"4.5 Ablation Experiments\", and \"4.6 Hyperparameter Effects\" in the revised manuscript.\n\n---\n\n## W2 & Q1\n\n**We have streamlined the description of two-level entropy-based sample filtering strategy and added a schematic diagram to aid intuitive understanding.** This mechanism is designed to prioritize the selection of **samples with high unimodal entropy and low multimodal entropy** for fine-tuning. Specifically, it includes the following two aspects:\n\n>I.  **Unimodal entropy $Ent(x_i)$ measures a sample's reliance on a single modality. Higher values indicate suitability for fine‐tuning.**\n>- *Low $Ent(x_i)$:* Model can predict using one dominant modality, implying limited multimodal information.\n>- *High $Ent(x_i)$:* Model likely integrates multiple modalities for prediction, indicating rich multimodal information.\n\n>II.  **Multimodal entropy $Ent(x)$ reflects prediction certainty. Lower values indicate suitability for fine‐tuning.**\n>- *Low $Ent(x)$:* Predictions are more certain. This helps reduce uncertainty in target domain predictions.\n>- *High $Ent(x)$:* Predictions are uncertain. The uncertainty in predictions may introduce misleading information and compromise model stability, therefore making such samples unsuitable for fine-tuning.\n\n*Manuscript revision:* We have updated \"3.3.1 Two-level Entropy-based Sample Filtering\" and added the schematic diagram.\n\n---\n\n## W3\n\n　　Thank you for the detailed feedback on Figure 1's colour scheme. We have simplified the colour palette and increased foreground-background contrast to more clearly delineate components and overall structure.\n\n---\n\n## Q2\n\n**We determine the number of iterations $iter$ experimentally.** \n>In the experiment labeled \"The value of the iter,\" shown in the lower-left panel of Figure 2, we systematically evaluate performance under various settings and ultimately set $iter=7$. Using this selected value, Figure 6 then illustrates how model performance evolves from iteration 1 through 7, showing steady improvement and peak performance at iteration 7.\n\n---\n\n## Q3\n\n　　Thank you for your question. Below is our response:\n\n1.  **The entropy threshold is determined jointly by batch‐level statistics and hyperparameter tuning.** \n>Inspired by Chebyshev's inequality, we set the threshold to the batch's mean entropy plus $n$ times its variance, where $n$ is a hyperparameter controlling the strictness of sample filtering. This design aims to retain high‑quality samples while suppressing interference from outlier samples. The detailed methodology for selecting the entropy threshold is provided in \"A.3 Selection of Entropy Threshold\", and we will emphasize this discussion more prominently in the main text to clarify its role within the overall method.\n\n2.  **Both overly strict and overly loose thresholds degrade performance, and thus an appropriate threshold needs to be determined through empirical evaluation.**\n>-  *Too strict:* Too few samples pass the filter, limiting learning of the target‐domain distribution.\n>-  *Too loose:* A large number of samples are accepted, including outliers, which increases the risk of adapting to noisy data."}]

Author rebuttal:

Meta-review: This work addresses two challenges in multimodal emotion recognition from physiological signals: (1) modeling long-range intra-modal dependencies and inter-modal correlations, and (2) handling distribution shifts and missing data at test time. The paper presents a new model (a multimodal bidirectional Mamba -BiMamba- network with test-time adaptation -TTA-) that achieves state-of-the-art performance on the DEAP and MAHNOB-HCI datasets.

All the reviewers are positive about the paper. The reviewers value both the innovative use of BiMamba and TTA to address the two challenges of emotion recognition from physiological signals studied in the paper, and the detailed experimental validation (including ablation studies, iterative TTA analysis, Grad-CAM, and t-SNE visualizations). Based on these considerations we recommend to accept the paper.
Decision: Accept (poster)