Coupled Mamba: Enhanced Multimodal Fusion with Coupled State Space Model

Authors: wenbing  li
hang  zhou
junqing  yu
zikai  song
wei  yang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-07 07:00:41

Abstract: The essence of multi-modal fusion lies in exploiting the complementary information inherent in diverse modalities.However, most prevalent fusion methods rely on traditional neural architectures and are inadequately equipped to capture the dynamics of interactions across modalities, particularly in presence of complex intra- and inter-modality correlations.Recent advancements in State Space Models (SSMs), notably exemplified by the Mamba model, have emerged as promising contenders. Particularly, its state evolving process implies stronger modality fusion paradigm, making multi-modal fusion on SSMs an appealing direction. However, fusing multiple modalities is challenging for SSMs due to its hardware-aware parallelism designs. To this end, this paper proposes the Coupled SSM model, for coupling state chains of multiple modalities while maintaining independence of intra-modality state processes. Specifically, in our coupled scheme, we devise an inter-modal hidden states transition scheme, in which the current state is dependent on the states of its own chain and that of the neighbouring chains at the previous time-step. To fully comply with the hardware-aware parallelism, we obtain the global convolution kernel by deriving the state equation while introducing the historical state.Extensive experiments on CMU-MOSEI, CH-SIMS, CH-SIMSV2 through multi-domain input verify the effectiveness of our model compared to current state-of-the-art methods, improved F1-Score by 0.4%, 0.9%, and 2.3% on the three datasets respectively, 49% faster inference and 83.7% GPU memory save. The results demonstrate that Coupled Mamba model is capable of enhanced multi-modal fusion.

Main paper: 
 introduction: 
 Real-world data captured and processed across multiple modalities, such as text, image, video, and sensor data, yield a rich tapestry of information that is inherently complementary. This complementarity profoundly enhances the capacities of deep learning models, facilitating more nuanced interpretations and predictions. As a result, deep learning models that integrate multi-modal data have shown substantial superiority over their uni-modal counterparts in various domains, including visual-language learning [1, 2, 3] , multi-modal classification/segmentation [4, 5, 6, 7] , sentiment analysis [8, 9, 10, 11] and etc. Given these advantages, the development of effective multi-modal fusion techniques has emerged as a center of attention. A variety of works have explored this topic on convolution or Transformer -based models, and developed specified mechanisms as early, middle, and late fusion, depending on position of fusion been conducted. A more prevalent practice is to first extract features using modality-specific backbones and then devise a fusion module to exploit the complementary information from all modalities. Existing fusion paradigms either aggregate modal-specific features into one by neglecting individual intra-modal propagation [12] or align modal-specific features into a united representation space through regulation while failing to exploit complementary inter-modal information exchange for difficulty in alignment supervision [13] .Recently, the state space models, advanced by the LSSL [14, 15, 16] , S4 [17] , GSS [18] , and S4D [19, 20] , use state variables to explicitly model the sequential evolving neural states, have being emerged as compelling alternatives to Transformers for its efficiency in modeling long-range sequences [21] . Particularly, Mamba [22] , improves with a selective scanning mechanism and hardware-aware parallelism to enable very efficient training and inference, achieving comparable performances to Transformers on large-scale data. Yet, existing explorations focus on processing uni-modal data, and the multi-modal fusion mechanism on SSMs is still under-investigated.In this paper, we observe that the explicit state variables in SSMs provide great fusion anchors, i.e., from which we can extract inter-model complementary information, and to where we can fuse the complementary information into a unified representation. Inspired by the effective Coupled Hidden Markov Model (CHMM) [23] , we investigate the multi-modal fusion problem of the Mamba model from a state transition perspective. For multi-modal fusion on Mamba, the bruteforce way is to direct aggregate features from all multi-modalities into one feature, i.e., the aggregation approach, and process with a sole Mamba model. However, such an approach neglects the individual intra-modal propagation. Instead, we propose the Coupled Mamba model, for coupling state chains of multiple modalities while maintaining independence of intra-modality state processes. Specifically, in our coupled scheme, we devise an inter-modal hidden states transition scheme, in which the current state is dependent on the states of its own chain and that of the neighbouring chains at the previous time-step. Another challenge is to fully comply with the hardware-aware parallelism for efficiency, we achieve parallel computing by deriving multi-modal global convolution kernels. As shown in Figure 1 , the entire Coupled Mamba model consists of N layers, and is finally adapted to downstream tasks through pooling. Each layer has M Coupled Mamba blocks, where M is the number of modalities. Each Coupled Mamba block receives sequence data of multiple modalities as input, aggregate states from multiple modalities, and then transits into the state at next time of each individual modality. We conduct extensive experiments on CMU-MOSEI, CH-SIMS [24] , CH-SIMSV2 [25] datasets through multi-domain input, and verify the effectiveness of our model compared to current state-of-the-art methods, with 0.4%, 0.9%, 2.3% F1-Score increase, 49% faster inference and 83.7% GPU memory save. The results demonstrate that our Coupled Mamba model enhances the multi-modal fusion with state coupling.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: The abstract and introduction of this paper accurately reflect the contribution of the paper, such as solving the problem that Mamba cannot be parallelized when using multi-modal input, and verifying the effectiveness of our method.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Multi-modal Fusion Multi-modal fusion focuses on combining features from various modalities into unified representations to tackle multi-modal learning challenges. Traditionally, fusion methods are categorized into feature-level early fusion and decision-level late fusion, based on where fusion occurs within the model [26] . Early fusion techniques are employed by [27] to merge features from diverse modalities such as audio, text, and vision. [28] introduce a method using two separate branches for spatial and temporal modalities with a straightforward post-fusion for video action recognition. Other notable post-fusion approaches include works like [9] and [29, 30] , which suggest robust late fusion via rank minimization. Recent advances in deep learning have expanded the concept of early fusion to mid-term fusion, which integrates features at multiple levels [31] . For instance, [32] develop a fused representation by progressively combining multiple fusion layers. Similarly, [33] propose a multi-layer fusion method that connects all modality-specific networks through a central network. [? ] introduce an architecture search algorithm to identify the optimal fusion architecture. Furthermore, [34, 35] incorporate attention mechanisms into multi-modal fusion, while [13] suggest exchanging feature channels between modalities. Additionally, [36] integrate bilinear pooling into attention blocks, showing its effectiveness in capturing higher-level feature interactions by stacking multiple attention blocks for image captioning. The focus has recently shifted towards dynamic fusion, which selects the optimal fusion strategy from various candidate operations based on inputs from different modalities [37, 38] . This dynamic approach offers greater flexibility for different multi-modal tasks compared to static methods. Inspired by the success of dynamic fusion designs and higher-level feature interaction capture in multi-modal fusion, our work aims to dynamically capture hidden states both within and between modalities using coupled state space models via state diffusion, enabling more efficient modality fusion for complex multi-modal tasks.State Space Models State Space Models (SSM) are exceptionally effective at learning the complex correlations inherent in language sequences. The seminal work of [17] introduced the structured state space model (S4), which aims to encapsulate the extended dependency characteristics of language sequences. Conceptually, S4 combines the unique properties of CNNs and RNNs to create a powerful framework for sequential data processing. Building on the foundation laid by S4, subsequent research efforts have been devoted to solving the problem of linearly scaling sequence lengths. In this regard, [39] introduced S5 utilizing MIMO-SSM and parallel scan technology, while [40] proposed H3, which greatly improved the performance of SSM, and [18] introduced GSS, which demonstrated faster training and competitive performance. Furthering the current state of research, [22] developed a novel language model called Mamba. This model uniquely combines a data-selective SSM layer and a parallel scanning algorithm to solve Transformer's quadratic complexity calculation problem in long sequence modeling and Transformer's inability to model data outside the attention window. This also illustrates the huge potential of Mamba in processing sequence data.Coupled Hidden Markov Model Hidden Markov Model (HMM) is a probabilistic model that simulates a sequence of hidden states to generate a sequence of observations. The core components of the model include the state transition matrix A, the observation probability matrix (emission matrix) B and the initial state probability vector π. This model assumes the existence of Markov chains between hidden states, and observation events are independently generated by hidden states. To address specific needs, researchers have developed several HMM variants. For example, the Hierarchical Hidden Markov Model (HHMM) [41] introduces a state hierarchy based on standard HMMs, while the Mixed Hidden Markov Model (MHMM) [42] combines multiple HMMs to Build complex distributions. These extensions improve the applicability of HMM in various scenarios and further promote the application of sequence data analysis in multiple fields. Coupled Hidden Markov Models (CHMM) [23] are a class of tools capable of modeling multiple interrelated time series. In multimodal fusion, we usually focus on signals from different channels, such as audio, text, and facial expressions, which are all time-correlated. Coupled HMMs can effectively model such data because they can consider dynamic correlations between multiple channels simultaneously.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: We have discussed the limitations in the end of the paper.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
coupled state space model: 
 In this section, we introduce Coupled Mamba method for multi-modal fusion in detail, which performs multi-modal fusion by introducing multi-modal historical states. As shown in Figure 2 , it contains two parts: state coupling and state space model.In recent years, the state space model has developed rapidly [17, 19, 40] . Mamba introduced a selectivity mechanism based on S4, which converted the original time-invariant characteristics. Mamba is based on the concept of continuous systems by introducing hidden states h (t) ∈ R N to map a series of inputs x (t) ∈ R L to obtain output y (t) ∈ R L , where N denotes the number of hidden states. The continuous system can be expressed as:h ′ (t) = Ah(t) + Bx(t), y(t) = Ch(t).(1) where A ∈ R N ×N represents the state transition matrix of the system, and B ∈ R N ×1 ,C ∈ R N ×1 are projection matrices. Mamba uses a time scale parameter ∆ to discretize the continuous parameters A, B into A, B, the zero-order hold (ZOH) principle is adopted by default. The discretized statespace equation is: Then the discretized version of Eq. ( 1) with step size ∆ can be rewritten as:EQUATIONEQUATIONFinally, by expanding h t-1 layer by layer, the global convolution kernel K ∈ R L can be obtained, and K is used to calculate the output y, which is defined as follows:EQUATION)where L is the length of the input sequence x and ⊗ denotes the convolution operation.For algorithm 1, L denotes the sequence length, E denotes the extended dimension, D denotes the feature dimension, and B denotes the batch size.For multi-modality data input, one naive way is to aggregate the multi-modal features into one feature and process using a single Mamba model. However, such approach neglects intra-modal propagation.Inspired by the Coupled Hidden Markov Model (CHMM) [23] , a more elegant solution is to model mutual modality transition probability as follows:P i=1:M,j = P h j t |h 1 t-1 , h 2 t-1 , ..., h M t-1where P i=1:M,j is the probability transition matrix from all modalities to current modality j. For SSM with M multi-modal Algorithm 1: Coupled Mamba Data: Input: Ht-1 = h 1 t-1 , h 2 t-1 , ..., h M t-1 , xt-1 : h m t-1 ∈ R N , xt-1 ∈ (B,EQUATIONwhere A i,m denotes the state transition matrix from modality i to m.Taking into account the memory overhead and computational efficiency, such modeling increase the number of parameters and computational complexity greatly. We propose a more memory efficient way by conducting summation before state transition, which achieves similar performance and is much more efficient. So our formation of Coupled SSM is:EQUATIONWhere we use S m ∈ R B×L×D×N to model the overall state transition after states summation. One minor drawback of this modeling is that we require all modalities to have the same state, which can be easily addressed by using projection layers.The main difference between Mamba and traditional recurrent neural networks (RNNs) is that the transition between states does not rely on any activation function. This feature enables it to precalculate intermediate results through the iterative Eq.( 3), thereby achieving parallel computing. However, Coupled Mamba adds multi-modal state information based on Mamba, which brings new challenges to the ability to maintain the Mamba parallelization algorithm. In order to solve this problem, we derived a global convolution kernel suitable for Coupled Mamba to ensure that Coupled Mamba can continue to enjoy the advantages brought by Mamba parallel computing, thereby effectively improving the throughput and inference speed of the model. Detailed analysis on throughput and inference speed will be discussed in depth in subsequent sections.After introducing the state information of different modals, we learned about the entire state transfer process (6) through 3.2. By deriving Eq.( 6), that is, disassembling h m t-1 , we can get the following results:EQUATIONwhere P ∈ R B×L×D×N . According to Eq.( 7) which can be extended to the state information of each modal, we use the following formula to calculate the output.EQUATIONFrom this, the global convolution kernel K = CP 0 , CP 1 , ..., CP t-1 , CP t suitable for Coupled Mamba can be obtained.The global convolution kernel K can be used to perform convolution operations on sequence data.In the convolution operation, the calculations of each convolution kernel and the input sub-region are independent of each other, allowing parallel processing of different convolution kernels or input blocks.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
experiment: 
 To evaluate the effectiveness of our proposed Coupled Mamba in multi-modal fusion, we conduct extensive experiments, with special focus on the multi-modal sentiment analysis (MSA) task as it relies heavily on multi-modal data and is in sequential form. The MSA task aims to predict people's emotional polarity by fusing audio, text, and visual information. To fully evaluate the advantages of our approach, we conduct extensive experiments on both classification and regression tasks.Datasets We conduct experiments on five benchmark datasets (CMU-MOSEI, CH-SIMS [24] , CH-SIMSV2 [25] , MM-IMDB and BRCA To fully validate the performance of Coupled Mamba, we conduct extensive comparisons with the following baselines [43, 30, 27, 11, 44, 45] in Table 1 . We ran five times and reported the average value. We use bold text to show the best results. Traditionally, models that use aligned corpora tend to perform better [27] . In our experiments, we achieve significant improvements on all evaluation metrics compared to unaligned models. Our unaligned method is able to achieve better results even when compared with aligned models. In multi-modal sentiment analysis tasks, language is a key factor because different languages may have different ways of expressing the same emotion. However, Table 2 shows that our Coupled Mamba shows robustness in both English and Chinese sentiment analysis tasks. Even with unaligned data, our method still achieves highest performance.The results of the classification task are given in Table 3 4. It can be seen from the results of the 1 that our proposed fusion method achieves state-of-the-art (SOTA) regardless of whether the data are aligned and from the results of the 4, we find that Coupled Mamba also performs well on Chinese datasets. This is sufficient to demonstrate the effectiveness and robustness of our method. Table 2 : Results on CH-SIMS (Chinese). All models are based on language features extracted by BERT, and the results are compared on unaligned data. Acc-N represents N-level accuracy.Model CH-SIMS Acc -2 ↑ Acc -3 ↑ Acc -5 ↑ F 1 -Score ↑ M AE ↓ TFN [9]78. 4 : Classification task results on CH-SIMS. All models are based on language features extracted by BERT and the results are performed on unaligned data. We ran it five times and report the average results. tasks, Coupled Mamba can show excellent performance. We expect that coupled mamba can be extended to other multimodal tasks.Model CMU-MOSEI Acc -2 ↑ Acc -3 ↑ F 1 -Score ↑ F 1 -Score -3 ↑ EF-LSTM [Acc -2 ↑ Acc -3 ↑ F 1 -Score ↑ F 1 -Score -3 ↑ EF-LSTM [Table 7 : Results on CH-SIMSV2, consistent across all experimental settings, using unaligned data. We run it five times and report the average results.Model CH-SIMSV2 Acc -2 ↑ Acc -3 ↑ Acc -5 ↑ F 1 -Score ↑ M AE ↓ Corr ↑ TFN [9]80. We evaluated the impact of each component in Coupled Mamba to verify the effectiveness of our design. It is worth noting that in order to reduce the impact of randomness on the experimental results, our entire ablation experiment was conducted on the CMU-MOSEI dataset.We use the cross-attention mechanism instead of the fusion strategy for comparison. The results are shown in Table 8 . Coupled Mamba filters input through a selective mechanism and uses historical modal information to remember and perceive global context, so Coupled Mamba also performs modal fusion well on unaligned data. In contrast, cross-attention is sensitive to misaligned data, and this spatio-temporal inconsistency will lead to insufficient integration between modalities and poor performance. The number of hidden states and size of ∆ will have an impact on the results. The size of ∆ affects SSM's ability to retain historical information. An increased size of ∆ focuses more on the present input while disregarding past data, and it also raises the count of hidden states. This escalation in complexity might result in overfitting, higher computational expenses, and may not enhance the model's actual effectiveness. In order to explore the impact of these parameters on the results, we conducted multiple experiments, and the experimental results showed that the model performance changed under different parameter settings. Detailed results can be found in Tables 9, 10 . With ∆ = dstate/8 and dstate = 64, Coupled Mamba achieves the best performance than other configurations. In order to verify the effectiveness of our state coupling, we adopt the splicing fusion, average fusion, and native Mamba blocks for experiments. Average Fusion and Concat Fusion refer to averaging and concatenating the features of different modalities and then sending them to a single Mamba Block for processing. Mamba Fusion refers to using a Mamba Block to process each modality, and finally weighting the results of the three blocks for downstream tasks. The result is shown in Table 11 , our Coupled Fusion obtains the best performance than others. Traditional modal fusion methods, such as averaging and concatenation, fail to fully cope with the inherent heterogeneity of multi-modal data. Such methods ignore the different influences that different modalities may have on specific tasks, thereby failing to effectively reveal the intrinsic correlation between multi-modal data. Simple Mamba blocks are not enough to dynamically grasp the semantic relationships. The introduction of state coupling mechanism based on Mamba can make up for this shortcoming and achieve significant improvements in multiple performance indicators. Compared to Transformers, our approach improves performance by 1% ∼ 2% as shown in Table 8 and decreases memory consumption by more than 83. 7% for sequences length 500 according to Figure 3 . When the sequence length increases, the GPU memory usage of the Transformer-based method increases exponentially. In comparison, our method exhibits linear growth. As the sequence grows, the advantages of Coupled Mambas become more apparent.As shown in Figure 4 , we compared Coupled Mamba and Transformers with five different sequence lengths, and the results show that our inference speed is twice as fast as Transformers under the same sequence length. However, as the sequence length continues to grow, the inference speed of Coupled Mamba will far exceed that of Transformers. GPU Memory (GB) To verify the robustness of our proposed method, we conducted experiments on the CMU-MOSEI dataset with missing data. Specifically, we created a random mask with the same shape as the original tensor, where each element is taken from the Bernoulli distribution B(1-p). This means that each element has a p% probability of being 1 (retained) and a (1-p)% probability (i.e., the missing rate (MR)) of being 0 (missing). We then multiplied this random mask with the original tensor so that the regions with masked values of 0 result in missing data in the original tensor. The results are shown in Table 12 , and the numbers of other baselines are from [63] . Our method shows the best performance. Note that the left side of / shows Acc-2, while the right side indicates the F1-score. 
conclusion and discussion: 
 In this paper, we introduce Coupled Mamba, a novel approach to enhance multi-modal fusion by leveraging state evolution chains within state space. Our method integrates intermediate information from various modalities, capturing dynamic multi-modal interactions over time. This addresses challenges in parallel SSM with multiple inputs. Both quantitative and qualitative experiments confirm the effectiveness of Coupled Mamba. Code is available at https://github.com/hustcselwb/coupledmamba.[68] Zheng Lian, Lan Chen, Licai Sun, Bin Liu, and Jianhua Tao. Gcnet: Graph completion network for incomplete multimodal learning in conversation. IEEE Transactions on pattern analysis and machine intelligence, 45(7):8419-8432, 2023.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: We submit our experimental code in the supporting material. The environment we use is python 3.10, cuda12.1, torch 2.12. We will indicate how to run the code in the supporting material. For the processing of the data set, we give details in the appendix.We refer to the MMSA library to reproduce the baseline on different data sets, and all experimental environments are the same.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We give our training details in detail in the implementation details chapter, such as hyperparameter learning rate, batch size settings, and optimizer selection. In order to be more detailed, we also conducted more detailed comparative experiments on different hyperparameters. This can be viewed in the appendix. We have also discussed the splitting of the data set in the data set chapter, including which data sets to use, why, and the sizes of the test set, validation set, and training set. We have explained them in detail.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: The data we report in the paper is the average data of five report runs, so the error bars are not explicitly written in the paper. Guidelines: • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: Yes, we have discussed the positive and negative impacts of our work. Generally speaking, our research has a positive effect on society and can promote human-computer interaction and better enable computers to serve humans in life.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [No] Justification: Our research does not design data sets with high risk of misuse, and our model is also very safe. For this reason, we do not elaborate on the corresponding protection measures in the article.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets• Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: The following content is the entire process of pushing to the Coupled Mamba parallelization guarantee. First, we first define the symbols. Assume that the total number of modes is M, h m t-1 represents the hidden state at time t-1, where m is any mode, A ∈ R D×N represents the state transition matrix, B ∈ R B×L×N represents the selective matrix obtained by mapping from the current input, C ∈ R B×L×N is the same as B, where superscript B is the batch size, L is the input time series length, and N is the number of hidden states.Since the equations of the state space model and Mamba's core processes, such as discretization processing, hardware-aware algorithms, and parallel execution theory, have been discussed in the text, they will not be repeated below.The core of Coupled Mamba is to introduce multi-modal information while ensuring the parallel computing advantages of Mamba block. After introducing multi-modal information, we have:EQUATION)where A m ∈ R B×L×D×N , G m ∈ R N ×N is a coupling matrix, which can be understood as a shared state transition matrix, which transfers the coupling state based on a certain probability based on the comprehensive state at time t -1. By integrating G m into A m , we can get its unified representation S m ∈ R B×L×D×N .Therefore, the above formula can be expressed asEQUATIONNext, let us derive it step by step starting from time 0, when t = 0 we have:EQUATION)when t = 1, we can get:EQUATION)EQUATION)The subscripts of S and B represent different modalities, and the superscript of x m t represents different modalities.Through this recursive formula we can getEQUATIONIn the same way, we bring h 1 t=1 , h 2 t=1 , ..., h M t=1 into the formula for calculating each mode h m t=2 , we can get:EQUATION)By disassembling h 1 t=1 , h 2 t=1 , ..., h M t=1 , we can get:EQUATION)whereM m=1 h m 0 = B 1 x 1 t=0 + B 2 x 2 t=0 + ... + B M x M t=0 (17) Let B 1 x 1 t + B 2 x 1 t + ... + B M x 1 t be U t , P = M m=1S m ,U t can be expressed asEQUATION)Similarly we can getEQUATION)EQUATIONTherefore, by recursively recursing this formula, we can getEQUATION)Finally we calculate the output through y = C ⊗ M m h m t , we haveEQUATION)where ⊗ represents the convolution operation, and the convolution kernel is K = CP 0 , CP 1 , ..., CP t-1 , CP t . At this point, we have completed the derivation of the entire parallelized calculation.In order to fully verify the effectiveness of this research method, we further performed classification experiments on the CMU-MOSI. The experimental results are displayed in Table 13 , as follows:Table 13 : Results on the CMU-MOSI dataset for classification task, all results are performed under the same conditions, and the average results are reported after five runs.CMU-MOSI Acc-2↑ Acc-3↑ F1-Score↑ F1-Score-3↑ EF-LSTM [49] 46 The CH-SIMSV2 [25] dataset is a Chinese data set for multi-modal sentiment analysis and is an extension of the CH-SIMS data set. The dataset contains audio, text, and video clips from different emotion categories, and each clip is labeled with emotional polarity, such as happy, sad, angry, etc. Each emotion category has corresponding speech, text, and video clips, as well as emotion labels associated with them.Feature extraction CMU-MOSEI uses the pre-trained BERT model to extract language features and obtains 768-dimensional hidden states as word embeddings. For the visual modality, each video frame is encoded using Facet to represent the presence of a total of 35 facial action units. The acoustic model is processed by COVAREP to obtain 74-dimensional features. CH-SIMS uses pre-trained Chinese BERTbase word embeddings to obtain word vectors from text records, and finally represents each word as a 768-dimensional word vector. Acoustic features at 22050Hz were extracted using the LibROSA speech toolkit with default parameters. A total of 33-dimensional frame-level acoustic features are extracted. Extract aligned faces using MTCNN face detection algorithm. The MultiComp OpenFace2.0 toolkit was then used to extract a collection of 68 facial landmarks, 17 facial action units, head pose, head orientation, and eye gaze. Finally, a total of 709-dimensional frame-level visual features were extracted.Exploring Coupled Mamba Layers We investigated the number of layers in Coupled Mamba by performing experiments shown in Table 14 . The optimal performance of our Coupled Mamba was observed at layer = 3. NeurIPS Paper Checklist† Indicates co-corresponding author. 


Decision: This work proposes Coupled SSM (State Space Models) to fuse multiple modalities effectively with SSM. Instead of fusing multi-modal features directly, the proposed method couples state chains of multiple modalities while maintaining the independence of intra-modality state processes. Specifically, they first propose an inter-modal hidden states transition scheme to fuse multiple modalities effectively. Then, they propose an expedited coupled state transition scheme to adapt the hardware-aware parallelism of SSMs for efficiency. Experimental results on classification task on CMU-MOSEI, CH-SIMS, CH-SIMSV2 show promising performance of the proposed method.

Reviews: review: paper_summary: paper_summary: The paper introduces a coupled mamba model for multi-modal fusion. The multi-modal hidden states are fused inside of Mamba, so the current state learns not only from a single modality but the correlation of all modalities. The experiments on multi-modal sentiment analysis show that the proposed model outperforms other baselines.
strength_weakness: strength_weakness: 1. Lack of speed and memory analysis: An important aspect of the proposed method is the memory and speed overhead. It's expected that the overhead is not more than cross-attention (as shown in Figures 3 and 4). However, the paper missed comparisons with other fusion methods, especially the ones that use the inputs directly instead of hidden states. 

2. The experiments are only on multi-modal sentiment analysis. I believe the proposed method is pretty general for any multi-modal data and tasks. Showing a variety of applications will make the paper more convincing and appealing.
questions: questions: 1. The original Mamba uses a parallel scan (recurrent representation) for parallel computing. However, the authors converted Mamba to the convolutional representation. Since eq 6 mainly adds the addition operation (to combine hidden states from different modalities), it should work with the parallel scan. What is the reason to convert to the convolutional representation for parallelization? 
2. The proposed method is compared with average, concat, and mamba fusions in the ablation study, but the explanation of these methods is missing. I assume the average and concat are done directly on the input. How does the mamba fusion work? 
3. What does unaligned/aligned mean in the experiments (Tables 1 and 6)?
4. Some details about the dataset are missing: 
    - What's the sequence length of each dataset?
    - What modalities do the datasets contain?
limitations: limitations: The limitation is discussed as a part of the ethical implications discussion.
paper_summary: paper_summary: This work proposes Coupled SSM (State Space Models) to fuse multiple modalities effectively with SSM. Instead of fusing multi-modal features directly, the proposed method couples state chains of multiple modalities while maintaining the independence of intra-modality state processes. Specifically, they first propose an inter-modal hidden states transition scheme to fuse multiple modalities effectively. Then, they propose an expedited coupled state transition scheme to adapt the hardware-aware parallelism of SSMs for efficiency.  Experimental results on classification task on CMU-MOSEI, CH-SIMS, CH-SIMSV2 show promising performance of the proposed method.
strength_weakness: strength_weakness: 1. The improvement in the regression task on CMU-MOSEI seems marginal compared to the baselines.
2. In line 201, the authors conclude that the results in Tables 3 and 4 show the SOTA performance of the proposed method regardless of whether the data are aligned or not. However, both results in Tables 3 and 4 are performed on unaligned data. Besides, the robustness of the proposed method needs further verified, since I cannot find the results that support the claims about robustness.
questions: questions: Please refer to the weaknesses.
limitations: limitations: The discussions about the limitations of this work would be further improved.
paper_summary: paper_summary: The paper addresses the challenge of multi-modal fusion in deep learning. Current fusion methods struggle to capture complex intra- and inter-modality correlations. Recent state space models like Mamba show promise but are limited in fusing multiple modalities efficiently. The paper propose Coupled Mamba, key aspects include: A coupled state transition scheme that allows information exchange between modalities while maintaining individual propagation. A state summation and transition mechanism to enable parallel computing with multiple inputs. Derivation of a global convolution kernel for efficient parallelization.
The model architecture consists of multiple Coupled Mamba blocks, each processing different modalities and aggregating states before transitioning to the next time step.

Experiments were conducted on three multi-modal sentiment analysis datasets: CMU-MOSEI, CH-SIMS, and CH-SIMSV2. 
Results show improved performance over existing methods as well as faster inference and GPU memory savings compared to transformer-based approaches. Effective performance on both aligned and unaligned multi-modal data.
strength_weakness: strength_weakness: The authors seem to have combined two Mamba models without providing enough insights into how this combination actually improves multi-modal fusion. 

There's no clear analysis or interpretability of what the model is learning or how it's fusing information across modalities. This makes it difficult to understand why their approach is fundamentally better than existing methods, including transformers. 

The speed and linear complexity benefits are inherited from the original mamba model, so they are apparent advantages for all state space-related works. 

The experiments are limited to sentiment analysis tasks. The generalizability of the method to other multi-modal tasks or domains is not demonstrated or discussed.
questions: questions: 1. In table 9, what does Mamba Fusion mean?

2. Can the authors provide more interpretability behind combining to mamba structure and explain how the modalities interact between the two mamba?
limitations: limitations: The authors have addressed the limitations
paper_summary: paper_summary: This paper proposes the coupled mamba to address the problem of multimodal data fusion. The core architecture of the coupled mamba is derived in the form of Equation (6) by improving upon Equation (5). Extensive ablation experiments validate the effectiveness of the proposed coupled mamba.
strength_weakness: strength_weakness: 1、I noticed that some of the references in the article are incomplete and inconsistent in format. Please handle this carefully. \
2、Why does $S_o$ in Algorithm 1 have the same form as in SSM? Is it reasonable to keep it consistent with SSM?\
3、The relationship between Equation (5) and the probability transition matrix in CHMH seems insignificant, although they are similar in form.\
4、The Equation (6) is a clever maneuver, but directly summing indicates that the coefficients in front of the hidden states of each modality are all 1. Is this design reasonable? Moreover, there is a lack of corresponding ablation experiments to validate this design.\
5、In Algorithm 1, the dimensions of $S_o$ : (B, L, E, N) and those of $S_m$ mentioned on page five are inconsistent.\
6、Some experiments, such as Table 1 and Table 2, lack corresponding inference time, FLOPs, or parameter data.\
7、In the Introduction, there is an extra space in 'Transformer-based'.
questions: questions: 1、Why is $S_o$ consistent with the form in SSM, rather than being obtained through input? What is its relationship with the probability transition matrix in Coupled Hidden Markov Models?\
2、What is the relationship between Formula 5 and the probability transition matrix in CHMH? Are they merely similar in form?\
3、In Formula 5, the term $\bar{A}_{m,m}$ should differ from other $\bar{A}$ terms, as it is most relevant to the current state $h_t^m$. Has this aspect been considered in this paper?\
4、Are there relevant ablation experiments to verify the validity of the simplification from Equation (5) to Equation (6)?
limitations: limitations: Please refer to Weaknesses and Limitations
paper_summary: paper_summary: This paper extends the state space model Mamba into the multi-modal domain. The authors propose utilizing separate Mamba blocks to process each modality and suggest conditioning the state of each modality on the others to facilitate modality interaction. They further introduce a parallelism technique for the coupled Mamba transition. The framework is evaluated on three benchmark datasets.
strength_weakness: strength_weakness: 1. The demonstration of the parallelism is not clearly illustrated. The original Mamba model's essence lies in the selection mechanism and the hardware-aware parallel scan. The parallelism achieved through the global convolution kernel in S4 is no longer as effective in the Mamba framework, given that the transition matrices at each time step are conditioned on the input. This inspired the introduction of the parallel scan method in the Mamba paper. While the authors show that coupled Mamba can be computed using a global convolution, the question remains, how is the parallelism achieved in the context of Mamba's scanning process?

2. The idea of using a sequential model to model each modality and then employing a simple interaction scheme (in this paper, sum) to enable modality fusion is straightforward, but maybe too simple to model the complex modality interaction.

3. Comparison with existing multi-modal Mamba models, for instance, [1-2], should be presented.

[1] VL-Mamba: Exploring State Space Models for Multimodal Learning
[2] Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inference
questions: questions: 1. In Table 9, does 'average (concat) fusion' means averaging (concating) the features from all modality and then go through a single mamba block?
limitations: limitations: Yes

Comments: title: response to the rebuttal
comment: Thank you very much for the supportive feedback. We will meticulously revise the paper and clarify the points concerning efficiency and performance.
Thank you for the additional experiments. This is very helpful. 
As my last concern is addressed, I raise my rating from 5 to 6. 

Please add the answers and additional results to the revised version. Also, the conclusion of the last discussion regarding efficiency should be clarified in the paper. It's important to note that the proposed approach is not necessarily efficient compared to mamba-based fusion models but improves performance.
Thank you for your positive feedback. We need to make a correction to our response to W3. While we do partially inherit the speed and linear complexity advantages of the original Mamba, our focus was on developing a fusion mechanism to extend Mamba's capability for better handling cross-modal data.

When compared to naive adaptation methods like 'concat' and 'mamba fusion,' our Coupled Mamba has similar inference speed but consumes more memory due to the specialized mechanism we developed for cross-modality fusion. Despite the increased memory usage, the performance improvement from Coupled Mamba is relatively significant compared to these naive adaptations. Please refer to the 2nd-5th tables we presented to Reviewer JRDw for more details.
Thank you for your thoughtful comments. We acknowledge that Coupled Mamba consumes more memory with similar inference speed. This is due to the specialized mechanism we developed for cross-modality fusion, which increases memory usage.

Regarding the minimal accuracy improvement, this is because our comparison was based on the CMU-MOSEI dataset (Table 9 in our paper), which involves relatively short sequences. Our approach is more suitable for modeling longer sequences. (Reviewer pbQ3 also mentioned this issue in W1; please also check our response.) To provide a more comprehensive understanding, we present memory, speed, and accuracy results in the table below for all other datasets used in the paper, i.e., the CH-SIMS, CH-SIMSV2, BRCA, and MM-IMDB datasets. It can be seen that the improvement from Coupled Mamba is not marginal compared to Mamba Fusion for datasets with longer sequences.

|          | **CH-SIMS**       |        |               |
|----------|---------------|---------------|---------------|
| **Model** | **Acc_2**     | **F1-Score**  |**Infer(s)/Mem(MB)**|
| Mamba Fusion | 80.1          | 79.7          |0.0070/93.18|
| **Coupled Mamba** | **81.8** | **81.3** |0.0072/177.15|

|      | **CH-SIMSV2**      |      |               |
|------|----------------|----------------|---------------|
| **Model** | **Acc_2** | **F1-Score** |**Infer(s)/Mem(MB)**|
| Mamba Fusion | 81.5          | 81.7          |0.0071/89.01|
| **Coupled Mamba** | **83.4** | **83.5** |0.0074/171.01|

| **BRCA** | **Acc**      | **WeightedF1** | **MacroF1**    |**Infer(s)/Mem(MB)**|
|------|----------|------------|------------|---------------|
| Mamba Fusion           | 84.7      | 84.9      | 81.7      |0.0067/67.59|
| **Coupled Mamba** | **88.1**  | **88.5**  | **85.4**  |0.0069/126.97|

| **MM-IMDB** | **MicroF1** | **MacroF1** |**Infer(s)/Mem(MB)**|
|-------------|-------------|-------------|-------------|
| Mamba Fusion | 60.27       | 49.84       |0.051/171.01|
| **Coupled Mamba** | **62.41** | **52.58** |0.052/343.04|

We also apologize for the misleading conclusion regarding memory and speed on 'concat' and 'Mamba Fusion.' We will clarify this point to Reviewer FSJZ and make the necessary corrections in the paper.
Thank you for the additional table. 

I understand that coupled Mamba outperforms other naive fusion approaches, but the proposed appraoch requires almost 2x more memory (and similar inference speed when the sequence length=500) compared to mamba fusion while the accuracy improvement is minimal. 

The paper does not clearly indicate that the efficiency of the approach is by using Mamba for multi-modal fusion, not by introducing coupled Mamba. The response to reviewer FSJZ10 also indicates that the coupled design is more efficient than other fusion approaches. 
> When naively adapted for multi-modal data, as indicated by the 'concat' and 'mamba' fusion approach shown in Table 9 in the main paper, they require higher memory and computational resources, resulting in inferior performance compared to our coupled design.

The authors should clarify this point in the paper and to other reviewers as it can be misleading.
Thanks for the response, especially for the part that clarified the process of parallelism. I've raised my score to borderline accept.
Thank you for your thoughtful feedback. Follow your suggestion, we conducted additional experiments on memory and inference speed using five methods: Average, Concat, Mamba Fusion, IMDer, and DynMM. We present the result in the below table.

|              | Seq=100 | Seq=200 | Seq=300 | Seq=400 | Seq=500 |
|--------------|----------|----------|----------|----------|----------|
| **Average**      | 24.58/0.0006 | 41.98/0.0006 | 59.39/0.0006 | 76.80/0.0006 | 94.20/0.0006 |
| **Concat**       | 49.15/0.0006 | 95.23/0.0006 | 131.04/0.0007   | 174.08/0.0007 | 205.84/0.0007 |
| **Mamba Fusion** | 41.98/0.0068  | 60.42/0.0068  | 80.90/0.0070 | 103.42/0.0071  | 134.57/0.0073 |
| **IMDer**        | 464.37/0.7154   | 492.74/0.7352  | 535.41/0.7587  | 650.87/0.7994  | 693.54/0.8124  |
| **DynMM**        | 62.46/0.2821  | 226.30/0.3133 | 301.06/0.3314 | 353.28/0.3546 | 467.97/0.3915 |
| **Coupled Mamba**| 58.36/0.0069  | 103.42/0.0071 | 151.00/0.0073   | 205.82/0.0074 | 244.73/0.0075 |

The data on the left and right sides of the **/** represent memory usage (MB) and inference speed (s), respectively. The results show that Coupled Mamba is significantly faster (by 50 to 100 times), and consuming less memory than both IMDer and DynMM. Moreover, our Coupled Mamba outperforms Mamba Fusion (Table 9 in main paper) with similar speed and slightly higher memory usage, demonstrating that our design is both efficient and effective.

**Here are some articles that supplement the baseline comparisons in the experiments:**

**CRidge** : Van De Wiel M A, Lien T G, Verlaat W, et al. Better prediction by use of co‐data: adaptive group‐regularized ridge regression[J]. Statistics in medicine, 2016, 35(3): 368-381.

**GMU** : Arevalo J, Solorio T, Montes-y-Gómez M, et al. Gated multimodal units for information fusion[J]. arXiv preprint arXiv:1702.01992, 2017.

**CF**: Hong D, Gao L, Yokoya N, et al. More diverse means better: Multimodal deep learning meets remote-sensing imagery classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 59(5): 4340-4354.

**MOGONET**: Wang T, Shao W, Huang Z, et al. MOGONET integrates multi-omics data using graph convolutional networks allowing patient classification and biomarker identification[J]. Nature communications, 2021, 12(1): 3445.

**TMC** : Han Z, Zhang C, Fu H, et al. Trusted multi-view classification[C]//International Conference on Learning Representations. 2020.

**MM-Dynamics** : Han Z, Yang F, Huang J, et al. Multimodal dynamics: Dynamical fusion for trustworthy multimodal classification[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 20707-20717.

**LRMF**: Liu Z, Shen Y, Lakshminarasimhan V B, et al. Efficient low-rank multimodal fusion with modality-specific factors[J]. arXiv preprint arXiv:1806.00064, 2018.

**MFM** : Tsai Y H H, Liang P P, Zadeh A, et al. Learning factorized multimodal representations[J]. arXiv preprint arXiv:1806.06176, 2018.

**MI-Matrix** : Jayakumar S M, Czarnecki W M, Menick J, et al. Multiplicative interactions and where to find them[C]//International conference on learning representations. 2020.

**RMFE** : Gat I, Schwartz I, Schwing A, et al. Removing bias in multi-modal classifiers: Regularization by maximizing functional entropies[J]. Advances in Neural Information Processing Systems, 2020, 33: 3197-3208.

**CCA**: Sun Z, Sarma P, Sethares W, et al. Learning relationships between text, audio, and video via deep canonical correlation for multimodal language analysis[C]//Proceedings of the AAAI conference on artificial intelligence. 2020, 34(05): 8992-8999.

**RefNet**: Sankaran S, Yang D, Lim S N. Multimodal fusion refiner networks[J]. arXiv preprint arXiv:2104.03435, 2021.

**DynMM** : Xue Z, Marculescu R. Dynamic multimodal fusion[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 2575-2584.
Thanks for the responses. I am leaning towards accepting.
Thank you for your further feedback.

**On the unrigorous conclusions from Tables 3, 4, and 10**: We apologize for the misunderstanding. You are correct that the expression was unrigorous if it referred only to Tables 3, 4, and 10. In the revision, we will update the phrase to 'It can be seen from Tables 1, 3, 4, and 10 ...' for a more accurate and precise expression.

**On the evaluation of robustness**: We apologize for the confusion. To validate the robustness of our model, we tested our Coupled Mamba under conditions where part of the data was missing, and with noisy input (by adding a certain level of Gaussian noise).

For testing on missing data, we followed the experimental setup in [1]. We conducted experiments on the CMU-MOSEI dataset by creating a random mask with the same shape as the original tensor, where each element is drawn from the Bernoulli distribution B(1-p). This means each element has a p% probability of being 1 (retained) and a (1-p)% probability (i.e., the missing rate (MR)) of being 0 (missing). We then multiplied this random mask with the original tensor, causing the areas with a mask value of 0 to result in missing data in the original tensor. The results are presented in the table below, with the numbers for other baselines copied from [1] due to rebuttal time constraints. Our method demonstrates the best performance. Please note that the left side of **/** shows Acc_2, while the right side denotes the F1 Score.

| Datasets   | MR | DCCA   | DCCAE  | MCTN   | MMIN   | GCNET  | **Coupled Mamba** |
|------------|----|--------|--------|--------|--------|--------|------------------|
| CMU-MOSEI  | 0.0 | 80.7/80.9 | 81.2/81.2 | 84.2/84.2 | 84.3/84.2 | 85.2/85.1 | **85.5/85.6**    |
|            | 0.1 | 77.4/77.3 | 78.4/78.3 | 81.8/81.6 | 81.9/81.3 | 82.3/82.1 | **82.6/82.7**    |
|            | 0.2 | 73.8/74.0 | 75.5/75.4 | 79.0/78.7 | 79.8/78.8 | 80.3/79.9 | **81.1/80.9**    |
|            | 0.3 | 71.1/71.2 | 72.3/72.2 | 76.9/76.2 | 77.2/75.5 | 77.5/76.8 | **81.0/81.0**    |
|            | 0.4 | 69.5/69.4 | 70.3/70.0 | 74.3/74.1 | 75.2/72.6 | 76.0/74.9 | **78.4/78.5**    |
|            | 0.5 | 67.5/65.4 | 69.2/66.4 | 73.6/72.6 | 73.9/70.7 | 74.9/73.2 | **77.4/77.7**    |
|            | 0.6 | 66.2/63.1 | 67.6/63.2 | 73.2/71.1 | 73.2/70.3 | 74.1/72.1 | **75.1/75.4**    |
|            | 0.7 | 65.6/61.0 | 66.6/62.6 | 72.7/70.5 | 73.1/69.5 | 73.2/70.4 | **74.1/74.2**    |
| Average    |     | 70.3/71.2 | 72.6/71.2 | 77.0/76.1 | 77.3/75.4 | 77.9/76.8 | **79.4/79.5**    |

For the evaluation on noisy input, we added Gaussian noise to the input at three noise levels, with standard deviations of 1, 2, and 3. Due to the limited time available for the rebuttal, we were only able to compare our results with Mult [2] as a reference. The performance of Coupled Mamba declines much slower as the noise level increases. The left and right sides of **/** represent Acc_2 and F1 Score, respectively.

|                       | Std=0            | Std=1            | Std=2            | Std=3            |
|-----------------------|------------------|------------------|------------------|------------------|
| Mult                  | 82.5 / 82.3      | 79.8 / 80.1      | 77.1 / 76.9      | 74.6 / 74.8      |
| CrossAttention       | 84.6 / 84.5      | 82.3 / 82.2      | 80.2 / 80.4      | 78.4 / 78.6      |
| **Coupled Mamba**     | **85.6 / 85.5** | **84.6 / 84.3** | **83.2 / 83.3** | **81.4 / 81.3** |

The experiments on missing data and noisy input demonstrate that our Coupled Mamba model is robust.

**DCCA** Andrew G, Arora R, Bilmes J, et al. Deep canonical correlation analysis[C]//International conference on machine learning. PMLR, 2013: 1247-1255.

**DCCAE** Wang W, Arora R, Livescu K, et al. On deep multi-view representation learning[C]//International conference on machine learning. PMLR, 2015: 1083-1092.

**MCTN** Pham H, Liang P P, Manzini T, et al. Found in translation: Learning robust joint representations by cyclic translations between modalities[C]//Proceedings of the AAAI conference on artificial intelligence. 2019, 33(01): 6892-6899.

**MMIN** Zhao J, Li R, Jin Q. Missing modality imagination network for emotion recognition with uncertain missing modalities[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021: 2608-2618.

**GCNET** Lian Z, Chen L, Sun L, et al. GCNet: Graph completion network for incomplete multimodal learning in conversation[J]. IEEE Transactions on pattern analysis and machine intelligence, 2023, 45(7): 8419-8432.

[1] Wang Y, Li Y, Cui Z. Incomplete multimodality-diffused emotion recognition[J]. NeurIPS,2024,36.

[2] Tsai Y H H, Bai S, Liang P P, et al. Multimodal transformer for unaligned multimodal language sequences[C],ACL 2019.
Thank you for the answers. 
I still have a concern regarding speed and memory analysis. 
The authors added Figures 1-3 in the rebuttal pdf (FLOPS, inference speed, and memory comparisons). However, the comparing methods are not SSM/Mamba-based, and none of these models are recent SOTA. It's unclear whether the memory and speed benefits come from Mamba. Could authors add these additional models to Figures 1-3? 

1) different fusion methods (avg. concat, and mamba fusion from Table 9)
2) more recent methods like [44]
3) some models from Tables 1 and 2 (rebuttal pdf)

Also, please add citations for Tables 1 and 2 in the rebuttal pdf. It's unclear whether these models are comparable.
First, I mean the claim "It can be seen from the results...the proposed fusion method achieves state-of-the-art (SOTA) regardless of whether the data are aligned or not" is not rigorous, as experiments in Tables 3,  4, and 10 are conducted on unaligned data. It would be better to consider the results from Table 1 and make such a conclusion.

Second, I think the author may have confused robustness with generalization, and I believe robustness is about the model performance under noisy inputs.
Thanks for the responses. My concerns are resolved. I'll raise the score to WA.