Coupled Mamba: Enhanced Multimodal Fusion with Coupled State Space Model
Authors: Wenbing Li
Hang Zhou
Junqing Yu
Zikai Song
Wei Yang
Venue: NeurIPS 2024 poster (2024)
Keywords: Mamba
multi-modal fusion
multi-modal sentiment analysis
Primary area: deep_learning_architectures
TL;DR: Enhanced Multimodal Fusion with Coupled State Space Model

Abstract: The essence of multi-modal fusion lies in exploiting the complementary information inherent in diverse modalities.However, most prevalent fusion methods rely on traditional neural architectures and are inadequately equipped to capture the dynamics of interactions across modalities, particularly in presence of complex intra- and inter-modality correlations.Recent advancements in State Space Models (SSMs), notably exemplified by the Mamba model, have emerged as promising contenders. Particularly, its state evolving process implies stronger modality fusion paradigm, making multi-modal fusion on SSMs an appealing direction. However, fusing multiple modalities is challenging for SSMs due to its hardware-aware parallelism designs. To this end, this paper proposes the Coupled SSM model, for coupling state chains of multiple modalities while maintaining independence of intra-modality state processes. Specifically, in our coupled scheme, we devise an inter-modal hidden states transition scheme, in which the current state is dependent on the states of its own chain and that of the neighbouring chains at the previous time-step. To fully comply with the hardware-aware parallelism, we obtain the global convolution kernel by deriving the state equation while introducing the historical state.Extensive experiments on CMU-MOSEI, CH-SIMS, CH-SIMSV2 through multi-domain input verify the effectiveness of our model compared to current state-of-the-art methods, improved F1-Score by 0.4%, 0.9%, and 2.3% on the three datasets respectively, 49% faster inference and 83.7% GPU memory save. The results demonstrate that Coupled Mamba model is capable of enhanced multi-modal fusion.

Coupled Mamba: Enhanced Multi-modal Fusion with Coupled State Space Model
Wenbing Li Hang Zhou Junqing Yu Huazhong University of Science and Technology (wenbingli, henrryzh, yjqing, skyesong, veiyanges) Chust. edu. cn
Zikai Song! Wei Yang!
Abstract
The essence of multi-modal fusion lies in exploiting the complementary infor- mation inherent in diverse modalities. However, prevalent fusion methods rely on traditional neural architectures and are inadequately equipped to capture the dynamics of interactions across modalities, particularly in presence of complex intra- and inter-modality correlations. Recent advancements in State Space Models (SSMs), notably exemplified by the Mamba model, have emerged as promising contenders. Particularly, its state evolving process implies stronger modality fusion paradigm, making multi-modal fusion SSMs appealing direction. However, fusing multiple modalities is challenging for SSMs due to its hardware- aware parallelism designs. To this end, this paper proposes the Coupled SSM model, for coupling state chains of multiple modalities while maintaining independence of intra-modality state processes. Specifically, in coupled scheme, devise an inter-modal hidden states transition scheme, in which the current state is de- pendent the states of its chain and that of the neighbouring chains at the previous time-step. To fully comply with the hardware-awaad devise expedite coupled state transition scheme and derive its corresponding global convolution kernel for parallelism. Extensive experiments on CMU-MOSEI, CH-SIMS. CH-SIMSV2. BRCA. MM-IMBB through multi-domain input verify the effectiveness of our model compared to current state-of oifie-of-thhattt the--art methods, im- proved F1-Score by 0.4%. 0.9%. and 2.3% the CMU-MOSEL CH-SIMS and CH-SIMSV2 datasetes respectively. 49% faster inference and 83.7% GPU memory save. The results demonstrate that Coupled Mamba model is capable of enhanced multi-modal fusion.
parallelism, we
Introduction
Real-world data captured and processed across multiple modalities, such text, image, video, and sensor data, yield rich tapestry of information that is inherently complementary This comple- mentarity profoundly enhances the capacities of deep learning models, facilitating more nuanced interpretations and predictions. As a result, deep learning models that integrate multi-modal data have shown substantial superiority over their uni-modal counterparts in various domains, including assual-anguage learning [ 2 3]. multi-modal daasificattoonssemenntte
4
5 6 77 sentiment
analysis [8] 9 10 and etc. Given these advantages, the development of effective multi-moda
fusion techniques has emerged as center of attention. A variety of works have explored this topic on convolution Transformer -based models, and developed specified mechanisms as early, middle, and late fusion, depending on position of fusion been conducted. A more prevalent practice is to first extract features using modality- sppcific backbones and then devise a fusion module exploit the complementary information from all modalities. Existing fusion paradigms either aggregate
Indicates correspoonding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

modal-specific features into one by neglecting individual intra-modal propagation [12 or align modal-specific features into united representation space through regulation while failing to exploit complementary inter-modal information exchange for difficulty in alignment supervision [3]
Recently, the state space models, advanced by the LSSL [14
15 16,
S4 17, GSS 18 and
S4D [19 20], use state variables to explicitly model the sequential evolving neural states, have being emerged as compelling alternatives to Transformers for its efficiency in modeling long-range sequences [2T] Particularly, Mamba [22]. improves with a selective scanning mechanism and hardware-awe aware parallelism to enable very efficient training and inference, achieving comparable performances to Transformers on large-scale data. Yet, existing explorations focus on process- ing uni-modal data, and the multi-modaa fusion mechanism on SSMs is still under-investigated.
In this paper, we observe that the explicit state variables in SSMs provide great fusion anchors, Le., from which we can extract inter-model com- plementary information, and to where we can fuse the complementary information into uni- fied representation. Inspired by the effective Cou- pled Hidden Markov Model (CHMM) (23,, we investigate the multi-modal fusion problem of the Mamba model from a state transition perspec- tive. For multi-modal fusion on Mamba, the brute- force way is to direct aggregate features from all multi-modalitie into one feature, i.e., the aggre- gation approach, and process with a sole Mamba model. However, such an approach neglects the individual intra-modee propagation. Instead, we propose the Coupled Mamba model, for coupling state chains of multiple modalities while main- taining independence of intra- untra-modality state pro- cesses. Specifically, in our coupled scheme, we
Pooling Layer
Coupled Mamba Layer N
Coupled Mamba Layer
Coupled Mamba Layer
Cat
Coupled Mamba Block
Coupled Mamba Block
Coupled Mamba Block
Figure 1: Architecture of Coupled Mamba.
devise an inter-modal hidden states transition scheme, in which the current state is dependent on the states of its own chain and that of the neighbouring chains at the previous time-step. Another challenge is to fully comply with the hardware-awand parallelism for efficiency, we achieve parallel computing by deriving multi-modal global convolution kernels. As shown in Figure [ the entire Coupled Mamba model consists of N layers, and is finally adapted to downstream tasks through pooling. Each layer has M Coupled Mamba blocks, where M is the number of modalities. Each Coupled Mamba block receives sequence data of multiple modalities as input, aggregate states from multiple modalities, and then transits into the state at next time of each individual modality. We conduct extensive experiments on CMU-MOSEI, CH-SIMS (24), CH-SIMSV2 25) datasets through multi-domain input, and verify the effectiveness of our model compared to current state-of-the-an methods, with 0.4%, 0.9%, 2.3% F1-Score increase, 49% faster inference and 83.7% GPU memory save. The results demonstrate that our Coupled Mamba model enhances the multi-modal fusion with state coupling.
2 Related Work
Multi-modal Fusion Multi-modal fusion focuses on combining features from various modalities into unified representations to tackle multi-modal learning challenges. Traditionally, fusion methods are categorized into feature-level early fusion and decision-level late fusion, based on where fusion occurs within the model [26). Early fusion techniques are employed by (27] to merge features from diverse modalities such as audio, text, and vision. 28 introduce method using two separate branches for spatial and temporal modalities with a straightforward post- post-fusion for video action recognition. Other notable post-fusion approaches include works like [9] and 22 330, which suggest robust late fusion via rank minimization. Recent advances in deep learning have expanded the concept of early fusion
to mid-term fusion, which integrates features at multiple levels (3I) For instance, [32 develop a
fused representation by progressively combining multiple fusion layers. Similarly, [33 propose
lylti-layer fusion method that connects all modality-specin networks through a central network. I?
introduce an architecture search algorithm to identify the optimal fusion architecture. Furthermore,

---

incorporate attention mechanisms into multi- modal fusion, while [3] suggest exchanging feature channels between modalities. Additionally, [36] integrate bilinear pooling into attention blocks, showing its effectiveness in capturing higher-level feature interactions by stacking multiple attention blocks for image captioning. The focus has recently shifted towards dynamic fusion, which selects the optimal fusion strategy from various candidate operations based on inputs from different modalities [37 38). This dynamic approach offers greater flexibility for different multi-modal tasks compared to static methods. Inspired by the success of dynamic fusion designs and higher-level feature interaction capture multi-modal fusion, our work aims to dynamically capture hidden states both within and between modalities using coupled state space models via state diffusion, enabling more efficient modality fusion for complex multi-modal tasks.
34
35
State Space Models State Space Models (SSM) are exceptionally effective at learning the complex correlations inherent language sequences. The seminal work of [77 introduced the structured state space model (S4), which aims to encapsulate the extended dependency characteristics of language sequences. Conceptually, S4 combines the unique properties of CNNs and RNNs to create powerful framework for sequential data processing. Building the foundation laid by S4, subsequent research efforts have been devoted to solving the problem of linearly scaling sequence lengths. In this regard, [39] introduced S5 utilizing MIMO-SSM and parallel scan technology. while [40] proposed H3, which greatly improved the performance of SSM, and [18] introduced GSS, which demonstrated faster training and competitive performance. Furthering the current state of research, (22] developed a novel language model called Mamba. This model uniquely combines a seta-selective SSM layer and parallel scanning algorithm to solve Transformer's ' quadratic complexity calculation problem in long sequence modeling and Transformer's inability to model data outside the attention window. This also illustrates the huge potential of Mamba in processing sequence data.
Coupled Hidden Markoy Model Hidden Markov Model (HMM) is a probabilistic model that simulates sequence of hidden states to generate a sequence of observations. The core components of the model include the state transition matrix A, the observation probability matrix (emission matrix) B and the initial state probability vector и. This model assumes the existence of Markov chains between hidden states, and observation events are independently generated by hidden states. To address specific needs, researchers have developed several HMM variants. For example. the Hicrarchical Hidden Markov Model (HHMM) [41 introduces a state hicrarchy based on standard HMMs, while the Mixed Hidden Markov Model (MHMM) complex distributions. These extensions improve the applicability of HMM in various scenarios and further promote the application of sequence data analysis in multiple fields. Coupled Hidden Markov Models (CHMM) [23] are a class of tools capable of modeling multiple interrelated time series. In multimodal fusion, we usually focus on signals from different channels, such as audio, text, and facial expressions, which are all time-correlated. Coupled HMMs can effectively model such data because they can consider dynamic correlations between multiple channels simultaneously.
42 combines multiple HMMs to Build
3 Coupled State Space Model
In this section, we introduce Coupled Mamba method for multi- mottil fusion in detail, which performs multi-modal fusion by introducing multi-modal historical states. As shown in Figure it contains two parts: state coupling and state space model.
3.1 Preliminary
In recent years, the state space model has developed rapidly 17 19 40
Mamba introduced a selectivity mechanism based S4, which converted the original time-invarianl characteristics. Mamba is based on the concept of continuous systems by introducing hidden states h (t) E R to map series of inputs (t) E B- to obtain output y (t) E R-,, where N denotes the number of hidden states. The continuous system can be expressed as:
M(()) Ah(() + Ba((), v(()= = Ch()).
(I)
where E R.N represents the state transition matrix of the system, and B E N C E Nx1 projection matrices. Mamba uses time scale parameter A to discretize the continuous parameters A, B into A, B, the zero-order hold (ZOH) principle is adopted by default. The discretized state- space equation is:
A = exp (AA), B = (AA) (exp (AA) I) AB.
(2)
3

---

h
N
11
M
22
X
B
y
X
B
C
y
Xt-1
Coupled
Xt
Coupled
h
Figure 2: Coupling Mamba receives input 2,-1, and performs internal state switching and output through three key parameter matrices, where B, C and S are respectively represented the input matrix, output matrix and state transfer matrix. The hidden states are summed across modalities and used for state transition input to generate next time states. The state is propagated sequentially in time.
Then the discretized version of Eq. ① with step size A can be rewritten as:
he Ah--1 Br,, = Ch+-
(3)
Finally, by expanding 14-1 layer by layer, the global convolution kernel K E TR can be obtained, and K is used calculate the output which is defined as follows:
Algorithm 1: Coupled Mamba
Data: Input:
H [hi-1, h2 13-,, h x
h E R I.:-1 E (B. L, D)
K (B,, CAB. CA L-
B
Result: Output: yt: (B, D)
(4)
Require :equire:Input
& K.
Ensure :uutput
where L is the length of the input sequence and & denotes the convolution operation. For algorithm [ L. denotes the sequence length, E denotes the extended dimension, D denotes the feature dimension, and B denotes the batch size.
Normalize the input sequence:
X4-1 (B, L, D) + (arerNoom(())1)))
(B, I. F) Linear,(al_1)
z (B, L, E) + Linear, Linear_(2$-11)
Process with Coupled Mamha: for in forward
3.2 Coupled State Transition
uy (B, L, F)) + SiLUU(onvvld.WW
For multi- modality data input, one naive way is to aggregate the multi-modal fea- tures into one feature and process using single Mamba model. However, such ap- proach neglects intra-modal propagation. Inspired by the Coupled Hidden Markov Model (CHMM) [23], a more elegant solu- tion is to model mutual modality transition probability as follows:
10
B. (B, N) Linear (u6):
C. (B, N) <- Linear ((ncar)(((::;
Ao (B, E.) log(1 +
exp(Lincars, (w() + Parameter )))
13
S. (B, L, E, N) + AN @ Parameter
B. LB, E, +- AN & B.:
14
15
yo: (B, E) +
CSSM(S., B., (H, nb);;
16 end
PI=1.M. I
M
18 Get gated you;
19 Viorward (B, L, E) + 3forward c SiLU(2);
where P is the probability transi- tion matrix from all modalities to current modality j. For SSM with M multi-modal input, we have M state propagation sequences. In alignment with CHMM, we can model the state transition of modality by coupling all the modality states as:
Residual connection:
21 Yt (B, L, D) + Linear (1'fowand + X
hm = Σ
A A1.h h Ag h A, + BmIT v."h = Ch
(5)

---

where Aim denotes the state transition matrix from modality i to m.
Taking into account the memory overhead and computational efficiency, such modeling increase the number of parameters and computational complexity greatly. We propose a more memory efficient way by conducting summation before state transition, which achieves similar performance and is much more efficient. So our formation of Coupled SSM is:
M
h
Sm
h + B B
(6)
Where use Sm E BBXLLDXN to model the overall state transition after states summation. One minor drawback this modeling is that we require all modalities to have the same state, which can be easily addressed by using projection layers.
3.3 Parallelism and Efficiency Analysis
The main difference between Mamba and traditional recurrent neural networks (RNNs) is that the transition between states does not rely on any activation function. This feature enables it to pre- calculate intermediate results through the iterative Eq.(3), thereby achieving parallel computing. However, Coupled Mamba adds multi-modal state information based on Mamba, which brings new challenges to the ability to maintain Mamba parallelization algorithm. order to solve this problem, we derived a global convolution kernel suitable for Coupled Mamba to ensure that Coupled Mamba can continue to enjoy the advantages brought by Mamba parallel computing, thereby effectively improving the throughput and inference speed of the model. Detailed analysis throughput and inference speed will be discussed in depth in subsequent sections.
After introducing the state information of different modals, we learned about the entire state transfer process (6) through 3.2
By deriving (q. that is, disassembling h we can get the following
results:
1-1
P
Σ S.n m=l
U.
Σ m=m1
B Bm
h.T S.n Σ P'U,-11: + Bma,"
where P E B0XXXDXN According to Eq. which can be extended to the state information of each modal, we use the following formula calculate the output.
M
C
8
h
U
k=0
From this, the global convolution kernel K (CPo, CP1 ...., Cpt-1 CPt) suitable for Coupled Mamba can be obtained.
The global convolution kernel K can be used to perform convolution operations sequence data, In the convolution operation, the calculations of each convolution kernel and the input sub-region are independent of each other, allowing parallel processing of different convolution kernels or input blocks.
Experiment
To evaluate the effectiveness of our proposed Coupled Mamba in multi-modal fusion, conduct extensive experiments, with special focus on the multi-mode sentiment analysis (MSA) task as relies heavily multi-modal data and is in sequential form. The MSA task aims to predict people's emotional polarity by fusing audio, text, and visual information. To fully evaluate the advantages of approach, we conduct extensive experiments both classification and regression tasks.
4.1 Datasets and Implementation Details
Datasets We conduct experiments on five benchmark datasets (CMU-MOSHI, CH-SIMS 24), CH- SIMSV2 (25). MM-IMDB and BRCA). CMU-MOSEI dataset is extension of CMU-MOSI,
5

---

contains 22856 samples of movie review video clips. In this dataset, 16326 samples are used as the training set, and the remaining 1871 and 4659 samples are used the validation set and test set respectively. CH-SIMS contains 2281 video clip samples, 1368 samples are used as the training set, and the remaining 456 and 457 samples are used as the validation set and test set respectively. CH-SIMSV2 is an extension of CH-SIMS, which contains 4402 video clip samples, of which 2722 samples are used as the training set, and the remaining 647 and 1034 samples are used as the validation set and test set respectively. For the feature extraction method of the dataset, please refer to the Appendix for more information. The MM-IMDB dataset used for the movie genre classification task, which classifies movies based posters and text descriptions. The BRCA dataset includes mRNA expression, DNA methylation, and miRNA expression data for predicting PAM50 subtype classification of breast cancer.
Evaluation metrics For regression tasks, we use the mean absolute error (MAE), which is the average absolute difference between the predicted value and the true value, and the Pearson corre- lation coefficient (Corr), which measures the degree of deviation of the prediction according to the following formula: The positive/negative and neganaiiive/neeatvvee classification results calculate the binary classification accuracy (Acc-2) and FI-Score, where Acc-2 and FI-Score are more important indicators. For classification tasks, we use Acc-2, Acc-3 and F1-Score (Weighted-Fl, Macro-F1, Micro-F1, F1-score3) as evaluation indicators. Fi-score3 is the overall performance evaluation of all categories, and Fi-score is the performance evaluation of two categories. At the same time, the neutral category is ignored. All experiments were conducted in the same environment.
Implementation details We use hidden dimension size of 128, expansion coefficient of 2, convolution kernel size of 4, number of 3 to train our Coupled Mamba. We use Adam to optimize the model and set the learning rate to 0.0005, weight decay coefficient 0.0005, epoch is 150, the batch size set to 1024, 128, 256 on CMU-MOSEI, CH-SIMS, and CH-SIMSV2. L1 loss is used as the loss function for the regression task, and cross entropy is used as the loss function for the classification task. All experiments were conducted on Linux workstation equipped with single NVIDIA 32GB V100GPU and a 32-core Intel Xeon CPU. More experimental details can be found in the Appendix.
dssaaee$ 8 as the configuration of each Mamba block, and layer
4.2 Comparison with the state-of-the-arts
To fully validate the performance of Coupled Mamba, we conduct extensive comparisons with the following baselines 43 30 27
45 in Table We ran five times and reported the average value. We use bold text to show the best results. Traditionally, models that use aligned corpora tend to perform better [27]. In experiments, achieve significant improvements all evaluation metrics compared to unaligned models. Our unaligned method is able to achieve better results even when compared with aligned models.
Table 1: Results on CMU-MOSEL. All models are based on language features extracted by BERT.
The one with * indicates that the model reproduces under the same conditions.
CMU-MOSEI
Model
Data Setting
MAE J
Corr 1
Acc - 21
F1 Score +
TFN
0.593
0.700
82.5
82.1
Unaligned
LMF 30
0.623
0.677
82.0
82.1
Unaligned
MFN
76.0
76.0
Aligned
MFM
46
0.568
0.717
84.4
84.3
Aligned
MulT
27
0,580
0.703
82.5
82.3
Aligned
MAG-BERT
47
84.7
84.5
Aligned
ICCN
0,565
0.713
84.2
84.2
Aligned
MISA
0.555
0.756
85.5
85.3
Aligned
TETEN 45
0.551
0.748
85.1
85.2
Unaligned
DMD
4F1
84.8
84.7
Unaligned
IMDer3
4
85.1
85.1
Unaligned
MAG-BERT
47
0.549
0.753
85.2
85.1
Aligned
Coupled Mamba (Ours)
0.547
0.756
85.6
85.5
Unaligned
Coupled Mamba (Ours)
0.547
0.758
85.7
85.6
Aligned

---

In multi-modal sentiment analysis tasks, language is key factor because different languages may have different ways of expressing the same emotion. However, Table shows that our Coupled Mamba shows robustness in both English and Chinese sentiment analysis tasks. Even with unaligned data, our method still achieves highest performance.
The results of the classification task are given in Table 3 It can be seen from the results of the that our proposed fusion method achieves state-of-the-an (SOTA) regardless of whether the data aligned and from the results of the we find that Coupled Mamba also performs well on Chinese datasets. This is sufficient to demonstrate the effectiveness and robustness of our method.
Table Results on CH-SIMS (Chinese). All models are based on language features extracted by BERT, and the results are compared on unaligned data. Acc-N represents N-level accuracy.
CH-SIMS
Model
Acc
Acc
Ace 5
F1 Score
MAE
TFN
9
78.4
65.1
39.3
78.6
0.432
LMF
30
77.8
64.7
40.5
77.9
0,411
MFN 10
77.9
65.7
39.5
77.9
0.435
MulT
[27
78.6
64.8
37.9
79.7
0.453
Self-MM
8
80.0
65.5
41.5
80.4
0.425
TETEN
45
81.2
63.2
41.8
80.2
0.420
IMDer
43
76.3
50.7
76.4
Coupled Mamba (Ours)
81.8
68.7
42.1
81.3
0.409
Table 3: Results of classification tasks CMU-MOSEL. All models are based language features extracted by BERT, and the results are performed on unaligned data. We ran five times and report the average results.
CMU-MOSEI
Model
Acc 21
Ace f
F1 Score
F1 Score 3
EF-LSTM
49
26.73
66.09
28.12
63.68
Graph-MFN
50
28.47
66.39
28.77
64.00
TFN 9
28.66
66.63
28.75
63.93
LMF
28.66
66.59
28.92
64.86
MFN 10
28.61
66.59
28.70
64.31
MulT
27
27.38
67.04
28.67
65.01
MISA [1
28.50
67.63
29.03
65.39
Self-MM
29.67
68.15
28.86
66.53
TETEN 45
29.54
67.95
28.47
66.33
Coupled Mamba (Ours)
32.02
68.95
29.72
67.76
Table 4: Classification task results CH-SIMS. All models based language features extracted
by BERT and the results are performed on unaligned data. We ran it five times and report the average results.
CH-SIMS
Model
Acc - 2
Acc 31
31
F1
Score 1
F1 Score
EF-LSTM
49
56.27
54.27
49.85
38.18
Graph-MFN
50
57.99
68.44
54.66
63.44
TFN 9
53.56
65.95
52.79
62.04
LMF
30
57.06
66.87
53.83
62,46
MFN 10
56.96
67.57
54.14
62.37
MulT
27
56.34
68.27
54.26
64.23
MISA
I
57.27
67.05
53,99
60.98
Self-MM
g
58,65
67.56
55,88
65.95
TETFN
45
57.77
66.83
55.15
65.23
Coupled Mamba(Ourg)
60.12
68.75
56.15
67.47
Table 7 shows the results the CH-SIMSV2 dataset, which currently only supports regression tasks. can seen from the table that the method we proposed has achieved a huge improvement of 2.3%, 2, 3% in Acc-2 and F1-Score respectively, indicating the effectiveness of our method.
The results of Coupled Mamba on BRCA and MM-IMDB datasets are shown Table Whether in multimodal sentiment analysis tasks or in movie genre classification tasks biology classification

---

Table 5: Result on the BRCA benchmark: mR, D, and miR denote mRNA expression, DNA methylation, and miRNA expression data respectively. The best results are in bold.
Modality Acc(%)
WeightedF (%))
MacroFI
GRridg [51]
mR+D+miR
74.5
72.6
65.6
GMU [52
mR+D+miR
80.0
79.8
74.6
CF [53
mR+D+miR
81.5
81.5
77.1
MOGONET 54
mR+D+miR
82.9
82.5
77.4
TMC [55]
mR+D+miR
84.2
84.4
80.6
MM- MM-Dynnaics [6
mR+D+miR
87.5
87.6
83.9
Coupled Mamba(Ours)
mR+D+miR
88.1
88.5
85.4
Table 6: Result on the MM-IMDB benchmark. I and T denote image and text respectively. The best results are in bold.
Modality
MicroFI(%))
MacroFI(%o]
LRMF 57
I+T
58.95
50.73
MFM 46
I+T
56.44
48.53
MI-Matrix 158
I+T
55,87
46.77
RMFE [59
I+T
58.67
49.82
CCA 660
I+T
60.31
50.45
RefNet 61
I+T
59.45
51.51
DynMM
I+T
60.35
51.60
Coupled Mamba (Ours)
I+T
62.41
52.58
tasks, Coupled Mamba can show excellent performance. We expect that coupled mamba can be
extended to other multimodal tasks.
Table Results on -IMSI, consistent across all experimental settings, using unaligned data. We run it five times and report the average results.
CH-SIMSV2
Model
Acc 21
Acc f
Acc
51
Fl
Score
MAE
Corr 1
TFN
9
80.1
72.3
52.5
80.1
30.3
70.7
LMF
30
74.1
64.9
47.8
73.8
36.7
55.7
MFN
10
81.1
73.7
54.5
81.2
29.5
72.6
MulT 27
80.7
73.1
54.8
80.7
29.1
73.8
MAG-BER
47
79.8
73.5
53.7
79.8
33.4
69.1
Self-MM
8
79.7
72.6
52.8
79.7
31.1
69.5
TETEN 45
79.7
73.6
54.4
79.8
31.1
69.5
Coupled Mamba
83.4
75.0
55.1
83.5
28.7
75.8
4.3
Ablation study
We evaluated the impact of each component in Coupled Mamba to verify the effectiveness of our design. is worth noting that order to reduce the impact of randomness on the experimental results, entire ablation experiment
conducted the CMU-MOSEI dataset.
We use the cross-attention mechanism instead the fusion strategy for comparison. The results are shown in Table Coupled Mamba filters input through selective mechanism and uses historical modal information remember and perceive global context, so Coupled Mamba also performs modal fusion well on unaligned data. In contrast, cross-attention is sensitive to misaligned data, and this performance.
inconsistency will lead to insufficient integration between modalities and poor

---

Table All things being equal, replacing Coupled Mamba with Cross attention, we execute it five times and report the average results.
CMU-MOSEI
Method
Data Setting
MAE J
Corr
Acc 21
F1 Score
Cross Attention
55,9
73.3
84,6
84.5 85.5
Unaligned Unaligned
Coupled Mamba (Ours)
54.7
75.6
85.6
The number of hidden states and size of A will have impact on the results. The size of A affects SSM's ability to retain historical information. An increased size of A focuses more on the present input while disregarding past data, and also raises the count of hidden states. This escalation in complexity might result in overfitting, higher computational expenses, and may not enhance the model's actual effectiveness. In order explore the impact of these parameters on the results, we conducted multiple experiments, and the experimental results showed that the model performance changed under different parameter settings. Detailed results can be found in Tables g 10 With A = dstale/8 and dslate = 64, Coupled Mamba achieves the best performance than other configurations.
Table 9: Performance on CMU-MOSEI with
Table 10: Performance on CMU-MOSEI with
different timescale A
different dstate
CMU-MOSEI
CMU-MOSEI
A
dstate
Corrt
Acc-2t
F1-Scoret
Corrt
Acc-21
FI-Scoret
dstate/16 16
75.3
85.2
85.0
128
74.1
84.2
84.1
dstate/8
75.6
85.6
85.5
64
75.6
85.6
85.5
dstate/4 74.2
85.0
84.9
32
75.0
84.9
84.9
In order to verify the effectiveness of our state coupling, we adopt the splicing fusion, average fusion, and native Mamba blocks for experiments. Average Fusion and Concat Fusion refer to averaging and concatenating the features of different modalities and then sending them to a single Mamba Block for processing. Mamba Fusion refers to using Mamba Block to process each modality, and finally weighting the results of the three blocks for downstream tasks. The result is shown in Table 11 our Coupled Fusion obtains the best performance than others. Traditional modal fusion methods, such as averaging and concatenation, fail to fully cope with the inherent heterogeneity of multi-modal data. Such methods ignore the different influences that different modalities may have on specific tasks, thereby failing to effectively reveal the intrinsic correlation between multi-modal data. Simple Mamba blocks are not enough dynamically grasp the semantic relationships. The introduction of state coupling mechanism based on Mamba can make up for this shortcoming and achieve significant improvements in multiple performance indicators.
Table 11: Comparison of fusion methods
CMU-MOSEI
Model
MAE 4
Corr 1
Acc 2 f
F1 Score 1
Average Fusion
56.4
73.6
84.2
84.1
Concat Fusion
56.2
72.8
84.8
84.5
Mamba Fusion
55.3
74.9
85.3
85.3
Coupled Fusion
54.7
75.6
85.6
85.5
Compared to Transformers, our approach improves performance by 1% 2% as shown in Table g and decreases memory consumption by more than 83. 7% for sequences length 500 according to Figure When the sequence length increases, the GPU memory usage of the Transformer-batt method increases exponentially. In comparison, our method exhibits linear growth. As the sequence grows, the advantages of Coupled Mambas become more apparent.
As shown in Figure 4 we compared Coupled Mamba and Transformers with five different sequence lengths, and the results show that our inference speed is twice as fast as Transformers under the same sequence length. However, as the sequence length continues to grow, the inference speed of Coupled Mamba will far exceed that of Transformers.

---

GPU Memory Usage by Sequence Length
Comparson Inference Speed
14
0.10
Cross Atsention
12.61
Cross Attention
12
Coupled Mamba
Cougled Mamba
0.0741
0.08
0,0726
GPU Momory (GB)
6,0648
8-41
0.06
5:20
0.04
5.0390
2.42
0.043
5,0363
0.02
77
2.05
100
0.00
200
300
400
Sequence Length
Sequence Length
Figure 3: GPU usage comparison
Figure 4: Inference speed comparison
To verify the robustness of our proposed method, we conducted experiments on the CMU-MOSEL dataset with missing data. Specifically, we created random mask with the same shape the original tensor, where each element is taken from the Bernoulli distribution B(1 B(1-p). This means that each element has 5% probability of being (retained) and (1-p)% % probability (i.e., the missing rate (MR)) being (missing). We then multiplied this random mask with the original tensor so that the regions with masked values of 0 result in missing data in the original tensor.
Table 12:
Performance of Coupled Mamba on CMU-MOSEI dataset when data is missing. Other
baselines are from (63]
MR
DCCA (64]
DCCAE [65
MCTN (66
MMIN 66
GCNET 688
Coupled Mamba
0.0
80.7/80.9
81.2/81.2
84.2/84.2
84.3/84.2
85.2/85.1
85.5/85.6
0.1
77.4/77.3
78.4/78.3
81.8/81.6
81.9/81.3
82.3/82.1
82.6/82.7
0.2
73.8/74.0
75.5/75.4
79.0/78.7
79.8/78.8
80.3/79.9
81.1/80.9
0.3
71.1/71.2
72.3/72.2
76.9/76.2
77.2/75.5
77.5/76.8
81.0/81.0
0.4
69.5/69.4
70.3/70.0
74.3/74.1
75.2/72.6
76.0/74.9
78.4/78.5
0.5
67.5/65.4
69.2/66.4
73.6/72.6
73.9/70.7
74.9/73.2
77.4/77.7
0.6
66.2/63.1
67.6/63.2
73.2/71.1
73.2/70.3
74.1/72.1
75.1/75.4
65.6/61.0
66.6/62.6
72.7/70.5
73.1/69.5
73.2/70.4
74.1/74.2
Average
70.3/71.2
72.6/71.2
77.0/76.1
77.3/7544
77.9/76.8
79.4/79.5
The results shown in Table 12 and the numbers of other baselines from 63 Our method shows the best performance. Note that the left side of shows Acc-2, while the right side indicates the Fl-score.
5 Conclusion and Discussion
In this paper, we introduce Coupled Mamba, a novel approach to enhance multi-modal fusion by leveraging state evolution chains within state space. Our method integrates intermediate information from various modalities, capturing dynamic multi-modal interactions over time. This addresses challenges in parallel SSM with multiple inputs. Both quantitative and qualitative experiments confirm effectiveness Coupled Mamba. Code available at http://ggthubb.oommmssssssw/o/od mamba.
Acknowledgments
This work is supported by the National Natural Science Foundation China (NSFC No. 622721 and No. 62402189), the China Postdoctoral Science Foundation under Grant Number G7C20230894, and the China Postdoctoral Science Foundation (Certificate Number: 2024M751012) The computa- tion is completed in the HPC Platform of Huazhong University of Science and Technology.
10

---

References
[1] Kyu Han Koh, Ashok Basawapatna. Vicki Bennett, and Alexander Repenning. Towards the automatic recognition of computational thinking for adaptive visual language learning. In 2010 IEEE symposium on visual languages and human-centric computing. pages 59-66. IEEE, 2010.
[2] Jean-Bapae Sann-Batistte Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in neural information processing systems, 35:23716-23736, 2022.
[3] Andrew Rouditchenko, Angie Boggust, David Harwath, Brian Chen, Dhiraj Joshi, Samuel Thomas, Kartik Audhkhasi, Hilde Kuehne, Rameswar Panda, Rogerio Feris, et al. Avinet: Learning audio-visual language representations from instructional videos. arXiv preprint MiXiv:2006.09199 2020.
[4] Douwe Kiela, Edouard Grave, Armand Joulin, and Tomas Mikolov. Efficient large-scale mudti-modal classification. In Proceedings of the AAAI conference on artificial intelligence. volume 32, 2018.
[5] Weiyao Wang. Du Tran, and Matt Feiszli. What makes training multi-modal classification networks hard? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12695-12777 16695-12705, 2020.
[6] Jose Dolz, Karthik Gopinath, Jing Yuan, Herve Lombaert, Christian Desrosiers, and Ismail Ben Ayed. Hyperdense-nee
hyper-densely connected enn for multi- image segmentation. IEEE transactions medical inaging. 38(5): 1865):1116-1126, 2018.
[7] Di Feng, Christian Haase-Schütz, Lars Rosenbaum, Heinz Hertlein, Claudius Glaeser, Fabian Timm, Werner Wiesbeck, and Klaus Dietmayer. Deep multi-modal object detection and semantic segmentation for autonomous driving: Datasets, methods, and challenges. IEEE Transactions on Intelligent Transportation Systems,
2020.
[8] Wenmeng Yu, Hua Xu, Ziqi Yuan, and Jiele Wu. Learning mosality-specifie representations with self-supervised multi-task learning for multimodal sentiment analysis. Proceedings of the AAAI Conference on Artificial Intelligence, page 10790-10797, Sep 2022.
[9] Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Phili-p Morency. Tensor fusion network for multimodal sentiment analysis. arXiv: Computation and arrguage,arXiv: Computation and Language, Jul 2017.
[10] Amir Zadeh, Paul Pu Liang, Navonil Mazumder, Soujanya Poria, Erik Cambria, and Louis- Philippe Morency. Memory fusion network for multi-view sequential learning. Proceedings of the AAAI Conference on Artificial Intelligence, Jun 2022.
[11] Devamanyu Hazarika, Roger Zimmermann, and Soujanya Poria. Misa: Modality-invarian and -specific representations for multimodal sentiment analysis. Cornell University arXiv, Cornell University - arXiv, May 2020.
[12] Chaoqun Wang, Chunyan Xu, Zhen Cui, Ling Zhou, Tong Zhang, Xiaoya Zhang, and Jian Yang. Cross-modal Conference on computer vision and pattern recognition, pages
for rgb-t tracking. In Proceedings of the IEEE/CVF
2020.
[13] Yikai Wang, Wenbing Huang, Fuchun Sun, Tingyang Xu, Rong Yu, and Junzhou Huang. Deep multimodal fusion by channel exchanging. Cornell University arXiv, Cornell University arXiv, Nov 2020.
[14) Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Re. Combining recurrent, convolutional, and continuous-time models with linear state space layers. Advances in neural information processing systems, 14:572-583 2021.
[15] S Sundhar Ram, Venugopal V Veeravalli, and Angelia Nedic. Distributed and recursive param- eter estimation in parametrized linear state-space models. IEEE Transactions on Automatic Control, 55(2):488-495 492, 2010.

---

[16] Vincent Verdult, Lennart Ljung, and Michel Verhaegen. Identification of composite local linear space models using projected gradient search. International Journal of Control, 75(16-17)13885-1398 2002.
[17] Albert Gu, Karan Goel, and Christopher Re, Efficiently modeling long sequences with structured state spaces.
[18] Harsh Mehta, Ankit Gupta, Ashok Cutkosky, and Behnam Neyshabur. Long range language modeling via gated state spaces. Jun 2022.
[19] Albert Gu, Karan Goel, Ankit Gupta, and Christopher Re, On the parameterization and initialization of diagonal state space models. Advances Neural Information Processing Systems, 35:35971-35500 -339933, 2022.
[20] Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint arXiv:2401.09417 2024.
[21] Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, and Timothy P Lillicrap. Compressive transformers for long-range sequence modelling. arXiv preprint 11X1v:1911.05507,
[22] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. Dec 2023.
[23] M. Brand, N. Oliver, and A. Pentland. Coupled hidden markov models for complex action recognition. In Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Nov 2002.
[24] Wenmeng Yu, Hua Xu, Fanyang Meng, Yilin Zhu, Yixiao Ma. Jiele Wu, Jiyun Zou, and Kaicheng Yang. Ch-sims: A chinese multimodal sentiment analysis dataset with fine-grained annotation of modality. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3718-377 1118-3777,20 2020.
[25] Yihe Liu, Ziqi Yuan, Huisheng Mao, Zhiyun Liang, Wanqiuyue Yang, Yuanzhe Qiu, Tie Cheng, Xiaoteng Li, Hua Xu, and Kai Gao. Make acoustic and visual cues matter: Ch-sims v2.0 dataset and av-mixup consistent module. Aug 2022.
[26] Pradeep K Atrey, M Anwar Hossain, Abdulmotaleb El Saddik, and Mohan S Kankanhalli. Multimodal fusion for multimedia analysis: survey. Multimedia systems, 16:345-379, 2010.
[27] Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the conference. Association for computational linguistics. Meeting, volume 2019, page 6558. NIH Public Access, 2019.
(28) Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. Cornell University- - arXiv, Cornell University arXiv, Jun 2014.
[29] Guangnan Ye, Dong Liu, I-Hong Jhuo, and Shih-Fu Chang. Robust late fusion with rank minimization. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, Jun 2012.
[30] Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshminarasimhan, Paul Pu Liang, AmirAli Bagher Zadeh, and Louis-Philippe Morency. Efficient low-rank multimodal fusion with -spelity- factors. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), Jan 2018
[31] Dhanesh Ramachandram and Graham W. Taylor. Deep multimodal learning: A survey recent advances and trends. IEEE Signal Processing Magazine, page 96-108, Nov 2017.
[32] Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. Large- scale video classification with convolutional neural networks. In 2014 IEEE Conference Computer Vision and Pattern Recognition, Jun 2014.
12

---

[33] Valentin Vielzeuf, Alexis Lechervy. Stephane Pateux, and Frederic Jurie. Centralnet: multi- layer approach for multimodal fusion. Coreell University arXiv, Cornell University arXiv, Aug 2018.
[34] Chiori Hori, Takaaki Hori, Yeng-Yok Lee, Kazuhiro Sumi, JohnR. Hershey, and TimK. Marks. Attention-basee multimodal fusion for video description. Comell University arXiv,Comnel University arXiv, Jan 2017.
[35] Arsha Nagrani, Shan Yang, Anurag Arna,, Aren Jansen, Cordelia Schmid, and Chen Sun. Attention bottlenecks for multimodal fusion. Dec 2021.
[36] Yingwei Pan, Ting Yao, Yehao Li, and Tao Mei. X-linear attention networks for image captioning. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2020.
[37] Zihui Xue and Radu Marculescu. Dynamic multimodal fusion.
[38] Zongbo Han, Fan Yang, Junzhou Huang, Changqing Zhang, and Jianhua Yao. Multimodal dynamics: Dynamical fusion for trustworthy multimodal classification.
[39] JimmyT.H. Smith, Andrew Warrington, and ScottW. Linderman. Simplified state space layers for sequence modeling. Aug 2022.
[40] Tri Dao. DanielY. Fu, KhaledK. Saab, ArminW. Thomas, Atri Rudra, and Christopher Re. Hungry hungry hippos: Towards language modeling with state space models. Dec 2022.
[41] Shai Fine, Yoram Singer, and Naftali Tishby. The hierarchical hidden markov model: Analysis and applications. Machine learning, 32:41-62, 1998.
[42] Rachel MacKay Altman. Mixed hidden markov models. Journal of the American Statistical Association, 102(477):201-210 2007.
[43] Yuanzhi Wang, Yong Li, and Zhen Cui. Incomplete mulimmoaaliy-ddifuused emotion recognition. Advances in Neural Information Processing Systems, 36, 2024.
[44] Yong Li, Yuanzhi Wang, and Zhen Cui. Decoupled multimodal distilling for emotion recognition. Mar 2023.
[45] Di Wang. Xutong Guo, Yumin Tian, Jinhui Liu, Lihuo Hc, and Xuemei Luo. Tetfin: text enhanced transformer fusion network for multimodal sentiment analysis.
[46] Yao-Hung Hubert Tsai, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency, and Ruslan Salakhutdinov. Learning factorized multimodal representations. In ICLR, 2019.
[47] Wasifur Rahman, Md Kamrul Hasan, Sangwu Lee, AmirAli Bagher Zadeh, Chengfeng Mao, Louis-Philippe Morency, and Ehsan Hoque Integrating multimodal information in large pretrained transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2359-2344 2369, Online, July 2020. Association for Computational Linguistics.
[48] Zhongkai Sun, Prathusha Sarma, William Sethares, and Yingyu Liang. Learning relationships between text, audio, and video via deep canonical correlation for multimodal language analysis. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9992-8999, 2020.
[49] Jennifer Williams, Steven Kleinegesse, Ramona Comanescu, and Oana Radu. Recognizing emotions in video using multimodal dnn feature fusion. In Proceedings of Grand Challenge and Workshop on Human Multimodal Language (Challenge-HMMe pages 11-1. Association for Computational Linguistics, 2018
[50] AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-PPilippe Morency. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for 2018.
Computational Linguistics (Volume 1: Long Papers), pages
13

---

[51] Mark A Van De Wiel, Tonje G Lien, Wina Verlaat, Wessel N van Wieringen, and Saskia M Wilting. Better prediction by use of co-data: adaptive group-regularized ridge regression. Statistics in medicine, 35(3):368-33) -811188,88 2016.
[52] John Arevalo, Thamar Solorio, Manuel Montes-y Gómez, and Fabio A González. Gated multimodal units for information fusion. arXiv preprint 100.v:700..0000. 2017.
[53] D Hong, L Gao. N Yokoya. J Yao, Chanussot, Q Du, and B Zhang More Diverse Means Better. Multimodal deep learning meets remote-sensing imagery classification, 2021, 59. DOI: https:/dooi. org/10. 109/TGRS, pages 4340-4354, 2020.
[54] Tongxin Wang, Wei Shao, Zhi Huang, Haixu Tang, Jie Zhang, Zhengming Ding, and Kun Huang. Mogonet integrates multi-omics data using graph convolutional networks allowing patient classification and biomarker identification. Nature communicationss, 12(1):3445, 2021.
[55] Zongbo Han, Changqing Zhang, Huazhu Fu, and Joey Tianyi Zhou. Trusted multi-view classification. In International Conference on Learning Representations, 2020.
[56] Zongho Han, Fan Yang, Junzhou Huang, Changqing Zhang, and Jianhua Yao. Multimodal dynamics: Dynamical fusion for trustworthy multimodal classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 20707-20717, 2022.
[57] Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshminarasimhann Paul Pu Liang. Amir Zadeh, and Louis-Philippe Morency. Efficient low-rank multimodal fusion with modality-speciinn factors. arXiv preprint arXiv: 1806.00064, 2018.
[58] Siddhant M Jayakumar, Wojciech M Czameeki, Jacob Menick, Jonathan Schwarz, Jack Rae, Simon Osindero, Yee Whye Teh, Tim Harley, and Razvan Pascanu. Multiplicative interactions and where to find them. In International conference on learning representations, 2020.
[59] Itai Gat, Idan Schwartz, Alexander Schwing, and Tamir Hazan. Removing bias in multi-modal classifiers: Regularization by maximizing functional entropies. Advances in Neural Information Processing Systems, 33:3197-3208, 2020.
[60] Zhongkai Sun, Prathusha Sarma, William Sethares, and Yingyu Liang. Learning relationships between text, audio, and video via deep canonical correlation for multimodal language analysis. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 8992-8999, 2020.
[61] Sethuraman Sankaran, David Yang, and Ser-Nam Lim. Multimodal fusion refiner networks. arXiv preprini 3:Xi::2104.0343: 2021.
[62] Zihui Xue and Radu Marculescu. Dynamic multimodal fusion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2575-2584, 2023.
[63] Yuanzhi Wang, Yong Li, and Zhen Cui. Incomplete sigiimodaaityydifffsse emotion recognition. Advances in Neural Information Processing Systems, 36, 2024.
[64] Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livescu. Deep canonical correlation anal- ysis. In Sanjoy Dasgupta and David McAllester, editors, Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pages 1247-1255, Atlanta, Georgia, USA, 17-19 Jun 2013. PMLR.
[65] Weiran Wang, Raman Arora, Karen Livescu, and Jeff Bilmes. On deep multi-view representation learning. In International conference on machine learning, pages 1083-1092. PMLR, 2015.
[66] Hai Pham, Paul Pu Liang, Thomas Manzini, Louis Phillppe Morency, and Barnabás Póczos. Found in translation: Learning robust joint representations by cyclic translations between modalities. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 6892-6899, 2019.
[67] Jinming Zhao, Ruichen Li, and Qin Jin. Missing modality imagination network for emotion recognition with uncertain missing modalities. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2608-2618, 2021.
14

---

[68] Zheng Lian, Lan Chen, Licai Sun, Bin Liu, and Jianhua Tao. Genet: Graph completion network for incomplete multimodal learning conversation. IEEE Transactions on pattern analysis and machine intelligence, 45(7):8419-844 8432, 2023.
15

---

A Appendix supplemental material
The following content is the entire process of pushing to the Coupled Mamba parallelization guarantee. First, we first define the symbols. Assume that the total number of modes is M. AT-1 represents the hidden state at time t-1, where is any mode, A E R represents the state transition matrix, B E R BxLx represents the selective matrix obtained by mapping from the current imput, Ce R B is the same B, where superscript B is the batch size, L is the input time series length, and N is the number of hidden states.
Since the equations of the state space model and Mamba's s core processes, such discretization processing, hardware-aware algorithms, and parallel execution theory, have been discussed in the text, they will not be repeated below.
The core of Coupled Mamba is to introduce multi-modal information while ensuring the parallel computing advantages of Mamba block. After introducing multi-modal information, we have:
M Σ m==
h = Am Gm
h + X,
where Am RBXLLDDN G, E NXN is coupling matrix, which can be understood shared state transition matrix, which transfers the coupling state based on certain probability based on the comprehensive state at time f - 1. By integrating Gm into Am. we can get its unified representation Sm E IR BxLxxx Therefore, the above formula can be expressed as
M
hTth
Sm
X
(10)
m=
Next, let us derive step by step starting from time O, when f we have:
hg = Bm=
(11)
when f = 1, we can get:
M
h/1 S₁
Σ
h B
(12)
m=1
M S2
h2_
Σ
hm
Br
(13)
m=1
The subscripts of S and B represent different modalities, and the superscript of z!" represents different Soddaaities.Thruggee this recursive formula we can get
M
h SM
Σ h + B Ba
(14)
m=1
In the same way, bring h/-1, 1 ... h into the formula for calculating each mode h1 can get:
M S
h
Σ hm + B Biat
(15)
m=l
By disassembling
h hMM we can get:
M
M
1 Si
Σ m=
hm
Σ
Sm
B B + Boxi + .... BaE
(16)
m=1
where
B BiIL + Br + ... + M
(17)
16

---

Let B1x] + B2zt +... BasØl be Up, P= Em=1 Sm Uú can be expressed
U+= = Σ
m
(18)
m=1
Similarly we can get
2 =1 S UU) ) B1
(19)
h S1 (P²U PU; + U2) B1
(20)
Therefore, by recursively recursing this formula, we can get
t-1
hTTh
Sm
P' + Bm=?
(21)
1=0
Finally we calculate the output through
C 2 Em hit, we have
y C U
(22)
where X represents the convolution operation, and the convolution kernel is K (CPo CP1 .P..... (ptt-1cPtt) At this point, we have completed the derivation of the entire parallelized calculation.
In order fully verify the effectiveness of this research method, we further performed classification experiments on the CMU-MOSI. The experimental results are displayed in Table [3] as follows:
Table 13:
Results on the CMU-MOSI dataset for classification task, all results are performed under the same conditions, and the average results are reported after five runs.
CMU-MOSI
Model
Acc-21
Acc-31
F1-Scoret
Fl-Score-3t
EF-LSTM [49]
46.40
74.43
46.55
72.74
Graph-MFN 150
46.13
75.34
46.96
73.71
TFN 9
43.78
73.44
45.55
71.86
LMF 30
45.76
74.11
46.34
72.50
MFN 10
46.68
74.99
46.78
74.22
MulT
27
48.93
74.99
47.29
73.06
MISA 11
44.15
76.30
46.75
74.57
Self-MM
g
51.48
77.74
48.37
76.25
TETEN
45
50.74
77.67
47.88
75.85
Coupled Mamba (Ours)
53.76
78.59
49.21
76.76
The CH-SIMSV2 25 dataset is a Chinese data for multi-modal sentiment analysis and is an extension of the CH-SIMS data set. The dataset contains audio, text, and video clips from different emotion categories, and each clip is labeled with emotional polarity, such as happy, sad, angry, etc. Fach emotion category has corresponding speech, text, and video clips, as well as emotion labels associated with them.
Feature extraction CMU-MOSEI uses the pre-trained BERT model to extract language features and obtains 768-dimensional hidden states as word embeddings. For the visual modality, each video frame is encoded using Facet to represent the presence of total of 35 facial action units. The acoustic model is processed by COVAREP to obtain 74-dimensional features. CH-SIMS uses pre-trained Chinese BERTbase word embeddings to obtain word vectors from text records, and finally represents each word as -o8-dimensional word vector. Acoustie features at 22050Hz were extracted using the LibROSA A speech toolkit with default parameters. A total of 33-dimensional frame-level acoustic features are extracted, Extract aligned faces using MTCNN face detection algorithm. The MultiComp OpenFace2.0 toolkit was then used to extract collection of 68 facial landmarks, 17 facial action
17

---

units, head pose, head orientation, and eye gaze. Finally, a total of 709-dimensional frame-level visual features were extracted.
Exploring Coupled Mamba Layers We investigated the number of layers in Coupled Mamba by performing experiments shown in Table 14 The optimal performance of our Coupled Mamba was observed at layer = 3.
Table 14: Performance on CMU-MOSEI with different layers
CMU-MOSHI
Model
MAET
Corrt
Acc-21
F1-Scoret
Coupled Mamba( Mamba(Layer 1)
55.1
74.7
84.7
84.8
Coupled Mamba( Mamba(Layer = 2)
55.3
74.9
84.9
84.8
Coupled Mamba( Layeer 3)
54.7
75.6
85.6
85.5
Coupled Mamba( = 4)
56.6
74.5
84.8
84.7
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction of this paper accurately reflect the contribution of the paper, such as solving the problem that Mamba cannot be parallelized when using multi-modal input, and verifying the effectiveness of our method.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations in the end of the paper.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
19

---

Answer: [Yes]
Justification: We fully prove the entire derivation process of our algorithm in the appendix. If you want to know more, you can refer to the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
We put the main experimental results in the main text, but in order to fully prove the effectiveness of our method, we also provide some additional experimental results in the appendix. In order to ensure the reproducibility of the experiment, we have set up an Implementation details chapter in the article, including our experimental environment, batch size, loss function and other details.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
20

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We submit experimental code in the supporting material. The environment we use is python 3.10, cuda12.1, torch 2.12. We will indicate how to run the code in the supporting material. For the processing of the data set, we give details in the appendix. We refer to the MMSA library to reproduce the baseline on different data sets, and all experimental environments are the same.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (kttps://nnps.co cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https //nips cc/ cac//pulic//ae mublic/guides:
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We give our training details in detail in the implementation details chapter, such as hyperparameter learning rate, batch size settings, and optimizer selection. In order to be more detailed, we also conducted more detailed comparative experiments on different hyperparameters. This can be viewed in the appendix. We have also discussed the splitting the data set in the data set chapter, including which data sets to use, why, and the sizes of the test set, validation set, and training set. We have explained them in detail.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
21

---

Answer: [No]
Justification: The data we report in the paper is the average data of five report runs, so the error bars not explicitly written in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.: Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We indicate the type of graphics card, CPU type, memory and other information we use in the implementation details, and also provide visualization of memory usage in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics atttp://neurrr: httpeeerrrps..cppuuleeeeeeeeeeee
Answer: [Yes]
Justification: We have read the NeurlPS Code of Ethics and strongly agree with it, and we fully followed the code during our research. We have not violated any guidelines and hope everyone will follow them.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
22

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: Yes, have discussed the positive and negative impacts of our work. Gener- ally speaking, our research has a positive effect on society and can promote human-computer interaction and better enable computers to serve humans life.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairesss considerations (e.g., dep.lymeen of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification: Our research does not design data sets with high risk of misuse, and our model is also very safe. For this reason, do elaborate the corresponding protection measures in the article.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
23

---

Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Any assets have the permission and approval of their original owners, and we fully understand and agree with the protection of any intellectual property rights. We used CC-BY 4.0 Asset license and for the datasets used, we cited them in the References section.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We proposed the Coupled Mamba model, and we have fully explained its detailed architecture in the text. If our paper can be accepted, we will publish the model and code strictly in accordance with the response standards.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/model as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our research does not involve any research or experiments with human subjects, and publicly available data sets for experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
24

---

Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our research did not involve any human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "LyD9qZG6Pd", "reviewer": "Reviewer_JRDw", "summary": "The paper introduces a coupled mamba model for multi-modal fusion. The multi-modal hidden states are fused inside of Mamba, so the current state learns not only from a single modality but the correlation of all modalities. The experiments on multi-modal sentiment analysis show that the proposed model outperforms other baselines.", "questions": "1. The original Mamba uses a parallel scan (recurrent representation) for parallel computing. However, the authors converted Mamba to the convolutional representation. Since eq 6 mainly adds the addition operation (to combine hidden states from different modalities), it should work with the parallel scan. What is the reason to convert to the convolutional representation for parallelization? \n2. The proposed method is compared with average, concat, and mamba fusions in the ablation study, but the explanation of these methods is missing. I assume the average and concat are done directly on the input. How does the mamba fusion work? \n3. What does unaligned/aligned mean in the experiments (Tables 1 and 6)?\n4. Some details about the dataset are missing: \n    - What's the sequence length of each dataset?\n    - What modalities do the datasets contain?", "limitations": "The limitation is discussed as a part of the ethical implications discussion.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The proposed method makes sense, and it outperforms other baselines.\n- The motivation is reasonable.\n-  The ablation study shows that the coupled fusion is better than other common fusion approaches (cross-attention, average, concat, and simple fusion).", "weaknesses": "1. Lack of speed and memory analysis: An important aspect of the proposed method is the memory and speed overhead. It's expected that the overhead is not more than cross-attention (as shown in Figures 3 and 4). However, the paper missed comparisons with other fusion methods, especially the ones that use the inputs directly instead of hidden states. \n\n2. The experiments are only on multi-modal sentiment analysis. I believe the proposed method is pretty general for any multi-modal data and tasks. Showing a variety of applications will make the paper more convincing and appealing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough and insightful feedback. We genuinely appreciate the effort and expertise you invested in reviewing our paper.\n\n**W1: Lack of speed and memory analysis**\n\nWe thank the reviewer for this insightful feedback. In our rebuttal PDF, we have included a comparison of memory usage(**Fig 3**), inference speed(**Fig 2**), flops(**Fig 1**), and parameter size (**Table 3**) with Mult[1], MISA[2], TFN[3], and LMF[4], all using inputs directly instead of hidden states. Additionally, Tab. 3 presents a detailed comparison of parameters. From both Fig. 1 and Tab. 3, it is evident that our Coupled Mamba consumes significantly less memory and achieves the fastest inference speed for sequences from short to long.\n\n**W2: Expriments on more multi-modal data and tasks to demonstrate generalizability**\n\nThanks for this helpful suggestion! We conducted additional experiments on the **BRCA benchmark** and **MM-IMDB benchmark**, and the results are presented in **Tab 1,2** in our rebuttal PDF. The BRCA benchmark focuses on predicting the PAM50 subtype classification of breast cancer from mRNA, DNA, miRNA data, while the MM-IMDB dataset is used for movie genre classification from text and images. Despite the differences in data modalities and tasks, our Coupled Mamba achieves the best performance on both benchmarks compared to existing SOTA methods, demonstrating its generalizability.\n\n**Q1: Reason of converting to the convolutional representation for parallelization**\n\nThe parallel scanning scheme in Mamba computes independent partial states recurrently from inputs and then combines these partial states using a global convolution in a hierarchical manner to achieve parallelism. In our Coupled Mamba, we adapt this approach by conducting recurrently computation of partial states within each modality. We enhance this process with a summation operation through a global convolutional kernel, enabling multi-modal fusion capabilities while preserving the inherent advantages of parallelism. To be more specific, the parallel scanning process generates intermediate results, which we fuse using a summation scheme with a global convolutional kernel across modalities. This scheme is inherently parallelizable and accelerates the fusion of multimodal data. We will clarify this point further in the revision.\n\n**Q2: Explanation of *average*, *concat*, and *mamba* fusions in the ablation study**\n\n**average** fusion involves directly averaging the multi-modal features. **concat** fusion concatenates the multi-modal features. The fused feature is then fed into a Mamba model, followed by a pooling and head layer for the final output. For **Mamba fusion**, we create a separate Mamba branch for each modality. The output features of each branch are then combined using a weighted summation (following late fusion schemes as in LMF, Mult). We will clearify this in revision.\n\n**Q3: The meaning of *unaligned* and *aligned* in experiments**\n\n**aligned** data refers to data from different modalities that are precisely synchronized in both spatial and temporal dimensions, such as synchronized audio and video. In contrast, **unaligned** data refers to instances where different modalities are not synchronized, presenting a more challenging scenario. Our method demonstrates superior performance on both aligned and unaligned data.\n\n**Q4: Some details about the dataset are missing**\n\nOur experiments utilize datasets comprising data from **video, audio, and text modalities**. The CMU-MOSEI dataset includes both aligned and unaligned data, with sequence lengths of 50 for aligned text, audio, and video, and 50, 500, and 375 for unaligned text, audio, and video, respectively. The CH-SIMS and CH-SIMSV2 datasets contain exclusively unaligned data, with sequence lengths of 39, 400, and 55 for text, audio, and video in CH-SIMS, and 50, 925, and 232 in CH-SIMSV2.\n\n[1] Tsai, Yao-Hung Hubert, Shaojie Bai, Paul Pu Liang, J. Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019. “Multimodal Transformer for Unaligned Multimodal Language Sequences.” In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. doi:10.18653/v1/p19-1656.\n\n[2] Hazarika, Devamanyu, Roger Zimmermann, and Soujanya Poria. 2020. “MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis.” Cornell University - arXiv,Cornell University - arXiv, May.\n\n[3] Zadeh, Amir, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2017. “Tensor Fusion Network for Multimodal Sentiment Analysis.” arXiv: Computation and Language,arXiv: Computation and Language, July.\n\n[4] Liu, Zhun, Ying Shen, Varun Bharadhwaj Lakshminarasimhan, Paul Pu Liang, AmirAli Bagher Zadeh, and Louis-Philippe Morency. 2018. “Efficient Low-Rank Multimodal Fusion with Modality-Specific Factors.” In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). doi:10.18653/v1/p18-1209.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NpQJx7hFEz", "reviewer": "Reviewer_pbQ3", "summary": "This work proposes Coupled SSM (State Space Models) to fuse multiple modalities effectively with SSM. Instead of fusing multi-modal features directly, the proposed method couples state chains of multiple modalities while maintaining the independence of intra-modality state processes. Specifically, they first propose an inter-modal hidden states transition scheme to fuse multiple modalities effectively. Then, they propose an expedited coupled state transition scheme to adapt the hardware-aware parallelism of SSMs for efficiency.  Experimental results on classification task on CMU-MOSEI, CH-SIMS, CH-SIMSV2 show promising performance of the proposed method.", "questions": "Please refer to the weaknesses.", "limitations": "The discussions about the limitations of this work would be further improved.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method couples the state chains of multiple modalities while maintaining the independence of intra-modality state processes.\n2. The proposed method is more memory efficient compared to cross-attention with the increasing sequence length.\n3. Results on several benchmark datasets show the superior performance of the proposed method.", "weaknesses": "1. The improvement in the regression task on CMU-MOSEI seems marginal compared to the baselines.\n2. In line 201, the authors conclude that the results in Tables 3 and 4 show the SOTA performance of the proposed method regardless of whether the data are aligned or not. However, both results in Tables 3 and 4 are performed on unaligned data. Besides, the robustness of the proposed method needs further verified, since I cannot find the results that support the claims about robustness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed and insightful comments and feedback, and we are deeply grateful for the time and expertise you dedicated to reviewing our paper..\n\n**W1: The improvement in the regression task on CMU-MOSEI seems marginal**\n\nThe CMU-MOSEI dataset mostly consists of short sequences: most of the videos, audios, and texts are only 1-5 seconds in duration. Mamba is more suitable for modeling relatively long sequences, which results in marginal improvement of Coupled Mamba on the CMU-MOSEI dataset. In contrast, for datasets with longer sequences, such as data in CH-SIMS, CH-SIMSV2, BRCA, and MM-IMDB datasets, our Coupled Mamba can effectively enhance multi-modal fusion and outperforms the state-of-the-arts by a relatively large margin. We will include a discussion on the impact of sequence lengths in the limitations section.\n\n**W2: Less experiments on aligned data and evidences of robustness:**\n\nThe fewer experiments on 'aligned' data are influenced by the composition of the datasets used, where there is a predominance of unaligned data. This is reasonable, as aligned data is more difficult to collect. Moreover, many existing classification methods for the CMU-MOSEI dataset primarily focus on unaligned data; hence, we use unaligned data for a fair comparison. Besides, experiments on aligned data in the main paper Table 1 have demonstrated our excellent performance for aligned data input. Furthermore, unaligned data is more challenging to process, and our approach achieves superior performance regardless of aligned or unaligned data.\n\nFor robustness, we have conducted experiments on aligned/unaligned data (Table 1 of the main paper), various modalities (text, audio, video, mRNA, miRNA, DNA, as shown in Tables 1 and 2 in the rebuttal PDF), sequence lengths (ranging from 100 to 500), and text in different language domains (English vs. Chinese). All experiments show that our Coupled Mamba achieves superior performance, demonstrating its robustness across various data input.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S7eQjpB13v", "reviewer": "Reviewer_FSJZ", "summary": "The paper addresses the challenge of multi-modal fusion in deep learning. Current fusion methods struggle to capture complex intra- and inter-modality correlations. Recent state space models like Mamba show promise but are limited in fusing multiple modalities efficiently. The paper propose Coupled Mamba, key aspects include: A coupled state transition scheme that allows information exchange between modalities while maintaining individual propagation. A state summation and transition mechanism to enable parallel computing with multiple inputs. Derivation of a global convolution kernel for efficient parallelization.\nThe model architecture consists of multiple Coupled Mamba blocks, each processing different modalities and aggregating states before transitioning to the next time step.\n\nExperiments were conducted on three multi-modal sentiment analysis datasets: CMU-MOSEI, CH-SIMS, and CH-SIMSV2. \nResults show improved performance over existing methods as well as faster inference and GPU memory savings compared to transformer-based approaches. Effective performance on both aligned and unaligned multi-modal data.", "questions": "1. In table 9, what does Mamba Fusion mean?\n\n2. Can the authors provide more interpretability behind combining to mamba structure and explain how the modalities interact between the two mamba?", "limitations": "The authors have addressed the limitations", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The method demonstrates consistent performance and good efficiency compared with existing works.\n2. The paper provides a detailed mathematical derivation of the coupled state transition process,\n3. The idea of coupling mamba is novel.", "weaknesses": "The authors seem to have combined two Mamba models without providing enough insights into how this combination actually improves multi-modal fusion. \n\nThere's no clear analysis or interpretability of what the model is learning or how it's fusing information across modalities. This makes it difficult to understand why their approach is fundamentally better than existing methods, including transformers. \n\nThe speed and linear complexity benefits are inherited from the original mamba model, so they are apparent advantages for all state space-related works. \n\nThe experiments are limited to sentiment analysis tasks. The generalizability of the method to other multi-modal tasks or domains is not demonstrated or discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed and insightful comments and feedback, and we are deeply grateful for the time and expertise you dedicated to reviewing our paper..\n\n**W1: Provide insights over combining Mamba branches for multi-modal fusion**\n\nEffective multi-modal fusion hinges on balancing inter-modality information exchange while preserving independent intra-modality learning [1,2]. Our approach, Coupled Mamba, achieves this by integrating historical information across modalities while maintaining intra-modal state autonomy. Specifically, Coupled Mamba fuses the state of each modality's chain with adjacent chains from previous time steps across modalities. This integration ensures that each modality's current memory incorporates crucial historical context from multimodal data, progressively building a comprehensive model over time. We will make this point more clear.\n\n**W2: Lack analysis or interpretability of the model's multi-modal fusion mechanism**\n\nThanks for the insightful feedback! In our Coupled Mamba, we use a separate Mamba branch for each modality, and fuses the state of each modality's chain with adjacent chains from previous time steps using a learnable state transition matrix ($A_{i, m}$ in Eqn. 5 and $S_{m}$ in Eqn. 6) to ehance multi-modality fusion. The state transition matrix selectively integrates important historical information from other modalities, while neglects information that's insignificant. By utilizing the coupling transition  between states, Coupled Mamba is tightly connected in time steps, effectively modeling multimodal data. \n\nThis mechanism is fundamentally different from existing multimodal fusion schemes, where existing fusion methods can be categorized into \n1. **attention mechanisms.** this method achieves multimodal information fusion through cross attention mechanisms. \n2. **the encoder decoder method.** This method extracts high-level features of different modalities through an encoder, maps them to a low dimensional space, and then uses a decoder to generate predictions from these latent representations. \n3. **Graph neural network method.** This method models multimodal data by constructing graph structured data. \n\nThese designs all focus on learning good multi-modality features. In constrast, Coupled Mamba ensures intra-modal indenpence (as we use separate Mamba branches for each modality), and fuse important information from multi-modalities using the learnable state transition scheme, which emphasize more on the information fusion process. We believe that's the reason why our approach outperforms across all multi-modal tasks in experiments.\n\nWe will explain this point clearer in the revision.\n\n**W3: The speed and linear complexity advantage is from Mamba**\n\nWe only partially inherit the speed and linear complexity advantage of the original Mamba, by preserving the parallel scan scheme within each modality. However, the original Mamba and other state-space models were originally designed to process unimodal data and cannot handle cross-modal fusion well. When naively adapted for multi-modal data, as indicated by the 'concat' and 'mamba' fusion approach shown in Table 9 in the main paper, they require higher memory and computational resources, resulting in inferior performance compared to our coupled design.\n\n**W4: Experiments are limited to sentiment analysis tasks**\n\nThanks for this helpful suggestion! We conducted additional experiments on the **BRCA benchmark** and **MM-IMDB benchmark**, and the results are presented in **Table 1,2** in our rebuttal PDF. The BRCA benchmark focuses on predicting the PAM50 subtype classification of breast cancer, based on three complex data modalities, i.e., the mRNA expression, DNA methylation and miRNA. while the MM-IMDB dataset is used for movie genre classification from movie intro, images. Despite the differences in data modalities and tasks, our Coupled Mamba achieves the best performance on both benchmarks compared to existing SOTA methods, demonstrating its generalizability.\n\n**Q1: Meaning of \"mamba fusion\" in Table 9**\n\n\"mamba fusion\" means establishing a seperate Mamba branch for each modality. Subsequently, the output features of these Mamba branches are merged through a weighted aggregation mechanism, consistent with the late-fusion paradigm embodied by methods such as LMF and Mult. We will further elaborate on this process in the revision.\n\n\n**Q2: Provide more interpretability of combining Mamba structures**: \n\nPlease refer to our response to **W1**\n\n[1] Wang, Yikai, Wenbing Huang, Fuchun Sun, Tingyang Xu, Rong Yu, and Junzhou Huang. 2020. \"Deep Multimodal Fusion by Channel Exchanging.\" NeurIPS 2020.\\\n[2] Li, Yaowei, Ruijie Quan, Linchao Zhu, and Yi Yang. n.d. \"Efficient Multimodal Fusion via Interactive Prompting.\" CVPR 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QKc2XuT6ts", "reviewer": "Reviewer_qhp2", "summary": "This paper proposes the coupled mamba to address the problem of multimodal data fusion. The core architecture of the coupled mamba is derived in the form of Equation (6) by improving upon Equation (5). Extensive ablation experiments validate the effectiveness of the proposed coupled mamba.", "questions": "1、Why is $S_o$ consistent with the form in SSM, rather than being obtained through input? What is its relationship with the probability transition matrix in Coupled Hidden Markov Models?\\\n2、What is the relationship between Formula 5 and the probability transition matrix in CHMH? Are they merely similar in form?\\\n3、In Formula 5, the term $\\bar{A}_{m,m}$ should differ from other $\\bar{A}$ terms, as it is most relevant to the current state $h_t^m$. Has this aspect been considered in this paper?\\\n4、Are there relevant ablation experiments to verify the validity of the simplification from Equation (5) to Equation (6)?", "limitations": "Please refer to Weaknesses and Limitations", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The article's logic is clear, the charts are coherent, and the proposed method alleviates the shortcomings of the original Mamba in handling multimodal tasks, contributing to the community.", "weaknesses": "1、I noticed that some of the references in the article are incomplete and inconsistent in format. Please handle this carefully. \\\n2、Why does $S_o$ in Algorithm 1 have the same form as in SSM? Is it reasonable to keep it consistent with SSM?\\\n3、The relationship between Equation (5) and the probability transition matrix in CHMH seems insignificant, although they are similar in form.\\\n4、The Equation (6) is a clever maneuver, but directly summing indicates that the coefficients in front of the hidden states of each modality are all 1. Is this design reasonable? Moreover, there is a lack of corresponding ablation experiments to validate this design.\\\n5、In Algorithm 1, the dimensions of $S_o$ : (B, L, E, N) and those of $S_m$ mentioned on page five are inconsistent.\\\n6、Some experiments, such as Table 1 and Table 2, lack corresponding inference time, FLOPs, or parameter data.\\\n7、In the Introduction, there is an extra space in 'Transformer-based'.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed and insightful comments and feedback, and we are deeply grateful for the time and expertise you dedicated to reviewing our paper..\n\n**W1: Incomplete and inconsistent references**\n\nWe thank the reviewer for pointing out this issue. We will carefully proofread and correct all incorrect references, formatting issues, and typos.\n\n**W2: The form of $S_0$ is the same with SSM** \n\nIn our Coupled Mamba, $S_0$ is obtained by multiplying the input-related $\\overline{A}$ and the devised coupling matrix $G$. Please refer to the detailed proof in the appendix. It can be interpreted as a shared state transition matrix that transfers the coupling states based on a certain probability derived from the comprehensive state at time $t-1$. We use this same form for consistency with SSMs for unification and better interpretability.\n\n**W3: The relationship between Eqn. 5 and the probability transition matrix of CHMM**\n\nThank you for the valuable feedback. CHMM has been widely adopted for multi-modal fusion [1, 2], utilizing the probability transition matrix to integrate information from multiple modalities. Our design of Equation 5 is inspired by CHMM, as the state propagation sequences in SSM resemble the state chains in CHMM. Equation 5 can also be interpreted as the probability transition between continuously evolving states, analogous to the transition of discrete states in CHMM.\n\n**W4: Eqn. 6 sets weights of hidden states from all modalities to 1** \n\nIn our implementation, we use a learnable scalar to weight each modality in Eqn. 6. This has not been reflected in the paper, and we will clarify this point in the revision. Additionally, in response to your suggestion, we conducted a comparative experiment by setting the weights of all historical states to 1. The results are shown in Table 5 (in the PDF). By incorporating a learnable scalar to weight each modality in the equation, we address the issue of varying contributions from different modalities to the task, thereby improving the performance of multimodal fusion.\n\n**W5: Inconsistent dimensions of $s_0$ and $s_m$:**\nThanks for pointing this issue out. Sorry for the typos. The correct dimensions for $S_{m}$ are [B,L,E,]. We will correct in the revision.\n\n**W6: Lack corresponding inference time, FLOPs, or parameter data in Table 1**\nWe apologize for the missing information. In addition to providing only the inference time, FLOPs, or parameter data, we further compared our Coupled Mamba with several SOTA methods on inference time, FLOPS, and parameters. The results are shown in Fig. 2, Fig. 1, and Tab. 3 of the rebuttal PDF. Additionally, we also compared memory usage in Fig. 3. Our method exhibits the lowest memory consumption and fastest inference speed.\n\n**W7: In the Introduction, there is an extra space in 'Transformer-based** \n\nThank you for your careful review. We will make revisions in future versions.\n\n**Q1**: Please refer to the reponse to **W2**.\n\n**Q2**: Please refer to the reponse to **W3**.\n\n**Q3:** $\\bar{A}_{m,m}$ should differ from $\\bar{A}$.\n\nThank you for your suggestion and feedback. Note that all $\\bar{A}{i,m}$ $, i \\in 1 \\ldots M$ in Eqn. 5 are learned parameters during network training. To preserve the generalizability of the model, we do not specifically emphasize the intra-modal transition $\\bar{A}_{m,m}$, and instead let the network learn to prioritize modalities.\n\n**Q4: Ablation experiment on simplification from Eqn. 5 to Eqn. 6**\n\nThank you for the valuable suggestion. We have conducted ablation study follow your suggestion, as illustrated in **Table 4** in the rebuttal PDF. The results indicate that this simplification reduces both the number of parameters and memory usage, and also improves inference speed, with minimal impact to the performance.\n\n[1]Garg A, Naphade M, Huang T S. Modeling video using input/output Markov models with application to multi-modal event detection[J]. Handbook of Video Databases: Design and Applications, 2003: 23-44.\n\n[2]ZHANG Yu-zhen,DING Si-jie,WANG Jian-yu,DAI Yue-wei,CHEN Qian.Event Detection by Fusing Multimodal Objects Using HMM[J].Journal of System Simulation,2012,24(8):1638-1642.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "77qpdapT09", "reviewer": "Reviewer_nXDG", "summary": "This paper extends the state space model Mamba into the multi-modal domain. The authors propose utilizing separate Mamba blocks to process each modality and suggest conditioning the state of each modality on the others to facilitate modality interaction. They further introduce a parallelism technique for the coupled Mamba transition. The framework is evaluated on three benchmark datasets.", "questions": "1. In Table 9, does 'average (concat) fusion' means averaging (concating) the features from all modality and then go through a single mamba block?", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The attempt to extending Mamba model to multi-modal domain is a motivating direction, which may facilitate the efficiency of future multimodal foundation model.\n\n2. The model achieved good evaluation results on 3 benchmark datasets, while reducing the GPU memory usage to linear cost.\n\n3. Detailed mathematical derivation of the new global convolutional kernel is provided.", "weaknesses": "1. The demonstration of the parallelism is not clearly illustrated. The original Mamba model's essence lies in the selection mechanism and the hardware-aware parallel scan. The parallelism achieved through the global convolution kernel in S4 is no longer as effective in the Mamba framework, given that the transition matrices at each time step are conditioned on the input. This inspired the introduction of the parallel scan method in the Mamba paper. While the authors show that coupled Mamba can be computed using a global convolution, the question remains, how is the parallelism achieved in the context of Mamba's scanning process?\n\n2. The idea of using a sequential model to model each modality and then employing a simple interaction scheme (in this paper, sum) to enable modality fusion is straightforward, but maybe too simple to model the complex modality interaction.\n\n3. Comparison with existing multi-modal Mamba models, for instance, [1-2], should be presented.\n\n[1] VL-Mamba: Exploring State Space Models for Multimodal Learning\n[2] Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inference", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed and insightful assessment of our paper; we are deeply grateful for the time and expertise you dedicated.\n\n**W1: Parallelism through global convolution:** \n\nS4 relies on a global convolution kernel to achieve parallelism, while Mamba iterates on Equation 3 to compute intermediate results, ultimately accelerating computation using a global convolution kernel. In our Coupled Mamba, we retain the parallel scanning scheme within each modality. During the parallel scanning process, we iteratively combine intermediate results from each modality, integrating historical states across modalities. Simultaneously, we derive a multimodal global convolution kernel to fuse results from intra-modal parallel scanning. This design ensures that computations for each convolution kernel and input subregion remain independent, facilitating parallel processing of different convolution kernels or input blocks.\n\n**W2: Summation is too simple to model complex modality interaction** \n\nOur method utilizes the state transition matrix $S_m$ to evolve the dynamic process of multimodal fusion. While summing up historical states might seem straightforward, our model effectively filters historical information from different modalities through the coupling matrix $G$, retaining only the filtered, crucial information. Additionally, we introduce self-learning scalars that enable the model to adaptively learn the contribution of each modality to the task. Numerous experiments have demonstrated that, compared to other widely used fusion schemes such as averaging and cascading, our summation scheme not only enhances speed but also delivers superior performance.\n\n**W3: Comparison with other multi-modal Mamba models** \n\nThank you for suggesting the concurrent multimodal Mamba work, including VL-Mamba and Cobra. Both VL-Mamba and Cobra are based on large language models (LLM) and are designed for VQA tasks. Our Coupled Mamba introduces Mamba to general multimodal tasks instead. In addition, VL-Mamba uses VSS Block to extract features from visual input, and then aligns textual features with visual features for fusion and inputs them into LMM for training. Cobra uses two visual models, DINOv2 and SigLIP, to extract visual features, and then aligns visual features with textual features for multimodal fusion. The Mamba models in both VL-Mamba and Cobra serve merely as feature extractors, and the multi-modal feature fusion is conducted within a LLM. They do not present any new schemes for multi-modal fusion. We will add both qualitative and quantitative comparisons with other concurrent multimodal Mamba works, including the proposed VL-Mamba and Cobra in the revision.\n\n**Q1: meaning of 'average (concat) fusion'** \n\nYes, 'average fusion' directly averages multimodal features before processing them with a single Mamba, while 'concat fusion' concatenates multimodal features. We will make this point clearer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their insightful feedback and appreciate the time and expertise they have dedicated to evaluating our work. We are encouraged by the positive comments highlighting the strengths of our approach: "superior and consistent performance" (JRDw, pbQ3, FSJZ, qhp2, nXDG), "demonstrates good efficiency" (pbQ3, FSJZ, nXDG), "clear logic & motivating direction" (JRDw, qhp2, nXDG), "novel idea" (FSJZ), and "detailed mathematical derivation" (FSJZ, nXDG).


Reviewers expressed concerns regarding the lack of speed and memory analysis (JRDw, FSJZ, qhp2), and the need for experiments on other multi-modal tasks besides sentiment analysis (JRDw, FSJZ), lack ablation on Eqn. 5 and Eqn. 6 (qhp2). In response, we have conducted additional experiments and included detailed figures and tables in the rebuttal PDF to address these concerns, coupled with our reviewer-specific replies.


**Summary of figures and tables in the rebuttal PDF in response to reviewers' feedback:**

**Table 1**: Experimental results on the **BRCA benchmark**, classifying breast cancer PAM50 using mRNA (mR), DNA methylation (D), and miRNA (miR) expression data.

**Table 2**: Experimental results on the **MM-IMDB Benchmark**, classifying movie categories using image (I) and text (T) modalities. Our method achieved the best results in both MicroF1 and MacroF1 indicators, with a **2.06%** improvement in MicroF1.


**Table 3**:  Comparison of parameters across different methods under identical conditions (same number of layers, hidden dimensions, etc.), demonstrating Coupled Mamba's superior performance with reduced complexity.

**Figure1, 2, and 3**: Showing FLOPs, Inference Time, and Memory Usage across varying sequence lengths for different models. Coupled Mamba exhibits lower computing resource requirements, faster inference speed, and significantly reduced memory usage.

**Table 4**: Comparison between Eqn. 5 and its simplified version Eqn. 6, showing effective reduction in model parameters, inference time, and memory usage with our simplification.

**Table 5**: Results comparing fixed weights (all modalities set to 1) and adaptive scalar learning weights in Eqn. 6 on the CH-SIMS dataset. Adaptive learning scalars improve model performance by enabling nuanced multimodal fusion, learning varying contributions of modalities to the task.

**Here are some articles that supplement the baseline comparisons in the experiments:**\
**CRidge** : Van De Wiel M A, Lien T G, Verlaat W, et al. Better prediction by use of co‐data: adaptive group‐regularized ridge regression[J]. Statistics in medicine, 2016, 35(3): 368-381.\
**GMU** : Arevalo J, Solorio T, Montes-y-Gómez M, et al. Gated multimodal units for information fusion[J]. arXiv preprint arXiv:1702.01992, 2017.\
**CF**: Hong D, Gao L, Yokoya N, et al. More diverse means better: Multimodal deep learning meets remote-sensing imagery classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 59(5): 4340-4354.\
**MOGONET**: Wang T, Shao W, Huang Z, et al. MOGONET integrates multi-omics data using graph convolutional networks allowing patient classification and biomarker identification[J]. Nature communications, 2021, 12(1): 3445.\
**TMC** : Han Z, Zhang C, Fu H, et al. Trusted multi-view classification[C]//International Conference on Learning Representations. 2020.\
**MM-Dynamics** : Han Z, Yang F, Huang J, et al. Multimodal dynamics: Dynamical fusion for trustworthy multimodal classification[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 20707-20717.\
**LRMF** : Liu Z, Shen Y, Lakshminarasimhan V B, et al. Efficient low-rank multimodal fusion with modality-specific factors[J]. arXiv preprint arXiv:1806.00064, 2018.\
**MFM** : Tsai Y H H, Liang P P, Zadeh A, et al. Learning factorized multimodal representations[J]. arXiv preprint arXiv:1806.06176, 2018.\
**MI-Matrix** : Jayakumar S M, Czarnecki W M, Menick J, et al. Multiplicative interactions and where to find them[C]//International conference on learning representations. 2020.\
**RMFE** : Gat I, Schwartz I, Schwing A, et al. Removing bias in multi-modal classifiers: Regularization by maximizing functional entropies[J]. Advances in Neural Information Processing Systems, 2020, 33: 3197-3208.\
**CCA** : Sun Z, Sarma P, Sethares W, et al. Learning relationships between text, audio, and video via deep canonical correlation for multimodal language analysis[C]//Proceedings of the AAAI conference on artificial intelligence. 2020, 34(05): 8992-8999.\
**RefNet** : Sankaran S, Yang D, Lim S N. Multimodal fusion refiner networks[J]. arXiv preprint arXiv:2104.03435, 2021.\
**DynMM** : Xue Z, Marculescu R. Dynamic multimodal fusion[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 2575-2584.

Meta-review: This work proposes Coupled SSM (State Space Models) to fuse multiple modalities effectively with SSM. Instead of fusing multi-modal features directly, the proposed method couples state chains of multiple modalities while maintaining the independence of intra-modality state processes. Specifically, they first propose an inter-modal hidden states transition scheme to fuse multiple modalities effectively. Then, they propose an expedited coupled state transition scheme to adapt the hardware-aware parallelism of SSMs for efficiency. Experimental results on classification task on CMU-MOSEI, CH-SIMS, CH-SIMSV2 show promising performance of the proposed method.
Decision: Accept (poster)