EMMA: Empowering Multi-modal Mamba with Structural and Hierarchical Alignment

Authors: Yifei Xing, Xiangyuan Lan, Ruiping Wang, Dongmei Jiang, Wenjun Huang, Zheng Qingfang, Yaowei Wang
Primary area: foundation or frontier models, including LLMs
Keywords: Multimodal models, State space models, Efficient architectures, Mamba, Computational Efficiency, Multimodal Alignment

Abstract:
Mamba-based architectures have shown to be a promising new direction for deep learning models owing to their competitive performance and sub-quadratic deployment speed. However, current Mamba multi-modal large language models (MLLM) are insufficient in extracting visual features, leading to imbalanced cross-modal alignment between visual and textural latents, negatively impacting performance on multi-modal tasks. In this work, we propose Empowering Multi-modal Mamba with Structural and Hierarchical Alignment (EMMA), which enables the MLLM to extract fine-grained visual information. Specifically, we propose a pixel-wise alignment module to autoregressively optimize the learning and processing of spatial image-level features along with textual tokens, enabling structural alignment at the image level. In addition, to prevent the degradation of visual information during the cross-model alignment process, we propose a multi-scale feature fusion (MFF) module to combine multi-scale visual features from intermediate layers, enabling hierarchical alignment at the feature level. Extensive experiments are conducted across a variety of multi-modal benchmarks. Our model shows lower latency than other Mamba-based MLLMs and is nearly four times faster than transformer-based MLLMs of similar scale during inference. Due to better cross-modal alignment, our model exhibits lower degrees of hallucination and enhanced sensitivity to visual details, which manifests in superior performance across diverse multi-modal benchmarks. Code provided at https://github.com/xingyifei2016/EMMA.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730084776728

thread_type:
official_review

<thread_object>
creator: reviewer_yy1f

<content>
summary:
Mamba-based architectures are promising for deep learning models but current Mamba multi-modal large language models (MLLM) are insufficient in extracting visual features. This paper proposes Empowering Multi-modal Mamba with Structural and Hierarchical Alignment (EMMA). It includes a pixel-wise alignment module for structural alignment at the image level to extract fine-grained visual information autoregressively. Additionally, a multi-scale feature fusion (MFF) module is proposed for hierarchical alignment at the feature level to prevent the degradation of visual information. Extensive experiments on various multi-modal benchmarks show that the model has lower latency than other Mamba-based MLLMs and is nearly four times faster than transformer-based MLLMs of similar scale during inference.

strengths:
1. The imbalance in the quality of visual and texture latent in the Mamba-based VLM does make sense to investigate and this paper proposes an effective method, named Empowering Multi-modal Mamba with Structural and Hierarchical Alignment (EMMA) to solve this problem.

2. The experiments are sufficient to verify the effectiveness of the proposed method.

3. The paper is well-written and easy to follow.

weaknesses:
1. EMMA essentially enhances the visual representation ability of the model. Therefore, the comparison with peers should be conducted, e.g. the contrastive learning in CLIP and SigLIP. Besides, does the masked image construction loss achieve a function similar to EMMA? 

2. EMMA uses the visual features from the vision encoder as the target of the Pixel-wise Alignment Loss. However, visual features from the vision encoder may lose the fine-grained information. Does this problem affect the performance of EMMA?

3. High-resolution image is an important direction for MLLM. Structural constraints on the high-resolution visual features are an interesting point to discuss. Authors are suggested to conduct the experiment under a high-resolution setting.

4. Authors are suggested to display more qualitative results.

questions:
Please see the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730565403325

thread_type:
official_review

<thread_object>
creator: reviewer_t8kc

<content>
summary:
The paper introduces Empowering Multi-modal Mamba with Structural and Hierarchical Alignment (EMMA), enhancing Mamba multi-modal large language models (MLLM) by improving their ability to extract fine-grained visual information. EMMA uses a pixel-wise alignment module for better structural alignment and a multi-scale feature fusion (MFF) module to maintain visual information during cross-modal alignment. Extensive experiments demonstrate that EMMA achieves lower latency and faster performance compared to other Mamba-based MLLMs, with superior cross-modal alignment and reduced hallucination. The code for this model will be made available.

strengths:
1. Well Written: The paper is clearly articulated, making the complex concepts accessible and easy to understand for the reader.

2. Motivation Well Clarified: The authors firstly identify the imbalance in the quality of visual and textual latent features in MLLM Mamba models. They propose a pixel-wise alignment approach to autoregressively enhance the learning and processing of structured visual features, leading to improved cross-modal alignment between visual and textual latents.

3. Experiments Solid: The paper presents comprehensive experiments conducted on a variety of multi-modal benchmarks. These experiments include thorough comparisons with current state-of-the-art models based on both Mamba and transformer architectures, demonstrating the robustness of the proposed approach.

weaknesses:
1. Why is there an imbalance in the quality of visual and textual latents in the Mamba LLM, where coarse visual features are ineffectively aligned with higher-quality textual features? How does this differ from transformer-based models? Do transformer-based models face the same issue? 

2. Could you provide a more objective, quantitative analysis of the loss of fine-grained visual cues in the LLM, as the current visualizations with just a few images seem insufficient? Is this phenomenon common across MLLMs, or is it specific to the use of the Mamba LLM?

3. Why do the current results show that the Mamba LLM still lags behind the transformer architecture? In this case, does the speed advantage gained by using the Mamba architecture sufficiently compensate for the performance loss?

questions:
1. Currently, models use ViT as the image encoder. Is it possible to build an MLLM entirely based on Mamba? What potential advantages and challenges might this approach entail?

2. In transformer-based LLMs, is there a similar issue with pixel-level alignment? Would the method proposed in the paper also be applicable to transformer architectures?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730681604998

thread_type:
official_review

<thread_object>
creator: reviewer_5nfq

<content>
summary:
This paper proposes a Mamba based integration for vision-language models. Similar to the VLM based framework where there are one vision encoder, one projector, and one LLM. The proposed method takes one image and its corresponding caption as input, followed by multi-scale feature extraction and fusion. A pixel-wise alignment loss is applied to the visual features for structural visual cues preservation, and an autoregressive NLP loss is applied to the textural feature. Experiments are conducted on several benchmark datasets.

strengths:
1. Pixel-wise alignment loss is interesting. The LLM outputs are text tokens and are further decoded into the image domain for measuring pixel-level similarity with the input image. 
2. The hierarchical alignment via multi-scale fusion is developed to retain fine-grained visual features
3. The experimental results seem promising on the benchmark datasets.

weaknesses:
1. While the main contribution is set for the loss design of model training, the proposed loss function seems rather general and not specifically for Mamba based structure. The general pixel-wise alignment loss seems functional for all VLMs, rather than only mamba structure. Can you clarify how the proposed loss function leverages or is tailored to the unique properties of Mamba architectures compared to transformer-based VLMs?
2. The inputs require both images and captions, how the captions are acquired are not illustrated well. A more detailed description of the data preparation process is desired here. Also, the decoded procedure via Mamba is not illustrated clearly. Is SFT or a similar finetuning process still needed for training VLM? Is related VQA data still required? Such technical details are not shown clearly in the manuscript to show the overall model tuning picture.  It is better to provide a step-by-step explanation of the decoding procedure and clarification on the fine-tuning process to help address the ambiguity.
3. The pixel-alignment loss is set for structure preservation, the role of LLM here is not clear as the caption has been sent to it. So LLM seems to function as a single mapping to fulfill this objective loss. Can you elaborate on the specific role of the LLM in the context of the pixel-alignment loss? 
4. Feature fusion is not motivated well and seems a general operation for representation enhancement. It would be better to provide a more detailed justification for the choice of feature fusion technique. Specifically, an explanation is expected of how your approach differs from or improves upon standard feature fusion methods, particularly in the context of Mamba-based architectures.

questions:
Overall, the pixel-alignment loss design is interesting but there are many ambiguious technical details to make the contribution clear. Further clarification is required to better position the proposed method within the scope of the visual instruction tuning design of VLMs
</content>
</thread_object>

</thread>
<thread>
creation date:
1730770174214

thread_type:
official_review

<thread_object>
creator: reviewer_ycea

<content>
summary:
Mamba has seen widespread use in large language models (LLMs) due to its exceptional efficiency. However, Mamba struggles with processing visual signals, limiting its application in multi-modal LLMs. To address this issue, this paper proposes improving Mamba’s ability to handle visual signals through structural and hierarchical alignment. The authors argue that Mamba’s difficulty with visual data stems from the lack of positional encoding, which causes the gradual loss of fine-grained spatial information during processing. To mitigate this, the paper introduces the EMMA method, which enhances the structural consistency between Mamba’s output visual features and the original image by reconstructing the image through a decoder. Additionally, EMMA incorporates cross-attention within Mamba, combining multi-level intermediate features via cross-attention to form the final output, thereby preventing the loss of fine-grained details in deeper layers. The results show that EMMA leads to improved performance.

strengths:
- The proposed method is simple yet effective.

weaknesses:
1. **Increased Complexity Due to Cross-Attention**:  
   EMMA introduces cross-attention operations into the Mamba network, reverting the model’s original sub-quadratic complexity back to quadratic complexity, which undermines the original intent of using Mamba. Could the authors provide the computational complexity of the cross-attention operations compared to the overall method? Is cross-attention necessary for fusing intermediate features? Could simpler alternatives be explored to achieve the same goal?

2. **Training Costs and Fair Comparisons**:  
   EMMA involves additional training steps, which should be explicitly detailed in the paper. Furthermore, when comparing EMMA to other methods, the additional training costs should be listed separately to ensure a fair comparison (training time, GPU hours and memory requirement). Since EMMA alters the original MLLM architecture, its inference complexity will also increase. Table 4 should include a breakdown of the increased complexity due to hierarchical alignment. For fairness, Table 4 should also provide EMMA’s inference speed when using Mamba LLM-2.8B.

3. **Limited Technical Contribution**:  
   EMMA’s approach is somewhat simplistic, as it merely enhances the consistency between deep and shallow features and the original image at the output stage. This method requires extra training, substantial computational resources, and introduces additional structures during deployment. As a result, the technical contribution appears somewhat trivial. It would be better if this paper could provide more detailed analysis of how their method addresses specific challenges in multimodal learning that previous methods have struggled with.

questions:
Please refer to the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731590995254

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
A grateful thanks to all reviewers.

comment:
We express our gratitude to all the reviewers for their meticulous reviews and valuable feedback on our paper. The effort put forth in posing insightful questions will greatly aid us in refining and clarifying our work. We are pleased to learn that the motivation has been effectively clarified (as noted by Reviewer T8Kc and yy1F), the paper is well-written (highlighted by Reviewer T8Kc and yy1F), and the experiments substantiate the efficacy of our approach (acknowledged by all reviewers). Over the next few days, we will diligently work towards addressing all raised concerns in order to enhance the quality and comprehensiveness of this work. Thanks again for reviewing our work!
</content>
</thread_object>

</thread>
<thread>
creation date:
1732538269719

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
A huge thanks to all reviewers, we look forward to hearing from you.

comment:
We thank the reviewers again for their valuable feedback. We are grateful that most reviewers are quite affirmative of our overall contributions, including the novelty and motivation ("Pixel-wise alignment loss is interesting" by 5nFQ, "Motivation Well Clarified” by T8Kc, "The imbalance ... does make sense to investigate ..." by yy1F), the extensiveness of empirical evaluation ("The paper presents comprehensive experiments" by T8Kc, "The experiments are sufficient" by yy1F), the effectiveness of the proposed method ("simple yet effective" by yCEA, "experimental results seem promising" by 5nFQ, "robustness of the proposed approach" by T8Kc, "effectiveness of the proposed method" by yy1F), and clarity and soundness of the paper ("Well Written" by T8Kc, "well-written and easy to follow" by yy1F).

We summarize the main contributions of our paper:
1. We observe and discover the difference in performance of mamba-based models in vision vs. language tasks, especially in large-scale settings. This renders the design of mamba-specific architectures in order to bring out the full potential of mamba models in MLLM tasks.
2. We are the first to extend from the vanilla MLLM framework for mamba models and propose mamba-specific modules to address the imbalance between visual and textual latents, enabling better multi-modal fusion of large-scaled mamba-based models. 
3. Due to hierarchical and structural alignment, EMMA is able to achieve better performance than current Mamba-based MLLMs and competitive performance with transformer-based approaches with similar scales. Furthermore, our model demonstrates superior capabilities in reducing hallucinations, where it outperforms all current models <4B in POPE in the OpenCompass VLM leaderboard and outperforms all current models <10B in HallusionBench in the OpenCompass VLM leaderboard. In terms of inference speed, our model is 3 times faster than current similar-sized transformer models. This presents Mamba as a promising contender in the realm of efficient general AI models that require fast response speeds.

We have also made a number of comments to address all reviewers' suggestions and concerns. A short summary of the
replies are made as:

1. Computation analysis: We provide a comparison of each module in terms of computational cost during training and inference. Our newly proposed structural and hierarchical alignment only occurs during training and is discarded during the inference stage.
2. Training stage clarification: Our method only includes a supervised fine-tuning stage that tunes all LLM, MLP projector, and decoder parameters in an end-to-end fashion. A summary of the preprocessing of training data is also included in the revised paper.
3. Motivation clarification: We analyze mamba models both architecturally and empirically to demonstrate the imbalance in the quality of visual and textual latents. This phenomenon renders the need for more effective visual processing in large-scale Mamba models to ensure that visual and textual features are adequately aligned.
4. Additional visualizations: We include additional visualizations to further demonstrate the effectiveness of our method.

Since the rebuttal deadline is incoming, please let us know if our replies have addressed your concerns. We are more than delighted to have further discussions and improve our manuscript. If our response has addressed your concerns, we would be grateful if you could re-evaluate our work.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces Empowering Multi-modal Mamba with Structural and Hierarchical Alignment (EMMA), enhancing Mamba multi-modal large language models (MLLM) by improving their ability to extract fine-grained visual information. This paper is well written and easy to understand. Most of reviewers point out that the pixel-wise alignment loss is interesting and useful, which is demonstrated by doing extensive experiments and showing good performance. The authors are required to update their final version of papers considering the reviews. All reviewers agree to accept this paper, thus I lean to make the recommondation of acceptance.