EMMA: Empowering Multi-modal Mamba with Structural and Hierarchical Alignment
Authors: Yifei Xing
Xiangyuan Lan
Ruiping Wang
Dongmei Jiang
Wenjun Huang
Zheng Qingfang
Yaowei Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Multimodal models
State space models
Efficient architectures
Mamba
Computational Efficiency
Multimodal Alignment
Primary area: foundation or frontier models, including LLMs

Abstract: Mamba-based architectures have shown to be a promising new direction for deep learning models owing to their competitive performance and sub-quadratic deployment speed. However, current Mamba multi-modal large language models (MLLM) are insufficient in extracting visual features, leading to imbalanced cross-modal alignment between visual and textural latents, negatively impacting performance on multi-modal tasks. In this work, we propose Empowering Multi-modal Mamba with Structural and Hierarchical Alignment (EMMA), which enables the MLLM to extract fine-grained visual information. Specifically, we propose a pixel-wise alignment module to autoregressively optimize the learning and processing of spatial image-level features along with textual tokens, enabling structural alignment at the image level. In addition, to prevent the degradation of visual information during the cross-model alignment process, we propose a multi-scale feature fusion (MFF) module to combine multi-scale visual features from intermediate layers, enabling hierarchical alignment at the feature level. Extensive experiments are conducted across a variety of multi-modal benchmarks. Our model shows lower latency than other Mamba-based MLLMs and is nearly four times faster than transformer-based MLLMs of similar scale during inference. Due to better cross-modal alignment, our model exhibits lower degrees of hallucination and enhanced sensitivity to visual details, which manifests in superior performance across diverse multi-modal benchmarks. Code provided at https://github.com/xingyifei2016/EMMA.

Published as a conference paper at ICLR 2025
EMMA: EMPOWERING MULTI-MODAL MAMBA WITH STRUCTURAL AND HIERARCHICAL ALIGNMENT
Yifei Xing! Xiangyuan Lan Ruiping Wang! Dongmei Jiang?, Wenjun Huang*
Qingfang Zheng", Yaowei Wang²
Institute of Computing Technology. Chinese Academy of Sciences, ?Pengcheng Laboratory, UUniversity of Chinese Academy of Sciences, *Sun Yat-sen University 'Pazhou Laboratory (Huangpu) xingyifed feell@eee22@mal 2maii2m uuas..a..... ac. cr, (lanxy, jiangdm, zhengqf01, wangyw)@pcl.co ac. cn wangraipingfict.acc huangwj98emai12 sysu edu.cn
ABSTRACT
Mamba-based architectures have shown to be a promising new direction for deep learning models owing to their competitive performance and sub-quadratic de- ployment speed. However, current Mamba multi-modal large language mod- els (MLLM) are insufficient in extracting visual features, leading imbalanced cross-modal alignment between visual and textural latents, negatively impacting performance on multi-modal tasks. In this work, we propose Empowering Multi- modal Mamba with Structural and Hierarchical Alignment (EMMA). which en- ables the MLLM to extract fine- fine-grained visual information. Specificallyy we pro- pose a pixel-wise alignment module to autoregressively optimize the learning and processing of spatial image-level features along with textual tokens, enabling structural alignment the image level. In addition, prevent the degradation visual information during the cross-model alignment process, we propose a multi-scale feature fusion (MFF) module to combine multi-scale visual features from intermediate layers, enabling hierarchical alignment at the feature level. Ex- tensive experiments are conducted across a variety of multi-modal benchmarks. Our model shows lower latency than other Mamba-based MI.LMs and is nearly four times faster than transformer-based MILMS of similar scale during infer- ence. Due to better cross-modal alignment, our model exhibits lower degrees of hallucination and enhanced sensitivity to visual details, which manifests su- perior performance across diverse multi-modal benchmarks. Code provided at https://iihub.com/iiiigfffoo 6/EMAA
INTRODUCTION
Recently there has been notable increase in the development domain-general AT agents Kalla et al. (2023): Zhao et al. (2023a) which can simultaneously solve diverse range of tasks and ex- hibit superior performance. Among them, multi-modal large language models (MLLMs) Achiam al. (2023), Team et al. 2023 Liu et al. (2024b) have emerged as promising direction due to their effectiveness in visual perception and logical reasoning. MLLMS usually consist of an image encoder that converts images visual tokens and strong large language model (LLM) backbone to process the visual and textual tokens concurrently. This integration of visual and textual informa- tion only enhances the understanding of visual content but also provides more comprehensive context for language understanding and generation. As a result, these cross-modal models have consistently achieved saate-of-the-ar performances in tasks such image captioning Hossain et al. 2019)) visual reasoning Johnson et al. (2017), and visual question answering Antol et al. (2015).
However, gravid challenge for current MLLMs is the substantial computational with training and deployment /Achiam et al. (2023); Han et al. (2022). In fact, current MLLMs are predominately transformer-baseee based, and consequently suffer from an input-dependent attention mech- anism that is quadratic in complexity Katharopoulos et al. (2020b). Transformers also struggle with
associated
CCorreponndng authors

---

Published as a conference paper at ICLR 2025
capturing long-raun ranged dependencies in data due to limited context window Zimerman & Wolf (2023); ; Tay et al. (2020). Many works have attempted to address these issues and propose novel ar- chitectures to increase effectiveness in -nngeeeeencc processing while reducing both computational
and memory costs Wang et al. 2020); Shen et al. (20215) Li et al. 2020);; Gu & Dao (2023)
Dao & Gu 2024 is structured spate-spcce sequence model (SSM) proposed as an alternative for transformers Aoki (2013); : Gu et al. (2021a (b) and has demonstrated
Mamba Gu & Dao 2023
superior performance in sequential tasks such as language and genomics Ma et al. (2024b); Gu & Dao (023). Inspired by classical state-space models Kalman (1960), Mamba is efficiently formu- lated as either recurrence or convolution with near-linear scaling in sequence length, while also possessing principled mechanisms for modeling long-ranged dependencies. Furthermore, [Do & Gu (2024); ; Ali et al. (2024) have addressed the close connection between Mamba and variants of atren- tion mechanisms in transformers, suggesting a promising new underlying architecture for previously transformer-dominate models that is more efficient and achieves competitive performances.
However, while Mamba models possess superior performance sequential textual data, they re- main ineffective in processing visual data, especially in large and huge models et al. (2024); Zhu et al. (024a): Liu et al. (2024c) where performance degrades quickly when in the pattmmeeer scale. This causes an imbalance in the quality of visual and texture latents in the Mamba LLM, where coarse visual features are ineffectively aligned with better-quality textual features. Further- more, while transformer-bant
L.M models contain positional embeddings to guide the learning of spatial information, Mamba LLM layers lack structural constraints the visual features, causing deeper layers of the LLM to gradually lose fine-grained spatial integrity in visual features, as shown in Fig. D This eventually leads to an increasingly blurred feature during the final layers of the LLM, which weakens the cross-modal alignment between visual and textual cues, ultimately resulting in suboptimal performance and higher degrees of hallucination (Fig. D With respect to improving the quality of visual latents, current Mamba-bask rating multiple vision encoders /hho et al. (2024a) or integrating visual selective sean within the projection MLP et al. (2024) to impose structural constraints. However, these approaches pri- marily focus on preserving the visual feature prior to it's integration into the Mamba LLM, lacking mechanisms to maintain its quality within the MLLM, where the visual latents participates in the autoregressive generation of textual tokens.
MLLMs employ various strategies such as incorpo-
Building these observations, we propose Empowering Multi-mode Mamba with Structural and Hierarchical Alignment (EMMA) that addresses the insufficient extraction of visual information inside the Mamba L.M.. First, note the autoregressive nature of Mamba models that allows for an analogous structure for next-token prediction in the visual domain, which has been shown to be effective in vision-related tasks Ren al. (2024); [EI-Nobby et al. (2024). Thus, we extend the prediction of text tokens to the prediction of the visual image as well, to serve as a structural constraint on the visual latents for the Mamba LLM. A pixel-wise alignment is proposed enforce the preservation of key spatial and structural information during training. Next, to further mitigate the gradual loss of fine-gras fine-grained visual features in intermediate LLM layers, we propose Multi- scale Feature Fusion (MFF) module that hierarchically preserves fine-grained visual features. By building additional contribution of intermediate layers the final pixel-wise alignment loss, the MFF effectively alleviates gradual information loss, resulting in finer-grained visual features. The major contributions of this paper are three-fold:
We observe the imbalance between the quality of image and text latents within MLLM Mamba models and propose pixel-wise alignment to autoregressively encourage the learning and processing of structural visual features, ultimately enabling better cross-modal alignment of visual and textural latents.
To mitigate the gradual loss of fine- ine-grained visual cues the LLM, we propose a Mamba- based multi-scale fusion module that combines visual features from multiple intermediate layers, resulting richer representation for the final alignment phase.
We conduct extensive experiments on various multi-modal benchmarks, comparing against current state-of-the- multi-modal LLMs. Due to better moos- alignment, EMMA achieves competitive performance to similar-scaled transformers and outperforms other Mamba models variety of tasks while achieving high inference speed.
2

---

Published as a conference paper at ICLR 2025
"Describe the image dotail."
Cobra Intermediate Visual Features
large deep dish pizza with pepperoni.
0
The image shows a pizza with slicp of pizza silver metal tray with spatula,
0.0
EMMA
EMMA Intermediate Visual Features (Ours)
Figure 1: Given an image of a pizza and the prompt "Describe the image in detail", we visualize intermediate visual features and their corresponding textual responses in Mamba-based MLLMs. Each upper row represents the magnitude each bottom highlights these patches on the original image. Each column (from left to right) rep- resents intermediate layers with increasing depth. Cobra Zhao et al. (2024a) experiences a gradual loss of visual features as visual cues become blurred and unrecognizable, resulting ineffective cross-modal alignment and producing an overly generalized (and hallucinated) answer. On the other hand, due to better cross modal alignment, EMMA is capable of preserving visual details even in deeper layers of the LLM, highlighting areas such as the perimeter of the pizza tray, the overall pizza, and the spatula on the top right of the image. The resulting text demonstrates higher alignment to the image data in the form of sensitivity to visual details and spatial relationships.
reconstructed spatial activations on each image, and
2 RELATED WORKS
2.1 MAMBA-BASED MODELS
Mamba Dao Gu (2024) : Gu & Dao (2023) is form of state-space model that originally stemmed
from classical signal processing theory Kalman (1960). These sequential models inherently excel at capturing long-range dependencies and are designed be computationally efficient Gu et (2020) 202 a): Gupta et al. (2022). Expanding upon these works, Mamba Gu & Dao (2023) introduces more effective selection mechanism that is input-dependent and a hardware-
framework that is computationally efficient. Since then, the Mamba architecture has been successfully applied towards a variety of sequential tasks including NLP Yuan et al. (2024); Behrouz et al. (2024). speech Zhang let al. (2024a); Chen et al. (2024c), and motion Wang al. (2024); Zhang et al. (2024b). rivaling the performance of transformer-based models while being more computationally efficient Qu et al. (2024). Mamba has also been adapted non-sequential data such vision and point clouds Xu et al. (2024a); Yang et al. (2024); Xing et al. (2024) Han et et (2024). which differs by not adhering any particular ordering Huang & Schneider (2011). These tasks are more challenging due to the lack of suitable and appropriate structural constraints in Mamba layers to preserve order-invariam in- formation, such as position encoding in their transformer counterparts Vaswani (0117). Visual State Space model Liu et al. (2024c) introduces novel 2D Selective Scan (SS2D) module that gathers contextual visual information from various perspectives. Vision Mamba Zhu et al. (2024a) applies Mamba to the Vision Transformer architecture and proposes a bi-directional SSM to better process visual features. However, despite gaining success in small-scale scenarios with less than 100M pa- rameters, these vision Mamba models often falter when scaled-up in more complex scenarios. Ren jet al. (2024) enhances Mamba's 's visual capability through autoregressive pretraining, which results in higher classification accuracy its supervised-trained counterparts and successfully scales vi-
3

---

Published as a conference paper at ICLR 2025
sual Mamba models to large and even huge model sizes. In this work, explore ways apply the Mamba architecture to multi-modal learning tasks, where large-scale processing of visual features in Mamba layers is required for aligning visual and language modalities.
2.2
MULTI-MODAL. LARGE LANGUAGE MODELS (MLLMS)
MLLMs take advantage of powerful large language model (LLM) Chang et al. (2024); Achiam cet al. 2023) to simultancously process data from both visual and textual modalities. Cho et al. (2021): Wang al. (2022) initially proposed unified transformer architecture for joint visual and language reasoning tasks that demonstrated better generalizability than conventional vision models. Since then, studies on MLLMs have established strong baselines for general-AI assistants Liu al. (2024b): AAchiam et al. (2023); Alayrac et al.] (2022), which typically consist of a vision encoder, a cross-modal projector, and an LLM backbone. While most current MLLMS utilize text-only loss function for optimization Li et al. (2024b); Zhu et al. (2024b); Lin et al. (2024), EMU [Sun et al.) (2023a 2024) takes interleaved visual and textual inputs in a unified training object of predicting next text or image token in an autoregressive way. However, they require multiple training stages and additional computational cost for training Stable Diffusion Rombach et al. (2022) visual decoder.
A major drawback of current MLLMs resides in the inherent computational overhead Katharopoulos al. a (2020b), predominantly attributed to their foundation transformer architectures. Ma et al. (2024a); al. (2024) propose architectural changes enable more efficient processing of data in MLLMs. Transformer-
MLLMs also struggle with learning long-range dependencies Zhu et al. (202T): Gu & Dao (2023). which is crucial for vision-language tasks. Chen et al. (2024b) repurpose the image embedding projector to encode long textual context, improving the MLLM's ability to handle longer sequences. Nevertheless, current progress in improving the computational complexity and learning long-range dependencies mostly depends on architectural changes in the transformer backbone. In this work, explore how effectively integrate the Mamba model into the maansformerddmmminaee MLLM domain. Mamba offers three to five times higher throughput than its transformer counterparts, while capable of processing million-length sequential data.
2.3 MAMBA-BASED MLLMS
Mamba has been extended to MLLMS due to their capabilities in both vision and NLP. Current multi-modal Mamba models Qiao et al. (2024); Zhao et al.) (2024a) follow a simple LLaVA-based regimen Liu et al. (2024b) through pretrained image encoder, Mamba LLM backbone, and llama-like textual loss. These models utilize efficient down-sampling techniques Chu et al. (2024);
Zhao et al. (0224) and multi-direction scanning mechanisms Liu al. (2024c); Qiao al. (2024)
as cross-modal projection modules to align visual and textual information. However, they lack explicit visual supervision during training. Given that Mamba models are inherently less effective
in processing visual information in large-scale settings Ren et al. (024). this results in poor-quality visual feature that weakens cross-modal alignment in Mamba MLLMs that easily surpass billions of parameters. Our work seeks ways to address the current bottleneck in mamba-based MLLMs where the mamba LLM extracts insufficient visual feature details that results in disparity the alignment between images and text features. To this end, impose additional structural constraints visual features generated by the Mamba LLM, effectively enhancing the overall efficacy of multi-modal representations and cross-modal alignment.
3 METHOD
In this section, present the overall framework for EMMA. EMMA aims to empower the insuf- ficient extraction of visual information in Mamba MLLMS through structural and hierarchical alignment. We first provide the model architecture in Sec. 3.1 Then, we describe our pixel-level visual alignment for preserving structural visual features in Sec. 3.2 and multi-scall fusion module that hierarchically constrains intermediate features for retaining fine-grained visual cues in Sec. 3.3
4

---

Published as a conference paper at ICLR 2025
Mulb-scalo Feature Fusion Module
Pixelwwie Alignment Loss
Visual Feature &,
Intermeciddd
LLM Foetures
Features
Vision Encoder
Multi-scale Feature Fuslon Module
Input image
Aftention
Corresponding Caption Ingut Text X
Mamba Projection
Tokenizer
Backbone
Visual Feature &
Text Response
Figure 2: Overview of EMMA. In addition the textual responses, our model extracts holistic visual feature from the Mamba LLM through a multi-scale feature fusion module that hierarchically combines intermediate visual features through cross-attentin and Mamba projections. A pixel-wise alignment loss is calculated between the final visual feature and the original image to enable the learning of more fine-grained features and increased multi-modal alignment.
3.1 ARCHITECTURE
Inspired by Zhao et al. (2024a); Qiao al. (2024); Liu al. (2024b), the architecture of EMMA consists of a pre-trained vision encoder Ju. a projection MI.P Mprej- a pre-trained Mamba LLM fø parameterized by and multi-scale fusion module vh, shown in Fig. [ Taking an image
X, and its corresponding tokenized caption X, as input, we first obtain visual Teatures through the visual encoder:
X. = fo(o))
(1)
The MLP then maps visual tokens into the word embedding space, which is concatenated with the tokenized captions to form multi-modal input. The combined tokens are fed into the Mamba LLM:
XLLM concat(X,, Xi), X fo
where the multimodal response can be separated into a textual response X, and a visual feature X.. Note that the visual feature is generated through the multi-scale fusion module which combines intermediate LLM features to alleviate the gradual loss of fine-grained visual features. Lastly, pixel-wise alignment loss is applied the visual feature X, for preserving structural visual cues, and autoregressive NLP loss applied to the textural feature X.. More implemennation details can be found in Appendix.
3.2 STRUCTURAL ALIGNMENT VIA PIXEL-WISE ALIGNMENT Loss
We first consider the standard Mamba MLLM training procedure, given multimodal input XLLM and Mamba LLM fø:
fo(LLL)) [X,, (,, =
The target text token sequence X, = ( length L is generated by computing the following probability and optimized by minimizing the corresponding negative log-likelihodd function Creati
p (Xe Xx.
III Po (Ær,s X., (Xes 15 <<())) Liest log
(xe Xo.
(4)
where i,,; depends the visual features X,, and the set of previous text tokens (Xt,; 1j < i). We observe the lack supervision on image-level features, as they serve solely as conditional prior for predicting text tokens X,, As Mamba models inherently struggle with visual tasks in larger models Ren et al. (2024), this may result in poor quality image features during moss-modal alignment, negatively impacting model performance. It becomes crucial to construct additional constraints on the image-level features for better cross-modal alignment. Our insight is to condition the generation

---

Published as a conference paper at ICLR 2025
of visual features structurally, which concern the organization, arrangement, spatial relationships present within visual data. To achieve this, we extend the paradigm of next-token prediction to enable the joint generation of textural tokens and visual images. Thus, similar formulation arises for the generation of target visual sequence X, ( of length K, given by the following probability function:
K pxx, X.) = II po (fn, {Xxs 1j < <i)
(5)
We then project the generated visual feature X2 from word embedding space back to the image domain through Mamba-based decoder fee to retrieve the final generated visual sequence. Using the original image as reference, a common loss function for measuring the quality and consistency between two images is L2 distance. The L2 loss considers global features of the image, focusing on overall similarity that helps in preserving structures and shapes in the image. We formulate our Pixel-wise Alignment Loss Lpizet to constrain the decoded image to match the original image as:
Lpizel = IIfdec 2x,12
(6)
By imposing our pixel-wise alignment loss, we constrain the output visual feature to match closely with the original image, forcing the model to understand and preserve structural features of the visual input, and effectively increasing the quality of visual latents for cross- modal alignment. Unlike Zhan et al. (2024); Sun et al. (2023a), our visual decoder is extremely small-scaled and efficient, and does not require a separate stage for training.
3.3 HIERARCHICAL ALIGNMENT VIA MULTI-SCALE FUSION MODULE
While the pixel-wise alignment loss forces the Mamba LLM retain structural characteristics of the visual input, we find that Mamba LLMs have a tendency to gradually lose fine-grained visual details through intermediate layers, as shown in Fig. [ This could be due to the inherent lack of "positional embedding" -analouuus within the Mamba L.MM backbone, such that spatial and fine- grained information is more easily distorted inside the LLM. Consequently, we contend that the visual feature derived from the final layer of Mamba LLM alone proves inadequate in preserving fine visual intricacies. To enable the Mamba LLM to extract more sufficient visual details and prevent gradual feature loss from the image modality, devise Multi-scale Feature Fusion (MFF) module that hierarchically integrates multiple intermediate features of the pretrained visual encoder for the visual alignment. Combining multi-level features that concern different levels of granularity enables the model to effectively capture intricate details at various scales, enhancing its ability to comprehend and interpret complex visual information with greater precision. Specifically, our multi-scale feature fusion module consists of I fusion blocks & {B.} where each B can be regarded as tiny Mamba network consisting combines intermediate features in a pairwise fashion. For instance, given hidden visual features XX,, x from layers i, j. & and the corresponding two-layer MFF & {B,)2, 2, the final aggregated output of the features X, is generated as follows:
cross-attention module and a Mamba layer, which
X + xX, = (B,
where each block B is given by residually Mamba and cross attention layers:
B(X,Y) = 1 Y) + Mamba(B((x B(X,Y) = X + cross_aftn(XXX
(8)
In practice we utilize three intermediate layers along with the final output layer for producing the aggregated visual feature for pixel-wise alignment. Thus, the final pixel-wise alignment becomes:
Spizel = || (X.) x,112
By introducing additional contribution of intermediate features in the pixel-wise alignment loss, we force the model to retain structural and fine- ame-grained information in these layers and alleviate the gradual loss of visual features. This results in better-quality visual features, as evident in Fig. I Additionally, we note that the feature fusion and visual decoding stage only occurs during training where loss calculations are needed, and poses additional computational overhead in inference.

---

Published as a conference paper at ICLR 2025
4 EXPERIMENTS
In this section, we conduct extensive experiments from multiple perspectives to demonstrate the effectiveness of our method. First, we provide experiment settings including training data, training recipes, and evaluation benchmarks in Sec. 4.1] Next, we provide experimental results from the aforementioned benchmarks and compare with contemporary MLLMs in Sec. 4.2 EMMA sur- passes other Mamba-based models majority of tasks and is highly competitive with transformer models of similar size. Then, we conduct detailed inference speed comparison between EMMA
and other Mamba and transformer-basee models of similar size in Sec. 4.3 At last, we present
ablation studies to investigate the effectiveness of our model design choices in Sec. 4.4
4.1 EXPERIMENT SETTINGS
Training Data. Following Zhao et al. (2024a), we train EMMA on a combination of datasets consisting of LLaVA-vI 5-mixed-665k Liu et al. (2024b), LVIS-Instruct-44 [Wang et al. (2023). and LRV-Insrruet Liu al. (2023b). These are conversational multi- modal and pure textual data that contain roughly 1.2 million images and dialogues.
Backbone Models. We select Cobra as our baseline model. FollowingZhane et al. (2024a), we utilize SigLIP and DINOv2 as the visual encoder, where the final output is the concatenated feature of both models. The image decoder consists projector consists of stacked linear layers. We provide two versions of our model with MambaV1- 2.8b & Dao 2023) and Mamba 22-2.7b Dao & Gu (024) backbones.
combination of Mamba and linear layers. The MLP
Training Recipes. We directly finetune the Mamba LLM backbone, the multi-scale fusion module, the image decoder, and the MLP projector on the training data for two epochs, discarding the pretrain phase. The visual encoder is frozen all times. We select global batch size of 128 and starting learning rate of 2e-5 with AdamW optimization. Our models are trained on eight 40G A100 GPUs with fully sharded data parallelism Zhao al. (023bb)
Evaluation Benchmarks. We evaluate our model variants on four open-ended visual question- answer benchmarks: VQAv2 Goyal al. (2017) and VizWiz Gurari et al. (2018) test general visual reasoning, GQA Hudson & Manning (2019) validates spatial reasoming, and Text Singh et al. (2019) assesses reasoning around text. We also evaluate our models on nine comprehensive closed- set benchmarks: VSR Liu et al. (2023a) tests coarse object spatial relationships. POPE Li et al. (2033b) tests object hallucinations while HallusionBench Guan et al. (2024) assesses object illusion and visual hallucinations. MMB Liu al. (2023c) and MME Fu et al. (2024) are both robust and holistic evaluations of MLLMs. We use reproduced results from the Cobra codebase Zhao et al. (024a) and obtain the performance of other models in their respective papers. More details for experimental settings can be found in the Appendix.
4.2 EVALUATION RESULTS
We compare model with variety of Mamba-based and transformer-based MILMs. These include large-scaled MILMMs OpenFlamingo Awadalla et al. (2023). BLIP-2 Li et al. (2023a).
MiniGPT-4 Zhu et al. (2023). EMU [Sun et al. (2023a). EMU 2 Sun al. (2024). InstructBLiP
Wenliang et al. (2023). Shikra Chen et al. (2023a). IDEFICS Hua & Artzi (0224) Qwen-VL Bai al. (2023). LLa VA Liu al. (2024b), Prism Karamcheti et al. (2024). ShareGPT4V Chen al. (2023)), VLLVA Lin et al. (2024). We also include transformers with similar scales, which encompass VLa Zhu et al. (2024b). Mobile Chu et al. (2023), Mobile V2 Chu et al. (2024), and LLaV Zhou et al. (2024). Lastly. we also compare with current Mamba- based MLLMs, consisting of VI. Mamba Qiao al, (2024) and Cobra Zhho al. (2024a). We also provide the backbone LLM of each model, as well as the training data size for more fair comparison.
Comparison with similar-scaled Mamba MLLMs. Performance comparisons between EMMA and other Mamba-based MLLMs can be found in the bottom group of Tab. as well as 6 We underline the best- persfomefon models this category. All models utilize a similar amount of data and Mamba backbone. We surpass the performance of Cobra, our baseline model, every evaluated metric, demonstrating better generalizability as well as visual and spatial reasoning. The increase in performance is most apparent in MME, where there is a noticeable increase of 279 in terms of

---

Published as a conference paper at ICLR 2025
Model
LLM
Data Size
VQA
GQA
VizWiz
VQA"
VSR
MME
MMB
OpenHaamingo
MPT-7B
2B
52.7
27.5
33.6
BLIP-2
Vicuna-130
129M
41.0
19.6
42.5
50.9
1293.8
MiniGPT-4
Vicuna-7B
5M
32.2
581.7
EMU
LLMMA-13B
3.4B
62.0
46.0
38.3
EMU
LLMMA-3BB
4.1B
84.9
66.6
InstructBLIP
Vicuna-7B
130M
49.2
34.5
50.1
54.3
36.0
InstructBLIP
Vicuna-138
130M
49.5
33,4
50.7
52.1
1212.8
Shikra
Vicuna-13B
6.1M
77.4
58.8
IDEFCCS
LLaMA-7B
354M
50.9
35.5
25,9
48.2
IDEFICS
LLaMA-75B
354M
60.0
36.0
30,9
54.5
Qwen-VL
Qwen-7B
1.5
78.2
35.2
63.8
38.2
LLaVA v1.5
Vicuna-7B
1.2M
78.5
62.0
50.0
58.2
1510.7
64.3
Prism
LLaMA-7B
1.2M
81.0
65.3
52.8
$9.7
59.6
ShareGPT4V
Vicuna-7B
1.2M
80.6
57.2
1567.4
68.8
MoE-LLLVA
StableL-M-1.6B
2.2M
76.7
60.3
36,2
50.1
59.4
MoF-IJLaV
Phi2-2.7B
2.2M
77.6
61.4
51.4
65.5
LLaVA-Phi
Phi2-2.7B
1.2M
71.4
35.9
48.6
1335.1
59.8
MobileVLM-33
LabilcLLaMA-2.78
1.2M
59.0
47.5
1288.9
59.6
MohileVLM v2
Mahilel.IaMA-2.11
3.6M
61.1
57.5
1440.5
63.2
TinLLaaV
Phi2-2.7B
1.2M
76.6
60.3
51.4
1464.9
66.9
VL-Mamba
Mamba 1-288B
1.2M
76.6
56.2
48.9
1369.6
57.0
Cobra
Mamba V1-2.8B
1.2M
74.9
59.1
52.0
52.4
51.7
1294.3
51.5
EMMA-VI
Mamba 1--2.8B
1.2M
76.3
60.5
52.1
57.2
51.5
1572.8
53.2
EMMA-V2
Mamba V2-2.7B
1.2M
75.7
59.4
54.1
56.2
52.7
1454.9
60.8
Table I: Experimental results on seven multi-modal benchmarks: VQA GQA, VizWiz, TextVQA, VSR, MME and MMB. We separate the models into three groups by horizontal lines: the first group consists of large MLLMs, the second group consists of small-scaled transformer MLLMs, and the third group consists of Mamba MILMs. Our method surpasses other Mamba-based frameworks in all but the VQA v2 task, where VL Mamba has slight advantage of less than 1%. Our method also achieves competitive results with other transformer- based methods of similar scale, while attaining much smaller computational complexity, detailed in Sec. 4.3
the general perception and cognition abilities of the model. There also a 5% increase in the open- ended VQA task TextVQA, which reflects the attentiveness of fine-grained details of our model. The performance gain is least apparent in VSR due to it consisting solely of true/false answers, which are inherently coarse nature and do not necessitate fine- grained details. Nevertheless, our model still improves upon the Cobra baseline on these benchmarks with small gains. We also outperform VL Mamba on all but the VQAv2 benchmark, where our best model underperforms by 0.3. These results suggest that Mamba MILMS benefit from better-quality visual features through structural and hicrarchical alignment, resulting in overall gains across the majority of metrics.
fine-grained
EMMA and other transformer-based MLLMs can found Performance comparisons between middle group of Tab. TT We bold- font the best- best-performing models for all similar-scaled MLLMs, Mamba and transformer-bitt basee..rrrbdd......o Note that although these models have similar parameter sizes to ours, their backbone LLM was trained on 1.3T (Mobilel_LMMA) and 1.4T (Phi-2-2.7B) textural tokens prior the MILM pretrain and finetune stage. On the other hand, Mamba V1-2.8b was trained on the SlimPj Shen et al. (2023) and Mamba V2-2.7b on the Pile Gao al. (2020), with only 627B and 300B tokens respectively. Furthermore, ML V2 trains on nearly three times as much data as ours prior to evaluation. Other networks such TinyLLaVA train the more diverse ShareGPT4V dataset, which greatly improves model performance. Nevertheless, our model achieves the best VizWiz, VSR, and MME scores across all models of similar scale. Our model is also competitive on other metries, many of which are only around 1 1 below the best-performing model.
Comparison with similar-scaled transformer MILMs.
Comparison with large-scaled transformer MLLMs.. Lastly, we also compare EMMA with other large-scaled MILMs in the upper group of Tab. [ Although there is significant gap in model size and training data, EMMA still achieves competitive performance some benchmarks. highlight EMU and EMU2, which is a transformer MILM that also jointly generates visual and textual tokens in an autoregressive fashion. However, both EMU and EMU2 require a sizable stable diffusion decoder translate visual tokens into image space, along with additional training stages to optimize their latent space. These transformer based models also demand a substantial amount of data and possess significantly larger parameter counts compared model. Regardless,
also
8

---

Published as a conference paper at ICLR 2025
model consistently outperforms EMU across all benchmarks while utilizing nearly one-fourth of its parameters. EMMA also achieves the best MME across all competing models. EMMA is second in terms of VizWiz, only 0.8 behind the best best-performing model EMU2, which is nearly ten times bigger than our model. These results demonstrate the effectiveness of EMMA and show the potential of Mamba-based MILLMs as alternative transformer-based models.
EMMA-VI
EMMA-V2 V2
Cobra
GPT4V
LLaVA-I.5
Claude3
BLIP2-T3
Qwen-VL
MiniGPTS
POPE
88.0
87.3
87.2
85.9
HBench
51.0
47.5
41.4
65.28
47.1
56.9
48.1
39.2
40.3
Table 2: Performance comparisons for Gallucination-relaii benchmarks selection of MLLMs. Hßench denotes HallusionBench. Our model achieves the best performance in POPE and is closely competitive with large-sized models.
Analysis for Hallucination. We analyze the degree of hallucination in our models and report cor- responding performances on the POPE and HallusionBench benchmark Tab. 2 POPE concerns object hallucination which refers to generated contents inconsistent with ground-truth objects in the input, which reflect hallucination leaning towards the language modality. HallusionBench focuses on diagnosing both the visual illusion and knowledge hallucination of MLLMs. Visual illusion refers to the misinterpretation of accurate visual information, and knowledge hallucination denotes percep- tions formed without relevant visual input. Thus, HallusionBench reveals hallucinations more the effective extraction of information in the visual modality. Our model achieves the highest score on POPE, demonstrating the superior NLP capabilities of the Mamba LLM backbone. We also achieve significant gain (51.0 vs 41.4) in HallusionBench from our baseline, demonstrating the effectiveness of higher-quality visual features in reducing visual hallucinations. Our model is also competitive in HallusionBench with ALaVA-1.5 1.5 and BLIP2-T5, a 7B and 12.4B model respectively.
4.3 MODEL LATENCY
We evaluate model latency between EMMA and other similar-sized MLLMs, which include Cobra, TinyLLaVA, and MobileVLM V2 in Tab. While VL Mamba is not open-sourced yet, we assume the latency of VL Mamba to mirror that of Cobra, as both models leverage the Mamba CambbaV1-2.88 1-2.8b backbone architecture. All models are given the same sample image and the exact textural prompt of "Describe this image in detail". Each model is then forced to generate 256 tokens in response to this prompt for total of 200 times, and the overall time cost is recorded as Toverall- Finally, calculate the average time cost for each model to generate 256 tokens (Tavg) as well as the number of tokens generated per second (N Nang) as follows:
T Toog T Toerrat/ /200; Navg 256/Taag
(10)
All evaluations are conducted on single 40G A100 GPU. EMMA has a significant time advantage over transformer-based MLLMs and a non-trivial edge over other Mamba-based MLLMs. Mamba models, given their better theoretical guarantees, are three to five times faster than current state- of-the-art art transformer- based models of similar sizes. Our model achieves even better runtime than Cobra due to more efficient processing in the Mamba L.M backbone.
Model
L.M Backbone
Tokens per second
Average Time
EMMA (Ours)
Mamba V2 LLM-2.7B
149.96
1.71
EMMA (Ours)
Mamba LLM-2.8B
138.95
1.84
Cobra
Mamba LLM-2.8B
138.95
1.84
TinyLLaVA
Phi2-2.7B
41.46
6.17
MobileVLM V2
MobileLLaMA-2.7B
39.36
6.50
Table 3: Latency table for similar-scaled MLLMs, both Mamba & transformer architectures.
4.4 ABLATION STUDIES
In this section, we present ablation studies for the design choices of our model in Tab. Component-wise Ablation. We first explore the impact of the two proposed components on the overall performance of our model. By removing the multi-scale feature fusion module, we observe

---

Published as a conference paper at ICLR 2025
noticeable reduction in MME, well slight degradation in VQAv2, GQA, POPE, and Hallu- sionBench. This shows that the multi-scale feature fusion module plays a crucial role in enhancing performance across multiple tasks, by preventing visual information loss in the LLM, thereby en- hancing the overall quality of visual features. We then remove the pixel-wise alignment loss, which is equivalent to training the plain Cobra model. We observe substantial decline in TextVQA and HallusionBench,, along with marginal decreases in GQA, VQA-V2, and MMB. Notably, TexiVQA involves open-set VQA, relying heavily on intricate visual details, whereas HallusionBench ad- dresses visual hallucinations. This outcome underscores the impact of our pixel-wise alignment loss in improving the quality of visual representations, leading to enhanced processing of fine visual details and a reduction in visual hallucinations.
Model
VQA
GQA
VizWiz
VQA
VSR
POPE
MME
MMB
HBench
EMMA (Ours)
76.25
60.5
52.1
57.2
51.5
88.0
1572.8
53.2
51.0
MPFF
75.9
59.3
52.0
57.0
51.5
87.1
1294.1
53.1
50.7
PAL
74.9
59.1
52.0
52.4
51.7
87.2
1294.3
51.5
41.4
+CSM
75.8
59.0
54.2
55.6
51.5
87.6
1420.5
52.7
50.5
+AVE 52.8
44.8
45.0
41.1
51.6
69.7
984.8
25.0
47.2
Table 4: Ablation table for our model. Without loss of generality, we utilize the MambaV1 version of our model for these experiments. HBench denotes HallusionBench. MFF denotes the multi- scale feature fusion module, PAL denotes the pixel-alignment loss, CSM denotes the cross-scan mechanism, and AVF denotes align visual features.
Preserving Structural Information in MambaLLM. We note that many works Qiao et al. (2024); Liu et al. (222c) utilize a visual selective sean mechanism to incorporate structural information and enhance the quality of visual features. Consequently, we opt to integrate this mechanism on top of our method (before the projection MLP) assess its impact on the quality of visual representa- tions and model performance (+CSM). The results demonstrate a marginal change in performance, suggesting that our structural constraint framework might already suffice in generating high-quality structural visual features. The introduction of the cross-scan module may appear to be unnecessary.
Pixel ys. Feature Alignment. Given that the Mamba LLM receives visual features from the vision encoder, we also explore directly aligning these processed features instead of the raw image pixels (+AVF). However, the resulting performance experiences a notable decline across various metrics. We attribute this degradation contrasting with the extracted features from the visual encoder that may lack such details. However, despite the severe performance degradation, the model achieves a high hallusionBench score which implies the efficacy of supervising visual features in mitigating visual hallucinations.
the robust structural information inherent in pixel-level images,
5 CONCLUSION
In this study, we introduce EMMA, a novel approach designed to rectify the imbalanced quality be- tween visual and textual latents within the MLLM framework, which adversely impacts mooal alignment and leads to suboptimal performance. By expanding the autoregressive generation of text tokens to operate image-wist wise patches, we leverage this process as visual supervision for the visual features within the Mamba LLM. Subsequently, we introduce a pixel-wise alignment loss to align the generated visual features with the original image, thereby preserving crucial structural informa- tion. On the other hand, the absence of structural constraints in the Mamba LLM leads to a gradual loss of fine-gras aiee-graind details in the visual features. To address this issue and counteract the gradual information loss, we propose multi-scale feature fusion module that hierarchically integrates mul- tiple intermediate L.LM features, effectively preserving fine- ane-grained information across these layers. Experimental results showcase that EMMA not only significantly enhances performance across di- verse benchmarks but also exhibits reduced levels of hallucination. The Mamba V2 iteration of our model further reduces model latency compared to current Mamba MLLMs and achieves nearly four times the speed of similarly scaled transformer models during inference. We hope that this study opens up new ways for enhancing the visual latent quality of Mamba-base MLLMs, presenting more efficient and ios demanding rapid responses, such as autonomous driving and robotics control.
coarmannc-cooparalae alternative to transformer models, especially in scenar-
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We express our gratitude Zhiyuan Wen and Zhilin Huang for their valuable and insightful dis- cussions during the composition of this paper. This work is partially supported by Natural Science Foundation of China under contract No. U21B2025, 62402252, and Pengcheng Laboratory Re- search Project No. PCL2023A08, PCL2024Y02.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint
2023.
Jean-Baye MannBBppistt Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716- 23736, 2022.
Ameen Ali, Itamar Zimerman, and Lior Wolf. The hidden attention of mamba models. arXiv preprint arXiv:2403..m 51590, 2024.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2423-,433, 2015.
Masanao Aoki. State space modeling of time series. Springer Science & Business Media, 2013.
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: open- source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308. 31390, 2023.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang. Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint Xrviv22308. /2966, 2023.
Ali Behrouz, Michele Santacatterina, and Ramin Zabih. Mambamixer: Efficient selective state space models with dual token and channel selection. arXiv preprint arXiv:2403. 198888 2024.
Yonatan Bitton, Gabriel Stanovsky, Michael Elhadad, and Roy Schwartz. Data efficient masked language modeling for vision and language. arXiv preprint arXiv:2109.02040, 2021.
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang. Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ACM Transactions Intelligent Systems and Technology, 15(3):1-43 2024.
Kaibing Chen, Dong Shen, Hanwen Zhong, Huasong Zhong, Kui Xia, Di Xu, Wei Yuan, Yifei Hu, Bin Wen, Tianke Zhang, et al. Evim: An efficient vasion-language model for visual understanding. arXiv preprint
2024a.
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal Ilm's 's referential dialogue magic. arXiv preprint arXiv:2306. /5/95, 2023a.
Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793, 2023b.
Wei Chen, Zhiyuan Li, Shuo Xin, and Yihao Wang. Dolphin: Long context as new modality for energy-efficient on-device language models. arXiv preprint arXiv:2408. 15518, 2024b.
Yujie Chen, Jiangyan Yi, Jun Xue, Chenglong Wang, Xiaohui Zhang, Shunbo Dong, Siding Zeng, Jianhua Tao, Lv Zhao, and Cunhang Fan. Rawbmamba: End-to-end bidirectional state space model for audio deepfake detection. arXiv preprint arXiv:2406.06mm 2024c.

---

Published as a conference paper at ICLR 2025
Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying vision-and-languaar tasks via text generation. In International Conference on Machine Learning, pp. 1931-1942. PMLR, 2021.
Xiangxiang Chu, Limeng Qiao, Xinyang Lin, Shuang Xu, Yang Yang, Yiming Hu, Fei Wei, Xinyu Zhang, Bo Zhang, Xiaolin Wei, et al. Mobilevim: A fast, reproducible and strong vision language assistant for mobile devices. arXiv preprint arXiv:2312. 2023.
Xiangxiang Chu, Limeng Qiao, Xinyu Zhang, Shuang Xu, Fei Wei, Yang Yang, Xiaofei Sun, Yiming Hu, Xinyang Lin, Bo Zhang, et al. Mobilevim v2: Faster and stronger baseline for vision language model. arXiv preprint wrXiv:2402.03766, 2024.
Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality, arXiv preprinl arXiv:2405.2 2024.
Soham De, Samuel L Smith, Anushan Fernando. Aleksandar Botev, George Cristian-Murarn Al- bert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, al. Griffin: Mix- ing gated linear recurrences with local attention for efficient language models. arXiv preprint arXiv:2402.19427 2024.
Wenhao Dong, Haodong Zhu, Shaohui Lin, Xiaoyan Luo, Yunhang Shen, Xuhui Liu, Juan Zhang, Guodong Guo, and Baochang Zhang. Fusion mamona for cross-modality object detection. arXiv preprint arXiv:2404.000 09146, 2024.
Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
Alaaeldin El-Nouby. Michal Klein, Shuangfei Zhai, Miguel Angel Bautista, Alexander Toshev, Vaishaal Shankar, Joshua M Susskind, and Armand Joulin. Scalable pre-training of large au- toregressive image models. arXiv preprint (xXi::4401.0844: 2024.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: comprehensive evaluation benchmark for multimodal large language models, 2024. /
23366. 13394]
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.000 00007, 2020.
Paolo Glorioso, Quentin Anthony, Yury Tokpanov, James Whittington, Jonathan Pilault, Adam Ibrahim, and Beren Millidge. Zamba: A compact 7b ssm hybrid model. arXiv preprint arXiv:2405.16712, 2024.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6904 6913, 2017.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv.2312.00752. 2023.
Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. Hippo: Recurrent memory with optimal polynomial projections. Advances in neural information processing systems, 33: 1474-1487, 2020.
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00300 2021a.
Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Com- bining recurrent, convolutional, and continuous-tinn models with linear state space layers. Ad- vances in neural information processing systems, 34:572-585, 2021b.
12

---

Published as a conference paper at ICLR 2025
Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattrnn Recognition, pp. 14375- 14385, 2024.
Zonghao Guo, Ruyi Xu, Yuan Yao, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, and Gao Huang. Llava-uhd: Imm perceiving any aspect ratio and high-resolution images. In European Conference on Computer Vision, pp. 390-406. Springer, 2025.
Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmen- tation. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5356-5364, 2019.
Ankit Gupta, Albert Gu, and Jonathan Berant. Diagonal state spaces are as effective as structured state spaces. Advances in Neural Information Processing Systems, 35:22982 2022.
Danna Gurari, Qing Li, Abigale Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3608-3617, 2018.
Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, al. A survey on vision transformer. IEEE transactions on paltern analysis and machine intelligence, 45(1):87-110, 2022.
Xu Han, Yuan Tang, Zhaoxuan Wang, and Xianzhi Li. Mamba3d: Enhancing local features for 3d point cloud analysis via state space model. arXiv preprint arXiv:2404. 14966, 2024.
Tuomo Hiippala, Malihe Alikhani, Jonas Haverinen, Timo Kalliokoski, Evanfiya Logacheva, Sera- fina Orekhova, Aino Tuomainen, Matthew Stone, and John A Bateman. Ai2d-rst: multimodal corpus of 1000 primary school science diagrams. Language Resources and Evaluation, 55:661- 688, 2021.
MD Zakir Hossain, Ferdous Sohel, Mohd Fairuz Shiratuddin, and Hamid Laga. A comprehensive survey of deep learning for image captioning. ACM Computing Surveys (CsUR), (1))::1336. 2019.
Yilun Hua and Yoav Artzi. Talk less, interact better: Evaluating in-context conversational adaptation in multimodal llms. arXiv preprint arXiv:2408.01417, 2024.
Tzu-Kuo Huang and Jeff Schneider. Learning auto-regressive models from sequence and non- sequence data. Advances in Neural Information Processing Systems, 24, 201
Drew Hudson and Christopher Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference computer vision and patlern recognition, pp. 6700-6709, 2019.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2901-2910, 2017.
Dinesh Kalla, Nathan Smith, Fnu Samaah, and Sivaraju Kuraku. Study and analysis of chat gpt and its impact on different fields of study. International journal of innovative science and research technology, 8(3), 2023.
Rudolph Emil Kalman. A new approach to linear filtering and prediction problems. 1960.
Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh. Prismatic vlms: Investigating the design space visually-conditione language models. arXiv preprint 202.:2240..0786.. 2024.
13

---

Published as a conference paper at ICLR 2025
A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret. Transformers mns: Fast autoregressive transformers with linear attention. In Proceedings of the International Conference on Machine Learning (ICML), 2020a. httphh////aaatt
org/ abs 201//2066.110 16236
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference ma- chine learning, pp. 5156-5165. PMLR, 2020b.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting lan- guage and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-73, 2017.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. Advances in neural information processing systems, 25, 2012.
Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench Benchmarking multimodal large language models. In Proceedings the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 13299-13308, 2024a.
Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Nau- mann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training large language-and assis- tant for biomedicine in one day. Advances in Neural Information Prucessing Systems, 36, 2024b.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference machine learning, pp. 19730-19742. PMLR, 2023a.
Rui Li, Jianlin Su, Chenxi Duan, and Shunyi Zheng. Linear attention mechanism: An efficient attention for semantic segmentation. arXiv preprint arXiv:2007.14902, 2020.
Wentong Li, Yuqian Yuan, Jian Liu, Donggi Tang, Song Wang. Jie Qin, Jianke Zhu, and Lei Zhang- Tokenpacker: Efficient visual projector for multimodal 1lm. arXiv preprint arXiv:2407.02339, 2024c.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305. 2023b.
Zhe Li, Haiwei Pan, Kejia Zhang, Yuhua Wang, and Fengming Yu. Mambadfuse: mamba-based dual-phase model for multi-modality image fusion. arXiv preprint arXiv:2404.084066 2024d.
Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Mcirom, Yonatan Belinkov, Shai mamba language model. arXiv preprint arXi::2403 2024.
et al. Jamba: A hybrid transformer-
Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Junwu Zhang, Munan Ning, and Li Yuan. Moe-llava: Mixture of experts for large vision-language models. arXiv preprint 15X19 2024.
Hezheng Lin, Xing Cheng, Xiangyu Wu, and Dong Shen. Cat: Cross attention in vision transformer. In 2022 IEEE international conference on multimedia and expo (ICME), pp. 1-6. IEEE, 2022.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Aision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning. Transactions of the Associ- ation for Computational Linguistics, 11:635-651, 2023a.
Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang. Yaser Yacoob, and Lijuan Wang. Mitigating hal- lucination large multi-modal models via robust instruction tuning. In The Twelfth International Conference Learning Representations, 2023b.
14

---

Published as a conference paper at ICLR 2025
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 26296-26300 2024a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in nearal information processing systems, 36, 2024b.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint 23Xiv:3307.0628.. 2023c.
Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang. Qixiang Ye, and Yunfan Liu. Vmamba: Visual state space model. arXiv preprint arXivv2440.10166, 2024c.
Lucidrains. Linear attention transformer. 2021. URL
Feipeng Ma, Yizhou Zhou, Hebei Li, Zilong He, Siying Wu, Fengyun Rao, Yueyi Zhang, and Xi- aoyan Sun. Ee-mllm: data-efficient and compute-efficient multimodal large language model. arXiv preprint arXiv:2408.I 11795, 2024a.
Jun Ma, Feifei Li, and Bo Wang. U-mamba: Enhancing long-range dependency for biomedical image segmentation. arXiv preprint arXiv:2401.0472. 2024b.
David Mizrahi, Roman Bachmann, Oguzhan Kar, Teresa Yeo, Mingfei Gao, Afshin Dehghan, and Amir Zamir. 4m: Massively multimodal masked modeling. Advances in Neural Information Processing Systems, 36, 2024.
Zhenyu Ning, Jieru Zhao, Qihao Jin, Wenchao Ding, and Minyi Guo. Inf-mlim: Efficient streamiin inference of multimodal large language models on a single gpu. arXiv preprint arXiv:2409.0908 2024.
Yanyuan Qiao, Zheng Yu, Longteng Guo, Sihan Chen, Zijia Zhao, Mingzhen Sun, Qi Wu, and Jing Liu. VI-mamba: Exploring state space models for multimodal learning. arXiv preprint arXiv:2403. 13600, 2024.
Haohao Qu, Liangbo Ning, Rui An, Wenqi Fan, Tyler Derr, Xin Xu, and Qing Li. A survey of mamba. arXiv preprint arXiv:2408.011 129, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp- 8748-8763. PMLR, 2021.
Sucheng Ren, Xianhang Li, Haoqin Tu, Feng Wang, Fangxun Shu, Lei Zhang, Jieru Mei, Linjie Yang, Peng Wang, Heng Wang. al. Autoregressive pretraining with mamba in vision. arXiv preprint arXiv:2406.07555 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Tanik Saikh, Tirthankar Ghosal, Amish Mittal, Asif Ekbal, and Pushpak Bhattacharyya. Scienceqa: A novel resource for question answering on scholarly articles. International Journal on Digital Libraries, 2383):289-31,, 2022.
Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer. How much can clip benefit vision-and- lnd-uuga tasks? arXiv preprint arXiv:2107.06383, 20
Zhiqiang Shen, Tianhua Tao, Liqun Ma, Willie Neiswanger. Zhengzhong Liu, Hongyi Wang, Bowen Tan, Joel Hestness, Natalia Vassilieva, Daria Soboleva, et al. Slimpajama-dc: Understanding data combinations for Ilm training. arXiv preprint a309v:2309. 10818, 2023.
15

---

Published as a conference paper at ICLR 2025
Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Efficient attention: Attention with linear complexities. In Proceedings of the IEEE/CVF winter conference on appli- cations of computer vision, pp. 3531-3539, 2021b.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Emu: Generative pretraining in multimodality. In The Twelfth International Conference on Learning Representations, 2023a.
Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative pretraining in multimodality. arXiv preprint arXiv:2307.0522) 2023b.
Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 14398-1444 14409, 2024.
Yi Tay, Mostafa Dehghani. Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. arXiv preprint arXiv:201 1.04006, 2020.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, al. Gemini: family of highly capable multimodal models. arXiv preprint 2rXiv:2312./18000 2023.
Jamba Team, Barak Lenz, Alan Arazi, Amir Bergman, Avshalom Manevich, Barak Peleg, Ben Aviram, Chen Almagor, Clara Fridman, Dan Padnos, et al. Jamba-1.5; Hybrid transformer- mamba models at scale. arXiv preprint arXiv:2408. 12570, 2024.
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Junke Wang, Lingchen Meng. Zejia Weng, Bo He, Zuxuan Wu, and Yu-Gang Jiang. To see is to believe: Prompting gpt-4v for better visual instruction tuning. arXiv preprint arX1v:2311.07574. 2023.
Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04766. 2020.
Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng. Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for all vision and vision-languar tasks. arXiv preprint arXiv:2208. 200.:0422, 2022.
10442,
Xinghan Wang, Zixi Kang, and Yadong Mu. Text-controlled motion mamba: Text-instructed tem- poral grounding of human motion. arXiv preprint 2nXiv:2404.113: 2024.
Dai Wenliang, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang. Boyang Li. Pascale Fung, and Steven Hoi. Instructblip: Towards general- purpoos vision-language models with instruction tuning. arXiv preprint 23X5:2300506500 2023.
Xinyu Xie, Yawen Cui, Chio-In Ieong, Tao Tan, Xiaozhi Zhang, Xubin Zheng, and Zitong Yu. Fusionmamba: Dynamic feature enhancement for multimodal image fusion with mamba. arXiv preprint arXiv:2404.0944 2024.
Zhaohu Xing, Tian Ye, Yijun Yang, Guang Liu, and Lei Zhu. Segmamba: Long-range sequential modeling mamba for 3d medical image segmentation. arXiv preprint arXiv:240I. 13560, 2024.
Xu, S Yang, Y Wang, Cai, B Du, and H Chen. Visual mamba: A survey and new outlooks. arXiv preprint 2404 18861, 2024a.
16

---

Published as a conference paper at ICLR 2025
Rui Xu, Shu Yang, Yihui Wang. Bo Du, and Hao Chen. A survey vision mamba: Models, applications and challenges. arXiv preprint arXiv:2404. 2024b.
Yijun Yang, Zhaohu Xing, and Lei Zhu. Vivim: a video vision mamba for medical video object segmentation. arXiv preprint arXiv:2401.44166 2024.
Danlong Yuan, Jiahao Liu, Bei Li, Huishuai Zhang, Jingang Wang. Xunliang Cai, and Dongyan Zhao. Remamba: Equip mamba with effective long-sequence modeling. arXiv preprint arXiv:2408. 15496, 2024.
Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, et al. Anygpt: Unified multimodal IIm with discrete sequence modeling. arXiv preprint arXiv:2402. 2024.
Xiangyu Zhang, Qiquan Zhang, Hexin Liu, Tianyi Xiao, Xinyuan Qian, Beena Ahmed, Eliathamby Ambikairajah, Haizhou Li, and Julien Epps. Mamba in speech: Towards an alternative self- attention. arXiv preprint arXiv:2405. 12609, 2024a.
Zeyu Zhang, Akide Liu, Ian Reid, Richard Hartley, Bohan Zhuang, and Hao Tang. Motion mamba: Efficient and long sequence motion generation with hierarchical and bidirectional selective ssm. arXiv preprint arXiv:2403.07487, 2024b.
Han Zhao, Min Zhang, Wei Zhao, Pengxiang Ding, Siteng Huang, and Donglin Wang. Cobra: Extending mamha to multi-modal large language model for efficient inference. arXiv preprint arXiv:2403. /4520, 2024a.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong. et al. A survey of large language models. arXiv preprint arXiv:2303. 18223, 2023a.
Xiangyu Zhao, Xiangtai Li, Haodong Duan, Haian Huang, Yining Li, Kai Chen, and Hua Yang- Mg-llava: Towards multi-granularity visual instruction tuning. arXiv preprint arXiv:2406. 17777 2024b.
Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint arXivv2304.1123
2023b.
Baichuan Zhou, Ying Hu, Xi Weng, Junlong Jia, Jie Luo, Xien Liu, Ji Wu, and Lei Huang. Tinyllava: A framework of small-scale large multimodal models. arXiv preprint arXiv:2402. 14289, 2024.
Chen Zhu, Wei Ping, Chaowei Xiao, Mohammad Shoeybi, Tom Goldstein, Anima Anandkumar, and Bryan Catanzaro. Long-short transformer: Efficient transformers for language and vision. Advances in neural information processing systems, 34:17723-17736, 2021.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing languala understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vi- sion mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint arXiv:2401.09417 2024a.
Yichen Zhu, Minjie Zhu, Ning Liu, Zhicai Ou, Xiaofeng Mou, and Jian Tang. Llava-phi: Efficient multi-modal assistant with small language model. arXiv preprint arXiv:2401.02330. 2024b.
Itamar Zimerman and Lior Wolf. On the long range abilities of transformers. arXiv preprint arXiv:2311.16620. 2023.
17

---

Published as a conference paper at ICLR 2025
uneer praeedos
Sequeeer tandtrration
Moniin eanty (oclatiio ear na wolioo nultpiicaio))
Mamba Block
Mamba2 Block
Figure 3: Comparison between Mamba and Mamba2 blocks Dao & Gu (2024
A APPENDIX
A.1 MAMBA PRELIMINARIES
Our MILM is composed of Mamba Gu & Dao (023) blocks, which are form of structured state- space sequence models (S4) Gu al. (2021a) with input-dependent selective scanning function. We provide preliminaries regarding S4 models and the Mamba architecture this section. Consider sequential input R", the S4 model is defined by a latent state h(t) R" with four parameters (A, A, B, C) that maps x to another sequence E R", given by the following equation:
(t(t) Ah(t) + Br(() vtt) = Ch(()
(11)
Where (A, A, B) are continuous parameters that require discretization rules transform into discrete parameters that enable parameter computations, given by:
= exp(AA) B (AAA) AB
Then, the discretized form of S4 can be computed as follows:
h Ah4-1 Brt
(12)
yt Ch+
This endows S4 models with beneficial properties such as resolution invariance and automatic nor- malization, and can be connected to gating functions in RNN. also allows these models to be computed as equivalently linear recurrence or a global convolution:
K = CC
(13)
Building upon the S4 framework, Mamba introduces a selective-scan mechanism (SSM) that is input-dependent. The main difference between S4 and Mamba is simply making the parameters (Á, B, C) as functions of the input sequence I, which allows the model to selectively remember or forget information depending on the current data in sequence. Due to the efficient computation for linear recurrence, Mamba is efficiently implemented and achieves significantly lower through- put time compared to transformers. Even without self-attention and MLP layers, Mamba achieves state-of-the-art performance in language, audio, and genomics tasks. Recently, there has also been ongoing research to further improve the architecture and efficiency of the Mamba layer. [ao & Gu (2024) draws the association between linear recurrence and the attention mechanism, and pro- poses Mamba2 with slight modifications to the Mamba architecture, Specifically, they remove the sequential linear projections and produce the SSM parameters A, B, C at the beginning of the block
18

---

Published as a conference paper at ICLR 2025
instead of functions of the input They also introduce additional normalization layer for better optimization stability. Mamba2 achieves even lower throughput than Mamba and outperforms on language benchmarks with slightly fewer parameters. A comparison between Mamba and Mamba2 architecture shown in Fig.
A.2 ADDITIONAL. TRAINING DETAILS
In this section, we provide additional training details for our model. Given that Cobra serves as our base model, the chosen values closely mirror those employed within the Cobra framework. Model configurations and hyperparameter selection can be found in Tab. 5
Configuration
Vision Encoder
DINOv2 + SigLIP VÝT-SO
Backbone LLM
MambaV1-24. MambaV2-2.
Projector
GeLU-MLP 384 384
Img Resolution
Img Tokens
729
Global Batch Size
128
Training Steps
19K
Optimizer
Adam W
LR scheduler
Cosine Decay
Initial LR
2e-5
Weight Decay
0.1
Warmup Ratio
0.03
Number Epochs
Table 5: Detailed training configurations used for EMMA.
Model Architecture and Hyperparameters. We directly use off-the-shelf DINOv2 and SigLIP models as our vision encoder. The GeLU-MLP is simply three-layer MLP with the GeLU acti- vation function in between. For the MambaV1 model, we use the Mamba-2. 2.t.-.22.b--epy version of it. The Mamba2 model is directly taken from the official github page, which is trained on the 300B Pile dataset. We select the version without attention.
Training Data. We provide more details regarding the training data for our model, which consists of three subsets:
1. ALaVA-Mixed--636 is mixture of cnttruction-olllwwiii multimodal data that contains a variety of VQA, OCR, region-level VQA, visual conversation, and language conversation data. This is the same dataset used for finetuning the LLaVA-v1.5 model Liu et al. (2024a). We provide the breakdown for this dataset as a comma-separated list with the origin and corresponding number of samples: LLaVA 158K, ShareGPT 40K, VQAv2 83K, GQA 72K, OKVQA 9K. OCRVQA 80K, A-OKVQA 66K, TextCaps 22K, RefCOCO 48K, and VG 86K, for total of 665K data samples. According to the I.L paper, these data are preprocessed follows: 1. For all VQA datasets, QA pairs from the same training image are merged into a single conversation. 2. For ShareGPT, they filter out invalid conversations. They also truncate long conversations that surpass 2048 tokens. 3. Each QA pair in A-OKVQA is augmented k times, where k is the number of choices per question, to counterbalance the lack of multiple-choice data. They sample 80K conversations from OCRVQA. 5. For Visual Genome, they sample 10 annotations for images with additional annotations. 6. For RefCOCO, conversations are dissected into segments, each containing fewer than 10 conversations.
7. All data splits are concatenated together to form the final 665K dataset.
IVSS-INST contains 220K pairs of image-text instructions that are generated by GPT- 4v. These instructions are visually aligned and context-aware to improve cross-modal alignment. This is the dataset presented by Wang et al. (0023). in an attempt to curate a fine-grained instruction- following dataset by explicitly conditioning the generation of instruction data both language and visual inputs. To generate this dataset, they leverage image data from LVIS Gupta et al. (2019). as well their fine-gra -eeggra annotations prompt the GPT-4V model undertake two key tasks: (1) generate conversational question- answer lists through seasonnas and (2) produce high-quality image descriptions guided by precise bounding box information. To the end, they 11KK images
19

---

Published as a conference paper at ICLR 2025
from LVIS and generate 220K high-quality visual instructions, which consist of 1 10K conversational data and 110K descriptional data. These images along with GPT-AV-generated captions form this dataset.
LRV-Instruction covers 16 vision-and and-language tasks with open-ended instructions and answers. It includes both positive and negative instructions for more robust visual instruction tuning. This dataset [iu et al. (2023b) leverages GPT4 to cover open-ended positive and negative instructions in different linguistic styles for images in the Visual Genome dataset. They use GPT4 create cestructioo-followiiie data with the image size, bounding boxes, and dense captions as input, and generate caption instances in both declarative and interrogative formats. Afterwards, they remove instances with answers longer than 30 words and those with unneeded content. This results a total of over 400k image-visual instruction pairs after filtering-
Data Preprocessing. After combining these three datasets, we finetune our entire model on it. Because we utilize pre-trained vision encoders and Mamba LLM, for data pre-processing, we also adopt the same tokenizer and image transforms from the respective components. For Mamba LLM, we utilize the GPTNeoX TokenizerFast tokenizer to convert input text into textual tokens. For both the SigLIP and D2 ViTs, we use their respective image transforms, as listed below: 1. SigLIP: Resize to (384, 384), center crop. normalize by mean 0.5 and std 0.5. Dino Resize to (384, 384), center crop, normalize by mean 0.485 and std 0.225.
A.3 ADDITIONAL EXPERIMENTS
We present further experimental results on the following datasets showcase the effectiveness of our method: SEED-IMG Li et al. (2024a) evaluates generative comprehension. A12d Hiippala jet al. (202I) and ScienceQA Saikh et al. (2022) both evaluate science-related topics. CCBench is
benchmark on Chinese culture Liu et al. (002cc).
Model
SEED
ScienceQA
A12D
CCBench
Cobra
61.5
63.0
48.6
11.6
EMMA-V1 (Ours)
62.9
66.3
50.5
16.7
EMMA-V2 (Ours)
64.3
65.0
49.1
29.21
Table 6: More performance comparisons across MLLMs with similar sizes. Since none of the other 3B-scale MLLMs report performance on these benchmarks, we report the results for Cobra and ours.
As shown, both versions of EMMA achieves consistent performance gains from its baseline Cobra, demonstrating the effectiveness of pixel-wise and hierarchical alignment in mamba-based VLMs.
A.4 ADDITIONAL EVALUATION DETAILS
In this section, we offer a concise summary of the key aspects that each benchmark emphasizes when assessing the model in order to give an overview of the strengths of our model.
"QAA2/Goyal et al. (2017) consists of 265K image-question suneeeeonosoon-nsseeee answer tuples sourced from the COCO dataset Lin et al. (2014). The evaluation of models in the VQAv2 test set centers on visual recogni- tion, visual grounding, spatial reasoning, and language comprehension.
GQA Hudson & Manning (2019) consists of 22M questions of daily images, where each image is associated with a scene graph of the image's objects, attributes, and relations, generated from the Visual Genome dataset Krishna et al. (2017). The GQA test set rigorously assesses the models' proficiency in visual and compositional reasoning.
WizWiz Gurari et al. (2018) is derived from natural visual question question-answering answering scenario. where visually impaired individuals captured images and verbally posed questions about them. Each image then accompanied by 10 answers from crowd-soureing. This benchmark tests predicting answers to visual questions and determining whether a visual question is unanswerable.
TAA Singh et al. 2019 is multimodal dataset consisting of 45K questions 28K images. This benchmark not only tests recognizing textual information the given images but also reasoning over them.
20

---

Published as a conference paper at ICLR 2025
VSR et al. (2023a) comprises caption-image pairs with true/false labels. Each caption delineates the spatial relationship between two distinct objects within the image. this task, the model is tasked with determining whether the caption accurately describes the image not.
MME Fu et al. (2024) stands as a comprehensive benchmark for evaluating MLLMs, measuring both perception and cognition across total of 14 subtasks. The perception portion includes the recognition coarse-graines and fine-grained objects. The cognition part includes commonsense reasoning, numerical calculation,
translation, and code reasoning. When evaluating our model, we add the scores from perception and cognition tasks as our final score.
MMB Liu al. 2023c) is also comprehensive benchmark for evaluating MILLMs. It surpasses existing similar benchmarks in terms of the number and variety of evaluation questions and abili- ties. Answers are recorded through multiple-choice questions in both English and Chinese versions, enabling performance comparisons under bilingual context.
SEED et al. (2024a) encompasses 19K multiple-choice questions with precise human annota- tions. It covers 12 evaluation dimensions, encompassing the understanding of both image and video modalities. We only select the image subset of the SEED benchmark to evaluate our models.
ScienceQA Saikh et al. (2022) features scientific questions and answers gleaned from academic sources. Models are evaluated on their ability to reason with scientific knowledge using questions, answer choices, and relevant contexts.
AI2D Hiipaaaa et al. (2021) a dataset comprising more than 5K grade school science diagrams with over 15K corresponding multiple-choice questions. This benchmark, like ScienceQA, also tests knowledge of MLLMs on science-related topics.
CCBench Liu et al. (203cc) is part of the MMBench project that evaluates the models on the
domain of Chinese Culture.
POPE Li al. (2023b) targets the evaluation of object hallucination in MLLMs. Test samples include both positive and negative objects (non-existent objects), requiring models to accurately recognize positives and identify negatives. This evaluation benchmark measures hallucination more on the textual modality.
HallusionBench Guan et al. (2024) focuses diagnosing both the visual illusion and knowledge hallucination MILMs. Visual illusion refers to the misinterpretation of accurate visual informa- tion, and knowledge hallucination denotes perceptions formed without relevant visual input. This benchmark ties more closely with testing hallucination on the visual modality.
A.5 ADDITIONAL VISUALIZATIONS
We also present additional visualizations for intermediate features in Mamba MLLMS to demon- strate the effectiveness of our method.
21

---

Published as a conference paper at ICLR 2025
"Describe Image detaill"
Cobra Intermediate Visual Features
Here line plot cellecKW Distribution employment by 2020.
Cobra
line shows distribution emplay ment by economic sector in Kuwat from 20 10 2020.
EMWA
EMMA Intermediate Visual Features (Ours)
Figure 4: As shown, our method better highlights important characteristies of the image, such as the captions and horizontal and vertical axes. Cobra, on the other hand, gradually loses its visual activations on these characteristics, resulting in incorrect reference to the plot as "calledK
"Describe the image detail."
Cobra Intermediate Visual Features
image contains sign writing on it, rusted r.
The image shows building with balcory sign the wall.
0.0
EMMA
EMMA Intermediate Visual Features (Ours)
Figure 5: Similar the previous example, EMMA is able retain focus important visual details even during later intermediate layers, resulting in better sensitivity to fine- one-ggained visual details and less visual ballucination.
22

---

Published as a conference paper at ICLR 2025
"Describe the image close detail."
Cobra Intermediate Visual Features
Two polar bears are playing the water, splashing water, with rocks background.
Cobra
open and black oluject in its moulh.
EMMA
EMMA Intermediate Visual Features (Ours)
Figure 6: EMMA is also able to better grasp objects that span large areas in an image. This example showcases that EMMA is less prone to hallucinations of large objects, opposed to Cobra.
"Describe image in close detail"
Cobra Intermediate Visual Features
AaNahaAskskaAalala
Dober
yellow
around hanging from tree branch.
0.0
EMMA
EMMA Intermediate Visual Features (Ours)
Figure 7: Another qualitative comparison between EMMA and Cobra. While Cobra fails to gen- erate logical answer, EMMA is able to accurately describe the image due its ability capture finegrained visual details.
23

---

Published as a conference paper at ICLR 2025
"Describe
image in close detail."
Cobra Intermediate Visual Features
Cotra
surting white surfooard blue ocean with white waves and rocky shoreline.
0
EMWA
EMMA Intermediate Visual Features (Ours)
Figure 8: Another fail case for Cobra, while EMMA successfully answers the input prompt. Notice that EMMA retains the visual structure of intermediate features, where the human silhouette has been preserved.
A.6 ADDITIONAL. COMPONENT ANALYSIS
Computation cost analysis. We provide the following table for parameter counts of different com- ponents in our model. TimeTrn and Memoryyye denote vraining-wise time and memory require- ments, while Temeeral and MemoryEvl denote evaluation-wise time and memory requirements. Decoder refers to the decoder for generating visual features for pixel-wise alignment, while MFF refers to the multi-scale feature fusion module for the hierarchical alignment. CA refers to the sum of all additional cross-attentinn modules introduced, to separately account for the computational complexity cross-attention operations. Note that this row marked gray as these parameters are also accounted for in MFF. Memory is in terms of MiB, and time is in terms of seconds. The memory of each component calculated by
memor modut memor Saamorrypree-oo module
The time of each component is calculated by
modduue lime pre-
The training results include additional loss calculation and optimization passes, which introduce additional time and memory usage. Note that the inference time reported here multiplied by 256 does not give the actual model inference time as presented in Table (in original paper), this is due to the incorporation of cached operations during inference (which are common in all MLLM models, including transformers). which only necessitate one pass through the ViT and reduced tokens for the LLM during the text generation phase. Here we report actual processing times instead of cached times for the sake of performance measure, but note that cached times are more suitable and realistic for deployment. The following breakdown is performed uniformly with batch size of 1, evaluated on a NVIDIA A.00 GPU.
A.6.1
ANALYSIS
Cross-attention background and overview. The cross attention module Lin et al, (0022) replaces key and value vectors in self-attention with inputs from different sequences, enabling the calculation of attention scores based on alternative information. Currently, cross-attention is commonly used in multimodal models for representation fusion, evidenced by various methods from basic MILM
24

---

Published as a conference paper at ICLR 2025
Components
Parameters
T Timermn
Memoryrrn
Time Fool
MemoryEval
Visual Encoder (ViT)
731M
2e-2
3,600
2e-2
3,600
Projection MLP
47M
2c-3
498
4c-4
460
MambaV] V1-2.88 8B LLM
2.8B
7c-1
25,266
7c-2
11,140
MambaV2-2.7B LLM
2.7B
7e-1
26,210
6e-2
11,054
Decoder
174M
5e-1
1,204
N/A
N/A
MFF
266M
6e-3
1,346
N/A
N/A
CA
10SM
le-3
892
N/A
N/A
Table 7: EMMA Component Analysis
approaches like Flamingo Alayrac et al. (002)) and EVLM Chen al (2024a) advanced repre- sentation fusion MILM methods such as LLAVA-UHD Guo et al. (2025) and MG-LLAVA (Zhao jet al.] (2024b). Cross-attention has demonstrated its capabilities in fusing both same-modal and
multi-modal features in these works, and is the reason why we have also opted for cross-attention.
The use of ceantiionmmehanismss in Mamba. The use of the attention mechanism in Mamba- based architecture is not uncommon. The authors for the original Mamba Gu & Dao (2023) paper has provided a version of attention-augmented Mamba2 model in their follow-up work Mamba2 Dao & Gu (2024). which shows improved performances without introducing significant computational overhead. Furthermore. hybrid Mamba- tranbbb-rrnffrmeer architectures such as Jamba Lieber al. (2024); Team et al. (2024), 7amba Glorioso et al. (2024), and Griffin De et al. (2024) have been proposed to combine the efficiency Mamba and state-space models with the benefits of transformer-based attentions.
Cross-attention in EMMA. We observe that current Mamba-based multimodal models exhibit a gradual loss of fine-grained visual cues in deeper layers of the MILM. Thus, in an attempt to reduce this information loss, we propose to use a hierarchical alignment module to counteract this loss of information. The essence of using cross-attention in work lies in merging information from mul- tiple layers of representation, ensuring that information from intermediate layers is also involved in visual representation alignment to prevent loss of image information at intermediate layers. There- fore, cross-attention is not mandatory, but only serves as a means to merge layer-wise information. Thus, the replacement of with more efficient and simpler attention mechanisms, usage of dif- ferent representation fusion modules can be explored. Nevertheless, due to the minimal number of parameters involved, there may not be significant difference computational efficiency, and hence not explored in this work.
As shown in Tab. 7 cross-attention modules only account for very small proportion (roughly 3 percent) of the model's optimization parameters. Additionally, the cross-attention here is only quadratic in terms of the length of image patches and does not vary based on the length of text, hence not truly quadratic in the strictest sense. Hence the memory and time usage of the cross attention module is extremely small. Also, since the decoder and MFF modules are not utilized during inference time, they pose additional computation cost during testing.
Alternative Attention Mechanisms in EMMA. As an example, we replace the cross-attention modules with lucidrains version Lucidrains (2021) of lincar attention Wang et al.) (2020); Katharopoulos et al. (2020a) in the EMMA- VI model. This enables lincar-wise computation of attention, and achieves the following performances:
Model
GQA
VizWiz
VQA
POPE
MME
MMB
HBench
EMMA-VI-CA
60.5
52.1
57.2
88.0
1572.8
53.2
51.0
EMMA-VI-LA
60.2
52.1
57.7
88.1
1560.0
52.5
46.9
Table Cross-attention vs. Linear Attention
EMMA-VI-S VI-CA denotes the original proposed model with cross-attention, while EMMA-VI-LA denotes EMMA but with linear attention instead. EMMA-VI-LA performs similarly to EMMA- VI-CA on the majority of benchmarks, besides HallusionBench, which experiences a 4% drop. This could be attributed the trade-off of sub-quadratic complexity and overall performance of
25

---

Published as a conference paper at ICLR 2025
linear attention. Nevertheless, EMMA- still outperforms the baseline on HallusionBench by
5%.
A.6.2 FEATURE-FUSION ANALYSIS
Mamba-based feature-fusion methods. To our knowledge, current Mamba-based feature fusion methods include MambaDFuse Li et al. (2024d), fusion-Mamba Dong et al.) (2024) and Fusion- Mamba Xie et al. (2024). MambaDFuse proposes a dual-phase feature fusion module that consists of shallow fuse and deep fuse modules. The shallow fuse module achieves lightweight exchange of features from multiple modalities, while the deep fuse module, a newly designed Multi-modal Mamba (M3) block, guides the generation of modality-fused features and incorporates local detail characteristics from different modalities. Fusion-Mambe proposes a Fusion-Mamba block (FMB) to map cross-moda. features into a hidden state space for interaction, thereby reducing disparities between cross moos- features and enhancing the representation consistency of fused features. Fu- sionaamba designs a Mamba model suitable for fusion tasks by integrating the Visual State Space Model with dynamic convolution and channel-wise attention.
Improvements from previous Mamba-based methods. We believe the novelty of our work lies in two perspectives. First, we are the first to examine feature-fusion of Mamba models in the con- text of multimodal LLMs. Previous feature fusion works only focus on multi-stream multimodal models which do not utilize a Mamba-based LLM, but rather consist of separate branches with similar sizes to encode both modalities. Thus, the differences in underlying model architectures renders different design choices for feature fusion methods. Previous Mamba-based multimodal LLM methods, on the other hand, adapts the Mamba LLM to typical MILMM frameworks without considering feature fusion. Our work the first that considers feature fusion in MI.LMs to boost the quality of visual features to ensure better multimodal alignment, justified with noticeable per- formance gains empirically. Secondly, we are the first to examine feature-fusion of same modality features in Mamba-base modal features, rather than augmenting useful information of same modality features from different scales. Our work proposes novel Mamba- based multi-scee scale feature fusion module that effectively combines visual features from different layers, alleviating the gradual loss of visual information in the Mamba LLM.
models. Previous feature fusion works only consider the fusion of multi-
A.6.3 ADDITIONAL COMPUTATIONAL COSTS ANALYSIS
Training procedure of MLLMs. We first present the training procedure of vision-language models (VLMs) and present the training paradigm of EMMA. Here, we are specifically referring to models inspired by the LLaVA Liu et al. (2024b) architecture, which consists of a vision encoder, a multi- modal projector and backbone LLM, of which our work is based on. The training paradigm for VLaVA-like VLMs consists two portions, pretraining and supervised finetuning. In the pretrain- ing stage, all but the multimodal projector is frozen, where the projector is then trained image-text pairs to learn the mapping from visual to token space. Then, in the supervised finetuning stage, only the vision encoder is frozen, while both the projector and backbone LLM is tuned, extending the LLM to multi-modal tasks.
Training procedure of EMMA. Due the effectiveness of the Mamba LLM, Cobra discovered that Mamba-based MLLMs do not benefit much from the pretraining stage. Thus, we follow the same procedure of Cobra and discard the pretraining stage altogether, and only conduct supervised finetuning for two epochs. For the additional structures introduced, we simply add the structural and hierarchical modules our model, and jointly optimize with the original text generation objective. This simplifies the training of our methods, where only finetuning data is used to end-to-end finetune our entire model. We also do not require separate stage nor additional data for the training of decoder or MFF module, unlike transformer-basee EMU Sun et al. (2023a) and EMU2 Sun et al. (0033)).
Fair comparison of training costs. We have provided the complete breakdown of training and testing costs in Tab. 7 The visual encoder has the same train-time and evaluation values due to it being frozen and un-optmmized during either training and testing. All other components utilize less memory and time resources in evaluation, compared to training. The newly introduced structures, decoder and MFF, are discarded during testing. Note that the training time for decoder is relatively
26

---

Published as a conference paper at ICLR 2025
large compared to other structures given its small size. This is due to the calculation of respective gradients that concern the update of LLM parameters due to pixel- alignmennt loss.
Inference speed clarification. Because both the structural alignment and hierarchical alignment considers visual features, which are used to boost the quality of visual latents and encourage better multi-modal alignment during the training stage, they are discarded during inference they not required for visual question answering. Thus, as shown in Tab. 7 they do not introduce additional complexity during inference. Since EMMA-VI and Cobra uses the Mamba LLM-2.8B backbone, they possess the same inference speeds in Tab.
A.7 FURTHER CLARIFICATION OF MOTIVATION
Our research is grounded in the observation that Mamba models exhibit an imbalance in handling visual and textual information. We argue for the imbalanced quality of visual and textual latents in Mamba models through two perspectives, architecturally and empirically.
Architectural imbalance. The inherent linear scanning mechanism of Mamba poses challenges in effectively extracting visual features on the same level as textual features. On one hand, transformer- based Dosovitskiy (2200) approaches utilize the self-attention mechanism that extracts global re- lationships between different parts of the input sequence. They also utilize a position encoding for learning spatial relationships of visual tokens. On the other hand, CNN-based approaches Krizhevsky et al. (2012) utilize local receptive fields to connect each neuron to local region of the input data, allowing interaction of neighboring pixels in all directions. The effectiveness of both methods in processing visual information lies in their ability to account for global, or to the least, local clusters of the visual input. However, Mamba utilizes selective-scan mechanism that only processes sequences in linear order. While Mamba models achieve excellent performances in tex- tual data, effectively capturing long-tern dependencies in multiple language benchmarks TTy et al. 2020), Mamba models possess difficulties in handling image data, where pixel-wise relationships may span multiple rows and columns. Visual Mamba models have been proposed to counteract this issue through combining additional scans from different directions Liu et al. (2024c) and utilizing a similar position encoding as ViTs Zhu et al. (2024a). However, they nevertheless still utilize the linear selective-scan mechanism. In the case where features possess large embedding dimensions, which is typical in MLLM scenarios, this becomes hard for the model to extract visual cues that are far apart spatially, as the degree of freedom between each visual token across different rows and columns become huge.
Empirical imbalance. Mamba models experiences an imbalance in the empirical evaluation of vision tasks and textual tasks, when scaled in parameters. |Ren et al. (2024) observes the per- formance degradation of visual Mamba models Xu al.] (2024b) when scaled up. Below is table taken directly from their paper.
Model Size (M)
VIM-B
VIM-L
VIM-H
98
340
755
ImageNet Acc.
81.2
81.0
Collapsed
Table 9: Degradation of visual Mamba models when scaled up, sourced from Ren et al. (2024).
As shown, plain Mamba models experience severe degradation in processing visual features when scaled up. In the 700M parameter-scale, the visual Mamba collapses entirely, rendering the need for effective approaches alleviate this performance degradation, specifically in Mamba-based models.
On the other hand, plain Mamba models do not experience performance degradation in processing textual data, as shown from the experiment results in the original Mamba papers Gu & Dao (023);
Dao & Gu 2024)..
Model
Mamba- 790M
Mamba-1.44
Mamba-2.8B
Mamba2-780R
Mamba2-1.3B
Mamba2-2.7B
Size (M) HellaSwag LAMBADA 62.7
790
1,400
2,800
780
1,300
2,700
55.1
59.
66.1
54.9
59.9
66.6
64.9
69.2
61.7
65.7
69.7
Table 10: Performance of textual Mamba models when scaled up, taken from Gu & Dao 2023
Dao & Gu (2024).
27

---

Published as a conference paper at ICLR 2025
Thus, we observe an imbalance of the processing capabilities of Mamba on visual and textual in- puts, when scaled up in parameerr models. This phenomenon renders the need for more effective visual processing in large-scaled Mamba models to ensure that visual and textual features are adequately aligned.
Motivation behind loss of fine-grained visual cues. We discovered the problem of the loss of fine- grained visual cues when directly extracting and comparing intermediate features of the Cobra model Zhao et al. (2024a). Specifically, we find that visual features in deeper layers of the Mamba LLM exhibit increased portions of noise (manifested in unnatural specs in the image) and loss of structural integrity (manifested in the loss of topologiaal structures) the Cobra baseline model, which can be found in our visualizations in the main paper. Subsequently, we designed the hierarchical alignment loss to enhance the participation of intermediate visual features to alleviate the gradual loss of visual information. Qualitative examples are shown in Appx. A.5
A.7.1 SIGNIFICANCE OF EMMA
Application of Mamba in MLLMs. To deploy Mamba models in MLLM settings, where large- scaled Mamba models are necessary in processing both textual and visual information, it is crucial to preserve a balance in the quality of visual and textual features to achieve multi-modal alignment. Previous Mamba MLLM works such as VL Mamba and Cobra directly replaces the transformer LLM with the Mamba LLM model without addressing this imbalance of feature quality. Still, the performance of these models show the potential of extending mambas to the realm of MILLMs, especially with their fast inference speeds.
Ineffectiveness of transformer-basee techniques. Furthermore, as a preliminary study for this work, we have also tried transformer-basee multimodal alignment approaches that enhance visual features or encourage multimodal alignment, such as visual-feature compression Li et al. (2024c). masked-alignment Mizrahi et al. (2024); Bitton et al. (202T), and contrastive losses |Shen et al. (2021);; Radford et al. (2021), all of which fail in the Mamba LLM setting. Here, we show the respective model performances the TextVQA dataset, which we find is good indicator for overall model performance.
Model
EMMA-VI EMMA-V2
Cobra
Compression
Masked
Contrastive
TextVQA 57.2
56.2
52.4
45.8
46.1
50.2
Table 11: Ablation with current transformer-based approaches on TextVQA
As shown, transformer- approaches tailored for multimodal alignment in MILM settings may in fact negatively impact model performance, which renders the necessity to devise Mamba-specific multi-modal alignment approaches.
Novelty of our approach. Autoregressive pretraining on the visual features are shown to improve visual Mamba models Ren et al. (2024), and for the first time allows visual Mamba models to scale up to large-sized models (600M). Given the effectiivnness of autoregressive pretraining in promoting the single-modal visual Mamba model, we extend this approach to the multimodal L.M setting. Subsequently, we propose the pixel-wise alignment loss to encourage the generated autoregressive features to structurally align with the input image. Furthermore, when visualizing the LLM fea- tures, we notice a gradual decline of the intermediate visual features, causing the final generated visual feature to degrade. In response, we propose the multi-scale feature fusion module tackle this issue, leading to better preservation of intermediate visual features, inducing better multimodal alignment of visual and textual features. In essence, our method specifically tackles the issue of multi-modal alignment in Mamba models, while the direct application of transformer-based multi- modal methods have failed. As result, FMMA demonstrates improved performances on a variety of metrics, and most importantly, significantly reduces the degree of visual hallucinations in the Hal- lusionBench benchmark, surpassing transformer models with twice the size. We believe this study provides groundwork for further research of Mamba models in the MLLM framework poten- tial replacement for transformer-bast approaches, with their comparable performance and superior inference latency.
Potential impact and application of our observations and techniques. On one hand, we believe the insights introduced in this paper will help inspire future research of mamba models in the realm
28

---

Published as a conference paper at ICLR 2025
of MLLM and other downstream tasks. Specifically, approach strengthen the quality of vi- sual features in large-scaled mamba models may provide future researchers with a framework for multimodal alignment in Mamba models. Our weight Mamba-bases decoder also serves as an efficient means to conduct feature reconstruction in the visual modality. Furthermore, our multi- scale feature fusion module can be inspiration for combining same-modality features in future Mamba-based frameworks, On the other hand, we aim for our research to pitch Mamba as poten- tial contender among ransformer-based architectures. Given Mamba's 's swift inference capabilities, we show in this work that former models, and underscores its potential for widespread adoption and integration into diverse Al applications.
MLLMs can achieve competitive performance with trans-
29


Reviews:
[{"review_id": "Y5XeK4IpFd", "reviewer": "Reviewer_yCEA", "summary": "Mamba has seen widespread use in large language models (LLMs) due to its exceptional efficiency. However, Mamba struggles with processing visual signals, limiting its application in multi-modal LLMs. To address this issue, this paper proposes improving Mamba’s ability to handle visual signals through structural and hierarchical alignment. The authors argue that Mamba’s difficulty with visual data stems from the lack of positional encoding, which causes the gradual loss of fine-grained spatial information during processing. To mitigate this, the paper introduces the EMMA method, which enhances the structural consistency between Mamba’s output visual features and the original image by reconstructing the image through a decoder. Additionally, EMMA incorporates cross-attention within Mamba, combining multi-level intermediate features via cross-attention to form the final output, thereby preventing the loss of fine-grained details in deeper layers. The results show that EMMA leads to improved performance.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The proposed method is simple yet effective.", "weaknesses": "1. **Increased Complexity Due to Cross-Attention**:  \n   EMMA introduces cross-attention operations into the Mamba network, reverting the model’s original sub-quadratic complexity back to quadratic complexity, which undermines the original intent of using Mamba. Could the authors provide the computational complexity of the cross-attention operations compared to the overall method? Is cross-attention necessary for fusing intermediate features? Could simpler alternatives be explored to achieve the same goal?\n\n2. **Training Costs and Fair Comparisons**:  \n   EMMA involves additional training steps, which should be explicitly detailed in the paper. Furthermore, when comparing EMMA to other methods, the additional training costs should be listed separately to ensure a fair comparison (training time, GPU hours and memory requirement). Since EMMA alters the original MLLM architecture, its inference complexity will also increase. Table 4 should include a breakdown of the increased complexity due to hierarchical alignment. For fairness, Table 4 should also provide EMMA’s inference speed when using Mamba LLM-2.8B.\n\n3. **Limited Technical Contribution**:  \n   EMMA’s approach is somewhat simplistic, as it merely enhances the consistency between deep and shallow features and the original image at the output stage. This method requires extra training, substantial computational resources, and introduces additional structures during deployment. As a result, the technical contribution appears somewhat trivial. It would be better if this paper could provide more detailed analysis of how their method addresses specific challenges in multimodal learning that previous methods have struggled with.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bjWkBcv7wr", "reviewer": "Reviewer_5nFQ", "summary": "This paper proposes a Mamba based integration for vision-language models. Similar to the VLM based framework where there are one vision encoder, one projector, and one LLM. The proposed method takes one image and its corresponding caption as input, followed by multi-scale feature extraction and fusion. A pixel-wise alignment loss is applied to the visual features for structural visual cues preservation, and an autoregressive NLP loss is applied to the textural feature. Experiments are conducted on several benchmark datasets.", "questions": "Overall, the pixel-alignment loss design is interesting but there are many ambiguious technical details to make the contribution clear. Further clarification is required to better position the proposed method within the scope of the visual instruction tuning design of VLMs", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Pixel-wise alignment loss is interesting. The LLM outputs are text tokens and are further decoded into the image domain for measuring pixel-level similarity with the input image. \n2. The hierarchical alignment via multi-scale fusion is developed to retain fine-grained visual features\n3. The experimental results seem promising on the benchmark datasets.", "weaknesses": "1. While the main contribution is set for the loss design of model training, the proposed loss function seems rather general and not specifically for Mamba based structure. The general pixel-wise alignment loss seems functional for all VLMs, rather than only mamba structure. Can you clarify how the proposed loss function leverages or is tailored to the unique properties of Mamba architectures compared to transformer-based VLMs?\n2. The inputs require both images and captions, how the captions are acquired are not illustrated well. A more detailed description of the data preparation process is desired here. Also, the decoded procedure via Mamba is not illustrated clearly. Is SFT or a similar finetuning process still needed for training VLM? Is related VQA data still required? Such technical details are not shown clearly in the manuscript to show the overall model tuning picture.  It is better to provide a step-by-step explanation of the decoding procedure and clarification on the fine-tuning process to help address the ambiguity.\n3. The pixel-alignment loss is set for structure preservation, the role of LLM here is not clear as the caption has been sent to it. So LLM seems to function as a single mapping to fulfill this objective loss. Can you elaborate on the specific role of the LLM in the context of the pixel-alignment loss? \n4. Feature fusion is not motivated well and seems a general operation for representation enhancement. It would be better to provide a more detailed justification for the choice of feature fusion technique. Specifically, an explanation is expected of how your approach differs from or improves upon standard feature fusion methods, particularly in the context of Mamba-based architectures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ymQcjGNqrS", "reviewer": "Reviewer_T8Kc", "summary": "The paper introduces Empowering Multi-modal Mamba with Structural and Hierarchical Alignment (EMMA), enhancing Mamba multi-modal large language models (MLLM) by improving their ability to extract fine-grained visual information. EMMA uses a pixel-wise alignment module for better structural alignment and a multi-scale feature fusion (MFF) module to maintain visual information during cross-modal alignment. Extensive experiments demonstrate that EMMA achieves lower latency and faster performance compared to other Mamba-based MLLMs, with superior cross-modal alignment and reduced hallucination. The code for this model will be made available.", "questions": "1. Currently, models use ViT as the image encoder. Is it possible to build an MLLM entirely based on Mamba? What potential advantages and challenges might this approach entail?\n\n2. In transformer-based LLMs, is there a similar issue with pixel-level alignment? Would the method proposed in the paper also be applicable to transformer architectures?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Well Written: The paper is clearly articulated, making the complex concepts accessible and easy to understand for the reader.\n\n2. Motivation Well Clarified: The authors firstly identify the imbalance in the quality of visual and textual latent features in MLLM Mamba models. They propose a pixel-wise alignment approach to autoregressively enhance the learning and processing of structured visual features, leading to improved cross-modal alignment between visual and textual latents.\n\n3. Experiments Solid: The paper presents comprehensive experiments conducted on a variety of multi-modal benchmarks. These experiments include thorough comparisons with current state-of-the-art models based on both Mamba and transformer architectures, demonstrating the robustness of the proposed approach.", "weaknesses": "1. Why is there an imbalance in the quality of visual and textual latents in the Mamba LLM, where coarse visual features are ineffectively aligned with higher-quality textual features? How does this differ from transformer-based models? Do transformer-based models face the same issue? \n\n2. Could you provide a more objective, quantitative analysis of the loss of fine-grained visual cues in the LLM, as the current visualizations with just a few images seem insufficient? Is this phenomenon common across MLLMs, or is it specific to the use of the Mamba LLM?\n\n3. Why do the current results show that the Mamba LLM still lags behind the transformer architecture? In this case, does the speed advantage gained by using the Mamba architecture sufficiently compensate for the performance loss?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "viSOUT2PDo", "reviewer": "Reviewer_yy1F", "summary": "Mamba-based architectures are promising for deep learning models but current Mamba multi-modal large language models (MLLM) are insufficient in extracting visual features. This paper proposes Empowering Multi-modal Mamba with Structural and Hierarchical Alignment (EMMA). It includes a pixel-wise alignment module for structural alignment at the image level to extract fine-grained visual information autoregressively. Additionally, a multi-scale feature fusion (MFF) module is proposed for hierarchical alignment at the feature level to prevent the degradation of visual information. Extensive experiments on various multi-modal benchmarks show that the model has lower latency than other Mamba-based MLLMs and is nearly four times faster than transformer-based MLLMs of similar scale during inference.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The imbalance in the quality of visual and texture latent in the Mamba-based VLM does make sense to investigate and this paper proposes an effective method, named Empowering Multi-modal Mamba with Structural and Hierarchical Alignment (EMMA) to solve this problem.\n\n2. The experiments are sufficient to verify the effectiveness of the proposed method.\n\n3. The paper is well-written and easy to follow.", "weaknesses": "1. EMMA essentially enhances the visual representation ability of the model. Therefore, the comparison with peers should be conducted, e.g. the contrastive learning in CLIP and SigLIP. Besides, does the masked image construction loss achieve a function similar to EMMA? \n\n2. EMMA uses the visual features from the vision encoder as the target of the Pixel-wise Alignment Loss. However, visual features from the vision encoder may lose the fine-grained information. Does this problem affect the performance of EMMA?\n\n3. High-resolution image is an important direction for MLLM. Structural constraints on the high-resolution visual features are an interesting point to discuss. Authors are suggested to conduct the experiment under a high-resolution setting.\n\n4. Authors are suggested to display more qualitative results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)