MM-LLMs: Recent Advances in MultiModal Large Language Models (2024)

Abstract: In the past year, MultiModal Large Language Models (MM-LLMs) have undergone substantial advancements, augmenting off-the-shelf LLMs to support MM inputs or outputs via cost-effective training strategies. The resulting models not only preserve the inherent reasoning and decision-making capabilities of LLMs but also empower a diverse range of MM tasks. In this paper, we provide a comprehensive survey aimed at facilitating further research on MM-LLMs. Initially, we outline general design formulations for model architecture and training pipeline. Subsequently, we introduce a taxonomy encompassing 126 MM-LLMs, each characterized by its specific formulations. Furthermore, we review the performance of selected MM-LLMs on mainstream benchmarks and summarize key training recipes to enhance the potency of MM-LLMs. Finally, we explore promising directions for MM-LLMs while concurrently maintaining a real-time tracking website1 for the latest developments in the field. We hope that this survey contributes to the ongoing advancement of the MM-LLMs domain.

heading: 1 Introduction
text: MultiModal (MM) pre-training research has witnessed significant advancements in recent years, consistently pushing the performance boundaries across a spectrum of downstream tasks (Li et al., 2020; Akbari et al., 2021; Fang et al., 2021; Yan et al., 2021; Li et al., 2021; Radford et al., 2021; Li et al., 2022; Zellers et al., 2022; Zeng et al., 2022b; Yang et al., 2022; Wang et al., 2022a,b). However, as the scale of models and datasets continues to expand, traditional MM models incur substantial computational costs, particularly when trained
*Equal contributions. †Corresponding authors. ‡This work was done when Duzhen Zhang was interning
at Tencent, AI Lab, Beijing, China. 1https://mm-llms.github.io
from scratch. Recognizing that MM research operates at the intersection of various modalities, a logical approach is to capitalize on readily available pre-trained unimodal foundation models, with a special emphasis on powerful Large Language Models (LLMs) (OpenAI, 2022). This strategy aims to mitigate computational expenses and enhance the efficacy of MM pre-training, leading to the emergence of a novel field: MM-LLMs.
MM-LLMs harness LLMs as the cognitive powerhouse to empower various MM tasks. LLMs contribute desirable properties like robust language generation, zero-shot transfer capabilities, and In-Context Learning (ICL). Concurrently, foundation models in other modalities provide highquality representations. Considering foundation models from different modalities are individually pre-trained, the core challenge facing MM-LLMs is how to effectively connect LLMs with models in other modalities to enable collaborative inference. The predominant focus within this field has been on
12401
refining alignment between modalities and aligning with human intent via an MM Pre-Training (PT) + MM Instruction-Tuning (IT) pipeline.
With the debut of GPT-4(Vision) (OpenAI, 2023) and Gemini (Team et al., 2023), showcasing impressive MM understanding and generation capabilities, a research fervor on MM-LLMs has been sparked. Initial research primarily focuses on MM content comprehension and text generation, encompassing tasks such as image-text understanding, exemplified by projects like BLIP-2 (Li et al., 2023e), LLaVA (Liu et al., 2023e), MiniGPT4 (Zhu et al., 2023a), and OpenFlamingo (Awadalla et al., 2023); video-text understanding, as demonstrated by initiatives such as VideoChat (Li et al., 2023f), Video-ChatGPT (Maaz et al., 2023), and LLaMA-VID (Li et al., 2023j); and audio-text understanding, as seen in projects like QwenAudio (Chu et al., 2023b). Later, the capabilities of MM-LLMs have been expanded to support specific modality outputs. This includes tasks with image-text output, such as GILL (Koh et al., 2023a), Kosmos-2 (Peng et al., 2023), Emu (Sun et al., 2024), and MiniGPT-5 (Zheng et al., 2023b); as well as speech/audio-text output, exemplified by projects like SpeechGPT (Zhang et al., 2023a) and AudioPaLM (Rubenstein et al., 2023). Recent research endeavors have focused on mimicking human-like any-to-any modality conversion, shedding light on the path to artificial general intelligence. Some efforts aim to amalgamate LLMs with external tools to reach an approaching any-to-any MM comprehension and generation, such as VisualChatGPT (Wu et al., 2023a), HuggingGPT (Shen et al., 2023), and AudioGPT (Huang et al., 2023b). Conversely, to mitigate propagated errors in the cascade system, initiatives like NExT-GPT (Wu et al., 2023d), CoDi-2 (Tang et al., 2023c), and ModaVerse (Wang et al., 2024d) have developed end-to-end MM-LLMs of arbitrary modalities. The timeline of MM-LLMs is depicted in Figure 1.
In this paper, we present a comprehensive survey aimed at facilitating further research on MM-LLMs. To provide readers with a holistic understanding of MM-LLMs, we initially delineate general design formulations from model architecture (Section 2) and training pipeline (Section 3). We break down the general model architecture into five components: Modality Encoder (Section 2.1), Input Projector (Section 2.2), LLM Backbone (Section 2.3), Output Projector (Section 2.4), and Modality Generator (Section 2.5). The training pipeline elu-
cidates how to enhance a pre-trained text-only LLM to support MM input or output, primarily consisting of two stages: MM PT (Section 3.1) and MM IT (Section 3.2). In that section, we also provide a summary of mainstream datasets for MM PT and MM IT. Next, we establish a taxonomy encompassing 126 State-of-the-Art (SOTA) MM-LLMs, each characterized by specific formulations, and summarize their development trends in Section 4. In Section 5, we comprehensively review the performance of major MM-LLMs on mainstream benchmarks and distill key training recipes to enhance the efficacy of MM-LLMs. In Section 6, we offer promising directions for MMLLMs research. Moreover, we have established a website (https://mm-llms.github.io) to track the latest progress of MM-LLMs and facilitate crowdsourcing updates. Finally, we summarize the entire paper in Section 7 and discuss related surveys on MM-LLMs in Appendix A. We aspire for our survey to aid researchers in gaining a deeper understanding of this field and to inspire the design of more effective MM-LLMs.

heading: 2 Model Architecture
text: In this section, we provide a detailed overview of the five components comprising the general model architecture, along with the implementation choices for each component, as illustrated in Figure 2. MM-LLMs that emphasize MM understanding only include the first three components. During training, the Modality Encoder, LLM Backbone, and Modality Generator are generally maintained in a frozen state. The primary optimization emphasis is on Input and Output Projectors. Given that Projectors are lightweight components, the proportion of trainable parameters in MM-LLMs is notably small compared to the total parameter count (typically around 2%). The overall parameter count is contingent on the scale of the core LLM utilized in the MM-LLMs. As a result, MM-LLMs can be efficiently trained to empower various MM tasks.

heading: 2.1 Modality Encoder
text: The Modality Encoder (ME) is tasked with encoding inputs from diverse modalities IX to obtain corresponding features FX , formulated as follows:
FX = MEX(IX). (1)
Various pre-trained encoder options MEX exist for handling different modalities, where X can be
image, video, audio, 3D, etc. Next, we will offer a concise introduction organized by modality.
Visual Modality For images, there are various optional encoders: NFNet-F6 (Brock et al., 2021), ViT (Dosovitskiy et al., 2020), CLIP ViT (Radford et al., 2021), Eva-CLIP ViT (Fang et al., 2023), BEiT-3 (Wang et al., 2023d), OpenCLIP (Cherti et al., 2023), Grounding-DINOT (Zhang et al., 2022b) with Swin-T (Liu et al., 2021b) backbone, DINOv2 (Oquab et al., 2023), SAM-HQ (Kirillov et al., 2023) with MAE (He et al., 2022), RAM++ (Zhang et al., 2023i) with Swin-B backbone, InternViT (Chen et al., 2023j), and VCoder (Jain et al., 2023). For videos, they can be uniformly sampled to 5 frames, undergoing the same pre-processing as images.
Audio Modality is typically encoded by CFormer (Chen et al., 2023b), HuBERT (Hsu et al., 2021), BEATs (Chen et al., 2023g), Whisper (Radford et al., 2023), and CLAP (Wu et al., 2023e).
3D Point Cloud Modality is typically encoded by ULIP-2 (Salesforce, 2022) with a PointBERT (Yu et al., 2022) backbone.
Moreover, to handle numerous heterogeneous modal encoders, some MM-LLMs, particularly any-to-any ones, use ImageBind (Girdhar et al., 2023), a unified encoder covering six modalities, including image/video, text, audio, heat map, inertial measurement units, and depth. We provide a brief introduction to some mainstream modality encoders in Appendix B.

heading: 2.2 Input Projector
text: The Input Projector ΘX→T is tasked with aligning the encoded features of other modalities FX with the text feature space T . The aligned features as prompts PX are then fed into the LLM
Backbone alongside the textual features FT . Given X-text dataset {IX , t}, the goal is to minimize the X-conditioned text generation loss Ltxt-gen:
argmin ΘX→T
Ltxt-gen(LLM(PX ,FT ), t), (2)
where PX = ΘX→T (FX). The Input Projector can be achieved directly
by a Linear Projector or Multi-Layer Perceptron (MLP), i.e., several linear projectors interleaved with non-linear activation functions. There are also more complex implementations like Cross-attention, Q-Former (Li et al., 2023e), PFormer (Jian et al., 2023), and MQ-Former (Lu et al., 2023a). Cross-attention (Perceiver Resampler) (Alayrac et al., 2022) uses a set of trainable vectors as queries and the encoded features FX as keys to compress the feature sequence to a fixed length. The compressed representation is then fed directly into the LLM or further used for X-Text cross-attention fusion. Q-Former extracts relevant features from FX with learnable queries, and the selected features are then used as prompts PX . Meanwhile, P-Former generates “reference prompts," imposing an alignment constraint on the prompts produced by Q-Former. MQ-Former conducts a fine-grained alignment of multi-scale visual and textual signals. However, Q-, P-, and MQ-Former require an additional PT process for initialization.

heading: 2.3 LLM Backbone
text: Taking LLMs (Zhao et al., 2023c; Naveed et al., 2023; Mo et al., 2024) as the core agents, MMLLMs can inherit some notable properties like zero-shot generalization, few-shot ICL, Chain-ofThought (CoT), and instruction following. The LLM Backbone processes representations from various modalities, engaging in semantic understanding, reasoning, and decision-making regarding the
inputs. It produces (1) direct textual outputs t, and (2) signal tokens SX from other modalities (if any). These signal tokens act as instructions to guide the generator on whether to produce MM contents and, if affirmative, specify the content to produce:
t, SX = LLM(PX ,FT ), (3)
where the aligned representations of other modalities PX can be considered as soft Prompt-tuning for the LLM. Moreover, some works have introduced Parameter-Efficient Fine-Tuning (PEFT) methods, such as Prefix-tuning (Li and Liang, 2021), LoRA (Hu et al., 2021), and LayerNorm tuning (Zhao et al., 2024). In these cases, the number of additional trainable parameters is exceptionally minimal, even less than 0.1% of the total LLM parameter count. We provide an introduction to mainstream PEFT methods in Appendix C.
The commonly used LLMs in MM-LLMs include Flan-T5 (Chung et al., 2022), ChatGLM (Zeng et al., 2022a), UL2 (Tay et al., 2022), Persimmon (Elsen et al., 2023), Qwen (Bai et al., 2023a), Chinchilla (Hoffmann et al., 2022), OPT (Zhang et al., 2022c), PaLM (Chowdhery et al., 2023), LLaMA (Touvron et al., 2023a), LLaMA-2 (Touvron et al., 2023b), and Vicuna (Chiang et al., 2023). We provide a brief introduction to some representative LLMs in Appendix D.

heading: 2.4 Output Projector
text: The Output Projector ΘT→X maps the signal token representations SX from the LLM Backbone into features HX understandable to the following Modality Generator MGX . Given the X-text dataset {IX , t}, t is first fed into LLM to generate the corresponding SX , then mapped into HX . To facilitate alignment of the mapped features HX , the goal is to minimize the distance between HX and the conditional text representations of MGX :
argmin ΘT→X
Lmse(HX , τX(t)). (4)
The optimization only relies on captioning texts, without utilizing any audio or visual resources X , where HX = ΘT→X(SX) and τX is the textual condition encoder in MGX . The Output Projector is implemented by a Tiny Transformer with a learnable decoder feature sequence or MLP.

heading: 2.5 Modality Generator
text: The Modality Generator MGX is tasked with producing outputs in distinct modalities. Commonly,
existing works use off-the-shelf Latent Diffusion Models (LDMs) (Song et al., 2021; Bao et al., 2022; Zhao et al., 2022), i.e., Stable Diffusion (Rombach et al., 2022) for image synthesis, Zeroscope (Cerspense, 2023) for video synthesis, and AudioLDM2 (Liu et al., 2023b,c) for audio synthesis. The features HX mapped by the Output Projector serve as conditional inputs in the denoising process to generate MM content. During training, the ground truth content is first transformed into a latent feature z0 by the pre-trained VAE (Kingma and Welling, 2013). Then, noise ϵ is added to z0 to obtain the noisy latent feature zt. A pre-trained Unet (Ronneberger et al., 2015) ϵX is used to compute the conditional LDM loss LX-gen as follows: LX-gen := Eϵ∼N (0,1),t||ϵ− ϵX(zt, t,HX)||22, (5)
which optimizes parameters ΘX→T and ΘT→X by minimizing LX-gen.

heading: 3 Training Pipeline
text: MM-LLMs’ training pipeline can be delineated into two principal stages: MM PT and MM IT.

heading: 3.1 MM PT
text: During the PT stage, typically leveraging the XText datasets, Input and Output Projectors are trained to achieve alignment among various modalities by optimizing predefined objectives. For MM understanding models, optimization focuses solely on Equation (2), while for MM generation models, optimization involves Equations (2), (4), and (5). In the latter case, Equation (2) also includes the ground-truth signal token sequence.
The X-Text datasets include Image-Text, VideoText, and Audio-Text, with Image-Text having two types: Image-Text pairs (e.g., <img1> <txt1>) and interleaved Image-Text corpus (e.g., <txt1><img1><txt2><txt3><img2><txt4>). Details of X-Text datasets are shown in Table 3.

heading: 3.2 MM IT
text: MM IT is a method that entails fine-tuning pretrained MM-LLMs using instruction-formatted datasets (Wei et al., 2021). Through this process, MM-LLMs can generalize to unseen tasks by adhering to new instructions, thereby enhancing zeroshot performance. This straightforward yet impactful concept has catalyzed subsequent success in the field of NLP, exemplified by works such as InstructGPT (Ouyang et al., 2022), OPT-IML (Iyer et al., 2022), and InstructBLIP (Dai et al., 2023).
MM IT comprises Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF), aiming to align with human intents and enhance the interaction capabilities of MM-LLMs. SFT converts part of the PT stage data into an instruction-aware format. Using visual Question-Answer (QA) as an example, various templates may be employed like (1) “<Image>{Question}" A short answer to the question is; (2) “<Image>" Examine the image and respond to the following question with a brief answer: “{Question}. Answer:"; and so on. Next, it fine-tunes pre-trained MM-LLMs using the same optimization objectives. SFT datasets can be structured as either single-turn QA or multi-turn dialogues.
After SFT, RLHF involves further fine-tuning of the model, relying on feedback regarding the MM-LLMs’ responses (e.g., Natural Language
Feedback (NLF) labeled manually or automatically) (Sun et al., 2023b). This process employs a reinforcement learning algorithm to effectively integrate the non-differentiable NLF. The model is trained to generate corresponding responses conditioned on the NLF (Chen et al., 2023i; Akyürek et al., 2023). The statistics for SFT and RLHF datasets are presented in Table 4 of Appendix G.
The datasets used by existing MM-LLMs in the MM PT and MM IT stages are diverse, but they are all subsets of the datasets in Tables 3 and 4.

heading: 4 SOTA MM-LLMs
text: As shown in Figure 3, we classify the 126 SOTA MM-LLMs from both functional and design perspectives. In the design division, “Tool-using” denotes treating the LLM as a black box and provid-
ing access to certain MM expert systems to perform specific MM tasks via reasoning, while “Endto-End” signifies that the entire model is trained jointly in an end-to-end manner. Based on the previously defined design formulations, we also conduct a comprehensive comparison of the architectures and training dataset scales for 43 of these SOTA MM-LLMs, as illustrated in Table 1. Next, we will summarize their developmental trends and briefly introduce the core contributions of some representative models in Appendix E.
Trends in Existing MM-LLMs: (1) Progressing from a dedicated emphasis on MM understanding to the generation of specific modalities and further evolving into any-to-any modality conversion (e.g., MiniGPT-4 → MiniGPT-5 → NExT-GPT); (2) Advancing from MM PT to SFT and then to RLHF, the training pipeline undergoes continuous refine-
ment, striving to better align with human intent and enhance the model’s conversational interaction capabilities (e.g., BLIP-2 → InstructBLIP → DRESS); (3) Embracing Diversified Modal Extensions (e.g., BLIP-2 → X-LLM and InstructBLIP → X-InstructBLIP); (4) Incorporating a HigherQuality Training Dataset (e.g., LLaVA → LLaVA1.5); (5) Adopting a More Efficient Model Architecture, transitioning from complex Q- and P-Former input projector modules in BLIP-2 and DLP to a simpler yet effective linear projector in VILA.

heading: 5 Benchmarks and Performance
text: To provide a comprehensive performance comparison, we have compiled a table featuring major MM-LLMs across 18 Vision-Language (VL) benchmarks, as reported in various papers (Li et al., 2023e; Chen et al., 2023d,f; Lin et al., 2023). This
information is presented in Table 2, with detailed descriptions of these benchmarks available in Appendix F. Given the numerous benchmarks available, we focus on evaluating and comparing different MM-LLMs based on OKVQA, IconVQA, VQAv2, and GQA.
OKVQA includes questions requiring reasoning with a variety of knowledge types, such as commonsense, world knowledge, and visual knowledge. MiniGPT-v2 and MiniGPT-v2-chat perform best in this benchmark, showcasing their outstanding reasoning abilities. IconVQA emphasizes the importance of abstract diagram comprehension and holistic cognitive reasoning in real-world diagram-based word problems, requiring both perceptual acumen and versatile cognitive reasoning. MiniGPT-v2 and MiniGPT-v2-chat also excel in this benchmark, highlighting their exceptional perception and cognitive reasoning capabilities. VQAv2 is a more balanced VQA dataset where each question is paired with a series of images. VILA-13B performs best in this benchmark, demonstrating its superior ability to comprehend multimodal information and its resistance to language biases in the knowledge it acquires. GQA is a VQA dataset focusing on image scene graphs, offering impartial compositional questions derived from real-world images. Each question is associated with a structured representation of its meaning and the detailed logical steps required to answer it. LLaVA-1.5 and VILA-7B perform best in this benchmark, illustrating their excellent reasoning abilities in this domain.
Following this, we will outline training recipes that enhance the effectiveness of MM-LLMs, drawing insights from SOTA models.
Training Recipes Firstly, higher image resolution can incorporate more visual details for the
model, benefiting tasks that require fine-grained details. For example, LLaVA-1.5 and VILA employ a resolution of 336 × 336, while Qwen-VL and MiniGPT-v2 utilize 448 × 448. However, higher resolutions lead to longer token sequences, incurring additional training and inference costs. MiniGPT-v2 addresses this by concatenating 4 adjacent visual tokens in the embedding space to reduce length. Recently, Monkey (Li et al., 2023l) proposed a solution to enhance the resolution of input images without retraining a high-resolution visual encoder, utilizing only a low-resolution visual encoder, supporting resolutions up to 1300× 800. To enhance the understanding of rich-text images, tables, and document content, DocPedia (Feng et al., 2023) introduced a method to increase the visual encoder resolution to 2560 × 2560, overcoming the limitations of poorly performing low resolutions in open-sourced ViT. Secondly, the incorporation of high-quality SFT data can significantly improve performance in specific tasks, as evidenced by the addition of ShareGPT4V data to LLaVA-1.5 and VILA-13B, as shown in Table 2. Moreover, VILA reveals several key findings: (1) Performing PEFT on the LLM Backbone promotes deep embedding alignment, crucial for ICL; (2) Interleaved Image-Text data proves beneficial, whereas ImageText pairs alone are sub-optimal; (3) Re-blending text-only instruction data (e.g., unnatural instruction (Honovich et al., 2022)) with image-text data during SFT not only addresses the degradation of text-only tasks but also enhances VL task accuracy.

heading: 6 Future Directions
text: In this section, we explore promising future directions for MM-LLMs across the following aspects:
More General and Intelligent Models We can enhance the MM-LLMs’ strength from the following four key avenues: (1) Expanding Modalities: Current MM-LLMs mainly support the following modalities: image, video, audio, 3D, and text. However, the real world involves a broader range of modalities. Extending MM-LLMs to accommodate additional modalities (e.g., web pages, heat maps, and figures&tables) will increase the model’s versatility, making it more universally applicable; (2) Diversifying LLMs: Incorporating various types and sizes of LLMs provides practitioners with the flexibility to select the most appropriate one based on their specific requirements; (3) Improving MM IT Dataset Quality: Current MM IT datasets have ample room for improvement and expansion. Diversifying the range of instructions can enhance the effectiveness of MM-LLMs in understanding and executing user commands; (4) Strengthening MM Generation Capabilities: Most current MMLLMs are predominantly oriented towards MM understanding. Although some models have incorporated MM generation capabilities, the quality of generated responses may be constrained by the capacities of the LDMs. Exploring the integration of retrieval-based approaches (Asai et al., 2023; Gao et al., 2023a; Kang et al., 2024) holds significant promise in complementing the generative process, enhancing the overall performance of the model.
More Challenging Benchmarks Existing benchmarks may not sufficiently challenge the capabilities of MM-LLMs, as many datasets have appeared to varying degrees in the PT or IT sets. This implies that the models might have already learned these tasks during training. Moreover, current benchmarks predominantly focus on the VL subfield. Therefore, it is crucial for the development of MM-LLMs to construct a more challenging, largerscale benchmark that includes additional modalities and employs a unified evaluation standard. For instance, GOAT-Bench (Lin et al., 2024b) is designed to assess the capability of various MMLLMs in discerning and responding to nuanced aspects of social abuse depicted in memes. MMCode (Li et al., 2024a) evaluates the algorithmic problem-solving skills of MM-LLMs in visually rich contexts. DecodingTrust (Wang et al., 2024a) measures the trustworthiness of MM-LLMs. MathVista (Lu et al., 2024) evaluates the mathematical reasoning ability of MM-LLMs within visual contexts, while GeoEval (Zhang et al., 2024b; Li et al.,
2024f; Song et al., 2024) assesses their proficiency in tackling geometry math problems. Moreover, MMMU (Yue et al., 2023) and CMMMU (Zhang et al., 2024a) have respectively introduced English and Chinese versions of a comprehensive multidiscipline MM understanding and reasoning benchmark for expert artificial general intelligence. Additionally, Fan et al. (2024) have challenged MMLLMs with multipanel VQA, and BenchLMM (Cai et al., 2023) benchmarks the cross-style visual capability of MM-LLMs. Furthermore, Liu et al. (2023h) have conducted an in-depth study on the optical character recognition capabilities of MMLLMs. These efforts highlight the need for more sophisticated and diverse benchmarks to truly gauge the advanced capabilities of MM-LLMs.
Mobile/Lightweight Deployment In order to deploy MM-LLMs on resource-constrained platforms and achieve optimal performance, such as lowpower mobile and IoT devices, lightweight implementations are of paramount importance. A notable advancement in this realm is MobileVLM (Chu et al., 2023a). This approach strategically downscales LLaMA, allowing for seamless off-the-shelf deployment. MobileVLM further introduces a lightweight downsample projector consisting of fewer than 20 million parameters, contributing to improved computational speed. Recently, there have been many similar studies on lightweight MMLLMs, achieving efficient computation and inference with comparable performance or minimal loss, including TinyGPT-V (Yuan et al., 2023b), Varytoy (Wei et al., 2024), Mobile-Agent (Wang et al., 2024c), MoE-LLaVA (Lin et al., 2024a), and MobileVLM V2 (Chu et al., 2024). Nevertheless, this avenue necessitates additional exploration for further advancements in development.
Embodied Intelligence The embodied intelligence aims to replicate human-like perception and interaction with the surroundings by effectively understanding the environment, recognizing pertinent objects, assessing their spatial relationships, and devising a comprehensive task plan (Firoozi et al., 2023). Embodied AI tasks, such as embodied planning, embodied visual question answering, and embodied control, equip robots to autonomously implement extended plans by leveraging real-time observations. Some typical works in this area are PaLM-E (Driess et al., 2023) and EmbodiedGPT (Mu et al., 2023). PaLM-E introduces a multi-embodiment agent through the training of
an MM-LLM. Beyond functioning solely as an embodied decision-maker, PaLM-E also demonstrates proficiency in handling general VL tasks. EmbodiedGPT introduces an economically efficient method characterized by a CoT approach, enhancing the capability of embodied agents to engage with the real world and establishing a closed loop that connects high-level planning with low-level control. While MM-LLM-based Embodied Intelligence has made advancements in integrating with robots, further exploration is needed to enhance the autonomy of robots.
Continual Learning Due to the large training costs associated with their massive scale, MMLLMs are not amenable to frequent re-training. However, updates are necessary to endow MMLLMs with new skills and keep them up-to-date with rapidly evolving human knowledge (Wu et al., 2024). Thus, Continual Learning (CL) is needed to make the model flexible enough to efficiently and continually leverage emerging data while avoiding the substantial cost of retraining MM-LLMs. CL for MM-LLMs can be classified into two stages: continual PT and continual IT. Recently, a continual MM IT benchmark has been proposed to continuously fine-tune MM-LLMs for new MM tasks while maintaining superior performance on tasks learned during the original MM IT stage (He et al., 2023). It introduces two primary challenges: (1) catastrophic forgetting, where models forget previous knowledge when learning new tasks (Robins, 1995; McCloskey and Cohen, 1989; Goodfellow et al., 2013; Zhang et al., 2023d,c,b; Zheng et al., 2023a), and (2) negative forward transfer, indicating that the performance of unseen tasks declines when learning new ones (Zheng et al., 2024; Dong et al., 2022, 2024a, 2023b,a).
Mitigating Hallucination Hallucinations entail generating textual descriptions of nonexistent objects without visual cues, which manifest in diverse categories (Liu et al., 2024a) such as misjudgments and inaccuracies in descriptions. The origins of these hallucinations are multifaceted (Liu et al., 2024a), including biases and annotation errors in training data. Additionally, Skip \n (Han et al., 2024) highlights semantic drift biases associated with paragraph separators, which can induce hallucinations when deliberately inserted. Current methods to mitigate these hallucinations involve leveraging self-feedback as visual cues (Lee et al., 2023). However, challenges persist, necessitating
nuanced discernment between accurate and hallucinatory outputs, as well as advancements in training methodologies to enhance output reliability.
Biases and Ethical Considerations Despite the strengths of MM-LLMs, ensuring their safe and efficient application remains crucial. Information generated by MM-LLMs can perpetuate stereotypes and cause harm to vulnerable populations. Since MM-LLMs learn from patterns in MM training data, they can reproduce biases present in these data, potentially leading to representational harm. To address this, we can develop new benchmarks specifically designed to evaluate biases in MMLLMs (Luo et al., 2024). Additionally, designing more effective and fine-grained alignment methods is essential. For instance, using RLHF can help calibrate MM-LLMs to produce answers that align with human values and desires (Li et al., 2024c).

heading: 7 Conclusion
text: In this paper, we have presented a comprehensive survey of MM-LLMs with a focus on recent advancements. Initially, we categorize the model architecture into five components, providing a detailed overview of general design formulations and training pipelines. Subsequently, we introduce various SOTA MM-LLMs, each distinguished by its specific formulations. Our survey also sheds light on their capabilities across diverse MM benchmarks and envisions future developments in this rapidly evolving field. We hope this survey can provide insights for researchers and contribute to the ongoing advancements in the MM-LLMs domain.
Social Impact
MM-LLMs have the potential to impact society. They can enhance accessibility for individuals with disabilities by improving voice recognition and visual aids, fostering equal access to information. In education, MM-LLMs can revolutionize learning with more interactive experiences, catering to diverse learning styles. In media, they can create more engaging content, enriching the consumer experience. However, the widespread adoption of MM-LLMs also poses data risks and user privacy risks. Privacy concerns arise from the training data, raising issues of security. On the other hand, exacerbating biases also arise, which can lead to biased outputs. Overall, while MM-LLMs offer promising opportunities, it is essential to address these challenges to ensure their responsibility.

heading: Acknowledgments
text: We express our gratitude to the anonymous reviewers for their valuable and insightful comments. This work was supported by JSPS KAKENHI Grant Number JP23K28144.
Limitations
In this paper, we embark on a comprehensive exploration of the current MM-LLMs landscape, presenting a synthesis from diverse perspectives enriched by our insights. Acknowledging the dynamic nature of this field, it is plausible that certain aspects may have eluded our scrutiny, and recent advances might not be entirely encapsulated. To tackle this inherent challenge, we’ve established a dedicated website for real-time tracking, using crowdsourcing to capture the latest advancements. Our goal is for this platform to evolve into a continuous source of contributions propelling ongoing development in the field. Given the constraints of page limits, we are unable to delve into all technical details, so we have provided concise overviews of the core contributions of mainstream MM-LLMs. Looking ahead, we commit to vigilant monitoring and continual enhancement of relevant details on our website (https://mm-llms.github.io), incorporating fresh insights as they emerge.

heading: A Related Surveys
text: Prior to the emergence of LLMs, several surveys on traditional MM PT have been conducted (Ruan and Jin, 2022; Du et al., 2022a; Long et al., 2022; Chen et al., 2023a). Most of these models entail a substantial computational cost during the PT phase, attributable to end-to-end training using large-scale models and datasets. As a consequence of not incorporating LLMs, these models suffer from deficiencies in instruction following, ICL, CoT, and interactive capabilities. Moreover, the training pipeline solely encompasses the PT phase without the inclusion of an IT stage.
In recent times, several surveys have emerged on MM-LLMs. Yin et al. (2023a) and Wu et al. (2023c) exclusively delve into early VL understanding models. Huang et al. (2023a) place a primary emphasis on visual IT, while Song et al. (2023) focus on modal alignment methods. Lastly, Cui et al. (2024) provide a comprehensive review of the applications of MM-LLMs within the realm of autonomous driving.
Compared with their works, the main distinctions are outlined as follows:
• We have comprehensively covered nearly all MM-LLMs over the past year, totaling around 120 or more, including not only understanding models but also generative models. Our coverage extends beyond VL modalities to encompass various modes such as audio and 3D point cloud;
• To offer readers a comprehensive understanding of MM-LLMs, we have introduced a general model architecture that incorporates anyto-any modality transformations, offering a detailed overview of the functional roles and implementation choices for each component;
• We have summarized the developmental trends of existing MM-LLMs and provided some training recipes that can enhance effectiveness;
• We have established an open-source website (https://mm-llms.github.io) for MM-LLMs researchers, supporting crowdsourced updates and aiming to facilitate collaboration in the MM-LLMs field. We anticipate that this survey will illuminate future research in the MMLLMs domain.

heading: B Modality Encoder
text: In the following, we provide a brief introduction to some mainstream modality encoders.
B.1 Visual Modality NFNet-F6 (Brock et al., 2021) is a normalizerfree ResNet (He et al., 2016), showcasing an adaptive gradient clipping that allows training on extensively augmented datasets while achieving SOTA levels of image recognition.
ViT (Dosovitskiy et al., 2020) applies the Transformer (Vaswani et al., 2017) to images by first dividing the image into patches. It then undergoes linear projection to flatten the patches, followed by encoding via Transformer blocks.
CLIP ViT (Radford et al., 2021) builds connections between text and images, comprising a ViT and a text encoder. With a vast amount of text-image pairs, it optimizes ViT by contrastive learning, treating paired text and images as positive samples and others as negative ones.
Eva-CLIP ViT (Fang et al., 2023) stabilizes the training and optimization process of the massive CLIP, offering new directions in expanding and accelerating the expensive training of MM base models.
B.2 Audio Modality C-Former (Chen et al., 2023b) employs the CIF (Dong and Xu, 2020; Zhang et al., 2022a; Han et al., 2022, 2023) for sequence transduction and a Transformer to extract audio features.
HuBERT (Hsu et al., 2021) is a self-supervised speech representation learning framework based on BERT (Kenton and Toutanova, 2019), achieved by the masked prediction of discrete hidden units. It has the capability to convert continuous speech signals into a sequence of discrete units.
BEATs (Chen et al., 2023g) is an iterative audio pre-training framework designed to learn Bidirectional Encoder representations from Audio Transformers.

heading: C Mainstream PEFT Methods
text: PEFT entails maintaining the pre-trained LLM in a frozen state while adjusting a small number of additional trainable parameters. In the following section, we revisit several representative PEFT methods, where x and h represent the input and output
of the original module, and h′ signifies the output of this module when attached with PEFT.
Prefix-tuning (Li and Liang, 2021; Lester et al., 2021) involves the addition of learnable tokens to the keys and values of the attention module. This process is formulated as follows:
h′ = Attn (xWq, [Pk,xWk], [Pv,xWv]) , (6)
with Pk,Pv ∈ Rl×d representing two sets of prefix tokens. [·, ·] denotes concatenation, and Attn is defined as:
Attn (Q,K,V) := softmax ( QKT√
d
) V.
Adapter (Houlsby et al., 2019; He et al., 2021; Rebuffi et al., 2017; Zhang et al., 2020) is typically a residual block consisting of a down-projection matrix A, a nonlinear activation function σ(·), and an up-projection matrix B. It can be inserted into any layer of the pre-trained LLM, formulated as follows:
h′ = h+ σ(xA)B. (7)
LoRA (Hu et al., 2021) is the most commonly used PEFT method. It assumes that the change in parameters occurs within a low-rank space. Given a pre-trained matrix W ∈ Rc×d, LoRA learns an incremental update ∆W and decomposes ∆W into a matrix multiplication between two low-rank matrices A ∈ Rc×r and B ∈ Rr×d, where r ≪ min(c, d). LoRA follows the forward process as outlined below:
h = Wx+∆Wx = Wx+ABx. (8)
QLoRA (Dettmers et al., 2023) is a quantized LoRA. The underlying principle of QLoRA includes the quantization of pre-trained weights to 4 bits, followed by the execution of PEFT using LoRA.
LayerNorm tuning (Zhao et al., 2024) presents an efficient strategy to transform LLMs into MMLLMs, which tunes LayerNorm in attention block, yielding strong MM performance compared with full parameter finetuning or LoRA.
In addition to the aforementioned PEFT methods, there are several others, including P-tuning (Liu et al., 2022), P-tuning v2 (Liu et al., 2021a), AdaptBias (Fu et al., 2022), Compacter (Karimi Mahabadi et al., 2021), AdapterFormer (Chen et al., 2022a), XTuner (Contributors, 2023), PLoRA (Dong et al., 2024c), MoLE (Chen et al., 2024), and Delta-LoRA (Zi et al., 2023).

heading: D Representative LLMs
text: The representative LLM Backbones in existing MM-LLMs research are as follows:
Flan-T5 (Chung et al., 2022) investigates IT for T5 (Raffel et al., 2020), an encoder-decoder architecture using unified text-to-text training for all natural language processing issues, exhibiting robust zero-shot and CoT capabilities.
ChatGLM is a Chinese-English bilingual dialogue model,2 optimized by an auto-regressive mask infilling objective. It is based on the GLM (Du et al., 2022b; Zeng et al., 2022a) architecture, optimized for Chinese question answering and dialogues.
InternLM (Team, 2023) is a multilingual trillion-parameter foundation model trained on over a trillion tokens of data. Based on this foundation, the model utilizes high-quality human-annotated dialogue data combined with RLHF to respond to complex instructions during human interactions, exhibiting responses that align with human ethics and values.
UL2 (Tay et al., 2022) is an encoder-decoder model trained to utilize a mixture of denoisers objectives, surpassing T5 on numerous benchmarks.
Qwen (Bai et al., 2023a) is trained on large-scale and diverse datasets, with a primary focus on Chinese and English. It employs SFT and RLHF techniques for alignment, resulting in dialogue models like Qwen-Chat.
Chinchilla (Hoffmann et al., 2022) is a causal decoder trained on extensive text data. It posits that model size should double for every doubling of training tokens.
OPT (Zhang et al., 2022c) is a GPT-3 (Brown et al., 2020) clone, striving to release an opensource model that replicates the performance of GPT-3.
PaLM (Chowdhery et al., 2023) is a causal decoder structure with parallel attention and feedforward layers, enabling training speeds up to 15 times faster. Notable changes contain RoPE embeddings, SwiGLU activation, multi-query attention, etc.
2https://github.com/THUDM/ChatGLM-6B
LLaMA (Touvron et al., 2023a) comprises decoder-only models with efficient causal attention.
LLaMA-2 (Touvron et al., 2023b) focuses on fine-tuning a superior and safer LLaMA-2-Chat model for conversation generation, incorporating 40% more training data with grouped-query attention and a larger context length.
Vicuna (Chiang et al., 2023) is a model built on top of LLaMA, utilizing user dialogue data obtained from ShareGPT.com and trained by SFT.

heading: E SOTA MM-LLMs
text: In the following, we will provide a brief introduction to the core contributions of some representative MM-LLMs.
Flamingo (Alayrac et al., 2022) represents a series of Visual Language (VL) Models designed for processing interleaved visual data and text, generating free-form text as the output.
BLIP-2 (Li et al., 2023e) introduces a more resource-efficient framework, comprising the lightweight Q-Former to bridge modality gaps and the utilization of frozen LLMs. Leveraging LLMs, BLIP-2 can be guided for zero-shot image-to-text generation using natural language prompts.
LLaVA (Liu et al., 2023e) pioneers the transfer of IT techniques to the MM domain. Addressing data scarcity, LLaVA introduces a novel open-source MM instruction-following dataset created using ChatGPT/GPT-4, alongside the MM instruction-following benchmark, LLaVA-Bench.
MiniGPT-4 (Zhu et al., 2023a) proposes a streamlined approach where training only one linear layer aligns the pre-trained vision encoder with the LLM. This efficient method enables the replication of the exhibited capabilities of GPT-4.
mPLUG-Owl (Ye et al., 2023b) presents a novel modularized training framework for MM-LLMs, incorporating the visual context. To assess different models’ performance in MM tasks, the framework includes an instructional evaluation dataset called OwlEval.
X-LLM (Chen et al., 2023b) is expanded to various modalities, including audio, and demonstrates strong scalability. Leveraging the language transferability of the Q-Former, X-LLM is successfully applied in the context of Sino-Tibetan Chinese.
VideoChat (Li et al., 2023f) pioneers an efficient chat-centric MM-LLM for video understanding dialogue, setting standards for future research in this domain and offering protocols for both academia and industry.
InstructBLIP (Dai et al., 2023) is trained based on the pre-trained BLIP-2 model, updating only the Q-Former during MM IT. By introducing instruction-aware visual feature extraction and corresponding instructions, the model enables the extraction of flexible and diverse features.
PandaGPT (Su et al., 2023) is a pioneering general-purpose model with the capability to comprehend and act upon instructions across 6 different modalities: text, image/video, audio, thermal, depth, and inertial measurement units.
PaLI-X (Chen et al., 2023h) is trained using mixed VL objectives and unimodal objectives, including prefix completion and masked-token completion. This approach proves effective for both downstream task results and achieving the Pareto frontier in the fine-tuning setting.
Video-LLaMA (Zhang et al., 2023e) introduces a multi-branch cross-modal PT framework, enabling LLMs to simultaneously process the vision and audio content of a given video while engaging in conversations with humans. This framework aligns vision with language as well as audio with language.
Video-ChatGPT (Maaz et al., 2023) is a model specifically designed for video conversations, capable of generating discussions about videos by integrating spatiotemporal vision representations.
Shikra (Chen et al., 2023e) introduces a simple and unified pre-trained MM-LLM tailored for Referential Dialogue, a task involving discussions about regions and objects in images. This model demonstrates commendable generalization ability, effectively handling unseen settings.
DLP (Jian et al., 2023) proposes the P-Former to predict the ideal prompt, trained on a dataset of single-modal sentences. This showcases the feasibility of single-modal training to enhance MM learning.
BuboGPT (Zhao et al., 2023d) is a model constructed by learning a shared semantic space for a comprehensive understanding of MM content. It
explores fine-grained relationships among different modalities, such as image, text, and audio.
ChatSpot (Zhao et al., 2023b) introduces a simple yet potent method for finely adjusting precise referring instructions for MM-LLM, facilitating fine-grained interactions. The incorporation of precise referring instructions, consisting of image- and region-level instructions, enhances the integration of multi-grained VL task descriptions.
Qwen-VL (Bai et al., 2023b) is a multi-lingual MM-LLM that supports both English and Chinese. Qwen-VL also allows the input of multiple images during the training phase, improving its ability to understand the vision context.
NExT-GPT (Wu et al., 2023d) is an end-to-end, general-purpose, any-to-any MM-LLM that supports the free input and output of image, video, audio, and text. It employs a lightweight alignment strategy, utilizing LLM-centric alignment in the encoding phase and instruction-following alignment in the decoding phase.
MiniGPT-5 (Zheng et al., 2023b) is an MMLLM integrated with inversion to generative vokens and integration with Stable Diffusion. It excels in performing interleaved VL outputs for MM generation. The inclusion of classifier-free guidance during the training phase enhances generation quality.
LLaVA-1.5 (Liu et al., 2023d) reports simple modifications to the LLaVA framework, including applying an MLP projection and introducing VQA data tailored for academic tasks, along with simple response formatting prompts. These adjustments result in enhanced capabilities for MM understanding.
MiniGPT-v2 (Chen et al., 2023d) is an MMLLM designed as a unified interface for diverse VL multi-task learning. To create a single model proficient in handling multiple VL tasks, identifiers are incorporated for each task during both training and inference. This facilitates clear task distinction, ultimately enhancing learning efficiency.
CogVLM (Wang et al., 2023b) is an open-source MM-LLM that bridges the gap between modalities via a trainable visual expert module within the attention and feedforward layers. This allows for a deep fusion of MM features without compromising performance on NLP downstream tasks.
DRESS (Chen et al., 2023i) introduces a method using natural language feedback to enhance alignment with human preferences. DRESS extends the conditional reinforcement learning algorithm to integrate non-differentiable natural language feedback, training the model to generate appropriate responses based on feedback.
X-InstructBLIP (Panagopoulou et al., 2023) introduces a cross-modal framework with instructionaware representations, scalable enough to empower LLMs to handle diverse tasks across multiple modalities, including image/video, audio, and 3D. Notably, it achieves this without the need for modality-specific PT.
CoDi-2 (Tang et al., 2023b) is an MM generation model excelling in modality-interleaved instruction following, in-context generation, and usermodel interaction by multi-turn conversations. It enhances CoDi (Tang et al., 2023c) to process intricate modality-interleaved inputs and instructions, generating latent features autoregressively.
VILA (Lin et al., 2023) outperforms vision tasks and shows remarkable reasoning ability while maintaining text-only capabilities. It achieves this by harnessing the full capabilities of LLM learning, using the interleaved attributes of image-text pairs, and implementing meticulous text data re-blending.
F VL Benchmarks
The 18 VL benchmarks presented in Table 2 include OKVQA (Schwenk et al., 2022), IconVQA (Lu et al., 2021), VQAv2 (Goyal et al., 2017), GQA (Hudson and Manning, 2019), VizWiz (Gurari et al., 2018), SQAI: ScienceQA-IMG (Lu et al., 2022), VQAT: TextVQA (Singh et al., 2019), POPE (Li et al., 2023k), MMEP: MME Perception (Fu et al., 2023), MMEC: MME Cognition (Fu et al., 2023), MMB: MMBenchmark (Liu et al., 2023g), MMBCN: MMBench-Chinese (Liu et al., 2023g), SEEDI: SEED-Bench (Image) (Li et al., 2023c), LLaVAW: LLaVA-Bench (In-theWild) (Liu et al., 2023a), MM-Vet (Yu et al., 2023c), QBench (Wu et al., 2023b), HM: HatefulMemes (Kiela et al., 2020), and VSR (Liu et al., 2023a).

heading: G Training Dataset
text: The statistics for MM PT and MM IT datasets are presented in Table 3 and Table 4, respectively.

heading: Future Work
text: , 2023f) pioneers an efficient chat-centric MM-LLM for video understanding dialogue, setting standards for future research in this domain and offering protocols for both academia and industry. InstructBLIP (Dai et al. , 2023) is trained based on the pre-trained BLIP-2 model, updating only the Q-Former during MM IT. By introducing instruction-aware visual feature extraction and corresponding instructions, the model enables the extraction of flexible and diverse features. PandaGPT (Su et al. , 2023) is a pioneering general-purpose model with the capability to comprehend and act upon instructions across 6 different modalities: text, image/video, audio, thermal, depth, and inertial measurement units. PaLI-X (Chen et al. , 2023h) is trained using mixed VL objectives and unimodal objectives, including prefix completion and masked-token completion. This approach proves effective for both downstream task results and achieving the Pareto frontier in the fine-tuning setting. Video-LLaMA (Zhang et al. , 2023e) introduces a multi-branch cross-modal PT framework, enabling LLMs to simultaneously process the vision and audio content of a given video while engaging in conversations with humans. This framework aligns vision with language as well as audio with language. Video-ChatGPT (Maaz et al