# From Large Language Models to Large Multimodal Models: A Literature Review
## 3 Large Multimodal Models
### Taxonomy of Vision-Language LMMs
#### 3.5.1 Image+Text to Text  
The \"Image+Text to Text\" model, also referred to as the image understanding model, constitutes both the starting point and a focal point of extensive research interest within the LMM community. Notably, it has the highest prevalence in terms of model quantity. In this section, we have selected 28 representative image understanding models as a basis for our study, as outlined in Table 2. A summary of these models' frameworks is depicted in Figure 9. Figure 10 shows an example of dialogue in an image understanding model. Upon statistical analysis of the components employed in image understanding models from Table 2, it emerges that CLIP-ViT is currently the most widely adopted visual encoder, followed closely by Eva-CLIP-ViT. These encoders demonstrably outperform alternative types of visual encoders. This superiority may stem from the contrastive learning paradigm employed by CLIP, coupled with its 400 million high-quality image-text pairs, which together endow it with a robust capability for visual feature extraction. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Model** & **Vision Encoder** & **Input Modal Aligner** & **Upstream LLM Backbone** \\ \hline BLIP-2 [60] & CLIP ViT & Q-Former + Linear Mapper & Flan-T5/OPT \\ \hline BLIVA [94] & CLIP ViT & Q-Former + Linear Mapper & Vicuna-7B/Flan-T5 XXL \\ \hline ChatSpot [95] & CLIP ViT & Linear Mapper & Vicuna-7B/LLaMA \\ \hline CogVLM [96] & Eva-2-CLIP ViT & MLP & Vicuna-v1.5-7B \\ \hline DRESS [97] & Eva-CLIP ViT & Linear Mapper & Vicuna-v1.5-13B \\ \hline DLP [77] & CLIP ViT & \begin{tabular}{c} Q-Former + P-Former + \\ Linear Mapper \\ \end{tabular} & OPT/Flan-T5 \\ \hline IDEFICS [98] & OpenCLIP & Cross Attention Layer & LLaMA \\ \hline InternLM-XComposer [99] & Eva-CLIP ViT & Cross Attention Layer & Intern-LM \\ \hline InternLM-XComposer2 [100] & CLIP ViT & Cross Attention Layer & Intern-LM2 \\ \hline Lyrics [78] & CLIP ViT & MQ-Former + Linear Mapper & Vicuna-13B \\ \hline LLaVA [101] & CLIP ViT & Linear Mapper & Vicuna-7B/13B \\ \hline LLaVA-1.5 [102] & CLIP ViT & MLP & Vicuna-v1.5-7B/13B \\ \hline LLaVAR [88] & CLIP ViT & Linear Mapper & Vicuna-13B \\ \hline MiniGPT-v2 [103] & Eva-CLIP ViT & Linear Mapper & LLaMA-2-Chat-7B \\ \hline MiniGPT-4 [61] & Eva-CLIP ViT & Q-Former + Linear Mapper & Vicuna-13B \\ \hline mPLUG-Owl [104] & CLIP ViT & Cross Attention Layer & LLaMA-7B \\ \hline mPLUG-Owl2 [105] & ViT & Modality-Adaptive Module & LLaMA-2-7B \\ \hline mPLUG-DocOwl [106] & CLIP ViT & Cross Attention Layer & LLaMA-7B \\ \hline MobileVLM [107] & CLIP ViT & \
\begin{tabular}{c} Lightweight Downsample \\ Projector (LDP) \\ \end{tabular} & MobileLAMA \\ \hline MobileVLM V2 [108] & CLIP ViT &
\begin{tabular}{c} Lightweight Downsample \\ Projector v2 (LDPv2) \\ \end{tabular} & MobileLAMA \\ \hline Otter [109] & CLIP ViT & Cross Attention Layer & LLaMA-7B \\ \hline Osprey [110] & ConvNeXt-Large & MLP & Vicuna \\ \hline PandaGPT [111] & ImageBind & Linear Mapper & Vicuna-13B \\ \hline PaLI-X [112] & ViT & Linear Mapper & UL2-32B \\ \hline Qwen-VL [113] & ViT & Cross Attention Layer & Qwen-7B \\ \hline RLHF-V [114] & BEiT-3 & Linear Mapper & Vicuna-v1-13B \\ \hline Silkie [115] & ViT & Cross Attention Layer & Qwen-7B \\ \hline VILA [116] & ViT & Linear Mapper & LLaMA-2-7B/13B \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Display of architecture components across different image understanding models, ordered by initial of model name. Figure 9: Common architecture of “Image+Text to Text” models. Additionally, the linear mapper and cross-attention layer are frequently employed as modal aligners, possibly due to their structural simplicity and performance efficiency when compared to aligners such as P-former. Another approach involves employing custom modal aligner designs. For instance, mPLUGOw12 [105] introduced a modality-adaptive module designed to accurately compute and compare the similarity between different modalities within a shared semantic space.