# From Large Language Models to Large Multimodal Models: A Literature Review
## 3 Large Multimodal Models
### Taxonomy of Vision-Language LMMs
#### 3.5.3 Image+Text to Text+Image  
Models in the \"Image+Text to Text+Image\" category should not be considered as mere image generation models; instead, they are bifurcated into two principal classes, namely, image editing models and generative models. Figure 13 presents an overview workflow of \"Image+Text to Text+Image\" models. As depicted in Table 4, among the 23 selected models, models such as CogCoM [127], DetGPT [128], Shikra [86], and SPHINX-X [129] (which does not contain a vision generator component) consequently do not possess the inherent ability to synthesize images. However, they are endowed with the ability to perform a variety of operations on input images, encompassing extraction, annotation, and segmentation, leading to their classification as image processing models. For instance, CogCoM is capable of executing cropping and zoom-in operations to acquire detailed local visual content, and can identify textual information within images through OCR while performing reasoning based on visual input. Although image editing does not equate to image generation, it does manifest the capacity of LMMs to comprehend and manipulate visual input data. Figure 12: Example showcasing how a model understands and replies to video, taken from the VideoChat paper [87]. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Model** & **Vision Encoder** & **Input Modal Aligner** & **Upstream LLM Backbone** \\ \hline Dolphins [117] & CLIP ViT & Cross Attention Layer & LLaMA/MPT \\ \hline InstructBLIP & ViT & Q-Former + Linear Mapper & Vicuna/Flan-T5 \\ \hline InternVL [118] & InternViT-6B & Cross-attention + QLAMA + MLP & Vicuna-13B \\ \hline Lynx [119] & Eva-CLIP ViT & Cross Attention Layer & Vicuna-7B \\ \hline mPLUG-video [120] & TimeSformer [121] & Cross Attention Layer + Linear Mapper & Chinese GPT-3 1.7B [122]/2.3B [123] \\ \hline VAST [124] & ViT & Cross Attention Layer & Vicuna-13B \\ \hline VideoChat [87] & ViT & Q-Former + Linear Mapper & Vicuna \\ \hline Video-ChatGPT [125] & CLIP ViT & Linear Mapper & Vicuna-v1.1 \\ \hline Video-LLaMA [126] & Eva-CLIP ViT & Q-Former + Linear Mapper & Vicuna/LLaMA \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Display of architectural components across different video understanding models. The remaining models incorporate a vision generator within their architectural composition, signifying that they are equipped for both image editing and for generating new image content or style transfer conditioned upon either textual or visual inputs. A case in point is LLaVA-Plus, which introduces 'Thought' and 'Action' fields to its input data and exists in two configurations: LLaVA-Plus (All Tools) harnesses all visual tools except semantic segmentation to enrich image features during both training and evaluation phases, whereas LLaVA-Plus (Fly) dynamically invokes and executes relevant tools for object detection, semantic segmentation, OCR, and conditional image generation according to the provided instructions. Figure 14 shows the capacities of LLaVA-Plus. Furthermore, a discernible trend among certain \"Image+Text to Image+Text\" models is their development incorporating a vision generator into the foundational architecture of prior image understanding models. Illustrative examples include Kosmos-G [63], which builds upon Kosmos-1 [130], along with LLaVA-Plus and LISA [131], both of which are constructed atop the LLaVA [101] framework.