# From Large Language Models to Large Multimodal Models: A Literature Review
## 3 Large Multimodal Models
### Taxonomy of Vision-Language LMMs
#### 3.5.4 Video+Text to Text+Video  
\"Video+Text to Text+Video\" models can be conceptualized as video generation models, as listed in Table 5, where six representative models are presented. Among these, models such as CoDi-2 [143], ModaVerse [144], and NExT-GPT [145] exemplify 'any-to-any' large multimodal models capable of understanding or generating text, audio, image, and video content. Figure 15 provides an overview framework for these models. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Model** & **Vision Encoder** & **Input Modal** & **Upstream LLM** & **Output Modal** & **Vision Generator** \\ \hline GILL [79] & CLIP ViT & Linear Mapper & OPT-6.7B & GILLMapper & Stable Diffusion-1.5 \\ \hline GLaMM [135] & CLIP ViT & MLP & Vicuna-7B/13B & MLP & Stable Diffusion \\ \hline GPT4 ROI [136] & CLIP ViT & Linear Mapper & Vicuna-7B & - & - \\ \hline Kosmos-G [63] & CLIP ViT & Linear Mapper & MAGNETO & AlignerNet & Stable Diffusion-1.5 \\ \hline LaVIT [137] & ViT & \begin{tabular}{c} Cross Attention \\ Layer \\ \end{tabular} & LLaMA-7B & \begin{tabular}{c} Designed- \\ quantizer \\ \end{tabular} & Stable Diffusion \\ \hline LISA [131] & ViT & - & LLaVA-7B/13B & - & - \\ \hline LLaVA-Plus [138] & - & - & LLaVA-7B/13B & - & Stable Diffusion \\ \hline MiniGPT-5 [139] & Eva-CLIP ViT & \begin{tabular}{c} Q-Former + Linear \\ Mapper \\ \end{tabular} & Vicuna-7B & \begin{tabular}{c} Transformer + \\ MLP \\ \end{tabular} & Stable Diffusion-2 \\ \hline MM-Interleaved [80] & CLIP ViT & \begin{tabular}{c} Cross Attention \\ Layer \\ \end{tabular} & Vicuna-13B & Transformer & Stable Diffusion-2.1 \\ \hline PixelLM [140] & CLIP-ViT & MLP & LLaMA2-7B/13B & - & - \\ \hline SEED [141] & CLIP ViT & \begin{tabular}{c} Q-Former+Linear \\ Mapper \\ \end{tabular} & OPT-2.7B & \begin{tabular}{c} Q-Former+Linear \\ Mapper \\ \end{tabular} & Stable Diffusion \\ \hline Shikra [86] & CLIP ViT & Linear Mapper & Vicuna-7/13B & - & - \\ \hline SPHINX-X [129] & \begin{tabular}{c} DINOV2/CLIP- \\ ConvNeXt \\ \end{tabular} & Linear Mapper & \begin{tabular}{c} TinyLlama- \\ 1.1B/LLaMA2- \\ 13B/Mixxtral- \\ 8x7B/InterLM2- \\ 7B \\ \end{tabular} & - & - \\ \hline Visual ChatGPT [90] & Visual Foundation & - & ChatGPT & - & \
\begin{tabular}{c} Visual Foundation \\ Models \\ \end{tabular} \\ \hline VL-GPT [142] & CLIP ViT &
\begin{tabular}{c} Causal \\ Transformer \\ \end{tabular} & LLaMA-7B & Transformer & Stable Diffusion \\ \hline \hline \end{tabular}
\end{table} 
Table 4: _Cont._Among the employed components, ImageBind [75] is the most commonly utilized vision encoder, whereas LLaMA 2 [9] represents the prevailing choice of upstream LLM backbone. For the video generator, the mainstream option is the ZeroScope series of open-source models from HuggingFace, which is a video generation model developed by Alibaba DAMO Academy. This model automatically generates videos consistent with user-provided text descriptions, incorporating visual elements (scenes), audio elements (music and sound effects), and subtitles. Furthermore, HuggingGPT [149] adopts the design concept of Visual-ChatGPT [125], employing ChatGPT [54] as its upstream LLM backbone to comprehend, process, and refine user prompts. In addition, it harnesses the capabilities of various existing external models within the community to facilitate multimodal input-output interactions. At present, video generation models can only produce videos of several seconds in duration, regardless of whether they are based on LMM, Latent Diffusion Models (LDM) [147, 150, 151, 152], or GANs [153, 154], and often fall short in terms of both logical coherence and authenticity. Against this backdrop, OpenAI introduced Sora in February 2024, a model that, by understanding textual descriptions, can generate high-definition video content up to one minute long while adhering to the laws of physics in the real world and maintaining logical continuity. Currently, neither the technical details nor the inner workings of the Sora model have been publicly disclosed.