# From Large Language Models to Large Multimodal Models: A Literature Review
## 3 Large Multimodal Models
### LMM Prompt Engineering  
In LLMs, prompt engineering [29; 31; 32; 33; 34; 89] refrains from updating the model parameters and instead seeks to augment its functionality by employing prompt tokens optimizing the performance of the LLM. Within the realm of LMMs, prompt engineering includes the inheritance of in-context learning [33] and chain-of-thought (CoT) [34] derived from LLMs, evolving into the multimodal in-context learning and multimodal chain-of-thought paradigms.  
**Multimodal In-Context Learning** LLMs leverage in-context learning (ICL) techniques, enabling them to achieve few-shot performance comparable to fine-tuning during the inference stage, relying on only a small number of demonstrations for downstream tasks [4]. In LMMs, as shown in Figure 8, in-context learning can enhance performance when extended to multimodal in-context learning through the provision of demonstrations across modalities. Typically, there are two distinct application scenarios of multimodal ICL within LMM; the first involves using a small set of demonstrations to guide an LMM in modal reasoning tasks, while the second entails teaching the LMM how to utilize external foundational tools to solve tasks through a series of exemplary steps, as presented in [18].  
Figure 8: Comparison of in-context learning in LLMs and LMMs. Here, <BOS> and <EOS> signs are tokens indicating the beginning and end of the input demonstrations.  
**Multimodal Chain-of-Thought** The concept of chain-of-thought (CoT) originally emerged as a technique in LLM to tackle complex reasoning tasks [34]. It involves guiding the LLM to break a complex problem into a series of subproblems and solve them iteratively, thereby significantly enhancing the performance of LLM [4; 5]. CoR led to an expansion of the prompt paradigm from <Input, Output> to <Input, CoT Rationale, Output> [34].  
Several works have advanced single-modal CoT to multimodal CoT (M-CoT), which can be categorized under different prompt paradigms such as zero-shot M-CoT [90], few-shot M-CoT [91], and fine-tuning M-CoT [92; 93]. In zero-shot M-CoT, no examples are used to guide the LMM through the CoT process; instead, the model's reasoning ability is triggered during the inference phase with a simple textual instruction such as _\"Do it step by step.\"_. Few-shot M-CoT resembles in-context learning, where a small number of exemplars are provided that detail the intermediate steps of reasoning. On the other hand, fine-tuning M-CoT necessitates the use of a specific dataset for fine-tuning the LMM's reasoning capabilities. Typically, the former two approaches are more applicable to larger LLMs, while the latter is often employed for smaller ones.  
Regarding modal alignment, M-CoT can further be divided into translation mapping and learnable mapping. In translation mapping, non-text modal inputs are straightforwardly converted into text descriptions, effectively transmitting non-text modalities to the text modality via 'translation' before engaging in CoT reasoning (although this process inherently entails considerable loss of modality-specific information). By contrast, learnable mapping involves constructing a trainable model that integrates features from other modalities into the textual feature space, forming a joint embedding that serves as input to the LLM for CoT reasoning.  
For instance, Multimodal-CoT [92] has been utilized with fine-tuning and learnable mapping to elicit CoT capabilities in T5-770M. By devising a two-stage learnable reasoning framework, T5-770M surpassed GPT-3.5 on the ScienceQA benchmark [91] while concurrently mitigating the occurrence of hallucination-induced errors inherent in LLMs.