# From Large Language Models to Large Multimodal Models: A Literature Review
## 2 Large Language Models
### LLM Prompt-Engineering  
Next, we summarize the techniques in prompt engineering, which treat the prompt as a learnable parameter without updating the parameters of pretrained models. By optimizing only a minuscule number of parameters, prompt engineering can improve the performance of pretrained models in various downstream tasks while approaching the efficacy of full fine-tuning. **Prefix Tuning [29]:** Incorporating concepts from prompting and in-context learning [4; 30], prefix tuning optimizes a set of continuous task-specific vectors, denoted as \"prefixes\". For decoder-only architectures, distinct prefix vectors are prepended to the Key (\(K\)) and Value (\(V\)) matrices at each layer of the decoder's attention mechanism. Conversely, for encoder-decoder structures, separate prefix vectors are appended to the Query (\(Q\)) and Key (\(K\)) matrices preceding each layer of the attention mechanism in both the encoder and decoder parts. Unlike full fine-tuning, prefix tuning freezes all parameters of the pretrained language model and focuses solely on optimizing the small subset of prefixed parameters that have been introduced. **P-Tuning [31]:** P-tuning maps discrete prompts into trainable continuous prompt embeddings, it utilizes LSTM and MLP to construct a prompt encoder. Distinct from prefix tuning [29], P-tuning inserts prompt tokens at arbitrary positions within the input layer; these are consecutively transformed into hidden states via the prompt encoder and jointly trained alongside the input embeddings. Nonetheless, in order to realize enhanced performance outcomes it is imperative to undertake the optimization of prompt embedding in concert with the overall model tuning process. **Prompt Tuning [32]:** Prompt tuning can be regarded as a simplified version of P-tuning [31] and Prefix Tuning [29], as illustrated in Figure 1. Prompt tuning concatenates a sequence of prompts with the input sequence to form the model's input. The embedded prompts (\(P_{e}\)) and input embeddings (\(X_{e}\)) collectively constitute a parameter matrix \([P_{e}:X_{e}]\) which is processed by the model, wherein only the parameters of the prompts \(P_{e}\) are updated; the original weights of the pretrained model remain unaltered. This method exhibits heightened parameter efficiency, and is becoming increasingly competitive as model parameter counts grow. It is able to attain performance comparable to full fine-tuning even when model parameters exceed billions. Through prompt tuning, a single pretrained model can be efficiently repurposed for multiple downstream tasks merely by training different prompt parameters for each respective task. **In-Context Learning:** The few-shot performance demonstrated by GPT-3 [4] highlights the potential of in-context learning (ICL), enabling language models to master downstream tasks without supplementary model tuning by relying solely on a handful of exemplary demonstrations. The survey by [33] decomposes in-context learning into two distinct stages, namely, training and inference. During the training stage, the model acquires the capability for in-context learning from the pretraining objectives. At the inference  
Figure 1: Schematic representation of the technical architectures for prompt tuning, P-tuning, and prefix tuning. stage, the manifestation of the model's in-context learning prowess is showcased through the strategic design of examples and the selection of appropriate evaluation mechanisms. **Chain-of-Thought:** The essence of the chain-of-thought (CoT) concept [34] lies in mimicking the human cognitive process for tackling complex issues by presenting the model with a modest collection of exemplars. Solutions are broken down into a progression of intermediate reasoning steps articulated in natural language while clearly delineating the logical trajectory from inquiry to resolution. Within the GSM8K [35] mathematics problem benchmark, the PaLM 540B model utilizing chain-of-thought prompting achieved a substantial leap beyond fine-tuned GPT-3 and other previous best performers.