# From Large Language Models to Large Multimodal Models: A Literature Review
## 2 Large Language Models
### Representative Pretrained LLMs
LLaMA's characteristic efficiency in its utilization of parameters has rendered it a popular choice as a large language model (LLM) backbone for numerous downstream tasks. **PanGu-\(\Sigma\) (2023):** Pangu-\(\Sigma\)[11] represents the inaugural trillion-parameter sparse language model, inheriting the decoder-only architecture and innate parameters of its predecessor, PanGu-\(\alpha\)[10], with a peak parameter count reaching 1.085 trillion. It introduces an input mechanism that employs distinct embeddings for distinct domains, which is coupled with a two-level routing design in the Random Routed Experts (RRE) framework. The pretraining corpus of Pangu-\(\Sigma\), totalling 329 billion tokens, primarily encompasses diverse data formats of bilingual Chines-English, content from [24; 47; 48] and code from [49; 50]. During pretraining, Pangu-\(\Sigma\) employs an Expert Computing and Storage Separation (ECSS) strategy, harnessing heterogeneous computing to enhance training throughput by a factor of 6.3 compared to PanGu-\(\alpha\). In experimental evaluations, Pangu-\(\Sigma\) demonstrates superiority over previous state-of-the-art models across sixteen downstream NLP tasks in the Chinese domain, encompassing dialogue, question answering, and machine translation, among others. However, due to variations in the pretraining datasets employed, the paper does not present comparative results against GPT-3 [4] or other contemporary SOTA large-scale models. **PaLM-2 (2023):** PaLM-2 [7] is based on the transformer [21] architecture, although the specific architectural details are not mentioned in the technical report. While the report does not reveal the exact parameter count for the PaLM-2 family, it states that the models in this series have a smaller number of parameters compared to PaLM 540B [6]. Despite this reduced parameter count, PaLM-2 demonstrates improved reasoning abilities across various tasks. The report highlights the significance of scaling laws from recent work [41]. In addition, through empirical validations, it underscores the criticality of considering data scale and model size as co-determinants of equal weight in driving performance improvements [41]. A distinguishing innovation in PaLM-2 lies in breaking away from the conventional approach used for pretraining large language models, which typically uses a single objective such as causal or masked language modeling. Building upon the achievements of UL2 [51], PaLM-2 adopts a hybrid of multiple pretraining objectives. Additionally, the pretraining dataset employed for PaLM-2 has been expanded to cover several hundred languages, thereby broadening its multilingual capacity. **LLaMA 2 (2023):** The architecture of LLaMA 2 [9] adheres to that of its predecessor LLaMA [8], adopting a decoder-only paradigm with a maximum parameter count scaling up to 70 billion. Notably, LLaMA 2 introduces certain enhancements over LLaMA, including doubling the context window size from its original 2048 tokens to an extended 4096 tokens and incorporating the utilization of grouped-query attention(GQA) [52] (shown in Figure 5), which economizes cache space required for Key-Value (KV) caching while effectively sustaining model accurac. In terms of the pretraining corpus, the LLaMA 2 ensemble of models is trained on a newly constructed publicly accessible hybrid online dataset encompassing 2 trillion tokens, thereby augmenting the training volume by 42% compared to the original LLaMA. Despite demonstrating superior performance across multiple benchmark evaluations relative to all open-source models, LLaMA 2 exhibits a discernible performance gap when juxtaposed against closed-source contemporaries such as GPT-4 [5] and PaLM-2-L [7].