# A Review of Current Trends, Techniques, and Challenges in Large Language Models (LLMs)
## 4 Pretraining--Strategies and Objectives
### Learning Strategies  
#### 4.2.1 Multitask Pretraining  
In multitask learning (MTL), parameters are shared between multiple tasks during pretraining. This leads to better generalization and performance improvement of related tasks. MTL helps to improve performance on new domains by leveraging the knowledge and representation learned from related tasks during pretraining. MTL uses a single model to perform many downstream tasks simultaneously. However, unlike the adapter layers, MTL requires simultaneous access to the tasks during pretraining. The networks' lower MTL layers (and their weights) are shared among the tasks, using specialized higher layers based on the downstream tasks. During'multitask learning', datasets from different tasks are mixed and used. As experimented in T5 [25], multitask learning involves pretraining the model on multiple tasks simultaneously. Although multiple tasks were used during pretraining, the T5 model was finetuned separately on supervised downstream tasks. One crucial factor to consider in multitask learning is how much data the model should be trained on from each task. There needs to be a proper balance where the model sees enough data to perform well regarding the task while not exposing it to more data such that it starts memorizing (overfitting) the dataset. Additionally, the proportion of data also depends upon factors such as dataset sizes, difficulty of learning the task, regularization, and task interference since performing better on one task might degrade the performance on another task. In T5 [25], as the same training objective was used for every task, only a single set of hyperparameters was required for effective finetuning on all downstream tasks. In MTL, as the same model performs many different tasks, the language model becomes conditioned on the input and the task to be performed. Such task conditioning can be implemented at the architecture level. However, a recent technique from GPT-2 [28] suggests a simplified mechanism where tasks, inputs, and outputs can all be specified as a sequence of symbols. That is, to be architecture-independent, the input can be transformed to incorporate task-aware information as a context (added as task-prefix) to the input sequence. Also, as stated in T5, every text processing problem can be mapped to \"text-to-text\" format, where the input and output are both text. For instance, to translate an English sentence \"I am good\" to French, the prefix \"translate English to French: I am good. Target:\" will be used, where the model will then be asked to generate the remainder \"je vais bien\" of the sequence in an autoregressive manner. So, similar to a translation of a sequence of (translate to French, English sentence, French sentence), a reading comprehension example can likewise be written as a tuple consisting of (answer the question, document, question, answer). Using this framework, the same encoding and decoding procedure is used across \begin{table}
\begin{tabular}{l l l l l} \hline \hline
**Model** & **Architecture** & **Objectives** & **Pretraining Dataset** & **Tokens, Corpus Size** \\ \hline \multirow{2}{*}{WeLM [60]} & - & - & Common Crawul, news, books, \\  & - & - & forums, academic writings. \\  & & & ROOTS corpus (46 natural \\  & & & languages and 13 programming \\  & & & languages) \\ GLM [61] & \
\begin{tabular}{l} bidirectional encoder \\ and unidirectional \\ decoder \\ \end{tabular} & GLM & & & \\ GPT-J & Mesh Transformer JAX & LM & PLE & 402 B \\ YaLM & & &
\begin{tabular}{l} online texts, The File, books, other \\ resources (in English, Russian) \\ \end{tabular} & 1.7 TB \\ Alpaca & & & & \\ Falcon & encoder-only & LM & RefinedWeb, Reddit & IT \\ (Xmer) XXXL [62] & encoder-decoder & MLM & C4 & 1T tokens \\  & & & BookCorpus, English Wikipedia, \\
[63] & decoder-only (MoE) & LM & CC-News, OpenWebText, & 300 B \\  & & & CC-Stories, CC100 & \\ XLM-R [64] & encoder-only & Multilingual MLM & CommonCrawl (CC-100) & 2.5 TB \\ \hline \hline \end{tabular}
\end{table}