# A Review of Current Trends, Techniques, and Challenges in Large Language Models (LLMs)
## 7 Scalability
### Training Tokens and Data Size  
Although [PERSON] et al. [97] showed a power law relationship between the number of model parameters and its performance, they did not consider pretraining tokens or  
\begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Model** & **PT, FT Batch-Size** & **Context Size** & **PT, FT Epochs** & **Activation, Optimizer** & **Finetuning Methods** \\ \hline YalLM & & - & - & - & FT, IT (instruction Tuning) \\ Alpaca & & 2048 & - & - & - \\ Falcon (Xmer) XXXL [62] & & & 2\({}^{\text{\tiny{29}}}\) & & FT \\
[63] & & 2048 & & FT, zero-shot, few-shot \\ \hline \hline \end{tabular}
\end{table}
Table 3: _Cont._corpus data size. [PERSON] et al. [40] also reached the same conclusion but recommended that large models should be trained for many more training tokens. Specifically, given a 10\(\times\) increase in computational budget, [PERSON] et al. [97] suggest that the model size should increase 5.5\(\times\) while the number of training tokens should only increase 1.8\(\times\). Instead, [PERSON] [40] finds that, as the computation budget increases, model size and the number of training tokens (training data) should be scaled in approximately equal proportions. Although large models achieved better performances, [PERSON] on the other hand demonstrated how, for a given computation budget, the best performance is achieved by smaller models trained on larger or more data as compared to large models. For instance, although LLaMA had 13 B parameters, it outperformed GPT-3 with 175 B on most benchmarks despite being ten times smaller. [PERSON] helps to answer how, given a fixed computational budget, one should balance model size and the number of training tokens.  
With one-fourth fewer parameters and four times more data than Gopher, Chinchilla could significantly outperform Gopher on a large range of downstream evaluation tasks. Not only does Chinchilla outperform its much larger counterpart, Gopher, but, because of its smaller model size, it uses less computing for finetuning and inference, which reduces finetuning and inference costs considerably and greatly facilitates downstream usage on smaller hardware. Although [40] it determines how to scale the model size and dataset for a given computation budget, it disregards the inference budget, which is crucial since the preferred model is the one that is fastest at inference and not at training. As per [57], a smaller model trained longer is cheaper at inference. For example, Falcon-40B requires 70 GB of GPU memory to make inferences, whereas Falcon-7B needs only 15 GB, making inference and finetuning accessible even on consumer hardware.  
Additionally, although [40] recommended training a 10 B model on 200 B tokens, [57] demonstrated how the performance of a model with 7 B parameters continued to improve even after it was trained on 1 T tokens. Furthermore, unlike Chinchilla, PaLM, or GPT-3, LLaMA demonstrated how it can train models and achieve SOTA performance using publicly available datasets without relying on proprietary and inaccessible datasets. WeLM [60], a Chinese LM, demonstrated how, by carefully cleaning, balancing, and scaling up the training data size, WeLM outperformed models with similar or larger sizes. For instance, on zero-shot evaluations, it matched the performance of ERNIE 3.0 Titan, which is 25\(\times\) larger.