# A Review of Current Trends, Techniques, and Challenges in Large Language Models (LLMs)
## 3 Transformer
### Prefix (Non-Causal) Language Model  
Prefix language models are also decoder-only-based models but differ in the masking mechanism. Instead of a causal mask, a fully visible mask is used for the prefix part of the input sequence, and a causal mask is used for the target sequence.  
For example, to translate an English sentence \"I am doing well\" to French, the model would apply a fully visible mask to the prefix \"translate English to French: I am doing well. Target:\", followed by causal masking while predicting the target \"je vais bien\". Also, unlike causal language models where the targets-only paradigm is used, the prefix language model uses the input-to-target paradigm. Both causal and prefix model architectures areautoregressive as the objective is to predict the next token. However, the causal model uses a unidirectional attention mask, while the prefix model modifies the masking mechanism to employ bidirectional attention over prefix tokens. Figure 4 demonstrates the mechanism of the above architectures. The lines represent the attention visibility. Dark lines represent the fully visible masking (bidirectional attention), and light gray lines represent causal masking (unidirectional attention).  
As shown in Figure 4, in the encoder-decoder architecture, fully visible masking is used in the encoder and causal masking is used in the decoder. In a decoder-only model, the input and target are concatenated, and then a causal mask is used throughout. A decoder-only model with a prefix allows fully visible masking over part of the input token (prefix), followed by causal masking on the rest of the sequence. In general, autoencoding models learn bidirectional contextualized representation suited for NLU tasks, whereas autoregressive models learn to generate the next token and hence are suited for NLG tasks. Table 1 details architectural information of prominent LLM models, such as their parameter size, hardware used, number of encoder (E) and decoder (D) layers, attention heads, etc.  
\begin{table}
\begin{tabular}{l l l l l l} \hline \hline
**Model** & **Param Size** & **Layers** & **d-Model** &
\begin{tabular}{l} **Attention** \\ **Heads** \\ \end{tabular} & **Hardware** \\ \hline Transformer-base [24] & - & 6 E, 6 D & 512 & 8 & 8 NVIDIA P100 GPUs \\ Transformer-big [24] & - & 12 E, 12 D & 1024 & 16 & 8 NVIDIA P100 GPUs \\ BERT-base [26] & 110 M & 12 E & 768 & 12 & 4 Cloud TPUs \\ BERT-large [26] & 340 M & 24 E & 1024 & 16 & 16 Cloud TPUs (64 TPU chips) \\ GPT-1 [27] & 117 M & 12 D & 768 & 12 & - \\ GPT-2 [28] & 117 M to 1.5 B & 24 D to 48 D & 1600 & 48 & - \\ GPT-3 [29] & 175 B & 96 & 12,288 & 96 & V 100 GPUs (285 K CPU cores, 10 K GPUs) \\ T5 [25] & 220 M–11 B & (12 E, 12 D) & - & - & 1024 TPU v3 \\ REALM [30] & 330 M & - & - & - & 64 Google Cloud TPUs, 12 GB GPU \\ Jurassic-1 [31] & 178 B & 76 & 13,824 & 96 & - \\ mT5 [32] & 13 B & - & - & - & - \\ Pangu-Alpha [33] & 207 B & 64 & 16,384 & 128 & 2048 Ascend 910 AI processors \\ CPM-2 [34] & 198 B & 24 & 4096 & 64 & - \\ Yuan 1.0 [35] & 245 B & - & - & - & - \\ HyperClova [36] & 82B & 64 & 10,240 & 80 & 128 DGX servers with 1024 A100 GPUs \\ GLaM [37] & 1.2 T (96.6) & 64 MoE & 8192 & 128 & 1024 Cloud TPU-V4 chips (Single System) \\ ERNIE 3.0 [38] & 10 B & 48, 12 & 4096,768 & 64, 12 & 384 NVDIA v100 GPU cards \\ Gopher [39] & 280 B & 80 & 16,384 & 128 & 4 DCN-connected TPU v3 Pods (each with 1024 TPU v3 chips) \\ Chinchilla [40] & 70 B & 80 & 8192 & 64 & - \\ AlphaCode [41] & 41.1 B & 8 E, 56 D & 6144 & 48, 16 & - \\ \hline \hline \end{tabular}
\end{table}
Table 1: Architecture details of LLMs.  
Figure 4: Different transformer architectures in LLMs.