# A Survey on Hardware Accelerators for Large Language Models
## 2 Computational and Energy Requirements  
### Computational Complexity  
Large language models (LLMs) are among the most computationally intensive applications in contemporary artificial intelligence research. The computational intensity of LLMs is primarily attributed to their architecture and the scale of the data they process. These models typically consist of hundreds of millions to billions of parameters, which are the learned weights that determine how the model processes and generates language. Training these parameters requires iterating over vast datasets multiple times, a process that demands substantial computational resources.  
Moreover, the complexity of the tasks LLMs are designed to perform exacerbates their computational demands. Tasks such as contextual understanding, nuanced language generation, and handling ambiguities in human language require deep neural networks with multiple layers, each contributing to the overall computational load. The iterative nature of model training and fine-tuning, involving continuous forward and backward propagation through these deep networks, further amplifies the computational requirements.  
One of the most computationally intensive parts of the LLMs is the Transformer networks that were introduced in 2017 [4]. Transformer networks use a technique called attention. Attention, adopted by the field of neuroscience, is the ability to selectively concentrate on specific data while ignoring other data of the environment. In deep learning, we imitate this technique through attention mechanisms. One approach is to encode a sequence not into a single fixed vector, but rather to create a model that produces a unique vector for each output step, by adding a set of weights that will later be optimized.  
Consequently, it does not simply learn what to produce in the output, but also how to selectively weigh specific input data, maximizing the probability of the correct output. For example, BERT uses multiple attention blocks. Each attention block converts the input using GEMM (general matrix multiplication) operations and then uses both GEMM and non-linear functions such as softmax, layer normalization, and Gaussian error linear unit (GELU) to produce the output.  
**Softmax**: Softmax is a mathematical function that normalizes input scores into a probability distribution, making the outputs sum to 1. In attention mechanisms, it is used to compute attention weights, highlighting the relative importance of different elements in a sequence. Mathematically, it is defined as follows:  
\[\text{Softmax}(x_{i})=\frac{e^{x_{i}}}{\sum_{j}e^{x_{j}}}\]  
where \(x_{i}\) represents the input scores.  
**ReLU (rectified linear unit):** ReLU is an activation function that introduces non-linearity by allowing only positive values to pass through and setting negative values to zero. It is computationally efficient and helps prevent the vanishing gradient problem. The ReLU function is expressed as follows:  
\[\text{ReLU}(x)=\max(0,x)\]  
**GELU (Gaussian Error Linear Unit):** GELU is a smooth, non-linear activation function that combines the benefits of ReLU and sigmoid functions. It enables models to capture nuanced patterns by scaling inputs based on their value. The GELU function is approximated as follows:  
\[\text{GELU}(x)=x\cdot\Phi(x)\]where \(\Phi(x)\) is the cumulative distribution function of a standard normal distribution. GELU is commonly used in Transformers as it provides better gradient flow and performance compared to ReLU.  
An approximation of \(\Phi(x)\) is often used in practice, as follows:  
\[\text{GELU}(x)\approx 0.5\cdot x\cdot\left(1+\tanh\left(\sqrt{\frac{2}{\pi}}(x+0.04 4715\cdot x^{3})\right)\right)\]  
These building blocks work in tandem to enable attention networks to model complex relationships within data, contributing to their effectiveness in natural language processing and other tasks. Together with GEMM, these building blocks are often the most computationally intensive parts of the LLM that are often mapped to specialized hardware resources.