# A Review of Current Trends, Techniques, and Challenges in Large Language Models (LLMs)
## 8 LLM Challenges
### Cost and Carbon Footprints  
As stated in CPM-2 [34], the cost of using pretrained language models increases with the growth of model sizes. The cost consists mainly of three parts.  
1. Computational cost for pretraining: large language models require thousands of GPUs with several weeks of pretraining.
2. Storage cost for finetuned models: a large language model usually takes hundreds of gigabytes (GBs) to store, and as many model copies as the number of downstream tasks need to be stored.
3. Equipment cost for inference: it is expected to use multiple GPUs to infer a large language model.  
So, as the model size increases, they become hard to use with limited computational resources and unaffordable for most researchers and institutions.  
Furthermore, the pretraining phase of large language models consumes massive energy responsible for carbon dioxide emissions. The formulas used in LLaMA to estimate the Watt hour (Wh) and carbon emissions are listed in Equations (6) and (7), where 0.385 in Equation (7) is the US national average carbon intensity factor (0.385 kg \(CO2 eq/\)KWh) and PUE represents power usage effectiveness.  
\[Wh=(GPU-h)\times(GPU\ power\ consumption)\times(PUE) \tag{6}\]  
\[tCO2 eq=MWh\times 0.385 \tag{7}\]  
As stated in LLaMA [57], carbon emission also depends on the data center's location used during pretraining of the network. For instance, BLOOM uses a grid that emits 0.057 kg \(CO2 eq/\)KWh, leading to 27\(tCO2 eq\), and OPT uses a grid that emits 0.231 kgCO2 eq/KWh, leading to 82 tCO2 eq. As stated in [106], a couple of factors are involved in computing the electricity required to run an NLP model, such as algorithm, program, number of processors running the program, speed and power of those processors, a data center's efficiency in delivering power and cooling the processors, and the energy supply mix (renewable, gas, or coal). Cloud data centers can also be \(1.4-2X\) more energy-efficient than typical data centers. A more detailed and granular formula stated in Equation (8) was presented in [106] that captures the carbon footprint of an NLP model:  
\[\begin{split} Footprint&=(electrical\ energy\ train+\\ &=+\ queries\times electrical\ energy\ inference)\\ &\times CO2e\ datacenter/KWh\end{split} \tag{8}\]  
To decrease the footprint of training, an ML researcher should pick the DNN model, the processor, and the data center carefully. The above Equations (6) and (7) can be restated in terms of energy consumption and CO2 emissions as Equations (9) and (10) below.  
\[\begin{split} KWh=Hours\ to\ train\times Number\ of\ Processors\times\\ & Average\ Power\ per\ Processor\times PUE\div 1000\\ & tCO2e=KWh\times kg\ CO2e\ per\ KWh\div 1000\end{split} \tag{9}\]  
To address the cost and carbon footprint problems, there is a need to improve the energy efficiency of algorithms, data centers, software, and hardware involved in implementing NLP models. Emphasis should be placed on reducing carbon footprint by building more efficient LLMs. For example, OPT [49] is comparable to GPT-3 and requires only 1/7 th of the carbon footprint to develop.  
Ref. [106] also recommends three suggestions that could eventually help to reduce the CO2e footprint:  
1. report energy consumed and CO2e explicitly;
2. reward improvements in efficiency as well as traditional metrics at ML conferences;
3. to help everyone understand its cost, include the time and number of processors used during training.  
As highlighted in [106], large but sparsely activated DNNs can consume <1/10 th the energy of large, dense DNNs without sacrificing accuracy despite using as many or even more parameters.