# A Review of Current Trends, Techniques, and Challenges in Large Language Models (LLMs)
## 4 Pretraining--Strategies and Objectives
### Learning Strategies
#### 4.2.3 Mixture of Experts (MoE)-Based Pretraining  
Pretraining LLMs requires significant computing power and resources. To address this issue, sparse experts were proposed, incurring substantially decreased training costs compared to dense models. The same parameters are reused and applied to all the inputs in a traditional static neural network. Instead, a mixture of experts (MoE)-based network enables dynamic selection of parameters for each incoming input and improves model capacity without incurring additional computation costs. In MoE, although a large number of weights are used during training, only relevant experts are needed to compute a small subset of the computational graph at inference time. Additionally, in static networks, as the entire model becomes activated for every example, the training cost is increased (roughly quadratically) with the increase in model size and training examples [65]. Meanwhile, ST-MoE [66] demonstrated how a 269B sparse parameter model has comparable or similar computational cost to an encoder-decoder transformer model with only 32B parameters and still achieves SOTA performance across a variety of NLP tasks. However, in MoE, when the model size is scaled by increasing the number of sparsely gated experts, it can significantly enlarge the parameter size, requiring more storage memory (it can reach the order of hundreds of GBs). In MoE, a trainable gating network determines which combination of sparse experts needs to be selected to process the given input. [PERSON]. [65] introduced MoE and demonstrated how conditional computation using sparsely gated experts improved model capacity by 1000 times, with a minor loss in computational efficiency. This is helpful, especially for language modeling and machine translation tasks, where the model capacity is essential to assimilate or absorb large amounts of information from the corpora. Using MoE, [PERSON]. [65] performed better on language modeling and machine translation tasks than prior studies. Similarly, with MoE, GShard [67] was able to efficiently perform training and inference using conditional computation, where only a subnetwork is activated on a per-input basis. Additionally, the translation quality of GShard increased with model size, but, due to MoE, the wall time of training increased only sublinearly. GShard, pretrained on multilingual, when translating text from 100 languages to English, was able to achieve better translation quality compared to prior cases. Additionally, an annotation technique was used by GShard to annotate the tensors either for distribution or replication across a cluster of devices. MoE-based models incur additional storage space. This might create difficulty in the model training and inference phase if GPUs capacity is exceeded. To address this issue, [PERSON] [34] proposed the INFMOE framework. This framework uses a dynamically scheduled offloading strategy and enables MoE model inference on a single GPU. The parameters of experts from MoE layers are offloaded to CPU memory to enable the inference of the model on a single GPU. As demonstrated in [63], for model training and inference, MoEs yield competitive zero- and few-shot performance (except full-shot finetuning) at a fraction of the computation. MoEs can match the dense model performance with four times less computing. Furthermore, the performance gap between MoE and dense models varies greatly across domains and tasks, indicating that MoE and dense models might generalize differently. GLaM [37] also used sparsely activated MoE architecture to achieve competitive few-shot task results compared to SOTA-dense models while being more computationally efficient. Although GLaM (1.2 T parameters) is seven times larger than GPT-3 in parameters, it activates a subnetwork of 96.6B (8% of 1.2 T) parameters, consumes only one-third of the energy used to train GPT-3, requires only half of the computation flops for inference, and achieves better overall zero-, one-, and few-shot performances across 29 NLP tasks.