## 4 Experimental Testing
### Theoretical Analysis
#### 4.2.1 Low Rank Adaptation  
We have introduced \"LoRA: Low-Rank Adaptation of Large Language Models\" [17; 52] in Section 3.5. The motivation for the use of 'LoRA' with 'BLOOM' (introduced in Section 3.3) in particular and the transformer model in general is that only a very small proportion of the parameters need to be trained when compared to the original model. Moreover, the performance of the model can achieve similar or even better results than training all the parameters in the original model.  
As discussed in Section 4.1 (training objectives), the 'BLOOM' model has been selected based on the parameter set and layers along with the demonstrable successful use of the model in other similar studies. More specifically, based on Equation (4), the number  
\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**Hyperparameter** & **BLOOM-560M** & **BLOOM-1.1B** & **BLOOM-1.7B** & **BLOOM-3B** & **BLOOM-7.1B** & **BLOOM** \\ \hline \multicolumn{8}{c}{**Architecture Hyperparameters**} \\ \hline Parameters & 559 M & 1065 M & 1722 M & 3003 M & 7069 M & 176,247 M \\ Precision & & & float16 & & & & bfloat16 \\ Layers & 24 & 24 & 24 & 30 & 30 & 70 \\ Hidden dim & 1024 & 1536 & 2048 & 2560 & 4096 & 14,336 \\ Attention Heads & 16 & 16 & 16 & 32 & 32 & 112 \\ \hline Vocab size & & & & 250,680 & & \\ Sequence length & & & & 2048 & & \\ Activation & & & GELU & & & \\ Position emb & & & ALiBi & & & \\ Tied emb & & & TRUE & & & \\ \hline \hline \end{tabular}
\end{table}
Table 1: BLOOM training hyperparameter specification.  
\begin{table}
\begin{tabular}{l c c} \hline \hline  & **Time/Epoch** & **Global Batch Size** & **Memory** \\ \hline BLOOM & 54 h & 1 & 39 GB \\ BLOOM + LoRA & 4 h & 1 & 39 GB \\ BLOOM + LoRA + DeepSpeed & 4 h & 1 & 36 GB \\ BLOOM + LoRA + DeepSpeed & 3 h & 2 & 39.5 GB \\ \hline \hline \end{tabular}
\end{table}
Table 2: Comparison of training time, batch size, and memory consumption for full-fine-tuning, LoRA when combined with DeepSpeed.  
of trainable parameters depends on the following parameters: \((r)\), \((d_{in})\), \((d_{out})\), and the number of layers \(\left(n_{layer}\right)\) in each backbone.  
In line with the sources cited in this paper, we set hyperparameter \((;=\,16)\) along with the other parameters that depend on the 'BLOOM' model including \(\left(n_{layer}\ =\ 30\right)\) and \((d_{in}\ =\ d_{out}=4096)\). Based on Equation (4), the number of training parameters can be calculated with the choice of \((r=16)\) which, at approximately 7.5 million parameters (which accounts for only 0.11% of the total parameters that will be trained), contains less than the original 7 billion parameters.  
The training time for the model is described in Table 2. For the original model, when training the entire dataset on a single A100 40 GB card, the time to train one epoch consisting of 100 k samples, with a batch size of 1, takes 54 h/100 k samples. With LoRA, the training time for one epoch is reduced to 4 h/100 k samples. Correspondingly, the time to train one epoch is reduced by nearly 14 times, which can help us evaluate a dataset more efficiently and perform full-fine-tuning on that dataset after testing LoRA.