# Enhanced Viral Genome Classification Using Large Language Models
## 5 Experimental Results and Discussion
### Model Evaluation and Performance Metrics  
The performance of each pre-trained model is measured using accuracy, precision, recall, F1-score, and ROC curve. Accuracy measures the overall correctness of the model by calculating the ratio of correctly predicted instances to the total number of instances. Precision, also known as positive predictive value, measures the ratio of correctly predicted positive instances to the total predicted positives. Recall, also known as sensitivity or true positive rate, measures the ratio of correctly predicted positive instances to actual positives. The F1-score is the harmonic mean of precision and recall. The formula for calculating accuracy, precision, recall, and F1-score is given in Formulas (2)-(5). The ROC curve plots the true positive rate and false positive rate of classification for different thresholds; thus, it portrays the quality of the model. \[\text{Accuracy}=\frac{TP+TN}{TP+TN+FP+FN} \tag{2}\]  
\[\text{Precision}=\frac{TP}{TP+FP} \tag{3}\] \begin{table}
\begin{tabular}{l l} \hline \hline
**Parameter** & **Value** \\ \hline learning rate & \(5\times 10^{-6}\) \\ training epochs & 10 \\ Optimizer & \(\text{adamw\_torch}\) \\ Stratified K Fold (k-fold) & 10 \\ weight\_decay & 0.01 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Hyper-parameter for the models. \[\text{Recall}=\frac{TP}{TP+FN} \tag{4}\]  
\[\text{F1}-\text{Score}=\frac{2*Precision\;*\;Recall}{Precision+Recall} \tag{5}\]  
The performance of the pre-trained models is presented in Table 2. Each model is trained using the cross-validation technique with a 10-fold split, and each fold is trained for 25 epochs. Initially, the model is trained with genome sequences of length 100 base pairs (bp). GENA-LM and DNAGPT achieved the highest accuracy of 95%, compared to DNABERT at 92% and DNABERT-2 at 94%. In terms of precision, recall, and F1 score, DNAGPT outperformed GENA-LM. DNAGPT has a slightly higher precision rate when compared to GENA-LM. The precision scores of both Bat and COVID classes are 0.01 higher than the GENA-LM model. The higher recall rates of the DNAGPT model indicate fewer false negative errors. Additionally, DNAGPT also has a slightly higher F1-score value for all three classes when compared to GENA-LM. To further enhance the model's performance, the evaluation is conducted by increasing the genome sequence length to 150 bp. The model is trained with a 150 bp sequence length for a cross-validation split of 15 folds for 10 epochs. The performance metrics of the pre-trained model, after being trained with 150 bp, are presented in Table 3. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{**Family**} & \multirow{2}{*}{**Model**} & \multirow{2}{*}{**Classes**} & \multicolumn{4}{c}{**Performance Metrics**} & \multirow{2}{*}{**Weighted Average**} \\ \cline{4-6}  & & & **Precision** & & & **Recall** \\ \hline \multirow{4}{*}{GENA-LM} & \multirow{2}{*}{BigBird} & Bat-Corona & 0.96 & 0.94 & 0.95 & \\  & & MERS & 0.96 & 0.90 & 0.93 & 0.95 \\  & & COVID-2 & 0.91 & 0.99 & 0.95 & \\  & & Bat-Corona & 0.97 & 0.93 & 0.95 & \\ \multirow{4}{*}{DNAGPT} & \multirow{4}{*}{DNAGPT} & MERS & 0.97 & 0.92 & 0.94 & 0.95 \\  & & COVID-2 & 0.90 & 0.99 & 0.94 & \\  & & Bat-Corona & 0.95 & 0.90 & 0.93 & \\ \multirow{4}{*}{DNABERT} & \multirow{4}{*}{DNABERT-2} & MERS & 0.92 & 0.90 & 0.91 & 0.92 \\  & & COVID-2 & 0.89 & 0.96 & 0.92 & \\ \cline{1-1}  & & Bat-Corona & 0.95 & 0.93 & 0.94 & \\ \cline{1-1}  & & MERS & 0.96 & 0.91 & 0.93 & 0.94 \\ \cline{1-1}  & & COVID-2 & 0.91 & 0.98 & 0.95 & \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Class-wise performance of pre-trained model with cross-fold validation (splits \(n=10\)) with a genome sequence length of 100 bp.