# Enhanced Viral Genome Classification Using Large Language Models
## 5 Experimental Results and Discussion
### Model Evaluation and Performance Metrics
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{**Family**} & \multirow{2}{*}{**Model**} & \multirow{2}{*}{**Classes**} & \multicolumn{4}{c}{**Performance Metrics**} & \multirow{2}{*}{**Weighted Average**} \\ \cline{4-6}  & & & **Precision** & & **Recall** & **F1** & **Accuracy** \\ \hline \multirow{4}{*}{GENA-LM} & \multirow{2}{*}{BigBird} & Bat-Corona & 0.97 & 0.92 & 0.94 & \\  & & MERS & 0.97 & 0.92 & 0.94 & 0.95 \\  & & COVID-2 & 0.90 & 1.00 & 0.95 & \\  & & Bat-Corona & 0.96 & 0.97 & 0.96 & \\ \multirow{4}{*}{DNAGPT} & \multirow{4}{*}{DNABERT-2} & MERS & 0.98 & 0.91 & 0.94 & 0.96 \\  & & COVID-2 & 0.93 & 0.99 & 0.96 & \\ \cline{1-1}  & & Bat-Corona & 0.95 & 0.90 & 0.93 & \\ \cline{1-1}  & & MERS & 0.92 & 0.90 & 0.91 & 0.92 \\ \cline{1-1}  & & COVID-2 & 0.89 & 0.96 & 0.92 & \\ \cline{1-1}  & & Bat-Corona & 0.96 & 0.94 & 0.95 & \\ \cline{1-1}  & & MERS & 0.97 & 0.92 & 0.94 & 0.95 \\ \cline{1-1}  & & COVID-2 & 0.92 & 0.99 & 0.95 & \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Class-wise performance of pre-trained model trained for cross-fold validation (splits \(n=15\)) with genome sequence of length 150 bp. The DNAGPT model achieves an improved accuracy of 96% after increasing the DNA sequence length to 150 bp. Similarly, DNABERT-2 accuracy also improved to 95%, whereas the accuracy of the other two models, GENA-LM and DNABERT, remains the same irrespective of the length of the genome sequence. The advanced contextual understanding of the genomic sequence allowed the DNAGPT model to capture more complex patterns. Additionally, the DNAGPT model is trained with genomes from all mammals, including viruses and bacteria. This specialized training data have enabled the model to exhibit improved performance. The confusion matrix represents the correctly classified instances in the diagonal and misclassified instances in the off-diagonal. The DNAGPT model has more correctly classified instances for BAT and MERS, while GENA-LM has slightly more correctly classified instances for COVID than DNAGPT. GENA-LM correctly classified more instances of COVID class than other models. The confusion matrix of all the pre-trained models and their variants is shown in Figure 4. The observation that DNAGPT classified MERS and BAT classes well, even after SMOTE-based data augmentation, suggests a greater robustness to learning from synthetic data compared to GENA-LM. GENA-LM's higher accuracy on the COVID class likely reflects the initial imbalance in the dataset, where COVID instances were more abundant. The subsequent addition of synthetic data to the MERS and BAT classes through SMOTE seems to pose a more significant challenge for GENA-LM to learn effective representations, unlike DNAGPT. The training loss and validation accuracy for the pre-trained models and their variants are shown in Figure 5a-d. During training, as shown in Figure 5a, the GENA-LM model exhibits fluctuations in loss across epochs. Initially, the training loss decreased; however,  
Figure 4: Confusion matrix of the pre-trained model. in later epochs, after epoch 23, it began to increase. In contrast, the DNAGPT, DNABERT, and DNABERT-2 models, as shown in Figure 5b-d, exhibit a gradual and steady decrease in training loss over the epochs. Additionally, the validation accuracy for the DNABERT model steadily increases, whereas for GENA-LM and DNAGPT, the validation accuracy fluctuates, with periods of increase followed by a drop. Figure 5: \(Cont.\)  
The Area Under the Curve (AUC) of the Receiver Operating Characteristic (ROC) curve is a performance measurement for classification models at various threshold settings. It plots the True Positive Rate (TPR) against the False Positive Rate (FPR) as shown in Figure 6, and measures the model's ability to distinguish between classes. A higher AUC value indicates that the model can better differentiate between positive and negative classes.