# Comparative Analysis of BERT and GPT for Classifying Crisis News with Sudan Conflict as an Example
## 5 Results
### K-Fold CV  
To address concerns with the limited test size from the original 80/20 split (43 articles), 5-fold cross-validation was applied to strengthen the reliability of the evaluation. This was performed across four BERT configurations: MLC and SL-MLG using both bert-base-uncased and bert-large-uncased.  
As shown in Table 7, performance improved across all metrics when using bert-large-uncased, particularly for MLC, which achieved an F1-score of 70.35 \(\pm\) 2.12% and an accuracy of 39.24 \(\pm\) 4.01%. In comparison, the original 90/10 split yielded lower MLC F1-scores of 62.85% (standard bert-base) and 54.62% (standard bert-large). Similarly, SL-MLG accuracy improved under cross-validation, with bert-large achieving 85.83 \(\pm\) 3.53%, compared to 76.74% in the original split. These results underscore the importance of using cross-validation, especially in low-resource settings, to produce more robust and generalizable performance estimates.  
Due to computational constraints, hyperparameter tuning via Optuna was excluded from the cross-validation runs. As a result, only standard BERT configurations without tuning were evaluated. Despite repeating training and inference across multiple folds, the average fold times remained reasonable, ranging from 9 m 16 s (bert-base) to 31 m 17 s (bert-large), further supporting the practicality of K-fold cross-validation for evaluating classification models in this domain.  
\begin{table}
\begin{tabular}{c c} \hline \hline \multirow{2}{*}{**Incident Narrative**} & Clashes between the Sudanese Armed Forces (SAF) and its paramilitary counterpart, the Rapid Support Forces (RSF), continue in various parts of the country. On April 18, artillery shelling led to the death of a person in El Obeid, the capital of North Kordofan. \\ \hline \multirow{2}{*}{**Ground Truth**} & Indiscriminate use of weapons, Willful killing of civilians, Military operations (battle, shelling) \\  & Willful killing of civilians, Military operations (battle, shelling), Indiscriminate use of weapons \\  & \multirow{2}{*}{**LLM Output**} & 100\% \\  & & 100\% \\
**ACC.** & & 100\% \\
**PREC.** & & 100\% \\
**REC.** & & 100\% \\
**F1** & & 100\% \\
**TP** & Military operations (battle, shelling); Willful killing of civilians; Indiscriminate use of weapons \\
**FP** & N/A * \\
**FN** & N/A * \\ \hline \hline \end{tabular}  
* False Positives (FP) and False Negatives (FN) are marked as N/A because the model in the example achieved a perfect prediction. All ground truth categories were correctly identified, and no incorrect labels were predicted, resulting in zero false positives and zero false negatives.  
\end{table}
Table 6: Example of an MLC result generated by Llama3.3-70B using the RAG ICL method.  
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**MODEL** & **TASK** & **ACC. \(\pm\) SD** & **F1 \(\pm\) SD** & **PREC. \(\pm\) SD** & **REC. \(\pm\) SD** & **AVG. FOLD TIME** \\ \hline bert-base-uncased & MLC & 27.90 \(\pm\) 2.90\% & 56.94 \(\pm\) 4.76\% & 59.50 \(\pm\) 7.62\% & 57.29 \(\pm\) 4.37\% & 9 m 16 s \\ bert-large-uncased & MLC & 39.24 \(\pm\) 4.01\% & 70.35 \(\pm\) 2.12\% & 76.85 \(\pm\) 4.52\% & 69.53 \(\pm\) 2.20\% & 30 m 7 s \\ bert-base-uncased & SL-MLG & 77.10 \(\pm\) 5.95\% & — & — & — & 9 m 23 s \\ bert-large-uncased & SL-MLG & 85.83 \(\pm\) 3.53\% & — & — & — & 31 m 17 s \\ \hline \hline \end{tabular}
\end{table}
Table 7: K-Fold CV results for BERT models on MLC and SL-MLG without HP tuning.