## 5 Comparative Analysis Results and Discussion
### Hyperparameter Finetuning Results
#### 5.3.5 Randomness Impact  
[PERSON] et al. suggested that when fine-tuning BERT utilizing a small dataset, it can result in an unstable model [6]. This stems from the fact that the training outcome can be heavily influenced by the randomly selected initial weights and biases [68]. The training process for our model utilizing the optimized hyperparameters was therefore repeated for 100 different random seeds. The purpose of this was to accomplish a statistically reliable result in regards to the accuracy of our model's performance measurements.  
The resulting standard deviation, confidence interval (CI), mean, and the margin of error for the 100 random seeds are presented in Table 8. The Figure 10 displays the distributions of the training accuracy, validation accuracy, and testing accuracy for all 100 random seeds. According the results from Figure 10 and Table 8, the ClaimsBERT's true accuracy was within an interval of 0.952 to 0.956 with a 95% CI. As the results shows in Table 8, the mean accuracy value for both validation and testing improved significantly in ClaimsBERT model.  
The obtained results also suggested that the random seed value selection can impact the ClaimsBERT's accuracy nominally. Hence, based on the literature recommendation [6], the random seed that results in the greatest validation accuracy for ClaimsBERT was selected.  
Despite the fact that our dataset contained only a relatively small number of labelled entries, the results we are showing in Figure 10 and Table 8 clearly illustrate that ClaimsBERT, with optimized configuration, achieves high classification accuracy at a high degree of confidence when identifying cybersecurity feature claims.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Architecture** & & **Filter Size** & **LR** & **Accuracy** & **F1-Score** \\ \hline \multirow{3}{*}{BERT + 1 Convolution} & 1 Dense & (256) & \(6\times 10^{-5}\) & 0.95 & 0.94 \\  & 2 Dense & (256) & \(2\times 10^{-7}\) & 0.94 & 0.93 \\  & 3 Dense & (256) & \(3\times 10^{-5}\) & 0.94 & 0.92 \\  & 4 Dense & (256) & \(7\times 10^{-5}\) & 0.94 & 0.93 \\ \hline \multirow{3}{*}{**BERT + 2 Convolution**} & 1 Dense & (256,128) & \(3\times 10^{-5}\) & 0.94 & 0.93 \\  & **2 Dense** & **(256,128)** & \(9\times 10^{-5}\) & **0.97** & **0.96** \\  & 3 Dense & (256,128) & \(2\times 10^{-7}\) & 0.93 & 0.93 \\  & 4 Dense & (256,128) & \(3\times 10^{-7}\) & 0.91 & 0.89 \\ \hline \multirow{3}{*}{BERT + 3 Convolution} & 1 Dense & (256,128,64) & \(7\times 10^{-5}\) & 0.93 & 0.91 \\  & 2 Dense & (256,128,64) & \(2\times 10^{-6}\) & 0.94 & 0.92 \\ \cline{1-1}  & 3 Dense & (256,128,64) & \(5\times 10^{-5}\) & 0.92 & 0.91 \\ \cline{1-1}  & 4 Dense & (256,128,64) & \(1\times 10^{-6}\) & 0.94 & 0.92 \\ \hline \hline \end{tabular}
\end{table}
Table 7: The impact of the number of convolution and dense layers on fine-tuning ClaimsBERT. The bold row indicates best model parameters with highest accuracy.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Model** & **Dataset** & **SD** & **Mean** & **CI (95\%)** & **Margin of Error** \\ \hline BERT + CNN & Training & 0.007 & 0.991 & 0.99 to 0.993 & 0.00142 \\ (ClaimsBERT) & Validation & 0.009 & 0.953 & 0.952 to 0.956 & 0.00.0018 \\  & Testing & 0.009 & 0.951 & 0.95 to 0.953 & 0.00183 \\ \hline \hline \end{tabular}
\end{table}
Table 8: ClaimsBERT Accuracy Performance Results for Investigating the Seed Randomness Effect.