## 6 Conclusions  
In this paper, we introduced a new concept of ClaimsBERT, a classifier model generated by incorporating feature maps via CNN into BERT, which resulted in significantly improved accuracy and performance. The proposed classifier was established by fine-tuning BERT using two convolution layers--a fully connected dense layer and a classification layer stacked on transformers. This classifier model marks a significant improvement over BertForSequenceClassification, which is a sequence classifier introduced by BERT. We use our curated cybersecurity claim sequences dataset to train our claim classifier. Our ClaimsBERT model increases the accuracy of the claim sequence classifier compared to BertForSequenceClassification, improving it from 72 to 97 percent. The ClaimsBERT model is established based on the feature map generated via CNN. These convolution layers are able to generate a feature map with smaller overall dimensionality, which helps the model understand the input features more efficiently. We also provided an in-depth comparative analysis of ClaimsBERT to show the effectiveness of our fine-tuning strategies and our hyperparameter selection method. The extensive experimental results demonstrate the effectiveness, efficiency and robustness of our ClaimsBERT classifier. The results demonstrate that the performance of our ClaimsBERT is better than all other language models we tested for this paper. The development of our novel classification model was inspired by our ongoing research activities that aim to create a new unbiased, objective, and semi-supervised cybersecurity vetting approach named CYPET that focuses on feature set claims for ICS devices, in order to obtain insights into the impact these ICS devices have on an operator's overall cybersecurity posture. This is based on the detection and verification of vendor claims that are found in device documentation, identified using NLP. In this context, the research advancement presented here represents a fundamental cornerstone of our CYPET program for vetting cybersecurity claims in the broad domain of industrial control systems. We wish to clarify that while our claims classifier is highly successful in identifying sequences that contain claims, it does not provide an indication of whether this is a claim of interest, or a generic claim unrelated to cybersecurity features. To accomplish this finer distinction, our research efforts are also now focusing on a claims type classifier, which requires a subsequent NLP processing step that follows the use of our ClaimsBERT classifier to detect claim sequences. Furthermore, this classification does not provide an indication of whether the claim represents a statement in support of this feature, or expressing the lack of this feature. For this support indication, our research has contributed a sentiment analysis capability to our vetting framework that provides this indication. Nevertheless, we note that these steps \begin{table}
\begin{tabular}{l l l l l l l l} \hline \hline
**Model** & **Best** & \multirow{2}{*}{**Accuracy**} & **Macro** & \multirow{2}{*}{**Meighted F1**} & **AUC** & **Training** & **Testing** & **Trainable** \\
**(ClaimsBERT)** & & & & & & & & \\
**2 Dense** & & & & & & & \\ \hline BERT + NN & 12 Encoder & 0.954 & 0.93 & 0.948 & 32,970 & 97 & 108,647,026 \\ (CyBERT) [8] & 3 Dense & & & & & & \\ \hline BERT + BiLSTM & 12 Encoder & 0.951 & 0.941 & 0.937 & 51,211 & 142 & 116,693,570 \\  & 2 Dense & & & & & & \\ \hline BERT + LSTM & 12 Encoder & 0.947 & 0.937 & 0.929 & 47,176 & 135 & 112,915,970 \\  & 2 Dense & & & & & & \\ \hline BERT + MLP & 12 Encoder & 0.952 & 0.942 & 0.940 & 10,487 & 85 & 1,093,800,482 \\  & 3 MLP & & & & & & \\  & 2 Dense & & & & & & \\ \hline BERTSequence & 12 Encoder & 0.76 & 0.72 & 0.773 & 7832 & 77 & 109,482,240 \\ Classifier & 1 Dense & & & & & & \\ \hline \hline \end{tabular} \({}^{\$}\) Times are in seconds.  
\end{table} 
Table 10: Comparison across all tested models.