# BVTED: A Specialized Bilingual (Chinese-English) Dataset for Vulnerability Triple Extraction Tasks
## 4 Experiments and Evaluation--Results
### Experimental Result
#### 4.3.2 Baseline Models of Cybersecurity Relation Extraction  
To train and test the BVTED, the experiments use two groups of common relation extraction models. The first group are classification-based relation extraction approaches, which include classification methods based on BiLSTM and classification methods based on CNN. In the second group, CasRel and OneRel are the joint relation extraction state-of-the-art baselines. \begin{table}
\begin{tabular}{l c c c c c} \hline \multirow{2}{*}{**Entity Types**} & \multicolumn{2}{c}{**CRF**} & \multicolumn{4}{c}{**BiLSTM + CRF**} \\ \cline{2-6}  & **BERT** & **Word2 Vec** & **BERT** & **ERNIE** & **Multilingual BERT** \\ \hline Product & 0.9154 & 0.9504 & 0.8997 & 0.9497 & **0.9537** \\ Definition & 0.9906 & 0.9911 & 0.9911 & 0.9920 & **0.9962** \\ Vendor & 0.8867 & 0.9283 & 0.8690 & 0.9323 & **0.9410** \\ Location & 0.9978 & 0.9961 & 0.9978 & 0.9934 & **0.9978** \\ Vulnerability & 0.9639 & 0.9756 & 0.9546 & 0.9842 & **0.9874** \\ Version & 0.8909 & 0.9286 & 0.8660 & 0.9430 & **0.9462** \\ Part & 0.8740 & 0.9185 & 0.8309 & 0.9155 & **0.9196** \\ Consequence & 0.9127 & 0.9577 & 0.8550 & **0.9922** & 0.9844 \\ Attacker & 0.9901 & 0.9975 & 0.9901 & **0.9999** & 0.9988 \\ Method & 0.8494 & 0.9280 & 0.7732 & 0.9777 & **0.9779** \\ Reason & 0.8619 & 0.9402 & 0.7368 & 0.9699 & **0.9787** \\ Condition & 0.6753 & 0.9483 & 0.7153 & 0.9464 & **0.9913** \\ Time & 0.6190 & 0.5641 & 0.3673 & 0.7692 & **0.8000** \\ \hline Whole F1 & 0.9290 & 0.9575 & 0.9113 & 0.9628 & **0.9667** \\ \hline \end{tabular}
\end{table} 
Table 6: Comparison of F1 values of our cybersecurity entity extraction model and other baseline methods, i.e., CRF and BiLSTM-CRF, with multiple embedding models based on BVTED. ERNIE + BiLSTM: [PERSON] et al. [39] proposed bidirectional long short-term memory networks (BiLSTM) to model the sentence and two entities with complete, sequential information about all words and classified relations using a softmax classifier. We utilize the knowledge-enhanced pre-training model ERNIE as an embedding layer to provide rich semantic context for the following feature extraction step. These two modules are combined to form the first baseline model ERNIE + BiLSTM. PCNN: [PERSON] et al. [40] proposed a novel model dubbed the Piecewise Convolutional Neural Network (PCNN), with multi-instance learning for a distant supervised relation extraction task. This baseline model solved the wrong label problem and error propagation or accumulation problem, avoiding feature engineering and capturing the structural information between two entities using a CNN architecture with a piecewise max pooling method. CasRel: [PERSON] et al. [41] proposed a novel cascade binary tagging framework (CasRel) that models relationships as mapping subjects and objects, instead of classifying discrete relation labels of entity pairs. This architecture addresses the overlapping triple problem using three components. First, a BERT encoder module is used to extracted the feature information of the input sentence. Second, a subject tagger of a cascade decoder is designed to detect all possible subjects in the input sentence. Third, a relation-specific object tagger of a cascade encoder, a high level tagging module, is used to identify objects and the involved relations with respect to the subjects obtained from a lower level. OneRel: [PERSON] et al. [42] proposed a novel model named OneRel, designed for joint entity and relation extraction. This model frames the joint extraction task as a fine-grained triple classification problem. Specifically, OneRel incorporates a classifier to assess whether a token pair and a relation form part of a factual triple. Additionally, it features a relation-specific horn tagging strategy, ensuring a straightforward yet effective decoding process. Table 7 shows the experimental results of four baselines on the dataset BVTED. The experiments use precision, recall and the F1-score. to evaluate the performance of each model on the dataset BVTED.