## 3 Results
### Machine Learning and Deep Learning Applications
#### 3.3.1 Deep Learning Methods in Data Extraction
The authors evaluated the model on the Microsoft Security Bulletins dataset and a malware dataset from CTI reports, observing that it outperformed several state-of-the-art and baseline models in terms of recall, precision, and F1-score. [PERSON] et al. [93] created a model based on Bidirectional Encoder Representations from Transformers (BERT) [94], BiLSTM networks, and CRF to automate the analysis of APTs. Extracted threat actions were used to construct a cyber threat ontology. The ontology can be used for IoC and TTP generation. The model was evaluated on a small sample-relevant dataset and outperformed other threat action extractors, scoring 97% and 96% in recall and accuracy, respectively. [PERSON] et al. [29] proposed a DL-based entity and relationship extraction system for CTI. Unstructured threat intelligence text data are used as input, with BERT, BiLSTM, and Convolutional Neural Network (CNN) architectures used for the extraction process. The model ultimately returns a KG stored as a Neo4j graph database. Its performance was assessed against 227 threat intelligence documents and compared to different baseline models, where it achieved the highest F1-score on the entity extraction, co-reference resolution, and relation extraction tasks. [PERSON] et al. [95] created a framework for extracting CTI triples from ATP reports, which they named APTKG. The authors also constructed APTOnt, an ontology based on STIX representation. A BiLSTM model is used for entity extraction along with CRF, BERT, CNN, and StanfordCoreNLP [96], while a BiGRU is used for inferring relations between the extracted CTI entities. An APT group-focused KG is ultimately constructed. When compared to alternative DL models, APTKG achieved better precision, recall, and F1-scores in CTI entity recognition. The authors characterized the joint model for entity and relationship inference as \"effective\", highlighting its relatively high F1-score (70.86%). A method for automating the CTI extraction of threat actions from unstructured data was developed by [PERSON] et al. [97]. K-CTIAA detects professional knowledge in a CTI KG through knowledge queries and uses it as an input into pretrained models for better understanding of the semantics of security-related terms. Another feature of K-CTIAA is the suggestion of countermeasures for rapid response against cyber attacks. The proposed method was tested on APTNotes [98], an open-source APT report repository, and outperformed CTI analysis tools such as ActionMiner [99], TTPDrill [59], and different NN architectures, achieving high precision (0.931), recall (0.951), and F1-score (0.941). [PERSON] and [PERSON] [100] used a different approach in knowledge extraction, leveraging Large Language Models (LLMs) such as ChatGPT for CTI KG construction. They created a ChatGPT-based pipeline for the collection of metadata and appropriate relationships, which were later used for the generation of triples. A CTI ontology was also used in this process. The created triples were ultimately parsed again through the LLM to ensure their quality and validity. The proposed method was tested on thirteen threat intelligence reports, outperforming AttackKG [57] and REBEL [101] in terms of recall and F1-score. However, the authors reported that ChatGPT occasionally generated relations that diverged from the structure of the ontology. Finally, [PERSON] et al. [102] created CSKG4 APT, a platform based on a novel algorithm utilizing BERT for threat knowledge recognition from bilingual documents. The platform results in the construction of an APT KG that adheres to different threat intelligence standards. During evaluation, information from English and Chinese CTI reports was extracted. CSKG4 APT outperformed other relevant models such as BERT-LSTM, BERT-BiLSTM, and BERT-BiLSTM-CRF in terms of both micro- and macro-evaluation metrics. The works related to DL methods for knowledge extraction from text data combined with ontology or KG construction or usage are summarized in Table 6.