CTIKG: LLM-Powered Knowledge Graph Construction from Cyber Threat Intelligence
Authors: Liangyi Huang
Xusheng Xiao
Venue: COLM (2024)
Keywords: Large Language Model, Machine Learning and Security, Knowledge Graph, Information Extraction
TL;DR: Collaboration of multiple LLM Agents for knowledge extraction from articles in the computer security field

Abstract: To gain visibility into evolving threat landscape, knowledge of cyber threats has been aggressively collected across organizations and is often shared through Cyber Threat Intelligence (CTI). While knowledge of CTI can be shared via structured format such as Indicators of Compromise (IOC), articles in technical blogs and posts in forums (referred to as CTI articles) provide more comprehensive descriptions of the observed real-world at- tacks. However, existing works can only analyze standard texts from mainstream cyber threat knowledge bases such as CVE and NVD, and lack of the capability to link multiple CTI articles to uncover the relationships among security-related entities such as vulnerabilities. In this paper, we propose a novel approach, CTIKG, that utilizes prompt engineering to efficiently build a security-oriented knowledge graph from CTI articles based on LLMs. To mitigate the challenges of LLMs in randomness, hallucinations and tokens limitation, CTIKG divides an article into segments and employs multiple LLM agents with dual memory design to (1) process each text segment separately and (2) summarize the results of the text segments to generate more accurate results. We evaluate CTIKG on two representative benchmarks built from real world CTI articles, and the results show that CTIKG achieves 86.88% precision in building security-oriented knowledge graphs, achieving at least 30% improvements over the state-of-the-art techniques. We also demonstrate that the retry mechanism makes open source language models outperform GPT4 for building knowledge graphs.

Published as a conference paper at COLM 2024
CTIKG: LLM-Powered Knowledge Graph Construction from Cyber Threat Intelligence
Liangyi Huang, Xusheng Xiao School of Computing and Augmented Intelligence Arizona State University, Tempe, AZ, USA (1huan139, xusheng. xiao edu
Abstract
To gain visibility into evolving threat landscape, knowledge of cyber threats has been aggressively collected across organizations and is often shared through Cyber Threat Intelligence (CTI). While knowledge of CTI can be shared via structured format such as Indicators of Compromise (IOC), ar- ticles in technical blogs and posts in forums (referred to CTI articles) provide more comprehensive descriptions of the observed real-world at- tacks. However, existing works can only analyze standard texts from mainstream cyber threat knowledge bases such as CVE and NVD, and lack of the capability link multiple CTI articles to uncover the relationships among security-related entities such as vulnerabilities. In this paper, we propose a novel approach, CTIKG, that utilizes prompt engineering effi- ciently build security-oriente knowledge graph from CTI articles based on LLMs. To mitigate the challenges of LLMs randomness, hallucinations and tokens limitation, CTIKG divides an article into segments and employs multiple LLM agents with dual memory design (1) process each text segment separately and (2) summarize the results of the text segments to generate more accurate results. We evaluate CTIKG on three representative benchmarks built from 500+ real world CTI articles, and the results show that CTIKG achieves 86.88% precision in building security-oriented knowl- edge graphs, achieving at least 30% improvements over the state-of-the-as techniques. We also demonstrate that the retry mechanism makes open source language models outperform GPT4 for building knowledge graphs.
1 Introduction
In recent years, advanced cyber attacks often exploits multiple vulnerabilities and progres- sively penetrates the enterprise network to compromise security and confidentiality. such as the 12CE3 ransomware attack chain (Keshavarzi & Ghaffary, 2020). To effectively counter these attacks, knowledge of cyber threats has been aggressively collected across organi- zations and is often shared through Cyber Threat Intelligence (CTI), which models cyber threats including their actors, tactics and techniques McMillan (2013); Wagner et al. (2019). CTI has been primarily collected in the form of Indicators of Compromise (IOCs) Obrst et al. (2012); Liao et al. (2016), which are forensic artifacts of an intrusion such as malicious file/ prrccss names and IPs/ domains of botnets. For example, variety of security websites have published their own IOC reports of the Log4J vulnerability (Kim, 2022; ElastZris, 2022; Azure, 2023). Automated security detection solutions can also benefit from IOCs. For instance, the Microsoft Malicious Software Removal Tool (MSERT) employed relevant IOCs following the cyber attack the Chilean bank regulator (Onyegbula, 2023). Additionally, CrowdStrike Falcon, Microsoft Sentinel and Microsoft Defender XDR integrate IOC as component of their detection systems (CrowdStrike, 2023; Microsoft, 2023).
Besides structured formats like IOCs, recent studies Liao et al. (2016); Dong et al. (2019) show that the information of cyber threat delivered by traditional blacklists Group; MX is rather thin, which covers only a limited set of 1OC classes (URL, domain, IP and MD5) and cannot reveal the relations between IOCs. Instead, descriptions of cyber threats, such articles

---

Published as a conference paper at COLM 2024
in technical blogs and posts in forums, referred to as CTI articles, are more favorable for security practitioners since they provide more comprehensive descriptions of the observed real-world attacks. For example, Figure 1 shows three CTI articles that describe the malware Bedep. Based on these articles, Bedep has relationship with Angler and Magnitude, and performs click-fraud, which can not be found in professional databases. Thus, existing techniques Obrst et al. (2012); Liao et al. (2016); Catakoglu et al. (2016); Dong et al. (2019) that focus extracting loCs is insufficient to represent the cyber threat knowledge in CTI since they overlook the IOCs' behaviors and their relationships. To enhance the representation of the knowledge accumulated in CTI, specialized knowledge graph that depicts domain- specific knowledge through entity-entity relationships, which has been widely used among large enterprises for data management (Yan et al., 2018) and Natural Language Processing (NLP) tasks (Liu et al., 2021), can be employed.
Constructing knowledge graph out of CTI articles entails automatic techniques for extracting triples from the sentences within these articles, and linking these triples to form graph. Al- though existing works based pre-trained language mod- els show promising results in extracting triples from gen- eral purpose texts, they show limited capabilities in dealing
Angler E.K Malverts Expose
US Userst "the second of which delivers the Angler Expioit kit the exploit kit proceeds download BEDEP variant"
Angler
TeshCrypt
Razsemware
download
"Badep dowulead additional malware
paylonds and inclist advertising click-faad activity."
Independent" Hacked, Leads to IestaCrypt Remsonware
Bedep
loads
initiate
"Bedee also silendl loads the Magnitude exploit
Click-fraud
Maguitude
activity
Figure 1: Correlated CTI articles for Bedep
with the complex domain-specific sentence in these articles (Satvat et al., 2021; Rossiello et al., 2023). Fortunately, large language models (LLM) have recently been shown to per- form much better in NLP tasks such as text comprehension, triple extraction, coreference resolution, and context reading (OpenAI, 2023a; Brown et al., 2020; 01.AI, 2023; 01-ai, 2023; Achiam et al., 2023), and thus becomes a potential solution to analyze the complex entities and relationships from CTI articles.
Goal. Recognizing the importance of CTI and potential of LLMs, we aim to build a novel LLM approach, CTIKG, that effectively analyzes descriptions of security-related entities in CTI articles to build a knowledgge graph and reveals relationships and behaviors among security-related entities across CTI articles. In particular, the nodes in the knowledge graph represent security- related entities and the edges in the graph represent the relationships among these security- related entities across different CTI articles.
Challenges. We encounter several key challenges due to the limitation, hallucinations and inaccurate comprehension of LLMs:
Token Limitation: Since some CTI articles have long contexts, and LLM prompts need to include chain of thoughts with examples, LLMs cannot process an entire CTI article at once due to the token limitation, and may miss important CTI information.
Incorrect Triple Format: Even explicitly specified in the prompts, LLMs may not follow the required formats, and will produce extracted information as tuples containing more than two elements.
Incorrect Output: LLMs may extract inaccurate information from the given text, such as the information from the given example but not the given text or wrong triple information due to misunderstanding the text.
Misunderstood Task: LLMs may misunderstand the task as the task for generating subse- quent text of the input CTI article and do not produce triples.
Contributions. To address these challenges, CTIKG performs text segmentation on CTI arti- cles and employs multiple LLM agents with dual memory design to process text segments separately (results stored in shorf-term memory) and summarize results to produce final results (results refined in the long-term memory). In this way, CTIKG can effectively address the token limitation and use long chain of thoughts to guide the LLM agents to extract triples from CTT articles in every text segment. Moreover, CTIKG employs multiple LLM agents to
2

---

Published as a conference paper at COLM 2024
Short-term Memory Construction
Long-term Memory Constraction
Knowledge Graph Constraction
Short-temm Memory Triples
Long-term
Multi-article
CTI Articles
Memory of Triples
Knowledge
Graph
Artiele Segment
Nonnalization Resolution
Text Segments
Integruted Triples
Merger Modification
Update
Merger Modification
Update
Weekers Extraction
Integrator Summay
Singlo-artiele Knowledge
Modified
Preliminary Triples Lists
Triples
Gruph
Figure 2: Architecture CTIKG
perform triple extraction with different temperature settings and then integrates the results using another LLM agent, mitigating the hallucinations and randomness of LLMs. More importantly, CTIKG employs checker LLM agent that instructs other LLM agents to redo their tasks if the results are incorrect, greatly improving the robustness of the constructed knowledge graph.
Evaluations. We evaluate CTIKG on three benchmarks constructed from 500+ real world CTI articles. For triple extraction from 255 sentence cross 13 cyber attack tactics defined by ATT&CK knowledge base (Corporation, 2022), CTIKG achieves 91.89% precision and 89.39% recall, which is better than Extractor (Satvat et al., 2021), REBEL (Cabot & Navigli, 2021) and KnowGL (Rossiello et al., 2023) by more than 10%. For knowledge graph construction from 30 CTI articles, CTIKG achieves 86.88% precision and 70.86% recall, which is at least 30% better than these three approaches in both precision and recall. We also evaluate CTIKG on 478 correlated CTI articles that describe 15 CVE entities, and show that CTIKG can averagely find 39.2 entity behaviors for each CVE entity by connecting a CTI article to 14.6 correlated CTI articles, compared 8.6 entity behaviors that can be found by analyzing a single CTI article. We also demonstrate that local LLM models (01-ai, 2023) deployed on single GPU can outperform GPTT-4 through prompt engineering, LLM agents cooperation, and chain of thoughts. Our implementation of CTIKG and the evaluation datasets are available at our project website (CTIKG, 2024). We also provide fine-tuned models at our project website (Research, 2024a). The latest model is currently based on Swwen1.--70B (Research, 2024b).
Overview
Figure 2 shows the architecture of CTIKG, consisting of three phases:
Short-term Memory Construction. CTIKG deploys multiple LLM agents extract the knowledge within a CTT article segment in the form of triples. This phase employs four types of LIM agents: worker, integrator, refiner, and checker. The worker agents extract the knowledge from the natural language text to form the triples. The integrator agent combines the multiple results on the same text from multiple workers. The refiner agent achieves the normalization and resolution tasks on the combined triples. The checker agent inspects the triples and instructs the other agents re-work when it detects common errors of the LLMs.
Long-term Memory Construction. As CTIKG segment processing to overcome LLMs' token limitation, the merger agent is responsible for assembling the triples from different article segments into one long-term memory.

---

Published as a conference paper at COLM 2024
Knowledge Graph Construction. CTIKG builds knowledge graph using multiple articles. To ensure consistency across different articles for the same entity (eg., a specific malware), CTıKG employs another merger agent to achieve unified entity description.
3 Approach
3.1 Design of Triple Extraction
CTIKG extracts knowledge from article as triples in the form of (subject, relation, object), where subject and object are entities and relation is their relation (e.g., (Linux, is, Operating System)).
Short-Term Memory and Long-Term Memory. Inspired by existing LLM studies (Park et al., 2023), CTIKG divides the article into segments and extracts the triples from each segment as the short-term memory. Meanwhile, CTIKG maintains list of all extracted triples from different segments as the long-term memory. Figure 3 shows the algorithm for building a knowledge graph with these two memories.
Text Segmentation. To segment a CTI article, CTIKG first divides into paragraphs at line breaks and merges these paragraphs into segments. For paragraphs exceeding 600 characters, CTIKG uses NLTK (Loper & Bird, 2002) further split the paragraphs into sentences and reassemble the sentences to form segments. The rationale behind such text segmentation shown in the Appendix A.
Entity Types and Relationships. Unlike the ontology-based knowledge graph, which can only extract predefined relationships and entity types, CTıKG is designed dynamically capture all types of relationships related to any cyber threat in CTI articles. As shown in the Appendix G.3, the similarity between CVE-2012-0158 and CVE-2017-1188 was discovered based on the entity relationship "for" and "memory corruption vulnerability". Our further investigation revealed that Magniber and Cerber have "mutual ransomware payload" relationship, and Emotet and Trickbot have "switch to" relationship. These entity relation- ships are determined by the author's 's writing style and cannot be anticipated in advance and defined in the ontology.
3.2 Construction of Short-Term Memory
To construct the short-term memory of a text segment, CTIKG leverages four types of agents: worker, integrator, refiner, and checker. The prompts for these agents can be found at Appendix B.
LLM
Worker Agent. Worker agents, told as experts in information extraction and computing, are tasked with extracting security-related data from texts (Lines 4-7). In particular, to instruct LLMs to focus on security-related data, we specify "Only extract triples that are related to cyber attacks. If a sentence does not contain any triple about cyber attacks, skip the sentence and do not include in your output" and "Focus on malware, Trojan horses, CVEs, or hacking organizations
the subjects of the triples" in the prompts of worker agents, and provide multiple security-related examples. CTIKG deploys three worker agents with the same prompt and different temperature settings (1, 0.5, and 0.2). If at least two worker agents identify no cyber security-related triples in a segment, CTIKG skips the other agents and returns empty result for that segment directly (Line 11). Specifically, the prompt for the worker agents contain the following parts:
Background: An introduction that defines the task of triple extraction with examples. Rules: An instruction that asks the worker agent to extract only entity relations relevant cyber attacks, and provides output format requirements.
Few-shots: Three complete chat conversations, showing example inputs and responses. Input: An instruction that asks the worker agent to perform triple extraction on given text.
Integrator Agent. The integrator agent extracts a final triple from the worker agents' results (Line 8). Due to the randomness of LLM (Lee et al., 2022), multiple workers may use
4

---

Published as a conference paper at COLM 2024
Algorithm 1: Single-article Knowledge Graph Construction
Input: Single Article
Output: Knowlodge Craph &
Function faooleedeergpaoossrrrcco...ll
Split A into segments
foreach segment in do T-initial empty list.
foreach worker agent do
T.i 4- W AgrkerAgent(a)
Appoud T. Tinitial
T Tiintegrate t- Integrator Ageentt Agent -iiittia))
T Treefine + Refiner Agennnrr integrate) if NonSecurity SesSeuurttFFeedbac([ (enttaac(Tiiiiittta) .initial) <2 then
10
11
L Skip to next segment.
12
if Checker (gent(T refine) is True then
if long-term memory L does not crist then L T Tiniiiall
else
T mmodified Merger (e if Checker modifice True then
17
18
L L 4- L Tmodified fied
19
else
20
Recall (ergor.Agen((Treefiiee L) up to three times
21
else
22
Recall WorkorAgents) to three times
return L
Figure 3: Algorithm for single-article knowledge graph construction
different triples to express the same entity relation in the text. The integrator is informed to summarize the valid triples with identical meanings and output representative triple. Specifically, the prompt for the integrator agent contains the following parts:
Background: An introduction that defines the task and provides examples to illustrate which triples assumed to have the identical meanings.
Rules: An instruction that asks the integrator agent to integrate the triples with identical meanings, and provides expected input and output format.
Few-shots: A description that consists of three complete chat conversations, showing how
integrator successfully does its job with the right outputs.
Inpuf: An instruction that provides the outputs from the three worker agents as the input for the integrator agent to process.
Refiner Agent. The refiner performs the normalization and resolution tasks (Line 9) (Satvat et al., 2021), and leverages chain of thoughts (Wei al., 2022) for complex task. The refiner agent takes the triples produced by the integrator agent, and performs the following tasks:
Simplification: Some triple's subject and object parts have modifiers or unnecessary pre- fixes.
Splitting Complex Triples: Some parts of the triples may contain conjunctions.
IOC Conversion: Some phases may contain IOCs and should be highlighted in the triple. Coreference Resolution: The refiner agent will replace the implicit references in a triple with the actual name according the context.
Normalization: The refiner agent will standardize the relation in triples.
Checker Agent. Although prompts emphasize requirement several times, LLMs may still output the incorrect results. Through empirical observations of the incorrect outputs, summarize some common patterns. The checker agent CTIKG checks whether the outputs from other agents based the given rules and determines whether the output is correct based on the summarized patterns of incorrect outputs (Line 12). If the output of an LLM agent is incorrect, CTIKG then instructs the LLM agent to re-perform the task. Due to resource constraints, we limit such retry to be up to three times.

---

Published as a conference paper at COLM 2024
3.3 Construction of Long-Term Memory
CTIKG uses the merger agent to further refine short-term memory triples to fit the long-teni memory (Line 16). After refinement and check, CTIKG directly adds the triples into the long-term memory (Line 18). The merger agent deter- mines whether the triples' subjects or objects in short-term and long- term memory refer to the same en- tities. It then modifies the corre- sponding subject or object in the short-term memory triple to unify representations of identical enti- ties. Since these tasks require plex reasoning, the merger also leverages chain of thoughts to im- prove effectiveness (Wei et al., 2022). Specifically, the merger agent performs the following tasks:
Algorithm 2: Multi-article Knowledge Graph Coustruction
Input: Long-tore Long-term memory of cach CTI article Li...., L.n
Output: Knowledge graph G
Function crossledggeaap.oosnnnnttooollldd
2 Initialize as an empty graph
3 forcach CTI article L; in Li...., Lu do foreach triple in L; do
5
VectorFmbbeddinn(()) N = FindSimilarNNdes/ V)
Tindified MergerAgent( N)
Add Tmodi fed as new edge
return G
Figure 4: Algorithm for multiple CTT articles
Unifying Terms: Subjects and objects that contain partially identical terms and refer to the same entities are unified by removing prefixes, suffixes, or modifiers. Modifier Removal: Specific names, such as those of malware or hacker organizations, should have all unnecessary modifiers removed.
Hallucination Restriction: The creation of triples through LLM's imagination is strictly prohibited.
3.4 Building Knowledge Graph
Based on the long-term memory of each CTT article, CTIKG builds knowledge graph using the subjects and the objects nodes and the relations as edges. Figure 4 shows the algorithm for knowledge graph construction. Specifically, CTIKG use triples from first article to build the basic graph, then gradually incorporates triples from other CTI articles. To maintain consistency of entity names across articles, CTIKG leverages another merger agent. CTIKG first uses the RoBERTa model function (Liu et al., 2019) to achieving the text embedding of subject and object (Line 5). For each triple to be added, CTIKG searches five most similar nodes in the knowledge graph based on the embedding vector similarity (Line 6). These five node names are used by the merger agent to refine the triple (Line 7). Finally, CTIKG connects the refined triple to the knowled graph (Line 8).
4 Evaluation
In the evaluations, we aim to answer the following research questions:
RQ1: How effectively can CTIKG extract triples from CTI articles?
RQ2: How effectively can CTIKG generate knowledge graphs for CTI articles?
RQ3: How effectively can CTIKG reveal relationships of security-related entities from conrelated CTI articles?
4.1 Evaluation Setup
Implementation. We implement CTIKG ( (0K characters prompts) on Ubuntu 20.04.6 server with AMD 5955WX CPU, RTX 6000 GPU and 128GB memory upon VLLM, OpenAI, Networkx, Stanford NLP tool suite, and NLTK (Kwon et al., 2023; OpenAl, 2023b; develop- ers, 2023; Lab, 2003; Loper & Bird, 2002). We choose Yi-34B-4bits model as the core LLM model. We provide more details LLM selection in Appendix D

---

Published as a conference paper at COLM 2024
Evaluation Subject. We curate large dataset with 72, CTI articles from security websites and cyber threat knowledge base, and construct three benchmarks:
RQ1 Sentence Benchmark: For each enterprise tactic defined by the Mitre ATT&CK (Cor- poration, 2022), select over 20 representative sentences from the CTI articles, resulting in 255 CTI sentences with 699 manually extracted triples.
RQ2 CTI Article Benchmarrk: We category CTI articles into three types: standalone (high- lighting a single cyber threat), chain (introducing multiple interlinked cyber threats), and overview (introducing multiple cyber threats that are not directly related) categories. Ten articles were selected from each category for manual analysis, resulting in 30 knowledge graphs with total of 315 edges.
RQ3 Correlated CTI Article Benchmark: Using the Stanford NLP tool (Lab, 2003), we perform entity name extraction on the all CTT articles, identify 15 most common CVE entities, and construct a knowledge graph based on the 478 articles containing these entities. The average length of an article is 5, 200 characters. The graph has 27, ,70 unique nodes, 26, edges and 7, 623 unique edges. In total, 39.45% of the nodes are related to computer science topics, and 51.94% of the edges are related to computer science topics.
We provide more details of the dataset in Appendix E.
4.2 RQ1: Effectiveness of Triple Extraction
To evaluate CTIKG's triple extraction, we compare CTIKG with three baseline approaches and two variants of CriKG powered by GPT-4 (Achiam et al., 2023) and GPT-3.5 (OpenAI, 2023a). The evaluation is conducted on the triple extraction benchmark constructed from the CTI article dataset.
Baseline Approaches. We compare CTIKG with two state-of-the-art entity relation extrac- tion approaches: REBEL (Cabot & Navigli, 2021) and KnowGL (Rossiello et al., 2023), both of which are transformer-based seq2seq models for generic text. We also compare CTIKG with secure text-oriented approach, Extractor (Satvat et al., 2021), which are based BERT (Devlin et al., 2018) and BiL.STM (Hameed & Garcia-Zapirain, 2020). We also use two variants of CTIKG (CTIKG based on GPT-4 and GPT-3.5) to evaluate the impact of the underlying LLM models. These two variants of CTIKG use the same prompts, except for the checker agent and the corresponding retry mechanism. This is due to the fact that OpenAI only provides the paid API, and applying the retry mechanism will result in significant expenses.
Effectiveness Comparison. Table 1 shows the precision and the recall of CTIKG and the baseline approaches. Overall, CTIKG with the YI model achieves the best performance with 91.89% precision and 89.39% recall, and CTIKG with GPT-4 achieves the second best performance: 84.26% precision and 83.66% recall. The precision and the recall of CTIKG with GPT-3.5 are only about 77.88%. Meanwhile, two approaches for general texts, REBEL and KnowGL, completely fail the task of triple extraction the security-relater sentences, achieving both precision and recall below 21%. Extractor achieves a slightly worse performance with 82.56% precision and 79.35% recall. For specific cyber attack tactics, CTIKG achieves the best performance (> 94%) for the tactics of Persistence, Discovery, and Lateral Movement, and worst for the tactics of Impact (about 84%). The second best approach, CTIKG with GPT-4, has a larger variant for different tactics, achieving only 71.59% for the tactics of Initial Access.
While GPT-4 is a better-trained model with top MMLU scores, thus CTIKG with GPT-4 suffers from the randomness and hallucination due to lacking of checker agent and retry mechanism. CTIKG with GPT-3.5 has the lowest MMLU score model, and lacking the checker agent and retry mechanism, achieves the worst results among these three versions of CTIKG. While Extractor is competitive with CTIKG with GPT-4 in terms of accuracy due to its well-designed internal components and multi-step pipeline, its text comprehension capability is less powerful than that of LLMs. When processing sentences like "The first thread is responsible for finding the CHD in the process, writing the results to a file, and preparing the files for exfiltration", " Extractor cannot find the relationship between the first thread and the file. This makes its recall lower than CTIKG with YI and GPT-4
7

---

Published as a conference paper at COLM 2024
Table 1: Comparison of CTIKG and baseline approaches in triple extraction
Taclics
Precision
CTIKG
Recall
CTIKG
Prediion
Recall
Precision
Recall
Precision
Recall
Precision
Recall
Precision
Recall
lmple Senience
Initial Access
91.20 91.8%%
83,6%%
71.50 71.29%
66
H1 11.77%
42.29%
73,65%
75.35%
17.87%
17.4%%
4.70%
1,5%
22
Executicn
88 27%
32.15%
62.25%
52.01%
65.35%
07.61%
84.96%
43.75%
17.4%
17.5%
18.2%
5.5%
Peststince
00.35
77.87
703315
0130%
3032%
71.0%
1,39%
Faviloge Excalation
8588%%
7138%
71025
67.35%
37.9%
2133%
1838%
15.0%
5.8%
Condenital Accens
80.9%
72.09
2760%
7400%
20335
87.0%%
76305
1238%
1838%
1500%
119
Discovery
85.1%
422%
8833%%
7638%
8525%
79.28%
753%%
1000
995%
835%
1025
31
Labral Vlavement
14.2%
15.3%%
09.1%
09.0%
70.07
80205
8504%
50.0%
33.8%
17388
26.9%
4338%
Collecson
19.00
28355
1700%
1787
111
4.17%
Exfiltation
Neerage
01.00%
89.30%
$4.70%
76.45%
77385
87.30%
79.0%%
21.00
1809
15.8%%
630%
31:1
Error Analysis. Since LLMs are essentially text generation models, the next word generated is dominated by probability (OpenAI, 2023a). We summarize some LLM-specific errors in triple extraction as follows:
Hallucination: Hallucination is the most common problem. For example, for sentence "Both apps upload users phone book to remole server and use for SMS spam GPT-4 generates new non-existent sentence and incorrect triple { (AwwareY, is linked to, GhostNet).
Misunderstood Task: Another common error is misunderstood task. For example, for the sentence "This image file exists on the third page of the document, so the user would have to scroll down in the document to this third page to get the SWF file to run", CTIKG with GPT-4 may outputs the incorrect triple FFommooo,, is, malware) based on the few-shot example. 12. and 6.95% of the outputs of CTIKG with GPT-3.5 and GPT-4, respectively, incorrectly use few-shot sample text due to lack of checker agent.
Text Comprehension Ability: The level of text comprehension ability has a great impact on the outputs. CTIKG with GPT-3355 often misunderstands the content, while CTIKG with the YI model and GPT-4 are much less likely to do so. For the sentence "Octopus uses wmic for local discovery information." ", only CTIKG with GPT-335 produced the incorrect result ( Octopus, uses, information..
Recent studies show that evaluations using LLMs can match or surpass the accuracy of manual inspection (McAleese et al., 2024; Zheng et al., 2024; Kenton et al., 2024). In addition to manual inspection, we use an automatic evaluator based on GITT-, which has an accuracy rate of 93%, to evaluate the correctness of triples and categorize their error types using few-shot learning and debating mechanism. The results demonstrate that CTiKG, when utilizing the YI model and checker agent, extracts fewer hallucinated triples compared to CHıKG based on GPT-4 and GPT-3.5. This finding demonstrates that the checker agent effectively filters out hallucinated triples. Additionally, the number of triples with incorrect relationships lower than that produced by CTIKG using the other two LLM models. Hal- lucinations often cause other triples within the same sentence to be incorrect. Consequently, the checker, despite not being specifically designed to address such errors, can reduce the number of incorrect triples by filtering out hallucinated triples. The detailed figure is shown in Appendix F.2.
Impacts of Sentence Complexity. We measure the complexity of a sentence using the number of ground truth triples and word count. As shown in Figure 6, with the complexity increases, the precision of CTIKG with the YI model improves while its recall decreases, and thus the overall effectiveness does not change much. It mainly due the check agent that leverages retry to minimize errors. CTIKG with GPT-4 achieves similar performance with the complexity increases but has a larger variance due to the randomness of GPT-4. Both Extractor and CTIKG with GPT-3.55 achieve worse results when the complexity increases. Although the syntax of these sentences is relatively simple, it requires Extractor to be able to detect relationship between two entities far apart. For example, Extractor fails to extract triples from the sentence "The overall purpose of Cannon to use several email accounts to send system data (system information and screenshot) to the threat actors and to ultimately obtain a payload from an email from the actors". CTIKG with the YI model and CTIKG with GPT-4 have better context reading capabilities, which allows them to find these relationships correctly. Figure/ shows a similar trend for Extractor: with the increase of the words in sentence, Extractor performs worse. Overall, CTIKG with the YI model and CTIKG with GPT-4 achieve the most robust results for sentences of different complexity. We notice that the performance of CTIKG with GPT-4 does not degrade as the number of triples or the
8

---

Published as a conference paper at COLM 2024
Table 2: Comparison of CrIKG and baseline approaches in knowledge graph construction
CTIKG
Extractor
REBEL
KnowGL
Type
Precision
Recall
Precision
Recall
Precision
Recall
Precision
Recall
Standalone
90.21%
79.63%
50.83%
34.03%
31.67%
6.11%
39.58%
9.87%
Chain 82.38%
62.89%
66.00%
38.67%
46.67%
16.67%
0.00%
0.00%
Overview 88.04%
70.08%
50.97%
50.13%
15.64%
5.05%
5.28%
2.27%
Average
86.88%
70.86%
55.93%
40.94%
31.32%
9.28%
14.95%
4.03%
number of words within sentence increases. This is mainly due to GPT-4's larger token limit, which is 131, 072 tokens, larger than GPT-3.55 and the YI model's default 4, 096 tokens.
Entity Types and Relationships. We have manually 699 cyber security-related triples of the RQ1 sentence benchmark, and fount out that 51.96 96% of nodes cannot be categorized into common entity types such malware names, and 54.93% of edges cannot be categorized into computer sclence-related relationship types. These results show that most of the entities and their relationships cannot be easily anticipated in advance and defined in the ontology.
4.3
RQ2: Knowledge Graph Construction
To evaluate CTIKG's knowledge graph construction, we compare CTIKG with the three base- lines: Extractor (Satvat et al., 2021), REBEL (Cabot & Navigli, 2021), and KnowGL (Rossiello et al., 2023) on the knowledge graph benchmark. Since demonstrated that CTIKG performs better than CTIKG with GPT-4 and GPT-3, we no longer include them in RQ2.
Effectiveness Comparison. As shown in Table 2, CTIKG achieves the best precision (86.88%) and recall (70.86%), which is at least 73% better than the second best approach, Extractor, in both precision and recall. Extractor achieves a precision of 55.93% and a worse recall of 31.32%. Both REBEL and KnowGL perform poorly on these security-related CTI articles, with precision and recall below 35%. This is consistent with their triple extraction effective- ness since inaccrrate triple extraction will cause the built knowledge graph to be even more inaccurate.
Across article types, CTIKG performs best with standalone articles and least effectively with chain articles. This variance is due to the complexity of chain articles, which detail multiple interrelated threat entities, making them more difficult to analyze than standalone articles, which focus on single cyber threat and maintain cohesion on a topic. Despite these challenges, CTIKG's precision and recall significantly outperform baseline methods, with improvements of at least 24.8%. We provide more details on the impacts of article type in Appendix C.
Error Analysis. By inspecting the final results of CTIKG, and the intermediate results of the short-term memories and the long-term memories, we found that besides those common errors observed in triple extraction, CTIKG also suffered from the following problems:
Length limit: Although the LLM context length limit is extended to 16, 384 tokens, CTIKG may still need more space during the chain of thoughts process (Wei et al., 2022). If there are too many triples in the short-term memory, the merger agent will reach the length limit when performing the chain of thoughts and output unfinished results. This limitation cannot be easily addressed by performing retry.
Incorrect Coreference Resolution: If an article segment uses demonstrative pronoun instead of a specific name throughout the text, the merge agent may incorrectly replace those pronouns with another specific name from the long-term memory. This error can be mitigated by recording the original sentence for each triple, but this method will make the problem of length limit worse.
4.4 RQ3: Revealing Entity Relationships Across Correlated CTI Articles
In this RQ, we compare the knowledge graph constructed from correlated CTI articles with the knowledge graph constructed from single CTI article, and use the number of edges connected to a specific node in the knowledge graph as a metric to represent the number of that entity's behaviors described within the graph.

---

Published as a conference paper at COLM 2024
Effectiveness Comparison. For the 15 CVE entities in RQ3 benchmark, the knowledge graph based on single article has an average of only 8.6 edges per CVE entity, representing 8.6 entity behaviors extracted from the article. The multi-article knowledge graph has averagely 39.2 entity behaviors per CVE entity. It indicates that by integrating the knowledge from correlated CTI articles, each CVE entity gains an additional 30.6 entity behavior descriptions, which is a 428.76% improvement. On average, each CVE is connected to 14.6 other articles, with 2.1 additional entity behaviors found by connecting to correlated article. Appendix G shows the detailed effectiveness data for each CVE, and a corresponding case study.
Discussion
Retrieval Augmented Generation. We used Retrieval Augmented Generation (RAG) (Jeong, 2023) to dynamically generate examples for each prompt. However, since LLM sometimes uses these examples part of the input, we cannot predict the wrong output that may occur when hallucinations occur when using RAG. Currently, we use fixed examples to eliminate the hallucination phenomenon.
Performance and Transferability The current CTıKG based on YI-32B processes an article in 5 minutes on dual RTX A6000. The newly released model, LLaMA3-8B, has shown better performance with a smaller model size according LMSYS Chatbot Arena Leader- board (Chiang et al., 2024). Since CTIKG can seamlessly switch to any LLM model as the backend, tool with LLaMA3-SB takes only 50 seconds to process an article. In the long run, the CTIKG with improved LLMs will achieve better performance with more advanced GPUs.
Other Domains While it is not within our current focused scope, in future work, we plan to verify the effectiveness of CTIKG with adaptation to other domain knowledge other domains like SciERC (Luan et al., 2018).
6 Related Work
Knowledge Graph Construction. Nidhi et al. Rastogi et al. (2020) propose an ontology to generate knowledge graphs for malwares. Aritran et al. Piplai et al. (2020) executes malware and record the behavior in a knowledge graph and merge with knowledge extracted from CTI blogs. Shenqi et al. Qin & Chow (2019) and Anders et al. Høst et al. (2023) build CVE- related knowledge graphs based on the NVD database's context. Rossiell et al. Rossielll et al. (2023) introduce End-to-End knowledge graph construction approach for general text. Some recent research Hu et al. (2024); Yao et al. (2023) also use LLM knowledge graph construction. CodeKGC Bi al. (2024), knowledge graph construction approach based on LLMs, can extract only pre-defined entity relationships and handle each sentence without considering the context of other sentences in the article, while CTIKG can automatically uncover different types of security-oriented entity relationships and process the whole article with the help of the dual memory design.
Conclusion
We have proposed CTIKG, which employs multiple LLM agents and dual memory design to build a knowledge graph from CTI articles with high effectiveness. CTIKG divides long CTI article into text segments and processes each text segment separately using multiple LLM aganets with different temperature settings for mitigating the randomness LLMs. CTIKG then summarize the results of the text segments to generate more accurate results. Our evaluations on two representative benchmarks derived from real world CTI articles demonstrate the superiority of CTIKG over the otate-of-the-art approaches.
10

---

Published as a conference paper at COLM 2024
References
01-ai.
01-ai/yii-4b-ccat--bbs,, 2023. Vi-34B-Chat-4bit.
URL httoss//huegiingaacecc///1-111
01.AL. Yi-34b-chat-4bits, 2023. URL https: https::/huggiiggface. /
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXie preprint ar Xrf::2303308877, 2023.
Azure.
Log4j vulnerability exploit aka log4shell ip ioc, 2023. URL https: ootpr//ons///paaaee220o000// 12021. yaml.
Zhen Bi, Jing Chen, Yinuo Jiang, Feiyu Xiong, Wei Guo, Huajun Chen, and Ningyu Zhang- Codekge: Code language model for generative knowledge graph construction. In Pro- ceedings of the ACM Transactions Asian and Low-Resource Language Information Processing (TALLIP), volume 23, pp- 1-16. ACM, 2024.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Proceedings of the Advances in Neural Information Processing Systems (NIPS), volume 33, pp. 1877-1901, 2020.
Pere-Lluís Huguet Cabot and Roberto Navigli. Rebel: Relation extraction by end-to-end language generation. In Proceedings of the Findings of the Association for Computational
Linguistics (EMNLP), pp. 2370-2381, 2021.
Onur Catakoglu, Marco Balduzzi, and Davide Balzarotti. Automatic extraction of indicators of compromise for web applications. In Proceedings of the International Conference on World Wide Web (WWW), pp. 333-343, 2016.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating lims by human preference, 2024.
The MITRE Corporation. Mitre att&ck,, 2022. https://attack.mitre.rrg//
Kim Crawley. A brief history of malware: Part three (1993-1999), 2016. URL https://medium.
CrowdStrike. How to ingest iocs and integrate with siem solutions, 2023. URL https: (ttinyurl.com/5nns5kk Accessed: 2024-08-07.
CTIKG. Ctikg reseach, 2024.
Darknet.
unix-privesc-check tion scanner, 2015.
user privilege escala-
URI. https://wwws https://www.darkket. org. wtt.eet..rrr.uu/11///e00080 uk/2215/06/
NetworkX developers. Networkx, 2023. URL httptt://neewwwrx...oo/
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-
training of deep bidirectional transformers for language understanding. arXiv preprint arXio:1810.04805, 2018
Ying Dong, Wenbo Guo, Yueqi Chen, Xinyu Xing, Yuqing Zhang, and Gang Wang. Towards the detection of inconsistencies in public security vulnerability reports. In Proceedings of the USENIX Security Symposium (USENIX Security), pp. 869-885, 2019.
ElastZris. The complete list of log4j indicators of compromise (ioc) to date, 2022. URL https://otx. htt.en/www...coooooo///eeeeeeees
11

---

Published as a conference paper at COLM 2024
Thomas Etheridge. Ir team investigations uncover ecrime use of nation- state attack methods, 2018. ir-team-investiie
URL https://www, https://ww..crowdstrkkkk cww.dstrike..eeeeeee com/blog/
Cisco Talos Intelligence Group. Phishtank. http://www..phisaank.com
Zabit Hameed and Begonya Garcia-Zapirain. Sentiment classification using single-layered bilstm model. IEEE Access, 8:73992-74001, 2020.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arxiv preprint arXiv:2009.0333 033000 2020.
Anders Molmen Høst, Pierre Lison, and Leon Moonen. Constructing a knowledge graph from textual descriptions of software vulnerabilities in the national vulnerability database. aiv preprint ar Xiv:2300.00038, 2023.
Yuelin Hu, Futai Zou, Jiajia Han, Xin Sun, and Yilei Wang. Lim-tikg: Threat intelligence knowledge graph construction utilizing large language model. Computers & Security, pp. 103999, 2024.
Cheonsu Jeong. Generative ai service implementation using 1lm application architecture: based on rag model and langchain framework. Journal of Intelligence and Information Systems, 2944)129911 2023.
Zachary Kenton, Noah Y Siegel, János Kramár, Jonah Brown-Cohen, Samuel Albanie, Jannis Bulian, Rishabh Agarwal, David Lindner, Yunhao Tang, Noah Goodman, et al. On scalable oversight with weak Ilms judging strong llms. arviv preprint ar Xii:2447.004222 2024.
Masoudeh Keshavarzi and Hamid Reza Ghaffary. 12ce3: A dedicated and separated attack chain for ransomware offenses as the most infamous cyber extortion. Computer Science Review, 36:100233, 2020.
Christopher Kim.
Log4j indicators of compromise to date, 2022.
URL
https://blogs. infoblox. int.s//o..ooo..oootttteaaaaaiaae Log4j-indicators- of woopoooimiiieeeeeeoooooooeeee to-date/.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the Symposium on Operating Systems Principles (SOSP), pp. 611-626, 2023.
Stanford NLP Lab. The Stanford Parser: A statistical parser, 2003. URL http://http: shtml.
1/nlf.stanford wawanford.eu//aeeeee
Mina Lee, Percy Liang, and Qian Yang. Coauthor: Designing human-ai collaborative writing dataset for exploring language model capabilities. In Proceedings of theACM Conference on Human Factors in Computing Systems (CHI), pp. 1-19, 2022.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-followinn models. https://gittuu: coom/tatsu-lab/aae aaaab//laca.evall....
Xiaojing Liao, Kan Yuan, XiaoFeng Wang, Zhou Li, Luyi Xing, and Raheem Beyah. Acing the ioc game: Toward automatic discovery and analysis of open-source cyber threat intelligence. In Proceedings of the ACM Conference Computer and Communications Security (CCS), pp- 755-766, 2016.
Xiaochen Liu, Yang Su, and Bingjie Xu. The application of graph neural network in natural language processing and computer vision. In Proceedings of the International Conference on Machine Learning, Big Data and Business Intelligence (MLBDDBI, pp. 708-714. TEFF, 2021.
12

---

Published as a conference paper at COLM 2024
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zer,, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. arXiv preprint ar arXie:1907.11692, 2019.
Edward Loper and Steven Bird. Nltk: The natural language toolkit. arXiv preprint cs/0205028, 2002.
Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. arXiv preprint ar Ar::::1008.0900 2018.
Nat McAleese, Rai Michael Pokorny, Juan Felipe Ceron Uribe, Evgenia Nitishinskaya, Maja Trebacz, and Jan Leike. Lim critics help catch Ilm bugs. arXiv preprint ar arXi::2407.00215, 2024.
Rob McMillan. Open threat intelligence, 2013. URL https://ww.gartneer.cmm 2487216/
Microsoft. What are indicators of compromise (iocs)?, 2023.
URL
https:/ https://ww..iccroooftt wttpruuussurtttuuuuneeessssss
Clean MX. Clean mx viruswatch mailing list. URL http://ssss..llllmmmommooo////// leelmmm/issinnooviruuusi
Leo Obrst, Penny Chase, and Richard Markeloff. Developing an ontology of the cyber security domain. In Proceedings of the International Conference Semantic Technology for Intelligence, Defense, and Security (STIDS), pp. 49-56, 2012.
Blessing Onyegbula. Tocs: Indicators of compromise explained, 2023. URL https:///ww splunk. com/en ss/bllo//leamm/oociiniiccccllll
html.
OpenAI. Chatgpt: Applications, opprrrunttssss and threats. arXiv preprint arXiv:2304.09103, 2023a.
OpenAI. The official python library for the openai api, 2023b. httptss: com/ openai/openai-pythiee
Joon Sung Park, Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the Annual ACM Symposium on User Interface Software and Technology (UIST), pp. 1-22, 2023.
Aritran Piplai, Sudip Mittal, Mahmoud Abdelsalam, Maanak Gupta, Anupam Joshi, and Tim Finin. Knowledge enrichment by fusing representations for malware threat intelligence and behavior. Proceedings of the JEEE International Conference on Intelligence and Security Informatics (ISI), pp- 1-6, 2020.
Shengzhi Qin and K. P. Chow. Automatic analysis and reasoning based on vulnerability knowledge graph. In Proceedings of the Cyberspace Data and Intelligence, and Cyber-Living, Syndrome, and Health (CyberDI), pp. 3-19. Springer Singapore, 2019.
Nidhi Rastogi, Sharmishtha Dutta, Mohammed J. Zaki, Alex Gittens, and Charu Aggarwal. Malont: An ontology for malware threat intelligence. In Gang Wang, Arridhana Ciptadi, and Ali Ahmadzadeh (eds.), Deployable Machine Learning for Security Defense, pp. 28 44, Cham, 2020. Springer International Publishing. ISBN 978-3-030-59621-77
CTIKG Research. Ctikg models, 2024a. https///hhgginngaeeeoo/rreelll
CTIKG Research. Ctikg model based on qwen1.5-70b, 2024b.
13

---

Published as a conference paper at COLM 2024
Gaetano Rossiello, Md Faisal Mahbub Chowdhury, Nandana Mihindukuulasooriyy,, Owen Cornec, and Alfio Massimiliano Gliozzo. Knowgl: Knowledge generation and linking from text. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 37, Pp. 16476-16478, 2023.
Kiavash Satvat, Rigel Gjomemo, and VN Venkatakrishnan. Extractor: Extracting attack behavior from threat reports. In Proceedings of the IEEE European Symposium on Security and Privacy (EuroS&P), pp. 615. IEFE, 2021.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Sounya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint 7XXx:2307.09288, 2023.
Thomas D Wagner, Khaled Mahbub, Esther Palomar, and Ali E Abdallah. Cyber threat intelligence sharing: Survey and research directions. Computers & Security, 87:101589, 2019.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the Adoances in Neural Information Processing Systems (NIPS), volume 35, pp. 24824 24837, 2022.
Jihong Yan, Chengyu Wang, Wenliang Cheng, Ming Gao, and Aoying Zhou. A retrospective of knowledge graphs. Frontiers of Computer Science, 12:55-74, 2018.
Liang Yao, Jiazhen Peng, Chengsheng Mao, and Yuan Luo. Exploring large language models for knowledge graph completion. arxiv preprint Xiv:23308 13916, 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. In Proceedings of the Advances in Neural Information Processing Systems (NIPS), volume 36, 2024.
A Research on Text Segmentation Parameter
We used data-driven method to set the text segment size, analyzing 13,385 CTI-related sentences with an average length of 141 characters. Open source LLMs, such as the YI model (01-ai, 2023), accurately handle up 600 characters or about 4 sentences. Hence, limit segments to 600 characters.
B Prompts for LLM Agents
B.1 Prompt for Worker Agent
"As an AI trained in entity extraction and relationship extraction. You're an advanced AI expert, so even if give you a complex sentence, you'll still be able to perform the relationship extraction task. A triple is a basic data structure used to represent knowledge graphs, which are structured semantic knowledge bases that describe concepts and their relationships in the physical world. A triple must has three elements: /Subject, Relation, Objectl. In entity extraction, you follow those rules:"
User:
Rule 1:
"Only extract triples that are related to cyber attacks. If a sentence does not have any triple about cyber attacks, skip the sentence and do not print in your output."
Rule 2: "Make sure your results is python dictionary format. One example is [subject1, relation1, object1]."
14

---

Published as a conference paper at COLM 2024
Preset assistant responses: got it."
User: "Here is one sentence from example articlerattempts to infiltrate target networks through various means of intrusion: watering hole websites, vulnerability scans of network services on the internet, and silee-forrediiciioonn login attempts."
Preset assistant responses: "[Leafminer, attempts to infiltrate, target networks], [Leafminer, use, watering hole websites), [Leafminer, use, vulnerability scans of network services on the Leaffie,,,,eeffviinr, use, brute-force,, [Laffinner, use, dictionary login attempts]."
User: "Here is another sentence from example article: Kismet is also a powerful tool for penetration testers that need to better understand their target and perform wireless
LAN discovery."
Preset assistant responses: "[Kismet,is powerful tool for, penetration testers], [testers, understand, their target], [testers, perform, wireless LAN discovery]."
User: "Here my sentence, extract all possible entity triples from it. Now, I start to give you sentence. [Input Text] .Nw,, my input text are over. You MUST follow the rules I told you before."
B.2 Prompt for Integrator Agent
"You responsible for combining the three different entity extraction results from three different assistants extracting from the same sentence into one. The final results is python list format. When you encounter multiple triples with the same meaning but slightly different words, choose the one that expresses most succinctly and do not add the others to your final result. Just answer me the final python dictionary with triple format without any other words."
User:
B.3 Prompt for Refiner Agent
"You play the role an entity extraction expert and hetiiiyy/mmmlfy/////e /split the text (extracted multiple triples) in the entity extraction result I gave you. according to the following rules. A triple basic data structure used represent knowledge graphs, which are structured semantic knowledge bases that describe concepts and their relationships the physical world. Here are all rules you must follow."
User:
"If the subject or object in triple contains pronouns such as it, they, malware, trojan, attacker, ransomware, or group, replace them with specific name as much as possible according to the context."
Rule 1:
Rule "Focus on malware, Trojan, CVE, or hacking organization or as the subject of the triples, if subject with malware or Trojan or CVE or hacking organization is found and has additional suffixes, remove the suffixes."
Rule 3: "Split a complex triple into multiple simpler forms. For example, [Formbook and XLonder, are, malware] should be split into [Formbook, is, makwarel and [XLoader,is, malware)."
Rule 4: "If the [subject, relation] in a triple can be formed into a new [subject, relation, object] triple, create new triple while keeping the original one."
Rule 5: "If the object can be simplified to a more concise, generic expression, create new triple while keeping the original one. For example, [Formbook, save, XLoader in desktopl must has a new triple /Formbook, save, XLoader] due to the object XLoader in desktop can be simplified to XLoader."
Rule 6: "Simplify the subject, object, and relation into a more concise, generic expres- sion."
15

---

should Published as a conference paper at COLM 2024
"When you encounter a plural or past tense form, convert to singular or present tense. For example, [Windows users] should be converted to [Windows user]."
Rule 7:
Rule "When you encounter an MD5, registry, path, or other identifier that contains prefixes, remove them. For example, [md5 be simplified to
User: "Here is my entity extraction result: finput text]. Now, you apply the rules I told you before. First write down your though, think it step by step. In the end, you musttell me the final new entity extraction result."
B.4 Prompt for Refiner Agent
"You play the role an entity extraction expert and modify /simplify/////// /split the text (extracted multiple triples) in the entity extraction result I gave you. according to the following rules. A triple is basic data structure used to represent knowledge graphs, which are structured semantic knowledge bases that describe concepts and their relationships the physical world. Here are all rules you must follow."
User:
"If the subject object in triple contains pronouns such as it, they, malware, trojan, attacker, ransomware, or group, replace them with a specific name as much as possible according to the context."
Rule 1:
Rule 2: "Focus on malware, Trojan, CVE, or hacking organization or as the subject of the triples, if subject with malware or Trojan or CVE or hacking organization is found and has additional suffixes, remove the suffixes." "
"Split a complex triple into multiple simpler forms. For example, [Formbook and XLoader, are, malware] should be split into [Formbook, is, malware] and [XLonder,is, malware]."
Rule 3:
Rule 4: "If the [subject, relation] in triple can be formed into a new [subject, relation, object] triple, create a new triple while keeping the original one."
Rule 5: "If the object can be simplified to more concise, generic expression, create new triple while keeping the original one. For example, [Formbook, save, XLoader in desktop/ must has a new triple [Formbook, save, XLoader] due to the object XLoader in desktop can be simplified to XLoader."
Rule 6: "Simplify the subject, object, and relation into a more concise, generic expres- sion."
Rule 7: "When you encounter a plural or past tense form, convert it to singular or present tense. For example, [Windows users] should be converted to [Windows user]."
Rule "When you encounter an MD5, registry, path, or other identifier that contains prefixes, remove them. For example, [md5 be simplified to
should
User: "Here is my entity extraction result: [imput text]. Now, you apply the rules I told you before. First write down your though, think it step by step. In the end, you musttell me the final new entity extraction result."
B.5 Prompt for Merger Agent
User: "You are a triples integration assistant. Triple is a basic data structure, which describes concepts and their relationships. But you can only see part of the article time. In order to record all the triples from article, you have the following long-term memory area record the triples from the entire article. long-term memory stores information on the article parts you have already read.
-The start of the long-term memory area-
16

---

Published as a conference paper at COLM 2024
Triples will added here-
The end of the short-term memory area-
Second, you now see part of this article. Based on this part, you already extract such triples and place them in your short-term memory:
-The start of the short-term memory area-
-Triples will be added here-
-Th end of the short-term memory area-
Third, now review your long-term memory and short-term memory. Modify the short- term memory into new short-term memory. You should follow following rules to modify triples in short-term memory to make them consistent with triples in long- term memory. You should write down how you the rule to modify the triples in short-term memory. In additional, you find any triples in long-term memory also need to modify based the rule, you should also write down how you the rule to modify the triple in long-term memory, and then add new modified triples in short-term memory
triple.
"You notice that in these triples, some triples have subjects and objects that contain partially identical terms and refer to the same specific nouns, but these specific nouns have prefixes/ suffixes modifiers that make them not identical. You should delete the prefixes Seffixes/suffixess suffixes /oodifiers and unify them into the same specific nouns." "
Rule 1:
Before rule: [the Formbook, is designed to run as, a deleter] [Formbook sample, is designed to run as, one-time encryptor]
After rule: [Formbook, designed run as, deleter] [Formbook, designed to run as, one-time encryptor]
The words the Formbook and Formbook sample refer the same entity, so they are unified to use the exact same subject Formbook for consistency.
Explanation:
"Be especially careful that when you meet specific names of malware, CVE, Trojans, hacker organizations, etc., always use their specific names and remove the prefixes moodifeers."
Rule 2:
Before rule: [Malware Formbook, is, malware]
After rule: [Formbook, is, malware]
The word Formbook is specific name of malware, so it should be used as the subject of the triple and the prefix Malware should be removed.
Explanation:
Rule 3: "Don't I add non existing triples to your new short-term memory."
Suppose you find in long-term memory: [the malware, downlond, Leafminer] and in short- term memory: [Formbook, is, malware]. You cannot add new triple in new short term memory: [Formbook, download, Leafmniner. Because you don't have evidence that the malware in the long-term memory specifically refers to Formbook.
"Don't add non existing triples that don't exist in long-term memory or short- term memory to your new short-term memory. You should add triples from long- term memory short-term memory to your new short-term memory, not from your imagination and self creation"
Rule 4:
Rule 5: "Don't add any example word like Formbook, XLoader, Leafminer, FinSpy, Kismet in your new short-term memory area, they are just example words not the real triples in the long term memory area or short term memory area."
Rule 6: "new short-term memory area must be started with The start of the new short-term memory area and ended with The end of the new short-term memory area."
17

---

Published as a conference paper at COLM 2024
Preset assistant responses: "Yes, understand and totally follow the rules." "
User:
"-The start of the long-term memory area-
[HackerAAA, disrupts, network operations], [HackGroup CCC, targets, financial in- stitutionsl, |Malware BBB, steals, personal data [CyberAttack 2042, compromised, database systems], |A sample of BBB, save, a payload]
-The end of the long-term memory area-
-he start of the short-term memory area-
(The HackerAAA Trojan, created by, sophisticated hacker group), [CCC hackers, launched, series of phishing campaigns], [BBB malware program, encrypts, files for ransom], |Attack 2042, led to, sensitive leaks),/BBB, run, payload]
-The end of the short-term memory area-"
Preset assistant responses: "Modifications required based on the rules: Apply Rule and Rule 2: Unify subjects and objects with partially identical terms and remove any prefixes /stffiusff/////////e modifiers, particularly for specific names like malware and hacker groups. Modified short-term memory:
[The HackerAAA Trojan, created by, a sophisticated hacker group] - [HackerAAA, created by, sophisticated hacker group] ICCC hackers, launched, series of phishing campaigns] [HackGroup CCC, launched, phishing campaigns] [BBB malware program, encrypts, files for ransom] - [Malware BBB, encrypts, files for ransom] [Attack in 2042, led to, sensitive leaks] + +[yyberAttack 2042, led to, sensitive data leaks]
I also find a triple in long-term memory that needs to be modified based on Rule and Rule 2:
[A sample of BBB, save, a payload] + (BBB, save, payload]
Apply Rule and 4 and 5: We must ensure we do not create non-existing triples when merging the memories. After applying the modifications, we now create new short-term memory area:
-The start of the new short-term memory area-
[HackerAAA: created by, sophisticated hacker groupl, [HackGroup CCC, launched, phishing campaigns], [BBB, encrypts, files for ransom), [CyberAttack 2042, led to, sensitive data leaks], [BBB, run, payload], [BBB, save, payload]
-The end of the new short-term memory area-"
User: "Good. Now, lets switch to another article.
-The start of the long-term memory area-
[Input Text]
-The end of the long-term memory area-
-The start of the short-term memory area- [Imput Text]
-The end of the short-term memory area-
Now, follow the rules. Write down how you the rule to modify the triples in short-term memory. Then, write down new short-term memory which must be started with -The start of the new short-term memory area- and ended with -The end of the new short-term memory area-"
18

---

Published as a conference paper at COLM 2024
B.6 Prompt for Checker Agent
"You are responsible for check the result from another Al agent. You job is to check whether the result contains following words.[Input Textj. The text may in different order different form obfuscated. Now,here is that result [Input TextJ. You answer should just be 'No error' or 'I found error" "
User:
C Impacts of Article Type on Knowledge Graph Construction
We evaluate the performance of CTIKG on different article types. In general, the standalone type the simplest, where the author describes only one security threat, such as malware or CVE, in an article. As a result CTIKG maintains high recall. The reason for the lower recall is that some authors tend to mention the specific name of the threat only at the beginning of the article, and use the indicative pronouns to refer to it in all other parts of the article. For example, the article "Unix-privese penrxx-priesc-check Unix/Linux User Privilege Escalation Scanner" Darknet (2015) mentions the name Unix-privesc-cher article. For CTIKG, even if the specific name of is not in the short-term memory, it still obtains its name and completes the knowledge graph by retrieving the name from the long-term memory. As for Extractor, even though it also performs coreference resolution, quite number of edges in the knowledge graph still link to "it" node.
only twice in the entire
The overview article tends describe different cyber threats in different parts. For example, the article "A Brief History of Malware: Part Three (1993-1999)* Crawley (2016) describes different malware in different paragraphs. For this type of article, CTIKG completes the construction of the whole article's 's knowledge graph through the direct stacking of the short-term memories from article segments.
For both CTIKG and Extractor, the chain type is definitely the most challenging to pro- cess. This type of articles often describe how malicious actor or team utilizes multiple tools to reach an intrusion and complete the final task, often stealing data or ransomware. For example, the article "IR Team Investigations Uncover eCrime Use of Nation-State Attack Methods" Etheridge (2018) describes how an attacker can start an intrusion at the beginning and maintain the persistence mechanism eventually install ransomware on the target computer by using multiple attack methods. For each attack method, the article provides some information that can be extracted as triples. When the long-term memory contains large number of information about different cyber threats, the merger agent of CTIKG is more likely produce inaccurate results, making the final result inferior to the result of the other two types.
D LLM Selection
We prefer open source LLMs that can be deployed on our own server. We finally chose the YI 34B model (01-ai, 2023) that achieved top scores on the Alpaca Eval Leaderboard (Li et al.) and the MMLU benchmark (Hendrycks et al., 2020). We also attempted to use LLaMa 2 (Touvron et al., 2023). However, in early experiments, even the largest LLaMa 2 70B model cannot strictly follow CTIKG's prompt instructions, and result in an accuracy of merely 30%. This phenomenon is consistent with its lower score on the MMLU benchmark, and thus CTIKG uses only the YI 34B model for CTIKG.
E Setup of Evaluation Dataset
We collect 88, 131 articles from 67 websites and 672 articles from the ATT&CK knowledge base (Corporation, 2022). Since some articles contain very short descriptions, such as just listing URLs or file addresses, we further apply NLP tools to filter out them and obtain 72,538 meaningful articles. These articles have average 52 sentences, where each sentence has an average of 15 words or 100 characters (Figure 5). With CTIKG capacity to process sentences at a time, most articles require processing in 10 segments.
19

---

Published as a conference paper at COLM 2024
Numbar Articles Number Sentences
3050
1100
X010
1.10
11.00
71.30
10-40
W
7181
60.00
184
oesentancss
Figure 5: Distribution of articles by sentence length
F Statistics of RQ1 Effectiveness Comparison
E1
The Impact of Text Complexities
Figure 6 and Figure 7 show the accuracy of models at different text complexities.
Precision with Various Triple Counts
Recall with Va rooious Triple Counts
0.95
0.95
0.90
0.90
0.85
Recall
0.80
0.75
0.75
Y1
0.05
GP13
0.70
GPT3
Number Trpples Ground Truth
Numbow Ground Trulh Triples Senterne
Figure 6: Evaluation based on varying sizes of triples in one sentence
Precision on Sentences with Various Word Counts
Recall on Sentences with Various Word Counts
0.30
0.90
0.85
0.85
Frectsion
0.80
0.75
0.75
GPT4
GPT4
Extractor
Extractar
1:15 16:20 21:25 26.30
31:35 26-40
40+
2:15
16:20 21-25 26:30 31:35 36-40 40+ Number of Words Sentence
Number of Words Sentence
Figure 7: Evaluation based on varying number of words in one sentence
20

---

Published as a conference paper at COLM 2024
E2
Analysis of Error Triples
Distribution of Error Triples
100
99
LLM Model
95
Yi
GPT4
GPT3
80
Tyo
72
Frequency
60
33
33
20 88
15
16
Triple with incorrect Relationship
B. Triple with Unrelated Subject and Object
Triple with incorrec Re Relationship Merging with Object
Triple with incorrect Number Elements
Triple with Hollucinated Subject
F. Triple with Hallucinated Object
G. Triple with Hollucinated Relationship
H. Triple with Complete Hallucination
Error Type
Figure 8: Distribution of Error Triples
G
Statistics of RQ3 Effectiveness and Case Study
G.1 Characteristics of Knowledge Graph
Figure 9 and Figure 10 show the accuracy of the models for different text complexity. In the graph, there are 6301 nodes related to cyber security topics, categorized under "Attack Vector And Techniques", "Malware Name" and "CVE". Additionally, there are 4361 nodes related to other computer science topics, which include "Software/Tooe TTools,, "File Type", "Domain Names", "Hashes", "IP Addresses" and "Email Addresses". The remaining 16365 nodes are classified under "Other", " "Organization Name" and "Geographic Region". Overall, 39.45% of the nodes are related to computer science topics.. As for the edges, the graph includes 562 edges related to cyber security topics, 1355 edges related to other computer science topics, and total of 1774 general edges. Overall, 51.94% of the edges pertain to computer science topics.
21

---

Published as a conference paper at COLM 2024
Distribution of Categories
14000
12000
10000
Count
8030
3965
2413
2000
1774
1355
8/0
698
S67
229
67
Attack Vector And ootware rechiigues aftarrr Gdls
Other
Matware Nome
Organization Name File Type
Geographical Region
Domain Names
CVE
Hashes
Addresses
Email Addresses
Category
Figure 9: Distribution of Nodes Categories
22

---

Published as a conference paper at COLM 2024
Distribution of Categories
1774
1750
1500
1355
1250
Count
1000
750
562
500
250
General Entity Relationships
Computer Science Science-Related Related but Non- Security Relationships
Cybersecurity Related Relationships
Non-Security
Category
Figure 10: Distribution of Edges Categories
G.2 Comparison of Knowledge Graph Effectiveness
Table shows the effectiveness of cyber threat revelation specifically for each CVE.
Table 3: Effectiveness of CTIKG
Name
Fol Behaviors
Max of Behaviors per Article Added Behavion
Fol Articles
Avg Added Behavions per Article
-VE-321--228
67
56
21
2,33
CVE-2018-174
CVE-2012-0158
2.11
LVV-2011-19781
60
51
18
CVE-2017-11881
47
35
1.67
CVF:2017-0199
LVE.
41
25
1.79
CVVE2012-1507
33
28
2,15
CVE-21221 34527
CVE-2021-2698 -66555
LV1-2011-2558
25
16
2.2
LCVE-010-1188
22
1.1
-VVE-2021-26855
CVE-2016-0189
13
1.3
CVE-2015-1878
17
13
1.66
Column "# of Behaviors" shows the total number of behaviors of with a specific CVE, as identified from multiple articles. Column "Max # of Behaviors per Article" shows the highest number of behaviors identified a single article. Column " "Added Behaviors" shows the number of additional behaviors identified from other articles. Column "# of
23

---

Published as a conference paper at COLM 2024
Articles" shows the total number of articles that describe the behaviors of the specific CVE. Column "Avg Added Behaviors per Article" shows the average number of additional behaviors contributed by each article.
G.3
Case Study
We identified two cases that illustrate how the knowledge graph reveals cyber threat intelligence. Figure 11 and Figure 12 show the correlations between the original article text and the corresponding knowledge graphs.
Reveal Hidden Information From Complex Text: CTIKG can reveal information that does not exist in the NVD database by analyzing the CTI article. For example, the knowledge graph shows that CVE-2021 1012201127700 --20665 is similar to CVE-2021-26856 and can overwrite any system file that isn't listed in NVD or other database.
CVE-2021
-27065
Zero day vulnerabilities in Microsoft Exchange Server
similar to
allows
"CVE-2021-27065 similar to CVE-2021 222222258288888 and allows an authorized attacker to overwrite any system file on the Exchange server..."
CVE-2021 -26858
attacker
overwrite
on
Exchange server
any system file
Figure 11: Cyber threat intelligence of CVE-2021-27065
Reveal Hidden Information From Multiple Articles: CTIKG can reveal the path from one entity to another entity based different articles. For example, the graph shows that both CVE-2012-0158 and CVE-2017-1188) are related to Microsoft Office, which informs the researcher who encounters the CVE-2017-1 1--017-11882 to check if the system is infected by CVE-2012-0158, since they both utilize Office.
Modifled Enfal Varlants
CVT-2012
Compromised Systems
-0158
"The attachment is the malicions
document Special General
Meeting.dos that exploits Microsoll Office vulnerability (CVE-2012-0158 )..""
for
The Independent" Hacked,
Leads TestaCrypt Ransomware
Microsoft Office
"Source. msre, www.m/osso..oooooppaaaaaaaaoaaaa
com/update- guide/en- quidd/cn-US//vllerrabb US/vuleerabbbiity
Memory
Cormuttoon
CVE-2017-1 71-120 Microsofi
Vuluerability
Office Memory Corruption
Vulnerability"
VE-201 R -11882
Figure 12: Cyber threat intelligence of CVE-2012-0158 and CVE-2017-11882
24


Reviews:
[{"review_id": "gnjAxXNTso", "reviewer": "Reviewer_hhCn", "summary": "The paper presents an approach using LLMs to automate the process of constructing a knowledge graph of cyber-security-related entities from text documents related to cyber threat intelligence. The approach is evaluated on a new, manually curated dataset of articles extracted from security-related websites and knowledge bases.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "## **Novelty**\nAs described in the introduction, directly using LLM in knowledge graph construction faces four challenges due to randomness and hallucination: _token limitation_, _incorrect triple format_, _incorrect output_, and _misunderstood task_. To address these challenges, we propose a novel design equipped with dual-memory design, multiple LLM agents, and retry mechanism. Dual-memory design overcomes the challenge of _token limitation_, multiple agents mitigate the challenges of _incorrect format_ and _incorrect output_ due to randomness, and retry mechanism improves the robustness by addressing the challenge of _misunderstood tasks_ caused by hallucination. CodeKGC, a knowledge graph construction approach also based on LLM, can only extract pre-defined entity relationships and handle each sentence independently without considering the context of other sentences in the article, while CTIKG can automatically uncover different types of security-oriented entity relationships and process the whole article with the help of dual memory design.\n\n## **Entity Types and Relationships**\nAs described in the introduction, unlike the otology-type knowledge graph that can only extract pre-defined relationships and entity types, our goal is to dynamically capture all types of relationships related to any cyber threats in CTI articles. We will add the statistics of unique entity types and relationships in the Appendix of the camera-ready version.\n\n## **Dataset**\nThe data was labeled by at least two authors. As stated in the introduction, we anonymously published the ground truth and the results of triple extraction and knowledge graph construction at https://github.com/ctikgresearch/GTIKGResearch.\n\n## **LoRA Support**\nRegarding the switching of lora during processing, Vllm needs 1-2 minutes to load the model, which makes switching LoRA very time consuming. In addition, to maximize GPU utilization, we used multiple threads to process different articles at the same time, so they call different agents at the same time. In our recent development, we updated the model to the SOTA model LLaMA3, and we have successfully enabled CTIKG to support LoRA training, LoRA switching, and merging the trained LoRA back into the base model as a new model. The new code, LoRA file, and the new model will be updated in the Appendix of the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- the paper is clear and well-written\n- the proposed approach seems sound and is evaluated on three distinct benchmarks (including an error analysis)\n- the dataset employed for the evaluation is a nice contribution to the field (although it's unclear whether it will be released publicly)", "reasons_to_reject": "- the scientific novelty of the approach is limited to the use of existing techniques for the use of LLMs in information extraction.\n- as the paper focuses on the construction of knowledge graphs for the field of cyber-threat intelligence, it is unfortunate that the authors never actually describe in detail the types of entities and relations that they seek to extract.This should, in my view, feature much more prominently in the paper. \n- the paper should include more information about the dataset of CTI articles employed for the evaluation. How was the annotation performed? If several annotators were involved, did you compute their inter-annotator agreement?", "questions_to_authors": "I'm not sure I understood correctly the reason for not testing parameter-efficient fine-tuning methods such as LoRA. On page 9, the authors mention the inability of loading multiple LoRA in VLLM. But why would you need to load multiple LoRA simultaneously? Couldn't you (at least for the purpose of your experiments) perform each task one at a time, each time storing intermediate results ?", "extra_scores": {}}, {"review_id": "MxouACBRcl", "reviewer": "Reviewer_APiJ", "summary": "The paper introduces a novel approach, CTIKG, for constructing a security-oriented knowledge graph from Cyber Threat Intelligence (CTI) articles. The authors address the limitations of existing work and propose the use of LLMs to overcome these challenges. CTIKG employs prompt engineering and multiple LLM agents with a dual memory design to efficiently extract triples from the articles and generate results. The effectiveness of CTIKG is evaluated on three benchmarks constructed from real-world CTI articles, demonstrating significant improvements over state-of-the-art techniques.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the novelty and effectiveness of CTIKG. \n\n## **Scalability**\nIn our evaluations, we demonstrate the effectiveness on the benchmark with an average length of 5200 characters. In particular, due to the dual-memory design, CTIKG can handle large documents by splitting them into segments as described in the Approach section and will not be limited by the document length. \n\n## **Performance**\nWe used YI-32B, the SOTA open source model before the submission, to process an article in 5 minutes on a server with dual RTX A6000 (equivalent to dual RTX 3090). Now the newly released model, LLaMA3-8B, has shown a better performance with a smaller model size according to LMSYS Chatbot Arena Leaderboard. Since CTIKG can seamlessly switch to any LLM model as the backend, in our new experiments, the new CTIKG with LLaMA3-8B only needs __50 seconds__ to process __a full article__ on a server with dual RTX 6000 ada (equivalent to dual RTX 4090). In the long run, the CTIKG with newer LLMs will achieve better processing performance with more advanced GPUs.\n\n## **Other Domains**\nWhile it is not within our focused scope in the submitted version, in future work, we plan to verify the effectiveness of CTIKG with adaptation to other domain knowledge in other domains like SciERC.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Novel approach. The paper addresses an important problem in cybersecurity by proposing a novel approach that leverages LLMs for knowledge graph construction from CTI articles.\n2. Clear problem statement. The authors clearly identify the limitations of existing work and motivate the need for a specialized knowledge graph that captures relationships among security-related entities.\n3. Robust evaluation. The paper provides comprehensive evaluations on multiple benchmarks, showcasing the effectiveness of CTIKG in terms of precision and recall compared to other approaches.", "reasons_to_reject": "Limited discussion on generalizability. The authors focus solely on cybersecurity-related text analysis using LLMs, and it would be beneficial to discuss potential applications or extensions of their approach beyond this specific domain.", "questions_to_authors": "How scalable is your approach when working with larger-scale datasets or real-time applications? Did you encounter any performance issues during evaluation?", "extra_scores": {}}, {"review_id": "0MhmmEhsps", "reviewer": "Reviewer_dKSc", "summary": "In this paper, the authors proposed a system that utilizes multiple LLM agents and a dual memory design to efficiently construct a knowledge graph from CTI articles. This advanced system breaks down lengthy CTI articles into smaller text segments, which are then processed individually by various LLM agents operating at different temperature settings. This approach serves to minimize the inherent randomness of LLMs. Following this segmented processing, the CTIKG summarizes the outcomes to produce more precise results. Evaluations conducted on three benchmarks, derived from actual CTI articles, have shown that CTIKG outperforms other cutting-edge methods.\n\nStrengths:\n\n1.The proposed multi-agent collaborative method for constructing knowledge graphs, as outlined in the paper, possesses practical value, particularly in the realm of knowledge representation and reasoning;\n\n2.The paper is very well written with clarity, easy-to-follow logic, and adequate discussions;\n\n3.After conducting extensive computational experiments, the results clearly demonstrated the effectiveness of the method.\n\nWeaknesses:\n\n1.The depth and breadth of the research conducted on knowledge graph construction(section 6) is disappointingly insufficient, lacking substantial content and comprehensive analysis;\n2.In the process of collaboratively constructing the knowledge graph, for intelligent agents playing different roles, what are their contribution rates? Analysis results of this part of the content also need to be given.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the novelty and the effectiveness of CTIKG. \n\n## **More Comprehensive Analysis**\nDue to page limit, we could not put in all the detailed analysis of knowledge graph construction in the submitted version, and will provide that in the Appendix of the camera-ready version. We have further analyzed CodeKGC, which is also an LLM-based approach for knowledge graphs. It can only extract pre-defined entity relationships and process a single sentence without considering the context of the whole article, while CTIKG can automatically uncover different types of security-oriented entity relationships and process the whole article with the help of dual memory design.\n\n## **Contribution of Agents**\nAs stated in the Approach section, each type of agents have their unique and indispensable task:\n* Multiple worker agents are used to extract diverse entity relationships with different temperature settings, mitigating the effect of randomness.Without them, CTIKG extracts fewer correct entity relationships due to randomness, resulting in a lower recall.\n* The integrator agent's role is to summarize the valid triples with identical meanings and output a representative triple. CTIKG without the integrator would produce redundant triples.\n* The refiner agent performs the normalization, resolution, and splitting of the composite triple with multiple entity relationships. The refiner agent assists the merger agent in unifying identical entities across multiple segments or articles. If the merger agent performs these tasks of the refiner during long-term memory construction, it would be constrained by token limitations and unable to produce complete results. \n* The checker agent detects common error patterns due to hallucination and activates the retry mechanism. As stated in Section 4.2,  even using the GPT4 as the backend, 6.95% of the results were affected by hallucination. For the GPT3.5 model, the rate of hallucination increases to 13.42%. \n*  The merger agent is responsible for merging the triples from the short-term memory to form the long-term memory and performs coreference resolution to merge the triples with identical entities, which is an indispensable function for short-term memory construction and long-term memory construction.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.The proposed multi-agent collaborative method for constructing knowledge graphs, as outlined in the paper, possesses practical value, particularly in the realm of knowledge representation and reasoning;\n\n2.The paper is very well written with clarity, easy-to-follow logic, and adequate discussions;\n\n3.After conducting extensive computational experiments, the results clearly demonstrated the effectiveness of the method.", "reasons_to_reject": "1.The depth and breadth of the research conducted on knowledge graph construction (section 6) is disappointingly insufficient, lacking substantial content and comprehensive analysis;\n2.In the process of collaboratively constructing the knowledge graph, for intelligent agents playing different roles, what are their contribution rates? Analysis results of this part of the content also need to be given.", "questions_to_authors": "In the process of collaboratively constructing the knowledge graph, for intelligent agents playing different roles, what are their contribution rates?", "extra_scores": {}}, {"review_id": "MpD6sybJtE", "reviewer": "Reviewer_yYHM", "summary": "This paper presents an LLM-based system to extract triples and resolve them into a knowledge graph for a specific domain, namely cyber threat intelligence. The system involves a short-term memory constructor with multiple steps to create initial triples, a long-term memory constructor to merge and modify those triples, and finally the KG constructor.   In addition to the system itself, the paper presents a gold standard set of extracted triples in order to measure performance.  Several variants of the proposed system as well as several existing systems (both general KG construction and CTI specific) are compared against existing benchmarks, with the proposed system performing much better than any of the other solutions.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the novelty and the effectiveness of CTIKG. \n\n## **Dataset and Code**\nAs stated in the introduction, they have already been published on Github, the anonymous link is https://github.com/ctikgresearch/GTIKGResearch. \n\n## **Writing Issues**\nThanks for the suggestion. We will ask native English speakers to refine it, and solve other problems you mentioned in the review.\n\n## **IOC**\nAs stated in the introduction, IOCs are forensic artifacts of an intrusion, and we will add more details in the camera-ready version.\n\n## **Error Analysis**\nAs stated in Section 4.2, for the error rate, we found that even using GPT4 as LLM backend, the CTIKG still has 6.95% of error rate that can be detected and fixed by the checker. For the GPT3.5 model, the error will increase to 13.42%. More error analysis will be provided with case study in the camera-ready version.\n\n## **Table 1 and 2**\nWe will update the number of strategy types in Table 1 and provide the significance analysis in Tables 1 and 2 in the camera-ready version.\n\n## **Node Merging**\nAs stated in Section 3.4, CTIKG lists the five nodes that are most similar to the entities based on text embedding, and uses the merger agent to determine if these nodes refer to the same entities and merge them.\n\n## **Command & Control and Impact**\nAs stated in Section 4.2, CTIKG with GPT-4 demonstrates a superior text comprehension capability among all CTIKG variants. The texts in the Command & Control and Impact tactics exhibit greater diversities in writing, and our goal is to extract all entity relationships mentioned in the text, rather than based on the limited relationships of pre-defined ontology. In this case, CTIKG with GPT-4 can better utilize its comprehension capability to extract potential entity relationships, resulting in better accuracy.\n\n## **Coreference Resolution**\nAs stated in Section 3, CTIKG’s worker, refiner and merger agents are instructed to resolve coreference (prompts provided in Appendix.B). In the long run, we can add a new agent specifically for resolving coreference by consuming the whole article and collaborating with other agents.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Highlights the complexity of using LLMs for specific tasks, especially in domain-specific applications. This approach is likely to extend to many related domains.\n- Provides a practical combination of techniques and is able to leverage LLMs in different ways. Includes use of standard NLP modules such as NER and text breaking.\n- Interesting use of short and long term memory constructors with KG.\n- Contains an error analysis for key components.", "reasons_to_reject": "- The English is off, making the paper difficult to read due to distracting ungrammaticalities and odd phrasing.  This can be fixed by having the final paper proof-read by a native speaker of English.\n- Needs to add statistical significance of results (given the major improvements, it is highly likely that these are statistically significant)", "questions_to_authors": "In order of the paper.\n\nPlease have the paper proofread by a native speaker of English. There are issues with determiner placement, verb agreement and word usage.\n\nWill the datasets and code (e.g. for evaluation) be released?  If so, state that and include under what license.\n\nSec 2: Add a sentence about how the system targets which entities and which relations. This becomes clearer later, but at this point it is confusing whether all subj-verb-obj triples are extracted and then later filtered out if not relevant.\nAlso add roughly how many unique nodes there are (e.g. 100, 1000, 100000) and how many unique relations (e.g. 10, 100, 1000).\n\nSec 3.2: What is IOC?\nChecker Agent: Can you provide some rough indicator of how often the LLMs output incorrect results?  Is this 1%, 10% or 50% of the time?  Later in the paper with the error analysis (which is very helpful), if you have a rough estimate of how often each error type occurs, that would be interesting to include.\n\nSec 3.4: How does the system decide whether the node is a new one or should be merged with an existing one?  Often thresholding on similarity is hard (although if the node names are relatively the same length, this becomes easier) with embeddings and so some additional factor is needed.\n\nsec 4.1: \"sta\" looks odd in the citation; use the name of the lab (e.g. Stanford NLP Lab or whatever the official name is)\n\nTable 1: Include statistical significance of best result against next best.\nPerhaps bold face the best results.\nIt would be great to know the number of triples of each type (row); could these be provided in parentheses after the name of the tactic?\nWhy does the GPT4 variant do better than CtiKG for Command & Control and Impact? Is there something different about these two compared to all the others?\n\nSec 4.3: In the long run, systems will have to overcome the coreference resolution issue if they are to work over textual documents. Any suggestions for future work for how to deal with this?\n\nTable 2: Add statistical significance of best system over next best\n\nSec 6: Just merge this into the intro.  It will be more helpful there and this will save space for other analysis.", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper is a solid investigation of the application of LLMs to a realistic information extraction setup, dealing with issues such as hallucinations and context limitations. The evaluation is thorough. The reviewers agreed on these points and the authors addressed their questions in their rebuttals adequately.
Decision: Accept