AttackQA: Development and Adoption of a Dataset for Assisting Cybersecurity Operations using Fine-tuned and Open-Source LLMs

## Abstract

Abstract Retrieval-augmented generation (RAG) on specialized domain datasets has shown improved performance when large language models (LLMs) are fine-tuned for generating responses to user queries. In this study, we develop a cybersecurity question-answering (Q&A) dataset, called AttackQA, and employ it to build a RAG-based Q&A system designed for analysts in security operations centers. The dataset comprises 25,335 Q&A pairs, accompanied by rationales to facilitate fine-tuning and evaluation. 80% of the dataset was generated with help of a lightweight open-source LLM (LLama 3 8B), which produced over 1100 tokens per second with full 16-bit precision on SambaNova System’s SN40L specialized hardware. To ensure dataset quality, we fine-tuned LLama 3 70B to detect and reject low-quality Q&A pairs. In using the dataset for RAG, we demonstrate that fine-tuning open-source embeddings and LLMs can yield superior accuracy compared to OpenAI’s state-of-the-art proprietary embedding and LLM (GPT-4o). Furthermore, we use Llama 3.1 405B as a judge to evaluate answer correctness, enabling the creation of a fully open-source, high-speed RAG and evaluation pipeline with a benchmark for model accuracy.

## 1 Introduction

Security operations centers (SOCs) house information security teams who are responsible for detecting, investigating, and responding to cybersecurity incidents using a variety of tools, technologies, and processes. As of 2024, firms with at least $1 billion in revenue spend $14.6 million on SOCs each year (Sadovi, ) and 80% of SOC budgets are spent on labor (). The cost of training a team of 10 SOC analysts to master 7 security tools is estimated at $3.69 million (Cohen, ). According to the Hershberger () survey, the top challenges facing SOCs include a lack of expertise in security, too much time spent in investigating alerts, and a slow response time to advanced threats. To address those challenges and to enable quicker attack prevention and recovery, we propose a question-answering (Q&A) system leveraging artificial intelligence to help SOC analysts get quick answers to time-sensitive questions about cyberattacks. Our solution leverages entirely open-source large language models (LLMs) that are becoming increasingly powerful and, on domain-specific datasets, can be tuned to exceed the performance of proprietary LLMs that are many times as large.

We used the MITRE ATT&CK® (The MITRE Corporation, ) knowledge base of cyberattack techniques, tools, campaigns, detection approaches, and mitigation approaches to generate a Q&A dataset called AttackQA for use in Q&A systems or general-purpose chatbots. That knowledge base, grounded in real-world observations and updated biannually, was chosen because the ATT&CK® framework is widely adopted for cyber threat intelligence across the private sector, government, and the broader cybersecurity product and service community (Roy et al., ; AL-SADA et al., ; Al-Shaer et al., ). It is stored in an esoteric database format called Structured Threat Information Expression (STIX), making it ill-suited for direct use in Q&A systems. Hence, we extracted the data and processed it in a way that makes it easier for training and inferencing with LLMs.

Figure: Figure 1: Illustration of dataset generation, quality control, and adoption in RAG
Refer to caption: extracted/5972253/fig/approach.png

The structure of this paper and our approach are outlined in Fig. 1. The first phase involves the creation of the AttackQA dataset. Initially, we generated 28,686 Q&A pairs derived from the MITRE knowledge base. Subsequently, we fine-tuned Llama 3 70B to perform quality control (QC) on those Q&A pairs, retaining 25,335 high-quality examples. In the second phase, AttackQA was used to fine-tune both Microsoft’s E5 Large V2 embedding (Wang et al., ) and Meta’s Llama 3 8B LLM AI@Meta () for retrieval-augmented generation (RAG). The accuracy of the results was assessed using Llama 3 405B, leveraging a G-Eval (Confident AI, ) correctness metric within the DeepEval Framework.

In summary, our contributions are as follows:

- •
We demonstrate the use of a compact, open-source LLM (Llama 3 8B Instruct) to generate a high-quality question-answer dataset from the MITRE ATT&CK® knowledge base.
- •
We perform an evaluation that shows that a fine-tuned Llama 3 70B model is better than OpenAI’s GPT-4o at identifying questions and answers that are of low quality, so they can be removed from AttackQA as part of an automated dataset quality control process.
- •
We demonstrate that fine-tuning an embedding model significantly enhances context recall in retrieval tasks, outperforming OpenAI’s state-of-the-art (SOTA) embedding model, Text-Embedding-3-Large.
- •
We utilize Llama 3 405B as a judge to evaluate answer correctness. Using its evaluation scores, we found that fine-tuning Llama 3 8B as a generation model in RAG improves correctness, surpassing the performance of OpenAI’s GPT-4o, which is many times as large.
- •
We developed an accurate and low-latency end-to-end RAG pipeline, utilizing fine-tuned open-source embeddings and LLMs to serve as a Q&A system to support security analysts.

By employing Llama 3 8B at speeds exceeding 1100 tokens/s (at full 16-bit precision), Llama 3 70B at over 550 tokens/s, and Llama 3 405B at 132 tokens/s, we were able to develop a highly responsive end-to-end solution for SOCs. Those model throughputs were achieved using the free SambaNova Cloud platform (SambaNova Systems, ) on specialized hardware (Prabhakar et al., ).

## 2 Related Work

The use of LLMs for synthetic dataset generation, curation, and evaluation has been surveyed by Long et al. (). Although AttackQA is synthetically generated, it is grounded in the widely reputed MITRE ATT&CK® knowledge base.

The work of Hsieh et al. () demonstrates that fine-tuning a 770M T5 model, using extracted rationales during the fine-tuning process, can outperform a few-shot prompted 540B PaLM model. Similarly, Zhang et al. () fine-tuned the generation model within a RAG pipeline, enabling it to predict rationales alongside answers. Moreover, they fine-tuned the model using context that included both relevant and irrelevant (distractor) documents to improve its ability to answer questions. We employ the same set up in our work and show that greater accuracy improvements can be obtained by fine-tuning the embedding in addition to the LLM.

Yu et al. () fine-tuned large language models to generate answers while simultaneously ranking context based on relevance. Izacard et al. () introduced the pretrained LLM Atlas, specifically designed for retrieval-augmented generation, which achieved a 3% performance improvement over a 540B parameter model, despite having 50 times fewer parameters. Our fine-tuned embeddings perform well on ranking without requiring any unique training approaches and our generation models produce a 9% improvement on proprietary SOTA models that are much larger.

The fine-tuning of embeddings has been previously shown to enhance performance on tasks involving domain-specific datasets (Fabian et al., ). Synthetic dataset generation for the explicit purpose of fine-tuning such embeddings has been explored by Wang et al. ().

## 3 Dataset Creation for Q&A

In this section, we describe our methodology for creating a Q&A dataset using the MITRE ATT&CK knowledge base.

### 3.1 Summary of the source data

**Table 1: Types of Entries in the MITRE ATT&CK knowledge base**
| Dataset | Description | # of Entries |
| --- | --- | --- |
| Techniques | Attack techniques | 637 |
| Tactics | Attack tactics. One attack technique may employ multiple tactics | 14 |
| Software | Software tools used by cyber attackers when executing techniques | 677 |
| Groups | Names of well-known attacker groups responsible for well-known attack campaigns | 148 |
| Campaigns | Attack campaigns that may leverage one or more techniques | 28 |
| Mitigations | Mitigation strategies used for the attacks | 43 |
| Relationships | Relationships between techniques, software, groups, campaigns, detection and mitigation approaches | 17324 |

The MITRE ATT&CK knowledge base encompasses seven categories of information, which are detailed in Table 1 along with their corresponding cardinalities.

The data for techniques, tactics, software, groups, campaigns, and mitigation approaches include a unique ID, name, description, and URL (an example is provided in Appendix A.1). From that data, we extracted the descriptions as text documents for use in Q&A tasks. The relationships table maps a *source type* to a *target type* via a *mapping type*. Source types include ‘software’, ‘group’, ‘data component’, ‘mitigation strategy’, and ‘campaign’, while target types consist of ‘technique’, ‘software’, and ‘group’. The mapping types include ‘uses’, ‘detects’, ‘mitigates‘, and ‘attributed-to’. A mapping description was also provided and included in our set of Q&A documents.

AttackQA was partially generated using manual heuristics, with the remainder produced by LLMs. Each Q&A pair was derived from a single document, eliminating the need for multi-hop reasoning because comprehensive answers did not require information from multiple documents.

### 3.2 Document preprocessing

Newline characters were removed from within individual documents, ensuring that they appeared only between documents in the final retrieval context presented to the generation model. In all the documents, hyperlinks and special tags were replaced with plain text to ensure that neither the embeddings nor the generation models needed to process special tags that would not be encountered in questions and not be expected in answers.

From each document, one to three triplets of $\{question,thought,answer\}$ were generated, where $thought$ represents the rationale necessary to accurately answer the question. Additional metadata was included in the dataset to enable hybrid retrieval approaches that use a combination of vector search, keyword search, relational querying, etc.

### 3.3 Manual Q&A Generation

Twenty percent of the Q&A pairs were generated by humans using heuristics embedded in code, relying solely on the relationships table for their creation.

The human-generated questions resemble “What campaigns used attack technique ’T1562.001: Disable or Modify Tools’?” The corresponding answers resemble “The campaigns that used attack technique ’T1562.001: Disable or Modify Tools’ were: ’C0002: Night Dragon’, ’C0024: SolarWinds Compromise’, ’C0028: 2015 Ukraine Electric Power Attack’, ’C0029: Cutting Edge’”. Because that answer was not available in any single document in the source dataset, we synthetically generated a document to match the answer. That ensured that the full list of relationships for a given source type, target type, and mapping type were available in a single document for ease of retrieval. To generate the document, it was sufficient to query the relationships table, filtering on the relevant entities (e.g., campaigns, software, techniques, etc.). The questions were generated to ensure comprehensive coverage of source types, target types, and mapping types. Notably, no list of relationships was long enough to cause the answer to exceed 1000 tokens in length.

**Table 2: AttackQA entry with human-generated question and answer**
| question | What campaigns used attack technique ’T1562.001:<br>Disable or Modify Tools’? |
| --- | --- |
| thought | To answer the question, I need to know what campaigns<br>used attack technique ’T1562.001: Disable or Modify Tools’ |
| answer | The campaigns that used attack technique<br>’T1562.001: Disable or Modify Tools’ were:<br>’C0002: Night Dragon’, ’C0024: SolarWinds Compromise’,<br>’C0028: 2015 Ukraine Electric Power Attack’,<br>’C0029: Cutting Edge’ |
| document | The campaigns that used attack technique<br>’T1562.001: Disable or Modify Tools’ were:<br>’C0002: Night Dragon’, ’C0024: SolarWinds Compromise’,<br>’C0028: 2015 Ukraine Electric Power Attack’,<br>’C0029: Cutting Edge’ |
| subject_id | T1562.001 |
| subject_name | Impair Defenses: Disable or Modify Tools |
| subject_type | T1562 |
| url | https://attack.mitre.org/techniques/T1562/001 |
| source | relationships_campaigns_for_technique |
| references | NaN |
| human_question | True |
| human_answer | True |
| field | NaN |
| relation_id | NaN |
| relation_name | NaN |

An example entry is presented in Table 2, for which the question and answer were generated using human-defined heuristics ($human_question=True$ and $human_answer=True$). Questions similar to that example would have been difficult for models to answer if we had not constructed documents summarizing the relationships between campaigns and techniques (see the $source$ field). The reason is that, without the summary documents, one document would have needed to be retrieved for each relation and presented to the LLM in the context. If the number of relations exceeded $k$, which was 5 in our case, the LLM would not have had all the information on relations to comprehensively answer the question. An alternative to presenting summaries of relations in a single document is to use an agentic approach, wherein the model has relational capabilities to navigate joins (e.g., text-to-SQL) and it can perform retrieval using those relational queries (e.g., through the use of a tool).

### 3.4 LLM-based Automated Q&A Generation

Utilizing Llama 3 8B Instruct AI@Meta (), we generated Q&A based on the processed documents. To speed up the process, we leveraged SambaNova’s free cloud offering (SambaNova Systems, ), which runs Llama 3 8B at over 1100 tokens/s with full 16 bit precision Kerner (). Although we experimented with Llama 3 70B, the results were not significantly different, leading us to continue with Llama 3 8B. In some instances, Llama 3 70B produced overly verbose responses that were difficult to quickly comprehend.

Half of the Q&A pairs comprised human-generated questions and LLM-generated answers. The questions encompassed those that we anticipated end users would most likely ask and were structured as follows: “Describe X”, “How can X detect Y?”, “How can X mitigate Y?”, and “How does attack software X use attack technique Y?”, where X and Y were obtained from the entities listed in Table 1. Each of those questions could be answered by a specific document in AttackQA. Therefore, the generation model simply had to summarize that document in response to the question.

**Table 3: AttackQA entry with human-generated question and LLM generated answer**
| question | How does attack software ’S0467: TajMahal’<br>use attack technique ’T1123: Audio Capture’? |
| --- | --- |
| thought | To answer the question, I need to understand how<br>TajMahal, an attack software, utilizes the ’T1123: Audio Capture’<br>technique. |
| answer | TajMahal has the ability to capture VoiceIP application<br>audio on an infected host. |
| document | How attack software ’S0467: TajMahal’ uses<br>attack technique ’T1123: Audio Capture’:<br>TajMahal has the ability to capture VoiceIP application<br>audio on an infected host. |
| subject_id | T1123 |
| subject_name | Audio Capture |
| subject_type | techniques |
| url | https://attack.mitre.org/techniques/T1123 |
| source | relationships_uses_software |
| references | [{’source’: ’T1123/TajMahal: https://attack.mitre.org/techniques/T1123’,<br>’citation’: ’TajMahal has the ability to capture<br>VoiceIP application audio on an infected host.’},<br>{’source’: ’T1123/TajMahal: https://attack.mitre.org/techniques/T1123’,<br>’citation’: ’TajMahal has the ability to capture<br>VoiceIP application audio on an infected host.’}] |
| human_question | True |
| human_answer | False |
| field | NaN |
| relation_id | S0467 |
| relation_name | TajMahal |

An example of an entry, with the question generated using human-defined heuristics ($human_question=True$) and answer generated using an LLM ($human_answer=False$), is presented in Table 3. The $thought$ and $references$ were also generated by the LLM. The references include citations to the document to ensure that the LLM-generated answer is grounded in the document. The questions of this type were generated using heuristics that aimed at mimicking potential user questions, while ensuring that all entities in the knowledge base were covered. In this example, the $subject_id$ and $subject_name$ refer to a technique and the $relation_id$ and $relation_name$ refer to the related software that the question mentions. In other examples of this type, the subject and relation may refer to other entities mentioned in Table 1.

The remaining 30% of Q&A pairs were generated using Llama 3 8B Instruct, where the question, answer, and rationale were all derived from a given document. Depending on the document length, up to three sets
of $\{question,thought,answer,references\}$ were generated in a single LLM completion. The precise prompt utilized for that generation is given in Appendix A.2.1.

**Table 4: AttackQA entry with LLM-generated question and answer**
| question | How does Akira initially access victim environments? |
| --- | --- |
| thought | To answer the question, I need to understand the initial access<br>mechanism used by Akira as described in the document. |
| answer | Akira uses compromised credentials to access single-factor<br>external access mechanisms such as VPNs for initial access |
| document | Description of attack group ’G1024: Akira’:<br>Akira is a ransomware variant and ransomware deployment<br>entity active since at least March 2023.<br>Akira uses compromised credentials to access single-factor<br>external access mechanisms such as VPNs for initial access,<br>then various publicly-available tools and techniques<br>for lateral movement. Akira operations are associated with<br>”double extortion” ransomware activity, where data is<br>exfiltrated from victim environments prior to encryption,<br>with threats to publish files if a ransom is not paid.<br>Technical analysis of Akira ransomware indicates<br>multiple overlaps with and similarities to Conti malware. |
| subject_id | G1024 |
| subject_name | Akira |
| subject_type | groups |
| url | https://attack.mitre.org/groups/G1024 |
| source | groups |
| references | [{’source’: ’groups/G1024/description:<br>https://attack.mitre.org/groups/G1024’,<br>’citation’: ’Akira uses compromised credentials to<br>access single-factor external access mechanisms<br>such as VPNs for initial access’}] |
| human_question | False |
| human_answer | False |
| field | description |
| relation_id | NaN |
| relation_name | NaN |

The example entry presented in Table 4, has both question and answer generated using an LLM ($human_question=False$ and $human_answer=False$). The $thought$ and $references$ were also generated by the LLM, similar to the previous example. The $subject$ of the question is an attack group and its description $field$ was used to generated the $document$ (from which the $question$, $thought$, and $answer$ were generated).

### 3.5 Ensuring Quality of LLM-generated data

Through careful prompt engineering and post-processing of special characters (e.g., the ‘$\backslash$’ character in file paths), we achieved a 99% success rate in ensuring that Llama 3 8B produced valid JSON in the required format. That was despite the present lack of a JSON mode in the SambaNova API.

To ensure the quality of the LLM-generated questions and answers, we employed three strategies:

- 1.
We mandated that the LLM generate a citation for each question-answer-rationale entry that included verbatim text from the document supporting the answer. That requirement ensured that the entry was grounded in the source material.
- 2.
All instances of duplicated questions were removed from the dataset, allowing the remaining questions to act as a unique index. The reason is that the presence of duplicate questions implies that the same inquiry could pertain to two or more different documents, indicating that the question lacked specificity to any particular document.
- 3.
We leveraged an independent grader LLM to grade each entry in the dataset on the quality of the question and answer. We refer to that LLM as the quality control (QC) LLM and the remainder of this subsection will describe that automated curation approach.

#### 3.5.1 Criteria for assessing Q&A quality

We employed the G-Eval metric Confident AI, in the DeepEval framework to facilitate automatic curation using the quality control (QC) LLM. The G-Eval metric is a custom metric that scores each Q&A pair by assessing it in conjunction with the retrieved context.
In addition to generating a score, the prompts formulated by DeepEval require the LLM to provide a rationale for the assigned score. The scoring is based on specific evaluation steps that establish the scoring criteria. We developed distinct metrics for assessing both question quality and answer quality.

For the assessment of question quality, we evaluated whether the questions were ambiguous, failed to reference specific topics in the context, referred to topics not present in the context, or were so broad that they could be answered with information outside of the provided context.

For the assessment of answer quality, we examined whether the answers were irrelevant to the question, did not reference pertinent content from the context, lacked comprehensiveness, were vague, or included information not contained within the context.

In DeepEval, the model is prompted to generate a score ranging from 0 to 10. However, that score is subsequently divided by 10 during post-processing to yield a normalized range from 0 to 1.

#### 3.5.2 Fine-tuning the QC LLM

We manually annotated 400 Q&A pairs, assigning scores along with reasons to justify the scores. We kept 80 of those pairs as a hold-out validation set to evaluate models ensuring that the score distributions were preserved (see Fig. 2). A high score indicates that the Q&A pair is of high quality and worth retaining, whereas pairs with low scores were filtered out.

Figure: Figure 2: Distribution of scores in the manually-annotated QC dataset
Refer to caption: extracted/5972253/fig/judging_trainval.png

According to our annotations, a score of 1 denoted a perfect example, while scores of 0.8 and 0.9 were deemed acceptable, but not flawless. Any score below 0.7 indicated significant quality issues and the example was marked for removal from the dataset. We classified examples from the dataset that are deemed worth retaining (i.e., $score>0.7$) as positives, and those that are better suited for removal (i.e., $score\leq 0.7$) as negatives. Under this classification, $31\%$ of the training set and $32.5\%$ of the validation set were identified as negatives.

At the time of writing, OpenAI’s GPT-4o is recognized as a SOTA proprietary model. However, its ability to judge data quality was found to be inadequate. Despite the validation set containing 26 negatives, GPT-4o identified only 2 as negatives, while Llama 3 70B did not predict any negatives. That resulted in $(precision,recall)$ values of $(79\%,98\%)$ for GPT-4o and $(71\%,100\%)$ for Llama 3 70B. Neither model demonstrated proficiency in predicting negatives, and the high recall values were largely attributable to the trivial strategy of predicting all examples as positives.

To ensure that the LLM-based QC process would yield a filtered dataset with a high precision (in which case most predicted positives turn out to be true positives), we opted to fine tune Llama 3 70B on the annotated training set. Full parameter tuning was conducted on SambaNova’s SambaStudio enterprise platform using the AdamW (Loshchilov & Hutter, ) optimizer with a fixed learning rate of $10^{(-5)}$ and a weight decay of $0.1$. As previously mentioned, the base model achieved a perfect recall due to its failure to predict any negatives. However, as we fine-tuned the model over additional steps, we observed a decrease in recall accompanied by an increase in precision, as illustrated in Fig. 3. The final QC assessment was performed using a checkpoint of Llama 3 70B that was fine-tuned for 65 steps, resulting in a validation precision of $84.2\%$ and a recall of $89\%$. By prioritizing precision over recall, we chose a model that was better at detecting negatives (bad examples) and minimizing their presence in the final dataset. The consequent reduction in recall led to the exclusion of some positive examples (along with the negatives) from the final dataset, a trade-off that we preferred because it ensured quality over quantity.

Figure: Figure 3: Precision & recall of QC LLM on annotated validation set
Refer to caption: extracted/5972253/fig/judging_pr.png

### 3.6 Dataset Summary

After performing QC, we present AttackQA’s summary statistics in Table 5. The breakdown of the 25,335 Q&A pairs by human vs LLM generation is also specified. The token lengths of the documents were measured by the cl100k_base tokenizer with Tiktoken (Open AI, ).

**Table 5: Dataset Summary**
| Dataset Summary |  |
| --- | --- |
| Total Number of Q&A Pairs | 25,335 |
| Human Question, Human Answer | 20% |
| Human Question, LLM Answer | 50% |
| LLM Question, LLM Answer | 30% |
| Number of Unique Documents | 17,760 |
| Max Document Length | 3,103 tokens |
| Min Document Length | 15 tokens |
| Mean Document Length | 75 tokens |

In using the dataset, some of the documents may need to be chunked for use with models with small context windows. Note that only 104 out of 17,760 (0.6%) of documents have greater than $500$ tokens in length and the rest could be used directly with a model of $4096$ context length. In the analyses presented in the following sections, we did not chunk any of the documents the open-source LLMs we used had context windows of length $8192$ tokens.

## 4 Model Fine-tuning for RAG

A basic RAG framework is illustrated in Fig. 4. In this section, we used AttackQA to fine-tune the LLMs and embeddings using SambaStudio to improve answer accuracy in that framework.

Figure: Figure 4: A basic retrieval augmented generation (RAG) framework
Refer to caption: extracted/5972253/fig/rag.png

Prior to any user interaction, documents are embedded by the embedding model and stored in a vector database. When a user asks a question, the question is embedded by the same embedding model and $k$ documents relevant to the question are retrieved from the vector database. In our analysis, we retrieve the $k$ documents based on the similarity between their embeddings and the question’s embeddings. The $k$ documents are then presented to the generation model in a prompt that asks the model to answer the user’s questions using information from the documents. The answer is then returned to the user.

Note that there a more complex implementations of the framework involving multiple generation models, re-ranking models, and multiple types of data stores. Such implementations are beyond on the scope of this analysis, which seeks to measure the contributions of fine-tuning individual models on the overall accuracy.

### 4.1 Training and Evaluation Split

We split the $25,335$ Q&A pairs into a training set ($90\%$) and an evaluation set ($10\%$) using uniform random sampling. Similar to Zhang et al. (), we ensured that all documents were represented in the training set so that the trained models would be familiar with the knowledge base from which questions would be asked. However, the questions in the evaluation set were not present in the training set. That resembles a live production usage setting, in which the end user wants to ask questions of a dataset, and the chatbot is familiar with the source documents but may not have previously seen the questions.

When fine-tuning the models, we used 10% of the training set for validation to ensure that we would eventually evaluate on a checkpoint that was not over-fitting on the training set.

### 4.2 Embedding Model

We performed full parameter fine-tuning on Microsoft’s E5 Large V2 embedding model (Wang et al., ), which has 335M parameters and encodes up to 512 tokens into an embedding of length 1024. The training dataset comprised of a list of questions from the training set. For each question, a list of positive documents (containing the answer) and negative documents (not containing the answer) were provided. By construction of the dataset, only one positive document existed for each question (since the question was generated from that document). The dataset was uploaded to SambaStudio and the job was run through the user interface.

Having negative documents helps the model learn to distinguish between relevant and irrelevant documents for a given question using contrastive learning (Chopra et al., ).
The negative documents were randomly sampled from a set that excluded documents whose entities were related to the entity associated with the question. That ensured that the answers could not accidentally be obtained from the negative documents, leading to poor contrastive learning. Related entities can be identified in the MITRE dataset based on their IDs (e.g., T1562.001 and T1562.002 are related techniques and should not be included in negative documents for any question relating to T1562.xxx).

### 4.3 Generation Model

The generation model was fine-tuned on SambaStudio using the same questions that were used to train the embedding model, but the dataset preparation for training was different. For each question, a set of $k$ documents were retrieved using Microsoft’s E5 Large V2 embedding. We denote the retrieved set of documents by $\displaystyle{\bm{d}}(k)=\{d_{1},...,d_{k}\}$.

Let $d^{*}$ denote the golden document, which contained the answer to the question. For all the 22,802 Q&A pairs in the training set, we used post-processing to ensure that their corresponding $d^{*}\in{\bm{d}}$. Specifically, when ${\bm{d}}$ did not contain $d^{*}$ from the retrieval, we used code to replace $d_{k}$ with $d^{*}$. The remaining $k-1$ documents, $d_{i}\neq d^{*}$, were distractor documents that the LLM would need to learn to ignore because the model would be presented with all $k$ documents even in production. We shuffled the $k$ documents in ${\bm{d}}$ to ensure that the model did not learn to pick $d^{*}$ based on its retrieval rank in ${\bm{d}}$, and focused on the contents of the documents instead of the ordering.

Each prompt comprised of an instruction with a one-shot example, and the retrieved list of documents, ${\bm{d}}(k)$, with $k=5$. The completions included a thought, answer, and references. The thought was included to ensure that the model’s answers were well-reasoned and the references ensured that the right document in the ${\bm{d}}$ was being used in answering the questions. An example of a prompt-completion pair is given in Appendix A.2.2.

We augmented the training set with 3,323 additional examples (amounting to one-eighth of the total training set) to train the model not to hallucinate. In doing so, we re-used questions in the training set for which $d^{*}\notin{\bm{d}}$ and had the completions modified with answers that read “I am sorry, I do not have the answer to the question.” and an empty references list.

## 5 Model Evaluation

In this section we present the approach to and results of our model evaluations. All results are presented on the hold-out evaluation set comprising 2,533 examples.

### 5.1 Retrieval Model

The retrieval component of the pipeline refers to steps 2-4 in Fig 4. We evaluated that component using the context recall metric, which captures whether or not the retrieved context contains the golden document.

**Table 6: Context recall in top $k$ documents for retrieval**
| Top $k$ | Base E5 Large V2 | Tuned E5 Large V2 | OpenAI TE-3-Large |
| --- | --- | --- | --- |
| $k=10$ | 72.8% | 93.29% | 83.58% |
| $k=5$ | 69.84% | 92.18% | 80.85% |
| $k=1$ | 57.6% | 81.48% | 65.1% |

Based on a metric, which in our analysis was the similarity metric, a vector database can be configured to return the top $k$ results, ${\bm{d}}(k,q_{i})$, for a given query $q_{i}$. We seek a metric to check if $d^{*}(q_{i})\in{\bm{d}}(k,q_{i})$ for all $q_{i}$ and their associated golden documents in the evaluation set. For an evaluation set of $N$ queries, the context recall (denoted by $R$) is computed as follows:

$$ $R(k)=\frac{1}{N}\sum_{i=1}^{N}{\bm{1}_{d^{*}(q_{i})\in{\bm{d}}(k,q_{i})}}$ (1) $$

The results for $k\in\{1,5,10\}$ are summarized in Table 6. In all cases, the fine-tuned E5 Large V2 model significantly outperformed both the base E5 Large V2 model and Open AI’s SOTA embedding model, Text Embedding 3 Large. The reason is that the dataset contained a lot of domain-specific jargon relating to cybersecurity that the base embedding models were not able to encode. Furthermore, the tuned embedding returned $d^{*}$ in top 5 ranks in ${\bm{d}}$ 92.18% of the time, indicating that a re-ranker model would not have been necessary to bump $d^{*}$ from the top 10 to the top 5. Finally, the tuned embedding produced $d^{*}$ at the top rank 81.48% of the time, indicating strong ranking ability.

### 5.2 Generation Model

Because the answers generated by the generation models are all free-form text, it was difficult to come up with an objective evaluation of their correctness. Objective metrics like Bleu (Papineni et al., ) and Rouge (Lin, ) perform N-gram comparisons between expected and actual answers and may not recognize when the two are semantically equivalent if they use different words. For that reason, we used an LLM-as-a-judge to score the answers for correctness.

Once again, we used the G-Eval metric with DeepEval to score answers and provide reasons for the scores. With regard to evaluation criteria, we required that the generated answers be penalized for correctness if they 1) contradicted the true answer, 2) omitted details from the true answer that were relevant to the question, and 3) included irrelevant detail that were not present in the true answer.

**Table 7: Pipeline evaluation of different embedding and generation model configurations**
|  | TE-3-L<br>GPT-4o | Base Emb<br>Base Gen | Base Emb<br>Tuned Gen | Base Emb<br>GPT-4o | Tuned Emb<br>Base Gen | Tuned Emb<br>Tuned Gen | Tuned Emb<br>GPT-4o |
| --- | --- | --- | --- | --- | --- | --- | --- |
| % Context Recall at k=5 | 80.85 | 69.84 | 69.84 | 69.84 | 92.18 | 92.18 | 92.18 |
| % Answer parsing success | 99.96 | 98.18 | 99.88 | 99.96 | 99.53 | 100.00 | 99.84 |
| % Correct reference | 85.47 | 74.97 | 74.97 | 75.96 | 91.04 | 88.91 | 92.10 |
| % Mean Correctness (soft) | 81.58 | 75.10 | 78.49 | 82.27 | 78.60 | 88.12 | 82.87 |
| % Mean Correctness (hard) | 72.38 | 60.97 | 70.24 | 63.99 | 76.03 | 86.07 | 79.08 |

We used Llama 3 405B (Zhou et al., ) for the aforementioned evaluation with DeepEval for its SOTA judging ability (Raju, ), speed, and cost (it is provided at 132 tokens/s for free by SambaNova Cloud). Seven combinations of embedding and generation models in the RAG framework were evaluated and the evaluation results are summarized in Table 7. “Base Emb” and “Tuned Emb” refer to the base and fine-tuned versions of E5 Large V2 embedding model, respectively. “Base Gen” and “Tuned Gen” refer to the base and fine-tuned versions of Llama 3 8B generation model, respectively. TE-3-L refers to Open AI’s SOTA ‘Text Embedding 3 Large’ model.

The first row of Table 7 recaps the context recall from Table 6 for $k=5$ to show how the other results may be impacted by it. The answer parsing success relates to the generation model’s ability to produce JSON-formatted answers with the required fields. That all combinations have at least a 98% parsing success indicates that the one-shot prompts (given in Appendex A.2.2) were adequately engineered to ensure correct completions most of the time. “% Correct reference” refers to the number of examples for which the correct reference was produced by the generation model. The references comprise URLs that are included in the retrieved context.

Two correctness scores are provided and both use the same G-Eval metric with Llama 3 405B. In the case of “mean correctness (soft)”, if $d^{*}(q_{i})\notin{\bm{d}}(k,q_{i})$ for any $q_{i}$ and the generated answer is “I am sorry, I do not have the answer to the question,” then we mark the answer as 100% accurate. That metric compensates for an inaccurate embedding in the retrieval component, explaining why it makes no difference to the result when we switch the embedding from base to tuned while keeping the same base generation model (either Base Gen or GPT-4o).

The “mean correctness (hard)” metric requires that the generated answer match the true answer, regardless of the embedding’s retrieval accuracy. No concessions are given for the generation model not admitting to knowing the answer. Therefore, soft correctness scores are higher because some of the answers that were marked as incorrect by hard correctness were forgiven my soft correctness.

The biggest gain on hard correctness, an improvement of 26 percentage points, was achieved when going from a Base Emb/Base Gen combination to a Tuned Emb/Tuned Gen combination. An improvement of 16 percentage points was achieved by swapping out the base embedding with a tuned one, for the same generation model.

Tuning the generation model allows it to correctly answer questions even if the answer is not present in the retrieved context leading to an improvement of 10 percentage points when going from a base generation model to a tuned generation model while keeping the embedding the same.

The first column in Table 7 refers to a solution using Open AI’s SOTA embedding and generation models. On hard correctness, that combination outperforms all other combinations that use the base embedding, but it underperforms those that use the tuned embedding. Therefore, tuning the embedding model is essential to beating proprietary SOTA models using open source SOTA models on our evaluation set.

### 5.3 Case Studies

In this section, we present three evaluation case studies to take a deeper look at the evaluation results. Each case study presents the results for a specific Q&A pair in the evaluation set. A tabular format is used in which the generated answers came from either GPT-4o, Llama 3 8B (base), or Llama 3 8B (fine-tuned), as specified in the column headers. The row $d^{*}$ rank in context refers to the ranked position in ${\bm{d}}$ when the retrieval succeeds (otherwise it reads $d^{*}\notin{\bm{d}}$). The hard correctness score and reason were both produced by Llama 3 405B, which we used for judging.

**Table 8: Evaluation Case Study: What is the purpose of KOPILUWAK?**
| Question | What is the purpose of KOPILUWAK? |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| True answer | KOPILUWAK has been used for victim profiling and C2 since at least 2017. |  |  |  |  |
| Document | Description of attack software ’S1075: KOPILUWAK’: KOPILUWAK is a JavaScript-based reconnaissance tool that has been used<br>for victim profiling and C2 since at least 2017. |  |  |  |  |
|  | Base Emb, Base Gen | Base Emb, Tuned Gen | Tuned Emb, Base Gen | Tuned Emb, Tuned Gen | OpenAI Emb, GPT 4-o |
| Generated answer | I am sorry, I do not<br>have the answer<br>to the question. | KOPILUWAK is a<br>command-line utility<br>used to compress and<br>encrypt data collected<br>prior to exfiltration,<br>such as tar. | KOPILUWAK is a<br>JavaScript-based<br>reconnaissance tool that<br>has been used for victim<br>profiling and C2 since<br>at least 2017. | KOPILUWAK has been<br>used for victim profiling<br>and C2 since at least<br>2017. | I am sorry, I do not<br>have the answer<br>to the question. |
| $d^{*}$ rank in context | $d^{*}\notin{\bm{d}}$ | $d^{*}\notin{\bm{d}}$ | 1 | 1 | $d^{*}\notin{\bm{d}}$ |
| Correctness score<br>(hard) | 0.0 | 0.0 | 0.8 | 1.0 | 0.0 |
| Correctness reason | The actual output<br>does not provide any<br>relevant information<br>about KOPILUWAK,<br>whereas the expected<br>output provides<br>specific details<br>about its purpose. | The actual output provides<br>unrelated information<br>about data compression<br>and encryption, whereas<br>the expected output<br>mentions its use for<br>victim profiling and C2<br>since 2017, indicating a<br>contradiction. | The actual output provides<br>extra information about the<br>programming language of<br>KOPILUWAK, which is<br>not present in the expected<br>output. However, it does<br>convey the same core<br>information as the expected<br>output regarding the<br>purpose of KOPILUWAK. | The actual output<br>matches the expected<br>output exactly,<br>with no omissions<br>or irrelevant details. | The actual output does<br>not provide any relevant<br>information about<br>KOPILUWAK, whereas<br>the expected output<br>provides specific details<br>about its purpose. |

The case study presented in Table 8 was among the 30% of all pairs for which the question and the answer were generated from the document using LLama 3 8B. The configurations with the tuned embedding both returned $d^{*}$ at the highest rank, which is desirable. The others did not return $d^{*}$ at all, likely because they were not able to properly embed the domain-specific term “KOPILUWAK”. Consequently, the configurations with the tuned embeddings produced correct answers, whereas the others did not. Llama 3 405B produced a score of 0.8 for the “Tuned Emb, Base Gen” configuration and its reasoning is clear that the answer includes irrelevant details. The scores for the “Tuned Emb, Base Gen” and Open AI configurations would have been set to 1 for soft correctness for their admission to not knowing the answer. The “Base Emb, Tuned Gen” configuration, however, would have received a soft correctness score of 0 for hallucinating.

Table 9 presents a case study in which all the embeddings find the $d^{*}$. Although the OpenAI embedding achieved a higher rank of 1 (all others had a rank of 3), GPT-4o generated a less accurate answer than the tuned generation Llama 3 8B. Like GPT-4o, even the base Llama 3 8B failed to mention that “testing and debugging” are purposes of the ‘Office Test’ registry key. That is despite the fact that the name of the key implies the purpose and the purpose is explicitly stated in $d^{*}$. That case study highlights that tuning the generation model with rationales can help improve reasoning.

**Table 9: Evaluation Case Study: What is the purpose of the ‘Office Test’ Registry key?**
| Question | What is the purpose of the ’Office Test’ Registry key? |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| True answer | The ’Office Test’ Registry key is used by Microsoft to load DLLs for testing and debugging purposes while developing Office applications. |  |  |  |  |
| Document | Description of attack technique ’T1137.002: Office Application Startup: Office Test’:\nAdversaries may abuse the Microsoft Office<br>”Office Test” Registry key to obtain persistence on a compromised system. An Office Test Registry location exists that allows a user<br>to specify an arbitrary DLL that will be executed every time an Office application is started. This Registry key is thought to be used<br>by Microsoft to load DLLs for testing and debugging purposes while developing Office applications. This Registry key is not created by<br>default during an Office installation.\n\nThere exist user and global Registry keys for the Office Test feature, such as:\n\n<br>HKEY_CURRENT_USER\Software\Microsoft\Office test\Special\Perf\n* HKEY_LOCAL_MACHINE\Software\Microsoft<br>\Office test\Special\Perf\n\nAdversaries may add this Registry key and specify a malicious DLL that will be executed whenever an<br>Office application, such as Word or Excel, is started. |  |  |  |  |
|  | Base Emb, Base Gen | Base Emb, Tuned Gen | Tuned Emb, Base Gen | Tuned Emb, Tuned Gen | OpenAI Emb, GPT 4-o |
| Generated answer | Adversaries may abuse<br>the Microsoft Office<br>’Office Test’ Registry key<br>to obtain persistence on<br>a compromised system. | The ’Office Test’ Registry<br>key is used by Microsoft<br>to load DLLs for testing<br>and debugging purposes<br>while developing Office<br>applications. | Adversaries may abuse<br>the Microsoft Office<br>’Office Test’ Registry key<br>to obtain persistence on<br>a compromised system. | The ’Office Test’ Registry<br>key is used by Microsoft to<br>load DLLs for testing and<br>debugging purposes while<br>developing Office<br>applications. | The purpose of the ’Office<br>Test’ Registry key is to<br>allow a user to specify an<br>arbitrary DLL that will be<br>executed every time an<br>Office application is<br>started. |
| $d^{*}$ rank in context | 3 | 3 | 3 | 3 | 1 |
| Correctness score<br>(hard) | 0.0 | 1.0 | 0.0 | 1.0 | 0.4 |
| Correctness reason | The actual output<br>contradicts the expected<br>output, stating it’s used<br>for persistence by<br>adversaries instead of<br>testing and debugging<br>purposes by Microsoft. | The actual output<br>perfectly matches the<br>expected output,<br>providing the same<br>information without<br>any omissions or<br>contradictions. | The actual output<br>contradicts the expected<br>output, stating it’s used<br>for persistence by<br>adversaries instead of<br>testing and debugging<br>purposes by Microsoft. | The actual output<br>perfectly matches the<br>expected output, providing<br>the same information<br>without any omissions<br>or contradictions. | The actual output<br>provides a partially correct<br>purpose of the ’Office Test’<br>Registry key but includes<br>arbitrary user specification<br>which is not present in the<br>expected output, and omits<br>the testing and debugging<br>purposes mentioned in the<br>expected output. |

Table 10 presents a case study in which the ‘Base Emb, Tuned Gen’ configuration is able to answer a question accurately even in the absence of $d^{*}$. That shows that fine-tuning the generation model on questions that cover a document, can allow the model to answer unseen questions about that document even when it is not presented in the LLM prompt context.

**Table 10: Evaluation Case Study: When was MoleNet first observed in use?**
| Question | When was MoleNet first observed in use? |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| True answer | MoleNet has been observed in use since at least 2019. |  |  |  |  |
| Document | Description of attack software ’S0553: MoleNet’:<br>MoleNet is a downloader tool with backdoor capabilities that has been observed in use since at least 2019. |  |  |  |  |
|  | Base Emb, Base Gen | Base Emb, Tuned Gen | Tuned Emb, Base Gen | Tuned Emb, Tuned Gen | OpenAI Emb, GPT 4-o |
| Generated answer | I am sorry, I do not have<br>the answer to the question. | MoleNet was first<br>observed in use in 2019. | MoleNet is a downloader<br>tool with backdoor<br>capabilities that has<br>been observed in use<br>since at least 2019. | MoleNet has been observed<br>in use since at least 2019. | I am sorry, I do not have<br>the answer to the question. |
| $d^{*}$ rank in context | NaN | NaN | 0.0 | 0.0 | NaN |
| Correctness score (hard) | 0.0 | 0.8 | 0.9 | 1.0 | 0.0 |
| Correctness reason | The actual output does not<br>provide any information<br>about when MoleNet was<br>first observed in use,<br>whereas the expected<br>output states it has been<br>observed since at least<br>2019. | The actual output<br>provides a specific year<br>that matches the<br>expected output, but<br>it implies a specific start<br>date, whereas the<br>expected output<br>leaves room for earlier<br>usage with ’at least’. | The actual output<br>provides the correct<br>year MoleNet was first<br>observed in use, but<br>also includes irrelevant<br>details about it being a<br>downloader tool with<br>backdoor capabilities. | The actual output matches<br>the expected output exactly,<br>with no omissions or<br>contradictions. | The actual output does<br>not provide any<br>information about when<br>MoleNet was first<br>observed in use, whereas<br>the expected output<br>states it has been<br>observed since at least<br>2019. |

## 6 Conclusion

In this work, we created a Q&A dataset based off the MITRE ATT&CK® database of cyberattack techniques, software, campaigns, mitigation approaches, and detection approaches. The dataset, AttackQA, can used to train models and create a chatbot to help security operations center analysts decrease their time to mitigate cyberattacks by giving them fast and accurate answers to questions that they may have about the attacks. We presented an approach to automatically generate data and perform quality control on that data using SOTA open-source LLMs.

We evaluated a RAG pipeline using our dataset and showed that fine-tuning both the generation and embedding models can lead to an increase in hard accuracy of 26 percentage points. Fine-tuning the embedding model alone can lead to an improvement of 16 percentage points. Finally, fine-tuning the generation model alone, as proposed by Zhang et al. (), leads to an accuracy improvement of 10 percentage points. Open AI’s SOTA models produced high accuracy but could be outperformed by tuning openly available embedding models. Even when GPT-4o was combined with our tuned embeddings, it underperformed a fine-tuned Llama 3 8B model, which was many times smaller.

The results establish a benchmark for modeling with AttackQA. The AttackQA dataset and associated benchmarking code are made openly available (Badrinath Krishna, ).

#### Acknowledgments

The author would like to thank Amit Kushwaha, Chen Wu, Meenakshi Swaminathan, James Valentine, and Nidhi Hiremath for helping improve the presentation of content in this paper.

The AttackQA dataset is derived from the MITRE ATT&CK® knowledge base, which bears the following copyright notice: © 2024 The MITRE Corporation. This work is reproduced and distributed with the permission of The MITRE Corporation.

## Appendix A Appendix

### A.1 Examples of tables from source dataset (MITRE)

In this appendix we present two examples of entries extracted from the MTIRE knowledge base. The first is an example of a software tool used by attackers and is presented in Table 11. There are 677 such entries in the MITRE knowledge base. Entries for techniques, tactics, groups, campaigns, and mitigation approaches also include unique ID, name, description, and URL.

**Table 11: Example software table entry from source data**
| Field | Value |
| --- | --- |
| ID | S0066 |
| name | 3PARA RAT |
| description | [3PARA RAT](https://attack.mitre.org/software/S0066 ) is a remote access tool (RAT) programmed in C++ that has been used by [Putter Panda](https://attack.mitre.org/groups/G0024 ). (Citation: CrowdStrike Putter Panda) |
| url | https://attack.mitre.org/software/S0066 |
| contributors | NaN |
| platforms | Windows |
| aliases |  |
| type | malware |
| relationship citations | (Citation: CrowdStrike Putter Panda),,(Citation: CrowdStrike Putter Panda) |

In creating AttackQA, we preprocess descriptions like “[3PARA RAT](https://attack.mitre.org/software/S0066 ) is a remote access tool (RAT) programmed in C++ that has been used by [Putter Panda](https://attack.mitre.org/groups/G0024 ). (Citation: CrowdStrike Putter Panda)” to “3PARA RAT is a remote access tool (RAT) programmed in C++ that has been used by Putter Panda.”

An example of a relationship entry in the MITRE knowledge base is presented in Table 12.

**Table 12: Example relationships table entry from source data**
| Field | Value |
| --- | --- |
| source ID | M1036 |
| source name | Account Use Policies |
| source type | mitigation |
| mapping type | mitigates |
| target ID | T1110 |
| target name | Brute Force |
| target type | technique |
| mapping description | Set account lockout policies after a certain number of failed login attempts to prevent passwords from being guessed. Too strict a policy may create a denial of service condition and render environments un-usable, with all accounts used in the brute force being locked-out. Use conditional access policies to block logins from non-compliant devices or from outside defined organization IP ranges.(Citation: Microsoft Common Conditional Access Policies) |

### A.2 Example Prompts

In this section, we present the exact prompts that were used for dataset generation and for the RAG application.

#### A.2.1 Prompts used for Dataset Generation

The Python function used to construct the prompts for dataset generation using LLMs is presented below. Valid JSON, containing up to three entries, was generated 99% of the time even when the API did not support token sampling constraints for JSON outputs. The prompt template is customized for Llama 3 models.

[
breaklines,
breaksymbolleft=
]python
def get_prompt_for_doc(doc, count=”three sets”):
prompt = ”””¡—begin_of_text—¿¡—start_header_id—¿system¡—end_header_id—¿
You are a JSON generator who generates machine-readable JSON¡—eot_id—¿¡—start_header_id—¿user¡—end_header_id—¿
Based on the following document, follow the instruction below
Document:
Instruction:
Generate JSON format:
[
”question”: ”¡generated question¿”,
”thought”: ”¡generated thought on what is needed to answer the question. Start with ’To answer the question, I need’¿”,
”answer”: ”¡generated answer¿”,
”references”: [
”¡verbatim text from document that supports the answer¿”,
”¡verbatim text from document that supports the answer¿”
]
]
The first character of the response must be ’[’ and the last character must be ’]’. No header text should be included.
¡—eot_id—¿¡—start_header_id—¿JSON list¡—end_header_id—¿
”””return prompt

#### A.2.2 Prompts for generating answers in RAG

The RAG prompt contains instructions, a one-shot example to illustrate the required response format, and the entire list of $k=5$ documents from the retrieval model. The prompt template was used for both fine-tuning the model and for inference on the base or fine-tuned models. The tags in the example prompt, which are specific to Llama 3, were removed when performing inference using GPT-4o.

Prompt:
<—begin_of_text—><—start_header_id—>system<—end_header_id—>
You are an assistant for generating JSON formatted responses
<—eot_id—><—start_header_id—>user<—end_header_id—>
Respond with a JSON dictionary that includes a thought, answer, and references
The answer must contain text obtained strictly from the given documents.
Avoid any text that is not in the given documents.
Answer using concise, self-contained, grammatically complete sentences.
The answer must be a string with less than four sentences.
Do not mention the documents by number or the context in the answers.
Answer the question strictly using the provided documents.
If you cannot answer the question using the documents, the answer should be ”I am sorry, I do not have the answer to the question.”
Along with the answer, include a thought that begins with ”To answer the question, I need”.
The references must contain URLs that exactly match the full URLs in the document headers relevant to by the answer.
There may be multiple references in the references list.
Follow the example below:
Document 1: https://attack.mitre.org/techniques/T1562/001
The campaigns that used attack technique ’T1562.001: Disable or Modify Tools’ were: ’C0002: Night Dragon’, ’C0024: SolarWinds Compromise’, ’C0028: 2015 Ukraine Electric Power Attack’, ’C0029: Cutting Edge’”
Document 2: https://attack.mitre.org/techniques/T1562/002
The campaigns that used attack technique ’T1562.002: Disable Windows Event Logging’ were: ’C0024: SolarWinds Compromise’, ’C0025: 2016 Ukraine Electric Power Attack’
Document 3: https://attack.mitre.org/techniques/T1070/001
The campaigns that used attack technique ’T1070.001: Clear Windows Event Logs’ were: ’C0014: Operation Wocao’
Question: What campaigns used attack technique ’T1562.002: Disable Windows Event Logging’?
Response:
{
“thought”: “To answer the question, I need to know what campaigns used attack technique ’T1562.002: Disable Windows Event Logging’. The answer is contained in the provided document with URL ’https://attack.mitre.org/techniques/T1562/002’.”,
“answer”: “The campaigns that used attack technique ’T1562.002: Disable Windows Event Logging’ were: ’C0024: SolarWinds Compromise’, ’C0025: 2016 Ukraine Electric Power Attack’”,
“references”: {[“https://attack.mitre.org/techniques/T1562/002”]}
}

Document 1: https://attack.mitre.org/techniques/T1539
How data component ’Process Access’ can be used to detect attack technique ’T1539: Steal Web Session Cookie’:
Monitor for attempts by programs to inject into or dump browser process memory.
Document 2:
https://attack.mitre.org/techniques/T1539
The following 2 data components can be used to detect attack technique ’T1539: Steal Web Session Cookie’: File Access, Process Access
Document 3: https://attack.mitre.org/techniques/T1539
The software procedures that use attack technique ’T1539: Steal Web Session Cookie’ are: ’S0467: TajMahal’, ’S0492: CookieMiner’, ’S0531: Grandoreiro’, ’S0568: EVILNUM’, ’S0631: Chaes’, ’S0650: QakBot’, ’S0657: BLUELIGHT’, ’S0658: XCSSET’
Document 4: https://attack.mitre.org/techniques/T1539
Tactics used in attack technique ’T1539: Steal Web Session Cookie’: Credential Access
Document 5: https://attack.mitre.org/techniques/T1539
Description of attack technique ’T1539: Steal Web Session Cookie’: An adversary may steal web application or service session cookies and use them to gain access to web applications or Internet services as an authenticated user without needing credentials. Web applications and services often use session cookies as an authentication token after a user has authenticated to a website. Cookies are often valid for an extended period of time, even if the web application is not actively used. Cookies can be found on disk, in the process memory of the browser, and in network traffic to remote systems. Additionally, other applications on the targets machine might store sensitive authentication cookies in memory (e.g. apps which authenticate to cloud services). Session cookies can be used to bypasses some multi-factor authentication protocols. There are several examples of malware targeting cookies from web browsers on the local system. Adversaries may also steal cookies by injecting malicious JavaScript content into websites or relying on User Execution by tricking victims into running malicious JavaScript in their browser. There are also open source frameworks such as ‘Evilginx2‘ and ‘Muraena‘ that can gather session cookies through a malicious proxy (e.g., Adversary-in-the-Middle) that can be set up by an adversary and used in phishing campaigns. After an adversary acquires a valid cookie, they can then perform a Web Session Cookie technique to login to the corresponding web application.
Question: What is a potential indicator of the ’T1539: Steal Web Session Cookie’ attack technique?
The response must be formatted as a JSON instance that conforms to the JSON schema above.
No text should appear before or after the JSON instance.
Response:
<—eot_id—><—start_header_id—>machine-readable JSON<—end_header_id—>

Completion:
{
”thought”: ”To answer the question, I need to understand how to detect the ’T1539: Steal Web Session Cookie’ attack technique. The answer is contained in the provided document with URL ’https://attack.mitre.org/techniques/T1539’.”,
”answer”: ”A potential indicator of the ’T1539: Steal Web Session Cookie’ attack technique is monitoring for attempts by programs to inject into or dump browser process memory.”,
”references”: [”https://attack.mitre.org/techniques/T1539”]
}
The above completion contains a $thought$, $answer$, and $references$. It is only used for fine-tuning the generation model and is constructed using fields from AttackQA. The $thought$ describes the rationale and is included to help the model learn to find the right document and use it to answer the question.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: I appreciate the author's efforts throughout the paper, this work has a clear contribution and recursive structure, my main concern is the limitations of the contribution. Specifically, all of evaluations are based on AttackQA dataset, despite the author splits it to training set and the evaluation set, but they are also generated in the same source and have similar distribution. Then it is not surprising that the improvement in experimental results is achieved on the test set after fine-tune the model, the evaluation experiments can only show that the AttackQA fine-tuned model performs better on AttackQA after the model has been integrated into the RAG pipeline, but cannot show that the model has improved its overall performance in the cybersecurity domain. This paper is an excellent technical report, containing detailed parameters and specific results, but the overall contribution is not sufficient to support it as a research paper.

In addition, as a security-related paper that might raise ethical concerns, having a relevant discussion indicates that due diligence has been made to minimise potential harm has become a common practice, hope the authors can provide this information.


*minors*:
- Unify the terms. `The MITRE ATT&CK knowledge base` and `The MITRE ATT&CK@ knowledge base`
- Line173-175, the use of inverted commas is not standardised, it is recommended to present the dataset example by a separate text box rather than plain text
soundness: 3
strengths: - Build high-quality, reliable dataset AttackQA for cybersecurity community
- Plans for dataset and benchmark open-source 
- In addition to the dataset generation, a significant amount of work has been done to illustrate feasible application scenarios for AttackQA, enriching the hierarchy of the whole paper
summary: This paper presents a detailed process for dataset construction and model capability enhancement, centred around the topic of cybersecurity Q&A. The paper describes the experimental details to illustrate its reliability, and gives specific experimental results with case studies. When the fine-tuned models based on AttackQA are integrated into related RAG pipeline, it can effectively improve the model performance in cybersecurity domain.
weaknesses: - The discussion of related datasets is missing in the Related Work section
- Lack of ethics review as a cybersecurity domain paper that might raise ethical concerns

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 1
questions: 1.  What is the performance of the existing Q&A systems for network security datasets, and  what advantages does AttackQA provide?

2. Given the abundance of cybersecurity datasets, why did the author choose MITRE ATT&CK® .  If AttackQA were trained on a different  dataset, would the performance vary significantly? How can AttackQA be scaled to other datasets?

3. How did the authors choose the LLM models through the entire process? What are the criteria and reasons for choosing LLM for different stages and tasks?

4. Could the authors elaborate on the specific techniques and strategies employed during the fine-tuning of the Llama 3 8B and Llama 3 70B models? What were the main challenges encountered, and how were they addressed?

5. In section 3.1, the paper claims each Q&A pair is derived from a single document because answers do not require information from multiple documents. Why? I question the validity of this assertion.
soundness: 2
strengths: Main strengths:
+ Reduce both the cost and complexity of the system by utilizing open-source LLMs and embedding models.
+ Fine-tuning open-source LLM and embedded models achieves better performance than using proprietary models.
summary: The paper introduces AttackQA, a cybersecurity Q&A dataset comprising 25,335 Q&A pairs with rationales, generated using the open-source LLM Llama-3 8B. A larger Llama-3 70B model was fine-tuned to improve dataset quality by filtering out low-quality pairs. The dataset was used to fine-tune open-source embeddings and LLMs, demonstrating better accuracy compared to OpenAI's GPT-4o. This work aims to support analysts in security operations centers by enhancing the efficiency and effectiveness of cybersecurity operations.
weaknesses: Main weaknesses:
-  Since AttackQA mainly focuses on textual knowledge, it cannot address the challenges of rapid response to cybersecurity incidents. In practice, Security Operations Centers (SoCs) also need to check  various tools' result to investigate the incidents.
- The experimental results may not be universally applicable, as AttackQA relies on a single dataset named MITRE ATT&CK® .
- The methodology for selecting techniques at each stage is not explained in sufficient detail.
- The experiments lack a comparison with existing solutions in this field.
- Partial grammatical errors and incomplete legends, such as the first paragraph in the conclusion section and  in the legend of Figure 2.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: 1. How were the Q&A pairs chosen to ensure that they reflect the most relevant types of
questions SOC analysts would encounter?

2. Could the paper provide more details on the specific metrics, thresholds,
and methods used in the quality control process to filter Q&A pairs?

3. In what types of scenarios or question types does AttackQA outperform GPT-4o,
and could the paper elaborate on the specific strengths in these contexts?

4. Has the model’s performance been tested on consumer-grade hardware,
and if so, could these results be included?

5. Did the paper do any user studies or
tests with SOC analysts to gather real-world feedback
on the tool's performance and utility?

6. Could the paper discuss any potential performance
trade-offs or adjustments when deploying AttackQA on hardware with limited resources?

7. Did the paper consider testing AttackQA’s performance on
more complex, multi-step SOC questions?
 If so, what were the findings, and/or is this an area of future exploration?

9. Could the paper share preliminary findings on common
errors or failure cases, and what strategies might address these limitations?

10. How could this dataset be useful for the cybersecurity research community?

11. In what ways can researchers use this dataset for open science or further studies?

12.  What specific research applications and practical uses does AttackQA offer for cybersecurity researchers?

13. How does the dataset add value and relevance to research in cybersecurity?
soundness: 4
strengths: 1. Domain-Specific Dataset

The reviewer notes the paper’s primary strength in creating a reasonably large,
cybersecurity-specific Q&A dataset that
leverages the MITRE ATT&CK knowledge base, a widely adapted resource in the cybersecurity community.
The paper has included detailed
descriptions of the dataset generation process,
including both human-generated and LLM-generated Q&A pairs,
and a quality control process.
The review believes this structured approach increases confidence in the dataset's
quality and applicability for SOC needs.

However, the reviewer wonders about the representativeness
of the Q&A pairs, particularly how well they reflect real SOC inquiries.
The authors might elaborate on how they determined which types of questions
would be most beneficial to SOC analysts.

2. Quality Control Mechanism

The reviewer appreciates the paper’s attention to quality control,
which involves fine-tuning LLMs (e.g., Llama 3 70B) to identify and
filter low-quality Q&A pairs.

The reviewer wonders if the paper could provide more
insight into how the quality control model was trained and
the rationale behind choosing G-Eval metric for
quality acceptance.


3. Model Deployment with High-Throughput Performance

The paper highlights that AttackQA models achieve high token generation speeds
using specialized hardware, such as the SambaNova Cloud,
which is particularly beneficial for SOC environments where latency is critical.

The reviewer wonders if the paper has considered testing the model
on consumer-grade hardware. Including performance benchmarks on less specialized
hardware could make the paper more broadly applicable to organizations with limited resources.
summary: The paper proposes a dataset, AttackQA,
designed to assist analysts in Security Operations Centers (SOCs)
with timely and accurate answers to cybersecurity-related questions.
The dataset is created using the MITRE ATT&CK knowledge base
and features final 25,335 question-answer (Q&A) pairs fine-tuned to
improve retrieval-augmented generation (RAG) pipelines.
The proposed system is intended to streamline SOC workflows,
enabling analysts to quickly access high-quality,
contextualized information, thereby addressing key challenges in cybersecurity operations,
such as knowledge gaps and slow response times.

The reviewer acknowledges the relevance of the dataset and the potential impact of a fast,
reliable RAG pipeline that can be deployed using accessible,
open-source technology. The reviewer thinks this work is a valuable addition to the cybersecurity
field by providing a high-quality, open-source tool for cybersecurity-specific question answering.

However, the paper does not explain how this dataset could be useful for the cybersecurity research community. The reviewer recommends adding a discussion and/or subsection to discuss specific research applications and practical uses of AttackQA for cybersecurity researchers. This addition would help readers realize the dataset’s value and relevance for research in cybersecurity.
weaknesses: 1. Limited Validation in Real-World SOC Settings

The reviewer observes that the dataset and RAG pipeline have not yet been validated
in an operational SOC environment,
limiting the understanding of AttackQA’s real-world effectiveness.

The reviewer suggests that a small-scale deployment or pilot study could greatly enhance
the paper’s credibility.
Even preliminary data on SOC analysts’ feedback or performance improvements in
simulated SOC scenarios would substantiate the system's value.



2. Dependence on Specialized Hardware

The reported performance relies on access to high-performance hardware,
such as the SambaNova Cloud, which may not be available to all SOCs.
This could limit the scalability and accessibility of the proposed solution.

To increase accessibility, the reviewer recommends that the authors benchmark AttackQA on
widely available hardware, such as NVIDIA consumer GPUs,
and report these results as part of the analysis.
This would demonstrate the adaptability of the pipeline to a range of environments.


3. Scope of RAG Framework

The reviewer notes that while the RAG framework is effective,
it may not fully address the complexity of real SOC inquiries that
require multi-hop reasoning or cross-document synthesis.
AttackQA, in its current form, may be limited in answering such complex questions accurately.

The reviewer recommends that the authors may consider extending the framework to
include multi-hop reasoning or explore alternative retrieval models that can
handle cross-document synthesis. This could make the model more adaptable to complex,
real-world SOC questions.

4. Error Analysis and Failure Case Discussion

The reviewer finds that the paper lacks an analysis of common failure cases,
which could help in understanding potential limitations of AttackQA and
provide directions for improvement.

The reviewer suggests conducting an error analysis to identify and
categorize common failure cases, such as instances where the model’s answers are incomplete,
incorrect, or hallucinated.
Outlining these challenges and discussing potential improvements would strengthen the paper.

5. Practicality and Utility in the Cybersecurity Research community

The paper does not discuss how AttackQA might be used by researchers,
leaving questions about its broader utility in the cybersecurity field.
Note that the reviewer is focusing on the cybersecurity research community, not its application in the industry.

- How could this dataset be useful for the cybersecurity research community?
- In what ways can researchers use this dataset for open science or further studies?
- What specific research applications and practical uses does AttackQA offer for cybersecurity researchers?
- How does the dataset add value and relevance to research in cybersecurity?

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: See comments above
soundness: 3
strengths: + It takes some efforts to curate the dataset
+ The security dataset is needed
summary: This paper curated a dataset named AttackQA based on  MITRE ATT&CK knowledge base and fintune the LLM and embedding model to show that the performance of QA and retrieval can be improved by fine-tuning.
weaknesses: - The paper is a bit hard to follow for readers unfamiliar with the cybersecurity
I would say, for a machine learning conference, the paper should be able to be understood by general audience, instead of only readers who already have a strong background of cybersecurity. For example, in the related work section, the paper could expand on giving a detailed example of one entry of MITRE ATT&CK dataset since this work is based on that. The same for your dataset. You should provided detailed examples for your dataset, instead of leaving them in appendix. Although the paper mentions the  attack technique ’T1562.001' as example in the main body, it is hard for general readers to understand what is this for (I have some bg for cybersecurity but I also have no idea before searching for it). I would suggest replacing it with a more common CWE example such as SQL injection or integer overflow.

- The paper writing could be improved.
I think the paper writing could be enhanced if the authors put more efforts on that. For example, there are many double quotes like this ’T1562.001’. Please check how to properly use that in latex. Some figures like Fig4 is too small to see the texts on that. Moreover, the table alignment for some tables make it painful to read, especially Tab9. 

- Irrelevant contribution about decoding speed.
This paper claims the high decoding speed by SambaNova Cloud platform, and mentions it a lot in paper. However, I don't think using a cloud platform can be deemed as contribution. When we refer to inference efficiency, we usually refer it to serving framework such as vllm, sglang, or attention mechanism such as flash attention 2, or decoding strategy such as lookup decoding. If you really want to show the overhead is small and this system can be real-time, you should at least apply these commonly used techniques since they are open-sourced. 

- Unclear technique details
Due to the missing example, I found section 3.1-3.4 hard to understand. For example, what is your manual heuristics? why do you choose Llama-3-8B instead of gpt-4o or others? what is the instruction prompting? The missing of these details make readers hard to follow.

- Missing insights of model choice
The paper choose Llama-3 family for the whole work, while not even mentioning some sota open-sourced model such as Qwen-2.5-72B. There is no insights of such model prefernce. Also, in Section 3.5.2, the paper mentions that the Llama-3-70B(before finetuning) can outperform GPT-4o without any explanation nor detailed examples. It has no insights for the community. 

- Missing training details
In section 4, a lot of training details for RAG for embedding model are missing. 

- Choice of judgement model without justification
Again, this works chooses Llama-3 family (Llama-3-405B) for judgement. Typically, for the evaluation part, it needs justification for why such a method could suit. It won't be a problem if you pick gpt-4o since many LLM papers are doing that. It is commonly known as SOTA and able to judge. You CAN justify that gpt-4o is not accurate if you could provide detailed explanation and examples. Otherwise, it is not convincing to pick a 405B model for judgement as it is too large to run locally and unclear whether it is the SOTA method for judgement.

Meta-review: The paper presents AttackQA, a cybersecurity Q&A dataset with 25,335 question-answer pairs derived from the MITRE ATT&CK knowledge base and refined using open-source LLMs like LLaMA-3. Fine-tuning LLMs and embedding models with AttackQA improves performance in retrieval-augmented generation (RAG) pipelines, surpassing GPT-4o.

The paper has merits in building a domain-specific dataset for cybersecurity and improving model performance based on the dataset. But the paper still has weaknesses:
- The presentation is not good enough to illustrate the contributions.
- The model selection issue raised by multiple reviewers, that only LLaMA is used for comparison.
- The experiments are limited to the MITRE ATT&CK, which may not be generalizable.

The paper received mixed reviews with one borderline accept recommendation, one borderline reject recommendation, and two reject recommendations. After author-reviewer discussion and AC-reviewer discussion, the paper still has some weaknesses despite its contributions on building the cybersecurity dataset and LLM. Therefore, the AC considers that the paper falls short of the ICLR acceptance threshold and recommends rejection.

Decision: Reject