Title: CTIBench: A Benchmark for Evaluating LLMs in Cyber Threat Intelligence
Benchmark: CTIBench
Summary: This paper introduces CTIBench, a benchmark designed to assess LLMs’ performance in cyner threat intelligence (CTI) applications. CTIBench includes multiple datasets focused on evaluating knowledge acquired by LLMs in the cyber-threat landscape.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Cyber threat intelligence
Phenomenon defined by authors: Yes
Phenomenon definition: Cyber Threat Intelligence (CTI) is the ability to process and analyze vast amounts of unstructured threat and attack data; allowing security analysts to utilize more intelligence sources. It involves the collection, analysis, and dissemination of information about potential or current threats to an organization’s cyber systems can provide actionable insights to help organizations defend against these attacks. 
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: The CTIBench benchmark evaluates LLMs on five cybersecurity tasks: answering multiple-choice CTI questions (CTI-MCQ), mapping vulnerabilities to their root causes (CWEs), predicting severity scores (CVSS), extracting attack techniques (MITRE ATT&CK), and attributing threats to actors or malware. Each task provides a text input (e.g. vulnerability descriptions or threat reports) and expects structured CTI-relevant outputs.
Task item definition: 1. CTI-MCQ: One row contains a multiple-choice question with a question string, four answer options, and the correct answer label.
2. CTI-RCM:  A CVE description (free-text) and the corresponding CWE label representing the root cause.
3. CTI-VSP: One row has a CVE description and the associated CVSS v3.1 vector string with detailed severity metrics.
4. CTI-ATE: One row contains a description of a threat behaviour and a list of MITRE ATT&CK technique IDs mentioned in the report.
5. CTI-TAA: Each row contains a threat report from a reputed vendor mapped to an Avanced Persistent Threat (APT) group.
Task source: Real task examples (e.g. GitHub issues), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Partial real task (e.g. answering medical questions collected from real people), Representative task (e.g. answering medical licensing exam questions)
Dataset size: 4947
Dataset metadata available: No
Sampling method: Random sample (creators defined a task space and sampled from it), Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Multiple choice, Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric subscores: Yes
Metric subscores detail: Scores are presented separately for each task subset
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Somewhat. Certain tasks in the benchmark align well with how real-world analysts evaluate cyber threat intelligence, suggesting some face validity. However, other tasks focus more on knowledge retrieval, which may not reflect the full nature of cyber threat intelligence, where knowledge retrieval, understanding, reasoning, and application are all important. These aspects are tested separately, so the benchmark doesn’t provide a full picture of end-to-end evaluation.
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Author validity detail: The authors sample task items from real-world cyber threat issues and reformat them for evaluation. While this means the tasks are grounded in real-world problems, it remains unclear whether the evaluation of LLMs aligns with how cyber threat analysts would perform such assessments.
Taxonomy: Alignment / Safety
New taxonomy: General Capability / Safety