AthenaBench: A Dynamic Benchmark for Evaluating LLMs in Cyber Threat
  Intelligence
Authors: name: Md Tanvirul Alam; name: Dipkamal Bhusal; name: Salman Ahmad; name: Nidhi Rastogi; name: Peter Worth
Published: 2025-11-03 01:45:29

Abstract: Large Language Models (LLMs) have demonstrated strong capabilities in natural
language reasoning, yet their application to Cyber Threat Intelligence (CTI)
remains limited. CTI analysis involves distilling large volumes of unstructured
reports into actionable knowledge, a process where LLMs could substantially
reduce analyst workload. CTIBench introduced a comprehensive benchmark for
evaluating LLMs across multiple CTI tasks. In this work, we extend CTIBench by
developing AthenaBench, an enhanced benchmark that includes an improved dataset
creation pipeline, duplicate removal, refined evaluation metrics, and a new
task focused on risk mitigation strategies. We evaluate twelve LLMs, including
state-of-the-art proprietary models such as GPT-5 and Gemini-2.5 Pro, alongside
seven open-source models from the LLaMA and Qwen families. While proprietary
LLMs achieve stronger results overall, their performance remains subpar on
reasoning-intensive tasks, such as threat actor attribution and risk
mitigation, with open-source models trailing even further behind. These
findings highlight fundamental limitations in the reasoning capabilities of
current LLMs and underscore the need for models explicitly tailored to CTI
workflows and automation.