CyberThreat-Eval: Can Large Language Models Automate Real-World Threat Research?
Authors: Xiangsen Chen
Xuan Feng
Shuo Chen
Matthieu Maitre
Sudipto Rakshit
Diana Duvieilh
Ashley Picone
Nan Tang
Venue: Accepted by TMLR ()

Abstract: Analyzing Open Source Intelligence (OSINT) from large volumes of data is critical for drafting and publishing comprehensive CTI reports. This process usually follows a three-stage workflow---triage, deep search and TI drafting. While Large Language Models (LLMs) offer a promising route toward automation, existing benchmarks still have limitations. These benchmarks often consist of tasks that do not reflect real-world analyst workflows. For example, human analysts rarely receive tasks in the form of multiple-choice questions. Also, existing benchmarks often rely on model-centric metrics that emphasize lexical overlap rather than actionable, detailed insights essential for security analysts. Moreover, they typically fail to cover the complete three-stage workflow. To address these issues, we introduce CyberThreat-Eval, which is collected from the daily CTI workflow of a world-leading company. This expert-annotated benchmark assesses LLMs on practical tasks across all three stages as mentioned above. It utilizes analyst-centric metrics that measure factual accuracy, content quality, and operational costs. Our evaluation using this benchmark reveals important insights into the limitations of current LLMs. For example, LLMs often lack the nuanced expertise required to handle complex details and struggle to distinguish between correct and incorrect information. To address these challenges, the CTI workflow incorporates both external ground-truth databases and human expert knowledge. TRA allows human experts to iteratively provide feedback for continuous improvement. The code of CyberThreat-Eval benchmark is available at https://github.com/secintelligence/CyberThreat-Eval.

Published in Transactions on Machine Learning Research (11/2025)
CyberThreat-Ewe Can Large Language Models Automate Real-World Threat Research?
Xiangsen Chen Xuan Fengt, Shuo Chent, Matthieu Maitre?, Sudipto Rakshit?, Diana Duvieilh?, Ashley Picone?, Nan Tang
Miccosoott Research
Microsoft
Honn Kong University of Science and Techonology (Guangshou)
'Hong Kong University of Science and Techonology
Reviewed on OpenReview: https: htto:s//pppnreewwww.ee////rrttts
Abstract
Analyzing Open Source Intelligence (OSINT) from large volumes of data is critical for draft- ing and publishing comprehensive CTI reports. This process usually follows a three-stage workflow-triage. deep search and TI drafting. While Large Language Models (LLMs) of- fer a promising route toward automation, existing benchmarks still have limitations. These benchmarks often consist of tasks that do not reflect real- worlw analyst workflows. For exam- ple, human analysts rarely receive tasks in the form of multiple-choice questions. Also, ex- isting benchmarks often rely on model-centric metrics that emphasize lexical overlap rather than actionable, detailed insights essential for security analysts. Moreover, they typically fail to cover the complete threc-stage workflow. To address these issucs, we introduce CyberThreat--val which is collected from the daily CTI workflow of world-leading com- pany. This expert-annotated benchmark assesses LLMs on practical tasks across all three stages as mentioned above. utilizes analyst-centric metries that measure factual accuracy, content quality, and operational costs. Our evaluation using this benchmark reveals impor- lant insights into the limitations of current LLMs. For example, L.MS often lack the nuanced expertise required handle complex details and struggle to distinguish between correct and incorrect information. To address these challenges, the CTI workflow incorporates both ex- ternal ground-truth databases and human expert knowledge. TRA allows human experts to iteratively provide feedback for continuous improvement. The code of CyberThreat-Fi - benchmark is available at https:// http://ggghuuuboo/eiiiiiiii SCailiggnnne///berreer CCryeryerretteeeeeee Evvl..
1 Introduction
Open Source Intelligence (OSINT) reports play a critical role in identifying and researching emerging cy- bersecurity threats and supporting Cyber Threat Intelligence (CTI) tasks Zhon et all (2022); Ainslic ct (2023). Analysts rely substantial volumes of OSINT data, including threat reports, vulnerability disclo- sures, and information from social networks |aan Puyvelde & Tabarez Rienzi (2112)) to draft and publish comprehensive reports. In our production environment, this process unfolds in throe stages (Figure W): triage, in which articles are prioritized; deep search, where supplementary evidence is gathered; and TI drafting, where narratives containing elements such as Indicators of Compromise (IoCs) [Merrosolt. Corpora- hon (2125a) and ATT&CK techniques are compiled into the final report. According survey conducted with analysts from team, analyzing incident described in given article typically requires several hours.
LLMs OpenAl (218 Hosst et all (2I2E) Achiam et all (АлК Wi et al (2IVER) Lountonn et all (2IK))
offer significant potential to automate these traditionally manual analyscs. Recent research demonstrates
Work done during an internship Microsoft. Research Asia.

---

Published in Transactions on Machine Learning Research (11/2025)
LLM capability in tasks such as IoC extraction Chen ettneeetea all (2025), cross-source correlation Wi (2124))
Perrina al (2028), and malware or threat actor identification Siracusano et all (2123). To quantify progress benchmarks, including
more systematically, the community has also released a series of dedicated CTI
CTIBench for multi-choice reasoning and ATT&CK attribution Alam of all (2ize), CTISum for attack-
process summarisation Peng et all (2022)), SECURE for ATT&CK and CVE reasoning Bhusal el all (2n22)); CyberMetric for general cybersccurity knowledge evaluation Tihanyi et all (20221), and CyberBench for report summarisation and threat-category classification Lin el
(2024T)
New problem. Although current benchmarks cover various OSINT and CTI subtasks using diverse au- tomatic metrics, several research gaps remain. (1) Unrealistic task formats. Many previous evaluations employ unrealistic tasks, such as multiple-choice or fact-recall questions Alam (2022)) eh (21zZ)); Tihanyi et all (20Zzl); ; Liu et all (20245).. Security analysts almost never attribute breaches to threat ac- tors by selecting from options AA/B/C/D. In addition, Q&A tasks reward rote memory (e.g., What does CVE-2021-26855 refer to?) Such formats fail to evaluate critical analyst skills, including linking evidence to threat actors, assessing business risks, and producing coherent incident summaries. This highlights the need to design more realistic tasks that align with the actual workflows and decision-making processes of security analysts. (2) The evaluation metrics for some tasks are model-centric, not analyst-centric. For example, ROUGE Lin (2TRET and BERTScore Zhang et all (2014) are the main metrics for summary tasks like CTISum Peng el. al. (2022).. However, in our pilot test a sparse summary outscored a detailed one de- spite analyst preference for the latter (full example in Appendix [BN)). Although human A-B evaluation Peng all (2024) is used assoss summarics, it covers only a small portion of the benchmark, lacks clear scoring criteria, and is difficult to scale consistently. Therefore, a new benchmark with more analyst- analyst-centric centric metrics (3) The current benchmarks miss evaluations in key workflow stages. As Figure 01 shows, the typical OSINT analysis workflow consists of the three stages: triage, deep search, and TI drafting. However, as Table Z shows, no public benchmark covers all three steps that analysts take in the end-to-end workflow. Current, benchmarks mostly focus on single tasks like testing the LLMs to read one threat incident and answer simple questions like 'What is the threat actor of XXX incident?" Therefore, existing benchmarks typically cover only certain lasks within the three workflow stages mentioned above, while a benchmark that spans the entire end-to-end workflow is still lacking. With the limitations mentioned above, current rescarch benchmarks are inadequate in the real-world setting of threat intelligence analysis.
should be constructed.
Our work. This work is conducted within the CTI division of one of the world's leading technology companies, whose platforms and services reach billions of users globally and shape the infrastructure of modern computing. Security is heavily invested because it protects the company's business bottom-line. The data and insights we present in this paper are within this real-world context.
Leveraging workflow data from professional security analysts, we construct CyberThreat-Evae an expert- annotated benchmark. It addresses three key gaps in existing evaluations: (1) We design tasks that reflect real analyst workflows rather than simplified quizzes or Q&A formats; (2) We introduce analyst-centric metrics such as factual accuracy, time spent, and token cost; (3) We evaluate performance across all three stages of the CTI workflow workflow-triage, deep search, and TI drafting in an end-to-end manner. We assess four LLMs: two base models (GPT-40, 03-mini) and two models fine-tuned on a CTI-specific corpus. Results show that while LLMs demonstrate strong recall in triage, they suffer from low precision. In deep search, base models (GPT-4o, o3-mini) retrieve more useful URLs. During TI drafting. IoC extraction reveals a trade-off between speed and recall, and all models struggle with accurately extracting and mapping MITRE ATT&CK TTPs, However, LLMs are notably better at generating coherent root-cause narratives. These findings suggest that current LLMs are effective at retrieving relevant information but need significant improvements in Iriage precision, TTP reasoning, and cost efficiency. To address these challenges, the CTI workflow needs to utilize external knowledge databases and human experts in the loop. We will explain how these elements are built into our workflow, namely TRA (Figure B). We will release both Cyber SybeaTh - and TRA to support the community advancing analyst-oriented CTT automation.
2

---

Published in Transactions on Machine Learning Research (11/2025)
Rejected
Input Article
Accept
Deep Search
Related Articles
TI Drafting
Output Report
Triage
Several hours
Figure 1: Illustration of the threat analysis workflow.
2 Background and Related Works
2.1 End-to-end Threat Research Workflow
To effectively process OSINT for threat analysis, a structured end-to-end workflow essential to handle the scale of cyber threats. As illustrated in Figure M1, a typical production-leve workflow transforms raw data feeds into actionable, customer-ready threat intelligence through three distinct stages: triage, deep search, and TI drafting. The process begins with data ingestion, where crawler collects articles. These articles then enter the triage stage, where each is assessed based on its subject matter and target to determine its relevance and potential impact. Articles that do not meet the criteria are rejected and removed from the workflow. Accepted articles are assigned an incident priority to guide the allocation of analytical resources. For each prioritized incident, the workflow continues into two parallel yet interconnected stages: deep search and TI drafting. the deep search stage, analysts generate related queries for the incident and follow hyperlinks from the seed article. They gather materials from open- web indices, commercial feeds, and public or proprictary knowledge bases, evaluating each document for relevance and quality. This results in curated list of URLs that enrich the original case file. In the TI drafting stage, analysts create a working report based the seed article, ileratively incorporating evidence oblained from the deep search. Elements such as IoCs, MITRE ATT&CK TTPs, timelines, and mitigations are added. The narrative is refined each time new, high- value information is discovered. Iteration ends when further searches no longer yicld substantive additions, resulting in a structured, publication-ready report. The final output comprehensive, customer-readd article containing key intelligence components. Despite ongoing research and industry efforts to improve these workflows, challenges such incomplete coverage and inconsistent data quality persist [in et. al (20281).
2.2 LLM Benchmarks for OSINT
Recent research has leveraged LLMs to automate a variety of CTI tasks. As summarized in Table these methods predominantly focus specific CTT subtasks. For example, IntelEX Xu all and threat actors from unstructured threat reports, while Cylens Lin et all (2125) specializes in threat actor profiling and malware clustering. Other approaches, such as HPTSA Fang et all (20ZZ)) identify zero-day vulnerabilities from forum discussions, and CASEY Torkamani et all
(20221 extracts TTPs
212 automatically identifies software
weaknesses (CWEs) and assesses their severity from codebases. Existing benchmarks such as CTIBench Alam (1123) CTISum Peng et al. (20221) SECURE Bhusal all (20221), and CyberMetric Tihanyi et al. (2024] assess performance in threat research tasks and cover a broad range of challenges from knowledge- based question answering to threat actor attribution and vulnerability classification. SEvenLLM-Bench I et all (2024) defined a targeted evaluation with 1,300 test questions covering both English and Chinese threat intelligence content. Although each method demonstrates strengths in isolated tasks, they typically fail to offer comprehensive solutions spanning multiple CTI tasks simultaneously.
When considering coverage across a realistic threat intelligence workflow, consisting of triage, deep search, and TT drafting stages, most existing I.LM-based approaches exhibit significant limitations (see Table 2 reviewed methods reveals systemic gaps. Specifically, of the surveyed methods, only HPTSA Fang et al.
3

---

Published in Transactions on Machine Learning Research (11/2025)
(0224) explicitly includes capabilities for both triage and deep search, and only CyberForumCTI Clairoux Trpaanier et all (2122]] covers both triage and TI drafting stages comprehensively. Notably, no single approach adequately addresses all three stages simultaneously. This fragmented coverage prevents these methods from fully automating the end-to-end threat intelligence workflow critical for operational effectiveness.
Beyond task and workflow coverage, current benchmarks and approaches employ model-centric metries (e-g. ROUGE Lmm (20ne), BERTScore Zhang al (2uи), Exact-Match) that reward lexical overlap yet overlook analyst-centric costs such token usage and latency. Furthermore, task formats often quiz-style (like multi-choice questions, Q&A), which do not mirror how practitioners triage incidents compile intelligence briefs.
Table 1: Existing CTT benchmarks and approaches: tasks and metrics
Benchmark
Task Scenario
Reported Metrics
CTIBeench Alam et al (20221
CTI Context Understanding and Analyzing
Accuracy:
Macro-F1;
Exact-Match;
MAD
CTISum Peng
Attack Process Summarization
ROUGE-1/2/L4 BERTScore;
Human
A/B Ratings Exact-Match;
SEvenLLM-Ben chan
(01e))
Incident Analysis (like Vulncrability Ex- ploitation Analysis, Attack Intent Analy- ...) and Understanding (like Cyberse- curity Event Classification, Malware Extrac- tion, ...)
Micro/Macro-FF;
ROUGE-L; BLEU
CyberMetric Thhany et all (AEZE]
Cybersecurity Knowledge Q&
Accuracy; Human vs. Model Accuracy Accuracy; ROUGE-I MAD
SECURE Biussll el al 122
Cybersecurity Knowledge Extraction and Reasoning
CyberBench Lu ml
Threat Report Summarization: Threat- related QA and Category Classification Identification of Recent LLM- Vulner-
ROUGE; Accuracy; Micro/ Macro-/
HIPTSA sng sl
Overall Success Rate
abilities
Table 2: Workflow-stage coverage of existing benchmarks and approaches
TI Drafting
Benchmark
Triage
Deep Search
Threat Actors
Root Cause
loCs
TTPs
CTIBench Alam et all (AIиE)
X
X
X
V
CTISum Peng et all (
X
X
X
X
SEvenLLM-Bent Bench ailala (MIKE)
X
X
J
V
V
CyberMetric Tihanyi
(2118)
X
X
X
X
X
SECURE Bhusal et all (M1)
X
X
X
V
CyberBench Lin (212417
X
V
HPTSA Fang et al.
(212
X
X
X
X
X
X
3 Cyber Syberthreat-Err
3.1 Overview
As outlined in the Introduction (Sec M), existing benchmarks for CTI exhibit several critical limitations. To address these gaps and provide a more practical and comprehensive framework for evaluating LLMs automated CTI research, we present the Eyveattreat-Eval benchmark. Our contributions are threefold. First, we design practical tasks that closely mirror real-world analyst workflows. Second, we implement analyst-centric evaluation methods that go beyond lexical metries to emphasize practical utility and real- world relevance. Third, we enhance workflow coverage by targeting three critical stagestriage, deep search, and threat intelligence (TT) draftingwith task designs tailored to each stage. Our CyberThreat aims to

---

Published in Transactions on Machine Learning Research (11/2025)
establish rigorous, end-to-end standard for assessing an LLMs ability to perform the complex, high-value tasks essential to security analysts.
3.2 Data Construction
To ensure that our benchmark reflects the practical challenges faced by security professionals, construct it using representative data sources drawn from real-world, industry- -yradd threat intelligence operations. Each task is designed to evaluate the extraction and classification of threat-related information, with ground-truth annotations provided by security research experts. Table @ summarizes the data distribution across the three primary workflow stages, For the triage stage, experts assigned priority scores to 488 articles. The deep search component evaluated using an initial set of 55 input URLs. The TI drafting stage, which involves more complex and multi-faceted tasks, is assessed through several representative evaluations of core capabilities. In the IoC extraction task, boound-truth answers straightforward to identify, as they consist of concrete, structured artifactssuch file hashes, IP addresses, domain namesthat clearly indicate malicious activity, In contrast, the TTP assignment task requires expert judgment: human annotators manually review articles to identify and label relevant Tactics, Techniques, and Procedures (TTPs), evaluate their alignment with the incident's ' subject and target, and assign an appropriate priority score. For the content generation evaluation, 412 articles were selected by experts for their richness in analytical depth. This evaluation specifically tests the LLMs' ability to generate coherent and informative narratives about the threat actor and root cause of each incident. These two clements demanding aspects of threat analysisrequiring models to go beyond extraction and synthesize the who and why behind an incident. To support focused evaluation, some tasks distinguish between two types of inputs: "description' (a concise summary of the incident) and 'article" (the full original source).
chosen because they represent some of the most cognitively
3.3
Task Description
Table 3: Data Description for Benchmark Tasks by Workflow Slage.
This benchmark evaluates LLMs through series of tasks that re- flect the end-to-end workflow of cy- ber threat research. The tasks are organized into three key stages: triage, deep search, and TI drafting. Each task is designed to assess the specific skills required for effective and comprehensive threat analysis at each stage
Category
Task Component
Data Size
Triage
Triaging & Assigning Priority Scores
488 articles
Deep Search
Identifying Related Articles
55 articles
ToCs Extraction
1310 IoCs
T1 Drafting
TTPs Identification
1565 TTPs
Threat Actor Generation
412 articles
Root Cause Generation
412 articles
3.3.1 Triage
This stage evaluates the LLMs' ability to prioritize incidents using only readily available information without extensive analysis. To systematically measure their effectiveness, we design two-part evaluation focusing on incident prioritization based on subject matter (e.g., mobile malware) and the affected target (e.g., a single system). Effective prioritization requires synthesizing multiple dimensions of threat intelligence, guiding analysts to identify and address the most critical incidents first. Specifically, we define two tasks for ovaluation:
Task 1: article triage. In this task, LLMs are asked to determine whether an article should be 'accepled" (i.e,, it provides valid, actionable subject and target information about threat incidentsufficient to support identifying threat actors, root causes, etc.) or "rejected" (if it lacks relevant or actionable insights) for further analysis.
Task 2: priority score assignment. For articles in the 'accepted" category, the task is to assign an appropriate priority score that reflects the severity and urgency of the incident. LLMs are evaluated based on their ability to classify articles in Task 1 and to assign accurate priority scores in Task according predefined criteria.
5

---

Published in Transactions on Machine Learning Research (11/2025)
Human expert judgments are used to validate the scoring system and assess the reliability of the LLMs' prioritization. These scores help analysts assess the relative importance of each incident and prioritize their responses accordingly. The detailed scoring criteria are provided in Appendix B..
3.3.2 Deep Search
Following triage, analysts typically collect additional information to deepen their understanding of an inci- dent. The deep search evaluation assesses the capability of LLMs to gather and analyze relevant external resources that provide valuable context about a particular threat. Specifically, given an initial input URL describing a cybersecurity incident, the LLM is tasked with searching the web to identify additional URLs that contain supplementary or supporting details about the same incident. To evaluate the effectiveness of this retrieval process, we employ an ALseb comparative analysis. For each candidate URL, its content is systematically compared against the original seed article. An LLM, acting prompled to determine if the retrieved article contains genuinely new and valuable "additional information," such as novel facts, data points, different analytical perspectives, or extra technical details that are absent from the original source. The evaluating LLM outputs a structured decision ('truc* or 'false),, allowing us to quantify the 'average URLs with additional information' and thereby measure the quality and utility of the deep search results.
cybersecurity expert, is
3.3.3 TI Drafting
The final stage assesses the model's ability to synthesize information (from the initial article and potentially from the deep search stage) into structured and insightful threat intelligence reports. To measure the effectiveness, we design the following tasks below, which are categorized into two main areas.
(1) Understanding and reasoning of key threat elements. This section evaluates how well LLMs can extract and interpret core elements of cyber threat intelligence from unstructured articles. It focuses on two key types of information: IoCs and MITRE ATT&CK TTPs.
IoC extraction. This lask assesses the LLMs' ability to accurately identify and extract explicit ToCs from cybersecurity articles. loCs are essential for identifying malicious activity, attack vectors, and compromised systems. The core objective is to evaluate the LLMs' proficiency in recognizing and extracting these technical indicators from text.
MITRE TTP identification and mapping. This task evaluates LLMs' ability to automatically detect adversary behaviors described in incident reports and map them to the appropriate TTPs within the MITRE ATT&CK framework. Accurate TTP identification is critical for understanding how threat actors operate and for designing effective defense strategies. LLMs must process each article to identify and associate relevant TTPs, which is more challenging than loC extraction.
(2) Contextual intelligence evaluation based on human expert criteria. This area evaluates the ability of LLMs to generate detailed, accurate, and coherent narrative content, providing analysts with deeper insights into incidents. It comprises two primary tasks: (1) producing descriptive profiles of identified threat actors by synthesizing article context with broader domain knowledge, and (2) creating detailed narratives on the incident's root cause, including attack mochanisms, exploited vulnerabilities, and event timelines. Outputs are evaluated by human experts on relevance, accuracy, comprehensiveness, clarity, coherence, and proper source attribution. The goal is to produce intelligence summaries significantly richer and more actionable than the original sources alonc.
4 Experimental Results
4.1 Experiment Setup
We evaluate diverse LLMs representing varying capabilities and deployment scenarios. For general-purpose models, we select the pre-trained GPT-4o and the reasoning-orienttr o3-mini. Additionally, we fine- tune GPT-4o and GPT-4o-mini analyst-triaged data from 2024, denoted respectively as GPT-4o (FT)
6

---

Published in Transactions on Machine Learning Research (11/2025)
We employ a standard Supervised Fine-Tuning (SFT) approach designed to bolster the I.L.Ms' overall security- related capabilities across diverse CTI tasks. The SFT dataset is a curated aggregation of multiple high- quality, proprictary datasets derived from our security operations. This multi-task corpus encompasses a variety of CTI-related functions, including but not limited to Question Answering (QA), CTI Mapping, BaturalL.angungctto--uuur Translation, and Function Calling. For rigorous monitoring of the fine-tuning process, the aggregated dataset is partitioned into training (79%), validation (1%), and testing (20%) sets. All models operate at the temperature of 0.01 and seed as 42 for consistency. To evaluate narrative qual- ity (threat actors and root causes), we employ the L.IM-as-Judge paradigm Lin et all (2023); |Verga et al. (2024)). This evaluation method provides the LLM judge with the ground-truth article, candidate output, and a scoring rubric covering relevance, factual accuracy, comprehensiveness, clarity, coherence, and attri- bution. Each dimension is rated on a 1-5 scale. To ensure the reliability of our LLM-as-Judge evaluation framework, we collaborate with senior sccurity exports to draft rulc-based criteria, requiring the judge LLM to output a justification field alongside its decision. Subsequently, we conducted a validation study using a diverse set of approximately 100 article examples. Human experts review the LLM's judgments and provide detailed qualitative feedback instances where the LLM's reasoning deviate from expert consensus. Based on this feedback, we systematically refine the evaluation details and prompt instructions over two rounds of testing. This calibration process yields an agreement rate between the LLM-as-Judge and human experts exceeding 95%. This high level of agreement shows that the LLM-as-Judge can be a reliable method for expert evaluation in this specific domain.
(fine-tuned gpt-4o-2024-08-66 and GPT-4o-mini (FT) (fine-tuned Ept-40-min1-2224-07--1
Metrics align explicitly with workflow stages. For triage, models perform two sequential tasks: (i) deciding whether article should be accepted (measured by precision and recall), and (ii) assigning priority score to accepted articles. Priority assigument performance is quantified by pass rate (percentage of exact matches with analyst-provider scores) and average bias (mean signed deviation from ground truth). Operational efficiency is assessed via processing time and token consumption per article. In the deep search stage, we measure: the average number of newly discovered URLs per incident, and the subset of URLs providing additional information (e.g., ToCs, exploit code, or mitigations). Efficiency metrics (processing time and tokens used) are also recorded. In the T1 drafting stage, two task categories are evaluated. For loC extraction report precision, recall, latency, and token usage. For generating threat actor and rool cause narratives, we apply the six-dimension evaluation rubric mentioned earlier to comprehensively assess content quality.
and TTP identification,
4.2 Overall Results
This section presents the detailed experimental results in our comprehensive threat research benchmark for each stage of the workflow. Each subsection elaborates on a specific task, and describing the performance of various LLMs. The result table is shown in Table H!
Triage performance. Triage evaluation focuses on two components: article classification ( ('accepted" or 'rejected") and priority score assignment. Results are shown in Table HI. For article classification, LLMs exhibit high recall (often >0.90), indicating strong sensitivity to potentially relevant articles. However, precision remains low (typically <0.40), reflecting a tendency to over-accept, which may increase analysts workload. For priority scoring, models achieve 55-60% accuracy, with fine-tuned variants like GPT-4o (FT) and GPT-4o-mini (FT) occasionally aligning more closely with expert judgments. Efficiency varies significantly: GPT-4o the fastest and most token-efficient, while 03-mini and finc-tuned models consume more time and tokens, especially when processing full-article inputs. These results highlight a trade-off between accuracy and operational cost LLMS can broadly identify relevant content but need improvement in precision and more nuanced prioritization.
Deep search performance. The deep search stage evaluates LLMs' ability to retrieve and rank contextual information that enriches incident understanding. As shown in Table EL base models like GPT-4o and o3- mini process more URLs on average compared to finc-tuned models. They also identify more URLs with additional relevant information (3.54 and 3.00, respectively). In contrast, fine-tuned LLMs are much more conservative. This suggests that fine-tuning may lead more focused retrieval, possibly because these
7

---

Published in Transactions on Machine Learning Research (11/2025)
Table 4: Performance Metrics for Vanilla LLMs by Task Category, Task, Metric, and Model
Category
Task
Metric
GPT-4o
03-mini
GPT-4o (FT)
GPT-lo-mini (FT)
Procision (Accpptdd)
0.3590
0.3982
0.3392
0.3477
Recall (Accepted)
0.9899
0.9091
0.9798
0.9798
Priority Sooring (Description)
Pass Rate (Score)
0.566
0,550
0.
0.590
Avg. Bias (Score)
0,505
0,680
0.170
0.520
Total Time (s)
708.23
5178.38
19399.3
4424.71
Total Tokens
430124
450093
453260
553547
Triage
Precision (Aocopled)
0.3037
0.3802
0.2717
0.2964
Rocall (Accpledd)
1.0000
0.9293
0.9798
1.0000
Priority Scoring (Article)
Rate (Scor))
0.566
0.580
0.590
0.600
Avg. Bins (Score)
0.505
0.550
0.500
0.450
Total Time (s)
785.64
60488-17
6031.71
7283.13
Total Tokens
1438118
1390052
1522969
1891248
Avg. URLs proceseed (w. inpur)
6.22
4.93
1.75
1.25
Avg. URLs w. additional info
3.54
3.00
0.38
0.22
Deep Search
URLLs Extraction
Aug. Processing Time(s)
484.94
322.95
345.24
Precision
0.8210
0.8503
0
0.6941
Time (s)
580
7533
727
808
loC Extraction
Tokens
339080
448152
331766
319700
Precision
0.2787
0.3480
0.2387
0.1771
Recall
0,2270
0.1759
0.1846
0.1414
MITRE ATT&CK TTP Ideutification
[s)
5097
2254
TI drafting
Tokens
466426
432593
453282
493860
Relevance
1.547
3.964
3.964
2.475
Accuracy
1.528
3.656
3.655
2.405
Comp,
1.145
3.165
3.165
1.755
Content: Threat Actor
Clazity
2.019
1.753
1.752
3,699
Coherence
1.731
4.731
4.731
3.405
Altribution
1.140
2.968
2.967
1.832
Relevance
3,686
4.733
4.533
3,808
Accurey
3.458
4.627
4.371
3.819
Comp.
3.362
4.576
4,286
3.441
Content:
Root Causse
Clarity
3.932
4.818
4.629
4.163
Coherence
3.753
4,800
4.586
3,988
Artribution
3.612
1.418
4.323
3.667
GIPT-40 (FT) to Finciuued 20-244 06; Go-T-4--mini (FT) relers Finetuned GPT DOPT-4o-mini 2024--0224-0000
Comp.: Comprohensiveness. Time (s) and Tokens rounded. Content seores (Relevance, Arruracy, Comp.) averagrs.
models possess greater inherent knowledge of specífic incidents due to specialized training. As a result, they may perform less external searching or consider fewer external sources as offering truly novel insights.
TI drafting performance This stage evaluates the LLMs' ability to synthesize information from given articles into structured, insightful threat intelligence components. Our findings, sunmarized in Table EI indicate notable variability in LLM performance across drafting tasks, broadly categorized into key element extraction and narrative content generation.
Key elements extraction. Extracting fundamental threat elements like loCs and MITRE ATT&CK TTPs reveals distinct L.LM strengths and weaknesses. Base models like GPT-4o and 03-mini demonstrate strong precision (around 0.82-0.85). However, efficiency is a concern: for example, while o3-mini is accurate, is significantly more resource-intensive in terms of both time and token usage compared to GPT-4o. Smaller models, such as GPT-4o-mini (FT), perform worse in precision (0.6944), highlighting that model size and fine-tuning are critical for reliable and cost-effective IoC extraction. In contrast, the TTP identification task remains & substantial challenge. All evaluated vanilla LLMs exhibit low precision and recall (mostly below 0.35). Although o3-mini achieves the highest TTP precision (0.3480), it also has the lowest recall, indicating it misses many relevant TTPs. Fine-tuning does not consistently improve performance, suggesting that this task demands deeper reasoning about adversarial bchavior than current LLMs are typically capable of.
Content generation quality. When generating descriptive narratives, LLM performance varies significantly between explaining threat actors and identifying root causes. For threat actor content, base GPT-4o performs poorly. In contrast, oi-mini and GPT-4o (FT) produce markedly better results, with high relevance and
8

---

Published in Transactions on Machine Learning Research (11/2025)
accuracy scores (around 3.7-4.0). However, their ability to provide fully comprehensive and well-attributed details (comprehensiveness and attribution scores around 3.0) still needs improvement. For root cause content, LLMs generally perform better, achieving acceptable scores across all six evaluation criteria. This suggests that LLMs are currently more adept at explaining the more factual 'how" of an incident (root cause) than the often more inferential and complex 'who' (threat actor). Their ability to generale clear and logical root cause analyses stands out as a notable strength.
4.3 Results Summary and Analysis
The experimental results highlight both the promising capabilities and current limitations of LLMs in auto- mated cyber threat rescarch tasks.
Performance varies in different tasks. In the triage stage, LLMs exhibit strong recall capabilities, consistently retrieving most relevant articles; however, they frequently struggle with precision, accepting many irrelevant entries. During the deep search stage, general-purpose LLMs such as GPT-4o and o3- mini typically identify more URLs containing valuable additional information compared to their fine-tuned counterparts. This implies that fine-tuned LLMs might be more selective or conservative, potentially due to their specialized training reducing the need for extensive external information searches. Additionally, processing efficiency (measured by retrieval time) varies signíficantly, with smaller or fine-tuned LLMs oc- casionally outperforming larger basc LLMs. In the TI drafting stage, the extraction of IoCs is relatively robust, although improvements in recall and processing efficiency remain necessary. By contrast, identifying and mapping TTPs presents substantial challenges for all evaluated LLMs. For narrative content generation, LLMs demonstrate notable proficiency in creating clear, coherent explanations of root causes. However, con- sistently generating comprehensive, well-attributed descriptions of threat actors proves considerably more challenging.
The performance-cost trade-off challenge. LLMs consistently exhibit. a fundamental trade-off across threat research tasks: enhancing recall (identifying all potentially relevant information, as seen prominently in triage) often diminishes precision (the accuracy of the identified information), simultaneously increasing operational costs (time and token consumption). Conversely, attempts to boost precision typically result in lower recall or demand greater resources. The TTP identification task exemplifies this clearly: the o3- mini achieves the highest precision observed (0.35 compared to GPT--oo's 0.28) but consumes approximately six times more total processing time and 1.3 times more tokens. On the other hand, fine-tuned GPT-4o substantially reduces latency by nearly two-thirds, but its precision correspondingly drops to 0.24. Analysts are thus confronted with clear operational dilemma: they must either accept slower, resource-intensive processes for enhanced accuracy, or prioritize faster, cost-effective solutions at the expense of potentially overlooking relevant introducing inaccurate information.
No one-fits-all solution of fine-tuning (specialization vs. generalization). Our results indicate that fine-tuning LLMs on domain-specific data can significantly enhance their performance on targeted tasks, such as generating detailed threat-actor profiles. However, this approach does not universally improve all performance metrics and may occasionally degrade certain capabilitics. For example, finc-tuning appears to limit LLMs exploratory breadth, making it less effective in tasks requiring exlensive external information retrieval, as evidenced in our deep search evaluations. Additionally, fine-tuning alone does not substantially mitigate fundamental reasoning limitations, such as the persistent challenges observed in TTP mapping tasks.
Poor rcasoning on cybersccurity-related knowledge. LLMs consistently face difficulties when tasks demand deeper reasoning and inference over ey cybrrsccurity knowledge. Two specific examples highlight this challenge. First, the identification and mapping of MITRE ATT&CK TTPs require LLMs to infer adversarial intent from subtle contextual clues. Current LLMs frequently miss these nuances or produce inaccurate mappings; for instance, GPT-4o achieves only 0.28 precision and 0.23 recall on our benchmark's TTP identification task. Second, generating comprehensive threat-actor profiles demands synthesizing disparate pieces of incident-related evidence into a coherent narrative, Baseline GPT-40 receives notably low scores
9

---

Published in Transactions on Machine Learning Research (11/2025)
from expert judges in this area, averaging just 1.55 for relevance and 1.14 for comprehensiveness on five-point scale. The resulting summaries are often superficial and, in worse cases, contain plausible yet incorrect statements (ballucinations). These shortcomings illustrate that merely optimizing for recall or precision is insufficient. Future rescarch must focus on enhancing the model's capability to reason across multiple documents and to rigorously ground every assertion verifiable evidence, both essential qualities for trustworthy CTT analyses.
4.4 Qualitative Analysis: Illustrating LLM Performance Gaps
In addition to quantitative evaluations, qualitative assessments of LLM-gennrated outputs reveal significant operational shortcomings in realistic threat intelligence scenarios. These limitations underscore the necessity of developing more sophisticated and structured approaches to achieve comprehensive automation of CTI tasks. Specifically, as illustrated in Figure Ø observe two main types of performance gaps: (1) the generation of sparse incomplete content for critical intelligence elements, such threat actor profiles and root cause expleseteppiiiio explanations, when compared to expert expectations; and (2) inaccuracies or hallucinations in technical details, such as misidentifying ToCs or incorrectly classifying MITRE ATT&CK TTPs. These qualitative examples indicate that although LLMs demonstrate foundational competencics, their standalone deployment in complex CTI operations may yield intelligence that is incomplete, imprecise, or potentially m misleading, Therefore, there is a clear need for integrated frameworks capable of supplementing LLMs with with effective knowledge retrioval, domain-specific reasoning, and rigorous verification methods.
Sparse content for threat actor/root cause
LLM generated content
WIRTE, aPT group linked Hamas Gasa Cybergang Qlet malware uses SVG file to perforn IIML proggiing and lecalty assemble a malicious payloas on vierims devices.
Hallucinated loC
LLM Extracted loCs:
hash_sha1: 4
Expected Content from Human Experts
WIRTE Advanced Persssent Threat (APT) group believed be lasked Hamas
Published: 2024-01-01 [In this link, , not included in VT]
and the Gaza Cybergang The group has been active since at least 2018 and primarily targets Middle Eastern governments. financial institutions, and other entities of strategic interett WIRTE known for using spear-phishigg compaigne gain to target systems, often delivering
Truth: A commit id from A page page At Git
malicious Microsoft Office The group employs custom tools and tactics to steal sensitive data, conduct repionage. and maintain persistence on compromised aetworks. Recent activity atributeed to WIRTE includes campaigns leveraging document based malware and advanced evasion techniques to anfiltrate taspets' systems. group focuses oa zatherina intelligence and has bees observed taggtiigg eatities in sectors such as govemmeut, military, and diplomacy within the Middle East and North Africa regions.
96ad4 2tw.
android kernel common
conntt 9cadte759ffaas aet.s
[lag] [ige]
author Benoltt Sevens rbsevensggoogle.coo com-
Thu Nov 07 14:22:02 2
tree fest wtt.eaaeettttttttttttttttttttaaa
Grog
eg' ogg
(d1##)
QDet also known as QukBor, is a banking Trojan and information stealing malware thread
that evolved hijacking, credential theft, and serving ransomware, Originally designed to steal basking credentials, QBot now plays a critical role cybercriminal operations by providing foothold for payloads. Recent activity shows Qut being distributed through phishing campaigus, oftes via malicious esail attachments or embedded link. Theeat actors leverage QBot to gain initaal access to setworks, thes deploy follow-on on malware like ransomarre. malware is known for constautly evolring tactics, incloding using leginimate tools such as PowerShell and exploiting valneraaliies to evade detection. In late 2023 and 2024, law enforcement crackdowns on QBot infisatructur lomporarily disupied in operations, but campaigns have since emsrgee, showeasing resilience and adaptability.
moltifunctioal
capable facilitating loader for other malware, incloding
Hallucinated TTPs
malicious
LLM
X
"T1556.004". Bypass User Account Control
"T1556.004": Modify Authentication
Truth
Process: Network Device Authentication
Illustaative examples of LLM performance gaps: sparse content generation for threat actors/root causes, and hallucination of ToOs and TTPs compared to ground truth or expert expectations.
Figure 2:
Absence of domain expert experience. Figure & (left) highlights a recurring shortcoming of vanilla LLMs when they are asked to provide descriptive context for entities such as threat actors or to explain the root cause of an incident. The LLMs frequently produce brief, high-level statements for example, "WIRTE, an APT group linked to Hamas..' or 'QBot malware relies on SVG files..' Although these remarks may be factually correct, they lack depth. Key dimensions that domain experts expect, historical campaigns, typical victim profiles, and broader strategic impact, are absent. Expert reviewers consistently flag this deficit in their feedback. Typical comments include 'This is generally accurate but sparse; more context
10

---

Published in Transactions on Machine Learning Research (11/2025)
would be helpful," or 'For the root cause, add subsection that provides additional detail on the malware or tool,' when they compare the model output with the richer reference text the right side of Figure B. These observations indicate that, without explicit domain conditioning, current LLMs struggle to synthesize scattered technical evidence into a coherent, multi-layered narrative. Consequently, analysts must still invest substantial effort to expand and validate L.L--generated content before is suitable for professional CTI reporting.
Absence of external ground truth to verify technical details. Internal knowledge encoded within LLMs alone is insufficient to reliably verify technical details, such as IoCs or MITRE ATT&CK TTP map- pings, due the complexity, specificity, and rapidly evolving nature of cybersecurity threats. Accurate verification typically requires authoritative external databases (e.g., Virus'Total) that maintain up-to-date and validated intelligence. Without integrating such external sources, LLMs are prone to hallucinate or misinterpret technical artifacts. The tightight example in Figure & illustrates an LLM extracting commit ID ("96ade...') and misrepresenting it as a SHA1 hash ToC. This type of errors can mislead analysts onto an incorrect investigative path. Similarly, the bottom-right example shows an LLM incorrectly mapping an observed behavior to a MITRE ATT&CK TTP ('T1556.004 004:: Bypass User Account Control') when the correct mapping is different ('T1556.004: Modify Authentication Process: Network Device Authentication" MITRE Corporation (D)). Such inaccuracies in identifying specific IoCs or TTPs undermine the reliability of LLM-generated intelligence.
5
Threat Research Agent (TRA)
5.1 Method
Within the CTT division of the company, we have developed the Threat. Research Agent (TRA), an end-to- end threat research framework that has been fully integrated into the organization's CTI workflow. TRA operates as an iterative, human-in-the-loop agent system designed to improve information inlegration, en- able streamlined verification, and produce structured, actionable intelligence for analysts. Its architecture, illustrated Figure & emphasizes continuous cycle of analysis driven by LLMs and subsequently refined through expert feedback. Specifically, TRA starts with an LLM performing preliminary research on a seed article, generating relevant queries, and retrieving supplementary content from various knowledge sources such public platforms like Bing and Google, as well as proprietary databases. After gathering the relevant information, an LLM-based selector filters these links to ensure they are pertinent to the seed article. This curated content is then assessed by an LLM evaluator lo refine the selection further. The LLM synthesizes the refined information into an initial draft, which is specifically structured for human review. By embedding domain expert feodback into the workflow (via prompt enginecring), TRA dircctly addresses the challenge of lack of domain-expert human experience, ensuring that the generated content is enriched with the depth and context required by human analysts. In addition, TRA integrates external authoritative knowledge sources, such VirusTotal. into the workflow for cross- chess-checiigg and validating LLM outputs. This integration min- imizes hallucinations and ensures that the technical details, such as IoCs and MITRE ATT&CK TTPs, are accurate and reliable, thus enhancing the overall credibility of the threat intelligence.
Embedding domain-expert feedback. The TRA framework is designed to scamlessly integrate domain- expert feedback throughout the entire automated threat research process, as depicted Figure The out draft of TRA, rich with citations and supporting evidence, serves as a dynamic interface where domain experts can easily validate the findings, provide corrections, and suggest further areas of exploration. This iuman--n-the-loogg approach is crucial for ensuring the accuracy, relevance, and completeness of the final intelligence report. As an integral part of this workflow, expert foedback is continuously captured and incorporated into the system, enabling the LLM to refine future outputs. Additionally, multiple verifiers and checkers cross-check the technical details within the report, such as IoCs and TTPs, ensuring they align with authoritative external data sources. This expert-driven iterative cycle improves the overall quality of the intelligence generated, addressing the gaps in LLMs ability to fully synthesize complex cybersocurity information.
11

---

Published in Transactions on Machine Learning Research (11/2025)
Iuput seed article
Related acticles (descrihing
(Driginal the souced]
7 ae 184ttee 5452
[-[Link sounce anl) hoppolbame anderd crmvleen berurig
2ull 02-11
-Linkk to source will bere sias oo frew oo bitiia
Human Feedback for improvement
3025921000ua aid kaves oualited snlamaailli
Accepted
cottlog
Triage
1+] to enunce wil) hepeelleee peppreemmmeterrr comminaad
Sccemtby /e feeeonndd heme ammleesoo soddly appaiiiaaiiie
Accogred article
-liik to scurco will lapolaad in andiulalenn
CVE 1021-51001
Verifiers/ checkers
Incident
Fariched Die (enrichments marked with contoortllekije
LLM Exaluntorr
Kondl Bug Exploite Altacee
LLMY protennnary Ionernptiona
Unique
Bost cause
Suagested liska
LLM Druft
checker
related articles
The coot cazae
an ou--afhound
valterbbility is
(UVC)
Reievanl links
Does relevant the sord article
M.-M-baool Iudger
Well-crffeed
sultiiional
llww physical asaaliin priellegs withot reuuring
eepaished Eaciera bosr
text
driver's fubiliiy sccunidly
Suggested queries
LLM Selector
UVC Vs UNDEFINED
lending fianse bufler siee miscalllaaioss pateutial bobudds
Viruiistal Estal TIF DU LLM chedkes
Valid IoCsTTPS
cunbimeettd Breatt Neares entraned by 12me Kesah.
Rarman The
tunada of Aeredl sarvic 2.226 and dogg 19 124, 344.6 and
Knowledge Sources
refines each sections
Anad susippl 3.6.6
Public
Bing Search
Google
Noo specifed
Threat aeterigrapeemppiig
Progrictary
Virest Intellieence
Eacyclopolii
Malwus
Human Expert Guicance
U.S fadool agrmiee
Start date- - Rad date
Nox specifial
Figure& 3: The illustration of how TRA works
Integrating external ground truth. As mentioned in Section ! a key limitation of LLMs in cyberse- curily analysis is their inability to reliably verily technical details such IoCs and MITRE ATT&CK TTPs. This limitation arises from the internal knowledge constraints of LLMs, which often prevent them from ac- curately handling such tasks on their own. To address this, we integrate authoritative external knowledge sources like VirusTotal into our workflow. The verifier /rciieerchhekke component cross-checks and validates LLM outputs, correcting misidentified ToCs or TTPs, refining generated content, and adjusting priority scores. This process ensures that the information provided by the LLMs aligus with trusted, up-to-date data sources, thereby minimizing the occurrence of hallucinations and enhancing the accuracy of the intelligence.
5.2 TRA Results Analysis
TRA is used by the analysts in our team, and the outputs it generates are reviewed by a team of experts. Their detailed feedback, including corrections and refinements, is systematically collected. The additional feedback captured by TRA further automates threat research. A representative example illustrates this improvement clearly. TRA successfully highlighted that many Mikro devices shipped with hardcoded admin account and blank password that were exploited by threat actors, critical fact initially overlooked by the human analyst. The expert reviewing this case remarked, "I was impressed that the model surfaced the fact that 'Many Mikro devices shipped with a hardcoded admin account with blank password, which was exploited by the threat actors.' I actually read several articles myself but did not surface this detail." * Additionally, experts noted that integrating ToC',, detections, mitigations, and MITRE ATT&CK TTPs into a single, cohcsive report significantly reduced the need for analysts to switch between multiple tools, thus making the final intelligence "more coherent than the original articles.' This feedback underscores TRA's practical value in day-to-day workflows. Detailed examples and expert comments illustrating this iterative feedback loop are provided in Appendix [Е The expert review process not only validates the quality of individual putputs but also generates a valuable dataset of human-labeled examples, facilitating the alignment of the underlying models more closely with expert judgment.
TRA's performance is further rigorously evaluated using the CyberThreat-Ewe benchmark. The quantitative results presented in Table B (Appendix DD) clearly illustrate the benefits of our human-in-the-looe approach. Specifically, TRA improves ToC extraction precision by 26 percentage points across all base LLMs. For the more challenging TTP identification task, TRA raises precision significantly: from 0.28 to 0.42 for 03-mini and from 0.24 to 0.31 for fine-tuned GPT-4o. Additionally, TRA reduces latency for GPT-4o
12

---

Published in Transactions on Machine Learning Research (11/2025)
by approximately 70 seconds, demonstrating gains in both quality and efficiency. The most substantial improvements appear in content generation for threat actor narratives, where expert evaluations of relevance, clarity, and coherence consistently rise above 4.5 on a five-point scale. In practical terms, TRA effectively transforms initial, sparse drafts into content that experts regard as 'publish-ready. Nevertheless, TRA currently retains limitations, as cannot yet. fully automate the entire threal-reseachh workflow without human oversight.
6
Conclusion
This paper introduces CyberThreat-Evar a novel benchmark designed for a holistic, practical, and analyst- centric evaluation of LLMs in automated threat intelligence research. CyberThreat-Eval incorporates tasks closely aligned with real-world analyst workflows and employs human-centric assessments alongside cost- aware performance metrics. Our evaluation reveals that while LLMs can substantially reduce the workload of human analysts, achieving expert-level performance in specific areas remains challenging. In particular, the delicate balance between recall and precision during the triage stage continues to pose difficulties, and tasks requiring complex reasoning, such as identifying and mapping MITRE ATT&CK TTPs, remain sig- nificant hurdles. Furthermore, we introduce the Threat Research Agent (TRA), an end-to-end framework that produces verifalee outputs. Case studies highlight TRA's effectiveness in supporting analysts through automated threat research, and it has been successfully integrated into the company's 's daily CTI workflow. Future work will focus expanding the CyberThreat--EE benchmark and further enhancing the robustness and efficiency of the TRA framework.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprial arXiw:2303.08772 2023.
Scott Ainslie, Dean Thompson, Sean Maynard, and Atif Ahmad. Cyber-threat intelligence for security decision- making: A review and research agenda for practice. Compulers & Security, 132:103352, 2023.
Md Tanvirul Alam, Dipkamal Bhusal, Le Nguyen, and Nidhi Rastogi. Ctibench: A benchmark for evaluating 1lms in cyber threat intelligence. arXiv preprint arXiv:2406.0755 2024.
Dipkamal Bhusal, Md Tanvirul Alam, Le Nguyen, Ashin Mahara, Zachary Lightcap, Rodney Frazier, Romy Fieblinger, Grace Long Torales, Benjamin Blakely, and Nidhi Rastogi. Secure: Benchmarking large language models for cybersecurity. arXiv preprint ar Xix:24005.200 20441, 2024.
Minghao Chen, Kaijie Zhu, Bin Lu, Ding Li, Qingjun Yuan, and Yuefei Zhu. Accr: Automatic atlack technique intelligence extraction based on fine-tuned large language model. Computers & Security, 150:104213, 2025.
Vanessa Clairoux- Trepanier, Isa- May Beauchamp, Estelle Ruellan, Masarah Paquet-Clouston, Olivier Paquette, and Eric Clay. The of large language models (Ilm) for cyber threat intelligence (cti) in cybercrime forums. arXiv preprint arXix:2488.0 03354, 2024.
Richard Fang, Rohan Bindu, Akul Cupta, Qiusi Zhan, and Daniel Kang. Teams of 1lm agents can exploit zero-day vulnerabilities. arXis preprint arXie:2406.0163 2024.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Cpt-1o system card. arXiv preprint 210::2410.21176, 2024.
Hangyuan Ji, Jian Yang, Linzheng Chai, Chaoren Wei, Tiqun Yang, Yunlong Duan, Yunli Wang. Tianzhen Sun, Hongcheng Guo, Tongliang Li, et al. Sevenlim: Benchmarking, eliciting, and enhancing abilities of large language models cyber threat intelligence. arXiv preprint arXio:2405.03444 2024.
Beomjin Jin, Famsoo Kim, Hyunwoo Lee, Elisa Bertino, Doowon Kim, and Hyoungshick Kim. Sharing cyber threat intelligence: Does it really help? In Proceedings of the 31st Annual Network and Distributed System Security Symposium (NDSS), 2024.
13

---

Published in Transactions on Machine Learning Research (11/2025)
Wentao Li, Congcong Wang, Xiaoxiao Cui, Zhi Liu, Wei Guo, and Lizhen Cui. Cosint-agent: A knowledge-driven mullimodal agent for chinese open source intelligence. Xiv preprial arXiv:2503.022: 2025.
Chin-Yew Lin. Rouge: A package for automatic evaluation of sunmaries. In Teat summarizzation branches oud, pp. 74-81, 2004.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 v3 technical report. arXiv preprint arXiu:2412 19487, 2024a.
Xiaoqun Liu, Jiacheng Liang, Qiben Yan, Jiyong Jang, Sicheng Mao, Muchao Ye, Jinyuan Jia, and Zhaohan Xi. Cyber defense reinvented: Large language models as threat intelligence copilots, 2025. URL https://arxiv.org/
abs/2502.28/91
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: Nlg evaluation using gpt-4 with better human aligmment. arXiv preprint UrXie:2903.16600 2023.
Zefang Liu, Jialei Shi, and John F Buford. Cyberbench: A multi-task benchmark for evaluating large language models in cybersecurity. In AAAI 2024 Workshop on Artificial Intelligence for Cyber Security, 2024b.
Microsoft. Corporation. What are indicators of compromise (ioc)? https://www htw.:/rrsstt..cmmeemuussssrrrrrr
unitied unitied-secoss an compoooosee--ooooooooooooieeeeee 1oc 2025a.
compromise-
Microsoft Corporation. How microsoft names threat actors?
htt.:tttrrratttttttaaaaaoo actor-naming 2025b.
MITRE Corporation. Mitre att&cck. https://attack.mtrreorr a.
MITRE Corporation. Mitre att&eck t1536.004.
b.
OpenAT. Openai gpt-4.5 system card. https://cdn.oppnen www.aai...penniioooooo comm 1opp--p---sstemmm---------- 2or 2025.
Wei Peng, Junmei Ding, Wei Wang, Lei Cui, Wei Cai, Zhiyu Hao, and Xiaochun Yun. Ctisum: A new benchmark dataset for cyber threal intelligence summarization. arXie preprinl arXi::2488.06576, 2024.
Filippo Perrina, Francesco Marchiori, Mauro Conti, and Nino Vincenzo Verde. Agir: Automating cyber threat intelligence reporting with natural language generation. In 2023 IEEE International Conference on Big Data (BigData), pp. 8053-3062, IEEE, 2023.
Samaneh Shafee, Alysson Bessani, and Pedro Ferreira, Evaluation of Ilm chatbots for esint-based cyber threat awareness. arXiv preprint arXiv:2401.1122 15127, 2024.
Giuseppe Siracusano, Davide Sanvito, Roberto Gonzalez, Manikantan Srinivasan, Sivakaman Kamatchi, Wataru Takahashi, Masaru Kawakita, Takahiro Kakumaru, and Roberto Bifulco. Time for action: Automated analysis of cyber threat intelligence in the wild. arXin preprint mXXh:2307.10214, 2023.
Norbert Tihanyi, Mohamed Amine Ferrag, Ridhi Jain, and Merouane Debbah. Cybermetric: A benchmark dataset for evaluating large language models knowledge in cybersecurity. orXiv preprint arXiv:2402.0768 2024.
Mohammad Jalili Torkamani, Joey Ng. Nikita Mehrotra, Mahinthan Chandramohan, Padmanabhan Krishnan, and Rahul Purandare. Streamlining security vulnerability triage with large language models. arXie preprinl 2rXth:2501.18000 2025.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models, Xiv preprind ar 230: 13971, 2023.
Damien Van Puyvelde and Fernando Tabárez Rienzi. The rise of open-source intelligence. European, Journal of International Security, pp. 115, 2025. doi: 10.1017/eis.2021.611
Pat Verga, Sehastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis. Replacing judges with juries: Evaluating Ilm generations with panel of diverse models. arXie preprint 2rXiv:2404.1877 2024.
Zongzong Wu, Fengxiao Tang, Ming Zhao, and Yufeng Li. Kgy: Integrating large language models with knowledge graphs for cyber threat intelligence credibility assessment. arXin preprint arXim:2408.0800 2024.
14

---

Published in Transactions on Machine Learning Research (11/2025)
Ming Xu, Honglai Wang, Jiahao Liu, Yun Lin, Chenyang Xu Yingshi Liu, Hoon Wei Lim, and Jin Song Dong. Intelex: A llm-driven atlack-level threat intelligence extraction framework. arXiv preprinl arXio:2412. 10872, 2024.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. XrXiv preprint arXiv:1904.0000 19057, 2019.
Yinghai Zhou, Tang, Ming Yi, Chuanyu Xi, and Hai Lu. Cti view: Apt threat intelligence analysis system. Security and Commusication Networks, (129(1)),,,, 2022.
A Background
A.1 OSINT
Open Source Intelligence (OSINT) refers to the collection and analysis of information from publicly available sources for intelligence purposes. In cyber threat analysis, OSINT encompasses data from open web content, social media, technical blogs, hacker forums, dark web marketplaces, and other public repositories that can reveal IoCs or ad- versary MITRE ATT&CK TTPs MITRE Corporation (a). By leveraging OSINT, organizations can quickly gather IoCs Microsoft Corporation (M1250) (such as IP addresses, domain names, malware signatures, etc.) and strategic insights (e.g., threat actor profiles Microsoft Corporation (2025)) or root causes) that would be difficult to obtain through proprietary sources alone. However, the volume of data increases, effectively harnessing OSINT for threat intelligence presents a significant challenge. Analysts must address issues related to the validity and credibility vast amount of data when using OSINT, along with ethical and legal considerations surrounding the monitoring of public information. Therefore, careful filtering and validation to separate credible signals from noise have become critical tasks.
A.2 LLM for Threat OSINT
LLMs present new opportunities for handling OSINT data at scale. Trained on vast corpora and capable under- standing and generating human language, LLMs are well-suited for tasks like processing unstructured text, extracting insights, and producing summaries or reports. In the OSINT context, researchers have begun to leverage LLMs for various tasks: for example, classifying whether social media posts indicate cybersecurity threats, extracting entities (e.g,, names, IoCs) from technical reports, or translating and summarizing foreign news articles. Recent studies show that several LLM-based chatbots, when applied to Twitter cybersecurity data, can achieve an F1 score of up 0.94 on a threat-related classification task Shatee all (2I2e). LLMs can reliably distill noisy OSINT sources (such as illicit forums) into structured intelligence, greatly accelerating what analysts could do manually Clairoux-Trreanter umal (212)) Beyond classification and extraction, LLMs are also being used to synthesize and generate threat intelligence reports Perrma all (2023). As research this area continues to progress, promising improvements in performance emerging. However, while LLMs offer significant potential for OSINT applications, they are not without limitations. For example, LLMs can produce erroneous or unverified information (hallucinations) and often lack up-to-date knowledge in OSINT all (212H). Thus, in addition simply applying an LLM, domain adap- tation and human oversight are necessary for effective OSINT analysis and generation. Moreover, there is a need for more granular prompt engineering and validation ensure that LLM-driven intelligence is both accurate and trustworthy.
B Scoring Criteria
LLMs accelerate the evaluation process and reduce the need for extensive manual analysis. To better quantify incident priority and further ease the workload for analysts, we introduce a scoring mechanism that maps threat context to numerical values. The underlying principle is that a lower numerical score corresponds to higher priority. A score of 1 indicates the highest-priority category, while a score of 5 indicates that the threat category not considered to exist. Numeric values are assigned based specific attributes of the threat and its target. The scoring criteria are presented in Table E. In the table, the horizontal columns represent the target categories of the incidents, and the vertical rows represent the subject matter categories of the incidents.
15

---

Published in Transactions on Machine Learning Research (11/2025)
Table 5: Priority Scoring for Threat Categories
Threat Category
Unknown/NA
Singular System Singular Company
Singular Country
Multiple Countries
Defacement Spam
Mobile Malware
Malware Updates
New Malware
Vuluerabiity Exploitation (OVE <
2
Cryptominer Resource Hijacking
Phishing Campaign
D-Day Vuluerability Exploitation
Vulnerability Exploitation (CVE 2
Threat Category
Industry/Senne /Sector
Platform Platform/Service SService
Drive-by
ICS(Industrial Control System)
Defacement / Spam
Mobile Malware
Malware Updates
New Malware
Vulnerability Exploitation (CVE
5
Cryptominer / Ressurce Hijacking
2
3
Phishing Campaign
2
O-Day Vulnerability Exploitation
5
Vulnerability Exploitation (CVE 9)
Content Evaluation C
C.1 Deep Search Evaluation
The deep search stage evaluates the LLMs' ability to retrieve relevant external information to enrich the understanding of an incident. A key component of this evaluation involves assessing whether the retrieved URLs offer genuinely new and valuable information beyond that contained in the initial seed article. To quantify this, we employ an LLM-baase comparative analysis. For each URL retrieved by a model (referred to the "comparison blog'), content is systematically compared against the original seed article (the 'reference blog"). The evaluation prompt is as follows:
You are an expert cybersecurity information analysis. are tasked with comparing the content of two blogs: "reference blog" and "comparison blog".
Task: decide whether the comparison blog contains "additional information" beyond the reference blog. "Additional information" any NEW, concrete content that increases understanding, such as:
New facts or data (numbers, dates, CVEs, malware names, actors, tools, geography, ...).
New analysis or interpretations (novel links, causes, trends, forecasts, mi mitigation advice,
...).
Extra technical contextual details (methods, background,
studies, impact elaboration,
..
Any other additional details absent from the "reference blog".
Provide your answer as either "True", (the comparison blog has additional information) or "False" (it does not). the answer "True", briefly justify your decision by listing examples the additional information found in the comparison blog.
Your response MUST be valid JSON format with these fields:
("has_addi aidnallinalliiiniin true/false, "justification"; "Your justification if tisnalll is true"}
C.2 Content Evaluation
Content evaluation criteria. In this subsection, we introduce the criteria used to evaluate the generated content for the threat actor and root cause. These criteria essential to ensure that the generated context is accurate, relevant, and comprehensive. Each criterion helps assess specific aspect of the content's quality. The full of criteria is shown below.
Evaluation details. We evaluate the results using GPT-4o based on the criteria provided for each task (e.g., threat actor, rool cause). The temperature is set to 0.01, and the output format specified JSON. The oulput
16

---

Published in Transactions on Machine Learning Research (11/2025)
template is: "Relevance"; <score?, "Accuracy": <score>, "Comprehensiveness" <score>, "Clarity": <score>, "Coherence": <score>, "Attribution": <score>. For each evaluation method, we calculate the average score for each criterion by taking the mean of the individual scores across all entries.
D TRA Evaluation Results
This section illustrates the detailed TRA results from Section Table presents a comprehensive comparison between vanilla LLMs and their counterparts enhanced with TRA in multiple tasks, including loC extraction, TTP identification, and content generation for the description threats and root canses. key insight that TRA's modular architecture systematically enhances LLM outputs for CTI tasks. For loC extraction, TRA consistently boosts precision over vanilla LLMs (e.g., finetuned GPT-40: 0.9034 TRA vs. 0.8846 vanilla), although with a trade-off in recall and increased operational costs. This highlights TRA's verification strength, especially "hard" articles where it achieved near-perfect precision and high recall. In TTP identification, a more challenging area, TRA generally provides modest precision gains and improve efficiency (e.g., TRA with GPT-4o faster and used fewer tokens than vanilla, GPT-4o). The most significant impact of TRA is observed in content generation quality. For both threat actor and root cause narratives, TRA substantially elevates all six expert-evaluated criteria (relevance, accuracy, comprehensiveness clarity, coherence, attribution) compared to vanilla oulputs across all tested base LLMs. instance, TRA with 03-mini for root cause analysis achieved average scores often exceoding 4.5, underscoring its ability to henthe richer, more accurate, and coherent intelligence. Although small decreases in comprehensiveness are present for some TRA configurations, the overall trend confirms strong capability TRA in producing actionable and reliable threat intelligence.
Table 6: Performance Overview: Vanilla LLMs vs. TRA (with Corresponding Base LLMs) on Key Metrics
GPT-1o
03-mini
GPT-1o (FT)
GPT-4o-mini (FT)
Task Category
Metric
Vanilla
TRA
Vanilla
TRA
Vanilla
TRA
Vanilla
TRA
Precision
0.8240
0.8496
0.8503
0.8864
0.8846
0.9034
0.6944
0.7469
Time (s)
580.13
2285.65
7534.92
9517.04
726.95
1938.19 19
808.67
1966.16
loC Ext.
Tokens
339080
458093
448152
599989
331766
452677
319700
395396
Precision
0.2787
0.3065
0.3480
0.4190
0.2387
0.2508
0.1771
0.2111
Recall
0.2270
0,2163
0.1759
0.1558
0.1846
0,1839
0,1414
0.1358
TTP Identification
Time (s)
817.62
745.63
5097.8 .32
3582.67
2254.21
1614.50
1889.00
1869. 12
Tokens
466426
455265
432503
432576
453282
449729
493860
498774
Relevance
1.547
2.902
3.964
4.229
3.964
4.229
2.475
4.098
Accuracy
1.528
2.738
3.656
3.889
3.655
3,889
2.405
3.685
Comp-
1.145
2.603
3.165
3.631
3.165
3.631
1.755
3.678
Content: Threat Actor
Clarity
2.019
3.776
4.753
4.814
4.752
4.814
3.699
4.678
Coherence
1.734
3.584
4.731
4.810
4.731
4.810
3.405
4.608
Attribution
1.140
2.149
2.968
3.394
2.967
3.394
1.832
3.280
Relevance
3.686
3.737
4.733
4.809
4.533
4.442
3.808
3.889
Accuracy
3.458
3.840
4.627
4.733
4.371
4.467
3.819
3.966
Comp. 3.362
3.147
4.576
4.685
4,286
4.380
3.441
3.344
Content: Root Cause
Clarity
3.932
4.000
4.818
4.897
4.629
4.731
4.163
4.229
Coherence
3.753
3.885
4.800
4.885
4.586
4.711
3.988
4.080
Attribution
3.612
3.907
4.418
4.491
4.323
4.456
3.667
3.768
GPT-40 (FT) refers Finetuned GPP-44-2024-08--00 GPT-4o-mini (FT) refers Finetuned CPTT-44-mimi-202---00 202---77-8-
Comp.: Comprehensiveness Time (s) and Tokens are rounded. Content scores are averages.
ToC Extastand for ToC Extraction.
17

---

Published in Transactions on Machine Learning Research (11/2025)
E Case Study Examples
E.1 Metric Comparision
This example to illustrate why model-centric metrics fall short in analysis. The reference article is about threat.
actor Water Gamayun:
Researchers at Trend Research uncovered campaign by the Russian threat actor Water Gamayun, exploiting zero-day vulnerability the Microsoft Management Console (MMC), tracked as CVE-2025-26636 The group, also known as EncryptHub and Larva-205 208, utilized a technique called MSC EvilTwin to manipulate .usc files and the Multilingual User Interface Path (MUIPath) to download and execute malicious payload, maintain persistence, and steal sensitive data from infected systems. The MSC EvilTwin technique involves creating two .nsc files with the same nameone legitimate and one maliciousand using the MUIPath feature to load the malicious file when the legitimate one is executed. The MSC EvilTwin loader, trojan loader written in PowerShell, was employed to weaponize these techniques. masquerades legitimate software, which then fetches the loader from the attacker's server. The loader mock trusted directories and manipulates whitespace in file paths to trick applications into loading malicions files from alternate locations. The malicious .msc file, containing the attacker's server address, is loaded when the non-malicious .usc file is executed, leading to the execution of commands such as downloading and executing the Rhadamanthys stealer downloader. Water Gamayun's arsenal includes the EncryptHub stealer, DarkWisp backdoor, SilentPrism backdoor, MSC EvilTwin loader, Stealc, and Rhadamanthys stealer. The campaign noted for its active development, employing multiple delivery methods and custom payloads designed to maintain persistence and exfiltrate sensitive data to the attackers servers. Microsoft and Trend Zero Day Initiative (ZDT) disclosed the vulnerability and released patch on March 11 mitigate the threat.
starts with a digitally-signed MSI file that
The summarized context and results are shown as S-1 and S-2. As shown in Table M. despite nearidentical BERTScore and slightly lower ROUGE-L, the longer S-2 is favoured by analysts because the S-2 itself provide additional context. for deeper analysis for the threat actor Water Gamayun and analysts are able to use this context.
E.2 TRA Case Study
This section illustrates some case study examples and more granular feedback comments from senior human experts, showcasing the advantages of our TRA framework and corresponds the Section T. in our main paper: (1) TRA handle cases that contain information ignored by human experts, such the case detailed below in Example B.1: 'I was impressed that the model surfaced the fact that 'Many Mikro'I'k devices shipped with a hardcoded admin account with a blank password, which was exploited by the threat actors.' 1 actually read several articles myself but did not surface this detail." (2) TRA can handle tricky issues like Example B.2.1 and Example B.2.2. And human experts give comments for Example B.2.1: "The Ivanti article that Castlin and just published was Bittle tricky becouse it could be Silk Typhoon but it was only medium confidence. The output did a great job of articulating that and providing information about the suspected groups. I was surprised to see they pulled info about the CVE from another 3rd parly and not CISA directly (they did pull reas from CISA though). And Example B.2.2: It a little bit more convoluted and pulled "loo much" in.. it was a similar siluation where there was suspected overlap but unconfirmed. However, the (miligation) recommenndations pulled in for this one the feedback commments, TRA can handle tricky issues for threat research, and achieve the expert level.(3) TRA performs with better automation and more clear: I asked my team before this meeting. One of the people said that it is actually more clear than the original articles in some cases. Thats great feedback. It feels easier to me having all of the pieces loCs, Detections, Mitigations, MITRE TTPs one place instead of having to click around and switch belween different lools.
really good! Based
18

---

Published in Transactions on Machine Learning Research (11/2025)
Table 7: Metrics vs. human assessment for two Water Gamayun summaries.
Summary
BERT-Score Fi ROUGE-L Fr Human Expert verdict
S1: Researchers uncovered the attack was carried out by the Russian threat actor Water Gamayun, also known as EncryptHub and Larva-208. The group; exploiting a zero-day vulnerability in Microsoft Management Console (MMC), tracked as CVE-2025-26632 utilised technique called MSC EvilTwin to manipulate two .msc files with the same name, legitimate and malicious, and abused the MULPath feature to fetch and execute the malicious file, maintain persistence, and exfiltrate sensitive data. S2: The attack was carried by the Russian threat actor Water Gamayun, also known as EncryptHub and Larva-208. Water Gamayun is Russian threat actor known for sophisticated cyber espionage tactics. Recently, they have been linked to the exploitation of zero-day vulnerability in Microsoft Management Console (MMC), identified as EVE-2025-26632 This vulnerability allows allackers manipulate .use files and the Multilingual User Interface Path (MUIPath) execute malicious code, enabling data theft and persistence on compromised systems. The group employs advanced techniques, such using trojan loader introduce both clean and malicious .msc files, tricking the MMC into executing the harmful version. This method helps them bypass traditional security measures. The MSC EvilTwin loader, written PowerShell, facilitates the download and execution of various malware payloads, including EncryptHub stealer and DarkWisp backdoor, which are designed to steal sensitive information and maintain control over infected systems. Water Gamayun's operations characterized by their ability
0.710
0.342
Sparse / Not actionable
0.715
0.306
Preferred / Actionable
blend malicious activilies with normal system operations, making detection challenging. They have been active since at least April 2024, continuously refining their methods and leveraging legitimate Windows components to execute their payloads stealthily. Microsoft has released security updates to address this vulnerability, and organizations are urged to apply these updates and implement additional security measures to mitigate the threat.
19

---

Published in Transactions on Machine Learning Research (11/2025)
Criteria for Threat Actor
Measures how closely the generated context aligns with the key details of the original article (e.g,, aliases, motivations, tactics).
Relevance:
1: Unrelated or fails to mention key aspects.
2: Limited relevance; misses critical details.
3: Moderately relevant; covers some aspects but lacks depth.
4 Mostly relevant; minor omissions inaccuracies.
5: Highly relevant; fully aligns with the original article.
Accuracy: Assesses the factual correctness of the generated context compared to the original article.
1: Factually incorrect or inconsistent.
2: Contains significant. inaccuracies.
3: Moderately accurate; minor inconsistencies.
4: Mostly accurate; very few errors.
Complctely accurate; perfectly reflects the original article.
Comprehensiveness: Evaluates the extent which the generated context covers all critical details from the original article.
1: Highly incomplete; critical details missing.
2: Covers only minimal details; significant gaps.
3: Moderately comprehensive; some key details missing.
4: Comprehensive; minor omissions.
5: Fully comprehensive; captures all essential details.
Clarity: Mcasures how clear and understandable the generated context is.
1: Poorly written, unclear, and difficult to understand.
2: Significant. clarity issues; partially understandable.
3: Moderately clear; some ambiguities.
4: Mostly clear; minor issues.
5: Perfectly clear; highly readable and easily understandable.
Coherence: Assesses the logical structure and flow of the generated context.
1: Disorganized and difficult to follow.
2: Significant. coherence issues; scattered information.
3: Moderately coherent; inconsistent flow.
4: Mostly coherent; well-organized with minor issues.
5: Fully coherent; logically structured and casy to follow.
Attribution: Evaluates whether the generated context properly attributes information the origi- nal article.
1: Information is unverified or unattributed.
2: Major attribution issues; many details are not clearly linked.
3: Moderately attributable; some details lack clear source references.
4: Mostly attributable; minor gaps in linking information.
Fully attributable; all details are clearly linked the original article.
20

---

Published in Transactions on Machine Learning Research (11/2025)
Criteria for Root Cause
Relevance:
1: Unrelated to the root cause of the incident or fails to mention key aspects the malware's
role,
2: Limited relevance; partial or tangentially related information but misses key aspects.
3: Moderately relevant; covers some important aspects but lacks depth.
4: Mostly relevant; covers most key aspects with minor omissions or inaccuracies.
5: Highly relevant; fully aligns with the incident's 's root cause, addressing all key aspects comprehensively.
Accuracy:
1: Factually incorrect or inconsistent with known details about the malware and the incident 's root cause.
2: Significant inaccuracies or contradictions, though some elements are correct.
3: Moderately accurate; factual but contains minor inconsistencies.
4: Mostly accurate; aligns well with known information with very few errors.
5: Completely accurate; perfectly reflects the root cause of the incident.
Comprehensiveness:
1: Highly incomplete; critical details missing.
2: Covers only minimal details; significant gaps remain.
3: Moderately comprehensive; includes critical details but missos others.
4: Comprehensive; covers most critical details with minor omissions.
5: Fully comprehensive; captures essential details.
Clarity:
1: Poorly written, unclear, and difficult understand.
2: Significant clarity issues; partially understandable.
3: Moderately clear; some ambiguities.
4: Mostly clear; minor issues.
5: Perfectly clear; highly readable and easily understandable.
Coherence:
1: Explanation lacks logical flow.
2: Some coherence, but there are significant gaps.
Generally coherent; some weak links.
4: Mostly coherent; strong logical connection.
5: Fully coherent; clear and logical connection.
Attribution:
1: Completely incorrect attribution.
2: Significant attribution errors; misidentified threat actor.
3: Basic attribution; minor inaccuracies.
4: Mostly correct attribution.
Perfect altribution; clearly identifies the threat actor.
21

---

Published in Transactions on Machine Learning Research (11/2025)
Example B.1: MikroTik Botnet Incident via Misconfigured SPF DNS Records
Source: https:/
Related articles (describing the same threat)
Original URL of the source: http:///wo...oooooo/oo//o//////a
Link to source url: htt.rrreiiattt/////////////i/iia
malspam-operation
Link to source url:
mikrofiikddexied
Enriched Doc (enrichments marked with * *content** (link)) (linten
Incident:
Mikro'Tik Botnet via Misconfigured SPF DNS Records
The root cause of the incident the misconfiguration of SPF (Sender Policy Frame- work) DNS records. Specifically, the SPF records for about 20,000 domains were configured with the overly permissive '+all' option, which allows any server send emails on behalf of those domains. This misconfiguration enabled the threat actors spoof emails and bypass email protections.
Root Cause:
htt/////aa/aaaiinnin//// Threat actor / httgg/roupp/e// campaign: The threat actor is linked to botnet operation involving ap-
proximately 13,000 hijacked MikroTik devices. The command and control (C2) server used in the attack is tied to Russian hackers. [Te botnet also configured the devices as SOCKS4 proxies to launch DDoS attacks, send phishing emails, exfiltrate data, and mask the origin of malicious traffic. The C2 server IP is 62. 60. 137 hosted by Global Connectivity Solutions. The emails imperson- ated DHL and contained a zip file with an obfuscated JavaScript file that exccuted PowerShell script connect the C2 server, The campaign also exploited critical vulnerabilittis in Mikro Tik routers, stemming from outdated firmware misconhgured security settings. The botnet comprises MikroTTik routers with various firmware versions, including recent ones, and exploits remote code execution vulnerability. Intoblox Threat Intel highlighted the use of trojan malware in the campaign. Many Mikro'Tik lik devices shipped with a hardcoded admin account with a blank password, which was exploited by the threat actors.
Drganization/Induns /industryy /location: Not specified
Start date End date: The malspam campaign was active late November 2024.
More information next page...
22

---

Published in Transactions on Machine Learning Research (11/2025)
Example B.1: MikroTik Botnet Incident via Misconfigured SPF DNS Records
MITRE TTPs:
T1071.001: Application Layer Protocol: Web Protocols; Confidence: High. Justification: The report mentions the use of PowerShell scripts to establish a connection to the threat actors command and control (C2) server.
T1566.002: Phishing: Spcarphishing Link; Confidence: Medium. Justification: The report describes the use of malspam emails impersonating DHL Express to deliver fake freight in- voices with a malicious payload.
External Proxy; Confidence: Medium. Justification: The botnet config- ured MikroTik devices SOCKS4 proxies to mask the origin of malicious traffic.
T1090.002: Proxy:
T1072:
Software Deployment Tools; Confidence: Medium. Justification: The use of Power- Shell scripts to deploy malware indicates the use software deployment tools.
T1090.001: Proxy: Internal Proxy; Confidence: Medium. Justification: The botnet's con- figuration as SOCKS4 proxies allowed tens or even hundreds of thousands of compromised machines to use them for nelwork access.
T1059.007: Command and Scripting Interpreter: JavaScript; Confidence: High. Justification: The report mentions the use of an obfuscated JavaScript file to execute a PowerShell script.
T1078:
Valid Accounts; Confidence: High. Justification: The threat actors used stolen credentials to access and exploit Mikro'Tik devices.
T1204.002: User Execution: Malicious File; Confidence: High. Justification: The report mentions the use zip file attachments containing obfuscated JavaScript files.
Impact: The attackers have created botnet of 13,000 Mikro'Tik devices and are conducting spoofing of roughly 20,000 web domains.
Mitigation Steps: Based on OSINT recommendation dictionary (Guidance for Botnets), the recommendations are:
Restrict automatic prompts for non-user-ae inittaid file downloads.
Enable Sale Links protection for links in email messages.
Enable Safe Attachments in block mode.
Enable Zero-hour auto purge (ZAP] in Office 365 to quarantine sent mail in response newly-acquired threat intelligence and retroactively neutralize malicious phishing, spam, malware messages that have already been delivered to mailboxes.
Run endpoint detection and response (EDR) block mode so that Microsoft. Defender for Endpoint can block malicious artifacts, even when your non-Microsoft antivirus does detect the threat when Microsoft. Defender Antivirus is running in passive mode.
IoCs:
IP: 13.133.60.137 Publish date: 2025-01-14 [I this link], Verified via VirusTotal
23

---

Published in Transactions on Machine Learning Research (11/2025)
Example B.2.1: Ivanti Connect Secure Zero-Day Exploitation
Source: hitpss
in-zeno-day-ataacks
Related articles (describing the same threat)
https:
ypn-zerooday--affaks
[link to source url]
.omm
exploileed-in--le-wiidd [link source ur!]
exploited-iin-te-wwill [link to source url
wiz.
wikd link to source url]
http:////...toooooo////////////. linkk to source wrl
hitpps:
https//www..Vautt.cm//booee www.:ivatii.coomceoo
Zta-gateways (Original link with source]
Enriched Doc (enrichments marked with *content*
Incident:
Ivanti Connect Secure Zero-Day Exploitation
The root cause of the incident is critical stack-based buffer overflow vulnerability (CVE- 2025-0282) in Ivanti Connect Secure before version 22.7...5, Ivanti Policy Secure before version 22.7R1.2, and Tvanti Neurons for ZTA gateways before version 22.7R2.3. This vulnerability allows maattennicattd attacker to remotely execute code on affected devices. Additionally, CVE22025-028, a privilege escalation vulnerability, affects Ivanti Connect Secure appliances and is critical to patch as might be exploited conjunction with CVE-2025-0282 for more complex attack chains.
Root Cause:
The specific threat actor or group behind the exploitation of this vul- nerability is not mentioned in the report. However, Ivanti is working with Mandiant, the Microsoft Threat Intelligence Center, external securily partners, and law entorcement agencres to investigate the atlacks. Man- diant suspects the involvement of China-linked espionage groups UNC5337 and UNC5221
Threat actor actor/group/ htt///rruppeeeee
ccampaign::: /ree//rrpoccccainnn
Organization/iine ioanizaationindussryyy /industry/ /location: The victims are organizations using Tvanti Connect Secure appliances. The specific organizations industries affected not disclosed in the report. However, Ivanti mentioned limited number of customers were exploited.
Start date End date: The exact start date of the attacks is not mentioned, but the report published on Jamuary 8, 2025, indicating that the attacks were detected and disclosed around this time. Mandiant observed attackers leveraging the vuluerability since tuid-Decembet.
24

---

Published in Transactions on Machine Learning Research (11/2025)
Example B.2.1: Ivanti Connect Secure Zero-Day Exploitation
MITRE TTPs:
T1203:
Exploitation for Client Execution; Confidence: High. Justification: The report mentions that the vulnerability allows unauthenticated attackers to remotely execute code on devices, which aligns with this TTP.
T1078: Valid Accounts; Confidence: Medium. Justification: The report mentions a second vulnera- bility (CVF-2025-0283) that allows an authenticated local attacker escalate their privileges, which aligns with this TTP.
T1071: Application Layer Protocol; Confidence: Medium. Justification: The report mentions that the attackers used the vulnerability to install malware on appliances, which aligns with this TTP.
Indicator Removal Host; Confidence: High. Justification: The attackers modified iptables rules and remounted the drive as 'read-write' to allow malware deployment, which aligns with this TTP.
T1070:
T1059: Command and Scripting Interpreter; Confidence: High. Justification: The attackers de- ployed web shell to compromised components, which aligns with this TTP.
T1003: Credential Dumping: Confidence: High. Justification: The attackers used the Dryhook malware to capture usernames and passwords during standard authentication processes, which aligns with this TTP.
The exact number of records or devices impacted not specified in the report. However, the exploitation of the vuluerability allowed attackers to install malware on affected appliances. The attackers installed custom malware called [Drylookk' and Phasejam]. Additionally, The SPAWN ecosyslem of malward, including SPAWNANT, SPAWNMOLE, and PAWNSNAIL, was identified on compromised systems. Mitigation Steps:
Impact:
Based OSINT recommendation dictionary (Mitigate zero-day vulnerabilities), the recommendations
Use Microsoft Defender Vulnerability Management to identify and address zero-day vulnerabilities. IoCs:
hash_md5: c7024411555574457755114411111111 Publish date: 2025-01-08 In this link], Verified via VT
bash_mdd: md5: 468891225ccc55000000444440000000 Publish date: 2025-01-08 In this link not in- cluded in VT database
hash md5: d1886225ccc0000cceg0000077711113 Publish date: 2025-01-08 In this link not in-
cluded in VT database
hash md5: Blbb5GGicc44taaD0leecccaaaaaaass Publish date: 2025-01-08 In this link not in-
cluded in VT database
25

---

Published in Transactions on Machine Learning Research (11/2025)
Example B.2.2: EAGERBEE Malware: Updated Arsenal for Attacking ISPs & Government Entities
Source: http:///bbaaceeeccco///eeeereeee ward
Related Articles (Describing the Same Threat)
https: htgp://ww............/eoeeeoeeee [link to source url]
middle
easterm--govt-orgg---ps [link to source url]
https: httpsss//feeldefffcc.ccom /
Eoverunnent-Lrreets [link to source url]
Originnal Link with Source]
htbps:/ htt////aaackkeewss.or..... (012500100 htg.:///..........///.//////aaar
llnkk to source url]
Enriched Doc (Enrichments Marked with * *content* (link))
Incident: EAGERBEE Malware Updated Its Arsenal to Attack ISPs & Government Entities
Root Cause: The root cause of the incident is the exploitation the ProxyLogon vulnerability in Exchange servers (CVE-2021-26855) which allowed attackers deploy the EAGERBEE backdoor. Additionally, the attackers used service manipulation, DLL hijacking techniques, and privilege escalation techniques to load malicious DLLs and maintain persistence. The atlackers deployed service injector ('Isvipsry and the 'ntusersO.datt payload, which leveraged the SessionFny' service to execute. The backdoor appears on the infected system 'dllosder1x64.dd and collects extensive system information, such as the local computers NetBIOS namd, OS details, and network addresses. establishes communication with command -adeaaadd--d--ooooooo- (C2) server, leveraging encrypted pro- tocols like SSL and TLS encryption.
'ssss.dll' manage plugins. Once deployed, the backdoor
Additional context for ProxyLogon, EAGERBEE:
ProxyLogon refers to critical vulnerability (CVE-2021-26855) in Microsoft Exchange Server, which allows attackers to perform server-side request forgery (SSR)) attacks. This vulnerability has been leveraged by multiple threat actors, inchuding state-sponsored groups and financially motivated cybercriminals, to gain horite access to vulnerable Exchange servers, leading to further exploitation and malware deploy- ment. ment.EAGERBEES is sophisticated backdoor malware discovered targeting ISPs and governmental entities in the Middle East. It uses a unique service injector to embed itself into running services, employing a DLL file ("tsviperv.dllly and payload file ("ntusersO.datt) for installation via the SessionEny service. Once in- stalled, it collects system information and establishes a connection with its C2 server, often using SSL/TLS encryption. EAGERBEE continuously operates, retrieving configurations, and using the Plugin Orchestrator module to manage additional plugins for malicious activities, such as file system manipulation, process man- agementt and remote access. The malware injects malicious code into legitimate processes evade detection and shares similarities with the CoughingDown threat group. Some inslances of EAGERBEE were deployed through ProxyLogon vulnerability, highlighting the necessity of patching known exploits. Researchers have observed the use of legitimate Windows services for loading malicious DLLs, complicating attribution efforts.
26

---

Published in Transactions on Machine Learning Research (11/2025)
Example B.2.2: EAGERBEE Malware: Updated Arsenal for Attacking ISPs & Government Entitics
The attack is linked the CoughingDown threat group, also known as (T428,, suggested consistent service creation and C2 domain overlap. The CoughingDown Core Module was also identified. The use of the most recent malware iteration attributed with medium confidence hacking group tracked CoughingDown. EAGERBEF was initially identified by Elastic Security Labs, linked to state-sponsored cyber-espionage group known as REF5961. It was observed in cyber- espionage attacks against Southeast. Asian government agencies and linked to the Chinese nation-backed hacking collective, which Sophos tracked as Crimson Palace. Previous researchers had attributed EAGERBEE Chinese threat group Iron Tiger (APT27), one of numerous groups that often collaborate with other China- backed slate-sponsored actors. Western ISPs and telecommunicationn service providers (TSP) were heavily largeted by PRC-backed groups, including Salt Typhoon, who potentially obtained inteaaaa assomalld willi communication habité of millions of the compromised providers customers. The new variant of EAGERBEE (aka (Chumtais) was observed attacks by Chinese state-aligned threat cluster tracked as Cluster A Alpha which overlaps with groups like Backdoor,, REF5961, Worok, and 'TA428. Backdoor exhibits tactical similarities with another Chinsse-speaing group codenamed CloudComputating (aka Faking Dragon), attributed to multi-plugin malware framework referred QSC.
Threat Caltor/Goou//Caaapai
Organization/Inde /Industry//// Eeristry/Loccaai LLocation: The targeted victims are ISPs and government entities the Middle East and East Asia.
Start Date End Date: The exact start and end dates of the attack are not specified in the report.
MITRE TTPs:
T1059: Command and Scripting Interpreter; Confidence: Medium;; Justification: The use of service injector and manipulation services to load malicious DLLs suggests the use of scripting or command execution.
The exact number of records leaked financial losses is not specified in the report. Mitigation Steps: Based on MDTI profile, we recommends the following mitigations to reduce the impact of this threat:
Impact:
Keep public-facing servers up to date to defend against malicious activity.
Utilize vulnerability management system such as |Mcrosoftt Detender Vuinerability Management,
Turn on cloud-delivered protection in Microsoft Defender Antivirus.
Enable real-Lime protection Microsoft Defender Antivirus.
Implement anomaly detection policies in Microsoft Defender for Cloud Apps.
loCs:
ip: 45.90.58.10 103, Publish date: 2025-01-07 In this link Verified via VT
ip: 185.82.217.164, Publish date: 2025-01-07 In this link
Verified via VT
ip: 62.233.57.94 54,94 Publish date: 2025-01-07 In this link
Verified via VT
ip: 12.11..2,,230 Publish date: 2025-01-07 In this link
Verified VT
domain: www.soccilentertaammennttt.errr Publish date: 2025-01-06 In this link], Verified via VT
27


Reviews:
[{"review_id": "5lDnak5s2d", "reviewer": "Reviewer_LWxd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work introduces a benchmark for cyber threat intelligence that mirrors a real analyst workflow as well as an agentic, human-in-the-loop system (TRA) to execute it. Across commercial and fine-tuned LLMs, they find high recall but noisy precision in triage, consistently weak TTP/IoC extraction, and modest gains in explanatory reporting.\nTRA improves precision via external tools and expert checks but at higher latency/token cost.\n\n### Strength\n1. Automating CTI workflows represents a high-impact and timely application area for LLM-based agents.\n2. The study evaluates the entire CTI pipeline rather than isolating individual components.\n3. Beyond proposing a benchmark, the work implements TRA as a real-world instantiation of agentic methods grounded in the benchmark.\n\n### Weaknesses\n1. The LLM-as-judge framework lacks reported checks for reliability. In particular, measuring the inter-judge agreement would improve the evaluation setup significantly. Without this, the observed gains may reflect judge-specific biases rather than genuine task improvements.\n2. The evaluation metrics do not report the variance across generation seeds, making it hard to judge the robustness of the reported numbers.\n3. The fine-tuning setup is under-specified: the paper does not detail tuning objectives, data curation and volume. The mixed results of fine-tuned models are very interesting but hard to interpret without additional details on how fine-tuning was performed.\n4. The live web-search component is inherently sensitive to web content drift, but the paper does not describe snapshotting, versioning, or archival strategies to stabilize evaluation once the benchmark is public. As a result, the long-term comparability and fairness of the evaluation setup remain unclear.", "claims_and_evidence_yn": "Yes", "claims_explanation": "While there are some weaknesses, I do think the overall results are accurate.", "audience_yn": "Yes", "audience_explanation": "Applying LLM agents to CTI problems is of interest to the TMLR audience.", "requested_changes": "Reporting inter-judge agreement for experiments evaluating the LLM-as-judge would be a critical addition to the paper. Other additions that would improve the work are:\n1. Reported variance across generation seeds.\n2. Additional details on how fine-tuning was done.\n3. A discussion on how the web-search evaluation will be robust in a public version of the benchmark.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kbe1vYygO8", "reviewer": "Reviewer_Dz9a", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Contribution and strengths:\n\n1. The paper addresses a current gap by focusing on real-world analyst workflows rather than artificial benchmark tasks. The three-stage workflow appears well-motivated and grounded in actual practice.\n\n2. Using actual workflow data from a world-leading technology company provides authenticity that synthetic benchmarks lack. The expert annotations add credibility.\n\nWeaknesses:\n\n1. The entire benchmark is derived from one company's workflow, different organizations likely have substantially different CTI processes.\n\n2. Another concern is that the sample size is quite small, this may limit robust evaluation, and limit the applicability.\n\n3. Human expert annotations are treated as gold standard, but inter-annotator agreement isn't reported. CTI analysis involves significant subjectivity, which may also limit the faithfulness of this benchmark.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Yes, the expriments and description support the claims.", "audience_yn": "Yes", "audience_explanation": "The benchmark address some serious gaps in the field, to my best understanding.", "requested_changes": "I hope the authors could address my concerns as following:\n\n1. Would the sample size limit statistical power and generalizability?\n\n2. Would the individual evaluation not be robust and not sufficient for the benchmark?\n\n3. Would LLM-as-judge for content evaluation introduce  bias and inconsistency?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JB0TkDvTG7", "reviewer": "Reviewer_HcGN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Cyber Threat Intelligence (CTI) plays a crucial role in identifying and analysing emerging cybersecurity threats. Analysts performing CTI are often required to process large amounts of data in order to make comprehensive and well-informed reports. With the emergence of Large Language Models (LLMs), the automation of CTI tasks no longer seems infeasible; however, we need well-designed, reliable, and comprehensive benchmarks to evaluate the performance of such systems. There are several existing benchmarks designed for this task, but as the authors point out, they either 1) have unrealistic task formats or 2) propose model-centric, not analyst-centric, performance measures. The authors also point out that (3) current benchmarks do not cover the end-to-end CTI workflow, which they define as three consecutive stages: *triage*, *deep research* and *Threat Intelligence (TI) drafting*. \nTo overcome the above three limitations, the authors propose the CyberThreat-Eval benchmark, a novel framework for evaluating LLMs on realistic (1), expert-annotated (2) end-to-end CTI reporting tasks, which cover all three stages (3). \nThe authors proceed to demonstrate the effectiveness of their benchmark by evaluating different variations of the GPT-4o and o3 families of models. Their results consistently show limitations of current LLMs, which they further demonstrate through detailed case studies in the appendix. \n\nFinally, they describe the high-level architecture of an LLM-based threat research framework, called Threat Research Agent (TRA), which outperforms vanilla LLMs across practically all metrics on the CyberThreat-Eval benchmark.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Claims about the limitations of current benchmarks are well-founded and supported by tables and figures. To my knowledge, they did not leave any existing work out. \nClaims about the CyberThreat-Eval benchmark are well-demonstrated, albeit their evaluation criteria are not always sufficiently described (see W1 below).\nClaims about the TRA are also supported by evidence, and the authors clearly demonstrate extensive testing in this regard", "audience_yn": "Yes", "audience_explanation": "Automation of CTI tasks is a relevant application of LLMs and has the potential to enhance the cyber resilience of many industry actors. However, as the authors rightly point out, this problem is not yet solved. CyberThreat-Eval offers a reliable, realistic and objective metric to evaluate existing and new solutions to this problem.", "requested_changes": "W1: In Section 4.1, the authors mention using the LLM-as-Judge paradigm to evaluate the final outputs of the different language models in their experiments. \nI believe it would benefit the paper if the authors included more details on the exact setup of this evaluation method and what actions they took to minimise the likelihood of incorrect judgments. I would also be interested to know to what extent the generated judgements agree with human expert judgements.\nW2: The \"Data Size\" column in Table 3 is difficult to comprehend. Please name the metrics in each row (eg, Triage: 488 *articles*) \nW3: To me, it was somewhat unclear whether in the Deep Search phase, the task is to evaluate a *URL* or the *article* to which the URL points. Similarly, I find it difficult to understand what \"Avg. URLs processed (w. input)\" means in Table 4. \nMy intuition tells me that the evaluated model first needs to decide whether a given URL should be investigated (URLs processed metric) and then, whether the document behind the URL contains relevant new information (URLs w. additional info metric). However, I do not find a clear description of this in the article. I believe Section 3.3.2 could be improved in this regard.\n\nIn my opinion, all of these modifications would be beneficial to the article, but I leave it to the author's judgment to implement them. I will recommend this article for acceptance as I believe it presents a well-established and novel evaluation benchmark for an important and impactful branch of LLM research.", "broader_impact_concerns": "None", "additional_comments": "None", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)