Ricky-dev/halluvision-cyner-xlmr-base

HalluVision fine-tuned CyNER (xlm-roberta-base)

Token-classification checkpoint used to produce the reference entities in

Song, Lee, Kim, Choi, Kim. Measuring Hallucination in Large Language Models for Cyber Threat Intelligence: An Exploratory Study. Computers and Electrical Engineering, 2026 (in press).

It is CyNER (Alam et al., 2022) fine-tuned on 50 MITRE ATT&CK software articles annotated with five STIX-aligned labels. Every entity-based number in the paper (answer-matching precision/recall, frequency-based entity similarity, the Threatpost and The Hacker News ground truth, the Llama-3 summary NER) was produced by this exact checkpoint, so it is released to make those numbers regenerable.

Labels

BIO tagging over five entity types:

id	tag		id	tag
0	B-Malware		6	B-Organization
1	O		7	I-Malware
2	B-Indicator		8	I-Organization
3	I-Indicator		9	B-Vulnerability
4	B-System		10	I-Vulnerability
5	I-System

Label definitions follow Table 3 of the paper (Malware, System, Indicator, Vulnerability, Organization).

Training

Fine-tuned with the CyNER training loop (cyner.TransformersNER) from xlm-roberta-base. Hyperparameters, verbatim from parameter.json:

parameter	value
dataset	dataset_new (MITRE, CoNLL BIO; released as ner_finetune/ on Mendeley Data)
transformers_model	xlm-roberta-base
random_seed	1
lr	5e-6
epochs	30
warmup_step	0
weight_decay	1e-7
batch_size	32
max_seq_length	128
fp16	false
max_grad_norm	1
lower_case	false

Held-out test split: precision 74.69, recall 77.84, F1 76.23 (paper Section 4.2). A manual audit of the checkpoint on 70 Threatpost articles (204 entity judgements, two raters, Cohen's kappa 0.726) is reported in Section 4.8 and released as ner_audit/ on Mendeley Data. Vulnerability is the weakest label (strict precision 0.0 on the audited sample); see the paper before relying on that label.

Usage

The paper's extraction recipe (sentence split, CyNER heuristics on, flair NER on, keep the five target types, order-preserving dedup, drop spans of two characters or fewer) is implemented in scripts/run_cyner_ner.py of the code release:

pip install git+https://github.com/aiforsec/CyNER nltk
huggingface-cli download Ricky-dev/halluvision-cyner-xlmr-base --local-dir cyner_ckpt
CYNER_CKPT=cyner_ckpt python scripts/run_cyner_ner.py \
    --input llm_outputs/llm_results_gpt4o_mini_2503.json --text-key summary_text \
    --output summary_ner_gpt4o_mini.json

Minimal direct use:

import cyner
model = cyner.CyNER(transformer_model="cyner_ckpt", use_heuristic=True, flair_model="ner")
for ent in model.get_entities("Evil Corp pivoted to LockBit ransomware after OFAC sanctions."):
    print(ent.entity_type, ent.text)

Related resources

• Data: Mendeley Data, doi:10.17632/z8sfdnch2c (corpus, LLM outputs, reference entities, per-document metrics, human validation labels, NER audit sheets, fine-tuning data).
• Code: https://github.com/SeungHo0422/HalluVision

License

MIT, matching CyNER and xlm-roberta-base. The MITRE fine-tuning data is derived from publicly available MITRE ATT&CK pages.

Citation

@article{song2026halluvision,
  title   = {Measuring Hallucination in Large Language Models for Cyber Threat Intelligence: An Exploratory Study},
  author  = {Song, Seungho and Lee, Donggyu and Kim, Jaehan and Choi, Jiyoung and Kim, Jinwoo},
  journal = {Computers and Electrical Engineering},
  year    = {2026},
  note    = {in press}
}