attack-technique-tagger (experimental v0.1)

Tags a sentence from a threat report with the MITRE ATT&CK techniques it describes (50 techniques, multi-label). Meant as an assistant for CTI analysts and detection engineers who triage reports; it is only moderately accurate (sealed test micro F1 0.59), so use its output as suggestions to review, not as ground truth.

Try it in your browser: a static page that runs the model locally (transformers.js, ONNX). The text you paste never leaves your device.

Usage

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification
repo = "Project-Bastion/attack-technique-tagger"
tok = AutoTokenizer.from_pretrained(repo); model = AutoModelForSequenceClassification.from_pretrained(repo).eval()

sentences = ["The malware dumps credentials from LSASS memory using a Mimikatz-like routine.",
             "It then creates a scheduled task to maintain persistence on the host.",
             "Command and control traffic is sent over HTTPS to a hard-coded domain."]  # sentences of ONE report, in order

def tag(i, threshold=0.2):
    prev = sentences[i - 1] if i > 0 else ""; nxt = sentences[i + 1] if i + 1 < len(sentences) else ""
    ctx = prev[-300:] + " | " + nxt[:300]          # neighbouring sentences are part of the input (the model was trained this way)
    enc = tok(sentences[i], ctx, truncation="longest_first", max_length=256, return_tensors="pt")
    with torch.no_grad(): p = torch.sigmoid(model(**enc).logits[0])
    return [(model.config.id2label[j], round(float(p[j]), 3)) for j in torch.nonzero(p >= threshold).flatten().tolist()]

for i, s in enumerate(sentences): print(s[:60], "->", tag(i))

Details that matter: input is the sentence plus its previous and next sentence from the same report (adding them improved validation F1 slightly); outputs are independent sigmoids over 50 technique ids (see config.json; look any id up at attack.mitre.org/techniques/<ID>); the threshold 0.2 was chosen on validation data and trades precision for recall (raise it for fewer false positives).

Evaluation (sealed test)

Data: the TRAM2 annotated threat-report sentences (Center for Threat-Informed Defense, Apache-2.0): 19,178 sentences from 151 public reports, 4,070 of them labelled with 1 or more of 50 techniques. Split by report (so no report is in two splits): 105 reports train, 20 validation, 26 test (2,920 sentences, 683 with a technique). Two techniques that occur in only 1 and 5 reports were pinned to train, so test covers 48 of the 50. All model and threshold choices (about 15 training runs over 4 base models, learning rates, class weights, context and seeds) used the validation split only. The test split was scored once, on the final model.

System Micro F1 Macro F1 "has any technique" F1
attack-technique-tagger (this model) 0.593 0.537 0.730
TF-IDF + logistic regression (same split) 0.536 0.496 0.672
nanda-rani/TTPXHunter (MIT; best existing open model I found; its confidence threshold tuned on validation) 0.296 0.243 0.499

Micro precision / recall of this model on test: 0.595 / 0.590. Validation micro F1: 0.605 (three seeds: 0.597, 0.606, 0.596). TTPXHunter was trained on a different corpus (ATT&CK procedure examples), covers all 50 of these techniques among its 193 classes, and was run off the shelf; its low score reflects that domain shift as much as model quality. Averaging three seeds scored 0.612 on validation (not shipped).

Training

cisco-ai/SecureBERT2.0-base (Apache-2.0, ModernBERT architecture) fine-tuned with binary cross-entropy (positive-class weight exponent 0.5), lr 6e-5, 12 epochs, max length 256, bf16, one A10G; the epoch with the best validation micro F1 is kept. Code: GitHub Project-Bastion/models, scripts/ttp/.

ONNX / browser

onnx/model.onnx (fp32, 599 MB) is published for transformers.js / onnxruntime. Checked against PyTorch on 200 validation sentences: onnxruntime fp32 differs by <1e-5 in probability; the in-browser code path (run in Node with transformers.js) gives identical decisions at the 0.2 threshold on all 200 sentences, with probabilities identical for 197 of them and within 0.014 on 3 long inputs (the demo works around a transformers.js quirk that drops the closing [SEP] when it truncates; a small residual on very long inputs remains, cause not fully isolated). int8 quantization does not work for this model: four dynamic-quantization recipes kept only 91-94% of decisions, so no int8 file is published. The demo therefore downloads the full 599 MB once.

Known limits

  • 50 techniques only (the TRAM2 label set); sub-technique distinctions are limited to the ones in that set; anything else is invisible to the model.
  • Sentence-level and context-limited; technique labels in the training reports are themselves noisy and sometimes depend on the whole report.
  • Test set is small (26 reports); numbers will move on other report styles, languages (English only) or newer malware reports. Not evaluated on non-report text such as logs or alerts.
  • Moderate accuracy: expect many missed and extra techniques. Do not use it to automate decisions.

License and credits

Apache-2.0. Training data: TRAM2 data from the Center for Threat-Informed Defense TRAM project (Apache-2.0), derived from public threat reports labelled with MITRE ATT&CK techniques (ATT&CK is a trademark of The MITRE Corporation; use per the ATT&CK Terms of Use).

Downloads last month
1
Safetensors
Model size
0.1B params
Tensor type
F32
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for Project-Bastion/attack-technique-tagger

Quantized
(2)
this model

Space using Project-Bastion/attack-technique-tagger 1

Evaluation results

  • micro F1 on TRAM2 sealed test (26 held-out reports
    self-reported
    0.593
  • macro F1 on TRAM2 sealed test (26 held-out reports
    self-reported
    0.537
  • any-technique detection F1 on TRAM2 sealed test (26 held-out reports
    self-reported
    0.730