Bwen-8B-Cyber

A cybersecurity-tuned (red team + blue team + CTF + network forensics) fine-tune of allenai/Bwen-8B, the AI2 Bolmo byte-level retrofit of Qwen3-8B-Base. This is a byte-level language model: max_new_tokens counts bytes, and it tokenizes over raw bytes instead of subwords — which is particularly suited to security artifacts (pcap hex dumps, packet listings, shellcode, obfuscated payloads) where BPE tokenizers are weakest.

Method

Parameter-efficient byte-level SFT following the Bolmo stage-2 recipe, adapted:

• Frozen: the entire global transformer (36 Qwen3 blocks), final norm, lm_head, and the subword-suffix embedding table (622M).
• LoRA (r=32, α=64) on all linear projections of the local byte encoder/decoder (mLSTM q/k/v/gates/out_proj, MLP gate/up/down, in/out projections).
• Fully trained: byte embedding + boundary predictor (~15M + LoRA ≈ 47.5M trainable, 0.57%).
• Losses: L = CE + 0.1 · BCE_boundary. CE over assistant bytes with boundary-symbol-fused targets (vocab 520 = 256 bytes + 256 byte+boundary + specials); boundary BCE supervises the predictor with the correct log-probability formula (p = exp(logprob)), target convention = ends (see below).
• Key correction (empirically validated): the pretrained boundary predictor uses an ends convention — pred[j] ≈ patch_start[j+1] (F1 0.985, off-by-one vs naive starts). Training pools patches with the true-ends mask, identical to the CE fused-flag mask, keeping pooling/targets/boundary-BCE fully consistent. Before this fix, training degraded eval CE (0.276→0.46); after it, eval CE improves monotonically.
• 3000 steps (micro-batch 1 × accum 2), LR 5e-5 cosine, seq ≤ 16384 bytes, global shuffle, single A100-80GB, ~57 min. Baseline acceptance gate before training: boundary BCE 0.0198 (<0.69), ce_oracle 0.2773 ≈ ce_pred 0.2760.

Training results (held-out eval, byte CE on assistant spans)

step	ce_oracle	ce_pred	boundary BCE
0 (baseline)	0.2773	0.2760	0.0198
3000	0.2585	0.2665	0.0169

(-6.8% / -3.4% / -14.6%). Training dashboard: 0xCra/bwen-8b-cyber-sft-trackio.

Evaluation — CyberMetric-500 (MCQ, byte-level letter logprob scoring)

model	plain format	User:/Assistant: template
base Bwen-8B	0.7860	0.3860
Bwen-8B-Cyber	0.6240	0.8360

McNemar (paired, exact): template format 14 (base-only-correct) vs 239 (SFT-only-correct), p<0.0001; plain format 81 vs 0, p<0.0001. In its trained format the model gains +45pp over base and +5pp over the base model's best format. Caveats: at 60-70% accuracy the SE on 500 questions is ≈ ±2pp; the full-vocab argmax almost never lands directly on a bare A/B/C/D for either model (letter mass 0.7-17.5%), so absolute MCQ scores are format-sensitive — within-format paired comparison is the reliable signal. The finetune is format-specialized: it underperforms base on prompts outside its User:/Assistant: template. Use the template below.

Training data — 0xCra/cyber-byte-sft

33,050 rows (~161MB bytes, 3.86 bytes/patch), attack-heavy mix, all Apache-2.0 except CTI (CC-BY-NC-SA):

source	rows	share	role
cyberstrike-sft-120k	14,940	45.2%	red team / offense
CTF-Instruct (cyber-filtered)	2,447	7.4%	CTF
self-built pcap (tshark + dapt-2020)	3,303	10.0%	network forensics
Trendyol v2 + Fenrir v2.1	7,920	24.0%	blue team / defense
threat-intelligence	4,440	13.4%	CTI

Dedup (exact + near), quality gates, literal-\n repair, cyber-keyword filter for CTF. The pcap/Wireshark portion is self-built from real captures (tshark hex dumps, conversation tables, port-scan/beaconing detection, TCP-stream analysis) with programmatically grounded answers — a data type with no existing Hub dataset and where byte-level models have a structural edge over BPE.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("0xCra/Bwen-8B-Cyber", trust_remote_code=True).to("cuda")
tok = AutoTokenizer.from_pretrained("0xCra/Bwen-8B-Cyber", trust_remote_code=True)

prompt = "User: Explain how to detect a SYN-port-scan in a pcap and write a Wireshark display filter for it.\n\nAssistant: "
ids = tok(prompt, return_tensors="pt")["input_ids"].to("cuda")
out = model.generate(ids, max_new_tokens=1024, do_sample=True, temperature=0.1)  # bytes, not tokens!
print(tok.decode(out[0], skip_special_tokens=True))

Limitations

• Base model is English-only, a base LM without safety post-training; this SFT adds no safety filtering (per the source datasets' red-team content, intended for educational/CTF/authorized-pentest use).
• Format-specialized: degraded outside the User:/Assistant: template (see eval above).
• Trained on ~9% of one epoch (3000/33k rows) under a $30 compute budget — more steps/data would likely help.
• MCQ knowledge eval only; agentic CTF (NYU CTF / InterCode) execution benchmarks not run.

Files

• model.safetensors — merged full model (bf16, 16.6GB)
• adapter/ — PEFT LoRA adapter + trained byte embedding/boundary predictor
• training/ — exact training, data-prep and eval scripts used

Citations

Bolmo (base architecture & byteification): arXiv:2512.15586 · BLT: arXiv:2412.09871 · H-Net: arXiv:2507.07955 · Distillation Scaling Laws: arXiv:2502.08606 · CyberMetric: arXiv:2402.07688

Datasets: cyberstrike-sft-120k, Trendyol Cybersecurity v2.0, Fenrir v2.1, CTF-Instruct, threat-intelligence-dataset (see dataset repo for full provenance), DAPT-2020 pcaps.