phishing-detector-encoder: legitimate / phishing / spam email classifier trained from scratch

No external pretrained weights were used. The encoder, the tokenizer and every parameter in model.safetensors were trained from random initialisation inside this project, in plain PyTorch, without the transformers library. The only external inputs are the public email corpora listed below. Initialisation of this checkpoint: pretrained:/home/nathank/BITS/dev/Phishing Model/checkpoints/pretrain/large-24L/final.

What it is

A 337,444,867-parameter BERT-style pre-LayerNorm Transformer encoder (24 layers, d_model 1024, 16 heads, FFN 4096, byte-level BPE vocabulary of 32,768, 512 positions) with a CLS-pooled classification head that maps an email (subject + "\n\n" + body) to three classes: 0 = Legitimate, 1 = Phishing, 2 = Spam. The binary verdict shown by the demo is the roll-up Unwanted (phishing or spam) = P(phishing) + P(spam) (config.json["binary_rollup"]). Phishing = Nazario phishing corpus + Nigerian-fraud (advance-fee) corpus + synthetic BITS-style credential/BEC lures; Spam = SpamAssassin and Enron-Spam spam folders; the TREC-05 / TREC-07 / CEAS-08 positives never separated spam from phishing and were trained as 'unwanted' only (probability mass of phishing + spam), never as either class. The architecture is defined in model.py, which ships with the weights. Weights are stored in bf16 safetensors (~675 MB); the header carries the architecture config and the label names as JSON.

How it was trained

1. Masked-language-model pretraining from scratch on the deduplicated Enron email corpus (corbt/enron-emails: 517,401 rows -> 226,273 documents after exact/near-duplicate removal and quote stripping), 15% dynamic masking (80/10/10), sequence packing to 512 tokens, AdamW (betas (0.9, 0.98), weight decay 0.01), peak LR 0.00015, bf16 autocast on one RTX 3090 Ti. Run summary (from the pretraining run's final/config.json):
   • run_name: large-24L
   • status: complete
   • source_checkpoint: step-0009552
   • steps: 9552
   • total_steps_planned: 9552
   • steps_per_epoch: 796
   • epochs_planned: 12
   • epochs_completed: 12.0
   • tokens_seen: 1291124736
   • total_training_seconds: 54185.9
   • total_training_time: 15h 03m 05s
   • final_loss: 1.523698394948786
   • val_loss: 1.9862932266429436
   • val_masked_acc: 0.6291136684837473
   • peak_vram_reserved_gb: 17.939
   • peak_vram_allocated_gb: 17.494
   • micro_batch: 24
   • grad_accumulation: 11
   • effective_batch: 264
   • precision: bf16_autocast
   • optimizer: AdamW(fused) {"lr": 0.00015, "betas": [0.9, 0.98], "eps": 1e-06, "weight_decay": 0.01, "grad_clip": 1.0, "mlm_probability": 0.15, "mask_replace_probs": [0.8, 0.1, 0.1], "schedule": "cosine_to_10pct", "effective_batch_sequences": 256}
   • schedule: {'kind': 'cosine', 'peak_lr': 0.00015, 'warmup_steps': 573, 'total_steps': 9552, 'floor_ratio': 0.1}
   • mean_tokens_per_s: 23889
   • data: {'n_windows': 210368, 'n_val_windows': 1999, 'tokenizer_sha1': '9f50c5e775089d2ca3efc82e100cd6b3396ed7e7', 'n_docs': 226273, 'limit_docs': None, 'window': '[CLS] + 511 stream tokens'}
   • weights_dtype: float32
   • exported_at: 2026-10-09T03:31:14
2. Fine-tuning on puyang2025/seven-phishing-email-datasets with our own split: TREC-05, TREC-07 and CEAS-08 (positives trained as unwanted only), Enron-Spam and SpamAssassin (spam folders = Spam), the Nazario and Nigerian-fraud corpora of kudzaiprichard/aura-phishing-email-corpus (= Phishing) and synthetic BITS-style lures (= Phishing) form train (157,113) / val (17,643), grouped by near-duplicate cluster so no template straddles the two; TREC-06 (16,302 rows, 23.9% positive) is held out entirely as the unseen binary test set and never touched by the tokenizer, pretraining or fine-tuning (1,652 pool rows that near-duplicate TREC-06 templates were removed). A second holdout, test_holdout3, holds clean-provenance three-way rows never seen by the tokenizer or the pretraining corpus for the 3x3 confusion matrix. Counts come from data/processed/splits_manifest.json. Rows with a clean three-way label train with 3-class cross-entropy; TREC / CEAS positives train on the probability mass of {phishing, spam} (partial-label loss: -log(P(phishing) + P(spam))), so they shape the legitimate-vs-unwanted decision without asserting a kind they were never labelled with. Fine-tune dials (from the checkpoint's config.json):
   • run_name: ft-focal
   • init: pretrained:checkpoints/pretrain/large-24L/final
   • n_layers: 24
   • freeze_layers: 0
   • loss: focal
   • focal_gamma: 2.0
   • lr: 3e-05
   • batch_size: 32
   • micro_batch: 16
   • epochs: 2
   • max_steps: None
   • eval_every: 1250
   • log_every: 20
   • eval_batch_size: 64
   • warmup_fraction: 0.06
   • weight_decay: 0.01
   • betas: [0.9, 0.98]
   • eps: 1e-06
   • grad_clip: 1.0
   • patience: 2
   • schedule: linear_decay
   • selection_metric: val_f1_unwanted
   • pooling: cls
   • num_labels: 3
   • synthetic: off
   • limit_rows: None
   • seed: 42
   • cpu: False

Metrics

TREC-06 binary holdout (Unwanted (phishing or spam))

Evaluated by src/evaluate.py on the test split (test_holdout.parquet, 16,302 rows, threshold 0.5). Positive class = Unwanted (phishing or spam), P(unwanted) = P(phishing) + P(spam); these definitions are identical to the binary-era runs, so the numbers are comparable.

metric	value
Recall (unwanted)	0.9741 (101 missed of 3,894)
Precision (unwanted)	0.8794
F1 (unwanted)	0.9243
Accuracy	0.9619
Specificity	0.9581
ROC-AUC	0.9946
PR-AUC	0.9848
Recall at 1% FPR	0.8955
Recall at 0.1% FPR	0.6497
Credential-lure slice recall / precision (n=398)	0.9959 / 0.9758
Latency, batch 1, GPU bf16 (avg / p50)	10.08 ms / 8.9 ms
Latency, batch 1, CPU fp32, 16 threads (avg / p50)	458.57 ms / 445.01 ms
Instructor benchmark	74.0 ms per sample

Recall on the unwanted class is the headline number: in a defensive triage aid a missed phish is the costly error, while a false positive costs a human one glance.

TREC-06 carries no phishing-vs-spam ground truth, so the three-way head can only be described here: of the 3,793 true positives it flagged, the model called 974 phishing and 2,819 spam (informational, not a metric).

Three-way holdout (legitimate / phishing / spam)

Evaluated by src/evaluate.py --split test3 on test_holdout3.parquet (3,220 rows with a full three-way label, threshold 0.5). Decision rule rollup_then_argmax: a row is unwanted iff P(phishing) + P(spam) >= threshold, then the kind is argmax(P(phishing), P(spam)), so this table collapses exactly onto the binary roll-up.

metric	value
Macro-F1 (3 classes)	0.9867
Accuracy	0.9885
Legitimate: precision / recall / F1 (n=1,999)	0.9900 / 0.9935 / 0.9918
Phishing: precision / recall / F1 (n=521)	0.9961 / 0.9866 / 0.9913
Spam: precision / recall / F1 (n=700)	0.9785 / 0.9757 / 0.9771
Phishing-vs-spam accuracy on the 1,221 true unwanted rows (kind decision alone)	0.9959 (4 phishing read as spam, 1 spam read as phishing)
Binary roll-up on this split: recall / precision / F1	0.9836 / 0.9893 / 0.9864

Confusion matrix (rows = true label by corpus provenance, columns = predicted):

true \ predicted	Legitimate	Phishing	Spam
Legitimate	1,986	1	12
Phishing	4	514	3
Spam	16	1	683

source	rows	Legitimate	Phishing	Spam	accuracy
Assassin	236	164	0	72	0.9534
CEAS-08	736	736	0	0	0.9905
Enron	628	0	0	628	0.9841
Nazario	222	0	222	0	0.9865
Nigerian_Fraud	299	0	299	0	0.9866
TREC-07	1,099	1,099	0	0	0.9982

Label provenance. Phishing = Nazario phishing corpus + Nigerian-fraud (advance-fee) corpus + synthetic BITS-style credential/BEC lures; Spam = SpamAssassin and Enron-Spam spam folders; the TREC-05 / TREC-07 / CEAS-08 positives never separated spam from phishing and were trained as 'unwanted' only (probability mass of phishing + spam), never as either class. The three-way holdout is drawn from text the tokenizer never saw and the pretraining corpus does not contain (Nazario / Nigerian-fraud phishing, SpamAssassin / Enron-Spam spam, CEAS-08 / TREC-07 / SpamAssassin legitimate mail from the former validation split). Because the kind labels come from which corpus a message sits in, part of any phishing-vs-spam confusion measures corpus style rather than lure content.

Intended use

A defensive triage aid for small-business inboxes: rank or flag incoming mail for a human to review, surface a probability and the attention-highlighted tokens that drove it, and shorten the time between a phishing wave landing and someone noticing. It is a classroom/final-project model and is not a substitute for a mail gateway.

Limitations

• Era: the labelled data is 1998-2008 spam-track mail (TREC, CEAS, SpamAssassin, Enron), plus Nazario phishing (1995-2022) and Nigerian-fraud mail. Modern lures (OAuth consent phishing, QR codes, MFA fatigue, SaaS-brand impersonation) are under-represented; expect lower recall on them.
• Labels follow corpus provenance: Phishing = Nazario phishing corpus + Nigerian-fraud (advance-fee) corpus + synthetic BITS-style credential/BEC lures; Spam = SpamAssassin and Enron-Spam spam folders; the TREC-05 / TREC-07 / CEAS-08 positives never separated spam from phishing and were trained as 'unwanted' only (probability mass of phishing + spam), never as either class. A phishing-vs-spam confusion therefore partly measures corpus style, and the credential-lure slice remains the closest proxy for phishing-proper recall.
• Length confound: legitimate emails in these corpora average ~2x the tokens of positives. Inputs are truncated to 512 tokens, and the error analysis checks whether errors follow length, but a short legitimate note can still look suspicious.
• English only; byte-level BPE will tokenize other scripts but the model has not seen them.
• Headers are not used (TREC rewrote them); the model sees subject and body text only. Sender reputation, SPF/DKIM and URL reputation are out of scope and should be combined with this score in any real deployment.
• Attention highlights are a signal of what the model attended to, not an explanation of why it decided.

How to load

model.py from this repository is the only code dependency (plain PyTorch).

import json, torch
from safetensors.torch import load_file
from tokenizers import Tokenizer
from model import EncoderConfig, EncoderForClassification   # model.py, shipped with the weights

cfg = json.load(open("config.json"))
model = EncoderForClassification(EncoderConfig.from_dict(cfg["model"]))   # num_labels = 3
model.load_state_dict(load_file("model.safetensors"), strict=True)   # bf16 weights cast into the fp32 module
model.eval()

tok = Tokenizer.from_file("tokenizer.json")
tok.enable_truncation(max_length=cfg["max_seq_len"])
enc = tok.encode("Your account has been suspended\n\nClick here to verify your identity within 24 hours.")
ids = torch.tensor([enc.ids])
mask = torch.ones_like(ids, dtype=torch.bool)
with torch.inference_mode():
    probs = model(ids, mask).float().softmax(-1)[0]
p_unwanted = sum(float(probs[i]) for i in cfg["binary_rollup"]["unwanted"])   # P(phishing) + P(spam)
print("unwanted" if p_unwanted >= 0.5 else "legitimate", {cfg["labels"][str(i)]: round(float(p), 4) for i, p in enumerate(probs)})

On a GPU wrap the forward in torch.autocast("cuda", dtype=torch.bfloat16). Measured single-email latency (batch 1, tokenize + forward + softmax, see the metrics table): GPU bf16 10.08 ms avg on NVIDIA GeForce RTX 3090 Ti, CPU fp32 (16 threads) 458.57 ms avg.

Files

• model.safetensors — bf16 weights, metadata format=pt, dtype=bf16, architecture=EncoderForClassification, config=<json>, labels=<json>
• config.json — model config, num_labels, labels, binary_rollup, label provenance, max_seq_len, export dtype, source checkpoint, fine-tune init, binary metrics (TREC-06), three-way metrics (test_holdout3), pretraining summary (null for scratch init)
• tokenizer.json — byte-level BPE, NFKC, case preserved, specials [PAD]=0 [UNK]=1 [CLS]=2 [SEP]=3 [MASK]=4
• README.md — this card

Provenance

Built for the Skillspire AI & Machine Learning course final project (BITS, 2026). Data decisions, measured VRAM / throughput / latency and every challenge hit along the way are recorded in the project's RUN_LOG.md.