Protocol Guardian
Protocol Guardian is a tiny BERT-compatible classifier trained from scratch to rank industrial-control commands as routine or hazardous. It uses SNIP's 512-token byte-level BPE vocabulary but does not inherit language-model weights.
The project also trains a transfer-learning variant from Google's two-layer, 128-hidden-dimension BERT miniature hosted on the Hugging Face Hub.
The final test contains command templates never present in training or validation. This measures phrasing transfer within a synthetic domain rather than memorization of identical strings.
Reproduce
uv run python projects/protocol-guardian/generate_data.py
uv run python projects/protocol-guardian/train.py
uv run python projects/protocol-guardian/train_pretrained.py
uv run python projects/protocol-guardian/train_embedding.py
This is a research classifier, not a substitute for deterministic safety controls, authorization checks, or an engineering review.
Verified results
The benchmark evolved through three increasingly strict template regimes. The final split contains 1,200 examples from templates absent from both training and semantic validation.
| Variant | Held-out accuracy | Precision | Recall | F1 |
|---|---|---|---|---|
| 118K BERT from scratch, first corpus | 65.92% | 97.51% | 32.67% | 0.4894 |
| 118K BERT from scratch, broader corpus | 40.75% | 42.34% | 51.17% | 0.4634 |
| Google two-layer BERT transfer | 50.00% | 0.00% | 0.00% | 0.0000 |
| MiniLM embeddings + linear head | 88.58% | 100.00% | 77.17% | 0.8711 |
The winning head has only 385 fitted parameters over 384-dimensional normalized
embeddings from sentence-transformers/all-MiniLM-L6-v2. Its test confusion matrix
was [[600, 0], [137, 463]]. The failed transformer variants are retained to document
how misleading same-template validation can be.