CDT5058/phi-redact-distilbert-teacher

phi-redact — compact English PII/PHI redaction model

phi-redact is an 11M-parameter BERT-mini token classifier for detecting PII and PHI in English text. It covers 29 entity types with BIOES labels and is designed for on-device deployment (ONNX, Core ML, LiteRT) where data must not leave the device.

It was knowledge-distilled from a 66M-parameter DistilBERT teacher and trained on ~157k documents spanning clinical notes, government benefits text, student essays, and Wikipedia-style prose.

Not a certified de-identification solution. This model assists redaction workflows. It is not certified under HIPAA Expert Determination or Safe Harbor methods. Evaluate it on your own data before relying on it in compliance-sensitive contexts. English only.

---

Model details

Property	Value
Architecture	BERT-mini (google/bert_uncased_L-4_H-256_A-4), 4L/256H/4A
Parameters	11.13M
Labels	121 BIOES tags (30 entity types × 4 positions + O)
Max sequence length	256 tokens
Vocabulary	30,522 WordPiece (bert-base-uncased)
fp32 size	44.5 MB
fp16 size	22.3 MB
Est. 4-bit Core ML	~5.6 MB
Est. int8 LiteRT	~11.2 MB
Training method	Knowledge distillation (α=0.7) from distilbert-base-uncased teacher
Language	English only

---

Entity types (30)

NAME_PATIENT · NAME_FAMILY · NAME_PROVIDER · DATE · ADDRESS · PHONE · EMAIL · SSN · MRN · HEALTHPLAN · ACCOUNT · IDNUM · LICENSE · EMPLOYER · PROFESSION · AGE · CITY · STATE · COUNTRY · ZIP · URL · IP · DEVICE · ROUTING_NUMBER · IBAN · CREDITCARD · HOSPITAL · FAX · PASSPORT · VIN

Note on TAX_ID: included in the schema design but not in any commercial-safe training source for v7 (ai4privacy excluded). TAX_ID predictions will be absent. Fix queued for v8 (add EIN templates to gen_synth.py).

---

Benchmark results (v7)

Strict span-level scoring: a prediction counts only if both the character boundaries and the label match exactly.

Standard benchmarks

Benchmark	Recall	F1	Notes
WikiANN (EN, name recall)	84.3%	0.775	CC-BY-SA 3.0; direct comparison with Desert Ant ✓
FewNERD (EN, name recall)	85.9%	0.782	CC-BY-SA 4.0; Wikipedia-prose names
LA essays (Micro F1 / Recall)	0.757 / 65.9%	—	CC-BY 4.0; informal student prose
govsynth (government benefits)	100%	1.000	Synthetic; primary deployment domain
Composite (5-benchmark recall avg)	86.1%	—	See note below

Composite = (WikiANN + FewNERD + LA essays recall + govsynth + MultiNERD†) / 5.

†MultiNERD (CC-BY-NC-SA 4.0) is included for completeness: v7 recall = 94.3%, within −0.6 pt of Desert Ant's 94.9% on the same dataset.

Nemotron-PII structured benchmark (CC-BY 4.0)

First valid run — 13,134 entities across 25 entity types in 2,000 test documents. MICRO F1 = 0.860, Recall = 0.843.

Selected per-entity F1 scores:

Entity	F1	Entity	F1
NAME_PATIENT	0.944	SSN	0.908
NAME_FAMILY	0.937	EMAIL	0.905
MRN	0.951	CREDITCARD	0.905
ADDRESS	0.949	IDNUM	0.908
LICENSE	0.920	DATE	0.859
COUNTRY	0.919	URL	0.828
CITY	0.904	DEVICE	0.791
ROUTING_NUMBER	0.876	PROFESSION	0.494

Known weak spot — PROFESSION (F1=0.494, Precision=0.413): the model over-fires on job-title nouns in non-PII context ("She saw a doctor" vs "Occupation: Doctor"). Not a concern for most PHI/PII redaction use cases where PROFESSION is rarely the sensitive entity, but flag it if your data is occupation-heavy.

URL in informal prose: Nemotron URL F1=0.828 (formal text) but LA essays URL recall=0.318 (informal prose). The model handles URLs well in structured documents; it misses them in casual first-person writing.

Comparison to Desert Ant Redact v0.4.0 (English)

Metric	phi-redact v7	Desert Ant (EN)
WikiANN name recall	84.3%	69.5%
MultiNERD name recall	94.3%	94.9%
Composite (5 benchmarks)	86.1%	86.5%
Parameters	11.13M	23M
Est. 4-bit on-device size	~5.6 MB	11.6 MB
Languages	English only	27 languages
License	Apache-2.0	Source-available

phi-redact v7 is essentially tied with Desert Ant's English composite at half the parameters and half the on-device footprint.

---

Training data

All sources are commercial-safe (CC-BY 4.0 or CC-BY-SA 4.0). No real patient data was used.

Source	Size	License	Entity coverage
Synthetic clinical notes (gen_synth.py)	20k	Synthetic (ours)	Full 29-type schema, clinical PHI
Learning Agency Lab PII (Kaggle)	~5,400	CC-BY 4.0	Names, emails, URLs in student essays
govsynth (synthetic govt. benefits)	~20k	Synthetic (ours)	SNAP/Medicaid/WIC case management PII
essay_names.jsonl (synthetic prose)	10k	Synthetic (ours)	Names in biography, news, legal, informal prose
FewNERD train split (DFKI-SLT/few-nerd)	50k	CC-BY-SA 4.0	Wikipedia person entities
WikiANN train split (EN)	~20k	CC-BY-SA 3.0	Wikipedia person, location, org
WNUT-17 train split	~3.4k	CC-BY 4.0	Emerging entities in social media
Nemotron-PII train split (nvidia)	50k	CC-BY 4.0	30 PII/PHI types in synthetic documents

Excluded: ai4privacy/pii-masking-400k (non-commercial license — not used in training or evaluation). MultiNERD (CC-BY-NC-SA 4.0 — evaluation only, not in training corpus).

---

Training procedure

Teacher (distilbert-base-uncased, 66M params):

epochs 10, batch 32, lr 3e-5, cosine LR, label-smoothing 0.05, device A100

Student (this model — BERT-mini, 11.13M params):

epochs 60, batch 64, lr 5e-5, cosine LR, label-smoothing 0.05,
distill-alpha 0.7, device A100 SXM4-40GB (~6h)

Knowledge distillation loss = 0.7 × KL(teacher logits ∥ student logits) + 0.3 × cross-entropy.

---

How to use

from transformers import AutoTokenizer, AutoModelForTokenClassification
import torch

tokenizer = AutoTokenizer.from_pretrained("CDT5058/phi-redact-distilbert-teacher")
model = AutoModelForTokenClassification.from_pretrained("CDT5058/phi-redact-distilbert-teacher")
model.eval()

text = "Patient Maria Garcia, DOB 1985-03-22, MRN 4471829, called 415-555-0189."

enc = tokenizer(text, return_tensors="pt", truncation=True, max_length=256)
with torch.no_grad():
    logits = model(**enc).logits          # [1, seq, 117]

pred_ids = logits.argmax(-1)[0].tolist()  # [seq]
id2label = model.config.id2label

# Decode BIOES tags to character spans
tokens    = tokenizer.convert_ids_to_tokens(enc["input_ids"][0])
offsets   = enc.encodings[0].offsets      # (char_start, char_end) per token

spans = []
span_start, span_label = None, None
for i, tag_id in enumerate(pred_ids):
    tag = id2label[tag_id]
    prefix, *label_parts = tag.split("-") if "-" in tag else (tag, [])
    label = "-".join(label_parts)
    if prefix == "S":
        spans.append((offsets[i][0], offsets[i][1], label))
    elif prefix == "B":
        span_start, span_label = offsets[i][0], label
    elif prefix == "E" and span_start is not None:
        spans.append((span_start, offsets[i][1], span_label))
        span_start = None
    elif prefix in ("O", "B"):
        span_start = None  # reset on unexpected transition

for start, end, label in spans:
    print(f"[{start}:{end}] {label!r:20s}  {text[start:end]!r}")

For production use, layer the deterministic rules.py (Luhn, ABA routing checksum, SSN, IBAN, regex patterns) on top — it wins on structured identifiers regardless of model confidence.

---

Limitations

• English only. The bert-base-uncased WordPiece vocabulary has no meaningful cross-lingual representations. French, Spanish, or other languages will produce silent misses — not safe degradation.
• PROFESSION over-fires (F1=0.494 on Nemotron). Treat PROFESSION predictions with lower confidence or disable the entity type if not needed.
• URL in informal prose (recall=0.318 on student essays vs 0.828 on formal text). Check URL recall on your specific document type.
• Trained on synthetic data. Real-world typos, OCR noise, and unusual formatting will degrade performance relative to the benchmarks above.
• Not a certified de-identification system. Does not satisfy HIPAA Expert Determination or Safe Harbor. Use as a detection aid alongside human review.

---

License

Apache-2.0. Training data licenses: CC-BY 4.0 (Nemotron-PII, LA essays, WNUT-17), CC-BY-SA 4.0 (FewNERD), CC-BY-SA 3.0 (WikiANN). CC-BY-SA ShareAlike requirements apply to the datasets, not the model weights, but consult your legal counsel for commercial deployment.

---

Card generated 2026-09-25 · Repo: CDT5058/phi-redact-distilbert-teacher