TensorGreed/brewslm-sensitive-data-tagger

Sensitive-data tagger (PCI / GDPR / HIPAA + custom) span tagger

A token-classification model that finds spans of text and labels each with a type, fine-tuned from microsoft/deberta-v3-base. Trained with BrewSLM (project “Sensitive-data tagger (PCI / GDPR / HIPAA + custom)”, run #143).

Model

• Base model: microsoft/deberta-v3-base
• Task: token classification (BIO tags → character spans)
• Window: 512 tokens (the tokenizer's model_max_length); longer texts are read in overlapping windows

Entity types

30 types: address, api_key, bank_account, card_expiry, credit_card, customer_id, cvv, date_of_birth, device_id, diagnosis, email, employee_id, health_plan_id, iban, ip_address, licence_number, license_plate, medical_record_number, medication, national_id, passport, person_name, phone, provider_name, routing_number, tax_id, ticket_id, transaction_id, username, visit_date. Labels are BIO: O, then B-<type> / I-<type> per type.

Training data

• Training rows: 1620 (1612 with at least one span)
• Validation rows: 202
• Data kinds: synthetic
• Row sources: passage_labels:cloud:anthropic:claude-sonnet-5-5 919; teacher_llm 335; passage_labels:ollama 223; passage_labels:cloud:anthropic:claude-haiku-4-5-20251001 101; document_labels:ollama 32; teacher_llm+regex 10
• Labelled by: cloud:anthropic:claude-sonnet-5-5 1620
• Augmentation: 1616 OCR-noise copies (character noise rate 0.1)
• Epochs: 5
• Seed selection: Picked seed 45 (run #143) of 3 by validation span F1 (tuned thresholds) (higher is better; values: seed 45 0.9467, seed 46 0.9452, seed 47 0.9437); test examples, the answer key and external benchmarks were not used to pick.

Evaluation

Strict span match: a prediction counts only when its type, start and end all equal a reference span.

Answer key (44 texts)

Reference spans written by cloud:anthropic:claude-sonnet-5-5.

Human review: 11 of 44 rows checked by a person (1 corrected). On the checked rows: precision 0.90 · recall 0.95 · F1 0.92 (11 texts).

Overall: precision 0.97 · recall 0.98 · F1 0.97

0 of 24 types have enough reference spans to show a 0.05 F1 difference; the others can only show 0.82 or more (medical_record_number: 1 span, needs 152).

Type	Precision	Recall	F1	Gold spans
address	1.00	1.00	1.00	4
api_key	1.00	1.00	1.00	2
bank_account	0.62	1.00	0.77	5
card_expiry	1.00	1.00	1.00	16
credit_card	1.00	0.91	0.95	23
cvv	1.00	1.00	1.00	9
date_of_birth	0.70	1.00	0.82	7
diagnosis	1.00	1.00	1.00	16
email	1.00	1.00	1.00	27
employee_id	1.00	1.00	1.00	3
health_plan_id	1.00	1.00	1.00	7
iban	1.00	1.00	1.00	7
ip_address	1.00	1.00	1.00	6
medical_record_number	1.00	1.00	1.00	1
medication	0.86	0.90	0.88	20
national_id	1.00	1.00	1.00	13
passport	1.00	1.00	1.00	6
person_name	1.00	1.00	1.00	37
phone	1.00	1.00	1.00	25
provider_name	1.00	1.00	1.00	23
routing_number	0.00	0.00	0.00	3
ticket_id	1.00	1.00	1.00	9
username	1.00	1.00	1.00	1
visit_date	0.94	1.00	0.97	16

Test examples (44 texts)

Overall: precision 0.96 · recall 0.98 · F1 0.97

0 of 24 types have enough reference spans to show a 0.05 F1 difference; the others can only show 1.00 or more (medical_record_number: 1 span, needs 152).

Type	Precision	Recall	F1	Gold spans
address	1.00	1.00	1.00	4
api_key	1.00	1.00	1.00	2
bank_account	0.62	1.00	0.77	5
card_expiry	1.00	1.00	1.00	16
credit_card	0.95	0.87	0.91	23
cvv	1.00	1.00	1.00	9
date_of_birth	0.70	1.00	0.82	7
diagnosis	1.00	1.00	1.00	16
email	1.00	1.00	1.00	27
employee_id	1.00	1.00	1.00	3
health_plan_id	1.00	1.00	1.00	7
iban	1.00	1.00	1.00	7
ip_address	1.00	1.00	1.00	6
medical_record_number	1.00	1.00	1.00	1
medication	0.86	0.90	0.88	20
national_id	1.00	1.00	1.00	13
passport	1.00	1.00	1.00	6
person_name	1.00	1.00	1.00	37
phone	1.00	1.00	1.00	25
provider_name	1.00	1.00	1.00	23
routing_number	0.00	0.00	0.00	2
ticket_id	1.00	1.00	1.00	9
username	1.00	1.00	1.00	1
visit_date	0.94	1.00	0.97	16

External benchmarks

Labelled sets made by someone else, mapped onto this model's types; only the mapped types are scored (a prediction of another type is neither a hit nor a false positive).

Benchmark	Source	Licence	Texts	Types scored	Precision	Recall	F1
CoNLL-2003 people (news, human-labelled)	eriktks/conll2003:test	other	400	person_name	0.93	0.70	0.80
Gretel PII (synthetic, independent)	gretelai/gretel-pii-masking-en-v1:test	apache-2.0	500	address, api_key, bank_account, credit_card, date_of_birth, email, employee_id, health_plan_id, ip_address, license_plate, medical_record_number, national_id, person_name, phone	0.76	0.80	0.78
CoNLL-2003 people, large (validation, 2000 rows)	eriktks/conll2003:validation	other	2000	person_name	0.93	0.74	0.83

Usage

from transformers import pipeline

tagger = pipeline("token-classification", model="path/to/model", aggregation_strategy="simple", stride=128)

def find_spans(text):
    spans = []
    for ent in tagger(text):
        start, end = ent["start"], ent["end"]
        # The tokenizer can attach the preceding space to a span; trim it.
        while start < end and text[start].isspace():
            start += 1
        spans.append({"type": ent["entity_group"], "start": start, "end": end,
                      "text": text[start:end], "score": float(ent["score"])})
    return spans

stride=128 matters: without it the pipeline cuts a text at the 512-token window and never tags anything after it. With it, long texts are read in overlapping windows.

thresholds.json holds per-type minimum scores (all 0.0 = keep every predicted span) and a recall_bias. To favour precision for a type, drop its spans whose score is under that type's value. BrewSLM's redact endpoint and scored decoder read this file; there the score is the span's mean per-token probability of its type, and recall_bias (0–1) is subtracted from each token's O probability so borderline tokens are tagged (more recall, less precision).

Limitations

• Scores come from small held-out sets — 44 texts (answer key), 44 texts (test examples) — so expect a few points of noise either way.
• Types with fewer than 10 reference spans in an evaluation (address, api_key, bank_account, cvv, date_of_birth, employee_id, health_plan_id, iban, ip_address, medical_record_number, passport, routing_number, ticket_id, username) have unreliable per-type scores.
• The training data is synthetic or model-labelled, not human-annotated; real documents with other layouts, formats or languages may score lower.
• It finds what it was trained to find: entity types outside the list above are never tagged.
• Not a compliance guarantee: this model can miss sensitive data and can tag text that isn't. Using it does not by itself make a system PCI DSS, GDPR, HIPAA or CCPA compliant. Use it as one layer, with review and other controls, not as the only safeguard.
• On the Gretel PII benchmark (103 employee_id, 55 license_plate, 17 bank_account reference spans — directional numbers): employee_id precision 0.95 / recall 0.70, license_plate F1 0.69, bank_account precision 0.50 (false positives are mostly routing numbers, device identifiers and timestamps that Gretel leaves unlabelled). Compared with the previous export of this project (run #135), employee_id recall is 0.06 lower and bank_account precision 0.21 lower; employee_id precision is 0.15 higher and license_plate 0.10 higher. Plate false positives are still UK National Insurance numbers, medical record numbers and postcodes.
• Benchmark hygiene: 32 Gretel gold values (identifiers, plates, street names) had been copied verbatim into earlier training rows by the data-seeding prompts (591 occurrences in 248 rows). They were replaced with invented values of the same shape before this run trained, so its Gretel figures are measured on data the model never saw. The previous export's employee_id figure was overstated by an estimated 0.03–0.15 F1 for that reason.