aitrainingdatabroker/hipaa-phi-detector

HIPAA PHI Detector

Finds HIPAA identifiers in healthcare administration text: front desk notes, billing notes, prior authorizations, claims, referrals and patient portal messages. Its labels follow the HIPAA Safe Harbor identifier list (45 CFR 164.514(b)(2)): names, dates, ages over 89, phone and fax numbers, emails, addresses and ZIP codes, Social Security numbers, medical record numbers, health plan member IDs, account numbers, license numbers, vehicle identifiers, device serial numbers, URLs and IP addresses.

Every name, number and address it was trained on is made up. Nothing comes from a real person, patient, customer or business.

Why it exists

HIPAA Safe Harbor removes 18 kinds of identifiers from protected health information (PHI). In practice the hard part is free text: a chart number in a fax note, a member ID in a claim comment, a caregiver's phone number in a scheduling message. This model flags those candidates so a person can review them. Photos, biometrics and other unique codes on the Safe Harbor list are not text and are out of scope.

Use it

from transformers import pipeline

tagger = pipeline("token-classification", model="aitrainingdatabroker/hipaa-phi-detector", aggregation_strategy="simple")
for hit in tagger("Prior authorization for Marisol Abernethy, member ID XQH482915733, was denied; call back at (312) 555-0147."):
    print(hit["entity_group"], hit["word"], round(hit["score"], 3))

How it was trained

Fine-tuned from distilroberta-base for 8 epochs on one RTX 3070. Training data: 1380 invented sentences from source/build.py in this repository, with test sentences built from templates the model never saw in training. The scripts that built the data, trained the model and scored it are in source/; the full scores are in scores.json.

Test scores

Exact entity match on 220 held-out rows: precision 0.829, recall 0.898, F1 0.862.

Type	Precision	Recall	F1	Count
ACCOUNT_NUMBER	1.0	1.0	1.0	25
ADDRESS	1.0	1.0	1.0	50
AGE_OVER_89	1.0	1.0	1.0	25
DATE	1.0	1.0	1.0	50
DEVICE_ID	0.694	1.0	0.82	25
EMAIL	1.0	1.0	1.0	25
FAX	0.194	0.24	0.214	25
HEALTH_PLAN_ID	1.0	1.0	1.0	25
IP_ADDRESS	1.0	1.0	1.0	25
MRN	0.88	0.88	0.88	25
NAME	0.949	1.0	0.974	150
PHONE	0.118	0.16	0.136	25
SSN	1.0	1.0	1.0	25
URL	1.0	1.0	1.0	25
VEHICLE_ID	0.353	0.48	0.407	25

Limits

• Trained only on invented text from templates, with test sentences built from templates it never saw. Real clinic text varies far more.
• Phone and fax numbers look the same, so it often mixes up PHONE and FAX; treat both as one finding. Vehicle identifiers are weak too (see the scores).
• Finding identifiers is only part of Safe Harbor: the rule also requires no actual knowledge that what remains could identify a person. Expert Determination is the other route.
• This model is not legal advice and not a certification. Whether a de-identification method meets HIPAA Safe Harbor or Expert Determination, CCPA and CPRA, GDPR or UK GDPR anonymisation, PIPEDA, Quebec Law 25 or another rule is a question for your counsel and, where a rule requires one, a qualified expert. A person should review its output before any data is shared.

Read more

• Can a medical or dental practice license data to AI?
• HIPAA Safe Harbor: the 18 identifiers for business data
• De-identification for HIPAA, CCPA and GDPR

License and credit

Apache 2.0. Please credit AI Training Data Broker and link to https://aitrainingdatabroker.com/ when you share or build on it.

About

AI Training Data Broker is a broker and technical advisor for companies that sell or license their data to AI labs. It guides the technical side, including export, format, de-identification and compliance, and publishes free checklists on de-identifying data and keeping it within the privacy rules that apply.