CobuildX-AI

/

cbx-pii-extractor-17M

Like 2

Follow

CobuildX AI 10

Evaluation on Indian PII + a small post-processing fix (word snapping)

Hi CobuildX team, thanks for releasing these models. I tested cbx-pii-extractor-17M and -68M (v0.2) on Indian text and wanted to share what I found, along with one small change that closes some leaks without retraining.

Setup

• Test set: 82 short texts, 111 labelled values, all fake. Covers PAN, Aadhaar, voter ID, Indian phone formats, UPI IDs, Indian names and addresses, Hinglish, Telugu-English, Devanagari/Telugu script, and 20 English sentences as a regression check. Labelling follows your v0.2 conventions (full addresses, MRN-style prefixes, order/ticket numbers not PII).
• Scoring: exact = span and label both right. Fully redacted = every character of the value is covered by some prediction, whatever the label (i.e. nothing leaks).
• Run on CPU (Apple M5), --revision v0.2.

Overall

By category (fully redacted %)

Some categories have only 4–9 values, so per-category numbers are indicative; the overall direction is more reliable.

Findings

1. Indian ID formats are not learned at either size. The models usually find the value but the label changes between sizes, which suggests guessing rather than a learned format:

• PAN: CUSTOMER_ID (17M) vs NATIONAL_ID (68M), sometimes missed
• 12-digit Aadhaar: PAYMENT_CARD (17M) vs TAX_ID / PHONE (68M)
• Voter ID: PATIENT_ID (both)
• The word "Aadhaar" itself is sometimes tagged PERSON or ADDRESS (68M)

2. Unspaced 10-digit Indian mobiles are truncated or missed. e.g. 9848012345 → 98480123 on 17M, so redaction leaves 45 visible. 98480 12345 (with a space) gets partly tagged as NATIONAL_ID.

3. Code-mixed text produces false positives. Hindi/Telugu words such as karo, batana, hai, andi, cheppandi get tagged as PERSON, while names like neha, Srinu, Pooja are missed. 68M is much better on Telugu-English than 17M.

4. Native scripts: names in Devanagari and Telugu script are missed by both (expected given English-only training, but worth noting for Indian deployments).

5. Lowercase names in English (17M): send dan the code and cc tom, anna and Lee Chen were all missed by 17M. 68M finds them.

6. Non-PII false positives: a PIN code alone (500081) tagged SECRET, a train number tagged ADDRESS (both sizes).

Root cause of the truncation: partial word tagging

Looking at per-token tags, the model tags some sub-word pieces of a value and not others. pii.py decodes this faithfully, so this is model behaviour, not a decoding bug:

' 98'   B-PHONE
'48'    I-PHONE
'0123'  I-PHONE
'45'    O          <- leaks

'R'     B-PERSON
'avi'   O          <- leaks

Proposed fix: word snapping (post-processing only)

If any piece of a word is tagged, extend the span to the whole word, then merge spans that now overlap (majority label wins). A "word" runs until whitespace or punctuation such as , ; : ( ) " ' < >, and trailing . - / & are trimmed, so emails in brackets, URLs before a full stop and Slack mentions are unaffected.

Results above: +6.3 points fully redacted on 17M, +1.8 on 68M, no new false positives, and no change on the English regression set (incl. emails in <>, URLs with ?&, possessives, hyphenated names). It fixed every truncated mobile number and UPI ID on 17M. The gain is largest on 17M, the size you recommend for CPU.

Known limits: it can't join values split by spaces (98480 12345), it doesn't fix wrong labels, and it makes some existing false positives longer (k → karo).

Happy to open a PR adding it to pii.py behind an option (e.g. join="snap") if useful.

Reproduce

The test set (all fake data), evaluation script and per-value results are ready to share. I'll publish them shortly, or happy to send them directly if you'd like them sooner. With them, everything above reruns with:

python india_eval/run_eval.py cbx-pii-extractor-17M cbx-pii-extractor-68M --snap-words
python india_eval/run_eval.py --tokens 19 36   # per-token tags

Suggestion

An Indian-format training slice (PAN, Aadhaar, voter ID, unspaced mobiles, UPI IDs, Hinglish/Telugu-English chat) would likely fix findings 1–3. I'd be glad to help build it.

Note

I built the test set and scripts with help from Claude, then ran and checked all results myself.