ScienceSoft/scnsoft-pii-encoder

ScienceSoft PII encoder

A multilingual token-classification model that detects personal data without a fixed format — the kinds that regular expressions cannot find: names of people and organisations, locations, addresses and dates. It covers seven languages (English, German, French, Italian, Spanish, Russian, Arabic), four Arabic registers and Arabizi.

It is the named-entity component of a data-loss-prevention agent that inspects outbound requests to AI assistants on a user's device and redacts personal data before the request leaves it.

Architecture: xlm-roberta-base, fine-tuned on a fully synthetic corpus, exported to ONNX and quantised to int8 — 278 MB, CPU inference.

Labels

PERSON ORGANIZATION LOCATION ADDRESS DATE_TIME (BIO tagging, 11 classes).

Identifiers with a fixed format — email addresses, telephone, payment-card, IBAN and passport numbers — are deliberately not labels. They have a defined structure and usually a checksum, so a regular expression recognises them exactly; a classifier would add a second, less reliable decision about something already decided. The model is trained to leave them unmarked, and the touch rate below measures how well it does so.

Evaluation on PII Bench

PII Bench (piibench) scores detection of the five entity types above in text serialised the way outbound traffic carries it — JSON, XML, SQL, CSV, key=value, log lines, Markdown tables, OCR output — as well as prose. It comprises 3,500 rows, 500 from each of seven third-party datasets. Precision, recall and F1 are computed over characters and are label-agnostic, micro-averaged within each dataset and then averaged across the seven. Live results for every entrant: leaderboard.

Conflict of interest. PII Bench is built and published by ScienceSoft, the developer of this model. Four of its seven datasets (TAB, FactRuEval, MEDDOCAN, IAHLT) informed changes to this model's training corpus; the other three did not. Both facts should be weighed when reading the comparison below.

Comparison with the strongest independent entrant, nvidia/gliner-PII, on the same rows and the same host. Intervals are 95 % percentile bootstrap intervals over rows, resampled within each dataset (1,000 resamples for each model's F1; 2,000 for the paired difference below):

	this model	nvidia/gliner-PII
F1	0.777 [0.769, 0.784]	0.753 [0.746, 0.761]
precision	0.710	0.744
recall	0.878	0.783
touch rate (not ranked; lower is better)	0.021	0.288
latency, median ms per row, batch size 1	20.3 †	140.6
latency, 95th percentile ms per row	33.5 †	199.7

† Measured on the previous release (v15, revision e98225e07225), which has the same architecture and is exported to int8 ONNX the same way; latency has not been re-measured for this build.

The paired difference in average F1 is +0.0233, 95 % CI [+0.0147, +0.0322] (2,000 resamples, seed 0); the interval excludes zero.

Three qualifications belong with that result:

• Recall is higher and precision is lower. For a data-loss-prevention agent a missed entity leaves the device unredacted while a false positive masks a word unnecessarily, so the trade is in the intended direction — but it is a trade, and the F1 average conceals it.
• Latency was measured on a shared workstation under a higher load average for this model (1-minute load 20.7) than for the comparison model (11.1), and for this model on the previous release (†). Absolute milliseconds depend on the host; the ordering is the informative part.
• Touch rate is the share of characters belonging to format-fixed identifiers that a model marks anyway. It is reported, not ranked.

Variability across training seeds

The result above describes this artefact — one set of weights, trained with one random seed — not the training recipe. The recipe was trained with three seeds, and each run was scored through the same production inference engine (the -engine-raw path on the leaderboard):

training seed	F1	Δ F1 vs nvidia/gliner-PII	95 % CI of Δ	CI excludes 0
1234 — this model (main)	0.7725	+0.0192	[+0.0105, +0.0278]	yes
4321 (branch seed-4321)	0.7590	+0.0057	[−0.0028, +0.0145]	no
7777 (branch seed-7777)	0.7543	+0.0011	[−0.0080, +0.0103]	no

Only the released run is separably better than the reference. This model is therefore the upper end of the recipe's observed spread, and another training run of the same recipe would not be expected to reproduce its lead. The weights of the other two runs are published on the branches named above so that this can be checked.

Release checks for this build (v23)

Run on 2026-09-09. Accuracy lines are scored through the production Rust inference engine, not through a Python decode of the same ONNX file. All eight checks passed for both training seeds.

check	seed 1234 (released)	seed 4321	threshold
known defect cases	21/21	21/21	all pass
format-fixed identifiers wrongly marked	0.1 %	0.0 %	≤ 15 %
recall on Russian PII (hivetrace)	0.9429	0.9357	≥ 0.90
F1 cost of int8 vs fp32	−0.0003	+0.0043	≤ 0.02
latency, 95th percentile at 512 tokens, CPU	192.6 ms	188.9 ms	≤ 200 ms
name-origin bias (max F1 gap between origins)	0.0000	0.0000	≤ 0.05
vs previous model, worst language (ru), Δ F1 [95 % CI]	+0.1538 [+0.1012, +0.2082]	+0.1308 [+0.0809, +0.1870]	CI lower bound ≥ −0.02
held-out register frames, engine, filtered F1	0.8870	0.8860	≥ 0.8786

Change from the previous release (v15). One corpus change: training frames in which a definite article or head noun immediately precedes an organisation name and lies outside its span. It targets the most frequent ORGANIZATION error measured on two real-text datasets that were not used to shape training: the previous build started 9–10 % of GermEval's gold organisations and 16 % of TAB's one word too early.

Other evaluations

External datasets (reported, not gated): hivetrace ru 0.8713 · TAB en, including quasi-identifiers, 0.7111 · IAHLT ar 0.5490 · Wojood ar 0.5384. hivetrace is itself synthetic, though expert-annotated.

In-distribution synthetic evaluation (n = 3,300; drawn from the same generator as the training data, so it overstates real-world accuracy): filtered F1 0.9353 (precision 0.9899, recall 0.8864); for the entities the product masks (PERSON and ADDRESS) 0.9795. On held-out register frames: raw F1 0.9438, filtered 0.8870.

Limitations

• Arabic is the weakest language, and the gap is recall — about 0.54–0.55 F1 on the Arabic datasets against 0.87 on Russian. The aggregate score is not uniform across languages.
• Trained only on synthetic data. Entity pools are finite, so scores on generated text overstate real-world ability; the external and PII Bench results are the better guide.
• Arabizi is a reconstruction. No corpus contains it; its conventions (3 = ʿayn, 7 = ḥāʾ, 2 = hamza) are a model of how people type, not observed data.
• DATE_TIME scores 0.000 on the in-distribution file by design. The product keeps a date only when an explicit birth-date cue (born on, DOB: …) is adjacent, and that file contains none. On held-out register frames, where such cues occur, filtered DATE_TIME F1 is 0.4933 at precision 0.9418.

Use

Required files: config.json, tokenizer.json, onnx/model_quantized.onnx.

import json, onnxruntime as ort
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("ScienceSoft/scnsoft-pii-encoder")
cfg = json.load(open("config.json"))
id2label = {int(k): v for k, v in cfg["id2label"].items()}
sess = ort.InferenceSession("onnx/model_quantized.onnx",
                            providers=["CPUExecutionProvider"])

enc = tok(text, return_offsets_mapping=True, return_tensors="np",
          truncation=True, max_length=256)
offsets = enc.pop("offset_mapping")[0]
logits = sess.run(None, {i.name: enc[i.name] for i in sess.get_inputs()})[0]
# argmax -> id2label -> merge BIO spans using `offsets` for character positions

Encode with the special tokens <s> and </s>. The model was trained with them; AutoTokenizer.__call__ adds them by default, whereas encode(..., add_special_tokens=False) does not, and omitting them measurably lowers accuracy.

Provenance

The training corpus is synthetic text produced by this project's own generator; no production traffic and no scraped text is used for training. Entity values inserted into that text come from project-written lists and from three third-party sources: Faker (person, company and city names), libpostal (address vocabulary) and GeoNames (Arabic-script city names) — see Licence. The generator configuration, seeds and library versions are recorded with each run. The organisation-name pool (REAL_ORGS) is off in this build: it was evaluated in an earlier run and cost more accuracy than it added.

Training stack: transformers==4.57.1, tokenizers==0.22.1, torch==2.9.0a0, faker==37.12.0; two epochs.

Licence

Proprietary to ScienceSoft. The weights are published so that the results on this page can be evaluated and reproduced; no licence to use them in production, to modify them or to redistribute them is granted. For licensing, contact ScienceSoft (https://www.scnsoft.com).

Basis for these terms. The model is derived from FacebookAI/xlm-roberta-base (MIT licence), and its training corpus includes entity values from Faker (MIT), libpostal (MIT) and GeoNames (CC BY 4.0). All four licences permit a derivative work to be distributed under proprietary terms, provided their notices are retained and GeoNames is attributed; none is share-alike and none restricts commercial use. The notices and the attribution are in THIRD_PARTY_NOTICES.md.