filename-classifier-ettin68m-v4

Multi-label filename classifier: given only a filename (no file content), predicts which of 114 sensitive-data categories are likely present. Fine-tuned from jhu-clsp/ettin-encoder-68m.

Supersedes CyberDesk/filename-classifier-ettin68m-v3 — v3 was trained with no Spanish data at all (0.477 Spanish micro-F1, masked by a high aggregate since Spanish was a small slice of its test set). v4 adds ~12.5k Spanish records (agent-generated + a downsampled synthetic-generator pool) on top of v3's full training set, kept via an additive-only rebuild: every v3 train/val/test record kept its exact split and labels, only new Spanish records were split and appended — verified by scripts/verify_additive_dataset.py (an earlier rebuild attempt had re-shuffled the whole pool and leaked 51.5% of v3's test set into train; this export is from the corrected rebuild).

Architecture — and why loading it needs modeling.py

Mean-pooled encoder output feeds a single 114-way multi-label classifier head. This is not a plain AutoModelForSequenceClassification checkpoint (transformers can't reconstruct the mean- pooling forward pass from config.json alone), so weights are split as backbone/ (standard HF AutoModel format) + classifier_head.pt, and the model class itself is bundled in this repo as modeling.py — download it alongside the weights (e.g. via snapshot_download, see Usage below). No other repo or private code is required to load or run this model.

Training data

data_v4: 65,328 train / 8,041 val / 8,151 test records. Two label-correctness bugs were found and fixed on this dataset after v4's original build: (1) the Spanish synthetic generator embedded a name in ~35% of records but computed labels independently of that insertion, so 81% of name-containing synthetic Spanish records shipped with no PERSON label; (2) a smaller, cross-language gap where a name embedded only inside an email address's local-part (e.g. kathleen_anderson@example.com) didn't get PERSON either. This checkpoint was trained before the second (smaller) fix — ~1,135 of 81,520 records (1.4%) had a label corrected after this model's training run. Not yet retrained against that patch; flagged here for provenance.

Hyperparameters

Reused from v3's own 40-trial Optuna sweep (this dataset didn't get a fresh sweep): task-lr=3.00e-04, batch-size=128, warmup-ratio=0.200, weight-decay=0.097, scheduler=linear, dropout≈5.8e-05, epochs=20 (early-stopped once gains fell below 1% relative for 3 consecutive evals). Best checkpoint: epoch 9.

Decision threshold

Calibrated against data_v4/val.json (never test), sweep 0.30–0.55. Best: 0.44.

Eval results (data_v4/test.json, n=8,151, threshold=0.44)

Split	P	R	F1 (micro)	macro-F1	Exact match
Overall	0.923	0.882	0.902	0.898	0.802
Spanish only (n=1,249)	0.883	0.817	0.849	0.821	0.626
en/de/it only (n=6,902)	0.933	0.898	0.915	0.916	0.834

For comparison, filename-classifier-ettin68m-v3 scores 0.928 micro-F1 / 0.857 exact-match on its own 100%-en/de/it test set (n=6,902) — v4 trades a small amount of en/de/it performance (0.928 → 0.915 micro-F1 on the same non-Spanish population) for closing the Spanish gap from 0.477 to 0.849. See CyberDesk/filename-classifier-mmbert-small-v4 for a second backbone trained on the identical data, which recovers most of that en/de/it cost while further improving Spanish.

Usage

import json
import sys
import torch
from huggingface_hub import snapshot_download
from transformers import AutoTokenizer

local_dir = snapshot_download("CyberDesk/filename-classifier-ettin68m-v4")
sys.path.insert(0, local_dir)
from modeling import FilenameClassifier  # bundled in this repo, see modeling.py

categories = json.load(open(f"{local_dir}/categories.json"))
tokenizer = AutoTokenizer.from_pretrained(local_dir)
model = FilenameClassifier.load_for_inference(local_dir, num_labels=len(categories))

encoding = tokenizer("q1_2024_ssn_export_backup.csv", return_tensors="pt")
with torch.no_grad():
    logits = model(**encoding)
probs = torch.sigmoid(logits)
predicted = [categories[i] for i, p in enumerate(probs[0]) if p >= 0.44]  # calibrated threshold
print(predicted)