THemidli/applied-ner-stage4-bert-mini-final

Applied NER Stage 4 — Final BERT-Mini

An eight-label English token classifier fine-tuned from prajjwal1/bert-mini. Repository: THemidli/applied-ner-stage4-bert-mini-final.

Results

Exact entity-level seqeval metrics:

Split	Precision	Recall	F1	Token accuracy
Train	0.9989	0.9987	0.9988	0.9998
Test	0.6189	0.6747	0.6456	0.8792

Label	Precision	Recall	F1	Support
PERSON	0.673	0.769	0.718	195
ORGANIZATION	0.470	0.429	0.448	147
LOCATION	0.686	0.748	0.716	143
TIMEDATE	0.873	0.904	0.888	167
PRODUCT	0.316	0.339	0.327	127
WORKOFART	0.368	0.546	0.440	97
JOB	0.810	0.859	0.833	99
AMOUNT	0.725	0.733	0.729	101

On 40 fresh, manually gold-labeled wild probes, exact span F1 was 0.7685 (precision 0.7429, recall 0.7959). Test F1 changed by +0.1771 versus Stage 3.

On the MPS backend, repeated runs with identical seed and config showed ±0.01–0.015 F1 variation (0.6331 vs 0.6455 across the two recorded runs); the seed is fixed, and the variation does not change the model ranking (both runs far above BERT-Tiny, slightly below ELECTRA-Small).

Training

• Dataset: THemidli/applied-ner-stage4-final
• Seed: 20260802
• Hardware: Apple MPS (macOS-27.0-arm64-arm-64bit)
• Runtime: 32.908 seconds
• Records/chunks: 841/865 train; 159/165 test
• Maximum length: 256; fast-tokenizer overflow chunks, no overlapping stride
• Hyperparameters: {"attention_dropout": 0.1, "classifier_dropout": 0.1, "epochs": 16, "eval_batch_size": 64, "hidden_dropout": 0.1, "label_smoothing_factor": 0.0, "learning_rate": 0.0005, "scheduler": "linear", "train_batch_size": 32, "warmup_steps": 45, "weight_decay": 0.02}
• No validation split and no test-driven checkpoint selection

Footprint and CPU benchmark

• Parameters: 11,109,137 (44.44 MB tensor storage)
• Saved artifact: 45.16 MB
• Model-load RSS delta: 32.29 MB
• End-to-end inference RSS delta: 58.52 MB
• CPU throughput: 4141.6 examples/s at batch 32 with 8 threads
• Mean latency: 0.2415 ms/example at that batch size

The benchmark covers tokenizer plus PyTorch CPU forward pass over 40 short probes, repeated 50 times. It is workload- and hardware-specific, not single-request latency.

Labels

PERSON, ORGANIZATION, LOCATION, TIMEDATE, PRODUCT, WORKOFART, JOB, AMOUNT using BIO encoding.

Limitations

This is an 11.11M-parameter uncased four-layer BERT trained on a small, heterogeneous dataset with controlled template additions. It is not a production privacy system. PRODUCT, WORKOFART, ORGANIZATION, title boundaries, and contextual site readings remain weak. The 40-probe wild set is diagnostic, not a population benchmark.