mindtype-tagger-v2.0
MindType's own grammar and spelling correction model, trained from scratch by Mindscend for an on-device keyboard (iOS and Android). Text never leaves the phone.
- 26.2M parameters, encoder-only transformer (8 layers, d=384), own 16k byte-level BPE tokenizer.
- Predicts one edit label per word (KEEP / DELETE / REPLACE_x / APPEND_x / case) in a single pass, so it is fast, tiny, and cannot invent words or rewrite correct text.
- Core ML package (8-bit weights) for iOS 18+, and ONNX (int8) for Android and servers.
What changed since v1.5/v1.6: training now includes phone-typing slips (fat-finger letters, short words, dropped letters), 40% already-correct text, and more casual messages. v2.0 is a weight average ("model soup") of three fine-tunes that share one initialization and differ only in data order, which removes most of the run-to-run luck. Use it with KEEP bias 3 (the default "Balanced" in MindType; 4 = "Only sure fixes").
| v1.6 (KEEP 2) | v2.0 (KEEP 3) | v2.0 (KEEP 4) | |
|---|---|---|---|
| Phone-typo set F0.5 | 0.689 | 0.912 | 0.910 |
| Phone typos fixed exactly | 22% | 77% | 74% |
| Changes to correct real messages (5,556 held out) | 1.64% | 1.26% | 0.90% |
| Evaluation suite F0.5 / false positives | 0.802 / 2.8% | 0.803 / 1.9% | 0.801 / 1.0% |
Results (MindType evaluation suite, 4,356 held-out cases, KEEP bias 3, plausibility guard)
| Set | n | F0.5 | Exact | False-positive rate | Lost protected spans |
|---|---|---|---|---|---|
| casual_clean | 118 | 0.000 | 0.983 | 0.017 | 0.000 |
| casual_noisy | 109 | 0.942 | 0.807 | 0.000 | 0.000 |
| coedit | 579 | 0.453 | 0.109 | 0.000 | 0.000 |
| formal_clean | 1000 | 0.000 | 0.976 | 0.024 | 0.000 |
| formal_noisy | 1000 | 0.876 | 0.655 | 0.000 | 0.000 |
| golden | 50 | 0.927 | 0.860 | 0.077 | 0.000 |
| preserve | 1000 | 0.838 | 0.646 | 0.000 | 0.000 |
| preserve_clean | 500 | 0.000 | 0.992 | 0.008 | 0.000 |
| overall | 4356 | 0.803 | 0.708 | 0.019 | 0.000 |
Training
- Pretraining: masked language modeling on ~20M sentences from FineWeb-Edu (ODC-BY).
- Correction: 12M (noisy, clean) pairs: synthetic typing and phone errors, CoEdIT GEC (Apache-2.0), casual messages written by a local open-weight model and filtered for correctness; 40% are already-correct text. 45,000 steps, three data orders from one shared head initialization, then the weights are averaged.
Hardware: one RTX 3080 Ti laptop GPU. log.jsonl has the three fine-tunes' validation curves.
Usage
Each word gets one label; apply the labels to the words (up to three passes), adding the KEEP bias to the KEEP
logit first. labels.json lists the labels, config.json the architecture; the Core ML and ONNX folders take
input_ids + attention_mask and return logits (see parity.json for reference outputs).
Inference settings used for the results: {"keep_bias": 3.0, "min_error_prob": 0.0}, with the plausibility guard
(plausible_pairs.json: replacements a typo could not have produced are skipped).
Limitations
English only, sentence-level, experimental. Fixes what's there; it doesn't do rewording or tone changes.
License
Apache-2.0 (weights, tokenizer, code). Training data licenses: FineWeb-Edu ODC-BY, CoEdIT Apache-2.0.
- Downloads last month
- 23