abhiram3040/simplewords-dictation-cleanup-v3

Post-ASR dictation cleanup for Pomvox. QLoRA fine-tune of mlx-community/Qwen3.5-2B-MLX-4bit, fused to bf16 and re-quantized to 8-bit.

• Fused model: abhiram3040/simplewords-dictation-cleanup-v3
• Adapter (separate repo, must NOT live in the fused repo): abhiram3040/simplewords-dictation-cleanup-v3-adapter
• Selected checkpoint: iter 14500 of 19,590 (3 epochs over 52,239 rows, batch 8, rank 16, scale 2.0, dropout 0.05, lr 1e-4 cosine, seed 42)
• Prompt: frozen system_v2.txt, shipped at the repo root with no leading or trailing whitespace. One user turn f"{SYSTEM}\n\n{raw}", one assistant turn, no system role, greedy, enable_thinking=False.

Where this model runs

This is the cleanup stage of Pomvox — free, MIT-licensed, fully on-device dictation for Apple Silicon. Hold a key, speak, and clean text lands at your cursor: Parakeet TDT on the Neural Engine transcribes, this model rewrites. It is the app's default cleanup model on 16 GB+ Macs.

· pomvox.ai · source (MIT)

# config.toml — use this model inside Pomvox
[cleanup]
model = "abhiram3040/simplewords-dictation-cleanup-v3"

Standalone, the frozen prompt must be applied exactly as trained — one user turn, no system role:

from mlx_lm import load, generate
from mlx_lm.sample_utils import make_sampler
model, tok = load("abhiram3040/simplewords-dictation-cleanup-v3")
SYSTEM = open("system_v2.txt").read().strip()   # shipped in this repo
p = tok.apply_chat_template([{"role": "user", "content": f"{SYSTEM}\n\n{raw}"}],
                            add_generation_prompt=True, enable_thinking=False,
                            tokenize=False)
out = generate(model, tok, prompt=p, max_tokens=320,
               sampler=make_sampler(temp=0.0))   # greedy

Prompt compatibility. This model ships system_v2.txt and uses it unchanged from v2 — the prompt is frozen on purpose so the corpus is the only variable across versions. Do not substitute v1's prompt, and do not retype it: load the file from this repo. There is no system_v3.txt.

Training data

Fully synthetic, generated and validated in a private corpus repo — no user dictation, no scraped speech. 52,239 training rows after de-leaking: the v2 behavioral corpus plus 3,900 additive rows covering cross-sentence corrections and interrogative-phrase corrections. Every row passes slot, preservation, no-content-loss/no-invention, production-guard and format gates at generation time; violations are discarded, never repaired. The only third-party data is Disfl-QA (CC BY 4.0), used strictly as a held-out evaluation set and never trained on.

What changed from v2

v2 handled intra-sentence self-corrections and failed cross-sentence ones — it emitted the superseded value as final. v3.1 fixes that, and additionally fixes a defect v2's eval set could not see: corrections that replace the interrogative phrase itself (Where are ... or what city has ...), which are 66% of real human disfluency and were 0% of the v1/v2/v3 training corpus.

Release gate

Gate	Threshold	v3.1	v2
cross-sentence self-correction (A/B/C/NAMED/G)	100%	PASS — 100.0% (122/122)	19.5% (15/77)
intra-sentence (regression.jsonl)	≥95%, no regression	MISS — 97.7% (43/44)	100.0% (44/44)
negative cases (D)	zero violations	PASS — 100.0% (30/30)	83.3% (25/30)
ordinary speech (F)	zero guard rejections	PASS — 100.0% (25/25)	100.0% (25/25)
out-of-distribution (Disfl-QA)	diagnostic	77.0% (231/300)	9.0% (27/300)

Shipped with a known gate miss — stated, not buried.

intra-sentence (regression.jsonl) is 43/44 against v2's 44/44. It clears the ≥95% threshold (97.7%) but fails the no regression clause by one row: reg_003, a chained triple correction in the v2-era lowercase/unpunctuated shape. That row is anti-correlated with the cross-sentence gate across the whole run — it passed at 7 of 30 checkpoints, nearly all before iter 5000, and never once alongside cross-sentence ≥121/122. Shipping was a deliberate call: one regressed row against 19.5% → 100% cross-sentence, 83.3% → 100% negatives and 9.0% → 77.0% on real speech. Tracked for v3.2.

Per-category (held-out eval_v3, fused model)

cat	what	n	slot	preserved	guard	PASS
A	cross-sentence corrections	48	100.0%	100.0%	100.0%	100.0%
B	corrections in lists	14	100.0%	100.0%	100.0%	100.0%
C	multiple corrections	10	100.0%	100.0%	100.0%	100.0%
NAMED	real production failures	5	100.0%	100.0%	100.0%	100.0%
G	interrogative-phrase corrections	45	100.0%	100.0%	100.0%	100.0%
D	negatives (must NOT correct)	30	100.0%	100.0%	100.0%	100.0%
E	v2 regression nits	18	100.0%	100.0%	100.0%	100.0%
F	ordinary cleanup	25	100.0%	100.0%	100.0%	100.0%

Scored per category, never as an aggregate — v2 shipped at 24/28 overall while scoring 0/5 on a category its eval set did not contain.

The five real production failures, verbatim on the fused model

prod_1_restate_next_sentence — PASS

raw      Let's meet Thursday. No, no, wait, uh we'll meet Friday actually.
expected Let's meet Friday.
got      Let's meet Friday.

prod_2_fragment_next_sentence — PASS

raw      Let's schedule a meeting for this Thursday. No, Friday at noon.
expected Let's schedule a meeting for this Friday at noon.
got      Let's schedule a meeting for Friday at noon.

prod_3_triple_no_then_fragment — PASS

raw      Let's schedule a meeting for this Thursday. No, no, no. Friday at noon.
expected Let's schedule a meeting for this Friday at noon.
got      Let's schedule a meeting for Friday at noon.

prod_4_list_item_corrected — PASS

raw      Let's do uh a shopping list. Uh we'll get bananas, apples and mangoes. No, no, oranges.
expected Shopping list:
- Bananas
- Apples
- Oranges
got      Let's do a shopping list:
- Bananas
- Apples
- Oranges

prod_5_correction_after_question — PASS

raw      Hi, how are you doing? Can we meet on Friday? Or actually, let's do Thursday, not Friday.
expected Hi, how are you doing? Can we meet on Friday? Actually, let's do Thursday.
got      Hi, how are you doing? Can we meet on Friday? Or actually, let's do Thursday.

Checkpoint selection

All 30 surviving checkpoints scored behaviorally; val loss was not used to select (it flatlines while behavior stays non-monotonic). Selected iter 14500. Five consecutive checkpoints (13000–15000) reached cross-sentence 122/122 with D and F perfect; iter 14500 wins on the OOD score.

Training was cut short at iter 15,000 of 19,590 (77%) by a GPU driver fault — [METAL] Command buffer execution failed: Discarded (victim of GPU error/recovery), an InnocentVictim reset, not an OOM and not a config problem. It was not resumed: --resume-adapter-file restores weights but not the LR schedule, so resuming would have restarted warmup at peak LR on an already-converged model. Behaviour had plateaued by then — cross-sentence sat at 122/122 for the last five checkpoints with LR already down to 1.5e-05.

	iter	x-sent	regr	D	REAL
selected	14500	122/122	43/44	30/30	231/300
runner-up	15000	122/122	43/44	30/30	228/300
runner-up	13000	122/122	43/44	30/30	226/300

Deployment notes

• Fused with --dequantize → bf16 → 8-bit. A plain 4-bit fuse rounds the low-rank delta away and collapsed v2 from 100% → 57.8% held-out self-correction while val loss and filler removal still looked perfect. A ten-case spot check will not catch this; the release process diffs full failure sets.
• The adapter must not live inside this repo. mlx-swift-lm enumerates the model directory recursively and merges every .safetensors it finds, then rejects unused keys — an adapter/ subfolder fails the load with unhandledKeys([... "lora_a", "lora_b"]).
• Any snapshot_download must pass allow_patterns=["model*.safetensors", "*.json", "*.jinja", "system_v2.txt"], or the adapter lands in the shared HF cache the app loads from.
• CleanupPromptProfile.frozenPromptIDs is an exact id set, not a prefix: the app must add abhiram3040/simplewords-dictation-cleanup-v3 and flip MemoryTier.standardCleanupModel in the same release, or v3.1 is prompted with the legacy few-shot prompt.

Homophone policy

Unambiguous homophones are fixed (their were → There were, more then → more than), consistent with the frozen prompt's "Fix capitalization, spelling, and punctuation". Already-correct cases are trained as negatives so the rule is not over-applied. This diverges from the app's original leave-them-alone rule, deliberately.

Known gaps

• Cross-sentence 122/122 — no residual eval_v3 failures in any gated category at this checkpoint.
• Out-of-distribution is 231/300 (77.0%), not ~100%: Disfl-QA is Wikipedia-QA register and is deliberately never trained on, so this measures generalization to a corpus the model has never seen.
• reg_003 (red one no the blue one actually the green one) — a chained triple correction; the only row of its shape in the eval set, marginal across all runs.
• One eval row (prod_1_restate_next_sentence) has a correct output the app's 0.30 length floor rejects (ratio 0.277). guards.CORRECTION_AWARE_FLOOR is the app-side fix and is off; flip it only in lockstep with the Swift app.
• Disfl-QA is a diagnostic, not a training source: its Wikipedia-QA register is deliberately kept out of the corpus, so the OOD number measures generalization.