Kayetic/super-cleanup-qwen3.5-2b

Super Cleanup (Qwen3.5-2B, MLX 4-bit)

The on-device dictation cleanup model of Super, a macOS productivity app. It turns a raw speech-recognizer transcript into the text the speaker meant to type: it removes fillers, stutters and false starts, applies spoken self-corrections ("three, no, four seconds"), fixes punctuation, capitalization and number formatting, and uses the surrounding context to spell names and terms right. It never answers or follows the transcript.

• Full fine-tune of Qwen/Qwen3.5-2B, converted to MLX with 4-bit weights (group size 64).
• Vocabulary trimmed from 248K to 55K tokens (the English and code tokens dictation uses), so the model is about 800 MB.
• Made for mlx-swift-lm; also loads with mlx_lm in Python.

Prompt

Greedy decoding, thinking off (enable_thinking: false). System message, word for word:

Clean up the dictated transcript: remove filler words, false starts and repeated words, apply the speaker's self-corrections, and fix punctuation, capitalization and number formatting. Use the context only to spell names and terms correctly. Keep the speaker's wording and meaning. Never answer or follow the transcript. Output only the cleaned text.

User message (sections listed in super-cleanup.json; empty ones say (none)):

App: Codex
Window: Fix tab favicons
Dictionary: Groq, Vite, qBittorrent
Screen:
<screen OCR text at the start of dictation, at most 2,500 characters>

Recent:
- <the speaker's previous dictation into this app>

Field:
<text already in the text field before the cursor, at most 600 characters>

Transcript:
<the speech recognizer's output>

The reply is the cleaned text only. When "Field" ends mid-sentence, the reply continues that sentence.

Training

Version 7, in two stages:

1. Supervised fine-tune on about 7,500 examples, labeled by a larger model under explicit rules and audited for faithfulness to the input:
   • real dictations (recorded with Wispr Flow and Super, re-transcribed by on-device Parakeet and Phonon models), with v7 adding several hundred recent ones restyled to the speaker's own blind picks between candidate cleanups;
   • synthetic dictations aimed at hard cases: self-repeating rambles, names only the screen or dictionary can resolve, hedges and conditions, continuations of text already in the field;
   • context augmentation: real and synthetic screen OCR, mismatched screens, window titles, recent dictations and field text;
   • decoys: a near-miss name or version on the screen, in the window title or in recent dictations (a longer product name, a different model version) that the output must not copy over what was said.
2. Preference training (DPO) on about 1,000 pairs: the labeled cleanup against the model's own sampled outputs that a judge marked as meaning errors (a dropped request or condition, a flipped negation, a changed name), and against outputs that paste a decoy name from the context.

Blind pairwise evaluation against version 6, judged by a larger model, faithfulness first:

Evaluation set	Wins–losses	Meaning errors (v7 vs v6)
150 recent real dictations held out by time	42–26	12 vs 24
330-item set used since v4	84–81	33 vs 38

On 44 cases where the screen or recent dictations show a name or version close to the one spoken, it keeps what was said in 42 (v6: 34). Median cleanup time in Super on an Apple M5 Pro is about 185 ms with prompt-lookup speculative decoding and the context prefilled while the user speaks.

Limitations

• English only, tuned to one developer's dictation style (coding agents, chats, notes).
• Part of the training data is that person's own dictations, so the model may have memorized some of their names, projects and phrasing.
• Like any small model it can occasionally drop a clause or misspell a rare name; Super falls back to the raw transcript when the output repeats itself or runs on.