llm.create_chat_completion(
messages = [
{
"role": "user",
"content": "What is the capital of France?"
}
]
)Quill: on-device dictation cleanup models
Quill is a family of small language models that turn raw speech-to-text output into clean, written text, entirely on your own device. It removes filler words (um, uh, like, you know), fixes punctuation and capitalization, repairs spoken self-corrections and false starts, and collapses the stutters and repeats that dictation produces, without changing your words or sending anything to the cloud.
Quill is the cleanup stage of Quobi, a private, offline dictation app for desktop and mobile.
What this is
When you dictate, a speech recognizer (e.g. Whisper) produces a literal, messy transcript:
"um so i was thinking like maybe we could you know meet up at three"
Quill rewrites that into what you actually meant to write:
"So I was thinking maybe we could meet up at three."
It is not a chatbot and not an instruction-following assistant. It does one job: clean dictated text. Feeding it questions or commands will not get answers; it will just clean the text.
Base model & credit
Quill is a fine-tune of Qwen3.5 by the Qwen team (Alibaba), used under the Apache 2.0 license. Qwen3.5 is a hybrid architecture interleaving Mamba-2 / state-space (SSM) layers with periodic full-attention layers, which makes the small sizes fast and memory-light, well suited to on-device, low-latency cleanup. All credit for the base models goes to the Qwen team; Quill only adds task-specific fine-tuning.
| Quill tier | Base model | Size (Q4_K_M) |
|---|---|---|
quill-0.8b-Q4_K_M.gguf |
Qwen/Qwen3.5-0.8B | 505 MB |
quill-2b-Q4_K_M.gguf |
Qwen/Qwen3.5-2B | 1.2 GB |
quill-4b-Q4_K_M.gguf |
Qwen/Qwen3.5-4B | 2.6 GB |
Which tier to use
| Tier | Best for | Behavior |
|---|---|---|
| 0.8B | Phones and any CPU (recommended default) | Verbatim: faithful cleanup, no rephrasing |
| 2B | Mid-range machines / a modest GPU | Verbatim + light tidying |
| 4B | Desktops with a GPU | Verbatim + tidying + light formatting |
The smaller tiers are deliberately conservative. The 0.8B is verbatim-only by
design: it is paired with a deterministic post-processing scaffold (symbol,
email, URL, and number normalization) so the model never has to guess at
conversions like "at" → @. This keeps the tiny model accurate and predictable;
the larger tiers take on more rewriting and structure.
Usage (llama.cpp)
llama-server -m quill-0.8b-Q4_K_M.gguf --host 127.0.0.1 --port 8080 -ngl 99
Prompt format (important). Use ChatML with the assistant turn pre-seeded with an empty think block so the model does not emit chain-of-thought:
<|im_start|>system
You clean up dictated text.<|im_end|>
<|im_start|>user
yeah so um the meeting is gonna be like at uh three thirty tomorrow i think<|im_end|>
<|im_start|>assistant
<think>
</think>
→ "The meeting is at 3:30 tomorrow."
⚠️ Do not pass
--jinja. It re-enables chain-of-thought leakage. Use the raw prompt above (or the/completionendpoint) with the pre-seeded empty<think></think>block. Greedy decoding (temperature = 0) is recommended.
Intended use & limitations
- Intended: post-ASR cleanup of first-person English dictation.
- Not intended: as a general assistant, translator, or summarizer; for languages other than English (non-English text is passed through, not cleaned); for safety-critical rewriting.
- Like any LM it can occasionally over- or under-edit. The verbatim tiers minimize this by preserving your wording; pair them with the deterministic scaffold for symbol/number normalization.
License
Apache 2.0, inherited from the Qwen3.5 base models (also Apache 2.0). You are free to use, modify, and redistribute, including commercially, under the terms of the license. Fine-tuned and released as part of the Quobi project.
- Downloads last month
- 75
4-bit
# !pip install llama-cpp-python from llama_cpp import Llama llm = Llama.from_pretrained( repo_id="Quobi/Quill", filename="", )