MarkusKaemmerer/Voxtral-Small-24B-2507-4bit-dense-encoder

Voxtral-Small-24B-2507, 4-bit with a bf16 audio encoder (MLX)

Voxtral Small (24B) quantised for Apple Silicon so that it runs on a 32 GB Mac: the language model and lm_head are 4 bit, and the audio encoder and its projector are left in bf16.

	value
size	15 GB
average bits/weight	4.82
peak memory, one 10-minute pass (M1 Max)	19.4 GB
speed (M1 Max)	~2x realtime

Why 4 bit is enough here

On the 24B model the audio encoder's precision matters most. An isolated sweep on hard German audio, everything else fixed, put the character error rate at 4.87 / 4.28 / 2.46 / 2.41 % for a 4 / 6 / 8 bit / bf16 encoder, and lm_head at 4 or 8 bit scored the same. So this build spends its bits on the encoder (which runs once per pass and costs no speed). That sweep held the language-model body at 4 bit; a separate paired comparison found a 6-bit body no better (four hand-corrected passages pooled: dWER +0.41 [−0.41, +1.69]). An 8-bit body was not measured; the 8-bit build (…-8bit-dense-encoder) needs ~34 GB. noScribe offers this 4-bit build so that the 24B model runs on a 32 GB Mac.

Measured on FLEURS German (100 recordings, 25 min): 2.78 % WER / 0.75 % CER, against 2.61 % for the unquantised model on the Open ASR Leaderboard and 4.89 % for our 3B Mini build.

Two 4-bit float formats were tried in its place and rejected: mxfp4 scored worse than the affine 4 bit used here, and nvfp4 (without a per-tensor global scale) broke the model outright — it answered with an endless "ist, ist, ist".

When to pick it over Mini

For clean audio: lectures, dictation, read-aloud speech, and material with many names. There it is clearly the better model. On conversational audio (interviews, podcasts, video calls) it was not more accurate than the 3B Mini build on any of four hand-corrected passages, stays closer to the verbatim wording (false starts and filler included), and runs at a quarter of Mini's speed.

It also loops more readily on podcast material: one passage came back with a sentence written twice, and on a ten-minute window a raw generate() ran into "Jetzt. Jetzt. Jetzt. …" after three minutes and never recovered, where Mini transcribed the whole window. noScribe's engine catches such loops (retry, split, keep the clean prefix) and returned the full text, but at 0.6–1.4x realtime instead of ~2x. If you call the model yourself, check for repetition. For interviews, Mini remains the recommendation.

Used in noScribe

These weights are one of the two Voxtral models in noScribe — a free, open-source app for high-quality interview transcripts (speaker diarization, timestamps, editable output) — offered as voxtral-small-4bit and downloaded on first use. noScribe sizes each pass from the machine's RAM and refuses the model before a run where it cannot fit; a bare generate() call has no such safety net.

Setup and the engine's own notes: VOXTRAL.md. The measurements behind this card: docs/voxtral-quantisation.md. The engine lives on the voxtral branch of this fork of kaixxx/noScribe.

Usage

from mlx_voxtral import load_voxtral_model, VoxtralProcessor
import mlx.core as mx, soundfile as sf

repo = "MarkusKaemmerer/Voxtral-Small-24B-2507-4bit-dense-encoder"
model, _ = load_voxtral_model(repo, dtype=mx.bfloat16)
proc = VoxtralProcessor.from_pretrained(repo)

audio, sr = sf.read("speech.wav", dtype="float32")   # 16 kHz mono
inp = proc.apply_transcrition_request(audio=audio, language="de", sampling_rate=sr)
out = model.generate(input_ids=inp.input_ids, input_features=inp.input_features,
                     max_new_tokens=4096, temperature=0.0, repetition_penalty=1.0)
print(proc.decode(out[0, inp.input_ids.shape[1]:], skip_special_tokens=True))

Set repetition_penalty=1.0. mlx-voxtral defaults to 1.2, a chat default that in verbatim speech penalises punctuation and function words.

Reproducing

python tools/quantize_voxtral.py mistralai/Voxtral-Small-24B-2507 \
    out-dir 4 64 dense-encoder --lm-head-bits 4

Quantisation is data-free (affine, group size 64), so the same command produces bit-identical weights.

License

Apache 2.0, inherited from mistralai/Voxtral-Small-24B-2507.