MarkusKaemmerer/Voxtral-Small-24B-2507-8bit-dense-encoder

Voxtral-Small-24B-2507, 8-bit with a bf16 audio encoder (MLX)

Voxtral Small (24B) quantised for Apple Silicon, built to be the highest-quality Voxtral build we could assemble rather than a practical everyday one: the language model and lm_head are 8 bit, and the audio encoder and its projector are left in bf16.

	value
size	25 GB
average bits/weight	8.71
minimum RAM to run	~34 GB (so 48 GB or more in practice)
speed (M1 Max, when it fits)	~0.8x realtime — slower than the recording

⚠️ Read this before downloading

We could not measure this build's accuracy. It was produced and tested on a 32 GB machine, where the 25 GB of weights do not fit in RAM alongside the OS: a benchmark run swapped continuously and did not finish a single 25-minute pass in four and a half hours. So unlike our other builds, there is no measured WER for this model — on our reference audio or on FLEURS.

What we can say:

• The architecture is deliberate, not accidental. The audio encoder is kept in bf16 for the same reason as in our Mini build: an isolated sweep of the encoder (4/6/8/bf16, everything else fixed) on hard German audio gave 10.19 / 9.72 / 8.29 / 8.29 % WER — accuracy that only quantising the encoder can throw away, at no cost to keep, because the encoder runs once per pass. lm_head is quantised because a matching sweep showed it makes no difference to accuracy while bf16 costs throughput.
• On clean audio the 24B model is genuinely better than the 3B one. Our 4-bit Small scored 2.82 % WER on FLEURS German against 4.81 % for the 3B build. This 8-bit build should be at least as good — but we have not confirmed it.

When this build makes sense

Only on a machine with 48 GB of RAM or more, for clean, single-speaker audio (lectures, dictation) where you want the best possible transcript and do not care that it runs slower than realtime. If any of those is not true, use the 3B Mini build instead — on hard conversational audio it is more accurate than the 24B model (4.27 % vs 7.82 % WER on our reference), and it runs on any Mac with 16 GB.

If your machine has under ~34 GB, this build will not run — it will swap until nothing progresses. There is no memory-safety net inside a bare generate() call; the check lives in noScribe, which refuses it before a run starts.

Used in noScribe

These weights are one of the two transcription models in noScribe — a free, open-source app for high-quality interview transcripts (speaker diarization, timestamps, editable output). This is the optional ceiling build, offered as voxtral-small-8bit and downloaded on first use; it appears in the model menu only where the machine can actually hold it, and is refused before a run starts where it cannot. For almost every recording the 3B Mini build is the better choice — see above.

Setup and the engine's own notes: VOXTRAL.md. The engine currently lives on the voxtral branch of this fork of kaixxx/noScribe.

Usage

from mlx_voxtral import load_voxtral_model, VoxtralProcessor
import mlx.core as mx, soundfile as sf

repo = "MarkusKaemmerer/Voxtral-Small-24B-2507-8bit-dense-encoder"
model, _ = load_voxtral_model(repo, dtype=mx.bfloat16)
proc = VoxtralProcessor.from_pretrained(repo)

audio, sr = sf.read("speech.wav", dtype="float32")   # 16 kHz mono
inp = proc.apply_transcrition_request(audio=audio, language="de", sampling_rate=sr)
out = model.generate(input_ids=inp.input_ids, input_features=inp.input_features,
                     max_new_tokens=4096, temperature=0.0, repetition_penalty=1.0)
print(proc.decode(out[0, inp.input_ids.shape[1]:], skip_special_tokens=True))

Set repetition_penalty=1.0. mlx-voxtral defaults to 1.2, a chat default that in verbatim speech penalises punctuation and function words.

Reproducing

python tools/quantize_voxtral.py mistralai/Voxtral-Small-24B-2507 \
    out-dir 8 64 dense-encoder --lm-head-bits 8

The conversion script and the full measurement write-up are in this noScribe fork under tools/ and docs/voxtral-quantisierung.md.

License

Apache 2.0, inherited from mistralai/Voxtral-Small-24B-2507.