distil-medium.en

ONNX builds of distil-whisper/distil-medium.en.

Distil-Whisper is a distilled version of Whisper from Hugging Face. The encoder is copied unchanged; the decoder is cut to two layers and retrained to match the original's output.

The result runs several times faster and weighs roughly half as much, at a small cost in accuracy. Because the decoder is shallow, it is more sensitive than Whisper to how long-form audio is stitched together — see the decoding parameters below.

This checkpoint: 394 M parameters — the encoder of whisper-medium with a two-layer decoder.

Variants

Every variant is the same checkpoint at a different weight precision. Pick a row, download that folder, and read onnx_manifest.json in it for the graph names and their checksums.

Variant	Size	vs fp16	WER	vs fp16
fp16	1,075 MB	1.0	9.68 %	reference
plain-int8	483 MB	÷2.2	11.45 %	+1.77 pt
plain-int4	289 MB	÷3.7	10.09 %	+0.41 pt
sym-int4-asym	290 MB	÷3.7	11.97 %	+2.29 pt
algo-int4-hqq	301 MB	÷3.6	13.42 %	+3.74 pt

Word error rate measured on ten minutes of English speech against a hand-checked transcript, greedy decoding, no voice-activity detection. It is a single corpus in a single language: use it to compare the rows against each other, not as an absolute score.

The rows above are measured with condition_on_previous_text=False, no_repeat_ngram_size=3 rather than the stock decoding settings.

What the names mean

• fp16 — Half-precision encoder, full-precision decoder. Smaller than fp32 and slower on CPU, where no float16 kernel exists — useful on GPU.
• plain-int8 — 8-bit weights, nothing excluded.
• plain-int4 — 4-bit weights, nothing excluded. Usually the smallest that still works.
• sym-int4-asym — 4-bit weights with an asymmetric scale, which does not assume the weights are centred on zero.
• algo-int4-hqq — 4-bit weights quantized with HQQ rather than round-to-nearest.

The int4 variants are meant for the CPU. On ONNX Runtime's CUDA provider the 4-bit matrix kernel (MatMulNBits) loses precision above 128 rows, which covers every encoder pass.

Licence

These are format conversions. The weights, and the licence they come with, are those of the original checkpoint — see the source repository.