RozzaCreat/whisper-large-v3-turbo-he-ggml

Whisper Large v3 Turbo (Hebrew) — Q4_K GGML

A 452 MB quantized GGML build of ivrit-ai/whisper-large-v3-turbo, the Hebrew fine-tune of OpenAI's Whisper Large v3 Turbo by the ivrit.ai project.

All the modelling work is ivrit.ai's. This repository only adds a Q4_K quantization of their published weights, so the model fits on machines where the 1.5 GB original does not, and measures what that costs on Hebrew.

Why this exists: stock Whisper treats Hebrew as one of 100 languages and does it poorly. On FLEURS Hebrew, stock Large v3 Turbo scores 28.59% WER. This file is 3.4× smaller than the stock model and scores 14.79% — roughly half the errors.

The file

File	Quant	Size	WER (FLEURS he test)
ivrit-ai-whisper-large-v3-turbo-q4_k.bin	Q4_K	452 MB	14.79%

sha256  c20109e585c1bdee9e3b046b92c9c5b76628bd4a125637f67509b5fb9105b7fc

Quantization cost, relative to the unquantized f16 upstream weights (1549 MB, 14.37% WER): +0.42 percentage points for 3.4× less disk.

If you want the last 0.42 points, quantize the upstream f16 yourself with whisper.cpp's whisper-quantize — Q5_0 measured identical to f16 here (14.37%) at 547 MB.

Important: set the language to Hebrew explicitly

The upstream model card warns that language detection was degraded during fine-tuning:

Language detection capability of this model has been degraded during training — it is intended for mostly-hebrew audio transcription. Language token should be explicitly set to Hebrew.

Translation was likewise not trained. Always pass -l he (or select Hebrew in your app). Auto-detect will cost you accuracy. Every number here was measured with the language pinned to Hebrew.

Usage

whisper.cpp

./whisper-cli -m ivrit-ai-whisper-large-v3-turbo-q4_k.bin -l he -f audio.wav

Handy

Handy is a free, open-source, offline speech-to-text app for Windows, macOS and Linux. It picks up GGML .bin files dropped into its models directory:

OS	Directory
Windows	%APPDATA%\com.pais.handy\models\
macOS	~/Library/Application Support/com.pais.handy/models/
Linux	~/.local/share/com.pais.handy/models/

Copy the .bin there, restart Handy (or press Rescan on the Models screen), then pick the model and set the language to Hebrew.

Note: legacy .bin files carry no capability metadata, so Handy cannot know the model's language and it will not appear under a Hebrew language filter. It works fine once selected.

Also works with Vibe and anything else built on whisper.cpp.

Benchmark

Measured on FLEURS he_il, test split.

• 40 utterances, 6.4 minutes of audio, sampled deterministically across the 347 unique sentences in the split
• whisper.cpp b4938, CPU + BLAS
• Greedy decoding (-bs 1 -bo 1), language pinned (-l he)
• Identical flags for every model; only -m differs
• WER: Levenshtein over whitespace-separated tokens, punctuation stripped, Hebrew niqqud removed, geresh folded to ASCII apostrophe

Model	Size	corpus WER	mean	median
Stock whisper-large-v3-turbo	1549 MB	28.59%	27.91%	22.65%
ivrit.ai f16 (upstream)	1549 MB	14.37%	13.87%	9.17%
ivrit.ai Q5_0	547 MB	14.37%	13.92%	8.01%
ivrit.ai Q4_K (this file)	452 MB	14.79%	14.57%	8.01%

Per-utterance, the fine-tune beat stock on 32 of 40 clips, tied on 7, lost on 1. A paired bootstrap over 10,000 resamples puts this file's mean-WER gain over stock at +13.4 points (95% CI 9.0–19.0).

Median WER is identical to Q5_0 (8.01%) — on a typical clip there is no difference; the gap only shows on harder ones.

Limitations of these numbers

Stated plainly, because they are easy to over-read:

1. All speakers are male. FLEURS he_il test and dev splits contain only male speakers (792/792). Female speakers exist only in train, which was excluded to avoid contamination. The comparison is controlled — every model heard identical audio — but these numbers say nothing about female speech.
2. Greedy decoding, not beam search. This inflates absolute WER for every model equally; relative comparisons are unaffected. Beam search would lower all figures.
3. n = 40, not the full 792-utterance split. The confidence interval accounts for this.
4. Q5_0 and f16 both scored 14.37%. That is a coincidence of integer error counts at this sample size, not identical output — their transcripts differ on 8 of 40 clips.

Credits

• ivrit.ai — the Hebrew fine-tune and all the training work, released under Apache-2.0. Trained on crowd-transcribe-v5, crowd-recital-whisper-training and knesset-plenums-whisper-training.
• OpenAI — the base Whisper Large v3 Turbo model.
• whisper.cpp — the ggml runtime and the whisper-quantize tool used to produce this file.

License

Apache-2.0, inherited from the upstream model. See ivrit-ai/whisper-large-v3-turbo for the original terms.