HitendraKawale/whispr-replication

whispr — a from-scratch Whisper replication (these models do not work)

⚠️ Read this before downloading

These models do not transcribe speech. Word error rate is 103% and 150% — worse than outputting nothing. They emit fluent English that loops and largely ignores the audio.

That is not a bug report, it is the result. They are published so the numbers in the accompanying write-up are checkable and so nobody has to repeat five hours of GPU time to reproduce them.

If you want to transcribe audio, use openai/whisper-tiny — same architecture, 6,800× more training data, and it actually works.

These are also not transformers-compatible. AutoModel.from_pretrained will not load them. See Usage.

What this is

A from-scratch replication of Robust Speech Recognition via Large-Scale Weak Supervision (Whisper), built and trained on an M1 MacBook Air as a way of learning audio ML from the ground up.

Code, a written note explaining every step, and the full method: https://github.com/HitendraKawale/whispr_replication

The architecture is verified faithful: OpenAI's real whisper-tiny checkpoint loads into this implementation with strict=True and produces bit-identical outputs (max difference 0.0). The log-mel frontend matches whisper.log_mel_spectrogram to 1.2e-7. What differs is the training data — 100 hours instead of 680,000.

The checkpoints

File	Train audio	Steps	Best val loss	WER (unseen speakers)
whispr-100h-step20000.pt	100.3 h train-clean-100	20,000 (~6 epochs)	4.028	103.4%
whispr-3.7h-step1250.pt	3.7 h dev-clean	1,250 (~4.7 epochs)	5.574	149.9%

Each .pt requires its matching tokenizer-*.json. The vocabulary is fitted per-run on that run's training transcripts, so the token ids differ between them — pairing a checkpoint with the wrong tokenizer decodes to nonsense silently, without raising.

~18M parameters each (Whisper Tiny's shape: 4 layers, width 384, 6 heads, with a 2,048-token vocabulary instead of GPT-2's 50,257).

What they actually output

On speakers they have never heard:

REF  SHE GOT UP ON HER KNEES AND WRUNG HER HANDS
HYP  I'LL NOT THINK IT'T YOU'T DO YOU'T                                  (3.7 h)

REF  I SAW THAT INTERVIEW IN THE PAPER YESTERDAY TELLING WHERE YOU WERE
HYP  AND THE OTHER TWO OF THE MEN WHO WERE IN THE SAME WAY THE OLD MAN   (100 h)

WER above 100% is possible because insertions are unbounded — the model emits more words than the reference.

Why they fail, which is the interesting part

The decoder learns two things at once: the distribution of English text, and the mapping from audio to text. The first is cheap — a few thousand transcripts teach you that "THE" is common and follows almost anything, and it requires no encoder at all. The second is expensive.

So gradient descent does the cheap thing first. Cross-entropy falls from 7.6 to ~5.5 almost entirely by learning unigrams and bigrams, then plateaus, because further progress needs the encoder to become useful.

At 3.7 hours the model never gets there and overfits — validation traced a textbook U, bottoming at step 1,250 then climbing while training loss kept falling. At 100 hours the U disappears: validation was still falling when the schedule ended, with training loss tracking it closely. The problem changed from overfitting to underfitting, which is a better problem — it says the next win is more epochs and more data, not more regularisation.

Whisper used 680,000 hours. That gap is the paper's central claim, measured from the wrong end.

Usage

These use a custom model class, not transformers.

git clone https://github.com/HitendraKawale/whispr_replication
cd whispr_replication && uv sync

import torch
from huggingface_hub import hf_hub_download

from whispr.audio import load_audio
from whispr.config import AudioConfig, Config, ModelConfig, TrainConfig
from whispr.decode import Decoder
from whispr.mel import log_mel_spectrogram
from whispr.model import build_model
from whispr.tokenizer import WhisprTokenizer

REPO = "HitendraKawale/whispr-replication"
ckpt_path = hf_hub_download(REPO, "whispr-100h-step20000.pt")
tok_path = hf_hub_download(REPO, "tokenizer-100h.json")   # must match!

ckpt = torch.load(ckpt_path, map_location="cpu", weights_only=False)
saved = ckpt["config"]
cfg = Config(
    audio=AudioConfig(**saved["audio"]),
    model=ModelConfig(**saved["model"]),
    train=TrainConfig(**saved["train"]),
)

model = build_model(cfg.model)
model.load_state_dict(ckpt["model"])
model.eval()

tokenizer = WhisprTokenizer.load(tok_path)
mel = log_mel_spectrogram(load_audio("clip.flac"), pad_to=cfg.audio.n_samples)
print(Decoder(model, tokenizer).greedy(mel)[0].text)

Each checkpoint carries its own config, so you don't have to remember that the 100 h model uses a 17 s window (850 encoder positions) and the 3.7 h model uses 15 s (750). Loading one with the other's config fails loudly.

Resuming training

Both include optimizer state, so training continues with the LR schedule intact:

cp whispr-100h-step20000.pt checkpoints/run_100h/best.pt
uv run python scripts/07_train.py --corpus train-clean-100 --window 17 \
    --steps 20000 --mel-cache --resume best.pt

Training details

Optimiser settings follow the paper's Table 17 exactly: AdamW, β=(0.9, 0.98), ε=1e-6, weight decay 0.1, gradient-norm clip 1.0, linear warmup then linear decay to zero, Gaussian fan-in initialisation.

Deviations, all forced by a single laptop and all documented with reasoning in whispr/config.py:

	Paper	Here
Training audio	680,000 h, multilingual	100.3 h English read speech
Updates	1,048,576 @ batch 256	20,000 @ batch 8
Vocabulary	50,257 (GPT-2 BPE)	2,048, fitted on our transcripts
Window	30 s (1500 enc positions)	17 s (850)
Augmentation	none	±6 dB gain jitter
Tasks	transcribe + translate + langID + VAD	English transcription only

Trained on an Apple M1 (16 GB) via MPS. Roughly 6 hours for the 100 h run.

Evaluation

Speaker-disjoint throughout. The 100 h model trains on train-clean-100 and validates on dev-clean — LibriSpeech's own partition, so the number is comparable to published results rather than only to itself. Greedy decoding, 400 held-out utterances, WER computed corpus-level (not averaged per utterance) after Whisper-style text normalisation.

Limitations

Everything. Specifically: English only, read audiobook speech only, no punctuation or casing (LibriSpeech labels have none), no timestamps, no translation, no robustness to noise or accents or spontaneous speech, and it does not produce correct transcripts of anything.

Reproducibility

MPS kernels are nondeterministic, so identical seeds do not give identical runs. An earlier 100 h run of this same code reached val 3.876 / 110.2% WER using a 25,000-step schedule. Expect the qualitative findings to reproduce and the third decimal not to.

Citation

The paper being replicated:

@article{radford2022whisper,
  title   = {Robust Speech Recognition via Large-Scale Weak Supervision},
  author  = {Radford, Alec and Kim, Jong Wook and Xu, Tao and Brockman, Greg
             and McLeavey, Christine and Sutskever, Ilya},
  journal = {arXiv preprint arXiv:2212.04356},
  year    = {2022}
}

Licence

MIT, matching the repository. LibriSpeech is CC BY 4.0.