whisper-large-v2-fp16

openai/whisper-large-v2 stored in fp16 instead of fp32, ready to serve with vLLM. The repo holds both the converted weights and the scripts that produced and verified them.

The checkpoint is bit-exact with the upstream fp32 release — see Why this is lossless.

	upstream fp32	this repo
model.safetensors	6.17 GB	3.09 GB
dtype	float32	float16
parameters	1.54 B	1.54 B
tensors	1259	1259
source revision	ae46427	—

Why this is lossless

OpenAI released Whisper large-v2 in fp16; the Hugging Face float32 checkpoint is an exact upcast of those fp16 weights. Every one of the 1,543,304,960 values in the upstream file has its low 13 mantissa bits set to zero, so rounding back to fp16 loses nothing:

• max absolute weight error: 0.0
• max relative weight error: 0.0
• values overflowing the fp16 range (|x| > 65504): 0 (largest weight magnitude is 4.64)
• values underflowing to zero: 0
• logits on identical input: identical to the fp32 model, same argmax tokens

Full numbers are in fp16_conversion_report.json. Halving the file removes storage and load time, not accuracy — but note this holds for this model because upstream was fp16 to begin with; do not assume it for other checkpoints.

Serving with vLLM

vLLM reads torch_dtype from config.json, so --dtype auto loads fp16 directly instead of upcasting to fp32.

vllm serve HTXDSAI/whisper-large-v2-fp16 --dtype auto

Then use the OpenAI-compatible transcription endpoint:

curl http://localhost:8000/v1/audio/transcriptions \
  -F "model=HTXDSAI/whisper-large-v2-fp16" \
  -F "file=@audio.ogg" \
  -F "language=en"

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
with open("audio.ogg", "rb") as f:
    print(client.audio.transcriptions.create(model="HTXDSAI/whisper-large-v2-fp16", file=f).text)

Whisper is a transcription-only model in current vLLM: serve it and call /v1/audio/transcriptions (or /v1/audio/translations) rather than the text completions API.

This checkpoint was smoke-tested end-to-end on vLLM 0.16.0 (CUDA 12.8, RTX 4090): the engine reported dtype=torch.float16 straight from config.json, loaded the weights in 0.30 s, and transcribed the mary_had_lamb sample correctly.

Using with transformers

import torch
from transformers import WhisperForConditionalGeneration, WhisperProcessor

model = WhisperForConditionalGeneration.from_pretrained(
    "HTXDSAI/whisper-large-v2-fp16", dtype=torch.float16
).to("cuda")
processor = WhisperProcessor.from_pretrained("HTXDSAI/whisper-large-v2-fp16")

fp16 needs a GPU — CPU inference in fp16 is unsupported or emulated in PyTorch. Load with dtype=torch.float32 to run on CPU.

Reproducing the conversion

The scripts use uv. uv run creates the environment from pyproject.toml/uv.lock on first use, pulling CPU-only torch (~250 MB) since the cast needs no GPU.

# fp32 from the hub -> fp16 in this directory (~10 s once the source is cached)
uv run scripts/convert_to_fp16.py

# structure, dtype, key-set and bit-exactness checks against the fp32 source
uv run scripts/verify_fp16.py

# ...plus a forward pass on both checkpoints, comparing logits
uv run --group verify scripts/verify_fp16.py --logits-parity

Useful flags:

flag	effect
--source PATH_OR_REPO	convert a local directory or a different hub repo
--revision SHA	pin the upstream commit to download
--output DIR	write elsewhere instead of this repo root
--dtype bfloat16	cast to bf16 instead (wider range, fewer mantissa bits)
--force	overwrite an existing model.safetensors
--allow-overflow	emit the checkpoint even if weights exceed the fp16 range

What the conversion does

scripts/convert_to_fp16.py streams the source model.safetensors tensor by tensor and:

1. audits each float tensor against the fp16 range and aborts if any weight would become inf (a single overflowed weight yields NaN logits at inference time);
2. casts float tensors to fp16, passing integer/bool tensors through untouched;
3. copies every non-weight file byte-for-byte, so the tokenizer, feature extractor and generation config are exactly upstream's;
4. edits only config.json, setting torch_dtype to float16;
5. asserts every file vLLM needs is present, then writes fp16_conversion_report.json.

Files

file	role
model.safetensors	fp16 weights (3.09 GB)
config.json	model architecture; torch_dtype: float16
generation_config.json	decoding defaults, language/task token maps
preprocessor_config.json	log-mel feature extractor (80 mel bins, 30 s window)
tokenizer.json, tokenizer_config.json, vocab.json, merges.txt, added_tokens.json, special_tokens_map.json, normalizer.json	multilingual tokenizer
fp16_conversion_report.json	provenance and numerical report for the conversion
scripts/	conversion and verification scripts

License and attribution

Weights are openai/whisper-large-v2, Apache-2.0, unmodified apart from the dtype cast. Model behaviour, languages, training data and limitations are documented in the upstream model card.