bilalgcm/nb-whisper-small-onnx

nb-whisper-small ONNX (with KV-cache)

ONNX export of NbAiLab/nb-whisper-small for use in the browser with Transformers.js.

Unlike earlier ONNX conversions of this model, this export includes the merged decoder with KV-cache (decoder_model_merged.onnx). Without it the decoder recomputes attention over all previous tokens on every step, which makes generation roughly an order of magnitude slower.

Measured in Chrome on WebGPU, 3-second audio chunks, Apple M-series laptop:

Export	Inference per chunk
Earlier ONNX conversions	10 000 – 18 000 ms
This export (fp32)	~1 250 ms

Usage

import { pipeline } from "@huggingface/transformers";

const transcriber = await pipeline(
  "automatic-speech-recognition",
  "REPLACE_WITH_YOUR_USERNAME/nb-whisper-small-onnx",
  { device: "webgpu", dtype: "fp32" }
);

Use fp32. An 8-bit quantized build was tested and was ~10× slower on WebGPU, since there is no fast native int8 path and weights are dequantized per operation.

How it was made

python -m optimum.exporters.onnx \
  --model NbAiLab/nb-whisper-small \
  --task automatic-speech-recognition-with-past \
  ./out

-with-past is the part that produces the cached decoder.

Built for Eleanor, a live conference transcription tool, as a student project at OsloMet.