collectiveai/canary-1b-v2-onnx-split-int8

Canary-1B-v2 ONNX — split decode-loop + INT8 (encoder & decoder)

A modified ONNX export of nvidia/canary-1b-v2 (CC-BY-4.0), built on top of istupakov/canary-1b-v2-onnx (also CC-BY-4.0). Two independent changes on top of that export:

1. Split decode-loop (graph surgery): the original fused decoder-model.onnx recomputes 16 cross-attention key/value projections from scratch on every decode step (~144 times per 30 s window), even though they never change within a window. This repo splits that graph in two — xattn_kv.onnx (K/V, computed once per window) and decoder_step.onnx (the per-token loop, now takes K/V as inputs) — and verifies the split composes back to the original fused graph bit-for-bit (max output difference 0.0 across a held-out set). Zero transcription drift, pure speed fix.
2. INT8 quantization, independently, on each half:
   • decoder_step.dynamic_v1_quint8.onnx: dynamic quantization (onnxruntime.quantization.quantize_dynamic, QuantType.QUInt8, MatMul-only, no calibration data needed).
   • encoder-model.static_qdq_v4_pct_excl.onnx (+ .onnx.data): static QDQ INT8 (Conv/MatMul/Gemm, per-channel, reduce_range=True), with percentile (99.999%) activation calibration over a 40-clip, 5-language calibration set, and 32 nodes measured and excluded via onnxruntime.quantization.qdq_loss_debug (late-layer convolutions and MatMuls whose per-tensor quantization SNR was a 6–22 dB outlier against a ~33 dB median — leaving those in fp32 is what actually closes the quality gap; percentile calibration alone was not enough).

This repository ships only the modified/quantized artifacts — the unmodified fp32 encoder-model.onnx, vocab.txt and config.json are included here too (byte-identical to the base repo) purely for convenience, so this directory is self-contained and you don't need to fetch two repos to use it.

Quality and speed (full 48-window mTEDx long-form validation, cpWER)

All numbers below are the normalized cpWER on the same held-out, real-world long-form Spanish talk (mTEDx, 48 non-overlapping 30 s windows, scored via meeteval's cpWER) — not a short clean-clip screen, which this project's own history has twice shown can diverge sharply (in both directions) from full-scale results.

Configuration	norm cpWER	Δ vs fp32	RTFx	Notes
fp32 encoder + fp32 split decoder	0.0513	—	1.62	Split-decode baseline; byte-identical transcript to the original fused graph
fp32 encoder + INT8 decoder (dynamic_v1_quint8)	0.0529	+3.1%	2.33	Fully validated, independently
INT8 encoder (static_qdq_v4_pct_excl) + fp32 decoder	0.0508	−1.0%	(see caveat)	Beats the fp32 baseline outright

Caveat on RTFx: the encoder-INT8 row's RTFx was measured in the same run as its own fp32 control (1.2112 → 1.2600, +4.0% relative) but on a different host-load condition than the split-decode baseline's RTFx 1.62 figure, so do not read those two RTFx numbers as directly comparable in absolute terms — only the relative (fp32-vs-INT8, same run) comparisons are apples-to-apples. The combined configuration (INT8 encoder and INT8 decoder together, one clean run) has not yet been measured — that's the natural next benchmark for anyone building on this.

Important: not a drop-in onnx_asr model

Because of the split decode-loop, this artifact set is not loadable by plain onnx_asr.load_model(...) the way the base repo is — it needs a decode loop that knows about the xattn_kv.onnx / decoder_step.onnx split and the two independent quantization selectors. Reference implementation: onnx-canary-split in collectiveai-team/coro (GitHub org; the HF org for this repo is collectiveai, not collectiveai-team) (onnx-canary-split backend, quantization= / decoder_quantization= selectors). The export/quantization recipes that produced these exact artifacts are also in that repo: coro/recipes/canary_split_decoder/, coro/recipes/canary_encoder_static_qdq/, coro/recipes/canary_decoder_dynamic_quantization/.

Status: comparative reference, not a recommended default

This is a research/benchmarking artifact from an internal ASR backend comparison, not a production recommendation. The project that produced it uses a different backend (onnx-asr's own default Parakeet integration) as its actual default — Canary is evaluated here specifically because it forces the transcription language natively (useful for language-constrained ASR use cases), which the default backend does not do reliably. If that's not a requirement for your use case, the base istupakov/canary-1b-v2-onnx (or an entirely different model) may be a better fit.

Files

File	What it is	Size
xattn_kv.onnx	Cross-attention K/V graph (new, graph surgery)	64 MB
decoder_step.onnx	Per-token decode graph, fp32 (new, graph surgery)	581 MB
decoder_step.dynamic_v1_quint8.onnx	Per-token decode graph, INT8 dynamic quant	197 MB
encoder-model.static_qdq_v4_pct_excl.onnx + .onnx.data	Encoder, INT8 static QDQ (32 nodes excluded)	971 MB
vocab.txt, config.json	Unmodified, copied from the base repo for convenience	<1 MB

License

CC-BY-4.0, same as nvidia/canary-1b-v2 and istupakov/canary-1b-v2-onnx. Attribution: NVIDIA (original model), istupakov (ONNX export this is built on), collective.ai (split decode-loop graph surgery + INT8 quantization in this repository).