Kn90688/FireRedASR2-AED-int8-batch-onnx

FireRedASR2-AED int8 (batch-capable) for sherpa-onnx

Batch-capable re-export of FireRedASR2-AED for sherpa-onnx, int8-quantized.

Supports Mandarin, English, and 20+ Chinese dialects/accents.

Differences from the officially released sherpa-onnx models

1. True batch decoding: all inputs have dynamic batch dims (the released decoder fixes tokens/offset to batch=1)
2. x_len attention mask in the encoder (the released encoder lost it, causing padding pollution in mixed-length batches)
3. Decoder cross-attention mask via enc_mask encoder output and cross_mask decoder input
4. Slim decoder graph: 991 nodes vs 8215 in the released one

Validation (macOS arm64, num-threads=2)

• int8 per-utterance == fp32 on all test files
• mixed-length batch == per-utterance: 8/8 (released model: 2/8)
• RTF: 0.133 (10.1s) / 0.164 (17.6s) — faster than the official int8 everywhere (8%/41%/39% faster on 10s/5s/17.6s)

Usage

Requires the fireredasr-batch-decoding branch of this sherpa-onnx fork (batched DecodeStreams + length bucketing + adaptive KV cache).

sherpa-onnx-offline \
  --num-threads=2 \
  --fire-red-asr-encoder=./encoder.int8.onnx \
  --fire-red-asr-decoder=./decoder.int8.onnx \
  --tokens=./tokens.txt \
  ./test.wav

Multiple files on the command line are decoded as length-bucketed batches.

QDQ Conv variant (smaller encoder)

encoder.qdqconv.int8.onnx is a drop-in replacement for encoder.int8.onnx where every Conv weight is additionally quantized to per-output-channel int8 (weight-only QDQ; compute stays fp32, so it is arm64-safe).

• Total encoder size: 1294 MB → 819 MB (-37%)
• Recognition output identical to encoder.int8.onnx on the test set
• ~1-4% slower decode

sherpa-onnx-offline \
  --num-threads=2 \
  --fire-red-asr-encoder=./encoder.qdqconv.int8.onnx \
  --fire-red-asr-decoder=./decoder.int8.onnx \
  --tokens=./tokens.txt \
  ./test.wav

Export/quantization scripts and full optimization report: scripts/fire-red-asr/ and docs/fireredasr-optimization.md in the fork.