Luigi/x-asr-zh-en-icefall-trainable

X-ASR zh-en — trainable PyTorch (icefall)

A faithful, fine-tunable PyTorch reconstruction of the deployed X-ASR bilingual (Traditional Chinese + English) streaming Zipformer2 transducer (~159.7M params, 5000 BPE, code-switching).

X-ASR ships only as inference ONNX. This checkpoint reconstructs the full trainable icefall AsrModel by extracting every weight directly from the deployed ONNX (GilgameshWind/X-ASR-zh-en, deployment/models/chunk-480ms-model) — so the community can fine-tune the real model.

⚠️ Note: the streaming_exp/pretrained.pt in the upstream repo is an unrelated checkpoint (its weights are uncorrelated with the deployed ONNX, Pearson r ≈ 0.01, and it decodes to garbage). This reconstruction is taken from the deployment ONNX, which is the only faithful X-ASR.

Fidelity (greedy, full 500-utt CommonVoice zh-TW + an English set)

forward config	zh-CER	en-WER (punct-stripped)
chunk_size=24, left_context=256 (deployed 480 ms streaming)	0.0685	0.040
chunk_size=-1 (offline / full-context)	0.0500	0.033

Decoder & joiner reconstruct bit-exactly vs the ONNX; the recovered per-channel chunk_scale reproduces the ONNX with maxdiff 0.0.

Files

• xasr_native.pt — {"model": state_dict}, fp32, trainable (159.7M params).
• tokens.txt, bpe.model — the X-ASR 5000-token SentencePiece tokenizer.
• load_xasr.py — builds the exact AsrModel and loads the checkpoint (+ a decode demo).
• xasr_from_onnx.py — the full, reproducible ONNX→PyTorch extractor.

Quick start

git clone https://github.com/k2-fsa/icefall
export PYTHONPATH=icefall/egs/librispeech/ASR/zipformer:icefall:$PYTHONPATH
python load_xasr.py xasr_native.pt audio_16k_mono.wav   # offline greedy decode

from load_xasr import load_xasr
model = load_xasr("xasr_native.pt", device="cuda")       # ready to train or eval

Fine-tuning

Standard icefall pruned-transducer fine-tuning. Targets are SentencePiece ids of Simplified text (tokens.txt/bpe.model); X-ASR emits punctuation, so strip punctuation before scoring WER/CER. Use the deployed streaming geometry (chunk_size=24, left_context_frames=256) or train multi-chunk. Re-export to streaming ONNX with icefall's export-onnx-streaming.py for deployment.

Notes for fine-tuners

• simple_am_proj / simple_lm_proj are training-only (absent from the inference ONNX). They are analytically seeded from the joiner here so the pruned-transducer loss starts sane and training is stable; they refine during fine-tuning.
• 19 vestigial layers.*.bypass_scale parameters are unused by the forward (the layers use the bypass/bypass_mid modules) and are left at init.
• Features: lhotse/kaldi 80-dim mel fbank, 16 kHz, snip_edges=False, dither=0.0.

License & attribution

Apache-2.0. Reconstructed from GilgameshWind/X-ASR-zh-en (X-ASR authors) using k2-fsa/icefall. Please credit the original X-ASR authors and icefall.