Small Amharic ASR
Two checkpoints from the same training setup, plus the language models used to decode them.
conformer_12layer_pool/is a 21.1M Conformer CTC model. The weight file is epoch 16 ofbest.ptwith the optimizer removed.zipformer_s_leyu/is a 23.4M streaming Zipformer-S (RNN-T and CTC). The weight file is epoch 23 oflast.ptwith the optimizer removed. The ONNX graphs are the live chunk: 32 embed frames, 128 frames of left context (640 ms).lm/word_5gram/is the word KenLM used with both models (α 0.5, β 0.8, beam 100).lm/char_6gram/is the character 6-gram the Zipformer demo uses as its second box (α 0.4, β 1.2, beam 32, token top-k 12). The tables below are not character-LM scores.
Both acoustic models use a 320-piece character inventory. The two tokenizer.json files are the same bytes. cmvn.pt is the global mean and variance for that checkpoint. model.pt is a dict with model, epoch, and vocab_size.
The word-LM binary in this folder is the file at exp/20m_char_12layer/lm_gpac/lm.bin (the one the demo loads, and the one named in the FLEURS gain run). Several studio logs call that LM exp/lm_gpac/lm.bin. The ARPA source and the LM training text are not included.
Try it
Amharic ASR runs in the browser. The Zipformer there is the int8 graph. The Conformer is the full-precision graph, because the int8 file uses ConvInteger, which that browser runtime does not include. The page decodes greedy, without the word 5-gram, so its text will not match the language-model rows below. With the Zipformer selected, the microphone is left unprocessed and captured at 16 kHz. It writes the transducer text every 640 ms and fills in CTC when the recording stops.
Results
WER / CER are percentages. Scoring folds the Dataset.ET homophone pairs (ሐ/ኀ→ሀ, ሠ→ሰ, ዐ→አ, ፀ→ጸ) and strips punctuation, which is the default in evaluation/metrics.py. Zipformer rows are the scores_folded block. Conformer rows are the scores block from evaluate.py.
Seen / unseen is against Dataset.ET train words, including on Leyu, so Leyu "unseen" includes words that were in the Leyu training text.
Dataset.ET test (1,548 clips)
| Model | Decoder | WER | CER | Seen | Unseen |
|---|---|---|---|---|---|
| Conformer pool | word LM | 18.61 | 6.91 | 14.44 | 51.84 |
| Conformer pool, later re-decode | word LM | 20.69 | 7.61 | 14.48 | 76.93 |
| Conformer pool | greedy | 26.05 | 8.49 | 21.74 | 59.69 |
| Zipformer Leyu, epoch 23, chunk 32 | CTC | 26.47 | 8.71 | 22.48 | 57.38 |
| Zipformer Leyu | transducer | 24.03 | 9.32 | 19.84 | 57.72 |
| Zipformer Leyu | word LM | 19.11 | 7.32 | 14.85 | 53.60 |
The Conformer word-LM row of 18.61 is eval_metadata_test_kenlm (24 Sep). A later decode of the same split, stored under eval_fixed/metadata_test_kenlm (5 Oct), is 20.69. The hypotheses differ on 916 of 1,548 clips. Both logs name the same LM weights (α 0.5, β 0.8). Dataset.ET dev with that word LM is 21.75 / 8.72 in eval_metadata_dev_kenlm. A second dev file, eval_dev_kenlm, is 27.87 / 10.79 on the same 1,526 clips.
Zipformer short utterances (20 clips) are 51.18 WER with the word LM.
FLEURS Amharic test (516 clips)
| Model | Decoder | WER | CER |
|---|---|---|---|
| Conformer pool | greedy | 29.30 | 8.81 |
| Conformer pool | word LM | 22.44 | 7.61 |
| Conformer pool, ONNX fp32 | greedy | 29.33 | 8.83 |
| Conformer pool, ONNX fp32 | word LM | 22.51 | 7.67 |
| Conformer pool, ONNX int8 | greedy | 29.46 | 8.86 |
| Conformer pool, ONNX int8 | word LM | 22.62 | 7.66 |
| Zipformer Leyu | CTC | 27.51 | 8.20 |
| Zipformer Leyu | transducer | 26.44 | 9.85 |
| Zipformer Leyu | word LM | 22.78 | 7.66 |
An earlier Conformer greedy file on this split is 45.60 WER. The 7 Oct rerun is 29.30, which matches the ONNX greedy numbers, so the table uses the rerun. A gain-normalized word-LM run, not the plain test, is 21.68 / 7.18.
Zipformer word-LM seen / unseen on FLEURS is 12.26 / 44.82.
Leyu
Prompt-held-out split. Speakers are not held out. Zipformer epoch 23, chunk 32, word LM α 0.5 β 0.8 beam 100.
| Split | Clips | CTC | Transducer | Word LM |
|---|---|---|---|---|
| dev | 2,391 | 29.44 / 9.48 | 27.28 / 10.18 | 24.95 / 9.12 |
| test | 2,407 | 28.16 / 8.96 | 26.12 / 9.62 | 24.23 / 8.61 |
Word LM on the dev set by dialect:
| Dialect | Clips | WER | CER |
|---|---|---|---|
| Addis Ababa | 507 | 18.83 | 6.18 |
| Wello | 497 | 22.32 | 7.97 |
| Shewa | 370 | 24.76 | 8.32 |
| Gonder | 514 | 29.14 | 11.59 |
| Gojjam | 503 | 31.56 | 12.36 |
Layout
conformer_12layer_pool/model.pt epoch 16 weights
conformer_12layer_pool/model.onnx fp32, full clip
conformer_12layer_pool/model.int8.onnx
conformer_12layer_pool/tokenizer.json
conformer_12layer_pool/cmvn.pt
conformer_12layer_pool/config.yaml
zipformer_s_leyu/model.pt epoch 23 weights
zipformer_s_leyu/onnx/ encoder, decoder, joiner, fp32 and int8
zipformer_s_leyu/tokenizer.json
zipformer_s_leyu/cmvn.pt
zipformer_s_leyu/config.yaml
lm/word_5gram/lm.bin
lm/word_5gram/unigrams.txt
lm/char_6gram/lm.bin
lm/char_6gram/unigrams.txt
lm/char_6gram/meta.json
MANIFEST.sha256
MANIFEST.sha256 is the checksum of every file except this README.