TujiaNLP/tujia-ctc-v0.1

tujia-ctc v0.1

Syllable-level speech recognition for Tujia (Northern dialect). Takes 16 kHz mono audio and returns a sequence of syllable labels with time spans. Tones are not predicted.

• Architecture: wav2vec2-xlsr-53-espeak-cv-ft (frozen, 50 Hz / 1024-dim) + LayerNorm + 2× BiLSTM(256) + Linear(512 → 264)
• Trainable parameters: 4,339,976 (head only, 17 MB)
• Training data: third-party Tujia recordings, ~2.6 hours (audio and transcripts are not distributed with this package)
• Best epoch: 35 (validation CER 0.4125)
• Encoder: facebook/wav2vec2-xlsr-53-espeak-cv-ft (~1.2 GB, not bundled, Apache-2.0)

Package contents

tujia-ctc-0.1/
├── model/
│   ├── tujia-ctc-v0.1.pt    weights (includes vocab, head_args, training hyperparameters, metrics)
│   ├── vocab.txt            264 labels, one per line
│   └── model_card.json      architecture, metrics, license
├── infer.py                 standalone inference script (no training repo required)
├── fetch_encoder.sh         downloads the encoder from Hugging Face
├── requirements.txt
├── SHA256SUMS
├── NOTICE
└── LICENSE                  MIT (code only)

Quick start

pip install -r requirements.txt

# Option 1: let transformers fetch the encoder from Hugging Face (~1.2 GB on first run)
python infer.py your_audio.wav

# Option 2: download once, then run offline
./fetch_encoder.sh ./models
TUJIA_XLSR=./models/w2v2-xlsr53-espeak-cv-ft python infer.py your_audio.wav

Output (one syllable per line):

    4.10     4.28  ta         0.981
    6.62     6.84  ʐa         0.954
    9.30     9.52  tʰa        0.977

--json out.json writes structured output; --start / --duration decode a single span.

Recommended: keep each input under ~60 seconds and split longer audio with --start / --duration.

Manual encoder setup (no fetch_encoder.sh)

--encoder and TUJIA_XLSR accept any directory containing a complete encoder (config.json plus weights). The path models/w2v2-xlsr53-espeak-cv-ft is only what fetch_encoder.sh creates — the directory name carries no meaning, so an existing copy can be pointed at directly without renaming.

fetch_encoder.sh is a POSIX shell script and does not run on Windows. On Windows, copy the encoder folder yourself and set the variable:

xcopy /E /I D:\somewhere\w2v2-xlsr53-espeak-cv-ft models\w2v2-xlsr53-espeak-cv-ft
set TUJIA_XLSR=models\w2v2-xlsr53-espeak-cv-ft
python infer.py your_audio.wav --device cpu

PowerShell: $env:TUJIA_XLSR="models\w2v2-xlsr53-espeak-cv-ft".

Input formats

infer.py reads audio with soundfile, which handles wav, flac and ogg. Container files such as mp4 or m4a must be demuxed first:

ffmpeg -i video.mp4 -vn -ac 1 -ar 16000 audio.wav

Metrics (all audio unseen during training)

Material	Reference	Syllable recall
In-house holdout (108 clips, decoding only)	3,273 syllables	80.26%
Cross-speaker read speech (21 utterances)	225 syllables	79.11%
Isolated syllables, read aloud (12 tokens)	12 syllables	91.67%
Narrowband field recording (single clip)	131 syllables	48.85%

Scoring: syllable shapes are normalized (nasal vowels treated as oral, coda ŋ dropped, ʨ/ʧ/ʦ collapsed, ɿ→ɨ, etc.), then aligned with Needleman-Wunsch. A reference syllable counts as a hit at edit similarity ≥ 0.5.

Known limitations

1. No tone prediction. All 264 labels are toneless syllables; tones must be filled in downstream from a lexicon. Do not treat this model as a tone recognizer.
2. Clear degradation on narrowband audio. On a recording with 85% of its energy below 390 Hz, recall drops to 48.85%, with under-segmentation: 104 syllables predicted against 131 in the reference, so deletions outnumber substitutions.
3. Very short isolated vowels are unstable. Single i / u tokens of 0.5–1.0 s often gain an onset (ni / li / ji), a pattern outside the training distribution.
4. Limited coverage. Training data is ~2.6 hours from a single source, with few dialect points and speakers. For other dialect points, use a ZIPA phone front-end as a fallback.

Data and license

• Code: MIT (see LICENSE).
• Model weights: research use only. No rights to the underlying training recordings or transcripts are granted.
• The training corpus is not distributed with this repository.
• Upstream encoder facebook/wav2vec2-xlsr-53-espeak-cv-ft is Apache-2.0 and must retain its attribution; it is not included here and is downloaded by the user.
• Parent project: https://github.com/ibaibing/TujiaNLP (field collection, annotation standards, orthography conversion; corpus terms are recorded there)
• Project page: https://github.com/ibaibing/tujia-asr

Version history

• 0.1.0 (2026-10): first public release. Frozen XLSR + BiLSTM syllable CTC head, 40 epochs, best at epoch 35.