TujiaNLP/tujia-ctc-v0.2

tujia-ctc v0.2

Syllable-level speech recognition for Tujia (Northern dialect) with tone marks. Takes 16 kHz mono audio and returns a sequence of toned syllable labels with time spans, e.g. la35, tsʰi53, ɕiɛ35.

• Architecture: wav2vec2-xlsr-53-espeak-cv-ft (fine-tuned, top 4 of 12 transformer layers trainable) + LayerNorm + 2× BiLSTM(256) + Linear(512 → 709). A 4-class tone auxiliary CTC head is used during training only.
• Labels: 708 toned syllables + blank; tones are 21 / 35 / 53 / 55
• Trainable parameters: 4,570,826 (head only, 18 MB); the fine-tuned encoder is distributed separately
• Training data: third-party Tujia recordings, ~2.6 hours (audio and transcripts are not distributed)
• Best epoch: 18 (validation SER 0.5527 on 108 held-out clips)
• Encoder: fine-tuned from facebook/wav2vec2-xlsr-53-espeak-cv-ft (Apache-2.0)

What changed from v0.1

	v0.1	v0.2
Labels	264 toneless syllables	708 toned syllables
Encoder	frozen	fine-tuned (top 4 layers)
Tone	must be filled in downstream	predicted as part of the label
Test SER	—	0.4767 (greedy) / 0.4634 (+ bigram)

The v0.2 encoder cannot be replaced by the original Hugging Face XLSR-53 weights: the features differ, and mixing them degrades recognition substantially. Use the fine-tuned encoder published as TujiaNLP/tujia-xlsr53-ft (fetched by fetch_encoder.sh).

Package contents

tujia-ctc-0.2/
├── model/
│   ├── tujia-ctc-v0.2.pt    head weights (includes vocab, head args, tones, metrics)
│   ├── vocab.txt            709 labels, one per line (blank is index 0)
│   └── model_card.json      architecture, metrics, license
├── infer.py                 standalone inference script
├── fetch_encoder.sh         downloads the fine-tuned encoder (~1.2 GB)
├── requirements.txt
├── SHA256SUMS
├── MANIFEST.json
├── NOTICE
└── LICENSE                  MIT (code only)

Quick start

pip install -r requirements.txt
./fetch_encoder.sh ./models            # Hugging Face by default
# mainland mirror: TUJIA_ENCODER_SOURCE=modelscope ./fetch_encoder.sh ./models
TUJIA_XLSR=./models/tujia-xlsr53-ft python infer.py your_audio.wav

The encoder is hosted on Hugging Face (TujiaNLP/tujia-xlsr53-ft) and mirrored on ModelScope under the same id. It is not bundled here because of its size.

Mainland China. Hugging Face is normally unreachable without a proxy. Two options:

TUJIA_ENCODER_SOURCE=modelscope ./fetch_encoder.sh ./models   # ModelScope mirror
HF_ENDPOINT=https://hf-mirror.com ./fetch_encoder.sh ./models # HF mirror

Output (one syllable per line):

    4.42     4.58  kɤ53        0.718
    4.56     4.70  tsʰeŋ21     0.320
    4.70     4.86  xu21        0.479

--json out.json writes structured output; --start / --duration decode a single span.

Manual encoder setup (no fetch_encoder.sh)

--encoder and TUJIA_XLSR accept any directory containing a complete encoder (config.json plus weights). The path models/tujia-xlsr53-ft is only the default that fetch_encoder.sh writes to — the name carries no meaning, so an existing copy can be pointed at directly without renaming it.

fetch_encoder.sh is a POSIX shell script and does not run on Windows. On Windows, copy the encoder folder yourself and set the variable:

xcopy /E /I D:\somewhere\encoder_best models\tujia-xlsr53-ft
set TUJIA_XLSR=models\tujia-xlsr53-ft
python infer.py your_audio.wav --device cpu

PowerShell: $env:TUJIA_XLSR="models\tujia-xlsr53-ft".

Input formats

infer.py reads audio with soundfile, which handles wav, flac and ogg. Container files such as mp4 or m4a must be demuxed first:

ffmpeg -i video.mp4 -vn -ac 1 -ar 16000 audio.wav

Metrics (strict protocol)

Scored as syllable error rate (SER): token sequences are aligned with Needleman–Wunsch and the edit distance is divided by the reference length. Tone error is computed the same way over the tone subsequence only. No fuzzy syllable-shape matching, no recall-only scoring.

Split: dev = 21 cross-speaker read utterances (wav21, 43 clips), test = 65 clips. The LM weight α is tuned on dev only; test numbers are reported once.

Configuration	dev SER	dev tone	test SER	test tone
Frozen features + greedy	0.6882	0.4590	0.5402	0.3944
Frozen + beam 32	0.6866	0.4598	0.5381	0.3923
Frozen + syllable bigram (α=0.8)	0.6753	0.4499	0.5274	0.3795
Fine-tuned (this release) + greedy	0.6646	0.4568	0.4767	0.3463
Fine-tuned + beam 32	0.6593	0.4514	0.4772	0.3463
Fine-tuned + syllable bigram (α=1.5)	0.6555	0.4439	0.4634	0.3376

The syllable bigram is trained on the training transcripts only (absolute discounting, unigram backoff; 693 unigram types, 692 contexts). A trigram was tried and gave no measurable gain (test SER 0.4624), which is expected at this corpus size.

Ablation: the tone auxiliary head

Re-running the identical recipe with lambda_tone = 0 (20 epochs, best at 15) gives validation SER 0.5561 against 0.5527 (best at 18) for the released model, and tone SER 0.4601 against 0.4204. The auxiliary loss is therefore worth ~4 points of tone and ~0.3 points of syllables at the 108-clip level.

Scored over all 685 clips with the v0.1 syllable-matching protocol, so the two checkpoints are directly comparable:

Checkpoint	all 685 recall	held-out 108 recall	train 577 recall	held-out tone acc
lambda_tone = 0	89.89%	82.80%	91.23%	65.05%
lambda_tone = 0.4 (this release)	89.90%	83.93%	91.03%	65.59%

The train/held-out gap narrows from 8.43 to 7.10 points, which is consistent with the tone head acting as a mild regularizer. The aggregate 685 number barely moves because training clips dominate it. Keep the auxiliary head: it costs nothing on the main task and supplies the per-syllable tone confidence that downstream fusion uses.

Do not compare the 80.26% in v0.1 with the numbers above

v0.1 reported syllable recall after normalizing syllable shapes (nasal vowels treated as oral, coda ŋ dropped, ʨ/ʧ/ʦ collapsed) with a hit threshold of edit similarity ≥ 0.5 — a lenient, recall-only metric. The table above is a strict SER on unnormalized toned labels. The same audio scores very differently under the two protocols; they are not interchangeable and must not be quoted side by side as if on one scale.

Known limitations

1. Over-segmentation. Predicted syllable counts exceed reference counts on slow or noisy speech (e.g. 38 predicted vs 24 reference on one dev clip). Deletions are comparatively rare.
2. In-domain data only. The corpus is ~2.6 hours from a single source with few speakers. Other dialect points are likely to degrade; use a ZIPA phone front-end as a fallback.
3. Tone errors are the dominant residual. Even after rescoring, about one tone in three is wrong on the test split. The confusions are spread almost evenly over 21 ↔ 35 ↔ 55 (each direction 38–44 occurrences); 53 is confused less often.
4. Selection leakage risk. The best epoch was chosen on the same 108 clips used for the dev/test table. The dev/test split is small (43/65 clips), so differences under ~1 point should be treated as noise.

Data and license

• Code: MIT (see LICENSE).
• Model weights: research use only. No rights to the underlying recordings or transcripts are granted.
• The training corpus is not distributed with this repository.
• Upstream encoder facebook/wav2vec2-xlsr-53-espeak-cv-ft is Apache-2.0 and must retain its attribution. The fine-tuned encoder weights are research-use only.
• Parent project: https://github.com/ibaibing/TujiaNLP (field collection, annotation standards, orthography conversion; corpus terms are recorded there)
• Project page: https://github.com/ibaibing/tujia-asr

Version history

• 0.2.0 (2026-10): toned 708-class label set, fine-tuned XLSR-53, warm-started head, syllable bigram rescoring.
• 0.1.0 (2026-10): first public release. Frozen XLSR + BiLSTM toneless syllable CTC head.