JacobLinCool/accent-detective

Accent Detective

Accent Detective scores how much a Mandarin recording sounds like Taiwan Mandarin. You give it the audio and the text it reads. One small network then does two things:

• aligns the syllables of that text to the audio
• scores how each syllable is realised

The output is an utterance score plus a per-syllable breakdown.

中文說明。 評估一段華語錄音聽起來有多像臺灣華語。輸入是語音，以及這段語音念的文字。模型會先把文字的音節對齊到語音上，再評估每個音節的實現方式，最後給出整句分數與逐音節的貢獻。

What it measures

The model looks at how syllables are realised:

• tone contours
• neutral tones
• initials and finals
• rhythm

It does not look at word choice, and it does not check whether a reading is "correct".

It was built to evaluate text-to-speech systems: comparing different renditions of the same sentence. It is not meant to classify people.

• Score: log-odds that the recording is Taiwan Mandarin. index is the same value on a 0–1 scale: 0.5 is undecided, and each 0.1 is about 7× the odds.
• Per-syllable values: how much each syllable pulls the judgement towards Taiwan (> 0) or away from it (< 0). They are not calibrated probabilities.

Usage

# pip install -r requirements.txt   (torch, torchaudio, librosa, soundfile, numpy, pypinyin, jieba, opencc, safetensors)
from huggingface_hub import snapshot_download
import sys
path = snapshot_download("JacobLinCool/accent-detective")
sys.path.insert(0, path)
from detective import load

det = load()                                               # CPU or GPU
det.score("clip.wav", "這麼長的時間")                         # {"logit", "index", "syllables": [...]}
det.score_recording("talk.wav", "第一句話。第二句話。")         # sentence by sentence, split at punctuation

• Text: must match the audio. Punctuation is used to split long recordings into sentences, because the model was trained on short sentences.
• Unalignable input: if the text cannot be aligned to the audio, score returns None.

Model

Everything is one checkpoint, model.safetensors, of 32.2 M parameters. One small encoder feeds the alignment and two families of judges, whose syllable log-odds are averaged.

• Encoder: a Conformer student encoder (4 layers, width 384, log-mel input), distilled from two teachers at once:
  • TencentGameMate/chinese-wav2vec2-base, of which it predicts hidden layers 4, 8 and 12
  • the encoder of openai/whisper-medium, of which it predicts hidden layers 6 and 12
• Alignment head: a convolutional CTC head over toneless pinyin letters, trained with a frame-level KL to the MMS forced-alignment model. It replaces a separate 315 M-parameter aligner.
• wav2vec 2.0 path: six syllable-sequence judges, one per seed, on the predicted wav2vec 2.0 layers.
• Whisper path: six judges on the predicted Whisper layers, after syllable-identity normalisation. The mean state of each syllable type is subtracted, so the judges see how a syllable is said rather than which syllable it is.
• Judges: each is a 2-layer transformer over per-syllable pooled states and the text context (tone, tone sandhi, Mainland neutral tone, phrase position). They are trained on nested layer sets (Matryoshka), and never see the identity of the characters.

The judges are trained with two kinds of content control:

• sentence-length and text-context importance weights
• a text-only product-of-experts bias model

These keep the judges from learning from content.

Evaluation

Compared with the 411 M-parameter pipeline it replaces (MMS alignment + 12-layer wav2vec 2.0 + one judge, mean of three seeds). Every system scores the audio end to end.

	Original	Accent Detective
Parameters	411 M	32.2 M
Length-matched AUC, held-out Common Voice speakers (526 Taiwan, 704 Mainland)	0.949	0.949 [0.925, 0.967]
Same text and voice prompt, BreezyVoice ranked above CosyVoice-300M (1,000 pairs)	72%	76%
The same contrast in standard deviations of the score on real speech	0.31	0.37
Score variance explained by the TTS voice (119 texts × 4 voices)	6.3%	7.2%
Expected calibration error, no recalibration	0.051	0.025
CPU latency per utterance (Apple M5 Pro, 6 threads, median)	172 ms	18 ms

• Held-out set: the held-out speakers were never used to train any model. Their sentences are harder than the original test set's: every system scores lower on them.
• Synthetic contrast: the 1,000 pairs are 250 held-out Taiwan sentences read with four voice prompts.

Limitations

• Southwestern Mandarin: standard Mandarin from speakers of Sichuan, Chongqing, Guizhou and Yunnan (KeSpeech) is judged Taiwan-accented 37% of the time, against 33% for the original pipeline. The overlap is largest on retroflex initials and velar nasal finals, which both varieties merge.
• Other recording conditions: on KeSpeech, standard Mandarin from the other Mainland regions is judged Taiwan-accented 6–14% of the time (10% on average), against about 5% for the original pipeline. The compact model is less robust than its teacher to recording conditions unlike Common Voice.
• Training domain: volunteer read speech from Common Voice.
• Not evaluated: Hong Kong, Singapore and Malaysian Mandarin.

Intended use

Research, and evaluating or steering speech synthesis towards Taiwan Mandarin. It scores speech, not people: do not use it to infer anyone's nationality or origin.

Training data and licences

• Training data: Common Voice 25 zh-TW and Common Voice 21 zh-CN, both CC0, with speakers disjoint across splits.
• Distillation audio: about 140 h, used without labels. It holds 84 h of the same public Common Voice audio, excluding every development, test and held-out speaker, and 55 h of AISHELL-1 and THCHS-30 (Apache 2.0) for more varied recording conditions.
• Code and weights: MIT (see LICENSE).
• Distillation teachers: TencentGameMate/chinese-wav2vec2-base (MIT) and openai/whisper-medium (MIT).
• Bundled lexicons:
  • lexicon/concised.json.xz: headwords and their pinyin extracted from 教育部《國語辭典簡編本》 (Ministry of Education, Taiwan), CC BY-ND 3.0 TW, converted to JSON.
  • lexicon/cedict.txt.gz: CC-CEDICT (MDBG), CC BY-SA 4.0.

Citation

@misc{lin2026accentdetective,
  title        = {Accent Detective: A Compact Judge of Taiwan Mandarin Accent},
  author       = {Lin, Jhen-Ke},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/JacobLinCool/accent-detective}},
  note         = {Hugging Face model}
}