A Self-Refining Framework for Enhancing ASR Using TTS-Synthesized Data
Authors: name: Cheng Kang Chou, name: Chan-Jan Hsu, name: Ho-Lam Chung, name: Liang-Hsuan Tseng, name: Hsi-Chun Cheng, name: Yu-Kuan Fu, name: Kuan Po Huang, name: Hung-Yi Lee

We propose a self-refining framework that enhances ASR performance with only
unlabeled datasets. The process starts with an existing ASR model generating
pseudo-labels on unannotated speech, which are then used to train a
high-fidelity text-to-speech (TTS) system. Then, synthesized speech text pairs
are bootstrapped into the original ASR system, completing the closed-loop
self-improvement cycle. We demonstrated the effectiveness of the framework on
Taiwanese Mandarin speech. Leveraging 6,000 hours of unlabeled speech, a
moderate amount of text data, and synthetic content from the AI models, we
adapt Whisper-large-v2 into a specialized model, Twister. Twister reduces error
rates by up to 20% on Mandarin and 50% on Mandarin-English code-switching
benchmarks compared to Whisper. Results highlight the framework as a compelling
alternative to pseudo-labeling self-distillation approaches and provides a
practical pathway for improving ASR performance in low-resource or
domain-specific settings.