A Self-Refining Framework for Enhancing ASR Using TTS-Synthesized Data
Authors: name: Cheng Kang Chou; name: Chan-Jan Hsu; name: Ho-Lam Chung; name: Liang-Hsuan Tseng; name: Hsi-Chun Cheng; name: Yu-Kuan Fu; name: Kuan Po Huang; name: Hung-Yi Lee
Published: 2025-06-10 17:30:32

Abstract: We propose a self-refining framework that enhances ASR performance with only
unlabeled datasets. The process starts with an existing ASR model generating
pseudo-labels on unannotated speech, which are then used to train a
high-fidelity text-to-speech (TTS) system. Then, synthesized speech text pairs
are bootstrapped into the original ASR system, completing the closed-loop
self-improvement cycle. We demonstrated the effectiveness of the framework on
Taiwanese Mandarin speech. Leveraging 6,000 hours of unlabeled speech, a
moderate amount of text data, and synthetic content from the AI models, we
adapt Whisper-large-v2 into a specialized model, Twister. Twister reduces error
rates by up to 20% on Mandarin and 50% on Mandarin-English code-switching
benchmarks compared to Whisper. Results highlight the framework as a compelling
alternative to pseudo-labeling self-distillation approaches and provides a
practical pathway for improving ASR performance in low-resource or
domain-specific settings.