Title: Whisfusion: Parallel ASR Decoding via a Diffusion Transformer
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Automatic Speech Recognition(ASR), Non-Autoregressive Models, Diffusion Transformers, Whisper, Speech-to-Text

Abstract: Fast Automatic Speech Recognition (ASR) is critical for real-time applications such as captioning and transcription. However, truly parallel ASR decoding remains challenging due to the sequential nature of autoregressive (AR) decoders and the context limitations of non-autoregressive (NAR) methods. While modern ASR encoders can process up to 30 seconds of audio at once, AR decoders still generate tokens sequentially, creating a latency bottleneck. We propose Whisfusion, the first framework to fuse a pre-trained Whisper encoder with a text diffusion decoder. This NAR architecture resolves the AR latency bottleneck by processing the entire acoustic context in parallel at every decoding step. A lightweight cross-attention adapter trained via parameter-efficient fine-tuning (PEFT) bridges the two modalities. We also introduce a batch-parallel, multi-step decoding strategy that improves accuracy by increasing the number of candidates with minimal impact on speed. Fine-tuned solely on LibriSpeech (960h), Whisfusion achieves a lower WER than Whisper-tiny (8.3% vs. 9.7%), and offers comparable latency on short audio. For longer utterances ($>$20s), it is up to $2.6\times$ faster than the AR baseline, establishing a new, efficient operating point for long-form ASR.

TL;DR: Whisfusion is a non-autoregressive ASR model combining a Whisper encoder and a diffusion transformer, delivering significant speed-ups over autoregressive models without sacrificing accuracy.