Title: Towards real-time BCI for speech with Whisper-based decoding of neural activity
Primary area: applications to neuroscience & cognitive science
Keywords: BCI, neuroAI, language decoding, invasive neural recordings

Abstract: Decoding continuous speech from neural activity is a central challenge for brain–computer interfaces (BCIs), with major implications for restoring communication in individuals with paralysis. While recent work has achieved impressive performance using recurrent neural decoders trained on multi-electrode array (MEA) recordings, these models remain brittle, data-hungry, and struggle to generalize across sessions or participants. In this work, we introduce Whisper-BCI, the first neural speech decoder to integrate high-resolution MEA recordings with a large pretrained automatic speech recognition (ASR) model. Our approach leverages interpretability findings showing that Whisper’s encoder layers learn phoneme-selective representations with localized attention. Building on this insight, we adapt Whisper to predict phoneme embeddings from neural signals into the third layer of Whisper's encoder and fine-tune the model end-to-end with a hybrid objective combining CTC loss on phoneme alignments and cross-entropy loss on word tokens. We further introduce domain-informed modifications including windowed self-attention to capture articulatory continuity, day-specific low-rank projections to address non-stationarity and reduce computational complexity, and subject-specific input embedders for cross-subject training. Evaluated on Card et al. and Brain-to-Text '25 data, Whisper-BCI performs on par with or outperforms baselines relative to prior MEA decoders, and achieves cross-subject generalization, opening the door to robust decoding with limited resources. Post-processing with rescoring and grammar-guided correction yields an additional relative improvement, and the use of windowed attention has the potential to significantly reduce latency, enabling near-real-time online decoding. Our results demonstrate that pretrained ASR models can serve as effective language backbones for neural decoding and suggest a scalable path toward foundation models for speech BCIs.

TL;DR: Fast sentence decoding from neural activity using Whisper