A cross-species neural foundation model for end-to-end speech decoding
Authors: Yizi Zhang
Linyang He
Chaofei Fan
Tingkai Liu
Han Yu
Trung Le
Jingyuan Li
Scott Linderman
Lea Duncker
Francis R Willett
Nima Mesgarani
Liam Paninski
Venue: ICLR 2026 Poster (2026)
Keywords: brain-computer interfaces
neuroscience
neural decoding
self-supervised learning
multimodal
transformer
masked modeling
Primary area: applications to neuroscience & cognitive science

Abstract: Speech brain–computer interfaces (BCIs) aim to restore communication for people with paralysis by translating neural activity into text. Most systems use cascaded frameworks that decode phonemes before assembling sentences with an n-gram language model (LM), preventing joint optimization of all stages simultaneously. Here, we introduce an end-to-end BraIn-to-Text (BIT) framework that translates neural activity into coherent sentences using a single differentiable neural network. Central to our approach is a cross-task, cross-species pretrained neural encoder, whose representations transfer to both attempted and imagined speech. In a cascaded setting with an n-gram LM, the pretrained encoder establishes a new state-of-the-art (SOTA) on the Brain-to-Text ’24 and ’25 benchmarks. Integrated end-to-end with audio large language models (LLMs) and trained with contrastive learning for cross-modal alignment, BIT reduces the word error rate (WER) of the prior end-to-end method from 24.69% to 10.22%. Notably, we find that small-scale audio-LLMs markedly improve end-to-end decoding. Beyond record-setting performance, BIT aligns attempted and imagined speech embeddings to enable cross-task generalization. Altogether, our approach advances the integration of large, diverse neural datasets, paving the way for an end-to-end decoding framework that supports seamless, differentiable optimization.

Published as a conference paper at ICLR 2026
A CROSS- NEURAL FOUNDATION MODEL FOR END- TO-END SPEECH DECODING
Yizi Zhang*:l.f Linyang He*.l.t, Chaofei Fan?, Tingkai Liu Han Yu', Trung Le Jingyuan Li Scott Linderman", 2 Lea Duncker', Francis R Willett", Nima Mesgarani' -iiam Paninski
1 CColumbia University, New York, NY, USA
Stanford University, Palo Alto, CA, USA Microsoff, New York, NY, USA University Washington, Seattle, WA, USA Amazon, Seattle, WA, USA
Equal contribution (yz4123, 1h3288
edu
ABSTRACT
Speech brain brain-comppnn computer interfaces (BCIs) aim to restore communication for peo- ple with paralysis by translating neural activity into text. Most systems use cas- caded frameworks that decode phonemes before assembling sentences with an n- n-gram language model (LM), preventing joint optimization all stages simulta- neously. Here. introduce end-to-en BraIn-to-Tes -ext (BIT) framework that translates neural activity into coherent sentences using a single differentiable neu- ral network. Central to approach is cross-task, cross-species pretrained neural encoder, whose representations transfer to both attempted and imagined speech. In cascaded setting with an n-gram LM, the pretrained encoder estab- lishes a new state-of-ther art (SOTA) on the Brain-to-Teas '24 and '25 benchmarks. Integrated end-to-end with audio large language models (LLMss) and trained with contrastive learning for cross-modal alignment, BIT reduces the word error rate (WER) the prior end-to-end method from 24.69% to 10.22%. Notably, we find that small-scale audio-LLMs markedly improve end-to-end decoding. Beyond record-setting performance, BIT aligns attempted and imagined speech embed- dings to enable cross-task generalization. Altogether, integration of large, diverse neural datasets, paving the way for an end-to-end de- coding framework that supports seamless, differentiable optimization.
approach advances the
INTRODUCTION
Speech neuroprostheties have experienced a recent revolution, enabling the decoding of attempted and imagined speech and restoring communication for those who have lost the ability to speak
(Akbari al. 2019 Makin et al. 2020 Moses al. 2021 Proix al. 2022 Willett al. 2023 Fan
Makin
et al. Metzger et al. 2023 Card et al. [2024] Littlejohn et al. 2025 Kunz et al. 2025). Despite this success, most existing systems rely on cascaded frameworks that cannot easily be optimized end-to-end. In such models, recurrent neural networks (RNNs) map neural activity to phonemes that then assembled into sentences with n-gram LM (Willett al., 2023). While efficient for small datasets, separate optimization of each component can result in dissociation between the performance of the RNN and the system as whole. For instance, lower phoneme error rates (PER) from the RNN do not always translate to lower word error rates (WER) when decoding with the n-gram model (Willett et al.) 2024). These limitations underscore the need for fully end-to-end optimized speech decoding framework.
Recently, FFng et al. (2024) connect RNNs with LLMs to translate neural activity into sentences in an end-to-end manner. While promising, this approach does not explore modern architectures, such as transformers, which may better capture complex neural representations and enhance decoding performance. In contrast, Feghhi al. (2025) incorporate transformers to decode phonemes and apply time masking to mitigate overfitting, achieving higher performance, but their study does not employ end-to-end training with L.LMs or leverage large-scale pretraining. Since transformers typi-

---

Published as a conference paper at ICLR 2026
cally require large datasets to reach their full potential, we hypothesize that combining transformer- based neural encoders with large-scale pretraining could further improve decoding performance and provide stable representations to effectively guide LLMs (Liu et al. 2023b).
Here, we present an end-to-ent BraIn-to- Molt--to-eex Text neural interface that we call BIT, which translates neural activity directly into coherent sentences. Our approach introduces several key innovations: transformer neural encoder pretrained with self-supervised masked modeling on 367 hours of Utah array recordings from humans and monkeys across speech and arm-related motor tasks; and, after fine-tuning, a flexible decoder that supports either a cascaded framework, where decoded phonemes are assembled into sentences with an n-gram LM, or an end-to-end framework, where an audio- LLM decoder directly generates sentences with contrastive alignment between neural and text em- beddings. Analogous to LLAVA (Liu et al., 2023b), which augments pretrained L.LM with an image encoder as its eyes for visual perception, BIT equips the L.M with brain to interpret neural activity. Through transformer- baseed pretraining and end-to-end L.LM integration, BIT overcomes the limitations of RNN-only encoders and small-scale datasets, yielding neural representations that generalize across tasks and align speech-related neural activity with the language structure of LLMs.
We evaluate our approach on the Brain-to-Tex Benchmark '24 and '25 datasets (Willett et al., 2023 Card et al. 2024), which consist of Utah array recordings from two human participants attempt- ing to speak sentences cued on computer monitor. First, we benchmark the pretrained encoder on attempted speech decoding, demonstrating that it outperforms both RNNs and transformers trained from scratch in the cascaded framework. Next, we evaluate LLM decoders, showing that small-scale audio-LLMs substantially improve end-to-end decoding performance, surpassing the text-based LLM used in prior work (Feng al.
2024). These gains are even more pronounced when transferring to imagined speech (Kunz et al. 2025), a low-data task with far fewer labeled ex- amples. Interpretability analysis reveals that BIT's neural embeddings for attempted and imagined speech align through shared semantic structure. These findings establish BIT as a new paradigm for integrating intracortical BCIs with L.MMs, enabling neural activity to guide sentence generation without relying on intermediate phoneme or character outputs. Overall, BIT represents step toward end-to-end speech BCIs that translate neural activity directly into coherent sentences.
RELATED WORK
Speech brain-computer interfaces. Early speech BCIs transform neural signals from auditory and motor cortices into intelligible outputs, enabling real-time and high-performance brain-to-text com-
munication (Bouchard et al. 2013 Akbari et al. 2019 Makin et al.|22200 Moses et al., 2021 Willett itt al. 20233 Fan et al. 2023 Card et al. 2024). These cascaded systems typically map activity to phonemes or characters before language model composition. Recent advances introduce transform- ers for long-range dependencies, streaming implementations, and end-to-end decoding with LLMs (Feghhi et al. [025] [Littlejohn et al. [dl]5002)) Feng et al., a 2024). Finally, inner speech decoding raises the possibility of communication without attempting to move (Kunz et al., (025). However, most prior work has relied on cascaded systems that cannot be trained end-to-end, while the previous end-to-end decoder, in turn, has relied RNN encoders without pretraining, not taking advan- tage of modern model architectures and training approaches. Our work closes this gap with a new end-to-end framework powered by cross-species, cross-task transformer-based pretraining-
Large-scale models for neural analysis. Increasing attention has been directed toward large-scale pretraining to learn generalizable neural representations across modalities including fMRI, calcium imaging, spiking activity, EEG, and EMG (Scotti et al., 2024 Thomas et al. [202]] Lurz et al. 2020
Azabou al.] [0233] 2024 Ryoo et al. 2025| Ye et al. [2023] 2025 Zhang et al. 2024 2025 Cui et al.] [024]] Kaifosh & Reardon 2025). ) These emerging "foundation models" " improve performance
while reducing reliance on task-specific data, and have shown success in behavior decoding, motor imagery, visual reconstruction, region and cell-type classification, mental state decoding, and neural activity prediction (Thomas et al. 2022; Scotti et al.) 2024, Cui et al., [0244] Azabou et al. 2024]
Yu et al. 2025; Wang et al. 2025). In spiking data, POSSM (Ryoo et al.) 2025) has demonstrated cross-species transfer from monkey to human imagined handwriting. Building on this progress, we explore cross-species, cross-task self- sulf-super pretraining for decoding human speech.
Modality alignment in multimodal language models. Recent advances in visual language models (VLMs) illustrate how large pretrained models can integrate disparate modalities. Early models such as CoCa (Yu et al.) 2022) rely on cross-attention projector for aligning billions of image- text pairs, achieving strong image-captioning performance but at high computational cost. Li et al. (023)
2

---

Published as a conference paper at ICLR 2026
Naural Activity from Utain Aray
Neural Representation
Willed
Decoded Text
Karpowicz
Nara-et
Pretrained
Neural Encodar
Audio-LLM Decodar
People should enjoy
Tocal (hours)
97.8
Churdtladd at 2012
15.8
c ('Debeeny at 2017
55.7
Autic-LMM Decodar
2013
Target egpp
Chowshury 2020
10.4
Evon Chon of 2002
tall 2003
21.2
15g
w.n22.5
Autoregressive Goneration
Dwosded
Text
Total (hours)
209.0
LoRA
Audio-LLM
Ne Neuraal Enooder
Peeinin
Finekune Neurl
Neusa Embeddings Projected to Text Lafort Space
Mone
Prompe Teot Embiactings
Projector
Modollty Nigner
Moan Pouling
Emtoedding Spece
Transformer Encodar
Transformer Encoder
Represeraaion
No Endlend
Beoomod Perck
LSE Loss
Neuraa Activtty
Figure 1: Schematic illustration of BIT. (A) BIT is an end-to-end speech decoding framework that translates neural activity directly into text by combining a cross-task, cross-species pretrained neural encoder with an audio-LLLM decoder. The data separately obtained and preprocessed from each study. (Appendix (A). (B) The neural encoder is transformer that embeds 20 bins of thresholded spikes and spike-band power into multi-bin time patches. It is pretrained using SSL with time-patch masking, reconstructing patch tokens via subject-specific linear read-in and read-out layers with an MSE loss. After pretraining, the masking module is removed, and the encoder is fine-tuned for phoneme decoding using linear classifier with CTC loss. (C) The neural encoder outputs mapped the text embedding space of an audio-LLM via shallow MLP projector. modality aligner trained with contrastive learning projects mean mean-poole pooled neural and text embeddings into a shared latent space for modality alignment. To guide decoding, we insert a prompt between neural and text embedding tokens: "decode the above neural activity into an English sentence:", ". During finetuning, we update the neural encoder, projector, and apply LoRA to the linear layers in the audio-I.LMMs attention and feed-forward blocks, while keeping other parameters frozen.
introduce a more efficient design by freezing pretrained visual and language backbones and training compact Q-former to align modalities. More recently, instruction-tuned L. such as LLaVA (Liu al. 2023a) replace query-based intermediates with direct projection layers mapping visual features into the LLM embedding space. This progression, from cross-attention to Q-formers to simple projections, demonstrates that as LLMs become more capable, modality alignment becomes less compute-intensive Inspired by these trends, we investigate whether LLMs can similarly generalize to speech-related neural signals.
3 METHODS
In this section, we present an end-to-end speech BCI consisting of transformer- neural en- coder and an audio-LLM decoder for speech translation (Figure [ The encoder and decoder are connected by a shallow MLP, trained with contrastive learning for modality alignment. The trans- former encoder is first pretrained with self-supervised masked modeling and then fine-tuned for phoneme decoding. Finally, this phoneme-awan encoder is fine-tuned to translate coherent sen- tences directly from neural activity using the audio-LLM..
3.1 DATASET
The Brain-to-Tex. Benchmark '24 and '25 datasets (Willett et al.) 2023 Card et al. 2024) are used for attempted speech decoding. The "24 dataset includes 12,100 sentences from participant TI2 across 25 sessions over four months, with 128-electrode Utah array recordings from ventral motor cortex. The '25 dataset contains 10,948 sentences from participant T15 across 45 sessions over 20 months, recorded with 256 electrodes in ventral motor cortex. For imagined speech, we the inner speech dataset (Kunz et al.) (025). comprising 500 and 712 sentences for T12 and T15, recorded with 128 or 256 electrodes while participants internally spoke cued sentences. For all
3

---

Published as a conference paper at ICLR 2026
datasets, neural features include thresholded spike counts and spiking-band power (SBP) Nason eet al. 2020). binned in 20 ms windows and normalized across days (z-scored) to address Utah array non-stationarity (Karpowicz et al.). For pretraining, we use ~ 98 hours of human Utah array recordings (including the above speech BCI datasets) and ~ 269 hours of monkey recordings during motor tasks (Willett al. [202]] 2023 Card al. [2024] Fan al. 2023 Karpowicz et al. 2024
Kunz et al. 2025 Churchland et al., 2012] Chowdhury et al.) [0200] Even-Cheee et al. 2019:] Ma
al. 2023
Perich al. 2018). Since SBP is unavailable in some pretraining datasets, we use only
thresholded spikes, normalized across days. See Appendix A for details.
3.2 ARCHITECTURE
Transformer neural encoder. We employ a transformer learn latent representations from neural activity. The original data has shape (T. C)), where T is the number of time bins and C' is the number of electrodes. Following Feghhi et al. (2025). we group every patch time bins into a patch, yielding data of shape (T TT/Tpii C x puich.. These patches are passed through a patch embedding module (LayerNorm, Linear, LayerNorm) to produce tokens for the transformer encoder with multi-heades attention. Using time patches rather than individual 20 ms bins aligns the finer temporal resolution of neural recordings with the slower timescale of speech production (30-60 words per minute). In addition, time patch tokens shorten the context length, minimizing information redundancy when feeding neural embeddings to LLMs for short sentence generation.
Each transformer block includes self-attention layer followed by feed-forward network (Ap- pendix QQ. Relative positional embeddings (RoPE) (Su et al. 2024) encode temporal information, and bidirectional attention mask allows each time patch to attend to all others. For self-supervised pretraining, neural activity from each subject is converted into transformer inputs via the patch em- bedding module, and the transformer outputs are reconstructed to the original neural data through the reversed patch embedding module, which projects latent tokens back to the input dimensionality. For phoneme decoding. transformer outputs are passed through a linear layer to generate logits over phonemes, the blank token, and the silence token.
LLM decoder. For text-LLMM,, encoder outputs are mapped into the text embedding space via a shallow MIP projector (Linear, ReLUU, Linear). Text is tokenized and embedded in the same space. A text prompt is inserted between neural and text embeddings to guide decoding, with design vary- ing depending on whether encoder outputs are treated as distinct modalities (Figure BB. During training, the model receives neural embeddings, text embeddings from prompt and target sentence, and generates text tokens via next-token prediction. inference, the model generates decoded text tokens autoregressively using only the neural embeddings and prompt. For audio-LLMs, neural en- coder outputs can be treated as a distinct neural modality and mapped into the text space, following the same procedure as for text-based LLMs. Since the encoder is pretrained on phoneme decoding, the resulting embeddings encode phonetic information. Even without audio tokens, the model can leverage speech knowledge acquired during LLM pretraining. When treated neural encoder outputs first pass through the MLP projector and are then mapped into the audio embedding space via the multimodal projector originally used for the audio encoder (Figure 3 In this case, neural activity can be treated as audio since it resembles speech waveforms.
audio modality,
We apply low-rank adaptation (LoRA) (Hu et al. 2022 to subset of LLM parameters, including at- tention projections and feed-forward layers, to enable fine-tuning with few trainable parameters. For audio-based L.MMs, LoRA is also applied to the multimodal projector, mapping neural embeddings into the audio embedding space to align with the model's 's acoustic representations.
Cross-modal alignment. After projecting encoder outputs into the text embedding space, a modality aligner projects mean-pooled neural and text embedding tokens into shared latent space using separate linear layers. The resulting 1.2-normalized vectors are optimized with contrastive loss to increase similarity between embeddings of matching sentences (Figure [C). See Section 3.3 and Appendix [ details.
3.3 TRAINING OBJECTIVES
Self-supervised pretraining using masked modeling. For pretraining, we utilize a temporal mask- ing strategy inspired by masked autoencoders (He et al., 2022). Neural activity is tokenized into temporal patches, and a subset is randomly replaced with learnable mask token. Masked patches can form contiguous spans of variable length, up predefined maximum timespan, while main-
4

---

Published as a conference paper at ICLR 2026
taining consistent overall masking ratio (Appendix Q The model is then trained to reconstruct the original neural activity from these partially masked sequences, encouraging the learning of rich contextual representations without external supervision. We use Mean Squared Error (MSE) as the reconstruction loss, since both thresholded spike counts and SBP are normalized. This masking strategy reduces overfitting through data augmentation and mitigates non-stationarity from probe drift (Karpowicz et al.) 2024). Additionally, pretraining is limited to human and primate Utah ar- ray data, allowing the model to learn stable probe representations that are robust to variations in electrode placement, subjects, and behavioral tasks.
Finetuning for phoneme-level decoding. After pretraining, the neural encoder is finetuned to pre- diet phoneme sequences from neural activity using a Connectionist Temporal Classification (CTC) loss (Graves et al.) 2006). The encoder transforms neural activity into latent embeddings, which are projected onto phoneme classes plus a blank token to produce phoneme logits. During fine- tuning, the time masking module is removed, since pretraining with extensive data augmentations has already mitigated overfitting. In the end-to-end model, phoneme logits are not fed the audio- L.M decoder; rather, they serve as intermediate targets that encode phonetic information into neural representations to guide L.LM fine-tuning, even when the goal is sentence prediction (Appendix (
Finetuning for sentence-level decoding. In the final stage, the neural encoder feeds neural represen- tations into the audio-LLM decoder, which is fine-tuned to predict the next token using cross-entropy loss against the ground-truth sentence. This objective aligns neural activity with the language struc- ture in LLMs, enabling coherent text generation. To reinforce modality alignment, we utilize a contrastive learning objective. The modality aligner in Section 3.2 pools neural and text embedding tokens into single "modality tokens" and projects them into shared latent space. Positive pairs are formed by neural and text embeddings from the same trial corresponding to the same sentence, while all other examples in the batch serve as negative pairs. This encourages the model to pull matching embeddings closer and push non-matching embeddings apart, allowing the model to learn representations that align with the semantic structure in the text embeddings. The total loss is the sum of the cross-entropy and contrastive losses (Appendix(J).
EVALUATION
4.1 TASKS
We evaluate our approach on two speech decoding tasks. For attempted speech, neural activity is decoded as the participant physically tries to articulate words, although no intelligible audio output is produced due to paralysis (Willett et al. 2023 Card et al. [024]. For imagined speech, neural activity is decoded as the participant silently imagines speaking without attempting to engage the speech muscles (Kunz et al. (025)_ Performance is measured by word error rate (WER) averaged across sentences, with lower WER indicating more accurate decoding.
4.2 BASELINES
We benchmark our method against baselines using two decoding strategies:
Cascaded decoder. We first decode phonemes from neural activity using neural encoder. Phoneme logits are passed into a 5-gram LM to generate sentence hypotheses, re-scored with OPT (Zhang eet al. 2022) (Appendix D.. Although not fully differentiable, this approach leads the lowest WER for Utah-array speech BCIs (Willett et al. 2024) and is used to benchmark baseline encoders:
1. RNN: Baseline RNN from prior speech BCIs (Willett et al. 2023 Card et al. 2024).
2. BIT-TFS: Transformer trained from scratch for phoneme decoding.
3. BIT-Human: Transformer pretrained on human data, fine-tuned for phoneme decoding.
4. BIT-All: Transformer pretrained on human monkey data, fine-tuned to decode phonemes.
5. BIT-Cross-Task--Onl Transformer is trained to decode attempted speech from the same subject, and then fine-tuned for decoding imagined speech to test cross-task transfer.
End-to-end decoder. The neural encoder trained for phoneme decoding is paired with a pre- trained LLM, fine-tuned to translate neural activity directly into sentences by feeding neural representations, rather than phoneme logits, into the LLM. We benchmark the same neural en- coders (RNN, BIF- BIT-Human BIT-All, B/T-Cross Cross-Task- (alsy-OOyy)) with various text- and audio- based LLMs (Section 5.2) using nucleus sampling (Holtzman et al. (2200) for sentence generation at inference time (Appendix F). Although end-to-end methods (Feng et al. 2024) have not yet sur- passed the cascaded decoder, our approach aims narrow this gap.
5

---

Published as a conference paper at ICLR 2026
A
Atterpited Speech
T32 (Cascaded)
T12 (Endito Eins
TI5 (Cascadiei
TES [End End
20.0
1.0
1.9
17.5
0.5
0.5
0.5
0.5
15.0
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
3.0
D.0
0.5
1.5
10.0
7.5
AAttempted Speech Exampies
Imagired Speech Examples
Encided
10 maalo de
Tagic
Dag clocer
Imagined Speech
Tagal
WWW
HrT-na BITAI
you Eigh goo
35.7
22
WTIPS
25.5
Togul
Taper
Docoled
acrolee dappticke 1oc 2uli Hurr the Dkot
Itar lesd Nedgaaccilig
Iraaded
Aaw you casi
20.6
Docoled
(e my ples
34.
Teget
Sapp
eim wy
10
Docited
ad werii horkil
1:1
bowiled
w/at
(Cascaded)
Endi
T15 (Gaocaeed
735 1ERDAG-EEN
Figure 2: Benchmarking BIT versus baxelines in atlempted and imagined speech decoding. (A) For attempted speech, the pretrained encoder (BIT-Human BIT-All) outperforms RNN and BIT-TFS using both cascaded and end-to-end approaches. Bar plots show mean WER across competition holdout sentences. (B) For imagined speech (50-word vocabulary), BIT-All outperforms all other baselines in both cascaded and end-to-end settings, Bar plots show mean WER across partitioned test sentences. SSL pretraining provides greater benefits for imag- ined speech than for attempted speech, since imagined speech has far fewer labeled examples. (C) Scatterplots compare BIT-All vs. B/T-Cross-Tic TaassTass--nni sentence and the green value showing relative improvement. Results show that SSL pretraining (cross-subject, unlabeled) yields larger transfer gains than SL pretraining within-subject, tross-task) after fine-tuning (D) Example decoded sentences from B/T-AIl using and-to-end approach. The imagined speech task has a smaller vocabulary (50 words) than attempted speech.
imagined speech decoding, with each dot representing test
EXPERIMENTS AND RESULTS
5.1 DECODING ATTEMPTED AND IMAGINED SPEECH
To benchmark attempted and imagined speech decod- Table 1: We benchmark BIT on the Brain-to- ing, models are trained and evaluated separately for each participant (T12 and T15). For attempted speech, we train on the Brain-to-Ted ing datasets, use the provided test set as the validation set for model selection, and report performance on the benchmark's 's holdout set (ground truth is unknown, but we obtain scores via submission to the competition). For imagined speech, we the training data provided by the original study, and split the evaluation set in half to create validation and test sets. WERs are computed the holdout or test trials for each participant. For abla- tion studies, we report performance on the validation set, as it is used to select the final model for benchmark sub- mission. See Appendix A for pretraining data splits.
Text '24 hold-out set (1200 sentences), com- paring it to other speech BCIs within the same decoding frroddiggfrmmewor framework (cascaded or end-to-end) for fairness. Background colors indicate parable methuds. See the |competttio leader- board for more details.
'24 and '25 benchmark train-
Competition Entry
WER
Feng et al. (0024) Hnd--ond-o--En
24.69%
15.67%
BIT End-to-End Ensemble
10.22%
RNN (Baseline)
9.76%
Feghhi et 2025 BIT Cascaded
7.98%
6.35% %
For attempted speech, we compare BIT-AlI, BIT-Human, BIT-TFS, and the RNN baseline. For imagined speech. we also include BIT-Cross-Task-OOe
Li al. (2024) Ensemble
5.77%
Feghhi et al. 22025 + Ensemble
5.68%
to evaluate within- subject, cross-task transfer. For both RNN and trans- former architectures, we perform hyperparameter tuning by initializing 30 models with optimizer hyperparameters randomly sampled from predefined ranges. We select the configuration with the best validation performance. See Appendix Q R for details. In our experiments, B/T-TFS outper- forms the RNN on attempted speech decoding (Figure 2 since the RNN does not use time mask- ing, this comparison reflects both architectural and training objective differences. Pretraining at scale -HTman,,,, BIT-All) further improves performance over training the encoder from scratch. For imagined speech (50-word vocabulary), BIT-Human and BIT-All outperform BIT-TFS by 39-
BIT Cascaded + Ensemble 5.10% %

---

Published as a conference paper at ICLR 2026
45% in WER, showing that SSL pretraining is especially beneficial in low-data settings. More- over, BIT-All exceeds B/T-Cross SIT-Crros-Task-Onf Task-Only (Figure[B-- B-CC), indicating that cross-subject, label-free SSL pretraining yields larger transfer gains than same-subject, cross-task supervised pretraining. Although BIT surpasses the prior end-to-end haseline (Feng et al., 2024) combining an RNN en- coder with text-based LLM decoder, gap in WER remains between cascaded and end-to-end approaches. Our method substantially narrows this gap, bringing end-to-end performance closer that of the cascaded decoder. Figure 2D shows example sentences decoded by BIT-All in the end-to-end framework.
To benchmark against other methods, we participate in the Brain-to-Text '24 and '25 competitions, where WER is measured on holdout set with ground-truth labels hidden from participants. For fair comparisons, we compare methods in the same decoding frame- work (cascaded end-to-end); see Appendix H for details. In Tables [ and 2 BIT Cascaded (5-gram LM) and BIT End (Aerol-Audio 1.5B) couple our pretrained encoder (BIT-AII) with different decoders. BIT Cascaded achieves a SOTA WER of 6.35% for non-ensembled models, surpassing the previous best of 7.98% (Feghhi et al., 2025). With a fine-tuned LLM that merges multiple decoded sentences from an ensem- ble of models trained with different seeds (Appendix G). BIT Cascaded + Ensemble further reduces WER to 5.10%, outperforming the prior 5.68% (Feghhi et al.]
Table 2: We benchmark BIT on the Brain-to- Text '25 hold-out set (1450 sentences), com- paring it other speech BCIs within the same decoding framework (cascaded or end- to-end) for fairness. Background colors indi- cale compuruble methods. See the competi- lion leaderboard for more details.
Competition Entry
WER
BIT End-to-End
11.06%
BIT End-to-End + Ensemble
7.76%
RNN (Baseline)
6.67%
BIT Cascaded
4.06%
RNN-TTA + Pseudo-Ensentn
4.42%
RNN Ensemble
3.09%
BIT Cascaded + Ensemble 1.76%
2025) and ranking first. BIT End-to-End Ensemble re- duces WER from 24.69% to 10.22% compared to the
previous end-to-end entry (Feng et al.]
2024). On the 225 public leaderboard. BIT Cascaded Ensemble leads with 1.76% WER, while BIT End-to-End + Ensemble achieves 7.76%.
5.2
ABLATING LARGE LANGUAGE MODEL DECODER
We conduct an ablation study to quantify the effects of LLM decoder type and model design on at- tempted speech decoding performance. Specifically, we compare text-based and audio-based LLMs, and test different prompt designs by treating neural embeddings as either audio or neural modalities (Section 3.2 and Figure B).. Finally, we evaluate how contrastive learning for cross-modal align- ment influences decoding performance. We focus on models with 1 7B parameters, which strike a balance between expressive capacity and adaptability for neural decoding with scarce labeled data. For text-based LLMs, we use Qwen2.5- 1.5B, Qwen2.5-7B (Qwen et al., 2025), Qwen3-0.66
and
Qwen3-1.7B (Yang et al. 2025). ). For audio-LLMs, we Acrol-Audis 1.5B (Li et al. (025), an audio-extended version of Qwen2.5-1.5B, and Qwen2-Audio-2 (Chu et al., 2024), a larger model capturing richer acoustic representations. Comparing these audio-extended models with their text-only counterparts allows us to isolate the impact of audio processing from general language modeling capacity.
Our experiments (Figure 3 BC-D) show that, for models of comparable size, audio-based LLMs out- perform text-based LLMS decoding, with Aerol-Audio 1.5B (Li et al. 2025), the audio-extended variant of Qwen2.5 1.5B, consistently achieving the best results. This suggests that pretraining LLMs on audio tasks provides inductive biases closer to the neural decoding problem, enabling a shallow MLP to align modality more effectively. Treating neural encoder outputs as a neural modal- ity performs slightly better than treating them as audio, indicating that while neural embeddings need not be interpreted as audio, LLMs still benefit from speech knowledge. We also find that smaller LLMs generally achieve lower WERs than larger LLMs when trained with limited labeled examples, since speech BCI tasks require English translation rather than advanced reasoning, which is typically seen in models above 7B parameters. Finally, using contrastive learning to align neural and text embeddings further improves performance.
5.3 ALIGNING NEURAL EMBEDDINGS FOR CROSS-TAS TASK GENERALIZATION
Although BIT is trained for speech decoding, we want to understand what semantic content is en- coded in its latent embeddings. To this end, we conduct sentence-level representational similarity
7

---

Published as a conference paper at ICLR 2026
LLM Decocier
Velidetion WER
itorstibuio INeurali
16.07
T12
LoRA
Audio-LLM
/NorST-Audia 1,3 (Andim
16.85
(B0/BB www.:/ JNPUEPO
Quenzza 1.3M Usvurat)
19.01
Quen2.5 (Neural)
19:45
Neural Audioo Embeddings
19:87
Embeddings
Qwenn 0..85 (Neural)
20.61
Muhlmmnde Projector from Audio Encodor
Cresnn/Auutiio (Neuroll
22.44
Qwen2-Audic (uudio)
22.77
Projector
Ammi-Andder (Audii)
8.79
T15
AccolAuuiio 1,58 CAudio)
9.19
Neurel Encoder
AFrAEF INPUEIO
9.80
Owwnzii
Neural Activity
Owie3 1.78 (Neural)
12:54
Prompt
Owon2:5 (Ncurat)
12.73
Project Audio: sjaudo, bosj-asjaudio cost-
franscribe the shove sudio into an English sentence:
Owon3 0.0B (Neural)
13:29
Project Neural: <neural wwt.iitt-sssinuurr..... aactvvlls decoci neural
Oweaz2.
(Neural)
English sentence:
Owanz-AAdie (Audio)
16.60
Figure 3: LLM decoder ablation across modality, model size, prompt design, and contrastive learning usage. (A) For audio-I.LMMs neural activity can be treated as either neural or audio modality. For neural modality, encoder outputs are projected directly into the emhedding space via MLP projector. For audio modality, neural encoder outputs pass through the MLP projector followed by a multimodal projector used for audio encoder. (B) Different prompts are used depending on whether neural encoder outputs are treated as neural or audio modality, with the encoder outputs inserted at the placeholder "#" (C-D) Bar plots show the mean WER across validation sentences for different LLM models, modality treatments, prompt designs, and contrastive learning usage. Here, we report validation WER, as it used select the final L.M decoder for benchmark submission. Colors distinguish text- (yellow) and audio-LLMs (bluc), with transparency indicating whether neural activity is treated as Neural or Audio modality, while diagonal hatching denotes that contrastive learning is not used.
analysis (RSA) (Kriegeskorte et al., 2008 to compare the representational geometry of neural em- beddings against that of LLMs. If neural embeddings preserve relational structures similar to LLMs, they may provide more informative features for decoding. Importantly, all neural encoders in the RSA analysis are taken prior to sentence-level fine-tuning to ensure that we compare the intrin- sic representational geometry rather than the effect of supervised adaptation (details Appendix L.I. Figure/44 shows the RSA scores between neural and audio-LLM, text embeddings. Pretrained encoder outputs are more similar to audio-LLM representations than those of RNN and BIT-TFS, indicating that large-scale pretraining facilitates linguistically structured representations.
We also want to understand which neural encoder learns the optimal mapping between attempted and imagined speech neural embeddings. As baseline, we apply PCA to neural activity, averaging embeddings across time and trials to obtain word-level embeddings, and visualize the top two PCs. For BIT-All, we apply PCA neural encoder outputs and project word-level embeddings onto the top two PCs. These embeddings are taken after sentence-leve fine-tuning to ensure they contain semantic information. For each word, we use Euclidean distance to quantify similarity between attempted and imagined embeddings (before PCA).
Figure 43 shows clear separation between attempted and imagined speech projections of neural activity. Since PCA focuses only on neural variance, this separation reflects differences unrelated sentence content. In contrast, BIT embeddings show much less separation (Figure 4C), indi- cating that BIT aligns neural embeddings across tasks after accounting for semantic information. Color intensity in Figure 4B-C indicates that embeddings from both tasks show higher similarity after pretraining. In addition, we fit a linear discriminant analysis (LDA) to find the projection that maximizes between-tasy separation while minimizing within-task variance. We then plot this dis- criminant axis as low-dimensions summary of task separability (Appendix L.2). This shows that task embeddings are linearly separable in original neural activity but not after alignment with BIT (Figure (Bggure4B-CC 4B-C).
To understand how neural embeddings relate to individual words over time, we use projectors that ML. projector in our experiments,
enable visualization of token-level interactions. While
8

---

Published as a conference paper at ICLR 2026
Imagined Attempted
Imagined Allempted
Imagined
Attertion
0.2
Pousun
0.2
you havc
0.4
T15
20-
PC
0.0
Aftomptod
2.5
Time In]
10.0
-2
02-
0.22
-4
SUCO866
0.0
2.5
Time
7.5
-2.5 PC1
2.5
PC
Haman
Figure 4: BIT aligns attempted and imagined speech neural embeddings to enable cross-task generalization. (A) Representational similarity analysis (RSA) scores between neural and audio-LLM text embeddings. (B) PCA embeddings of neural fcatures from participant T12 are visualized on the first two PCs. Word-level em- beddings averaged across time and trials and shown dots. The same words are shown for both tasks. Colors indicate tasks, and color intensity represents the distance between attempled and imagined speech em- beddings for each word (darker colors indicate higher similarity). The line shows the LDA linear discriminant.
(C) For BIT-AII, PCA is applied to neural encoder outputs from participant T12, with word-level embeddings from the top two PCs visualized as dots. Same plotting conventions as panel B. (D) Using cross- attention visualize attention weights, which reveal that neural text temporal alignment
projector in BIT allows similar across tasks.
linear and cross-attention projectors yield similar, slightly lower performance (Appendix K.. We visualize attention weights of a cross-attention projector, showing that embeddings from both tasks exhibit similar neural-text temporal alignment (Figure A)) These results further confirm that BIT can preserve semantic information to enable cross-task generalization.
DISCUSSION
Summary: In this work, we introduce an end-to-end speech decoding framework. Our method combines transformer neural encoder, pretrained on human and monkey Utah array recordings across speech and arm-related motor tasks, with either an n-gram LMM in cascaded framework or an audio-LLM for end-to-end optimization. In the cascaded setting, we show that cross-species, cross-task pretraining enables transfer to attempted and imagined speech after fine-tuning, achieving first place on the Brain-to-Texy '24 and '25 benchmarks. Integrated end-to-end with Aero lero--Audio 1.5B, audio-extended Qwen2.5-1.5B, BIT reduces WER by over 50% compared the previous best method. Finally, we demonstrate that BIT learns representations that are shared across imagined and attempted speech, enabling cross-task generalization.
Limitations and future directions: Our approach faces several challenges and opportunities for improvement. First, the end-to-end approach requires about 0.95 seconds per sentence average, which is slower than the cascaded approach (0.24 seconds) and inefficient for real-time BCIs. In addition, the neural encoder is trained with bidirectional attention to maximize performance, ren- dering it unsuitable for online decoding; switching to causal attention is feasible, albeit at some cost to decoding accuracy. Similarly, while we employ compact 1.5B audio-LLM, larger models cannot run on-device, further limiting real-time applications. Another consideration is pretraining: we find that human data yield larger gains than cross-species transfer, likely because human datasets include tasks more relevant to speech (and, to lesser extent, handwriting), whereas monkey reach- ing data are less applicable. Additionally, the neural encoder requires substantial unlabeled datasets to address sensor variability, and LLMs need large amounts of labeled data to outperform cascaded approaches. However, we have limited access to private human data. Finally, the LLM decoder itself can be improved through better modality alignment and prompt design, and there remains broader need to develop methods addressing non-stationarity (Le et al. 2025). neural plasticity, and user-interface co-adaptation for long-term BCI use.
Broader impact: This work improves current cascaded speech BCI systems to restore high- accuracy communication for users with paralysis. It also advances SOTA end-to-end models, mak- ing them easier to optimize and deploy in real-time applications. These developments could enable patients who are unable to communicate everyday tasks with Al assistance. At the same time, it raises important safety and privacy concerns, as decoding inner speech must require explicit user consent and reliable safeguards (Kunz et al.) 2025).
move interact with the outside world and perform

---

Published as a conference paper at ICLR 2026
REFERENCES
Hassan Akbari, Bahar Khalighinejad, Jose L Herrero, Ashesh D Mchta, and Nima Mesgarani. To- wards reconstructing intelligible speech from the human auditory cortex. Scientific reports, 9(1): 874, 2019.
Mehdi Azabou, Vinam Arora, Venkataramana Ganesh, Ximeng Mao, Santosh Nachimuthu, Michael Mendelson, Blake Richards, Matthew Perich, Guillaume Lajoie, and Eva Dyer. A unified, scalable framework for neural population decoding. Advances in Neural Information Processing Systems, 36:44937-44956, 2023.
Mehdi Azabou, Krystal Xuejing Pan, Vinam Arora, Ian Jarratt Knight, Eva Dyer, and Blake Aaron Richards. Multi-session, multi-task neural decoding from distinct cell-types and brain regions. In The Thirteenth International Conference on Learning Representations, 2024.
Kristofer E Bouchard, Nima Mesgarani, Keith Johnson, and Edward F Chang. Functional organiza- tion of human sensorimotor cortex for speech articulation. Nature, 495(7441):327-332, 2013,
Nicholas S Card, Maitreyee Wairagkar, Carrina Iacobacci, Xianda Hou, Tyler Singer-Clark, Fran- cis R Willett, Erin M Kunz, Chaofei Fan, Maryam Vahdati Nia, Darrel Deo, al. An accurate and rapidly calibrating speech neuroprosthesis, New England Journal of Medicine, 391(7):609- 618, 2024.
Raeed H Chowdhury, Joshua Glaser, and Lee E Miller. Area 2 of primary somatosensory cortex encodes kinematics of the whole arm. Elife, 9te48198, 2020.
Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, al. Qwen2- -audio technical report. arXiv preprint 10X55 2024.
Mark M Churchland, John Cunningham, Matthew Kaufman, Justin Foster, Paul Nuyujukian, Stephen Ryu, and Krishna V Shenoy. Neural population dynamics during reaching. Nature, 487 (7405):51-58) 2012.
Wenhui Cui, Woojae Jeong, Philipp Thölke, Takfarinas Medani, Karim Jerbi, Anand A Joshi, and Richard M Leahy. Neuro-gpt: Towards a foundation model for eeg. In 2024 IEEE International Symposium on Biomedical Imaging (ISBI), pp. 1-5. IEEE, 2024.
Nir Even-Chen, Blue Sheffer, Saurabh Vyas, Stephen Ryu, and Krishna V Shenoy. Structure and variability of delay activity in premotor cortex. PLoS computational biology, 15(2)ee0068088 2019.
Chaofei Fan, Nick Hahn, Foram Kamdar, Donald Avansino, Guy Wilson, Leigh Hochberg, Kr- ishna Shenoy, Jaimie Henderson, and Francis Willett. Plug-and-play stability for intracortical brain-computer interfaces: a one-year demonstration of seamless brain-to-text communication. Advances in neural information processing systems, 36:42258 42270, 2023.
Ebrahim Feghhi, Shreyas Kaasyap, Nima Hadidi, and Jonathan C Kao. Time-masked trans- formers with lightweight test-time adaptation for neural speech decoding. arXiv preprint arXiv:2507.02800, 2025.
Sheng Feng, Heyang Liu, Yu Wang, and Yanfeng Wang. Towards end-to-end framework for invasive brain signal decoding with large language models. arXiv preprint arXiv:2406. /1..006615566 11568, 2024.
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist tem- poral classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pp. 369-376, 2006.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
10

---

Published as a conference paper at ICLR 2026
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious of neural text degeneration. In International Conference on Learning Representations (ICLR), 2020. URL battps://arxiaa org/ abs 1904. 09751
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Patrick Kaifosh and Thomas Reardon. A generic non-invasive neuromotor interface for human- computer interaction. Nature, pp. 1-10, 2025.
Brianna M Karpowicz, Yahia Ali, Lahiru Wimalasena, Andrew R Sedler, Mohammad Reza Keshtkaran, Kevin Bodkin, Xuan Ma, Lee E Miller, and Chethan Pandarinath. Stabilizing brain- computer interfaces through alignment of latent dynamics. nov 2022. doi: 10.1101/2022.04. 06.487388. URL https:///www. biorxiv. org/contenn/0.11110002222
6:v2.
Brianna M Karpowicz, Joel Ye, Chaofei Fan, Pablo Tostado Marcos, Fabio Rizzoglio, Clay Wash- ington, Thiago Scodeler, Diogo de Lucena, Samuel R Nason- Tomaszewski, Matthew J Mender, et al. Few-shot algorithms for consistent neural decoding (falcon) benchmark. Advances in Neural Information Processing Systems, 37:76578-
2024.
Nikolaus Kriegeskorte, Maricke Mur, and Peter A Bandettini. Representational similarity analysis- connecting the branches of systems neuroscience. Fronfiers in systems neuroscience, 2:249, 2008
Erin M Kunz, Benyamin Abramovich Krasa, Foram Kamdar, Donald Avansino, Nick Hahn, Sconghyun Yoon, Akansha Singh, Samuel R Nason-Tommszenn Tomaszewski, Nicholas S Card, Justin Jude, et al. Inner speech motor cortex and implications for speech neuroprostheses. Cell, 2025.
Trung Le, Hao Fang, Jingyuan Li, Tung Nguyen, Lu Mi, Amy Orsborn, Uygar Sümbül, and Eli Shlizerman. Spint: Spatial permutation-invarian neural transformer for consistent intracortical motor decoding. arXiv preprint 20XXi:2507008402 2025.
Vladimir I. Levenshtein. Binary codes capable of correcting deletions, insertions, and reversals. Soviet Physics Doklady, 10:707-710, 1966.
Bo Li, Chen Change Loy, Pu Fanyi, Yang Jingkang, Zhang Kaichen, Hu Kairui, Thang Luu Minh, Trung Nguyen Quang, Cong Pham Ba, Liu Shuai, Wang Yezhen, and Liu Ziwei. Acro: Audio- enhanced large language models. 2025. URL https: www. www.w.lmmmn Imms - llab com/posts/ aero audio/l
Jingyuan Li, Trung Le, Chaofei Fan, Mingfei Chen, and Eli Shlizerman. Brain-to-text decoding with context-aware neural representations and large language models. Journal of Neural Engineering, 2024.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023. URLhtttss://axxiv.co http://pggaaaaa....///////
Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E Gonzalez, and lon Stoica. Tune: A research platform for distributed model selection and training. In Proceedings of the 2018 Workshop on ML Systems al NeurIPS, 2018.
Kaylo T Littlejohn, Cheol Jun Cho. Jessie R Liu, Alexander B Silva, Bohan Yu, Vanessa R An- derson, Cady M Kurtz-Miott, Samantha Brosler, Anshul Kashyap, Irina P Hallinan, et al. A streaming brain-to-voice neuroprosthesis restore naturalistic communication. Nature neuro- science, pp. 1-11, 2025.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Llava: Large language and vision sistant. arXiv preprint arXiv:2304.08485 2023a. URL
08485
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34916, 2023b.

---

Published as a conference paper at ICLR 2026
Iyy Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.0S101, 2017.
Konstantin-Klemenn Lurz, Mohammad Bashiri, Konstantin Willeke, Akshay K Jagadish, Eric Wang, Edgar Walker, Santiago A Cadena, Taliah Muhammad, Erick Cobos, Andreas Tolias, et al. Generalization in data-driven models of primary visual cortex. BioRxiv, pp. 2020-10, 2020.
Xuan Ma, Fabio Rizzoglio, Kevin L Bodkin, Eric Perreault, Lee E Miller, and Ann Kennedy. Using adversarial networks to extend brain computer interface decoding accuracy over time. elife, 12: e84296, 2023.
Joseph G Makin, David A Moses, and Edward F Chang. Machine translation of cortical activity to text with an encoder-decoder decoder framework. Nature neuroscience, 23(4)3755-52, 2020.
Sean L Metzger, Kaylo T Littlejohn, Alexander B Silva, David A Moses, Margaret P Seaton, Ran Wang, Maximilian Dougherty, Jessie Liu, Peter Wu, Michael A Berger, al. A high- performance neuroprosthesis for speech decoding and avatar control. Nature, 620(7976):0337- 1046, 2023.
David A Moses, Sean L Metzger, Jessic R Liu, Gopala Anumanchipalli, Joseph Makin, Pengfei F Sun, Josh Chartier, Maximilian E Dougherty, Patricia M Liu, Gary M Abrams, et al. Neuroprosthesis for decoding speech in a paralyzed person with anarthria. New England Journal of Medicine, 385(3):217-233 2021.
Samuel Nason, Alex K Vaskov, Matthew S Willsey, Elissa Welle, Hyochan An, Philip P Vu, Autumn J Bullard, Chrono S Nu, Jonathan C Kao, Krishna V Shenoy, et al. A low-power band of neuronal spiking activity dominated by local single units improves the performance of brain- machine interfaces. Nature biomedical engineering, 4(10):973-98) 2020.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. In arXiv preprint arXiv: 2018.
Matthew G Perich, Juan A Gallego, and Lee Miller. A neural population mechanism for rapid learning. Neuron, 100(4):964-99)) 2018.
Timothée Proix, Jaime Delgado Saa, Andy Christen, Stephanie Martin, Brian Pasley, Robert Knight, Xing Tian, David Poeppel, Werner K Doyle, Orrin Devinsky, et al. Imagined speech can be decoded from low-and cross-frequency intracranial eeg features. Nature communications, 13 (1):48, 2022.
Qwen, An Yang. Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025.
URL httpt:ttt///ttt /arpii orgg abs 2412 15115
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00026. 2021.
Avery Woon Ryoo, Nanda H Krishna, Ximeng Mao, Mehdi Azabou, Eva I. Dyer, Matthew G Perich, and Guillaume Lajoie. Generalizable, real-time neural decoding with hybrid state-space models. arXiv preprint urXiv:2506.05320, 2025.
Paul S Scotti, Mihir Tripathy, Cesar Kadir Torrico Villanueva. Reese Kneeland, Tong Chen, Ashutosh Narang, Charan Santhirasegaran, Jonathan Xu, Thomas Naselaris, Kenneth A Norman, et al. Mindeye2: Shared-subjed models enable fmri-to-image with 1 hour of data. arXiv preprint arXiv:2403.11207, 2024.
Jianlin Su. Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En hanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024.
12

---

Published as a conference paper at ICLR 2026
Hisham Temmar, Yixuan Wang. Nina Gill, Nicholas Mellon, Chang Liu, Luis Hernan Cubillos, Rio I Parsons, Joseph Costello, Matteo Ceradini, Madison M Kelberman, et al. Long-term intracorti- cal neural activity and kinematics (link): An intracortical neural dataset for chronic interfaces, neuroscience, and machine learning. The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025.
Armin Thomas, Christopher Ré, and Russell Poldrack. Self-supervised learning of brain dynamics from broad neuroimaging data. Advances in neural information processing systems, 35:21255- 21 21269, 2022.
Eric Y Wang, Paul Fahey, Zhuokun Ding, Stelios Papadopoulos, Kayla Ponder, Marissa A Weis, Andersen Chang, Taliah Muhammad, Saumil Patel, Zhiwei Ding, et al. Foundation model of neural activity predicts response to new stimulus types. Nature, 640(8058):400-477, 2025.
Francis R Willett, Donald T Avansino, Leigh R Hochberg, Jaimie M Henderson, and Krishna V Shenoy. High-performance brain-to-text communication via handwriting. Nature, 593(7858): 2021.
Francis R Willett, Erin M Kunz, Chaofei Fan, Donald T Avansino, Guy H Wilson, Eun Young Choi, Foram Kamdar, Matthew Glasser, Leigh R Hochberg, Shaul Druckmann, al. high- performance speech neuroprosthesis, Nature, 62017976) 2023.
Francis R Willett, Jingyuan Li, Trung Le, Chaofei Fan, Mingfei Chen, Eli Shlizerman, Yue Chen, Xin Zheng, Tatsuo S Okubo, Tyler Benster, et al. Brain-to-text benchmark 24: Lessons learned. arXiv preprini arXiv:2412.17227, 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09.00 09388, 2025.
Joel Ye, Jennifer Collinger, Leila Wehbe, and Robert Gaunt. Neural data transformer multi- context pretraining for neural spiking activity. Advances in Neural Information Processing Sys- tems, 30:803662-8037777 2023.
Joel Ye, Fabio Rizzoglio, Adam Smoulder, Hongwei Mao, Xuan Ma, Patrick Marino, Raeed Chowd- hury, Dalton Moore, Gary Blumenthal, William Hockeimer, et al. A generalist intracortical motor decoder. bioRxiv, 2025.
Han Yu, Hanrui Lyu, Ethan Yixun Xu, Charlie Windolf, Eric Kenji Lee, Fan Yang, Andrew M Shelton, Shawn Olsen, Sahar Minavi, Olivier Winter, al. In vivo cell-type and brain region classification via multimodal contrastive learning. bioRxiv, pp. 2024-11, 2025.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captiomers are image-text foundation models. arXiv preprint arXiv:2205.01917. 2022.
http:p/ss//a.iiv.oorrrrs///22/// 01917
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li. Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205. 2006000000000 01068, 2022.
Yizi Zhang, Yanchen Wang, Donato Jiménez-Benetó, Zixuan Wang, Mehdi Azabou, Blake Richards, Renee Tung, Olivier Winter, Eva Dyer, Liam Paninski, et al. Towards a" universal translator" for neural dynamics at single-cell, single-spike resolution. Advances in Neural Infor- mation Processing Systems, 37:80495-8000 -00221 2024.
Yizi Zhang, Yanchen Wang, Mehdi Azabou, Alexandre Andre, Zixuan Wang, Hanrui Lyu, The International Brain Laboratory. Eva Dyer, Liam Paninski, and Cole Hurwitz. Neural encoding and decoding at scale. arXiv preprint arXiv:2504.08201, 2025.
13

---

Published as a conference paper at ICLR 2026
A
DATASET DETAILS
This section documents the studies and sources of all datasets used for pretraining, as well as for attempted and imagined speech decoding, and provides a brief description of each.
A.1 HUMAN DATA
Willett et al. 2021). This dataset contains recordings from participant with hand paralysis who attempted and imagined handwriting, decoded in real time by an intracortical BCI. Data are publicly available at the DRYAD repository. Only threshold crossings from 192 electrodes are available.
Willett et al. (2023). This dataset is associated with the Brain-to- Text Benchmark '24 and comes from a study in which researchers developed speech BCI that decodes neural activity from intracor- tical microelectrode arrays to reconstruct speech in real time for participant with amyotrophic lat- cral sclerosis (ALS). The dataset contains sentences with both a 50-word vocabulary and a 125,000- word vocabulary. Neural recordings during attempted speech are publicly available at the DRYAD repository. Both threshold spikes and spike-band power (SBP) are provided from 256 electrodes. However, only 128 of them for decoding, as find that this empirically leads to the best performance.
Fan et al. (2023). This dataset comes from a study in which researchers introduced a framework al- lowing speech BCI to self-recalibrate without interrupting the user. Neural recordings are publicly available at the DRYAD repository, with only threshold crossings available from 192 electrodes.
Karpowiez et al. (0024). This dataset is from the Few-shot Algorithms for Consistent Neural De- coding (FALCON) Benchmark] which evaluates intracortical BCI decoders for stability across ses- sions. We only the human handwriting subset, also associated with Fan al. (2023), available at DANDI The threshold spike data in this subset also include 192 electrodes.
Card al. 2024). This dataset associated with the Brain-to- Text Benchmark '2 and comes from study in which speech BCI decoded neural activity from the left precentral gyrus of a person with ALS and severe dysarthria. Neural recordings during attempted speech, containing a 125,000-word vocabulary, publicly available at the DRYAD repository, with both threshold spikes and SBP available from 256 electrodes.
Kunz et al. 2025). This dataset comes from a study in which speech BCI decodes imagined speech from motor cortex activity to enable real-time communication without vocalization. It includes recordings and sentences from four participants during attempted, imagined and perceived speech tasks, publicly available at the DRYAD repository, with both threshold spikes and SBP from 256 electrodes. For pretraining, we use all four participants, but decoding uses only T12 and T15.
A.2 NON-HUMAN PRIMATES (NHP) DATA
Churchland et al. 2012). This dataset comes from a study in which neural population activity primary motor cortex (M1) recorded from non-human primates (NHPs) during center-out reaching tasks. It includes thresholded spikes from 192 electrodes, publicly available at DANDI.
O'Doherty al. (2017). This dataset originates from study in which researchers recorded neural activity from the sensorimotor cortex of NHPs during self-paced reaching tasks. It includes thresh- olded spikes from two monkeys, recorded from 192 electrodes. The data are publicly available at the Zenodo Z repository.
Perich et al. 2018). This dataset is from study in which researchers recorded neural activity from MI and dorsal premotor cortex (PMd) of four macaque monkeys during reaching tasks. Neural data are publicly available at the DANDI repository with thresholded spikes. The data contains spike sorted activity three subjects, and threshold crossings in the fourth subject from 192 electrodes.
Chowdhury et al. 2002)). This dataset comes from study in which researchers recorded neural activity from area 2 of the primary somatosensory cortex rhesus macaques during reaching tasks. It includes thresholded spike data from 92 channels, publicly available at the DRYAD repository.
14

---

Published as a conference paper at ICLR 2026
Even-Chen et al. (2019). This dataset originates from study in which researchers recorded neural activity from PMd in rhesus macaques during an instructed-delay center-out reaching task. It in- cludes thresholded spike data from 96-channel Utah arrays, publicly available at the DANDI repos- itory.
Ma al. (2023). This dataset from experiments in which researchers recorded neural activity from MI in rhesus macaques across many days while the animals performed various motor tasks (wrist force, grasping, manipulandum reaching, etc.). They used thresholded multi-unit spike data from 96-channel microelectrode arrays. The dataset is publicly available at the DRYAD repository.
Temmar al. 2025). This dataset comes from study in which researchers recorded neural ac- livity from MI in a rhesus macaque performing self-paced finger movement task. It includes thresholded multiunit spikes from 96-channel Utah microclectrode arrays, together with behavioral measurements of finger positions and velocities. The dataset is available at the DANDI repository.
A.3 DATA PREPROCESSING
For all datasets, resample neural activity into 20 ms time bins (if not already provided this resolution) and apply z-scoring across days to mitigate non-stationarity. Without this normalization, speech decoding WERs increase substantially due to severe data drift. In addition, when SBP is available, we include it in decoding, as we find empirically that SBP contributes more to decoding accuracy than thresholded spikes alone. Combining thresholded spikes and SBP leads to the lowest WER (Table 3). Human datasets generally contain higher-quality recordings, while some monkey datasets include dead channels. If fewer than two dead channels are detected, interpolate them with the mean neural activity over time. though we note that more careful handling would be prefer- able. Datasets with more than two dead channels are excluded from pretraining.
Participant
Thresholded
SBP
Thresholded SBP
T12
21.35%
18.64%
17.26%
Table 3: Ablation of features used for speech decoding. Validation PER is reported.
Regarding data quality, we also considered other human and monkey datasets not listed but ulti- mately excluded them from pretraining, as their inclusion led to reduced pretraining performance. This may be due to data quality issues that require more careful examination in future work, which is beyond the scope of the present study.
A.4 DATA SPLITS FOR PRETRAINING
As SSL. pretraining does not rely on labels, all neural data from the datasets can be incorporated into the training set, including the evaluation (validation) and holdout datasets provided by the Brain- to-Text '24 and '25 benchmarks. However, since our primary interest is evaluating whether the transformer effectively captures the neural representations of participants T12 and T15, use the validation datasets from these participants to select the optimal model checkpoint, based on the metric for neural activity reconstruction quality.
B METRICS
R2 for neural activity reconstruction. We use the coefficient of determination (R²) as the metric to evaluate neural activity reconstruction quality during SSL pretraining, The R² score measures the proportion of variance in the target neural activity that can be explained by the model's predictions, with higher values indicating better reconstruction. During pretraining, the validation R² is used to select the model checkpoint for downstream speech decoding.
PER. We use 41-token phoneme vocabulary, including all phonemes plus silence and the CTC blank token. The phoneme error count is first computed by removing repeats and blanks from the decoded phoneme sequences, then calculating the Levenshtein edit distance (Levenshtein, 1966) to the ground-truie truth sequences, counting substitutions, insertions, and deletions. The phoneme error rate is then obtained by normalizing the error count by the total number of ground-truth phonemes.
15

---

Published as a conference paper at ICLR 2026
WER. Word error rate is computed similarly to PER. Blank and special tokens removed from the decoded sentences to obtain word sequences. The Levenshtein edit distance between the predicted and ground truth word sequences is then calculated, counting substitutions, insertions, and deletions. WER obtained by normalizing the total error count by the number of words in the ground-truth sequences.
C PHONEME-LEVEL DECODING
For brevity, intermediate phoneme decoding results are omitted from the main text. Nonetheless, we observe that neural encoders achieving lower PERs consistently lead to lower WERs, whether combined with 5 gram LM in the cascaded framework with LLM decoder in the end-to-end framework. In the main text, we present the holdout or validation WERs for each baseline neural encoder; in this section, provide the validation PERs for the baseline encoders, summarized in Table[4
Neural Encoder
Attempted Speech
Imagined Speech
T12
T15
T12
T15
RNN
18.67%
9.64%
30.81
24.56%
BIT-TES
17.26%
8.87%
25.01% %
21.21%
BIT-Human
15.95%
7.61%
19.63%
18.83%
BIT-AIl
14.39%
7.12%
18.08%
17.94%
Caroos-Task-On
20.46%
19.58%
Table 4: Phoneme decoding benchmark. The metrics shown are the validation PER.
In the end-to-end model, phonemes serve intermediate targets that encode phonetic information into neural representations to guide LLM fine-tuning, even when the final goal is sentence prediction. We observe that, without chooodng training, simply pretraining the neural encoder large data corpus does not lead to low WERs during the sentence-level decoding stage, highlighting the importance of phonetic information for speech decoding tasks. This phenomenon is also reported in previous end-to-end methods (Feng et al. 2024), which employ a two-stage decoding strategy that trains the model for phoneme prediction before sentence prediction.
D CASCADED DECODER
The cascaded decoder consists of 5-gram LM and OPT model for re-scoring the LM outputs. In this section, we briefly describe the resources and procedures used: for comprehensive details, readers referred the original studies (Willett al. 2023 Card et al. 2024)
5-gram LMM. We use the 5-gram LM provided by the Brain-to- Brain-to-Text Benchmark '24 and '25 (Willett al. 2023 Card et al. (2024) to decode word sequences from neural encoder outputs. The pretrained model is publicly available on DRYAD. The 5-gram LM is created using Open Web'eex2 corpus and therefore has large vocabulary, constructed from 634 million sentences with 99 billion words. The LM decoder performs an approximate Viterbi (beam) search, which maps CTC phoneme label sequences from the encoder to words. During inference, the beam search integrates state transition probabilities with the encoder's predicted CTC phoneme label probabilities to find the most likely word sequences. See Section 8 of the supplementary matcrials for details on constructing the 5 S-gram report the parameters used for beam search LM and performing inference with it. In Table 5 with a 5-gram L.M
OPT re-scoring. Given the sentence hypotheses from the 5-gram LM, an OPT model (Zhang et al., 2022) is employed to rescore the candidate sentences and further improve decoding accuracy. A publicly available pretrained OPT with 6.7B parameters is used to rescore the n-best outputs from the n-gram LMM The re-scoring procedure considers both the probability of the sentence's 's corresponding
CTC phoneme label sequence output by the encoder (acoustic score) and the sentence probability estimated by the n-gram LM; Section 8.5 of the supplementary materials for details the
equation used for OPT re-scoring and the LM decoding parameters. In Table 5 we report the parameters used for language model re-scoring with OPT.
16

---

Published as a conference paper at ICLR 2026
Hyperparameter
Value
Beam Size (Number of Candidate Sentences)
100
Acoustic Score Scale Factor
0.325
Blank Penalty
90
OPT Re-Score Scale Factor (a:)
0.55
Table 5: Hyperparameters used for inference with the cascaded decoder (5-gram LM with beam
search + OPT sentence re-scoring).
E END-TO-END DECODER
Text-based LLMss We include models from the Qwen2.5 and Qwen3 families. Qwen2.5-1.5B
2025) is mid-sized LLM trained with an improved pretraining pipeline and instruc- tion tuning. We also include Qwen2.5-7B. To examine the effect of model size and architectural advances, we include
(Qwen et al.
recent models from the Qwen3 series: Qwen3-0.6B (Yang et al. 20225), a compact model optimized for efficiency, and Qwen3-1.7B, a larger variant designed to provide stronger language modeling capacity while remaining computationally tractable. Together, these models provide a representative set of text- text-based LLMs at approximately 1B scale, enabling com- parison with audio-augmented counterparts under similar parameter regimes.
Audio-based LLMs. In addition to text-only baselines, we consider audio-augmented LLMs. Aerol-Audio 1.5B (Li et al., 2025) extends Qwen2.5-1.5B with audio front-end enabling the model to process acoustic representations alongside text. To understand the effect of model scale in the audio domain, also include Qwen2-Audio 7B (Chu al. 2024), a larger audio-based LLM capturing richer acoustic and semantic features. Comparing Aerol-Auce Auddio 1.5B and Qwen2- 7B with text-only models of similar size allows us quantify the contributions of audio conditioning relative to purely textual modeling in speech decoding from neural activity.
F LLM INFERENCE FOR END-TO-END DECODING
Text generation uses nucleus (top-p) sampling with parameter of p 0.9 and temperature of 0.7, generating up to 25 tokens per sequence. Top-p sampling restricts token selection to the smallest set of tokens whose cumulative probability exceeds p, focusing on high-likelihood tokens while allowing some diversity. The temperature controls randomness: lower values produce more confident, deterministic outputs, while higher values increase variability. Although more advanced strategies such beam search can be used, they significantly increase inference time when the number of beams exceeds five. Therefore, we use nucleus sampling for efficiency. We acknowledge that better LLM inference strategies could further improve end-to-end decoding performance, but this is beyond the scope of the present work.
G MODEL ENSEMLBING USING LLM MERGER
To further improve the decoding accuracy of the cascaded and end-to-end decoders for benchmark submissions, we employ model ensembling approach with LLM merging (Li et al. (024). While the 5-gram and OPT models can partially correct phoneme decoding errors, their outputs are not always perfect, and different neural encoders produce varying sentence candidates. To mitigate these errors, a fine-tuned L.LM can be used to evaluate multiple sentence candidates from an ensemble of cascaded end-to-end encoders. Using both the transcription candidates and their corresponding phoneme sequences as inputs. Li et al. (2024) finetuned an LLM to generate the correct sentence. This setup enables the model learn the relationship between predicted phonemes and decoded sentences, as well as to identify common, model-specific errors across decoder outputs. For both cascaded and end-to-end approaches, we provide the candidate sentences along with the phoneme outputs from neural encoders that are randomly initialized with different seeds to the fine-tuned ILM, which then generates the final best sentence used for evaluation.
H DRAIN-TO-TEXT BENCHMARK '24 AND '25 DETAILS
Brain-to-Text Text Benchmark '24. The top scores the leaderboards for both cascaded and end-to- end methods are achieved using model ensembling with LLM merging. For this benchmark, we initialize 10 cross-species, cross-task pretrained encoders with different random seeds, each fine-
17

---

Published as a conference paper at ICLR 2026
tuned to decode sentences from participant T12 data. Each model is trained on the training data provided by the benchmark and uses the provided validation set for checkpoint selection. Each of the 10 models outputs its own phoneme and sentence sequences, which are then provided to a fine- tuned GPT-3.5 to select the final best sentence sequence. This procedure is applied to both the entries BFT Cascaded Ensemble and BIT End-to-End + Ensemble, which the top-ranked submissions for the cascaded and end-to-end approaches on the leaderboard. All of our other leaderboard entries based on a single model with single seed. For information on the other leaderboard entries, see Willett et al. (024)..
Brain-to-Tex Benchmark '25. The top scores on the public leaderboards for both cascaded and end-to-end methods are achieved using model ensembling with LLM merging. For this benchmark, we initialize 29 cross-species, cross-task pretrained encoders with different random seeds, each fine- tuned to decode sentences from participant T15 data. We fine-tune GPT-4 merge the phoneme and sentence outputs from these 29 models. All of our other leaderboard entries are based on single model with a single seed. In Table 2 the RNN (baseline) uses a single RNN encoder with 5-gram LM, labeled "UCD-NPL. causal RNN + 5gram." The RNN Ensemble combines mul- tiple RNNs with LLM merging. labeled "Stanford-NPTLL causal RNN Ensemble + Sgram." The RNN-TTA +
generates diverse outputs from a single RNN model via noise injec- tion and selects the final sentence with an LLM merger; it is listed as "Stanford-NPTL causal RNN TTA-Ensemble + 5gram," where TTA stands for test-time adaptation.
CONTRASTIVE LEARNING FOR MODALITY ALIGNMENT
We align neural and text embeddings in a shared latent space using contrastive learning. Let 2 RTxD XD and 2 E RLXD denote the neural and text embeddings. respectively, for the i-th sample in batch of B trials. Here, the neural embeddings are obtained by projecting the neural encoder outputs through the MLP projector into the text embedding space. Each of z E Rxx and z R is first mcan-pout -aoulp to a single vector by averaging across its sequence dimension (T' and L), then projected into shared latent space of dimension P via modality-specifff linear layers, followed by l normalization. We denote the resulting modality-spedn -ppeiiicc vectors as & E R and & E R... which we refer to the "modality tokens."
Contrastive learning is then applied to pull together corresponding neural and text modality tokens (positive pairs) while pushing apart tokens from different samples (negative pairs) within training batch. Following a symmetric InfoNCE loss (Oord et al., 2018 Radford et al. 202T), the loss for batch is defined as
C conntaattvve
1
Σ 1=1 Σ/=1
B log
exp(z] Zt/T)
log
exp(2 2 /T
2B
B
exp(X) (z) ///T) E/=1 exp(z) Z3/T)
where T is a learnable temperature parameter that scales the cosine similarities.
In experiments, apply a learnable scaling factor to the similarity logits when computing the contrastive learning objective. This factor is parameterized by a learnable temperature, initialized 0.1 and constrained maximum value of 100. Due to the large size of the LLMs and limited computational resources, we use small batch size (Table 13p for contrastive learning. While this batch size enables measurable performance gains in the current study, it may limit the effectiveness of contrastive learning, which typically benefits from larger batch sizes to better sample negative pairs. Future work should explore larger batch sizes to fully realize the potential of contrastive learning and further improve cross-modal alignment.
J Loss FUNCTION FOR BIT
We train the end-to-end model with cross-entropy loss for neural-to-text translation and contrastive loss for neural-text alignment. Let y; = (Ji.1..... L., denote the ground-truth token sequence for the i-th sample in batch of size B. The predicted probability distribution over vocabulary of size V at the t-th token is denoted as p.,. = ff (concat(z), 2()) E RRY, where f represents the I.LM decoder, and the neural and text tokens are concatenated along the sequence dimension. The
18

---

Published as a conference paper at ICLR 2026
cross-entropy loss for the batch is then defined
LCB
log (o..))
The total training objective is given by Lerr Lce Leontrastive
K
PROJECTOR FOR LLM DECODER
To project neural embeddings into the LLM embedding space, we explore different projectors, including linear, MLP, and cross aotos-attntionn modules. We benchmark these projectors to evaluate their impact on end-to-end decoding performance. Results are summarized in T
Projector Variant
Imagined Speech (T12) Imagined Speech (T15)
BIT (Lincar Projector)
19.47%
18.09 %
BIT CCross-Attention Projector)
17.33%
17.67%
BIT (MLP Projector)
16.39%
13.61
Table 6: Ablation of the projector in BIT: The metries shown validation WER.
Cross-attention projector: This module applies cross-attention from neural tokens (queries) to text tokens (keys and values), enabling token-level interactions between modalities. Input neural and text embeddings, z" and z', are first normalized with layer normalization and projected linearly to a shared hidden dimension dhisten. Multi-headee attention is then applied, followed by a learnable residual connection to stabilize training. The attended outputs are finally projected back to the text embedding dimension dext. Formally, the module computes
CrossAttenntionPooicctrrrt zz),
where z" are the aligned neural-text representations and A are the attention weights (Figure(4p). The architecture includes tunable parameters such as the hidden dimension dhidden. number of attention heads headss drop out ratio, and learnable residual scaling factor.
Hyperparameter
Value 256
Hidden Dimension (dhista)
Number Heads (Raeads)
Learnable Residual Scaling Factor
Dropout Ratio
0.1
Table Hyperparameter used for the cross cross-attention projector.
L INTERPRETABILITY ANALYSIS
L.1 REPRESENTATIONAL SIMILARITY ANALYSIS
We restrict the analysis to sentences with neural time-patch token lengths between 45 80 for participant T12 and 120 ~ 200 for participant T15, ensuring sufficient temporal resolution and bal- anced coverage (see Figure 5) Neural embeddings converted into fixed-length vectors using segmented mean pooling, where each sequence is divided into 10 segments and the segment av- erages are concatenated preserve coarse temporal dynamics. contrast, LLM embeddings are reduced to sentence vectors via mean pooling across tokens. For each modality (ncural and text), we compute a representational dissimilarity matrix (RDM) (Kriegeskorte et al.) 2008) using one minus the cosine similarity. To compare representational structures, we extract the upper-triangular entries of each RDM and compute the Pearson correlation coefficient between neural and LLM RDMs. The resulting RSA score quantifies how well the geometry of neural embeddings aligns with lan- guage structures in LLMs. Framing RSA as an interpretability tool. this analysis demonstrates that consistent with language-les structure than RNNs. RSA is applied only to attempted speech data from participants T12 and T15 the studies Card al. 2024).
the transformer encoder learns representations
Willet al. 2023
19

---

Published as a conference paper at ICLR 2026
Distribution Neural Token Lengtns
Distribution of Neura Token Lengths
T12
T15
used for RSA inteppetabillity analysis
used interpratabilly analysis
1000
800
100
600
Court
200
400
Token
Tovon
Figure 5: Distribution of neural token lengthy across sentences for RSA. We restrict RSA to sentences with token lengths between 45 and 80 (mean length 63) for participant T12 and between 120 and 200 (mean length 160) for participant T15, since neural embeddings are converted into fixed-length sentence vectors by dividing cach sequence into ten temporal segments and concatenating their averages. Sequences that are too short lack sufficient resolution, while overly long sequences introduce imbalance. These ranges ensure reliable and comparablc sentence-level embeddings for cross-modal RSA.
L.2 LDA AND EMBEDDING DISTANCES
We fit an LDA using scikit-learn. For each word, neural embeddings are first projected onto the PCA subspace and then averaged across time and trials to produce single feature vector. The resulting low-dimensional neural projections are used as features to fit the LDA to predict trial type (attempted vs. imagined speech for binary classification). We restrict LDA a single dimension to identify an axis that linearly separates neural projections from the two tasks for visualization purposes. However, when computing Euclidean distances between attempted and imagined speech embeddings, use the original high-dimensional embeddings for each word before PCA projection to ensure comparisons reflect both direction and magnitude in the embedding space. These analyses are applied only attempted and imagined speech data from participants T12 and T15 in the study Kunz et al. 2025).
M EFFECTS OF SUPERVISED CROSS-SUBJES PRETRAINING ON DECODING
We investigated whether supervised cross-subject pretraining could enhance downstream speech de- coding performance. Incorporating data from multiple subjects during phoneme- and sentence-level decoding led to significantly worse performance compared single-subject training. This per- formance degradation may arise from inter-subject functional variability (differences in how neural activity relates to behavior) rather than sensor variability (e.g.. electrode placement or noise), SSL cross-subject pretraining does improve performance after fine-tuning on single-subject data. These results indicate that, with the current model architectures and datasets, supervised cross-subject pre- training does not improve performance, and more sophisticated modeling strategies for addressing inter-subject functional variability will be needed in future work.
N DECODING ERROR ANALYSIS
N.1 PHONEME-LEVE: DECODING ERRORS OF THE NEURAL ENCODER
We performed a detailed decoding error analysis to characterize model behavior at the phoneme level. For each sentence, we first tokenized the predicted and ground truth phoneme sequences, then computed the edit distance. To study systematic substitution patterns, we aligned every pre- dicted sequence with its corresponding ground truth using a dynamic programming algorithm that explicitly models match, substitution, deletion, and insertion operations. Insertions and deletions were represented with a dedicated null token so that both sequences remain length matched after alignment.
20

---

Published as a conference paper at ICLR 2026
Phoneme Confusion Metrix (Percentage)
or
SH
Uk
1.
du
NG
Predicedd phonerme
Ar
H
AD
M-
EY
on
Thue shoneme
Figure 6: Phoneme-level decoding matrix. Predicted and ground truth phoneme sequences aligned, correct matches are removed, and all remaining errors are normalized to percentages. Phonemes ordered by total error frequency, with truc phonemes on the horizontal axis and predicted phonemes on the vertical axis. The visualization highlights dominant substitution patterns and systematic decoding errors.
Based on the aligned pairs, we aggregated all observed phoneme confusions across the validation For every occurrence of a true phoneme aligned with predicted phoneme, we incremented global confusion counter. This produced full phoneme by phoneme confusion matrix that includes substitutions, deletions, and insertions. To emphasize the structure of decoding errors, we sorted phonemes by their total off diagonal error counts. The matrix was then visualized with the most error prone phonemes placed the bottom of the vertical axis and the right side of the horizontal axis, which makes high error regions visually salient.
Inspection of the matrix further reveals several prominent confusion clusters. As show Figure /D/ and /T/ show frequent reversals, /B/ and /P/ exhibit substitution, and /S/ often confuses with /Z7. In addition, the central vowels /AH/, /IH/, and /EH/ are commonly misdecoded due to their acoustic proximity. Overall, the observed error structure matches known phonetic similarity patterns and appears consistent with expected decoder behavior.
N.2 WORD-LEVEL DECODING ERRORS OF THE LLM DECODER
To examine word level decoding errors, we first computed the full confusion matrix across all pre- dicted and true words. Figure a shows the complete matrix. Because includes every word arranged alphabetically, the matrix is very large and individual error pixels can only be seen at ex- tremely high resolution. Even so, the matrix exhibits a clear diagonal structure, indicating that the model achieves high word level decoding accuracy. Most predictions fall near the correct location, and the observed errors tend to occur between words that are similar in spelling or phonological form.
At the same time, we note that many errors occur only once. These sparse, non recurrent mistakes behave more like noise relatively more frequent error types, we filtered the matrix to retain only word pairs that appeared in least errors. After this filtering, the total number of error events dropped from 734 to 58. The
off events rather than consistent confusion patterns. To highlight the
21

---

Published as a conference paper at ICLR 2026
Word leve Confusion on Matrix (all errors)
b
Word-level Contusion Matrix (at least errors, sorted)
10
3,0
25
2.5
Predictod word
0.5
0.5
00
0.0
A
Z
Tuee word
word
Figure 7: Word-level decoding error matrix. (a) Word level confusion matrix computed over all decoding errors. Words are arranged alphabetically from to for both axes. Because the vocabulary large, the full matrix must viewed at a very large scale to see individual pixels clearly, A clear diagonal structure appears, indicating that most decoding mistakes occur between words with similar spellings or phonological forms. (b) Word level confusion matrix after filtering to retain only word pairs that exhibit at least two errors This highlights the systematic confusions and removes the many resulting matrix reveals the dominant recurrent error patterns more clearly.
entries present in panel a. The
WER (%)
Human (cascaded)
Human (end-to-end) Monkey (end-to-end)
5
Monkey (cascaded)
12
4
11
200
400
200
Data size (hours)
Data size (hours)
Figure 8: Impact of progressively increasing the proportion of human versus monkey pretraining data on attempted-speed decoding performance for participant T15. Across both cascaded and end-to-end models, "human" corresponds the fraction of human pretraining data progressively added, whereas "monkey" indi- cates the fraction of monkey pretraining data added on top of the human data.
filtered matrix is shown in Figure 7b.. Although the maximum count after filtering is still only three, which indicates that the model does not develop highly stable confusion patterns at the word level, these repeated errors least reveal small but observable tendencies in handful of word pairs.
Overall, these analyses show that the model performs well at word level decoding. Errors are limited number and largely reflect marginal or near neighbor confusions. Even the repeated errors occur only in a very small subset of word pairs, underscoring the generally high decoding accuracy.
O SCALING CURVE
We examine how gradually increasing the amount of human versus monkey pretraining data affects attempted speech decoding performance for participant T15. In the human-only condition, ex- pand the pretraining corpus by progressively adding human Utah-array datasets, starting from no human data and reaching the full 97.8 hours. In the cross-species condition, we begin with the complete human corpus and then incrementally incorporate monkey Utah-array datasets, ultimately extending the total pretraining duration to 366.8 hours across both species. Figure 8 shows how de- coding performance changes as these proportions vary, and highlights that adding human data yields larger performance gains than augmenting the model with monkey data.
22

---

Published as a conference paper at ICLR 2026
P ABLATION OF ENCODER PRETRAINING
Our pretraining procedure includes the human neural data used later for finetuning, but discards all labels. Whether to include such datasets during pretraining depends the type of generalization being evaluated: excluding them tests generalization to unseen data, whereas including them tests generalization to unseen tasks. Because our objective is to improve transfer to speech-decoding tasks, we choose to include the available human speech datasets during pretraining. This also al- lowed to make full use of the limited human data to help stabilize the learned representations. To test generalization to entirely unseen data, however, we also repeated pretraining while excluding all human speech datasets, As shown in Table including excluding the human neural data used for finetuning leads to no substantial difference in attempted speech decoding performance.
(a) Cascaded model
(b) End-to-end model
Finetune data included
Finetune data excluded Finetune data included
Finetune data excluded
T12 6.35%
7.24%
15.67%
15.34%
T15
4.06%
4.36%
11.06%
11.00%
Table Impact of including versus excluding fine-tuning datasets during pretraining on attemmted speech de- coding performance. Excluding the fine-tuning datasets tests gencralization to unseen data, whereas including them tests generalization to unseen tasks.
secluding.fne-tunning
In Figure 2 the comparison between BIT-All and BIT-Cross- Task- Kask-OOly for imagined speech decod- ing does not control for differences in data size, as our goal was to include additional data beyond same-participant data to fully benefit from SSL pretraining. To control for data size differences, we also conducted an experiment comparing SSL and supervised pretraining using equal data sizes, When controlling for data size, we find no substantial performance
with results reported in Table 9 difference between SL and SSL pretraining for imagined speech decoding.
T12 (Cascaded) T12 (End-to-End)
BIT- BIT-Cross- Trr--Cros-Tak---C Only BIT SameParticipant--S SSL
12.53%
15.71%
12.67%
15.64%
Table 9: Impact of SL versus SSL pretraining using equal amounts of human speech data on imagined speech decoding performance. WVT-Cros-TTask-Oe is pretrained using SI., whereas BIT-SamePParticipane is pre- trained using SSL, and both use the same human speech data from participant T12.
Q MODEL AND HYPERPARAMETER DETAILS
Transformer Encoder. We implement a transformer encoder with an architecture similar to Feghhi ct al. (2025). Since their official code is not publicly available at the time of this submission, we con- struct our own architecture using custom model parameters listed in Table TO To select the optimal model architecture and optimizer configuration for training single-subject data (T12 and T15), we use Ray Tune (Liaw et al.) 2018) to randomly sample 30 optimizer hyperparameter (batch size, weight decay, and learning rate) combinations from the ranges listed in Table [2] The optimizer hyperparameters for single-subject phoneme-level decoding is selected based on these ranges. For imagined speech decoding, we reuse the same model hyperparameters for both participants. For pretraining, we reuse the same model architecture while adjusting the learning rate, weight decay and batch size when scaling to human and monkey datasets, as SSL pretraining performance (evaluated using the validation RD metric for neural activity reconstruction) shows no decline, indicating that the model capacity is sufficient to handle the data variability. The transformer model comprises ~ 7 million pa- rameters. The inclusion of subject-specific patch embedding modules and linear decoders increases the total number of model parameters to 13 million.
x 10-4, x 10-5 and 64, respectively. We do not increase the model depth
RNN Encoder. For the RNN encoder, we use the Py Torch implementation provided by the bench- mark host in the official GitHub / repository. We retain the original model architecture (Table []] and apply Ray Time (Liaw et al. 2018) to randomly sample 30 optimizer hyperparameter (batch size, weight decay, and learning rate) combinations from the ranges listed in Table 12 using attempted
23

---

Published as a conference paper at ICLR 2026
Hyperparameter
Value
Embedding Dimension
384
Head Dimension
512
Number of Heads
6
Depth
7
Mask Ratio
0.5 (T12) and (T15)
Max Mask Time Span
15
Patch Size
5
Dropout Rate
0.2
Bidrectional
True
Attention Dropout Rate
0.4
White Noise Standard Deviation
0.2
Constant Offset Standard Deviation
0.05
Gaussian smoothing width
2.0
Table 10: Hyperparameters used for the transformer-basee encoder.
speech data from participants T12 and T15. For imagined speech decoding, we reuse the same model hyperparameters for both participants, except for the number of day-specific read-in layers, which differs across datasets.
Hyperparameter
Value
Embedding Dimension
1024
Depth
5
Mask Ratio
0.1
Dropout Rate
0.4
Stride Length
4
Kerne Length
32
Numher of Day Layers
24 (T12) and 45 (T15)
Bidrectional
True
White Noise Standard Deviation
0.8
Constant Offset Standard Deviation
0.2
Gaussian smoothing width
2.0
Table 11: Hyperparameters used for the RNN encoder.
Hyperparameter
Value Range
Batch size
[8, 32, 64]
Weight Decay
Log-Uniforte x 10 0.1)
Learning Rate
1og-Uniform55 10-5, 0.001)
Table 12: The range of possible optimizer hyperparameters for training the transformer and RNN for phoneme- level decoding, from which Ray Time randomly samples combinations.
LLM Decoder and LoRA. We employ the same configuration for the MLP projector and LoRA across all text- and audio-based LLM decoders (see details in Table 13). For text-based LLMss, the MLP hidden dimension is set to match the hidden dimension of the LLM. For audio-LLMs, the MLP hidden dimension is set to match that of the audio encoder's multimodal projector. Depending on the LLM decoder, LoRA is applied to different modules. For all decoders, we apply LoRA to the query, key, value, and output projection layers of the multi-headed self-attention mechanism, well as the projection layers in the feed-forward MLP. For audio-based Qwen models, we additionally apply LoRA to the linear layer of the multimodal projector. The hyperparameters for cross-modal alignment with contrastive learning are described in the Appendix[]
R TRAINING DETAILS
Transformer Encoder. For both pretraining and fine-tuning, we train the model using the Adam optimizer (Loshchilov & Hutter 2017). Encoders pretrained human-only human-and-monn datasets are trained on a single NVIDIA A100 GPU (80 GB memory) in under 2 days for total of 400 epochs. Phoneme decoding is performed single NVIDIA A40 (48 GB memory) or A100
24

---

Published as a conference paper at ICLR 2026
Hyperparameter
Value
Activation
ReLU
Learning Rate
5 10-5
Weight Decay
10-5
Batch Size
16 (T12) and 8 (T15)
Gradient Accumulation Step
(model < 7B) and (model 2 7B)
LoRA Rank
8
LoRA Scaling Factor
32
LoRA Dropout Ratio
0.2
Table 13: End-to-end model hyperparameters., including MLP projector, the LLM decoder LoRA.
GPU (40 or 80 GB memory) in less than 8 hours for T12 and less than 1 day for T15, for 800 epochs. For hyperparameter tuning, we use 4 NVIDIA A40 GPUs for phoneme decoding, completing the process in less than 2 days. During training. model checkpoint, based on the highest R² or lowest PER. We employ mixed precision (bfloat16) to improve training efficiency for both pretraining and fine-tuning, and all models fit single NVIDIA GPU.
use validation metrics (Rt2, PER) to select the best
RNN Encoder: We train the model using the A optimizer. Phoneme decoding is performed on single NVIDIA A40 (48 GB memory) or A100 GPU (40 or 80 GB memory) in less than day for T12 and less than 2 days for T15, for 600 epochs. For hyperparameter tuning, we use 16 NVIDIA A40 GPUs for phoneme decoding, completing the process less than 2 days. Checkpoint selection follows the same procedure as for transformer-baa based models.
LLM Decoder. For all decoders, train using the AdamW optimizer with bíloat precision for 150 epochs. For transformer encoders, sentence-level decoding requires less than day for T12 and less than 2 days for T15 single NVIDIA A40 A100 GPU. For RNN, this process takes less than 1.5 days for T12 and less than 4 days for T15 on single NVIDIA A40 or A100 GPU. Small-sized LLMs, including 0.6B, 1.5B, and 1.7B models, are trained on a single NVIDIA A40 or L40 GPU (48 GB memory). Larger LLMs around 7B parameters are trained on two NVIDIA A40 or L40 GPUs with DeepSpeed ZeRO-3 optimization. all cases, checkpoint selection is based on the validation WER.
THE USE OF LARGE LANGUAGE MODELS (LLMs)
We used LLMs to polish the writing of this paper. Specifically, LLMs were applied to improve grammar and clarity. No scientific contribution results was generated by LLM, and all AI- assisted edits were verified by the researchers.
25


Reviews:
[{"review_id": "2OTAN1ij2P", "reviewer": "Reviewer_GRt5", "summary": "Advancing speech brain-computer interfaces (BCIs) to restore communication for individuals with paralysis, this paper (under review at ICLR 2026, titled *Decoding Inner Speech with an End-to-End Brain-to-Text Neural Interface*) addresses the key limitation of existing cascaded frameworks—where recurrent neural networks (RNNs) decode phonemes first before assembling sentences with n-gram language models (LMs), preventing end-to-end optimization and often causing mismatches between phoneme error rate (PER) and word error rate (WER)—by proposing the end-to-end BraIn-to-Text (BIT) framework, whose core is a transformer neural encoder pretrained via self-supervised masked modeling on 367 hours of Utah array data (98 hours of human speech/handwriting recordings and 269 hours of monkey motor task recordings) to enable cross-task (attempted and imagined speech) and cross-species representation transfer; BIT supports two decoding configurations: a cascaded setting (pairing the pretrained encoder with an n-gram LM, achieving state-of-the-art (SOTA) word error rates (WERs) of 5.10% on the Brain-to-Text ’24 holdout set and 2.82% on the ’25 holdout set with model ensembling, outperforming prior methods like Feghhi et al. (2025)) and an end-to-end setting (integrating the encoder with audio large language models (LLMs) such as Aero1-Audio 1.5B, optimized via contrastive learning for cross-modal alignment between neural and text embeddings, which reduces the prior end-to-end WER from 24.69% (Feng et al., 2024) to 10.22% with ensemble, while small-scale audio-LLMs outperform text-based LLMs and BIT aligns neural embeddings of attempted and imagined speech to enable cross-task generalization—especially valuable for low-data imagined speech tasks); the work also notes limitations including slower end-to-end inference (~0.95 seconds per sentence vs. 0.24 seconds for cascaded methods), bidirectional attention unsuitable for real-time decoding, and reliance on limited human data, but overall advances a differentiable, integrable framework for direct neural activity-to-coherent text translation.", "questions": "1. Why include monkey motor task data in encoder pretraining? No clear link to human speech neural activity is given.\n2. For imagined speech, is \"data size difference\" controlled when comparing BIT-All and BIT-Cross-Task-Only?\n3. Without removing monkey data in ablation, is it proven such cross-species data is essential for pretraining?\n4. Why prioritize audio LLMs over text ones? Their bias for neural-to-text decoding lacks clear support.\n5. When comparing cascaded/end-to-end frameworks, are LLM differences (size, data) controlled for fairness?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Novel end-to-end BIT framework: Replaces traditional cascaded systems (phoneme decoding followed by n-gram language models) with a single differentiable neural network, solving the dissociation between phoneme error rate (PER) and word error rate (WER) while enabling joint optimization of all stages.\n2. Cross-task and cross-species pretrained encoder: Pretrained via self-supervised masked modeling on human (speech-related tasks) and monkey (motor tasks) Utah array data, it outperforms RNNs and Transformers trained from scratch, and sets a new state-of-the-art (SOTA) on the Brain-to-Text benchmarks.\n3. Audio LLM integration with contrastive alignment: Integrates audio large language models (LLMs) in the end-to-end setting, and uses contrastive learning to achieve cross-modal alignment between neural and text embeddings; small-scale audio LLMs show better performance in end-to-end decoding than text-based LLMs.\n4. Strong cross-task generalization: Aligns the embeddings of attempted and imagined speech (leveraging shared semantic structure), enabling effective decoding of the imagined speech task, which has scarce labeled data.\n5. Outstanding performance: In the cascaded setting (especially with model ensembling), it achieves SOTA results on the holdout sets of the Brain-to-Text benchmarks; the end-to-end setting significantly narrows the performance gap with cascaded frameworks.", "weaknesses": "1. Slow end-to-end inference: The end-to-end approach is less efficient than the cascaded one, making it unsuitable for real-time brain-computer interfaces (BCIs).\n2. Incompatibility with online decoding: The neural encoder uses bidirectional attention to maximize performance, which cannot be applied to online decoding; switching to causal attention will sacrifice some decoding accuracy.\n3. Limitations of large models in on-device use: Larger audio large language models (LLMs) cannot run on devices, further restricting real-time applications.\n4. Limitations of pretraining data: Human data contribute more to performance gains than cross-species transfer (monkey reaching data are less relevant to speech), and access to private human data for pretraining is limited.\n5. Room for improvement in LLM decoders and long-term BCI adaptation: The LLM decoder needs better modality alignment and prompt design; there is also a lack of solutions for non-stationarity, neural plasticity, and user-interface co-adaptation required for long-term BCI use.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nAhaicB3uS", "reviewer": "Reviewer_pds5", "summary": "This paper presents BIT, an end-to-end framework for brain-to-text speech decoding aiming to translate neural activity directly into coherent sentences. The core contribution include a transformer-based neural encoder pretrained on large-scale, cross-task and cross-species neural datasets, integration with Audio-LLM. The proposed method achieved a lower word error rate (WER) on the test set compared to the baseline.", "questions": "Q1: In the BIT loss, the cross-entropy loss and the contrastive loss are simply added together. What would be the effect if the weighting between these two loss terms were adjusted?\nQ2: The training details in Appendix O show that hundreds of epochs are used for training the different components. Is there any theoretical justification for such a large number of epochs, and does this raise concerns about potential overfitting?\nQ3: The motivation for using an Audio-LLM requires further explanation. Why not adopt a VLM or Omni model instead?\nQ4: The reason for adopting LoRA is not entirely clear. Would full-parameter training yield better performance?\nQ5: The concepts of attempted speech and imagined speech are mentioned multiple times early in the paper, but they are not explained until Section 4.1. It would improve readability to introduce and clarify these concepts earlier in the text.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "S1: This paper builds an end-to-end architecture, integrating neural signals into a multimodal LLM and performing modality alignment.\nS2: This paper conducts additional analyses, such as ablation studies across different models and comparisons of the representational geometry of neural embeddings against that of LLMs.\nS3: This paper provides a relatively thorough discussion of its limitations.", "weaknesses": "W1: The evaluation metrics only involve WER, which seems insufficiently comprehensive.  Although the paper introduces finetuning for sentence-level decoding, it evaluates performance solely at the word level. Incorporating sentence-level evaluation metrics, such as semantic similarity, would likely provide a more complete assessment. In addition, a more detailed analysis of the error words should be provided. For example, does the same word sometimes appear correctly and sometimes incorrectly? Are common words recognized more accurately? Are words that appear infrequently in the training data more prone to errors?\nW2: The paper introduces cross-species (monkey) data for training. Although this approach is relatively novel, the paper lacks further discussion on how the monkey data affect the training performance. I am also interested in understanding the impact of different proportions of monkey and human data on the results.\nW3: The end-to-end architecture proposed in the paper appears to underperform cascade systems in terms of both effectiveness and efficiency in practical applications, which raises doubts about the necessity of adopting an end-to-end framework for the brain-to-text task. In particular, given the limited amount of data available in this domain, it is questionable whether an LLM-based end-to-end architecture can be effectively trained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1xWAZj5SWo", "reviewer": "Reviewer_1Ni1", "summary": "This paper investigates the use of self-supervised learning (SSL) combined with Transformer architectures for decoding inner speech from EEG signals. The authors pre-train an SSL model on a large-scale EEG dataset and evaluate it on several downstream decoding tasks. The work aims to bridge the gap between recent advances in representation learning and neural decoding applications.", "questions": "Since the contribution is primarily empirical, it would be helpful to better understand the theoretical or conceptual motivation behind combining SSL and Transformer.   For instance, what properties of EEG data make Transformer architectures particularly suitable?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper addresses an important and technically challenging problem: decoding inner speech from EEG, a task that remains largely open and scientifically valuable.  \n- The experimental design is systematic and the paper provides clear descriptions of the model, datasets, and training procedures.  \n- Empirical results show consistent improvements over baseline methods, indicating that the approach is effective in leveraging large-scale pre-training for EEG representation learning.", "weaknesses": "1. **Potential Data Leakage in Pre-training**  \n   The SSL model was pre-trained on a large EEG dataset that includes data from some subjects who also appear in the test set. Although SSL does not rely on explicit labels, this overlap means that data from the test set contributed to the model’s representational space. As a result, the reported improvements might partially reflect familiarity with certain subjects rather than genuine generalization to unseen data. Clarifying or mitigating this issue would be important for ensuring the validity of the conclusions.\n\n2. **Characterization of Model Novelty**  \n   The Introduction suggests that prior work \"does not explore modern architectures, such as Transformer.\" This statement may be somewhat overstated. The Transformer architecture has been extensively studied across many fields, including neural data modeling. The novelty here lies more in its application to *inner speech decoding* rather than in the architecture itself. The framing could be refined to more accurately reflect this distinction.\n\n3. **Limited Theoretical Innovation**  \n   The paper primarily combines existing components—SSL pre-training, Transformer encoders, and standard EEG decoding techniques—on publicly available data. While the integration is well executed and empirically valuable, the theoretical innovation appears limited. The contribution may therefore be more in engineering practice than in advancing fundamental understanding in machine learning or neuroscience.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VmO7zd8L9b", "reviewer": "Reviewer_6BdX", "summary": "This work introduces BIT, an end-to-end brain-to-text speech decoding framework that translates intracortical neural activity into text by combining a transformer-based, cross-task/cross-species pretrained neural encoder with an audio large language model (LLM) decoder. The authors design an architecture that supports both cascaded and end-to-end decoding by leveraging self-supervised masked modeling pretraining on extensive human and monkey datasets, phoneme-level finetuning, and contrastive learning for cross-modal neural-text alignment. The method is evaluated on the Brain-to-Text Benchmark '24 and '25 for both attempted and imagined speech.", "questions": "See concerns.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1. The manuscript is well written and easy to follow. Numerous tables and figures provide detailed results, ablations, and interpretability analysis, helping readers understand model choices and results. The supplemental materials provide many clarifying details and results.\n\n2. Based on the well-tuned architecture in Time-Masked Transformer [7], MAE-based pre-training further improves `~0.5%` in two speech decoding datasets [2,8] in Table 1.\n\n3. Explore the effects of audio LLMs for end-to-end speech decoding, instead of LLMs used in [9].", "weaknesses": "1. The significant performance drop in the end-to-end model (Fig. 2A) casts doubt on the effectiveness of the proposed Audio-LLM Decoder (Fig. 1C). This finding is consistent with prior work [4].\n\n2. The pre-training approach lacks neuroscientific justification. It aggregates intracranial data from different brain regions (e.g., motor [1] vs. speech production [2]) and species, without explaining how such disparate neural computations can share a common latent representation. Perhaps applying more data augmentation (e.g., timing jittering) during pre-training and using a single subject’s data would be enough to compensate for the reported performance differences (Figure 2). Besides, the motivation for removing time masking during fine-tuning is unclear (Line 223), which is a core module in Time Masked Transformer [7].\n\n3. Using audio models for supervision is not novel [3], where the results indicate that audio latent guidance is less effective than sequential phoneme labels under phoneme error rate (PER) evaluation.\n\n4. The use of Masked Autoencoder (MAE) pre-training is a well-established technique and does not represent a significant methodological novelty [5,6].\n\n5. The proposed BIT model appears to have a significantly higher training cost than the Time-Masked Transformer baseline. To better contextualize this, the authors should include a runtime comparison, detailing the reported 260-hour pre-training time and the computational overhead of the Audio-LLM Decoder.\n\n**References**:\n\n[1] Willett F R, Avansino D T, Hochberg L R, et al. High-performance brain-to-text communication via handwriting[J]. Nature, 2021, 593(7858): 249-254.\n\n[2] Willett F R, Kunz E M, Fan C, et al. A high-performance speech neuroprosthesis[J]. Nature, 2023, 620(7976): 1031-1036.\n\n[3] Metzger S L, Littlejohn K T, Silva A B, et al. A high-performance neuroprosthesis for speech decoding and avatar control[J]. Nature, 2023, 620(7976): 1037-1046.\n\n[4] Jiang W B, Wang Y, Lu B L, et al. NeuroLM: A universal multi-task foundation model for bridging the gap between language and EEG signals[J]. arXiv preprint arXiv:2409.00101, 2024.\n\n[5] Ye J, Collinger J, Wehbe L, et al. Neural data transformer 2: multi-context pretraining for neural spiking activity[J]. Advances in Neural Information Processing Systems, 2023, 36: 80352-80374.\n\n[6] Kapoor J, Schulz A, Vetter J, et al. Latent diffusion for neural spiking data[J]. Advances in Neural Information Processing Systems, 2024, 37: 118119-118154.\n\n[7] Feghhi E, Kaasyap S, Hadidi N, et al. Time-Masked Transformers with Lightweight Test-Time Adaptation for Neural Speech Decoding[J]. arXiv preprint arXiv:2507.02800, 2025.\n\n[8] Card N S, Wairagkar M, Iacobacci C, et al. An accurate and rapidly calibrating speech neuroprosthesis[J]. New England Journal of Medicine, 2024, 391(7): 609-618.\n\n[9] Feng S, Liu H, Wang Y, et al. Towards an end-to-end framework for invasive brain signal decoding with large language models[J]. arXiv preprint arXiv:2406.11568, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)