Time-Masked Transformers with Lightweight Test-Time Adaptation for Neural Speech Decoding
Authors: Ebrahim Feghhi
Shreyas Kaasyap
Nima Ryan Hadidi
Jonathan Kao
Venue: NeurIPS 2025 poster ()
Keywords: Neuroscience
Speech Decoding
Speech Neuroprostheses
Brain Computer Interface
Primary area: neuroscience_and_cognitive_science

Abstract: Speech neuroprostheses aim to restore communication for people with severe paralysis by decoding speech directly from neural activity. To accelerate algorithmic progress, a recent benchmark released intracranial recordings from a paralyzed participant attempting to speak, along with a baseline decoding algorithm. Prior work on the benchmark showed impressive accuracy gains. However, these gains increased computational costs and were not demonstrated in a real-time decoding setting. Here, we make three contributions that pave the way towards accurate, efficient, and real-time neural speech decoding. First, we incorporate large amounts of time-masking during training. On average, over $50\%$ of each trial is masked. Second, we replace the gated recurrent unit (GRU) architecture used in the baseline algorithm with a compact Transformer. The Transformer architecture uses $83\%$ fewer parameters, cuts peak GPU memory usage by $52\%$, and is significantly faster to calibrate relative to the GRU. Third, we design a lightweight variant of an existing test-time adaptation method developed for decoding handwriting from neural activity. Our variant adapts the model using multiple time-masked augmentations of a single trial and requires only one gradient step per trial. Together, these contributions reduce word error rate by over $20\%$ and effectively mitigate performance degradations across held-out days in a real-time decoding setting while substantially lowering computational costs.

Time-Masked Transformers with Lightweight Test-Time Adaptation for Neural Speech Decoding
Ebrahim Feghhi 1.2 Shreyas Kaasyap² Nima Hadidi
Jonathan C. Kao 12,33
Neuroscience Interdepartmental Program
2 Department of Electrical & Computer Engineering
3 Department of Computer Science University of California, Los Angeles
Abstract
Speech neuroprostheses aim to restore communication for people with severe paral- ysis by decoding speech directly from neural activity. To accelerate algorithmic progress, recent benchmark released intracranial recordings from paralyzed participant attempting to speak, along with baseline decoding algorithm. Prior work on the benchmark showed impressive accuracy gains. However, these gains increased computational costs and were not demonstrated in a real-time decoding setting. Here, we make three contributions that pave the way towards accurate, efficient, and real-time neural speech decoding. First, we incorporate large amounts of time-masking during training. On average, over 50% of each trial is masked. Second, we replace the gated recurrent unit (GRU) architecture used in the baseline algorithm with a compact Transformer. The Transformer architecture uses 83% fewer parameters, cuts peak GPU memory usage by 52%, and significantly faster to calibrate relative the GRU. Third, we design lightweight variant of an existing test-time adaptation method developed for decoding handwriting from neural activity. Our variant adapts the model using multiple time-masked aug- mentations of a single trial and requires only one gradient step per trial. Together, these contributions reduce word error rate by over 20% and effectively mitigate performance degradations across held-out days in a real-time decoding setting while substantially lowering computational costs.
Introduction
Conditions including amyotrophic lateral sclerosis (ALS) and brainstem stroke can lead to severe paralysis, leaving individuals unable to speak or interact with the world. A promising path toward restoring communication for these individuals is speech neuroprostheses, which bypass the vocal apparatus and decode speech directly from neural activity (35). Speech neuroprostheses have made significant strides recent years, demonstrating the ability to decode speech with high accuracy over
large vocabularies (41 30 25
To be viable in real-world clinical settings, decoding algorithms for speech neuroprostheses should ideally satisfy several key criteria beyond accuracy. First, they be able to operate in a real-time "streaming" fashion. decoding speech with low-latency over short windows rather than waiting for the entire utterance to finish (25). Second, they should have low computational requirements to enable on-device inference and adaptation (8. minimizing reliance on external connections and preserving user privacy. Finally, decoding algorithms should be easily integrated with test-time adaptation methods that mitigate performance degradation across time (17)
"Correspondence: ebrahimfeghhi 0g. wwla. edu
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

To accelerate progress along these lines, the Brain-to Textt Benchmark '24 was released, an open- source dataset containing intracortical neural recordings while a participant with ALS attempted to speak sentences across 24 days. Along the with the dataset, the organizers provided a baseline decoding algorithm which consisted of a gated recurrent unit (GRU) architecture to decode neural activity into phonemes, followed by beam search guided by an n-gram language model (LM) (41).
A recent summary report outlined the findings of the top four entries submitted to the benchmark (4)). Several entries replaced the GRU architecture with Transformers, deep state space architectures, and convolutional neural networks (CNNss) and found that none of these architectures outperformed the baseline GRU. For example, the first place entry reported that the phoneme error rate (PER) for Transformers was more than double that of the baseline GRU (24. While researchers found that using a learning rate scheduler, layer normalization a) and different optimization algorithms helped improve accuracy, the modification that led to the largest improvement was "using an ensemble of neural decoders to generate a diverse set of sentence hypotheses, and then using fine-tuned large language model (LL.)) to merge these hypotheses into finalized sentence" (42), an idea that was first implemented by
6 in the context of neural speech decoding.
Although the accuracy gains from prior work are remarkable, there are several important caveats. First, the top four entries all used a bidirectional GRU, which requires access to future neural activity and therefore cannot decode speech in real-time. Furthermore, LLM merging was only applied after the entire text was decoded, as applying it to intermediate outputs is not straightforward. Second, some entries used up to 10 bidirectional GRUs and GPT 3.5, which makes inference and test-time adaptation on local resource- cessurcc-constrained devices challenging. Third, since the sentences used in the benchmark were from the publicly available Switchboard corpus (2), it is possible L.MS were trained on this corpus and so their contribution is overstated relative to conversational settings. These points highlight how optimizing for a single metric on machine learning benchmarks can lead to sacrifices along other dimensions important for real-world use.
In this work, we aimed to holistically improve speech neuroprostheses by focusing on multiple criteria: accuracy, capability for real-time streaming, low computational costs, and robustness to distribution shifts. In order to do so, we focused on improving the neural network that translates neural activity into phonemes rather than external language modeling components. Our improvements are based on two key observations. First, the baseline GRU overfits early in training (Figure [a). Second, the GRU processes highly redundant inputs at each time step in order to achieve optimal performance: consecutive inputs are 87.5% overlapping when using the settings proposed by Willett et al. 4 (Figure [p), which we hypothesized is source of inefficiency.
Based on these observations, we proposed two modifications to the baseline algorithm. First, in order to delay overfitting, we incorporated time-masking which is a regularization strategy that masks contiguous temporal chunks of the input neural activity during training (Figure [+) (32). Second, we replaced the GRU architecture with a compact, unidirectional Transformer (Figure
(38) We
hypothesized that the GRU requires overlapping inputs due to its lossy memory, which may result in increased computational costs. Since Transformers have perfect memory for a fixed context length, they are likely able to efficiently processes con-ovellapping inputs. To foreshadow the results, the Transformer trained with time-masking (time-masked Transformer) achieved a WER of 12.17% with a 3-gram LM and 8. 18% with the 5-gram LM setup, which is 20% and 26% lower than the baseline unidirectional GRU, respectively. The Transformer architecture also substantially reduces parameter count, peak GPU memory usage, FLOPs, and per-epoch training times.
We next leveraged the time-masked Transformers to improve upon the test-time adaptation method created by Fan et al. (17 for handwriting neuroprostheses. Their approach, Continual Online Recalibration with Pseudo-labels (CORP), refines the text produced by a GRU-based model with an n-gram LM,, and then adapts the GRU to output the LM-refined text during test-time. CORP utilizes multiple gradient steps per trial and maintains previous data to enable adaptation. Here, we present "DietCORP". a lightweight variant which enables test-time adaptation step without storing previous data by leveraging multiple time-masked augmentations of the same trial (Figure (1)) When combined with the low memory requirements and fast training speeds of the Transformer, DietCORP effectively adapts the model to distribution shifts while requiring only 1.3 GiB of peak GPU memory usage and 18 ms to adapt per trial.
single gradient
2

---

Text
D
Precktor
Trial Psaudo-label
CTC Loss
CTC Loss
Bearn search using n-gram language model
Early stopping
Val
P
P
Train
5x(
Transformer
Thall Procktene
Epoch
Trial Passudo-lbell
B
GRU Input at 4
GRU Input at N.
Patch embedding
CTC Loss
CTC Loss
Trii Aue. Preds
Teai Aug, Freds
Features
87,5 5% overlap
Features
MASK
MASK
*
frozen
weigts
Time
Time (s)
Time (s)
Figure 1: A. The GRU exhibits pronounced overfitting when training for long durations. Black dashed line indicates where training was stopped for the baseline model. B. Adjacent input windows to the GRU overlap by 87.5 97.5% when using the optimal baseline hyperparameters (window length 640 ms, stride 80 ms). We replaced the GRU with a lightweight Transformer-based model. The Transformer takes as input von-ovellap temporal patches of neural activity and outputs logits (denoted as Pi). Consecutive patches were replaced with MASK token during training. as denoted by dark coloring. We used the connectionist temporal classification (CTC) loss. D. An overview of DietCORP. the top panel, the Transformer architecture is run evaluation mode to generate logits, and these logits are integrated with a language model guided beam search to generate a labud.-label. In the bottom panel, the model is trained to produce the pseudo-label across Z masked augmentations with CTC loss. Only the patch embedding module is adapted during this process.
2 Related Work
Although we found success with using a Transformer, Transformers are typically not the architecture of choice for speech neuroprostheses or brain computer interfaces (BCIs) more broadly. Willsey (43 used convolutional neural network (CNN) to decode finger movements. Costello et al. (I2) followed up on this work, and showed that recurrent neural networks (RNNs) achieve better finger movement decoding performance than either CNNs or Transformers. The majority of speech neuroprostheses have employed some combination CNNs and RNNs (3 37 28
et al.
40 A1 4 g
36 25 although few studies have used Transformers 39 11 1O). In the discussion of the Brain-to-
Text benchmark results, the authors speculated that the relatively poor performance of Transformers may be attributed to the fact that 1) more optimization is required 2) phonemes represented over short context windows in neural activity, which suits GRUs because they naturally implement a locality bias whereas Transformers are better suited for modeling long-range dependencies (42).
Our Transformer architecture is most similar to that of Chen et IT) in the BCI literature. Specif- ically, we also used relative positional embeddings and temporal patches as input. However, each temporal patch in our work consists of data from all electrodes, whereas Chen et al. (10) assign each electrode to its own temporal patch. Assigning each electrode to its own patch can substantially increase the context size, and Chen et (10) use the Swin Transformer (26) to address this chal- lenge. Beyond model architecture, Chen et al. (10) displayed their results closed-source dataset consisting of electrocorticographic (ECoG) and depth electrodes (sEEG), and they translated neural activity directly into speech. Our results are displayed using open-source dataset consisting of microelectrode array (MEA) recordings, and we translated neural activity into text. Li et al. (24) also used Transformers on the Brain-to- Text Benchmark '24; however, to the best of our knowledge the authors did not release sufficient details for reproducing their Transformer results.
Structured input masking was initially popularized by Park et al. (32) for automatic speech recognition (ASR). Within speech neuroprostheses, Littlejohn et al. (25) masked entire channels and cropped
each trial by applying a temporal window. Metzger et al. (30 29) applied a single temporal mask to
each trial. The masking augmentations used by Littlejohn al. (25); Metzger et al. (30) 29) were
primarily applied on close-soureed, ECoG datasets and mask much smaller percentage of the input
3

---

than in this work. Furthermore, the contribution of masking relative to other augmentations is not emphasized in these works. Beyond speech neuroprostheses, Saced et al. (34;; Ding al. (15) applied structured input masking for EEG-based BCIs, and Fu et al. (19) applied it for intracranial monkey BCIs.
DictCORP is most directly inspired by CORP However, the idea to use multiple augmentations, rather than previous data, for test-time adaptation was inspired by Zhang al. (46) and Yao al. 44. Zhang et al. (46) encouraged models to make consistent and confident (low entropy) predictions across several augmentations of the same image at test-time. Yao al. (44 add KL divergence loss term across two Lime-masedd versions of the same audio sample to the connectionist temporal classification (CTC) (2 loss during training.
3
Methods
3.1 Neural Dataset
The Brain-to-Tex: '24 benchmark dataset consists of microclectrode array (MEA) recordings from the ventral premotor cortex (area Gv) of a single participant with ALS. Data was recorded from two microclectrode arrays with 64 channels each, a ventral 6v array and a dorsal 6v array. Spike band power and threshold crossings were extracted for each channel, leading to a total of 256 features. Neural activity was recorded while the patient attempted to speak 10,850 sentences. In each trial, the subject was shown a sentence, and attempted speak the sentence the onset of the "go" cue. All analyses were done on neural activity during the "go" phase while the participant attempted to speak the sentence. The neural activity was provided in 20 ms time bins (50 Hz resolution), and z-scored within each block (20-50 sentences). For additional details, refer to
3.2
Word error rate and phoneme error rate
We computed the word error (WER) the Levenshtein (edit) distance between the predicted and target word sequences, normalized by the total number of words in the target. Formally, WER is defined as
WER S + +++
N
where S. D, and denote the number of substitutions, deletions, and insertions, respectively, and N is the total number words in the target transcription. Similarly, the phoneme error rate (PER) was computed in the same manner but the level of phonemes rather than words. We applied an n-gram guided beam search process when computing WER (Section E. and computed PER using greedy decoding.
3.3 Train, validation, and test splits
The benchmark provided train, validation, and test splits. There were 8800 sentences in train, 880 sentences in validation, and 1200 sentences in test. Train and validation sentences were recorded on 24 days (collected over almost 4 months), and test sentences were recorded on 15 out of the 24 days. We refer the days with testing data as "competition" days. All hyperparameter tuning was performed on the validation set, and the validation set was not used for training after hyperparameter tuning for the main results (Section 4.1 4.2). For DietCORP (Section 4.3). we were interested in evaluating its effectiveness on days with no training data. Since the original splits contained training data for all days, we modified the splits by holding out 5 or 8 competition days entirely from the training set. To evaluate performance, we used the original training and validation data from these held-out days as our new test set since the competition platform does not allow for evaluating WER on subset of the competition days. We used all data from the preceding days for training.
3.4 Baseline gated recurrent unit-based model
The dataset was accompanied by a baseline gated recurrent unit (GRU)-based model. This model first passed the input features (spike band power and threshold crossings) through a day-specific linear layer followed by softsign non-linearity, which served to account for day-specific differences in neural activity. At each step, the GRU received vector of these tay-traanformed inputs with

---

dimension F- Tin, where F is the number of features (256) and Tin (the window length) is the number of neural time bins. The model consisted of GRU layers. The hidden state of the final layer was passed through linear layer to produce logits over phonemes, the CTC blank token, and a silence token. Logits were output every Tout (stride length) steps. For optimal performance, the baseline algorithm set in 32 and Tout 4, resulting in an 87.5% overlap between consecutive inputs.
We followed the same training procedures as listed in the Pytorch codebase provided by Willett et al. 41 for the baseline GRU model. Specifically, we used the Adam optimizer (22) and comnectionist temporal classification (CTC) loss (2I. During training, white noise and baseline shift augmentations were added to the neural activity, followed by causal Gaussian smoothing. Training was performed for 10,000 batches (~ 73 epochs). The full set of hyperparameters is listed in Table We used a unidirectional GRU for all results unless otherwise stated.
3.5 Transformer-based model
Inspired by Dosovitskiy et al. (T6) and Chen et al. (O)) we replaced the GRU architecture with Transformer. We segmented the input neural data into non-overlapping temporal patches, each consisting of Tin time bins and all (256) features. Tin was set to 5, such that each patch captured 100 of neural activity. Each patch was then flattened into F Tm (F is the number of neural features) dimensional vector and passed through a patch embedding module. consisting of the following layers in sequence: LayerNorm (5), Linear, and LayerNorm. The linear layer projected the patches from F Tin to the transformer hidden dimension size. Time-masking (Section 3.6 and input dropout was applied to the output of the patch embedding module, followed by L. Transformer blocks (I= 5). Each Transformer block comprised self-attention layer followed by We applied LayerNorm before the self-attention layer, and the feed-for forward network included the following layers in sequence: LayerNorm, Linear, GeLU activation, Dropout, Linear, and Dropout. Residual connections were added around both the self-attention layer and the feed-forward network.
network.
We used relative positional embeddings similar to the T5 architecture (3 Specifically, we added learned bias the attention scores based on the relative distance between two patches. A causal attention mask was also used to ensure that each patch could only attend to itself and previous patches. The attention operation was then defined as:
QK
Attention( Q, K. V) softmax
+B+M
where B,, = bbi- and b 2 was learnable vector containing scalar bias values for each possible relative position. The index (i - j) represents the relative distance between query position i and key position j. allowing attention scores to incorporate relative position information up to maximum absolute distance of I, 1. The matrix M was the causal attention mask, defined as:
0
i S
M.,, -00 otherwise
The outputs of the causal Transformer were passed through final LayerNorm followed by an output linear layer to obtain logits over phonemes, the CTC blank token, and silence token. Logits were output every Tour time bins, and we set Tout = 5.
When training the Transformer, we applied a log-transformation to the neural data before z-score normalization and trained for 250 epochs. We also used learning rate scheduler, where decreased the learning rate by a factor of 10 after 150 epochs. Training was performed with the AdamW optimizer (27 and CTC loss, and also applied white noise and baseline shift augmentations followed by causal gaussian smoothing. The full set of Transformer hyperparameters is listed Table 6. The code used for the Transformer model is largely based on the following GitHub repository: vit-pytorch
3.6 Time-masking
We performed time-masking by masking several contiguous temporal patches (Transformer) time bins (GRU) of each trial. The rationale for applying contiguous masking addition to input dropout is that input dropout does not mask contiguous input chunks, and is therefore likely weaker form
5

---

Algorithm 1: TIMEMASK(X, N, M)
Input: Trial x E RR (L patches, C' features);
N number of masks;
M max-mask length as a fraction of trial length (0 < M S 1)
Output: Masked trial
F < [M L]
// maximum mask length
for k in 1:N do
S (0, I. T.-F) F);
// start index // mask length
D ~ U(0,, F);
x(S S DJ <- LEARNABLE-MASK TOKEN
return x
of regularization. description of the time-masking algorithm is provided Algorithm for the Transformer. For the masked GRU, all details are identical except masking applied at the time bin level, and masked bins were replaced with 0 rather than a special MASK token. We do not enforce time-masks to be non-overlapping.
For all analyses, we set N 20 (number of masks) and M 0.075 (max mask length as a fraction of trial length). Under these settings 53% of a trial is masked out on average (Section D. We report performance with other hyperparameters in Section !|
3.7 DietCORP
For each trial, we generated a "pseudo-label" by applying language model guided beam search (Section E to the model predictions, as done in CORP (17). "Pseudo-label" generation is already performed for real-world use, and so it does not incur additional computational costs. In CORP, the pseudo-label is combined with labeled training data well as previously generated pseudo-labeled data to update model weights with CTC loss until either the loss decreases below a threshold or 200 gradient steps completed. DietCORP reduces computational costs and complexity by eliminating the need to store previous data and performing adaptation with one gradient step per trial. This is done by adapting the model across Z augmentations of the same trial. Model weights not reset across trials (i.e., calibration is continuous), as done in CORP.
When applying DietCORP, we only updated the patch embedding module because 1) neural dis- tribution shifts across days are likely form of input distribution shift (as opposed to feature or output distribution shifts) and selectively fine-tuning input layers is effective when dealing with input distribution shifts (23). and 2) test-time adaptation can rapidly degrade when adapting the entire model (23). Augmented trials were generated by creating Z copics of given trial, and applying white noise, baseline shift, and time-masking to each copy. Time-masie mane-ma served as the main form of augmentation. We set Z 64 since this the batch size used for training, although DietCORP was robust to lower values of Z as discussed in the results. All hyperparameters were set to the values used during training. The learning rate was set to the mean of the learning rate before and after the learning rate step decay was applied, which was 5e - 4. We additionally clipped the gradient norm, as done in CORP, to 0.5.
4 Results
4.1
Comparison with baseline unidirectional GRU
We first compared the causal Transformer trained with time-masking (Section 3.5 masked Transformer, with the baseline unidirectional GRU model, or baseline GRU (Section 3.4). When using the 3- 3-gram language model (LM) (Section E}. the mimke Transformer achieved word error rate (WER) that was 3.08 absolute percentage points better than the baseline GRU (or 20.2% relative improvement), which was a significant decrease (p < 0.05; one-sided independent t-test across 10 seeds) (Table D. When using the stronger 5-gram LM setup, with additional second-pass rescoring using an unpruned LM and large language model (LLM) (Section F the
3.6 time-
time tine-masked Transformer performed 2.94 absolute percentage points better than the baselinc GRU

---

Table I: Performance comparison using different language modeling setups. The values represent word error rates (WER) as percentage (mean + SEM across seeds). An em dash (-) denotes settings that were not evaluated. Values for the Linderman Lab and diphone decoding models are provided by the original authors. Results are reported across 10 seeds except for the "Fine-tuned LLMs" setting (N = 1), diphone decoding with the 5-gram LM setup (N = 5), and Linderman Lab GRU (N 1).
Model
3-gram LM Setup
S-gram LM Setup
Fine-tuned LLM
Baseline Unidirectional GRU
15.25 0.16
11.12 + 0.13
Time-Masked Causal Transformer
12.17
5.68 Llama 8B
Linderman Lab Bidirectional GRU
8.0
Diphone Decoding with Bidirectional GRU
8.39 1: 0.22
5.77 w/ GPT 3.5
6.85 w/ Llama 3.1 70B
(or a 26.4% relative improvement) (p < 0.05; one-sided independent t-test across N = 10 seeds) (Table D. This our knowledge the largest improvement accuracy over the baseline GRU on this neural dataset when using a streaming compatible architecture.
Beyond gains in accuracy, the lime-masked Transformer also substantially reduced computational costs relative to the baseline GRU. The Transformer used 83% fewer parameters, cut peak GPU memory usage by 52%, reduced mega floating pointing operations (mFLOPP)) by 43% (Section C. while shortening per-epoch training times by 58% (Table 4.1p. These efficiency gains make the Transformer architecture well-suited for on-device test-time adaptation (8) and motivate the experiments in Section 4.3 We also observed that beam search decoding times, measured after producing logits, were approximately 3 times faster when using the time-masked Transformer relative to the baseline model (Section H.. Furthermore, the Transformer does not require day-specific parameters unlike the baseline GRU, which we speculate is due the layer normalization layers in the patch embedding module.
4.2 Comparison against top-performing benchmark entries
We next compared the time-masked Transformer with of the top-performing. entries in the Brain-to-Text Benchmark '24: the Linderman Lab bidirectional GRU (Section F (42) and diphone decoding with a bidirectional GRU (4).. Both these entries are not streaming compatible, and we used the WER values provided by the original authors. When using the 5-gram LM, there was no significant difference in performance between either the time-masked Transformer (N 10 seeds) and the Linderman Lab GRU (N 1 seed) (p 0.05; one-sample t-test) or the time-masked Transformer and the diphone decoding approach (N 5 seeds) (p > 0.05; independent t-test).
We next evaluated the time-masked Transformer with the "Fine-tuned LLM" setup (Section G) which involved fine-tuning LLM to correct the outputs of ensemble of models. We fine-tuned Llama 3.1 8B (20) on the outputs of 10 seeds of the causal time-masked Transformer. The Transformer + Llama 3.1 8B combination performed on-par with the best performing entry in Willett al. (42). specifically the diphone decoding GRU + GPT 3.5 ( or Llama 3.1 70B [].
Beyond accuracy, the Transformer architecture provides large gains in computational efficiency over the bidirectional GRU (Table 4.1,, and we used a smaller, open-source LLM (Llama 3.1 8B) relative to previous entries that used either GPT 3.5, which is not open-source, or Llama 3.1 70B (6) 24)..
Table 2: Values are reported using a Nvidia GeForce RTX 3090 GPU. Epoch time is reported mean + standard deviation (N 1200). MFLOPS stands for mega floating point operations per second.
Model
Parameters (M)
Peak memory (GiB)
MFLOPS
Epoch time (s)
Bidirectional GRU
135.4
11.21
1563.29
55.63 + 0.69 25.88 + 0.55
Unidirectional GRU
56.7
5.55
634.81
Causal Transformer
9.4
2.66
364.2
10.81 + 0.17
7

---

No Adaptation
DietCORP
35
Word error rate (%)
held-out days
45
30
35
25
5 held-out days
8 held-out days
25
5 held-out days
20
25
7
28
30
12 21
34 49 51
64
32
16
Days since last training day
Days since last training day
Number of augmentations
Figure 2: Results are for the time-masked Transformer with a 3 3-gram LM. Points show the mean over four seeds; shading indicates SEM. A. WER across five held-out days without adaptation and with DietCORP. B. Same panel A with evaluation on eight held-out days. C. Average WER across held-out days as function of the number of augmentations used by DietCORP. Green points are when using DietCORP: the purple dashed line is when no adaptation is performed. Lower curves correspond to five held-out days, upper curves to eight held-out days.
4.3 Test-time adaptation across held-out days
We next used the lightweight Transformer architecture perform test-time adaptation (TTA) with DietCORP. In brief, DietCORP adapts the decoder to predict LM-refined pseudolabels (with the 3-gram LM) using multiple augmented versions of the current trial. DietCORP only requires one gradient step per trial and minimal hyperparameter tuning (Section 3.7). We applied DierCORP under two settings in which entire days were held-out from the training set: training 15 days and testing on the next 5 competition days, and a more challenging setting where we trained on 12 days and tested on the next 8 competition days. In both cases, we compared performance on the held-out competition days when applying DietCORP versus when no TTA method was applied.
When training on 15 days, there was substantial rise in WER across the 5 held-out competition days when no TTA method was applied, starting from a WER of 22.74 + 0.30% (mean + SEM across N = 4 seeds) the first day and rising to WER of 32.58 = 0.60% by the fifth day. By contrast, when applying DictCORP the increase in WER was substantially smaller, beginning at 21.24 + 0.29 and increasing to 22.97 + 0.72% across the five days (Figure 2 Under the more challenging evaluation setting, WER rose from 28.87 + 0.54% to 66.47 + 0.67% across the eight held-out days when TTA was applied. DietCORP again substantially ameliorated the deterioration in performance, with WER ranging from 26.32 + 0.32% to 31.74 + 1.41% across held-out days (Figure 2))
We next evaluated how DietCORP's benefit scales with the number of time-masked augmentations. Reducing the augmentation count progressively diminished its impact, supporting the idea that adaptation across multiple augmentations of the same trial helps buffer against test-time distribution shifts (46) (Figure 2c). Nevertheless, DictCORP still outperformed the adaptation setting when using only a single augmentation, indicating that there are still benefits when using a low number of augmentations. We finally quantified the memory requirements and adaptation times for DietCORP. When using the Transformer, DietCORP required 18 ms per trial and its peak memory
Table 3: Metrics were collected on an NVIDIA GeForce RTX 3090 over 2040 trials. Peak memory is the highest GPU memory recorded during DietCORP calibration across all trials. Adaptation time (reported as mean + SD) the time needed update the model on a single trial with DietCORP. Only the patch-embedding layer was trainable for the Transformer, for a fair comparison, only the day-specific linear layer was trainable for the baseline GRU.
Model
Peak memory (GiB)
Adaptation time (ms)
Time-masker Transformer
1.33
18.21 + 5.34
Baseline GRU
2.91
28.44 + 8.33
8

---

Applying DieiCORP to the baseline GRU increased adaptation times by 56% and peak memory usage by 118%, underscoring the benefit of the Transformer for on-device test-time adaptation.
usage was 1.3 GiB (Table 13
4.4 Ablations to the time-masked Transformer
We conducted an ablation study to isolate the contributions of each component in our time-masked Transformer model (Table D All ablation results are reported on the validation set with a 3-gram L.M. First, we removed two components from training pipeline for the time-masked Transformer: the neural data log-transformation and the learning rate scheduler (Section 3.5 Removing either component independently increased the WER by 4% relative to the original model. Next, we replaced the TS-style relative positional embeddings (Section 3.5 with absolute sinusoidal embeddings used in prior BCI studies (2).. This change resulted in 10% WER increase, suggesting that relative positional information is important for focusing on local features relevant to phoneme decoding. We next removed time- masking, and restored regularization hyperparameters (dropout, input dropout, white noise, baseline shift) to the baseline levels used for the GRU (Table Removing time- time-masking increased WER by 18%. Since the Transformer model without time- mankemn was trained for 250 epochs, this indicates that the performance benefits of the time-masked Transformer over the baseline GRU (which was trained for 73 epochs) cannot be attributed to extended training time alone. Finally, we replaced the Transformer architecture with the GRU architecture. When training the time-masked GRU, we used the same regularization hyperparameters as the time-masked Transformer (Table 6) and trained for 250 epochs. The time-masked GRU only performed 4% worse, whereas the baseline GRU performed 20% worse, than the time-masked Transformer suggesting that time-masking is a generally useful augmentation for neural speech decoding across network architectures.
4.5 Optimizing the time-masked GRU with non-overlapping inputs
Results from the ablation study suggest that the mimeed GRU performs competitively with the time-masked Transformer. However, key benefit of the Transformer is that it requires lower memory resources and is faster calibrate. We hypothesized that the computational efficiency benefits of the Transformer are related to its ability to effectively process non overllaplg inputs, reducing processing redundancy. While prior work showed the baseline GRU performs optimally with overlapping inputs (41), we tested if this was also the case for the time-masked GRU. To investigate this, we trained the time-maseed GRU with non-overlapping inputs by setting both the window length and stride length to 80ms (Section 3.4 We observed that performance was significantly worse with non-overlapping inputs, with validation WER when using a 3-gram LM increasing from 17.85 + 0.13% to 19.66 + 0.14% (p < 0.05; two-sided independent t-test, N = 4 seeds). To account for the possibility that a smaller hidden state would be more optimal with the smaller, non-overlapping inputs, we lowered the hidden state size from 1024 to 768 or 512, but this further worsened performance. Thus, consistent with Willett et al. (4I, our results suggest that the GRU performs optimally with overlapping inputs even with time-masking We leave it to future work to further investigate whether a smaller GRU architecture can perform competitively with the Transformer.
Table 4: Ablation study results for the Masked Causal Transformer. Values represent word eиror rates (WER) on the validation data with 3-gram LM as percentage (mean + SEM across seeds). 10 seeds were run for the original model, and 4 seeds were run for the ablations.
Ablation
Validation WER (%)
Time-masked Transformer
17.15 + 0.15
No Log Transform
17.85 + 0.15
No Learning Rate Scheduler
17.85 - 0.21
No T5 Positional Encoding
18.89 = 0.06
No Time-Masking
20.17
Transformer + GRU
17.85 = 0.12

---

Discussion
There are three noteworthy limitations to our results. First, all results are on single participant. At the time of writing this manuseript, there were no other open-source microelectrode array datasets for speech neuroprostheses, and evaluation on one participant is standard of the field. However, it is still important to evaluate the effectiveness of time-masked Transformers and DietCORP across multiple participants. Second, our use of beam search allows previously decoded text to be revised, which complicates integration with text-to-speech systems and may also provide suboptimal experience for users if textual revisions are significant. Third, even the smallest language modeling setup used in this study, the 3-gram L.M requires about 60 GB of CPU memory to operate. Such large memory requirement makes it challenging to run the current decoding algorithm on local, resource-constraing devices.
In summary, this work delivers three practical advances for the Brain-to- text Benchmark '24 and. more broadly, for the development of speech neuroprostheses. First, we showed that large amounts of mame-masking serves as highly effective data augmentation method for improving neural speech decoding accuracy. Second, replacing the baseline GRU with Transformer provides substantial reductions in computational costs, making the Transformer an ideal architecture for on-device test- time adaptation. Third, we introduced DietCORP, a simple, lightweight, and fast method that utilizes multiple time-masked augmentations of the same trial to effectively adapt the Transformer at test-time. Together, these innovations lower word error rate while cutting resource demands, moving closer to building robust, on-device speech neuroprostheses that restore fluent communication for patients with severe paralysis.
There are several exciting avenues for further improving speech neuroprostheses. First, future research can investigate avenues to reduce memory costs associated with the n-gram LM or explore alternative strategies integrate text-based knowledge. For instance, Feng al. (18) and a recent study (I) project the outputs of GRU or Transformer to the token space of an LLM for text decoding, obviating the need for the n-gram LMM Second, a recent trend in automatic speech recognition is to build a single model that can operate in both streaming and non-streaming modes (45). Such an approach could be useful for speech neuroprostheses, since the user may have different preferences the context. Third, as more microelectrode array datasets
for accuracy versus latency depending are released, an emerging research question is whether integrating microelectrode array recordings across multiple participants other neural recording modalities (T).. Finally, future work should explore achieving similar accuracy levels without using beam search, so that previous outputs cannot be revised for easier integration with text-to-speech systems (25).
be used to boost performance. This question has been explored with
We hope that our study encourages future speech neuroprostheses benchmarks, such the Brain- to- to-Text Benchmark '25, to offer more holistic evaluation criteria for decoding algorithms beyond only accuracy. By evaluating decoding algorithms along multiple criteria such ability to decode in real-time, computational costs, and integration with existing test-time adaptation methods, these benchmarks can more effectively spur innovation and assist paralyzed patients with communication impairments.
Acknowledgements
This work was supported by the following awards. To JCK: NSF CAREER 1943467, NIH DP2NS122037, NIH ROINS121097. To EF: NIH 732251115533.
10

---

References
[1] Decoding inner speech with an end-to-end brain-to-text neural interface. Accessed: 2025-11-2.
https://
[2] Switchboard-1 release 2.
SwapennnedduLLDC997566 Accessed: 2025-
5-16.
[3] Miguel Angrick, Christian Herff, Emily Mugler, Matthew C Tate, Mare W Slutzky, Dean J Krusienski, and Tanja Schultz. Speech synthesis from ECoG using densely connected 3D convolutional neural networks. J. Neural Eng., 16(3):036019, June 2019.
[4] Miguel Angrick, Shiyu Luo, Qinwan Rabbani, Daniel N Candrea, Samyak Shah, Griffin W Milsap, William S Anderson, Chad Gordon, Kathryn R Rosenblatt, Lora Clawson, Donna Tippett, Nicholas Maragakis, Francesco V Tenore, Matthew S Fifer, Hynek Hermansky, Nick F Ramsey, and Nathan Crone. Online speech synthesis using a chronically implanted brain- computer interface in individual with ALS. Sci. Rep., 14(1):9617, April 2024.
[5] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv [stat.ML.]. July 2016.
[6] Tyler Benster, Guy Wilson, Reshef Elisha, Francis R Willett, and Shaul Druckmann. A cross- modal approach to silent speech with LLM-enhanced recognition. arXiv [cs.HC], March 2024.
[7] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. arXiv [cs. CLJ, May 2020.
[8] Han Cai, Chuang Gan, Ligeng Zhu, and Song Han. TinyTL: Reduce memory, not parameters for efficient on-device learning, Neural Inf Process Syst, 33:1 1285-11297177 July 2020.
[9] Nicholas S Card, Maitreyee Wairagkar, Carrina lacobacci, Xianda Hou, Tyler Singer-Clark, Francis R Willett, Erin M Kunz, Chaofei Fan, Maryam Vahdati Nia, Darrel Deo, Aparna Srinivasan, Eun Young Choi, Matthew F Glasser, Leigh R Hochberg, Jaimie M Henderson, Kiarash Shahlaie, Sergey D Stavisky, and David M Brandman. An accurate and rapidly calibrating speech neuroprosthesis. N. Engl. J. Med., 391(():609-6) 618, August 2024.
[10] Junbo Chen, Xupeng Chen, Ran Wang, Chenqian Le, Amirhossein Jensen, Patricia Dugan. Werner Doyle, Orrin Devinsky, Daniel Friedman, Adeen Flinker, and Yao Wang. Transformer-basedd neural speech decoding from surface and depth electrode signals.
-Galtian,,,,,,aii,, Erika
J. Neural Eng., 22(1), January 2025.
[11] Xupeng Chen, Ran Wang, Amirhossein Mhaliian-Gourtani Leyao Yu, Patricia Dugan, Daniel Friedman, Werner Doyle, Orrin Devinsky, Yao Wang, and Adeen Flinker. A neural speech decoding framework leveraging deep learning and speech synthesis. Nat. Mach. Intell. 6(4): 467-480, April 2024.
[12] Joseph Costello, Hisham Temmar, Luis H Cubillos, Matthew Mender, Dylan M Wallace, Matthe S Willsey, Parag G Patil, and Cynthia A Chestek. Balancing memorization and generalization RNNs for high performance brain-machine interfaces. bioRxiverg, May 2023.
[13] Michael Han Daniel Han and Unsloth team. Unsloth, 2023. URL http://github.cc com/ umslothai/umsloth
[14] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient finetuning of quantized LLMs. arXiv [cs. LG), May 2023.
11

---

[15] Wenlong Ding, Aiping Liu, Ling Guan, and Xun Chen. A novel data augmentation approach using mask encoding for deep learning-base based asynchronous SSVEP-BCI. IEEE Trans. Neural Syst. Rehabil. Eng., 32:875-888 886, February 2024.
[16] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv [cx.CV], October 2020.
[17] Chaofei Fan, Nick Hahn, Foram Kamdar, Donald Avansino, Guy H Wilson, Leigh Hochberg, Krishna V Shenoy, Jaimie M Henderson, and Francis R Willett. Plug-and-play stability for intracortical brain-computer interfaces: one-year demonstration of seamless brain-to-text communication. Adv. Neural Inf. Process. Syst., 0144422884227000 December 2023.
[18] Sheng Feng, Heyang Liu, Yu Wang, and Yanfeng Wang. Towards end-to-end framework for invasive brain signal decoding with large language models. In Interspeech 2024, pages 1495 1499, ISCA, September 2024. ISCA.
[19] Haotian Fu, Peng Zhang, Song Yang, Herui Zhang, Ziwei Wang, and Dongrui Wu. Effective and efficient intracortical brain signal decoding with spiking neural networks. arXiv HCs. December 2024.
[20] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ah- mad Al-Dahle, Aicsha Aicsha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits. Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith. Filip Radenovic, Francisco Guzmán, Frank Zhang. Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang. Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang. Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi. Jianyu Huang, Jiawen Liu, Jie Wang. Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Va- suden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala- Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kam- badur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi. Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng. Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang. Saghar Hos- seini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Scohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan. Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou. Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do Vish Vogeti, Vítor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine
12

---

Babaci, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim. Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanann Hakan Inan. Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph. Helen Suk, Henry Aspegren, Hunter Goldman. Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang. Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang. Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang. Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang. Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Raf Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li. Rebekkah Hogan, Robin Battey, Rocky Wang. Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto. Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang. Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala. Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho. Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li. Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Con- stable, Xiaocheng Tang, Xiaojian WU, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniy Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang. Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The llama herd of models. arXiv [cs.Al], July 2024.
[21] Alex Graves, Santiago Fern 'andez, Faustino J Gomez, and Schmidhuber. Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks.
13

---

International Conference on Machine Learning. 2006.
[22] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv [cs.LG], December 2014.
[23] Yoonho Lee, Annie S Chen, Fahim Tajwar, Ananya Kumar, Huaxiu Yao, Percy Liang, and Chelsea Finn. Surgical fine-tuning improves adaptation to distribution shifts. International Conference on Learning Representations, 2023.
[24] Jingyuan Li, Trung Le, Chaofei Fan, Mingfei Chen, and Eli Shlizerman. Brain-to-text decod- ing with context-aware neural representations and large language models. arXiv [eess.SP]. November 2024.
[25] Kaylo Littlejohn, Cheol Jun Cho, Jessie R Liu, Alexander B Silva, Bohan Yu, Vanessa R Anderson, Cady M Kurtz-Miott, Samantha Brosler, Anshul P Kashyap, Irina P Hallinan, Adit Shah, Adelyn Tu-Chan, Karunesh Ganguly, David A Moses, Edward F Chang, and Gopala K Anumanchipalli. A streaming brain-to-voice neuroprosthesis to restore naturalistic communication. Nat. Neurosci., 28(4):902-99) 912, April 2025.
[26] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv CVJ, March 2021.
[27] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv [cs.LG), November 2017.
[28] Joseph G Makin, David A Moses, and Edward F Chang. Machine translation of cortical activity to text with an mncoder-decoder framework. Nat. Neurosci., 23(4):575 -82552 April 2020.
[29] Sean L Metzger, Jessie R Liu, David A Moses, Maximilian E Dougherty, Margaret P Seaton, Kaylo T Littlejohn, Josh Chartier, Gopala K Anumanchipalli, Adelyn Tu-Chan, Karunesh Ganguly, and Edward F Chang. Generalizable spelling using a speech neuroprosthesis in an individual with severe limb and vocal paralysis. Nat. Commun. 13(1);6510, November 2022.
[30] Sean L Metzger, Kaylo T Littlejohn, Alexander B Silva, David A Moses, Margaret P Seaton, Ran Wang. Maximilian Dougherty, Jessie R Liu, Peter Wu, Michael Berger, Inga Zhuravleva, Adelyn Tu-Chan, Karunesh Ganguly, Gopala K Anumanchipalli, and Edward F Chang. A high-performance neuroprosthesis for speech decoding and avatar control. Nature, 620(7976): 1037-1046, August 2023.
[31] Mehryar Mohri, Fernando Pereira, and Michael Riley. Speech recognition with weighted stite--tate transducers. Springer Handbook of Speech Processing, pages 559-584. Springer Berlin Heidelberg. Berlin, Heidelberg, 2008
[32] Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoe V Le. SpecAugment: A simple data augmentation method for automatic speech recognition. In Interspeech 2019, ISCA, September 2019. ISCA.
[33] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 2020.
[34] Aaqib Saced, David Grangier, Olivier Pietquin, and Neil Zeghidour. Learning from hetero- geneous EEG signals with differentiable channel reordering. In ICASSP 2021 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1255-
1259. IEEE, June 2021.
[35] Alexander B Silva, Kaylo T Littlejohn, Jessie R Liu, David A Moses, and Edward F Chang. The speech neuroprosthesis. Nat. Rev. Neurosci., 25(7):473 492, 2024.
[36] Alexander B Silva, Jessie R Liu, Sean L Metzger, Ilina Dougherty, Margaret Seaton, Kaylo T Littlejohn, Adelyn Tu-Chan, Karunesh Ganguly, David A Moses, and Edward F Chang. A bilingual speech neuroprosthesis driven by cortical articulatory representations shared between languages. Nat. Biomed. Eng., 888)9777991, August 2024.
Maximilian E
14

---

[37] Pengfei Sun, Gopala K Anumanchipalli, and Edward F Chang. Brain2Char: deep architecture for decoding text from brain recordings. J. Neural Eng., 17(6):066015, December 2020.
[38] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 3998-6008, 2017.
[39] Maitreyee Wairagkar, Nicholas Š Card, Tyler Singer-Clate Xianda Hou, Carrina Iacobacci, Leigh R Hochberg, David M Brandman, and Sergey D Stavisky. An instantaneous voice synthesis neuroprosthesis. bioRxivorg, September 2024.
[40] Francis R Willett, Donald T Avansino, Leigh R Hochberg, Jaimie M Henderson, and Krishna V Shenoy. hiihhppeforrmacce brain-to-text communication via handwriting. Nature, 593(7858): 249 254, May 2021.
[41] Francis R Willett, Erin M Kunz, Chaofei Fan, Donald T Avansino, Guy H Wilson, Eun Young Choi, Foram Kamdar, Matthew Glasser, Leigh R Hochberg, Shaul Druckmann, Krishna Shenoy, and Jaimie M Henderson. high-performance speech neuroprosthesis. Nature, 620 (7976):0011-1036, August 2023.
[42] Francis R Willett, Jingyuan Li, Trung Le, Chaofei Fan, Mingfei Chen, Eli Shlizerman, Yue Chen, Xin Zheng. Tatsuo Okubo, Tyler Benster, Hyun Dong Lee, Maxwell Kounga, E Kelly Buchanan, David Zoltowski, Scott W Linderman, and Jaimie M Henderson. Brain-to-text benchmark 24: Lessons learned. arXiv [cs. CLJ, December 2024.
[43] Matthew S Willsey, Samuel nasoo-Tomaszewkii. Scott R Ensel, Hisham Temmar, Matthew J Mender, Joseph T Costello, Parag G Patil, and Cynthia A Chestek. Real-time brain-machine interface in non-human primates achieves high-velocity prosthetic finger movements using shallow feedforward neural network decoder. Nat. Commun., (3(1),689, November 2022.
[44] Zengwei Yao, Wei Kang, Xiaoyu Yang, Fangjun Kuang. Liyong Guo, Han Zhu, Zengrui Jin, Zhaoqing Li, Long Lin, and Daniel Povey. CR-CTC: Consistency regularization on CTC for improved speech recognition. International Conference Learning Representations, 2025.
[45] Binbin Zhang, Di Wu, Zhuoyuan Yao, Xiong Wang, Fan Yu, Chao Yang, Liyong Guo, Yaguang Hu, Lei Xic, and Xin Lei. Unified streaming and non-streaming two-pass end-to-end model for speech recognition. arXiv (cs.SD)] December 2020.
[46] Marvin Zhang, Sergey Levine, and Chelsea Finn. MEMO: Test time robustness via adaptation and augmentation. Neural Information Processing Systems, 2022.
[47] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. OPT: Open pre-trained transformer language models. arXiv [cs. CLJ, May 2022.
15

---

A Computational resources
The majority of our results were generated using Ubuntu server with three Nvidia GeForce RTX 3090 GPUs and an AMD Ryzen Threadripper 3960X 24-Core CPU with 125 GiB of memory. For the 5-gram LM and LLM fine-tuning results, we used the "g6e. 6xxlaree" Amazon EC2 instance with one Nvidia L40S GPU and an AMD EPYC 64-core CPU with 512 GiB of memory.
B Code availability
Code is available at the following link: htth:///ggtthi...mmmmbaaaaafee/r
with_ dithddietcorp
C FLOPS
We computed the number of floating point operations per second (FLOPS) for the baseline GRU and Transformer. When computing FLOPS for the baseline GRU, included the day-specific linear layer, the 5 GRU layers, and the output linear layer. When computing FLOPS for the Transformer, we included the patch embedding linear layer, the Transformer layers, and the output linear layer. We removed all regularization for this calculation, and computed FL.OPS using 10 seconds of input, and then divided by 10.
D Hyperparameters
We list the hyperparameters for the baseline GRU in Table 3 and for the time-masker Transformer in Table x
E
Language model and beam search
We used the 3-gram and 5-gram language models (LMs) provided by Willett et (41), and use the same language model decoding setup the Willett et al. (II)) We provide brief description of the language model and beam search process here, with additional details provided in the original study.
The n-gram LM was converted to weighted finite state transducer (WFST) 31 This language model WFST, referred as the grammar WFST, was composed with two additional WFSTs representing the mapping from phonemes to words (lexicon WFST) and the mapping between phonemes and the CTC blank token model logits (token WFST). To compute the word error rate (WER), applied beam search on the composed WFST graph to obtain the most likely text sequence from
Table 5: Baseline GRU hyperparameters. Window length indicates the number of stacked neural time bins fed as input per GRU step, and stride size indicates the rate at which the GRU outputs phonemes.
Hyperparameter
Value
Window Length
32 (640 ms)
Stride Size
(80 ms)
Layers
5
Hidden Size
1024
LR
0.02
Epochs
73
White Noise
0.8
Baseline Shift
0.2
L2 Decay
le 5
Dropout
0.4
Input Dropout
0.0
Gaussian Smooth Kernel Size
20
Gaussian Smooth
2.0
16

---

Table 6: Transformer architecture and training hyperparameters. Feed-forward network (FFN) multiplier indicates the increase in dimensionality in the FFN module. The values N and M represent the number of time-masks and max mask length percentage of trial duration, respectively.
Hyperparameter
Value
Patch Length
5 (100 ms)
Patch Width
256
Transformer Dim
384
Layers
5
Heads
6
Dim Head
64
FFN Multiplier
4
LR
0.001
White Noise
0.2
Baseline Shift
0.05
N
20
M
7.5
Epochs
250
Dropout
0.35
Input Dropout
0.2
L.2 Decay
le - 5
Gaussian Smooth Kernel Size
20
Gaussian Smooth o
2.0
the GRU/Transfrmmer. We denote the GRU/Transformer as the "encoder" for this section. The probability of given beam (i.e., one possible decoded transcription), b, computed weighted sum between the probability assigned by the encoder and the probability assigned by the n-gram language model:
score (og((enc((b) + log( (Og((Pageie Pagram (b))
The beam with the lowest log probability was then selected as the final decoded transcription. Blank labels emitted by the encoder were additionally penalized by dividing their probability by a constant. When using the 3-gram LM, used beam size of 18, 0.8, and set the blank penalty log(2). For the 5-gram LM, we modified the blank penalty to log(7). All decoding hyperparameters were the same as the original Willett et
study.
When using the 5-gram LMM, two additional decoding steps were performed. First, a second pass was applied with unpruned 5-gram LM. The second pass replaces the original LM beam scores with updated, more accurate scores from the unpruned LM. Second, the top K beams were returned, and an LLM (OPT 6.7B (47)) was used score these top K beams. The updated score for each beam was then:
score(b) = a- (one + B loog (ongramm
ngrram (b)) + (1- (og( (b))
Following Willett et al. (41 we set K = 100, f = 0.5, and ran OPT 6.7B in 8-bit precision mode.
F Linderman Lab GRU
The Linderman Lab bidirectional GRU modified the original baseline GRU architecture to include the following stack of layers after the final (5th) GRU layer.
self fc decoder_ out = nn Sequential (
nn LayerNorm (h * 2),
nn. Dropout (p=dropout)
nn. Linear (h 2, h 2). nn ().
17

---

mn LayerNorm (h & 2),
nn. Dropout (p=dropout)
nn. Linear (h * 2, h 2),
nn. nn. S...iI
nn. Dropout (p=droputtt
nn. Linear (h 2, nclasses + 1),
Where h refers to the number of hidden units in the GRU (1024), dropout refers to the dropout probability (0.4) and nclasses refers to the number of phonemes (40). Beyond changes the model architecture, the GRU was trained using a linear learning rate scheduler, starting from 0.025 and ending at 0.0005 for 20,000 batches (~~ 146 epochs). An input dropout layer, with the dropout value set to 0.3 was also included before the day-specific linear layer. All other hyperparameters were the same the baseline GRU model.
G Large language model fine-tuning
To perform LLM fine-tuning, we first generated decoded transcripts using seeds of the time- masked Transformer and the 5-gram LM setup for the entire training and validation set. We then fine-tuned Llama 3.1 8B to predict the ground-truth transcript given the 10 decoded sentences for each trial. Fine-tuning was performed by "masking" the ground-trudh truth transcript and training the LLM to predict the masked tokens with cross-entropy loss. The L.LM was fine-tuned with QLoRA (4) the Unsloth package
(13). We used the default hyperparameters provided by Unsloth for Ilama 3.1 8B, which are listed in the table below. prompt, also provided below, was used to guide the LLM performed, our procedure consisted of fine-tuning the LLM first on the validation set, then on the training set, and finally once more on the validation set. We report results from single LLM fine-tuning seed.
for each trial. As no hyperparameter tuning
Table 7: Hyperparameters for model training. Lora-specifie parameters are listed at the bottom.
Hyperparameters
Values
Warmup Steps
5
Epochs
1
Learning Rate
2 x 10-4
Weight Decay
0.01
Learning Rate Scheduler
Linear
Batch Size
16
Lora Alpha
16
Lora Dropout
0
Lora Rank
16
LLM Prompt: Your task is to perform automatic speech recognition error correction. Below are multiple candidate transcriptions of the same utterance. These candidates were decoded from neural activity and may contain errors. Based on the candidates, produce the single most accurate, coherent, and grammatical transcription of the original utterance. Focus on key differences between candidates that change meaning or correctness, and avoid repetitive or nonsensical phrases. Respond with only the final corrected transcription-na explanations or extra text.
H Beam search decoding time
We measured the average time required for beam search decoding using the 3-gram language model, excluding the time to generate logits. With the baseline GRU, the average decoding time across 1200 test trials was 0.056 + 0.042 seconds per trial (mean + standard deviation). In contrast, the time- masked Transformer was over 3 times faster, achieving an average decoding time of 0.017 + 0.011 seconds. One potential explanation is that the Transformer outputs logits at Hz, compared to 12.5 Hz. for the GRU, resulting in fewer decoding steps. However, the time-masked GRU, which shares the same output resolution as the baseline GRU, achieved a decoding speed of 0.041 + 0.034 seconds, meaning the reduced output temporal resolution does not fully account for the faster decoding times.
18

---

Table 8: Performance under different masking strategies across 4 seeds. N is the number of masks,
and M is the max mask length as a percentage of trial duration.
N and M
Validation WER (%)
20 and 15
26.16 2.11
20 and 12.5
20.27 = 0.33
20 and 10.0
17.74 0.28
20 and 8.5
17.15
20 and 7.5
17.08
20 and 5.0
17.96 0.31
20 and 2.5
20.81 + 0.29
20 and 1.5
=
10 and 15.0
17.08 - 0.18
10 and 10.0
17.31 + 0.31
30 and 7.5
18.14 + 0.33
30 and 5.0
17.5 + 0.23
I Performance with other masking strategies
We report validation WER across different masking ratios for the Transformer when using the 3-gram LM (Table Sb. We set N = 20 and M = 7.5 throughout the study.
We additionally experimented with masking entire channels (channcl-masking) instead of time- masking. In order to do so, we applied N channel-masks for each microelectrode array. For each mask, selected starting channel i. Then, masked (replaced with 0) the closest p channels from channel i, where "closest" was quantified by the euclidean distance from channel i. was uniformly sampled between 0 and M 64 (there could range from 0 to 1. Our preliminary results suggested that channel masking was not as effective as time- maskina on this dataset, either when applied alone or in combination with time-mass mmmemmskingg
64 channels per microelectrode array), and M
Calculating average masking percentage
Refer to Algorithm 1 for the full variable definitions. Here we derive the expected fraction of patches that are masked. Given that each mask length is uniformly sampled between 0 and F, the expected mask length is F/2. Ignoring boundary effects, the probability that given patch T is not masked by a given mask Nj is:
F
F
ML
M
P(unmaskedy.x:) ~
=1
2T.
2T.
Then, the probability that patch / is unmasked after all masks are applied is:
NN
N
N
P(i unmasked) = II
P(unmasked), = M 2
=
2
k=1
To find the proportion of masked patches, take the complement. Using the linearity of expectation, we find that the expected proportion of masked patches is:
1 T, ==
E fraction of masked patches]
M
1-
M
For our given parameters, we approximate that 53.44% of patches are masked on average for each trial.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims from the abstract and introduction match the claims from the results and figures. We make three core contributions, and provide evidence for the effectiveness of each throughout our results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are described in the first paragraph of the Discussion.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
20

---

Answer:
[NA]
Justification:
There are no theoretical results in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: All methods and architectures are fully detailed in the Methods section. All relevant models that are used in the paper detailed. Hyperparameters and other details for reproducibility are listed in the appendix section. A link to the code base used to generate all results is provided.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
21

---

reproducible, they 5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes)
Justification: The dataset used is open-source and freely available online. A link to the code base is also provided.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
//iipscoom wttpcccpccpppcccccccceooocp httpa///guideee/ddeuubbbbiiiii ieriiisssonoooii
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specifies all the above information in either the Methods or Ap- pendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report standard error of the mean or standard deviation values across random seeds throughout the paper. We report the sample sizes for all statistical tests.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
22

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, the paper reports the resources used to run the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: tttttps://eeurtii http///uripp..e/ hww.rrpp..cpppll////hiiiiee cccppullic/thiiccciiddee......ee
Answer: [Yes]
Justification:
The paper follows all the criteria listed in the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discusses societal impacts in the introduction and discussion sec- tions.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
23

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The methods outlined by the paper pose no risk of misuse beyond speech neuroprostheses as whole.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The paper properly cites all data and models provided by other authors.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
24

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pawpess..ppwwiiii.... thcode. com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Code availability is stated in the appendix.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification:
The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not collect data from any study participants beyond what was available in an already available open-source dataset.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
25

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs were only used to edit writing.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: http:s://neurps.... http://errrrrecc/ooeerrreee///// cc/Coonferences/22255/LI. for what should or should be described.
26


Reviews:
[{"review_id": "Xcb1fIKypk", "reviewer": "Reviewer_1NPM", "summary": "This paper studies the ways to improve Speech Neuroprostheses. The ideas are technical sounds and practical. \n\nThe TTA CORT implementation is particularly noteworthy in BCI context as brain signals carry less information but vary more from trial to trial and from day to day than other signals such as speech and video signals. While the TTA technique is not entirely new (I consider it is new in BCI context), the successful CORT implementation is an important step towards practical and real-time deployment. The empirical improvements as presented in Fig 4 are noticeable and consistent. The technical is well justified and easy to repeat, therefore valuable. I consider that the experiments are well devised, and the results are generalizable.", "questions": "How does the online recalibration with time masking calibrate at the beginning for a new subject? As this paper studies the data from a single subject, the effect of recalibration is minimum. In general terms, the recalibration should cover variations across trials, settings, subjects etc,  authors may want to discuss the generalization ability of the techniques.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Both Masked Transformers with Test-Time Adaptation techniques are not new. But the implementation involves careful engineering. The study improved speech neuroprostheses for accuracy, capability for real-time streaming, low computational and memory costs, and robustness to distribution shifts, and achieved impressive results. \n\nThe paper didn't motivate how replacing GRU with Transformer improves, and what problem it solves by doing so.\n\nIt is assumed that the proposed architecture adopts the same language model strategy - \"The decoded phonemes logits are then integrated with an N-gram language model through shallow fusion. We use a 3-gram LM to replicate the NPTL Baseline (RNN + 3-gram\nLM) algorithm reported on the benchmark leaderboard (note the RNN is a GRU).\". A shallow fusion means that the decoding is done as a post-processing, that violates the streaming requirements.", "was_revised": true, "final_justification": "I appreciate the authors’ detailed and thoughtful rebuttal.\n\nThe additional justification for replacing GRU with Transformers—specifically for enabling more efficient and lightweight test-time adaptation—is clear and convincing. The new DietCORP method improves upon CORT meaningfully, and the reported WER reductions across large temporal gaps further validate the practicality and robustness of the proposed system.\n\nThe clarification on shallow fusion being compatible with streaming alleviates my concern regarding streaming constraints.\n\nI maintain my positive assessment and consider the paper a strong contribution to the speech neuroprosthesis and BCI community.", "rebuttal": "Thank you for the feedback. We appreciate that the Reviewer finds our test-time adaptation method useful, and we have since made additional improvements to it.  We provide a one sentence response for each question and weakness, with additional details provided below. Tables are provided at the end of the text. \n\n**Motivation for replacing the GRU with the Transformer**: Our primary motivation for replacing the GRU with the Transformer was to propose an architecture that was more suitable for test-time adaptation (TTA), where computational efficiency is very important since TTA is typically performed on-device while the participant is using the neuroprosthesis. \n\n- *Additional details:* A $5$-layer version of our time-masked Transformer, which performs slightly better than our original $7$ layer model, contains less then $10$ M parameters. By comparison, the bidirectional GRU used by the entries in Willett et al., 2025 [b] contain roughly $130$ M parameters. When applying our updated test-time adaptation method (see Updated Test-Time Adaptation Section), we find that the Transformer requires only $1.04$ GiB of peak GPU memory usage and $29.3$ ms to calibrate per trial, whereas the bidirectional GRU requires $5.6$ GiB of peak GPU memory usage and $163$ ms to calibrate per trial (computed on an Nvidia GeForce RTX 3090 GPU). These gains are important for applying TTA on local, resource-constrained devices. \n\n**Shallow fusion violates streaming requirements**: Shallow fusion is compatible with streaming systems, and was used in a streaming setting in Supplementary Video 1 from the Willett et al., 2023 [a] study. \n\n- *Additional details:* While we applied the shallow fusion step after the model logits were acquired for convenience, the shallow fusion stage itself does not require access to future logits. During shallow fusion, each beam is constructed one timestep at a time, and the most probable beam for that timestep is selected to be shown on the screen. Therefore, shallow fusion is streaming compatible because text is outputted in real-time, and the algorithm does not need to wait until the entire trial has completed. \n\n**How does online recalibration with time masking calibrate in the beginning for a new subject or in other settings**: We have included TTA results in a more challenging setting (Table 2). \n\n- *Additional details:* While we agree that online recalibration to new participants is a very interesting question, there was no other data from other participants at the time we wrote this study. We agree an interesting direction for future work is to calibrate a decoder to a new participant with minimal supervised trials, and then use online recalibration to maintain the performance of this decoder on the new participant. We will include this point in our discussion. We have included a more challenging setting where the model is evaluated on $8$ held-out days in our updated test-time adaptation method section. Under this setting, our updated method is able to maintain stable performance, and the effect of recalibration is quite large (Table 2). \n\n**Updated Test-Time Adaptation Method**: Using time masking and Transformers, we designed a computationally lightweight version of CORP (Li et al., 2023 [b]) which is a significant improvement over our old TTA method, CORT. \n\n- *Additional details:* While we did find a small reduction in WER when applying CORT relative to No CORT across days (average WER improvement across days shown in Figure 4, left), this improvement was small. We thus sought to examine if applying time masking and Transformers could provide efficiency gains to the pseudo-label based TTA method proposed by Fan et al., 2023 [c]. The method used by Fan et al., 2023, CORP, adapts a model at test-time by treating the language model (LM) refined model outputs as pseudo-labels, and then training the model with CTC loss using these pseudo-labels. Since the LM refined outputs are already being generated for real-world use, there is no additional cost to generate these pseudo-labels. However, CORP requires up to $200$ adaptation steps per test trial, setting a loss threshold hyperparameter, and including previous data to stabilize training. Here, we present \"DietCORP\", a lightweight version of CORP which only uses one adaptation step per trial and no previous data. Similar to CORT, DietCORP operates by generating several time masked augmentations of the same trial, and then it adapts the model to produce the same pseudo-label across all augmentations. We present results when applying DietCORP to a time-masked Transformer trained on $15$ days of data, and evaluated on the next $5$ competition days (Table 1), and a time-masked Transformer trained on $12$ days of data and evaluated on the next $8$ competition days (Table 2). Only the patch embedding module was adapted. DietCORP leads to very stable WER across days relative to when not applying it with minimal hyperparameter tuning. We provide additional comparisons with CORP in the updated manuscript. \n\n***\n**Tables**\n\nTable 1: Values are word error rate (mean ± SEM across N=$4$ seeds) computed on $5$ held-out competition days using a model trained on the previous $15$ days. \n\n| Days since last training day | No DietCORP | DietCORP  |\n|--------:|----------------:|----------------:|\n| 7       |  22.74 ± 0.26        | 21.14 ± 0.32        |\n| 13     |  25.48 ± 0.28        | 21.23 ± 0.21       |\n| 19     |  25.43 ± 0.33       | 21.40 ±  0.22       |\n| 28     |  30.13 ± 0.33     | 21.93 ±  0.65       |\n| 30     |  32.59 ± 0.52     | 22.26 ±  0.60        |\n\n\nTable 2: Values are word error rate (mean ± SEM across N=$4$ seeds) computed on $8$ held-out competition days using a model trained on the previous $12$ days. \n\n| Days since last training day | No DietCORP | DietCORP  |\n|--------:|----------------:|----------------:|\n| 5       | 28.87 ± 0.47        | 26.97 ± 0.28          |\n| 12     | 34.50 ± 0.44       | 26.38 ± 0.30         |\n| 21     | 44.79 ± 0.34       | 27.57 ±  0.13        |\n| 28       | 53.29 ± 0.25        | 30.46 ± 0.41         |\n| 34       | 59.98 ± 0.31        | 29.70 ± 0.20         |\n| 40       | 52.71 ±  0.52        | 29.08 ± 0.38         |\n| 49       | 62.96 ± 0.40         | 31.35 ± 0.76         |\n| 51       | 66.47 ± 0.58         | 31.91 ± 1.25        |\n\n[a] Willett et al., 2023, A high-performance speech neuroprosthesis\n\n[b] Fan et al., 2023, Plug-and-Play Stability for Intracortical Brain-Computer Interfaces: A One-Year Demonstration of Seamless Brain-to-Text Communication"}, {"review_id": "NZYEvcG7vJ", "reviewer": "Reviewer_oJzC", "summary": "This paper does a deep dive on the architecture of the GRU in Willett et al 2023, and makes a number of principled adjustments to the decoding model in other to improve performance and reduce computational load. They introduce a novel time masking strategy, which prevents overfitting, and shows a strong reduction in the fundamental error rate before any post hoc adjustments using ensembles or language models. They also switch out the GRU for a lightweight transformer, with no loss and maybe a small gain in performance. They then complete detailed analyses of whether their approach sacrifices any robustness for the improved performance, which is does not appear to.", "questions": "None", "limitations": "None", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Overall, this was an excellent paper. The literature review was well done, and the approach was highly principled. Code is included, and this analysis utilized a high quality public dataset. The performance results were strong, and it is particularly useful that the fundamental classification percentage has been improved, rather than the downstream corrections. Everything was very well explained, and strongly motivates the use of this approach. I didn’t note any weaknesses throughout the paper.", "was_revised": true, "final_justification": "After reading through the list of other reviews and rebuttals, I will maintain my high score due to the solid quantitative effect on performance.", "rebuttal": "Thank you so much for the positive review! We appreciate that the reviewer found our focus on improving the neural network that maps to phonemes useful."}, {"review_id": "hd2ctHki4y", "reviewer": "Reviewer_FhGX", "summary": "The paper investigates application of transformer architectures for brain to text decoding to be incorporated in speech neuroprostheses. The work proposes to replace the GRU baseline model previously developed for this problem with a transformer along with masking the input of neural activity during training. Furthermore, in testing implementation of test-time adaptation is proposed which acts as “muti augmented head” which selects the most confident prediction out of N variants. Such a procedure is expected to be efficient when considering generalization to multiple sessions and days. Results are reported on the brain-to-text 24’ benchmark and compared with baseline performance of NPTL+3gram in that benchmark.", "questions": "Q1. How would the proposed model perform with 5gram LM?\n\nQ2. What are the reasons for transformer based architectures to perform with lower accuracy than SOTA counterparts?\n\nQ3. How would the model perform with additional strategies proposed by recent work in brain to text described in R2, e.g. would incorporation of diphones in the input masked decoder improve PER and WER?\n\nR2. Willett, F. R., Li, J., Le, T., Fan, C., Chen, M., Shlizerman, E., ... & Henderson, J. M. (2024). Brain-to-Text Benchmark'24: Lessons Learned. arXiv preprint arXiv:2412.17227.", "limitations": "Somewhat yes though the authors did not address the limitations particularly related to using transformers with neural activity and text output.", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nS1. The work proposes and implements an masked input transformer for neural activity to speech problem and shows improvement of performance compared to baseline of NPTL + RNN + 3-gram LM.\n\nS2. The work proposes continual online recalibration with time masking (CORT) that allows improvement in generalization for new sessions/days.\n\nS3. CORT allows for lower memory use and faster calibration time than compared baseline GRU approach.\n\n\nWeaknesses:\n\nW1. The results are reported with respect to a single baseline (NPTL + RNN + 3-gram LM currently placed #25 in the benchmark). This baseline is significantly weaker than other baselines (eg NPTL + 5-gram+OPT #12 in the benchmark) and other approaches that are state of the art  (alternative architectures or enhancements to GRU approach). \n\nW2. Related to W1, the proposed approach achieves a modest improvement in WER with respect to other approaches and would be placed approximately #15 in the benchmark indicating a significant gap between accuracy of the proposed approach vs. other approaches.\n\nW3. Relation of Phoneme Error Rate to Word Error Rate rate in the model appears to be unclear. \n\nW4. Technical contributions in the paper are not new (input masking, multi-head consensus) and have been applied in general deep learning context and in the context of neural time series.  Furthermore, as the authors indicate some of the architectural variants (various transformers) might have been implemented by previous works and reported as less accurate, which the current work is able to reaffirm.\n\nW5. Input masking is an effective approach and has been applied in various datasets contexts, eg. see R1 and studies that followed. These are not described in related work and recent strategies developed for masking could turn out to improve accuracy of the masked transformer.\n\nR1. Chang, H., Zhang, H., Jiang, L., Liu, C., & Freeman, W. T. (2022). Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 11315-11325).", "was_revised": true, "final_justification": "The authors advanced the work during rebuttal from being placed in ~ 20 place in the benchmark competition in the original submission to being competitive with the state-of-the-art methods. Also significant amount of grasp of related work and the differences of proposed approach wrt to related work occurred during rebuttal and discussion periods.\n\nThe main contribution appears to be that this work succeeded to practically achieve the impressive performance with transformers, which mostly got advanced in the rebuttal and discussion phases, If these results were reported in the original submission I would rank the paper higher but to avoid giving an unfair advantage over other submissions I would like to keep my original score of 2.", "rebuttal": "Thank you for the feedback. We appreciate that the Reviewer finds our improvement relative to the NPTL 3-gram baseline noteworthy. We provide a one sentence response for each question and weakness, with additional details provided below. Tables are provided at the end of the text. \n\n**General Note**: We realized after submission that all the entries in Willett et al. 2025 [b] used a bidirectional GRU. A bidirectional GRU can only decode speech once the entire trial has finished, and roughly doubles memory usage, training times, and calibration times relative to the unidirectional GRU. All our models are either unidirectional GRUs or causal Transformers which only rely on present and past neural activity to generate outputs. \n\n**Low performance on benchmark + Results are not shown with 5-gram LM setup**: When using the 5-gram language modeling setup used by the entries in the Brain-to-Text Benchmark ‘24, the time-masked Transformer achieves a mean WER of 8.01%, which ties the 2nd place entry (Linderman Lab GRU) despite using ~$15$ times less parameters and decoding in real-time. (Table 1). \n\n- *Additional Details:* We have added results (Table 1) with the offline language modeling (LM) setup used by the entries in Willett et al., 2025 [b]. Specifically, the offline language modeling setup involved using a 5-gram LM, with a second pass using the 5-gram LM, and rescoring with OPT 6.7B. We refer to the setup where only the 3-gram LM is used with no second pass as the online LM setup. The only entry that performs substantially better than our entry is DConD-LIFT (Li et al., 2024 [d]) (WER = 5.77%). However, the primary performance gain from DConD-LIFT stems from applying GPT 3.5 to an ensemble of bidirectional GRUs, which requires cloud resources and is not streaming compatible. When focusing on the improvements made to the GRU itself by Li et al., 2024 [d], specifically diphone decoding, their bidirectional GRU achieves a WER of 8.06% which is slightly worse than our much smaller, causal time-masked Transformer. \n\n**How would the model perform with the additional advances described in Willett et al., 2025 [b]?**: Our preliminary data suggest that incorporating the post GRU block from the Linderman Lab to the Transformer does not improve performance, and our causal time-masked Transformer slightly outperforms the bidirectional GRU which decodes diphones from Li et al., 2024 [c]. \n\n**Transformer performs with lower accuracy than SOTA**: We would like to clarify that the time-masked Transformer outperforms competing GRU-based entries in the real-time decoding setting (Table 1), and that a major point of our study is that time-masked Transformers can decode speech with high accuracy and are well-suited for test-time adaptation.\n\n**Relation of phoneme error rate to word error rate is not clear**: We will expand our existing discussion of this (in the Limitations) for the updated manuscript. \n\n- *Additional details:* We used PER to select the best model during validation because that is the procedure used by Willett et al., 2023 [a] and because computing WER during training is computationally expensive. However, we agree that PER is not an ideal metric to optimize for, and we will include cases that highlight this in the updated manuscript. We believe exploring ways to directly optimize WER require a dedicated study. \n\n**Technical contributions are not new**: Time masking and Transformers were not used by any other entry in Willett et al., 2025 [b], and we further combine these contributions to provide an updated version of our previous TTA method (CORT). \n\n- *Additional details:* Time masking has been applied in three other neural speech decoding studies to our knowledge ([e,f,g]), however these studies used closed source ECoG datasets, used much smaller amounts of time masking, and did not quantify the impact of time masking relative to other augmentations. We will cite these studies, as well the reference the reviewer provided (Chang et al., 2022), in our updated related works section.\n\n- Previously we combined our two contributions, time masking and Transformers, to propose CORT. However, the improvement in WER from CORT is relatively small. We thus sought to examine if applying time masking and Transformers could provide efficiency gains to the pseudo-label based TTA method proposed by Fan et al., 2023 [c]. The method used by Fan et al., 2023, CORP, adapts a model at test-time by treating the language model (LM) refined model outputs as pseudo-labels, and then training the model with CTC loss using these pseudo-labels. Since the LM refined outputs are already being generated for real-world use, there is no additional cost to generate these pseudo-labels. However, CORP requires up to $200$ adaptation steps per test trial, setting a loss threshold hyperparameter, and including previous data to stabilize training. Here, we present \"DietCORP\", a lightweight version of CORP which only uses one adaptation step per trial and no previous data. Similar to CORT, DietCORP operates by generating several time masked augmentations of the same trial, and then it adapts the model to produce the same pseudo-label across all augmentations. We present results when applying DietCORP to a time-masked Transformer trained on $15$ days of data, and evaluated on the next $5$ competition days (Table 2), and a time-masked Transformer trained on $12$ days of data and evaluated on the next $8$ competition days (Table 3). Only the patch embedding module was adapted. DietCORP leads to very stable WER across days relative to when not applying it with minimal hyperparameter tuning.\n\n- DietCORP only requires $1.04$ GiB of peak allocated GPU memory usage and $29.3$ ms to calibrate per trial using the Transformer. When applying our TTA method to the unidirectional GRU, it requires $2.8$ GiB of peak allocated GPU memory usage and $76.9$ ms to calibrate per trial. These values further increase to $5.6$ GiB and $163.0$ ms when using the bidirectional GRU. We will also include comparisons when using CORP in the updated manuscript, where DietCORP applied to the Transformer exhibits even larger computational gains. \n\n- In summary, while we agree our three contributions are not entirely novel in the context of machine learning as a whole, we believe they represent a unique application on the first open source neural speech decoding dataset, and have been combined in novel ways. Furthermore, we would like to point out that the NeurIPs guidelines state that applications of existing methods to improve performance are equally as valuable as proposing novel methods. \n\n***\n**Tables**\n\nTable 1: We show results with a $5$ layer time-masked Transformer trained for $250$ epochs (Transformer - Ours) as well as comparisons with the baseline provided by Willett et al., 2023 [a] and the Linderman Lab GRU discussed in [b]. We run the baseline and Linderman Lab GRU in unidirectional mode, and show metrics in bidirectional mode in parenthesis when available. \n\n| Metric                    | Transformer - Ours | Baseline GRU       | Linderman Lab GRU   |\n|---------------------------|-------------------:|-------------------:|--------------------:|\n| Online LM Setup WER %   | 12.03 ± 0.24        | 15.25 ± 0.16        | 15.03 ± 0.19         |\n| Offline LM Setup WER % | 8.01 ± 0.23         | 11.12 ± 0.13 (9.76) | 10.77 ± 0.13 (8.0)   |\n| Parameters (M)          | 9.4                 | 56.7 (135.4)        | 58.8 (143.8)         |\n| Training time (hours)   | ~1                  | ~0.5  ( ~1)         | ~1  ( ~2)            |\n\nTable 2: Values are word error rate (mean ± SEM across N=$4$ seeds) computed on $5$ held-out competition days using a model trained on the previous $15$ days. \n\n| Days since last training day | No DietCORP | DietCORP  |\n|--------:|----------------:|----------------:|\n| 7       |  22.74 ± 0.26        | 21.14 ± 0.32        |\n| 13     |  25.48 ± 0.28        | 21.23 ± 0.21       |\n| 19     |  25.43 ± 0.33       | 21.40 ±  0.22       |\n| 28     |  30.13 ± 0.33     | 21.93 ±  0.65       |\n| 30     |  32.59 ± 0.52     | 22.26 ±  0.60        |\n\nTable 3: Values are word error rate (mean ± SEM across N=$4$ seeds) computed on $8$ held-out competition days using a model trained on the previous $12$ days. \n\n| Days since last training day | No DietCORP | DietCORP  |\n|--------:|----------------:|----------------:|\n| 5       | 28.87 ± 0.47        | 26.97 ± 0.28          |\n| 12     | 34.50 ± 0.44       | 26.38 ± 0.30         |\n| 21     | 44.79 ± 0.34       | 27.57 ±  0.13        |\n| 28       | 53.29 ± 0.25        | 30.46 ± 0.41         |\n| 34       | 59.98 ± 0.31        | 29.70 ± 0.20         |\n| 40       | 52.71 ±  0.52        | 29.08 ± 0.38         |\n| 49       | 62.96 ± 0.40         | 31.35 ± 0.76         |\n| 51       | 66.47 ± 0.58         | 31.91 ± 1.25        |\n\n[a] Willett et al., 2023, A high-performance speech neuroprosthesis\n \n[b] Willett et. al, 2025, Brain-to-Text Benchmark '24: Lessons Learned\n \n[c] Fan et al., 2023, Plug-and-Play Stability for Intracortical Brain-Computer Interfaces: A One-Year Demonstration of Seamless Brain-to-Text Communication\n\n[d] Li et al., 2024, Brain-to-Text Decoding with Context-Aware Neural Representations and Large Language Models\n\n[e] Metzger et al., 2023, A high-performance neuroprosthesis for speech decoding and avatar control\n\n[f] Metzger et al., 2022, Generalizable spelling using a speech neuroprosthesis in an individual with severe limb and vocal paralysis\n\n[g] Littlejohn et al., 2025, A streaming brain-to-voice neuroprosthesis to restore naturalistic communication"}, {"review_id": "2zM9B1XSuP", "reviewer": "Reviewer_VfR8", "summary": "This paper examines the application of masked Transformers in conjunction with test-time adaptation techniques for speech neuroprostheses. It addresses the real-time processing and memory-efficiency limitations of existing GRU-based models, as well as distribution shifts in neural signals over time. The core idea is to apply time masking to sequential neural activity during training to mitigate overfitting and to deploy a lightweight Transformer architecture that maximizes parallelism and reduces the memory footprint. At test time, the model adapts to ensure consistent predictions on time-masked inputs, thereby enhancing its robustness to changes in temporal distribution.", "questions": "- Would similar results be obtained using Dorsal 6v instead of Ventral 6v?\n- Why is a Transformer architecture more effective than GRUs when paired with time masking?\n- Are the default time masking parameters (N = 20, M = 0.075) optimal?", "limitations": "See above", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "1. Claims And Evidence\n    - Strengths\n        - Figure 2 demonstrates that applying time masking significantly narrows the gap between training and validation CTC loss, indicating its effect in mitigating overfitting.\n        - Table 7 shows the effectiveness of time masking across various choices of N and M.\n        - Figure 3 presents a clear comparison between the Transformer and a baseline GRU, revealing simultaneous gains in both efficiency and accuracy under diverse conditions.\n        - By relying solely on the Transformer model, without LLMs or ensemble strategies, the approach achieves optimized streaming performance.\n    - Weaknesses\n        - Time masking is effectively a form of data augmentation; however, the paper lacks a comparative evaluation against other augmentation methods, such as SpecAugment, additive white noise, and speed perturbation [a, b].\n        - The GRU-based TeamCyber model [c] attains a WER of 8.26%, whereas the proposed method yields 12.27 ± 0.15%, a substantially worse result that calls into question its significance.\n2. Methods And Evaluation Criteria\n    - Weaknesses\n        - Since time masking is a widely used technique, it is difficult to regard it as the paper’s main contribution. To make such a claim credible, the authors would need to benchmark it against alternative approaches (e.g., speckled masking [g]).\n        - Beyond CORT, the paper does not compare with other established test-time adaptation methods (e.g., pseudo-label–based adaptation, sample-efficient entropy minimization, test-time normalization) [d, e, f].\n        - The Method section lacks a detailed mathematical formulation for how test-time adaptation is applied; a more explicit derivation would strengthen reproducibility.\n3. Theoretical Claims\n    - Weaknesses\n        - The paper provides limited theoretical justification for why Transformers are inherently more efficient and robust than GRUs and lacks experiments comparing alternative architectures or variants to demonstrate generality.\n4. Experimental Designs Or Analyses\n    - Weaknesses\n        - An exploratory data analysis illustrating day-to-day distribution shifts in the neural signal data would have been informative.\n        - The proposed method updates only the weights of the patch embedding module during online TTA while freezing all other layers. In contrast, the method in [18] performs partial fine-tuning across every layer; however, the authors did not evaluate a similar multi-layer adaptation strategy—an experiment that would be necessary to assess the impact of broader fine-tuning.\n        - DConD-LIFT [c] achieved a WER of 5.77% by combining an RNN ensemble with an LLM; the proposed method’s performance appears to be far inferior. Although the paper cites memory usage as the key advantage of using a Transformer, it is unclear whether this benefit justifies the performance gap, and no supporting experiments are provided.\n5. Others\n    - Typographical error: “partiicpant” (Section 5: Limitations)\n\n> \n> [a] Park, Daniel S., et al. \"Specaugment: A simple data augmentation method for automatic speech recognition.\" *arXiv:1904.08779* (2019).\n> \n> [b] Huh, Mina, Ruchira Ray, and Corey Karnei. \"A Comparison of Speech Data Augmentation Methods Using S3PRL Toolkit.\" arXiv:2303.00510 (2023).\n> \n> [c] Willett, Francis R., et al. \"Brain-to-Text Benchmark'24: Lessons Learned.\" *arXiv:2412.17227* (2024)\n> \n> [d] Wang, Qin, et al. \"Continual test-time domain adaptation.\" *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2022.\n> \n> [e] Niu, Shuaicheng, et al. \"Efficient test-time model adaptation without forgetting.\" *International conference on machine learning*. PMLR, 2022.\n> \n> [f] Lim, Hyesu, et al. \"Ttn: A domain-shift aware batch normalization in test-time adaptation.\" ICLR (2023).\n> \n> [g] Keshtkaran, Mohammad Reza, and Chethan Pandarinath. \"Enabling hyperparameter optimization in sequential autoencoders for spiking neural data.\" *Advances in neural information processing systems* 32 (2019).\n>", "was_revised": true, "final_justification": "As stated in the introduction, the authors claim to propose a holistic approach, centered on the model architecture (including masking strategies). They explicitly note that GRUs tend to overfit in the early stages of training and that they process highly redundant inputs at each time step, which is a limitation.\n\nHowever, when GRUs are applied bidirectionally, the WER differs by only 0.09%, as also reported in Author Final Remarks. This result suggests that the importance lies more in incorporating bidirectional context rather than in the GRU architecture or the masking strategy itself. Consequently, I believe the validity of the authors’ claim is weakened, and the proposed technique lacks sufficient supporting evidence. Furthermore, in the context of Reviewer FhGX’s comments, I find the claim of being a “holistic” technique to be over-hyped.\n\nIn conclusion, while the research topic and the additional experiments presented during the review process are impressive, due to the significant issues outlined above, I have decided to assign a final rating of reject.", "rebuttal": "Thank you for the feedback. We appreciate that the Reviewer noted our focus on optimizing performance in a real-time streaming setting. We provide a one sentence response for each question and weakness, with additional details provided below. Tables are provided at the end of the text. \n\n**General Note**: We realized after submission that all the entries in Willett et al. 2025 [b] used a bidirectional GRU. A bidirectional GRU can only decode speech once the entire trial has finished, and roughly doubles memory usage, training times, and calibration times relative to the unidirectional GRU. All our models are either unidirectional GRUs or causal Transformers which only rely on present and past neural activity to generate outputs. \n\n**Low performance on benchmark**: When using the 5-gram language modeling setup used by the entries in the Brain-to-Text Benchmark ‘24, the time-masked Transformer achieves a mean WER of 8.01%, which ties the 2nd place entry (Linderman Lab GRU) despite using ~$15$ times less parameters and decoding in real-time. (Table 1). \n\n- *Additional Details:* We have added results (Table 1) with the offline language modeling (LM) setup used by the entries in Willett et al., 2025 [b]. Specifically, the offline language modeling setup involved using a 5-gram LM, with a second pass using the 5-gram LM, and rescoring with OPT 6.7B. We refer to the setup where only the 3-gram LM is used with no second pass as the online LM setup. \n\n**DConD-LIFT achieves 5.77% WER**: The primary performance gain in the first place entry, DConD-LIFT, stems from applying GPT 3.5, while their main improvement to the GRU itself (decoding diphones) achieves a WER of 8.06%.\n\n- *Additional Details:* DConD-LIFT (Li et al., 2023) shows remarkable accuracy gains by applying GPT 3.5 to an ensemble of bidirectional GRUs. However, since their submission requires access to cloud resources and is not streaming compatible, we believe a comparison with our method strictly based on accuracy is not entirely fair. \n\n**Benchmark time masking against other regularization strategies**: We have included a comparison with the Linderman Lab GRU which incorporates speckled masking (i.e. input dropout) in Table 1, and our original study also included comparisons with other regularization strategies (channel masking, additive white noise, baseline shifts). \n\n- *Additional Details*: We ran the Linderman Lab GRU, which uses specked masking (i.e. input dropout), in unidirectional mode. The time-masked Transformer performs substantially better than the Linderman Lab GRU in a real-time decoding setting (Table 1), and matches the performance of the bidirectional Linderman Lab GRU (Table 1). Furthermore, the baseline GRU from Willett et al., 2023 [a] used high amounts of additive white noise and baseline shift augmentations, providing a strong comparison with our proposed time masking augmentation. We also incorporated speckled masking (denoted as input dropout in our manuscript), additive white noise, and baseline shifts when training models with time masking. We mentioned these details in the Methods and Appendix, and will state them more clearly in the updated manuscript. Regarding SpecAugment, we note that SpecAugment consists of three parts: time masking, channel masking, and time warping on an audio spectrogram. Time masking was a major focus of our study, and we also provided preliminary using channel masking in Appendix Section 7.3. We did not implement time warping since it had a minor effect in the original SpecAugment study (Park et al., 2023 [e]), and we did not implement speed perturbation since we are working with data from one patient who speaks at a relatively constant rate.\n\n**Comparison with existing test-time adaptation (TTA) methods**: The main TTA method in the field of communication neuroprostheses is continual online calibration with pseudo-labels (CORP) (Fan et al., 2023 [c]), and in our updated manuscript we present a computationally lightweight version of CORP. \n\n- *Additional Details:* In CORP, the model is adapted at test-time by treating the language model refined model outputs as pseudo-labels. CORP requires up to $200$ adaptation steps per test trial, setting a loss threshold hyperparameter, and including previous data to stabilize training. We propose \"DietCORP\", a lightweight version of CORP which only requires one adaptation step per test trial and does not require maintaining previous data. Similar to CORT, DietCORP operates by generating several time masked augmentations of the same trial, and then it adapts the model to produce the same pseudo-label across all augmentations. We present results when applying DietCORP to a time-masked Transformer trained on $15$ days of data, and evaluated on the next $5$ competition days (Table $2$), and a time-masked Transformer trained on $12$ days of data and evaluated on the next $8$ competition days (Table 3). DietCORP leads to stable WER across days relative to when not applying it. DietCORP only requires $1.04$ GiB of peak allocated GPU memory usage and $29.3$ ms to calibrate per trial using the Transformer, versus $5.6$ GiB and $163$ ms to calibrate when using the bidirectional GRU. We will provide direct comparisons with CORP in the updated manuscript. \n\n**Only the patch embedding module is updated for TTA**: We only updated the patch embedding module because it requires less resources, which is important for edge devices, and we will show additional results when adapting a larger fraction of the weights in the updated manuscript. \n\n**Mathematical Formulation for TTA**: We will provide a detailed mathematical formulation of our updated TTA method in the manuscript. \n\n**Theoretical investigation into why Transformers are more efficient than GRUs**: We hypothesize that Transformers are more efficient because they were optimized to process non-overlapping inputs, and show efficiency gains with a GRU that processes non-overlapping inputs.\n\n- *Additional Details:* When optimizing the time-masked GRU to process non-overlapping inputs, we obtain competitive performance with a GRU that uses less than half the parameters as the baseline GRU ($21$ M versus $56$ M). We will provide a discussion of this in the updated manuscript. \n\n**Exploratory analysis of day-to-day shifts in neural activity**: We will discuss previous work (Willett et al., 2021 [f]) which shows that day-to-day shifts in neural activity can be represented as rotations from the original space into a new space in our updated manuscript. \n\n**Optimal hyperparameters for time masking**: We show results with other time masking hyperparameters in Appendix Section 7.3, and will reference this more clearly in the updated manuscript. \n\n**Dorsal 6v**: The time-masked Transformer performs better than the baseline GRU when using only Dorsal 6v, which we will include in the updated manuscript. \n\n***\n**Tables**\n\nTable 1: We show results with the time-masked Transformer trained for 250 epochs (Transformer - Ours) as well as comparisons with the baseline provided by [a] and the Linderman Lab GRU discussed in [b]. We run the baseline and Linderman Lab GRU in unidirectional mode, and show metrics in bidirectional mode in parenthesis when available. \n\n\n| Metric                    | Transformer - Ours | Baseline GRU       | Linderman Lab GRU   |\n|---------------------------|-------------------:|-------------------:|--------------------:|\n| Online LM Setup WER %   | 12.03 ± 0.24        | 15.25 ± 0.16        | 15.03 ± 0.19         |\n| Offline LM Setup WER %  | 8.01 ± 0.23         | 11.12 ± 0.13 (9.76) | 10.77 ± 0.13 (8.0)   |\n| Parameters (M)          | 9.4                 | 56.7 (135.4)        | 58.8 (143.8)         |\n| Training time (hours) | ~1                  | ~0.5  ( ~1)         | ~1  ( ~2)            |\n\n\nTable 2: Values are word error rate (mean ± SEM across N=$4$ seeds) computed on $5$ held-out competition days using a model trained on the previous $15$ days. \n\n| Days since last training day | No DietCORP | DietCORP  |\n|--------:|----------------:|----------------:|\n| 7       |  22.74 ± 0.26        | 21.14 ± 0.32        |\n| 13     |  25.48 ± 0.28        | 21.23 ± 0.21       |\n| 19     |  25.43 ± 0.33       | 21.40 ±  0.22       |\n| 28     |  30.13 ± 0.33     | 21.93 ±  0.65       |\n| 30     |  32.59 ± 0.52     | 22.26 ±  0.60        |\n\nTable 3: Values are word error rate (mean ± SEM across N=$4$ seeds) computed on $8$ held-out competition days using a model trained on the previous $12$ days. \n\n| Days since last training day | No DietCORP | DietCORP  |\n|--------:|----------------:|----------------:|\n| 5       | 28.87 ± 0.47        | 26.97 ± 0.28          |\n| 12     | 34.50 ± 0.44       | 26.38 ± 0.30         |\n| 21     | 44.79 ± 0.34       | 27.57 ±  0.13        |\n| 28       | 53.29 ± 0.25        | 30.46 ± 0.41         |\n| 34       | 59.98 ± 0.31        | 29.70 ± 0.20         |\n| 40       | 52.71 ±  0.52        | 29.08 ± 0.38         |\n| 49       | 62.96 ± 0.40         | 31.35 ± 0.76         |\n| 51       | 66.47 ± 0.58         | 31.91 ± 1.25        |\n\n[a] Willett et al., 2024, A high-performance speech neuroprosthesis \n \n[b] Willett et. al, 2025, Brain-to-Text Benchmark '24: Lessons Learned\n \n[c] Fan et al., 2023, Plug-and-Play Stability for Intracortical Brain-Computer Interfaces: A One-Year Demonstration of Seamless Brain-to-Text Communication\n\n[d] Li et al., 2024, Brain-to-Text Decoding with Context-Aware Neural Representations and Large Language Models\n \n[e] Park et al., 2019, SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition\n\n[f] Willett et al., 2021, High-performance brain-to-text communication via handwriting"}, {"review_id": "XRwhc5z4cc", "reviewer": "Reviewer_YQY9", "summary": "This paper proposes three improvements over the baseline algorithm in the brain-to-speech benchmark:\n\n1. Time masking to prevent overfitting\n2. Replace GRU with a transformer to reduce the memory footprint\n3. Test-time adaptation (TTA) via entropy minimization\n\nWith these improvements, the word error rate (WER) reduces from 15.39 to 12.27, which is quite significant compared to previous attempts. The results of TTA also look promising as a novel strategy to mitigate the non-stationarity problem inherent to many BCI decoders.", "questions": "1. Minimize in-session training time is a key objective for a clinically viable BCI. The transformer takes 600 epochs to train, which is ~2.3hr, while the original GRU baseline requires ~0.5hr.  Can you show how the transformer+maksing, gru+masking, and the gru baseline’s WER vary with training time? This can help the researchers pick the best model for the amount of time they have in session for training models.\n2. The baseline GRU’s parameters were optimized for the overlapping inputs and may not be optimal for the time-masked non-overlapping inputs. To really support the claim that the transformer uses less memory than the GRU, both models’ hyperparameters need to be optimized with the time-masked non-overlapping inputs.\n3. Can you run decoding with the 5gram LM with cloud computing resources?\n4. In Sec 4.3, if performance is measured in WER, is the difference between CORT and No CORT still similar?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n\n- Many people have tried to replace the GRU baseline in the brain-to-text benchmark without success. This paper identifies the core problem (overfitting) and finds a simple solution (time masking).\n- The first to apply TTA with entropy minimization to the BCI problems, promising results.\n- The paper is clearly written and easy to follow. Experiments are well designed to support the claims.\n\nWeakness:\n\n- The major benefits of using a transformer are less GPU memory usage and faster recalibration. But they are very marginal: 5GB to 3GB, and 17ms to 13ms. Even though the per-epoch training is faster, the overall training actually takes longer due to more training epochs.\n- The GRU’s hyperparameters are not re-optimized for the time-masked non-overlapping inputs. So it’s not fully convincing that the transformer is more efficient than the GRU.\n- Though testing with a 5gram LM requires significantly more computing resources, all other brain-to-text benchmark methods report their results on the 5gram LM. The authors need to show their results with a 5gram LM to really convince the field that a masked transformer is a better approach.", "was_revised": true, "final_justification": "Thanks for the update. With the 8.01 WER when using 5-gram, and the stable performance of DietCORP, I believe this paper makes a very valuable contribution to brain-to-text BCI decoding algorithms.\n\nOne minor point is that calling the 5-gram+LLM rescoring offline isn't accurate. This decoding approach was used here for online decoding. The rescoring step is quick enough that it doesn't cause a significant delay. I suggest naming the metric in Tab 1. after the language models you use.", "rebuttal": "Thank you for the feedback. We appreciate that the Reviewer finds our paper easy to follow and finds the use of time masking to reduce overfitting interesting. We provide a one sentence response for each question and weakness, with additional details provided below. Tables are provided at the end of the text. \n\n**General Note**: We realized after submission that all the entries in Willett et al. 2025 [b] used a bidirectional GRU. A bidirectional GRU can only decode speech once the entire trial has finished, and roughly doubles memory usage, training times, and calibration times relative to the unidirectional GRU. All our models are either unidirectional GRUs or causal Transformers which only rely on present and past neural activity to generate outputs. \n\n**Training time takes longer when using time masking**: Our time-masked Transformer showed no drop in performance when trained for the same time as the entries which used GRUs.\n\n- *Additional Details:* We find that a $5$-layer time-masked Transformer trained for $250$ epochs (~1 hour of training time) exhibits slightly better validation WER than our original $7$-layer time-masked Transformer trained for either $600$ epochs or $250$ epochs. This may be because the smaller Transformer trained for less epochs has a lower validation CTC loss than the larger variant. We had previously only focused on validation phoneme error rate (PER), as done in Willett et al., 2023 [a]. This Transformer requires roughly the same amount of training time as the baseline bidirectional GRU and about half the training time as the Linderman Lab GRU, which was the best performing entry with only modifications to the GRU in Willett et al., 2025 [b] (Table 1).\n\n**Evaluate whether time-masked GRU can process non-overlapping inputs, and if this yields efficiency gains**: A time-masked GRU trained with non-overlapping inputs and a smaller hidden state size reduces parameter count by $35$ M relative to the original GRU architecture while maintaining competitive performance with our original time-masked GRU. \n\n- *Additional details:* Our original hypothesis was that the GRU would not perform optimally with non-overlapping inputs due to its lossy hidden state. Similar to Willett et al., 2023 [a], we found this was the case with the original GRU size. However, we found that performance with non-overlapping inputs improved when reducing the hidden size, suggesting that models with non-overlapping inputs generalize better when smaller. We will continue optimizing the GRU with non-overlapping inputs for efficiency gains, and update our manuscript to discuss how processing non-overlapping inputs may be a good heuristic to follow when designing lightweight architectures. \n\n**Results are not displayed with the 5-gram LM setup**: We have included results using the updated models with the 5-gram LM setup, which show that our models outperform strong baselines in a real-time decoding setting, and even match the performance of strong baselines which rely on bidirectional GRUs (Table 1). \n\n- *Additional details:* We have added results with the offline language modeling setup used by the entries in the competition (Willett et al., 2025 [b]). Specifically, the offline language modeling setup involved using a 5-gram LM, with a second pass using the 5-gram LM, and rescoring with OPT 6.7B. We refer to the setup where only the 3-gram LM is used with no second pass as the online LM setup. Results are displayed in Table 1. \n\n**Is the difference between CORT and No CORT similar with WER?**:  We have revised our test-time adaptation method to a computationally lightweight pseudo-label based approach, which was inspired by Fan et al., 2023 [c], and there is a substantial reduction in WER when applying our improved TTA method relative to no TTA (Table 2 and Table 3). \n\n- *Additional details:* While we did find a small reduction in WER when applying CORT relative to No CORT across days (average WER improvement across days shown in Figure 4, left), this improvement was small. We thus sought to examine if applying time masking and Transformers could provide efficiency gains to the pseudo-label based TTA method proposed by Fan et al., 2023 [c]. The method used by Fan et al., 2023, CORP, adapts a model at test-time by treating the language model (LM) refined model outputs as pseudo-labels, and then training the model with CTC loss using these pseudo-labels. Since the LM refined outputs are already being generated for real-world use, there is no additional cost to generate these pseudo-labels. However, CORP requires up to $200$ adaptation steps per test trial, setting a loss threshold hyperparameter, and including previous data to stabilize training. Here, we present \"DietCORP\", a lightweight version of CORP which only uses one adaptation step per trial and no previous data. Similar to CORT, DietCORP operates by generating several time masked augmentations of the same trial, and then it adapts the model to produce the same pseudo-label across all augmentations. We present results when applying DietCORP to a time-masked Transformer trained on $15$ days of data, and evaluated on the next $5$ competition days (Table 2), and a time-masked Transformer trained on $12$ days of data and evaluated on the next $8$ competition days (Table 3). Only the patch embedding module was adapted. DietCORP leads to very stable WER across days relative to when not applying it with minimal hyperparameter tuning.\n\n**Reductions in memory usage and calibration times are marginal**: With our updated TTA method, DietCORP, we reduce peak GPU memory usage by $81$% and calibration time by $82$% relative to the bidirectional GRU.\n\n- *Additional details:* DietCORP only requires $1.04$ GiB of peak allocated GPU memory usage and $29.3$ ms to calibrate per trial using the Transformer. When applying our TTA method to the unidirectional GRU, it requires $2.8$ GiB of peak allocated GPU memory usage and $76.9$ ms to calibrate per trial. These values further increase to $5.6$ GiB and $163.0$ ms when using the bidirectional GRU. We will also include comparisons when using CORP in the updated manuscript, where DietCORP applied to the Transformer exhibits even larger computational gains. We accept that the reductions in memory usage and training times are perhaps marginal for offline training, where models are often trained using large servers while the neuroprosthesis is not in use. However, we believe these efficiency gains are important in the context of test-time adaptation, where calibration is performed on-device in the background while the patient is using the neuroprostheses. \n\n\n***\n**Tables**\n\nTable 1: We show results with the $5$-layer time-masked Transformer trained for $250$ epochs (Transformer - Ours) as well as comparisons with the baseline provided by Willett et al., 2023 [a] and the Linderman Lab GRU discussed in Willett et al., 2025 [b]. We run the baseline and Linderman Lab GRU in unidirectional mode, and show metrics in bidirectional mode in parenthesis when available. \n\n\n| Metric                    | Transformer - Ours | Baseline GRU       | Linderman Lab GRU   |\n|---------------------------|-------------------:|-------------------:|--------------------:|\n| Online LM Setup WER %    | 12.03 ± 0.24        | 15.25 ± 0.16        | 15.03 ± 0.19         |\n| Offline LM Setup WER %  | 8.01 ± 0.23         | 11.12 ± 0.13 (9.76) | 10.77 ± 0.13 (8.0)   |\n| Parameters (M).        | 9.4                 | 56.7 (135.4)        | 58.8 (143.8)         |\n| Training time (hours)   | ~1                  | ~0.5  ( ~1)         | ~1  ( ~2)            |\n\n\nTable 2: DietCORP is our computationally lightweight version of CORP, a test-time adaptation method proposed by Fan et al., 2023 [c]. Values are word error rate (mean ± SEM across $N=4$ seeds) computed on $5$ held-out competition days using a model trained on the previous $15$ days. \n\n| Days since last training day | No DietCORP | DietCORP  |\n|--------:|----------------:|----------------:|\n| 7       |  22.74 ± 0.26        | 21.14 ± 0.32        |\n| 13     |  25.48 ± 0.28        | 21.23 ± 0.21       |\n| 19     |  25.43 ± 0.33       | 21.40 ±  0.22       |\n| 28     |  30.13 ± 0.33     | 21.93 ±  0.65       |\n| 30     |  32.59 ± 0.52     | 22.26 ±  0.60        |\n\n\nTable 3: Values are word error rate (mean ± SEM across $N=4$ seeds) computed on $8$ held-out competition days using a model trained on the previous $12$ days. \n\n| Days since last training day | No DietCORP | DietCORP  |\n|--------:|----------------:|----------------:|\n| 5       | 28.87 ± 0.47        | 26.97 ± 0.28          |\n| 12     | 34.50 ± 0.44       | 26.38 ± 0.30         |\n| 21     | 44.79 ± 0.34       | 27.57 ±  0.13        |\n| 28       | 53.29 ± 0.25        | 30.46 ± 0.41         |\n| 34       | 59.98 ± 0.31        | 29.70 ± 0.20         |\n| 40       | 52.71 ±  0.52        | 29.08 ± 0.38         |\n| 49       | 62.96 ± 0.40         | 31.35 ± 0.76         |\n| 51       | 66.47 ± 0.58         | 31.91 ± 1.25        |\n\n\n[a] Willett et al., 2023, A high-performance speech neuroprosthesis \n\n[b] Willett et. al, 2025, Brain-to-Text Benchmark '24: Lessons Learned\n\n[c] Fan et al., 2023, Plug-and-Play Stability for Intracortical Brain-Computer Interfaces: A One-Year Demonstration of Seamless Brain-to-Text Communication"}]

Author rebuttal:

Meta-review: This paper proposes method that makes significant contributions for speech neuroprostheses. The main idea revolves around: (i) time masking of neural activity to mitigate overfitting and (ii) replacing GRUs with lightweight Transformers. Together with a lightweight test-time adaptation method (DietCORP), the approach reduces WER while lowering memory and calibration costs. The work is among the first to make Transformers competitive on the Brain-to-Text benchmark in a streaming setting.

Reviewers consistently agree that the integration is carefully engineered and domain-aware, offering practical improvements in efficiency and robustness. The work demonstrates stable performance across large temporal gaps, which is highly relevant for real-world BCI deployment.

It is also noted that masking and Transformers are not novel techniques, and the paper sometimes overstates conceptual contributions. Comparisons with alternative augmentations (e.g., SpecAugment) and TTA methods are limited, and study into hyperparameter sensitivity could be improved. Much of the performance gain was established during rebuttal however the architectural changes were minimal to be considered a major change. I request the authors to make changes to the manuscript to reflect the changes discussed during the rebuttal period. 

Overall, while the novelty is modest, the paper provides meaningful engineering advances and demonstrates competitive, practical performance in an important application domain. Therefore, I recommend accept as the contributions are valuable for the BCI community.
Decision: Accept (poster)