SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We next tested if online decoder performance follows the same ordering as offline accuracy. Figure 2e/f shows online performance on two exemplar days where all five decoders were tested in a 1-DoF and 2-DoF task with Monkey N, where performance followed the same ordering as offline accuracy for both comparisons (except for the TFM outperforming the FNN on the 1-DoF task). Interestingly, the FNN had only slightly lower online 2-DoF bitrate compared to the RNN decoders despite significantly lower offline position and velocity correlations. This suggests that offline performance on the same dataset may give a general idea about online performance ordering but does not indicate the specific performance differences during closed-loop control.
High Channel-Count, Simulated Decoder Comparison With the recent developments of higher channel count recording systems [29, 30], BMIs may soon have access to more input channels. To determine if recurrent architectures may be more accurate than feedforward architectures at higher channel counts, we trained decoders on simulated datasets with a varied number of artificial neural channels. As seen in Figure 2c, for 10-100 channels, the LSTM and GRU decoders have significantly higher offline correlation compared to FNN, TFM, and KF decoders (average LSTM correlation 0.1 higher than TFM). However, at higher channel counts the advantage of the LSTM and GRU shrinks, such that the difference in correlation at 400 channels is only 0.01 between LSTM and TFM. While actual neural data may differ from the simulated datasets used here, these results suggest that the choice of a specific nonlinear decoder may be less important at high channel counts.
3.2 Able-bodied performance by memorizing movements
As opposed to the fully randomized target postures in the previous section, we next investigated RNN decoder performance on more simple, stereotyped tasks with a limited set of targets. RNNs have the ability to memorize and reproduce learned patterns [14]; here, we aimed to let the RNN memorize movement patterns for a discrete set of targets, potentially allowing for more accurate decoding despite the potential for reduced generalization on a wider range of tasks. To test the capacity for RNNs to memorize targets, we trained and tested LSTM decoders on discrete target datasets, ranging from 2 targets (1D) up to 23 targets (2D). We note that the target order was randomized, which required the decoder to rely on neural input. Separate decoders were trained for each number of targets. We found that LSTM decoders could memorize the specific positions and movements between targets and that offline error increased as the number of targets increased (Figure 3a). Intuitively, this makes sense because as more targets are added, there are fewer example movements per target when keeping the number of trials constant. In additional simulations, we found that as more targets are added, more inputs or training data are needed to maintain accuracy (Supplemental Figure 3). While offline error was highest for random targets, this maximum value corresponds to relatively accurate predictions (as shown in the previous section).
We found a similar trend when testing decoders online with simple tasks (Figure 3b). For tasks with 2 to 4 targets, monkey N using an LSTM decoder achieved  $2\%$  higher bitrates than able-bodied manipulandum control on average (see Supplemental Video 2), where the performance improvement was due to a reduced orbiting time (in the 2-target case on average, brain control had a  $20~\mathrm{ms}$  slower move time but a  $96~\mathrm{ms}$  faster orbiting time). Move time is the time to reach the target and orbit time is time for extraneous movements after reaching the target (see Methods). As the number of targets increased, brain-control move times stayed approximately the same while orbiting time increased (Figure 3c). This suggests that with fewer targets, the RNN learns stronger internal dynamics to minimize extraneous movement. Overall, as more targets are added, thus decreasing offline performance, online performance drops down to approximately random-target performance. Therefore, by changing the training dataset the RNN decoder can smoothly transition between memorization with strong dynamics (similar to a classifier) and generalization with weaker dynamics.
Figure 3: LSTM decoders have increased performance on simple tasks. (a) Offline MSE (a.u.) for LSTMs trained on tasks with varied numbers of targets and 1-2 DoFs, where all tasks were performed on the same day. “2D random small” refers to random targets with  $25\%$  smaller size, requiring finer control. (b) Online performance for varied numbers of targets and DoFs across multiple sessions, with performance measured relative to able-bodied control (calculated as online-bitrate / hand-bitrate). Lines indicate tests performed within the same day. (c) Average move and orbiting times for hand control (gray bars) and brain control (blue bars) for 1-DoF tasks on an exemplar day. Orbiting times tend to increase with task complexity. Note that the 500 ms hold time is not depicted.
3.3 Recovering performance through memorization
In some cases, information on a particular DoF is limited (for example if a BMI has too few channels tuned to that DoF or movement) making the overall BMI unusable. To recover functional use, we hypothesized that the poor DoF could rely on memorization of a discrete movement set (for greater accuracy) while the other good DoF(s) maintain fully continuous movements. For example, Figure 4 shows the first three principal components of the hidden state of a 2-DoF GRU trained on two targets for index finger and random targets for MRS fingers (simulated data, see Supplemental Methods). In this setting, one DoF acts like a movement classifier jumping between line attractors, while the other DoF is continuous with positions continuously represented along each line attractor.
We tested this strategy with Monkey N, who, over a period of 6 months, had a reduction in control of the index finger (likely due to a reduction of neural channels tuned to index finger). When using an LSTM trained on a 2 finger group (2-DoF) random task, index finger had poor online control, resulting in a success rate of  $66\%$  (average bitrate 1.14, 112 trials). To recover performance, on the same day, we trained an LSTM on a modified task with only three targets for index finger (extend, center, flex) while maintaining random targets for MRS fingers. By reducing the number of distinct index finger movements, the number of training examples per movement is increased, allowing the RNN to learn more accurate movement dynamics. The LSTM trained on the modified task had substantially higher online performance (success rate of  $98\%$ , average bitrate of 4.18, 262 trials) and the finger groups were visibly more independent (see Supplemental Video 3). Thus, for BMI or prosthetics users with poor input quality or limited data, functional online performance may be recovered by modifying the decoder training dataset without necessarily tuning hyperparameters or parameters of the decoder itself.
3.4 Memorized RNNs partially generalize to continuous tasks
For RNNs trained to memorize simple postures, it is unclear how well these decoders can generalize to other tasks. One might expect that, when trained to output a discrete set of positions, the RNN may learn strong fixed points (or other dynamics) that enable the RNN to accurately output discrete positions, but may perform poorly when tasked with holding intermediate postures (random targets). To test this assumption and the ability of memorized RNNs to generalize, we trained LSTMs on tasks with a limited number of targets and tested the decoders on online random target tasks. We tested the generalization ability of positional (50% position, 50% integrated velocity) and velocity (1% position, 99% integrated velocity) variants of the same decoder (Supplemental Figure 4).
LSTM decoders trained on 1-DoF 2- and 3-target tasks had  $100\%$  and  $98\%$  online success rate (SR) respectively when tested on the same simple tasks (positional variant). When tested on the random target task, as expected, the positional decoders had difficulty stopping on targets that did not overlap with the learned targets, and failed to move outside of the learned flexion-extension range. The
Figure 4: Hidden states of a memorized RNN. (a) A GRU RNN was trained on a modified task with random targets for MRS fingers but only 2 targets for index finger (simulated data). (b, c) Plots depict the first three principal components of the RNN hidden state, with traces indicating the path over time. (b) and (c) plot the same data, just with different coloring. When the path is colored by the predicted index position (b), it is clear each line attractor corresponds to an index target. As in (c), the position along each line attractor corresponds to the predicted MRS position. Thus, the RNN can learn different dynamical structure for each DoF to act more like a classifier when beneficial.
2-target positional decoder was non-usable  $(\mathrm{SR} < 50\%)$ , while the 3-target positional decoder was more functional (SR  $87\%$ , bitrate 1.05). Interestingly, the velocity decoder variants were better able to generalize, where the 2-target velocity decoder had a  $98\%$  SR (bitrate 1.33) and the 3-target velocity decoder had an  $80\%$  SR (bitrate 0.93, lower performance likely due to low monkey motivation). While functional, the velocity variants had more difficulty making small corrective movements and had lower bitrates compared to random-trained decoders (bitrate 2.19). These results suggest that memorized RNNs can generalize to some degree, where training on more targets leads to better positional generalization.
A key component of BMIs that restore motor function is an accurate decoding algorithm that performs well in online settings. Unfortunately, for continuous movement decoding, offline accuracy does not incorporate closed-loop dynamics, meaning offline metrics may fail to predict online performance [9] and offline cross validation does not fully reveal decoder overfitting [10]. However, here, using a multi-DoF finger BMI, we found that offline accuracy generally predicted the relative online decoder performance ordering. We found that recurrent neural networks, namely LSTMs, outperform other neural network architectures in both offline and online settings. This high performance of RNNs follows previous work showing that decoders that learn neural or kinematic state dynamics (including RNNs) can more accurately predict kinematics from noisy neural data [31, 27, 6]. All the networks tested here were relatively small (5 layers or fewer) and could be efficiently run in real time (typically  $< 2$  ms per prediction), suggesting that very deep or complex architectures may not be necessary for high performance. Additionally, we did not perform intention-based recalibration (ReFIT; as described in [24, 13]) due to experimental trial count constraints. This could potentially benefit some decoders more than others. Further work may also investigate the performance trade-offs of decoders trained to de-noise neural data [7, 8] versus decoders directly trained to predict kinematics.
While the neural network decoders tested here were all highly usable with minimal apparent overfitting, Deo et al. [10] found that RNNs tended to significantly overfit to offline data when tested in humans. In humans with paralysis, decoding algorithms are trained from offline data by instructing participants to attempt movements in sync with an artificially moving effector. With able-bodied users (like the non-human primates used here), however, the direct relationship between neural activity and true movement can be used for training. These able-bodied movements more broadly sample the range of positions/velocities and neural activity needed to perform a task and often have errors like overshooting. When trained on only two able-bodied targets, enough variation was present to allow for generalization to other movements. These results suggest regularization of neural networks is necessary for high performing BMIs, whether inherent to the training data or by applying artificial data modifications.
In additional testing we found that using more time history in the FNN and TFM decoders (up to 20 bins/640 ms) resulted in poor online control (Supplemental Figure 5). This is likely due to overfitting to training data that lacks many of the corrective movements present in online control, and demonstrates how offline analyses can fail to predict the performance in the closed loop setting. Interestingly, the RNNs were trained with longer history but did not display this overfitting. Future work may investigate methods of using longer training sequences while limiting overfitting and lag, for example by discouraging attention to early bins.
Decoder memorization of training data is commonly associated with overfitting, suggesting poor closed-loop control. Here, we found that RNN decoders allowed to memorize a few movement patterns were highly controllable online and matched able-bodied performance for tasks with 2-4 target postures. With a reduced target set the RNN can learn stronger recurrent dynamics for higher accuracy, acting somewhat like a pseudo-classifier between movements. As task complexity (the number of targets and distinct movements) increases, offline and online decoder performance degrades gracefully down to fully continuous performance, partly due to the reduced number of training examples per movement. This is similar to neural word decoding, where word error rates increase with larger vocabularies [32]. Unlike algorithms that manually combine a classifier and continuous decoder [33, 34], RNNs are fully continuous yet can internally act as a classifier and generator for movements that lack input information, while maintaining high performance continuous output for other movements. As BMIs extend to more DoFs, our results suggest that more data or neural channels will be needed to maintain accuracy, and the degree of memorization can be controlled for each DoF independently. Future work may investigate the relative dependence on inputs vs autonomous dynamics of RNNs, and their effect on online performance.
There are several limitations to the present study of closed-loop BMI performance. First, we did not explore every possible decoder architecture. More optimal architectures likely exist, but here we found several key factors of each class (for example, the need for limited time history of feedforward models, and the ability of RNNs to strongly learn few movements). Future work may explore models that combine recurrence with feedforward attention layers to take advantage of the benefits of each (for example, [35]). Also, only one monkey was used for the online analyses; additional users may have different control strategies or decoder errors.
Future BMI decoders could be designed to be more explicitly tunable to allow the user to adjust the balance between memorization (with more reliance on internal dynamics) and generalization to a wider range of movements. A decoder could act more like a classifier when producing stereotyped movements such as sign language or functional grasps, but act as continuous controller when producing arbitrary movement. Users at the onset of BMI use could use a simple task decoder that works well with minimal training data (reducing cognitive fatigue and maintaining user motivation), then transition to higher complexity as more data is acquired. These strategies could also be employed for myoelectric and other prosthetic controllers which must function using noisy signals with large uncertainty in user intentions. BMIs that can produce accurate kinematic predictions across the range of few neural channels to thousands could speed the clinical translation of BMIs with current and future hardware.
We thank Eric Kennedy for animal and experimental support. We thank the University of Michigan Unit for Laboratory Animal Medicine for expert surgical and veterinary care. JTC was supported by NSF GRFP under grant 1841052. DMW was supported by Dan and Betty Kahn Foundation (Grant 2029755) and University of Michigan Robotics Institute. HT, MJM and PGP were supported by NSF grant 1926576. LHC was supported by Agencia Nacional de Investigación y Desarrollo (ANID) of Chile. MSW was supported by NIH grant T32NS007222. CAC was supported by Dan and Betty Kahn Foundation (Grant 2029755) and NSF grant 1926576.
[33] Jonathan C. Kao, Stephen I. Ryu, and Krishna V. Shenoy. Leveraging neural dynamics to extend functional lifetime of brain-machine interfaces. Scientific Reports, 7:1-16, 2017.
[34] Brian M. Dekleva, Jeffrey M. Weiss, Michael L. Boninger, and Jennifer L. Collinger. Generalizable cursor click decoding using grasp-related neural transients. Journal of Neural Engineering, 18, 2021.
[35] Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In International conference on machine learning, pages 4651-4664. PMLR, 2021.