Balancing memorization and generalization in RNNs for high performance brain-machine Interfaces
Authors: Joseph T Costello
Hisham Temmar
Luis H Cubillos
Matthew J Mender
Dylan M Wallace
Matthew S Willsey
Parag G Patil
Cynthia Chestek
Venue: NeurIPS 2023 spotlight (2023)
Keywords: brain computer interface
brain machine interface
neural decoding
prosthetic control
recurrent neural network
RNN
transformer
real time
closed-loop
user interface
TL;DR: Using an intracortical brain-machine interface, we show how RNNs outperform other neural networks for closed-loop neural decoding, and can act both like a classifier and a continuous decoder for decoding movement.

Abstract: Brain-machine interfaces (BMIs) can restore motor function to people with paralysis but are currently limited by the accuracy of real-time decoding algorithms. Recurrent neural networks (RNNs) using modern training techniques have shown promise in accurately predicting movements from neural signals but have yet to be rigorously evaluated against other decoding algorithms in a closed-loop setting. Here we compared RNNs to other neural network architectures in real-time, continuous decoding of finger movements using intracortical signals from nonhuman primates. Across one and two finger online tasks, LSTMs (a type of RNN) outperformed convolutional and transformer-based neural networks, averaging 18% higher throughput than the convolution network. On simplified tasks with a reduced movement set, RNN decoders were allowed to memorize movement patterns and matched able-bodied control. Performance gradually dropped as the number of distinct movements increased but did not go below fully continuous decoder performance. Finally, in a two-finger task where one degree-of-freedom had poor input signals, we recovered functional control using RNNs trained to act both like a movement classifier and continuous decoder. Our results suggest that RNNs can enable functional real-time BMI control by learning and generating accurate movement patterns.

Balancing Memorization and Generalization in RNNs for High Performance Brain-Machine Interfaces
Joseph T. Costello, Hisham Temmar, Luis H. Cubillos, Matthew J. Mender, Dylan M. Wallace, Matthew S. Willsey, Parag G. Patil, Cynthia A. Chestek Departments of Electrical and Computer Engineering, Biomedical Engineering, Robotics, and Neurosurgery, University of Michigan, Ann Arbor, MI, USA
Abstract
Brain-machine interfaces (BMIs) can restore motor function to people with paral- ysis but are currently limited by the accuracy of real-time decoding algorithms. Recurrent neural networks (RNNs) using modern training techniques have shown promise in accurately predicting movements from neural signals but have yet to be rigorously evaluated against other decoding algorithms in closed-loop set- ting. Here we compared RNNs to other neural network architectures in real-time, continuous decoding of finger movements using intracortical signals from nonhu- man primates. Across one and two finger online tasks, LSTMs (a type of RNN) outperformed convolutional and transformer-basee neural networks, averaging 18% higher throughput than the convolution network. On simplified tasks with a reduced movement set, RNN decoders were allowed to memorize movement patterns and matched able-bodied control. Performance gradually dropped as the number of distinct movements increased but did not go below fully continuous decoder performance. Finally, in two-finger task where one sfgree-of-freedone had poor input signals, we recovered functional control using RNNs trained act both like movement classifier and continuous decoder. Our results suggest that RNNs can enable functional real-time BMI control by learning and generating accurate movement patterns.
1 Introduction
Brain-machine interfaces (BMIs) have the potential to restore motor function to people with paralysis. While intracortical motor BMIs have successfully enabled paralyzed human patients to control computer cursors []] robotic arms (2), or even move their own muscles through functional electrical stimulation [3] the decoding algorithm that predicts intended movement from neural signals still lacks accurate and naturalistic movement outputs. Recently, non-linear decoding algorithms using artificial neural networks have demonstrated higher offline kinematic prediction accuracy compared to linear approaches, which could enable more functional BMIs. Glaser et al. 2020 found that recurrent neural networks (RNNs), specifically the long short-term memory (LSTM, 4 and gated-recurrent unit (GRU, (5), outperformed other decoding architectures for predicting movement in several brain regions 6 Other neural networks have improved decoding accuracy by de-noising neural signals using RNNs (LFADS, (7)) or transformers 8 However, offline accuracy does not necessarily predict real-time closed-loop (online) performance 9 10,, where errors are cumulative and the user can adjust neural activity based on feedback. With goals of improving function of real-time BMIs, it is unclear if the high offline accuracy RNNs translates the online setting.
In online prediction settings, BMIs using RNNs for continuous movement prediction have demon- strated limited success but have not been evaluated against other neural network architectures. Sussillo et al. 2016 showed that an online RNN was more robust to electrode perturbations and slightly outperformed linear Kalman filter in a reaching task (T] In a closed-loop simulation, Hosman
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

al. 2019 found that an LSTM decoder could outperform a Kalman filter in cursor-control task [12] Recently, Deo et al. 2023 found that RNNs tested with a human subject tended to significantly overfit to offline data and required novel dataset regularization for functional online use [10]. Alternatively,
our group found that an ReFIT convolutional feedforward network outperformed a ReFIT Kalman
filter in an online two finger group task by accurately predicting a large dynamic range of velocities [3]. Thus, there is a need to evaluate the online performance of RNNs against other high performing neural networks where all decoders have the possibility of overfitting to training data.
In addition high prediction accuracy, RNNs have the capacity to memorize and generate realistic sequences with minimal input [14 13 [6]. In the BMI context, an RNN decoder could memorize discrete postures and stereotyped movements between postures, and could help generate movement for degrees-of of-frredom (DoFs) where little information is contained in the input (for example, if few cortical channels are tuned to one movement direction). RNNs have previously been used to generate and other work has shown that RNNs can learn 16 However, it unclear if a memorized RNN controllable in closed-loop setting where overfit decoders may easily fail [0]
smooth, multi-DoF limb movements 17
14
18
hundreds of separate simple tasks 19
Here, we evaluate the online closed-loop performance of RNNs against other neural network archi- tectures in non-human primates performing a multi-DoF finger task. We find that offline accuracy generally predicts the relative ordering of online decoder performance, with LSTMs outperforming other architectures. Then, we show how reducing task complexity and allowing memorization of movements can enable the BMI to reach able-bodied performance, with a gradual transition between discrete and fully continuous movement outputs. We provide an example of using selective movement memorization to recover performance despite having poor neural inputs for a particular DoF. Finally, we find that RNNs trained to memorize a small set of movements still rely on user inputs and have some limited capacity to generalize when controlling velocity rather than position.
2 Methods
2.1 Microelectrode Array Implants
We implanted two male rhesus macaques (Monkeys N and W) with Utah microelectrode arrays (Blackrock Neurotech) in the motor cortex using the arcuate suleus as an anatomic landmark for hand area, described previously (20) 21]. In each animal, a subset of the 96 channels in M1, with threshold crossings morphologically consistent with action potentials, were used for offline recordings and closed-loop BMI control. Surgical procedures were performed in compliance with NIH guidelines as well as our institution's 's Institutional Animal Care& & Use Committee and Unit for Laboratory Animal Medicine.
2.2 Signal Processing and Feature Extraction
We recorded 96-channel Utah array data using a Cerebus neural signal processor (Blackrock Neu- rotech). The Cerebus sampled data at 30 kHz, applied 300-1000 Hz bandpass filter and downsampled to 2 kHz before streaming the data to computer running xPC Target version 2012b (Mathworks), which calculated spiking band power (SBP) by taking the magnitude of the signal and summing into 32 ms bins. We used SBP since it is highly correlated with neural firing rate and a high-performance BMI input feature (22]. During online experiments with neural network decoders, the binned SBP of each channel was normalized to have zero mean and unit variance, based on the mean/variance of initial training trials.
2.3 Behavioral Task for Finger Movement Decoding
We trained monkeys N and W to acquire virtual targets with virtual fingers shown on computer screen in front of the animal (Figure la). In hand control ("offline" trials), monkeys moved their fingers within a manipulandum that measured the angles of the index and middle-ring sindlee-ing-mmall (MRS) finger groups using bend sensors, and controlled the virtual fingers (Figure lb). During brain control (also known as closed-loop. "online" trials), monkeys controlled the virtual fingers using neural signals and decoder. The finger task required placing the virtual index finger and MRS fingers on the respective targets and holding for 750 ms during offline trials or 500 ms during online testing. The target size was 15% of the active range of motion. The total number of possible target locations
2

---

Dicoddr
Predicted Finger
Spiking
Addivyy
Finger Poslions
Postions
MRS Floxon (%)
Vituall Hend
Index Flexion (%)
Truc
Indax
Position
MRS Position
eriend
SBP
h2
pv
Velcoit
Desoder
MRS Volccity
Time
Figure 1: BMI for decoding finger movement. (a) Experimental setup. During hand-conter trials the monkey used a finger manipulandum to control a virtual hand and acquire targets, with visual feedback from the screen. Spiking-bant power (SBP) calculated in real-time from 96-channel Utah microelectrode arrays. During brain control, the decoded finger positions controlled the virtual hand. (b) Example movements for the 2-DoF task involving flexion and extension of the index finger and MRS fingers. In a random target task, the target randomly chosen between extension and flexion for each DoF. (c) SBP averaged into 32 ms bins and fed into RNN (GRU LSTM) decoder which predicted position and velocity for one or two finger groups. Other decoders not depicted here
also tested. (d) Example true and predicted positions and velocities from LSTM decoder for Monkey N performing 2-DoF random target task. The LSTM is able to predict both slow and fast velocities.
varied based on the task, ranging from 2 targets up fully random (any position could be chosen), with 1-DoF (just index finger) or 2-DoF (index and MRS fingers). Target locations were randomly chosen the start of each trial. We collected 500 offline trials for decoder training (although only 300 were collected for offline memorization, Figure 3a). We note that Monkey N typically continued move his hand even during brain control mode. In this work we only used offline historical data from Monkey W who did not perform online control.
2.4 Simulated Datasets
For some analyses we created simulated offline datasets of a virtual user performing the same target acquisition task. The goal of these simulations was test the relative impact of amount of training data, number of DoFs, and number of inputs on decoder performance, rather than measuring absolute performance. The simulated user moved with velocity proportional the distance to the target along each DoF. Artificial neural channels were generated such that each channel had a random relationship with the position, velocity, and acceleration at each timestep, suggested in 23 Further details and specifics can be found in Supplementary Methods.
2.5 Performance Metrics
Offline performance was measured by the Pearson correlation and mean-squared error (MSE) between predicted and true finger position and velocity. Offline velocities and positions were normalized to have unit standard deviation and zero mean (normalized using training data). We also calculated the mean-absolute error (MAE) between velocity predictions and the true velocity. To do true velocities were binned into 21 bins between to standard deviations, and the mean-absolute error was calculated for the predictions of each bin, where data across days were grouped together.

---

Online performance was measured using two sets of metrics: time and information throughput. Time metrics included the trial-time (the time for trial with hold time subtracted), move-time (the time until all fingers touch the target for the first time), and orbiting time (trial-time minus move-time, representing extraneous movements where fingers left the target). Information throughput, measured in bits per second (bitrate), has previously been used to measure finger task performance [13] and was calculated using Fitt's law which accounts for both task difficulty and completion time:
Ex log2
(1+
R
throughput
(I)
baog
where Dj is the distance of the kth virtual finger the center of the kith target, R is the target radius, and hacy is time to reach the target. Better decoders (fast and precise) have a higher throughput. Before calculating throughput, we removed initial trials from the performance calculation account for the decoder learning period before steady state performance, which took approximately 1-2 minutes. When more than 100 trials were performed we removed the first 50 trials, and when fewer than 100 trials were performed (lypically when the performance was poor and the monkey was unable to complete many trials) we removed the first 20 trials.
2.6 Decoders
We tested five decoder architectures, chosen for their high offline or online decoding performance in other studies. Decoders took in binned SBP features (neural features) from up to 96 channels with 1-5 bins of time history for each channel and were trained to predict the position and velocity of 1 or 2 finger groups (Figure lc). The neural network decoders were implemented and trained using PyTorch 1.12.1. Each decoder is detailed below; see Supplemental Figure 1 for more details on each network architecture. The total number of neural network decoder parameters varied from 479k for the LSTM to 923k for the TFM (see supplemental results).
Recurrent Neural Network (RNN) We tested two variants of RNNs: the LSTM 4 and GRU [5]. which have previously shown high performance in offline neural decoding 6 Decoders were implemented using the prrchnnn..TTI and torch. nn. GRU classes. As shown in Figure 1c, the RNN decoder takes in the current time bin of SBP features and the previous hidden state, updates the hidden state, and predicts position and velocities sequence length of 20 and zero- iero-iniiialization for the hidden state were used for each sample. The final output of each sample sequence was used compute the loss. During online decoding, the hidden state was stored in memory, such that the inputs at each timestep were the current neural bin and the previous hidden state.
linear function of the hidden state. During training.
Convolutional Feedforward Neural Network (FNN) We tested the FNN described in Willsey al. 2022 1 which previously outperformed ReFIT Kalman Filter [44 21 in online testing. The network uses an initial convolutional layer over time (for each channel) followed by several feedforward layers that use batch normalization, dropout, and ReLU. The network used bins of time history per channel (160 ms). Implementation details are described in [13] and Supplemental Figure and hyperparameters were optimized as described below. Notably, we did not perform ReFIT recalibration as was done in 13
Transformer (TFM) Transformers 25 have recently become popular for sequence processing
tasks and have performed well in offline neural decoding (Ye 2021). The transformer tested here used a cosine positional encoding layer (25] followed by multi-head attention and feedforward layers, and final linear layer to output kinematic predictions (see Supplemental Figure 1). The transformer layers were implemented using the torch.nn. Dorch.n..Trnsformerfii. TransformerFnnodder..yr Layer.......rr.......... The transformer used five bins of time history per channel (160 ms). We note that unlike the transformer in Ye et al. 2021 [ which was trained to predict smoothed firing rates, the transformer here was directly trained to predict kinematics.
Kalman Filter (KF) The kinematic Kalman Filter (KF) is a linear decoder widely used in BMI that optimally combines a state transition model (using the current position and velocity as a state) with an observation model (that relates the kinematics to neural activity) to estimate the current position and velocity. Like the other decoders, the KF takes in the current bin of neural features and the previous

---

state, and then outputs predicted finger velocities. The KF was implemented and trained as described in 120 26 using the NumPy library, but here we did not use ReFIT recalibration.
2.7 Neural Network Decoder Training
For offline decoders, datasets were split into 70% for training, 10% validation, and 20% testing. For online decoders, datasets were split into 80% training (400 trials) and 20% validation (100 trials) since a test set was unnecessary. Neural features and finger positions/veloocties were normalized based on the training set to have zero mean and unit variance. We first performed hyperparameter optimization for each decoder using two offline datasets for each monkey individually. Bayesian optimization was performed using the Optuna library, and decoder performance was generally robust to the specific choice of hyperparameters. See Supplemental Methods for details and Supplemental Table 1 for parameter values used. Decoders were trained using minibatch gradient descent with the Py Adam optimizer. The loss function was the mean-squared error (MSE) loss with an additional term to penalize finger co-dependence (see Supplemental Methods; the additional term was only used for online decoders). The learning rate was held constant until validation loss plateaued, then halved and held constant until validation loss platcaued a second time (using the ReduceLROon ROnPlateau scheduler). After training was complete, a linear regression was calculated to transform the normalized outputs back the initial position and velocity scale. Training took less than 5 minutes for each network.
2.8 Online Decoder Testing
During online experiments, decoders were run in Python 3.7 using dedicated linux computer with an RTX 2070 Super GPU (NVIDIA). After receiving binned neural features from the xPC, the decoder calculated position and velocity predictions, and final position for each finger was sent back the xPC to update the virtual hand (see [13] for more details). For the RNN decoders, we found that combining velocity and position predictions to generate the final displayed positions (98%, integrated velocity, 2% position) was necessary to prevent the virtual fingers from becoming biased (similar to that of [27 II), and we used this for all neural network decoders. For tasks with less than targets we increased the positional contribution up to 50%. During online comparisons, decoders were alternated in A-B-A A-B-A-B format, with performance averaged across sets. For the 2D decoder comparison with all 5 decoders, each decoder was only tested once (A-B) due to time. An online decoder test was stopped if the monkey was unable to acquire targets for more than 30 seconds or the monkey stopped attempting to acquire targets.
2.9 Training Optimizations
To optimize online performance, in early exploration we tested several machine learning techniques for improving accuracy and generalization (detailed in Supplemental Methods). When the initial online trial success rate was 90% or less, we found the following techniques improved individuated finger control and success rate (SR): modifying the loss function to encourage finger independence (11% increase in SR. 12% improvement in trial time), training on more single-finger movements (38% increase in SR, 27% improvement in trial time), and training on trials with added positional perturbations (1 1% increase in SR, 12% improvement in trial time). Adding small amount of noise to the neural data during training (as suggested in (8 improved average trial times by 6%. Finally, we found that training on more data improved offline accuracy (up through 2000 trials), but with only a small improvement after 500 trials. For online results detailed below, decoders used the modified loss function, additional single-finger movements, and added noise.
3 Results
3.1
High Performance finger BMI with RNNs
Offline Decoder Comparison Previous work has shown that RNN decoders, specifically LSTMs and GRUs, can outperform other feedforward neural networks for predicting offline reaching move- ments from intracortical signals (6. Here, we began by establishing a baseline for how well RNN decoders can decode dexterous finger movements, compared to other neural network architectures. One adult male rhesus macaque, Monkey N, was implanted with Utah arrays in the hand area of the
5

---

Offinn 2-DdF
Offiine 2-0dF
Simdited, Offine 2.0xF
1,00-
Megn Absolute Error
0--
M52
Cornistionn
0.P5
PMN
bectlriward networt
veleelyy pooiion
0.90
1TM
0.5
-aa (iree))
1.7TM
GRU
TFM
-3
-2
2
0:20
100
500
402
Truo Velocity
Number of Channels
Online 2-DeF
Online 1.DoF
10-
Online 2.0
Throuphput (bila)))
Exsnssio (S)
Throughpas
9.
Hand LSTM
PNN
1PM
Figure 2: Offiine and online decoder performance comparison. (a) Offline model performance measured by MSE (a.u) between predicted and actual movements. Values represent the average of ten models trained on ten separate days of the 2-DoF random task. (b) Mean absolute error of velocity predictions where the x-axis is the true velocity and the y-axis is the error in predicted velocity (lower is better). Units are normalized velocity and data is averaged across ten days. (c) Offline correlation between true and predicted movements for simulated datasets with varied channel count. As channel count increases, the difference in accuracy across decoders becomes smaller. (d) A representative online decode trace of Monkey N using an LSTM for a 2-DoF random target task. The fingers quickly move to the target and can hold with minimal overshoot. Boxes represent the target position for each finger and lines represent the finger positions. (e, f) Online model performance on exemplar days where all decoders were tested in the same day for 1-DoF (e) and 2-Dof (f) random tasks. Performance is measured by bitrate (higher is better). "Hand" is the performance during physical hand control (able-bodied performance). Across all figures, error bars denote one standard error of the mean (SEM), and data for a, come from Monkey N.
primary motor cortex, and simultancous neural activity and finger movements were recorded. For a continuous, random target task we compared the following decoders across ten days: LSTM, GRU, FNN, TFM, and KF. An example offline prediction on a representative day using an LSTM is shown in Figure 1d, where the predicted positions and velocities closely match the true finger movements, notably reaching both fast and slow velocities. To evaluate decoder accuracy, we calculated the MSE and correlation between predicted and true finger velocities and positions. As in Figure 2a, for Monkey N. LSTM decoders had the lowest positional and velocity error across all decoders and also the highest correlations of 0.86 and 0.77 for position and velocity respectively. The LSTM had a significant improvement in position and velocity MSE over all decoders (p<0.001 for all comparisons, one-sided paired t-test). All neural network decoders had significantly lower position and velocity MSE than the KF (p<le-5 for all comparisons, one-sided paired t-test). The same relative ordering of decoder was found when the analysis was performed on historical data from a second monkey (Supplemental Figure 2), albeit with slightly higher errors overall.
To further reveal the difference in decoder accuracy with Monkey N, we calculated the mean-absolus error (MAE) for each ground-truth velocity (Figure 2b). All neural network decoders had similar low error at slow velocities; however, RNNs had significantly lower error for the fastest 10% of velocities than other neural networks (p<le-5, one-sided t-test), suggesting they may enable faster real-time movements while maintaining mlow-movement accuracy. Thus, RNNs more accurately predict offline finger movements than other architectures, agrecing with previous literature (6).
Online Decoder Comparison Since real-time, closed-loop control requires the user to actively react to and correct errors, the online and offline distributions of neural activity and movements may differ, preventing a high performing offline decoder from functioning well online. Therefore, we next compared decoders closed-loop setting, where only the monkey's 's neural activity (rather than his hand) controlled the virtual hand movement. To evaluate online performance, we compared

---

decoders within multiple sessions with Monkey N, and measured performance using information throughput (bitrate). Decoders were trained using only same-day data. Figure 2d shows example online movements using a 2-DoF LSTM, where the monkey quickly moved each finger group and then held at the target position, for a median time of 1.3 sec per trial. The LSTM tended to overshoot index-flexion targets, which Monkey N also tended to do during hand-control. On all tests, the LSTM decoder matched or had the highest information throughput, with a median bitrate 15% higher than the GRU across 2 comparison days, and 18% higher than the FNN across 3 days. Supplemental Video I shows example usage of each decoder.
We next tested if online decoder performance follows the same ordering as offline accuracy. Figure 2e/f shows online performance on two exemplar days where all five decoders were tested in I-DoF and 2-DoF task with Monkey N, where performance followed the same ordering as offline accuracy for both comparisons (except for the TFM outperforming the FNN on the 1-DoF task). Interestingly, the FNN had only slightly lower online 2-DoF bitrate compared to the RNN decoders despite significantly lower offinn position and velocity correlations. This suggests that offline performance on the same dataset may give general idea about online performance ordering but does not indicate the specific performance differences during closed-loop control.
High Channel-Count Simulated Decoder Comparison With the recent developments of higher channel count recording systems 29 30 BMIs may soon have access to more input channels. To determine if recurrent architectures may be more accurate than feedforward architectures at higher channel counts, we trained decoders on simulated datasets with a varied number of artificial neural channels. As seen in Figure 2c, for 10-100 channels, the LSTM and GRU decoders have significantly higher offline correlation compared to FNN, TFM, and KF decoders (average LSTM correlation 0.1 higher than TFM). However, at higher channel counts the advantage of the LSTM and GRU shrinks, such that the difference in correlation at 400 channels is only 0.01 between LSTM and TFM. While actual neural data may differ from the simulated datasets used here, these results suggest that the choice of a specific nonlinear decoder may be less important at high channel counts.
3.2 Able-bodied performance by memorizing movements
As opposed to the fully randomized target postures in the previous section, we next investigated RNN decoder performance on more simple, stereotyped tasks with limited set of targets. RNNs have the ability to memorize and reproduce learned patterns [4]; here, we aimed to let the RNN memorize movement patterns for discrete set of targets, potentially allowing for more accurate decoding despite the potential for reduced generalization on a wider range of tasks. To test the capacity for RNNs memorize targets, we trained and tested LSTM decoders on discrete target datasets, ranging from 2 targets (1D) up to 23 targets (2D). We note that the target order was randomized, which required the decoder to rely on neural input. Separate decoders were trained for each number of targets. We found that LSTM decoders could memorize the specific positions and movements between targets and that offline error increased as the number of targets increased (Figure 3a). Intuitively, this makes sense because as more targets are added, there are fewer example movements per target when keeping the number of trials constant. In additional simulations, we found that as more targets are added, more inputs or training data are needed to maintain accuracy (Supplemental Figure 3). While offline error was highest for random targets, this maximum value corresponds to relatively accurate predictions (as shown in the previous section).
We found a similar trend when testing decoders online with simple tasks (Figure 3b). For tasks with 2 to 4 targets, monkey N using an I.STM decoder achieved 2% higher bitrates than able-bodied manipulandum control on average (see Supplemental Vidco 2), where the performance improvement was due to a reduced orbiting time (in the 2-target case on average, brain control had a 20 ms slower move time but 96 ms faster orbiting time). Move time the time to reach the target and orbit time is time for extraneous movements after reaching the target (see Methods). As the number of targets increased, brain-contro move times stayed approximately the same while orbiting time increased (Figure 3c). This suggests that with fewer targets, the RNN learns stronger internal dynamics to minimize extraneous movement. Overall, as more targets are added, thus decreasing offline performance, online performance drops down to approximately random-targe performance. Therefore, by changing the training dataset the RNN decoder can smoothly transition between memorization with strong dynamics (similar classifier) and generalization with weaker dynamics.
7

---

1:0xF
% of Able- food--ooiied Biteette
Hard More Drain Orbing
2-0uF
503%
63%
Time (ma)
503
MSE
79%
250
60%
E06
11
20 mwndor
2.34
20 random 10000
10 (AT2OTT
Targets
Number Targets
Targets
Figure 3: LSTM decoders have increased performance on simple tasks. (a) Offline MSE (a.u.) for LSTMs trained on tasks with varied numbers of targets and 1-2 DoFs, where all tasks were performed on the same day. "2D random small" refers to random targets with 25% smaller size, requiring finer control. (b) Online performance for varied numbers of targets and DoFs across multiple sessions, with performance measured relative to able-bodied control (calculated as online-bitrate hand-bitrate). Lines indicate tests performed within the same day. (c) Average move and orbiting times for hand control (gray bars) and brain control (blue bars) for 1-DoF tasks on an exemplar day. Orbiting times tend to increase with task complexity. Note that the 500 ms hold time is not depicted.
3.3 Recovering performance through memorization
In some cases, information particular DoF is limited (for example if BMI has too few channels tuned to that DoF or movement) making the overall BMI unusable. To recover functional use, we hypothesized that the poor DoF could rely memorization of discrete movement set (for greater accuracy) while the other good DoF(s) maintain fully continuous movements. For example, Figure shows the first three principal components of the hidden state of a 2-DoF GRU trained on two targets for index finger and random targets for MRS fingers (simulated data, see Supplemental Methods). In DoF acts like movement classifier jumping between line attractors, while the other
this setting, DoF is continuous with positions continuously represented along each line attractor.
We tested this strategy with Monkey N, who, over period of 6 months, had reduction in control of the index finger (likely due to a reduction of neural channels tuned to index finger). When using an LSTM trained on a finger group (2-DoF) random task, index finger had poor online control, resulting in a success rate of 66%, (average bitrate 1.14, 112 trials). To recover performance, on the same day, we trained an LSTM center, flex) while maintaining random targets for MRS fingers. By reducing the number of distinct index finger movements, the number of training examples per movement is increased, allowing the RNN to learn more accurate movement dynamics. The LSTM trained on the modified task had substantially higher online performance (success rate of 98%, average bitrate of 4.18 262 trials) and the finger groups were visibly more independent (see Supplemental Video 3). Thus, for BMI or prosthetics users with poor input quality or limited data, functional online performance may be recovered by modifying the decoder training dataset without necessarily tuning hyperparameters or parameters of the decoder itself.
modified task with only three targets for index finger (extend,
3.4 Memorized RNNs partially generalize to continuous tasks
For RNNs trained to memorize simple postures, is unclear how well these decoders can generalize to other tasks. One might expect that, when trained to output discrete set of positions, the RNN may Iearn strong fixed points (or other dynamics) that enable the RNN to accurately output discrete positions, but may perform poorly when tasked with holding intermediate postures (random targets). To test this assumption and the ability of memorized RNNs to generalize, we trained LSTMs on tasks with limited number of targets and tested the decoders online random target tasks. We tested the generalization ability of positional (50% position, 50% integrated velocity) and velocity (1% position, 99% integrated velocity) variants of the same decoder (Supplemental Figure 4).
LSTM decoders trained on 1-DoF 2- and 3-target tasks had 100% and 98% online success rate (SR) respectively when tested on the same simple tasks (positional variant). When tested on the random target task, as expected, the positional decoders had difficulty stopping on targets that did not overlap with the Icarned targets, and failed to move outside of the learned extenion range. The
8

---

Targets
Index
Finger
MRS
Fingers
PC2
% Flesson
PCE
Colored by Index Position
Colored by MRS Position
Figure 4: Hidden states of memorized RNN. (a) A GRU RNN was trained on a modified task with random targets for MRS fingers but only 2 targets for index finger (simulated data). (b, c) Plots depict the first three principal components the RNN hidden state, with traces indicating the path over time. (b) and (c) plot the same data, just with different coloring. When the path is colored by the predicted index position (b), it is clear each line attractor corresponds to an index target. As in (c), the position along each line attractor corresponds to the predicted MRS position. Thus, the RNN can learn different dynamical structure for each DoF to act more like a classifier when beneficial.
2-target positional decoder was non-usable (SR < 50%), while the 3-target positional decoder was more functional (SR 87%, bitrate 1.05). Interestingly, the velocity decoder variants were better able to generalize, where the 2-target velocity decoder had 98% SR (bitrate 1.33) and the 3-target velocity decoder had an 80% SR (bitrate 0.93, lower performance likely due to low monkey motivation). While functional, the velocity variants had more difficulty making small corrective movements and had lower bitrates compared to random-trained decoders (bitrate 291 These results suggest that memorized RNNs generalize to some degree, where training positional generalization.
targets leads to better
Discussion
A key component of BMIs that restore motor function is accurate decoding algorithm that performs well in online settings. Unfortunately, for continuous movement decoding, offline accuracy does not incorporate closed-loop dynamics, meaning offline metrics may fail to predict online performance [ and offline cross validation does not fully reveal decoder overfitting (10). However, here, using a multi-DoF finger BMI, found that offline accuracy generally predicted the relative online decoder performance ordering. We found that recurrent neural networks, namely LSTMs, outperform other neural network architectures in both offline and online settings. This high performance of RNNs follows previous work showing that decoders that learn neural or kinematic state dynamics (including RNNs) can more accurately predict kinematics from noisy neural data [31 27 6). All the networks tested here were relatively small (5 layers or fewer) and could be efficiently run in real time (typically <2 ms per prediction), suggesting that very deep or complex architectures may not be necessary for high performance. Additionally, we did not perform intention-based recalibration (ReFIT; as described 24 [3) due to experimental trial count constraints. This could potentially benefit some decoders more than others. Further work may also investigate the performance trade-offs of decoders trained de-noise neural data 7 versus decoders directly trained to predict kinematics.
While the neural network decoders tested here were all highly usable with minimal apparent overfit- ting, Deo al. [10] found that RNNs tended significantly overfit offline data when tested in humans. In humans with paralysis, decoding algorithms are trained from offline data by instructing participants to attempt movements sync with an artificially moving effector. With able-bodied users (like the non-human primates used here), however, the direct relationship between neural activity and true movement can be used for training. These able-bodied movements more broadly sample the range of positions/velloities and neural activity needed to perform a task and often have errors like overshooting. When trained on only two able-bodied targets, enough variation was present to allow for generalization to other movements. These results suggest regularization of neural networks is necessary for high performing BMIs, whether inherent to the training data or by applying artificial data modifications.

---

In additional testing we found that using more time history in the FNN and TFM decoders (up to 20 bins/640 ms) resulted in poor online control (Supplemental Figure 5). This is likely due to overfitting to training data that lacks many of the corrective movements present in online control, and demonstrates how offline analyses can fail to predict the performance in the closed loop setting. Interestingly, the RNNs were trained with longer history but did not display this overfitting. Future work may investigate methods of using longer training sequences while limiting overfitting and lag, for example by discouraging attention to early bins.
Decoder memorization of training data is commonly associated with overfitting, suggesting poor closed-loop control. Here, we found that RNN decoders allowed to memorize a few movement patterns were highly controllable online and matched able-bodied performance for tasks with 2- 4 target postures. With reduced target set the RNN can learn stronger recurrent dynamics for higher accuracy, acting somewhat like a pseudo-classifier between movements. As task complexity (the number of targets and distinct movements) increases, offline and online decoder performance degrades gracefully down to fully continuous performance, partly due to the reduced number of training examples per movement. This is similar to neural word decoding, where word error rates increase with larger vocabularies [32]. Unlike algorithms that manually combine a classifier and continuous decoder 33 [4
RNNs are fully continuous yet can internally act as classifier and generator for movements that lack input information, while maintaining high performance continuous output for other movements. As BMIs extend to more DoFs, results suggest that more data or neural channels will be needed to maintain accuracy, and the degree of memorization can be controlled for each DoF independently. Future work may investigate the relative dependence inputs vs autonomous dynamics of RNNs, and their effect on online performance.
There are several limitations to the present study of closed-loop BMI performance. First, we did not explore every possible decoder architecture. More optimal architectures likely exist, but here we found several key factors of each class (for example, the need for limited time history of feedforward models, and the ability of RNNs to strongly learn few movements). Future work may explore models that combine recurrence with feedforward attention layers to take advantage of the benefits of each (for example, (35.. Also, only one monkey was used for the online analyses; additional users may have different control strategies or decoder errors.
Future BMI decoders could be designed to be more explicitly tunable to allow the user to adjust the balance between memorization (with more reliance on internal dynamics) and generalization to wider range of movements. A decoder could act more like classifier when producing stereotyped movements such sign language functional grasps, but act as continuous controller when producing arbitrary movement. Users at the onset of BMI use could use a simple task decoder that works well with minimal training data (reducing cognitive fatigue and maintaining user motivation), then transition to higher complexity as more data acquired. These strategies could also be employed for myoelectric and other prosthetic controllers which must function using noisy signals with large uncertainty in user intentions. BMIs that can produce accurate kinematic predictions across the range of few neural channels to thousands could speed the clinical translation of BMIs with current and future hardware.
5
Acknowledgements
We thank Eric Kennedy for animal and experimental support. We thank the University of Michigan Unit for Laboratory Animal Medicine for expert surgical and veterinary care. JTC was supported by NSF GRFP under grant 1841052. DMW was supported by Dan and Betty Kahn Foundation (Grant 2029755) and University of Michigan Robotics Institute. HT, MJM and PGP were supported by NSF grant 1926576. LHC was supported by Agencia Nacional de Investigación y Desarrollo (ANID) of Chile. MSW was supported by NIH grant T32NS007222. CAC was supported by Dan and Betty Kahn Foundation (Grant 2029755) and NSF grant 1926576.
10

---

References
[1] Chethan Pandarinath, Paul Nuyujukian, Christine H. Blabe, Brittany I.. Sorice, Jad Saab, Francis R. Willett, Leigh R. Hochberg, Krishna V. Shenoy, and Jaimie M. Henderson. High performance communication by people with paralysis using an intracortical brain-computer interface. eLife, 6:1-27, 2017.
[2] Jennifer L. Collinger, Brian Wodlinger, John E. Downey, Wei Wang, Elizabeth C. Tyler-Kabara, Douglas J. Weber, Angus J.C. McMorland, Meel Velliste, Michael L. Boninger, and Andrew B. Schwartz. High- performance neuroprosthetic control by an individual with tetraplegia. The Lancet, 38::557-564, 2013.
[3] A. Bolu Ajihoye, Francis R. Willett, Daniel Young, William D. Memberg, Brian Murphy, Jonathan P. Miller, Benjamin L.. Walter, Jennifer A. Sweet, Harry A. Hoyen, Michael W. Keith, Hunter Peckham, John D. Simeral, John P. Donoghue, Leigh R. Hochberg, and Robert F. Kirsch. Restoration of reaching and grasping movements through brain-controlled muscle stimulation in person with tetraplegia: proof-of-concept demonstration. The Lancet, 389: 199118211880 2017.
[4] Sepp Hochreiter and Jürgen Schmidhuber. Long short-tern term memory. Neural Computation, 9(8):1335 170000000780 1997.
[5] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre. Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rmn encoder-decoder for statistical machine translation. arXiv preprint arXiv: 1000//106... 1078, 2014.
[6] Joshua I. Glaser, Ari S. Benjamin, Raeed H. Chowdhury, Matthew G. Perich, Lee E. Miller, and Konrad P. Kording. Machine learning for neural decoding. eneuro, 7:ENEURO.0506-119200 2020,, 2020.
[7] Chethan Pandarinath, Daniel J. O' Shea, Jasmine Collins, Rafal Jozefowicz, Sergey D. Stavisky, Jonathan Kao, Eric M. Trautmann, Matthew T. Kaufman, Stephen I. Ryu, Leigh R. Hochberg, Jaimie M. Henderson, Krishna V. Shenoy, L. F. Abbott, and David Sussillo. Inferring single-trial neural population dynamics using sequential auto-encoders. Nature Methods, 15:805-815, 2018
[8] Joel Ye and Chethan Pandarinath. Representation learning for neural population activity with neural data transformers. bioRxiv, page
2021.
[9] John P. Cunningham, Paul Nuyujukian, Vikash Gilja. Cindy A. Chestek, Stephen I. Ryu, and Krishna V. Shenoy. A closed-loop human simulator for investigating the role of feedback control in interfaces. Journal of Neurophysiology, 105: 1932-199994, 2011.
[10] Darrel Deo, Francis R Willett, Donald T Avansino, Leigh R Hochberg, Jaimie M Henderson, and Krishna V Shenoy. Translating deep learning to neuroprosthetic control. bioRxiv, pages 2023-04, 2023.
[11] David Sussillo, Sergey D. Stavisky, Jonathan C. Kao, Stephen I. Ryu, and Krishna Shenoy. Making brain-machine interfaces robust to future neural variability. Nature Commmnications, 7:1-12, 2016.
[12] Tommy Hosman, Marco Vilela, Daniel Milstein, Jessica N. Kelemen, David M. Brandman, Leigh R. Hochberg, and John D. Simeral. Bci decoder performance comparison of 1stm recurrent neural network and kalman filter retrospective simulation. International IEEE/EMBS Conference on Neural Engineering, NER,
10661 2019.
[13] Matthew S. Willsey, Samuel R.
Scott R. Ensel, Hisham Temmar, Matthew J. Mender, Joseph T. Costello, Parag G. Patil, and Cynthia A. Chestek. Real-time brain-machin interface in non- human primates achieves high-velocity prosthetic finger movements using a shallow feedforward neural network decoder. Nature Communications, 13, 12 2022.
[14] David Sussillo and L. F. Abbout. Generating coherent patterns of activity from chaotic neural networks. Neuron, 63:544-5555 8 2009.
[15] David Sussillo, Mark M. Churchland, Matthew T. Kaufman, and Krishna V. Shenoy. A neural network that finds naturalistic solution for the production of muscle activity. Nature Neuroscience, 18: 1025-13033,, 6 2015.
[16] Guangyu Robert Yang, Madhura Joglekar, Francis Song, William T Newsome, Xiao-Jing Wang. Task representations in neural networks trained to perform many cognitive tasks. Nature neuroscience, 22(2):297-306, 2019.
[17] R. Felix Reinhart and Jochen J. Steil. Reaching movement generation with a recurrent neural network based learning inverse kinematics for the humanoid robot icub. pages 323 330. IEEE, 12 2009.
11

---

[18] Philipp Kratzer. Mare Toussaint, and Jim Mainprice. Prediction of human full-body movements with motion optimization and recurrent neural networks. pages 1792-1798. IEEE, 5 2020.
[19] Benjamin Ehret, Christian Henning, Maria R Cervera, Alexander Meulemans, Johannes Von Oswald, and Benjamin F Grewe. Continual learning in recurrent neural networks. arXiv preprint arXiv:2006. 12109, 2020.
[20] ZT Irwin, KE Schroeder, PP Vu, AJ Bullard, DM Tat, CS Nu, A Vaskov, SR Nason, DE Thompson, JN Bentley, et al. Neural control of finger movement via intracortical brain-machine interface. Journal of neural engineering, 14(6):06004,, 2017.
[21] Alex K. Vaskov, Zachary T. Irwin, Samuel R. Nason, Philip P. Vu, Chrono S. Nu, Autumn J. Bullard, Mackenna Hill, Naia North, Parag G. Patil, and Cynthia A. Chestek. Cortical decoding of individual finger group motions using refit kalman filter. Frontiers in Neuroscience, 12, 2018
[22] Samuel R. Nason, Alex K. Vaskov, Matthew Willsey, Elissa Welle, Hyochan An, Philiip Vu. Autumn J. Bullard, Chrono S. Nu, Jonathan C. Kao, Krishna V. Shenoy, Tackwang Jang. Hun Scok Kim, David Blaauw, Parag G. Patil, and Cynthia A. Chestek. A low-power band of neuronal spiking activity dominated by local single units improves the performance of machmme interfaces. Nature Biomedical Engineering, pages 0- 1, 2020.
[23] Wilson Truccolo, Gerhard M. Friehs, John Donoghue, and Leigh R. Hochberg. Primary motor cortex tuning to intended movement kinematics in humans with tetraplegia. Journal of Neuroscience, 28:1 1633- 1178, 2008.
[24] Vikash Gilja, Paul Nuyujukian, Cynthia A. Chestek, and Krishna V. Shenoy. A high-performance neural prosthesis enabled by control algorithm design. Nature Neuroscience, 2012.
[25] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systemy, 30, 2017.
[26] Samuel R. Nason, Matthew J. Mender, Alex Vaskov, Matthew Willsey, Nishant Ganesh Kumar, Theodore A. Kung, Parag G. Patil, and Cynthia A. Chestek, Real-time linear prediction of simultaneous and independent movements of two finger groups using an intracortical brain-machine interface. Neuron, (09-3164-1177.e, e8, 2021.
[27] Jonathan C. Kao, Paul Nuyujukian, Stephen I. Ryu, Mark M. Churchland, John P. Cunningham, and Krishma V. Shenoy. Single-trial dynamics of motor cortex and their applications brain-machine interfaces. Nature Communications, 6, 2015.
[28] Francis R. Willett, Donald T. Avansino, Leigh Hochberg, Jaimie M. Henderson, and Krishna V. Shenoy. perfforrrfe brain-to-text communication via handwriting. Nature, 593:249-2555 222244 2021.
[29] Nicholas A. Steinmetz, Cagatay Aydin, Anna Lebedeva, Michael Okun, Marius Pachitariu, Marius Bauza, Maxime Beau, Jai Bhagat, Claudia Böhm. Martijn Broux, Susu Chen, Jennifer Colonell, Richard J. Gardner, Bill Karsh, Fabian Kloosterman, Dimitar Kostadinov, Carolina Mora-Lopez, John O' O'Callaghan, Junchol Park, Jan Putzeys, Britton Sauerbrei, Rik J.J. van Daal, Abraham Z. Vollan, Shiwei Wang. Marleen Welkenhuysen, Zhiwen Ye, Joshua T. Dudman, Barundeb Dutta, Adam W. Hantman, Kenneth Harris, Albert K. Lee, Edvard I. Moser, John O' Keefe, Alfonso Renart, Karel Svoboda, Michael Häusser, Sehastian Haesler, Matteo Carandini, and Timothy D. Harris. Neuropixels 2.0: A miniaturized high-density probe for stable, long- term brain recordings. Science 372, 2021.
[30]
Kunal Sahasrabuddhe, Aamir A. Khan, Aditya P. Singh, Tyler M. Stern, Yeena Ng. Aleksandar Tadić, Peter Orel, Chris Lareau, Daniel Pouzzner, Kurtis Nishimura, Kevin M. Boergens, Sashank Shivakumar, Matthew S. Hopper, Bryan Kerr, Mina Elraheb S. Hanna, Robert J. Edgington, Ingrid McNamara, Devin Fell, Peng Gao, Amir Fishai, Sampsa Veijalainen, Alexander V. Klekachev, Alison M. Stuckey, Bert Luyssaert, Takashi D.Y. Kozai, Chong Xie, Vikash Gilja, Bart Dierickx, Yifan Kong. Malgorzata Straka, Harbaljit S. Sohal, and Matthew R. Angle. The argo: A high channel count recording system for neural recording in vivo. Journal of Neural Engineering, 18, 2021.
[31] Wu, Black, Gao, Bienenstock, M Surruya, and J.P. Donoghue. Inferring hand motion from multi-cell recordings in motor cortex using a kalman filter. pages 66-73, 2002.
[32] Francis R Willett, Erin M Kunz, Chaofei Fan, Donald T Avansino, Guy Wilson, Eun Young Choi, Foram Kamdar, Leigh R Hochberg. Shaul Druckmann, Krishna V Shenoy, et al. A high nigh-performance speech neuroprosthesis. bioRxiv, pages 2023-00 01, 2023.
12

---

[33] Jomathan C. Kao, Stephen I. Ryu, and Krishna V. Shenoy. Leveraging neural dynamics to extend functional lifetime of brain-machine interfaces. Scientific Reports, 7:1-16, 2017.
[34] Brian M. Dekleva, Jeffrey M. Weiss, Michael I.. Boninger, and Jennifer L. Collinger. Generalizable cursor click decoding using grasp-related neural transients. Journal of Neural Engineering, 18, 2021.
[35] Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In International conference machine learning, pages 4651- 4664. PMLR, 2021.
13


Reviews:
[{"review_id": "OPIPnu6rw8", "reviewer": "Reviewer_3c6F", "summary": "This paper uses an RNN for motor control task. Specifically, it uses the RNN to decode finger movements using neural data recorded from NHPs. ", "questions": "I have following questions. \n- SBP vs spike thresholding. There was a single reference that was cited. Do you see a marked difference between the two. If SBP gives similar correlations then it would be highly beneficial from hardware perspective since the sampling rate can be dropped to 2KSPS?\n- How is higher channel count simulated? Is there a simulator that enables generating BMI data? Can you simulate and generated a longer dataset (more sessions) for training with same channel count (Does that have benefit on using RNN vs KF ). \n- Figure 3 and Section3.2 are the decoders retrained for different targets? For instance will you retrain/calibrate the decoders for different set of target or are you having a constant training set and then just re-evaluating the decoder for multiple targets?\n- For all the decoder are they trained and tested on the same session?\n\n", "limitations": "Check weakness and questions. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The following are the strengths of this paper:\n-Online decoding using RNNs. \n-Evaluation and comparison with other neural network algorithms. \n-Evaluation with higher channel counts. \n- Recovering performance using RNNs. ", "weaknesses": "-There have been several publications including previous NEURIPS papers that show use of RNNs for decoding neural data. \n-The algorithms are mostly used as is. The novelty and strengths of the manuscript lies in the system engineering. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their questions.\n\n\n**Do you see a difference between SBP and spike thresholding?**\n\nIn this work we did not directly compare SBP to spike thresholds but used SBP as the feature for all decoders. Anecdotally, in experiments prior to this study, Monkey N was unable to complete trials using a threshold-based Kalman filter but could successfully perform trials using an SBP-based Kalman filter. This is likely due to visibly low SNRs for most channels on Monkey N’s array, which may be more easily decoded using SBP than thresholds (Nason et al. 2020). \n\nAs you mention, SBP can substantially reduce the sampling rate (to 2 KSPS) for a large reduction in power (Irwin et al. 2015, Nason et al. 2021, Costello et al. 2023). Thus, because of the strong decoding performance and the ability to use SBP on battery-powered implantable systems, we used SBP for the decoder comparisons here. \n\n\n\n**How is the higher channel count data simulated? Can you generate more data?**\n\nWe appreciate the reviewer pointing out the lack of explanation for simulated data. This information was in the supplement, so we propose adding another methods section as noted in the Global Response. \n\nBriefly, we simulated data using a virtual user who performed the task by moving to targets, and then generated artificial channels where each channel had a random relationship with the virtual user’s position, velocity, and acceleration. This method was based on Trucollo et al. 2008, and is similar to random tuning methods used in other work (Churchland et al. 2012, Willet et al. 2019). Here, the purpose of the simulation was to simply generate datasets with any length, number of channels, number of degrees-of-freedom, or task complexity, which could then be used to measure relative decoder performance across these varied parameters (for example, Fig 2c and Supplemental Fig 3). However, since the simulated channels are randomly generated and do not match the real neural channels, they cannot directly be used for generating additional training data or extrapolating absolute performance to higher channel counts (although there are methods for generating additional realistic datasets).\n\n\n\n**Are decoders retrained for different targets? Are decoders trained/tested in the same session?**\n\nFor each different number of targets we collected a training set and trained a decoder on that specific setup. This means each datapoint in figure 3 is a fully separate decoder. The only exception to this is section 3.4 where we purposefully tested the ability of a decoder to generalize to more targets.\n\nAll decoders were trained and tested within the same session. While there are methods of incorporating previous datasets into training (like using a day-specific input layer) or recalibrating previous decoders, here, using able-bodied monkeys, we could obtain enough daily training data to saturate performance without added complexity.\n\nTo help clarify these points, we propose adding the following:\nResults 3.1, line 221, “Decoders were trained using only same-day data.”\nResults 3.2, line 252, “Separate decoders were trained for each number of targets.”\n"}, {"review_id": "9e8xHj5NWE", "reviewer": "Reviewer_WBvg", "summary": "The authors study the performance of various models (GRUs, LSTMs, TFMs, FNNs, KFs) in reading out neural population activity during one and two finger tasks in non-human primates. Overall, they find that RNNs (GRUs & LSTMs) outperform other models during online tasks and serve as the most promising model for real-time brain machine interfaces (BMI).\n", "questions": "What are the practical limitations of extending the claims here to more naturalistic BMIs? Balancing memory and generalization in 1 or 2 DoF tasks is an extremely special case. This should be discussed. \n\nDoes the performance of the transformer model improve if a sequence length of greater than 5 (i.e., longer than 160ms) is included in the transformer context window (i.e., the memory of the transformer)?\n\nWhat are the number of parameters per model? Are they comparable?\n\nAre there corresponding results for Figure 3 for the other models?\n\nI don’t quite grasp the discussion regarding the ability of LSTMs memorizing the positions and movements of targets when number of targets are low. Is this supposed to be a good thing? This seems to me more of a failure/limitation of the model, since it fails to abstract what the neural population is actually doing (controlling finger movements, rather than directing to specific targets).\n\nLine 339: “These results suggest regularization of neural networks is necessary for high performing BMIs”. It wasn’t clear to me which reported result this sentence was referring to.\n\nWhat are the theoretical reasons as to why RNNs may perform better than other models for continuous BMI?\n", "limitations": "The limitations are mostly in terms of the scalability of a ‘memorization’ strategy to ethological and naturalistic tasks. More generally, it’s not clear how these findings would scale to a more natural task.\n", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "\nThe paper demonstrates the real-world utility of contemporary machine learning models in BMI. The methodological approach seems sound. Despite the 10-page limitation of the NeurIPS submission, it is clear that a lot of work and effort (both experimentally and data analytically) was put into the making of this study.\n", "weaknesses": "I have one conceptual weakness, and several more methodological weaknesses. The primary weakness is that it’s unclear to what extent these claims will generalize to more interesting BMI tasks – their results already demonstrate the limitations of their experimental designs (i.e., when more movements exist, the capacity of RNNs are effectively reached). It’s also not clear how generalizable the solution of “balancing memory and generalization” is motor BMI, when the ecologically valid degrees of freedom are extremely large.\n\nMethodologically, I am skeptical about how/why transformers perform so poorly. Transformers are powerful precisely because they can learn dependencies in their context window. In the case of a BMI task, this might be temporal neural activity in the preceding time window, with the embedding dimension corresponding to the number of neurons/channels. In this case, why were the transformers limited to only 5 bins of time history per channel (160ms) (line 140-141). This seems extremely limiting, and it is likely that GRUs and LSTMs have a ‘memory’ that is significantly longer than this. A sequence of length 5 does not seem to play to the strengths of a transformer. There are also more recent ‘recurrent’ transformer models that may be relevant to test here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their questions and feedback.\n\n\n**What are the practical limitations of extending the claims here to more naturalistic BMIs? (More DoFs)**\n\nWe address this question in the global response and propose an addition to the discussion. Briefly, more DoFs may require more training data and/or more channels. With more channels (and more information), we might reduce the amount of memorization for more naturalistic control.\n\n\n\n**Does the performance of the transformer model improve if a sequence length of greater than 5 is included in the transformer context window?**\n\nThis is a good question since we’re not taking advantage of one of the transformer’s strengths. We found that using longer sequence lengths resulted in better offline accuracy but worse closed-loop control. See the global response for details and a proposed addition to the discussion.\n\n\n\n**What are the number of parameters per model? Are they comparable?**\n\nAll the neural network models had the same order magnitude number of parameters, but the LSTM used significantly fewer parameters than the TFM:\nGRU – 639k, LSTM – 479k, FNN – 821k, TFM – 923k, KF – 10k\nNotably, the feedforward models required more parameters than the RNNs. For each model, we chose the layer sizes that maximized offline accuracy, aiming to not overparameterize each model.\n\nWe propose adding the following line to results at line 198, “The total number of neural network decoder parameters varied from 479k for the LSTM to 923k for the TFM (see supplemental results).” And adding a table of parameter counts to the supplement: “GRU – 639k, LSTM – 479k, FNN – 821k, TFM – 923k, KF – 10k”\n\n\n\n**Are there corresponding results for Figure 3 for the other models?**\n\nFigure 3, which details the performance on varied numbers of targets, was only performed using LSTMs and not replicated with the other models. While it would be interesting to test other models on the varied tasks, we chose to test the highest performing model (LSTM) due to experimental time and paper length constraints. We hypothesize that similar trends would be found with the other models, albeit with lower performance. Since the feedforward models are limited to only 5 bins of history, one might expect them to have a lesser ability to generate smooth and precise movements compared to the RNNs with a longer time history.\n\n\n\n**Why is memorization of few targets a good thing? This seems like a model failure since the model fails to abstract to what the neural population is actually doing.**\n\nAs you mention, “memorization” limits the output space of the model, may prevent generalization to the continuous range of postures, and does not fully abstract the neural dynamics. However, from a BMI controls perspective, we want a controller that can accurately and robustly move between postures to enable functional use, even if this somewhat limits the output capability. For example, someone with perfect control of a prosthetic with a few distinct positions may have a functional advantage over someone with moderate control of a fully continuous prosthetic. When a user has more neural channels with more information, it probably makes sense to use regular continuous control. However, with current implants it is common to have poor signal quality, in which memorization can be useful. Also, it was interesting to find that even when the decoder “memorizes” outputs, it still relies on the user’s inputs vs fully relies on autonomous dynamics (which was initially unclear). \n\n\n\n**Line 339: “These results suggest regularization of neural networks is necessary for high performing BMIs, whether inherent to the training data or by applying artificial data modifications”. It wasn’t clear to me which reported result this sentence was referring to.**\n\nIn this discussion paragraph we compare a recent study that found when training on artificial movements in humans, significant data augmentation was required to prevent overfitting. Here, we trained on real movements that incorporate imperfections and we did not have a problem with overfitting. The “regularization” we refer to in line 339 is the natural variation in our training data that may help to regularize the network, resulting in the generally high performance overall (for example, matching able-bodied performance in section 3.2.\n\n\n\n**What are the theoretical reasons as to why RNNs may perform better than other models for continuous BMI?**\n\nModels that maintain an internal state may better “denoise” neural inputs. For example, a Kalman filter that incorporates a model of kinematics or physics can often outperform a simple linear regression. Similarly, RNNs can learn an internal state that helps it understand the neural dynamics and to smoothly generate continuous outputs, despite noisy input. Feedforward models without an internal state may be able to accurately predict continuous outputs when given enough time history. However, as we found in real-time decoding, using long time history results in overfitting and poor performance. \n"}, {"review_id": "0szaGcte01", "reviewer": "Reviewer_cHBb", "summary": "The work analyzes the performance of different decoders for finger movement prediction from monkeys in different offline and online settings. The authors evaluate recurrent networks, convolutional networks, transformers and Kalman filters. They find recurrent networks to perform best both for online decoding as well as offline decoding. The order of offline performance of the decoders predicts the order of online performance, but the magnitude of differences between decoders varies between offline and online settings. The work also investigates the effects of the degrees of freedom and the number of discrete movement targets (up to an infinite number/continuous movement decoding). Results for recurrent neural networks suggest that varying the number of discrete targets allows to interpolate between the behavior of a classifier and a continuous movement decoder, where for few targets, most of the actual movement to the target is memorized and with more and more targets, more of the movement is decoded from the brain signal.\n", "questions": "During online decoding experiments training part, is the movement 100% controlled from the start by the decoded brain signal? Or is it in any way  slowly transitioning  from  desired movement to decoded movement as some other works do?\nWhat exactly are the simulated data used in 3.1?\n\n\n\"When more than 100 trials were performed, we removed the first 50 trials from the performance calculation to account for a decoder learning period. When fewer than 100 were performed (typically when the performance was poor and the monkey was unable to complete many trials), the first 20 trials were removed from analysis.\"\nWas there some reason to choose exactly 50 and 20?", "limitations": "Main results are from a single monkey.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The topic of continuous movement decoders is scientifically interesting and the comparisons  are insightful. \nAdditional evaluations looking at number of targets etc. are also valuable.\nFigures help understand method and results, fairly clean (except a bit blurry Fig 4).\n\nThe model types (convolutional, recurrent, transformer, kalman) seem well-chosen to cover typically used decoder types.\n\nThere are a lot of subanalyses like higher-channel-count simulated data that help better understand the factors affecting the behavior/performance of the different decoding methods.\n\nThe manuscript is well-written and covers many questions and most technical details clearly. Results cover interesting aspects like move and orbiting time.", "weaknesses": "Some details could be made more clear, see Questions.\n\nReFIT is first used without an explanation what it means.\n\nFigure 4 is blurry and hard to read.\n\nA small detail, but making the \"Figure X\" tests into references, so you can click on them, would be nice.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their questions and helpful suggestions.\n\n\n**ReFIT is first used without an explanation.**\n\nWe propose modifying line 40 to briefly explain ReFIT:\n“Alternatively, [30] found that a convolutional feedforward network using intention-based recalibration (ReFIT) outperformed a ReFIT Kalman filter in an online two finger group task by accurately predicting a large dynamic range of velocities.”\n\n\n\n**Are training movements fully controlled by decoded neural signals?**\n\nIn this work, training movements were controlled by the monkey’s physical fingers in a manipulandum, rather than decoded neural signals. Then, after training the decoder, during test trials the movements were 100% controlled by the decoded neural signals. \n\nAs you mention, there are methods to gradually transition from idealized movements to full brain control (Velliste et al. 2008), which may be necessary when the user is paralyzed or is amputated. Here, we simply used the able-bodied training movements since the goal was to compare maximal decoder performance rather than training methods. Also, for a fair comparison across decoders, each decoder would need to be trained with the gradual method (taking 5-10 minutes each), which would be infeasible due to the monkey’s limited window of motivation.\n\n\n\n**What is the simulated data in 3.1?**\n\nWe appreciate the reviewer noticing this confusing reference.  The simulated data was outlined in the supplementary methods (removed from the main text due to space restrictions), but not referenced in the main text. We propose adding a methods section to describe the simulated data, as explained in the Global Response.\n\n\n\n**What is the reasoning behind the number of initial trials removed from analysis?**\n\nThe goal of removing initial trials from analysis was to account for the learning period (when the monkey was getting used to using the decoder) and only measure maximal steady-state performance. The monkey qualitatively took approximately 1-2 minutes to reach this steady-state period, which corresponded to ~50 trials when the decoder functioned well and ~20 trials when the decoder functioned poorly. \n\nWe propose the following change to line 110 for clarity:\n“Before calculating average bitrate, we removed initial trials from the performance calculation to account for the decoder learning period before steady state performance, which took approximately 1-2 minutes. When more than 100 trials were performed we removed the first 50 trials, and when fewer than 100 trials were performed (typically when the performance was poor and the monkey was unable to complete many trials) we removed the first 20 trials.”\n\n\n\n**Main results are from a single monkey.**\n\nWe agree that only having one monkey is a primary limitation of the study. The specific decoder errors and absolute closed-loop performance would likely vary between monkeys or human users. To bring more confidence to the results, we performed the offline decoder comparison (3.1) using historical data from a second monkey (briefly mentioned, with full results in Supplemental Figure 2) and using simulated artificial neural data (Figure 2c). In all datasets the same relative performance ordering of decoders was found.\n\nFor the analyses investigating RNN memorization, the interesting result is that memorized RNNs can be highly functional for closed-loop control despite the possibility to overfit. While one example user (monkey) is enough to show that is high performance is possible, it doesn’t fully explore aspects specific to the user (like absolute model accuracy or performance under different user control strategies). \n\nWe propose adding a paragraph to the discussion to mention this limitation, as explained in the Global Response.\n"}, {"review_id": "6GCGkiJB8C", "reviewer": "Reviewer_ZEFp", "summary": "This paper explores the utility of different classes of neural decoders for online brain machine interfaces. The authors first show that recurrent network decoders outperform transformers, CNNs and Kalman filters in an offline setting. They then validate that the superiority of RNN-based decoders also holds in the online setting. Finally, the authors show that such RNNs can flexibly interpolate between memorizing output trajectories in a few-target setting and integrating instantaneous neural activity in a many/continuous-target setting, and that this can be done independently for different actuators.", "questions": "I generally liked the paper, but here are a few suggestions/questions that I thought it would be helpful to clarify if the authors have the capacity to do so.\n\nL40-41: The authors seem to assume that the reader knows what a 'ReFIT' network is, but this would be useful to explain in half a sentence.\n\nEq 1: I'm not sure I fully understand how information throughput is computed, since t_acq is a single number for a given trial, while D_k continuously changes throughout a trial. Which D_k is used? and does the sum only run over fingers constrained by the task (equivalent to setting R=infinity for non-constrained fingers)?\n\nSection 2.5: Since the RNNs have an adaptive memory and can (in theory) learn to retain information over as long timescales as are necessary for solving the task, it might be interesting to explore different convolutional filter sizes for the CNN and different context windows for the transformer. Is the lower performance of these models because you need more than 160 ms of history to accurately predict movements? or is it something more fundamental? and if you can recover RNN-level performance with more temporal history, what is the resulting cost in computation speed? One could imagine a plot that shows performance against time-per-forward-pass for varying context sizes for both the CNN and transformer, and comparing this to the performance/speed of the LSTM/GRU networks.\n\nRelated to the above, it would be good to report the total number of parameters for each network, and potentially the time-per-forward-pass, in the main text, even if the full architecture details can be found in the appendix.\n\nFig 2: It would be nice to show online performance in terms of trial time in addition to information throughput, since that is a much more intuitive metric.\n\nL235: I found it confusing that this section came after the online decoding section, which is also inconsistent with the layout of Figure 2. I was also wondering whether a similar analysis could be done by subsampling the experimental data instead of using simulated data? On the whole, I found this analysis slightly tangential to the rest of the paper and think it could be moved to the supplement if the authors are low on space.\n\nFigure 3: Is there a way to more directly investigate the dependence of the RNN decoders on inputs vs autonomous dynamics? e.g. removing inputs part way through the trial and investigating the effect on performance for different conditions? Or comparing the relative magnitude of 'recurrent' inputs and 'external' inputs across time and conditions?\n\nSection 3.4: It would be useful to put these results in a figure (and potentially place it in the supplement if there is no room in the main text). For example, it would be informative to have a plot that shows performance as a function of target extension (in the continuous test setting), with the discrete training data points highlighted, to visualize how performance decreases away from the training data in the 2-target and 3-target cases.\n\nSupplementary videos: I found these videos informative but it would be helpful to add a section in the supplementary text/methods that provides a brief legend for each video.\n\nSupplementary Figure 3: the title says \"LSTM performance\" and the legend says \"a GRU decoder was trained\".", "limitations": "The limitations of the study are not discussed to a great extent. Things that may be interesting to touch on (if there is room) could include:\n\n(i) the impossibility of exploring the full space of network architectures within each class (RNN vs CNN vs transformer) and what that means for the conclusions about which is 'best'.\n\n(ii) thoughts about the generalizability of the results to more complicated tasks, e.g. N-DoF tasks or non-motor tasks. Would the authors expect RNNs to also be superior for BMI in brain regions that rely less on autonomous recurrent dynamics during natural behavior?", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "Investigating the consistency or discrepancy of different neural decoders between offline and online BMI settings is a very important question in modern systems neuroscience. This is especially true since many labs and researchers do not have the capacity for online BMI experiments, and we as a field therefore strongly rely on offline metrics for motivating different neural decoders.\n\nThe question of memorization in a few-target setting vs. flexible control in a many-target setting is also interesting, and the analyses are clear and informative.\n\nIn general, this is a paper that asks interesting questions, answers them with convincing data and analyses, and is well written and presented, and I very much enjoyed reading it.", "weaknesses": "A major difficulty with studies like this is to tease apart the _potential_ utility of different types of decoders/architectures from the _specific implementation choices_ made. Not much work is done to explore the range of speed/accuracy tradeoffs when e.g. varying network or context size (see below), leaving the reader with only a single data point per neural architecture to draw conclusions from.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their questions and helpful suggestions.\n\n\n**“ReFIT” is not explained.**\n\nWe propose modifying line 40 to briefly explain ReFIT:\n“Alternatively, [30] found that a convolutional feedforward network using using intention-based recalibration (ReFIT) outperformed a ReFIT Kalman filter in an online two finger group task by accurately predicting a large dynamic range of velocities.”\n\n\n\n**How is information throughput calculated for each trial since D_k continuously changes? (Eq. 1)  How many fingers does the sum include?**\n\nInformation throughput was calculated using the distance between finger and target at the start of the trial, so D_k has only one value per trial. We propose modifying line 108 to say “where D_k is the initial distance of the kth virtual finger to the center of the kth target…”. This calculation ignores fingers that are not being actively controlled, for example, in the 2-DoF task only 2 fingers are included in the calculation and all other DoFs are ignored. \n\n\n\n**Is the lower performance of the feedforward models due to the lack of time history?**\n\nWe found that adding more time history resulted in better offline prediction accuracy but worse real-time performance. See the global response for detail.\n\n\n\n**It would be good to report the total number of parameters for each network, and potentially the time-per-forward-pass.**\n\nWe propose adding the following line to results at line 198, “The total number of neural network decoder parameters varied from 479k for the LSTM to 923k for the TFM (see supplemental results).” And adding a table of parameter counts to the supplement:  “GRU – 639k, LSTM – 479k, FNN – 821k, TFM – 923k, KF – 10k”\n\nAll networks performed one decoding step in less than 2ms, such that this processing time is not a limiting factor even if faster screen refresh rates were used (than the 32ms update rate used here).\n\n\n\n**Figure 2: It would be nice to also show trial time in addition to bitrate since it is an intuitive metric.**\n\nWhile bitrate is not as intuitive as trial time, it takes into account individual trial difficulty as measured by the distance from finger to target at the start of the trial, for a single value that describes performance. However, we propose giving an example of trial time in the results text, line 221, “Figure 2d shows example online movements using a 2-DoF LSTM, where the monkey quickly moved each finger group and then held at the target position, for a median time of 1.3 sec per trial.”\n\n\n\n**Could the comparison of decoders at simulated high channel counts be done by subsampling the experimental data?**\n\nIn this analysis we aimed to get an idea of relative decoder performance at higher channel counts. One could imagine taking linear combinations of existing real channels to generate more artificial channels, however, this does not add information. Without introducing more channels with new information (like we can do in simulation), it’s unclear if this technique would be suggestive of decoder performance at high channel counts.\n\nAlthough the analysis is slightly tangential to the other results, it provides context about how this result may look with future high channel count systems. Here, we found RNNs to outperform other networks, but with future systems this might not be the case.\n\n\n\n**Is there a way to more direct way to investigate the dependence of the RNN decoders on inputs vs autonomous dynamics?**\n\nThis is an interesting question that we’ve begun to investigate. As you mention, we can compare the magnitude of the external vs recurrent inputs over time and conditions. We can also create stronger or weaker dependence on dynamics by adding a loss function penalty, and measuring the resulting effect on closed-loop control. Here we focused on functional examples of different decoder architectures, and in future work we plan to explore RNN dynamics more thoroughly.\n\nWe propose adding a line to the discussion, line 353, “Future work may investigate the relative dependence on inputs vs autonomous dynamics of RNNs, and their effect on online performance.”\n\n\n\n**Section 3.4: It would be useful to put these results in a figure (and potentially place it in the supplement if there is no room in the main text).**\n\nWe propose adding a figure to the supplement with this data, as shown in the attached global response pdf.\n\n\n\n**It would be helpful to add a legend to the supplemental videos.**\n\nWe will add a legend in the final version.\n\n\n\n**The impossibility of exploring the full space of network architectures within each class (RNN vs CNN vs transformer) and what that means for the conclusions about which is 'best'.**\n\nWe plan to add a discussion on the difficulty of exploring the full parameter space, as noted in the global response.\n\n\n\n**Thoughts about the generalizability of the results to more complicated tasks.**\n\nWe address this question in the global response and propose an addition to the Discussion.\n\n\n\n**Would the authors expect RNNs to also be superior for BMI in brain regions that rely less on autonomous recurrent dynamics during natural behavior?**\n\nIt is unclear whether feedforward or recurrent models may be superior in brain regions with weaker autonomous dynamics (meaning the region is more feedforward and/or input driven). One might expect feedforward models to predict continuous outputs with similar accuracy to RNNs since there is no benefit to internally modelling the neural dynamics. However, with relatively few or noisy neural channels, there is still the need to infer missing information about the output. In this case, a model (such as an RNN or a KF) that models realistic continuous movements and maintains a state may be more accurately “denoise” the output predictions. For example, a Kalman filter that incorporates a model of kinematics or physics can often outperform a simple linear regression. Further study is needed to investigate this and is out of scope for the current study.\n\n"}]

Author rebuttal:
We thank the reviewers for their support of this work and their helpful questions and suggestions. Our work demonstrates state-of-the-art control of a finger brain-machine interface, where we show the importance of closed-loop decoder testing and benefits of decoder memorization. The majority of reviewer comments were clarification questions and suggestions for further discussion, which we have addressed both here and in the individual comments.  

**Amount of Time history for Feedforward Models**

Multiple reviewers mentioned that the convolutional and transformer models may have lower performance due to using a time window of only 5 bins (160ms). For the transformer, during initial testing we found that using longer context windows of 10-20 bins (320-640ms) resulted in higher offline correlations, but the decoder was unusable during online control (the decoder worked for 1-2 trials before becoming stuck and was unable to move). Using a shorter window (~5 bins) resulted in better online control where the decoder did not get stuck or biased. We found a similar result with the convolutional network which had maximal closed loop performance with ~160ms of history and performance dropped as more history was added. This suggests that the feedforward models may tend to overfit to the hand-control training data and pay too much attention to earlier bins, showing the benefits of closed-loop testing. These problems could potentially be mitigated by training on more erroneous/imperfect movements or discouraging attention to earlier bins, but further experiments are needed. 

We propose adding the following paragraph to the discussion (line ~341):
“In additional testing we found that using more time history in the FNN and TFM decoders (up to 20 bins/640 ms) resulted in poor online control. This is likely due to overfitting to training data that lacks many of the corrective movements present in online control, and demonstrates how offline analyses can fail to predict the performance in the closed loop setting. Interestingly, the RNNs were trained with longer history but did not display this overfitting. Future work may investigate methods of using longer training sequences while limiting overfitting, for example by discouraging attention to early bins.”

**Generalizability to more complex tasks**

Multiple reviewers suggested discussing the ability of a “memorized” decoder for a simple task to generalize to more DoFs or other tasks, for more naturalistic BMI. First, we note that prosthetics and human interfaces can be highly functional with only 2-3 degrees of freedom. For example, a prosthetic hand with control of the index/thumb “pinch” and MRS fingers open/close can form many different hand grasps, or a 2d cursor can control a computer.

Our results suggest that the level of memorization can be controlled within each DoF independently (results 3.3) and that as more DoFs are added, slightly more training data or channels are needed to maintain accuracy (supplemental fig 3). Thus, we hypothesize that higher DoF tasks will need more data but can be trained and tuned using the same approaches used here. In future BMIs with more channels, the decoder may have high accuracy for fully continuous control without the need for memorization, for a larger output space and more naturalistic control.

We propose adding the following text to the discussion, line 353:
“As BMIs extend to more DoFs, our results suggest that more data or neural channels will be needed to maintain accuracy, and the degree of memorization can be controlled for each DoF independently.”


**Simulated Data Methods**

We propose adding the following methods section to better describe the simulated data. This information was in the supplemental methods but not referenced in the main text:

“For some analyses we created simulated offline datasets of a virtual user performing the same target acquisition task. The goal of these simulations was to test the relative impact of amount of training data, number of DoFs, and number of inputs on decoder performance, rather than measuring absolute performance. The simulated user moved with a velocity proportional to the distance to the target along each DoF. Artificial neural channels were generated such that each channel had a random relationship with the position, velocity, and acceleration at each timestep, as suggested in Trucollo et al. 2008. Further details and specifics can be found in Supplementary Methods.”



**Limitations – inability to explore the full parameter space**

Based on multiple reviewers pointing out limitations, we propose adding the following paragraph to the discussion:

“There are several limitations to the present study of closed-loop BMI performance. First, we did not explore every possible decoder architecture. More optimal architectures likely exist, but here we found several key factors of each class (for example, the need for limited time history of feedforward models, and the ability of RNNs to strongly learn few movements). Also, only one monkey was used for the online analyses; additional users may have different control strategies or decoder errors.”  


Meta-review: A comprehensive comparison was made for BMI decoding and finds great performance of RNNs. Many engineering challenges were tackled to enable a high performance system. Interesting observations were made, as one reviewer summarizes nicely, “…varying the number of discrete targets allows to interpolate between the behavior of a classifier and a continuous movement decoder, where for few targets, most of the actual movement to the target is memorized and with more and more targets, more of the movement is decoded…” This paper is of potential interest to a broad audience of NeurIPS.
Decision: Accept (spotlight)