Explainable Machine Learning Predictions for the Long-term Performance of Brain-Computer Interfaces
Authors: Morgan E Urdaneta
Nicole C Veit
Renae G Burke
Ian G Malone
Kaleb E Smith
Kevin J Otto
Venue: Submitted to ICLR 2023 (2023)
Keywords: SHAP
Explainability
feature importance
BCI
Neural interfaces
longitudinal intracortical recordings
neural engineering

Abstract: Brain computer interfaces (BCIs) can decode neural signals to control assistive technologies such as robotic limbs for people with paralysis. Neural recordings from intracortical microelectrodes offer the spatiotemporal resolution (e.g., sortable units) necessary for complex tasks, such as controlling a robotic arm with multiple degrees of freedom. However, the quality of these signals decays over time despite many attempts to prolong their longevity. This decrease in long-term performance limits the implementation of this potentially beneficial technology. Predicting whether a channel will have sortable units across time would mitigate this issue and increase the utility of these devices by reducing uncertainty, yet to-date, no such methods exist. Similarly, it would be useful to understand how variables like time post-implantation, electrochemical characteristics, and electrode design impact the long-term quality of these signals. Here, we obtained longitudinal neural recordings and electrochemical data from freely behaving rats implanted with a custom designed microelectrode array with varying site areas, shank positions, and site depths. This dataset was used to develop an explainable artificial intelligence pipeline that predicts with high accuracy the presence of sortable units on a given channel and elucidates the most important factors leading to these predictions. Our pipeline was able to predict whether a channel will be active with an AUC of 0.79 (95% C.I. 0.73–0.86) on unseen data. The most important features of the model were experimental subject, time post-implantation, and a channel’s previous spike metrics. Electrode site depth was the most important electrode design variable. Our results demonstrate the feasibility of implementing explainable artificial intelligence pipelines for longitudinal BCI studies and support previous reports on how factors like time, inter-animal variability, and cortical depth impact long-term performance of BCIs. These results are an important step forward in improving efficient decoding performance and guiding device development, which stand to advance the field and benefit the lives of human BCI patients.

Under review as a conference paper at ICLR 2023
EXPLAINABLE MACHINE LEARNING PREDICTIONS FOR THE LONG-TERM PERFORMANCE OF BRAIN COM- PUTER INTERFACES
Anonymous authors Paper under double-blind review
ABSTRACT
Brain computer interfaces (BCIs) can decode neural signals to control assistive technologies such as robotic limbs for people with paralysis. Neural record- ings from intracortical microclectrodes offer the spatiotemporal resolution (e.g.. sortable units) necessary for complex tasks, such controlling robotic arm with multiple degrees of freedom. However, the quality of these signals decays over time despite many attempts prolong their longevity. This decrease in long-term performance limits the implementation of this potentially beneficial technology. Predicting whether a channel will have sortable units across time would mitigate this issue and increase the utility of these devices by reducing uncertainty, yet to our knowledge, no such methods exist. Similarly, would be useful to under- stand how variables like time post- most-implantttion, electrochemical characteristics, and electrode design impact the long-term quality of these signals. Here, we ob- tained longitudinal neural recordings and electrochemical data from freely behav- ing rats implanted with custom designed microelectrode array with varying site areas, shank positions, and site depths. This dataset was used to develop an ex- plainable machine learning pipeline that predicts with high accuracy the presence of sortable units on a given channel recorrdddd and elucidates the most impor- tant factors leading to these predictions. Our pipeline was able to predict whether channel will be active with an AUC of 0.79 (95% C.I. 0.73-0.86) on unseen data. The most important features of the model were experimental subject, time post- implantation, and a channel's 's previous spike metrics. Electrode site depth most important electrode design variable. Our results demonstrate the feasibil- ity of implementing explainable machine learning pipelines for longitudinal BCI studies and support previous reports on how factors like time, inter-animal vari- ability, and cortical depth impact long-term performance of BCIs. These results are important step forward in improving efficient decoding performance and guiding device development, which stand to advance the field and benefit the lives of human BCI patients.
the
INTRODUCTION
Decoding neural signals through brain computer interfaces (BCI) can improve quality of life for people with paralysis. Every year, approximately 20,000 patients suffer from spinal cord injury in the US alone (Sekhon & Fehlings 2001). Most of these cases result in tetraplegia, causing paralysis from the neck down (Sekhon & Fehlings 200I). J. By interfacing directly with the brain, BCIs can be used as assistive technologies for these patients. Neural signals can be decoded and used to move computer cursors (Hochberg et al., 2006), control robotic limbs (Hochberg et al. 2012), and enrich exoskeleton (Benabid et al. 2019) and spinal cord stimulation technologies (Capogrosso et al.) (0166) that aim to restore locomotion. Neural signals can be recorded using variety of interfaces that vary in their degree of invasiveness (Eisinger et al. 2018), Recording signals directly from the brain cortex using implantable microelectrodes enables isolating spikes from individual neurons (single units). This level of precision is important for complex implementations such controlling a robotic arm with multiple degrees of freedom (Lebedev Nicolelis) (017) and has led to the creation of companies like Elon Musk's Neuralink (Muk]]2019)

---

Under review as a conference paper at ICLR 2023
One challenge for intracortical BCIs is the long-term stability of the neural signals. The ability to record sortable units gradually decreases over time, compromising decoding performance due to reduced quality of the recorded neural signals (Williams et al. 1999 Colachis et al.] [021] Downey et al. 2018). Potential solutions, such as replacing the electrode, might require a second surgery, hindering the feasibility and clinical implementation of BCI technologies. There have also been attempts to mitigate this decline in stability by using drug delivery systems, electrode coatings, and new electrode materials (Colachis et al., 2021), but the problem persists. To date, we still do not fully understand how these factors, and others such as the electrode design, inter-animal variability, or electrochemical features, might play role in the stability of these signals. Predicting when stability of neural signals will decrease and understanding which features of the interface contribute to this, would benefit clinicians and researchers and improve utility of intracortical BCIs.
Classical statistical techniques that focus on the individual impact of each of these factors may be ineffective due to the size and complexity of many BCT datasets. These rich datasets may benefit from more advanced machine learning (ML) techniques. In recent years, ML has been used used in medicine for the prediction of disease onset (Fleuren et al., 2020; Yahaya et al., 2020) and for the analysis of neurotherapeutics such as Deep Brain Stimulation (DBS) for Parkinson's Disease (Per- alta et al.] 2021). However, to our knowledge, no ML studies have been applied on the longitudinal stability of intracortical BCIs. In addition to predicting when a recording channel will be active (has sortable units), applying MI. explainability tools could also shed light on the most important factors involved in these predictions, such as time post-implantations electrochemical characteristics, and electrode design variables.
Here, we obtained longitudinal neural recordings and electrochemical data (voltage transient, impedance spectroscopy) from freely behaving rats implanted with a custom made intracortical microelectrode (Fig. [. This device has 16 channels with varying shanks, electrode-site areas, and cortical depths. We only selected features that were available prior to a given recording session and that are commonly available in clinical BCT studies. Following offline spike sorting, we developed an explainable MI. pipeline that predicts (pre-recordings) whether channel will be active and elu- cidate the most important features leading to these predictions (Fig. D. We bypassed the complex- ity/interpettaiility tradeoff of many black box ML models by using gradient boosting (Friedman 2001). This non-parametric model offers high accuracy on tabular datasets (Shwartz-Zir & Armon, 2022) and is still interpretable (Lundberg et al. 2018). Through the of Shapley additive expla- nations (SHAP), a method derived from game theory (Lundberg & Lee (017), we obtained local and global explanations that shed light on the most important factors for individual predictions, and the model as a whole, respectively. Similarly, through the use of decision paths, we elucidated how feature importance changes depending on the presence or absence of sortable units chronic timepoints. Overall, these findings demonstrate the feasibility of predictive modeling and ML ex- plainability tools for the longitudinal performance of BCIs and similar neurotechnologies. Imple- mentation of these tools can guide the design of future BCI studies (including non-intracortical BCI experiments) as well as the manufacturing of novel intracortical microelectrodes. Understanding the most important factors affecting the long-term stability of BCIs and predicting when channel is ac- tive has the potential to propel forward the translatability of these assistive technologies for millions of patients with paralysis.
2 METHODS
2.1 OVERVIEW
Following the collection of intracortical recordings and electrochemical characteristics over a period of 15 weeks with a custom-made microelectrode (Fig.[) []. we developed explainable ML pipeline that relied on gradient boosting and Shapley additive explanations. Importantly, all input features of the model are available prior to the recording session (Fig. I and are routinely collected in clinical BCI studies (i.e. impedance). This pipeline not only predicted with high accuracy whether a channel would have sortable units, it also elucidated the most important features leading to such prediction.
2

---

Under review as a conference paper at ICLR 2023
Implantation
Chronic data collection
Pre-Procee Processing
Noural recordings
N-8
Cuslomm-madee Intracorticl
For 15 weeks measure:
inioreeledrroe
Electrical
Filtering
Somatnoensory
Spike sarting (with commersill softwaro)
Neural Recordings
Electrode-deeign variables:
Shank Number
area (jmn))
Site depth (um)
Voltage Transient
Eoo
Sortable unit
2402
503
elsctroda-slt a sortable unit
700
21000
Active Channel
Exploratory Data Analysis
Model Development
ML Explanability
Pre-recordings
Use only festures available prior neuraal rocordings
SHAPley Additive exPlanations
channel will be active
Global Explanations
Propation aecte
0.4
Importance
Train models
subjects
Sucject
0.2
Using Leave one out CV
Local Explenations
0.1
Classification Models
Logasic Regression
600
2400
X00 203 710 90
K-nearcst neigi neighbars
Decision Trees
Decision Plots
(posll
boesting
Examd Viseaton
chronic timepoints
SSt
Figure 1: Data collection and Explainable Machine Learning Pipeline. Implantation: A silicon mi- croelectrode with custom design was implanted in the S1 of rats. The clectrode-sites of this device are combination of are shank, area, and depth. Chronic data collection: electrochemical features and neural recordings were obtained for up to 15 weeks instiimplantation. Pre-processing following offline sorting, channels that had sortable units were deemed active. EDA: Percentage active sites across electrode design variables. Model development: in order to predict whether channel will be active prior to the recording session, only features available prior to the collection of neural record- ings were selected, such as the channel's previous sortable unit metrics. The development set used to train each classifier was done using leave-one out cross- pross-valddation and performance was evaluated on external validation subject. ML Explainability: SHAP values computed to determine the most important predictors for the model, on chronic timepoints, and for individual trials.
2.2 SURGICAL IMPLANTATION AND CHRONIC MEASUREMENTS
A custom silicon microelectrode array with variable site sizes distributed across 4 shanks de- signed for this study (NeuroNexus, Ann Arbor, MI). Following sterilization with ethylene oxide, this device inserted into the somatosensory cortex adult Sprague Dawley rats using a micro-insertion system, resulting in a maximum electrode site depth of 900 μmm (see: Fig. D. All surgeries were carried by the same surgeon. The first 5 subjects were implanted within period of weeks. The sixth animal was implanted a month afterwards and was used as the external validation set in our machine learning pipeline (Fig. Specifics of the surgical procedure are explained elsewhere
(Urdaneta et al. 2021
Following implantation, neural recordings, electrical impedance spectroscopy (EIS), and voltage transients (VT) were measured for 15 weeks (Fig. [T. First, neural recordings from all 16 channels were taken for minutes at a sampling rate 24 kHz on freely behaving animals using a PZS amplifier (Tucker Davis Technologies, Alachua, FL). Immediately after, EIS 15 mV peak-to-peak measurements were obtained using logarithmic frequency sweeps from 10 Hz to 100 kHz using a PG-STAT- 128N Potentiostat (Metrohm, Utrecht, Netherlands). Lastly, VT was measured using μu symmetric waveforms with a phase duration of 50 μs using a IZ-32 stimulator (Tucker- Davis Technologies, Alachua, FI.) (Saldanha et al. 2021).

---

Under review as a conference paper at ICLR 2023
2.3 VARIABLES AND FEATURE ENGINEERING
To determine whether channel was active (presence of sortable units), raw neural signals were sorted for spike waveforms using Offline Sorter (Plexon Inc. Dallas, TX) (Fig. D. This supervised method was chosen over automatic alternatives since it allows for easy artifact removal (typical of recordings in freely behaving animals) and has been widely used in the field (0 (O'Doherty et al.. 2009). Only channels with spike waveforms that passed the inspection of two blind reviewers and had more than 200 spikes, were considered active. Importantly, our threshold for classifying channel as ac- tive was solely dependent on the presence of a sortable waveform. other words, having a sortable unit in channel does not imply that the extracellular recordings are coming from single, individ- ual neuron. Spike metrics (SNR, amplitude, and spike count) were calculated automatically with the Offfine Sorter software (Plexon Inc. Dallas, TX). Overall, 52. 1% of all channels across the du- ration of the study were active. Several features were engineered to potentially augment the model's predictive power. First, we obtained the electrochemical (EIS and VT) and spike metrics of the first recording session that had a sortable unit for a particular channel ("First S.U."). These spike metrics were amplitude, signal-to-noise ratio (SNR), and spike count. In the case of intracortical microstim- ulation (ICMS), metrics such as the amplitude of the first detection threshold, serves as a good proxy for the overall chronic performance of that channel (Urdaneta al. 2022). Therefore, obtaining the metrics of the first time the channel was active might be informative for the model. Similarly, we obtained electrochemical and spike metries of the previous experimental session. To avoid potential data leakage, features such as "week of previous active channel" were dropped. Electrochemical features (EIS and VT) were imputed using last observation carried forward (LOCF) imputation. Feature scaling was performed for the K-nearest neighbor (KNN) and logistic regression algorithms using min-max normalization (Patro & Sahu, 2015). However, given that decision tree models are insensitive to data variance (Dietterich & Kong 1995) normalization was not performed for either gradient boosting and decision tree classifier models. Lastly, cross-coree cross-correlate variables (i.e., day week vs. month post- cosplimmlantttion) were removed, while some highly correlated variables such as different impedance frequencies were left untouched for the explainability pipeline. Fig. 4 and table 2 (appendix) show a correlation matrix with the final list of features, and descriptions of those features, respectively.
2.4 MODEL DEVELOPMENT
In order to capture the nonlinear interactions from this type of dataset used gradient boosting. con-aarmettic ensemble model that uses gradient descent in the function space FFiedman (2001). We compared the performance of gradient boosting with baseline classifiers. Specifically, we used a KNN classifier with 5 neighbors and uniform weights, a Gini entropy decision tree classifier, and a logistic regression with ridge penalty. All these models were implemented from the Scikit-Learn library Pedregosa et al. (2011). We used XGBoost (extreme gradient boosting), an implementation of gradient boosting known for its improved speed (Chen & Guestrin 2016) as well as its ability to outperform even complex deep learning models on tabular datasets (Shwartz-Zn) Ziv & Armon 2022). Hyperparameter optimization was performed using Optuna, an automatic search space for tuning hyperparameters based on dynamically constructed spaces (Akiba et al., 2019). The final hyperpa- rameters for our gradient boosting model were: number of estimators: 1190, learning rate 0.003, subsample = 74.8%, and a depth-wise growth policy. To take into account the performance het- erogeneity observed across subjects in previous BCI studies (Williams et al. [9999] Downey et al.) 2018), and to better generalize planted subjects and an external validation set with the last-implanted animal. All models were trained using leave one out cross-validation (Kearns & Ron 1997) the development set, and its performance was compared to the external validation set.
unseen data, we created development set with the first 5 im-
2.5 MODEL. PERFORMANCE
We used different metrics to evaluate the performance of each classifier across the development and external validation sets. Accuracy and AUC-ROC (area under the ecceiving-operatiog characteristics curve) were used as performance metrics. Given that the dataset is slightly imbalanced, 52. % for the positive (channel active = 1) class. We decided to also use metrics that performed well in imbalanced datasets. For this we used both precision and recall, well as the F1 score, the harmonic
4

---

Under review as a conference paper at ICLR 2023
mean between precision and recall. Furthermore, each dataset bootstrapped (resample with replacement) 1000 times to obtain 95% confidence intervals for each metric.
2.6 EXPLAINABLE MACHINE LEARNING
The last step of our explainable ML pipeline was the use of SHAP (Lundberg & Lee 2017), a concept derived from game theory (Charnes et al.) 1988). This method quantifies how predictions for a particular model are affected after the removal of an individual feature an iterative fashion (Lundberg & Lee 2017). This way SHAP assigns a relative importance value for each feature individual predictions. Each SHAP value Φi is given by:
[S]((M - 1S1 1)! M!
Σ SEN(E
(LSSU f=(S
where S is the set of non-zero indexes, M is the number of input features, and N is the set of all input features (Lundberg & Lee 2017). Our gradient boosting model was passed through a Tree Explainer (Lundberg al.) 20188) in order to obtain the global importance of the model, individual instances, and decision paths.
3 RESULTS
3.1 THE PROPORTION OF ACTIVE CHANNELS MONOTONICALLY INCREASES WITH ELECTRODE SITE DEPTH
We calculated the proportion of active channels (channels with sortable units) for each distinct fea- ture (area, shank, depth) our custom-designed intracortical microelectrode (Fig. | exploratory data analysis). The results of this analysis showed that channel depth had a monotonic increase in the proportion of active channels as a function of cortical depth. Specifically, there were 28.4% more active channels for sites 900 um from the cortical surface (134) than 300 μm from cortical surface (96). Proportionally, only 41.7% of superficial sites (300 μm) were active compared to 60.6% of deep sites (900 jm). In contrast, the percent difference in the number of active sites between the largest site-area (21,600 um?) (105) and the smallest site-area (800 μm²) (118) was only 11.01 %. The site area with the highest proportion of active sites 7200 μm² the second largest area, with 57.5%. Lastly, the proportion of active sites for the most anterior and posterior and most shank was similar, with 46.5% and 47.9% for positions 1 and 4, respectively. The shank position with the highest proportion of active channels was the second most anterior (position 2) with 60.6%. Over- all, compared to site-size and shank position, electrode site-depth had a monotonic increase in the proportion of active sites as these channels go deeper into the cortical surface.
3.2 GRADIENT ROOSTING CAN PREDICT WITH HIGH PERFORMANCE THE PRESENCE OF ACTIVE CHANNELS
Table Predictive modeling performance across classifiers. Numbers in parentheses represent 95% CI from 1,000 bootstrapped iterations.
MODEL
SUBSET
ACCURACY
AUC-ROC
F1 SCORE
PRECISION
RECALL
Gradient Boosting
(0.68-0.79)
(0.67-0.81)
0.64 (0.54-0.72)
0.73 (0.63-0.84)
0.63 (0.52-0.72)
Gradient Boosting
ext. val.
0.73 (0.68-0.79)
0.79 (0.73-0.86)
0.79 (0.74-0.84)
0.83 (0.77-0.89)
0.76 (0.69-0.83)
Log. Regression
test
0.69 (0.63-0.75)
0.72 (0.64-0.80)
0.62 70)
0.66 (0.56-0.76)
0.65 (0.55-0.75)
Log. Regression
ext. val.
(0.3-0.42)
0.49 (0.44-0.54)
0.07 (0.03-0.12)
1.00 (1.00-1.00)
0.04 (0.01-0.06)
KNN
0.62 (0.56-0.68)
0.59 (0.50-0.88)
0.54 (0.46-0.62)
0.55 (0.46-0.64)
0.60 (0.49-0.71)
KNN
ext. val.
0.37 (0.30-0.43)
0.58 (0.51-0.644
0.12 (0.06-0.18)
0.88 (0.67-1.00)
0.07 (0.03-0.10)
Decision Tree
test
0.69 70.64-0.76)
0.65 (0.58-0.73)
0.60 (0.50-0.68)
0.65 (0.53-0.77)
0.58 (0.48-0.7)
Decision Tree
val.
0.66 (0.60-0.72)
0.66 (0.61-0.72)
0.73 (0.67-0.78)
0.79 (0.73-0.86)
0.67 (0.60-0.75)
Table sshows the results our predictive modeling across different metrics for different classifiers. The average accuracy on the holdout test of the development set 0.69 (95% CI: 0.63-0.75) for
5

---

Under review as a conference paper at ICLR 2023
logistic regression, 0.62 (0.56-0.68) for KNN, and 0.69 (0.64-0.76) for the decision tree classifier. On the other hand, the accuracy for the external validation was 0.36 (0.3-0.42) for logistic regression, 0.37 (0.3-0.43) for KNN, and 0.66 (0.6-0.72) for decision trees. In contrast, model had the best accuracy (0.73) for both the test and external validation set. Gradient boosting also had better overall performance among the test and external validation for other metrics. For instance, the test set had AUC-ROC and a FI score of 0.74 (0.67-088) and 0.64 (0.54-0.72 respectively, while the external validation set had AUC-ROC 0.79 (0.73-0.86) and FI score of 0.79 (0.74-0.84 Altogether, the non-pan oon-parmettic nature of gradient boosting allows us to predict with high performance when a channel will have sortable units. One of the advantages of gradi- ent boosting is that it is not as affected as other complex models in the typical trade off between performance and explainability (Lundberg et al., 2018). Hence, using SHAP's tree explainer, we decided to further investigate how gradient boosting reached these predictions in order to obtain global explanations and determine feature importance of this model.
gradient boosting
3.3 GLOBAL EXPLANATIONS INDICATE FEATURE IMPORTANCE
High
Subject
Time post-implantation
Channel's previous SNR
Channel's previous amplitude
0.3 kHz imp. magnitude
Voltage transient peak
Channel's previous spike count
E Electrode-site depth
Feature Importance
Channel's previous voltage transient peak
0.3 kHz imp. magnitude of first sortable unit
kHz imp. magnitude
Feature value
Voltage transient p2p
Channel's previous kHz imp. magnitude
Electrode design
5 kHz imp. magnitude
features
Eleectrode-ite shank
SNR of first sortable unit
Electrode-site area
Amplitude of first S.U.
5 kHz imp. magnitude of first S.U.
1 kHz imp. magnitude of first S.U.
Spike count of first S.U.
Vollage Transient p2p of first S.U.
Voltage Transient peak of first S.U.
Time post-implantation of first S.U.
Low
SHAP value (impact on model output)
Summary plot containing global explanations for the most important features across the study. Feature importance is represented in descending order, with the most important features at the top of the graph. SHAP values indicate the overall contribution of an individual feature, higher values push the prediction towards the channel being active.
Figure 2:
SHAP is a robust model-agnostic method to quantify the contribution of individual features for each prediction the model does (Lundberg & Lee, 2017). SHAP values are estimated by running a model with and without the feature in question while at the same time iterating the order in which the

---

Under review as a conference paper at ICLR 2023
feature is presented to reduce bias. This way, SHAP allows to elucidate the feature importance from otherwise black box models and obtain global explanations and elucidate the most important predictors (Lundberg et al., 2018). Fig. 2 shows beeswarm plot with the most important features for the prediction of active channels in our longitudinal recordings for BCI dataset. In this graph, positive SHAP values have positive impact on the model output, in other words, they drive the pre- diction towards a channel being active (having sortable units), while negative SHAP values push the prediction towards the channel not being active. Our results showed that Subject ID had the greatest impact in our model. Interestingly, the distribution of SHAP values for this feature shows bimodal distribution with some subjects having high SHAP values (red dots) while others having low SHAP values. This is consistent with the phenomenon of responders vs. non-responders observed across several BCI studies (Williams et al. 1999. Downey et al., 2018). Time post-implantation was the feature with the second largest impact in our model. Indeed, for the first half of the study 55.5% of channels were active while only 46.8% were active during the second half. The high feature values of Fig. 2 (dark red dots) had mostly negative SHAP values, suggesting that late timepoints tend to drive the prediction negatively (towards a channel not being active). This finding is consistent with the decay in the count of sortable units observed in chronic timepoints across the literature (Downey) iet al. 2018). Features based on the channel's ' previous sortable unit metrics (SNR, amplitude, and spike count) were among the top most relevant features, with previous SNR and amplitude being the 2nd and 3rd most important features, respectively. The channel's ' previous electrochemical fea- tures such as VT peak and and IkHz impedance magmitude, were ranked 9th and 13th for channels previous VT peak, respectively. These findings suggest that predictions rely heavily on the results of the previous experimental session, with channels with high SNR and amplitude in the previous recording having higher probability of being active. Electrochemical characteristics such as 0.3 kHz impedance and VT peak were the 4th and 9th most important features. For the most part, low impedance values and high VT peak values seem to push the prediction towards the channel being active. Indeed, the average 1kHz impedance magnitude for inactive channels was 554.6 = 468.3 s.d.and 480.7 = 399.6 s.d. for active channels. Being ranked 8th, electrode-sit depth was the elec- trode design variable that had the highest feature importance. Predominantly, red dots (light red: 700 μm, dark red 900 μm) had higher SHAP values, suggesting that deeper electrode-sites increased the probability of a channel being active. In contrast, both site shank and site area had similar con- tributions to the model but they are less important, with shank and area being the 15th and 17th most important feature in our model, respectively. Lastly, the electrochemical and spike metrics of the first sortable unit of a particular channel had an almost negligible importance 2 bottom). Many of the most important features of fig. 2 were consistently ranked among the top predictors across individual subjects and across classifiers. Collectively, the insights obtained from these findings, led us to further investigate how the model makes individual predictions on specific recording session.
3.4 LOCAL EXPLANATIONS CAN SHED LIGHT INTO INDIVIDUAL CONTRIBUTIONS FOR A PARTICULAR RECORDING SESSION
SHAP's local explanations allow us to elucidate how decisions are made for individual instances, in other words, how each individual feature contributed to reach prediction (Lundberg & l Lee| 2017] Lundberg et al. 2018). To contrast how local explanations change for an active and an inactive channel, we selected neural recording session on chronic timepoint (week 10) for one of the animals (subject 2) of the development test. Subsequently, we picked 2 out of 16 channels at ran- dom, one with sortable units and one without them. Figure (3a shows local explanations for channel 5, an active channel. The axis units are log of odds, with higher values increasing the probability of a channel being active. Features in red increase the probability of channel being active, while features in blue decrease it. Based on its magnitude (widest red bar), the most important feature driving the prediction positively was the previous SNR of the channel, which was 7.46. The other most important features in order of importance were the subject ID, the channel's previous ampli- tude, VT peak, and electrode-site depth. Conversely, time post-implantation was the most important factor driving the prediction towards a negative outcome. This is consistent with the observation of Fig. in which late timepoints drive the prediction towards the channel not being active. Altogether, the additive nature of these explanations leads to a log of odds of 4.98, which corresponds to a prob- ability of 0.99, in other words the model correctly classified this channel as active. On the other hand, Fig. Bp depicted local explanations for channel 10, channel with no sortable unit. The most important factors pushing this instance towards a negative prediction were 0.3 kHz impedance mag- nitude, time post-implantation, and site depth. Interestingly, compared to the prediction of channel
7

---

Under review as a conference paper at ICLR 2023
Suboed
Week 10
Log odte DO Priss of eechiiiiddee chermed 4.99
vilue
0.90
Poot-impeentdoon
Channel
Red
Prob active charnel
Charnel's
postinpporaate
Chorne's preious amplitude
Volage temaunt pusi 1,300 (V)
Sortatle
Active Channel
B
Log oods fo) Prob, ofoovi cianacl
-1.73
0.16
bose
Charnel
Prob. axtivv channe
Subject
kite
2191 1
Phewoze SPRC +
Dedtrdd site depth 300 (am)
Votage vonsinnt posk 1:235
Time www.irpaaaatto....o (wooks)
No scrtidii Active Channel @
10-D ringilade in 10P1 (7)
Figure 3: Local explanations depicting individual predictions for an active and an inactive channel for single session of neural recordings. A) Local explanations for a channel with an active channel. Red features pushing the probability of channel being active are more predominant Local explanations for a channel with an inactive channel. blue features, decreasing the probability of channel being active, more prominent. The final prediction in the form of log of odds and probability are present at the top of each graph.
this plot. B)
5 (channel with sortable unit) where site depth influenced the prediction positively, in this case, site depth had the opposite effect. This can be explained by the fact that the depth of channel was 700 μm, compared to 300 um for channel 10, the most superficial channel. As reported in Fig. 2 superficial channels predominantly had negative SHAP values, meaning that superficial channels (such channel 10) tend to drive the prediction towards not having sortable unit. The final pre- diction for channel 10 (Fig. (3 had log of odds of -1.73, which corresponds to probability of 0.15, in other words the algorithm correctly classified this channel as not active (0). The results of Fig. 3 showed that feature importance varied substantially from positive to negative predictions in just two instances. This raised the question: how does feature importance change across positive and negative predictions for more than two individual instances?
3.5 FEATURE IMPORTANCE CHRONIC TIMEPOINTS CHANGES DEPENDING WHETHER CHANNEL WAS ACTIVE OR NOT
To assess this question, we used decision plots for every negative and positive prediction for the second half of the longitudinal study. Similar to force plots (Fig. 33. decision plots can represent prediction paths for multiple individual predictions, allowing us to individually rank the most im- portant features for active and inactive channels (Fig. 5 The x-axis is in log-odds, with higher values representing a higher probability that the channel active. Each vertical line represents a single recording session for channel. By following the path of individual lines across different features, we can determine how the model reached an individual prediction. Similar to Fig. 2 fea- ture importance is sorted in descending order. Figure (Appendix) shows decision plot for all inactive channels after the second half of the study. The most important factors leading to a negative prediction were subject ID. time post-implantation, channel's previous spike metrics (SNR, ampli- tude, spike count), and electrode-site depth. On the other hand, Fig. 55p shows individual prediction paths for active channels. In this case, most important feature was the channels' previous SNR, followed by subject ID, and 0.3 kHz impedance magnitude. We saw shift in the importance of time post-implantation. This factor went from the second most important feature for inactive channels (Fig. 55) to the fifth most important feature for active channels (Fig. 5p). This shift in importance suggests that time ppplaaamlanttoion has higher impact in the prediction of inactive channels, con- sistent with the observations of figures 2 and 3 Regarding features related with the design of the electrode, only electrode-site depth had relatively high importance for negative and positive predic- tions, 6th, and 8th, respectively. Other electrode design variables (site-area, and shank) were out of
8

---

Under review as a conference paper at ICLR 2023
top 10 most important features. Overall, analyzing decision paths for multiple predictions, help us elucidate intrinsic differences in feature importance leading to the prediction of sortable units for BCIs.
4 DISCUSSION
In this study, we recorded neural signals from rats chronically implanted with a custom- deeigndd intracortical microelectrode. In order to better understand the long-term stability of these signals for BCIs, we developed an explainable ML pipeline to predict with high performance when a partic- ular channel was active and to elucidate how different factors influenced these predictions. Using gradient boosting, our model had an AUC-ROC of 0.79 (95% C.I: 0.73-0.86) predicting whether a channel was active unseen data. Through the use of SHAP, our pipeline identified that subject ID and time imslaaapaanttnonn were among the two important features in the model. This was followed by the spike metrics of the previous recording session. Importantly, out of the electrode design vari- ables of our unique device, electrode-site depth was the most important, ranking 8th most important feature overall. Moreover, through the use of local explanations and decision plots, we were able to elucidate how the model made predictions on chronic timepoints for individual instances and how different feature importance varied based on whether the channel was active not. For instance, time post- cost-implantation played more significant role in predicting whether a channel was not active than active. The significance and novelty of this study is two-fold: 1)
unique longitudinal
dataset with a custom-designes microelectrode array. 2) To our knowledge, this is the first time predictive modeling and/or MI. explainability tools have been applied to a chronic BCI dataset.
Our study had a few limitations and constraints. First, we limited our longitudinal study to 15 weeks only. Although this is a common timeframe for rats (Saldanha et al.) [02T]. human BCI studies might last several years (Hughes et al. 2021 Downey et al., 2018)) 8). However, animal and human recordings have shown similar trends over time (Downey et al. 2018: Williams et al. 1999). There- fore, this study might serve precedent for the implementation of ML tools on future human work. Another potential limitation is that for this study, we limited our dependent variable to the presence absence of sortable spikes (single units). Some modern BCIs might rely different neural signals using different filtering techniques. However, isolating single units remains an impor- tant metric for many BCIs (Lebedev & Nicolelis (011)) and other neuroscientific research (Buzsáki ett al., 2012). The feature importance of electrode-site area in our study might be limited to the design constraints of the microelectrode device. Even though our device had electrode-site sites with different areas ranging from 800 jm² to 21,700 pm? the width of the shank was held constant. Many studies have shown that reducing the thickness and width of the device to subcellular levels (< 10 um) can improve both the biocompatibility and chronic stability of its recordings (Luan et al.) 2017]. In conclusion, through the design of custom microelectrode device and the development of an explainable ML pipeline we were able to predict with high performance whether channel was active for BCI applications. This pipeline also allowed us to shed light on the most important factors in the model including the roles of inter-animal variability, time post-implantation, electrochemical features, and electrode design variables. Even though ML explainability tools such as the ones used in this study do not imply causation, our findings can inform the design of electrode manufacturing and future BCI studies. Similarly, this study can open the doors for the implementation of similar ML and explainability techniques the field of neural engineering. Combining high performance predictions with explainability tools can help both clinicians and researchers alike as well as improv- ing the long-term performance and wide implementation of BCI assistive technologies for millions of patients with paralysis.
ETHICS STATEMENT
This study involved handling, surgical implantation and experimentation of Oprague-Dawler rats. The study was reviewed and approved by the Institutional Animal Care and Use Committee (IACUC) of the last author's institution.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 2623-2631, 2019.
Alim Louis Benabid, Thomas Costecalde, Andrey Eliseyev, Guillaume Charvet, Alexandre Verney, Serpil Karakas, Michael Foerster, Aurélien Lambert, Boris Morinière, and Neil Abroug. An exoskeleton controlled by an epidural wireless brain-m interface a tetraplegic patient: a proof-of-concept demonstration. The Lancet Neurology, 18(22)11122-122, 2019. ISBN: 1474- 4422 Publisher: Elsevier.
György Buzsáki, Costas A. Anastassiou, and Christof Koch. The origin of extracellular fields and Currents-EEEE ECoG, LFP and spikes. Nature reviews neuroscience, 13(6):407-420, 2012. Publisher: Nature Publishing Group.
ISBN: 1471-0048
Marco Capogrosso, Tomislav Milekovic, David Borton, Fabien Wagner, Eduardo Martin Moraud, Jean-Baptiste Mignardot, Nicolas Buse, Jerome Gandar, Quentin Barraud, and David Xing. A brain-spine interface alleviating gait deficits after spinal cord injury in primates. Nature, 539 (7628):284-288, 2016. ISBN: 1476-4687 Publisher: Nature Publishing Group.
A. Charnes, B. Golany, M. Keane, and J. Rousseau. Extremal principle solutions of games in char- acteristic function form: core, Chebychev and Shapley value generalizations. In Econometrics of planning and efficiency, pp. 123-133. Springer, 1988.
Tianqi Chen and Carlos Guestrin. Xghoost: A scalable tree boosting system. In Proceedings the 22nd acm sigkdd international conference on knowledge discovery and data mining, pp. 785-794, 2016.
Samuel C. Colachis, Collin F. Dunlap, Nicholas V. Annetta, Sanjay M. Tamrakar, Marcia A. Bock- brader, and David A. Friedenberg. Long-term intracortical microelectrode array performance in human: A 5 year retrospective analysis. Journal of Neural Engineering, 18(4):0460d7, 2021. Publisher: IOP Publishing.
ISBN: 1741-2552
Thomas G. Dietterich and Eun Bae Kong. Machine learning bias, statistical bias, and statistical variance of decision tree algorithms. Technical report, Citeseer, 1995.
John E. Downey, Nathaniel Schwed, Steven M. Chase, Andrew B. Schwartz, and Jennifer L. Collinger. Intracortical recording stability in human brain-computer interface users. Journal of neural engineering, 15(4):046016, 2018. ISBN: 1741-2552 Publisher: IOP Publishing.
Robert S. Eisinger, Morgan E. Urdaneta, Kelly D. Foote, Michael S. Okun, and Aysegul Gunduz. Non-motor characterization of the basal ganglia: evidence from human and non-human primate electrophysiology. Frontiers in neuroscience, 12:385, 2018. ISBN: 1662-453X Publisher: Fron- tiers Media SA.
Lucas M. Flcuren, Thomas Klausch, Charlotte L. Zwager, Linda J. Schoonmade, Tingjic Guo, Luca F. Roggeveen, Elconora L. Swart, Armand RJ Girbes, Patrick Thoral, and Ari Ercole. Ma- chine learning for the prediction of sepsis: systematic review and meta-analysis of diagnos- tic test accuracy. Intensive care medicine, 46(3):383 400, 2020. ISBN: 1432-1238 Publisher: Springer.
Jerome H. Friedman. Greedy function approximation: a gradient boosting machine. Annals of statistics, pp. 1189-1232, 2001.
ISBN: 0090-5364 Publisher: JSTOR.
Leigh R. Hochberg. Mijail D. Serruya, Gerhard M. Frichs, Jon A. Mukand, Maryam Saleh, Abra- ham H. Caplan, Almut Branner, David Chen, Richard D. Penn, and John P. Donoghue. Neuronal ensemble control of prosthetic devices by human with tetraplegia. Nature, 442(7099):164-17) 2006. ISBN: 1476-4687 Publisher: Nature Publishing Group.
Leigh R. Hochberg, Daniel Bacher, Beata Jarosiewicz, Nicolas Masse, John D. Simeral, Joemn Vogel, Sami Haddadin, Jie Liu, Sydney S. Cash, and Patrick Van Der Smagt. Reach and grasp by people with tetraplegia using a neurally controlled robotic arm. Nature, 485(7398):372-375, 2012. ISBN: 1476-4687 Publisher: Nature Publishing Group.
10

---

Under review as a conference paper at ICLR 2023
Christopher L. Hughes, Sharlene N. Flesher, Jeffrey M. Weiss, John E. Downey, Michael Boninger, Jennifer Collinger, and Robert A. Gaunt. Neural stimulation and recording performance in human sensorimotor cortex over 1500 days. Journal of Neural Engineering. 18(4):045012, 2021. ISBN: 1741-2552
Publisher: IOP Publishing.
Michael Kearns and Dana Ron. Algorithmic stability and sanity-check bounds for leave-one-out cross-validation. In Proceedings of the tenth annual conference on Computational learning the- ory, pp. 152-162, 1997.
Mikhail A. Lebedev and Miguel AL Nicolelis. Brain- machie interfaces: From basic science to neuroprostheses and neurorehabilitatin. Physiological reviews, 97(2):767-837, 2017. ISBN: 0031-9333 Publisher: American Physiological Society Bethesda, MD.
Lan Luan, Xiaoling Wei, Zhengtuo Zhao, Jennifer J. Siegel, Ojas Potnis, Catherine A. Tuppen, Shengqing Lin, Shams Kazmi, Robert A. Fowler, and Stewart Holloway. Ultraflexible nanoelec- tronic probes form reliable, glial scar-free neural integration. Science advances, 3()::1600 2017. ISBN: 2375-2548 Publisher: American Association for the Advancement of Science.
Scott M. Lundherg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017.
Scott M. Lundherg, Gabriel G. Erion, and Su-In Lee. Consistent individualized feature attribution for tree ensembles. arXiv preprint / /002.038000330000 03888, 2018.
Elon Musk. An integrated brain-machine interface platform with thousands of channels. Journal of medical Internet research, 21(10)=e11144 2019. Publisher: JMIR Publications Inc., Toronto, Canada.
Joseph E. O'Doherty, Mikhail Lebedev, Timothy L. Hanson, Nathan Fitzsimmons, and Miguel AL Nicolelis. A brain-machine interface instructed by direct intracortical microstimulation. Frontiers in integrative neuroscience, pp. 20, 2009. ISBN: 1662-5144 115-51151T
Publisher: Frontiers.
S. Patro and Kishore Kumar Sahu. Normalization: arXiv:1503.06462. 2015.
A preprocessing stage. arXiv preprint
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, and Vincent Dubourg. Seikit-learn Machine learning in Python. the Journal of machine Learning research, 12:2825-2833 201 1. Publisher: JMLR. org.
ISBN: 1532-4435
Maxime Peralta, Pierre Jannin, and John SH Baxter. Machine learning in deep brain stimulation: A systematic review. Artificial Intelligence in Medicine, 122:102198, 2021. ISBN: 0933-3657 Publisher: Elsevier.
Ramya L. Saldanha, Morgan E. Urdaneta, and Kevin J. Otto. The Role of Electrode-Site Placement in the Long-Temm Stability of Intracortical Microstimulation. Frontiers in Neuroscience, 15, 2021. Publisher: Frontiers Media SA.
Lali HS Sekhon and Michael G. Fehlings. Epidemiology, demographics, and pathophysiology of acute spinal cord injury. Spine, 26(24S):S2-S12, 2001. ISBN: 0362-2436 Publisher: LWW.
Ravid Shwartz-ZD and Amitai Armon. Tabular data: Deep learning is not all you need. Information Publisher: Elsevier.
Fusion, 81:84-90, 2022. ISBN; 1566-253:
Morgan E. Urdaneta, Nicolas G. Kunigk, Francisco Delgado, Shelley I. Fried, and Kevin J. Otto. Layer-specific parameters of intracortical microstimulation of the somatosensory cortex. Journal of Neural Engineering, 18550055007, 2021. ISBN: 1741-2552 Publisher: IOP Publishing.
Morgan E. Urdaneta, Nicolas G. Kunigk, Seth Currlin, Francisco Delgado, Shelley I. Fried, and Kevin J. Otto. The long-term stability of intracortical microstimulation and the foreign body response are layer dependent. Frontiers in Neuroscience, 16:908858, 2022. ISBN: 1662-453X Publisher: Frontiers Media SA.
11

---

Under review as a conference paper at ICLR 2023
Justin C. Williams, Robert L. Rennaker, and Daryl R. Kipke. Long-term neural recording charac- teristics of wire microclectrode arrays implanted cerebral cortex. Brain Research Protocols, (3):303-313, 1999. ISBN: 1385-299X Publisher: Elsevier.
Lamido Yahaya, N. David Oye, and Etemi Joshua Garba. A comprehensive review heart disease prediction using data mining and machine learning techniques. American Journal of Artificial Intelligence, 4(1):20 22.2009922200 2020. Publisher: Science Publishing Group.
6 APPENDIX
Active
Previous SNR
0.33
Previous amplitude
0.27
0.63
Pearson's Correlation
Subject
Electrode-sile sile depth
0.17 0.18 0.21 0.02
0.88
Previous vollage transient peek
0.6
Previous spike count
0.04 0.04 0.14 0.08 -0.00
0.4
0.2
Electrode-site area
0.00 .0.23.0.02
0.0
Electrode-site site shank
0.01 -0.06 0.02 0.04 0.06 -3.01 0..00 0.02
-0.2
Voltage transient
-0.10 0.00 0.03 0.07 0.24 0.02
-0.4
-0.6
Voltage transient p2p
-0.00 -00000.06.00 -0.01 0.83 0.07 07 -.001
Previous kHz imp. magnitude
0.11 11.00.006 -0.04 0.01 -0.11 0.07 01 -0.31 0.31--0.08 -0.08 0.06 0.07
Spike count of first S.U.
0.45
1 KHz imp. magnitude
0.13 0.04 +0.08
-0.01
+0.10
0.31
-0 02 0.33 0.09 0.05 0.07 0.12 0.05
0.39
kHz imp. magnitude
Voltage Transient p2p of first S.U.
-0.14 1044-0.17-40 19.00.9
0,10 0.00
-0.11 -0.10 -0.22 0.00 0.10 0.02 0.38 0.08 0.11
Voltage Transient peak of first S.U,
1.00
0.3 kHz imp. magnitude of first S.U.
0.16 116--0.16-00 0.18
0.57
0.14
0.00
00 0.06 00.27 0.08 0.09 0.12 0.33 0.15 0:20
0.78 0.76
0.3 imp. magnitude
0.17-0.02 -0.08-0.05 0.24 -0 04 -0.20 0.20-0..07 0.11 0.12 0.14 0.80
0.74
0.12
0.11
0.14
Amplitude of first S.U.
-0.06
0.04
0.88
Previcus SNR
Provicus ampiltute
Subject
Electrode site depth
Previous vottage tramelent pesk
Provious spike count
Electrode site aron
Elecirode site shank
Voltage transiont p3p
Previous kHtz matgnildd
Spike count of firet S.U.
kHtz imp. magnitido
kHz imp magnitude
Vollage Transient p2p of first S.U.
0.3 kHz imp. magnitude of first S.U.
0.3 kihtz magnitutic
Amplitude of first U.U.
Vottage tramslent
Vollage Transient peak
Figure 4: Cross-correlation matrix across features. Pearson's 's correlation value for each model vari- able and the binary dependent variable: active channel. The diagonal of this matrix (1) has been hidden for simplicity. S.U.: Sortable units.
12

---

Under review as a conference paper at ICLR 2023
Table 2: Model's input features: S.U. stands for sortable units
FEATURE
DESCRIPTION
UNITS
Subject
Subject ID
Int
Time post-implantation
Time after the animal implanted
weeks
Electrode-site depth
Cortical depth of the electrode site
um
Electrode-site shank
Shank position in the array (1 most anterior 4 most posterior)
Int
Electrode-site area
Size of the electrode-site
um?
Channel's previous SNR
Signal-to-noise ratio of the last S.U. particular channel had Count of spikes of the last S.U. particular channel had Peak-to-peak amplitude of the last S.U. particular channel had
Float
Channel's previous spike count
Int
Channel's previous amplitude
uV
Channel's previous kHz imp. mag.
kHz impedance magnitude of the previous experimental session Voltage transient peak of the previous experimental session Impedance magnitude frequency response at 300Hz
Ω
Channel's previous VT peak
μV
0.3 kHz imp. magnitude
Ω
kHz imp. magnitude
Impedance magnitude frequency response at 1kHz
5 kHz imp. magnitude
Impedance magnitude frequency response at 5kHz
Ω
Voltage transient peak Voltage transient p2p
Voltage transient peak of the current experimental session Voltage transient peak-to-peak of the current experimental session Week in which a particular channel had its first S.U.
μV
μV
Time post-implantation of first S.U.
weeks
SNR of first S.U.
Signal-to-noise ratio of the first S.U. registered for channel Peak-to-peak amplitude of the first S.U. registered for a channel count of spikes of the first S.U. registered for a channel Impedance at 300Hz of the first S.U. registered for a channel Impedance 1kHz of the first S.U. registered for channel Impedance 5kHz of the first S.U. registered for channel Voltage transient peak of the first S.U. registered for a channel
Float
Amplitude of first S.U.
uV
Spike count of first S.U.
Int
0.3 kHz imp. magnitude of first S.U.
Ω
kHz imp. magnitude of first S.U.
5 kHz imp. magnitude of first S.U.
Ω
Voltage Transient peak of first S.U.
μV
Voltage Transient p2p of first S.U.
Voltage transient peak-to- to-peak of the first S.U. registered for channel
μV
13

---

Under review as a conference paper at ICLR 2023
A
Proh. channel
Increased active channel
Subject
Channe is previcus SNR
Time post-implanation
Subject
Previcus BNR
0.3 kHz imp. magnituue Previous tmplitudo Time poet-inplenietion
Previous amplitude
Previous spike count
depth
Volage transient pask
Feature Importacce
magnituce
Feature Imporfance
Previous splike
magniude
Dedrode-sie
Vatage transient
Previous magnitude
SNR first S.U
magnitude
Previous NHz magnilude
Previous vollage transient
HHz mp. magnitude
0.3 kHt magnilude first S.U
Previcus vollage transient peak
kitte imp. magnitude
5 HHz imp. magnitude
Valtage transient pōp
Electrode site shank
Electoddesiie area
Ampitude of first sortable unit
Vollage pōp
Spike
Spiik
Amplitude first
magnitude
kftz magnitude first S.U
Model output value
Model output value
Figure 5: Decision plots on chronic time points for predictions of channels with and without sortable units. All individual trials of the second half of the study were considered for this analysis. Feature importance increases from bottom to top. A) decision plot showing prediction paths for every inac- tive site. The lower the model output value the lower the probability that channel has sortable unit. B) decision plot showing prediction paths for every active site. The higher the model output value the higher the probability that channel has sortable unit. S.U.: sortable units.
14


Reviews:
[{"review_id": "y9xY3TVmJa", "reviewer": "Reviewer_eCAt", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a new unique longitudinal dataset with a custom-designed microelectrode array implanted in 5 rats to develop models for predicting microelectrodes arrays dead channels and, more importantly, explain what factors are the most influential for predicting dead channels.  \n", "summary_of_the_review": "The idea presented in this paper is interesting and novel, but the model should be refined (i.e. by using different features for prediction and explaining the choice of features a bit more). The conclusions drawn from the model could potentially be much more informative using a different set of features. I hence don't recommend acceptance of this paper.", "strength_and_weaknesses": "The collected data is very interesting and useful for the proposed task (predicting dead channels and explaining what factors influence dead channels). The analysis is sound and well-conducted. \n\nHowever, I think that the claims made in the paper are a little too strong. \n\nThe claim, in the abstract, that \"Predicting whether a channel will have sortable units across time would mitigate this issue and increase the utility of these devices by reducing uncertainty, yet to-date, no such methods exist\" is not true. This has been done (see for example Spike sorting pipeline for the International Brain Laboratory, IBL, 2022), with a method predicting bad channels in Neuropixels probes based on similarity with neighboring channels. I agree that this is different from the work in this paper but it would be worth explaining the difference a little better. \n\nAlso, I think it would be straightforward to extend the author's analysis to high-density probe recordings where dead channels can be detected, to evaluate the accuracy of the prediction. \n\nI would have liked to see a schematic of the experiments, showing a diagram of the probe (it is hard to know what the probe looks like from the text), and also explain the features that are input to the model in more depth. I couldn't understand if the authors use all the spikes info, and how they calculate SNR, amplitude or magnitude for the channels. \n\nThe novelty of the approach is that the model predicts dead channels using features like Subject/time post implantation, that have a high SHAP value. However, lots of additional features could have been used and the selection of features seems bit arbitrary. Why use channel's amplitude and SNR? Having a model that solely predicts the dead channels using info that is not contained in the signal would be much more impressive and informative. Moreover, I think it should be easy to get higher accuracy when using signal info. \n\nFinally, it seems to me that \"time post-implantation\" is obviously important to predict dead channels, as well as \"Subject\" and it does not really inform us about the reason why a channel dies (i.e. Subject could mean many different things - such as a problem when implanting the probe or else), which make the conclusions a little less informative. \n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "nmW8Ycjwln", "reviewer": "Reviewer_Mjwm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper uses explainable machine learning methods to study which factors impact the number of active channels in a BCI. The authors analyse a number of important factors such a intersubject variability, time post-implantation, electrochemical properties and electrode design. The analysis relies on a reasonable dataset in mice. ", "summary_of_the_review": "While more analysis can be done, I think the paper is a nice contribution to an important BCI question. I am not sure how much this aligns with the theme of ICLR. ", "strength_and_weaknesses": "I am not sure how this paper fits into a typical ICLR paper, bu this paper addresses an important problem in BCI. \n\nIn addition to the SHAP values associated with each parameter, it might be helpful to see the impact of each parameter directly on the % of active channels (similar to what is done for electrode design parameters in Section 3.2). This would help understand how the methods used in this paper help with better explainability. \n\nStability of SHAP values across different partitions of individuals would be useful to know. That is, how uniform are the contributions of different features across different individuals? \n\nDo the factor contributions (SHAP values) change with classifier type?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "-_gAQiEiolW", "reviewer": "Reviewer_9zRr", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on invasive brain-computer interfaces (BCIs) using intracortical electrodes that are susceptible to degradation of signal quality over time. The goal of the study is to predict whether channels in intracortical electrodes are active based on features from neural recordings, electrochemical data and physical characteristics of the electrodes and assess feature importance. Results are presented from an animal experiment.", "summary_of_the_review": "\nRecommendation is based on the strengths and weaknesses described above.\n\nOther Comments:\n\n-    While Figure 1 is well-annotated, caption should be more informative and direct reader through understanding the various components of the figure. Same for Figure 4.\n\n-\tOffline Sorter: Authors need to provide more details about Offline Sorter because it is confusing. Is it a software algorithm than can automatically identify spike waveforms or does it require manual review with “blind reviewers”? If Offline Sorter is automated, why is a ML algorithm need to predict active channels if similar information can be obtained using Offline Sorter?\n\n-\tCustom-designed intracortical microelectrode: Are some of the features able to be recorded automatically using conventional electrodes or are they available due to the custom designed microelectrode? \n      o\t \"First, we obtained the electrochemical (EIS and VT) and spike metrics of the first recording session with that had a sortable unit for a particular channel (”First S.U.”).\"\n", "strength_and_weaknesses": "Strength\n- Well designed study with novel dataset that is highly relevant to the neuroscience/BCI/machine learning community.\n\nWeaknesses\n\n- It is not clear that the paper answers the question of when a channel will become inactive or assesses the longitudinal behaviour of channels over time (\"Predicting when stability of neural signals will decrease and understanding which features of the interface contribute to this, would benefit clinicians and researchers and improve utility of intracortical BCIs.\")\n\n- Authors do not provide results demonstrating how predicting whether a channel is active, and consequently the proportion of active channels is related to the downstream BCI task (e.g., movement of robotic arm). Assuming the proportion of active channels can be predicted, how is this information relevant to predicting the longitudinal performance of the BCI task?\n\n- Not really clear what additional insight is revealed by some of the feature analysis (e.g., sections 3.4 and 3.5) and based on previous analysis in the literature (section 3.3, e.g., intuitively, the longer the implant duration, the higher probability of inactive channels due to build-up, high impedance, etc.). \n\n- Small sample size and short duration of study, but understandable given the nature of the experiment. Authors acknowledge limitations.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-XUXEzfsED", "reviewer": "Reviewer_byYz", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The problem tackled is that of the decay in performance of BCI systems over time. The article presents an analysis of an electrocorticography data set (n=6) used for brain-computer interface, and in particular discusses the importance of the different features used by a predictive model built with machine learning techniques.\nFour different models are put to the test and the one exhibiting highest prediction rates is further analyses to decode the importance of the different features.\nAlthough ML is popular for BCI applications, I agree with the statement that for the specific case of boosting longitudinal stability of the BCI its application is novel.\n", "summary_of_the_review": "+ Without an analysis of error propagation it is difficult to anticipate how far in the long term would the predictive power of the model hold within useful boundaries.\n+ There appears to be no measure of the degradation of the performance of the xML pipelines as the prediction horizon is further away.\n+ Cross-correlated variables were removed. Were corrections considered (as opposed to simply dropping) e.g. statistical whitening, Bayesian multivariate decoupling, ARIMA models with a cross-variate feed, etc?\n+ The explanatory approach methods taken here (SHAP and decision paths) is actually very insightful and in principle is generic, hence it could be translated to other BCI modalities; e.g. EEG based, fNIRS based, etc. What adaptations would be suggested to do so?\n+ Although the translation to humans is identified as one of the limitations of the study, I can see how rather than a limitation of this study that makes up for a different study altogether. In this sense, my question is a bit more humble, regardless of the length of the BCI intervention, and focusing strictly on the predictive horizon of the model (currently shown at 15 weeks for the rats), is it likely to be at least equally long in humans? Even if it is only for 15 weeks, that remains a quite reasonable horizon in humans, but unfortunately, with a much more complex nervous system, I reckon the decay of the performance of the predictive models might suffer more heavily?.\n\n\nMinor details\n+ Figures looks grainy on my end (although readable). Not sure whether this is only on my side, but just in case it does not cost much to double check.\n+ AI and ML are closely related but not the same. The authors seem to use explainable machine learning and explainable artificial intelligence interchangeably. If this is intentional, I’m fine with it. If not, why not stick to only one of them? Subjectively I would say this paper in particular is closer to xML rather than xAI but I wouldn’t argue with other perceptions.\n+ Some of the things explaining the figures are in the main text. Of course, every author has its style and I do not want to interfere. But if there is no strong preference, may I suggest to move such parts to the figure caption to make the images self-contained?\n\n", "strength_and_weaknesses": "Strengths\n\n+ The problem studied is clear and remains open and the study proposed in this draft is a clear contribution to the field.\n+ The introduction is entertaining and of adequate depth, the literature review is extensive and its use throughout the article is excellent, the methodology is solid, the results are clear, the discussion (embedded within the results) is interesting, and the article has high nomological validity.\n\nWeaknesses\n\nNot many…\n+ Absence of inferential statistics, and even the descriptive statistics is incomplete. For example, the standard deviations of the different statistics are not described, the statistical power analysis is not described, etc.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject