Title: SEQUENCE TRANSFER LEARNING FOR NEURAL DE-CODING

Abstract: A fundamental challenge in designing brain-computer interfaces (BCIs) is decoding behavior from time-varying neural oscillations. In typical applications, decoders are constructed for individual subjects and with limited data leading to restrictions on the types of models that can be utilized. Currently, the best performing decoders are typically linear models capable of utilizing rigid timing constraints with limited training data. Here we demonstrate the use of Long Short-Term Memory (LSTM) networks to take advantage of the temporal information present in sequential neural data collected from subjects implanted with electrocorticographic (ECoG) electrode arrays performing a finger flexion task. Our constructed models are capable of achieving accuracies that are comparable to existing techniques while also being robust to variation in sample data size. Moreover, we utilize the LSTM networks and an affine transformation layer to construct a novel architecture for transfer learning. We demonstrate that in scenarios where only the affine transform is learned for a new subject, it is possible to achieve results comparable to existing state-of-the-art techniques. The notable advantage is the increased stability of the model during training on novel subjects. Relaxing the constraint of only training the affine transformation, we establish our model as capable of exceeding performance of current models across all training data sizes. Overall, this work demonstrates that LSTMs are a versatile model that can accurately capture temporal patterns in neural data and can provide a foundation for transfer learning in neural decoding.

1. INTRODUCTION:

A fundamental goal for brain-computer interfaces (BCIs) is decoding intent. Neural decoders that predict behavior address this goal by using algorithms to classify neural patterns. The performance of a decoder is directly related to its ability to overcome sensor noise and intrinsic variability in neural response, particularly in situations with constraints on training data availability. Due to limitations imposed in clinical recording settings, available human subject datasets are commonly on the order of minutes to tens of minutes.

Limited by dataset duration, existing neural decoders achieve reasonable performance by focusing on constrained model designs (Yanagisawa et al., 2012; Vansteensel et al., 2016) . As such, the stateof-the art decoders are models which only need to learn a small set of parameters (Gilja et al., 2012; Collinger et al., 2013; Hotson et al., 2016) . A limitation of these models is that they rely heavily on the quality of the training data, informative features, and often have rigid timing constraints which limit the ability to model neural variability (Vansteensel et al., 2016) . Furthermore, these specific informative features and timing constraints must be hand-tailored to the associated neural prosthetic task and the corresponding neural activity. Deep learning algorithms, however, have been applied to similar problems with the goal of learning more robust representations with large amounts of training data (Graves et al., 2013; Sutskever et al., 2014) . Unfortunately, only few studies apply these techniques to neural decoding (Bashivan et al., 2015; Sussillo et al., 2016) ; however, they exploit different measurement modalities -EEG and single neuron recordings, respectively.

Traditionally hidden Markov models (HMMs) have been used in neural decoding while accounting for temporal variability (Wang et al., 2011; Wissel et al., 2013) . Exploring the limitations of these neural decoders and considering recent advancements in deep learning techniques, we propose a framework using Long Short-Term Memory (LSTM) networks for neural decoding. LSTMs have demonstrated an ability to integrate information across varying timescales and have been used in sequence modeling problems including speech recognition and machine translation (Graves et al., 2013; Sutskever et al., 2014; Cho et al., 2014) . While previous work using LSTMs has modeled time varying signals, there has been little focus on applying them to neural signals. To this end, we establish LSTMs as a promising model for decoding neural signals with classification accuracies comparable to that of existing state-of-the-art models even when using limited training data.

Furthermore, addressing the limitations of existing models to generalize across subjects, we propose a sequence transfer learning framework and demonstrate that it is able to exceed the performance of state-of-the-art models. Examining different transfer learning scenarios, we also demonstrate an ability to learn an affine transform to the transferred LSTM that achieves performance comparable to conventional models. Overall, our findings establish LSTMs and transfer learning as powerful techniques that can be used in neural decoding scenarios that are more data constrained than typical problems tackled using deep learning.

2. METHODS:



2.1. DATA DESCRIPTION:

Neural signals were recorded from nine subjects being treated for medically-refractory epilepsy using standard sub-dural clinical electrocorticography (ECoG) grids. The experiment was a finger flexion task where subjects wearing a data glove were asked to flex a finger for two seconds based on a visual cue (Miller et al., 2012) . Three subjects are excluded from the analysis due to mismatches in behavioral measurement and cue markers. Rejecting electrodes containing signals that exceed two standard deviations from the mean signal, two additional subjects are removed from analysis due to insufficient coverage of the sensorimotor region. All subjects participated in a purely voluntary manner, after providing informed written consent, under experimental protocols approved by the Institutional Review Board of the University of Washington. All subject data was anonymized according to IRB protocol, in accordance with HIPAA mandate. This data has been released publicly (Miller & Ojemann, 2016) .

Analyzing the neural data from the four remaining subjects, electrodes are rejected using the same criteria mentioned above. For each subject, 6 -8 electrodes covering the sensorimotor region are utilized for their importance in motor planning. They are conditioned to eliminate line noise, and then instantaneous spectral power in the high frequency band range (70 -150 Hz) is extracted for use as the classifier features (Miller et al., 2007) . The data is segmented using only the cue information resulting in 27 -29 trials per finger (5 classes). The signal power for each trial is binned at 150 ms yielding an average sequence length of 14 samples.

As model accuracy is evaluated as a function of the number of training samples, shuffled data is randomly partitioned into train and test sets according to the evaluated training sample count. For each subject, 25 -27 training samples per class were used allowing for a test set comprising of at least 2 samples per class. A validation set is not used due to the limited data size. All experiments report the average of 20 random partitions of the shuffled data.

2.2. BASELINE MODELS:

As linear discriminant analysis (LDA) and Hidden Markov models with Gaussian emissions (HMM) are commonly used to decode sequence neural signals (Hotson et al., 2016; Wang et al., 2011; Wissel et al., 2013) , we use them for baseline comparisons. Inputs to the HMMs are identical to those used by the LSTMs. However, as LDA does not have a representation of time, individual time bins for each electrode are presented as features to the model. LDA was regularized by automatic covariance shrinkage using Ledoit-Wolf lemma (Ledoit & Wolf, 2004 ) without using a separate validation set. The formulation of LDA using time bins as features has been shown to achieve high accuracies with longer trial lengths (Hotson et al., 2016) . As such, we emphasize that this formulation results in a strong baseline performance because it can explicitly model the temporal response as a function of neural activity relative to movement onset. To handle varying sequence lengths, an LDA model must be trained for each of the possible sequence lengths. It would be infeasible to construct a model for every possible sequence length, and is susceptible to overfitting as electrode count and trial duration increase. This is in contrast to the HMM and LSTM models for which the learned parameters do not change as a function of time relative to movement onset and, instead, temporal dynamics are captured by parameters that are constant across time. Therefore, while LDA works well for structured experimental tasks, it would not generalize to unstructured naturalistic scenarios.

The HMM baseline model is analyzed with 1-state and 2-states using Gaussian emissions to explore if more complex behaviors can be modeled with a more expressive temporal model. For both the 1 and 2-state HMM, a single HMM is trained per class and maximum likelihood decoding is used to identity the target class. While the 2-state HMM is a standard ergodic HMM (Rabiner, 1989) allowing transitions between both the states, a 1-state HMM is a special case, it does not have any state dynamics and makes an independence assumption for samples across time. Thus, the 1-state HMM is specified by

P ({x} |y) = P (x 1 |y)P (x 2 |y) . . . P (x t |y) x t |y ∼ N (µ y , Σ y )

where {x} denotes the sequential data and y denotes the class.

The 1-state HMM assumes the features over time are independent and identically distributed Gaussian. While this independence assumption is simplistic, it is closely related to the LDA model and thus has fewer parameters to estimate. The 2-state HMM on the other hand processes the data as a true time series model similar to LSTM.

2.3. NETWORK ARCHITECTURE AND OPTIMIZATION:

The single recurrent LSTM cell proposed by Hochreiter & Schmidhuber (1997) is utilized and shown in Figure 1A . The model is completely specified by

f t = σ(W xf x t + W hf h t-1 + b f ) j t = tanh(W xj x t + W hj h t-1 + b j ) i t = σ(W xi x t + W hi h t-1 + b i ) o t = σ(W xo x t + W ho h t-1 + b o ) c t = c t-1 f t + i t j t h t = tanh(c t )

o t where σ is the sigmoid function, the W terms are weight matrices, the b terms are biases, and represents Hadamard multiplication. To enable gradient flow, the forget gate bias term is initialized to 1 (Gers et al., 2000; Jozefowicz et al., 2015) . At every time step during training, the label is provided to allow error propagation during intermediate steps rather than only after the full sequence has been evaluated (Yue-Hei Ng et al., 2015) . This is depicted in Figure 1B .

Different weight initialization schemes were analyzed: randomly sampling from a zero-mean Gaussian distribution (Glorot & Bengio, 2010) , using a language model (Dai & Le, 2015) , and using a sequence autoencoder (Dai & Le, 2015) . Building on the sequence autoencoder, we utilize a modified version where the encoder and decoder weights are not shared, and is similar to the LSTM autoencoder from Srivastava et al. (2015) . Additionally, dropout is utilized in the recurrent states (Gal & Ghahramani, 2016) with a probability of 0.3, but is not used on the input layer due to the limited dimensionality. Different model complexities were also explored focusing on network sizes and stacked architectures. The model used for our analyses was constructed with 100 hidden units with no performance gain identified using larger or stacked networks. Gated recurrent units (Cho et al., 2014) were also evaluated with no observed improvement in performance.

Two curriculum learning strategies were identified as potential approaches for improving LSTM trainability. The standard approach uses the full-sequence during training across all epochs. Curriculum learning, originally proposed by Bengio et al. (2009) , increases the difficulty of the training samples across epochs and Zaremba & Sutskever (2014) proposed adding a probability for selecting the full-sequence or the curriculum-sequence. This approach is especially useful in situations where optimization using stochastic gradient descent is difficult (Zaremba & Sutskever, 2014) . In our task, difficulty is directly related to the length of the sequence. Defining L as the full-sequence length, the curriculum-sequence length, l, is scaled by the current epoch, k, using l = min {k/8 + 3, L}. Each sample in a batch is randomly chosen to be of length L or l with equal probability.

To prevent overfitting to the training set, training was stopped at 75 epochs for all evaluations and subjects. Models were trained through backpropagation using Adam (Kingma & Ba, 2014) and prediction is performed on the test data with accuracies reported using classification at the last time step, except when specified otherwise.

Briefly summarizing the LSTM model: a single single layer of 100 hidden units with 0.3 probability hidden unit dropout and weights initialized by a sequence autoencoder is used. Curriculum learning strategy of Zaremba & Sutskever (2014) with the parameters mentioned was employed for presenting the training samples. All hyperparameters including the number of epochs to stop training were obtained by optimizing a model on 75% of subject Bs data. These selected parameters were kept the same for all evaluations across all subjects. We emphasize that the chosen hyperparameters, though potentially suboptimal, help train a model that generalizes well across all subjects. For reporting accuracies we average results from an evaluation of 20 random partitions of the data. The standard error is at less than 0.02 for all the reported accuracies.

2.4. TRANSFER LEARNING ARCHITECTURE:

We next explored the possibility of transferring the representation learned by the LSTM from a subject, S 1 , onto a new subject, S 2 . Typical transfer learning approaches keep the lower-level layers fixed, but retrain the higher-level layers (Srivastava et al., 2015; Yosinski et al., 2014) . Due to the unique electrode coverage and count as well as physiological variability across subjects, this approach yielded poor results. Accounting for these factors, we propose using an affine transform to project the data from S 2 onto the input of an LSTM trained on S 1 as we might expect a similar mixture of underlying neural dynamics across subjects ( Morioka et al. (2015) ). The fully connected affine transformation is specified by

x S 2 t = W xx x S2 t + b x

where W x and b x are the weights and biases of the affine mapping, x S2 t and x S 2 t are the original and the transformed sequential data from S 2 respectively.

Using hyper-parameters outlined in the single subject LSTM model, a two-step training process, shown in Figure 2A , is utilized. The first step trains the LSTM using all of the data from S 1 . Upon fixing the learned weights, the fully connected affine layer is attached to the inputs of the LSTM and trained on S 2 data. To establish that the affine transform is only learning an input mapping and not representing the neural dynamics, a baseline comparison is utilized where the Step 1 LSTM is fixed to a random initialization and only the Sof tmax S1 is trained, this is shown in Figure 2B .

All code will be made available.

3. RESULTS:

We found that LSTMs trained on an individual subject perform comparable to state-of-the-art models with sufficient training samples. Additionally, using the proposed transfer learning framework we

A B:

Step 1

Step 2

Step 1

Step 2 observe that LSTMs provide a principled, robust, and scalable approach for decoding neural signals that can exceed performance of state-of-the-art models. First we establish the performance of the baseline models (Table 1 ). Interestingly, we observe that increasing the complexity of the HMM marks little improvement in the classification accuracy and typically results in decreased accuracy at low sample counts due to the increased complexity. Additionally, while LDA performs comparably for three subjects, it performs much better than the other models for Subject C due to the increased alignment between cue and observed behavior. This is expected because, as noted, the LDA formulation is better suited to take advantage of the time alignment in the experiment.

Examining the performance of the LSTMs (Table 1 ), we demonstrate that the proposed model is better able to extract information from the temporal variability in the signals than HMMs and is able to achieve performance comparable to the best baseline for each of the subjects. Consequently, we observe across most subjects LSTMs are able to exceed performance of both HMM models and LDA. Even for Subject C, the LSTM model is comparable to LDA without making the temporal alignment assumption.

Further investigating model time dependence, accuracy is evaluated using neural activity preceding and following behavioral onset. As LDA explicitly models time, a series of models for each possible sequence length are constructed. Depicted in Figure 3B , we observe that the LSTM is slightly better able to predict the behavioral class at earlier times compared to HMMs and is comparable to LDA across all times.

3.2. TRANSFER LEARNING PERFORMANCE:

Historically, neural prostheses must be tuned frequently for individual subjects to account for neural variability (Simeral et al., 2011; Pandarinath et al., 2017) . Establishing LSTMs as a suitable model for decoding neural signals, we explored their ability to learn more robust, generalized representations that can be utilized across subjects.

We demonstrate that learning the affine transformation for the input, it is possible to relax the constraint of knowing exactly where the electrodes are located without having to retrain the entire network. First examining the baseline condition in order to assess the ability for the affine layer to learn the underlying neural dynamics, the S 1 LSTM weights were randomly initialized and fixed as outlined in Figure 2B . Looking at Figure 4A , the model performs slightly above chance, but clearly is unable to predict behavior from the data. The TL model where only the affine layer is trained in

Step 2 for S 2 (Figure 2A ) performs comparably to the subject specific model for S 2 . The notable advantage provided by TL is that there is an increase in loss stability over epochs, which indicates a robustness to overfitting. Finally, relaxing the fixed LST M S1 and Sof tmax S1 constraints, we demonstrate that the TL-Finetuned model achieves significantly better accuracy than both the best existing model and subject specific LSTM.

Detailing the performance for each TL and TL-Finetuned, we evaluate all 3 remaining subjects for each S 2 . For the TL model, we found that the transfer between subjects is agnostic of S 1 specific training and performs similarly across all 3 subjects. The performance of TL-Finetuned is similarly evaluated, but has trends unique to TL. In particular, we observe that transferring from Subject A always provides the best results followed by transferring from Subject B. Accuracies using the maximum permissible training data for all four subjects comparing the two transfer learning approaches and the best existing model as well as the subject specific LSTM are reported in Table 2 .

3.3. MODEL ANALYSIS:

For the transfer learning models, we explored the stability of the affine layer and analyzed the learned LSTM weights between subjects. Examining the learned affine mapping, we can see that the layer resembles a projection of electrodes to the appropriate brain regions. In Figure 5A , we show the mapping for two cases: a projection from sensorimotor cortex (R M otor , R Sensory ) between the two subjects, and a projection also adding the occipital lobe electrodes (R Occipital ) for S 2 . As the occipital lobe is involved with integrating visual information, and contains information unique from the trained regions, we would expect and do observe, there to be an affine transformation with weights for R Occipital closer to zero indicating an absence of electrodes in S 1 that contain similar information. Considering the findings by Haufe et al. (2014) , it is important to note that we know apriori from neurophysiology that the occipital region in motor movement tasks should be less informative. Therefore, we would expect the affine mapping to learn to reject the signals from occipital region which we observe in our model. Additionally, we can qualitatively observe the stability of the affine transform after the fixed LST M S1 and Sof tmax S1 are relaxed. It is clear by looking between the left and right columns of Figure 5A that the learned weights from TL are a good representation and only require minor modification in TL-Finetuned.

Furthermore, exploring the use of LSTMs for transfer learning, a two-dimensional embedding of the LSTM output using t-SNE (Maaten & Hinton, 2008) on the training data was created. We use the LST M S1 outputs for S 1 data and LST M S1 outputs for S 2 data after passing it through the learned affine layer on S 2 . All the data between -300 ms to 900 ms, relative to behavioral onset, is embedded in the two-dimensional space. From the summary image, Figure 5B , it is clear that the model is able to separate classes for both subjects well and that the projections for both the subjects are clustered together. Identifying the source of the noise in the figure, we project the output at different time steps and see that the majority of the confusion occurs at the start of the sequences.

4. DISCUSSION:

Although many neural decoding models and processing techniques have been proposed, little work has focused on efficient modeling of time-varying dynamics of neural signals. In this work, we have shown that LSTMs can model the variation within a neural sequence and are a good alternative to state-of-the-art decoders. Even with a low sample count and comparably greater number of parameters, the model is able to extract useful information without overfitting. Moreover, LSTMs provide a robust framework that is capable of scaling with large sample counts as opposed to the the limited scalability provided by existing approaches. Considering the limitations imposed on our model by stopping at a fixed evaluation epoch, it would be possible to further boost performance by utilizing early stopping with a validation set. And while the input features were selected from empirical observations made in previous studies, the results could be improved by extracting the features in an unsupervised manner using autoencoders (Poultney et al., 2007; Le et al., 2011) or by training the decoder end-to-end using convolutional LSTMs (Shi et al., 2015; Zhang et al., 2016) .

Establishing the LSTM as a good approach for neural decoding, we explored utilizing the model in transfer learning scenarios. Exploring a less constrained model where the LSTM weights are relaxed, the performance is shown to exceed that of both the subject specific training and the best decoder models. This robustness against subject specific neural dynamics even when only the affine transform is learned indicates that the LSTM is capable of extracting useful information that generalizes to the new subject with limited impact due to the original subject's relative performance.

When exploring the tradeoffs between TL and TL-Finetuned, the latter provides performance that exceeds the current state-of-the-art models with fewer subject-specific training samples where the former is able to achieve comparable performance. However, because TL only requires training of the affine layer, the computation is less expensive than the TL-Finetuned. From Figure 4 , it could be seen when learning only the affine transformation the cross-entropy loss is still decreasing after 500 epochs suggesting that with better optimization methods, the TL model by itself may outperform the subject-specific model. This motivates the statement that the LSTM is capable of extracting a representation of the neural variability between behaviors that generalizes across subjects. While this may be specific to the behavior being measured, it posits potential scenarios for using sequence transfer learning.

Exploring the reasoning behind the affine layer, we consider relaxing the structured mapping of electrodes between the subjects required by typical transfer learning models. While the structured mapping would intuitively yield good results if electrode placement is the sole factor influencing neural variability, we see that it leads to suboptimal performance due to limited alignment capabilities as well as the underlying neural representation being unique. The addition of an affine layer, however, provides sufficient flexibility for the input remapping to account for this variability and matches the new subjects input electrodes based on error minimization. Moreover, the weights that are produced through backpropagation are able to optimize the weights regardless of S 2 input dimensionality and thus allows for eliminating use of non-informative electrodes. This input gating leads to model training stability and is shown to be a valid mapping due to the minimal weight update when transitioning from TL to TL-Finetuned. Furthermore, exploring the relationship between subjects 1 and 2, the t-SNE analysis shows that the learned parameters for the affine layer provide a meaningful mapping between the two subjects likely indicating an underlying physiological basis rooted in the structure of the sensorimotor cortex that has been shown to exist.

We believe that the approaches established in this paper provide techniques for decoding neural signals that had not yet been explored in detail. Particularly, the insights gained from exploring neural decoding leveraging the expressibility and generalizability of LSTMs yielded techniques that provide more accurate and robust models compared to current state-of-the-art decoders. Consequently, the strategies of applying LSTMs to sequence learning and sequence transfer learning problems will be useful in a variety of neural systems.

:



Figures and Tables:

Figure 1: A schematic depicting the Long Short-Term Memory (LSTM) model architecture. (A) Gating and activation functions for a single unit. (B) Unrolled LSTM network with Softmax output at every time step during training.

Figure 2: A schematic depicting the LSTM transfer learning model architecture. Where gray indicates fixed and white indicates learned weights, both (A) and (B) depict a 2-step training process with Subject 1 (S 1 ) training in step 1 and Subject 2 (S 2 ) training of only the affine layer in step 2. (A) Transfer learning model (TL) training the LSTM and Softmax layers for S 1 in step 1. (B) Randomly initialized LSTM layer and training only the Softmax layer for S 1 in step 1.

Figure 4: Transfer learning LSTM performance comparison for a representative subject. The TL-Finetuned, TL, and random LSTM all utilize Subject B as S 1 and Subject C as S 2 . The subject specific model uses Subject C. (A) as a function of the amount of training samples. (B) Cross-entropy error on the test set across epochs, using 10 training samples per class. Error bars show standard error of mean averaging results from 20 random partitions of the shuffled data.

Summary comparing the average accuracy of the LSTM model with existing approaches after 20 random partitions of all the shuffled training data.

Table content: <table><tr><td>Model</td><td colspan="4">Subject A Subject B Subject C Subject D</td></tr><tr><td>LDA</td><td>0.50</td><td>0.53</td><td>0.79</td><td>0.64</td></tr><tr><td>HMM -1s</td><td>0.51</td><td>0.61</td><td>0.69</td><td>0.65</td></tr><tr><td>HMM -2s</td><td>0.53</td><td>0.59</td><td>0.68</td><td>0.60</td></tr><tr><td>LSTM</td><td>0.51</td><td>0.62</td><td>0.75</td><td>0.69</td></tr></table>

Summary comparing the average accuracy of transfer learning with subject specific training, using all training data across 20 random partitions of the shuffled data. Visualizations of model parameters. (A) Weights of the learned affine mapping from S 2 (Subject C) to S 1 (Subject B). Electrodes mapped from motor to sensorimotor and sensorimotor + occipital to sensorimotor regions of S 2 to S 1 , respectively. Both TL and TL-Finetuned weights are depicted. (B) Two-dimensional t-SNE embedding of the learned signal representation corresponding to different fingers for both subjects.

Table content: <table><tr><td>Model</td><td colspan="4">Subject A Subject B Subject C Subject D</td></tr><tr><td>Best Existing Model</td><td>0.53</td><td>0.61</td><td>0.79</td><td>0.65</td></tr><tr><td>Subject Specific</td><td>0.51</td><td>0.62</td><td>0.75</td><td>0.69</td></tr><tr><td>TL (S 1 = Subject A)</td><td>-</td><td>0.66</td><td>0.72</td><td>0.60</td></tr><tr><td>TL (S 1 = Subject B)</td><td>0.40</td><td>-</td><td>0.70</td><td>0.46</td></tr><tr><td>TL (S 1 = Subject C)</td><td>0.44</td><td>0.63</td><td>-</td><td>0.59</td></tr><tr><td>TL (S 1 = Subject D)</td><td>0.44</td><td>0.67</td><td>0.73</td><td>-</td></tr><tr><td>TL-Finetuned (S 1 = Subject A)</td><td>-</td><td>0.71</td><td>0.82</td><td>0.70</td></tr><tr><td>TL-Finetuned (S 1 = Subject B)</td><td>0.46</td><td>-</td><td>0.75</td><td>0.62</td></tr><tr><td>TL-Finetuned (S 1 = Subject C)</td><td>0.44</td><td>0.63</td><td>-</td><td>0.66</td></tr><tr><td>TL-Finetuned (S 1 = Subject D)</td><td>0.53</td><td>0.71</td><td>0.79</td><td>-</td></tr></table>

Peer review (iclr2018):
 The paper describes an approach to use LSTM’s for finger classification based on ECOG. and a transfer learning extension of which two variations exists. From the presented results, the LSTM model is not an improvement over a basic linear model. The transfer learning models performs better than subject specific models on a subset of the subjects. Overall, I think the problem Is interesting but the technical description and the evaluation can be improved. I am not confident in the analysis of the model. Additionally, the citations are not always correct and some related work is not referenced at all. For the reasons above, I am not willing to recommend the paper for acceptance at his point. The paper tackles a problem that is challenging and interesting. Unfortunately, the dataset size is limited. This is common for brain data and makes evaluation much more difficult. The paper states that all hyper-parameters were optimized on 75% of subject B data. The actual model training was done using cross-validation. So far this approach seems more or less correct but in this case I would argue that subject B should not be considered for evaluation since its data is heavily used for hyper-parameter optimization and the results obtained on this subject are at risk of being biased. Omitting subject B from the analysis, each non-transfer learning method performs best on one of the remaining subjects. Therefore it is not clear that an LSTM model is an improvement. For transfer learning (ignoring B again) only C and D are improved but it is unclear what the variance is. In the BCI community there are many approaches that use transfer learning with linear models. I think that it would be interesting how linear model transfer learning would fare in this task. A second issue that might inflate the results is the fact that the data is shuffled randomly. While this is common practice for most machine learning tasks, it is dangerous when working with brain data due to changes in the signal over time. As a result, selecting random samples might inflate the accuracy compared to having a proper train and test set that are separated in time. Ideally the cross-validation should be done using contiguous folds. I am not quite sure whether it should be possible to have an accuracy above chance level half a second before movement onset? How long does motor preparation take? I am not familiar with this specific subject, but a quick search gave me a reaction time for sprinters of .15 seconds. Is it possible that cue processing activity was used to obtain the classification result? Please discuss this effect because I am do not understand why it should be possible to get above chance level accuracy half a second before movement onset. There are also several technical aspects that are not clear to me. I am confident that I am unable to re-implement the proposed method and their baseline given the information provided. LDA baseline: ————————— For the LDA baseline, how is the varying sequence length treated? Ledoit wolf analytic regularization is used, but it isn not referenced. If you use that method, cite the paper. The claim that LDA works for structured experimental tasks but not in naturalistic scenarios and will not generalize when electrode count and trial duration increases is a statement that might be true. However, it is never empirically verified. Therefore this statement should not be in the paper. HMM baseline ————————— How are the 1 and the 2 state HMM used w.r.t. the 5 classes? It is unclear to me how they are used exactly. Is there a single HMM per class? Please be specific. LSTM Model ————— What is the random and language model initialization scheme? I can only find the sequence auto-encoder in the Dai and Le paper. Model analysis ——————————- It is widely accepted in the neuroimaging community that linear weight vectors should not be interpreted directly. It is actually impossible to do this. Therefore this section should be completely re-done. Please read the following paper on this subject. Haufe, Stefan, et al. "On the interpretation of weight vectors of linear models in multivariate neuroimaging." Neuroimage 87 (2014): 96-110. References ———— Ledoit wolf regularization is used but not cited. Fix this. There is no citation for the random/language model initialization of the LSTM model. I have no clue how to do this without proper citation. Le at al (2011) are referenced for auto-encoders. This is definitely not the right citation. Rumelhart, Hinton, & Williams, 1986a; Bourlard & Kamp, 1988; Hinton & Zemel, 1994 and Bengio, Lamblin, Popovici, & Larochelle, 2007; Ranzato, Poultney, Chopra, & LeCun, 2007 are probably all more relevant. Please cite the relevant work on affine transformations for transfer learning especially the work by morioka et al who also learn an input transferm. Morioka, Hiroshi, et al. "Learning a common dictionary for subject-transfer decoding with resting calibration." NeuroImage 111 (2015): 167-178.