Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes using reservoir computing for classifying time segments. The key advantage is less training time. The approach, TRAKR, is evaluated on simulated timeseries, sequential MNIST, and ECoG with increased accuracy over conventional methods shown.
main_review: Strength
1. The paper is clearly written and easy to follow.
2. The proposed approach is simple, easy to implement, and already provides a gain over conventional methods.

Weaknesses
1. The technical contribution is unclear, since reservoir computing is not new. Would be interesting to see if we learn an embedding space (using simpler methods) with a given set of reference time segments and use that embedding to reconstruct other segments to generate error signals, whether we could achieve the same gain as reservoir computing. This experiment can help highlight the need for reservoir computing.
2. It is unclear what would be a good choice of “reference” time segments/sequences. For ECoG, rest period would make sense, but is there a way to decide in the general setting? Is the contribution of this work actually the use of a reference to generate error signals for classification, which tends to increase accuracy?
3. It is unclear what the simulated timeseries experiment is demonstrating. If we overfit the training time segments, by construction, we would get huge error signals on test segments, which would enable us to distinguish the two. 
4. For the sequential MNIST experiment, it is not clear which digit would be a good reference and how results change with this choice. Nevertheless, getting an AUC of 99% with such a simple approach is quite impressive. If we use 10-fold cross validation, does the high AUC stays? Sometimes leave-one-out provides error estimates similar to training errors. 
5. Though TRAKR is fitted in one shot, does performance improve with a few more rounds? In general, when deploying a model real time, we care more about the amount of time it takes to generate a prediction from the model, and not so much its training time (unless the amount of training time is impractical). Hence, comparisons with other DL models are needed to show TRAKR’s performance is at least on par.
6. Overall, seems like TRAKR works well for relatively simple classification tasks but breaks down on harder tasks like match vs. mismatch.

summary_of_the_review: My score is mainly based on TRAKR not evaluated on harder classification tasks, lack of comparison with state-of-the-art DL models, and unclear technical novelty.

Post-revision summary
The authors addressed some of my comments. However, the motivation as written remains to be faster runtime for real world applications, but accuracy is lower than MLP and runtime is not that much shorter. A little surprising, but would expect the state-of-the-art to be some form of transformers as opposed to MLP. Given MLP provides higher performance, I will keep my score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new method to classify neural time-series. The method is based on a recurrent reservoir, which includes a black box function (here a fixed random RNN) and a linear readout. The black box function produces an state vector at each time point of an input time-series. This state vector is then transformed with a linear readout, trained to reconstruct the input signal. In other words, the reservoir is an auto-encoder with a fixed random RNN encoder and a linear decoder trained to minimize the reconstruction error. The method then uses the reconstruction error of the reservoir as a representation of the signal, to be used in a RBF-kernel SVM classifier and perform a given task.

On a simple toy example, the authors show that the reservoir can learn to reconstruct a sinusoid signal, and that it fails to reconstruct a sinusoid with a frequency different than during training. This failure can be considered useful to identify regimes different that the ones seen during training. The method is then used on a small MNIST datasest, training the reservoir on one sample, and then training an SVM classifier on all samples based on the reconstruction error of the reservoir. The method is then compared with four baseline methods. Finally, the method is applied to neural recordings of a macaque OFC, training the reservoir on one epoch, and then training a classifier to predict behavioral states.
main_review: Strengths: The method is reasonably presented. The method seems novel, and valuable because it requires very little training to create a representation of the input that can be used for further tasks. 

Weaknesses:
The experiments proposed in the paper do not seem sufficient to assess the strengths of the proposed method. The proposed method is not compared against other reservoir-based methods, against random-projection methods, or against deep neural networks.
___
Sinusoid toy dataset:
1. The frozen section of the train set is confusing. Why include this section in the train set if the learned weights are frozen ? Wouldn't it be clearer to remove the frozen section of the training set ?

MNIST experiment:

2. MNIST is composed on images, which are not naturally represented as sequential time-series. Why use MNIST as a dataset for time-series classification ? 
3. Why is the comparison limited to the four baseline methods? Why not include other models known to be good at classifying MNIST (such as CNNs)? Why not include other methods based on reservoir computing ? Why not include other random-projection methods ?
4. How were these baseline methods applied? Were the distance metrics used as input of the RBF kernel? How is Naive Bayes used in conjunction with the SVM ?
5. Is the classification task a one-vs-one, one-vs-all, or multinomial task? If multinomial, how is the AUC computed ?
6. How are the error bars computed in Figure 3A ? How is the shaded area computed in Figure 3B ? Assuming this is some sort of standard deviation around a mean value, why does the mean AUC for TRAKR fluctuates more than the standard deviation (e.g. trough at noise level = 0.7)?
7. How much time does the method requires to (a) train the reservoir, (b) train the SVM, (c) predict a sample ? Which computational time is reported in Table 1?

The OFC experiment raises the same questions as the MNIST experiment in terms of choice of the baseline methods. Additionally, it is not clear how the task of predicting behavioral states is useful to understand how the brain works. The TRAKR method does not seem to give particularly interpretable representations. The TRAKR+PCA embedding presented in Figure 5 also lacks a comparison with other embeddings, such as PCA directly on the electrode signals.
summary_of_the_review: The comparison with other methods seems rather incomplete.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Here, the authors propose a reservoir-computing based framework for time-series applications. The proposed methodology involves training the output units of the echo state network to recapitulate, or autoencode, a test signal. Next, time-series are fed to the network and the error signal between the time-series and the network's response is used as input to an SVM that is used for classification.  Effectively the error signal is used as a distance metric.

The authors report that the network performs well on sequential MNIST and decoding of neural data. Additionally, they argue that a key advantage of their approach is that it is computationally lightweight – both training and inference are fast. 
main_review: Strengths:

1. The manuscript is straightforward and well-written
2. The notion of using the error signal as the input to a classifier as new and potentially interesting
3. The classification results are possibly compelling

Major issues:

1. A central claim of the paper is that echo state networks are both performant and computationally efficient. Firstly, to properly contextualize the accuracy of their technique, the authors should compare TRAKR against more state of the art methods. For instance, the author should consider implementing some of the methods from the Fawaz review they cite. Moreover, while deep-learning methods may be more expensive to train, they can be used as a more up-to-date benchmark. Additionally, this could present the authors with an opportunity to compare the performance of TRAKR with deep learning methods as a function of the size of the training dataset (I suspect TRAKR might look more favorable here).

2. Another major claim is that TRAKR is more "lightweight" than ensemble or deep-learning methods. Deep learning methods, especially for large network sizes, are particularly data-hungry in the training phase. However, inference with deep networks can be extremely fast, to the point of being sufficient for real-time applications. If one were to compare the inference time of a deep network on a GPU vs TRAKR, it is possible that TRAKR is not significantly faster than a better-performing deep network. This must be clarified since computational efficiency is touted as a major feature of TRAKR.  

Minor issues:

1. Figure 4 what do the error bars repesent
summary_of_the_review: While the manuscript is well-written and the ideas presented are fairly straightforward, the authors need to address the primary competitor to TRAKR, deep nets, head-on. First, it is likely that deep nets are on par if not better-performing than TRAKR, which the authors readily admit.  However, the authors also need to perform analysis of inference speed with deep networks.  If deep nets can run almost as quickly as TRAKR, then this complicates a central pillar of the paper.
review_title: null

Meta-review: Authors propose an autoencoding echo state machine for a one-shot one-class time series classification task. Their approach feeds a (one-dimensional) error signal over time relative to a reference training datum to SVMs. Training is very fast by design. OFC signal analysis has practical value in neuroscience. But only one benchmark (seq-MNIST) was used to evaluate their method. While the performance seem impressive, no explanation of why the internal representation learned by the proposed system is superior and robust to noise was provided. No sequential autoencoders or latent neural trajectory inference methods were compared. Although the manuscript has greatly improved through the review-rebuttal process, there are missing key details (e.g. length of E(t) used for classification--important for real-time application, initial state for the reservoir, choice of W_in --important since it seems to be a chaotic network that's driven by strong input). While there is novelty in the approach, there is a general lack of enthusiasm among the reviewers for the manuscript as is. The reviewers and AC strongly encourage the authors to further developed these ideas and add thorough analyses for another conference.

(BTW, perhaps it's worth citing https://doi.org/10.1109/IJCNN.2016.7727309, since autoencoder combined with reservoir computing has been used for anomaly detection.)
Decision: Reject