Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose a method for fast and efficient classification of sequential data. The guiding principle is that for some data modalities it is not necessary to see the whole sequence in order to make a fairly certain classification. Their model reduces inference time by learning a rank code that is inspired by spiking neural networks. Reported results show improved inference times in two toy sequence classification tasks, temporal MNIST, and in Google Speech Commands classification (compared to models without optimizing timing of inference through learning a rank code). Increasing inference speed comes with a minimal decrease in accuracy, the authors, however, introduce and show the effectiveness of a regularization term that allows for tuning of this speed-accuracy trade-off. 
main_review: PROS: 

I think the proposed method is very practical and the ideas of this paper are organized logically. 

The method is related to early-exit inference but their model not just exits early, it does so by a learned rank code (RC) that is inspired by spiking neural networks. This allows for adaptively decreasing computation and increasing speed during both training AND inference. 

For training, the idea of backpropagating from a strategically early time step of each sequence, determined by the time at which an output neurons activation crosses a threshold, is interesting and can lead to significantly shorter training time and lower compute resources.

It appears that prior and related work is adequately referenced throughout the paper. Unfortunately I am not familiar enough with this line of work, but I quick search revealed no glaring omission.

The empirical methodology appears standard, and is reported in sufficient detail to recreate the results. The resulting claims are justified by the performance of the method. Strengths and limitations are sufficiently discussed. The writing is clear and succinct.

CONS:

The authors do not compare their method to non-RC trained LSTMs in the temporal MNIST task. For this task, since it appears that the first frame contains most of the information, I suspect that an early-exit but non-RC trained LSTM would perform similar in terms of inference speed and accuracy. This is however just a guess, did the authors try this or have further insights? Comparing their model to SNNs in this task seems somehow unfair, also given that the authors state earlier that “ANNs are simpler to train, and usually achieve superior performance”.
However, I can follow the authors argumentation on why they have chosen to compare to SNNs. Comparing to early-exit but non-RC trained LSTMs would still be interesting but omitting it would not weaken the message delivered in the overall strong paper.

The threshold is a fixed hyper-parameter of the model. Would it be beneficial to learn this parameter?

summary_of_the_review: Overall, I vote for accepting. I like the idea of integrating specific isolated aspects of biological neurons into otherwise conventional ANNs. Here, the authors show that ANNs can benefit by such an approach and I think that further exploration of such methods may advance both spiking neural network and conventional deep learning research.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors introduce a new way to train RNNs using rank order coding (ROC). With ROC the label is given by the first readout unit to reach a threshold. As soon as this happens, the processing is stopped, and BPTT is used from that particular time step, using the predictions at that particular time step and the ground truth. This will encourage the neuron with the right label to be as active as possible at that particular time step, and thus its threshold will tend to be reached earlier in the future. This is desirable, as the latency of the decision will decrease. Furthermore, the speed-accuracy trade-off is tunable by varying the threshold.

The authors validate their idea using LSTMs on two toy problems, and then on MNIST and on the Google Speech Command dataset.
main_review: Strength:
* As far as I know the idea is new
* Well written

Weaknesses
* Experimental validations are well below modern ML standards
* The authors should test their idea on spiking neurons (eg LIF) as opposed to LSTMs

I think experimental validation falls short for ICLR. Only the Google Speech Command dataset is not toy, and the accuracy they get on this dataset is below the SOTA.

In addition, the authors seem to target the spiking neural network community (part of which uses ROC). See for example Table 1. But then, for a fair comparison, they should try to use a spiking neuron model (eg leaky integrate and fire, LIF) instead of the LSTM. In discrete-time, the LIF can be seen as a recurrent ANN unit, and BPTT could be used as well.

The authors seem to use batch processing, but I don't understand how it's possible. The number of timesteps used by BPTT is example-dependent. How can batch processing work with a varying number of timesteps? More insight is needed here.


summary_of_the_review: A potentially interesting idea, but not yet validated
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents the original idea of applying rank coding to classical LSTM networks in order to improve their performance.
main_review: The paper is very clearly written and has an extensive introduction that allows to introduce the scientific problem. The method is briefly described and then applied in experimental demonstrations. If the first example is particularly simple, the three following applications are more challenging and show the advantage of using this method compared to the state of the art. One. A particularly interesting point is the trade-off curves between speed and accuracy as well as the performance of the network depending on the use or not of rank coding.

I would like to point out some limitations of this work and how it could be improved. First of all, it seems that the rank coding used in this paper is radically different from the one proposed by Thorpe and Gautrais. Indeed, in the latter an analog vector is transformed into a vector in which its analog values are ordered from the highest to the lowest. This representation has many properties such as being invariant to continuous monotonic transformations of the analog values, as for example an image can be transformed by a change of its contrast. This transformation also keeps a very high complexity of possible representations which corresponds to the set of permutations of all ranks, that is to say, to the factorial of the dimension of this vector. However in this paper it seems that you only consider the maximum of the analog vector when it exceeds a threshold. This indeed allows to transform the calculation into a temporal calculation, but the complexity of this operation is much lower than the original rank coding.

summary_of_the_review: In the first practical example it seems that you use this coding in different time windows that correspond to different bins in a sequence and that these sequences are then processed independently. This information processing is far too simplified to qualify as temporal coding and thus to correspond to a spiking neuron model. The other experiments seem to show a clear advantage to the rank coding and seem promising to extend it to the coding of the maximum but also of the successive values. I encourage the authors to apply this kind of method to larger images as it has already been done in the literature.
review_title: null

Meta-review: The authors propose a rank coding scheme for recurrent neural networks (RNNs) - inspired by spiking neural networks - in order to improve inference times at the classification of sequential data. The basic idea is to train the RNN to classify the sequence early - even before the full sequence has been observed. They also introduce a regularisation term that allows for a speed-accuracy trade-off.

The method is tested on two toy-tasks as well as on temporal MNIST and Google Speech Commands.

The results are very good, typically improving inference time with very little loss in accuracy.

Furthermore, the idea seems novel and the paper is well written.

An initial criticism was that experiments with spiking neural networks (SNNs) were missing. The authors added a proof of concept for SNNs, which satisfied the reviewer. 

The authors also added some control experiments in response to the initial reviews, which improved the manuscript.

In summary, the manuscript presents a valuable novel idea with good experimental verification and interesting aspects both for ANNs and SNNs. The reviewers consistently vote for acceptance.
Decision: Accept (Spotlight)