Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this paper, the authors have introduced an event-based continuous-time variant of the gated recurrent model and derived an eve-based form of backprop to train the system. Extensive experiments on a wide variety of benchmarks show that the proposed model EGRU can achieve comparable performance with SOTA recurrent models and exhibits high levels of activity-sparsity during the training and inference phase. 
strengths_and_weaknesses: # Strengths
* Approach is novel
* Well-written paper
* Experiments are good

# Weakness
* Ablation study missing
* Experimental section is missing key details
* Missing some key citations
questions: * The comparison should be made with at least two models from [1-5], first, one proposed sparse RTRL to train neural networks, that can potentially solve the scalability issues shown by BPTT.  Other [2] focuses on other recurrent architecture and learning approaches to solve this issue. 

* For models only based on BPTT, prior work has shown that sparse versions of recurrent models have the potential to outperform dense versions. Hence it is important to show how efficient the current approach is compared to other sparse RNNs.

*Hyper-parameter optimization- No detail about hyper-parameter optimization, authors should provide reasoning how these parameters were chosen. 

* Statistical significance: - Experiments should be conducted for K trials and average performance and the standard error should be reported.
* Can authors comment on the robustness and stability of the given model? Based on different seeds, and settings, the optimizer is model stable compared to dense models trained using BPTT.

* Authors have mentioned that EGRU is computationally efficient, can authors provide numbers in FLOPS for both training and inference compared with other sparse and dense RNNs

[1] https://arxiv.org/abs/2006.07232
[2] https://arxiv.org/abs/1602.03032
[3] https://arxiv.org/pdf/1901.09208.pdf
[4] https://link.springer.com/article/10.1007/s00521-021-05727-y
[5] https://proceedings.mlr.press/v139/liu21p.html

limitations: Highlighted above
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes EGRU, an event-based GRU for activity-sparse inference and learning. The proposed model is more efficient as only a select set of units are active at any given time. 
strengths_and_weaknesses: **Originality**:

* To my knowledge, the idea proposed in this article is novel and original. 

**Quality**:

* The paper technically sounds correct and claims well supported by theoretical analysis and experimental results.
* Related works are covered and discussed.
* Experiments are conducted extensively in different type of tasks and the results are discussed thoroughly and compared in terms of accuracy,#MAC, and sparsity. 


**Clarity**:

* This paper is well written and organised.


**Significance**:
* the proposed EGRU, as the authors have suggested, can be more energy-efficient and scalable as it reduces the required compute for both inference and learning. 

questions: * In terms of activity sparsity during inference, how does it compare against pruning a GRU, such as the one mentioned in "EXPLORING SPARSITY IN RECURRENT NEURAL NETWORKS"?
limitations: The authors have discussed inheriting negative societal impacts and risks existed in GRU.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes an event-based continuous-time variant of the GRU model, named EGRU, motivated by building scalable, energy-efficient deep recurrent models. The EGRU unit only outputs events when the internal states reach a trainable threshold, which makes it activity-sparsity naturally. Furthermore, the authors derive corresponding BP algorithms for updating EGRU and extend it to a discrete-time version that is easier to implement by popular machine learning architecture. As the experiments show, the EGRU model exhibits comparable performance to state-of-the-art recurrent network architectures for several tasks.
strengths_and_weaknesses: This paper is well-written, consistent, and produces good results. By introducing the event generating mechanism, the EGRU model not only significantly reduces the multiply-accumulate operations but also preserves the performance that is comparable to the more computation-expensive GRU-based model. 

Strengths:
+ The event generating mechanism is a novel module inserted into the GRU model, which is proved to effectively reduce the computation of the original model.
+ The authors conduct three downstream applications to validate the effectiveness of the EGRU model, which are convincing and with comparable performances to previous methods.

Weakness:
- For the delay-copy task, the authors only show a simple example. How about increasing the number of EGRU units? Besides, did the events generating mechanism improve the memory ability, compared to the GRU model?
- In gesture prediction, the authors down sample frames into 32x32, which may affect the performance. I wonder how the input size affects the prediction accuracy with EGRU model. In general, the larger-resolution images contain more detailed information, which could be helpful for classification. 
- For language modeling experiments, the performance decreases with the increment of EGRU units. Can the authors give some explanation or the trend of performance related to the number of EGRU units?

questions: This paper presents a novel GRU architecture, which balances the performance and computational complexity. 
- However, in the downstream application with EGRU, some experiments’ setups are unclear, as listed in Weakness. I expect the authors to clarify these points. 
- Besides, the authors apply EGRU in some special cases. Since the LSTM/GRU are applied widely in CV/NLP applications, can they be replaced by EGRU to reduce the computation complexity in general? 
- As GRU is simplified from LSTM, can event generating mechanism transfer to the LSTM model?

limitations: The authors have addressed the limitations and potential negative societal impact of their work in discussion section.
review_title: null

Meta-review: metareview: This paper introduces an event-based GRU to obtain an efficient continuous-time RNN. Although the method is sound and can work on a series of small sequence modeling tasks, there are multiple issues with the significance of the results of the paper which I point out in the following:

1) There have been significant advances in recurrent neural networks designed to efficiently model sequences, and their scaling properties, which are overlooked in this paper. In particular, structural state-space models [1], diagonal state-space models [2], LSSL [3], Closed-form continuous-time networks [4], efficient memorization via polynomial projection [5], and Neural Rough DEs [6] are currently shaping the state-of-the-art sequence modeling frameworks that efficiently model tasks with long-range dependencies while significantly outperforming Transformers and their variants. Therefore, from the perspective of representation learning capabilities, there is a significant gap between what EGRU (proposed in this paper) could achieve compared to the state-of-the-art sequence modeling tools powered by recurrent networks. Before getting published, it is essential to compare performance and speed to these models on proper benchmarks such as Long Range Arena [7]. 

[1] Gu, A., Goel, K., & Ré, C. (2021). Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396

[2] Gupta, A. (2022). Diagonal State Spaces are as Effective as Structured State Spaces. arXiv preprint arXiv:2203.14343.

[3] Gu, A., Johnson, I., Goel, K., Saab, K., Dao, T., Rudra, A., & Ré, C. (2021). Combining recurrent, convolutional, and continuous-time 
models with linear state space layers. Advances in neural information processing systems, 34, 572-585.

[4] Hasani, R., Lechner, M., Amini, A., Liebenwein, L., Tschaikowski, M., Teschl, G., & Rus, D. (2021). Closed-form continuous-depth models. arXiv preprint arXiv:2106.13898.

[5] Gu, A., Dao, T., Ermon, S., Rudra, A., & Ré, C. (2020). Hippo: Recurrent memory with optimal polynomial projections. Advances in Neural Information Processing Systems, 33, 1474-1487.

[6] Morrill, J., Salvi, C., Kidger, P., & Foster, J. (2021, July). Neural rough differential equations for long time series. In International Conference on Machine Learning (pp. 7829-7838). PMLR.

[7] Tay, Y., Dehghani, M., Abnar, S., Shen, Y., Bahri, D., Pham, P., ... & Metzler, D. (2020, September). Long Range Arena: A Benchmark for Efficient Transformers. In International Conference on Learning Representations.

2) When it comes to the efficiency of computations on spatiotemporal tasks, especially when using a benchmark such as DVS Gesture detection, efficient models such as spiking networks must be accounted for. For instance, in [8], authors outperform EGRU on the DVS task with over 10x fewer parameters. In this case, EGRU+DA achieves 97% accuracy with 15.75M  (10.77M MAC) parameters, while the method proposed in [8] achieves 98% accuracy with 1.1M parameters. I believe it is essential to compare results with appropriate efficient models both in terms of computational efficiency and performance. 

[8] She, X., Dash, S., Mukhopadhyay, S.: Sequence approximation using feedforward spiking neural network for spatiotemporal learning: Theory and optimization methods. In: International Conference on Learning Representations (2022), https://openreview.net/forum?id=bp-LJ4y_XC

3) Selected benchmarks for testing EGRU are not appropriate. For instance, sMNIST is a solved problem already with 99% accuracy of CoRNN that even outperforms EGRU (We need more clarifications on this as well). Even DVS is almost solved (98% from [8]). Instead, I suggest the authors try to use more challenging and up-to-date benchmarks such as Long Range Arena [7], audio datasets, and larger language benchmarks. 


For the above fundamental reasons, I vote for the rejection of this paper and encourage the authors to incorporate these critical points in their next submission. 



confidence: Certain
Decision: Reject