Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose a biologically inspired event-generating mechanism that they integrate within GRU units for enhancing activity sparsity and reducing train/test compute budget. They show that the time complexity for computing weight updates in the continuous time limit is proportional to the number of events generated in EGRU. The authors show competitive performance of their sparse EGRU network on multiple machine learning benchmarks related to RNNs despite the network producing high activity sparsity and low effective MACs (highly desirable).
strength_and_weaknesses: Strengths:
1) While recurrent neural networks come with several advantages such as low parameters (owing to weight tying), ability to effectively integrate long-range dependencies and model sophisticated functions with nonlinear dynamics, their relevance for applications is restricted by their high computational budget especially during training caused by BPTT. Proposed work addresses this central issue of RNNs by taking inspiration from sparsity in biological neurons. 
2) Evaluations performed on DVS gesture recognition, sequential MNIST and language modeling highlight that EGRU can match/outperform related RNN models in task performance while using a relatively much lesser computational requirements.
3) The potential to use neuromorphic hardware to maximize the gains produced by sparse activations of EGRU is very exciting.

Weaknesses:
1) Some of the compared baselines aren't strong enough. E.g. in Table.1, it looks like DA produces a big gain in performance on EGRU. Why did the authors not evaluate GRU or LSTM with DA? What is the CNN+EGRU adding to the comparison? It looks like the effective MACs are much higher on CNN+EGRU networks in return for a marginal gain in test accuracy. On a related note in Table 3, I don't see any difference in number of parameters between EGRU and GRU with the same hidden dim in Table 1. Why is there a difference between matched GRU and EGRU model number of parameters in table 3?
2) What are some of the limitations of the proposed EGRU model? It would be great if the authors could please further highlight room for improvement of EGRU apart from combining with neuromorphic devices in the discussion.
clarity,_quality,_novelty_and_reproducibility: The proposed work is novel and technically sound. The authors have done a good job explaining clearly their proposed EGRU model and presented ample experimental evidence to support the high performance of EGRU under high sparsity and low computational requirements. 
summary_of_the_review: This work proposes an effective solution to training RNNs with less computational requirements using activity sparsity found in biological neurons. The authors have theoretically proven that the time complexity to calculate EGRU's weight updates grows linearly with number of events, and evaluated their EGRU model rigorously on a variety of RNN-relevant benchmarks and compared performance against a set of well chosen baselines. It is clear that the proposed EGRU model is capable of achieving high performance with high activity sparsity and potential gains are even larger when EGRU is implemented on suitable neuromorphic hardware. This work is highly promising and is of great relevance to ICLR audience and hence, I recommend acceptance of this paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Authors show that by including the biologically grounded constraint that neural activity is sparse, learning in a recurrent neutral network can be improved, both in the areas of computation (memory footprint,...) and accuracy. Using a continuous time extension of the recurrent neural network model, they theoretically prove the properties of the proposed model, and in particular the modulation of the computational cost with sparsity of the activity. The numerical results provide encouraging results for applying this methodology to generic networks and in particular their use in neuromorphic hardware.

strength_and_weaknesses: The strength of the paper lies in the focus on the sparsity inducing mechanism and its theoretical and numerical application. In particular, the success of several benchmarks such as DVS gesture, sequential MNIST and natural language processing clearly demonstrate the advantage of this method. 
One weakness of the paper lies in the lack of analysis of the encoding resulting from the introduction of the sparsity induction mechanism. For example, Figure 2 shows the activity produced by a segment of the DVS gesture dataset, the output of which in the different layers is qualitatively very different from the raster plots observed in biology. Further analysis of the average frequency of completion could help to understand how this could be improved, for example by incorporating homeostatic mechanisms that optimize the information carried by each neuron (for simplicity, a neuron that does not respond or always responds carries no information about the sensory input). Also, what is the influence of thresholding on accuracy? This would be an added value to the article.
clarity,_quality,_novelty_and_reproducibility: The article is generally very clearly written and the model is well presented and theoretically analyzed. Minor: one redundant sentence in the first sentences of paragraphs 4 and 4. 1. In addition, the same number of digits should be used for precision in Table 1. 
summary_of_the_review: In summary, the authors provide an important and novel contribution to recurrent neural networks that have important applications to neuromorphic hardware. The work could be further improved by showing mechanically why their solution is improved over SOTA.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors present an event-based version of GRUs that allows for sparse forward and backward passes that achieves similar (sometimes even better) task performance than vanilla GRU but at a lower compute cost. The authors also present theoretical results indicating that the computation complexity doesn't scale with the network size, but instead with the number of events. Although it is unclear how does the number of events scale with network size (I personally feel that it would depend on the task and training loss function, and is therefore hard to make a generic statement), the implications of this result are significant in terms of efficiently scaling up network size for solving complicated tasks. In this work, the authors support their claims with empirical evidence on gesture recognition, sequential MNIST and language modeling tasks. 
strength_and_weaknesses: Strengths:
1. The paper presents a strong motivations for their solution and is overall well-written.
2. The theoretical results present strong evidence of why their solution enables efficient scaling up of models.
3. The EGRU architecture contributes to reducing the memory footprint of BPTT, which is indeed a significant step towards making BPTT implementable in long sequences (instead of relying of truncated BPTT). It would be interesting to see how EGRU performs with some of the biologically-proposed temporal credit assignment rules, that rely on erasing out credit signals carried over long temporal horizons.

Weaknesses:
1. I feel the EGRU is quite similar to the Identity-RNNs proposed by Le, Jaitly & Hinton in 2015, wherein they used a ReLU non-linearity with RNNs (already cited by the author in the context of sequential MNIST experiments). Although EGRU has a mechanism to avoid multiple firing of the same unit, which is different from the Identity-RNN formulation, I would be curious to know any corresponding links/ comparison to the IRNN architecture. 
2. Since one of the central claims that the authors make relates to scalability, it would be nice to see how the performance and activity/backward sparsitiy changes when the number of hidden units in EGRU are changed. A trend to empiricially demonstrate how these quantities change with scaling up the network would be useful.
3. A minor point (feel free to ignore if it is more along the directions in future work): it would be interesting to see how the sparsity changes across training. The authors currently report the average sparsity of the network but I would be curious to know if the operational sparsity is lower than that of an untrained network because the network eventually converged to such a solution. 
clarity,_quality,_novelty_and_reproducibility: The writing is quite clear and the ideas and motivations are well-presented. 
Although the idea is not completely novel, I feel this is a valuable piece of work as it presents an architecture leveraging that idea. As mentioned above, I would be keen to see an connection/comparison with the IRNN architecture. 
The work seems to be reproducible, although I haven't read the experiment details in the appendix. Also, the authors propose to make the code base available which should be a valuable resource for the community.
summary_of_the_review: Overall, I would recomment accepting the paper. I feel the paper can be slightly improved by updating the motivation by talking about connections to ReLU non-linearity for RNNs, which are also supposed to sparsify the activations. I have currently rated the paper to be marginally above the acceptance threshold, but if the authors can address some of the weaknesses mentioned above, I am happy to increase it to 8.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper extends gated recurrent unit (GRU) models by adding an event generating mechanism (making an event-based GRU, or EGRU) that is more sparse and discrete in its activations over time. The authors provide theoretical and empirical data to show that this can permit much more computationally efficient recurrent neural network training without sacrificing performance.

The reviewers all agreed that this is a strong paper that makes an important contribution. There were some concerns related to whether the tests were extensive enough and comparisons to other work sufficient. After the authors' response, the reviewers were satisfied that their concerns were addressed, and there was a clear consensus that this paper should be accepted.
justification_for_why_not_higher_score: Though this paper provides a very nice, clear contribution for making RNNs more energy efficient it is not conceptually or empirically ground-breaking. Thus, although it is worthy of a spotlight, given its high quality, it is probably not appropriate for an oral.
justification_for_why_not_lower_score: The paper is well written and the work very solid. There is no doubt this paper should be accepted. Moreover, given the clear agreement between reviewers that it makes a worthwhile contribution, I think it is worthy of a spotlight.
Decision: Accept: notable-top-25%