Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: In this paper, the authors design a big-little dual-module inference to dynamically skip unnecessary memory access and computation to speedup RNN inference.
It cannot only mitigate the memory-bound problem to speedup RNN inference but also leverage the error resilience of nonlinear activation functions by using the lightweight little module to compute for the insensitive region and using the big module with skipped memory access and computation. They also conduct several experiments to evaluate their approaches.


Strength:
(1)	Well written in general.
(2)	Contributions clearly stated and justified

A couple of minor questions:

(1)	The organization in Section 3 can to be improved. It is better if the author can give a brief overview of their method first and then go into details.
(2)	Some of the technical details necessary for understanding the soundness of the techniques are either missing or are poorly explained. For example, in Section 3
a.	the authors did not mention how to construct the HH module
b.	the authors did not provide detailed information of how to conduct dimension reduction since this will affect the performance
c.	many mathematical notations and equations need to be revised to increase the readability. For instance, there is no information in the paper that explain why the authors design functions in such certain way (such as equation (2) and (3))
d.	the authors did not provide enough detailed information about how to select the quantization methods since there are lots of approaches such as static (uniform) or dynamic quantization, where different methods may have different impacts on the final performance 
e.	the authors mentioned that they have tried both sigmoid and tanh activation function to find the sensitive region. However, they do not provide enough reason to do so, how about using other non-linear activation functions

(3)	The organization in Section 4 can to be improved. It is better if the author can introduce the motivation of each experiment.

(4)	Parameters of the evaluation are unclear or missing. For example, 
a.	what is the data size, what is the dropout, learning rate, how many time stamps for the RNN modules
b.	why the authors only use single-layer LSTM and why they select 750 and 1500 hidden units in the experiments

(5)	While the authors have applied their models on other existing method, they do not provide good discussion of results and such model seems old (released in 2016). It would be great if this approach can also be applied on other newly released models
(6)	Some tables need to be reorganized. For instance, for table 6, there needs some space between the title and the table.
(7)	While the paper has good coverage of the prior work, I do suggest the authors can also cite or discuss some newly designed models (in 2019).

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This manuscript proposes an approach to reduce memory access and computation in Recurrent Neural Networks.  Specifically, they train a second "little" neural network to approximate a pre-trained "big" network and use simple rules to switch between the little and the big network.  The approach can provide some speedups while reducing the total number of memory accesses and the computational cost in exchange for a mild decrease in predictive performance.

While this manuscript proposes a reasonable contribution, it lacks real comparisons to many of the common competing methods that hinder the interpretation.

Weight sparsity/pruning is a very common approach that has shown the ability for larger speedups than what is shown here.  I disagree with the assessment that "those methods require extensive retraining via regularization." Realistically, you can take a pretrained model and add the penalties with mild re-training and extensive reuse of code.  The result is also simpler to implement.  I would argue that this is less work than the proposed approach, which requires switching rules and a second trained network.  I don't know which is better, but the authors should actually evaluate whether their approach improves over the more popular approach.

The authors should give better discussion and motivation on the random projections.  This is an area with very deep theory, yet the rules are provided without a rationale.  Realistically, where does the sparsity level in the random sparse matrix come from?  Why use the rule for k in (3)?  The authors should motivate and discuss this section more.

Also note that there is existing literature on learning multiple models and switching between them, for example:
Bolukbasi, Tolga, et al. "Adaptive neural networks for efficient inference." Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017.
As there is a lot of similarity in the motivations, you should discuss that line of research in your related work.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper attempts to compress the networks so as to accelerate the running procedure as well as save the storage. The authors propose a dual-module that is composed of a little module and big module. The big module use the full original data and parameters whereas the little module use small data and parameters by random projecting on the original ones. Through a statistical investigation, the authors provide a method to choose the little or big module dynamically. By applying this method on LSTM and GRU, the authors make them more efficient. Experimental results validate this point.

Overall, I think this paper is well written and easy to follow. The idea of dual-module is interesting and wise. The experiments is valid. However, I would like the authors to answer me two questions. 
1 How do you draw Figure 2? 
2 Why is the distribution of the outputs of sigmoid not bipolar? Intuitively, the distribution should be similar to that of tanh since their functions are similar.

review_title: Official Blind Review #1

Meta-review: This paper presents an efficient RNN architecture that dynamically switches big and little modules during inference. In the experiments, authors demonstrate that the proposed method achieves favorable speed up compared to baselines, and the contribution is orthogonal to weight pruning. 
All reviewers agree that the paper is well-written and that the proposed method is easy to understand and reasonable. However, its methodological contribution is limited because the core idea is essentially the same as distillation, and dynamically gating the modules is a common technique in general. Moreover, I agree with the reviewers that the method should be compared with more other state-of-the-art methods in this context. Accelerating or compressing DNNs are intensively studied topics and there are many approaches other than weight pruning, as authors also mention in the paper. As the possible contribution of the paper is more on the empirical side, it is necessary to thoroughly compare with other possible approaches to show that the proposed method is really a good solution in practice. For these reasons, I’d like to recommend rejection. 

Decision: Reject