Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: A main problem with RNN is to update all hidden dimensions in each time step. The authors proposed selective-activation RNN (SA-RNN), which modifies each state of RNN by adding an update coordinator which is modeled as a lightweight neural network. The coordinator, based on the incoming data, makes a discrete decision to update or not update each individual hidden dimension. A multi-objective optimization problem is defined to both solving a sequential learning task and minimizing the number of updates in each time step. The authors evaluated their networks on three public benchmark datasets and achieved good results compared to the state-of-the-art ones.
The papers is well-written. The idea proposed in this paper is interesting and it is presented very well. There is also an extensive evaluation.

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper attempts to reduce computation in recurrent neural networks. Instead of artificially determining the update pattern for updating the states, the authors propose SA-RNN to predict discrete update patterns automatically through optimization driven entirely by the input data. Experiments on publicly-available datasets show that the proposed method has competitive performance with even fewer updates.

Pros:
Overall, I think the idea of this paper is clear and the whole paper is easy to follow. The experiments clearly show the advantage of the proposed method claimed by the authors.

Cons:
1.	Some expressions need to be improved. For example, in “This way, representations can be learned while solving a sequential learning task while minimizing the number of updates, subsequently reducing compute time.” two “while”s are not elegant and there should be an “In” before “this way”. In “We augment an RNN with an update coordinator that adaptively controls the coordinate directions in which to update the hidden state on the fly”, the usage of “in which to” is not right. I suggest the authors to thoroughly proofread the whole paper and improve the presentation.
2.	Since this paper focuses on the efficiency of RNN, I suggest the authors could provide the time complexity comparisons. Merely the comparisons on skip of neurons cannot show the advantage on the efficiency.

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: Summary: This paper proposes selective activation RNN (SA-RNN), by using an update coordinator to determine which subset of the RNN’s hidden state dimensions should be updated at a given timestep. The proposed loss term is then a sum of the original objective (e.g. classification) and a weighted sum of the probability that each dimension will be updated for each timestep. The method is evaluated on 3 time series datasets: Seizures, TwitterBuzz, Yahoo. 

Decision: Weak Reject. Although the authors tackle a challenging problem, their empirical results are lacking to provably demonstrate that their approach outperforms existing baselines.

Supporting Arguments/Feedback: The authors compare SA-RNN to 5 baselines: random updates, clockwork RNN, phased LSTM, Skip RNN, and VC-GRU. Although I appreciated the authors’ comparison across the suite of methods with respect to various metrics (e.g. # FLOPS, proportion of neurons that weren’t updated, etc.), the experiments were conducted on datasets that were relatively simple. For example, in prior work, the empirical evaluations were on much larger-scale datasets such as Wikipedia [Shen et. al 2019], real clinical data sources [Liu et. al 2018], and Charades videos [Campos et. al 2018], among others. I would be very interested to see how this training procedure fairs when evaluated on much more complex tasks, and would make the results about computational speedups at train/test time much more convincing.

Questions:
- I’m curious if you tried different types of gradient estimators to get around the non-differentiability rather than the straight-through estimator. Also how was the slope-annealing conducted (e.g. annealing schedule)?

review_title: Official Blind Review #2

Meta-review: This paper introduces a new RNN architecture which uses a small network to decide which cells get updated at each time step, with the goal of reducing computational cost.  The idea makes sense, although it requires the use of a heuristic gradient estimator because of the non-differentiability of the update gate.

The main problem with this paper in my view is that the reduction in FLOPS was not demonstrated to correspond to a reduction in wallclock time, and I don't expect it would, since the sparse updates are different for each example in each batch, and only affect one hidden unit at a time.  The only discussion of this problem is "we compute the FLOPs for each method as a surrogate for wall-clock time, which is hardware-dependent and often fluctuates dramatically in practice."  Because this method reduces predictive accuracy, the reduction in FLOPS should be worth it!

Minor criticism:
1) Figure 1 is confusing, showing not the proposed architecture in general but instead the connections remaining after computing the sparse updates.

Decision: Reject