Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Quality: 
The motivation and experimentation is sound.

Originality:
This work is a natural follow up on previous work that used group lasso for CNNs, namely learning sparse RNNs with group-lasso. Not very original, but nevertheless important.

Clarity:
The fact that the method is using a group-lasso regularization is hidden in the intro section and only fully mentioned in section 3.2 I would mention that clearly in the abstract.

Significance:
Leaning small models is important and previous sparse RNN work (Narang, 2017) did not do it in a structured way, which may lead to slower inference step time. So this is an investigation of interest for the community.

Minor comments:
- One main claim in the paper is that group lasso is better than removing individual weights, yet not experimental evidence is provided for that.
- The authors found that their method beats "direct design". This is somewhat unintuitive, yet no explanation is provided. 
review_title: Nice work on RNN compression 

final_score: rating: 7: Good paper, accept
review_content: The authors propose a technique to compress LSTMs in RNNs by using a group Lasso regularizer which results in structured sparsity, by eliminating individual hidden layer inputs at a particular layer. The authors present experiments on unidirectional and bidirectional LSTM models which demonstrate the effectiveness of this method. The proposed techniques are evaluated on two models: a fairly large LSTM with ~66.0M parameters, as well as a more compact LSTM with ~2.7M parameters, which can be sped up significantly through compression.
Overall this is a clearly written paper that is easy to follow, with experiments that are well motivated. To the best of my knowledge most previous papers in the area of RNN compression focus on pruning or compression of the node outputs/connections, but do not focus as much on reducing the computation/parameters within an RNN cell. I only have a few minor comments/suggestions which are listed below:

1. It is interesting that the model structure where the number of parameters is reduced to the number of ISSs chosen from the proposed procedure does not attain the same performance as when training with a larger number of nodes, with the group lasso regularizer. It would be interesting to conduct experiments for a range of \lambda values: i.e., to allow for different degrees of compression, and then examine whether the model trained from scratch with the “optimal” structure achieves performance closer to the ISS-based strategy, for example, for smaller amounts of compression, this might be the case?

2. In the experiment, the authors use a weaker dropout when training with ISS. Could the authors also report performance for the baseline model if trained with the same dropout (but without the group LASSO regularizer)?

3. The colors in the figures: especially the blue vs. green contrast is really hard to see. It might be nicer to use lighter colors, which are more distinct.

4. The authors mention that the thresholding operation to zero-out weights based on the hyperparameter \tau is applied “after each iteration”. What is an iteration in this context? An epoch, a few mini-batch updates, per mini-batch? Could the authors please clarify.

5. Clarification about the hyperparameter \tau used for sparsification: Is \tau determined purely based on the converged weight values in the model when trained without the group LASSO constraint? It would be interesting to plot a histogram of weight values in the baseline model, and perhaps also after the group LASSO regularized training.

6. Is the same value of \lambda used for all groups in the model? It would be interesting to consider the effect of using stronger sparsification in the earlier layers, for example.

7. Section 4.2: Please explain what the exact match (EM) and F1 metrics used to measure performance of the BIDAF model are, in the text. 

Minor Typographical/Grammatical errors:
- Sec 1: “... in LSTMs meanwhile maintains the dimension consistency.” → “... in LSTMs while maintaining the dimension consistency.”
- Sec 1: “... is public available” → “is publically available”
- Sec 2: Please rephrase: “After learning those structures, compact LSTM units remain original structural schematic but have the sizes reduced.”
- Sec 4.1: “The exactly same training scheme of the baseline ...” → “The same training scheme as the baseline ...”
review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper spends lots of (repeated)  texts on motivating and explaining ISS. But the algorithm is simple, using group lasso to find components that are can retained to preserve the performance.  Thus the novelty is limited.

The experiments results are good.

Sec 3.1 should be made more concise. 
review_title: the text is verbose but method is simple

Meta-review: The reviewers really liked this paper. This paper presents a tweak to the LSTM cell that introduces sparsity, thus reducing the number of parameters in the model.

The authors show that their sparse models match the performance of the non-sparse baselines. While the results are not state-of-the-art but vanilla implementations of standard models, this is still of interest to the community.
Decision: Accept (Poster)