The authors propose a technique to compress LSTMs in RNNs by using a group Lasso regularizer which results in structured sparsity, by eliminating individual hidden layer inputs at a particular layer. The authors present experiments on unidirectional and bidirectional LSTM models which demonstrate the effectiveness of this method. The proposed techniques are evaluated on two models: a fairly large LSTM with ~66.0M parameters, as well as a more compact LSTM with ~2.7M parameters, which can be sped up significantly through compression. Overall this is a clearly written paper that is easy to follow, with experiments that are well motivated. To the best of my knowledge most previous papers in the area of RNN compression focus on pruning or compression of the node outputs/connections, but do not focus as much on reducing the computation/parameters within an RNN cell. I only have a few minor comments/suggestions which are listed below: 1. It is interesting that the model structure where the number of parameters is reduced to the number of ISSs chosen from the proposed procedure does not attain the same performance as when training with a larger number of nodes, with the group lasso regularizer. It would be interesting to conduct experiments for a range of \lambda values: i.e., to allow for different degrees of compression, and then examine whether the model trained from scratch with the “optimal” structure achieves performance closer to the ISS-based strategy, for example, for smaller amounts of compression, this might be the case? 2. In the experiment, the authors use a weaker dropout when training with ISS. Could the authors also report performance for the baseline model if trained with the same dropout (but without the group LASSO regularizer)? 3. The colors in the figures: especially the blue vs. green contrast is really hard to see. It might be nicer to use lighter colors, which are more distinct. 4. The authors mention that the thresholding operation to zero-out weights based on the hyperparameter \tau is applied “after each iteration”. What is an iteration in this context? An epoch, a few mini-batch updates, per mini-batch? Could the authors please clarify. 5. Clarification about the hyperparameter \tau used for sparsification: Is \tau determined purely based on the converged weight values in the model when trained without the group LASSO constraint? It would be interesting to plot a histogram of weight values in the baseline model, and perhaps also after the group LASSO regularized training. 6. Is the same value of \lambda used for all groups in the model? It would be interesting to consider the effect of using stronger sparsification in the earlier layers, for example. 7. Section 4.2: Please explain what the exact match (EM) and F1 metrics used to measure performance of the BIDAF model are, in the text. Minor Typographical/Grammatical errors: - Sec 1: “... in LSTMs meanwhile maintains the dimension consistency.” → “... in LSTMs while maintaining the dimension consistency.” - Sec 1: “... is public available” → “is publically available” - Sec 2: Please rephrase: “After learning those structures, compact LSTM units remain original structural schematic but have the sizes reduced.” - Sec 4.1: “The exactly same training scheme of the baseline ...” → “The same training scheme as the baseline ...”