Summary: This paper presents a thorough examination of the effects of pruning on model performance.   Importantly, they compare the performance of "large-sparse" models (large models that underwent pruning in order to reduce memory footprint of model) and "small-dense" models, showing that "large-sparse" models typically perform better than the "small-dense" models of comparable size (in terms of number of non-zero parameters, and/or memory footprint).   They present results across a number of domains (computer vision, language modelling, and neural machine translation) and model types (CNNs, LSTMs).   They also propose a way of performing pruning with a pre-defined sparsity schedule, simplifying the pruning process in a way which works across domains.   They are able to show convincingly that pruning is an effective way of trading off accuracy for model size (more effective than simply reducing the size of model architecture),  although there does come a point where too much sparsity degrades the model performance considerably;  this suggests that pruning a medium size model to 80%-90% sparsity is likely better than pruning a larger model to >= 95% sparsity. Review: Quality: The quality of the work is high  --- the experiments are extensive and thorough.   I would have liked to see "small-dense" vs. "large-sparse" comparisons on Inception (only large-sparse results are reported). Clarity: The paper is clearly written,  though there is room for improvement.  For example, many of the results are presented in a redundant manner (in both tables and figures, where the table and figure are often not next to each other in the document).   Also, it is not clear in several cases exactly which training/heldout/test sets are used, and on which partition of the data the accuracies/BLEU scores/perplexities presented correspond to.  A small section (before "Methods") describing the datasets/features in detail would be helpful.   Also, it would have probably been nice to explain all of the tasks and datasets early on, and then present all the results at once (NIT: include the plots in paper, and move the tables to an appendix). Originality: Although the experiments are informative,  the work as a whole is not very original.   The method proposed of using a sparsity schedule to perform pruning is simple and effective,  but is a rather incremental contribution.  The primary contribution of this paper is its experiments, which for the most part compare known methods. Significance: The paper makes a nice contribution,  though it is not particularly significant or surprising.   The primary observations are: (1) large-sparse is typically better than small-dense, for a fixed number of non-zero parameters and/or memory footprint. (2) There is a point at which increasing the sparsity percentage severely degrades the performance of the model,  which suggests that there is a "sweet-spot" when it comes to choosing the model architecture and sparsity percentage which give the best performance (for a fixed memory footprint). Result #1 is not very surprising,  given that Han et al (2016) were able to show significant compression without loss in accuracy;  thus, because one would expect a smaller dense model to perform worse than the large dense model,  it would also perform worse than the large sparse model. Result #2 had already been seen in Han et al (2016) (for example, in Figure 6). Pros: - Very thorough experiments across a number of domains Cons: - Methodological contributions are minor. - Results are not surprising, and are in line with previous papers.