Task: Language Identification
Input (Spanish, English): This paper investigates three simple weight-pruning techniques for NMT, and shows that pruning weights based on magnitude works best, and that retraining after pruning can recover original performance, even with fairly severe pruning.  The main strength of paper is that the technique is very straightforward and the results are good. Itâs also clearly written and does a nice job covering previous work.  A weakness is that the work isnât very novel, being just an application of a known technique to a new kind of neural net and application (namely NMT), with results that arenât very surprising.   Itâs not clear to me what practical significance these results have, since to take advantage of them you would need sparse matrix representations, which are trickier to get working fast on a GPU - and after all, speed is the main problem with NMT, not space. (There may be new work that changes this picture, since the field is evolving fast, but if so you need to describe it, and generally do a better job explaining why we should care about pruning.)  A suggestion for dealing with the above weakness would be to use the pruning results to inform architecture changes. For instance, figure 3 suggests that you might be able to reduce the number of hidden layers to two, and also potentially reduce the dimension of source and target embeddings.  Another suggestion is that you try to make a link between pruning+retraining and dropout (eg âA Theoretically Grounded Application of Dropout in Recurrent Neural Networksâ, Gal, arXiv 2016).  Detailed comments:  Line 111: âsoftmax weightsâ - âoutput embeddingsâ may be a preferable term  S3.2: Itâs misleading to call n the âdimensionâ of the network, and specify all parameter sizes as integer multiples of this number as if this were a logical constraint.  Line 319: You should cite Bahdanau et al here for the attention idea, rather than Luong for their use of it.  S3.3: Class-uniform and class-distribution seem very similar (and naturally get very similar results); consider dropping one or the other.  Figure 3 suggestion that you could hybridize pruning: use class-blind for most classes, but class-uniform for the embeddings.  Figure 4 should show perplexity too.  What pruning is used in section 4.2 & figure 6?  Figure 7: does loss pertain to training or test corpora?  Figure 8: This seems to be missing softmax weights. I found this diagram somewhat hard to interpret; it might be better to give relevant statistics, such as the proportion of each class that is removed by class-blind pruning at various levels.  Line 762: You might want to cite Le et al, âA Simple Way to Initialize Recurrent Networks of Rectified Linear Unitsâ, arXiv 2015.
Output (English): en