Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: Although I worked and published on network pruning in the past, I found little overlap between my knowledge and the content of this paper. I feel therefore that I can only provide a very superficial feedback.

The authors generalize recently-developed network pruning techniques that were developed in he context of feed-forward networks, and do not require iterative pruning-retraining cycles, to RNNs. Since RNNs apply the same matrix on their hidden vector multiple times, approximate closed-form expressions for pruning criteria can be derived analytically. The pruning criterion is based on the spectrum of the Jacobian matrix of the (N+1)-th hidden unit with respect to the N-th one. A closed-form algorithm for pruning is presented, and the method is evaluated on several datasets.

While I appreciate the neatness of deriving closed-form analytical expressions for a fast and simple pruning method, I feel that am not in a position to rate the paper. My apologies to the Authors and the Editors.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: 
Notes: 

  -RNN network pruning has proven to be challenging using the techniques often used with other network types.  

  -One issue is that the performance of an LSTM/GRU can hinge on a few activated gates and can lead to more concentration of influence than would be seen in a feedforward network without parameter sharing between layers.  

  -New objective uses 64 points to prune the network (I assume this is just the size of the minibatch).  

  -Result is a 95% sparse GRU cell.  

  -New idea is based on keeping weights that propagate information through many time steps.  

  -Encourages "singular values of the temporal jacobian with respect to network weights to be non-degenerate" (I suppose this means that the gradient flowing through time will contain multiple directions of variation)  

  -Introduction does a good job of introducing the key ideas.  

  -J_t is a temporal jacobian of size N x N (N is number units) at time step t.  

  -Chi is the spectral norm of this temporal jacobian.  I'm a bit confused by this, because my understanding is that the spectral norm is the largest singular value, but equation 3 looks like a sum over singular values, making it more like a frobenius norm?  

  -This jacobian isn't tractable, so paper approximates it using a first-order taylor expansion.  So basically the pruning just amounts to taking parameters with the largest gradient?  

  -Section 2.4 is confusing and seems to come out of nowhere.  Is this suggesting that the technique isn't just pruning but adding a new normalization scheme?  On second reading, this is a normalization scheme effecting which parameters to prune.  The motivation for why the gradients are normalized like this is still confusing.  If you're willing to make a linear assumption, it seems like it's enough to consider the gradient on the parameter multiplied by the magnitude of the parameter to see the overall effect of removing it?  

  -The results look good, but sequential mnist is a bit of a toy task.  I'd also like to see a more fine-grained analysis showing the tradeoff between the number of units removed and the performance.  

  -The paper claims that L2 pruning requires more data, but it's unclear if this really matters since the whole dataset was used to train both methods initially.  

  -On Table 2 the results of the technique don't seem that much better than "Random".  

  Review: This paper presented a fast pruning algorithm for RNNs, which uses the norm of the gradient as a guide to pruning.  I'm borderline on this paper.  The idea of using the gradient is good, but the explanation of some aspects like the normalization is confusing and felt random.  Additionally the results, while better than some other pruning techniques on RNNs, don't seem to be that much better than random.  
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: OVERALL:

I should first say that this is reasonably far outside my wheelhouse.
I have never worked on RNNs or pruning.
I also have no familiarity with the data sets used.

All these things being said, I can follow the derivations, and the idea
seems reasonable and well-motivated, and pruning is interesting
for both scientific and practical reasons, and this technique seems to help a substantial amount,
so I'm inclined to vote for acceptance, with the understanding that perhaps better informed reviewers
 will in the future point out something I have missed.

DETAILED NOTES:

> overparameterized networks require more storage capacity and are computationally more expensive than their pruned counterparts
I'm with you on the storage capacity, but do any of these pruned networks actually run faster than their non-pruned counterparts?
I thought you had to work really hard to prune to some kind of block-sparse representation to realize any speed gains.
This question is not rhetorical - I know very little about this topic.

> Work for the present volume began by asking the question...
I like this paragraph for motivation, but perhaps 'volume' is slightly overwrought?

> For our pruning objective, we simply take the
K weights with largest sensitivity scores, as those represent the parameters which most affect the
Jacobian objective near the initialization.
Is there some notion of redundancy, where certain sets of parameters affect the jacobian in the same way,
so that all but 1 element of the set could be pruned, or something?

In line 13 of algorithm 1, why do we need to sort if in the next step we just mask out
everything with sensitivity less than D tilde k?

Maybe this is a dumb question, but if you're pruning at initialization, why not
just initialize a smaller network in such a way that you wouldn't choose to prune any of its parameters?
Am I misunderstanding what you're doing?


> In the prequel, we postulated t
The prequel?

Fig 1 is interesting, but it raises the question of whether you could recover a simpler
algorithm by just modifying random pruning so that it evenly distributes 'prunes' across
gate and Type.

In fig 2, why are all the singular values less than 1?
It's not obvious to me why that should be true, unless you enforce it w/ the initialization.

review_title: Official Blind Review #2

Meta-review: Based on current unanimous reviews, the paper is accepted.
Decision: Accept (Poster)