Venue: ICLR 2017 workshop

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper uses the sparsity-inducing techniques of Han et al 2016 and Narang et al 2017 for the LSTM used in image captioning. This amounts to four techniques Type 1 ... Type 4, with 1,1,3,4 hyperparameters respectively. It is not clear from the paper how these hyperparameters are set or how they were tuned, except for the mentioned 80% sparsity. The experiments consist of running the 4 types and comparing them to a baseline, which uses the full LSTM. One has to squint quite hard to see which sparsity approach works best, and the result tables don't offer very consistent takeaways.

Overall, it is not clear if these results should be a paper by itself, or merely one section in either Han et al 2016 or Narang et al 2017 papers, both of which report experiments with RNNs already. Consistent with the results already presented in these much more thorough papers, the added sparsity appears to have a small regularizing effect. Therefore, it is not clear what value is added with this work.
review_title: null

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper explores regularization of LSTM-style recurrent networks using sparsity, showing improved image captioning performance from hybrid CNN-LSTM models and comparing different approaches to applying sparsification.


Pros:

Explores sparsity a regularization technique for LSTMs, a widely used recurrent network architecture.

Analyzes several sparsification protocols, varying the rate and timing of sparsification during training.

Experiments show that the simplest sparsification approach (Type I) where sparsification is applied only once after a single iteration of training gets consistently good performance, better than the baseline and mostly better than the more complex protocols (Types II-IV).


Cons:

Not much novelty -- the aim of the paper is to compare two existing RNN sparsification methods (Han et al 2016b & Narang et al. 2017) for image captioning with LSTMs.  It’s not clear that the experimental results would consistently generalize to other problems or non-RNN architectures. (Han et al. 2016b, the source of the sparsification method, includes experiments in other problem settings and with pure convnets, in addition to image captioning RNNs.)

The largest performance improvements from sparsification aren’t very big, and the runtime improvement isn’t quantified (I would guess it’s relatively small in this case, with the convnet being the bulk of the execution time).

Only the sparsification schedule is varied in the experiments.  The paper could explore the impact of other hyperparameters, such as the sparsity proportion (all experiments use 80% sparsity).

Minor: the “rough ranking” in Sec 3.1 claims Type III is generally the worst performing approach -- this is true for the MSCOCO results, but it seems to actually perform the best for Flickr8k (except with BLEU-4).

Minor: Figure 4 sparsity patterns -- if the only takeaway from these is that different LSTM gates have different degrees of sparsity, these visualizations could be replaced with a simple table or plot of that summary statistic for each gate.  The actual patterns (besides the overall “darkness” indicating the degree of sparsity) seem like noise.


Overall, the paper doesn’t propose anything new, and though the evaluation may be a useful reference for practitioners, its scope is too limited given that only the sparsification schedule is explored.
review_title: null

Decision: Reject