Venue: ICLR 2017 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Dyvedeep presents three approximation techniques for deep vision models aimed at improving inference speed.
The techniques are novel as far as I know.
The paper is clear, the results are plausible.

The evaluation of the proposed techniques is does not make a compelling case that someone interested in faster inference would ultimately be well-served by a solution involving the proposed methods.

The authors delineate "static" acceleration techniques (e.g. reduced bit-width, weight pruning) from "dynamic" acceleration techniques which are changes to the inference algorithm itself. The delineation would be fine if the use of each family of techniques were independent of the other, but this is not the case. For example, the use of SPET would, I think, conflict with the use of factored weight matrices (I recall this from http://papers.nips.cc/paper/5025-predicting-parameters-in-deep-learning.pdf, but I suspect there may be more recent work). For this reason, a comparison between SPET and factored weight matrices would strengthen the case that SPET is a relevant innovation. In favor of the factored-matrix approach, there would I think be fewer hyperparameters and the computations would make more-efficient use of blocked linear algebra routines--the case for the superiority of SPET might be difficult to make.

The authors also do not address their choice of the Xeon for benchmarking, when the use cases they identify in the introduction include "low power" and "deeply embedded" applications. In these sorts of applications, a mobile GPU would be used, not a Xeon. A GPU implementation of a convnet works differently than a CPU implementation in ways that might reduce or eliminate the advantage of the acceleration techniques put forward in this paper.

review_title: Why benchmark techniques for IoT on a Xeon?

final_score: rating: 7: Good paper, accept
review_content: This work proposes a number of approximations for speeding up feed-forward network computations at inference time. Unlike much of the previous work in this area which tries to compress a large network, the authors propose algorithms that decide whether to approximate computations for each particular input example. 

Speeding up inference is an important problem and this work takes a novel approach. The presentation is exceptionally clear, the diagrams are very beautiful, the ideas are interesting, and the experiments are good. This is a high-quality paper. I especially enjoyed the description of the different methods proposed  (SPET, SDSS, SFMA) to exploit patterns in the classifer. 

My main concern is that the significance of this work is limited because of the additional complexity and computational costs of using these approximations. In the experiments, the DyVEDeep approach was compared to serial implementations of four large classification models --- inference in these models is order of magnitudes faster on systems that support parallelization. I assume that DyVEDeep has little-to-no performance advantage on a system that allows parallelization, and so anyone looking to speed up their inference on a serial system would want to see a comparison between this approach and the model-compression approaches. Thus, I am not sure how much of an impact this approach can have in it's current state.

Suggestions:
-I wondered what (if any) bounds could be made on the approximation errors of the proposed methods?
review_title: Interesting ideas, but I'm not sure about the significance.

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors describe a series of techniques which can be used to reduce the total amount of computation that needs to be performed in Deep Neural Networks. The authors propose to selectively identify how important a certain set of computations is to the final DNN output, and to use this information to selectively skip certain computations in the network. As deep learning technologies become increasingly widespread on mobile devices, techniques which enable efficient inference on such devices are becoming increasingly important for practical applications. 

The paper is generally well-written and clear to follow. I had two main comments that concern the experimental design, and the relationship to previous work:

1. In the context of deployment on mobile devices, computational costs in terms of both system memory as well as processing are important consideration. While the proposed techniques do improve computational costs, they don’t reduce model size in terms of total number of parameters. Also, the gains obtained using the proposed method appear to be similar to other works that do allow for improvements in terms of both memory and computation (see, e.g., (Han et al., 2015)). It would have been interesting if the authors had reported results when the proposed techniques were applied to models that have been compressed in size as well.

S. Han, H. Mao, and W. J. Dally. "Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding." arXiv prepring arXiv:1510.00149 (2015).

2. The SDSS technique in the paper appears to be very similar to the “Perforated CNN” technique proposed by Figurnov et al. (2015). In that work, as in the authors work, CNN activations are approximated by interpolating responses from neighbors. The authors should comment on the similarity and differences between the proposed method and the referenced work.

Figurnov, Michael, Dmitry Vetrov, and Pushmeet Kohli. "Perforatedcnns: Acceleration through elimination of redundant convolutions." arXiv preprint  arXiv:1504.08362 (2015).

Other minor comments appear below:

3. A clarification question: In comparing the proposed methods to the baseline, in Section 4, the authors mention that they used their own custom implementation. However, do the baselines use the same custom implementation, or do they used the optimized BLAS libraries?

4. The authors should also consider citing the following additional references:
 * S. Tan and K. C. Sim, "Towards implicit complexity control using variable-depth deep neural networks for automatic speech recognition," 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Shanghai, 2016, pp. 5965-5969.
 * Graves, Alex. "Adaptive Computation Time for Recurrent Neural Networks." arXiv preprint arXiv:1603.08983 (2016).

5. Please explain what the Y-axis in Figure 7 represents in the text.

6. Typographical Error: Last paragraph of Section 2: “... are qualitatively different the aforementioned ...” → “... are qualitatively different from the aforementioned ...”
review_title: Interesting and clearly written paper. My main concerns about this paper, are about the novelty, and the advantages of the proposed techniques over related papers in the area.

final_score: rating: null
review_content: I would like to clarify something about the experiments reported in Section 4/5. On each of the test sets, do you tune the hyperparameters on a validation set, and then report performance improvements on a separate held-out set? Or do you tune and report results on the same set? Doing the latter might suggest stronger performance improvements/accuracies then might be obtained on held-out data.

Also, how sensitive is the performance improvement to the choice of these hyperparameters?

Thanks in advance.
review_title: Clarification about hyperparameter tuning

final_score: rating: null
review_content: -
review_title: -

Meta-review: The Area Chair recommends to reject this paper given the reviewers concern about the limited significance of this work and the lack of comparisons. We encourage the authors to take into account the reviewers feedback and resubmit.
Decision: Reject