Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a method to build a CNN in the Winograd domain, where weight pruning and ReLU can be applied in this domain to improve sparsity and reduce the number of multiplication. The resultant CNN can achieve ~10x theoretical speedup with little performance loss.

The paper is well-written. It provides a new way to combine the Winograd transformation and the threshold-based weight pruning strategy. Rather than strictly keeping the architecture of ordinary CNNs, the proposed method applied ReLU to the transform domain, which is interesting.  

The results on Cifar-10 and ImageNet are promising. In particular, the pruned model in the Winograd domain performs comparably to the state-of-the-art dense neural networks and shows significant theoretical speedup. 
The results on ImageNet using ResNet-18 architecture are also promising. However, no results are provided for deeper networks, so it is unclear how this method can benefit the computation of very deep neural networks 

A general limitation of the proposed method is the network architecture inconsistency with the ordinary CNNs. Due to the location change of ReLUs, it is unclear how to transform a pretrained ordinary CNNs to the new architectures accurately. It seems training from scratch using the transformed architectures is the simplest solution. 

The paper does not report the actual speedup in the wall clock time. The actual implementation is what matters in the end. 

It will be more informative to present Figure 2,3,4 with respect to the workload in addition to the weight density. 


review_title: A promising Method, though with some limitations

final_score: rating: 7: Good paper, accept
review_content: This paper proposes to combine Winograd transformation with sparsity to reduce the computation for deep convolutional neural network. Specifically, ReLU nonlinearity was moved after Winograd transformation to increase the dynamic sparsity in the Winograd domain, while an additional pruning on low magnitude weights and re-training procedure based on pruning is used to increase static sparsity of weights, which decreases computational demand. The resulting Winograd-ReLU
CNN shows strong performance in three scenarios (CIFAR10 with VGG, CIFAR100 with ConvPool-CNN-C, and ImageNEt with ResNet-18). The proposed method seems to improve over the two baseline approaches (Winograd and sparsity, respectively).

Overall, the paper is well-written and the experiments seems to be quite thorough and clear. Note that I am not an expert in this field and I might miss important references along this direction. I am leaving it to other reviewers to determine its novelty. 

Putting ReLU in the Winograd domain (or any transformed domain, e.g., Fourier) seems to be an interesting idea, and deserves some further exploration. Also, I am curious about the performance after weight pruning but before retraining).
review_title: Good paper with thorough experiments

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary: 
The paper presents a modification of the Winograd convolution algorithm that enables a reduction of multiplications in a forward pass of 10.8x almost without loss of accuracy. 
This modification combines the reduction of multiplications achieved by the Winograd convolution algorithm with weight pruning in the following way:
- weights are pruned after the Winograd transformation, to prevent the transformation from filling in zeros, thus preserving weight sparsity
- the ReLU activation function associated with the previous layer is applied to the Winograd transform of the input activations, not directly to the spatial-domain activations, also yielding sparse activations

This way sparse multiplication can be performed. Because this yields a network, which is not mathematically equivalent to a vanilla or Winograd CNN, the method goes through three stages: dense training, pruning and retraining. The authors highlight that a dimension increase in weights and ReLU activations provide a more powerful representation and that stable dynamic activation densities over layer depths benefit the representational power of ReLU layers.

Review:
The paper shows good results using the proposed method and the description is easy to follow. I particularly like Figure 1. 
I only have a couple of questions/comments:
1) I’m not familiar with the term m-specific (“Matrices B, G and A are m-specific.”) and didn’t find anything that seemed related in a very quick google search. Maybe it would make sense to add at least an informal description.
2) Although small filters are the norm, you could add a note, describing up to what filter sizes this method is applicable. Or is it almost exactly the same as for general Winograd CNNs?
3) I think it would make sense to mention weight and activation quantization in the intro as well (even if you leave a combination with quantization for future work), e.g. Rastegari et al. (2016), Courbariaux et al. (2015) and Lin et al. (2015)
4) Figure 5 caption has a typo: “acrruacy”

References:
Courbariaux, Matthieu, Yoshua Bengio, and Jean-Pierre David. "Binaryconnect: Training deep neural networks with binary weights during propagations." In Advances in Neural Information Processing Systems, pp. 3123-3131. 2015.
Lin, Zhouhan, Matthieu Courbariaux, Roland Memisevic, and Yoshua Bengio. "Neural networks with few multiplications." arXiv preprint arXiv:1510.03009 (2015).
Rastegari, Mohammad, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. "Xnor-net: Imagenet classification using binary convolutional neural networks." In European Conference on Computer Vision, pp. 525-542. Springer International Publishing, 2016.
review_title: Well-written paper introducing a novel method of reducing multiplications in CNNs with very minor loss in accuracy

Meta-review: The paper presents a modification of the Winograd convolution algorithm that reduces the number of multiplications in a forward pass of a CNN with minimal loss of accuracy. The reviewers brought up the strong results, the readability of the paper, and the thoroughness of the experiments. One concern brought up was the applicability to deeper network structures. This was acknowledged by the authors to be a subject of future work. Another issue raised was the question of theoretical vs. actual speedup. Again, this was acknowledged by the authors to be an eventual goal but subject to further systems work and architecture optimizations. The reviewers were consistent in their support of the paper. I follow their recommendation: Accept.

Decision: Accept (Poster)