Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposed a model compression method: Falcon and rank-k Falcon. Both are used to compress CNN type of models by replacing standard convolution layer with a compact Falcon or rank-k Falcon layer to compress the model. Falcon's main idea is to decompose the traditional convolution kernel K into two smaller tensors, one is depthwise convolution kernel D and pointwise convolution kernel P. And DP will reconstruct the original kernel K. Since D+P's memory is  D*D*M+N*M which is smaller than the original size D*D*M*N, and thus when N is large, the memory saving could be large. The paper is in general in good writing and very easy to read.

Below I have several concerns/suggestions for this paper:

1: Novelty. What is the main difference between this method and all the other tensor decomposition based methods for CNN compression? There are so many tensor decomposition based methods for CNN, and seems Falcon belongs to one of them. The one (maybe) special for Falcon is that it only decomposes along one dimension. Why this method could perform better than other tensor based decomposition methods(some of them are having even smaller memory footprint as they decompose more dimensions) or Falcon could be one special case of it? 

2: In Section 3.3 and 3.4, the proposed Falcon and rank-k Falcon seems is fully recovering the original K, see equations above Theorem 2 and above Section 3.5, should it be minimizing the reconstruction error as other tensor decomposition methods? And how to find the solutions P and D from K? how the model is getting trained if using Falcon or rank-k Falcon? Do you have retrain step after the decomposition of K?

3: In the experiment, no any other standard compression techniques such as quantization, low-rank, weight-sharing, sparse, etc are compared.  This makes us curious about the benefit of the proposed methods over other methods.

In summary, I am mostly worried about the novelty of the paper, and wondering how the model is getting trained, and the comparison with other compression techniques.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: Overview:
The paper is dedicated to studying fast and lightweight convolution for efficient compression and retaining original accuracy. In this paper, the authors interpret existing convolution methods based on depthwise separable convolution and derive FALCON. They claim their FALCON mathematically approximate the standard convolution kernel and achieves a better TA/efficiency tradeoff. They conduct extensive experiments to show that FALCON based method 1) outperforms previous state-of-the-art methods; 2) achieve 8X efficiency while ensuring similar TA.

Strength Bullets:
1. The authors give a detailed interpretation of the depthwise separable convolution method via EHP. Then they propose novel FALCON has better accuracy than competitors while having similar compression and computation reduction rates.
2. The paper is well organized and easy to read. The author conducts extensive experiments to show FALCON based method not only surpass other depthwise separable convolution models but also give up to 8 times efficiency while giving similar accuracy.

Weakness Bullets:
1. The authors need to provide more ablation experiments for two components of FALCON: 1) align depthwise and pointwise convolution; 2) initialize kernels. Moreover, for the order of performing depthwise convolution and pointwise convolution, it also needs experimental support.
2. I would like to see a comparison of the compression rate with previous filter compression methods. i.e, Soft weight-sharing (ICLR'17), Deep K-means (ICML'18) and so on.
3. [Minor] Even if there is computation reduction (Flops), it is not interesting enough. I am very curious about how much efficiency gains (i.e. latency) in the software implementation of FALCON. 

Recommendation:
Although there are a few flaws in experiment design, it still is a novel and good technique. This is a weak accept.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The paper proposes a CNN compression method, based on the so called EHP operation, which can be used to  analyze and generalize depthwise separable convolution. Based on EHP, the paper develops depthwise separable convolution to compress CNNs, and extend it to a rank-k approach with further improved accuracy. Some analysis is provided about the operation equivalence. The experiments on standard benchmark datasets show the effectiveness of the method. 

Probably the most important contribution of this work is that it proposes a new operation that can summarize/generalize the existing depthwise separable convolution and reveal the relationship with the standard convolution. The work is meaningful to me because a compact representation of CNNs can largely reduce the computational resources and storage.  
The experiments are extensive as well. 
review_title: Official Blind Review #2

Meta-review: The submission presents an approach to accelerating convolutional networks. The framework is related to depthwise separable convolutions. The reviews are split. R3 expresses concerns about the experimental evaluation and results. The AC agrees with these concerns. The AC also notes that the submission is 10 pages long. Taking all factors into account, the AC recommends against accepting the paper.
Decision: Reject