Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: ### Summary
This paper proposes a three-step compression of neural networks without fine-tuning, which comprises weight hashing, merging of similar weights, and depthwise separation via basis extraction. The compression is filter-based, data-free, and with minimal accuracy degradation.

main_review: ## Strengths
The combined approach appears to be effective and competitive against related works which includes some recent fine-grained and data-driven methods.

The reviewer believes that the method proposed is interesting with the use of density estimation for redundancy discovering, despite some presentations issues (as discussed below).

The experiments and ablation analyses appear to be sufficient to justify the claims in the paper.

## Weaknesses
* It is not immediately clear why hashing weights helps with identifying redundancy. Why not simply merge neurons with small $\ell^2$ distances? Perhaps providing a formal definition of how $\tau$ is used would clear up the confusion.
* It is also not clear how the KDE bandwidth $\Delta_l$ is determined. The hashing process is highly dependent on this.
* The proposed hashing method may have a scalability problem (“ranging from 45 to 413 seconds for ResNets on CIFAR-10 and up to 21 hours for a ResNet-50 on ImageNet”, line 202). It appears to not extrapolate well to larger models. What about ResNet-$N$ ($N \geq 101$)? Could you explain why this is the case or provide a discussion on its complexity?
* Minor issues:
	* Line 112 discreet -> discrete
	* Line 123 defines the contrast hyperparameter $\tau$, but provides no definition of how it is used.
	* Line 109: grammar: “for a each layer”.
* The combined method appear to be overall too complex, the reviewer would appreciate if the work will be open-source in the future.

limitations_and_societal_impact: Sufficiently discussed in the paper.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduce a data-free compression methods by combing three strategies: adaptive scalar hashing, merging similar neurons and uneven depth-wise separation method to prune conv layers, obtaining SOTA data-free compression performance.
main_review: 1. Novelty is pretty good. The observations of redundancies in DNN are pretty novel. 
2. The comparisons conducted on Imagenet are not very strong. There are many published data-driven compression papers. The listed paper's performances are not real SOTA. However, this method is data-free, I will buy it.  It's better to show more comparisons on other benchmark models (for example, MobilenetV2) on Imagenet. 
3. As mentioned in paper, this method can incorporate with other compression methods together for further compressions. 

I am curious about:  What if we apply three strategies in loop iteratively? Can it improve the performance?
limitations_and_societal_impact: Please provide Societal Impact discussion.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The author propose a data free pruning pipeline to compress a given deep neural network. The author propose to first apply adaptive weight hashing to force the weights tensors to be similar and then merge weight tensors based on a calculated similarity score. The proposed methods use kernel density function for hashing and do not require training data for hashing the weights tensors. The proposed pipeline (RED) is verified on CIFAR-10 and ImageNet.
main_review: Strength
1. The proposed pipeline (Hashing + Merging) is interesting and the author has demonstrated the effectiveness of hashing on introducing redundancies on the weights tensor.

2. Data-free compression is also worth to explore for its efficiency on computation resources.

Weakness
1. The writing of the paper need further refinement. For an example, table 1 in the paper seems to be incomplete as there is a missing of the column headers. I cannot figure out the meaning of those results based on the current version.

2. The impact of hashing on the model accuracy is not clear. While the hashing introduce tensor level redundancies, will it hurts the model representation capability? 

3. How is the proposed methods affected to different initialization methods? This is mentioned in the caption of Figure 5 but the impacts are not clear as there is not experiment results on this point. I thought this is worth to explore.

4. The proposed method is mainly verified on large models like ResNet. It is not clear if the proposed method will work on more efficient backbones such as MobileNet [1], EfficientNet [2] and MobileNeXt [3].

Overall, the proposed method is interesting to explore. However, the current presentation and ablation experiments do not convince me on the effectiveness of the proposed RED pipeline. Besides, the difference between the proposed hashing methods and the quantization algorithm (as there are a lot of well performing quantization algorithms) is also worth to explore.

I am willing to incease my score if the author could address mu concerns sufficiently.

[1] Sandler, Mark, et al. "Mobilenetv2: Inverted residuals and linear bottlenecks." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.

[2] Tan, Mingxing, and Quoc Le. "Efficientnet: Rethinking model scaling for convolutional neural networks." International Conference on Machine Learning. PMLR, 2019.

[3] Zhou, Daquan, et al. "Rethinking bottleneck structure for efficient mobile network design." Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16. Springer International Publishing, 2020.

limitations_and_societal_impact: The author do not provide social impact of the paper
review_title: null

Meta-review: I agree with the two of the reviewers that this contribution is novel and interesting. While the empirical evidence presented here could (and was) improved (in the rebuttals), I think that a sufficiently revised version of the paper (as suggested in the rebuttals) can be accepted for a poster at NeuriPS.
Decision: Accept (Poster)