Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper introduces a nonparametric score test to estimate the importance of network connections on the final output of Deep Neural Networks. They derive this by modeling the log-transformed joint density of each connection and final output in a tensor product reproducing kernel Hilbert space. They finally derive an asymptotic distribution of the proposed test statistics which only depends on the eigenvalues of the kernel. This importance test is applied to ranking the importance of each connection in Multilayer Perceptron Networks, and Convolutional Networks and sparsifying the networks by removing the least important connections. The method is applied post-training to fully trained networks but evaluated by retraining the sparsified network from scratch. The method is demonstrated in experiments on compressing three networks. 

I believe this paper is a borderline accept. It provides a more statistically principled method to examine the importance of connections in a Neural Network and rank them compared to existing compression methods. Due to this ranking, there is a clear method to easily adjust the target compression rate. They are able to achieve high lossless compression rates. However, the benefits shown in the empirical results could be more convincing. It lacks baselines on more complex networks and could benefit from more empirical analysis of the theoretical benefits and properties of this approach.

Pros: 
The method is able to maintain accuracy while achieving high compression rates on MNIST and CIFAR10. It does better than the baselines compared to the small networks. They show a capability for increasing generalizability by decreasing error rates. 

It provides a well derived statistically principled method to examine the importance of connections and rank them.

Indicates an additional use to visualize the importance of features.

Cons:
The empirical results could be clearer. It lacks baselines for larger models on Cifar10. Could you compare it with the published results of other algorithms? How does it do on larger networks like Imagenet?
Computational efficiency is mentioned but could be examined in greater detail.  How long does it take to run and how is that affected by model size?

The experimental setting is somewhat unclear. The baseline Louizos et al. (2017)  was designed to optimize group sparsity/speed, but the experimental results here only examine the compression rate. Was the baseline run to optimize speed or sparsity?

It would be interesting to examine the correlation of importance score with actual impact on network performance. This might be done with a comparison with random pruning or pruning higher importance connections. It might be useful to examine the performance of the networks after pruning nodes of differing importance without full retraining or just fine-tuning. It is unclear how important full retraining is in this method.

It would be interesting to visualize the importance of features at different depths in the deep convolutional networks.

Minor suggestions: In the introduction, you mention l0 and l1 norm methods, but cite Han et al. (2015) which compares l1 and l2 norm and found l2 norm to be better overall.

typos:
In Abstract: nonparemtric scoring test  -> nonparametric scoring test  
In 4.4: sample averarge -> sample average
In 4.5: ASYMPTOTICALLY DISTRIBUTION  ->   ASYMPTOTIC DISTRIBUTION 
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: The paper proposes a new way of evaluating the importance of neural connections which can be used for better model compression. The approach uses a non-parametric statistical test to detect the three way interaction among the two nodes and final output. Some small scale experiments show that the approach achieves better compression rate given the same test error.

The approach seems interesting in the sense that, unlike existing techniques, it explicitly measures the three way interaction among the two nodes and the output. Also, it removes the average effects and only considers (non-linear) correlation after removing the mean. The explicit link to the final output and removing average effects allows the method to remove more weights without decreasing the loss by much.

However, the drawback of the approach is the significantly increased computation due to 1) quadratic complexity for the kernel methods; 2) unable to cache computation among different pairs of nodes (for example, gradient-based approach can compute importance for all node connections in one forward-backward pass). This limits the applicability of the approach to more interesting cases of larger models (for example, models that work on ImageNet) where model compression is of more urgent need. As a result, the significance and impact of the approach is also limited.

The experiment results are interesting in that it shows the proposed approach can achieve better compression rates given the same test error tolerance on a few small datasets. However, as mentioned above, these experiments are less convincing than more complicated models on larger datasets, such as ImageNet where model compression has greater impact. In addition, the test errors on smaller datasets are easier to achieve, sometimes tuning the optimization settings such as learning rates can result in significant improvement. Therefore, such results are more like preliminary.

The paper is general clear and well written. The experiment section should include more important information such as what kernel bandwidth is used for Gaussian kernel, which greatly affects performance. Also, the main text introducing the proposed statistical test is a bit verbose, and dense in unnecessary notations. I think it can be made more succinct by presenting a high level idea (removing mean, three way correlations) first and then the final results. Some intermediate results can be put into Appendix.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: In this paper, the authors propose a new pruning technique that utilizes the statistical dependency between the corresponding nodes and outputs. The dependency is measured by a kernel based dependency measure which is closely related to MMD. The test statistics derived from the dependency measure have an asymptotic distribution which can be written as a weighted sum of chi-square random variables. The proposed method is numerically investigated using some datasets such as MNIST and CIFAR 10.

Pruning is one of important problems for practical deep learning operations. This paper gives an interesting idea for the pruning techniques. I think the idea is novel.

On the other hand, I also have the following concerns:
- Although applying the kernel type information measure is an interesting idea, its computational complexity would be large. It is not obvious that it works for large datasets such as ImageNet even if the sub-sampling technique is applied.
- The numerical experiments are conducted in small datasets. How it works in larger datasets such as ImageNet and (more importantly) how it is compared with SOTA pruning methods in more difficult datasets. Actually, performance comparison is not done for the CIRAR10-VGG16 setting.
- The asymptotic distribution can be seen as a corollary of existing researches for MMD and HSIC.

For these reasons, I was not completely convinced with the effectiveness of the proposed method.


Minor comment:
- The test statistics is more like HSIC. It would be nice if there were comments on the connection to HSIC.

===
Update: The computational cost for this method seems not so much demanding, and could be applied to large data-set. The  resultant performance also seems useful. I think more convincing comparisons are needed. However, its idea seems interesting and its practicality is ensured to some extent. Thus, I have raised  my score.
review_title: Official Blind Review #3

Meta-review: This paper proposes a novel approach for pruning deep neural networks using non-parametric statistical tests to detect 3-way interactions among two nodes and the output. While the reviewers agree that this is a neat idea, the paper has been limited in terms of experimental validation. The authors provided further experimental results during the discussion period and the reviewers agree that the paper is now acceptable for publication at ICLR-2020. 
Decision: Accept (Poster)