Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper authors propose a mechanism to prune a convolutional neural network model in a relatively data-free manner i.e., they do not utilize training data or loss function for retraining the pruned model. However unlike the actual data-free pruning techniques they assume the availability of moments of class-conditional distributions of the activations. They make a critical assumption that these distributions are Gaussian to exploit cheaply computable sufficient statistics in deriving tractable bounds which are used to guide the pruning process.

They propose a pruning method which exploits the proposed metric to both decide the extent of pruning for a given layer and the actual filters to prune in a given layer, without measuring the impact on the down-stream layer outputs either in terms of the deviations in the metric or in terms of actual performance. This is a reasonable operation under the fundamental hypothesis informing this paper, i.e., discriminability of a filter strongly correlates with performance impact on pruning it. However they do design an iterative version of their pruning method which explicitly measures the cumulative impact of pruning at all the layers on the over all model.

They show appreciable reductions in performance degradations for a given pruning budget compared to other data-free pruning methods. 
strength_and_weaknesses: Strengths
------------
The paper clearly calls out the hypothesis and constantly justifies the algorithmic decisions in the context of this hypothesis. 
It performs intermediate validation exercises for their hypothesis, which motivate the reader and guide them through the author's intuitions.
The supplementary material is very exhaustive and helpful in further clarifying the details of the proposed algorithm/metrics.

Weaknesses
---------------
The motivation for data free pruning is not clearly described in the paper. The readers are forced to rely on the references.
It would be very informative to the reader to compare the sparsification potential of this technique to a pruning method which exploits fine-tuning post compression.

clarity,_quality,_novelty_and_reproducibility: The paper is very clear. It explicitly calls out the hypothesis under which the proposed method is designed and provides reasonable intermediary validation steps.

The novelty of the paper lies in designing a cheap pruning method which is cheap due to well validated/referenced assumptions such as Gaussian nature of intermediate activations or strong correlation between discriminability of a layer's output and its prunability. 

Authors made appreciable effort to improve reproducibility of the results in the paper.
They provide a link to their implementation.
The authors describe the algorithm in great detail.
They utilize standard architectures to demonstrate their pruning method.
They utilize open datasets for experimental validation.
summary_of_the_review: This paper guides the reader through the design of the algorithm and motivates the design decisions with well validated/referenced hypotheses. It addresses a problem of critical interest to this community. The writing style is clear. The references are more than adequate. The proposed technique is has limited but sufficient experimental validation. Though there are minor possible improvements the paper in its current form is already useful to the readers.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a training-data free structure pruning method to prune deep neural networks. Specifically, the authors propose to measure the discriminating ability of filters through the total variation (TV) distance between the class-conditional distributions of the feature maps outputs by the filters. Then, based on the above TV-distance, the authors define the LDIFF score to decide the pruning ratio of each layer. Last, the authors proposed IterTVSPrune, which iteratively prunes the model to achieve greater sparsity. Experimental results on CIFAR10/100 and ImageNet demonstrate the effectiveness of the proposed method. However, there are still some issues in the paper. Detailed comments are as follows.
strength_and_weaknesses: ## Strength:

1. The authors propose a new paradigm called distributional pruning for pruning neural networks. It measures the discriminating ability of filters through the total variation (TV) distance between the class-conditional distributions of the feature maps outputs by the filters. 

2. Experiments on the image classification task demonstrate the effectiveness of the proposed method. 

3. The manuscript is easy to read and provides enough experimental details to reproduce.

## Weakness:

1. The proposed method may be hard to be applied in real data-free scenarios because it needs to assess thousands of images from the test set (3/4 of the training set) to calculate the distributions of feature maps. Can the proposed method use less number of images (or images from datasets with similar distribution) to calculate the distributions of feature maps? More explanations and experiments are required.

2. In Algorithm 1, the authors set the weights of the pruned filters as zero rather than remove them directly. In this way, the pruned model may still cost the same storage space to store these zero weights and consume the same computation cost to forward with the zero weights. How does the pruned model reduce the storage space and the inference time? More explanations and experiments are required.

3. In Section 5, the motivation of proposing LDIFF metric to identify which layers can be pruned is unclear. The authors claim that LDIFF metric tends to prune layers with a mixture of discriminative and non-discriminative filters and avoid pruning layers with a majority of discriminative filters or non-discriminative filters. Why should the LDIFF metric avoid pruning layers with a majority of non-discriminative filters? It would be better to visualize and analyze the fraction of discriminative filters $\tao(\eta)$ in each layer.

4. In Definition 1, the meaning of the function "sup|·|" in the equation is confusing. It would be better for the authors to explain the meaning of "sup|·|".

5. In Section 7, it would be better for the authors to provide the theoretical complexity measure and the time consumption of the proposed method.

6. The authors only show the results on heavy-weight models such as ResNet. It would be better for the authors to conduct more experiments on light-weight models such as MobileNet-V2 [1].

7. More ablations of the $\eta$ are required because the influence of $\eta$ is unclear. How does this hyper-parameter affect the performance of the proposed pruning method? 

8. The authors only present the experimental results on image classification task. It would be better for the authors to show the experimental results on more computer vision tasks such as object detection and semantic segmentation.

## Minor Issues：
1. Many references in this paper have been officially published, such as “Baykal et al. (2018)” and “Frankle & Carbin (2018)”. Please reference these papers in the formal format.

2. There are some typos in this paper:
(1) In Section 7.1, "in order to reduce the test error" should be "in order to increase the test error"?
(2) In Section 6.1, "Therefore, we use the LDIFF scores to decide We now present the TVSPRUNE algorithm." is an incomplete sentence.

## Reference:
[1] MobileNetV2: Inverted Residuals and Linear Bottlenecks. CVPR 2018.

clarity,_quality,_novelty_and_reproducibility: The manuscript is easy to read and is of high quality but requires more experimental results and explanations. The proposed new pruning paradigm called distributional pruning is novel. The idea of data-free pruning is not very novel. The manuscript provides enough experimental details to reproduce.
summary_of_the_review: The authors propose a training-data free structure pruning method to prune deep neural networks. The proposed new paradigm called distributional pruning for pruning neural networks is novel. The idea of data-free pruning is not very novel. The manuscript is easy to read and provides enough experimental details to reproduce. Experiments on the image classification task demonstrate the effectiveness of the proposed method. However, the manuscript still needs more experimental results and explanations.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The proposed work uses the hypothesis: Non-discriminative filters do not contribute a lot to the predictive performance of a network and can be removed safely. Keeping this in mind, along with a privacy based setting where the original training data and loss function are unavailable yet the a sample from the original distribution is available, the proposed work uses TotalVariation distance between class conditioned distributions as a measure to identify discriminative filters. In addition, the proposed work offers LDIFF score as a way to ascertain the extent to which a layer possesses a mixture of discriminative and non-discriminative filters. Overall, these two ideas are combined to provide a pruning algorithm.
strength_and_weaknesses: Strengths
- The technical contribution and its relevant information have been explained well.
- The overall structure of the manuscript is solid and helps the reader digest the information in a steady manner.

Weaknesses
- Could the authors comment on how to connect the proposed relaxation back to data-free structured pruning? (Pg. 2, Paragraph 1, Line 2-5)
- Could the authors contextualize the theoretical setting assumed in the proposed work, of class conditioned distribution from various layers of the network, given that storing such information consumes extra memory? In addition, when iteratively updating the pruning algorithm, how much do the distributions vary, post pruning of the previous layer? Are their effects exaggerated on the distribution of successive layers?
- Could the authors provide more insight in choosing Random Pruning as a baseline to understand the impact of TV distance? Since due to its weak assumptions it provides loose bounds and a measure that provides tighter bounds could inform the pruning better.
- Could the authors clarify and define the meaning of the notations used in Equation P? 
- Could the authors describe and discuss in detail the statement "Note that we also observe that some layers have features that cannot discriminate well, and yet cannot be pruned."? 
- Could the authors explain how values of $\eta$ were obtained?
- From an experimental perspective, could the authors clarify whether selecting the filter with the highest  TV distance matched with the largest drop in performance? (i.e., the values of TV distance corresponded to the expected ranking of discriminative filters)
- Could the authors clarify if a batch size of 30000 was used for experiments in Section 7.2?
- Given the extreme drop in performance, relative to a small amount of parameters being removed (Table 1), could the authors discuss if TV distance and LDIFF could be further highlighted when framed similar to more standard pruning setups?

Post Rebuttal
- Based on the responses provided by the authors, I have updated the final recommendation.
clarity,_quality,_novelty_and_reproducibility: Clarity
The context, problem domain and intent of the paper is well put forward and easy to digest.

Quality and Originality
While the finer points of the technical contribution are novel, the broader context and its viability are concerning, given the large amount of existing work in pruning and the current state of the domain.


summary_of_the_review: Certain choices in parameters and rationalization behind the problem domain are the main concerns when it comes to the proposed work. Addressing the weaknesses highlighted above should solves these concerns.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors proposed to solve the problem of formalizing and quantifying the discriminating ability of filters through the total variation (TV) distance between the class-conditional distributions of the filter outputs.
strength_and_weaknesses: +The setting of pruning without finetuning is kind of attractive.

-The calculation of MinTVS requires samples and their labels. It's hard to understand why authors emphasize they do not need the loss function. Compared to computing loss and using related channel or filter importance scores, MinTVS may not have advantages in terms of efficiency.

-The claim of 'not using training set' is also confusing. Authors resplit the test set to calculate MinTVS. If the whole test dataset is used for evaluation, then MinTVS must use samples from the training set. 

-Although the authors claim that they do not finetune the model, the performance of their method does not have significant advantages compared to regular pruning methods like CHIP.

-The experimental settings are also confusing. It's good to show accuracy without any finetuning process. However, some results are quite meaningless. For example, losing more than 30% accuracy for ResNet-50 on ImageNet when only reducing around 25% parameters. Authors should provide meaningful results by including the finetuning process.

-The proposed algorithm includes tuning several hyperparameters, and how to decide the final pruning rate is not straightforward.
clarity,_quality,_novelty_and_reproducibility: The writing is average, with some typos. The novelty is not significant. The reproducibility seems ok, but the experimental setting is questionable.
summary_of_the_review: The setting of pruning without finetuning is interesting. But many arguments and experimental settings of this paper are not valid or confusing.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The manuscript proposes a distributional pruning method to structurally prune deep neural networks. It uses the following hypothesis: Non-discriminative filters do not contribute a lot to the predictive performance of a network and can be removed safely. Specifically, it measures the discriminating ability of filters through the total variation (TV) distance between the class-conditional distributions of the feature maps outputs by the filters. This can be done when a sample from the original distribution is available while the original training data and loss are unavailable. Then, based on the above TV-distance, it defines the LDIFF score to decide the pruning ratio of each layer. These two ideas can be applied iteratively to prune the model for greater sparsity. Experimental results on CIFAR10/100 and ImageNet demonstrate the effectiveness of the proposed method. Such a distributional pruning idea is novel. There are various questions raised by reviewers about the writing and implementation details, where the authors provide clarification and improvement. Three reviewers agree to accept the paper while one reviewer keeps the negative attitude. Based on the majority reviews, the manuscript can be accepted had the authors included all the promised revision in the final version.  
justification_for_why_not_higher_score: Two reviewers agree to accept the paper, one reviewer weakly accept and one reviewer keeps rejection for some concerns on the claims. The manuscript is thus accepted if the authors made all the promised revision in the final version. 
justification_for_why_not_lower_score: Three reviewers agree to accept the paper while one reviewer keeps the negative attitude. All the reviewers agree that the idea in this paper is interesting. Based on the majority reviews, acceptance is suggested.
Decision: Accept: poster