Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a new method called  "The Cross(word) Puzzle" that helps in getting better pruning metrics to further improve pruned neural networks in accuracy for the same compute footprint. This is informed by what the authors call Fabulous Coordinates (the choice of pruning criteria) and Fabulous distribution (the target distribution of weights after pruning). 

The paper starts of by motivating what the useful properties of a good pruning function should look like and leverages it to build a search pipeline for semi-automating it. They evaluate the method with a couple of experiments on ImageNet and CIFAR-10


The brevity of the review is due to the fact that the paper is extremely incoherent and it took me over 5 hrs to digest the information. The paper is extremely poorly written, with no solid definitions, inconsistent naming (SFN vs FSN vs SNF -- what is this?), poor naming (Fabulous?), lack of details of the methods, no signs of reproducibility, and inconsistent experimentation and design motivations. 



strength_and_weaknesses: Strength:
1) The problem paper tries to solve is real and has real-world utility.

Weakness:
1) The abstract is extremely uninformative, so is the introduction.
2) There are terms used without any explanation or pointers to -- what is a coordinate? is something it too me a long to process
3) What is fabulous about the distribution?
4) The motivation for the design principles come from unstructured pruning and the paper experiments on structured pruning, I am not sure how that works. I know the connections between both styles exist, but connecting Han et al 2015 to the current paper does not gel well. 
5) I have no idea how the candidates for coordinates are even generated for search, the entire pipeline is so under-explained at required places and over-exposited at places with little impact on the entire method. 
6) Table 1 is extremely hard to comprehend. 
7) definitions of equations follow the same suit.
8) Figure 5 shows your method is more inaccurate than Han et al., I do not understand. 
9) The experiments are performed on limited evaluations and at 50% structured sparsity which is not even considered to be sparse because of the known redundancies. The usual comparisons happen at over 70% for ImageNet. 
10) FLOPS is not directly proportional to sparsity due to non-uniformity across layers, so that metric needs to be fixed. 
11) While the figure are slightly accessible, they are difficult to read due to poor placement and design. 
clarity,_quality,_novelty_and_reproducibility: Poor presentation and quality. While novelty exists in the motivation, it is executed in an unscientific fashion. It seems like a paper thrown all together at the last moment and I do not recommend publishing of the paper even after significant registrations.
summary_of_the_review: Poor presentation and quality. While novelty exists in the motivation, it is executed in an unscientific fashion. It seems like a paper thrown all together at the last moment and I do not recommend publishing of the paper even after significant registrations.

--------------------
Post Rebuttal: Given the need for a significant revision, I recommend rejecting the paper from ICLR 2023 but encourage the authors to make the material more accessible to readers in the next iteration.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: The paper proposes the Cross Puzzle to find something called the Fabulous Coordinate for pruning neural networks. 
strength_and_weaknesses: The paper is very difficult to read. Though experiments are performed, the experiments are not up to the quality expected. For example, there is no direct comparison with The Lottery Ticket Hypothesis paper (https://arxiv.org/pdf/1803.03635.pdf), which can reduce network size by over 90%. 

Terms are not clearly defined. I cannot find clear definitions of the Cross Puzzle, the Fabulous Coordinate or the Fabulous Distribution. 


clarity,_quality,_novelty_and_reproducibility: The paper is very difficult to read. 

A partial list of edits: 
edits: 
p. 2:
"cutting off by pruning" --> "cut off by pruning"
"nearly none redundancy " --> "nearly no redundancy" 
"intrigues lots of follow-up works" --> "inspired many follow-up works"
"amount of efforts" --> "amount of effort"
"We refer impact-based pruning to methods for ... ." --> "We refer to methods for ... as impact-based pruning."
"coarse-estimated second-order derivative" --> "a coarsely-estimated second-order derivative"?
p. 3:
" leverages empirical fisher matrix" --> " leverages the empirical Fisher matrix"
"constrains" --> "constraints"
", and etc" --> delete this.
"aware of/" --> delete this.
"should obeys Gaussian Distribution" --> unclear. Maybe: "should approximate a Gaussian Distribution"?
"fast narrow the neural network" --> unclear. Maybe "first narrow the neural network"?
"in Bayesian DNN " --> "in a Bayesian DNN "
"two-folded" --> "two-fold"
"as handstuned issues" --> unclear. Maybe "for hand-tuning".
"introduces new observation point" --> "introduces a new observation point"
"distribution itself is not the protagonist" --> unclear
"​​to parameter distribution" --> "to parameter distributions"?
p. 4: 
"weights in the form Laplace " --> unclear. Is the suggestion that the weights are distributed approximately as a Laplace distribution? 
"neuron as" --> "neurons as"?
 "represent each others" --> "represent each others"
"coordinate psi" --> Should this be "parameter psi"?
p. 5: 
"don’t fit our requirement" --> it is unclear to me what the requirement is. 
Figure 1 and 2: vertical axis would be better labeled "Count" than "Counting"

"Crossword Puzzle search engine" --> unclear to me what this is

summary_of_the_review: Overall, this paper id not ready for publication.
The key idea is not clearly communicated, the terms are not well defined, and there are not experiments that support the key idea.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: The authors proposed a so-called Crossword Puzzle method to find the optimal criteria for network pruning. The key idea is to find a so-called Fabulous Coordinate which satisfies three key properties for pruning. The authors validated their method on ImageNet and show that they can compress ResNet family by 50% without accuracy degradation.
strength_and_weaknesses: [Weakness]

The paper is poorly written. It looks like the paper is generated by some Google translation engine with lots of meaningless words. I tried my best but honestly speaking, I cannot understnad what the authors are writing about. In Section 3, the authors throwed up lots of formulars and figures without sufficient explanation. I have no clew what these numbers or equations mean.
clarity,_quality,_novelty_and_reproducibility: * Poor clarity, unable to read
* Not sure about novelty because it is difficult to get useful information from the paper.
summary_of_the_review: Strong reject.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Existing pruning approaches rely on intuition- or experience-based measures to determine the constraint based on which to prune neural networks. Instead, the current work proposes a framework, Crossword Puzzle, to guide the search of a pruning criterion given basic building blocks and evaluate the quality of the pruning criterion when applied on networks. Specifically, the proposed framework is based on the Fabulous coordinates, which indicate that when the distribution from a provided model match the constraints set for these coordinates, they are highly qualified to provide good pruning outcomes.
strength_and_weaknesses: Strengths
- The proposed work offers an interesting idea on measuring the quality of a proposed pruning metric.
- Combined with an almost meta-learning-like framework on identifying a high quality pruning metric, automation of this domain is certainly important and brings pruning a step closer to being part of the development cycle of DNNs.

Weaknesses
- Could the authors provide more quantitative bounds when mentioning "relatively-high pruning rate" in Pg. 1, footnote.
- When highlighting the improvement in acceleration, could the authors clarify whether the "3x" improvement is in theoretical FLOPs or real-time inference latency?
- I would encourage the authors to look closely into the literature discussing distribution-based pruning. Apart from the papers mentioned in the current manuscript, there are broad swathes of work on probabilistic pruning that can and should be referenced.
- Could the authors provide an intuition for the formulation of the LKL-CD, including how they solve for the final values?
- Across multiple instances there are mix ups between the terminology of SFN and FSN (E.g., Fig. 2, Fig. 3 and others). Please correct them.
- The LKL-CD of Raw Weights in Table 1 provide a much better approximation than the chosen SFN formulation, assuming lower values are better. Could the authors justify the selection of SFN?
- Could the authors clarify the meaning of the notation $C_W$?
- I encourage the authors to increase the font of legends and Axes ticks for readability.
- Quantifiably, could the authors provide metrics used to measure the collapse as well as highlight how exactly they were applied to measure the values provided in Fig. 5?
- In addition, could the authors provide more comparisons on the recovery process of the central hole across the various conv layers?
- From Tables 2 and 3,
     - Across VGG16 and Table 3, SFN provides better accuracy at a lower % of FLOPs removed when compared to previous approaches.
     - Could the authors clarify their choice in prior art and comparisons? Since there are a number of previous works with higher performances that haven't been included.
     - In addition, could the authors include the decrease in number of parameters as well, since it provides a slightly different perspective on the degree to which a network can be pruned.

clarity,_quality,_novelty_and_reproducibility: Clarity
The explanation of the method and important observations are good. However, certain portions of the measures and evaluation scheme is slightly obscure.
 
Quality and Novelty
The measure and overall search framework offer a requisite level of novelty and quality to the proposed work. However, a deeper evaluation is necessary to highlight these ideas.
summary_of_the_review: While the core ideas are relatively straightforward, the evaluation and explanation of certain aspects of the work are obscure. Addressing the weaknesses mentioned above will bring more clarity to the results and content of the proposed work.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a Crossword Puzzle engine for searching the optimal pruning metrics automatically. It first gives three properties of a good metric/coordinate: centralized distribution,  retraining recovers centralized distribution and central collapse. Then, it introduces a metrics called LKL-CD  for measuring the centralization degree. The paper finally presents their coordinate obtained from the engine and empirically evaluates the effectiveness of the coordinate system for pruning.
strength_and_weaknesses: Strengths

•	The paper gives a comprehensive introduction to previous work on pruning metrics categorized in 3 different groups, magnitude, impact, and distribution-based methods. These build the metrics blocks for search.
•	The paper introduces the Fabulous Coordinate with three properties, a new perspective of understanding the model redundancies.

Weaknesses

•	There is little explanation on why the proposed LKL-CD is better than other metrics. The paper attempts to explain it through Table 1, the four layers in ResNet-50. It might not be statistically enough to claim the advantage of it using one type of network. More validation should be done either empirically or theoretically. It would be better to see that comparisons between them are included in the experiment section.

clarity,_quality,_novelty_and_reproducibility: The paper needs clarifications in a number of places. 

•	How should one understand the second requirements of the Fabulous Coordinates, that retraining recovers centralized distribution? This seems to contradict to the third requirement (central collapse). In the Figure 7 pipeline, a coordinate goes through retraining recovery verification first, then to central collapse inspector. These two procedures seem to do similar things, i.e., pruning, retraining, and then test distribution. How can a coordinate gives recovered center in the first procedure, while gives collapsed center in the second procedure?

•	 In Section 3.1, the paper proposes the LKL-CD measurements. Table 1 highlights the value of it. Why does a lower value indicate a better measurement?

•	In Figure 4, how can one conclude that the degradation of centralization is more apparent for the proposed pruning method? From the figure, the LKL-CD value of the method is higher than the base method.

•	There is no information on how the pruning metric blocks are build. For example, in Section 4, the search engine can jump out of the given intuitive rule when trigged by the search guider, while it is unclear how the search guider behaves. The only description for it is the acceptance probability.

•	Following the previous question, the description for other parts of the pipeline is also ambiguous. It is strongly suggested that the paper should include more details of them. The implementation code is not included, which makes the reviewer harder to understand each part.

summary_of_the_review: The paper proposes novel perspective of pruning. Intensive experiment is done to search better coordinate systems to guide the pruning procedure. However, some key points need further and clearer explanation.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper was so difficult for the reviewers to understand that they could not make an adequate technical evaluation of the paper. I recommend rejection and encourage the authors to focus on writing so that the paper can get a full technical evaluation in a future submission.
justification_for_why_not_higher_score: This paper did not receive a higher score because the reviewers were unable to evaluate it on its technical merits due to poor writing.
Decision: Reject