Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper conducts a series of empirical studies, which aim at relating node/filter/channel pruning with the partition of input space by neural nets using spline operators as activation functions. In particular, the final decision boundary is only determined by a few splines defined by a few filters of the network. This is verified in the paper through case studies of both fully-connected networks and convolutional networks, which implies that pruning does not severely degrade the accuracy as long as the important splines are not removed by pruning. Another observation is the early-bird tickets: the paper argues that the important splines do not change too much after a few early epochs since the binary activation patterns of data converge rapidly, so prunning can be applied after only a few epochs. The paper then proposes a pruning strategy that sequentially finds the most similar filter pairs and removes them. In experiments, they show that the proposed prunning method is more efficient than some recent pruning methods to achieve similar accuracy with pruning ratio <=70% for CIFAR10/100 and <=50% for ImageNet. 
main_review: Please read the summary first. Here are detailed comments:

(1) Section 3.1 shows a trend in case studies that the pruning tends to preserve subdivision splines useful to the decision boundary and remove the redundant ones. As long as the decision boundary is preserved, the performance will not degrade. This is interesting. However, the two toy datasets here are too synthetic and no quantitative analysis is presented, especially for more complicated datasets or neural nets. Moreover, given the previous success of pruning on DNNs with piecewise linear activation functions like ReLU, the phenomenon observed here is somewhat obvious. I expected to find some in-depth analysis of why the trained decision boundary only depends on a small subset of splines and why the applied pruning method here can preserve them, which could be an important novelty of this paper, but only the phenomenon is described. The splines also lack a formal math definition in terms of the network parameters/filters/units, which could be very helpful to understand their relationship in pruning.  

(2) Section 3.2 claims that a novel metric based on those subdivision lines is developed to study the early-bird ticket phenomenon but the metric is simply the change of ReLU activation patterns between two consecutive epochs. It is obvious that if the activation patterns do not change too much, the decision boundary of the DNNs would not change too much, but this does not necessarily depend on the previous analysis of the spline partition of input space. The change of partition is only reflected via a toy example in Fig.4 without quantitative analysis and in-depth discussion, e.g., do all the splines or just the important ones (forming the boundary) not change since very early epochs, what happens to the partitioned regions that do not contain any training data, etc.

(3) I also have a similar concern about Section 3.3. It proposes an intuitive pruning strategy, i.e., removing the redundant units that are too similar to each other in terms of cosine similarity of parameters and difference between bias terms. However, this heuristic does not necessarily depend on any previous observations of spline subdivisions and input-space partitions. To achieve a pruning ratio > 50%, this strategy needs to be applied sequentially and thus cannot leverage parallel computation as most existing pruning methods. In each round, its complexity is quadratic in the number of units since it needs to compute all pairwise similarities, which is also costly. Hence, I am worried about its efficiency in practice. 

(4) The experiments try to show the proposed strategy's advantage over existing pruning methods on efficiency (when achieving comparable accuracies). However, they did not try high enough pruning ratios. As reported, almost all the evaluated pruning ratios achieve similar accuracies, indicating that the pruned DNNs are still very redundant and the pruning does not enter a trade-off regime between accuracy and pruning ratio. In the current regime, "better pruning strategies" can not be well distinguished from "worse pruning strategies" and thus cannot provide strong evidence to show the advantage of the proposed method. 

Additional questions:

(1) Did you compare to any iterative pruning methods, which keep removing unimportant units/weights over epochs since the very beginning? They might be as efficient as or even more efficient than the early-bird metric used in this paper.

(2) Did you consider the difference of layers on their contributions to the final decision boundary and their difference in the convergence of activation patterns? Will these differences be helpful to improve the pruning performance?

(3) Instead of sequentially finding the redundant pairs of units, as proposed in Section 3.3, why not apply clustering based on the metric in Eq. (2)? It is not clear how to get the "grouped partition trajectories" without using any clustering-type method. 

(4) Though starting with max-affine spline DNs in Eq. (1), the paper only considers ReLU activations. Can you add empirical analysis and experiments for other activations belonging to this family?

(5) The current related works lack discussion about the research area for linear regions of neural nets, which covers a great number of papers and is strongly related to the spline subdivisions in this paper and thus cannot be ignored.
summary_of_the_review: Although both the input space partition of neural nets with piecewise linear activations (sometimes called linear regions, convex polytopes, etc.) and node/structured pruning have been widely studied and well known in recent years, this paper provides an interesting and novel perspective to relate them, i.e., explaining why a high pruning ratio can still work, early-stopping for pruning, and pruning by removing redundant units. The case studies clearly explain the empirical observations and well motivate some conclusions. However, the main results are a little bit disappointing to me due to the lack of in-depth discussion of the phenomenons. Necessary math formulation of some important concepts (e.g., splines, grouped partition trajectories, etc) is missing. Both the early-bird metric and the pruning strategy lack a strong or insightful connection to the spline partition and decision boundary discussed in the first part of the paper. The experiments have not explored higher pruning ratio regimes, which are necessary since the reported accuracy does not change too much over the evaluated pruning ratios. 

-------------Update---------------

The authors address most of my major concerns in their new reported experiments and updated draft. Therefore, I raise my score to 6.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper mainly proposes a new angle to understand the deep neural network pruning based on Spline theory and propose a new pruning algorithm approach. First, the author introduces the back ground of the Spline theory and current pruning methods. Second, the author analysis the different pruning methods from space partition perspective and introduce the proposed algorithm. Third, the author run experiments to evaluate the proposed algorithm in different aspects. 
main_review: Strength: 
1.the paper provides a new perspective on deep neural network pruning, 
2.the analysis through spline space partition is interesting. 
3.The paper has good written and visualization.
Weakness: 
1.The author claim that ‘The observation consistently shows that only parts of subdivision splines are useful for decision boundary; and the goal of pruning is to remove those (redundant) subdivision splines and find winning tickets.’, however, in theoretical part, the author didn’t provide how the proposed algorithm in detail to remove the subdivision splines. Will the algorithm need extra computation cost for such space partition building? 
2. When the author introduces the proposed algorithm, the author didn’t analysis if such method has the same convergence guarantee as Lottery Ticket Hypothesis. If so, what is the bound of the error probability?
3.In the experiment, the author didn’t consider Vision Transformer, which is an important SOTA model in image classification. And it is unsure if such technique is still working for larger image dataset such as ImageNet. Will the pruning strategy will be different in self attention layers?

summary_of_the_review: The idea and the observation is interesting, and most of the experiment results are promising. However, it is unclear how the author finds the subdivision spline to remove in experiment implementation. And in the theoretical part, I suggest the author to provide the convergence proof of proposed algorithm’s under Lottery Ticket hypothesis.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to use the max-affine spline function to understand network pruning in the deep learning architecture. Specifically, it uses two examples (one in fully-connected NN and one in CNN) to show the splines visualization with different pruning rates. Inspired by the theory, it then proposes a policy that is based on calculating the cosine similarity between slope and biases to determine the importance. The experiment results show the proposed metric could achieve a similar or better accuracy with good energy efficiency in multiple datasets with both structured pruning and unstructured pruning.
main_review: Pros:
1. The proposed idea to explain network pruning is quite interesting and could have a significant impact.
2. The experiment results do show the proposed method could achieve similar or better accuracy with good energy efficiency. 

Cons:
1. While the paper claims it bridges the understanding between pruning and decision boundary, however, I find it not convincing. It is only through two simple examples to illustrate the relation. There lacks a systematic way how to derive which spline function could be affect the final decision boundary. 
2. Some of their main claims are not well-supported. Proposition 1 is supporting evidence to conduct their pruning however it does not clearly show why Proposition 1 is working and only gives a reference to the paper proposing the deep learning spline function. Also, the early bird detection is also through two simple trajectories which are not convincing to me either. 
3. To use the global spline pruning, it has to first use PCA to shrink the space. However, the dimension reduction method is crucial to calculating the cosine similarity score. It could be seen in the appendix the performance is fluctuating with a not very small margin, which makes the method not reliable.
summary_of_the_review: Although the paper has proposed a very interesting perspective on network pruning, it lacks supporting evidence on some of their main claims. Therefore, I vote for a borderline paper with a tendency towards reject.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a novel methodology for deep network pruning from the perspective of the max-affine spline.
The key idea of the paper is to remove the redundant subdivision splines and find winning tickets.
This is an interesting and very nicely written paper that bridges the max-affine spline formulation and empirical pruning techniques.
The authors provide a sufficient and straightforward introduction of the behind motivation.
Beyond these, the authors discuss robustness considerations and perform thorough experiments on various benchmarked models and datasets, showing the favorable performance of their method.
main_review: - The idea of pruning the deep network by removing the redundant subdivision splines is a very novel idea. What's more, the authors interpret this idea with many vivid figures which is very readable.

- The authors first bridge the connection between deep network spline theory and deep network pruning. The authors provide new insights into how pruning deep network nodes affect the decision boundary. Furthermore, they leverage this new insight to develop a principled and efficient pruning strategy.

- Training deep network is a remaining huge obstacle. The authors propose a new robust and efficient algorithm for deep network training by effective network pruning techniques, which outperforms state-of-the-art competitors.

- There remains a lack of explicit understanding of network pruning impact on a deep network’s decision boundary. The authors illustrate the effectiveness of their method from many aspects.

- I don't understand why the authors choose the pairwise redundancy measure in (2). They need to give readers some more motivation. Is there any other measure that may be better? Some simple analysis may help us recognize it intuitively. 

- In section 4.3, Global spline pruning, the big problem is the mismatch of the filter dimension in different layers impedes the cosine similarity calculation. The authors solve this problem by adopting PCA for reducing the feature dimensions to the same. Is there any reason to measure the similarity between different layers by PCA or FA?
They need to clarify it.

- In practice, units with small enough but nonzero $N_\rho^l(k,k')$ are also highly redundant and can be removed. How to decide the threshold? The authors need to give more details.

summary_of_the_review: The paper is well-written and the ideas are well-presented. However, some points listed above need to be clarified.
review_title: null

Meta-review: ### Summary

The paper investigates the relation between pruning and splines. The results show that the prunable nodes of networks correspond to splines that do not affect the ultimate decision boundary of the network. The results also show that splines stabilize early in training, in correspondence with the claims of early bird tickets. Finally, the results show the connection between similarity-based pruning and splines, with an evaluation of the quality of similarity-based pruning.

### Discussion

The paper demonstrates a very interesting connection with splines and promising results with the multiple analyses. However, there could be more precision in the stated connections between splines and pruning. Also, reviewers requested more elaboration of the base technical definitions (e.g., splines).

### Recommendation

I recommend Reject. While the first part of this paper is quite strong in that the spline view of networks provides an interesting analysis framework that draws out interesting connections to pruning. The latter part that develops the concrete pruning algorithm needs reenvisioning. Specifically, to the best of my reading, using the similarity of filters for pruning is not new [1,2,3]. That means the proposed spline pruning policy approach cannot be claimed as new. Though it is reasonable to claim the connection to splines as new.

Second, while the evaluation is extensive in its comparison to other techniques, it is quite murky. The results claim victory by including both spline and EB spline, neither of which strictly dominates the other. My suggestion is to remove the EB spline and, in the worst case,
claim that similarity-based pruning is (1) competitive with other approaches and (2) rests on a foundational theory that is a new alternative to typical importance-based techniques. That latter observation is an important one for the community that can lead everyone into new productive directions. 

Additionally, in line with some requests from the reviews for more analysis, simplifying the evaluation (with fewer and less ambitious claims) and, even perhaps, eliminating the EB analysis altogether, would make room available for additional analysis and elaboration.

[1] BCAP: An Artificial Neural Network Pruning Technique to Reduce Overfitting
Kiante Brantley. UMD Masters Thesis, 2016

[2] A dynamic CNN pruning method based on matrix similarity
Mingwen Shao, Junhui Dai, Jiandong Kuang, Deyu Meng. Signal, Image and Video Processing volume 15, pages 381–389 (2021). Published August, 2020

[3] Similarity-Based Filter Pruning for Efficient Super-Resolution Models.
Chu Chu; Li Chen; Zhiyong Gao. IEEE International Symposium on Broadband Multimedia Systems and Broadcasting, 2020
Decision: Reject