Max-Affine Spline Insights Into Deep Network Pruning

Abstract: State-of-the-art (SOTA) approaches to deep network (DN) training overparametrize the model and then prune a posteriori to obtain a "winning ticket'' subnetwork that can be trained from scratch to achieve high accuracy. To date, the literature has remained largely empirical and hence provides little insights into how pruning affects a DN's decision boundary and no guidance regarding how to design a principled pruning technique. Using a recently developed spline interpretation of DNs, we develop new theory and visualization tools that provide new insights into how pruning DN nodes affects the decision boundary. We discover that a DN's spline mappings exhibit an early-bird (EB) phenomenon whereby the spline's partition converges at early training stages, bridging the recently developed max-affine spline theory and lottery ticket hypothesis of DNs. We leverage this new insight to develop a principled and efficient pruning strategy that focuses on a tiny fraction of DN nodes whose corresponding spline partition regions actually contribute to the final decision boundary. Extensive experiments on four networks and three datasets validate that our new spline-based DN pruning approach reduces training FLOPs by up to 3.5x while achieving similar or even better accuracy than state-of-the-art methods. All the codes will be released publicly upon acceptance.

Reviews and author responses:

 I appreciate the authors for the effort to address my concerns! I believe most of my major concerns have been addressed, and I would like to raise my score.

This paper presents a novel methodology for deep network pruning from the perspective of the max-affine spline.
The key idea of the paper is to remove the redundant subdivision splines and find winning tickets.
This is an interesting and very nicely written paper that bridges the max-affine spline formulation and empirical pruning techniques.
The authors provide a sufficient and straightforward introduction of the behind motivation.
Beyond these, the authors discuss robustness considerations and perform thorough experiments on various benchmarked models and datasets, showing the favorable performance of their method. - The idea of pruning the deep network by removing the redundant subdivision splines is a very novel idea. What's more, the authors interpret this idea with many vivid figures which is very readable.

- The authors first bridge the connection between deep network spline theory and deep network pruning. The authors provide new insights into how pruning deep network nodes affect the decision boundary. Furthermore, they leverage this new insight to develop a principled and efficient pruning strategy.

- Training deep network is a remaining huge obstacle. The authors propose a new robust and efficient algorithm for deep network training by effective network pruning techniques, which outperforms state-of-the-art competitors.

- There remains a lack of explicit understanding of network pruning impact on a deep network’s decision boundary. The authors illustrate the effectiveness of their method from many aspects.

- I don't understand why the authors choose the pairwise redundancy measure in (2). They need to give readers some more motivation. Is there any other measure that may be better? Some simple analysis may help us recognize it intuitively. 

- In section 4.3, Global spline pruning, the big problem is the mismatch of the filter dimension in different layers impedes the cosine similarity calculation. The authors solve this problem by adopting PCA for reducing the feature dimensions to the same. Is there any reason to measure the similarity between different layers by PCA or FA?
They need to clarify it.

- In practice, units with small enough but nonzero $N_\rho^l(k,k')$ are also highly redundant and can be removed. How to decide the threshold? The authors need to give more details.
 The paper is well-written and the ideas are well-presented. However, some points listed above need to be clarified.

 I appreciate the authors for their efforts to address my concerns! After checking all the results posted in the rebuttal and the new draft, I think most of my major concerns have been addressed. Therefore, I am glad to raise my score.

This paper conducts a series of empirical studies, which aim at relating node/filter/channel pruning with the partition of input space by neural nets using spline operators as activation functions. In particular, the final decision boundary is only determined by a few splines defined by a few filters of the network. This is verified in the paper through case studies of both fully-connected networks and convolutional networks, which implies that pruning does not severely degrade the accuracy as long as the important splines are not removed by pruning. Another observation is the early-bird tickets: the paper argues that the important splines do not change too much after a few early epochs since the binary activation patterns of data converge rapidly, so prunning can be applied after only a few epochs. The paper then proposes a pruning strategy that sequentially finds the most similar filter pairs and removes them. In experiments, they show that the proposed prunning method is more efficient than some recent pruning methods to achieve similar accuracy with pruning ratio <=70% for CIFAR10/100 and <=50% for ImageNet.  Please read the summary first. Here are detailed comments:

(1) Section 3.1 shows a trend in case studies that the pruning tends to preserve subdivision splines useful to the decision boundary and remove the redundant ones. As long as the decision boundary is preserved, the performance will not degrade. This is interesting. However, the two toy datasets here are too synthetic and no quantitative analysis is presented, especially for more complicated datasets or neural nets. Moreover, given the previous success of pruning on DNNs with piecewise linear activation functions like ReLU, the phenomenon observed here is somewhat obvious. I expected to find some in-depth analysis of why the trained decision boundary only depends on a small subset of splines and why the applied pruning method here can preserve them, which could be an important novelty of this paper, but only the phenomenon is described. The splines also lack a formal math definition in terms of the network parameters/filters/units, which could be very helpful to understand their relationship in pruning.  

(2) Section 3.2 claims that a novel metric based on those subdivision lines is developed to study the early-bird ticket phenomenon but the metric is simply the change of ReLU activation patterns between two consecutive epochs. It is obvious that if the activation patterns do not change too much, the decision boundary of the DNNs would not change too much, but this does not necessarily depend on the previous analysis of the spline partition of input space. The change of partition is only reflected via a toy example in Fig.4 without quantitative analysis and in-depth discussion, e.g., do all the splines or just the important ones (forming the boundary) not change since very early epochs, what happens to the partitioned regions that do not contain any training data, etc.

(3) I also have a similar concern about Section 3.3. It proposes an intuitive pruning strategy, i.e., removing the redundant units that are too similar to each other in terms of cosine similarity of parameters and difference between bias terms. However, this heuristic does not necessarily depend on any previous observations of spline subdivisions and input-space partitions. To achieve a pruning ratio > 50%, this strategy needs to be applied sequentially and thus cannot leverage parallel computation as most existing pruning methods. In each round, its complexity is quadratic in the number of units since it needs to compute all pairwise similarities, which is also costly. Hence, I am worried about its efficiency in practice. 

(4) The experiments try to show the proposed strategy's advantage over existing pruning methods on efficiency (when achieving comparable accuracies). However, they did not try high enough pruning ratios. As reported, almost all the evaluated pruning ratios achieve similar accuracies, indicating that the pruned DNNs are still very redundant and the pruning does not enter a trade-off regime between accuracy and pruning ratio. In the current regime, "better pruning strategies" can not be well distinguished from "worse pruning strategies" and thus cannot provide strong evidence to show the advantage of the proposed method. 

Additional questions:

(1) Did you compare to any iterative pruning methods, which keep removing unimportant units/weights over epochs since the very beginning? They might be as efficient as or even more efficient than the early-bird metric used in this paper.

(2) Did you consider the difference of layers on their contributions to the final decision boundary and their difference in the convergence of activation patterns? Will these differences be helpful to improve the pruning performance?

(3) Instead of sequentially finding the redundant pairs of units, as proposed in Section 3.3, why not apply clustering based on the metric in Eq. (2)? It is not clear how to get the "grouped partition trajectories" without using any clustering-type method. 

(4) Though starting with max-affine spline DNs in Eq. (1), the paper only considers ReLU activations. Can you add empirical analysis and experiments for other activations belonging to this family?

(5) The current related works lack discussion about the research area for linear regions of neural nets, which covers a great number of papers and is strongly related to the spline subdivisions in this paper and thus cannot be ignored. Although both the input space partition of neural nets with piecewise linear activations (sometimes called linear regions, convex polytopes, etc.) and node/structured pruning have been widely studied and well known in recent years, this paper provides an interesting and novel perspective to relate them, i.e., explaining why a high pruning ratio can still work, early-stopping for pruning, and pruning by removing redundant units. The case studies clearly explain the empirical observations and well motivate some conclusions. However, the main results are a little bit disappointing to me due to the lack of in-depth discussion of the phenomenons. Necessary math formulation of some important concepts (e.g., splines, grouped partition trajectories, etc) is missing. Both the early-bird metric and the pruning strategy lack a strong or insightful connection to the spline partition and decision boundary discussed in the first part of the paper. The experiments have not explored higher pruning ratio regimes, which are necessary since the reported accuracy does not change too much over the evaluated pruning ratios. 

-------------Update---------------

The authors address most of my major concerns in their new reported experiments and updated draft. Therefore, I raise my score to 6.

 Thanks for the feedback. However, I am still not convinced that only two illustrations are enough to show the spine function to actually affect the final decision boundary, which is a kind of overclaim in my opinion. I will keep my score.

 We greatly appreciate your careful review and recognition of our novel idea, vivid visualization, new insights, efficient training, and explicit understanding through many experiments. We also thank you for giving constructive suggestions. Below are our detailed responses to your questions. 

---
**Q1: Why use pairwise redundancy measure in (2)? Is there any other measure that may be better?**

We empirically find that the pairwise cosine similarity measurement facilitates the spline pruning to achieve better performance than non-pairwise counterparts and other distance metrics, e.g., L1 and L2. Note that the pairwise measurement is only conducted on channel-wise pruning so that pruning overhead is also negligible as analyzed in Section 3.3.

---
**Q2: Is there any reason to measure the similarity between different layers by PCA or FA?**

We find that nodes within different layers can perform with similar subdivision lines, so that we extend the layer-wise pruning to global pruning, resulting in more accurate pruned networks. Also, in both cases of PCA and FA, their goal is just to preserve as much information as possible while enforcing the dimensions of different filters to be consistent, so that our spline based pruning method is not sensitive to the adopted dimension reduction methods as long as it can be used to measure the correlation between two units.

---
**Q3: In practice, units with small enough but nonzero $N^l_{\rho}(k, k’)$ are also highly redundant and can be removed. How to decide the threshold?**

Good question! We decide the threshold according to the target pruning ratios. The threshold will be set as the borderline to prune the required number of filters/channels.

 We greatly appreciate your careful review and constructive suggestions. Below are our detailed responses to your concerns. 

---
**W1: Only two simple examples? Lack a systematic way for identifying which spline could affect the final decision boundary?**

Note that the two examples only aim for clear verification and visualization, we have also shown the experiments on four SOTA DNN models on both CIFAR and ImageNet datasets. Our proposed spline pruning policy can be regarded as an empirical way to identify a set of splines that contribute to the final decision boundary while not being redundant. The theoretical analysis of which splines directly contribute to the final decision boundary is however challenging without current theoretical solutions beyond measuring the impact on the final performances.

---
**W2: The proposition 1 is not clearly demonstrated? Two simple trajectories are not sufficient to justify the detection of the early birds?**

We have provided a short proof right below the proposition to further confirm our result.  
Apart from the two trajectory visualization, we report the EB tickets emerge epochs and retraining accuracies in Table 6 and Table 7, across a wide range of models, datasets, optimizers, and initializations. The results show that our spline EB tickets consistently emerge at early training stages and perform on par with their unpruned counterparts, and thus are empirically observed to be universal to different optimization methods and initializations.

---
**W3: The dimension reduction method is crucial, but the performance is fluctuating a lot?**

Actually, the performance of spline pruning with different dimension reduction methods does not fluctuate a lot, i.e., **the standard deviation is less than 0.5%**, which is standard and acceptable for the CIFAR dataset. We also show the results in the table below, from which we can see that our spline based pruning method is not sensitive to the adopted dimension reduction methods as long as it can be used to measure the correlation between two units.

| Setting | Methods | Accuracy (p=30%) | Accuracy (p=50%) | Accuracy (p=70%) |
|---|---|:---:|:---:|:---:|
| PreResNet-101@CIFAR-10 | Spline (FA) | 94.27% | 94.26% | 91.97% |
| PreResNet-101@CIFAR-10 | Spline (PCA) | 94.13% | 93.92% | 92.06% |
| | Difference (FA vs. PCA)  | 0.14% | 0.34% | 0.09% |
| VGG-16@CIFAR-10 | Spline (FA) | 93.1% | 93.25% | 92.49% |
| VGG-16@CIFAR-10 | Spline (PCA) | 93.62% | 93.46% | 92.85% |
| | Difference (FA vs. PCA)  | 0.52% | 0.21% | 0.36% |
| PreResNet-101@CIFAR-100 | Spline (FA) | 74.37% | 72.94% | 68.02% |
| PreResNet-101@CIFAR-100 | Spline (PCA) | 73.79% | 72.04% | 68.24% |
| | Difference (FA vs. PCA)  | 0.58% | 0.90% | 0.22% |
| | | Accuracy (p=10%) | Accuracy (p=30%) | Accuracy (p=50%) |
| VGG-16@CIFAR-100 | Spline (FA) | 72.2% | 71.99% | 70.62% |
| VGG-16@CIFAR-100 | Spline (PCA) | 72.18% | 71.54% | 70.07% |
| | Difference (FA vs. PCA)  | 0.02% | 0.45% | 0.55% |


 We greatly appreciate your careful review and constructive suggestions. Below are our detailed responses to your comments/concerns. 

---
**W1: Theoretical measurement about how the spline pruning removes the subdivision splines? Will the pruning algorithm need extra computation cost for space partition building?**

We entirely agree on the crucial role of this question. While this is an extremely challenging task without current theoretical solutions beyond measuring the impact on the final performances, we try our best to supply the analysis on why there is redundant partition boundaries to Appendix J of the revised manuscript, and empirical evaluations/validations of spline pruning for removing such redundant splines as an important first step, that provides critical insights and inspirations for pushing our knowledge boundary and inspire in-depth theoretical study in future work.

As for the pruning overhead, we humbly point out that we analyzed the efficiency of the spline pruning in Sec. 3.3, the pruning overhead including the space partition building is negligible when evaluating across four models and three datasets. For example, ResNet-50 has about 20k channels and needs 100G FLOPs to prune, which is only 1/30,000,000 of the total training costs (3000P FLOPs).

---
**W2: Analysis on if the proposed pruning method has the same convergence guarantee as lottery ticket hypothesis (LTH)? What is the bound of the error probability?**

Good suggestion! The original LTH paper does not give such convergence guarantees, while recent work [1][2] attempts to prove the LTH towards guarantee that the pruned subnets can achieve a similar accuracy as original dense networks with strong assumption of sufficient over-parameterization and bounded weights; nevertheless, their analysis mostly focuses on only two-layer neural networks, it still remains challenging to prove the convergences of modern deep neural networks (e.g., VGG-16, ResNet-50) pruning and training. We leave more in-depth convergence analysis as a future work. To give more insights, we also supply the analysis about the connections between LTH and spline theory to Appendix G of the manuscript.

Meanwhile, we humbly emphasize that in this paper our main contributions instead focus on (1) how to relate the pruning of DN nodes or weights to the DN input space partition, the per-region affine parameters, and the decision boundary; (2) identify spline EB tickets that are invariant to pruning ratios; and (3) derive an efficient pruning strategy based on the aforementioned decision boundary analysis and spline EB tickets detection,  providing an explicit interpretation and visualization of existing empirical pruning strategies and our spline prunings at various granularity levels.

[1] Pensia et al., Optimal Lottery Tickets via Subset Sum: Logarithmic Over-Parameterization is Sufficient, NeurIPS’20

[2] Malach et al., Proving the Lottery Ticket Hypothesis: Pruning is All You Need, ICML’20

---
**W3: Will spline pruning work for vision transformer (ViT) and larger dataset such as ImageNet?**

For larger datasets such as ImageNet, we do provide the results as shown in Table 3 of the submitted manuscript. For self-attention layers in ViT, the spline theory is not directly applicable due to the softmax function which is not continuous piecewise affine. However, one possible direction would be to model such layers as ‘’smooth splines’’. We have added a reference to a possible paper that would allow such extension in the Appendix R of the revised manuscript and pointed this as an interesting future direction.

 **Q3: Why not apply clustering based on the metric in Eq. 2?**

This is a very interesting suggestion. Currently, our method does prune a group of units that have high similarity to one remaining unit after calculating the pairwise distance between each other. However, this is done sequentially. If first using a clustering method, one could identify the group of similar units at once, and then prune all but one. While this would reduce the amount of sequential steps, it would introduce an additional cost of first fitting a clustering method, whose choice might also impact the performances. Since we are already able to achieve improved performances with reduced computational cost (see Tables 1, 2, 3 and 4), we would prefer to defer such efficiency improvements for future work given the limited rebuttal time. However, we will gladly mention it in the conclusion as an interesting direction.

---
**Q4: Only consider ReLU activation? Can you add empirical analysis and experiments for other activations belonging to this family?**

Great request. Following your suggestions, we redraw Fig. 4 (spline trajectory) and Fig. 5 (early-bird phenomenon) and further supply the pruning experiment comparisons using networks with Leaky-ReLU activation functions. The figures are supplied to Appendix N of our revised manuscript; we still consistently observe that the splines will adapt during the early phase while converging at later training stages. The supplied experiments are shown in the table below, from which we see that the spline pruning consistently outperforms all baselines in terms of both accuracy and efficiency, leading to -0.44% ~ +1.38% accuracy improvement and up to 4x training flops savings. We add this set of experiments and discussion to Table 9 and Appendix N of the revised manuscript.

| Setting | Methods | Accuracy (p=30%) | Accuracy (p=50%) | Accuracy (p=70%) | Training FLOPs (p=30%) | Training FLOPs (p=50%) | Training FLOPs (p=70%) |
|---|---|:---:|:---:|:---:|:---:|:---:|:---:|
| PreResNet-101@CIFAR-10 | NS | 93.54% | 93.18% | 90.89% | 13.9P | 12.7P | 11.0P |
| PreResNet-101@CIFAR-10 | ThiNet | 92.18% | 91.57% | 90.25% | 13.2P | 10.6P | 8.65P |
| PreResNet-101@CIFAR-10 | Spline | 93.56% | 92.74% | 91.11% | 13.6P | 12.1P | 10.1P |
| PreResNet-101@CIFAR-10 | EB Spline | 92.76% | 92.13% | 90.66% | 6.00P | 4.26P | 2.74P |
|  | **Spline Improv.** |  | -0.44% ~ +1.38% |  |  | 1.1x ~ 4.0x |  |

---
**Q5: Missing related work discussion about the linear regions of neural networks.**

We thank the reviewer for pointing this out. We entirely agree on this point and have added references in the related work section to papers that study both the evolution in the number of linear regions, the complexity of those regions, along with analytical results on the geometry of those regions.

 **Q1: Compare with iterative pruning methods?**

We did consider the progressive pruning (iteratively training and pruning) methods, e.g., RigL [4], and another two pruning at initialization methods, GraSP [5] and SynFlow [6] in Appendix B of our submitted manuscript. To better address your comment, we further supply the comparison with network slimming with the mentioned iterative pruning method (NS-IP). As shown in the table below, our spline pruning again outperforms those methods in terms of both accuracy and efficiency, leading to -0.47% ~ +1.11% accuracy improvement and up to 7.1x training flops savings. We add this set of experiments and discussion to Table 11 and Appendix P of the revised manuscript.

| Setting | Methods | Accuracy (p=30%) | Accuracy (p=50%) | Accuracy (p=70%) | Training FLOPs (p=30%) | Training FLOPs (p=50%) | Training FLOPs (p=70%) |
|---|---|:---:|:---:|:---:|:---:|:---:|:---:|
| PreResNet-101@CIFAR-10 | NS-IP | 93.74% | 92.72% | 92.53% | 23.6P | 21.6P | 18.7P |
| PreResNet-101@CIFAR-10 | Spline | 94.13% | 93.92% | 92.06% | 13.6P | 12.1P | 10.1P |
| PreResNet-101@CIFAR-10 | EB Spline | 93.67% | 93.18% | 92.32% | 6.00P | 4.26P | 2.74P |
|  | **Spline Improv.** | +0.39% | +1.20% | -0.47% |  | 1.7x ~ 6.8x |  |
| PreResNet-101@CIFAR-100 | NS-IP | 72.68% | 71.82% | 69.86% | 23.3P | 21.3P | 17.5P |
| PreResNet-101@CIFAR-100 | Spline | 73.79% | 72.04% | 68.24% | 12.6P | 10.9P | 9.36P |
| PreResNet-101@CIFAR-100 | EB Spline | 72.67% | 71.99% | 69.74% | 5.44P | 3.84P | 2.46P |
|  | **Spline Improv.** | +1.11% | +0.22% | -0.12% |  | 1.8x ~ 7.1x |  |

[4] Evci et al., Rigging the Lottery: Making All Tickets Winners, ICML’20

[5] Wang et al., Picking winning tickets before training by preserving gradient flow. ICLR’20

[6] Tanaka et al., Pruning neural networks without any data by iteratively conserving synaptic flow, NeurIPS’20

---
**Q2: Different layers’ contribution and differences in the convergences of activation patterns? Will these differences be helpful to improve the pruning performance?**

Great question. To further provide quantitative analysis and in-depth discussion, we measure and record the quantitative distance between input space partitions of different layers, i.e., early, middle, or later layers, along the training trajectory. As shown in the table below, the quantitative distance of early layers converges faster than later layers, indicating that the base partition regions divided by the first few layers will not change too much since the very early epochs. Analysis of different layers’ contribution and differences definitely are helpful for guiding the pruning by (1) setting layer-wise pruning ratios according to the importance of different layers; and (2) progressively pruning different layers during training dense networks, e.g., prune early layers first and then later layers. For a theoretical understanding of this phenomenon, an entire dedicated study involving spline subdivision and approximation theory would be required. We would thus like to leave such exploration as future work considering that even the most recent results on deep networks approximation theory using splines (e.g. Nonlinear Approximation and (Deep) ReLU Networks, Daubechies et al., 2021) have not yet provided per-layer analysis of convergence rates separately from a given deep network. We will add this set of experiments and discussion in our revision. We add this set of experiments and discussion to the Table 12 and Appendix Q of the revised manuscript.

| Datasets | Models | Layers | Distance (10th epoch) | Distance (40th epoch) | Distance (80th epoch) | Distance (120th epoch) | Distance (160th epoch) | 
|---|---|---|:---:|:---:|:---:|:---:|:---:|
| CIFAR-100 | VGG-16 | Early (1st) | 0.247 | 0.197 | 0.061 | 0.021 | 0.003 |
| CIFAR-100 | VGG-16 | Medium (8th) | 0.538 | 0.424 | 0.183 | 0.024 | 0.003 |
| CIFAR-100 | VGG-16 | Later (16th) | 0.579 | 0.512 | 0.236 | 0.027 | 0.004 |
| CIFAR-100 | PreResNet-101 | Early (1st) | 0.621 | 0.562 | 0.296 | 0.023 | 0.002 |
| CIFAR-100 | PreResNet-101 | Medium (16th) | 0.704 | 0.612 | 0.273 | 0.025 | 0.003 |
| CIFAR-100 | PreResNet-101 | Later (32th) | 0.781 | 0.728 | 0.376 | 0.038 | 0.005 |

 **C2: Section 3.2 metric novelty? Relation between activation patterns and spline partition of input space? Quantitative analysis? What happens to the partitioned regions that do not contain training data?**

*Continued response:*

We provide in Figs 2,3 and 4 2-dimensional (slices) of the data space to visualize the evolution of the partition boundaries. Those representations consider the entire (slice) space regardless of the position of the training samples. Then, in Fig 5 we show the evolution (through the binary feature map metric) of the partition boundaries during training to detect early bird tickets. The latter needs to be computed based on data samples. For this figure, we opted to display it for the training set samples, as those are the most relevant ones. However, your suggestion to sample it at different parts of the space is indeed interesting and could reveal intricate properties on how a DN partition evolves around training samples and at other places. This is why we added in Appendix M of the revised manuscript a reproduction of Fig. 5 but now computed based on (1) test samples and (2) random samples. And very interestingly, we observe that the early-bird ticket phenomenon is also present and can be detected similarly. We carefully pointed to those new figures and conclusions in the main paper. 

---
**C3: Section 3.3 relates to previous observations? Is its complexity costly?**

Yes, we humbly clarify that the spline pruning indeed relates to both Section 3.1 and Section 3.2 since that (1) the pruning is based on the input space partition to identify “redundant” units that provide similar subdivisions; and (2) the pruning is based on the identified spline early-bird tickets. Furthermore, we do include the efficiency analysis in Section 3.3 of the original manuscript. Instead of using parameter-wise pruning, we perform channel-wise pruning. So for a convolutional layer, an entire filter is pruned at once, making the computational complexity quadratic only in the number of filters, and not the dimension of the feature maps. Hence, the pruning overhead is negligible when evaluating across four models and three datasets. For example, ResNet-50 has about 20k channels and needs 100G FLOPs to prune, which is only 1/30,000,000 of the total training costs (3000P FLOPs).

---
**C4: Missing high pruning ratios in the experiment?**

Thank you for your comment. We humbly point out that it is not common for channel-wise structural pruning to reach a 90% sparsity ratio as current SOTA methods [1][2][3] indicate. To better address your comment and fairly compare with the baselines, we compare our Spline pruning with network slimming (NS) and ThiNet baselines under a 90% pruning ratio. As shown in the table below, the proposed spline pruning outperforms the baselines across two models (PreResNet-101 and VGG-16) and two datasets (CIFAR-10 and CIFAR-100), leading to -0.35% ~ +63.87% accuracy improvements. Note that the NS method suffers from bottleneck layers due to the “over-pruning” channels on specific layers. We add this set of experiments and discussion to Table 10 and Appendix O of the revised manuscript.

| Datasets | Models | Methods | Accuracy (p=90%) |
|---|---|---|:---:|
| CIFAR-10 | PreResNet-101 | NS | 77.63% |
| CIFAR-10 | PreResNet-101 | ThiNet | 87.80% |
| CIFAR-10 | PreResNet-101 | Spline | 87.96% |
| CIFAR-10 | PreResNet-101 | EB Spline | 88.15% |
|  | **Spline Improv.** |  | +0.35% ~ +10.52% |
| CIFAR-10 | VGG-16 | NS | 89.13% |
| CIFAR-10 | VGG-16 | ThiNet | 86.90% |
| CIFAR-10 | VGG-16 | Spline | 89.11% |
| CIFAR-10 | VGG-16 | EB Spline | 89.24% |
|  | **Spline Improv.** |  | +0.11% ~ +2.34% |
| CIFAR-100 | PreResNet-101 | NS | 28.90% |
| CIFAR-100 | PreResNet-101 | ThiNet | 60.66% |
| CIFAR-100 | PreResNet-101 | Spline | 58.97% |
| CIFAR-100 | PreResNet-101 | EB Spline | 60.31% |
|  | **Spline Improv.** |  | -0.35% ~ +31.41% |
| CIFAR-100 | VGG-16 | NS | 1% |
| CIFAR-100 | VGG-16 | ThiNet | 56.31% |
| CIFAR-100 | VGG-16 | Spline | 59.37% |
| CIFAR-100 | VGG-16 | EB Spline | 64.87% |
|  | **Spline Improv.** |  | +8.56% ~ 63.87% |

[1] Liu et al., Learning Efficient Convolutional Networks through Network Slimming, ICCV’17

[2] Guo et al., Channel Pruning Guided by Classification Loss and Feature Importance, AAAI’20

[3] Guan et al., DAIS: Automatic Channel Pruning via Differentiable Annealing Indicator Search, arXiv’21 

 We thank the reviewer for carefully reading our paper. The reviewer’s summary is on point and well describes our submission. Overall concerns: lack of in-depth analysis/discussion/insights into the partitions, and needing more extreme pruning ratios. Overall rebuttal: we added many new figures/discussions to provide insights, and performed the requested experiments.
We answer in detail each of the reviewer’s concerns below.

---
**C1: Section 3.1 intuition on why the trained decision boundary only depends on a small subset of splines and why the applied pruning method here can preserve them?**

We agree that the 2-dimensional dataset of Fig. 2 is too simplistic. This is why we also provided Fig 3, which is done on the MNIST dataset. This is also quite simplistic, however it allowed us to use a relatively simple convolutional network and obtain all the partition boundaries from it that we then visualized in 2-dimension along with the decision boundary. However, we entirely agree that one fundamental question is “why do DN partition boundaries are in most cases irrelevant to the task at hand”. We have dedicated a section in the Appendix J of the revised manuscript to show a potential explanation. In short, to reach successful training, using gradient descent, we need to start from an over-parameterized model. Otherwise, the network gets stuck in a poor local minima. By over-parametrizing (giving more partition boundaries than needed) the likelihood of having at least a few of them at a good initialization is increased and training is successful. After training however, most of the added partition boundaries are redundant and can be removed without affecting the decision boundary. We illustrated this through a carefully designed pruning experiment in the Appendix K of the revised manuscript. We also added a sentence pointing to that discussion in the main paper. 

---
**C2: Section 3.2 metric novelty? Relation between activation patterns and spline partition of input space? Quantitative analysis? What happens to the partitioned regions that do not contain training data?**

We apologize for not clarifying our core metric contribution in a simpler way. Our core metric that guides pruning is given in section 3.3 in equation (2). This metric is (as far as we are aware) novel and non-obvious. It quantifies how close are the partition boundaries produced by different units. However, in section 3.2 we indeed proposed an early-bird ticket detection metric which is quite standard and has been used before in other contexts (similarity search, neural tangent kernel) whose goal is only to detect early bird tickets. That is, and as you well described, its role is solely to detect when changes in the partition boundary stop to occur after consecutive epochs. However, the goal of this metric/experiment/section is solely to point out that early-bird tickets can also be detected in the partition space, while our core metric actually used to prune units is proposed in section 3.3.

To better understand the relationship between activation patterns and spline partition of input space, we supply Fig. 12 and Appendix L to the revised manuscript for clarification. Basically, each hidden node of the network corresponds to one subdivision line for distinguishing non-zero activations from zeros in the input grid.

To further provide quantitative analysis and in-depth discussion, we measure and record the quantitative distance between input space partitions of different layers, i.e., early, middle, or later layers, along the training trajectory. As shown in the table below, the quantitative distance of early layers converges faster than later layers, indicating that the base partition regions divided by the first few layers will not change too much since the very early epochs. We add this set of experiments to Table 12 and Appendix Q of the revised manuscript.

| Datasets | Models | Layers | Distance (10th epoch) | Distance (40th epoch) | Distance (80th epoch) | Distance (120th epoch) | Distance (160th epoch) | 
|---|---|---|:---:|:---:|:---:|:---:|:---:|
| CIFAR-100 | VGG-16 | Early (1st) | 0.247 | 0.197 | 0.061 | 0.021 | 0.003 |
| CIFAR-100 | VGG-16 | Medium (8th) | 0.538 | 0.424 | 0.183 | 0.024 | 0.003 |
| CIFAR-100 | VGG-16 | Later (16th) | 0.579 | 0.512 | 0.236 | 0.027 | 0.004 |
| CIFAR-100 | PreResNet-101 | Early (1st) | 0.621 | 0.562 | 0.296 | 0.023 | 0.002 |
| CIFAR-100 | PreResNet-101 | Medium (16th) | 0.704 | 0.612 | 0.273 | 0.025 | 0.003 |
| CIFAR-100 | PreResNet-101 | Later (32th) | 0.781 | 0.728 | 0.376 | 0.038 | 0.005 |


This paper mainly proposes a new angle to understand the deep neural network pruning based on Spline theory and propose a new pruning algorithm approach. First, the author introduces the back ground of the Spline theory and current pruning methods. Second, the author analysis the different pruning methods from space partition perspective and introduce the proposed algorithm. Third, the author run experiments to evaluate the proposed algorithm in different aspects.  Strength: 
1.the paper provides a new perspective on deep neural network pruning, 
2.the analysis through spline space partition is interesting. 
3.The paper has good written and visualization.
Weakness: 
1.The author claim that ‘The observation consistently shows that only parts of subdivision splines are useful for decision boundary; and the goal of pruning is to remove those (redundant) subdivision splines and find winning tickets.’, however, in theoretical part, the author didn’t provide how the proposed algorithm in detail to remove the subdivision splines. Will the algorithm need extra computation cost for such space partition building? 
2. When the author introduces the proposed algorithm, the author didn’t analysis if such method has the same convergence guarantee as Lottery Ticket Hypothesis. If so, what is the bound of the error probability?
3.In the experiment, the author didn’t consider Vision Transformer, which is an important SOTA model in image classification. And it is unsure if such technique is still working for larger image dataset such as ImageNet. Will the pruning strategy will be different in self attention layers?
 The idea and the observation is interesting, and most of the experiment results are promising. However, it is unclear how the author finds the subdivision spline to remove in experiment implementation. And in the theoretical part, I suggest the author to provide the convergence proof of proposed algorithm’s under Lottery Ticket hypothesis. 

The paper proposes to use the max-affine spline function to understand network pruning in the deep learning architecture. Specifically, it uses two examples (one in fully-connected NN and one in CNN) to show the splines visualization with different pruning rates. Inspired by the theory, it then proposes a policy that is based on calculating the cosine similarity between slope and biases to determine the importance. The experiment results show the proposed metric could achieve a similar or better accuracy with good energy efficiency in multiple datasets with both structured pruning and unstructured pruning. Pros:
1. The proposed idea to explain network pruning is quite interesting and could have a significant impact.
2. The experiment results do show the proposed method could achieve similar or better accuracy with good energy efficiency. 

Cons:
1. While the paper claims it bridges the understanding between pruning and decision boundary, however, I find it not convincing. It is only through two simple examples to illustrate the relation. There lacks a systematic way how to derive which spline function could be affect the final decision boundary. 
2. Some of their main claims are not well-supported. Proposition 1 is supporting evidence to conduct their pruning however it does not clearly show why Proposition 1 is working and only gives a reference to the paper proposing the deep learning spline function. Also, the early bird detection is also through two simple trajectories which are not convincing to me either. 
3. To use the global spline pruning, it has to first use PCA to shrink the space. However, the dimension reduction method is crucial to calculating the cosine similarity score. It could be seen in the appendix the performance is fluctuating with a not very small margin, which makes the method not reliable. Although the paper has proposed a very interesting perspective on network pruning, it lacks supporting evidence on some of their main claims. Therefore, I vote for a borderline paper with a tendency towards reject. 

Meta-review: ### Summary

The paper investigates the relation between pruning and splines. The results show that the prunable nodes of networks correspond to splines that do not affect the ultimate decision boundary of the network. The results also show that splines stabilize early in training, in correspondence with the claims of early bird tickets. Finally, the results show the connection between similarity-based pruning and splines, with an evaluation of the quality of similarity-based pruning.

### Discussion

The paper demonstrates a very interesting connection with splines and promising results with the multiple analyses. However, there could be more precision in the stated connections between splines and pruning. Also, reviewers requested more elaboration of the base technical definitions (e.g., splines).

### Recommendation

I recommend Reject. While the first part of this paper is quite strong in that the spline view of networks provides an interesting analysis framework that draws out interesting connections to pruning. The latter part that develops the concrete pruning algorithm needs reenvisioning. Specifically, to the best of my reading, using the similarity of filters for pruning is not new [1,2,3]. That means the proposed spline pruning policy approach cannot be claimed as new. Though it is reasonable to claim the connection to splines as new.

Second, while the evaluation is extensive in its comparison to other techniques, it is quite murky. The results claim victory by including both spline and EB spline, neither of which strictly dominates the other. My suggestion is to remove the EB spline and, in the worst case,
claim that similarity-based pruning is (1) competitive with other approaches and (2) rests on a foundational theory that is a new alternative to typical importance-based techniques. That latter observation is an important one for the community that can lead everyone into new productive directions. 

Additionally, in line with some requests from the reviews for more analysis, simplifying the evaluation (with fewer and less ambitious claims) and, even perhaps, eliminating the EB analysis altogether, would make room available for additional analysis and elaboration.

[1] BCAP: An Artificial Neural Network Pruning Technique to Reduce Overfitting
Kiante Brantley. UMD Masters Thesis, 2016

[2] A dynamic CNN pruning method based on matrix similarity
Mingwen Shao, Junhui Dai, Jiandong Kuang, Deyu Meng. Signal, Image and Video Processing volume 15, pages 381–389 (2021). Published August, 2020

[3] Similarity-Based Filter Pruning for Efficient Super-Resolution Models.
Chu Chu; Li Chen; Zhiyong Gao. IEEE International Symposium on Broadband Multimedia Systems and Broadcasting, 2020

Decision: Reject