SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hoang Pham<sup>1</sup>, The-Anh Ta<sup>2</sup>, Shiwei Liu<sup>3,6</sup>, Lichuan Xiang<sup>4</sup>, Dung D. Le<sup>5</sup>, Hongkai Wen<sup>4</sup>, Long Tran-Thanh<sup>4</sup>
$^{1}$  FPT Software AI Center,  $^{2}$  CSIRO's Data61,  $^{3}$  University of Texas at Austin,  $^{4}$  University of Warwick,  $^{5}$  VinUniversity  $^{6}$  Eindhoven University of Technology hoang.pv1602@gmail.com, theanh.ta@csiro.au, {L.Xiang.2, Hongkai.Wen, Long.Tran-Thanh}@warwick.ac.uk dung_ld@vinuni.edu.vn, shiwei.liu@austin.utexas.edu
Pruning at initialization (PaI) aims to remove weights of neural networks before training in pursuit of training efficiency besides the inference. While off-the-shelf PaI methods manage to find trainable subnetworks that outperform random pruning, their performance in terms of both accuracy and computational reduction is far from satisfactory compared to post-training pruning and the understanding of PaI is missing. For instance, recent studies show that existing PaI methods only able to find good layerwise sparsities not weights, as the discovered subnetworks are surprisingly resilient against layerwise random mask shuffling and weight re-initialization. In this paper, we study PaI from a brand-new perspective – the topology of subnetworks. In particular, we propose a principled framework for analyzing the performance of PaI methods with two quantities, namely, the number of effective paths and effective nodes. These quantities allow for a more comprehensive understanding of PaI methods, giving us an accurate assessment of different subnetworks at initialization. We systematically analyze the behavior of various PaI methods through our framework and observe a guiding principle for constructing effective subnetworks: at a specific sparsity, the top-performing subnetwork always presents a good balance between the number of effective nodes and the number of effective paths. Inspired by this observation, we present a novel data-agnostic pruning method by solving a multi-objective optimization problem. By conducting extensive experiments across different architectures and datasets, our results demonstrate that our approach outperforms state-of-the-art PaI methods while it is able to discover subnetworks that have much lower inference FLOPs (up to  $3.4 \times$ ). Code is available at: https://github.com/pvh1602/NPB.
Deep neural networks have achieved state-of-the-art performance in a wide range of machine learning applications [3, 13, 39, 38]. However, the huge computational resource requirements limit their applications, especially in edge computing and other future smart cyber-physical systems [23, 48, 37, 47, 2]. To overcome this issue, a number of approaches have been proposed to reduce the size of deep neural networks without compromising performance, among which pruning has received voluminous attention [10, 24, 8]. Traditional pruning approaches mainly focus on accelerating inference, which usually require a pre-trained dense network [33, 20, 31].
As large language models (LLMs) [3, 43, 44] continue to gain popularity, endeavors start to explore the possibility to prune models before training while matching the dense performance. Lottery Ticket
Hypothesis (LTH) [15, 4, 5] provides empirical evidence for this research goal by discovering sparse subnetworks that can be trained from scratch to the comparable performance of the dense network. However, LTH typically involves the costly iterative pruning and re-training process, whose overall cost is much more than training a dense network.
This issue raises an intriguing research question: How to identify sparse, trainable subnetworks at initialization without pre-training? Specifically, a successful pruning before training method can significantly reduce both the cost of memory and runtime, without sacrificing performance much [46]. This would make neural networks applicable even in scenarios with scarce computing resources [1, 47]. As such, many methods for PaI have been proposed [26, 42, 9, 45, 1, 29]. While these methods are based on a number of intuitions (e.g., leveraging the gradient information [26, 45]), they typically measure the importance of network parameters. Their performance in terms of both accuracy and computational reduction is far from satisfactory compared to post-training pruning and the understanding of PaI is missing. More recently, Frankle et al. [17], Su et al. [40] observe a rather surprising phenomenon: for PaI methods, layerwise shuffling connections of pruned networks does not reduce the network's performance, which suggests that layerwise sparsity ratios might be more important than weight-level importance scores. This indicates that in searching for good subnetworks at initialization, the topology of subnetworks, particularly, the number of input-output paths and active nodes, plays a vital role and should be investigated more extensively.
In this paper, we first present a counter-argument against the previous findings: while Frankle et al. [17] show that PaI methods are insensitive to random shuffling, we find this is not true in the extreme sparsity regime ( $>99\%$ ), as the number of effective paths heavily suffers from random shuffling. In layerwise shuffling experiments (see Section 3.3), shuffling connections results in more effective nodes but substantially fewer input-output paths. In normal sparsity levels, shuffling weights in regular sparsities can maintain and even increase effective parameters [17] and the number of activated nodes [36]. This increases representation capacity and performance of the subnetwork. However, at extremely sparse levels, shuffling still preserves roughly the same number of effective nodes, but the performance of shuffled subnetworks drops significantly compared to their unshuffled counterparts. This is because random weight shuffling damages effective paths and hampers information flow within the subnetwork. These findings suggest that separately considering effective paths or nodes is insufficient to fully capture the behavior of subnetworks generated by PaI methods.
To underscore the critical importance of nodes and paths in designing network architecture, we conduct an analysis of their quantities in the context of Neural Architecture Search (NAS). For instance, Figure 1 illustrates the relationship between candidate's performance, nodes and paths in NAS-Bench-Macro [41] which has 6,561 models (more in Section 3.4). To ensure fairness in terms of model size, we compare networks with similar number of parameters (i.e., in a range of  $300\mathrm{k}$  parameters in Figure 1). Remarkably, architectures exhibiting a higher number of input-output paths and active neurons concurrently yield superior performance. This highlights the crucial role of simultaneously considering both node and path in the successful design of networks at initialization.
Figure 1: The accuracy of architectures in NAS-Bench-Macro benchmark along with the number of effective nodes and paths.
To improve upon the current understanding of PaI methods, we introduce a novel framework from the perspective of subnetwork topology to provide a more accurate and comprehensive explanation of the performance of different approaches. In particular, we propose the joint usage of both the number of input-output paths (a.k.a. effective paths) and activated (i.e., effective) nodes to interpret behaviors of different PaI methods in a more comprehensive way (see Section 3.3 for more details). Inspired by the guiding principle observed through our framework, we further introduce NPB, a novel pruning at initialization by solving a node-path balancing objective. We show that NPB outperforms state-of-the-art PaI methods in almost all settings, while achieving much lower inference FLOPs. In summary, our main contributions are:
- We propose a unified effect node-path perspective to understand the behavior of PaI, particularly considering metrics of effective nodes and paths as proxies for the performance of PaI methods (Section 3). We revisit the layerwise shuffling sanity check on PaI methods and provide unified explanations for their behaviors based on these metrics in a wide range of sparsities (Section 3.3).
- We discover a new relation between the proposed metrics and the performance of subnetworks, termed the Node-Path Balancing Principle, that suggests a non-trivial balance between nodes and paths is necessary for optimal performance of PaI (Section 4).
- We present a novel data-agnostic pruner, NPB, by solving a multi-objective optimization problem to give a proof-of-concept for our principle. With extensive experiments, we show that NPB outperforms the best state-of-the-art PaI methods, PHEW, on 11 out of 12 settings while ours produces more efficient subnetworks with much fewer inference FLOPs (up to  $3.4 \times$ ) and faster in pruning time than PHEW (up to  $14.5 \times$ ) (Section 5).
Neural Network Pruning. Neural network pruning methods [25, 21, 20, 12, 32] traditionally focus on pruning trained models based on pre-defined criteria, and then resulting subnetworks will be fine-tuned to converge. Recently, [15, 16, 4, 5] empirically show the existence of randomly initialized subnetworks (lottery tickets) which when trained from scratch or in early training iterations, that can achieve competitive performance with their dense counterparts. Unfortunately, finding lottery tickets is computationally expensive due to the train and prune cycle [16, 15]. Gradual pruning methods [49, 18] interleave the pruning and training, which are usually cheaper than pruning after training, but the network still needs to be trained to choose the ideal sparse subnetwork. Other methods [6, 7] apply one-shot pruning during training to further reduce the computational cost. Dynamic sparse training [30, 14, 28, 27], on the other hand, start with a (random) sparse network and update network connectivity during training. While pruning before training methods [26, 45, 36, 42, 1] determine subnetworks by the network initialization, gradient information, and network topology. However, experimental results done by Frankle et al. [17], Su et al. [40] show that current criteria of PaI methods may not be sufficient to obtain a subnetwork with good performance.
Pruning and Network Shape. Since PaI methods do not utilize training data [42, 36] or use only negligible portions of data [26, 45] to obtain gradient information without training, the configuration of nodes and connections is an essential source of information for optimizing the performance of pruned networks. It turns out that some PaI methods implicitly optimize certain aspects of network shape. In particular, SynFlow [42] preserves the number of input-output paths as synaptic strength, but often creates more isolated neurons in pruned networks. The works of Patil and Dovrolis [36] and Gebhart et al. [19] aim to preserve proxies in terms of path kernels which are also directly related to the shape of subnetworks. Furthermore, while PHEW [36] additionally implements random walks to increase the number of effective nodes, it unintentionally decreases the number of input-output paths. Our new point of view on node-path balancing would be helpful to systematically optimize network configuration for better performance. Other works also consider the number of effective nodes and effective paths to capture the capacity of pruned subnetworks [45, 34] where these numbers are considered separately.
3.1 Pruning at Initialization Methods
Given a  $L$  layer neural network, we denote  $\mathbf{w} = (w_{1},\dots ,w_{L})$  as the set of vectorized weights. Pruning generates binary mask vectors  $m_{\ell}\in \{0,1\}^{d_{\ell}}$  ( $d_{\ell}$  is the number of connections in layer  $\ell$ ) yielding sparse neural networks with sparse weights  $m_{\ell}\odot w_{\ell}$  - the elementwise product of masks and weights. Sparsity is defined as the fraction of weights being removed:  $s = 1 - \frac{\sum m_{\ell}}{\sum d_{\ell}}\in [0,1]$ .
A pruning method usually consists of two operations: score and remove, where score takes as input weights of the network and outputs an important score for each weight:  $z_{\ell} = \text{score}(w_{\ell}) \in \mathbb{R}^{\ell}$ ; then remove takes as input the scores  $\mathbf{z} = (z_1, \dots, z_L)$  and the sparsity  $s$ , and outputs the masks  $m_{\ell}$  with
overall sparsity  $s$ . Pruning can be done in one-shot or iteratively. For one-shot pruning, we only generate the scores once, then prune the network up to sparsity  $s$ . For iterative pruning, we repeat the processes of the score, then prune from sparsity  $s^{(t - 1) / T}$  to  $s^{t / T}$  repeatedly  $T$  times.
Random. This method assigns each connection with a random score from a uniform distribution  $\mathcal{U}(0,1)$ . Random pruning empirically prunes each layer to target sparsity  $s$  [29].
SNIP. SNIP was introduced by Lee et al. [26] with the pruning objective of reducing connection sensitivity to the training loss. One passes a mini-batch of data through the network and computes the score  $\mathbf{z}$  for weight  $\mathbf{w}$  of SNIP as  $\mathbf{z} = |\mathbf{w} \odot \nabla_{\mathbf{w}}\mathcal{L}|$ .
Iterative SNIP. This is an iterative variant of SNIP [9] with the same important score. But, iterative SNIP gradually prunes the remaining weights with lowest scores from sparsity  $s^{\frac{t - 1}{T}}$  to sparsity  $s^{\frac{t}{T}}$  iteratively  $T$  times for  $t = 1,2,\ldots ,T$ .
SynFlow. SynFlow [42] is an iterative and data-agnostic PaI method. The pruning objective of SynFlow is to make the network remains connected until the most extreme possible sparsity. The weight scores are computed as follows. One first replaces all weights in the network by their absolute values. Then, an 1 input tensor is passed through the network, and one computes the sum of the logits as  $R = \mathbf{1}^{\top}(\prod_{\ell=1}^{L}|w_{\ell}|)\mathbf{1}$ . Finally, the score of weight w is computed as  $\mathbf{z} = |\mathbf{w} \odot \nabla_{\mathbf{w}}R|$ . SynFlow prunes the network iteratively  $T$  times.
PHEW. PHEW [36] is also an iterative and data-independent PaI method. It selects a set of input-output paths to be preserved. These paths are chosen through random walks, biased towards higher-weight magnitudes. The selection starts with a unit that is selected through round robin procedure. This process continues until the subnetwork achieves the predefined sparsity.
ERK. Erdős-Rényi (ER) first used by Mocanu et al. [30] to sparsify Multilayer Perceptron (MLP) networks using a random topology that allocates higher sparsity to larger layers. Evci et al. [14] extends ER to a convolutional version called Erdős-Rényi-Kernel (ERK) which scales the sparsity of the convolutional layer in proportion to the number of neurons/channels in a layer.
3.2 Metric Definition
Figure 2: An example of effective paths and effective nodes.
In a sparse network, it is intuitively clear that one should arrange the connections into a configuration neither too thin nor too spread-out to have good information propagation during training. For a better measurement, we propose using two metrics to evaluate the quality of subnetworks. Please refer to Appendix B for detailed discussions and Python code for calculating the metrics.
**Effective Path.** We define a path to be effective if it connects an input node to an output node without interruption (see Figure 2). Metrics based on paths are mentioned in [42, 19] as  $l_{1}$  and  $l_{2}$  path norms, respectively. In this paper, we only take into account the number of paths.
Effective Node/Channel. A node/channel is effective if at least one effective path goes through it (demonstrated as the right part in Figure 2). This concept is also considered in [36, 17]. For convolutional layers, we consider a kernel as a connection, and a channel as a node, and then convert the convolutional layer into a fully connected layer (see the left part in Figure 2).
3.3 Layerwise Shuffling Phenomenon
In this section, we investigate the intriguing phenomenon that layer-wise reshuffling the subnetwork found by PaI methods still produces competitive accuracy [17, 40]. Based on metrics, we provide a
Figure 3: Layerwise shuffling results on various sparse subnetworks of ResNet20 produced by SNIP, SynFlow, and PHEW on CIFAR-10.
new way to understand why reshuffling subnetworks work and when they fail. We first use three PaI methods, i.e., SNIP [26], SynFlow [42], and PHEW [36], to find the subnetworks. Then, we randomly shuffle the pruning mask  $m_{l}$ . All the subnetworks (both unmodified and shuffled) are trained in the same setting. Finally, we compute the ratio of the number of active paths and nodes after and before pruning and visualize the average scores in Figure 3.
In SNIP and SynFlow, the number of effective nodes drops significantly as sparsity increases (blue lines in second columns in Figure 3). After reshuffling, the connections are distributed uniformly in each layer leading to wider subnetworks while the number of effective paths decreases. In contrast, PHEW focuses on increasing the number of effective nodes by gradually adding new paths such that the network is as wide as possible. Consequently, reshuffling hurts the concrete network configuration and then reduces both the number of effective nodes and effective paths as sparsity is higher.
At sparsity levels below  $99\%$ , layerwise shuffled subnetworks demonstrate competitive performance or even better than unmodified counterparts, especially with SynFlow and SNIP. In addition to effective connection preservation as discussed in prior work [17], the representation capacity of shuffled sparse networks is enhanced with SynFlow and SNIP, attributed to the increase in the number of effective nodes [35] while maintaining considerable input-output paths. In more details, Figure 3 shows that these two metrics for SynFlow and SNIP shuffled versions closely resemble to unmodified subnetworks of PHEW with corresponding sparsities.
However, when the network becomes more sparse, the number of input-output paths decreases substantially (along with the reduction of effective parameters see Appendix D). Even though layerwisely shuffled networks become wider, the limited number of effective paths damages the information flows in the subnetworks. These explain why the accuracy of shuffled subnetworks is reduced significantly compared to the unmodified ones in intensive sparsities.
These observations indicate that increasing the number of active paths or nodes alone might not be sufficient in the design of PaI methods. We hypothesize that to have good subnetworks, the number of effective paths and effective nodes should be concurrently considered. If we balance these two metrics well, the performance after the training of subnetworks will be enhanced.
3.4 NAS Observations
Our research delves into the fundamental aspects of network architecture design by investigating the interplay between nodes and paths. Going beyond the realm of pruning literature, we conduct an analysis of node and path metrics in the NAS benchmark. It is noteworthy that networks in NAS are dense networks. In particular, we focus on NAS-Bench-Macro [41], a macro benchmark comprising 8 searching layers. Each layer offers three candidate blocks, resulting in a staggering 6,561 distinct networks with parameter counts ranging from 300k to 3M. To ensure a fair comparison
among candidates, we specifically consider networks with similar parameter counts within the 300k range. We compute metrics across four different parameter ranges and visualize them in Figure 4.
In the context of NAS, it's important to consider that other aspects beyond node-path balance can contribute to final classification accuracy. For instance, in the case of NAS-Bench-Macro, networks with similar node-path balance but varying classification accuracies can be affected by other architectural configurations, such as differences in pooling layers, kernel sizes, and expansion ratios in the MobileNet-v2 block. These architectural variances often result in different numbers of parameters, influencing the overall network performance. However, in the sparse neural network context, pruning methods focus on maintaining the same network structure while pruning connections within the network based on specific sparsities. Consequently, our experiments did not explore other architectural elements beyond node and path balance. In general,
Figure 4: The accuracy of network candidates in NAS-Bench-Macro benchmark along with the number of activated nodes and paths in different parameter ranges.
we can see that networks with larger numbers of nodes and paths tend to exhibit higher performance. Our findings unveil a strong correlation between the two metrics (node, path) and the final performance of network candidates, both playing pivotal roles in achieving superior performance. This highlights the critical significance of both nodes and paths in the initial stages of designing subnetworks that yield exceptional results.
4 Node-Path Balancing Principle
4.1 Node-Path Balancing Principle
From observations in Section 3.3 and NAS observations in Section 3.4, both effective paths and nodes have shown their critical roles in the performance of subnetworks. We now formally state the Node-Path Balancing Principle (NPB): The combination of both the numbers of effective nodes and effective paths is a proxy for the potential performance of subnetworks under pruning. A pruning at initialization method which produces pruned subnetworks with too many effective paths (resp. effective nodes) will have less than necessary the number of effective nodes (resp. effective paths) and consequentially has suboptimal performance. It is necessary to balance the number of effective nodes and effective paths for better performance of pruned subnetworks.
4.2 Proposed Method: NPB
Building upon the aforementioned principle, we elegantly transform the pruning problem into a multi-objective optimization problem. Specifically, our objective is to maximize both the number of effective nodes and effective paths concurrently given an architecture and desired sparsity ratio. We formulate this intriguing problem as follows:
Given an architecture  $A$  with parameter  $\mathbf{W} \in \mathbb{R}^N$  where  $N$  is the total number of parameters and sparsity ratio  $s$ . Denote  $f_p$  as the total number of input-output paths,  $f_n$  as the number of activated nodes, and consider the mask for parameter  $\mathbf{M} = \{0,1\}^N$  as variable to solve.
$$
\underset {\mathbf {M}} {\text {M a x i m i z e}} \quad \alpha f _ {n} + (1 - \alpha) f _ {p}
$$
$$
s. t \quad \| \mathbf {M} \| _ {1} \leq N (1 - s)
$$
where  $\alpha$  is a coefficient. Solving node-path balancing objective globally over the whole neural networks is a non-trivial problem. We sidestep this challenging issue by solving a sequence of easy problems to obtain good approximated solutions. In particular, we propose an approximation for solving this problem by doing layer by layer through convex optimization. The approximation problem is solved efficiently via the available convex optimization library.
To be simple, we consider a linear layer as an example, in which,  $\nu$  denotes the node,  $l$  is the layer index,  $\mathbf{m}^{(l)}\in \mathbb{R}^{h^{(l)}\times h^{(l + 1)}}$  is the mask for layer  $l$  in which  $h^{(l)}$  the number of neurons in layer  $l$ . We denote  $P(\nu_i^{(l)})$  as the number of path go to node  $\nu_{i}$  in layer  $l$  (e.g., at input layer  $P(\nu_i^{(0)}) = 1$ ). We formulate the pruning problem in layer  $l$  as an optimization problem in which we maximize the number of paths from input to layer  $l + 1f_p^{(l + 1)}$  and the total activated nodes  $f_{n}^{(l + 1)}$  in two layers  $l$  and  $l + 1$ . For brevity, we denote them as  $f_{p}$  and  $f_{n}$ . We set  $\mathbf{m}^{(l)}$  as:
$$
m _ {i j} ^ {(l)} = \left\{ \begin{array}{l l} 0 & \text {i f p r u n e d} \\ 1 & \text {i f n o t p r u n e d} \end{array} \right.
$$
The number of paths to  $\nu_{j}^{(l + 1)}$  in layer  $l + 1$  is sum of all paths to nodes in layer  $l$  connecting to  $\nu_{j}^{(l + 1)}$ .
$$
P (v _ {j} ^ {(l + 1)}) = \sum_ {i} ^ {h ^ {(l)}} m _ {i j} ^ {(l)} P (v _ {i} ^ {(l)})
$$