SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Then the total number of effective paths to layer  $l + 1$  is:
$$
f _ {p} (m) = \sum_ {j} ^ {h ^ {(l + 1)}} P \left(v _ {j} ^ {(l + 1)}\right) \tag {1}
$$
A node  $\nu_{i}$  in layer  $l$  is activated if and only if there are paths pass through it and edges connect from it to nodes in layer  $l + 1$ , which is formulated as below:
$$
P \left(v _ {i} ^ {(l)}\right) \sum_ {j} ^ {h ^ {(l + 1)}} m _ {i j} ^ {(l)} \geq 1 \Leftrightarrow \min  \left(P \left(v _ {i} ^ {(l)}\right) \sum_ {j} ^ {h ^ {(l + 1)}} m _ {i j} ^ {(l)}; 1\right) = 1
$$
And a node  $\nu_{j}$  in layer  $l + 1$  is effective when there exist attached nodes in layer  $l$  that connect to it,
$$
\sum_ {i} ^ {h ^ {(l)}} m _ {i j} ^ {(l)} P \left(v _ {i} ^ {(l)}\right) \geq 1 \Leftrightarrow \min  \left(\sum_ {i} ^ {h ^ {(l)}} m _ {i j} ^ {(l)} P \left(v _ {i} ^ {(l)}\right); 1\right) = 1
$$
The layer-wise objective for the node becomes:
$$
f _ {n} (m) = \sum_ {i} ^ {h ^ {(l)}} \min  \left(P \left(v _ {i} ^ {(l)}\right) \sum_ {j} ^ {h ^ {(l + 1)}} m _ {i j} ^ {(l)}; 1\right) + \sum_ {j} ^ {h ^ {(l + 1)}} \min  \left(\sum_ {i} ^ {h ^ {(l)}} m _ {i j} ^ {(l)} P \left(v _ {i} ^ {(l)}\right); 1\right) \tag {2}
$$
In convolution layers with a kernel of height  $h$  and width  $w$ , we let the variable  $m_{ij}$  have the value from 0 to  $hw$  representing this kernel. After solving  $\mathbf{m}$ , in each kernel, we assign  $|m_{ij}|$  parameters to entries whose initialized weights have the highest magnitude.
Along with two objectives, we also consider a regularization term which aims to encourage activating as many kernels as possible in each layer. Besides, since we optimize the node and path per layer, the solution will not be the global one. We can view the regularizer as an adjustment term, which moves the local solution around to figure out the better ones.
$$
R = \sum_ {i} \sum_ {j} \min  \left(m _ {i j} ^ {(l)} - 1; 0\right) \tag {3}
$$
From Equations 1, 2, and 3 the final objective becomes:
$$
\underset {\mathbf {m} ^ {(1)}} {\text {M a x i m i z e}} \quad \alpha f _ {n} + (1 - \alpha) f _ {p} + \beta R \tag {4}
$$
$$
s. t \quad \| \mathbf {m} ^ {(l)} \| _ {1} \leq N ^ {(l)} (1 - s ^ {(l)}) \tag {5}
$$
where  $\alpha$  is a coefficient to control the balance,  $\beta$  is a hyperparameter, and the constraint is the number of unpruned parameters that satisfies the sparsity. To ensure the same scale between elements in the main objective function, we normalize the number of paths, nodes, and the regularizer in each layer with their maximum possible values, respectively. Note that, optimizing nodes is much easier than paths so we select small values of alpha which can be prior knowledge. In particular, in Section 5, we fix  $\alpha = 0.01$  and  $\beta = 1$  to all settings. We optimize  $^1$  the network structure sequentially layer by layer from input to output in which the layer-wise sparsity level is found by ERK method [14, 29]. We describe our method in Algorithm 1 and the pseudo code for optimizer in Appendix C.
<sup>1</sup>We use the default mixed integer programming solver in CVXPY library
Algorithm 1 Node-Path Balancing Pruner
1: Inputs: Final sparsity  $s$ , weights  $\mathbf{w}_0 = \mathbf{1}$ , balancing coefficient  $\alpha$ , and hyperparameter  $\beta$   
2: Obtain layer-wise sparsity  $s^{(l)}$  by using ERK method  
3: Define unit input  $\mathbf{x} = \mathbf{1} \Rightarrow P^{(0)} = \mathbf{1}$   
4: for  $l = 0, \dots, L$  do  
5:  $\mathbf{m}^{(l)} \gets \text{optimize}(\mathbf{m}^{(l)}, s^{(l)}, P^{(l)})$  # solve Eq.4  
6: Set  $\mathbf{w}_0^{(l)} = \mathbf{m}^{(l)}$   
7: Extract layer  $l + 1$  output when put  $\mathbf{x}$  to the network with  $\mathbf{w}_0$  to obtain  $P^{(l + 1)}$   
8: end for  
9: Return: Mask  $\mathbf{M}$
We conduct experiments on three standard datasets: CIFAR-10, CIFAR-100, and Tiny-Imagenet. Following [42], we use ResNet-20 for CIFAR-10, VGG-19 for CIFAR-100, and a variant ResNet-18 with 18 layers for Tiny-Imagenet. We treat weights of all convolutional and linear layers as prunable, but we do not prune biases and weights in batch normalization layers. We run five seeds with experiments on CIFAR-10, CIFAR-100, and three seeds on Tiny-Imagenet. The average results are used for visualizations. Since the number of active paths is orders of magnitude larger than that of active nodes, and both numbers are of exponential scales, we take logarithm base 10 of the number of active paths, and logarithm base  $e$  (natural logarithm) of the number of active nodes to visualize the results. To ensure the fair comparison between NPB and other baselines, we fix  $\alpha = 0.01$  and  $\beta = 1$  in NPB for all settings. More details on our experimental setting are in Appendix A.
Figure 5: The number of effective paths (log scale), nodes (ln scale), and the corresponding accuracy of different PaI methods on three datasets in different sparsity levels. Best accuracy is in blue.
5.2 Comparison with PaI Methods
Figure 5 shows the experimental results for different PaI methods: Random, SNIP, Iter-SNIP, SynFlow, PHEW, and our NPB, with different sparsities on three settings: VGG-19 on CIFAR-100, ResNet-20 on CIFAR-10, and ResNet-18 on Tiny-Imagenet. Overall, our method NPB shows significant improvements on standard PaI methods, both data-dependent like Iter-SNIP, SNIP (improve up to
$14\%$  ) and data-independent like SynFlow (increase up to  $7 \%$  ) in all three settings. Notably, NPB achieves better results than state-of-the-art PaI method PHEW in almost all settings (11 out of 12 settings). Importantly, we set the same alpha and beta hyperparameters for NPB across all three settings, thereby eliminating the need for extensive hyperparameter search. This demonstrates the robustness and generalizability of our method, making it easily applicable to various settings with consistently high performance. Ablation studies on alpha and beta can be found in Appendix G.
We discuss the experimental results from the point of view of the Node-Path Balancing principle and network shape optimization. In Figure 5, SynFlow and Iter-SNIP produce sparse networks with large number of input-output paths, which results from iterative pruning procedures. These methods gradually remove nodes of low connection degree, which makes the subnetworks become narrower while maintaining the high number of paths. As a consequence of a very high number of input-output paths, the width of resulting subnetworks is significantly reduced, limiting the representation capacity of the subnetworks [35, 36], leading to suboptimality.
Random pruning is a PaI method which produces subnetworks with the very high number of active nodes. On average, it distributes parameters uniformly to all layers and kernels, creating subnetworks of large width yet a low number of active paths. At sparsities below  $90\%$ , with an adequate number of input-output paths, subnetworks of Random pruning still perform well and are even competitive with more sophisticated PaI methods. This phenomenon is exhibited clearly in experiments on the more complex dataset Tiny-Imagenet. However, when sparsity increases over  $90\%$ , Random pruning creates a huge number of ineffective paths and parameters. Consequently, the performance of subnetworks drops significantly. We again observe the necessity to balance the optimization of the number of nodes and paths.
In Figure 5, our method NPB and PHEW produce sparse networks which, in visualization, lie in specific areas which have higher paths than Random and greater nodes than SynFlow, or Iter-SNIP. We call these are balancing regions. Subnetworks in these regions have broader widths and considerably many paths, which provides good representation capacity and preserves the information flow of the network. These two factors together contribute to better training of pruned subnetworks leading to superior performance. This illustrates the effectiveness of our principle.
One drawback of one-shot pruning methods like SNIP is creating a large number of ineffective parameters, which decreases the capacity and performance of subnetworks. Although our method is also one-shot, it efficiently distributes the parameters by direct optimization. Consequently, NPB enjoys a better node-path balancing level and better performance (up to  $14\%$ ) compared to SNIP.
Results of all settings in Figure 5 provide evidence to support our Node-Path Balancing principle and the existence of a specific balancing region between nodes and paths at given sparsity levels. Particularly, we posit that when the number of active neurons and input-output paths of a sparse network fall within this balancing range, it will probably have a good performance after training.
5.3 Pruning Time and FLOPs Reduction
Table 1: Accuracy, pruning time (in seconds) and FLOPs of subnetworks for different pruning methods and compression ratios on Resnet18 - Tiny-ImageNet.
Accuracy (%)	Pruning time (seconds)	FLOPs (108)
Sparsity (%)	68.38	90.00	96.84	99.00	68.38	90.00	96.84	99.00	68.38	90.00	96.84	99.00
SNIP	56.99	53.43	48.77	36.02	5.14	4.95	5.55	5.64	11.35	5.77	3.04	1.55
Iter-SNIP	56.73	53.60	48.55	36.42	229.16	235.34	233.19	231.23	10.73	7.05	3.98	1.97
SynFlow	56.71	54.68	49.03	39.79	108.17	96.18	91.15	92.60	14.71	8.91	4.24	1.50
PHEW	58.09	55.93	50.81	40.54	5511.20	1342.03	471.23	324.78	14.29	8.35	3.92	1.50
NPB	58.39	56.82	51.37	41.05	380.52	375.65	384.32	387.89	14.37	5.21	1.74	0.59
Since NPB optimizes layer by layer, the complexity of NPB depends on network architecture (i.e., the number of layers and the size of each layer). With large layers, we sidestep the time-consuming issues by making a further step where we divide the layer into chunks with the same sparsity constraint. In particular, we split nodes in the next layer into equal parts while the input node is fixed. Then, optimizing chunks in a layer can be solved in parallel. Thank to the available convex optimization libraries we can find subnetworks more efficiently and quickly. We have computed the pruning time of our proposed method and compared it with other PaI methods in Table 1. Our pruning time is not
significantly slow compared to those iterative approaches (e.g., Iter-SNIP, SynFlow) while it is much faster (up to  $14.5\times$ ) than PHEW in lower sparsity levels.
Besides pruning time, we find that the FLOPs reduction of subnetwork after pruning is more important in the context of pruning before training. We have measured FLOPs of subnetworks produced by different methods in Table 1. The result indicates that our NPB can produce subnetworks with lower FLOPs than other baselines while NPB outperforms PaI methods. More details are in Appendix E.
In this paper, we present a novel framework for studying PaI methods that systematically employs the configuration of pruned subnetworks based on two different metrics: the number of effective paths and the number of effective nodes. Through our framework, we discover a new relationship between these metrics, called the Node-Path Balancing Principle, which provides guidance for understanding and optimizing PaI methods. Our framework offers unified explanations for the intriguing layerwise connection reshuffling phenomenon [40, 17] of subnetworks produced by PaI methods in normal pruning sparsity regime, as well as the failure of this phenomenon in extreme sparsity levels. Furthermore, we propose a novel pruning method based on optimization approach, namely NPB, that demonstrates the potential of balancing the numbers of effective paths and nodes to improve the performance of PaI methods. Extensive experiments conducted on various model architectures and datasets show that NPB outperforms the best baseline, PHEW, on almost settings while ours produces more efficient subnetworks with much fewer inference FLOPs and faster in pruning time than PHEW. Our new perspective on the configuration of subnetworks, in terms of effective nodes and effective paths, provides new insights into the working mechanism of PaI methods and opens new research directions on neural network pruning methods, as well as designs of sparse neural network.
Part of this work was done while Hoang Pham was visiting University of Warwick. Dung D. Le was funded by Vingroup Innovation Foundation (VINIF) under project code VINIF.2022.DA00087.
[1] M. Alizadeh, S. A. Tailor, L. M. Zintgraf, J. van Amersfoort, S. Farquhar, N. D. Lane, and Y. Gal. Prospect pruning: Finding trainable weights at initialization using meta-gradients. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=AIgn9uwfcD1.
[2] P. Bithika, B. Arindam, K. Sudeshna, M. Pabitra, and B. Biswajit. A study on the ramanujan graph property of winning lottery tickets. In International Conference on Machine Learning, volume 162, pages 17186-17201, 2022.
[3] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[4] T. Chen, J. Frankle, S. Chang, S. Liu, Y. Zhang, Z. Wang, and M. Carbin. The lottery ticket hypothesis for pre-trained bert networks. Advances in neural information processing systems, 33:15834-15846, 2020.
[5] T. Chen, J. Frankle, S. Chang, S. Liu, Y. Zhang, M. Carbin, and Z. Wang. The lottery tickets hypothesis for supervised and self-supervised pre-training in computer vision models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16306-16316, 2021.
[6] T. Chen, B. Ji, T. DING, B. Fang, G. Wang, Z. Zhu, L. Liang, Y. Shi, S. Yi, and X. Tu. Only train once: A one-shot neural network training and pruning framework. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL https://openreview.net/forum?id=p5rMPjrcCZq.
[42] H. Tanaka, D. Kunin, D. L. Yamins, and S. Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in Neural Information Processing Systems, 33:6377-6389, 2020.
[43] R. Taylor, M. Kardas, G. Cucurull, T. Scialom, A. Hartshorn, E. Saravia, A. Poulton, V. Kerkez, and R. Stojnic. Galactica: A large language model for science. arXiv preprint arXiv:2211.09085, 2022.
[44] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
[45] C. Wang, G. Zhang, and R. Grosse. Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=SkgsACVKPH.
[46] H. Wang, C. Qin, Y. Bai, Y. Zhang, and Y. Fu. Recent advances on neural network pruning at initialization. In L. D. Raedt, editor, Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022, pages 5638-5645. ijcai.org, 2022. doi: 10.24963/ijcai.2022/786. URL https://doi.org/10.24963/ ijcai.2022/786.
[47] G. Yuan, X. Ma, W. Niu, Z. Li, Z. Kong, N. Liu, Y. Gong, Z. Zhan, C. He, Q. Jin, et al. Mest: Accurate and fast memory-economic sparse training framework on the edge. Advances in Neural Information Processing Systems, 34:20838-20850, 2021.
[48] R. Zhao, Y. Hu, J. Dotzel, C. De Sa, and Z. Zhang. Improving neural network quantization without retraining using outlier channel splitting. In International conference on machine learning, pages 7543-7552. PMLR, 2019.
[49] M. Zhu and S. Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.01878, 2017.
In this section, along with pruning at initialization (PaI) methods in the main text, we provide experimental results with GraSP [45]. In particular, GraSP is another gradient-based pruning PaI that aims to preserve the gradient flow of sparse networks obtained by pruning. The score  $z$  of weight  $w$  in GraSP is computed as  $\mathbf{z} = -\mathbf{w} \odot (\mathbf{H}\nabla_{\mathbf{w}}\mathcal{L})$ , where  $\mathbf{H}$  is the Hessian of the training loss after passing a mini-batch of data through the network.
We describe our experiment settings on architectures and datasets. We use Pytorch library and conduct experiments on a single GTX 3090Ti or A100 (depend on their available). We adapt from Tanaka et al. [42] source code for SNIP, GraSP, SynFlow, and Random, and change the prune epochs to 100 for Iterative SNIP instead of 1 in SNIP. And we use the official code of Patil and Dovrolis [36] for PHEW. For NPB method in the main experiments, we set  $\alpha = 0.01$  and  $\beta = 1$  for all settings.
Datasets. Our main experiments are conducted with CIFAR-10, CIFAR-100, and Tiny-Imagenet datasets, where:
- CIFAR-10 is augmented by normalizing per-channel, randomly flipping horizontally.
- CIFAR-100 is augmented by normalizing per-channel, randomly flipping horizontally.
- Tiny-ImageNet is augmented by normalizing per channel, cropping to 64x64, and randomly flipping horizontally.
Architectures. We use three different networks:
- VGG-19 is a CIFAR-100 network used in SynFlow [42]. We choose a batch-normalization version.
- ResNet-20 is a 20-layer CIFAR-10 version of ResNet created by [22]. This version has added batch normalization layers before each activation function.
- ResNet-18 is a ImageNet version with 18 layers adapted from SynFlow [42]. The first convolution has kernel size  $3 \times 3$  (instead of  $7 \times 7$ ) and max-pooling layer that follows has been removed.
We treat all of the weights from convolutional and linear layers of these networks as prunable parameters, but we do not prune the biases nor the weights in the batch normalization layers. The weights in convolutional and linear layers are initialized with Kaiming normal, while biases are initialized to be zero. We run five seeds with CIFAR experiments and three seeds with experiments on Tiny-Imagenet.
Training details With Iterative pruning methods SNIP and SynFlow, we use 100 pruning epochs. With methods using training data like SNIP, GraSP, and Iterative-SNIP, we randomly select 10 samples for each class, particularly, 100 data points for CIFAR-10, 1000 data points for CIFAR-100, 2000 data samples for Tiny-ImageNet. Other hyperparameters are chosen as follow:
Table 2: Summary of the architectures, datasets, and hyperparameters used in experiments.
Network	Dataset	Epochs	Batch	Optimizer	Momentum	LR	LR Drop, Epoch	Weight Decay
VGG-19	CIFAR-100	160	128	SGD	0.9	0.1	10x, [60,120]	0.0001
ResNet-20	CIFAR-10	160	128	SGD	0.9	0.1	10x, [60,120]	0.0001
ResNet-18	Tiny-ImageNet	100	128	SGD	0.9	0.01	10x, [30,60,80]	0.0001
$^{2}$ https://pytorch.org
<sup>3</sup>https://github.com/ganguli-lab/Synaptic-Flow
4 https://github.com/ShreyasMalakarjunPatil/PHEW
B Effective metrics calculation
Effective path. To exactly compute the number of effective paths, we remove the batch normalization layers, we initialize all the remaining parameters to 1. Then, we put the input vector one to the network, and the number of effective paths is the sum of logits on the output layer  $R = \mathbf{1}^{\top}(\prod_{\ell=1}^{L}|w_{\ell}|)\mathbf{1}$ .
More precisely, we face problems with pooling layers in convolutional neural networks. With max pooling layer, we simply do not modify the output of this layer. At that time, the result is the maximum number of paths in subnetworks. With average pooling layer, since all inputs of this layer contribute to the output, we change the average operator to the sum operator to exactly compute the number of effective paths. We all use ReLU activation functions in computing this metric since this function does not affect the results of calculations.
Effective parameter. We follow [17] when identifying which is effective parameter. Similar to computing effective paths, we make further steps. After having the sum of logits, we compute the gradients of this sum with respect to weights  $\nabla_{\mathbf{w}}R$ . Then, if an unpruned weight has a non-zero gradient, it is effective and vice versa. Effective parameters are dense edges that connect two effective nodes as visualized in Figure 2.
Effective node/channel. With fully connected layers, if all connections to one node or out of one node are pruned, this node is pruned node. If there exist connections to a node but all of these connections are ineffective, then this node becomes ineffective In convolutional layers, instead of nodes, we have channels. We consider a kernel as a connection, a channel as a node, and then convert the convolutional layer into a fully connected layer. The connection is pruned if and only if all parameters in the corresponding kernel are removed. Finally, identifying the effective nodes/channels is similar to the way in fully connected layers.
def metric Calculation(model, mask):
    def model: network architecture
        mask: mask for subnetwork
        n_eff_paths = 0
        n_eff_nodes = 0
        n_eff_parameters = 0
    # Initialize network with pruned weight = 0 and kept weight = 1
    for name, param in model named_parameters():
        param.copy_(mask[name])
    x = torch.ones((1, c, h, w)) # c: channel - h: height - w: width
    y = model(x)
    sum_logits = y.sum()
    n_eff Paths = sum_logits.item()
    sum_logits.backup()
    with torch.no_grad():
        for name, param in model named_parameters():
            eff.Param = torch.where(param.grad.data != 0, 1, 0)
            n_eff.params += torch.sum(affparam)
            eff_in_node = torch.where(torch.sum(affparam, d=0) > 0, 1, 0)
            n_eff幾個 += torch.sum(aff_in_node)
        # with output layer
            eff_out_node = torch.where(y>0, 1, 0)
            n_eff_nodes += torch.sum(aff_out_node)
    return n_eff_paths, n_eff_nodes, n_eff_parameters
Listing 1: Metric calculation example in fully connected neural networks
C Layer-wise Mask Optimization
We provide the pseudo code solving the mask optimization problem based on the cvxpy library [11]. We use the default solver provided in this tool.
import math
import cvxpy as cp
import numpy as np
import torch
def optimize_layerwise(mu, inp, sparsity, alpha=0.1, beta=0.1, max-param_per_kernel=None):
    ''''
    mask: mask of this layer
    inp: input of this layer
    sparsity: the sparsity of this layer
    alpha: is the balancing coefficient
    beta: is the regularizer coefficient
    max-param_per_kernel: max param in a kernel
    ''''
if len(mu.shape) == 4: # Convolution layer
    C_out, C_in, K, W = mask.shape
    max-param_per_kernel = K * W
    P_in = torch.sum(inp, dim=(1,2)).numpy()
else: # Linear layer
    C_out, C_in = mask.shape
    max-param_per_kernel = 1
    P_in = inp.numpy()
# Params in layer
nParams = int(math.ceil((1 - sparsity)*mask(numel)))
# This has to be integer
# Define mask variable
M = cp.Variable((C_in, C_out), integer=True)
sum_in = cp.sum(M, axis=1) * P_in # P_in * \sum_{j}^{}{C_out} M_{\{ij\}} => shape: C_in
sum_out = cp.sum(M*P_in, axis=0) # \sum_{i}^{}{C_in} M_{\{ij\}} * P_in[i] => shape: C_out
# If eff_node_in is small which means there is a large number of input effective node
sum_eff_node_in = C_in - cp-sum(cp_pos(1 - sum_in))
sum_eff_node_out = C_out - cp-sum(cp_pos(1 - sum_out))
# Optimize nodes
max_nodes = C_in + C_out
A = (sum_eff_node_in + sum_eff_node_out) / max_nodes # Scale to 1
# Optimize paths
max_path = compute_max_path(P_in, n.params, C_out,
max-param_per_kernel)
B = (cp-sum(P_in @ M)) / max_path
# Regulaziation
Reg = (n.params - cp-sum(cp_pos(1-M))) / n.params # encourage number of edges
# Constraint the total activated params
constraint = [cp-sum(M) <= n.params, M <= max-param_per_kernel, M >= 0]
# Objective function
obj  $=$  cp.Maximize(alpha \*A  $^+$  (1-alpha)\*B  $^+$  beta\*Reg) # Init problem prob  $=$  cp.Problem(obj, constraint) prob.solve() # Solving return M.value
Listing 2: Layer-wise mask optimization
53   
54   
55   
56   
57   
58   
59
D Layerwise Shuffling Experiments