SlowGuess/NeurIPS_2023_OCR huggingface dataset:
With each setting, at each sparsity ratio, we seek subnetworks with 5 different seeds, and with each seed, we randomly shuffle the subnetwork two times. In addition to effective path ratios and effective node ratios, we compute the number of effective parameters after pruning and the actual remaining ones, then calculate the ratio between these two values.
Similar to [17] results, the performance and the number of effective parameters of high-density subnetworks after permuting the connections are similar to or even higher (in SNIP) than the unmodified ones. However, when the sparsity level becomes more intensive, the configuration of subnetworks is more concrete. Randomly rearranging connections within layers destroys this strict structure by detaching important edges, which drastically reduces the number of effective paths. The shuffled subnetworks lack input-output paths to transfer information during training, leading to a drop in performance compared with unmodified ones.
Figure 6: Layerwise shuffling results on various sparse subnetworks of ResNet20 produced SNIP, SynFlow, and PHEW at initialization on CIFAR-10.
E Pruning Time and FLOPs Reductions
Since NPB optimizes layer by layer, the complexity of NPB depends on network architecture (i.e., the number of layers and the size of each layer). With large layers, we sidestep the time-consuming issues by making a further step where we divide the nodes in the next layer into chunks with the same sparsity constraint, and optimizing chunks in a layer can be solved in parallel. Particularly, we consider layer  $l$  with mask  $\mathbf{m}^{(l)} \in \mathbb{R}^{h^{(l)} \times h^{(l + 1)}}$  in which  $h^{(l)}$  and  $h^{(l + 1)}$  are the number of nodes in layer  $l$  and  $l + 1$ . We divide  $h^{(l + 1)}$  nodes into  $K$  equal chunks. Instead of directly solving for  $\mathbf{m}^{(l)}$ , we solve  $K$  problems  $[\mathbf{m}_1^{(l)}, \mathbf{m}_2^{(l)}, \dots, \mathbf{m}_K^{(l)}]$  where  $\mathbf{m}_k^{(l)} \in \mathbb{R}^{h^{(l)} \times h_k^{(l + 1)}}$ . Thank for available convex optimization libraries we can find subnetworks more efficiently and quickly. We have computed the pruning time of our proposed method and compared it with other PaI methods in Tables 3, 4, and 5.
Besides pruning time, we find that the FLOPs reduction of subnetwork after pruning is more important in the context of pruning before training. We have measured FLOPs of subnetworks produced by different methods in Tables 3, 4, and 5. The result indicates that our NPB can produce subnetworks with lower FLOPs than other baselines while NPB outperforms PaI methods.
Table 3: Accuracy, pruning time (in seconds) and FLOPs of subnetworks for different pruning methods and compression ratios on Resnet18 - Tiny-ImageNet.
Accuracy (%)	Pruning time (seconds)	FLOPs (108)
Sparsity (%)	68.38	90.00	96.84	99.00	68.38	90.00	96.84	99.00	68.38	90.00	96.84	99.00
SNIP	56.99	53.43	48.77	36.02	5.14	4.95	5.55	5.64	11.35	5.77	3.04	1.55
Iter-SNIP	56.73	53.60	48.55	36.42	229.16	235.34	233.19	231.23	10.73	7.05	3.98	1.97
SynFlow	56.71	54.68	49.03	39.79	108.17	96.18	91.15	92.60	14.71	8.91	4.24	1.50
PHEW	58.09	55.93	50.81	40.54	5511.20	1342.03	471.23	324.78	14.29	8.35	3.92	1.50
NPB	58.39	56.82	51.37	41.05	380.52	375.65	384.32	387.89	14.37	5.21	1.74	0.59
Table 4: Accuracy, pruning time (in seconds) and FLOPs of subnetworks for different pruning methods and compression ratios on VGG19 - CIFAR-100.
Accuracy (%)	Pruning time (seconds)	FLOPs (107)
Sparsity (%)	68.38	90.00	96.84	99.00	68.38	90.00	96.84	99.00	68.38	90.00	96.84	99.00
SNIP	71.31	70.65	67.82	61.77	5.15	4.96	5.12	4.55	17.952	7.806	3.686	1.816
Iter-SNIP	72.84	70.86	67.72	63.13	115.91	115.52	116.83	117.60	18.465	9.479	4.951	2.529
SynFlow	71.63	69.18	66.98	62.11	96.55	100.33	101.90	104.67	22.998	12.702	6.306	2.605
PHEW	73.18	70.70	68.18	64.43	6928.59	1699.80	605.65	417.25	22.108	11.746	5.611	2.340
NPB	74.05	71.76	68.87	64.82	430.52	438.20	412.16	425.33	22.035	8.773	2.874	1.046
Table 5: Accuracy, pruning time (in seconds) and FLOPs of subnetworks for different pruning methods and compression ratios on Resnet20 - CIFAR-10.
Accuracy (%)	Pruning time (seconds)	FLOPs (106)
Sparsity (%)	68.38	90.00	96.84	99.00	68.38	90.00	96.84	99.00	68.38	90.00	96.84	99.00
SNIP	87.88	84.02	76.72	62.03	1.42	1.40	1.87	1.68	17,952	8.323	3.470	1.709
Iter-SNIP	88.17	84.22	77.05	64.95	58.21	54.60	52.02	57.61	18,465	9.698	4.510	2.022
SynFlow	88.64	84.94	78.22	66.05	57.46	56.24	53.43	54.13	22,998	11.549	4.263	1.633
PHEW	90.38	87.41	81.05	70.44	78.31	18.09	4.78	2.58	22,108	10.690	4.110	1.640
NPB	90.69	87.61	80.55	70.70	20.10	23.91	21.51	21.13	22,035	7.642	2.645	1.122
F Additional Results with PaI comparison
Figure 7: Accuracy of subnetworks found by different PaI methods and LTH at various sparsity levels on two settings ResNet20 on CIFAR-10 and ResNet18 on TinyImagenet.
G Additional Results with Ablation Studies
In this section, we conduct experiments with different balancing coefficients (alpha) and regularization coefficients (beta) in setting ResNet18 on Tiny-Imagenet and visualize results in Figure 8. We run NPB with alpha in  $\{0.01, 0.05, 0.1, 0.5, 0.75, 0.9\}$  and beta in  $\{0.1, 0.5, 0.75, 1.0, 1.5, 2.0\}$ . In the first row, we visualize the performance of subnetworks generated by different alpha beta hyperparameters. In addition, to illustrate the correlation between subnetworks of NPB and PaI methods in terms of the number effective nodes and paths, we visualize alpha and beta subnetwork variants of NPB and PaI methods in the second row of Figure 8. Although NPB's variants have difference performance, they still outperform baselines. When visualize NPB variants with different PaI methods (SynFlow, Random, and PHEW) together, we can see that NPB variants concentrate around a specific region near PHEW which we hypothesize as a balancing area. Besides, networks in this region show a better results compared with SynFlow or Random, and have competitive performance with each others. This strengthens our Node-Path Balancing Principle.
Figure 8: Ablation study on alpha and beta hyperparameter with setting ResNet18 on Tiny-Imagenet
NPB depends on the layer-wise sparsity ratios, disparate ratios guide the pruner to a diverse set of solutions. Then, it is essential to investigate the effect of sparsity initializers on NBP's performance. We follow Liu et al. [29] to use ERK [14], Uniform, Magnitude ratio, SNIP ratio found by applying SNIP [26], and GraSP ratio by using GraSP [45] as five methods to produces layer-wise sparsity levels. Figure 9 illustrates results with these sparsity initializers for ResNet18 on Tiny-Imagenet with different sparsity levels. With the same value of  $\alpha$  and  $\beta$ , we observe that different layer-wise sparsifiers produce different network structures and performances as well. This difference raises a question for further work how we can specify ratios that help obtaining promising solutions.
Figure 9: Ablation study on different layerwise sparsity initializer with setting ResNet18 on Tiny-Imagenet
H NAS-Bench-Macro Experiments
NAS-Bench-Macro [41] is a macro benchmark which is used in Neural Architecture Search. It includes 8 searching layers, each having three candidate blocks which construct 6,561 different networks with the parameter varying from  $300\mathrm{k}$  to 3M. The network structure is presented in Table 6. The structure is inspired from MobileNetV2 search space. In particular, candidate blocks are (i) Identity connection, (ii) MB3_K3 which is MobileNetV2 block with kernel size 3 and expansion ratio 3, and (iii) MB6_K5 which is MobileNetV2 block with kernel size 5 and expansion ratio 6. To make a fair comparison between candidates, we only compare networks with a similar number of parameters (i.e., in a range of  $300\mathrm{k}$  parameters). Note that, each candidate here is a dense network. We compute metrics in four different parameter ranges and visualize them in Figure 10.
Table 6: Macro structure of search space on NAS-Bench-Macro.
n	input	block	channel	stride
1	32 × 32 × 3	3 × 3 conv	32	1
2	32 × 32 × 32	Choice Block	64	2
3	16 × 16 × 64	Choice Block	128	2
3	8 × 8 × 128	Choice Block	256	2
1	4 × 4 × 256	1 × 1 conv	1280	1
1	4 × 4 × 1280	global avgpool	-	-
1	1280	FC	10	-
Figure 10: The accuracy of network candidates in NAS-Bench-Macro benchmark along with the number of activated nodes and paths in different parameter ranges.