SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 2: Results on CIFAR-10 of PAGS (with different PaI methods), in comparison to vanilla PaI methods and LTH (as empirical performance "ceiling"). Baseline refers to the PaI-found sparse mask (at initialization) without modification by PAGS.  $\checkmark$  means no pre-training needed (i.e., ideal PaI).  $X$ ,  $XX$ , and  $XXX$ , represent low, high, and very high pre-training costs, respectively.
Method	Baseline acc. %	with PAGS (Best acc %)	(Avg. acc %)	No Pretraining needed
ResNet18
SNIP	89.54	90.17	89.80 ± 0.14	✓
GraSP	91.39	92.01	91.68 ± 0.16	✓
ERK	88.92	90.42	91.05± 0.17	✓
Random	85.43	86.03	85.72± 0.16	✓
LTH	91.22	—	—	XXX
ResNet34
SNIP	91.30	91.80	91.58± 0.13	✓
GraSP	91.27	91.85	91.48± 0.16	✓
ERK	91.18	92.29	91.98± 0.13	✓
Random	88.23	88.73	88.56±0.10	✓
LTH	92.76	—	—	XXX
3.2 Pruning Early as Graph Sampling (PEGs)
To further improve the classification accuracy at reduced compute, we extend the zero-shot PAGS, to present Pruning Early as Graph Sampling (PEGS), which incurs only a small amount of pre-training cost. In PEGS, we "pre-train" the dense network only for a small number of iterations (by default, for 500 iterations in all our experiments) and then apply PAGS to this lightly trained network. This draws similar inspiration as LTH rewinding [5] or early-bird (EB) ticket [23] in literature.
Compared to the PaI case, we note that it is not as straightforward to directly "sample" masks using LTH/EB or any other PaT method, since generating a different mask from those methods would require re-taking an expensive training process: otherwise, the pruning is deterministic on trained weights and there is no way to inject randomness. Hence, we do not treat PEGS as a "metaframework" on top of PaT methods. Instead, we discover that light enough pre-training, followed by cheap PaI-based sampling, can yield sparse masks with comparable quality to the top-performing PaT masks such as LTH, at a significantly cheaper cost. For example, compared to LTH with  $M$  rounds of iterative training,  $N$  epochs per training round, and  $I$  iterations each epoch ( $I = 500$  in CIFAR-10): PEGS in the same setting would only cost 500 iterations of pre-training, which is roughly  $\mathbf{MN} \times$  more compute-efficient and faster. For example, in CIFAR-10 experiments we have  $M = 23$  and  $N = 250$ , hence PEGS would yield  $> 5000$  times saving over LTH.
We leverage four PaI generators to facilitate our experiments (both PAGS and PEGS):  $\bullet$  Random [10] uniformly prunes every layer with the same pruning ratio assigned globally. Each parameter is randomly assigned a score based on the normal distribution.  $\bullet$  ERK [3, 14] initializes sparse networks with a Erdős-Rényi graph where small layers are usually allocated more budget, while bigger layers are assigned fewer parameter budget. Random pruning is then performed following those layer-wise ratios  $\bullet$  SNIP [9] for any layer  $l$  it issues scores  $s_l = |g_l \odot w_l|$  where  $g_l$  and  $w_l$  are gradients and weights respectively. The weights with the lowest scores after one iteration are pruned before training.  $\bullet$  GraSP [20] removes weights that impeded gradient flows by computing the Hessian-gradient product  $h_l$  and issue scores  $s_l = -w \odot h_l$ , for a layer  $l$ .
We also compare with two PaT methods: • Lottery Ticket Hypothesis (LTH) [4] iteratively prunes the lowest 20% of weights and rewind the remaining weights to some values in the past. To achieve the desired 99% sparsity, LTH would necessitate 23 rounds of full training and pruning. • Early Bird (EB) [23] utilizes one-shot “early pruning”, with a “mask-similarity” signal to automatically terminate pre-training as a cost-saving mechanism (typically happening around the first 15%-20% epochs). The original EB was implemented for structured pruning; for a fair comparison, the results shown in Table 3 are our re-implementation of EB onto unstructured pruning.
Table 3: Results on CIFAR-10 of PEGS (with light pre-training followed by different PaI methods), in comparison to vanilla PaI methods, LTH and EB (the later two come with much heavier pre-training costs). Baseline refers to the PaI-found sparse mask (after light pre-training) without modification by PEGS.  $X$ ,  $XX$ , and  $XXX$ , represent low, high, and very high pre-training costs, respectively.
Method	Baseline acc. %	with PEGS	No pre-training needed
(Best acc %)	(Avg. acc %)
ResNet18
SNIP	91.05	91.53	91.22± 0.14	X
GraSP	89.97	91.50	91.25± 0.23	X
ERK	89.87	91.31	91.05± 0.17	X
Random	85.37	86.03	90.13± 0.19	X
EB	90.10	—	—	XX
LTH	91.22	—	—	XXX
ResNet34
SNIP	92.38	92.75	92.51± 0.10	X
GraSP	92.84	92.90	92.70± 0.40	X
ERK	91.10	92.01	91.49± 0.23	X
Random	87.86	88.91	88.41±0.23	X
EB	92.00	—	—	XX
LTH	92.76	—	—	XXX
Table 4: Results on CIFAR-100 on ResNet18 using PAGS/PEGS in comparison to vanilla PaI methods, LTH and EB. Baseline refers to the PaI-found sparse mask.  $X$ ,  $XX$ , and  $XXX$ , represent low, high, and very high pre-training costs, respectively. @100 and @500 refer to different "pre-training" iterations using PEGS. @0 means we start from random initialization using PAGS.
Method	Baseline acc. %	with PAGS/PEGS (Best acc %) (Avg. acc %)	No pre-training needed
ResNet18@0
SNIP	64.60	65.39	64.89± 0.26	✓
GraSP	65.25	66.05	65.54± 0.21	✓
ERK	64.54	64.84	64.69 ± 0.12	✓
ResNet18@100
SNIP	63.22	64.92	64.34± 0.34	X
GraSP	63.27	65.17	64.35± 0.33	X
ERK	64.06	64.82	64.41± 0.22	X
ResNet18@500
SNIP	62.60	64.27	63.46± 0.30	X
GraSP	61.34	63.95	62.91± 0.59	X
ERK	64.06	65.05	64.56± 0.27	X
EB	62.45	—	—	XX
LTH	65.50	—	—	XXX
Unless otherwise stated, we use a high target sparsity of  $99\%$  in all our experiments. We demonstrate results on CIFAR-10/ CIFAR-100 in the main text. We use two representative models, Resnet18 and ResNet34, as the main backbones in this section<sup>1</sup>. Additional training details and results on Tiny-ImageNet are deferred to the Supplementary due to the space limit.
4.2 Comparison of PAGS and PEGS with Existing PaI methods
Tables 2 and 3 detail our pruning results for CIFAR-10 using PAGS and PEGS, while Tables 4 and 5 do so for CIFAR-100. As PAGS/PEGS operate on a sampling basis, we list both the peak performance of the best mask (Best acc) and the mean performance across all  $n$  sampled masks (Avg. acc). The latter is supplemented by the standard deviation. From the presented results, we deduce the following: ① In CIFAR-10/100 trials, our methods consistently surpass the baseline PaI generators in PAGS. A standout instance is the  $2.12\%$  advantage over ERK for ResNet18 using PaI, as seen in Table 2. ② With light pre-training, our methods still hold an edge against reference generators. For instance, Tables 4 and 5 indicate about  $1\%$  enhancement over the top-performing generator at both the 100 and
${}^{1}$  VGG16 is not included since some PaI methods and EB fail on it at 99% high sparsity ratio.
Table 5: Results on CIFAR-100 on ResNet34 using PAGS/PEGS in comparison to vanilla PaI methods, LTH and EB. Baseline refers to the PaI-found sparse mask.  $X$ ,  $XX$ , and  $XXX$ , represent low, high, and very high pre-training costs, respectively. @100 and @500 refer to different "pre-training" iterations using PEGS. @0 means we start from random initialization using PAGS.
Method	Baseline acc. %	with PAGS/PEGS (Best acc %) (Avg. acc %)	No pre-training needed
ResNet34@0
SNIP	69.48	70.73	69.83±0.27	✓
GraSP	67.88	70.59	69.64±0.74	✓
ERK	68.64	69.90	69.77±0.11	✓
ResNet34@100
SNIP	68.04	69.41	68.64±0.33	✗
GraSP	62.47	66.61	64.43±1.03	✗
ERK	68.91	69.92	69.50±0.16	✗
ResNet34@500
SNIP	67.41	69.23	68.53±0.36	✗
GraSP	67.18	68.95	68.03±0.41	✗
ERK	68.99	69.92	69.45±0.22	✗
EB	65.22	—	—	XX
LTH	68.05	—	—	XXX
500 iteration marks. ③ Interestingly, optimal outcomes typically manifest during the initialization phase, implying that light pre-training doesn't instantaneously refine the weight distribution.
4.3 Comparison with (Much More Expensive) LTH and the Early Bird Ticket
We further compare PAGS/PEGS with the two PaT methods, LTH and EB, and note that the latter two are significantly costlier. Overall, for PaI setting (table 2), we intend to include LTH as the "performance ceiling" and show PAGS can get close to or even outperform it often times. For example, Table 2 shows that ResNet18 utilizing GraSP marginally outperforms LTH, without any pre-training.
For PaT setting (table 3, table 4 and table 5), we show that with a small fraction of pre-training costs, our PEGS could solidly outperform both LTH and EB, using either SNIP or GRASP mask generators. Interestingly, in ResNet18, even the random ERK mask generator can be turned into LTG-level performance with the aid of PEGS. Our findings suggest that maximizing the full-spectrum's  $\ell_2$ -moving distance can significantly improve performance by optimizing both the graph's structure and weights. This approach allows us to achieve performance levels comparable to LTH but at a significantly lower cost. By leveraging the insights the Ramanujan perspective provides, we can achieve notable performance improvements across the board of pruning, while incurring minimal computational overheads.
Recent years have seen the rise of graph theory in analyzing and understanding sparse subnetworks. However, we still lack a crucial understanding of the role parameters play in graph connectivity. To fill this gap, in this paper, we study the weighted spectral gap of Ramanujan structures in sparse neural networks and investigate its correlation with the final performance. By examining the evolution of sparse structures under DST, we identify a strong correlation between Ramanujan bound, weighted spectral gap, and performance. Leveraging these observations, we proposed a new "full-spectrum coordinate" which comprehensively characterizes the complex interplay between various graph characteristics that further leads to actionable pruning methods both at initialization and after light pre-training. This unified perceptive is expected to invoke more future exploration into the complex interplay between topology and weights, not just for sparse NNs but for generic DNNs as well.
Z. Wang is in part supported by NSF Scale-MoDL (award number: 2133861) and the NSF AI Institute for Foundations of Machine Learning (IFML).
[19] D. T. Vooturi, G. Varma, and K. Kothapalli. Ramanujan bipartite graph products for efficient block sparse neural networks. ArXiv, abs/2006.13486, 2020.
[20] C. Wang, C. Wang, G. Zhang, and R. B. Grosse. Picking winning tickets before training by preserving gradient flow. ArXiv, abs/2002.07376, 2020.
[21] H. Wang, C. Qin, Y. Bai, Y. Zhang, and Y. Fu. Recent advances on neural network pruning at initialization. In International Joint Conference on Artificial Intelligence, 2021.
[22] L. Yin, V. Menkovski, M. Fang, T. Huang, Y. Pei, and M. Pechenizkiy. Superposing many tickets into one: A performance booster for sparse neural network training. In Uncertainty in Artificial Intelligence, pages 2267-2277. PMLR, 2022.
[23] H. You, C. Li, P. Xu, Y. Fu, Y. Wang, X. Chen, R. G. Baraniuk, Z. Wang, and Y. Lin. Drawing early-bird tickets: Towards more efficient training of deep networks. arXiv preprint arXiv:1909.11957, 2019.
A DNN as bipartite graphs
A bipartite graph is a graph consisting of two distinct sets of vertices  $L$  and  $R$  that are connected with  $E$  edges as  $G(L \cup R, E)$ . Let  $v$  denote the total number of vertices in both sets, the more common way to represent  $G$  is as a binary adjacency matrix  $A \in \mathbb{R}^{v \times v}$ .
A DNN is a series of bipartite graphs, one for each layer, to represent all of its compute graphs. For a convolutional layer with parameter tensor  $\pmb{\theta} \in \mathbb{R}^{C_{in} \times C_{out} \times k_{in} \times k_{out}}$ , we unfold the dimensions so that  $L = C_{in} * k_{in} * k_{out}$  and  $R = C_{out}$ . For a linear layer with parameter tensor  $\pmb{\theta} \in \mathbb{R}^{C_{in} \times C_{out}}$ , we can directly adapt its parameters where  $L = C_{in}$  and  $R = C_{out}$ .
An interesting and relevant property of a bipartite graph, is that when each node has the same number of  $d$  out-edges, its adjacency matrix  $\mathbf{A}$  has eigenvalues  $\mu(\mathbf{A})$  such that  $\mu_0 \geq \ldots \geq \mu_{v-1}$ , where  $\mu_0$  and  $|\mu_{v-1}|$  are equaled to  $d$ . We define  $\hat{\mu}(\mathbf{A}) = \max_{|\mu_i| \neq d} |\mu_i|$  as the largest nontrivial eigenvalue. In the context of unstructured pruning, we often find  $G$  to be **irregular**, in which case Hoang et al. [7] showed that  $d_{avg} \leq |\mu_0| \leq d_{max}$ .
B Supplementary results for PAGS and PEGS
B.1 Additional Experiments on CIFAR-10
We expanded our experiments on the CIFAR-10 dataset by utilizing weights pre-trained for 100 iterations with a batch size of 128 per iteration. The CIFAR-10 dataset consists of 50,000 training images and 10,000 testing images, divided into 10 different classes. The results of these experiments are summarized in Table 6.
We observed performance improvement relative to baseline. However, compared to other modes of pre-training for CIFAR-10, certain PaI generators exhibited higher-than-expected standard deviation and lower average performance, indicating some instability in generating sparse structures. Specifically, we observed this trend with GraSP in ResNet18 and SNIP in ResNet34.
B.2 Additional Experiments on Tiny-Imagenet
We expanded our experiments on the Tiny-Imagenet dataset by utilizing weights pre-trained for 100 iterations with a batch size of 128 per iteration. The Tiny-Imagenet dataset consists of 100,000 images, divided into 200 different classes. The results of these experiments are summarized in table 7 and table 8.
C Limitations and Societal Impacts
This work studies the effect of weights under the Ramanujan settings through observation using ITOP. By gaining insights from these observations, we empirically improve the performance of pruning methods using PAGS. We do not expect any negative societal impact from this work.
Table 6: Results on CIFAR10 on ResNet18 and Resnet34 using PEGS in comparison to vanilla PaI methods, LTH and EB. Baseline refers to the PaI-found sparse mask.  $X$ ,  $XX$ , and  $XXX$ , represent low, high, and very high pre-training costs, respectively. @100 refers to weight pretraining at 100 iterations with batch size 128.
Method	Baseline acc. %	with PEGS (Best acc %)	(Avg. acc %)	No pre-training needed
ResNet18@100
SNIP	90.39	91.30	90.06±0.17	✓
GraSP	86.09	91.11	80.38±3.14	✓
ERK	90.07	90.41	90.11±0.22	✓
EB	90.10	—	—	XX
LTH	91.22	—	—	XXX
ResNet34@100
SNIP	92.91	93.22	90.01±1.70	✓
GraSP	92.66	93.11	92.91±0.11	✓
ERK	92.04	92.19	91.88±0.18	✓
EB	92.00	—	—	XX
LTH	92.76	—	—	XXX
Table 7: Results on Tiny-ImageNet on ResNet18 using PAGS/PEGS in comparison to vanilla PaI methods, LTH and EB. Baseline refers to the PaI-found sparse mask.  $X$ ,  $XX$ , and  $XXX$ , represent low, high, and very high pre-training costs, respectively. @100 and @500 refer to different "pre-training" iterations using PEGS.
Method	Baseline acc. %	with PAGS/PEGS (Best acc %) (Avg. acc %)	No pre-training needed
ResNet18@100
SNIP	47.13	49.00	48.45 ± 0.44	X
GraSP	48.31	49.43	48.82 ± 0.32	X
ResNet18@500
SNIP	45.83	47.83	47.11 ±0.42	X
GraSP	48.89	50.26	49.48 ± 0.29	X
EB	47.43	—	—	XX
LTH	49.81	—	—	XXX
Table 8: Results on Tiny-ImageNet on ResNet34 using PAGS/PEGS in comparison to vanilla PaI methods, LTH and EB. Baseline refers to the PaI-found sparse mask.  $X$ ,  $XX$ , and  $XXX$ , represent low, high, and very high pre-training costs, respectively. @100 and @500 refer to different "pre-training" iterations using PEGS.
Method	Baseline acc. %	with PAGS/PEGS (Best acc %) (Avg. acc %)	No pre-training needed
ResNet34@100
SNIP	53.07	54.91	53.65 ± 0.4	X
GraSP	52.71	53.71	52.80 ± 0.41	X
ResNet34@500
SNIP	53.57	54.65	54.16 ± 0.25	X
GraSP	53.83	55.48	54.84 ± 0.35	X
EB	53.40	—	—	XX
LTH	54.00	—	—	XXX