SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Don't Just Prune by Magnitude! Your Mask Topology is Another Secret Weapon
Duc Hoang
hoangduc@utexas.edu
Souvik Kundu
souvikk.kundu@intel.com
Shiwei Liu
s.liu3@tue.nl
Zhangyang Wang
atlaswang@utexas.edu
Recent years have witnessed significant progress in understanding the relationship between the connectivity of a deep network's architecture as a graph, and the network's performance. A few prior arts connected deep architectures to expand graphs or Ramanujan graphs, and particularly,[7] demonstrated the use of such graph connectivity measures with ranking and relative performance of various obtained sparse sub-networks (i.e. models with prune masks) without the need for training. However, no prior work explicitly explores the role of parameters in the graph's connectivity, making the graph-based understanding of prune masks and the magnitude/gradient-based pruning practice isolated from one another. This paper strives to fill in this gap, by analyzing the Weighted Spectral Gap of Ramanujan structures in sparse neural networks and investigates its correlation with final performance. We specifically examine the evolution of sparse structures under a popular dynamic sparse-to-sparse network training scheme, and intriguingly find that the generated random topologies inherently maximize Ramanujan graphs. We also identify a strong correlation between masks, performance, and the weighted spectral gap. Leveraging this observation, we propose to construct a new "full-spectrum coordinate" aiming to comprehensively characterize a sparse neural network's promise. Concretely, it consists of the classical Ramanujan's gap (structure), our proposed weighted spectral gap (parameters), and the constituent nested regular graphs within. In this new coordinate system, a sparse subnetwork's  $\ell_2$ -distance from its original initialization is found to have nearly linear correlated with its performance. Eventually, we apply this unified perspective to develop a new actionable pruning method, by sampling sparse masks to maximize the  $\ell_2$ -coordinate distance. Our method can be augmented with the "pruning at initialization" (PaI) method, and significantly outperforms existing PaI methods. With only a few iterations of training (e.g 500 iterations), we can get LTH-comparable performance as that yielded via "pruning after training", significantly saving pre-training costs. Codes can be found at: https://github.com/VITA-Group/FullSpectrum-PAI.
Pruning [21] reduces the size of deep neural networks (DNNs) by generating sparse models suitable for compute and memory-limited applications while still preserving comparable accuracy as their dense counterparts. Existing research in this area can broadly be divided into three main components. Firstly, pruning after training  $(\mathbf{PaT})$  that involves creating sparse DNNs by leveraging information
from a trained dense model. The Lottery Ticket Hypothesis (LTH) [4], a notable work in PaT, suggests a sparse subnetwork within a dense trained model can achieve comparable performance. Based on this hypothesis, they propose iterative magnitude pruning (IMP) to identify such "lottery" subnetwork at an expensive cost. Secondly, Dynamic Sparse Training (DST) [14, 11, 2] starts with training a sparse network from scratch instead of using a pre-trained dense model. Here, the sparsity mask keeps on updating throughout the training to reach maturity while maintaining the target sparsity during each epoch, often referred to as "sparse-to-sparse" training [12]. The mask update in DST primarily relies on pruning and regrowing operations. Finally, Pruning at Initialization (PaI) [9, 20, 18] keeps the sparsity mask frozen throughout the training. Various proxies have been proposed [9, 20, 18, 2] to identify the sparsity mask before training, although the PaI-yielded performance so far falls clearly below LTH and other PaT methods [6] despite the former's appealing low overheads.
Despite the significant overlap between these methods, particularly in their use of various proxies like magnitude, they differ in their beliefs on the origin of the inherent knowledge represented by the proxy used to create the sparsity mask. PaI assumes that the knowledge emerges during the initialization phase, while PaT and DST hypothesize that the knowledge is acquired from a fully or partially trained model. Empirical studies ([16], [6]) have demonstrated the effectiveness of PaT over PaI under high sparsity conditions. However, such performance benefits often come at the cost of high training requirements due to the iterative nature of PaT. Interestingly, a recent work [10] highlighted that even randomly pruned subnetworks can achieve similar performance as the dense model, further raising questions about the applicability of different proxies. These often conflicting and counter-intuitive results intensify the debate of proxy knowledge generation through matured training vs. that identified at the beginning.
To unravel this mystery, we draw inspiration from a complementary perspective of graph theory to precisely understand the relationship between the connectivity of a neural network's architecture as a graph and the network's performance. Specifically, we focus on a subset of the expander graph family, namely the Ramanujan graphs. Ramanujan graphs have a maximal eigenbound, allowing the network to be highly sparse and highly connected. This aligns with the primary objective of pruning — to find a sparse yet highly connected network. Previous works [19, 15, 1, 17, 7] have provided empirical evidence supporting this intuition by utilizing the Ramanujan property to guide existing sparse generators. However, to our understanding, there exists several missing links:
1. Is the formation of the Ramanujan characteristic in sparse structures a natural occurrence during training? If so, how can we observe and quantify it?
2. What is the role of weight magnitudes in a Ramanujan graph and can they have a relation with the graph topology? If so, can there be a unified representation to encompass both?
Towards closing this gap, we first investigate the evolution of sparse structures under In-Time Over-Parameterization (ITOP) [13], a popular DST regime using magnitude pruning and random growth. Interestingly, we observe that DST's random growth is inherently a maximizer of Ramanujan graphs (though not a very efficient one). We then discover a negative correlation between the performance and the weighted spectral gap  $(\lambda)$ . To further leverage this observation, we propose to construct a new "full-spectrum coordinate" aiming to comprehensively characterize a sparse neural network's promise, by combining Ramanujan's bound  $\Delta r$  (structure), weighted spectral gap  $\lambda$  (parameters), and the constituent nested regular graphs within (as in the case of  $\Delta r_{imdb}$  [7]). Most strikingly, in the resultant coordinate system, we find that a sparse sub-network's  $\ell_2$ -moving distance from its original initialization has a nearly linear correlation with the network's performance. Inspired by this, we maximize this  $\ell_2$ -distance in the coordinate by greedily sampling sparse masks at initialization, yielding a new PaI approach. We empirically verify that our new PaI method, dubbed Pruning at Initialization as Graph Sampling (PAGS), can create significantly better "zero-shot" masks than the existing PaI methods. Our contributions are outlined as:
- We first uncover how a sparse topology is evolved in a representative dynamic sparse training scheme (ITOP) by analyzing the weighted spectral gap of Ramanujan structures. We discover that this mechanism is an inherent yet inefficient Ramanujan maximizer.
- We then establish a full-spectrum coordinate to jointly measure the structure and weight distance by combining the Ramanujan perspective with the weighted spectral gap, demonstrating a sparse subnetwork's  $\ell_2$ -moving distance from its initialization in the new coordinate as a strong (linear) performance indicator.
- We propose Pruning at Initialization as Graph Sampling (PAGS) by greedily maximizing the aforementioned  $\ell_2$ -moving distance. PAGS can be organically applied as a "zero-shot" PaI method and outperforms existing PaI methods with large margins. With only a few iterations of training (e.g. only 500 iterations), we can get LTH-comparable performance as that yielded via IMP in PaT, significantly saving pre-training costs.
2 Observing Weight and Topology Evolution via Dynamic Sparse Training
2.1 Notations and Definitions
We first introduce two important concepts that are used to explain key observations made later on in this section. We also summarized the various notations in Table 1 for easy reference.
Ramanujan gap: Belonging to a subset of the expander graphs, Ramanujan graphs are distinguished by their sparse yet highly interconnected nature. High connectivity in a graph implies smoother information flow, a trait coveted in sparse neural networks. Previous research [19, 15, 1, 17] interpreted DNNs as a series of bipartite compute graphs (check the appendix for more details). Here, each layer takes the form of a square adjacency matrix  $A$ . Pruning strategies, inspired by Ramanujan properties, aim to widen the gap between Ramanujan's upper-bound,  $2*\sqrt{d - 1}$ , where  $d$  is the average number of edge per node, and the non-trivial eigenvalue of the compute adjacency matrix,  $\hat{\mu}(A)$ . This gap can be viewed in two ways:
- Canonical perspective:  $\Delta r = 2*\sqrt{d - 1} -\hat{\mu} (A)$
- Iterative perspective:  $\Delta r_{imdb} = \frac{1}{|K|}\sum_{i = 1}^{|K|}(2\sqrt{d_i - 1} -\hat{\mu} (A_{K_i}))$
While the canonical perspective,  $\Delta r$ , indicates the ease of information propagation by measuring the network's degree of connectivity, a more recent take by [7] introduces the iterative perspective,  $\Delta r_{imdb}$ . This measures the average connectivity limit across all subgraphs  $K$  within  $A$ , potentially offering a more comprehensive insight into the network's connectivity. Note that "imdb" stands for "iterative mean difference of bound", following the same naming scheme in [7].
Weighted spectral gap: Of our own devise, the weighted spectral gap, denoted as  $\lambda$ , quantifies the separation between the trivial  $(\mu_0)$  and non-trivial  $(\hat{\mu})$  eigenvalues of the weighted adjacency matrix,  $W$ .  $\lambda$  provides insights analogous to the Cheeger constant, which evaluates if a graph possesses a "bottleneck". Mirroring the two previous perspectives for Ramanujan gap, the weighted spectral gap is similarly dual-formed:
- Canonical perspective:  $\lambda = \mu_0(|\pmb{W}|) - \hat{\mu} (|\pmb{W}|)$ ,
- Iterative perspective:  $\lambda_{msg} = \frac{1}{|K|}\sum_{i=1}^{|K|}\left(\mu_0(|\boldsymbol{W}_{K_i}|) - \hat{\mu}(|\boldsymbol{W}_{K_i}|)\right)$
In this context,  $W$  stands for the layer-specific weight elements. However, unlike  $\Delta r$  and  $\Delta r_{imdb}$ , the interpretations of  $\lambda$  and  $\lambda_{i msg}$  are not universally settled in relation to the network's sparsity and their performance correlation. However, later on in this section, we reveal them to be reliable indicators for performance. Note, here "msg" stands for "iterative mean spectral gap."
Table 1: Important graph notations, their equations, and descriptions.
Notation	Equation	Description
Δr	2*√d-1-μ(A)	Measure a graph's degree of connectivity using Ramanujan bound
Δrimbdb	1/K|∑i=1|K|(2√di-1-μ(AK_i))	Iterative mean of Ramanujan gap for set of subgraphs K in A
λ	μ0(|W|)-μ(|W|)	Spectral gap of weight matrix W
λmsg	1/K|∑i=1|K| (μ0(|W_K_i|)-μ(|W_K_i|))	Iterative mean of spectral gap for set of subgraphs K in W
2.2 The Role of Dynamic Sparse Training in Pruning with Ramanujan
Limitations of Ramanujan theory in model pruning: Recently Hoang et al. [7] extended the Ramanujan theory to PaI, achieving notable success in ranking PaI-pruned subnetworks with  $\Delta r_{imdb}$ . Yet, their reliance on graph spectrum properties introduces pressing challenges. These encompass the absence of a direct pruning strategy, an oversight of weight magnitude, and the inherent non-differentiability of graph spectrum properties. Such limitations substantially restrict the wider applicability of the Ramanujan theory to the straightforward pruning approaches.
Foundational assumption: This study is anchored on two pivotal hypothesis. Firstly, we posit that graph-based metrics can be strategically used as a sampling criterion, allowing for heuristic optimization and sidestepping issues related to non-differentiability. Secondly, we believe that the topology and weights of a sparse subnetwork jointly and complementarily influence performance. To substantiate these hypotheses, we aim to identify a correlation between a mask's topology, its corresponding weight, and the ensuing performance of the sparse subnetwork. Ultimately, our goal is to develop an actionable pruning method that can be leveraged to improve PaI as well as PaT.
Dynamic sparse training as an observational tool: DST [14, 2, 13, 22, 8] is an efficient sampler that (i) can swiftly navigate the mask-weight space through training dynamics evolution and (ii) is known to specialize at the "good performance" subspace (since DST always evolves to lower the training loss), rather than attempting to fit the full space. This saves the sampling burden considerably. Note that, focusing on the "good performance" subspace suffices for our scenarios since most pruning methods deliver "reasonably good" sparse masks. By using DST as an observational tool, we further provide tantalizing evidence on the natural formation of the graph properties (e.g., Ramanujan characteristic) during training.
2.3 Observing Temporal Evolution of Sparse DNN using DST
Setup: To generate and evaluate results in this section, we utilize the ITOP framework [13] for its reliable performance in monitoring sparse structures and their evolving graph characteristics. Using magnitude pruning and random growth (initial renewal rate of  $50\%$ ), we sample model masks and weights every 1500 iterations over 250 epochs. After training these unique sparse models, we assess their graph characteristics to base our subsequent observations. Notably, all our sparse subnetworks maintain a  $99\%$  unstructured sparsity (only  $1\%$  trained weights remain non-zero).
2.3.1 Correlation between Performance and Sparse Topology
From Table 1, we distill the distinctions between the canonical Ramanujan bound  $\Delta r$  and its iterative variation  $\Delta r_{imdb}$ . The correlation between these and performance in image classification across evolving sparse structures is visually captured in Figure 1, with time-sampled data color-coded on a gradient scale. Key takeaways include:
Effectiveness over time: Our findings show that the performance of fully fine-tuned sparse DNNs improves over time when sampled during the ITOP process. This reaffirms ITOP's (and DST's overall) capability in pinpointing the "good performance" subspace, as discussed in Section 2.2.
Contradiction to convention: The bottom row of Figure 1 displays a narrowing trend for  $\Delta r$  with increasing performance, challenging previously held beliefs from [19, 15]. This narrowing suggests that the paths for information flow between layers become progressively more constrained.
Patterned random-growth: Contrary to the notion that random growth is entirely unpredictable, Figure 1 indicates structured trends between accuracy and both  $\Delta r$  and  $\Delta r_{imdb}$ . This min-max pattern suggests maximizing or minimizing the expansion property, contingent on the chosen informational boundary metric. Interestingly, the increasing expansion of regular sub-graphs within each layer aligned with the intuitive link between information flow rate and performance.
2.3.2 Complementary Relationship between Weight and Topology Graphs
Table 1 itemizes one of our contributions in this paper, namely the proposal for a weighted spectral gap,  $\lambda$  and  $\lambda_{msg}$  as a comprehensive and reliable performance indicator. Figure 2 presents a heatmap that combines information from both the weight magnitudes and graph topology. This heatmap highlights the region of highly performative models in red, while other regions are shown in blue. By
(a) VGG16
(b) ResNet18
(c) ResNet34
Figure 1: The evolution of  $\Delta r_{imdb}$  and  $\Delta r$  over time as a performance function using ITOP.
correlating the graph's "unweighte" topology with our proposed weighted counterparts, we make the following observations:
Pareto curvature: Figure 2 shows a mutual dependency between the structure of the graph and corresponding weights. In particular, it depicts the Pareto curvature lines between  $\Delta r$  and  $\lambda$ , as well as  $\Delta r_{imdb}$  and  $\lambda_{msg}$ .
Area of performance: We identify the region of highly performative sparse models that lies within a range optimizing the forward expansion characteristic represented by  $\Delta r_{imdb}$  while minimizing the average weight magnitude of sub-graphs represented by  $\lambda_{msg}$ . Conversely, for the relationship between  $\Delta r$  and  $\lambda$ , improved performance is often associated with narrowing the canonical information flow while increasing the layer-wise weight average.
These observations emphasize the interplay between the structure of the graph and the weights in achieving optimal performance. The analysis in Figure 2 provides valuable insights into the trade-off between graph topology and weight magnitudes, shedding light on the relationship between these factors and performance outcomes. This complex interplay among multiple factors emphasizes the need for unified coordinates to describe the performance relations in sparse DNNs.
2.4 Full-Spectrum Coordinate Distance as a Strong, Linear Performance Indicator
In Section 2.3, we identified a robust interconnection between the Ramanujan gap and the weighted spectral gap, from both canonical and iterative angles. Figures 1 and 2 portray the combined influence of sparse topology and weights in optimizing sparse DNN performance, prompting us to investigate the central question: Is it possible to allow them to co-exist in the same representation?
To this end, we introduce a novel coordinate system, the "Full-spectrum", integrating  $\Delta r$ ,  $\Delta r_{imdb}$ ,  $\lambda$ , and  $\lambda_{imdb}$  as its four layer-wise axes, given their ties to both weights and topology. Formally, it's represented as  $\mathbb{R}^{L\times 4}$ , with  $L$  denoting the # layers of a model. Navigating this space entails navigating a complex trade-off landscape. We gauge its utility by assessing its linear correlation with overall performance via Pearson correlation. Figures 3 and 4 visualize the efficacy of the "Full-spectrum" in contrast to existing graph metrics. From these, we deduce:
Consistent linear correlation with performance: As evidenced by Figure 3 under ITOP's random growth regime, the  $\ell_2$ -moving distance from a sparse subnetwork's original position within the "Full-spectrum" aligns almost linearly with performance.
(c) ResNet34
Figure 2: Performance landscape between the structure and weights under a graph perspective. In the top panel, we examine the correlation between  $\Delta r_{imdb}$  and  $\lambda_{msg}$ . In the bottom panel, we explore the correlation between  $\Delta r$  and  $\lambda$ . The landscape represents the classification performance, with gradients coloring from dark blue to dark red, indicating the worst to best performance.
(a) VGG16  $(\rho = 0.71)$
Figure 3: Full-spectrum  $\ell_2$ -distance against classification performance on CIFAR-10 across different models. We denote the associated Pearson correlation  $(\rho)$  to performance in the parentheses.
(b) ResNet18  $(\rho = 0.96)$
(c) ResNet34  $(\rho = 0.91)$
Superiority over current graph metrics: Figure 4 confirms that, relative to  $\Delta r$  and  $\lambda$ , our "Full-spectrum" perspective offers a more direct linear correlation to performance. This affirms the harmonious co-existence of these metrics within a unified representation, and the potency of this integrated indicator paves the way for its application in tangible pruning scenarios.
3 Pruning as Graph Sampling
We now present our actionable pruning methodology as graph sampling, which essentially leverages the proposed  $\ell_2$ -moving distance metric associated with the "full-spectrum" coordinate, which shows to be a strong, linear performance predictor. Note that our proposed methodology is applicable to any pruning setting since it makes no specific assumption on the weight (e.g., randomly initialized or pre-trained weights) nor the mask (e.g., local layer-wise or global pruning mask). This makes our methodology generic that can be augmented with various "off-the-shelf" pruning methods.
In this section, two specific variants are introduced, one for Pruning at Initialization (PaI), and the other for Pruning after Training (PaT), which requires only a few training iterations.
$\Delta \mathbf{r}$ $(\rho = -0.85)$
ResNet18
$\lambda$ $(\rho = 0.95)$
Full spectrum  $(\rho = 0.99)$
$\Delta r$ $(\rho = -0.85)$
Figure 4: Illustration of the correlation between topology  $(\Delta r)$ , weights  $(\lambda)$ , and the combined "full spectrum" with respect to the classification performance on CIFAR-100 with ResNet18 (top row) and ResNet34 (bottom row).  $\rho$  indicates the Pearson correlation.
ResNet34
$\lambda$ $(\rho = 0.95)$
Full spectrum  $(\rho = 0.97)$
3.1 Pruning at Initialization as Graph Sampling (PAGS)
Pruning at Initialization as Graph Sampling (PAGS) is a lightweight pruning method that does not require any pre-training and aims to maximize the layer-wise full-spectrum  $\ell_2$ -moving distance. It directly "augments" any existing PaI method, by oversampling the PaI mask generator and selecting the top mask that tends to maximize the  $\ell_2$ -moving distance criteria. In other words, PAGS can be viewed as a "meta-framework" applicable on top of (and to improving) any existing PaI method.
For proxy-driven PaI methods such as [9, 20, 18], we first generate a population of sparsity masks: to create each mask, we simply leverage a randomly selected small minibatch of training data (GenerateMask().). For non-proxy-driven PaI like random pruning or ERK [10], the mask population is created by randomly assigning the non-zero values to random weight locations for
Algorithm 1: Pruning at Initialization as Graph Sampling (PAGS)
1: Initialize the weights θ
2: n ← population size
3: k ← mini-batch size
4: i ← sampling rate
5: A, W ← GenerateMask(data, k, θ)
6: // generate sparse topology and weights
7: while n - 1 ≥ 0 do
8: A, W ← GenerateMask(data, k, θ)
9: // sampling new mask
10: A, W ← CompareMerge((A, W), (A, W))
11: // compare and merge if conditions are satisfied
12: if n % i == 0 and A, W are modified then
13: Save A, W
14: end if
15: n ← n - 1
16: end while
each mask. From the population, we sample a mask, perform a layer-wise comparison to the current best mask, and update the best in case it improves the  $\ell_2$ -moving distance (CompareMerge().). For a population size of  $n$ , the process is repeated  $n$  times. During these iterations, we periodically save the current best mask at an interval of  $i$  steps. Algorithm 1 illustrates this process.
PAGS is computationally inexpensive, as each mask sampling requires only a forward pass over a minibatch. In all our experiments, we adopt the default  $n = 1,000$ ,  $i = 20$ , and minibatch size 128.