Title: Don’t just prune by magnitude! Your mask topology is a secret weapon
Venue: NeurIPS 2023
Accepted: yes
Keywords: Pruning at Initialization

Pruning at Training

LTH

DST

Ramanujan

graph
Abstract: Recent years have witnessed significant progress in understanding the relationship between the connectivity of a deep network's architecture as a graph, and the network's performance. A few prior arts connected deep architectures to expander graphs or Ramanujan graphs, and particularly,[7] demonstrated the use of such graph connectivity measures with ranking and relative performance of various obtained sparse sub-networks  (i.e. models with prune masks) without the need for training. However, no prior work explicitly explores the role of parameters in the graph's connectivity, making the graph-based understanding of prune masks and the magnitude/gradient-based pruning practice isolated from one another. This paper strives to fill in this gap, by analyzing the Weighted Spectral Gap of Ramanujan structures in sparse neural networks and investigates its correlation with final performance. We specifically examine the evolution of sparse structures under a popular dynamic sparse-to-sparse network training scheme, and intriguingly find that the generated random topologies inherently maximize Ramanujan graphs. We also identify a strong correlation between masks, performance, and the weighted spectral gap. Leveraging this observation, we propose to construct a new "full-spectrum coordinate'' aiming to comprehensively characterize a sparse neural network's promise. Concretely, it consists of the classical Ramanujan's gap (structure), our proposed weighted spectral gap (parameters), and the constituent nested regular graphs within. In this new coordinate system, a sparse subnetwork's L2-distance from its original initialization is found to have nearly linear correlated with its performance. Eventually, we apply this unified perspective to develop a new actionable pruning method, by sampling sparse masks to maximize the L2-coordinate distance. Our method can be augmented with the "pruning at initialization" (PaI) method, and significantly outperforms existing PaI methods. With only a few iterations of training (e.g 500 iterations), we can get LTH-comparable performance as that yielded via "pruning after training", significantly saving pre-training costs. Codes can be found at:  https://github.com/VITA-Group/FullSpectrum-PAI.

Don't Just Prune by Magnitude! Your Mask Topology is Another Secret Weapon

Abstract

Recent years have witnessed significant progress in understanding the relationship between the connectivity of a deep network's architecture as a graph, and the network's performance. A few prior arts connected deep architectures to expander graphs or Ramanujan graphs, and particularly,[7] demonstrated the use of such graph connectivity measures with ranking and relative performance of various obtained sparse sub-networks (i.e. models with prune masks) without the need for training. However, no prior work explicitly explores the role of parameters in the graph's connectivity, making the graph-based understanding of prune masks and the magnitude/gradient-based pruning practice isolated from one another. This paper strives to fill in this gap, by analyzing the Weighted Spectral Gap of Ramanujan structures in sparse neural networks and investigates its correlation with final performance. We specifically examine the evolution of sparse structures under a popular dynamic sparse-to-sparse network training scheme, and intriguingly find that the generated random topologies inherently maximize Ramanujan graphs. We also identify a strong correlation between masks, performance, and the weighted spectral gap. Leveraging this observation, we propose to construct a new 'full-spectrum coordinate' aiming to comprehensively characterize a sparse neural network's promise. Concretely, it consists of the classical Ramanujan's gap (structure), our proposed weighted spectral gap (parameters), and the constituent nested regular graphs within. In this new coordinate system, a sparse subnetwork's ℓ 2 -distance from its original initialization is found to have nearly linear correlated with its performance. Eventually, we apply this unified perspective to develop a new actionable pruning method, by sampling sparse masks to maximize the ℓ 2 -coordinate distance. Our method can be augmented with the 'pruning at initialization" (PaI) method, and significantly outperforms existing PaI methods. With only a few iterations of training (e.g 500 iterations), we can get LTH-comparable performance as that yielded via 'pruning after training", significantly saving pre-training costs. Codes can be found at: https://github.com/VITA-Group/FullSpectrum-PAI .

1 Introduction

Pruning [21] reduces the size of deep neural networks ( DNNs ) by generating sparse models suitable for compute and memory-limited applications while still preserving comparable accuracy as their dense counterparts. Existing research in this area can broadly be divided into three main components. Firstly, pruning after training ( PaT ) that involves creating sparse DNNs by leveraging information

Souvik Kundu

Zhangyang Wang

from a trained dense model. The Lottery Ticket Hypothesis ( LTH ) [4], a notable work in PaT, suggests a sparse subnetwork within a dense trained model can achieve comparable performance. Based on this hypothesis, they propose iterative magnitude pruning (IMP) to identify such 'lottery' subnetwork at an expensive cost. Secondly, Dynamic Sparse Training ( DST ) [14, 11, 2] starts with training a sparse network from scratch instead of using a pre-trained dense model. Here, the sparsity mask keeps on updating throughout the training to reach maturity while maintaining the target sparsity during each epoch, often referred to as 'sparse-to-sparse' training [12]. The mask update in DST primarily relies on pruning and regrowing operations. Finally, Pruning at Initialization ( PaI ) [9, 20, 18] keeps the sparsity mask frozen throughout the training Various proxies have been proposed [9, 20, 18, 2] to identify the sparsity mask before training, although the PaI-yielded performance so far falls clearly below LTH and other PaT methods [6] despite the former's appealing low overheads.

Despite the significant overlap between these methods, particularly in their use of various proxies like magnitude, they differ in their beliefs on the origin of the inherent knowledge represented by the proxy used to create the sparsity mask. PaI assumes that the knowledge emerges during the initialization phase, while PaT and DST hypothesize that the knowledge is acquired from a fully or partially trained model. Empirical studies ([16], [6]) have demonstrated the effectiveness of PaT over PaI under high sparsity conditions. However, such performance benefits often come at the cost of high training requirements due to the iterative nature of PaT. Interestingly, a recent work [10] highlighted that even randomly pruned subnetworks can achieve similar performance as the dense model, further raising questions about the applicability of different proxies. These often conflicting and counter-intuitive results intensify the debate of proxy knowledge generation through matured training vs. that identified at the beginning.

To unravel this mystery, we draw inspiration from a complementary perspective of graph theory to precisely understand the relationship between the connectivity of a neural network's architecture as a graph and the network's performance. Specifically, we focus on a subset of the expander graph family, namely the Ramanujan graphs. Ramanujan graphs have a maximal eigenbound, allowing the network to be highly sparse and highly connected. This aligns with the primary objective of pruning -to find a sparse yet highly connected network. Previous works [19, 15, 1, 17, 7] have provided empirical evidence supporting this intuition by utilizing the Ramanujan property to guide existing sparse generators. However, to our understanding, there exists several missing links:

Is the formation of the Ramanujan characteristic in sparse structures a natural occurrence during training? If so, how can we observe and quantify it?

What is the role of weight magnitudes in a Ramanujan graph and can they have a relation with the graph topology? If so, can there be a unified representation to encompass both?

Towards closing this gap, we first investigate the evolution of sparse structures under In-Time OverParameterization (ITOP) [13], a popular DST regime using magnitude pruning and random growth. Interestingly, we observe that DST's random growth is inherently a maximizer of Ramanujan graphs (though not a very efficient one). We then discover a negative correlation between the performance and the weighted spectral gap ( λ ). To further leverage this observation, we propose to construct a new 'full-spectrum coordinate' aiming to comprehensively characterize a sparse neural network's promise, by combining Ramanujan's bound ∆ r (structure), weighted spectral gap λ (parameters), and the constituent nested regular graphs within (as in the case of ∆ r imdb [7]). Most strikingly, in the resultant coordinate system, we find that a sparse sub-network's ℓ 2 -moving distance from its original initialization has a nearly linear correlation with the network's performance. Inspired by this, we maximize this ℓ 2 -distance in the coordinate by greedily sampling sparse masks at initialization, yielding a new PaI approach. We empirically verify that our new PaI method, dubbed Pruning at Initialization as Graph Sampling ( PAGS ), can create significantly better 'zero-shot' masks than the existing PaI methods. Our contributions are outlined as:

We first uncover how a sparse topology is evolved in a representative dynamic sparse training scheme (ITOP) by analyzing the weighted spectral gap of Ramanujan structures. We discover that this mechanism is an inherent yet inefficient Ramanujan maximizer.

We then establish a full-spectrum coordinate to jointly measure the structure and weight distance by combining the Ramanujan perspective with the weighted spectral gap, demonstrating a sparse subnetwork's ℓ 2 -moving distance from its initialization in the new coordinate as a strong (linear) performance indicator.

We propose Pruning at Initialization as Graph Sampling (PAGS) by greedily maximizing the aforementioned ℓ 2 -moving distance. PAGS can be organically applied as a 'zero-shot' PaI method and outperforms existing PaI methods with large margins. With only a few iterations of training (e.g. only 500 iterations), we can get LTH-comparable performance as that yielded via IMP in PaT, significantly saving pre-training costs.

2 Observing Weight and Topology Evolution via Dynamic Sparse Training

2.1 Notations and Definitions

We first introduce two important concepts that are used to explain key observations made later on in this section. We also summarized the various notations in Table 1 for easy reference.

Ramanujan gap: Belonging to a subset of the expander graphs, Ramanujan graphs are distinguished by their sparse yet highly interconnected nature. High connectivity in a graph implies smoother information flow, a trait coveted in sparse neural networks. Previous research [19, 15, 1, 17] interpreted DNNs as a series of bipartite compute graphs (check the appendix for more details). Here, each layer takes the form of a square adjacency matrix A . Pruning strategies, inspired by Ramanujan properties, aim to widen the gap between Ramanujan's upper-bound, 2 ∗ √ d -1 , where d is the average number of edge per node, and the non-trivial eigenvalue of the compute adjacency matrix, ˆ µ ( A ) . This gap can be viewed in two ways:

Canonical perspective: ∆ r = 2 ∗ √ d -1 -ˆ µ ( A )

Iterative perspective: ∆ r imdb = 1 | K | ∑ | K | i =1 (2 √ d i -1 -ˆ µ ( A K i ))

While the canonical perspective, ∆ r , indicates the ease of information propagation by measuring the network's degree of connectivity, a more recent take by [7] introduces the iterative perspective, ∆ r imdb . This measures the average connectivity limit across all subgraphs K within A , potentially offering a more comprehensive insight into the network's connectivity. Note that 'imdb' stands for 'iterative mean difference of bound', following the same naming scheme in [7].

Weighted spectral gap: Of our own devise, the weighted spectral gap, denoted as λ , quantifies the separation between the trivial ( µ 0 ) and non-trivial ( ˆ µ ) eigenvalues of the weighted adjacency matrix, W . λ provides insights analogous to the Cheeger constant, which evaluates if a graph possesses a 'bottleneck'. Mirroring the two previous perspectives for Ramanujan gap, the weighted spectral gap is similarly dual-formed:

Canonical perspective: λ = µ 0 ( | W | ) -ˆ µ ( | W | ) ,

Iterative perspective: λ imsg = 1 | K | ∑ | K | i =1 ( µ 0 ( | W K i | ) -ˆ µ ( | W K i | ) )

In this context, W stands for the layer-specific weight elements. However, unlike ∆ r and ∆ r imdb , the interpretations of λ and λ imsg are not universally settled in relation to the network's sparsity and their performance correlation. However, later on in this section, we reveal them to be reliable indicators for performance. Note, here 'imsg' stands for 'iterative mean spectral gap.'

2.2 The Role of Dynamic Sparse Training in Pruning with Ramanujan

Limitations of Ramanujan theory in model pruning: Recently Hoang et al. [7] extended the Ramanujan theory to PaI, achieving notable success in ranking PaI-pruned subnetworks with ∆ r imdb . Yet, their reliance on graph spectrum properties introduces pressing challenges. These encompass the absence of a direct pruning strategy, an oversight of weight magnitude, and the inherent nondifferentiability of graph spectrum properties. Such limitations substantially restrict the wider applicability of the Ramanujan theory to the straightforward pruning approaches.

Foundational assumption: This study is anchored on two pivotal hypothesis. Firstly , we posit that graph-based metrics can be strategically used as a sampling criterion, allowing for heuristic optimization and sidestepping issues related to non-differentiability. Secondly , we believe that the topology and weights of a sparse subnetwork jointly and complementarily influence performance. To substantiate these hypotheses, we aim to identify a correlation between a mask's topology, its corresponding weight, and the ensuing performance of the sparse subnetwork . Ultimately, our goal is to develop an actionable pruning method that can be leveraged to improve PaI as well as PaT.

Dynamic sparse training as an observational tool: DST [14, 2, 13, 22, 8] is an efficient sampler that (i) can swiftly navigate the mask-weight space through training dynamics evolution and (ii) is known to specialize at the 'good performance' subspace (since DST always evolves to lower the training loss), rather than attempting to fit the full space. This saves the sampling burden considerably. Note that, focusing on the 'good performance' subspace suffices for our scenarios since most pruning methods deliver 'reasonably good' sparse masks. By using DST as an observational tool, we further provide tantalizing evidence on the natural formation of the graph properties (e.g., Ramanujan characteristic) during training.

2.3 Observing Temporal Evolution of Sparse DNN using DST

Setup: To generate and evaluate results in this section, we utilize the ITOP framework [13] for its reliable performance in monitoring sparse structures and their evolving graph characteristics. Using magnitude pruning and random growth (initial renewal rate of 50%), we sample model masks and weights every 1500 iterations over 250 epochs. After training these unique sparse models, we assess their graph characteristics to base our subsequent observations. Notably, all our sparse subnetworks maintain a 99% unstructured sparsity (only 1% trained weights remain non-zero).

2.3.1 Correlation between Performance and Sparse Topology

From Table 1, we distill the distinctions between the canonical Ramanujan bound ∆ r and its iterative variation ∆ r imdb . The correlation between these and performance in image classification across evolving sparse structures is visually captured in Figure 1, with time-sampled data color-coded on a gradient scale. Key takeaways include:

Effectiveness over time: Our findings show that the performance of fully fine-tuned sparse DNNs improves over time when sampled during the ITOP process. This reaffirms ITOP's (and DST's overall) capability in pinpointing the 'good performance' subspace, as discussed in Section 2.2.

Contradiction to convention: The bottom row of Figure 1 displays a narrowing trend for ∆ r with increasing performance, challenging previously held beliefs from [19, 15]. This narrowing suggests that the paths for information flow between layers become progressively more constrained.

Patterned random-growth: Contrary to the notion that random growth is entirely unpredictable, Figure 1 indicates structured trends between accuracy and both ∆ r and ∆ r imdb . This min-max pattern suggests maximizing or minimizing the expansion property, contingent on the chosen informational boundary metric. Interestingly, the increasing expansion of regular sub-graphs within each layer aligned with the intuitive link between information flow rate and performance.

2.3.2 Complementary Relationship between Weight and Topology Graphs

Table 1 itemizes one of our contributions in this paper, namely the proposal for a weighted spectral gap , λ and λ imsg as a comprehensive and reliable performance indicator. Figure 2 presents a heatmap that combines information from both the weight magnitudes and graph topology. This heatmap highlights the region of highly performative models in red, while other regions are shown in blue. By

(a) VGG16

(b) ResNet18

(c) ResNet34

89.0

91.8

91.5

50

88.0

93.0

91.2

40

(%）

87.0

91.0

92.5

90.8

Test Acc(

86.0

Test

90.5

85.0

90.2

92.0

20

84.0

90.0

91.5

10

89.8

83.0

1.651.701.751.801.851.90

0.100 0.125 0.150 0.175 0.200 0.225 0.250 0.275

0.00

0.020.040.060.080.10

0.12

Iterative_Mean_Ramanujan_Bound

Iterative_Mean_Ramanujan_Bound

Iterative_Mean_Ramanujan_Bound

89.0

91.8

91.5

50

88.0

93.0

91.2

40

(%)

87.0

91.0

92.5-

90.8

Test Acc

86.0

Step

85.0

90.2

92.0

20

84.0

90.0

91.5

10

89.8

83.0

0.50-0.25 0.000.25 0.500.75 1.001.25

1.75 1.50-1.25 -1.00 0.75 0.50

Ramanujan_Bound

Ramanujan_Bound

Ramanujan_Bound

correlating the graph's 'unweighte' topology with our proposed weighted counterparts, we make the following observations:

Pareto curvature: Figure 2 shows a mutual dependency between the structure of the graph and corresponding weights. In particular, it depicts the Pareto curvature lines between ∆ r and λ , as well as ∆ r imdb and λ imsg .

Area of performance: We identify the region of highly performative sparse models that lies within a range optimizing the forward expansion characteristic represented by ∆ r imdb while minimizing the average weight magnitude of sub-graphs represented by λ imsg . Conversely, for the relationship between ∆ r and λ , improved performance is often associated with narrowing the canonical information flow while increasing the layer-wise weight average.

These observations emphasize the interplay between the structure of the graph and the weights in achieving optimal performance. The analysis in Figure 2 provides valuable insights into the trade-off between graph topology and weight magnitudes, shedding light on the relationship between these factors and performance outcomes. This complex interplay among multiple factors emphasizes the need for unified coordinates to describe the performance relations in sparse DNNs.

2.4 Full-Spectrum Coordinate Distance as a Strong, Linear Performance Indicator

In Section 2.3, we identified a robust interconnection between the Ramanujan gap and the weighted spectral gap, from both canonical and iterative angles. Figures 1 and 2 portray the combined influence of sparse topology and weights in optimizing sparse DNN performance, prompting us to investigate the central question: Is is possible to allow them to co-exist in the same representation ?

To this end, we introduce a novel coordinate system, the 'Full-spectrum' , integrating ∆ r , ∆ r imdb , λ , and λ imdb as its four layer-wise axes, given their ties to both weights and topology. Formally, it's represented as R L × 4 , with L denoting the # layers of a model. Navigating this space entails navigating a complex trade-off landscape. We gauge its utility by assessing its linear correlation with overall performance via Pearson correlation. Figures 3 and 4 visualize the efficacy of the 'Full-spectrum' in contrast to existing graph metrics. From these, we deduce:

Consistent linear correlation with performance: As evidenced by Figure 3 under ITOP's random growth regime, the ℓ 2 -moving distance from a sparse subnetwork's original position within the 'Full-spectrum' aligns almost linearly with performance.

(%) 3 151

(% 3 1591

(a) VGG16

(b) ResNet18

(c) ResNet34

91.75

68

91.50

0.00

.65

Iiterative_Mean_Ramanujan_Bound

88

93.0

91.25

0.02

1.70

87

Test Acc (%)

91.00

0.04

92.5

-86

90.75

1.80

90.50

92.0

85

90.25

0.10

90.00

.90

84

91.5

89.75

0.12

0.250 0.255 0.260 0.265 0.270 0.275 0.280 0.285

83

0.035 0.040 0.045 0.050 0.055 0.060 0.065 0.070

0275

0.0250 0.0275 0.0300 0.0325 0.0350 0.0375 0.0400 0.0425

Iterative_Mean_Spectral_Gap

Iterative_Mean_Spectral_Gap

Iterative_Mean_Spectral_Gap

Spectral_Gap

Spectral_Gap

Spectral_Gap

0;5

tztttot60g°0z09°0

↑.25

68

0.35

0.40

0.45

0.2

91.75

91.50

0200.220.240.260.280.300.32

0.4

f.00

88

0.4

91.25

0.6

93.0

F0.6

87

Test Acc (%)

91.00

-0.8

92.5

0.50

90.75

0:25

90.50

92.0

0.00

85

90.25

1.4

84

1.6

90.00

1.6

91.5

89.75

1.8

1.8

(a) VGG16 (

ρ

=

0

.

71

)

(b) ResNet18 (

ρ

=

0

.

96

)

ρ

=

0

.

91

)

89.0

91.8

91.5

50

88.0

91.2

93.0

40

Test Acc (%)

87.0

91.0

92.5

Test AcC

90.8

Test AcC

86.0

90.5

92.0

85.0

90.2

20

84.0

90.0

91.5

10

89.8

83.0

3.5

4.0

4.5

5.0

5.5

1.00

1.25

1.501.752.00

2.25

2.50

1.0

1.2

1.41.61.82.02.2

full_spectrum_12

full_spectrum_12

full_spectrum_12

Superiority over current graph metrics: Figure 4 confirms that, relative to ∆ r and λ , our 'Full-spectrum' perspective offers a more direct linear correlation to performance. This affirms the harmonious co-existence of these metrics within a unified representation, and the potency of this integrated indicator paves the way for its application in tangible pruning scenarios.

3 Pruning as Graph Sampling

We now present our actionable pruning methodology as graph sampling, which essentially leverages the proposed ℓ 2 -moving distance metric associated with the 'full-spectrum' coordinate, which shows to be a strong, linear performance predictor. Note that our proposed methodology is applicable to any pruning setting since it makes no specific assumption on the weight (e.g., randomly initialized or pre-trained weights) nor the mask (e.g., local layer-wise or global pruning mask). This makes our methodology generic that can be augmented with various 'off-the-shelf' pruning methods.

In this section, two specific variants are introduced, one for Pruning at Initialization (PaI), and the other for Pruning after Training (PaT), which requires only a few training iterations.

ResNet18

∆

r

(

ρ

=

-

0

.

85)

λ

(

ρ

= 0

.

95)

Full spectrum

(

ρ

= 0

.

99)

ResNet34

∆

r

(

ρ

=

-

0

.

85)

λ

(

ρ

= 0

.

95)

Full spectrum

(

ρ

= 0

.

97)

70.0

60

70.0

60

70.0

60

69.0

50

69.0

50

69.0

50

Test Acc (%)

68.0

40

Test Acc (%)

68.0

40

Test Acc (%

40

67.0

67.0

67.0

66.0

20

66.0

20

66.0

20

65.0

10

65.0

10

65.0

10

-0.700-0.600-0.500-0.400-0.300-0.200

0.300

0.400

0.500

0.600

0.700

0.300

0.400

0.500

0.600

Ramanujan_Bound

Spectral_Gap

full_spectrum_I2

60

60

60

73.0

73.0

73.0

50

50

50

Test Acc (%)

72.0

40

Test Acc (%)

72.0

40

Test Acc (%)

72.0

40

71.0

71.0

71.0

20

20

20

70.0

70.0

70.0

10

10

10

-0.900 -0.800 -0.700-0.600 -0.500 -0.400

0.200

0.250

0.3000.3500.4000.450

L

0.100 0.120 0.140 0.160 0.180 0.200 0.220

0

Ramanujan_Bound

Spectral_Gap

full_spectrum_I2

3.1 Pruning at Initialization as Graph Sampling (PAGS)

Pruning at Initialization as Graph Sampling (PAGS) is a lightweight pruning method that does not require any pre-training and aims to maximize the layer-wise full-spectrum ℓ 2 -moving distance. It directly 'augments' any existing PaI method, by oversampling the PaI mask generator and selecting the top mask that tends to maximize the ℓ 2 -moving distance criteria. In other words, PAGS can be viewed as a 'meta-framework' applicable on top of (and to improving) any existing PaI method.

For proxy-driven PaI methods such as [9, 20, 18], we first generate a population of sparsity masks: to create each mask, we simply leverage a randomly selected small minibatch of training data ( GenerateMask ( . ) ). For non-proxy-driven PaI like random pruning or ERK [10], the mask population is created by randomly assigning the non-zero values to random weight locations for

Algorithm 1: Pruning at Initialization as Graph Sampling (PAGS)

1: Initialize the weights θ

2: n ← population size

3: k ← mini-batch size

4: i ← sampling rate

5: A , W ← GenerateMask (data, k , θ )

6: // generate sparse topology and weights

7: while n - 1 ≥ 0 do

8: ˆ A ˆ A ˆ A , ˆ W ˆ W ˆ W ← GenerateMask (data, k , θ )

9: // sampling new mask

10: A , W ← CompareMerge (( A , W ), ( ˆ A ˆ A ˆ A , ˆ W ˆ W ˆ W ))

11: // compare and merge if conditions are satisfied

12: if n % i == 0 and A , W are modified then

13:

Save

A

,

W

14: end if

15: n ← n - 1

16: end while

each mask. From the population, we sample a mask, perform a layer-wise comparison to the current best mask, and update the best in case it improves the ℓ 2 -moving distance ( CompareMerge ( . ) ). For a population size of n , the process is repeated n times. During these iterations, we periodically save the current best mask at an interval of i steps. Algorithm 1 illustrates this process.

PAGS is computationally inexpensive , as each mask sampling requires only a forward pass over a minibatch. In all our experiments, we adopt the default n = 1,000, i = 20, and minibatch size 128.

3.2 Pruning Early as Graph Sampling (PEGS)

To further improve the classification accuracy at reduced compute, we extend the zero-shot PAGS, to present Pruning Early as Graph Sampling ( PEGS ), which incurs only a small amount of pre-training cost. In PEGS, we 'pre-train' the dense network only for a small number of iterations (by default, for 500 iterations in all our experiments) and then apply PAGS to this lightly trained network. This draws similar inspiration as LTH rewinding [5] or early-bird (EB) ticket [23] in literature.

Compared to the PaI case, we note that it is not as straightforward to directly 'sample' masks using LTH/EB or any other PaT method, since generating a different mask from those methods would require re-taking an expensive training process: otherwise, the pruning is deterministic on trained weights and there is no way to inject randomness. Hence, we do not treat PEGS as a 'metaframework' on top of PaT methods. Instead , we discover that light enough pre-training, followed by cheap PaI-based sampling, can yield sparse masks with comparable quality to the top-performing PaT masks such as LTH, at a significantly cheaper cost. For example, compared to LTH with M rounds of iterative training, N epochs per training round, and I iterations each epoch ( I = 500 in CIFAR-10): PEGS in the same setting would only cost 500 iterations of pre-training, which is roughly MN × more compute-efficient and faster . For example, in CIFAR-10 experiments we have M = 23 and N = 250, hence PEGS would yield > 5000 times saving over LTH.

4 Experiments

4.1 Experiment setup

We leverage four PaI generators to facilitate our experiments (both PAGS and PEGS): · Random [10] uniformly prunes every layer with the same pruning ratio assigned globally. Each parameter is randomly assigned a score based on the normal distribution. · ERK [3, 14] initializes sparse networks with a Erd˝ os-Rényi graph where small layers are usually allocated more budget, while bigger layers are assigned fewer parameter budget. Random pruning is then performed following those layer-wise ratios · SNIP [9] for any layer l it issues scores s l = | g l ⊙ w l | where g l and w l are gradients and weights respectively. The weights with the lowest scores after one iteration are pruned before training. · GraSP [20] removes weights that impeded gradient flows by computing the Hessian-gradient product h l and issue scores s l = -w ⊙ h l , for a layer l .

We also compare with two PaT methods: · Lottery Ticket Hypothesis (LTH) [4] iteratively prunes the lowest 20 % of weights and rewind the remaining weights to some values in the past. To achieve the desired 99 % sparsity, LTH would necessitate 23 rounds of full training and pruning. · Early Bird (EB) [23] utilizes one-shot 'early pruning', with a 'mask-similarity' signal to automatically terminate pre-training as a cost-saving mechanism (typically happening around the first 15%-20% epoches). The original EB was implemented for structured pruning; for a fair comparison, the results shown in Table 3 are our re-implementation of EB onto unstructured pruning.

Unless otherwise stated, we use a high target sparsity of 99% in all our experiments. We demonstrate results on CIFAR-10/ CIFAR-100 in the main text. We use two representative models, Resnet18 and ResNet34, as the main backbones in this section 1 . Additional training details and results on Tiny-ImangeNet are deferred to the Supplementary due to the space limit.

4.2 Comparison of PAGS and PEGS with Existing PaI methods

Tables 2 and 3 detail our pruning results for CIFAR-10 using PAGS and PEGS, while Tables 4 and 5 do so for CIFAR-100. As PAGS/PEGS operate on a sampling basis, we list both the peak performance of the best mask (Best acc) and the mean performance across all n sampled masks (Avg. acc). The latter is supplemented by the standard deviation. From the presented results, we deduce the following: 1 In CIFAR-10/100 trials, our methods consistently surpass the baseline PaI generators in PAGS. A standout instance is the 2 . 12% advantage over ERK for ResNet18 using PaI, as seen in Table 2. 2 With light pre-training, our methods still hold an edge against reference generators. For instance, Tables 4 and 5 indicate about 1% enhancement over the top-performing generator at both the 100 and

500 iteration marks. 3 Interestingly, optimal outcomes typically manifest during the initialization phase, implying that light pre-training doesn't instantaneously refine the weight distribution.

4.3 Comparison with (Much More Expensive) LTH and the Early Bird Ticket

We further compare PAGS/PEGS with the two PaT methods, LTH and EB, and note that the latter two are significantly costlier. Overall, for PaI setting (table 2), we intend to include LTH as the 'performance ceiling' and show PAGS can get close to or even outperform it often times. For example, Table 2 shows that ResNet18 utilizing GraSP marginally outperforms LTH, without any pre-training.

For PaT setting (table 3, table 4 and table 5), we show that with a small fraction of pre-training costs, our PEGS could solidly outperform both LTH and EB, using either SNIP or GRASP mask generators. Interestingly, in ResNet18, even the random ERK mask generator can be turned into LTGlevel performance with the aid of PEGS. Our findings suggest that maximizing the full-spectrum's ℓ 2 -moving distance can significantly improve performance by optimizing both the graph's structure and weights. This approach allows us to achieve performance levels comparable to LTH but at a significantly lower cost. By leveraging the insights the Ramanujan perspective provides, we can achieve notable performance improvements across the board of pruning, while incurring minimal computational overheads.

5 Conclusion

Recent years have seen the rise of graph theory in analyzing and understanding sparse subnetworks. However, we still lack a crucial understanding of the role parameters play in graph connectivity. To fill this gap, in this paper, we study the weighted spectral gap of Ramanujan structures in sparse neural networks and investigate its correlation with the final performance. By examining the evolution of sparse structures under DST, we identify a strong correlation between Ramanujan bound, weighted spectral gap, and performance. Leveraging these observations, we proposed a new 'full-spectrum coordinate' which comprehensively characterizes the complex interplay between various graph characteristics that further leads to actionable pruning methods both at initialization and after light pre-training. This unified perceptive is expected to invoke more future exploration into the complex interplay between topology and weights, not just for sparse NNs but for generic DNNs as well.

Acknowledgment

Z. Wang is in part supported by NSF Scale-MoDL (award number: 2133861) and the NSF AI Institute for Foundations of Machine Learning (IFML).

References

K. Bhardwaj, G. Li, and R. Marculescu. How does topology influence gradient propagation and model performance of deep networks with densenet-type skip connections? 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 13493-13502, 2021.

U. Evci, T. Gale, J. Menick, P. S. Castro, and E. Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning , pages 2943-2952. PMLR, 2020.

U. Evci, T. Gale, J. Menick, P. S. Castro, and E. Elsen. Rigging the lottery: Making all tickets winners. ArXiv , abs/1911.11134, 2020.

J. Frankle and M. Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv: Learning , 2018.

J. Frankle, G. K. Dziugaite, D. M. Roy, and M. Carbin. Stabilizing the lottery ticket hypothesis. arXiv preprint arXiv:1903.01611 , 2019.

J. Frankle, G. K. Dziugaite, D. M. Roy, and M. Carbin. Pruning neural networks at initialization: Why are we missing the mark? ArXiv , abs/2009.08576, 2020.

D. N. Hoang, S. Liu, R. Marculescu, and Z. Wang. Revisiting Pruning at Initialization Through the Lens of Ramanujan Graph. In The Eleventh International Conference on Learning Representations , 2023.

S. Kundu, M. Nazemi, P. A. Beerel, and M. Pedram. Dnr: A tunable robust pruning framework through dynamic network rewiring of dnns. In Proceedings of the 26th Asia and South Pacific Design Automation Conference , pages 344-350, 2021.

N. Lee, T. Ajanthan, and P. H. S. Torr. Snip: Single-shot network pruning based on connection sensitivity. ArXiv , abs/1810.02340, 2018.

S. Liu, T. Chen, X. Chen, L. Shen, D. C. Mocanu, Z. Wang, and M. Pechenizkiy. The unreasonable effectiveness of random pruning: Return of the most naive baseline for sparse training. ArXiv , abs/2202.02643, 2022.

S. Liu, D. C. Mocanu, A. R. R. Matavalam, Y. Pei, and M. Pechenizkiy. Sparse evolutionary deep learning with over one million artificial neurons on commodity hardware. Neural Computing and Applications , 33:2589-2604, 2021.

S. Liu and Z. Wang. Ten lessons we have learned in the new" sparseland": A short handbook for sparse neural network researchers. arXiv preprint arXiv:2302.02596 , 2023.

S. Liu, L. Yin, D. C. Mocanu, and M. Pechenizkiy. Do we actually need dense overparameterization? in-time over-parameterization in sparse training. ArXiv , abs/2102.02887, 2021.

D. C. Mocanu, E. Mocanu, P. Stone, P. H. Nguyen, M. Gibescu, and A. Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications , 9(1):2383, 2018.

B. Pal, A. Biswas, S. Kolay, P. Mitra, and B. Basu. A study on the ramanujan graph property of winning lottery tickets. In ICML , 2022.

M. Paul, F. Chen, B. W. Larsen, J. Frankle, S. Ganguli, and G. K. Dziugaite. Unmasking the lottery ticket hypothesis: What's encoded in a winning ticket's mask? ArXiv , abs/2210.03044, 2022.

A. Prabhu, G. Varma, and A. M. Namboodiri. Deep expander networks: Efficient deep networks from graph theory. In ECCV , 2018.

H. Tanaka, D. Kunin, D. L. K. Yamins, and S. Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. ArXiv , abs/2006.05467, 2020.

D. T. Vooturi, G. Varma, and K. Kothapalli. Ramanujan bipartite graph products for efficient block sparse neural networks. ArXiv , abs/2006.13486, 2020.

C. Wang, C. Wang, G. Zhang, and R. B. Grosse. Picking winning tickets before training by preserving gradient flow. ArXiv , abs/2002.07376, 2020.

H. Wang, C. Qin, Y. Bai, Y. Zhang, and Y. Fu. Recent advances on neural network pruning at initialization. In International Joint Conference on Artificial Intelligence , 2021.

L. Yin, V. Menkovski, M. Fang, T. Huang, Y. Pei, and M. Pechenizkiy. Superposing many tickets into one: A performance booster for sparse neural network training. In Uncertainty in Artificial Intelligence , pages 2267-2277. PMLR, 2022.

H. You, C. Li, P. Xu, Y. Fu, Y. Wang, X. Chen, R. G. Baraniuk, Z. Wang, and Y. Lin. Drawing early-bird tickets: Towards more efficient training of deep networks. arXiv preprint arXiv:1909.11957 , 2019.

A DNN as bipartite graphs

A bipartite graph is a graph consisting of two distinct sets of vertices L and R that are connected with E edges as G ( L ∪ R,E ) . Let v denote the total number of vertices in both sets, the more common way to represent G is as a binary adjacency matrix A ∈ R v × v .

A DNN is a series of bipartite graphs, one for each layer, to represent all of its compute graphs. For a convolutional layer with parameter tensor θ ∈ R C in × C out × k in × k out , we unfold the dimensions so that L = C in ∗ k in ∗ k out and R = C out . For a linear layer with parameter tensor θ ∈ R C in × C out , we can directly adapt its parameters where L = C in and R = C out .

An interesting and relevant property of a bipartite graph, is that when each node has the same number of d out-edges, its adjacency matrix A has eigenvalues µ ( A ) such that µ 0 ≥ ... ≥ µ v -1 , where µ 0 and | µ v -1 | are equaled to d . We define ˆ µ ( A ) = max | µ i |̸ = d | µ i | as the largest nontrivial eigenvalue. In the context of unstructured pruning, we often find G to be irregular , in which case Hoang et al. [7] showed that d avg ≤ | µ 0 | ≤ d max .

B Supplementary results for PAGS and PEGS

B.1 Additional Experiments on CIFAR-10

We expanded our experiments on the CIFAR-10 dataset by utilizing weights pre-trained for 100 iterations with a batch size of 128 per iteration. The CIFAR-10 dataset consists of 50,000 training images and 10,000 testing images, divided into 10 different classes. The results of these experiments are summarized in Table 6.

We observed performance improvement relative to baseline. However, compared to other modes of pre-training for CIFAR-10, certain PaI generators exhibited higher-than-expected standard deviation and lower average performance, indicating some instability in generating sparse structures. Specifically, we observed this trend with GraSP in ResNet18 and SNIP in ResNet34.

B.2 Additional Experiments on Tiny-Imagenet

We expanded our experiments on the Tiny-Imagenet dataset by utilizing weights pre-trained for 100 iterations with a batch size of 128 per iteration. The Tiny-Imagenet dataset consists of 100,000 images, divided into 200 different classes. The results of these experiments are summarized in table 7 and table 8.

C Limitations and Societal Impacts

This work studies the effect of weights under the Ramanujan settings through observation using ITOP. By gaining insights from these observations, we empirically improve the performance of pruning methods using PAGS. We do not expect any negative societal impact from this work.

Peer reviews:
comments: SUMMARY:
The paper suggests two pruning strategies: pruning at initialization as graph sampling (PAGS) and pruning early as graph sampling (PEGS). The PAGS approach is one of those that doesn't call for any pre-training dietary intake. The mask is oversampled each time, and the masks with the greatest l2 movement distance are chosen. With the PEGS approach, which is a slight enlargement of PAGS, the dense network undergoes very little pre-training. The article experiment performs SOTA on CIFAR10.

STRENGTHS:
* The context of the paper is very logical.
* This paper is rich in theoretical analysis.

WEAKNESSES:
* The actual method suggested is relatively naive, and too much length in the paper is devoted to analysis and introduction.

* The experimental part only contains the CIFAR-10/100 dataset, and such a small number of experiments cannot accurately reflect the effectiveness of the method. It is highly recommended that the author increase the validity experiments such as ImagNet.

* "only 500 iteration" does not support the statement of "a few iteration" on small datasets like CIFAR.

QUESTIONS:
Please see the weakness part.

LIMITATIONS:
No limitations discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new method for pruning deep neural networks that optimizes both the network's structure and weights. The experimental results show that it can be naturally applied as a “zero-shot\" PaI method and outperforms existing PaI methods with large margins. Alternatively, it can be applied after light pre-training to get LTH-comparable performance.\n\nSTRENGTHS:\nThe proposed approach to deep network pruning based on Ramanujan graphs is a technically innovative and interesting method that optimizes both the graph's structure and weights, achieving notable performance improvements across various pruning methods while incurring minimal computational overheads. As connecting graph theory to sparse network architecture is a recently emergent idea, this paper fills in a timely and important research gap. \n\nThe authors leveraged DST as an organic tool to sample sparse masks and observe its co-evolution with weights. The research observation sheds light on the relationship between the connectivity of a neural network's architecture as a graph and the network's performance, providing a complementary perspective to understand the effectiveness of different pruning methods. \n\nThe algorithm novelty lies in the use of a new \"full-spectrum coordinate\" that characterizes a sparse neural network's promise, consisting of the classical Ramanujan's gap, the proposed weighted spectral gap, and the constituent nested regular graphs within.\n\nThe experimental findings suggest that maximizing the full-spectrum's l2-moving distance can significantly improve performance by optimizing both the graph's structure and weights. The empirical evidence also shows that even the random ERK mask generator can be turned into LTH-level performance with the aid of PEGS.\n\nWEAKNESSES:\nDespite the appeal of theory, I don’t think the current experiments are sound enough. In general, all pruning methods in-use tend to perform similarly on CIFAR-10/100. As seen in Tables 1 and 2, different methods’ numbers are often within each other’s error bar, making it hard to judge whether one “outperforms” others with true statistical significance or just by chance/ random noise. \n\nNoteworthily, while the authors highlighted their success that “ResNet18 utilizing GraSP (+ PAGS) even marginally outperforms LTH, without any pretraining”, the GRASP baseline in Table 1 (top) and Table 2 (bottom) already outperformed LTH, hence disapproving LTH as “empirical performance ceiling” as authors assumed (or otherwise, implying there exists an unfair comparison among baselines).\n\nI also don’t feel the “Full-Spectrum Coordinate” is convincing nor sound enough. An ad-hoc construction by authors, it has two “topology-only” coordinates and two other “weight-topology” coordinates. So at least, the four two groups of coordinates are redundant, one dependent of the other. Why one cannot expect a better linear predictor via more “orthogonalized” coordinates, such as two “topology-only” and two “weight-only” coordinates combined?\n\nLastly, the experiment part contains many hyperparameters that were barely discussed or justified. For example, “I = 50”, “N = 250”, etc. How are they chosen and are they sensitive?\n\nQUESTIONS:\nThis is overall an interesting and theoretically sound paper, but several important concerns remain as discussed in “weakness”. Please address each of them.\n\nLIMITATIONS:\nno", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new approach to deep network pruning based on graph theory, specifically the use of Ramanujan graphs. The authors argue that their approach, which focuses on optimizing both the graph's structure and weights, can achieve notable performance improvements across various pruning methods while incurring minimal computational overheads. Empirical results demonstrate that their approach outperforms existing methods such as LTH and EB, even with a small fraction of pre-training costs.\n\nSTRENGTHS:\n-\tThe paper proposes a new approach to deep network pruning based on Ramanujan graphs, which are highly sparse yet highly connected graphs that align with the primary objective of pruning. Although prior arts have connected Ramanujan graphs with pruning/PaI quality assessment, this is first work deriving an actionable pruning method based on such.\n\n-\tThe approach focuses on optimizing both the graph's structure and weights, which can achieve notable performance improvements across various pruning methods while incurring minimal computational overheads. \n\n-\tEmpirical results demonstrate that the proposed approach outperforms existing methods such as LTH and EB, even with a small fraction of pre-training costs. It also significantly improve the previous random sampling-based pruning, e.g. ERK.\n\n-\tThe paper also sheds light on the relationship between the connectivity of a neural network's architecture as a graph and the network's performance, providing a complementary perspective to understand the effectiveness of different pruning methods.\n\nWEAKNESSES:\nI have two major critiques: (1) poor readability, and (2) unimpressive, sometimes confusing experiments.\n\nReadability: This paper is, at very least, quite hard to follow. That is partially owing to the dense content – the authors tried to pack too many observational stuff in one submission (sections 2-4, before finally reaching their “actionable pruning” results in section 5). That is along with a lot of self-defined jargons and unnecessary notations. I suggest merging sections 2-4 into two sections, one on observation and the other on pruning method, both being significantly shorter in length.\n\nExperiments: there are several presented results confusing me. \n\n-\tWhy VGG shows much poorer linear correlation in Figures 1-4, compared to ResNets? Also the authors didn’t report VGG results in Section 5. I wonder: is the proposed method only specifically effective on ResNets?\n\n-\tI am not sure I understand how to read Figure 3. There is no detailed analysis in main text either.\n\n-\tIn Table 2, the improvements are veryminor, usually within error bar. I am not convinced the improvement of PEGS is evident. Can you show more dataset results like ImageNet?\n\n-\tIn Tables 1 and 2: do you suggest it’s fairer to compare baseline acc with “Best acc”, or “Average acc”?\n\nQUESTIONS:\nPlease check questions enlisted above.\n\nLIMITATIONS:\nThe authors have not explicitly discussed any limitation.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies network pruning in a way that takes into account both the graph topology and values of weights. This connects purely magnitude-based and purely graph-based pruning methods. In particular, the authors find correlations between performance of masks and 4 properties derived per layer of the neural network that account for graph structure and weight magnitude. The authors then use these properties to propose a new method that augments existing pruning at initialization methods.\n\nSTRENGTHS:\n1. Sampling after a light pretraining is a good, efficient approach.\n2. Proposed PAGS method is simple in a good sense, and can be used with any other pruning at initialization method.\n3. Empirical contribution of measurements of various network properties during ITOP.\n\nWEAKNESSES:\n1. Page 5's \"physics intuition\" is hard to understand and unrigorous, and does not seem to do much to alleviate the mentioned \"lack of theory\" from the Pal et al. approach.\n2. [1] also uses weighted graph regularization to sparsify neural networks weights.\n3. I generally find the paper hard to read. It seems like there is a lot of content and jargon / abbreviations introduced very rapidly.\n4. Use of weighted spectral gap is not so novel.\n5. Experimental improvements are quite modest in Tables 1 and 2.\n6. It would be interesting to more directly optimize for the 4 properties of the coordinate system, instead of simplying sampling and choosing masks that improve on those properties.\n\n[1] Tam and Dunson. Fiedler Regularization: Learning Neural Networks with Graph Sparsity. ICML 2020.\n\nQUESTIONS:\nPage 3, the use of \"tantalizing\" is hard to understand for me here. What did you mean by this?\n\nLIMITATIONS:\nThere is a limitations section in Appendix B, but it does not actually discuss any limitations.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}