Understanding Sparse Neural Networks from their Topology via Multipartite Graph Representations
Authors: Elia Cunegatti
Matteo Farina
Doina Bucur
Giovanni Iacca
Venue: Accepted by TMLR ()

Abstract: Pruning-at-Initialization (PaI) algorithms provide Sparse Neural Networks (SNNs) which are computationally more efficient than their dense counterparts, and try to avoid performance degradation. While much emphasis has been directed towards \emph{how} to prune, we still do not know \emph{what topological metrics} of the SNNs characterize \emph{good performance}. From prior work, we have layer-wise topological metrics by which SNN performance can be predicted: the Ramanujan-based metrics. To exploit these metrics, proper ways to represent network layers via Graph Encodings (GEs) are needed, with Bipartite Graph Encodings (BGEs) being the \emph{de-facto} standard at the current stage. Nevertheless, existing BGEs neglect the impact of the inputs, and do not characterize the SNN in an end-to-end manner. Additionally, thanks to a thorough study of the Ramanujan-based metrics, we discover that they are only as good as the \emph{layer-wise density} as performance predictors, when paired with BGEs. To close both gaps, we design a comprehensive topological analysis for SNNs with both linear and convolutional layers, via (i) a new input-aware Multipartite Graph Encoding (MGE) for SNNs and (ii) the design of new end-to-end topological metrics over the MGE. With these novelties, we show the following: (a) The proposed MGE allows to extract topological metrics that are much better predictors of the accuracy drop than metrics computed from current input-agnostic BGEs; (b) Which metrics are important at different sparsity levels and for different architectures; (c) A mixture of our topological metrics can rank PaI algorithms more effectively than Ramanujan-based metrics.

Published in Transactions on Machine Learning Research (04/2024)
Understanding Sparse Neural Networks from their Topology via Multipartite Graph Representations
Elia Cunegatti
eioo.cunegati@@vmitn..
University of Trento, Italy
Mattco Farina
m.faarina@umitn..
University of Trento, Italy
Doina Bucur
d. w...uurr@uuuntt...e ml
University of Twente, The Netherlands
Giovanni Iacca
giovanni
iacoa iac.oaniiiccoe@eeee.
wwwiinn.n...nn....omomoo it
University of Trento, Iaay
Reviewed on OpenReview: https: // openreviee..........mm net/ forum? id- EgbOtU2n0Y
Abstract
(PaI) algorithms provide Sparse Neural Networks (SNNs) which are computationally more efficient than their dense counterparts, and try to avoid performance degradation. While much emphasis has been directed towards how to prune, we still do not know what topological metrics the SNNs characterize good performance. From prior work, we have layer-wise topological metrics by which SNN performance can be predicted: the Ramamujan-basee metrics. To exploit these metrics, proper ways represent network layers via Graph Encodings (GEs) needed, with Bipartite Graph Encodings (BGEs) being the de-facto standard at the current stage. Nevertheless, existing BGEs neglect the impact of the inputs, and do not characterize the SNN in an end-to-end manner. Additionally, thanks to a thorough study of the Ramanujan-based metrics, we discover that they are only as good as the layer-wise densily as performance predictors, when paired with BGFs. To close both gaps, we design a comprehensive topological analysis for SNNs with both linear and convolutional layers, via (i) a new input-aware Multipartite Graph Encoding (MGE) for SNNs and (ii) the design of new end-to-end topological metrics the MGE. With these novelties, we show the following: (a) The proposed MGE allows to extract topological metrics that are much better predictors of the accuracy drop than metrics computed from current input-agnostic BGEs; (b) Which metrics are important at different sparsity levels and for different architectures; (c) mixture of our topological metrics can rank Pal algorithms more effectively than baased metrics. The codebase publicly available at https://github.eo wwtp://cgaaattinnnnnnnooo mng-snn
1
Introduction
Pruning Dense Neural Networks (DNNs) has recently become one of the most promising research areas machine learning. Pruning removes portion of network parameters (i.e., weights) to reduce the computational resources and inference time, while avoiding performance degradation. The "winning tickets " hypothesis, a milestone discovery from Frankle & Carbin (2019), states that within randomly initialized DNN there exist subnetworks that can reach the performance of the overall DNN when trained in isolation. Thanks to this, among several alternatives, Pruning at Initialization (PaI) emerged as good pruning approach.
While many papers have investigated how find sparse architectures, only a few looked what characterizes performiig SNN from a topological viewpoint. An SNN is characterized by a unique topology, and many strategies can be employed to map this topology to a graph representation, allowing to conduct various

---

Published in Transactions on Machine Learning Research (04/2024)
kinds of analyses (Pal et al. 2022 Hoang et al. 2023b a.. For Convolutional Neural Networks (CNNs), state-of-the-art Graph Encodings Fs)) of SNNs rely on layer-based rolled representations, where nodes map to layer parameters, d.e., either a kernel channel (Prabhu et al.) 2018), or a kernel value (Pal et al. 2022; Hoang et al. 2023b). While highly informative, these studics neglect the impact of the input dimensionality on the graph topology, and overlook the importance of end-lo-end information processing within neural networks, solely employing layer-based Bipartite Graph Encodings (BGEs).
To overcome both limitations, propose novel unrolled input-aware Multipartite Graph Encoding (MGE) which fully captures the end-to-end relationship between the (pruned) network parameters and the imput dimensionality. Given an SNN and the dimensionality of its input, our GE works in two steps. First, we generate a BGE for each layer, where left/right nodes correspond to the unrolled inputs/outputs (e.g., pixels in feature maps) while edges correspond to the masked /unmasked relations between them, according to the pruned layer parameters. Since we anchor the dimensionality of our BGEs to the layers' inputs and outputs, with this strategy we generate consccutive bipartite graphs where the right nodes of layer I correspond to the left nodes of layer + 1. This enables extending our sequence of layer-based BGEs to a unique MGE, which fully captures the SNN in an end-to-end and data-aware manner.
We then analyze the accuracy drop of SNNs differently from previous works. Within the field, most efforts are currently aimed at understanding the relat ionship between SNN-derived GEs and Ramanujan graphs, a special class of graphs based on the theory of Expanders, i.e., graphs with high connectivity but few edges (Pal et al.] 2022; Hoang et al., 2023b a). We discover that the existing Ramamujan-based metrics often do not provide more information than a readily available quantity after pruning: the average layer- wise density. This finding has impact, since Ramamjja-based metries require the cumbersome process of building BGEs, while the average layer-wise density can be computed directly from the pruned weight matrices (or convolutional kernels) of the SNN. Hence, in this work we analyze SNNs from a broader graph theory perspective, and propose a different set of graph-based metrics for this purpose. The proposed MGE, paired with the proposed metrics, turns out to be crucial for the analysis of SNNs, underlining the importance of SNN-derived GEs.
Contributions. To summarize, the main contributions of this paper can be outlined as follows:
(a) We examine the link between the Ramamujan-based metries commonly used for graph-based SNN analysis and the average layer-wise density, finding that the former are seldom more informative than the latter.
(b) We present a new unrolled input-aware MGE that fully represents the end-to-end topology of an SNN while accounting for its input dimensionality.
(c) We shift the perspective of graph-based SNN analysis from Ramarujann-assed to a broader viewpoint grounded in graph theory, by proposing new topological metrics for SNNs.
(d) We experiment with large pool of 1, 260 SNNs, showing that the proposed MGE and metrics successfully enable predicting the accuracy drop of SNNs, and ranking PaI methods according to their expected performance, thus paving the way towards the adoption of SNN-derived GEs in practice. We discover that no metric alone can fully explain the performance drop of SNNs for every sparsity ratio and architecture setting, rather a combination of metrics is needed to understand the reason behind the performance drop of SNNs in an arbitrary scenario.
2 Related Work
In this Section, we first introduce the Pruning at Initialization algorithms, and then discuss how graph theory intersects with Deep Learning, specifically with SNN analysis.
Pruning at Initialization (Pal). This family of pruning algorithms aims at discovering the best-performing subnetwork by removing weights prior to training. Good solutions to PaI are very appealing, thanks to their potential to decrease the burden of dense optimization. In modern literature, the earliest approach of this kind is SNIP (Le et al. 2019), which selects the connections to preserve based on their estimated influence on the loss function. While SNIP works in one-shot, its itcrative version IterSNIP is presented in de Jorge et al. (2020). Using second-order information, GraSP (Wang et al., (020) applies gradient signal preservation mechanism based the Hessian-gradient product. Differently from data-dependent methods,
2

---

Published in Transactions on Machine Learning Research (04/2024)
SynFlow (Tanaka et al., 2020) prunes according to the synaptic strengths in data-agnostic setting, which was also explored later in PHEW (Patil & Dovrolis 2021). More recently, ProsPR (Alizadeh et al. 2022) extended SNIP by maximizing the trainability throughout meta-gradients on the first steps of the optimization process. Lastly, NTK-SAP (Wang et al. 2023) relies on the neural tangent kernel theory to remove the least, informative connections in both a weight- and dala-agnostic manner. These approaches are relatively cheap in terms of computational resources, since the mask is found before training in, at most, hundreds of iterations. However, as the sparsity requirement increases, performance deteriorates faster with Pal methods than it does when other categories of pruning algorithms are employed (Dynamic Sparse Training methods
above all, e.g., Dettmers & Zettlemoyer (209 Evci et al. (2020); Liu et al. (202Tb)), ), due to the difficulty of training SNNs from scratch (Evci et al.,
2019 2022). Moreover, the ability of Pal algorithms to uncover the
most effective sparse architectures has been firstly quest ioned using Sanity Checks in Su et al. (2020); Frankle et al. (2021). Subsequently, Liu et al.
(022)) showed that small perturbations to random pruning methods,
such as ER (Mocanu et al. (2177) and ERK (Evci et al.) 2020). can even outperform well-engincered PaI
algorithms. Consequently, this motivates us to shed light on what are the characteristics of a well-performing
SNN after Pal.
Graph Representation of SNNs and DNNs. Studying DNNs from the perspective of graph theory is a popular approach that commonly leverages weighted graph representations. Based on these, many advances to Deep Learning have been proposed, e.g., early-stopping criteria (Rieck et al. 2019), customized initialization techniques [Limnios et al. 2021), and performance predictors (Vahedian et al. 2021 2022).
Other aspects of graph theory have been shown to be successful also in the context of SNNs. In fact, the sparse topology can explain or hint at why the network can still solve given task with a minor accuracy drop w.r.t. its dense counterpart. Within the field, the random generation of small sparse structures for Multi-Layer Perceptrons (MLPs) has been proposed in Bourely et al. (2017); | Stier & Granitzer (2019) and their performance investigated via basic graph properties in |Sterr et al. (2022). Always with MLPs, the Graph-Edii-Distancee has been introduced a similarity measure in Liu et al. (2021a), to show how the graph topology evolves during Dynamic Sparse Training algorithms, such as SET (Mocanu et al. 2017).
Shifting towards more recent and deep architectures (i.e., belonging to the Resnet family), You et al. 2020 used relation graphs to show that the performance of randomly generated SNNs is associated with the clustering coefficient (i.e., the capacity of nodes to "cluster together") and the average path length (i.e., the average number of connections across all shortest paths) of its graph representation.
More recently, to study SNNs generated by Pal algorithms, a new line of research based on Expander graphs has emerged. Exploiting Ramanujan graphs, this research line indicates that the performance of an SNN correlates with graph connectivity (Prabhu et al., 2018: Pal et al. 2022) and spectral information (Hoang et al. 2023a). Additionally, this line suggests that such performance can be predicted thanks to well-designed metrics, such as the itcrative mean difference of bound (IMDB) (Hoang et al. 2023b). Based the theory of Ramanujan graphs, a few works further atlempted to generate efficient SNNs (Slewart el al. 2023 Laenen
2023). As final note, SNN connectivity has also been studied with approximations of the computational graph, to analyze how the data in input to a given sparse structure influence the "effective" sparsity (i.e., the fraction of inactivated connections). "effective" nodes (i.e., nodes with both input and output connections).
and "effective" " paths (i.e., connections connecting input and output layers) ) /Vysogorets Kempel 2023
Pham al., 2023 T
In all the aforementioned works analyzing Pal algorithms, convolutional layers are modeled either with rolled GE based on the kernel parameters (K) (Pal et al.) 2022 Hoang et al., 2023b), with rolled- channel encoding (Prabhu et al. 2018: Vahedian et al. 2022) depending the number of input/output channels (Cin and Cout, respectively).
In both cases, the relationship wetn the SNNs and the input data is not encoded. On the one hand, the rolled GE generates a graph G E ||4||||| for each layer, s.t.
of note, in Vysogorets & Kempel 2023); Pham et al. (2023) the authors used end-to-end computational graph to represent SNNs. Their proposed metrics (effective sparsity, nodes, and paths) are bused the l path norm computed using ones input values over the computational graph. To note that by definition computational graph represents mathematical operations where nodes and edges respectively correspond to operations and data between operations, while in our proposed multipartite graph nodes correspond neurons of the SNN and an edge represents the connection between neurons, that is present the connection not masked in the given sparse structure.
3

---

Published in Transactions on Machine Learning Research (04/2024)
= K Kr, and [R] Cout, with resulting number of graph edges |E| [W]. This GE reflects the topology of single layer (each non-pruned weight corresponds to an edge). but cannot be used to encode connectivity between consecutive layers, since [Ral # I. On the other hand, the rolled- channel encoding works with layer-based graphs G E RIC) where [C] = Cin and [R] Cout, leading to |E| = (c) (0}. Here, nodes correspond to channels. By design, for any inpul-oulput channel pair, an edge exists if and only if the corresponding kernel channel not fully pruned.
Unlike these rolled GEs, we employ an unrolled imput- awere MGE, which we describe in the next Section.
3 Methodology
In this Section, we first introduce the novel unrolled input-aware GE and its formulation in the bipartite version. We then show how this extended to multiportite version, which links consecutive layers and generales single graph that is able to represent the SNNs end-to-end. Finally, we propose set of topological metrics that can be extracted from the proposed encoding, Our notation is summarized in Table M
3.1 Bipartite Graph Encoding (BGE)
The proposed BGE, which takes into consideration the application of the convolutional steps over the inputs, encodes a neural network with N layers as a list of weighted, directed, acyelic bipartite graphs G (Gr,...,GNy ,,)). Due to its design, the bipartite graph construction differs for linear and convolutional layers.
Linear Layers. For lincar layers, we use the encod-
Table 1: Notation used in the paper. Note that we focus on SNNs for vision tasks.
ing proposed in (Rieck et al. 2019 Filan et al. 2021 Prabhu et al. 2018) Vahedian et al., 2022 Hoang et al. 2023b;: denoting with L, and RG respectively the left and right layer of the i-th bipartite graph, and given binary mask M2 {0, 1111,11|R.1, its corresponding GE is G2 (Li U R2 E.), where Eg is the set of edges present in Md i.e., (a, b) E, = M + O, where T. and b R.
Symbol
Definition
hipartite graph with left node ant I,, right node set R (for a total of [Z] + (R] nodes), and edge set
G M.L.R.
N
mumber of layers imput of layer
74
w
height, width of input feature map of i-th layer row, column indexes of input feature map of i-th layer
Convolutional Layers. For convolutional layers, our approach is substantially different from all prior work. Specifically, we devise our encoding based on the unrolled input size: given imput, for each i- th layer, set of feature maps I₁ X construct the corresponding bipartite graph as G = (L, R,, E,), where again L, and RG are the two layers of the bipartite graph, and Li corresponds to the flattened representation of the inputs based on the input size I, and the parameters W; E Remmeeaa Xha k of the i-th layer:
a,b
nodes of G with L and b It liuary mask of suueed/upprned weights
M,
binary mask of pruned/ump /unprmnedee weights of i-th layer
model parameters after pruning
model parameters after pruning of i-th layer model parameters
New
height and width of kernel
Cin Cout P.S.J
number of imput and outpur channels padding, stride, filters
[ The size of the layer R2, i.e., the output feature map, is calculated
LL = h4 X W. X Cin
|R
[14
liker
+ 11 x Wi Wker +
1) Cout-
(1)
S
Differently from the linear layer case, the set of edges Eg cannot be directly computed from the convolutional wher since the latter is dynamically computed over the input date
mask M; E {0, CC XCa Xhar Xhare
Σ
I'n u x in wt tug V out E [0, Cont)-
(2)
O
Faher
2Padding nodes included. 9The formula uses cors--orreeation.
4

---

Published in Transactions on Machine Learning Research (04/2024)
Feature Maps
Convolatioal Step
Convolutional Step
Sparse (sparsity 66)
4
Komels
Quiput
7
Feature
27
10
11
14
17
33
18
18
20
23
0
(
010
(N)
Cpa
Figure 1: Illustration of the proposed unrolled input-aware BGE with = x 3 and convolutional parameters (Cout = 2, C.n = 3, Wkke = 2, hiker = 2, P 0,5 1). (T) and (IT) show, respectively, the first and second convolutional steps and how the graph edges are generated assuming that all the kernel parameters are unmasked. (III) and (IV), respectively, show the complete graph representation after all the convolutional steps have been done in both the (IIT) dense and (IV) sparse cases.
From Eq. 11, we know that Ién u..,, and nout P44 (t.e. the output node of the convolutional step operation) respectively correspond to node (fucte,)xin E Li and a node ( E R2, so, in this case, the edges of the bipartite graph are constructed during the convolutional operation such that:
E; = {(a twp) xis (neas (uusttan +141)) out) |M www.tta + 0 A out, in, ue, v.}
(3)
where the ranges of out, in, us, is are defined according to Eq. (2,, in and out denote, respectively, the input and the output channel for that convolutional step, and (u; + u,+1, 8 + U1+1) corresponds to one kernel entry. Intuitively, given the i-th layer of neural network layer, each input element (e.g., in computer vision tasks, each pixel) represents node in the graph, and the connection between element of the input (denoted a) and element of the output feature map (denoted as b) is present if and only if during the convolutional operation, the contribution of for generating b is not set to zero by the mask M; in the kernel cell used to convolute the two pixels. An illustration of such encoding, which highlights the construction of the graph throughout the convolutional steps for both SNNs and DNNs, is shown in Figure 1
Equation 3 shows how the edges are added to the graph representation for its unweighted version. However, this equation can be easily extended generate weighted graph representation that does not only consider the binary mask M but also the values of model parameters after pruning W, computed W f M then, in Equation 4 it is possible to see how the weighted edge list is generated.
E: = ((( ((4 11 1 11) outs W ouut,nn, (111:; W wouts to # 0 A out, in, u. v/}.
(4)
3.2 Multipartite Graph Encoding (MGE)
The BGE described above has been devised to encode, independently, every layer (either convolutional linear) in network. However, the common limitation of BGEs (Pal et al. 2022 Hoang et al.) 2023b) lies in the lack of connections between any two consccutive (and, indirectly, non-consecutive) i-th and i 1-th layers. On the other hand, our proposed wnrolled input-aware GE, differently from the existing encodings, generates by design ionsecutive layers with the same number of nodes between the i-th right layer and the + 1-th left layer (f.e., [R,] = [(+1l) of consecutive bipartite graphs.
A multipartite graph, a.k.a. N N-partite where N represents the number of layers, is defined as G = (V, É) where V = Vi U V2 U...U Vr and E ((a,b) a Vi,be Vz j < i,j S N). Tn the previous Section,
5

---

Published in Transactions on Machine Learning Research (04/2024)
we showed how to generate set of bipartite graphs G = ,x) for network with N layers where G; = (I, U RA, F.). Given a set of bipartite graphs G, it is possible to concatenate all its elements into a single multiportite graph & = (V, É). This concatenation holds only because with our proposed GE by design |V++1 = | hence edges can be easily rewired as: E = {(a,b) E, 1 S < N} where a E Vi (i.e., it belongs to [4) and E Vi+1 (i.e. it belongs to R.).
However, an extension of the previous encoding is needed for connecting consecutive layers when a pooling operation is employed between them. Because pooling operations lead consecutive BGEs to have different dimensions (i.e., [Ra] # (ا-1)), a further step is required by the GE to make such layers match in size to allow concatenation. In a mutshell, L1+1 is reduced to 4, such that Gi+1 (L,41 U R Eig)) and [4اي = Rgl. The edge rewiring process to translate E to E adds to L a total of Ip(R.) times, each edge in G.+1 that links
I++1 to U E R4+1- This corresponds to each node in the pooling window Pw S RR, computed over R,, which ensures that encoding enables any pair of consecutive layers G; and Gi+1 to be linked in the MGE framework. We refer to Appendix [ for the complete formal explanation of the pooling encoding. It is also worth noting that this MGE also includes residual connections, which are a crucial element in many DNN architectures, while in the previous graph encodings (Pal el al., 2022; Hoang et al. 2023a) they were analyzed layer-wise without linking them between the two corresponding layers. The concatenation of residual connections straightforward. Given two consecutive layers linked by residual connections (e.g.; the last and the first layer of the blocks in Resnet architectures), modeled by their corresponding BGEs G, and G++1, the residual connection is modeled as separate bipartite graph G7 By construction, 4,4+1 = |T,]] and | = 11, hence the residual connection edge construction only adds the edges of G,++1 to the concatenation of G; and G iee. between layers L; and Ri+1.
L is linked to R4+1- Consequently, this pooling
3.3 Topological Metrics
The proposed unrolled MGE allows for the study of SNNs from a topological perspective, including first analysis of the network connectivily between consecutive layers. To this aim, we define a set of topological metrics (in the following, referred to as topometrics) over SNNs, summarized in Table 2
Local Connectivity. The Local metrics are graph metrics com- putable over individual nodes or edges. These metrics (i) are comn putationally inexpensive, and (ii) are able to capture some features of the graph connectivity between con- secutive layers. Node-based topo- metrics include the number of sink nodes, the number of source nodes, and the number of dis- connected nodes over the MGF. The sink and source nodes are, respec- tively, those with outdegree and in- degree of veree] The disconnected nodes are those with neither incom- ing nor outgoing connections. Con- sidering the sink and source nodes, it is possible to compute the number of removable (in/out) connections, The removable out-connections of the set of source nodes (denoted the removable in-connections of the
Table 2: Proposed topometrics with their bricf definitions, and their complexity (N: number of nodes, E: number of edges)
Category
Symbol
Definition
Complexity
sink
no. of nodes with sero outgoing connections
source
no. of nodes with zero incoming comnections
O(E) for all Incal metrics
Local
dise.
of nodes with connections
Connectivity
rout
no removable outgoing connections
r-in
no removable incoming connections
avg. mumber l-hop neighbors
O(E)
Neighboe
N2
avg. number of 2-hop neighbors
O E) O(N3*)
Connertivity
motif())
no. of motils of size & 4
Strength
E-core
centrality messure on maximal degree of subgraphs
O(E)
Connectivity
avg. nodes strength over input edges weights
O + E))
of connected componcnts
O(N 4 F)
Global
Care
avg. size of the connected components
O(1)
Connectivtty
cut-edges out-nodes
no. edges to discounected counected components
O + E)
no. nodes disconeected connected components
O(N 4 F)
7(A)
Spectral Gap of the Adjacency Matrix
Expansion
p(C)
Spectrol Radius of the Laplacian Matrix
O(kum)
which are bdge- topometrics.
here as a) are r-out
[. X Enea outdegreeen).. Complementary,
set of sink nodes (denoted here as (3) are r-in = T x Enes indegree(n)). Intuitively, both these types of connections are useless for the performance of the neural network, since they are ignored at inference. Usually, lower values of Local Connectivity metrics are related to highly connected structures.
The complexity is dependent of the implementation used to compute the cigenvalues. In our case the complexity is given as (krm), where n the size of the matrix, & the number of computed eigenvalues, and m is the number of Lanczos iterations. 4Surcee and sink nodes are respectively not computed over the first and last. layer.
6

---

Published in Transactions on Machine Learning Research (04/2024)
Neighbor Connectivity. To extend the Local Connectivity metries, which compute edge and node properties as a quantity w.r.t. the whole graph, we propose a set of metries that quantify the graph connectivity throughout the nodes' degree information. The first metric computes the size of the neighborhood for each node in the graph. This is the k-hop neighbor, which is denoted M&((a) and is the total number of nodes that are reachable & hops from the root node a. To have a single numerical value, we calculate N₂ as the average (a(a)) over all the graph nodes. In our experiments, we decided to focus on k = 1,2 to capture the connectivity both in the same layer (BGEs) and between consecutive layers (MGE), i.e., 6 Ni and N2, but in principle this metric can be extended to any k lower than the number of layers. We then propose a metric that does not evaluate the nodes' connectivity but rather computes the different and recurrent patterns of connections. In particular, we use 8 Motif(a). i..., the sum of the number of occurrences of any molif of size 4 (we decided to use the size of 4 since it captures connections among consecutive layers). The motifs, which are defined as statistically significant recurrent subgraphs, have been computed using the FANMOD algorithm (Wernicke & Rasche 2006).
Strength Connectivity. The previous metrics capture the node connectivity in fixed subgraph size (i.e,, hops and size of motifs). To better capture the node connectivity of the complete weighted graph structure, we introduce the node-strength, defined as S(a) = Eý=1 Waj where Wa, E E represents the weight of the edge connecting nodes a and j. We also introduce a centrality metric, 10 the k-core decomposition computed using Batagelj & Zaversnik (2003). Given a graph G, this is defined as its maximal subgraph G = (V, F)) such that Va E V dego(a) 2 k. Then, we define k-core(a) as the highest order of the core (k)) that contains a (f.e., the maximal degree of the subgraph that contains a). Both metrics are the average over all the graph nodes.
Global Connectivity, These metrics are computed over the complete MGE and are able capture the overall connectivity of the graph. They (1) are more expensive computationally, but (2) can better analyze the connectivity of the networks. These topometrics are: ①1 the mumber of weakly connected components (a.k.a. clusters), C, and 12 the average size of the clusters, Cang: computed as Tे Σ!d |cel, where o is the i-th component of the networks. Moreover, we identify two Global Connectivity metrics based on edges and nodes, namely 13 the number of cut-edges, and 14 the number cut-nodes, that are respectively the number of edges and nodes required to disconnect connected components. The latter, as well as connected components, have been computed using Depth-first search (DFS).
Expansion. In graph theory, it is standard to analyze the connectivity of a given graph based on its adjacency and Laplacian matrices (Coja-Oghlan, 2007: Tikhomirov & Youssef 2016 Hoffman et al. 2021 Given a graph G = (V, EV, its weighted adjacency matrix defined as Ag E R TXTVT where each entry A corresponds to edge weights EE. [] The Laplacian Matrix, instead, is defined by taking into consideration the nodes' degrees as LG = AG - DD where D is the degree matrix. We leverage these two matrices by extracting their n eigenvalues, formally >(Aa) (4..
s.s. 2 11,2 2 "nnn and X(La)= {1 //) s.t. Mo 2 11 2 ... 2 Inn = 0, using Lehoucq et al. (1998), to compute: 15 the Spectral Ga defined ((A)) = Ho fi, where ja = max Mi and 16 the Spectral Radius, computed as p(Ca) = Mo Preciidoo
et al. 2013 For both metrics, higher values correspond to more connected structures.
4 Experiments
Experimental Setup. To provide insights into the topological properties of SNNs, we first generate a large, heterogeneous pool of sparse topologies. The graph size of the proposed unrolled MGE is based on the shape of the input data, so we select three datasets sharing the input dimensionality: CIFAR-10, C00
(Krizhevsky et al.) 2009) and the downscaled Tiny-Image.Net (Chrabaszcz et al. 2017 , all with 32 x 32 images. We employ four different architectures: Conv-6 (Frankle & Carbin 2019), 20 and Resnet-32 (He et al. 2016), and Siide-Resnet-28-2 (Zagoruyko & Komodakis 2016). For pruning, we use total of seven algorithms: four PaI methods (SNIP (Lee ct al. 2019). GraSP (Wang ct al. 2020). Synflow (Tanaka
et al. 2020), ProsPr (Alizadeh el al. 2022)), ), and three instances of Layer-wise Random Pruning, namely
Uniform Zhu & Gupta. 2017). ER Mocanu et al. 2017), and ERK (Evii et al.
2020
Whhe computing this matrix, we use undirected graphs and the absolute values of the weights, in Pal ct 2022
6 similar metric has been used in Hoang et al. (2023a). from a layer- based perspective. Here, we compute the spectral gap w.r.t. the whole sparse architecture.
7

---

Published in Transactions on Machine Learning Research (04/2024)
We apply the aforementioned algorithms at five sparsity values, to cover broad spectrum, namely s E (0.6, 0.8, 0.9, 0.95, 0.98) (as in Hoang et al. (2023))) ). We train each combination of (pruning algorithm, dataset, architecture, sparsity) for 3 runs, obtaining a pool of 1, 26200 sparse architectures, which are characterized by the 16 topometrics defined above] More details, as well as reproducibility experiments of all pruning algorithms in all settings (all successful), are reported in Appendices A and B
4.1 Ramanujan-based metrics and network density
In this Section, we experiment with the metrics associated with rolled BGE for SNNs: the difference of bound (Pal et al. 2022), its follow-up relaxation, called iterative mean difference of bound (Hoang et al. 2023b), and the iterative mean weighted spectral gap (Hoang et al. 2023a). Since these metrics stem from the theory of Ramanujan graphs, we refer to them as "Ramanujan- basse. Our goal here is to evaluate if these metries provide more valuable insights than network density, which would motivale using the associated GEs in practical applications. Our findings suggest this is seldom the case.
Preliminaries. Given a d-regular graph G, its difference of bound is formally defined as Ar= 2 V ((G). Here, d is the graph regularity and (()) denotes the largest magnitude among the von-trivial eigenvalues of the adjaconcy matrix of g (for irregular graphs, the regularity d is commonly replaced by the estimated degree davg (Hoory) 2005)). This quantity gained popularity in the analysis of Pal algorithms thanks to its close tie with the Expansion properties of G itself. A value Ar 2 0 ensures that the Ramanujan property is held by G. In turn, for irregular graphs, this requires that (()) S 2 x / Vaav I, which is unlikely in high sparsity regimes and would discard valid Expanders from the analysis. Thus, to cope with the strict upper bound requirement of ((() for irregular graphs, HHann et al. (2023b) further propose a relaxalion: Arimdt = [K] E!K x vd; - 11 (K,)), where the difference of bound is averagod over K, í.e., the set of regular subgraphs located in G. The resulting metric is coined as the iterative mean difference of bound. However, both Ar and Arimab ignore the values of the parameters of the model. To include this knowledge, Hoang et al. (2023a) follows the same relaxation principle of Arimau: but introduces the iterative meon weighted spectral gap, defined as Ximog- = [ Σ! |Kl (Ho(IAg;))~ (l(lA In this context, Ag denotes the weighted adjacency matrix of a graph.
All three metries (Ar, Arimab, and Ameeg) exhibit good correlation with the SNN performance, thus reprosenting valid candidates for performance ranking and prediction. However, graph connectivity plays a central role in the theory of Ramanujan graphs and, as a consequence, these quantities may also stronggy correlate with network density, which can be trivially computed directly from the SNN as per layer [Wnonzerol [W]
Hence, we are interested in answering two crucial questions: actually linked to network density?; and (Q2) Are Ar, Arimdb; and Ximag more advantageous than density metrics to rank and/or or predict the performance of SNNs?
(Q1) To what extent are Ar, Arimdb, and Ximag
Ramamujan-based metrics vs. network density, Here we answer question (QI): what is the link between the Ramanujan-based metrics and network density? Since Ar, Arimal, and Ximag are computed separately for each BGE (each one encoding a different network layer), we compare them with the corresponding layer-wise density (i.e., the fraction of active parameters in a layer) induced by Pal algorithms (for brevity, refer to it as Layer-Density). To avoid biasing our observations towards only one architecture, and to account for the possible presence of residual connections, we make this comparison on Conv-6 and Resnet-32, We report the corresponding correlation values in Appendix D
as shown in Figure? 2
Both Figure 2 a) and Figure convey the same message: all Ramanujan metrics follow the same trend of the Layer-Density and are, thus, stouly linked to the overall pruning percentage. In other words, while being intuitive, these metrics provide essentially the same information provided by the Layer-Density.
Analyzing the performance of SNNs: Ar Arimat and Ximag vs. network density. We now answer question (Q2). Ramamujan-based metrics have been shown to correlate well with performance, é.e., SNNs whose BGEs have larger Ar, Arimdb, and Ximag tend to have better accuracy once fine-tuned. Consequently, given a (model, dataset, sparsity) triplet, this useful piece of information can be used to rank Pal methods. However, this largely holds also for the overall network density: the denser the network, the greater the
Each topometric normalized based the number of podes/edges present in the graph representation to prevent graph sixe//newwon network density from being a confounding variable for our topological study.
8

---

Published in Transactions on Machine Learning Research (04/2024)
Medo
Aveg
Layer Dansity GreSP
Ar
Ayop
Layer Donsity GraSP
SNIP
SNIP
Nometzed Vaka
1.3
042
ProsPR
SynFlow
ProsPR
SynFlow
Narmaaad Naka
Nomattee
0.5
EPK
ERK
ER
Nomatzed
040 Layer-Index
080
Layor-Irdex
Layer-hndx
(a) Conv-6 CIFAR-10 0.9
(b) Resnet-32 CIFAR-10 al 0.8
Figure 2: Comparison between the Ramanujan- based metrics and the Seayer-ensstt.. To enable visual comparison, we scale all values by their overall sum across layers s.t. they sum to 1.
performance. As consequence of these observations, we aim to understand whether using Ar, Arimabs Ximag gives better rankings than using the average network density across layers. The results of our analysis, shown in Table 4 provide a significant. finding: the average layer-wise sporsily is mostly comparable lo, and sometimes better than, Ramanajan--aan based metrics as a performance ranking measure.
While these results may suggest that SNN-dcrived BGEs are of limited practical interest, we argue that a key ingredient could still be missing: more informative graph metrics. Henceforth, in the next Section, we go beyond the Ramanujan-based metrics and show that large benefits to SNN analysis emerge when accounting for larger, more informative of graph features.
4.2 Accuracy drop of SNNs
In this Section, we experiment with the proposed MGE and topometrics. First, in Section 4.2.1 we show the potential of our MGE, which allows computing our proposed graph-based topological metrics over SNNs. The information extracted from these metrics is then used means regress the accuracy drop of the SNNs with respect to their corresponding DNNs (hereafter, always refer to the accuracy drop as acc 1 aces. where ace, and acca denote the SNN and DNN accuracy, respectively). Then, focus on understanding the relationship between the proposed topological features and the accuracy drop in Section 4.2.2 We divide all the experiments into two orthogonal scenarios.
acoa
In the Sparsity-Fixed scenario, we aim at understanding which topological features statistically associated with the accuracy drop of SNNs when the computational budget (f.e., the sparsity) is fixed. Importantly, we do not limit our analysis to a specific architecture, but rather we aim at general features that are important across architectural designs.
In contrast, the Architecture-
scenario we fix the architecture and examine how the topometrics- vs.-accuracy relationships evolve when increasing the sparsity ratio. Intuitively, we wish to determine if specific architectural designs lead striking differences the relevance of topological features.
4.2.1 Regression Analysis
We argue that the more informative the GE is, the easier it is to determine the accuracy drop of the SNNs it encodes. Hence, the following we compare the informativeness of different GEs by regressing the accuracy drop, using standard linear least-squares regression. Given an SNN, its respective GE, and the topological features therefrom, the task entails fitting a regression model prodict the accuracy drop between the SNN and its dense counterpart.
9

---

Published in Transactions on Machine Learning Research (04/2024)
Baselines. We compare wrrolled MGE with two encodings used basclines, namely (1) the rolled BGE used in (Pal et al. (2018); Vahedian et al. (2021). Then, from all types of encodings, we compute whenever possible the topometrics described in Section 3.3 It should be noted indeed that our MGE allows to extract end-to-end topological features that cannot be computed otherwise with rolled BGEs. Specifically, our encoding enables the extraction of additional features w.r.t. the existing rolled BGEs: the five Local Connectivity metrics, and the 2-hop Neighbor Connectivity (N2 18)
2022: Hoang et al. 2023b a), and (2) the rolled-channel BGE from Prabhu et al.
Experimental setup. Instead of focusing on unique regression model, we average the regrossion performance over 8 different models, to provide more stable and reliable results. All of these 8 regression methods are listed in Appendix A.66] For each regressor, we employ k-fold cross- pooss-alldation (A- = 5), and aggregate the results over 100 runs. The numerical values used as input of the regression analysis (i.e. the different topometrics) have been normalized separately in [o, 1] using min-max scaling. Regression performance is given in terms of the Adjusted- R22 coefficient, which has been shown to be the most informative measure for discovering associations between input features and predicted variables (Chicco et al. 2021 and the Mean Absolute Error (MAE), which is standard performance metric to compute the closeness between correct and predicted results. Both the train and test splits, obtained with k-fold cross-validation, contain a mixture of accuracy drop values measured on CIFAR-10, CIFAR-100, and Tiny-ImageNet. To provide generalizable conclusions and avoid inducing dataset-dependent behaviors, we do not separate the data points by dataset but use a mixture of datasets in all train and test splits.
Results. The results are reported in Table 3 for both the Sparsity-Fixed and the Architecture-Fixed scenarios. Our unrolled MGE outperforms the rolled BGEs for all cases in both scenarios, suggesting that our MGE is indeed more informative than existing rolled BGEs. Notably, in the Sparsity-Fixed scenario, the unrolled MGE exhibits stable results across sparsity ratios. This hints that the knowledge encoded in our MGE richer than sole network density. In contrast, the information encoded in rolled BGFs is much more sensitive to changes in the overall sparsity. Finally, the results show that the gap among all methods is smaller in the Architecture-Fiss Fixed case, which suggests that varying the architectural design is more relevant than varying the computational budget, regardless of the encoding. Note that this is also suggested by the overall better results in the Architecture-Fixed case.
Table 3: 1 acc regression analysis for both the Sparsity- Fixed and the Architecture Fixed scenarios. We avoid reporting the MAE standard deviation since, with 3 digits approximation, it always equals O.
Sparsity-Plxod
Architecturo-Pine
Metric
Encoding
0.6
0.8
0.9
0.85
0.98
Conv-6
Resuet -0
Resset-32
Wiile WiieeBessze--22 28-2
Rolled BGE
02 1: .01
.11 1: 002
19 02
.10 1: 002
28 1: 01
.65 .01
.75 1: .00
.70 .01
=
001
Adjusted-R (1)
colled-channel BGE
.10 1 .02
110 1: .33
25
03
20
4: .02
.7 1: .01
.06 1: 00
54 2: 001
.60 1: .01
722 =
Unrolled MGE (ours)
.03
.50
1. .01
.78 .01
.01
Rolled BCE
.020
.047
077
.122
.165
.043
.082
.091
.045
MAE
Rolled-channel BGE
.020
.047
.076
.111
.144
.040
.119
.113
.045
Unrolled MGE (ours)
.014
.032
.058
.091
.113
.035
.080
.089
.035
4.2.2 Feature Importance
In the previous Section, we empirically verified that our unrolled MGE extracts more meaningful topological information about SNNs compared to existing rolled BGEs. However, the relationship between the topometrics therefrom and performance remains unclear. Thus, we assess here the importance of each topometric, examining both the Caaritt-Fiieed and the Architecturo-Fixed scenarios. Towards this goal, we compute the Pearson correlation coefficient (commonly denoted by r) between each topometric and the output of each regressor employed in the previous Section, then display the average outcome in Figures 3 and 4
Remark. Note that all regressors have been fitted to predict the accuracy drop. As consequence, when increasing a feature with positive correlation coefficient (r> 0), then acc would likely increase as well (ie., the SNN would perform worse), and vice versa. for features with r < 0.
8 Moreaver, we should remark that the number of motifs (Motif) is bound at 3 for the rolled BGEs (since these encodings are based on directed bipartite graphs) while is set 4 for our MGE (as explained in Section g
10

---

Published in Transactions on Machine Learning Research (04/2024)
Sparsity-Fixed. Figure 3 shows the feature importance (i.e., the correlation coefficients), across increasing sparsity ratios. Interestingly, the importance of the Strength Connectivity and the Neighbor Connectivity topometrics decreases when increasing the sparsity. On the other hand, the Local Connectivity and the Global Connectivity connectivity metrics follow an inverse trend. An intuition for this behavior is that at lower sparsity ratios (i.e., 0.6 and 0.8) the MGE still strongly connected, thus its connectivity is better captured by nodes' degroe information e Strength and Neighbor Connectivity metrics). Conversely, high sparsity regimes, the network starts be more disconnected, and here both Local and Global Connectivity have primary role in explaining occ. Importantly, these observations
Local Connectivity metrics represent a successful extension to Vysogorets & Kempe (2023); Pham et al. 2023), which introduce "effective" nodes and paths. It is, in fact, easy to see the similarities between sink, source, and "effective" nodes, as well as those wetween the removable in/out connections and "effective" paths. Interestingly, the correlation of the Expansion topometrics roughly remains low, without a clear pattern across sparsity ratios.
0.6
0.9
0.95
0.98
k-00re
Topometrics
out-edges cul-nodes
0114
F-U
0.25
0.00
0.25
0.50
-0.25
0.00
0.25
0.50
0.25
0.00
0.25
0.50
-0.25
0.00
0.25
0.50
-0.25
0.00
0.26
0.50
Figure 3: Pearson correlation coefficients between 4 ace and each topometric (Sparsity-Fixed scenario).
Architecturo-Fit Fixeed. This scenario is analyzed in Figure from which we can make key observation: different architectures lead to striking differences in feature importance. Hence, informative GEs, from which diverse topometrics can be extracted, are essential for the analysis. For instance, in every Resnet extension (i.e., 20 and 32) the Local Connectivity metrics correlate strongly with the accuracy drop. In wider and more over- parameterized architectures (i.e., Conv-6 and SideeResnet--2-.. the importance of the Strength Connectivity and Neighbor Connectivity topometrics becomes prevalent. In contrast. to the Sparsity-Fixed scenario, Expansion topometrics generally present a strong, negative correlation with the accuracy drop. Since we do not only consider the Spectral Gap, but further examine the Spectral Radius, and compute both topometrics over the whole MGE, these findings successfully extend Hoang al. (2023a).
Conv-6
Resnet-20
Resnet-32
Wide Wide-Resnet
k-cor9:
mob
Topometrias
cut-edges cut-nodes
Caup
disc.
sink
-0.5
0.0
0.5
0.0
-0.5
Figure 4: Pearson correlation cocfficients between acc and each topometric (Architacturo-Fixedd scenario).
4.3 Ranking Pal algorithms via topometric mixture
In our previous analysis, we quantified the importance each topometric for predicting the accuracy drop of SNNs. We now use feature importance for a practical aim: ranking PaI algorithms. Ranking entails sorting a set of pruming algorithms according the expected performance of their SNNs, once fine-tuned. In light of the No- No-Free-Lunch Theorem for pruning (Frankle et al. 2021 given task, the optimal algorithm may save
11

---

Published in Transactions on Machine Learning Research (04/2024)
vast amounts of computational resources. If, instead, suboptimal Pal method is chosen, low performance is likely, and one may need to train new SNNs from scratch.
Given a (model, dataset, sparsity) triplet, denoted by (M,D,s) previous works (such as Pal et al. 2022
Hoang et al. (023b)) rank Pal methods according to a carefully chosen topometric 2, relying on its correlation lationation performance. However, this approach would discard a key finding from Section 4.2 namely that the importance of different topometrics varies with models and sparsity ratios. To close this gap; we propose to rank Pal methods with a topometric mixture. In detail, we propose to perform the ranking according to weighted sum of topometrics, with weights embodying proxies for topometric importance in the target scenario. Specifically, merge the importance of cach topomctric in the orthogonal Sparsity-Fixed and Architecture-Pie Fixed scenarios, which allows accounting for estimates of topometric importance given both a computational budget and network design. More formally, let x" E RR" denote the = 16 topometrics extracted from our MGE after encoding the SNN obtained with pruming algorithm p E P, packed into a single vector. Additionally, let wM E R" and w E R" be the estimated Architecture- Fixed and Sparsity-Five Fixed feature importance vectors. We then define the overall ranking coefficient of p as:
with
W
W 2
rP
x
(5)
Since w.M. and w are functions of the accuracy drop, we use the proposed ranking coefficient to sort Pal algorithms in ascending order. Hence, for any Pl,P2 E P, algorithm pl is preferred over p2 iff rP <
Basclines. We compare our ranking method to the following strategios: (1) the difference of bound (A,, Pal et al. (2022)) (2) the iterative mean difference of bound (Arinds, Hoang et al. (2023b)), and (3) the ilerative mean weighted spectral gap (Aamag) from Hoang et al. (2023a)- In light of the experimental results of Section 4.1 we additionally compare our ranking strategy to (4) the average layer-wise density. Note that we extract Ar, Arimdb and Ximag from the GEs used in the respective papers.
Experimental Sctup. For this analysis, we employ the four Pal algorithms listed at the beginning of Section 4 To measure the goodness of ranking, we use the Rank Biased Overlap (RBO) coefficient from Webber et al. (2010), which is computed as RBO b, = (1 - a) Σd= |dd-1le_anbss i.e., given any two ranked lists, this coefficient computes their similarity w.r.t. their length (d) and the order of the items. Here, a E (0, 1) is a real-valued hyperparameter weighting the contribution of the top-ranked elements. To avoid biasing the results with an explicit. choice for a, we average over evenly spaced values, i.e., over a [0.25, 0.50, 0.75]. Following the same rationale of Section 4.2.1. we further average over CIFAR-10, CIFAR-100, and Imageen
le.anbis)
Results. The ranking provided in Table shows that using the proposed topometric mixture outperforms all other ranking strategies on 3 out of 4 architectures (see the Avg. column). The only case where our proposed method struggles to reach the same performance the competitors is Conv-6. We hypothesize that the very shallow nature of this architecture, comprised of 6 layers only, decreases the significance of end-to-end topometrics, and, in general, of GEs. In fact, the trivial average of the layer- wise density is the best performer with this architecture. When ranking deeper models, however, our MGE and the proposed topometric mixture far surpass this trivial quantity, while also largely surpassing the other ranking stratogics. Finally, Table ! lets us conclude the analysis from Section 4.1 Ramanujan-based metrics (i.e., Ar, Arimab; Aimag) are seldom more informative than layer-wies wise density, the only exception being the case of 0.98.
5 Conclusions, Limitations, and Future Directions
In conclusion, in this paper, first identified and empirically verified the limitations of the existing layer- based GEs, as well as of the related Ramanujan-baser metrics. We tackled these drawbacks with a novel MGE, modeling the SNNs in an end-to-end and inpul-aware manner, along with a new set of lopological features for the grsph analysis of SNNs, We extensively studied the correlation between the proposed features and the accuracy drop of SNNs and showed that an effective strategy to rank Pal methods can be derived based on feature importance, paving the way toward the adoption of SNN-based GEs in practical use cases. We showed how topometric alone can accurately predict the performance of an SNN, but broader
12

---

Published in Transactions on Machine Learning Research (04/2024)
Table Mean RBO of the proposed topometric mixture (Ours) vs, existing ranking strategies. In total, each strategy must rank seven pruning algorithms, for all (model, sparsity) combinations, We provide average, per-model results the Avg. column.
Sparsity
Model
Strategy
Avg.
0.6
0.8
0.9
0.95
0.98
Jayer-Densiny
.56 +
.58 .34
.88
.05
.88
+ .06
.30
.44 + .29
.44 .29
.79
.02
+ .03
= .01
.67 .13
Conv-6
Arimdb
.49 = .26
.58
.22
+ .04
.80 = .02
.70 .14
Apnag
.17 = .00
.20
.58
.35
= .04
.84 = .00
.52 .09
Ours
.51 1 .29
.51 1 .28
.6
.01
.39 1 .01
.80 1 .01
.31 .12
Layer- Seyer-Density
.01
.31
.18 .06
+
.18 .01
.35
Ar
.91 + .06
.23
.15
.37
Resnet-20
Arsndd
+
.25
.37
Ximag
.91 + .06
.56 .55
.18 .99
= .41
.45 .16
Ours
.57 = .28
.92 .03
.07
.62 + .32
.92 = .07
.69 .15
Laycr-Density
.58 1 .33
.17 .04
11 .10
1 .04 1
.18 1 .01
.25 .10
Ar
.56 + .36
.16 .04
144 #
.40
.99 .18
Resnet-32
Arsmdi
.56 + -36
+
.37
Ximag
+
.77 .25
.49
Ours
.03
.60
.61
Layer-Density
.31 + .06
.3 .07
.48 + .33
.40 + .38
.25 = .11
.35 + .19
Ar
.28 1 .08
28 .08
27 .07
.00 1 .2
.11 1
.11
.25
.09
Wide-Resnet-28-2
Arimdb
+ .08
.99
24
1
11
24 .09
Xineg
+ .08
.69 .31
30
011
.11
.32
.11
Ours
.35 1
.07
graph-based viewpoint, based on multiple topometrics, turns out to be the key to success for predicting, and then ranking, Pal performance across different sparsity ratios and models.
Despite the many advantages, a limitation of our proposed GE is the time and space complexity for the bipartite encoding generation: for each layer of the MGE (L.e., for each BGE) the time complexity is O x Cout x step), while the space complexity is O(L + R + E), where step = (fe-des + 1)² and E Ci [W \ [0]] step, assuming square feature maps and kernels. Hence, when computing the BGEs concatenation to obtain the end-to-end MGF, the complexity is multiplied by the number of layers, plus any additional residual connection. Additionally, in this work we focused on the established experimental setups for the graph-based analysis of SNNs, thus examining Pal methods and CNNs, as in Pal et al. (0022) Hoang et al. (2023b). Nevertheless, analyzing different models, such as Transformers, and families of pruming algorithms will be needed to further enrich the field. Furthermore, the empirical evidence found in our observational study could provide elements for theoretical sludies about SNNs, as done for instance in Malach et (2200) Gadhikar et al. (0023). Finally, while devising pruning algorithms from graph-baser SNN representations (e.g., by optimizing certain topometrics according known sparsity ratio and model design) was out of the scope of this work, this may well be relevant rescarch direction.
References
Milad Alizadeh, Shyam A. Tailor, Luisa M Zintgraf, Joost van Amersfoort, Sebastian Farquhar, Nicholas Don- ald Lane, and Yarin Gal. Prospect Pruning: Finding Trainable Weights at Initialization using Meta- Gradients. International Conference on Learning Representations, 2022.
Vladimir Batagelj and Matjaz Zaversnik. An O(m) algorithm for cores decomposition of networks. arXiv preprint cs/0310049, 2003.
Alfred Bourely, John Patrick Boueri, and Krzysztof Choromonski. Sparse neural networks topologies, 2017. Nlv:1776.00683.
13

---

Published in Transactions on Machine Learning Research (04/2024)
Davide Chicco, Matthijs J Warrens, and Giuseppe Jurman. The coefficient of determination R-squared is more informative than SMAPE, MAE, MAPE, MSF and RMSE in regression analysis evaluation. PeerJ Computer Science, 7:e623, 2021.
Patryk Chrabaszez, Ilya Loshchilov, and Frank Hutter. A downsampled variant of ImageNet an alternative to the CIFAR datascts, 2017. mrXiv:1707.088111
Amin Coja-Oghlan. On the Laplacian cigenvalues of Gn, p. Combinatorics, Probability and Computing, 16 (6):923 946, 2007.
Pau de Jorge, Amartya Sanyal, Harkirat Singh Behl, Philip H. S. Torr, Grégory Rogez, and Puneet Ku- mar Dokania. Progressive Skeletonization: Trimming more fat from network at initialization, 2020. arXiv:2006.09081.
Tim Dettmers and Luke Zettlemoyer. Sparse Networks from Scratch: Faster Training without Losing Performance, 2019. wrXiv:1907.048400.
Utku Evei, Fabian Pedregosa, Aidan Gomez, and Erich Elsen. The Difficulty of Training Sparse Neural Networks. In ICML Workshop on Identifying and Underslanding Deep Learving Phenomena, 2019.
Utku Evei, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, pp. 2943 2052.952.2 PMLR, 2020.
Utku Evci, Yani Ioannou, Cem Keskin, and Yann Dauphin. Gradient flow in sparse neural networks and how lottery tickets win. In AAAI Conference on Artificial Intelligence, volume 36, pp. 6577-6588 2022.
Daniel Filan, Stephen Casper, Shlomi Hod, Cody Wild, Andrew Critch, and Stuart Russell. Clusterability in neural networks, 2021. XxXVv22003.03366
Jonathan Frankle and Michael Carbin. The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks. In International Conference Learning Representations, 2019.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Pruning Neural Networks at Initialization: Why Are We Missing the Mark? In International Conference on Learning Representations, 2021.
Advait Harshal Gadhikar, Sohom Mukherjee, and Rebekka Burkholz. Why random pruning is all we need to start sparse. In International Conference on Machine Learning, pp. 10542-10570. PMLRR 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving Deep into Rectifiers: Surpassing Huvan-Level Performance on ImageNet Classification. In International Conference Computer Vision, pp. 1026-1034, 2015.
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. IEEE/CVF Conference on Computer Vision and Pattern Recogrétion, pp. 770-778, 2016.
Duc N.M Hoang, Souvik Kundu, Shiwei Liu, and Zhangyang Wang. Don't 't just prune by magnitude! Your mask topology is a secret weapon. In Newral Information Processing Systems, 2023a.
Due N.M Hoang, Shiwei Liu, Radu Marculescu, and Zhangyang Wang. Revisiting Pruning At Tnitialization Through The Lens Of Ramanujan Graph. In International Conference Learning Representations, 2023b.
Christopher Hoffman, Matthew Kahle, and Elliot Paquette. Spectral gaps of random graphs and applications.
International Mathematics Research Notices, 2021 (11):8353-840))) 8404, 2021.
Shlomo Hoory. A lower bound on the spectral radius of the universal cover of a graph. Journal of Combinatorial Theory, Series B, 2005.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pp. 448-456. PMLR, 2015.
14

---

Published in Transactions on Machine Learning Research (04/2024)
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Master's thesis, University of Toronto, ON, Canada, 2009.
Steinar Laenen. One-Shot Neural Network Pruning via Spectral Graph Sparsification. In Topologicol, Algebraic and Geometric Learning Workshops, pp. 60-71. PMLR, 2023.
Namhhon Lee, Thalsiyasingam Ajanthan, and Philip Torr. SNIP: single-shot network pruning based on connection sensitivity. In International Conference on Learning Representations, 2019.
Richard Lehoueq, Danny C Sorensen, and Chao Yang. ARPACK users' guide: solution of large-scole eigenvalue problems with implicitly restarted Arnoldi methods. SIAM, 1998.
Stratis Limmios, George Dasoulas, Dimitrios M Thilikos, and Michalis Vazirgiannis. Hcore-init: Neural network initialization based on graph degeneracy. In International Conference on Pattern Recognition, pp. 5852 5858, 2021.
Shiwei Liu, Tim Van der Lee, Anil Yaman, Zahra Atashgahi, Davide Ferraro, Ghada Sokar, Mykola. Pechenizkiy, and Decebal Constantin Mocanu. Topological insights into sparse neural networks. In Joint. Europeon Conference on Machine Learning and Knowledge discovery in databases, pp. 279 294. Springer, 2021a.
Shiwei Liu, La Yin, Decebal Constantin Mocanu, and Mykola Pechenizkiy. Do actually need dense teeepaarrameterization?? In-time over-paraaeeeriizationn in sparse training. In International Conference Machine Learning, pp. 6989-7000. PMLR, 2021b.
Shiwei Liu, Tianlong Chen, Xiaohan Chen, Li Shen, Decebal Constantin Mocanu, Zhangyang Wang, and Mykola Pechenizkiy. The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training. In International Conference on Learning Representations, 2022.
Eran Malach, Gilad Yehudai, Shai Shalev-Shwarzz, and Ohad Shamir. Proving the Lottery Ticket Hypothesis: Pruning is All You Need. In International Conference on Machine Learning, pp. 6682-6691, 2020.
Decebal Constantin Mocamu, Elena Mocanu, Peter Stone, Phuong H. Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature Communications, 2017.
Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparamelerizaatin.. International Conference on Machine Learning, pp. 4646 4655. PMLR, 2019.
Bithika Pal, Arindam Biswas, Sudeshna Kolay, Pabitra Mitra, and Biswajit Basu. A Study on the Ramamujan Graph Property of Winning Lottery Tickets. In International Conference on Machine Learning, pp. 17186 17201. PMLR, 2022.
Shreyas Malakarjun Patil and Constantine Dovrolis. PHEW: Constructing sparse networks that learn fast and generalize well without training data. In International Conference on Machine Learning, pp. 8432 8442. PMLR, 2021.
Hoang Pham, The-Anh Ta, Shiwei Liu, Lichuan Xiang, Dung D. Le, Hongkai Wen, and Long Tran-Thanh. Towards Data-Agnosti Pruning At Initialization: What Makes a Good Sparse Mask? In Neural Information Processing Systems, 2023.
Ameya Prabhu, Girish Varma, and Anoop Namboodiri. Deep expander networks: Efficient deep networks from graph theory. In European Conference on Computer Vision, pp. 20-35, 2018.
Victor M Preciado, Ali Jadbabaie, and George C Verghese. Structural analysis of Laplacian spectral properties of largo-scale networks. IEEE Transactions on Automatic Control, 58(99:2338-23) 2013.
Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, and Mohammad Rastegari. What's Hidden in a Randomly Weighted Neural Network? In IEEE/CVF Conference on Computer Vision and Poltern Recognation, pp. 11890-1 11899, 2019.
15

---

Published in Transactions on Machine Learning Research (04/2024)
Bastian Rieck, Matteo Togninalli, Christian Bock, Michael Moor, Max Horn, Thomas Gumbsch, and Karsten Borgwardt. Neural Persistence: A Complexity Measure for Deep Neural Networks Using Algebraic Topology. In International Conference on Learning Representations, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition, 2014. 3AXiv:1409.1566
Kartik Sreenivasan, Jy yong Sohn, Liu Yang, Matthew Grinde, Alliot Nagle, Hongyi Wang, Eric Xing, Kangwook Lee, and Dimitris Papailiopoulos. Rare Gems: Finding Lottery Tickets Initialization. In Neural Information Processing Systems, volume 35, pp. 14529 14540, 2022.
James Stewart, Umberto Michieli, and Mete Ozay, Data-Free Model Pruning at Initialization via Expanders. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4518 4423, 2023.
Julian Stier and Michael Granitzer. Structural analysis of sparse neural networks. Procedia Compuler Science, 159:107-116, 2019.
Julian Stier, Harshil Darji, and Michael Granitzer. Experiments on properties of hidden structures of sparse neural networks. In International Conference on Machine Learning, Oplimization, and Data Science, pp- 380 394. Springer, 2022.
Jingtong Su, Yihang Chen, Tianle Cai, Tianhao Wu, Ruiqi Gao, Liwei Wang, and Jason D Lee. Sanity- checking pruning methods: Random tickets can win the jackpot. In Newral Information Processing Systems, volume 33, pp. 20390-20401, 2020.
Hidcnori Tanaka, Daniel Kunin, Danicl Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. Neural Information Processing Systems, volune 33, pp- 6377-6389, 2020.
Konstantin Tikhomirov and Pierre Youssef. The spectral gap of dense random regular graphs, 2016. arXiv:1610.01765.
Fatemeh Vahedian, Ruiyu Li, Puja Trivedi, Di Jin, and Danai Koutra. Convolutional Neural Network Dynamics: Graph Perspective, 2021. S.ii:2111.05411
Fatemeh Vahedian, Ruiyu Li, Puja Trivedi, Di Jin, and Danai Koutra. Leveraging the Graph Structure of Neural Network Training Dynamics. In International Conference on Information & Knowledge Management, pp. 4545 4549. ACM, 2022.
Artem Vysogorets and Julia Kempe. Connectivity Matters: Neural Network Pruning Through the Lens of Effective Sparsity. J. Mach. Learn. Res., 24:99-1, 2023.
Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking Winning Tickets Before Training by Preserving Gradient Flow. In International Conference on Learning Representations, 2020.
Yite Wang, Dawei Li, and Ruoyu Sun. NTK-SAP: Improving neural network pruning by aligning training dynamics. In International Conference on Learning Represenlations, 2023.
William Webber, Alistair Moffat, and Justin Zobel. A similarity measure for indefinite rankings. ACM Transactions on Information Systems, 28( (8(4))1-38, 2010.
Sebastian Wernicke and Florian Rasche. FANMOD: a tool for fast. network motif detection. Bioinformalics, 22(9):1152-111) 1153, 2006.
Jiaxuan You, Jure Leskovec, Kaiming He, and Saining Xie. Graph structure of neural networks. In International Conference on Machine Learning, pp. 10881-10891. PMLR, 2020.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks, 2016. arXiv:1605.07146.
Hattic Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask. In Neural Information. Processing Systems, 2019.
16

---

Published in Transactions on Machine Learning Research (04/2024)
Michael Zhu and Suyog Gupta. To prune, not to prune: exploring the efficacy of pruming for model compression, 2017. arX1v:1710.01878.
17

---

Published in Transactions on Machine Learning Research (04/2024)
A Experimental Setup
A.1 Datasets
HFFAR-10/100 Krizhevsky et al. (2009). These two datasets are composed of 60, 000 color images each. These images are divided, respectively for the two datasets, into 10 and 100 classes. Each image has size of 32 x 32 pixels. On both datasets, 50, 000 images are used for training, while the remaining 10, 000 are used for testing purposes.
Tiny-ImageNet Chrabaszez al. (2017) The Tiny-ImageNet dataset consists of 100, 000 color images of size 64 x 64 pixels. These images divided into 200 classes, with each class containing 500 images. The dataset is further split into three sets: training set, validation set, and test set. Each class has 500 training images, 50 validation images, and 50 test images. In our experiments, we further downsample the images to 32 x 32 pixels in order lo use the same architectures for all datasets. The downsampling has been devised using the Box algorithm with the same approach used in Chrabaszcz et al. (011)).
A.2 Architectures
Conv-6. This model is scaled-down variant of VGG Simonyan & Zisserman (014 with six convolutional and three linear layers. Max-Pooling operation is employed every two convolutional layers. Batch norm llffe & Szegody (2015) is employed after every convolutional/Iinn /lincar layer. This model was devised for the first lime Frankle & Carbin (2019) and used as a benchmark in many recent studies on sparse models Zhouu et al. (2019): Ramanujan et al. (2019): Pal al. (2022). The total number of learnable weights in the model is 2.3M.
Resnet-20 Seineet-2/RReeeeee Resnet-32 This model is a modification of the classical ResNet architecture. Originally proposed in He et al. 20166). it has been designed to work with images of size 32 x 32 pixels. The implementation is based on public codebase 10 This model has been benchmarked in Frankle et al. (2021) ; Liu et al.
2022 Alizadeh et al. (002); Srrennivasan et al. (2022). The total number of learnable weights is 270K for
Resnet- 20 and 460K for Resnet-32.
Wide-Resneet-28-2 This model is modification of Wide-ResNet, and was originally proposed in Zagoruyko
& Komodakis (2016). The implementation is based public codebase") This model has been benchmarked
in Mostafa & Wang 2019) Dettmers & Zettlemoyer (2019) Frankle et al. (2021) h: Sreenivasan et al. (0022
The total number of learnable weights in the model is ~ 1.4M.
Table 5: Architectures of the models used in our experiments.
Layer
Conv-6
Resnet-20
Resnet-32
Wiide-Resne-2255
3 x 3, 16 Padding 1
3 3, 16 Padding Stride 1
3 3, 16 Padding Stride 1
Conv 1
Stride
x 3, 64
3 x 3, 16
3 x 3, 16
3 x 3, 32 3 3,32
Layer stack1
x
x
x
3 3, 64
3 16
3 x 3, 16
Mool
3 x 3, 128
3 3, 32]
x 3,32
3 x 3, 64
Layer stack2
x
x
3 128
3 3, 32
3 x 3,32
3 x 3,
Max-Pool
3 x 3, 256
3 64
3 3, 64
3 x 3, 128
x
x
Layer stack3
3 x 3, 256
3 3, 64
3 x 3, 64
3 x 3, 128
Max-Pool
Avg Pool
Avg Pool
Avg Pool kernel size
kernel size
kernel size
FC
256, 256, Melasses
64, Dellsses
64, Delasses
128, Melasees
htttps//pstrrykchraaezzzzz http://wwwwaaaaaaabbbbbb....b... http:///hhcccccccco..ooo//o/aaaa
10 httte:///ghuu..cc//stmtttts////s wttp:errccreetetttteeeeeeeeeeeee
18

---

Published in Transactions on Machine Learning Research (04/2024)
A.3 Hyperparameters
For the experiments with Pal and Layer-wise Random Pruning algorithms, well as for the dense baselines, for CIFAR-10 and CIFAR-100 we trained the models for 160 epochs with SGD with momentum 0.9 using batch size of 128. The initial learning rate set to 0.1 and decreased by 10 at epochs 80 and 120. Weight docay was set to 1 x 10-4 for all models but trained the model for 100 epochs. The initial learning rate was set to 0.1 and decreased by 10 at epochs 30, 60, and 80. Momentum and weight decay hyperparameters were set as for CIFAR-10 10AARR--1100.--
where it was set to 5 x 10-4 For Tmay-ImageNel,, we
100. This experimental setup is based on the ones used in Wang et al. (2020): Tanaka et al. (0200): Liu et al. 2022). In all cases, the weights have been initialized using the standard Kaiming Normal He al. (2015). Concerning the input data, random cropping (32 x 32, padding 4) and horizontal flipping have been used for data augmentation in all the experiments.
A.4 Pruning Algorithms
Details of the tested Pruning Algorithms are listed in Table 6 for Pal algorithms and in Table 7 for Layer-wise Random oncs.
Table 6: Pruning at Initialization Algorithms.
Pruning Method
Drop
Sanily Check
Training Data
Iterative
SNIP Lee al. (019
(oL(O)
GraSF Wang et al. (2020
H (()
X
Synflow Tanaka et al.
2020
R
1T 10
X
ProsPr Alizadeh al.
2022
(o.L((e))
Table 7: Layer-wise Random Pruning Algorithms.
A Agoorih
Layer-wise Sparsity
Uniform Zhu & Gupta (2017)
V E (0 N)
ER Mocanu et al.
2017
ERK
Evei et
2020
n Xn X.O XR
A.5 Implementation
In order to replicate the results of all the pruning algorithms used in this paper, we based our code the original PyTorch algorithm implementation whenever possible. The used sources are the following:
SNIP http:://gittabbcom///lllaaaaaa/
GraSP
httpss//gthuub.coo////ooa/ggggg
SynFlow httppp//gsttiiicoc//aaaaaa/a///s
ProsPR
Uniform, ER and ERK
On the other hand, for the IMDB metric and the rolled graph encoding Hoang et al. 2023b we relied on the official implementation available at htttps://git http///tttttuttt.//// www.VVTA--rooup........
an on pai
A.6 Models used in regression analysis of performance drop of SNN
In the performance drop analysis carried out in Section 4.2 we tested the topometrics over 8 different regressor model namely: ! Linear Model, ② Ridge, 3 Lasso, ElasticNet, 5 Huber, 6 Principal Component Regression, 7 Bayesian Ridge, and ( Automatic Relevance Determination (ARD).
19

---

Published in Transactions on Machine Learning Research (04/2024)
B Results
In this section, the results of the experiments explained in Appendix A are shown separately for each dataset, see Tables 8 10 Each combination (dataset, archilecture, sparsily, pruning algorithm) has been evaluated over 3 runs. In the tables, the symbol **" indicates that the sparse network is not able to outperform the random dense baseline (i.e., if the probability of correctly guessing sample class is I C the network after pruning has an average accuracy S )). In each lable, for any combination (architecture, sparsity).
Table 8: Accuracy results on CIFAR-10.
Saunly Rala
E55
18:
5XIP
P111
965444
STALRE
512.04
12.4455
085483
BOLES
663431
123//22
166433
17.2033
011:131
223:33
707283
523511
113505
881153
965633
86543
533805
S64683
36466
04463
02:002
123600
Uagan
411
523101
9.146
111111
825483
75342
547481
STALA
513411
003901
19300
96
93580
16:-03
558-00
115-55
087997
80-61
1833
012883
641901
013405
18:003
Table 9: Accuracy results on CIFAR-100.
Dmne
Hease-20
Homeel-33
Wiie VMe-Bome--552
GUALEA
16ALE1
111003
542183
83.01
30.31.3
90L04
63318.1
525134
25111
06113
634043
90.13
102.00
36133
TE5181
03113
03111
643383
MALE]
02305
193181
6831811
518125
122081
022144
153005
063124
6.101
6.2.1
30818
647183
BULLE2
124234
067111
153302
0.1153
BUICSS
112012
Table 10: Accuracy results on Tiny- ImageNe...
Conss
Boeash-32
Soundty Bate
E6
105
19
655
16
84
65
106
1.6
0.5
1190
0.54
11
15
11
135
1UB
SNI
A00
253093
30010 6
318115
10
00:10
591114
130551
37.500
Gud
412453
111
411484
212-12
118481
36045
2374LL
ITALE5
133333
241683
31.2000
4TH1
20033
84431
11-1A
113917
01661
W5111
411
03303
35307
35360
M3-64
1991
11055
03409
153005
11104
01141
255.02
153181
122233
211822
0134
15114
287235
C Graph Encoding Pooling Layers
Suppose the network has two consecutive layers 4 and 441 and a pooling operation P is employed between them. The two layers have respectively bipartite graph representation G; and Ge41. However, the MGE is not directly applicable since [Ra] * Li+1]- Nevertheless, we know a priori that [p(R,) |++1] This encoding extension aims at concatenating two layers R2 and L,+1 with different dimensions. To do so, we transform Lit1 to Li such that | |R&,l, to allow the nodes' concatenation between consecutive layers. Then, the edge construction is straightforward: each edge in Gi+1 which links u E Li+1 to v E Ri+1 is added [p(R:)] times, one for each node in the pooling window Pw C Ri which is computed over R& (since RG has the same dinension of L'+1 and the two have been concatenated) such that u E L is linked to v E R,+1- Then, based such pooling encoding, any pair of consecutive layers G¹, and G441 can be linked together in the MGE.
D Correlation Expander-related metrics and network density
we report the Pearson Correlation (r) between the Ramantjan- metrics and the Layer- Density. The correlation has been computed over the metrics on all layers of the given barchttcctures.spastty. considering as dataset CIFAR-10. It can be seen how the correlation for Ar Pal et al. (2022) and Arimah Hoang et al. (2023b) is almost always above 0.9, with several peaks at ~ 1, while it is slightly lower for Ximeg Hoang et al. 2023a). To note that empty entries refer to SNNs that cannot be analyzed using the Ramanujan- based metrics, since in those cases the graph representation violates the Ramanujan constraints of min(di, d,) > 3 Hoang et al. (022bb for every layer.
In Figure 5
20

---

Published in Transactions on Machine Learning Research (04/2024)
Layer-Density Density Ar
Leyer-Dereily dlasb
Layer Density
Aueso
Plcanst
Ploret
Bearet-2 S1-32
Resneid0
Reyoer-P
6:4E
300
0.4
8:0
000
507
040
CHI
5:0
0.74
6:99
060
559
0.0
CU7
355
0.0
6.6
070
Pruning Atporitinn
Pruning Agoerre
Pruning Aggoittiin
Pruning Algon Pm
Pruting Algothm
Pruning Aigortin
(a) Ar Pal al. (2022)
(b) Ar Armmdb Hoang et (2023b)
(c) Ainag Hoang et al. 12023a
Figure Pearson Correlation (r) between Ramantjan-based metrics and Layer-Density
E Rank Prediction
In this section, include the numerical values (in terms of mean RBO) of the rank prediction devised with topometric mixture, separately for Pruning at Instialization and Layer- Wise Random Pruming algorithms. Table 11 shows the results for the Pal case, where we see that the numerical values follow the same pattern presented in the main text, i.e., our proposed topometric mixture outdoes the other metrics by 30%. On the other hand, in Table 12 we report the results for the Layer-Wise Random Pruming algorithms, where no clear better approach visible. The numerical values of the mean RBO turn out to be high for almost all metrics (ours included), which indicates almost perfect matching the ranking prediction.
Table 11: Mean RBO of the proposed topometric mixture (Ours) vs. existing ranking strategies. In total, each strategy must rank four Pruning at Initialization algorithms (see Scction 4 for all (model, sparsity) combinations. We provide average, per-model results in the Avg. column.
Sparsity-wise
Model
Encoding
Avg.
0.6
0.8
0.9
0.95
0.98
.60 + .26
.62 + .34
.91 + .05
.90 + .06
.69 = .29
.74 + .20
Ar
.60 + .26
.62 .34
.91 - .05
.86 + .00
.8 + .00
.77 + .13
Conv-6
Arimab
.60 +:2 .26
.62 + .34
.91 + .05
.86 + .00
.66 + .00
.77 + .13
Ximag
.60 +:2
.62
.34
.91 .05
.06
+ .00
+ .14 .54 .14
Ours
.54 + .28
.53 + .29
.37 + .02
.34 + .07
.93 + .06
Layer-Density
62 + .34
.53 .29
.36 + .00
.36 + .00
.35 1 .08
.44
Ar
.42 1 .07
.66 .10
.66 + .00
.26 + .00
.33 1 .06
.33 .05
Resnet-20
Arsmth
.42 + .07
.6 110
.26 - .00
.26 .00
.33 = .06
33 + .05
Xamag
.42 + .07
.6 + 110
.26 - -00
.29 + .06
.3 + .06
33 + .06
Ours
.90 + .06
.54 .31
.2 + .28
.93 + .06
.53 + .29
.74 + .20
Layer-Densee
.55 + .30
.50 .43
.6
.66 + .00
.42 .16
Ar
.72 + .28
.54 + .31
.99 + .06
.26 .00
.26 + .00
.1 .13
Resnet-32
Arimab
.72 +
.29
.26 +
.26 + .00
37 .08
Aimeg
.59 + .33
.50
.34
.29 1 .06
.36 1 .00
.36 1 .00
422 .15
Ours
.59 +: .36
.33 + .06
.93 + .06
.97
.73 .32
.71 .16
Layer-Density
.61 + .22
.44 + .04
.76 + .35
.53 .29
38 + .02
.54 .88
Ar
1 .02
.99
.29 1 .06
34 .03
Wide-Resnet-28--
Arimd
.37 + .02
37 4: .02
.33 - .06
29 .06
.29 1 .66
33 .04
Aumag
.37 + (12
.37
.29 .06
33 + .06
34
Ours
.63 + .29
.82 .31
.66 + 299
.73 .29
.94 + .07
.74 .25
21

---

Published in Transactions on Machine Learning Research (04/2024)
Table 12: Mean RBO of the proposed topometric mixture (Ours) va, existing ranking strategies. total, each strategy must rank three Layer- Wise Random Pruning algorithms (sce Section for all (model, sparsity) combinations. We provide average, per-model results in the Avg. column.
Sparsity-wise
Model
Encoding
Avg.
0.6
0.8
0.9
0.95
0.98
Layer-Density
.04 = .00
.40 + .00
.56 +: .29
1.0 + .00
.67 = .29
.61 = .12
Ar
.40 + .00
1.0 .00
.67 .29
Conv-6
Aimu
.97 + .06
.97 + .06
.43 i .06
1.0 1 .00
.67 + .29
.81 + .09
Xinag
.40
.40 + .00
.56
.29
.83 .29
.63 .32
.56 .18
Ours
.97 + .06
.97 + .06
.43 + .06
.83 + .29
.67 + .29
.77 = .15
Layer-Densic
1.0 .00
1.0 + .00
.83 1 .29
.67 + .29
1.0 + .00
.90 - .12
Ar
1.0 .00
1.0 1 .00
.83 + .29
.50 + .00
.83 + .99
.83 .12
Resnet-20
Aimdb
1.0 + .00
1.0 + .00
.83 1 .29
.67 + .29
.83 + .29
.87 = .17
Aimsg
1.0 .00
.67 + 29
1.0 + .0
.67 + .29
.67 = .29
.80 = .17
Ours
.83 .99
1.0 + .00
.67 + .29
.67 + .29
1.0 + .00
.83 = .17
Layer-Density
.83 .99
+ .29
1.0
.00
+ .00
.83 .29
.90 .17
Ar
83 29
.83 + .29
1.0 + .00
.83 + .29
1.0 + .00
.90 .17
Resnet-32
Asmdl
=
1.0
.00
.83 29
.00
.90 .17
Ximeg
1.0 - .00
.83 + .29
1.0 + .00
.83 + .99
1.0 + .00
.93 = .12
Ours
1.0 .00
.67 + .99
.83 .29
.83 .99
Layer-Density
.83
.29
1.0 +
.83
.29
1.0 + .00
.50 .
.83 .12
Ar
.83
.29
1.0 + .00
.83
.29
.83 + .29
.83 + .29
86 = .3
Wide-Resnet-28-2
Aumdb
.83
.29
1.0 + .00
.83
.29
1.0 + .00
.83 + .29
.90 .17
Ximag
.83 + .29
1.0 + .00
.83 + .29
.83 + .99
.67 + 29
.83 = .3
Ours
.83 .29
1.0 - -00
.67 = 29
.83 = .29
.67 = .29
.80 = .23
22


Reviews:
[{"review_id": "x6qRB5kD6B", "reviewer": "Reviewer_782F", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n1. This paper challenges a commonly used heuristics of using Ramanujan graph metrics to predict the performance. Results of this paper (Section 4.1) show that Ramanujan graph metrics are not so better than graph density while requiring more computation costs. \n2. This paper is generally well written and easy to follow. As a non-expert in sparse neural networks, I can follow the main ideas of this paper easily. \n3. Experiments are extensive. The authors test different neural architectures, datasets, and sparse training methods with pretty consistent results. \n4. As a non-expert of sparse neural networks, I like the motivation of this work to better predict the performance of an SNN from some general structural metrics. It provide insights about how the sparse training algorithms should be designed. \n\n## Weaknesses\n1. It seems that the proposed MGE contains a significantly larger number of nodes than single layer ones. Therefore, I am wondering what is the price we pay for generating such a large graph and calculating these structural metrics. Maybe the authors can shed some lights by giving some computation cost analysis (especially, calculating the global connectivity and expansion of a very large graph takes some time). \n2. The authors mention supporting residual layers in Section 3.2. While I can somehow roughly figure out the procedure, it is always better to show how to explicitly include residual layers into the analysis, as residual layers are important components in modern neural networks. \n3. Section 4.1 can be enhanced with numeric correlation values. Although visually, these values do show the same trend, it is always better to give some numeric values. \n4. In Section 4.2, the authors say that 'compute the pearson correlation coefficient between each topometric and the output of each regressor\" to assess the importance of each topometric. I am wondering why not just use the regression coefficient? Also, regarding the section, I am wondering whether there are variances between different datasets (in terms of regression accuracy). If the variance is small, it would better enhance the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a method to predict the performance of sparse neural networks. The proposed method involves a multipartite graph construction of the layers, which allows an end-to-end modeling of a sparse neural network instead of a single layer. The proposed method also analyzes the impact of multiple graph topology metrics that are derived upon the multipartite graph to see how these metrics can predict the performance loss of sparse neural networks.  Experimental results suggest that the proposed MGE graph encoding leads to structural metrics that can better predict SNN performances, while metrics resultant from bipartite graph encoding (with only a single layer) cannot.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the Weaknesses 1 through 4.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fbK6Ejhqyv", "reviewer": "Reviewer_deP5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is very well written and easy to follow. It explains the necessary basics to understand the setting and context at an appropriate level of detail, and clarifies the proposed method and metrics very clearly and as far as I can see also technically correctly, adding also more appropriate imagery and free text description to help the reader to build the appropriate intuition. \n\nThe experiments are well chosen and well designed to document the performance and the results are well communicated to allow the reader to pick the most important messages.\n\nWhile the MGE construction seems relatively simple and the proposed metrics are (I believe) standard in graph analysis, I can see the interest of the community in exploring these for the SNN analysis.\n\nThe only weakness is in the description of the topometrics: I presume these are mostly standard in graph analysis. I would like to see this stated more clearly with providing the appropriate references. Also, apart from mentioning the FANMOD algo for Motif, there is not mention of the algorithms to get these graph metrics and their respective complexity. The complexity is mentioned as one of the limitations of the work in section 5 but it is not clear where do these numbers come from - which of the topometrics? I would like to see the connection more clearly.\nTogether with a missing link to a code-base (is this because of the anonymization of the submission) this reduces the reproducibility of the results and possibility for future re-use. I would strongly encourage the authors to include a link to their code-base in the final version.\n\nA few minor comments are listed in the Requested Changes but these are all fairly cosmetic and should be easy to include into the final version.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes to use a set of graph topological metrics calculated from a mutlipartie graph encodening (MGE) of a neural network to analyse and predict performance of pruned-at-initialization (PaI) sparse neural networks (SNNs). The MGE itself is also new and is build from an unrolled (input-aware) bi-partie graph encodings. The MGE construction is covered for MLP and CNN architectures including residual connections and pooling operations, architectures such as transformers remain for future work. The newly proposed topometrics are analysed in a set of experiments and compared to the previously proposed Ramanujan-based metrics and simple layer-density concluding that Ramajuan-based metrics are rarely more informative than the simple layer-density and that the proposed topometrics can provide more valuable information to understand the pruning effects.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see weaknesses paragraph before: complement topometrics description and provide link to code-base.\n\nMinor change requests:\n- What happens in your BGE and MGE if padding is not zero? Would these padding nodes be included in the graphs? Please clarify.\n- section 3.2, second paragraph, second line, definition of $\\hat{E}$ should it read $i \\neq j$ or rather $i < j$?\n- section 3.2, second paragraph, pooling operation graph - you refer to Appendix C. Please give at least a brief description here to help the intuition without the need to read the appendix.\n- section 3.2, second paragraph, residual connections - how are these included in the graph? This is probably trivial but a diagram would help.\n- r-out and r-in - are this no. (as in counts) or rather ratios? Please correct if these are ratios.\n- source nodes - Do I understand correctly that these can only exist in the 1st layer? Or can they appear in some later layers due to the sparsifying operations? Please comment in the text to help the reader.\n- neihbourhood connectivity - are the k-hop neihborhoods directional?\n- Expansion - formal definitions of eigenvalues $\\lambda(\\mathcal{L}_G) = \\{ \\ldots \\}$ has a typo in the last element of the list. Should be $n$ and not $0$.\n- Spectral gap - is this correct? Isn't the $\\hat{\\mu}$ simply the 2nd biggest eigenvalue and hence $\\mu_1$?\n- section 4.1 - network density is introduced as a term which is, however, never explained. I could only understand what it is when coming to page 8 which says under Ramanujan-based metrics paragraph that it is the fraction of active parameters. \n- section 4.2.1 - I believe this is a standard linear ordinary-least squares regression model. Please state it clearly to avoid confusion. \n- Page 11 last paragraph - please spell out RBO before using the abbreviation.", "broader_impact_concerns": "No concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "oljbvIPq2q", "reviewer": "Reviewer_nJmv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n- This paper focuses on the interesting research problem of how the topological metrics of sparse neural networks (obtained by prune-at-initialization methods) are related with the performance of such pruned network. \n- The idea of using unrolled input-aware MGEs makes sense. \n\nWeakness\n- The pearson correlation in Fig.3 and 4 is positive for several metrics, but we also have some metrics with negative correlation. Do we have some guideline on which metric we should rely on?\n- Although this paper has a through experimental results, it is not based on some theoretical analysis. \n- The claim of this paper is bit unclear. Are the authors claiming that \"Regarding SNNs obtained by Pruning at Initialization methods, metric XXX is a good indicator of the performance of the SNN\"? What is XXX in this claim?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focuses on the relating the performance of pruning at initialization schemes, with the metrics developed in graph theory. The authors discovered that the existing Ramanujan-based metrics (obtaining which requires high computational complexity) often do not provide more information than the readily-available layer-wise density metric. Motivated by this finding, the authors consider measuring the metrics for unrolled input-aware Multipartite Graph Encoding (MGE) setup, which is empirically shown to have high correlation with the accuracy drop of sparse neural networks (SNNs).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I guess the theoretical analysis is bit hard to cover, but giving some insight on the experimental results is required. Moreover, the clarification of the results and conclusion is needed.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers appreciated the direction of using topological metrics to better understand and predict the performance of sparse neural networks (SNNs), as well as the proposed MGE methodology and the extensive experiments conducted. They listed as strengths the fact that the paper is challenging common metrics, and single-layer approaches; it is proposing a simple but effective idea with MGE. 

At the same time, they highlighted a number of issues, mostly dealing with presentation. Specifically, they asked to revise the description of the topological metrics to make them more clear for the TMLR audience, and list the complexity of the discussed approaches. Other criticisms regarded missing details from the experiments and on how to interpret results in tables. 

Authors exhaustively answered all the concerns during the rebuttal and provided a revised version of the manuscript that satisfied all the reviewers. As such, the paper is accepted as is.
Decision: Accept (poster)