The Graphon Limit Hypothesis: Understanding Neural Network Pruning via Infinite Width Analysis
Authors: Hoang Pham
The-Anh Ta
Tom Jacobs
Rebekka Burkholz
Long Tran-Thanh
Venue: NeurIPS 2025 spotlight ()
Keywords: Pruning Network
Graphon
Neural Tangent Kernel
Primary area: theory
TL;DR: We study the training dynamic of sparse neural networks through the lens of Graphon where masks of pruning methods converge to graphons as networks' width approach infinite

Abstract: Sparse neural networks promise efficiency, yet training them effectively remains a fundamental challenge. Despite advances in pruning methods that create sparse architectures, understanding why some sparse structures are better trainable than others with the same level of sparsity remains poorly understood. Aiming to develop a systematic approach to this fundamental problem, we propose a novel theoretical framework based on the theory of graph limits, particularly graphons, that characterizes sparse neural networks in the infinite-width regime. Our key insight is that connectivity patterns of sparse neural networks induced by pruning methods converge to specific graphons as networks' width tends to infinity, which encodes implicit structural biases of different pruning methods. We postulate the *Graphon Limit Hypothesis* and provide empirical evidence to support it. Leveraging this graphon representation, we derive a *Graphon Neural Tangent Kernel (Graphon NTK)* to study the training dynamics of sparse networks in the infinite width limit. Graphon NTK provides a general framework for the theoretical analysis of sparse networks. We empirically show that the spectral analysis of Graphon NTK correlates with observed training dynamics of sparse networks, explaining the varying convergence behaviours of different pruning methods. Our framework provides theoretical insights into the impact of connectivity patterns on the trainability of various sparse network architectures.

The Graphon Limit Hypothesis: Understanding Neural Network Pruning via Infinite Width Analysis
Hoang Pham', The-Anh Ta², Tom Jacobs Rebekka Burkholz', Long Tran-Thanh'
1 University of Warwick, 2 CSIRO's Data61, CISPA Helmholtz Center for Information Security
hoang. pham@warwick.en
boang.phamovarwickcc ac. uk, theanh. ta@csiro... au, tom tom. www.jjaccisssa....o..
burkholz@cispa..d, long. tran-thanh@warwick ac. uk
Abstract
Sparse neural networks promise efficiency, yet training them effectively remains fundamental challenge. Despite advances in pruning methods that create sparse architectures, understanding why some sparse structures are better trainable than others with the same level of sparsity remains poorly understood. Aiming to de- velop a systematic approach to this fundamental problem, we propose a novel theoretical framework based on the theory of graph limits, particularly graphons, that characterizes sparse neural networks in the infinite-width regime. Our key insight is that connectivity patterns of sparse neural networks induced by pruning methods converge to specific graphons as networks' width tends infinity, which encodes implicit structural biases of different pruning methods. We postulate the Graphon Limit Hypothesis and provide empirical evidence to support it. Leveraging this graphon representation, we derive Graphon Neural Tangent Kernel (Graphon NTK) to study the training dynamics of sparse networks the infinite width limit. Graphon NTK provides a general framework for the theoretical analysis of sparse networks. We empirically show that the spectral analysis of Graphon NTK corre- lates with observed training dynamics of sparse networks, explaining the varying convergence behaviours of different pruning methods. Our framework provides theoretical insights into the impact of connectivity patterns the trainability of various sparse network architectures.
Introduction
Deep neural networks have achieved remarkable success wide range of machine learning tasks, [7]. and scientific modeling [38]
including computer vision 33 18] natural language processing 63
key ingredient this success overparameterisation 4
9
37 5 networks are often trained with
many parameters than are strictly necessary, which facilitates optimization and generalization. However, this overparameterisation introduces substantial computational and memory overheads 25 61 56 69), hindering deployment in resource-constrained environments such as mobile/edge devices, embedded systems, or real-time applications.
To address these challenges, network pruning has emerged a fundamental approach for compressing deep networks by removing redundant weights [4 31 23 25). By identifying and eliminating parameters that contribute little to the model's output, pruning can produce sparse subnetworks that retain high performance while offering significant efficiency gains. Empirically, the Lottery Ticket Hypothesis (LTH) 24 demonstrates that randomly initialised dense networks contain sparse subnetworks (winning tickets) that, when trained isolation, can match or exceed the performance of the original network. This observation has sparked significant research efforts including iterative pruning methods for finding such winning tickets 24
[4
13
28
post- trainini pruning approaches and dynamic sparse
29 various Pruning at Initialisation (Pal) techniques
43
66
56
69
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

training [5] 21 44 45]. Parallel to this empirical progress, theoretical work has sought to prove
the existence of effective sparse neural networks 49 TT [O]. Other works [4% [2] attempt to
understand the effectiveness of sparse neural networks by analyzing gradient flow during training. Through the lens of the Ramanujan graph, (4 7 [5] indicate that maximizing the graph connectivity
of sparse networks improves the performance of subnetworks. Recently, 774) have studied the training
dynamics of randomly pruned networks via NTK in an infinitely wide neural network setting.
Despite these advances, a comprehensive theoretical framework for understanding neural network pruning and the training dynamics of general sparse neural networks remains elusive. A particularly challenging aspect is explaining why sparse networks are often difficult to train effectively [26]. In this paper, we aim to develop a systematic framework for analysing sparse networks obtained by Pal methods. While NTK and other large width limit approaches [37 S 42 provide valuable tools for analysing dense neural network training, they are not suitable for the sparse network settings due to a fundamental difficulty of defining the limit of sparse networks of different sizes.
To address this problem, we develop a novel approach by leveraging graph limit theory. In particular, a pruning method produces binary masks that determine the active connections between neurons in layers. These masks can naturally be interpreted as the adjacency matrices of graphs defined over neural network layers. As the width of the network increases, the size of these adjacency matrices grows, and their structure becomes more regular and well-defined. In this view, the sequence of pruning masks generated by increasingly wide networks forms a sequence of graphs.
We hypothesize that, in the infinite-widhh width limit, this sequence of graphs converges to graphon that serves as a limit object for dense or structured sparse graphs 47 g. Informally, any large graph can be viewed random sample from some underlying graphon W(u, v) encoding the probability of an edge between node indices u and (57). We propose the Graphon Limit Hypothesis for sparse neural networks obtained by a given pruning method in the large width limit and provide empirical evidence to support it. Informally, the hypothesis states that:
Given a sparsity level, each network pruning method defines sequences of binary masks (M,²) that converges layer-wisely to gruphons W in cut distance as the network width n -> Oc.
This new perspective allows us to unify diverse pruning techniques under a common mathematical framework. For example, Random Pruning at initialisation produces subnets whose configuration is an Erdös-Rényi random graph, hence its graphon limit corresponds to a constant graphon, while other Pal methods produce graphons with different patterns. Crucially, these graphons can be used to define an infinite-width model, analogous to how fully-connected networks yield NTK in the limit 37 S 42). We formalise this idea by introducing the Graphon Neural Tangent Kernel (Graphon NTK), kernel that captures the infinite-widie width behaviour of pruned network specified by graphon.
Our framework offers a new approach for analysing sparse neural networks through their graphon limits, provides theoretically grounded tool for understanding how pruning affects training dynamics, and offers a principled basis for comparing or designing pruning strategies via their associated NTK behaviour. This framework generalises previous work on randomly pruned networks [74] where Random Pruning corresponds to constant graphons. Our contributions are summarised as follows:
We introduce the concept of graphon limits of pruning masks and propose the Graphon Limit Hypothesis that each pruning method corresponds to a distinct graphon in the infinite- width limit, supported by empirical evidence showing that pruning masks exhibit structural convergence and generate characteristic graphons.
We formalise the Graphon NTK, new kernel framework that combines pruning structure with infinite-width analysis, and show how can be derived from the graphon representation of the mask, highlighting the key differences between NTK and Graphon NTK.
We empirically establish a connection between the spectral properties of the Graphon NTK and training dynamics of sparse networks, providing theoretical insights into how connectivity patterns affect the trainability of sparse networks.
The remainder of this paper is organised as follows: Sections 2 3 discuss related works and provide preliminaries on graph limits, NTK, and pruning methods, respectively. In Section 4 we formalise the Graphon Limit Hypothesis and provide empirical evidence for its validity. Section derives the Graphon NTK and its special case of Random Pruning. Section pressents our experimental results. Section 7 concludes with summary of our contributions and directions for future research.

---

2
Related work
Neural network pruning. Pruning has been extensively studied as means of reducing the com- putational and memory demands of deep networks. Early works propose magnitude saased pruning. where weights with the smallest absolute values are removed after training [41 32 31 The Lottery Ticket Hypothesis (LTH)) 23 represents a paradigm shift by demonstrating that dense networks contain sparse subnetworks that can be trained in isolation to match or exceed the performance of the original network. Follow-up works expand this observation across architectures [14] [3] and training regimes [24] 28). while theoretical analyses [49] [5] [53 I0 [ seek to establish formal conditions under which such "winning tickets" exist. However, finding these winning tickets typically requires computationally intensive iterative srain-prune-reraile cycles. To address this computational burden, Pruning at Initialisation (Pal) methods have emerged, which identify sparse subnetworks before training begins. Subnetworks can be found based on the magnitude, gradient, or hessian information T6 3 or WTTK-based scores (46 54 30 67], while other methods 662 36 69 try to identify these tickets based on the subnetworks" configuration only. Despite these advances, analyses of why different pruning methods yield varying performance at the same sparsity level remain largely empirical. Works such [48] [2] (6) have analysed gradient flow in sparse networks, while [74] examines the NTK limit behaviour of randomly pruned networks. Our work advances these efforts by providing a theoretical framework for understanding the limiting behaviour of pruned networks as their width increases.
43 65
Graphon and graph limit. Graphons are limit objects of graph sequences and have become central tool in the study of large networks 8]. A graphon is a symmetric measurable function W: [0, 1]2 - [0. 1] that can be viewed the continuum analogue of adjacency matrix. Graphons have been widely used in, e.g., modelling social networks [60 588, and community detection [39
In machine learning, graphons have found use in graph neural network (GNN) analysis. Graphon neural networks extend traditional GNNs by considering their behaviour in the limit of large graphs [5 52 50 34). In particular, graphon neural networks operate on functions over [0, 1] rather than on inite-dimension
vectors, which makes it possible to model infinitely large graphs or to generalise across graphs of different sizes. Another line of work focuses on learning graphon from collection of observed graphs 12 77 68). Specifically, to estimate graphons, SAS [12] reorders adjacency matrices by degree before applying smoothing, SBA (2 fits a stochastic block model to the graph data, GWB [70] relaxes the cut distance by wasss--Wassersteinn distance and minimizes this distance between observed graphs, and IGNR [68] directly uses neural networks. However, to the best of our knowledge, no prior work has connected graphons to the behaviour of pruning methods in neural networks. Prior studies on pruning GNNs [15 [59 75 focus on removing edges in input data graphs to enhance computational efficiency or task performance. In contrast, we study pruning within the model's own weight graph, which form an implicit connectivity graph within the neural network architecture itself. Our work introduces novel interpretation: pruning masks of increasingly wide neural networks define a sequence of binary graphs that converge to graphons, and these graphons encode structural priors in the fffnite-width limit.
Neural tangent kernel and widtnite-width limit. The neural tangent kernel (NTK) 37 S
characterises the training dynamics of infinitely wide neural networks under gradient descent. In this regime, training a network becomes equivalent to solving a kernel regression problem using the NTK. The NTK has since been extended and studied across wide variety of architectures, including CNNs, RNNs, transformers, GNNs [5] 71 72 40 20 Among these, Graph Neural Tangent Kernels (GNTKs) 200 allow study on the limiting behaviour of infinitely wide (the number of features) GNNs trained via gradient descent. Later, 44) demonstrates that as the size of the graph increases, the GNTK converges to graphon neural tangent kernel While most NTK analyses assume dense architectures, recent studies have begun to explore sparse NTK models. [74 shows that Random Pruning preserves the NTK in the infinite-widtth limit up to a scaling factor, with convergence improving at larger widths. In particular, they apply pre-defined random masks on layers' weights to derive the NTK. Separately, [73] analyses sparsity induced by large bias initialisation, demonstrating that it leads to structured sparse activations and improved NTK conditioning, yielding faster convergence and tighter generalization bounds. Our work extends this line of research by developing framework for analysing arbitrary pruning methods through their limiting graphons. This enables a more nuanced understanding of how different pruning strategies affect network trainability in the large width limit.
'This kemel, also named Graphon NTK in 40 is for graph data and is different from our kemel.
3

---

3 Preliminaries
3.1 Neural tangent kernel
The neural tangent kernel (NTK) 37 19 5
provides a theoretical framework for understanding
the training dynamics of overparameterized neural networks. For network f(;;0, 0) mapping input I E Rd to output in IR, the NTK is defined as: (zz x²) = (of(x, O) Tof 0)..
In the limit where the width of each hidden layer tends to infinity and the parameters are initialised randomly (e.g.. with Gaussian scaling), the NTK converges to a deterministic kernel Oo(a, x²). independent of training: for any given / 0, Ox x²) (0(2, xx), as π oo. More recent works have shown that even networks with finite width, and any depth follows the NTK behaviour (5 42 Under gradient flow on the mean squared error loss, the predictions evolve according to a linear differential equation, which admits the solution: f(X;0,) = J(X: Og)e-m8u + (I e: - 9.
where is the training label. This formula shows that the network predictions interpolate exponentially toward the training labels g. with convergence behaviour governed by the spectral properties of Og. In particular, convergence is faster along directions corresponding to larger eigenvalues of the NTK.
Different architectures yield different limiting NTKs. For example, fully connected ReLU networks admit slosed-form recursive formulas for the entries of O, [37]. while convolutional and residual architectures induce structured NTKs that encode translation invariance and hierarchical composition 5 Thus, the NTK framework bridges deep learning and kernel methods, offering theoretical insights into optimisation and generalisation.
3.2 Graphon and graph limit theory
Graph limit theory provides an analytic framework for analysing large graphs. For a sequence of graphs with an increasing number of nodes (G,), one can often associate a limit object known as graphon which is symmetric, measurable function W: [0, 1|2 - [0. 1] that encodes the connectivity pattern between infinite set of nodes [47 The function value W(x, y) represents the probability of an edge between nodes indexed by x and in the limit. Graphon theory is particularly useful for capturing the limiting behaviour of a graph sequence (G,) as the number of nodes + 0o.
Each finite graph G, with n vertices can be represented (up to relabelling) by step function WG.. on [0, 1/2 where the unit interval is partitioned into equal parts and edge presence is encoded by: f-11
1=1 1)), with A being the adjacency matrix of G., A sequence (G.) is said to converge to graphon W if the sequence (Wa.) converges to W in the labelled cut distance:
Wo., (3r, (2r,w) v) Aij, V(, v)
V (u(x,y) y) W(x, v))d
du(,, sup S.TC[O.1] SXT
where 44 and W are two graphons, and the supremum is taken over all measurable subsets S, T C [0, 1]. To account for vertex relabelling, we define the cut distance between two graphons up to measure- mncasure-preserving transformations. Let & be the set of all measure- preeuurr- bijections ø 10, + [0, 1]. The (label-invariant) cut distance between graphons U and W is modified be: (1 W) = inf dn(,, W where W* y) W(o(z), @(v)). A sequence of graphs (G.)) with [V) + oc is said to converge to a graphon W if lim,- (nW... W) = O.
The cut distance quantifies the similarity between two graph adjacency matrices or functions by considering all possible labellings of their nodes 47 Intuitively, two graphs are close in the cut distance if their global connectivity structures are similar. An equivalent characterization of the convergence of a graph sequence (G,,) to a graphon is based on homomorphism counting 88 57). Particularly, for every finite subgraph F. the homomorphism densities t(F, Gn) converge to t(F,, W). The homomorphism density t(F, G)) represents the probability that a random map from the vertices of F those of G preserves all edges.
4 Graph limit and sparse neural networks
Neural network pruming produces binary masks over the network's 's layers to specify which weights removed from each layer. These masks naturally define bipartite graphs between adjacent layers, with the mask matrix (1) {0, 1) m acting as the biadjacency matrix. We hypothesize that

---

pruning-induced induced structures can be modelled in the infinite- width limit using graphons, enabling unified geometric and functional interpretation of sparse neural networks.
4.1 Pruning masks graphs
In a fully connected network, each neuron in layer - is connected to every neuron in layer 1, resulting in dense weight matrix W()) q A pruning method replaces this weight matrix with a masked version W = W @ MQ) where M E {0, 1)* is the binary mask and @ denotes elementwise multiplication. The binary mask M defines a bipartite graph, and in the -iftt limit, we model it with a bipartite graphon W [o, 1]2 + [0, 1]. From this point onward, use W to denote a bipartite graphon, unless explicitly mentioned otherwise.
Despite being called "sparse", most pruning methods retain a constant fraction of weights (e.g.. 10%), resulting in (n²)) connections for width (although with very small constants). From graph-theoretic perspective, these remain dense graphs, where graphon theory applies. Thus, to ensure well-defined graphon limits, we assume: (i) the pruning masks retain constant fraction p > 0 of weights (normally satisfied in pruning context), and (ii) the width of hidden layers remains comparable across layers - Under these assumptions, given pruning method, for sequence of pruning masks (M!,) with increasing width, we can define their convergence to a bipartite graphon W analogously: lim 6n W() where W Mg is the bipartite graphon representation of the mask M at layer 1 (see Section 3.2 for definition of graphon representation of graphs). As the network width - oc, each mask matrix becomes larger and more structured graph. Different pruning methods induce different sequences (M,))) of such biadjacency matrices.
4.2 Graphon Limit Hypothesis for neural network pruning
Graphon Limit Hypothesis. Given sequence of neural networks (Nn) from fixed architecture class A with widths tending infinity, the application of a pruning method fixed sparsity level p > O produces sequences of binary masks (M that converge layer-wisely to deterministic graphons W( in the cut distance. These limiting graphons depend only on A.p.P P and characterize the connectivity structure of the pruned networks the infinite-width limit.
The graphon limit has the following geometry interpretation. fundamental result from graph limit theory states that a sequence of graphs (G,, converges to a graphon W if and only if the density of any fixed subgraph F in (G,,) converges to the density of in W. In the context of neural networks, this means the graphon limit asymptotically captures geometric patterns including path densities (effective paths), and other structural motifs that influence the network's computational properties.
This hypothesis has several important implications: (i) graphons WI() serve as structural priors for the pruned network in the wnddhite-width setting: (ii) each pruning method corresponds to a distinct region in the space of graphons, determined by the method's design; and (iii) structural differences between pruning methods can be captured and compared through their limiting graphons.
4.3 Experiments on graph limit of pruning at initialisation methods
We empirically validate the Graphon Limit Hypothesis by examining whether pruning methods converge to distinct, characteristic graphons as network width increases. We analyse four pruning at initialisation methods: Random, SNIP 43 GraSP (55. and Synflow [2] across varying network
widths n E {100, 500, 1000, 2000]. number of layers (4. 5}. and sparsity levels {70%, 80%, 90%}, }. We conduct 100 independent trials per configuration and collect layer masks except for masks from input and output layers. To visualise the emergent graphons, we employ the SAS method [12] that: (I) Sorts nodes based on degree centrality (out-degree for layer I, in-degree for layer ( 1); (2) Partitions the sorted bipartite graph into a grid of intervals; (3) Computes the average edge density within each interval. This degree-basee sorting serves as an approximate measure- peeesurr-rreerving transformation, revealing underlying structural patterns while maintaining invariance to node permutations. We refer to Appendix
C for more details.
Figure D displays the estimated graphons with increasing network widths. Each pruning method converges to a distinct pattern. In particular, Random Pruning converges to constant graphon
5

---

3.3
0.
Random
0.0
17.5
SNIP
10.0
0.0
12.5
0.8
GraSp
0.0
0.8
Synfow
15
10.3
Hidden 28
10
0.2 053
06
28 10
1.2 d3 08
100
1006
2005
Hidden
Figure Graph limit of subnetworks' mask produced by Pal methods at 80% sparsity and the corresponding convergence of graphons via Euclidean distances.
(Erdōs-Rén) Rényi random graph), with uniform connection probability across all node positions. SNIP and GraSP exhibit structured, non-uniform graphons with density gradients, preferentially connecting high-centrality nodes. Synflow converges prioritising connections among high-centrality neurons. This method encourages sparse networks with high number of paths, which is also indicated in [56 (4).
block-like graphon with sharp transitions, strongly
To quantify convergence, we also show the Euclidean distance between density matrices at width n and reference matrices at n 2000 in Figure rright. Since all histograms are aligned via degree- based sorting, we use the Euclidean distance between density matrices as proxy for cut distance. All methods demonstrate monotonic convergence, with distances decreasing width increases, confirming that the limiting graphon structure is an intrinsic characteristic of each pruning algorithm.
These results provide compelling evidence that each pruning method induces a unique (up to rela- belling), stable connectivity pattern in the wrgth limit, validating our graphon hypothesis and establishing foundation for analysing pruning methods through graph limit theory.
Neural tangent kernel with graphon structure
In this section, we present the derivation of NTK for neural networks with graphon structure. This novel formulation extends the standard NTK theory to accommodate networks where connectivity patterns are modulated by graphon functions, providing insights into how sparse network architecture influences learning dynamics.
5.1 Network structure and setup
In standard neural networks, weights between layers are typically sampled from identical independent distributions. However, sparse networks exhibit connectivity patterns, where connection strength depends on neuron positions. Graphon provides mathematical tool to model such connectivity patterns. Intuitively, we can view each neuron as having a position in [0, 1]. and the graphon value W (u, v) represents the expected connection strength/probabiiity between neurons at positions and w. Fach layer's structure is thus defined by its corresponding graphon W(O,, providing continuous representation of the network's connectivity.

---

Graphon-structured structured networks naturally connect to neural network pruning, where certain connections eliminated. In pruning, binary mask M0) is applied to weights: W W @ M For weights initialised as w
ij NN, )w), pruning effectively modifies their variance: Var ) ())
Var(W.;; 1j M = Var(W,
1j M de M
Based on the Graphon Limit Hypothesis in Section 4 as network width increases, pruning masks
converge to graphons layer-wisely W 41-1) ), representing the probability density of connections
between positions. In a finite network, the weights are sampled as: W (1)
ij ~ N (0, vo() ( -+==-))
where ny represents the width layer I, and we consider o = 1 for simplicity [5]. Similar to the standard NTK setting [37 [ remove the bias terms and consider single output network. Unlike standard networks where weights are i.i.d., our graphon- craphon-modulated weights have position-dependent variances while maintaining independence. As we take layer widths infinity n4 cc, discrete neuron indices approach to continuous positions in [0, 1]: i/n; - 4 E [0, 1] (position in layer 1) and j/m4- - ut-1 E [0, 1] (position in layer 1 1). Summations over neurons become integrals positions, and the scaling term 1 1/4-1 will be absorbed into the integral. The forward pass of an hidden layers network below illustrates this approach from discrete to continuous:
Discrete network
Continuous network
nz-1
2f (x) = M
1 (x),
(t-1)
(w, = L/' W (wr,
1) (w)-1, du-
402 (x) oz (x)).
11)
h0 (u, x) = (u),)),
f(x)= = S w(L+1) (UL+I, (L (L)
f(x)
w (((+t))/ (L) (x).
x)) duz.
VTL
where (w,, u111~ (O, ( 1)11) is random field modulated by the graphon. In addi- tion, since the statistical properties of the network now is affected by the graphon structure, weights are independent but not identically distributed random variables. We need additional assumptions to ensure the Law of Large Numbers (LLN) and Central Limit Theorem (CLT) still apply. For the LLN to hold in this non-iid setting, we assume: (i) the graphon values Wil) (wt, ur-1) 1) are bounded, and (ii) the average connectivity Jo. W (u, u 1 are well-behaved for all positions Similarly, for the CLT to apply to pre-activations, we assume the Lindeberg- Feller condition:
Σ
1 n2 (x))
lim RI 000
-1 -1 ( = 0,
for all c >> 0. This condition ensures that no single allowing the pre-activations to still converge to Gaussian processes, albeit with position-dependent covariance structures shaped by the graphon.
product dominates the sum,
5.2 Graphon neural tangent kernel
We extend the NTK theory to networks with graphon-structured connectivity. We begin by character- ising the limiting behaviour of pre-activation and activations in the widtnite-width limit.
Proposition 1. For a neural network with layers structured by graphons W(O, [0, 1]2 + [o. 1),
Lipschitz nonlinearity and in the limit as fi,...,TL - oc, the pre-activations 200 (u, x) at every hidden layer converge to centred Gaussian processes with covariance E1(1), where 510), is defined recursively by:
I 2) x')s; (1),,,, "/,,""",,,"") 8(mi -w1) n1) 1 E wal
(1)
(m, u/, 2/, x') ) 8(m u/) 1.' wil ui (2)(0 (wi-1 41 x' d')du1
(2)
7

---

where (x 10-20 x'); represents the imput correlation at position j. the activation covariance E() is:
" (ui, I, x',x 6(m h1lDB//sshhoNNoooM0oooooo0000000
(3)
6(m 1d) is the Dirac delta function, and A A (ui) is the position-dependeee covariance matrix:
50 (ut, uy, 2)) 500 (u, ut, 2; x:
(w))= 50 (un, uy x' x²,x)) 20 uy x
The key difference between the standard NTK and Graphon NTK lies in the pre-activation covariance formulation. In standard NTK, the pre-activation covariance Σ(0 equals the previous layer's activation covariance E!--1), In contrast, the Graphon NTK modulates this with the graphon function Wil,, creating position-dependent covariance structures. This causes signals to propagate non-uniformly through the network, with connectivity strength determined by the graphon values.
Remark
The NTK characterises how network outputs change with respect to parameters during training. Our key result shows that the Graphon NTK also converges to deterministic limit in the infinite-width regime, but with a structure determined by the graphon connectivity patters..
Theorem 1 (Graphon NTK). For a neural network with layers structured by graphons Wil) : [0, 1J² + [0, 1}, Lipschitz nonlinearity a, in the limit as Riy.,, RJ, + 0o, the Graphon Neural Tangent Kernel (Gruphon NTK) (xx,x)) converges to deterministic kernel:
0(x,,')
=
1:00 (M,,,,, f
II 10.1p--43 m=141
L+1
moim Uma 1)) (Um; Ums I. x²) du
1=1
(1 14 14 yduce dui,
(4)
where (u,, ut x, x²x = ())) (wI, represents the expected correlation between activation derivatives, and dui+1 ddu++idur, duu+1.
Remark 2. The Graphon NIK explicitly shows how guncion W shape the kemel through desieion-dependentt connectivity, contrast to the standard NTK. This structure gives insights on the heterogeneous learning dynamics of sparse model training, and allows modelling diverse connectivity patterns.
functions
The Graphon NTK provides a powerful framework for analysing how the pattern of connectivity between neurons in sparse neural networks affects learning dynamics in the infinite-width limit. By modulating the weight variances according to the graphon, we can model a wide range structured connectivity patterns, including those that arise from various network pruning strategies. We refer to Appendix A for detailed proofs.
5.3 Graphon neural tangent kernel of Random Pruning
Our Graphon NTK framework encompasses broad class of structured sparsity patterns with Random Pruning being a special case. Specifically, when the underlying graphon constant, i.e., W(u, v) = where с E (0. 1]. the resulting kernel simplifies to scaled version of the standard NTK: O x²) c² sa, x'), where Ozd denotes the standard NTK of a fully- conle-connected network with hidden layers. This special case was previously studied in 74 by applying random masks to weight matrices. Our general framework of Graphon NTK not only recovers this result, but also flexible enough to analyse more complex sparsity connectivity patterns beyond Random Pruning.
This scaling directly influences network training dynamics. If d is the k-thh eigenvalue of + then this k-th eigenvalue of the pruned network's NTK becomes At- Notably, while the absolute learning speed reduced, the relative dynamics between modes remain unchanged. This offers principled explanation for the empirical observation that sparse random networks converge more slowly than their dense counterparts 277
We refer to Appendix for more discussions.
6 Numerical experiments
We illustrate the relationship between spectral properties of Graphon NTK and training dynamics of finite sparse networks using three pruning methods: Random, SNIP 43 and Synflow [62] at
8

---

Taining Loss Sparsity 60%
Trining Locs Sparsity 70%
Training Less Sparsity 80%
Taining Loss Sparsity 93%
baskin
Syneer
1.5
0.
100
LSocabe steps
Upd00d steps
Up:00t strps
Utciade steps
Figure 2: The training loss in the first 200 gradient update steps of training sparse networks produced by Random. SNIP, and Synflow with different sparsity levels. At the beginning of the training phase, subnetworks generated by SNIP and Synflow show faster convergence speed than Random across sparsity levels.
Eigenvalue Decay Rate
Effective Rank
Spectrol Gap
Energy Concentration
2.5
Expomnnt
2.0
Effectee Baank
Specii Cap Gurial
10-
0.65
2.6
0.60
1.-
0.55
0.-
Synfow
Spanity
Spansity
Spaniit
Spanity
Figure 3: Spectral metries of the Graphon NTK with different graphon functions and sparsity levels.
sparsity levels from 50% to 95%. Subnetworks are pruned from 4- 4-layer networks with hidden size 1024, then trained on MNIST. We approximate the Graphon NTK by graphon functions found in Section 4 In particular, given the sparsity, we generate masks for 4-layer networks with hidden dimension n = 1024 based on graphon functions. Then we compute the Graphon NTK based on a batch of 1024 data samples from 10 classes in MNIST, and analyse four spectral metrics: eigenvalue decay rate (a),, effective rank (race(O)/A, spectral gap (A1/A2), and the energy concentration in top-5 eigenvalues
We refer Appendix D for further details.
Results and discussion. Figure reveals that Random Pruning maintains relatively consistent spectral properties across the sparsity levels, with stable decay rate, high effective rank, and broad spectral spread. This reflects uniform eigenvalue scaling which is consistent with the constant graphon analysis in Section 5.33 where Random Pruning acts global downscaling of the kernel. In contrast, SNIP and Synflow increasingly concentrate their Graphon NTK energy in top eigenvalues as the sparsity level grows, despite reduced effective rank and spectral gaps. This suggests stronger focus on dominant eigen-directions, aligning with their faster training loss reduction at the beginning when compared with random subnetworks at the same sparsity in Figure
The observed correlation between kernel spectral properties and training dynamics demonstrates an initial insight of our Graphon NTK framework for sparse neural network training. These findings offer both theoretical insight and practical guidance: Graphon NTK can serve as principled, training-free indicators for evaluating pruning quality, and preserving key spectral characteristics should be design goal for future pruning algorithms, as also verified [67].
7 Conclusion
In this paper, we introduce novel theoretical framework for analysing sparse neural networks through the lens of graph limit theory and neural tangent kernel. Our Graphon Limit Hypothesis establishes a connection between pruning methods and their limiting graphons in the infinite width regime, providing mathematical framework for understanding the structural properties of sparse networks. We derive Graphon NTK which offers meaningful tool for analysing how these structural properties affect training dynamics of sparse networks. This paves the way for the theoretical study of sparse models using tools from kernel and graph limit theories.

---

8 Limitations and future research directions
Our work introduces the Graphon NTK as a new theoretical tool for understanding neural network pruning. While it establishes a solid foundation, several limitations remain, and each suggests promising avenues for future research.
Formalising the Graphon Limit Hypothesis. A foundational aspect of our work is its reliance on the Graphon Limit Hypothesis, which is the proposition that pruning masks converge to limit graphon. A key limitation is that we establish this hypothesis empirically, providing experimental evidence across MI.P architectures and pruning methods, rather than through formal mathematical proof. The theoretical framework of the Graphon NTK, which we use to study training dynamics and spectral properties, consequently built upon this compelling, yet unproven, foundation. This immediately opens critical avenue for future theoretical research to formally prove the existence and uniqueness of these graphon limits.
From Static Analysis to Dynamic Pruning and Complex Architectures, Our analysis currently focuses on the clean theoretical setting of pruning at initialisation using fixed masks on MLPs. This controlled environment was essential for developing the core theory but does not yet encompass more complex, real-world scenarios. This limitation presents clear research trajectory which extends the framework to dynamic pruning and advanced architectures. For iterative or continuous sparsification, one can envision modelling the pruning process as an evolution of the graphon over time, potentially recasting the problem as gradient flow on the continuous space of graphons. A parallel effort will be to generalise the Graphon NTK beyond MLPs to architectures like CNNs and Transformers.
Bridging the Theory- Practice Gap with Principled Algorithm Design. While our framework provides a powerful lens for analysing existing pruning methods, a key limitation is the inherent gap between its infinite-width theoretical predictions and the practical realities of finite-width networks. Furthermore, our analysis explains why certain structures are effective but does not yet yield prescriptive algorithm for practitioners to create them. Future work can focus on creating graphon- guided pruning algorithms that directly optimise for a target graphon with desirable spectral properties (e.g., high energy concentration for faster convergence), thereby closing the loop from theory back sparse
to practice. Beyond just designing masks, these insights could inform graphon-informe training methodologies, such as developing learning rate schedulers or weight initialisations based on the eigenfunctions of the Graphon NTK. Such methods would be designed to explicitly accelerate training and improve robustness, helping to bridge the theory-practice gap for finite networks.
Acknowledgements
RB acknowledges the Gauss Centre for Supercomputing V. for providing computing time the GCS Supercomputer JUWELS at Jülich Supercomputing Centre (ISC). RB also grateful for funding from the European Research Council (ERC) under the Horizon Europe Framework Programme (HORIZON) for proposal number 101 16395 SPARSE-ML.
References
[1] Emmanuel Abbe. Community detection and stochastic block models: recent developments. Journal of Machine Learning Research, 18(177):1-86, 2018
[2] Edo M Airoldi, Thiago B Costa, and Stanley H Chan. Stochastic blockmodel approximation of a graphon: Theory and consistent estimation. Advances in Neural Information Processing Systems, 26, 2013.
[3] Milad Alizadeh, Shyam A. Tailor, Luisa M Zintgraf, Joost van Amersfoort, Sebastian Farquhar, Nicholas Donald Lane, and Yarin Gal. Prospect pruning: Finding trainable weights at ini- tialization using meta veta-ggaddeens. In International Conference on Learning Representations, 2022.
[4] Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparame- terized neural networks, going beyond two layers. Advances neural information processing systems, 32, 2019.
10

---

[5] Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. Advances in neural information processing systems, 32, 2019.
[6] Krishna B Athreya and Soumendra N Lahiri. Measure theory and probability theory, volume 19. Springer, 2006.
[7] Pal Bithika, Biswas Arindam, Kolay Sudeshna, Mitra Pabitra, and Basu Biswajit. A study the ramanujan graph property of winning lottery tickets. In International Conference on Machine Learning, volume 162, pages 17186 17201, 2022.
[8] Christian Borgs, Jennifer T Chayes, László Lovász, Vera T Sós, and Katalin Vesztergombi. Convergent sequences dense graphs i: Subgraph frequencies, metric properties and testing. Advances in Mathematics, 101-1551, 2008
[9] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 33:1877-1901, 2020.
[10] Rebekka Burkholz. Convolutional and residual networks provably contain lottery tickets. In International Conference on Machine Learning, pages 24 14-2433. PMLR, 2022.
[11] Rebekka Burkholz. Most activation functions win the lottery without excessive depth. Advances in Neural Information Processing Systems, 35: 15:1777778,720, 2022.
[12] Stanley Chan and Edoardo Airoldi. consistent histogram estimator for exchangeable graph models. In International Conference on Machine Learning, pages 208-216. PMLR, 2014.
[13] Tianlong Chen, Jonathan Frankle, Shiyu Chang. Sijia Liu, Yang Zhang, Michael Carbin, and Zhangyang Wang. The lottery tickets hypothesis for supervised and self-supervised pre-training in computer vision models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16306-16316, 2021.
[14] Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. The lottery ticket hypothesis for pre-trained bert networks. Advances in neural information processing systems, 33: 13:15844-15846, 2020.
[15] Tianlong Chen, Yongduo Sui, Xuxi Chen, Aston Zhang, and Zhangyang Wang. A unified lottery ticket hypothesis for graph neural networks. In International conference on machine learning, pages 1695-1706. PMLR, 2021.
[16] Pau de Jorge, Amartya Sanyal, Harkirat Behl, Philip Torr, Grégory Rogez, and Punect K. Dokania. Progressive skeletonization: Trimming more fat from a network at initialization. In International Conference Learning Representations, 2021.
[17] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguisties: human language technologies, volume (long and short papers), pages 4171-44 141814 2019.
[18] Alexey Dosovitskiy., Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[19] Simon Du and Wei Hu. Width provably matters in optimization for deep linear neural networks. In International Conference on Machine Learning, pages 1655-1664. PMLR, 2019.
[20] Simon S Du, Kangcheng Hou, Russ R Salakhutdinov, Barnabas Poczos. Ruosong Wang, and Keyulu Xu. Graph neural tangent kernel: Fusing graph neural networks with graph kernels. Advances in neural information processing systems, 32, 2019.
[21] Utku Evei, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. International Conference on Machine Learning, pages 2943-2952 PMLR, 2020.

---

[22] Utku Evci, Yani Ioannou, Cem Keskin, and Yann Dauphin. Gradient flow in sparse neural networks and how lottery tickets win. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 6577-6586, 2022.
[23] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03633. 2018.
[24] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In Internationall Conference on Machine Learning, pages 3259-3269. PMLR, 2020.
[25] Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pages (0323-10337) PMLR, 2023.
[26] Advait Gadhikar, Tom Jacobs, Chao Zhou, and Rebekka Burkholz. Sign-in to the lottery: Reparameterizing sparse training from scratch. arXiv preprint urXiv:2504. /2801, 2025.
[27] Advait Gadhikar, Sree Harsha Nelaturu, and Rebekka Burkholz. Cyclic sparse training: Is it enough? arXiv preprint vrXiv:2406.02772 2024.
[28] Advait Harshal Gadhikar and Rebekka Burkholz. Masks, signs, and learning rate rewinding. In The Twelfth International Conference on Learning Representations, 2024.
[29] Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity deep neural networks. arXiv 2019.
preprint arXiv:1902.099
[30] Thomas Gebhart, Udit Saxena, and Paul Schrater. A unified paths perspective for pruning at initialization. arXiv preprint wwxXi:2201.105S5 2021.
[31] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
[32] Babak Hassibi, David G Stork, and Gregory J Wolff. Optimal brain surgeon and general network pruning. In IEEE international conference on neural networks, pages 293-299. IEEE, 1993.
[33] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[34] Daniel Herbst and Stefanie Jegelka. Higher-order graphon neural networks: Approximation and cut distance. In The Thirteenth International Conference on Learning Representations, 2025.
[35] Due N.M Hoang, Shiwei Liu, Radu Marculeseu, and Zhangyang Wang. REVISITING PRUN- ING AT INITIALIZATION THROUGH THE LENS OF RAMANUJAN GRAPH. In The Eleventh International Conference on Learning Representations, 2023.
[36] Tom Jacobs and Rebekka Burkholz. Mask in the mirror: Implicit sparsification. In The Thirteenth International Conference Learning Representations, 2025.
[37] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
[38] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron- neberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. nature, 596(7873):583-589, 2021.
[39] Florian Klimm, Nick S Jones, and Michael Schaub. Modularity maximization for graphons.
SIAM Journal on Applied Mathematics, 82(6):130--195 2022."
[40] Sanjukta Krishnagopal and Luana Ruiz. Graph neural tangent kernel:
Convergence on large graphs. In International Conference Machine Learning, pages 17827-17841, PMLR, 2023,
12

---

[41] Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989.
[42] Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. Advances in neural information processing systems, 32, 2019.
[43] Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. Snip: Single-shot network pruning based connection sensitivity. In International Conference Learning Representations, 2019.
[44] Shiwei Liu, Tianlong Chen, Xiaohan Chen, Zahra Atashgahi, Lu Yin, Huanyu Kou, Li Shen, Mykola Pechenizkiy, Zhangyang Wang. and Decebal Constantin Mocanu. Sparse training via boosting pruning plasticity with neuroregeneration. Advances in Neural Information Processing Systems, 34:9908-9922, 2021.
[45] Shiwei Liu. Lu Yin, Decebal Constantin Mocanu, and Mykola Pechenizkiy. Do we actually need dense over- over-parameten parameterizaaion? in-time over peer-paramttrizatioon in sparse training. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learming, volume 139 of Proceedings of Machine Learning Research, pages 6989-7000 PMLR, 18 24 Jul 2021.
[46] Tianlin Liu and Friedemann Zenke. Finding trainable sparse networks through neural tangent transfer. In International Conference on Machine Learning, pages 6336-6347. PMLR, 2020.
[47] László Lovász and Balázs Szegedy. Limits of dense graph sequences. Journal of Combinatorial Theory, Series B, 96(6):933-955 2006.
[48] Ekdeep Singh Lubana and Robert Dick. A gradient flow framework for analyzing network pruning. In International Conference on Learning Representations, 2021.
[49] Eran Malach, Gilad Yehudai, Shai Shalev-Sehwarle and Ohad Shamir. Proving the lottery ticket hypothesis: Pruning is all you need. In International Conference on Machine Learning, pages 6682-6691. PMLR, 2020.
[50] Sohir Maskey, Ron Levie, and Gitta Kutyniok. Transferability of graph neural networks: an extended graphon approach. Applied and Computational Harmonic Analysis, 63:48-83, 2023.
[51] Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibeseu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications, 9(1):1-12, 2018. 8.
[52] A Martina Neuman and Jason J Bramburger. Transferability of graph neural networks using graphon and sampling theories. arXiv preprint arXiv:2307. 13206, 2023.
[53] Bithika Pal, Arindam Biswas, Sudeshna Kolay, Pabitra Mitra, and Biswajit Basu. A study on the ramanujan graph property of winning lottery tickets. In International Conference Machine Learning, pages 17186-1777 17201. PMLR, 2022.
[54] Shreyas Malakarjun Patil and Constantine Dovrolis. Phew: Constructing sparse networks that learn fast and generalize well without training data. In International Conference Machine Learning, pages 8432-8442. PMLR, 2021.
[55] Ankit Pensia, Shashank Rajput, Alliot Nagle, Harit Vishwakarma, and Dimitris Papailiopou- los. Optimal lottery tickets via subset sum: Logarithmic over-paraaetrrizzaionn is sufficient. Advances in neural information processing systems, 33:2599-2610, 2020.
[56] Hoang Pham, Shiwei Liu, Lichuan Xiang, Dung Le, Hongkai Wen, Long Tran-Thanh, et al. Towards data-agnostic pruning at initialization: what makes a good sparse mask? Advances in Neural Information Processing Systems, 36:800444800000 2023.
[57] Luana Ruiz, Luiz Chamon, and Alejandro Ribeiro. Graphon neural networks and the trans- ferability of graph neural networks. Advances Neural Information Processing Systems, 33:1702-1712, 2020. 2020.
13

---

[58] Mahalakshmi Sabanayagam, Leena Chennuru Vankadara, and Debarghya Ghoshdastidar. Graphon based clustering and testing of networks: Algorithms and theory. arXiv preprint wrXiv:21 2021.
[59] Hyunjin Seo, Jihun Yun, and Eunho Yang. TEDDY: Trimming edges with degree-based discrimination strategy. In The Twelfth International Conference on Learning Representations, 2024.
[60] Yi Su, Raymond KW Wong, and Thomas CM Lee. Network estimation via graphon with node features. IEEE Transactions on Network Science and Engineering, 7(3):078-2068 2089, 2020.
[61] Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A simple and effective pruning approach for large language models. In The Tweifth International Conference on Learning Representations, 2024.
[62] Hidenori Tanaka, Daniel Kunin, Daniel L Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in Neural Information Processing Systems, 33:6377-6388, 2020.
[63] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[64] Dharma Teja Vooturi, Girish Varma, and Kishore Kothapalli. Ramanujan bipartite graph products for efficient block sparse neural networks. Concurrency and Computation: Practice and Experience, 35(14):c6363, 2023.
[65] Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations, 2020.
[66] Huan Wang, Can Qin, Yue Bai, Yulun Zhang, and Yun Fu. Recent advances on neural network pruning initialization. In International Joint Conference on Artificial Intelligence, 2022.
[67] Yite Wang, Dawei Li, and Ruoyu Sun. NTK-SAP: Improving neural network pruning by aligning training dynamics. In The Eleventh International Conference on Learning Representations, 2023.
[68] Xinyue Xia, Gal Mishne, and Yusu Wang. Implicit graphon neural representation. In In- ternational Conference on Artificial Intelligence and Statistics, pages 10619-10634. PMLR, 2023.
[69] Lichuan Xiang, Quan Nguyen-Tri -rii Lan-Cuong Nguyen, Hoang Pham, Khoat Than, Long Tran- Thanh, and Hongkai Wen. DPai: Differentiable pruning at initialization with node-path balance principle. In The Thirteenth International Conference on Learning Representattins, 2025.
[70] Hongteng Xu, Dixin Luo, Lawrence Carin, and Hongyuan Zha. Learning graphons via struc- tured grssesewasserstein barycenters. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 10505-10513, 2021.
[71] Greg Yang. Tensor programs ii: Neural tangent kernel for any architecture. arXiv preprint arXiv:2006. 14548, 2020.
[72] Greg Yang and Etai Litwin. Tensor programs iib: Architectural universality of neural tangent kernel training dynamics. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Leurning Research, pages 11762-11772. PMLR, 18-24 Jul 2021.
[73] Hongru Yang, Ziyu Jiang, Ruizhe Zhang, Zhangyang Wang, and Yingbin Liang, Convergence and generalization of wide neural networks with large bias. arXiv preprint arXiv:2301 00337, 2023.
14

---

[74] Hongru Yang and Zhangyang Wang. On the neural tangent kernel analysis of randomly pruned neural networks. In Francisco Ruiz, Jennifer Dy, and Jan Willem van de Meent, editors, Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Proceedings of Machine Learning Research, pages 1513-1553, PMLR, 25-27 Apr 2023.
[75] Guibin Zhang. Kun Wang, Wei Huang, Yanwei Yue. Yang Wang, Roger Zimmermann, Aojun Zhou, Dawei Cheng, Jin Zeng, and Yuxuan Liang. Graph lottery ticket automated. In The Twelfth International Conference Learning Representations, 2024.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main claims in the abstract and the listed contributions in Section 1 reflect the main contributions made the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have provided the limitation discussion in Appendix E.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
16

---

Justification:
We have provided the necessary assumptions and proofs in the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have provided the information for reproducing the main experimental results and experimental settings in Sections 4, 6 and in the Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [Yes]
Justification:
We have provided the code in the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppp.cc/pulli//guiii wwwiiccgguie///eeeeeee CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We have provided details in experiment settings in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We have conducted tests for the statistical significance of the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
18

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: We only provide the type of compute workers, but not the memory and time of execution.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We make to conform to the NeurIPS Code of Ethics in every respect of the paper.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper focuses on understanding the training dynamics of sparse neural networks, which is irrelevant societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper focuses understanding the training dynamics of sparse neural networks, which has no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited the original paper that produced the code package used in our paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
20

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[NA]
Justification:
We did not release new assets
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper doesn't 't involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
21

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: this paper does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standart components.
Please refer to our LLM policy (https:/ /ttps//////u.p...eeee. wwtparrpppsccooofffeeeeeeeee ///Cooffreences/2222 for what should or should be described.
22

---

Contents
Introduction
2
Related work
3
Preliminaries
Neural tangent kernel
3.2
Graphon and graph limit theory
Graph limit and sparse neural networks
4.1
Pruning masks as graphs
Graphon Limit Hypothesis for neural network pruning
Experiments on graph limit of pruning at initialisation methods
5
Neural tangent kernel with graphon structure
5.1
Network structure and setup
5.2
Graphon neural tangent kernel
5.3
Graphon neural tangent kernel of Random Pruning
Numerical experiments
7
Conclusion
8
Limitations and future research directions
10
9
Acknowledgements
10
Details on graphon neural tangent kernel
24
A.1
Network structure and setup
24
A.2 Forward propagation: covariance structure (proof of proposition
26
A.3 Graphon neural tangent kernel convergence (proof of theorem
28
Details on graphon neural tangent kernel of Random pruning
31
Experiments graph limit of pruning at initialisation methods
33
Details on numerical experiments
37
23

---

A Details on graphon neural tangent kernel
In this section, we present a comprehensive derivation of the NTK for neural networks with graphon structure. This novel formulation extends the standard NTK theory to accommodate networks where connectivity patterns are modulated by graphon functions, providing insights into how network architecture influences learning dynamics. In particular, we first describe the network setting and the transition from discrete continuous network in Appendix A.I Then we provide the proof for the Proposition I in Appendix A.2 and for Theorem T in Appendix
A.3 respectively.
A.1 Network structure and setup
Discrete neural network with graphon Consider single output neural network with L hidden layers where weights are modulated by graphon function W [0, + [0, 1]:
W ~~(0)
N(.
(o, w
(i my n4-1 )
The graphon function governs the statistical structure and strength of synaptic connections between neurons. Specifically, it describes how the variance of random weights varies based on the positions of the connected neurons, thereby shaping the connectivity patterns and signal propagation across layers. Unlike in standard neural networks where weights identically distributed, graphon- moopuln-dddultedd weights have position-dependent variances while maintaining their independence.
Pre-activations and activations computed as:
2, (x) 1 12-1
Σ
1 bj n1
(111( (z),
(6)
1=1
h 10 (x) = al (z))
(7)
The network output for simplicity is single output:
f(z)
1
(1++ h (x).
(8)
VLL
Connection to network pruning The graphon framework can be directly connected to network pruning. standard pruning. a binary mask M is applied the weights:
W w M(()
where @ denotes element wise multiplication and M ( E {0, 1) indicates whether the connection is kept (1) or pruned (0). In the finite-width case, the variance of the pruned weights becomes;
Var (Wa()) o²,M M))
(10)
which equals either of (kept) or O (pruned). As network width increases (nu, 71-1 -> oc), the pruning mask can be viewed step function over [0. 1]2, with MD) corresponding the value mip ) the limit, this mask converges to the graphon y0 u-11,, which represents the density or probability of connections in each region of the unit square. Different pruning methods yield different graphon structures. Thus, the modeling the effect of pruning directly within the network initialization, enabling analysis of pruned network behavior in the infinite-width limit.
modulated weights can be interpreted as
Continuous limit formulation continuous indices:
As layer widths approach infinity (ni - 20), transition to
i/ni + u [0. (position in layer l),
24

---

j/m-11 - 41-1 [0, 1] (position in layer 1 1).
The continuous network becomes:
2 2 x) x)= = [/¹ (u. (1 (u-11, dUU
(11)
h 100 (u,, x) =0(z (2),
(12)
f(z) !
(13)
where WW u1-1) (O, wi u-1)).
Statistical properties and convergence conditions In graphon-structured networks, weights become independent but identically distributed (non-i.i.d.) random variables. This change requires careful consideration of the conditions under which the Law of Large Numbers (LLN) and Central Limit Theorem (CLT) apply.
Law of Large Numbers in Graphon Networks
For the LLN to hold in the non-i.id. setting of graphon networks, the following conditions are required:
Bounded Graphon Values: W0 (u. w-1) < 1 ensures that Var(W (1) is bounded, pre- venting any term from dominating the sum (satisfied by graphon definition).
fj.
Well-Defined Average Connectivity: The integral Jo' ()()() 4t 1 must be well- defined for all positions us, ensuring the "average effect" connections per neuron stable.
Since the graphon values are bounded, the variances of weights and related quantities remain bounded, ensuring the LLN applies empirical averages throughout the network.
Central Limit Theorem and the Lindeberg- Feller Condition
For pre-activations to converge to Gaussian processes in graphon networks, we assume the Lindeberg- Feller condition
6 is satisfied:
lim 11
[X3 [(ks))>ean) 0,
(14)
o
1=1
02 Σx Var(X,), and the pre-activation 2 =E Xj. This condition ensures that no single term in the sum disproportionately influences the total variance. Then, the CLT applies despite the non-i.i.d nature of the weights, allowing pre-activations to converge to Gaussian processes with position-dependee
for all > O, where X, -ター w
covariance structures.
In our setting, the Lindeberg condition is satisfied due to the following structural properties of graphon networks: (i) When network weights are initialized using a bounded graphon (My() (u, v) < 1), each connection's 's variance is controlled, preventing any single weight from becoming dominant; (ii) Activations from the previous layer h (111)) (x:) are bounded with sigmoid and tanh function, while activations like ReLU tend to remain within reasonable ranges when networks are properly initialized and inputs are normalized; (iii) The scaling factor of কলー in the pre-activation formula ensures that individual pre-activation variances decrease proportionally Together, these properties imply that as the layer width n4-1 - co, the influence of any single term X, becomes negligible relative to the total variance. Hence, the Lindeberg condition is met, and the pre-activations converge distribution a Gaussian process with position depeedenn covariance structure induced by the graphon.
the network widens, scaling as
O )ーー
25

---

A.2 Forward propagation: covariance structure (proof of proposition 1)
We first establish the statistical behaviour of signals as they propagate through the network. Our main result for the forward pass is captured in the Proposition] T the proposition here again:
For the sake of convenience, we re-state
Proposition 2. For a neural network with layers structured by graphons W() [0, 1]2 + [0. 1, Lipschitz nonlinearity and in the limit as ni,..., - oc, the pre-activations 200) (n, x) at every hidden layer converge to centred Gaussian processes with covariance 51(1,, where 5)(1) is defined recursively by:
(u, "1, 8(u1 -n1) d D willlu,, 2 ('))))))
z²)s.
(15)
(m, u4. 8(m v/) L (wl up
4-1, x ddu
(16)
where (x - x')s represents the imput correlation at position j, and the activation covariance 5 is:
(ur, uhs x, = ō(m -ooeeteaaaooooooooooooooooaat
(17)
where 8(m ul) is Dirac delta function, (w)) the position-dependent covariance matrix:
500 un u1, x., Σ E0 (n, x',x) (ws, U1 x x²))
Σ (m,, x,,)
(w))= =
Proof:
We prove the proposition by induction on the layer index l. The key insight to analyze how the graphon structure affects the statistical properties of pre-activations as signals propagate through the network. The proof works follows:
Base case: first layer 1 7 1 For the first layer with graphon modulation:
with N (0. (1
(18)
where d is the input dimension. The pre-activation covariance becomes:
B|4 (x) (x') (=') =
(1) 23
1
(1)
Ik
(19)
J=1
Wl w W
(20)
Since weights are independently sampled:
E(W:Sw ij w 0 when (i.j) A k)),
[[(W(g)))] (0 - 2 when and
Therefore:
E () (x)2 (2))) (x')) = d.r
Σ w(
-
(21)
m1
26

---

As + oo in the continuous limit with i/n1 - u1:
(un, u/, x, 8(u1 - -() Σ wa 2 2) d'},
(22)
where is the dot- product, (a x'), represents the input correlation position j. The activation covariance at position W1 is:
( (ui, 21 x') ) 8(u1 - w4
('))))
(23)
Since (u1, x), 2 (un, x²)) follows joint Gaussian distribution according to the CLT, this can be computed
(m) x,) 8(u1 Ba.aaaaaaae.raaooaaaaaa
(24)
Where (u1) is the position-dependent covariance matrix:
(un, x,,) (mn, un 2, x²) (un) = 50 U1, (u, u1, x²)
(25)
Different from fully connected network, where the pre-activation covariance of the first layer becomes covariance of the input layer ( (z, 2²) x.x', in graphon networks, the pre-activation covariance depends on the structure of the graphon.
Inductive step: subsequent layer > 1 For layer / with graphon structure:
E 9
(2')
1
w
Σ w wi
(26)
71-1
ni-1
14-1
Σ
B
W
El E (c))) (o))) (x'))- (x'))
(27)
n
Since weights are independently sampled and B((W =(0) will m
E E 13) (x)2 = der
W
j
Bihg (1- (z)h (x))
(28)
n--1
n
In the continuous limit as j/nn-1 + u1-1. and 1/m--1 is absorbed into the integral:
(m,, uhi x, x') = 8(m u4) / 4-1141111
" (w 4-1, x, x )du
(29)
The activation covariance is: I()) (m,, u/, x, x') = 6(m oui
(30)
where A (2x) is the covariance matrix for pre-activations at position uy:
500 (u. un, xr, x') 2 (ut, ue, x)
(w))=
(31)
50 (u, un
50 (m, uy x., xx)]
This completes the proof of Proposition by induction. A critical insight is how the graphon structure W (u,, w1-1) directly modulates signal propagation, creating non-uniform information flow across different network regions.
27

---

A.3 Graphon neural tangent kernel convergence (proof of theorem 1)
Our main theoretical result characterises the NTK for graphon-structune networks. For convenience, we re-state the Theorem heee again:
Theorem 2 (Graphon NTK), For a neural network with layers structured by graphons W(0)
[0. 1]2 [0, 1). Lipschitz nonlinearity a, in the limit as ni...., nn + 3c. the Graphon Neural Tangent Kernel (Graphon NTK) (zx, x') converges to deterministic kernel:
(1x,x')= = Σl
L
00 00 (u), u. u. x, 22) (0.1)--1 111 m=/+1
/
L+1
II
W(n) (um, Um um ())( (Ums Umms 2, x') cuu++1
1=1
(32)
21, 2dd'uce du
represents the expected correlation between activation derivatives, and du = dur+idur du1+1.
where E (ug, Ui x') (lo'(2(0))
Proof:
We prove the Graphon NTK by decomposing the derivation into three key steps: (1) characterizing gradient flow through graphon-structured layers, (2) analyzing gradient correlations under infinite- width statistical independence, and (3) integrating these correlations to derive closed-form NTK expression. This structured approach yields an interpretable kernel that reflects how the graphon structure shapes learning dynamics across the network.
A.3.1 Backward propagation: gradient flow
Gradient recursion For layer L, applying the chain rule:
(f(x) 0z (2(4)(uL,z) ahll) x) (ul,)
0f(x) Dh (1z,) w(++)) (uL+I. (e (uz, x)).
(33)
For earlier layers (I < L), the gradient with respect to pre-activations is:
0f(x) o' d'= [' (m(,
Of(z) Wil+1) (u141, u))du+1.
(34)
Parameter gradients
The gradients with respect to weights are:
0f(z) O200 x) aw u-1)
0f(z) (1 (1-1) (wi-1,
(35)
A.3.2 Graphon neural tangent kernel derivation
The Neural Tangent Kernel is defined as the inner product of gradients with respect to all parameters:
O = Σ 1=1
f
af(x) Ow (w, 14-1) aw() 1-11)
0f(z')
E
dundu=1-
(36)
Layer-wise graphon NTK contribution
The contribution the Graphon NTK from layer is:
(()
1
0f(x) 0f(z')
(1(x,x') = jo J
E
duydu-1
(37)
@W( (ut, w1-1) OW (wi, ui 11
0f(x) (1 (111( Oz 0z (ur,
0f(x')
E
1e 111-1) (u)1)))))
(21)
(38)
28

---

At this point, we apply key insight: In the infinite-width limit, by the LLN, the gradients (24,a) and the activations (t 1) (u)- become statistically independent for distinct positions uq and Wt-1- This allows us to factorize the expectation:
af(a)
[(B
af(x') 02l)) (ui, x) (2(1) (') x²)
af(x)
] (%/ (u) (m x' Idus- du
01(x,x²)==
E
0
(39)
(5)))
0f(x')
] (Sp I((1)) (w- (MM u-1, x² ydu- du-
E
(40)
02(0) (21, x) 02 (21, 2")
Gradient correlation analysis gradients at different inputs:
To compute the NTK, we need analyze the correlation between
0f(x)
as(x')
E
(41)
(2,0 x) O2 (u),"")
From our backward propagation analysis in Equation 34 we have:
0f(x)
0f(x')
E
= E
[0 x))oo (z
02l0(us x)
(2, x²)
So 1-' (2((+))()44,))))))) x) (2(44)))//4:)4)))
0f(z)
0f(x')
W (u141, Un w)) 4 u)du+
(42)
In the infinite-width limit, (z4, .)) 200)
(m,, (u), follows bivariate Gaussian distribution, allowing
us to define:
10 (un, un, u,, x') = (2 (u,, x')]]
(43)
The weights W(I+1) (w141, un) and w(++)) (1/4 ui) are independent for u+11 # u/41 with:
(1)(+))(441) un) ( u)] (0(+))(444)) w)8(u++1 11(11)
(44)
Substituting this into our gradient correlation:
0f(x) 0f(x))
5) un 24,2,,))
(45)
020) x) (2),
L' ((l+1)((441,)) M))
0s(x)
as(x')
dutti.
Oz (1n Oz 0z(++) (w)) (x(,)))))
Closed-form expression for Graphon NTK
To derive the closed-form expression, we define:
0f(x) 0f(x') 12 (4, x) Oz (u, (4)xx)) x²)
(un x')
(46)
From our derivation, GL follows the recursion:
((++)) G0, 1100 (ui, Un, 2,, /' ((l+))(( (C (14+1. x, x' d')duu+.
(47)
With the base case (( =
29

---

The general form for any layer becomes:
L+1
Z'), (1, (u,, U1 22) JoAII-1 m=l+1
II
W(m) (mms Um ()(m) (Ums Ums I. x²) du
(ums
(48)
where duj+1 du ddy ... du and E(++1) (uL+1, 21,,,, Substituting this into our formula for θ₁
01(x, = S/¹ (wn, (fo Iリ- IU- (u-1, (M- 11 x. ddu du
(49)
1/') (S0 2, 10.1)4-141
L+1
II
W(m) (um, Um 1) (um, Ums F. x²) du
m=l+1
(50)
(o y(l-1) 21-1, I, x' )duu dus.
(51)
The full Graphon NTK is the sum over all layers:
(x,xx') =
, (x,x²)
(52)
[=1
Σ
/
J
II 1o.1a in m=l+1
L+1
m( (Umm Um (10m-1)) {m (um, Uony 3. x² dui+
1=1
(53)
JL"
(U-1)( 1-1,,, z')duu du
(54)
This expression reveals how the graphon structure at each layer shapes the Neural Tangent Kernel through multiple integrals involving the graphon functions. The resulting kernel is position-deeendent and reflects the specific connectivity patterns encoded by the graphons.
A.3.3 Discussion and implication
The Graphon NTK provides a powerful analytical framework for understanding how structured weight patterns influence neural network learning dynamics. Several key insights emerge:
Non-uniform signal propagation: The graphon structure creates ssittonnddependen infor- mation flow, with regions of higher graphon values propagating signals more strongly.
2. Relationship to pruning: The graphon formulation provides a continuous limit perspective on network pruning, where the graphon y0t (w, 44-1 can be interpreted as the density or probability of connections.
3. Position-dependent learning dynamics: Different regions of the network effectively learn at different rates based on their connectivity patterns.
These theoretical results establish the foundation for analysing learning behaviours in neural networks with connectivity patterns, providing insights that may guide the development of more efficient architectural designs.
30

---

B Details on graphon neural tangent kernel of Random pruning
With Random pruning, pruning masks converge constant graphons (the Erdös- random graph) as the width tends to infinity. When the underlying graphons are constants, we observe uniform scaling effect on training dynamics.
For a constant graphon Wuu, u) = C, the resulting NTK scales uniformly as:
(zz x²) c² (z,
(55)
where (( denotes the standard NTK of a fully-connected network.
Proof:
For the first hidden layer (1 1), pre-activation covariance becomes:
50 (u, 5(u u') (c(x- x'))
(56)
For simplicity, assuming of 0 (no biases):
5:(1) 50 (u, u', ,0,,,, c and (u, u' "',,,'''',
(57)
This leads activation covariance at first layer:
(u, c - (u, ul x,,').
(58)
By induction, for any layer I:
(0, ",,,,,,,,,), xx,x)== - 8(u uu) u')
L (1-1) (1,,,,,,,,
(59)
1' d 1(1-1)
= 5(u u¹)
(v, 1.
11
8(u u')
Egd
2')J
d' Sill (u, u/, x,,').
And the activation covariance layer is:
Σ (u, (,,,',,,,')) e¹ I:0l (u, ul x,,
(60)
We also assume that the activation function is ReLU, then the relationship between activation derivative covariance:
(x, = (11) (x, xx)
(61)
For the gradient correlation function:
Gl (u, = 5000 (1, u, x, J' wi(+)) () 2v)o.
(62)
For homogeneous graphon, each backward step contributes factor of c. Going from output layer L back to layer I:
31

---

(lL+tt (u, = sdd [1, 1, =1
(,, (u, U,, (
d')oo
(I, Σ10 (,,
(u, G(L) 2,,
G(L-1) (u, x') _(1-1) U,, c I 20,
G (u, = (² (u,
By induction, we have:
ch+1-1 G (1, x').
(63)
The layer-wise NTK contribution is:
(1(x,)')= - [ (', (f" (U du.
(64)
Substituting our findings:
01(2,x')= = S'' (it1--s((,,,,, (L df "0 x, 2,, z'dd)) du.
(65)
When the activation integral dominates the constant (which is typical in deep networks):
O1(2, (11 = 1 d-1 O x') c. O1,aa (x, x²).
(66)
Summing over all layers:
(xx,²') Σ O1(x,x') dh Σ 0 = ch Ozd(z,²)).
(67)
This uniform scaling directly impacts convergence rates during training. If A are the eigenvalues of Oud then the eigenvalues of θ become c² Ag. Consequently, residual modes during training decay as:
a = au (0)e
(68)
This scaling directly influences network training dynamics. If A the eigenvalues of Qud- then the eigenvalues of the pruned network's 's NTK become c Ag. Notably, while the absolute learning speed reduced, the relative dynamics between modes remain unchanged. This offers principled explanation for the empirical observation that sparse random networks converge more slowly than their dense counterparts 22).
More broadly, our framework extends beyond this homogeneous case, allowing for arbitrary position- dependent graphons that induce heterogeneous learning dynamics. This highlights the generality and flexibility of Graphon NTK in modeling structured sparsity and its effect on ncural dynamics.
32

---

C Experiments on graph limit of pruning at initialisation methods
We empirically validate the graphon hypothesis by examining whether pruning methods converge to distinct, characteristic graphons as network width increases. We analyze four
methods: Random pruning. SNIP [4], GraSP 65], and Synflow [62] across varying network widths E {100, 500, 1000, 2000), layers 4, 5, and sparsity levels f70%, 80%, 90%}. We conduct 100 independent trials per configuration and exclude masks from input and output layers. We follow
Synflow [62] source code to produce masks [
All the experiments are run single Nvidia A10
GPU (24GB).
To visualize the emergent graphons, we employ the SAS method [12]
that:
Sorts nodes based on degree centrality (out-degree for layer I, in-degree for layer ++ 1). For 2D histogram, nodes in layer and / + 1 are in the x-axis and y-axis, respectively.
2. Partitions the sorted bipartite graph into a grid of intervals. Each axis is split into = 64 intervals.
3. Computes the average edge density within each interval.
This degree-based sorting serves as an approximate measure- presurrvinee transformation, revealing underlying structural patterns while maintaining invariance node permutations.
We visualise graph limits subnetworks' masks produced by Pal methods at different sparsity levels 8
7 and [ In particular, Random pruning converges to constant graphon (Erdös-Rényi random graph), with uniform connection probability across all node positions. SNIP and GraSP exhibit structured, non-uniform graphons with density gradients, preferentially connecting high-centrality nodes. Synflow converges to a block-like graphon with sharp transitions, strongly prioritising connections among high-centrality neurons. In Figure | large part of neurons is eliminated in hidden layers creating subnetwork with high paths. This observation is also indicated in [56 [4], in which after each iteration, Synflow prunes weights connected to low-degree nodes.
in 4- and 5-layer networks in Figures 5
To quantify convergence, we show the Euclidean distance between density matrices at width N and reference matrices N 2000 as in Figure Since all histograms are aligned via degree-based sorting, we used the Euclidean distance between the density matrices as proxy for the cut distance
d (WN. W2000) = (H)( H (2.00(i.))) (i.
where H is the histogram.
All methods demonstrate monotonic convergence, with distances decreasing as width increases, confirming that the limiting graphon structure an intrinsic characteristic of each pruning algorithm.
These results provide compelling evidence that each pruning method induces a unique (up to rela- belling), stable connectivity pattern the large-width limit, validating our graphon hypothesis and establishing foundation for analysing pruning methods through graph limit theory.
33

---

CraSp
Sparsit:
Eudidomn Blpaace
Sparsity
15
Sparsity
15
10
10
100
1.0
500
1000
Middin on
Middin
(a) Histogram convergence via Euclidean distance in 4-layer networks setting.
Synffow
Saarohy
12.5
30
Excfeenn Dibarce
Kearshy
Sparsty
10.0
3
10
100
500
1030
2000
100
500
1000
2000
100
500
1000
2000
100
500
1000
2000
Mdden Dimersion
Hicden Dimerston
Fidden Dimetsioon
Midden Dimans
(h) Histogram convergence via Euclidean distance in S-layer networks setting.
Figure 4: Histogram convergence via Euclidean distance.
(a) Graph limit of subnetworks' mask produced by Random pruning different sparsity levels in 4-layer networks.
(b) Graph limit of subnetworks' mask produced by SNIP pruning at different sparsity levels in 4-yayer networks.
f.
(c) Graph limit of subnetworks' mask produced by GraSP pruning at different sparsity levels 4-layer networks.
(d) Graph limit of subnetworks' mask produced by Synflow pruning at different sparsity levels 4-layer networks.
Figure 5: Graph limit of subnetworks' mask produced by Pal methods at different sparsity levels in 4-layer networks.
34

---

dacoers hare mhrndde
(a) First hidden layer.
(b) Second hidden layer.
Figure 6: Graph limit of subnetworks' mask produced by Random pruning at different sparsity levels in 5-layer networks.
20 linp 1ily
(a) First hidden layer.
(b) Second hidden layer.
Figure 7: Graph limit of subnetworks' mask produced by SNIP pruning different sparsity levels in 5-layer networks.
necpens nenideer
20 corooaonn for oie seoet oyt
First hidden layer.
(b) Second hidden layer.
Figure 8: Graph limit of subnetworks' mask produced by GraSP pruning at different sparsity levels 5-layer networks.
diggeres Ploaee syllem
20 spales, Miteii
(a) First hidden layer.
(b) Sccond hidden layer.
Figure 9: Graph limit of subnetworks' mask produced by Synflow pruning at different sparsity levels in 5-layer networks.
35

---

20 Hislograms for Pruner: mag
Hic Oee 183
Hidtin Den SCE
14:den Dies 1000
11:den Cirs 2039
63
08
62
02
Figure 10: Graph limit of subnetworks' mask produced by Magnitude pruning at different sparsity levels in 4-layer networks.
Magnitude Pruning at Initialisation. Through the experiment, we find that Magnitude produces masks converge to constant graphons as shown in Figure [O] From a purely structural perspective, Magnitude pruning is identical to that of Random pruning in case all the weights are i.id. initialised. In particular, when all weights [we] > Threshold is the same for all (i, j). Let this probability of preserving connection be By definition, graph where every edge exists with an independent and identical probability e an Erdós Erdös-Rényí Rényi random graph, G(n, c). The well-know limit of sequence of Erdōs-Rénny graphs is a constant graphon W(u, v) = c.
i.i.d., the probability that any single weight w: has a magnitude
36

---

D Details on numerical experiments
Experimental setup We evaluate the relationship between Graphon NTK spectral properties and sparse networks training dynamics using three pruning methods: Random Pruning, SNIP [3], and SynFlow [62] at sparsity levels from 50% to 95%. Subnetworks are pruned from 4 hidden layers network with width n. = 1024, then trained on MNIST. Subnetworks are then trained on MNIST with Adam Optimizer with 0.001 learning rate. All experiments are run times. We illustrate the training loss in the first 200 update steps during training.
We approximate the Graphon NTK by graphon functions provided in Section In particular, given the sparsity, we generate the mask for 4 hidden layers network with width n = 1024 based on graphon functions. Then we compute the Graphon NTK based on batch of 1024 data samples from 10 classes in MNIST. Since the eigenvalues of the NTK (on the training data) quantify how much the kernel emphasizes certain directions in function space, we analyze four spectral metries based on eigenvalues:
Eigenvalue decay rate (ar): The decay exponent a describes how fast these eigenvalues fall off, and it upproximated by fitting into the pow-law curve 2 k=
Effective rank trace(()/\r: Quantifies the functional "dimensionallty" of the kernel, show- ing how many independent directions meaningfully contribute to learning; lower values indicate stronger concentration
few key patterns
Spectral gap (A1/ /2): The ratio between the first and second eigenvalues, revealing how dominant the primary learning direction compared to others; larger gaps indicate the network will prioritize one function class extensively.
Measures what fraction of the kernel's total power resides in Energy concentration سس by
the top cigenvalues; higher concentration means the kernel heavily emphasizes a few key patterns. We use k = 5 in our experiments.
Randam Pruning
SMP fruring
Syrflow fruring
Sacrlyy 70%
Suerihy 70%
Sterlyy 20%
Kantty ann
kanstyy an%
Laardyy 67%
15-
Delic
Denec
Densa
1.0
123
130
120
Update
Update
Update
Figure 11: The training loss in the first 200 gradient update steps of training sparse networks produced by Random, SNIP, and Synflow with different sparsity levels compared with dense networks.
The training curves in Figure II reveal distinctive learning dynamics across pruning methods that can be elegantly explained through the Graphon NTK framework. For all methods, increased sparsity generally slows convergence, with dense networks typically converging fastest, but the magnitude of this effect varies significantly between approaches. Random pruning exhibits the strongest degradation with increased sparsity, showing a clear separation between different sparsity levels. Meanwhile, SNIP and Synflow maintain better convergence than random pruning equivalent sparsity levels. These differences emerge from how each pruning method shapes the graphon structure ( (ill) functions) the kernel formula.
37


Reviews:
[{"review_id": "iAJw5ACRCa", "reviewer": "Reviewer_Q1h3", "summary": "This paper studies the infinite width limit of pruned neural networks by introducing Graphons to analyze the infinite width limit of pruning masks. The paper first shows that various pruning at initialization masks converge to corresponding graphons. Then, the paper introduces the continuous limit of pruned networks and shows that they converge to gaussian processes with covariances modulated by the graphon that defines the pruning. Finally the paper empirically measures how pruning affects the NTK.", "questions": "See weaknesses for more questions. My basic issue with the paper is in its current state it is not clear how computing the graphon NTK can inform the choice of pruning strategies.\n\nCan the graphon NTK only be used to analyze pruning strategies at initialization? Or can it be adapted to pruning trained networks?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths: The idea of studying pruning through graphons in the infinite width limit is promising. The experiments in section 4 demonstrate that pruning masks indeed converge to graphons. The results in section 5 help demonstrate that NTK analysis can be adapted with graphons.\n\nWeaknesses:\nWhile the motivation is interesting, I do not think the theoretical/empirical benefits of graphon NTK analysis have been demonstrated in this paper.\n\n1. While the graphons are measured empirically, the authors do not characterize them except for the trivial case of random pruning (which leads to a constant function). Deriving the graphons, and then connecting their spectral properties to the properties of the pruned NTK can lead to a comparative analysis of pruning strategies. How do we choose pruning strategies that \"preserve key spectral characteristics\", as claimed in the paper?\n\n2. The appearance of $\\delta(u_1 - u_1')$ in the graphon NTK covariances means that functions at $u_1$ and $u_1'$ are uncorrelated (which tracks with the idea that they are independent draws from the same GP). From this observation, It seems that the role of the graphon basically boils down to a constant multiplying the covariance of the original NTK. Can you illustrate with a clear example how the NTK and Graphon NTK will differ?\n\n3. Section 6 performs empirical measurements of the spectral properties of the graphon NTK, but does not connect them to faster convergence rates, informing the choice of hyperparameters, etc.", "was_revised": true, "final_justification": "I appreciate the author's clarification about how the Graphon reshapes the covariance of the gaussian process at each layer. This was a gap in my understanding that was resolved by their rebuttal. While my other questions were not completely addressed, I believe this paper can be accepted as a foundation for answering the questions I raised.\n\nI have updated my score accordingly.", "rebuttal": "We thank the reviewer for your constructive feedback. Below are our point-to-point responses to your concerns and questions:\n\n**Q1**: “I do not think the theoretical/empirical benefits of graphon NTK analysis have been demonstrated in this paper.”, “While the graphons are measured empirically, the authors do not characterize them …“\n\n**A1**: The primary benefit of our framework is that it provides a **new theoretical foundation and a unified analytical tool** for studying sparse neural networks. To our knowledge, this is the first work to connect the structure of pruning masks to graph limit theory, postulating the Graphon Limit Hypothesis and providing empirical evidence for it (Section 4). We then derive the Graphon NTK (Section 5), which explicitly links the structure in the limit to training dynamics.\n\nWhile we do not provide closed-form expressions for the graphon limits of methods like SNIP or Synflow, this is a common challenge in deep learning theory where the objects of study are complex. Instead, our experiments in Figures 1 and Appendix C empirically reveal their distinct and stable structures, which is a crucial first step for any analysis.\n\n**Q2**: “How do we choose pruning strategies that \"preserve key spectral characteristics\", as claimed in the paper?”, “My basic issue with the paper is in its current state it is not clear how computing the graphon NTK can inform the choice of pruning strategies.”\n\n**A2**: Regarding the choice of pruning strategies, our work lays the necessary groundwork for a more principled approach. As an example of its utility, we have linked the spectral properties of the Graphon NTK to the trainability and convergence speed of training different masks. For instance, in the case of high energy concentration (and thus faster learning of dominant features), we expect faster convergence, as shown in Figure 3. Accordingly, we have explained why pruning is more difficult in the high sparsity regime, especially in the case of random pruning.\n\nFurthermore, by understanding how a method's graphon shapes the NTK's spectrum, one could also move towards *designing pruning strategies by optimising for a target graphon* that yields desirable spectral properties (e.g., high energy concentration for faster convergence). This moves the field beyond pure heuristics. Recently, NTK spectral properties have already been leveraged to design pruning methods that preserve the eigenspectrum of “Dense” NTK [1]. It is expected that our Graphon NTK limit will also enable optimisation of sparse networks’ configurations and lead to concrete pruning methods. While detailed algorithms for these are a substantial future work, our framework establishes its theoretical feasibility.\n\n**Q3**: “The appearance of $\\delta(u_1, u’_1)$ in the graphon NTK covariances means that functions at $u_1$  and $u’_1$ are uncorrelated, …, It seems that the role of the graphon basically boils down to a constant multiplying the covariance of the original NTK. Can you illustrate with a clear example how the NTK and Graphon NTK will differ?”\n\n**A3**: The $\\delta(u_1, u’_1)$ appears because in the continuous limit, pre-activations at different positions evolve independently due to the infinite-width limit. Specifically:\n\n**In standard NTK**: $\\widetilde{\\Sigma}^{(l)}(x,x') = \\Sigma^{(l-1)}(x,x')$\n\n**In Graphon NTK** (Eq. 2 in paper):  $\\widetilde{\\Sigma}^{(l)}(u_l, u_l', x, x') = \\delta(u_l - u_l') \\int^1_0 \\mathcal{W}^{(l)} (u_l, u_{l-1}) \\Sigma^{(l-1)}(u_{l-1}, u_{l-1}, x, x') du_{l-1}$\n\nThe key difference is that standard NTK has **homogeneous** covariance (same for all neurons), while Graphon NTK has **position-dependent** covariance induced by graphon. The $\\delta$ function enforces that only neurons at the same position are correlated in the infinite-width limit. Interpretably, in dense networks, all neurons learn uniformly, while in graphon-structured networks, neurons at position u learn at a rate proportional to $\\int \\mathcal{W}(u,v)dv$\n\n**In summary, the graphon does not simply multiply the covariance, it reshapes it via integration.** The only case where it becomes a simple scalar multiplier is when the graphon itself is a constant (e.g., Random Pruning), which our framework correctly recovers the result from [2] as a special case. For any structured graphon, the resulting Graphon NTK is fundamentally different from the standard NTK.\n\n**Q4**: “Section 6 performs empirical measurements of the spectral properties of the graphon NTK, but does not connect them to faster convergence rates, informing the choice of hyperparameters, etc.”\n\n**A4**: The connection to convergence rate can be made more explicit as follows: (1) Figure 2 shows that networks pruned with SNIP and Synflow converge faster than those from Random Pruning; (2) Figure 3 indicates that the graphons from SNIP and Synflow produce Graphon NTKs that **concentrate more energy in their top eigenvalues** than Random Pruning; (3) and a kernel that concentrates more of its energy in its top eigenvalues will drive faster convergence, as these correspond to the dominant learnable functions [3].\n\nThis analysis opens the door to designing more efficient training schemes for sparse networks. For example, one could design learning rate schedulers or weight initialisations based on the eigenfunctions of the Graphon NTK to further accelerate training for a given sparse architecture. While identifying the correspondence between specific weights and eigenfunctions is a non-trivial challenge, our work establishes the theoretical pathway.\n\n**Q5**: “Can the graphon NTK only be used to analyze pruning strategies at initialization? Or can it be adapted to pruning trained networks?”\n\n**A5**: We focused on pruning at initialisation (PaI) because it provides the cleanest setting for developing the theory. Fixed masks enable a rigorous infinite-width analysis. Extending our Graphon NTK analysis to dynamic or post-training pruning is also possible, but it requires significant additional ideas and technical details. For example, we would need to handle weight-mask co-evolution, significantly complicating the analysis. We believe understanding the \"base case\" of fixed masks is essential before tackling time-varying masks.\n\nOverall, we believe establishing rigorous theoretical foundations is valuable even without immediate algorithmic payoffs, as demonstrated by the impact of works like NTK theory and mean-field theory in deep learning. Our graphon framework provides a new lens for understanding sparse networks that we hope will inspire future algorithmic innovations.\n\n### References\n\n[1] Wang et. al., 2023. \"NTK-SAP: Improving neural network pruning by aligning training dynamics.\"\n\n[2] Yang et. al., 2023. \"On the neural tangent kernel analysis of randomly pruned neural networks.\"\n\n[3] Kopitkov et. al., 2020. \"Neural spectrum alignment: Empirical study.\"\n\n[4] Mocanu et. al., 2018. \"Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science.\""}, {"review_id": "4ful21V0aU", "reviewer": "Reviewer_Z3zQ", "summary": "This paper introduces a novel theoretical framework for analyzing sparse neural networks. The authors argue that such pruned networks exhibit unique structural properties in the infinite-width regime, depending on the pruning method. To analyze these structures, they propose a Graphon Neural Tangent Kernel framework, which captures how structural differences affect training dynamics. Empirical results show that popular pruning methods like SNIP and SynFlow increasingly concentrate their NTK energy on top eigenvalues as sparsity grows, suggesting a focus on dominant eigendirections. This aligns with their faster initial training loss reduction compared to random pruning at the same sparsity level.", "questions": "- Could the authors elaborate on why they chose to focus on pruning at initialization? Is this primarily due to compatibility with the NTK assumptions (infinite-width, lazy training)? Would the proposed framework extend to post-training pruning or iterative sparsification?\n- The paper notes that SNIP and GraSP yield structured, non-uniform graphons, while SynFlow leads to block-like graphons that emphasize connections between high-centrality neurons. Could the authors elaborate on the intuitive explanation behind these observations? Are these structures a direct result of the pruning criteria used, and what does that imply?\n- The authors use Euclidean distance between density matrices as a proxy for cut distance. Could they provide more justification or intuition for why this is a reliable proxy in this context? Are there limitations or assumptions that make this approximation valid?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- The paper provides a novel theoretical tool (Graphon NTK) that offers insight into the structure and training dynamics of sparse networks.\n- It bridges the gap between structural properties of pruned networks and their training behavior, helping explain why some pruning at initialization methods outperform random pruning even at high sparsity levels.\n- The alignment between theory and empirical findings (e.g., energy concentration in top eigenvalues) strengthens the validity of the proposed framework.\n\nWeaknesses\n- The analysis is built upon the infinite-width NTK regime, which is known to diverge from practical finite-width networks in many important ways. Besides, in practice, the pruning is conducted on finite-width networks instead of infinite-width ones.\n- The focus on pruning at initialization limits the applicability of the theory to real-world pruning workflows, which often involve iterative or post-training pruning.", "was_revised": true, "final_justification": "This is an interesting and novel theory paper. It provides a useful tool for studying the structure and training dynamics of sparse networks. The authors' response fully resolved my concerns. I recommend its acceptance.", "rebuttal": "We thank the reviewer for your constructive feedback and for recognising the novelty and quality of our work. Below are our point-to-point responses to concerns and questions:\n\n**Q1**: “The analysis is built upon the infinite-width NTK regime, which is known to diverge from practical finite-width networks in many important ways. Besides, in practice, the pruning is conducted on finite-width networks instead of infinite-width ones.”\n\n**A1**: It is true that the NTK regime diverges from practical finite-width settings. However, NTK analysis is one of the only few available approaches for theoretical analysis of neural networks. Its value has been demonstrated recently in practice, for instance $\\mu$P methods [1, 2, 3] for hyperparameter transfer.\n\nOur framework follows this spirit. By introducing the Graphon NTK, we provide a new, theoretically sound approach for analysing the training dynamics of sparse neural networks, analogous to what the standard NTK provides for dense networks. It opens up several promising practical research directions. For example, one could develop a $\\mu$P-like transfer method for sparse networks or leverage spectral properties of Graphon NTK to accelerate the training of sparse networks.\n\n**Q2**: “The focus on pruning at initialization limits the applicability of the theory to real-world pruning workflows, which often involve iterative or post-training pruning.”; “Could the authors elaborate on why they chose to focus on pruning at initialization? Is this primarily due to compatibility with the NTK assumptions (infinite-width, lazy training)? Would the proposed framework extend to post-training pruning or iterative sparsification?”\n\n**A2**: We focused on pruning at initialisation (PaI) because it provides the cleanest setting for developing the theory, fixed masks enable rigorous infinite-width analysis. Extending this framework to dynamic or post-training pruning is also possible to carry out Graphon NTK analysis, but it requires significant additional ideas and technical details. For example, we would need to handle weight-mask co-evolution, significantly complicating the analysis. We believe understanding the \"base case\" of fixed masks is essential before tackling time-varying masks.\n\nIn particular, with iterative pruning, one could study the *evolution of the graphon W(u, v, t)* at each pruning step t. Our framework provides the tools to analyse the spectral properties of the network at each stage, allowing one to ask: Which evolutionary paths for the graphon preserve trainability? For Dynamic Sparsification, it has been observed that some dynamic sparse training methods evolve from random to scale-free networks [4]. From our perspective, this can be viewed as the graphon evolving from a constant to a structured, power-law pattern. One could even recast dynamic sparsification as a *gradient flow problem on the continuous space of graphons.* These extensions require significant theoretical advancements, which we believe are well worth pursuing\n\n**Q3**: “The paper notes that SNIP and GraSP yield structured, non-uniform graphons, while SynFlow leads to block-like graphons that emphasize connections between high-centrality neurons. Could the authors elaborate on the intuitive explanation behind these observations? Are these structures a direct result of the pruning criteria used, and what does that imply?”\n\n**A3**: The emergent graphon structures are a reflection of each pruning method’s underlying philosophy:\n\n- SNIP and GraSP are two pruning methods based on the gradient/Hessian information on a batch of sample data. Therefore, neurons which have high gradient magnitudes or preserve second-order information will be kept, creating non-uniform and “hub”-like graphon patterns.\n- Synflow is an iterative data-agnostic pruning method which maximises the sum of path strengths. As a result, at each iteration, Synflow removes edges connecting to low-degree neurons, gradually narrowing the network  [5,6]. Hence, networks produced by Synflow converge to block-like graphons.\n\n**Q4**: “The authors use Euclidean distance between density matrices as a proxy for cut distance. Could they provide more justification or intuition for why this is a reliable proxy in this context? Are there limitations or assumptions that make this approximation valid?”\n\n**A4**: The cut distance quantifies the similarity between two graph adjacency matrices or functions by considering all possible labellings of their nodes. It requires finding optimal alignment between graphons, which is computationally intractable (NP-hard).\n\nWe follow a degree-based alignment algorithm, sorting-and-smoothing (SAS) [7], which includes two main steps:\n\n(1) Alignment, where we sort neurons in each layer by their degree. This provides a canonical alignment that is provably near-optimal for graphons with monotone degree sequences [7]. For pruning methods that inherently rank connections by 'importance', this degree-based sorting is particularly effective and leads to structures with near-monotone degree sequences.\n\n(2) Distance computation where the cut distance between graphs is directly related to the Euclidean distance between their corresponding density matrices after alignment. Density matrices are constructed by partitioning sorted graphs into a grid of intervals, then computing the average edge density within each interval.\n\nBy doing this, we are no longer comparing arbitrary node labellings but are instead comparing the aligned structures of graphs. The Euclidean distance then measures the difference in edge density between structurally equivalent regions. While this is not formally the cut distance, it is a highly reliable proxy for measuring differences in large-scale connectivity patterns, which is precisely the goal of our analysis.\n\n### References\n[1] Yang et. al., 2020. \"Feature learning in infinite-width neural networks.\"\n\n[2] Yang et. al., 2022 \"Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer.\"\n\n[3] Blake et. al., 2025 \"u-$\\mu $ P: The Unit-Scaled Maximal Update Parametrization.\"\n\n[4] Mocanu et. al., 2018. \"Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science.”\n\n[5] Pham et. al., 2023 \"Towards data-agnostic pruning at initialization: what makes a good sparse mask?.”\n\n[6] Patil et. al., 2021 \"Phew: Constructing sparse networks that learn fast and generalize well without training data.\"\n\n[7] Chan et. al., 2014 \"A consistent histogram estimator for exchangeable graph models.\""}, {"review_id": "gAM2iHCpqV", "reviewer": "Reviewer_xZi5", "summary": "This submission proposes to study the sequence of masks produced by a given\npruning method as the width of networks goes to infinity.\nAuthors put forward the hypothesis that each pruning method yields masks which\nadmit a limit (in the form of a graphon), somehow caracteristic of the pruning\nmethod. The paper does not provide theoretical arguments hinting at the\nexistence of such limits, but proposes experimental evidence supporting this\nclaim for MLPs of various depths, across a wide range of pruning methods.", "questions": "What *are* the challenges in training sparse networks (cited as early as L2\nin the abstract and throughout the paper) ? Are there really any ?\nThe claim on L4 that different pruning mechanisms yield slightly different results\nis backed by evidence, but the true difficulty inherent to sparse network\ndoes not seem to have such support, as best seen perhaps on the paper's own\nexperiments in Figure 2, where all training methods consistently reach nearly-identical\nloss values.\nThe claim that particular subnetworks are both harder to train but as good\nas the unpruned networks they are extracted from (popularized by Frankle 2019\nunder the \"Lottery Ticket Hypothesis\" name), has been contested ever since its\nintroduction, including the very same year, by Zhuang Liu et al (2019, \"Rethinking\nthe value of network pruning\") able to train any random sparse network, at\nscales much larger than the initial Lottery Ticket experiments, and consistently so,\nprovided hyperparameters are correctly tuned.\n\nFollowing up on the previous question, how are the hyperparameters selected in\nthe training experiments presented ? If one subscribes to the idea that very\nsparse networks are not particularly harder to train, but on the contrary just\nrequire a new hyperparameter sweep, the results of Figure 2 are insufficient\nto reach conclusions on the loss of accuracy obtained per sparsity level.\nCan you extend these experiments to at least include small variations around\nthe hyperparameters selected, and would those alter the conclusions regarding\nspectral properties ?\n\nIf the submission is accepted, and if the camera-ready limit is increased\nto ten pages, as was previously the case, it seems to me that Figures 8 & 9\nare worth the space they will take in the main text, they offer more valuable\ninsights into qualitatively different shapes across layers.\nHowever, regardless of their position, the size of these figures needs to be\nincreased, they are hardly readable in their current state and I highly doubt that\nthe legends would be readable in print.", "limitations": "The discussion of limitations is only present in the appendix, which is a\nquestionnable choice, and although the gap between finite widths are graphon\nlimits of infinite widths is addressed, the question of whether the experiments\nwith relatively small width are really representative of the near-infinite\nwidth regime is left unaddressed.\nThe restriction of these methods to (essentially) MLPs is not stressed enough,\nneither in the main text, nor in the appendix\n\nThe restrictions to bounded value and well-behaved connectivity to make graphon\nlimits definable are substantial, but clearly outlined in the text.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "The hypothesis and the underlying intuition are clearly laid out,\nwith convincing a-priori arguments, and backed by a thorough litterature review.\nThe experiments cover a wide range of pruning methods, although the historical\nmethods such as magnitude-based and hessian-based pruning are absent,\nand all results support not only the idea that there is indeed a limit graphon,\nbut additionally that this graphon is very informative, allowing for instance\nto clearly distinguish the SynFlow method from earlier SNIP and GraSP algorithms.\n\nThe computation of the Graphon NTK is soundly carried out, although\nthe collision in the choice of name with a previous use of the same term\nis unfortunate. The estimations of its spectrum yield valuable insights into\npruning methods, and both tools have a high potential to become standard\nanalysis tools in the design of new pruning algorithms.\n\nIf anything, the experiments are a little lacking in size. In Figure 1,\na width of 2000 is not particularly large for a network which additionally\ndoes not have to be trained since all pruning methods are compared at initialization,\nsurely this is an experiment which can be pushed to 10^4 width, given a reasonable\ncomputation budget.\nIn Figure 3, only 128 samples to estimate the NTK is also a much smaller\nbatch size than the limit of what should be reasonably practical even\nwithout GPUs on a day-long experiment.\nIn Figure 2, the choice to include the whole training window\nobscures the comparison of values obtained. Having one or many plots at 200 update\nsteps with performance vs sparsity for each method would have been more striking\nand easier to read values from.\nAppendix C does provide details on material and methods, but does not report\ntraining times, limiting the evaluation of whether it is reasonable to ask\nfor experiments at larger scale.\n\nI wholeheartedly support the publication of this work, this is clearly\namong the top papers that I expect to read from the conference.", "was_revised": true, "final_justification": "This submission proposes new tools to analyze pruning methods in deep learning models, by observing that many pruning methods of the literature admit a specific kind of limit, defined through a graphon. The existence and qualitatively different behaviors of different pruning methods are supported by reproducible experiments, and results are consistent with the presented theory and intuition.\n\nThe theoretical justifications for the existence of the limit are not particularly strong, and the definition presented is restricted to multi-layer perceptrons, which could limit impact. However, MLPs do exhibit very similar behaviors to richer architectures under pruning, which make it a good model to develop new tools to understand precise effects of various pruning methods. The connection of the limit graphon-ntk with observed properties such as convergence speed at high sparsity is also backed by convincing evidence. This is largely sufficient to justify the use of this tool to understand the effects of structured pruning at large widths, and further research efforts to extend this construction to richer architectures.\n\nThe spectral analysis of the limit graphon-ntk and the visualization of the shape of the limit graphon have potential to impact theoretical understanding of neural network convergence, inform domain-specific choices of pruning methods by practitioners, and even help develop new pruning methods with specific properties such as ensuring fast convergence at high sparsities or preserving specific eigenspaces of tangent kernels after pruning.", "rebuttal": "We sincerely appreciate the reviewer for recognising our work. We are also truly grateful for your constructive feedback. Below are our answers to your concerns and questions:\n\n**Q1**: “In Figure 1, a width of 2000 is not particularly large, …, surely this is an experiment which can be pushed to 10^4 width?”; “Appendix C does provide details on material and methods, but does not report training times, limiting the evaluation of whether it is reasonable to ask for experiments at larger scale.”\n\n**A1**: We have pushed the size of the network to $10^4$, unfortunately, this year NeurIPS does not allow submitting additional files/links. Thus, we can not provide the visualisation of graphon convergence. To illustrate the convergence, in Table 1, we present the Euclidean distance between histograms, in which we consider a network with size $10^4$ as a reference at 80% sparsity. All methods show a decreasing trend as width increases.\n\nRegarding the complexity, the SAS [1] algorithm requires sorting neurons based on degree taking O(nlogn) and O(n^2) in terms of memory. In practice, given a sparsity and pruning method, it takes ~120 minutes for n=$10^4$ in our machine with Dual AMD EPYC 7443 CPU and Nvidia A10 GPU.\n\nTable 1: Euclidean distance between histograms taking n=10000 as reference at 80% sparsity.\n| Method   | 100   | 500   | 1000  | 2000 | 10000 |\n|----------|:-----:|:-----:|:-----:|:-----:|:-----:|\n| SNIP     | 17.16 | 4.52  | 1.52  | 0.21 | 0.00  |\n| GraSP    | 17.82 | 2.56  | 2.22  | 2.23 | 0.00  |\n| SynFlow  | 30.74 | 16.39 | 12.72 | 9.73 | 0.00  |\n\n**Q2**: “In Figure 3, only 128 samples to estimate the NTK is also a much smaller batch size than the limit of what should be reasonably practical”\n\n**A2**: We have scaled the samples to 1024 with a hidden size of 1024 and also 512 with a 2048 hidden size. We observe a similar trend in the spectral metrics, where SNIP and SynFlow have higher energy concentration at top eigenvalues as illustrated in Table 2,3 below. We will incorporate this new result into the next version of the manuscript.\n\nTable 2: Energy Concentration in top5 eigenvalues in setting 1024 samples and 1024 hidden size.\n| Method \\ Sparsity (%) | 50 | 60 | 70 | 80 | 90 | 95 |\n|:----------------------|:---:|:---:|:---:|:---:|:---:|:---:|\n| Random | 0.4509 | 0.4584 | 0.4538 | 0.4756 | 0.4694 | 0.4644 |\n| SNIP | 0.4584 | 0.4789 | 0.4851 | 0.4996 | 0.5379 | 0.6027 |\n| SynFlow | 0.4641 | 0.4733 | 0.4853 | 0.4985 | 0.5121 | 0.6592 |\n\nTable 3: Energy Concentration in top5 eigenvalues in setting 512 samples and 2048 hidden size.\n| Method \\ Sparsity (%) | 50 | 60 | 70 | 80 | 90 | 95 |\n|:----------------------|:---:|:---:|:---:|:---:|:---:|:---:|\n| Random | 0.4652 | 0.4641 | 0.4629 | 0.4641 | 0.4658 | 0.4709 |\n| SNIP | 0.4670 | 0.4671 | 0.4768 | 0.4957 | 0.5480 | 0.6064 |\n| SynFlow | 0.4597 | 0.4687 | 0.4942 | 0.4858 | 0.5154 | 0.6433 |\n\n**Q3**: “In Figure 2, the choice to include the whole training window obscures the comparison of values obtained. Having one or many plots at 200 update steps with performance vs sparsity for each method would have been more striking and easier to read values from”\n\n**A3**: We thank the reviewer for the suggestions. The reason we included the training window is to compare the convergence rate of networks produced by different pruning methods.\nWe add a table to show the final training loss at step 200 versus the sparsity levels for each pruning method as below:\n\nTable 4: Loss at step 200 with different sparsity levels and PaI methods.\n| Method \\ Sparsity (%) | 0 | 60 | 70 | 80 | 90 |\n| --- | :---: | :---: | :---: | :---: | :---: |\n| Random | _ | 0.24 ± 0.07 | 0.27 ± 0.07 | 0.30 ± 0.08 | 0.34 ± 0.08 |\n| SNIP | _ | 0.24 ± 0.06 | 0.25 ± 0.06 | 0.26 ± 0.06 | 0.31 ± 0.07 |\n| SynFlow | _ | 0.25 ± 0.05 | 0.26 ± 0.05 | 0.28 ± 0.05 | 0.30 ± 0.06 |\n| Dense | 0.21 ± 0.07 | _ | _ | _ | _ |\n\n\nBesides, we would like to refer the reviewer to Appendix D in the manuscript for the comparison of training curves of different sparsity levels for each pruning method. Those figures indicate that increasing sparsity generally slows the convergence.\n\n**Q4**: “What are the challenges in training sparse networks...? The claim that particular subnetworks are both harder to train but as good as the unpruned networks... has been contested... by Zhuang Liu et al (2019, 'Rethinking the value of network pruning') able to train any random sparse network... provided hyperparameters are correctly tuned.”\n\n**A4**: We thank the reviewer for bringing up this important work by Zhuang Liu et al [2], which has shown that even random sparse networks can be trained to high accuracy if hyperparameters are meticulously tuned for each specific sparsity level. Our claim about the 'challenge' of sparse training is not that it is impossible, but that structured sparsity, as induced by methods like SNIP and Synflow, leads to architectures that are inherently more trainable with standard, less-tuned hyperparameters.\n\nOur framework provides a theoretical explanation for this phenomenon:\n\n- Random Pruning: Yields a constant graphon, which uniformly scales down the NTK. This slows learning for all functional modes equally, which is also observed in [3].\n- Structured Pruning (e.g., SNIP, Synflow): Yields structured graphons that concentrate the NTK's energy into dominant eigenmodes. This means the network tends to learn key functions quickly, even with a standard set of hyperparameters.\n\nSo, the 'challenge' is one of trainability and robustness to hyperparameters. Our Graphon NTK analysis helps explain why some sparse structures are inherently 'better' initialisations for training than others. We will revise our introduction to state this more precisely, referencing the work of [2] to provide proper context.\n\n**Q5**: “How are the hyperparameters selected in the training experiments presented?”, “Can you extend these experiments to at least include small variations around the hyperparameters selected, and would those alter the conclusions regarding spectral properties?”\n\n**A5**: For the experiments in Figure 2, we used a fixed, standard set of hyperparameters (e.g., Adam optimiser [4] with a learning rate (lr) of 0.001) across all pruning methods and sparsity levels. Our goal was not to find the peak possible performance for each sparse mask, which would indeed require a full hyperparameter sweep per setting. Instead, our goal was to compare the inherent trainability of the architectures produced by each method under a single, consistent training protocol.\n\nFollow your suggestion, we have run with different lr values and we report the training loss of the first 100 steps with two lr 0.01 and 0.005 in Table 5,6, respectively (due to space limit). The first (resp. last) two rows are 80% (resp. 90%) sparsity.  \n\nTable 5: Training loss with 0.01 lr\n|  | 0    | 10   | 20   | 30   | 40   | 50   | 60   | 70   | 80   | 90   | 100  |\n| - |:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|\n| Random  | 2.30 | 1.61 | 1.07 | 0.70 | 0.51 | 0.42 | 0.38 | 0.34 | 0.35 | 0.33 | 0.31 |\n| SNIP    | 2.31 | 1.61 | 0.98 | 0.67 | 0.51 | 0.42 | 0.36 | 0.33 | 0.32 | 0.33 | 0.30 |\n| Random  | 2.30 | 1.75 | 1.12 | 0.78 | 0.58 | 0.46 | 0.41 | 0.36 | 0.31 | 0.31 | 0.32 |\n| SNIP    | 2.31 | 1.53 | 0.93 | 0.63 | 0.49 | 0.40 | 0.34 | 0.32 | 0.34 | 0.30 | 0.28 |\n\nTable 6: Training loss with 0.005 lr\n|  | 0    | 10   | 20   | 30   | 40   | 50   | 60   | 70   | 80   | 90   | 100  |\n| - |:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|\n| Random  | 2.30 | 1.70 | 1.09 | 0.76 | 0.56 | 0.45 | 0.39 | 0.36 | 0.31 | 0.30 | 0.31 |\n| SNIP    | 2.31 | 1.48 | 0.91 | 0.60 | 0.46 | 0.40 | 0.33 | 0.31 | 0.30 | 0.28 | 0.26 |\n| Random  | 2.30 | 2.01 | 1.25 | 0.83 | 0.61 | 0.51 | 0.45 | 0.41 | 0.36 | 0.34 | 0.35 |\n| SNIP    | 2.31 | 1.52 | 0.93 | 0.64 | 0.50 | 0.43 | 0.36 | 0.32 | 0.31 | 0.29 | 0.28 |\n\n\nAlthough increasing lr speeds up the training of subnetworks, results from Table 5,6 still align with Figure 2, which show clear differences in convergence speed between Random and SNIP, therefore directly support our argument: with hyperparameters held constant, the network's structural properties (as encoded by the graphon) become the important factor in its training dynamics.\n\n**Q6**: “It seems to me that Figures 8 & 9 are worth the space they will take in the main text, they offer more valuable insights into qualitatively different shapes across layers. … the size of these figures needs to be increased, they are hardly readable ….”\n\n**A6**: We are grateful for these suggestions. We will definitely improve the visualisation and incorporate it into the next version.\n\n**Q7**: Absence of magnitude-based pruning methods\n\n**A7**: We have run additional experiments with Magnitude Pruning at Initialisation (PaI). Through the experiment, we find that Magnitude produced masks converge to constant graphons. From a purely structural perspective, Magnitude Pruning is identical to that of Random Pruning in case all the weights are i.i.d. initialised. In particular, when all weights are i.i.d., the probability that any single weight $w_{ij}$ has a magnitude $|w_{ij}| > Threshold$ is the same for all (i, j). Let this probability of preserving a connection be $s$. By definition, a graph where every edge exists with an independent and identical probability $s$ is an Erdős–Rényi random graph, $G(n, s)$. The well-known limit of a sequence of Erdős–Rényi graphs is a constant graphon $\\mathcal{W}(u, v) = s$. We will incorporate these results into our next version.\n\n\n### References\n\n[1] Chan et. al., 2014 \"A consistent histogram estimator for exchangeable graph models.\"\n\n[2] Zhuang Liu, et. al., 2019 \"Rethinking the value of network pruning.\"\n\n[3] Yang et. al., 2023. \"On the neural tangent kernel analysis of randomly pruned neural networks.”\n\n[4] Kingma et. al., 2014. \"Adam: A method for stochastic optimization.\""}, {"review_id": "fWGYGI3tMS", "reviewer": "Reviewer_xsUw", "summary": "This paper introduces a theoretical framework for analyzing sparse neural networks obtained through various pruning methods. The authors link pruning masks to graphons in the infinite-width limit to examine connectivity patterns induced by different pruning techniques. Building on this, they propose Graphon NTK to study the training dynamics and spectral properties of sparse networks. Overall, this paper is well-structured and provides compelling experimental results.", "questions": "- The authors validate their method under only a single architecture and dataset. I am curious whether consistent patterns shown in the experiments appear when using different datasets or advanced architectures like transformers.\n- I am also curious about the graph limit and spectral results of magnitude-based pruning approach.", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- The connection between graphon limit theory and pruning mask is sound and novel. I believe this work would be a valuable contribution as analyzing pruning behaviors has been still underexplored.\n- The experimental results are convincing, particularly the distinct differences in graph limit and spectral results between random pruning and other methods.\n\n**Weaknesses**\n\n- While the paper introduces an elegant theoretical tool to analyze pruning methods, it does not translate this framework into clear practical guidelines. For instance, it remains unclear how one can leverage graphon NTK analyses to select or design pruning strategies in real-world scenarios. The paper would benefit from connecting its theoretical insights to concrete, practical contributions.\n- It would be interesting to add a discussion comparing the proposed graphon limit analysis with edge-pruning methods [1-3] in the GNN domain. Although the purpose is different, analyzing pruning masks as continuous adjacency matrices in latent space shares conceptual similarities with edge pruning, where one explicitly removes edges from the graph adjacency matrix in data space.\n- If I understand correctly, the proposed framework does not appear to be applicable to dynamic pruning [4-5], which may limit its applicability.\n\n**References**\n\n---\n\n[1] Chen, Tianlong, et al. \"A unified lottery ticket hypothesis for graph neural networks.\" ICML 2021\n\n[2] Seo, Hyunjin, Jihun Yun, and Eunho Yang. \"TEDDY: Trimming edges with degree-based discrimination strategy.\" ICLR 2024\n\n[3] Zhang, Guibin, et al. \"Graph lottery ticket automated.” ICLR 2024\n\n[4] Evci, Utku, et al. \"Rigging the lottery: Making all tickets winners.”, ICML 2020\n\n[5] Kusupati, Aditya, et al. \"Soft threshold weight reparameterization for learnable sparsity.\" ICML 2020", "was_revised": true, "final_justification": "After the rebuttal, I decide to maintain my rating in favor of acceptance. The authors provide rigorous and well-reasoned responses, along with clear explanations and additional experimental results that address most of my concerns.\n\nResolved issues include clarification of the practical relevance, relation to GNN pruning, application to dynamic pruning, and magnitude pruning analysis. However, the proposed work remains limited to MLPs, which constrains its broader applicability.\n\nNevertheless, I would like to assign more weight to the originality of the authors' contributions and in-depth rebuttal responses. I believe this paper makes self-contained contribution for future principled pruning analysis.", "rebuttal": "We thank the reviewer for your constructive feedback and recognition of our work. Below are our responses to the potential concerns and questions:\n\n**Q1**: “While the paper introduces an elegant theoretical tool to analyze pruning methods, it does not translate this framework into clear practical guidelines. For instance, it remains unclear how one can leverage graphon NTK analyses to select or design pruning strategies in real-world scenarios. The paper would benefit from connecting its theoretical insights to concrete, practical contributions.”\n\n**A1**: Regarding the choice of pruning strategies, our work lays the necessary groundwork for a more principled approach. By understanding how a method's graphon shapes the NTK's spectrum (as shown in Figure 3), one can move towards *designing pruning strategies by optimising for a target graphon* that yields desirable spectral properties (e.g., high energy concentration for faster convergence). This moves the field beyond pure heuristics. Recently, NTK spectral properties have already been leveraged to design pruning methods that preserve the eigenspectrum of “Dense” NTK [6]. It is expected that our Graphon NTK limit will also enable optimisation of sparse networks’ configurations and lead to concrete pruning methods.\n\nFurthermore, our analysis opens the door to designing more efficient training schemes for sparse networks. For example, one could design learning rate schedulers or weight initialisations based on the eigenfunctions of the Graphon NTK to further accelerate training for a given sparse architecture. However, identifying how specific weights contribute to particular eigenfunctions is inherently non-trivial due to the non-linearity of networks. While detailed algorithms for these are left for future work, our framework establishes its theoretical feasibility.\n\n**Q2**: “It would be interesting to add a discussion comparing the proposed graphon limit analysis with edge-pruning methods [1-3] in the GNN domain. Although the purpose is different, analyzing pruning masks as continuous adjacency matrices in latent space shares conceptual similarities with edge pruning, where one explicitly removes edges from the graph adjacency matrix in data space.”\n\n**A2**: We thank the reviewer for this insightful suggestion. The conceptual parallel between our work and edge-pruning in GNNs is indeed interesting, and we will add a discussion to the related work section in the final version. The critical difference lies in the object of pruning. In GNN pruning [1-3], one prunes the edges of the input data graph to improve efficiency or performance on that graph. In our work, we analyse the pruning of the model's own weight connections, which form an implicit connectivity graph within the neural network architecture itself. While both can be viewed through the lens of graph theory, the former relates to data preprocessing/regularisation, while our work relates to finding efficient model architectures. This distinction is crucial, and exploring the transfer of ideas between these two domains is a promising direction.\n\n**Q3**: “If I understand correctly, the proposed framework does not appear to be applicable to dynamic pruning [4-5], which may limit its applicability.”\n\n**A3**: Our current analysis focuses on pruning at initialisation (PaI) because it provides the cleanest setting for developing the theory, fixed masks enable rigorous infinite-width analysis. Extending this framework to dynamic pruning [4,5] is also possible to carry out graphon NTK analysis, but it requires significant additional ideas and technical details. For example, we would need to handle weight-mask co-evolution, significantly complicating the analysis. We believe understanding the \"base case\" of fixed masks is essential before tackling time-varying masks.\n\nFor dynamic sparsification, it has been observed that some dynamic sparse training methods evolve from random to scale-free connectivity [7]. From our perspective, this can be viewed as the graphon evolving from a constant to a structured, power-law pattern. One could even recast dynamic sparsification as a *gradient flow problem on the continuous space of graphons.* These extensions require significant theoretical advancements, which we believe are well worth pursuing.\n\n**Q4**: “The authors validate their method under only a single architecture and dataset. I am curious whether consistent patterns shown in the experiments appear when using different datasets or advanced architectures like transformers.”\n\n**A4**: Our choice of MLP architectures provides a controlled setting to clearly establish our central theoretical claims. Extending this framework to more complex architectures like CNNs or Transformers is a primary goal for future work, but it presents significant and non-trivial challenges. To draw an analogy, extending the original dense NTK [8] from MLPs to architectures with convolution or attention required significant new technical developments [9,10] . Similarly, deriving a Graphon NTK for these architectures would involve handling concepts like convolution (for CNNs) or data-dependent connectivity (for Transformers), each constituting a substantial research project. However, we believe the fundamental principle that pruning masks converge to graphon limits is a general one.\n\n**Q5**: “I am also curious about the graph limit and spectral results of magnitude-based pruning approach.”\n\n**A5**: We thank the reviewer for this insightful question. Through the experiment, we find that Magnitude produced masks converge to constant graphons. From a purely structural perspective, Magnitude Pruning is identical to that of Random Pruning in case all the weights are i.i.d. initialised. In particular, when all weights are i.i.d., the probability that any single weight $w_{ij}$ has a magnitude $|w_{ij}| > Threshold$ is the same for all (i, j). Let this probability of preserving a connection be $s$. By definition, a graph where every edge exists with an independent and identical probability $s$ is an Erdős–Rényi random graph, $G(n, s)$. The well-known limit of a sequence of Erdős–Rényi graphs is a constant graphon $\\mathcal{W}(u, v) = s$. We will incorporate these results in the next version of our paper.\n\nHowever, Magnitude PaI is not just a masking operation, it is a *selection procedure that biases the initial weight distribution.* The surviving weights are from a truncated distribution with a strictly larger variance than the original. Hence, while Magnitude PaI’s graphons are identical to Random Pruning, a larger variance in initial weights leads to a larger variance in pre-activations, which in turn produces \"stronger\" kernels with larger eigenvalues. This helps subnetworks produced by Magnitude learn in low-rank spaces faster, leading to faster initial training compared to Random Pruning.\n\nTo demonstrate this, in Table 1, 2, we provide new experimental results at 80% sparsity (we will provide full results in our next version since we can not provide additional files/urls for NeurIPS rebuttal this year). This trend is consistent across different sparsity levels.\n\nTable 1: Top 10 eigenvalues (values in log scale)\n| Method | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Random | 19.09 | 16.71 | 16.48 | 16.30 | 16.17 | 15.99 | 15.87 | 15.71 | 15.60 | 15.44 |\n| Magnitude | 23.16 | 20.79 | 20.59 | 20.43 | 20.31 | 20.14 | 20.03 | 19.79 | 19.71 | 19.59 |\n| SNIP | 20.23 | 18.03 | 17.90 | 17.73 | 17.51 | 17.31 | 17.13 | 17.02 | 16.89 | 16.76 |\n\nTable 2: Training loss\n| Method | 0 | 10 | 20 | 30 | 40 | 50 | 60 | 70 | 80 | 90 | 100 | 110 | 120 | 130 | 140 | 150 | 160 | 170 | 180 | 190 |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Random | 2.30 | 2.21 | 1.80 | 1.22 | 0.85 | 0.63 | 0.52 | 0.47 | 0.42 | 0.40 | 0.40 | 0.41 | 0.37 | 0.38 | 0.35 | 0.36 | 0.32 | 0.31 | 0.33 | 0.32 |\n| Magnitude | 2.30 | 2.16 | 1.67 | 1.10 | 0.76 | 0.58 | 0.49 | 0.45 | 0.40 | 0.38 | 0.38 | 0.39 | 0.35 | 0.36 | 0.34 | 0.34 | 0.31 | 0.30 | 0.31 | 0.30 |\n| SNIP | 2.31 | 1.85 | 1.12 | 0.74 | 0.59 | 0.51 | 0.43 | 0.41 | 0.40 | 0.37 | 0.34 | 0.32 | 0.32 | 0.30 | 0.28 | 0.27 | 0.28 | 0.27 | 0.25 | 0.24 |\n\n\nAlthough kernels from Magnitude-pruned networks have higher eigenvalues than SNIP, SNIP provides higher energy concentration in the top eigenvalues, as illustrated in Table 3, which helps SNIP learn faster than Magnitude\n\nTable 3: Energy Concentration in top eigenvalues\n| Method    | Top 5      | Top 10     |\n|-----------|:-------------------:|:-------------------:|\n| Random    | 0.4756 ± 0.0140     | 0.5404 ± 0.0139     |\n| Magnitude | 0.4673 ± 0.0097     | 0.5335 ± 0.0097     |\n| SNIP      | 0.4996 ± 0.0186     | 0.5758 ± 0.0171     |\n\n\n### References\n[6] Wang et. al., 2023. \"NTK-SAP: Improving neural network pruning by aligning training dynamics.\"\n\n[7] Mocanu et. al., 2018. \"Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science.”\n\n[8] Jacot et. al., 2018. \"Neural tangent kernel: Convergence and generalization in neural networks.\"\n\n[9] Arora et. al., 2019 \"On exact computation with an infinitely wide neural net.\"\n\n[10] Greg Yang, 2020. \"Tensor programs ii: Neural tangent kernel for any architecture.\""}]

Author rebuttal:

Meta-review: This paper proposes to study the sequence of masks produced by a given pruning method as the width of networks goes to infinity. The paper puts forward the hypothesis that some pruning methods yield masks which admit a limit in the form of a graphon, somehow characteristic of the pruning method. The paper does not provide theoretical arguments hinting at the existence of such limits, but proposes experimental evidence supporting this claim for MLPs of various depths, across a wide range of pruning methods. Under this hypothesis, the paper proposes a graphon NTK approach to study the training dynamics and spectral properties of networks under pruning. 

Overall, the reviewers were consistently positive about this paper, praising the novel idea of analyzing pruning via graphon analysis, which is backed by elegant theory, and the supportive experimental results. 

This is a high quality paper and a clear “accept.”
Decision: Accept (spotlight)