CoNNect: Connectivity-Based Regularization for Structural Pruning of Neural Networks
Authors: Christian P.C. Franssen
Jinyang Jiang
Yijie Peng
Bernd Heidergott
Venue: Accepted by TMLR ()

Abstract: Pruning encompasses a range of techniques aimed at increasing the sparsity of neural networks (NNs). These techniques can generally be framed as minimizing a loss function subject to an $L_0$ norm constraint. This paper introduces CoNNect, a novel differentiable regularizer for sparse NN training that ensures connectivity between input and output layers. We prove that CoNNect approximates $L_0$ regularization, while preserving essential network structure and preventing the emergence of fragmented or poorly connected subnetworks. Moreover, CoNNect is easily integrated within established structural pruning strategies. Numerical experiments demonstrate that CoNNect can improve classical pruning strategies and enhance state-of-the-art one-shot pruners, such as DepGraph and LLM-pruner.

Published in Transactions on Machine Learning Research (09/2025)
CoNNect: Connectivity-Base Regularization for Structural Pruning of Neural Networks
Christian Franssen*
c.p.c.franssen@mm..ll
Vrije Universiteit Amsterdam
Jinyang Jiang' Peking University
Yijie Peng Peking University
pengyüjieppu..ddu.cmm
Bernd Heidergott
b.f.heidergott@vv..ll
Vrije Universiteit Amsterdam
Reviewed on OpenReview: https: //
// operreveew.eee.... net/ forum? id- 7BuEp
Abstract
Pruning encompasses a range techniques aimed at increasing the sparsity neural net- works (NNs). These techniques can generally be framed minimizing loss function subject to To norm constraint. This paper introduces CoNNect, novel differentiable regularizer for sparse NN training that ensures connectivity between input and output layers. We prove that CoNNect approxinates Lo regularization, while preserving essential network structure and preventing the emergence of fragmented or poorly connected subnetworks. Moreover, CoNNect casily integrated with established structural pruming strategies. Numerical ex- periments demonstrate that CoNNect can improve classical pruning strategies and enhance slate-of-the-arl one-shot pruners, such as DepGraph and LLM-pruner.
1 Introduction
Machine learning models, such neural networks (NNs), have seen rapid growth in recent years, leading to significant increases in model performance. However, as the footprint of these models increases (Patterson)
le al. 2021), there is a growing need to develop more energy-efficient approaches to machine learning that can balance computational performance with environmental sustainability.
An effective technique for reducing a model's computational effort and memory burden neural network pruning. Pruning refers to the process of systematically eliminating parameters that contribute little network performance. The resulting sparse NNs have attracted significant interest in recent years due to their ability to boost computational efficiency and minimize memory consumption while preserving even improving model performance {LeCun et al. 1989 Hassibi et al. 1993 Frankle & Carbin 2019).
Varions techniques have been proposed to achiove sparsity in NNs, such as unstructured pruning, which involves selectively removing individual weights from the network. Pruning weights based on their individual magnitudes is a classic example of unstructured pruning [Leuun al. 1989 Hassibi et al. 1993 Hagiwara
1993 Han al. 2015), where connections are removed solely based on the absolute value of each weight, Although it provide highly sparse networks, it often results in irregular memory access patterns, which can be difficult to optimize in hardware implementations. Recently, comi-structured pruning has emerged as an approach to balance granularity and efficiency by removing weights within predefined patterns or
*Corresponding authors.

---

Published in Transactions on Machine Learning Research (09/2025)
Sun et al. 2024 Fang et al. 2024). While this is promising approach for realixing high accuracy at various sparsity ratios, it presents nuanced trade-offs in inference speed and therefore the computational efficiency of the model. Finally, structured pruning (e.g. see Yuan & Lin (2006;; Huang & Wang (2018); : Anwar et al. (017)) offers a systematic method to remove entire groups or channels of neurons. Techniques like Group Lasso (Yuan & Lin 2006; [Hoeller el. al. 2021] and other
groups [Frantar & Alistarh 2023
structured sparsity learning (Wen et al. 2016 Zhuang et al. 2020) fall into this category; see He & Xiao (2023) for a review. The structured removal of parameters generally leads to an almost equal reduction in computational complexity and inference speed, thus immediately improving computational efficiency. For instance, both at 50% pruning rate, structured pruning (aa et al.) 2023) achieves a 1.85x end-to-end latency acceleration on LLaMA-7B (Touvron et al. (023), whereas semi-structured pruning (Sun et al. 2024) only achieves a 1.24x speedup. Structured pruning is the only paradigm that enables a universal neural network compression without requiring special hardware or software, thereby addressing the primary objective of pruming: accoleration (Cheng al. 2024).
This highlights the practical value of structured pruning, motivating a deeper investigation into the principles that should guide pruning strategies. We believe that pruming should obey the following two axioms (where we identify a NN with a directed, weighted graph):
Axiom (Delete Weights to Improve Computational Efficiency). The graph should be 'small': pruming must significantly reduce the number of weights while minimally impacting accuracy and marimizing computational efficiency.
Axiom 2 (Preserve Neural Network Connectivity). The pruning process must prevent disruptions in the conneclivily of the newral nelwork and preserve the flow of information from impul to oulpul.
The extensive research on pruning neural networks, as more elaborately outlined the literature overview in Section 2 and particularly in review works such as Hoefler al. (2021); He & Xiao (2023), predominantly aligns with the first axiom. However, few methods address Axiom 2 as the impact of weight. removal on overall network connectivity is rarely considered. This negligence
result in pruning that produces highly disconnected networks (Vysogorets & Kempel 2023), or in the most extreme case so-called layer collapse, see Figure [ where the NN becomes completely dysfunctional. A notable exception is SynFlow pruning (Tanaka et al. 2020), a method designed for unstructured pruning at initialization that explicitly considers connectivity by preserving signal flow through the network. We explore SynFlow in more detail in Section 333.1
Figure 1: Magnituude-bsed pruning of NN (left) leads to layer collapse (right).
In this paper, we propose a new regularizer, called CoNNect, that can be used to satisfy both axioms simul- taneously and (i) is differentiable (except in the point zero) and allows for gradient descent optimization, (ii) effectively approximates Lo regularization and guarantees maximally connected network structures stable stationary points, avoiding issues such as layer collapsc. CoNNect is best understood as generalization of the SynPlow principle into a training-time regularizer that promotes network connectivity throughout optimization. It utilizes a weight normalization for its measurement, resulting in weights being restricted to (0, 1), so that the contribution of a path from the input to the output layer to the overall conncctivity of the network goes exponentially quickly to zero unless the weights along the paths are (close to) 1. Hence, when maximizing the connectivity for the normalized weights, we find weight association that prefers fewer
2

---

Published in Transactions on Machine Learning Research (09/2025)
direct paths over many paraltel paths, while focusing on the connectivity of the imput with the output layer. Importantly, we show how CoNNect is applicable for structural pruning, and seamlessly integrates within one-shot pruming pipelines.
We demonstrate CoNNect's efficacy through a series of numerical examples. First, we provide an illustrative unstructured pruning example in which we show that pruning strategies like magnitude-pruiigg (LeCun et al. 1989; Hassibi et al., 1993) and SynFlow (Tanaka et al. 2020) can benefit from CoNNect regularization during training. Here, it outperforms L and L2 regularization in terms of both accuracy and stability (measured by the frequency of layer collapse). Then, we specifically conduct numerical experiments the integration of CoNNect within structured pruning, given its potential to achieve substantial improvements in computational efficiency, thus satisfying Axiom]] [] We apply CoNNect regularization on a channel level when training a Graph Neural Network (GNN), achieving improved performance compared to L1 and L2 regularization. Then, we integrate CoNNect into state-of-the-art methods for structural pruning of pre- trained models, such as DepGraph (Fang ct al., 2023), and LLM-pruner (Ma ct al. 2023), a one-shot pruning method for Large Language Models (LLMs). Our numerical results demonstrate consistent performance improvements compared with other methods obtaining similar gains in computational efficiency.
2 Related Work
The concept of pruning NNs dates back to the carly 1990s. The seminal work by LeCun et al. 19899 on Optimal Brain Damage introduced the idea of pruning by removing weights that contribute least. lo performance, thus simplifying the network. Hassibi et al. (193) extended this concept with Optimal Brain Surgeon, which provided more sophisticated method for determining which weights to prune based on their impact on the error function. These carly methods laid the foundation for modern pruning techniques, focusing on reducing network complexity while maintaining accuracy.
Regularization-Bassr Pruning (Soft Pruning). Regularization methods play crucial role in pro- moting sparsity by extending the loss function with penalty function that discourages overly complex models. These methods typically either encourage certain components (e.g., weights or filters) to become ineffective, thereby yielding sparsity, or drive them to become functionally redundant, enabling safe removal after training (Ding et al. 2019 Valverde et al., 2024). In the case where regularization encourages sparsity, regularization does not explicitly set the weights to zero but instead reduces their magnitude, allowing them to remain non-zero and potentially become active again if needed. This leads to what is termed soft pruning, where sparsity is encouraged but not strictly enforced through hard weight removal during training. After training concludes, unimportant weights, typically those with the smallest magnitudes, are then pruned Hagiwara, 1993; Gale et al.) 2019). One of the simplest and most widely used methods, L, regularization Tibshirani 1996 He et al. 2017 Yang et al. 2019 De & Doostan 2022 Ziyin & Wang 2023), penalizes the sum of the absolute values of the weights, encouraging many weights to become zero. Moreover, In regularization fails to incorporate considerations from Axiom [ which emphasizes the preservation of neural network connectivity and functionality. This lack of consideration for connectivity can lcad to network that, while sparse, may suffer from disrupted information flow, ultimately impairing its performance. Sim- ilarly, La regularization, another common regularization technique, penalizes the sum of the squares of the weights (e.g., see Hinton (2012); PPhaisangittsaaaul (2016); Loshchilov et al. (2017)). While Le regularization is effective at discouraging large weights, it does not push small weights towards zero, thus failing to induce sparsity in the network. As a result, T22 regularization typically produces networks with small but non-zero weights, which do not benefit from the same computational efficiency gains that a sparse network would offer. Moreover, like L1 regularization, La regularization does not address the need to maintain critical con- nections highlighted by Axiom 2 making it less suitable for tasks where maintaining network connectivity is essential.
Stage-Based Pruning (Hard Pruning). Stage-based pruning strategies are utilized as separate, discrete actions during various stages of model training. These techniques can be implemented before training (Lee) Tanaka et al. 2020 Wang et al. (020). during training (Frankle & Carbin] 2019 Mocanu
et al.]
2019
et
2018
Jayakumar et al. 2020), or after training (Hagiwara) 1993] Thimm & Fiesler 1995 Gale
et al.
2019
He et al.) 2019 Ma et. al. 2023). Stage-based pruning generally does not fundamentally alter
3

---

Published in Transactions on Machine Learning Research (09/2025)
the objective function or the descent direction like regularization does, but instead acts on the model's structure or parameters at specific moments. These kinds of pruning methods can be considered hard pruning approaches, as parameters are explicitly removed. Many different criteria for pruning have been introduced, such as magnitudo-based pruming (Hagiwara 1993 Gale et al. 2019 which involves removing weights with the lowest absolute values and based on the idea that. these weights have the least. impact on the overall performance of the model. More complex criteria have been constructed determine the impact of weight removal, such as first-order (e.g., see Zhou & Si (1999); Molchanov et al. (2017); Sanh et al. (2020) and second-order expansions (LeCun et al.) 1989 Hassibi et al.) 1993 Ma et al. 2023) of the training objective. Specifically, SynFlow (Tanaka et al. 2020 is a method that acheres closely to the principles of Axiom [ focusing on retaining the network's connectivity and functionality during pruming, Unlike magnitude- based Lechniques, SynFlow utilizes a first-order expansion of signal flow to pinpoint and remove weights with minimal impact on the network's overall information flow. This approach ensures that while the network is being pruned, its structural integrity is preserved and the critical pathways in terms of connectivity remain intact. Another approach adopting network-theoretic perspective is Li et al.] (2020), who employ Katz centrality to prune neural network nodes in nonlinear system modeling. Although this method highlights the potential of network measures for guiding pruning decisions, our methodology is fundamentally different and further extends to large-scale NNs.
We conclude the above discussion by noting that the CoNNect regularizer, to be introduced in the next section, can both be used as a soft pruning approach and integrated in hard pruming approaches.
3 Methodology
3.1 Preliminaries
We define a graph g = (V, E), where V denotes the set of vertices (or nodes) and E represents the set of directed links that connect these vertices. A weighted graph has weights Wij for links (i.j) E, where we let W., 0, for (i.j) & E. Neural networks can be described using graph theory by representing them as directed, weighted graphs. In this setting, the vertices V = U... UK. in the graph correspond to the neurons in the network which are organized into distinct subsets corresponding to the different layers V&, for k = 1,..., K. Here, the input nodes V1 represent the neurons in the input layer, the hidden nodes Ve, for k = 2 1,, 1, represent the neurons in the hidden layers, and the output nodes VK represent the neurons in the output layer.
K-
Throughout the paper, we describe neural network 9 using the tuple (W, where W E RIVIIVI is the weighted adjacency matrix of the weights, such that Wiy connects node Ve with node j E Ve+1, and b (bi,..., bivi) is the bias vector. Moreover, we denote the activation of the A + 1th layer by the tensor (I+11) (W) the weights between nodes in Vk, and Vk+1, and b(++1) the biases for the nodes in V4+1- Finally, we denote (1x(),; W,b) as forward pass through the network. Note that, for nolational simplicity, we omit untrainable structures in neural networks, such as residual connections, which will be dealt with in later sections. However, they can be incorporated by treating them as edges with fixed weights.
(R) (()) + b(k+1),, where is the activation function, WK( is the submatrix containing
3.2 Problem Formulation
Let {(x, y denote the training set, where x, x(") represents the input data and ys represents the corresponding label for each of the N samples. Fitting the parameters of a neural network g involves optimizing the network's weights to minimize a loss function v),,y), where و = f(x; b) is the predicted output given an input x.
In this paper, our objective to train a sparse neural network, which can be achieved by inducing sparsity in the network's parameters. A commonly employed approach sparsification is regularization. Regularization involves augmenting the loss function with an additional term that penalizes non-zero elements in the network
4

---

Published in Transactions on Machine Learning Research (09/2025)
parameters. Specifically, the optimization problem can be formulated as:
min W.b
L(0.0) + AR(W),
(1)
where R(W) |,, However, this Lo norm non-convex and leads to combinatorial optimization problem, which is generally NP-hard and computationally intractable for large-scale problems. A more practical alternative is L1 regularization, as in Lasso regression, where R(W) = [|W],,, L1 regularization induces sparsity by shrinking weights to zero, approximating the Lo norm while remaining convex and suitable for gradient-based optimization. However, L1 regularization primarily satisfics Axiom by reducing connections but fails to address Axiom 2 which focuses on preserving network connectivity and ensuring efficient signal flow. This limitation can result in a disconnected or underperforming network when key pathways not maintained.
3.3 CoNNect
To overcome the aforementioned issues, we propose CoNNect, a regularizer that considers the network's overall connectivity, ensuring that the structure contributes to optimal performance. While we first define CoNNect in an unstructured form establish it from the ground up Section 3.3.I| this formulation serves foundation: scaling factors can be introduced to control the granularity of pruning and scamlessly transition to structured regularization, as we detail in Section 3.3.2 This design enables CoNNect to directly support structured pruning objectives while maintaining flexibility and precision.
3.3.1 Weight-Level Regularization
Katz centrality a measure used in network analysis to determine the relative connectivity of a node network by considering both the number and the quality of connections (Katz 1953). Inspired by the connectivity mcasurement in Katz centrality, let us consider the following connectivity matrix for neural network:
((W) =X
where (s indicates the connectivity from node i to node j. The matrix 0(W) is normalized repre- sentation of the network's parameterized weights between successive layers, capturing the relative strength of connections. For E Va and VR+1, the normalized weight is defined as
WWLl
(O(W)) = E EK,DE WE,l
(2)
where Fg denotes the set of edges connecting the kth layer V& to layer V and Wa5 the parameterized weight the connection between nodes i and j. Then, in the context of neural network, we can denote the connectivity by taking the sum of connectivity values between the input and output layers:
roo (W) = Eiev, Ejevx (c((W))
Finally, we argue for the preservation of connectivity (as per Axiom 2 so we aim to maximize the network's overall connectivity. Consequently, we choose the regularizer as:
R(W) = - tot (W),
(3)
which we will refer to as the CoNNect regularizer.
It is important to note that Equation 3 exclusively considers parameterized weights of the nelwork. Since structures like residual connections, which are commonly used in modern neural network architectures, do not contain learnable parameters, they are therefore excluded. This ensures that the CoNNoct regularizer exclusively influences the learnable paraneters of the network without conflating it with architectural short- cuts that not subject to optimization. In practice, this means CoNNect promotes sparse yet effective
5

---

Published in Transactions on Machine Learning Research (09/2025)
parameter usage without penalizing being affected by static residual pathways, thereby exchusively focus- ing the regularization on the components that matter for model capacity and generalization. We refer the reader to Appendix [ for additional implementation details, including the treatment of activation functions, pooling layers, and other architectural components.
CoNNect is effectively the (negative of the) sum of all (multiplicative) normalized weighted paths between nodes in the imput layer Vi and the output layer Vk. It follows that - path with positive parameterized weights between the imput and output layer. Moreover, -
if and only there is
be efficiently computed using single forward pass f(I, W, 0), where I is vector of ones imput, 0 is a vector of zeroes for the biases, and finally taking the sum of the output values. Thus, for batch size of M, the additional time used for computing t (W) is proportional to N Hence, CoNNect can be efficiently applied to large-scale neural networks without incurring significant computational overhead.
In the following, we show that -opo (W) can be used as a surrogate regularizer for the Lo norm to induce sparsily. Taking R(W)- [|Wl,,, in Equation ( it is easy to show that any neural network W that minimizes ||W while connecting the imput layer to the output layer via parameterized weights, i.e., gtot > O, has K - non-zero weights. As the following theorem shows, similar result holds for the CoNNect regularizer as any W minimizing -stta (W) has only K - 3 non-zero weights between layers 2 and 1.
Theorem 3.1. Consider the problem
min W
(4)
for a given network with number of layers K All solutions W to Equation have at most [V]
+
|Kl + K non-zero weights.
Proof. See Appendix A.1
Theorem 3.1 demonstrates that Lo norm regularization can be effectively achieved through the CoNNect regularizer, as the induced sparsity in large neural networks is comparable. Importantly, the difference in the number non-zero elements becomes negligible practice when most input nodes contribute valuable predictive information, and all output nodes are used for accurate classification. Also, the regularizer does not cause input nodes to disconnect, as it does not depend on how many input nodes are connected the second layer. This is beneficial feature. If certain input nodes were disconnected, might happen with other regularizers such L1 regularization, important data features could be disregarded, potentially resulting in suboptimal model performance.
We now show that CoNNect is a well-behaved regularizer in the sense that it docs not have stable stationary points other than its global optima. This ensures that a gradient descent gets stuck in a stationary point of the regularizer, the loss function will always push the solution to leave the stationary point unless the loss function itself is stationary at that point. In the following, we exclusively consider connected W, that is, "da (W)) 0. We do so because with the log transformation the objective diverges as connectivity vanishes, preventing disconnection during training.
First, consider for some (i,j) E Eg let
EmSEE [Wrel W ((no)eE, [Wrel)2
Ow., (B(W))
and, specifically for (1 + (i,5) E Fxi
-IWis
Ow., (O()))) = (E(r)ee, [W

---

Published in Transactions on Machine Learning Research (09/2025)
Observe that differentiating 8(W) with respect to Way only affects the weights in the same layer Wa, Thus, stationary point to Equation 4 solves the following first-order conditions:
Σ (ne)ER,
3w (((W)) ae =0, A (1,5) E E1,
(5)
Σ (ne)e
arr - Ow., (8(W)) a az. (1,5) E E2,
Σ (x
arr Sw,, (8W))) = =0, A (1.J) E Ex-1:
where
h1-1 (O(W)) de Σ mEVK 7ET k=1
hl-1
0. Σ VEV
II 7ET k=
Σ
II
(O(W))
are the connectivity from input layer to a node r and connectivity from node a the output layer, respectively. To satisfy Equation we need:
the weights for the edges En must be assigned to all (O(W)) where j arg maxp Op.
the weights for the edges in Ex, k 2...., K must be assigned to (O(W))ss) where (i,j) e
arg (ax a.pag.:
the weights for the edges in EK-1 must be assigned to (B(W)) where E arg max, a.g.
The set of weight matrices W that satisfy Equation (5) can be more precisely formulated as in Lemma 3.2
Lemma 3.2. Assume neural network with K 3 layers. All stationary points W to Equation 4 that
are connected, i.e., (tot(W) > O, have paths with equal subsequent weights between layers 2 and K on its non-zero paths. That is, for each two paths ,n,"" E Usevi,mevs i,,, such that
II (0(W*)) O, PE
d..., both paths have positive weight, we have (0(W*)),; = (0(W)))))) for all k = 2...., "....,K 2.
Proof. See Appendix
A.2
Using Lemma 3.2 we note that all non-optimal stationary points, ie., ((o((W) < have multiple directions of improvement by simply transferring mass from one path to another. It follows that these solutions are inherently unstable and thus are not a local oplimum. We present a precise statement in Theorem 3.3 where we the proof is omitted as it follows directly form the previous observation.
Theorem 3.3. Assume neural network: with K 3 layers. All stable stationary points W to Equation that are conmected, i.e., slot (W) > O, are global minsmiizers.
As Theorem 3.3 shows, the only stable stationary points of CoNNect those where the weight matrix has only K - non-zero weights between layer 2 and K - 1. Moreover, CoNNect does not have regions of attraction, and thus is well-behaved regularizer for gradient scarch.
As argued earlier, it is recommended to take the logarithm of the connectivity regularizer, i.e.,
- log tot (W))).
(6)

---

Published in Transactions on Machine Learning Research (09/2023)
Channel
d1
Input figure
Channel 2
82
Output feature
Scaled feature
Conv kerne
Figure 2: Illustration of CNN with the scaling factor.
as it ensures that if the neural network tends to disconnect during training, i.e., 4 - 0, Equation (6) approaches oo, hence preventing layer collapse. Moreover, it enhances numerical stability, ensuring that the regularization term remains well-behaved even for varying scales of connectivity.
Once we have trained model with CoNNect regularization, many of the redundant weights will have been pushed to zero. Consequently, we can hard prune the regularized model using pro-ostablished pruning strategies. A well-known strategy is simple magnitude-based pruning (LeCun et al. 1989). which prunes the smallest weights in absolute value. Alternatively, we can use SynFlow pruning (Tanaka et al. 2020), which prunes the neural network's weights according to synaptic saliency scores:
4J (f (0(W)) a. (0((W) az-
and eliminate the weights with the smallest 4J values.
3.3.2 Channel-Level Regularization
The regularizer introduced in Section 3.3.1 was explicitly defined on the weights of the neural network (effectively making it unstructured pruning approach). In this section, show the required modifications to extend it to structured pruning. To this end, (e.g,, neurons, channels, elc.) that we want to prune (Huang & Wang 2018). In the following, we explain how to include structured pruning at the channel level in, e.g., Convolutional Neural Networks (CNNs) and Graph Ncural Networks (GNNs), but this can be naturally extended to any parallel structures neural networks, such as nodes, but also entire block structures.
introduce a scaling factor for the output of structures
Neural networks that utilize channels are designed to process multi-dimensional data where information is structured across multiple fcature dimensions, such as color channels in images or frequency bands in audio. For example, CNNs are specialized type of neural network designed to process grid-like data such as images. These images can be represented using tensor X E Rdxhxw where d refers to the number of channels (e.g., RGB for color images) and h and w refer to the height and width of the image respectively. A standard CNN consists of (several) convolutional layers followed by an activation function (e.g., ReLU), and pooling layers that reduce spatial dimensions while preserving important features. Convolutional layers transform the tensor into a set of fcature maps through a scries of learned filters (also known as kernels). Each convolutional layer in the CNN applies these filters to local regions of the input, capturing spatial hierarchies and patterns like edges, textures, and more complex shapes as the network deepens.
For performing regularizing at the channel level, we introduce a set of learnable parameters that scale the oulput of each channel after a convolutional layer. More formally, for every ( E Rdxhxx which is the activation after the k-th convolutional layer, we scale the channels with 8(k) e Rd so that ( 8(4) @ (1)) where O denotes clement-wise multiplication so that the scaling factor 8(k) broadcast across the height h and width w. The inclusion of scaling factors (kk) is simple linear transformation and so can be perceived the introduction of additional layer to the neural network W, see Figure 2 resulting in extended
8

---

Published in Transactions on Machine Learning Research (09/2025)
neural network denoted by W As the normalization in Equation (2) will also be applied the scaling factors, the unstructured CoNNect regularizer in Equation (3) carries over to a structured regularization, where the scaling factors of less informative channels are pushed to 0 and more informative channels are retained.
Once a regularized neural network is obtained, we can do pruning a similar fashion as Section 3.3. Specifically, we can prune its channels via calculating an importance scores for each channel. To that end, aim determine the contribution of channel in layer l in terms of the connectivity of the neural network, denoted by T More formally, let g (8) 1/ |I denote the normalization of the scaling factors with index e for convolutional layer k so that IN. can be determined via
(8) Ereeio,aa) an) (6) (Erevigg ar))
I
where v the subset of nodes in a layer corresponding to channel index c. Simply put, I denotes the total connectivity that flows through channel layer k. Consequently, a simple pruning strategy is to prune the channels with lowest values of Lg,e.
(a) No regularization.
(b) L1 regularization.
(c) CoNNect regularization.
Figure 3: Trained (top) and fino-tuned (bottom) models. Thicker and darker colors correspond to stronger values. Red and blue edges correspond to positive and negative values respectively.
4 Numerical Experiments
We now provide several numerical experiments. In Section 4.1 we show results for CoNNect regularization during training. In Section 4.2 we show how CoNNect can be further scaled for pruning pre-trained models through integration in hard pruning strategies, such as DepGraph (Fang et al. 2023) and LLM-pruner (Ma et al. 2023 Our code is available at /tttps://githb...
4.1
Regularized Training with CoNNect
4.1.1 Unstructured Pruning of MLPs
In the following, we want to study the effects integrating CoNNect regularization in an unstructured pruning task. Let us consider a small multilayer perceptron neural network with ReLU activations. The network has 6 input nodes, three hidden layers of five nodes, and single output node. We sample input values H, (fi.1y.-.,,iig) ~ (O, E), where Σ is matrix with the value 2 the diagonal. Furthermore,
9

---

Published in Transactions on Machine Learning Research (09/2025)
let the output values be di if 24,1 + 21,2 + & O, and y otherwise, where ~ (V,, 0.25). To find sparse network representation, we train the network with Ln and CoNNect regularization. To that end, we solve
X2 min L(.. v) W - X2 log (W)) + +lallWlla.
(7)
W,b
We fit Uhree different models for 200 epochs following Equation Table B see Appendix C.1 We show the resulting NNs on the top row in Figure g for a single neural the bottom row, we present the fine-tuned NNs after SynFlow pruning. As can seen, the CoNNect regularizer is capable of identifying the relevant paths, where the other methods fail. We provide more details of this experiment. in Appendix C.1 including more extensive results.
for which we provide coefficients in
network initialization.
4.1.2 Channel-Level Pruning on GNNs
In this section, we demonstrate CoNNect for structured pruning on the channel level. Specifically, we prune Graph Convolutional Network (GCN, Kipf & Welling 2017) containing seven layers with learnable parameters, where the hidden feature dimensions are 211-250-256-256-222
Each GCN layer followed by ReLU activation function. We train the model on the Cora dataset (Sen et al. 2008), graph-based dataset consisting of 2,708 academic papers (nodes) and 5,429 citation links (edges), with each paper categorized into one of seven topics and represented by 14433-dimensional binary feature vector.
0.8
0.8
0.7
0.7
0.6
0.6
Accuracy
0.5
0.5
0.4
0.4
0.3
No Reg.
0.3
No Reg.
L1 Reg.
L1 Reg.
0.2
CoNNect Reg.
0.2
CoNNect Reg.
2
2
2a
23
2
2
2
2
2
24
25
2
Compression Ratio
Compression Ratio
(a) Without fine-tuning.
(b) With fine-tuning.
Figure 4: Accuracies of GNNs for given pruning ratios.
We train GCNs following Equation 7 for 300 epochs using the parameters shown in Table in Appendix and fine-tune each model after pruning for 100 epochs. We conduct 10 repeated experiments, and as shown Figure 4 our method outperforms I, regularization, especially for high compression ratios, which are computed by total channels/ (total channels pruned channols). The shaded regions represent 98% confidence intervals. We refer the reader to Appendix D.2 for an exlensive ablation on the regularizer coefficients.
4.2 Pre- Trained Model Pruning with CoNNect
To further demonstrate the versatility and scalability of CoNNect, we integrate into DepGraph (Fang al. 2023) and LLM-Pruner (Ma et al. 2023). These frameworks ensure all parameters are divided into several groups according to the dependency relationships in the computation process. Then, the importance score under the objective function J(.) is calculated by 4, Jww., -W) s jåw. lōw.,J((W) J(W) Wayl We integrate our CoNNect approach through the objective, i.., J(W) L(D) X Alog(pfo((W)) (W)), where D denotes the dataset. The importance of each group is aggregated through summation, and the least important groups are pruned. Connectivity, as currently defined CoNNect, is by default not affected by modules such as activation functions or biases. However, when keeping such modules in place within the connectivity computation, CoNNect can be understood in terms of signal flow. the remainder of this section, we simplify the connectivity computation and only redefine (0(W)),s [ to enhance both
10

---

Published in Transactions on Machine Learning Research (09/2025)
numerical stability and computational efficiency, while also setting the biases to b6. The result is that we aim to prune the groups that minimally affect the loss function, while also preserving signal flow.
4.2.1 One-shot Pruning CNNs
Using our integration in DepGraph, we perform structural pruning on ResNet-56 (He et al. 2016) and VGG-19 (Simonyan & Zisserman 2015), which are pro-trained and fine-tuned on CIFAR-10 and CIFAR- 100 datasets (Krizhevsky, 2009), respectively (see Appendix C.3). DepGraph framework iteratively prunes the model until the predefined speed-up targets are achieved, which is calculated as the ratio of multiply- accumulate operations before and after pruning, We first follow the pruning intensities tested in Fang et al. (2023), and then verify CoNNect further with extreme cases, Thus, the pruning is set to target speed-ups of 2.5x and 16x for ResNet-56 on CIFAR-10 and 8x, and 16x for VGG-19 on CIFAR-100. As shown in Table D CoNNect cxhibits advantages across various pruning ratios, with benefits being more pronounced in more extreme cases.
Table 1: Pruning results on ResNet-56 and VGG-19.
MODEL & DATASET
BASE Acc.
METHOD
PRUNED Acc.
SPEED UP
PRUNING RATIOS
DEPGRAPH
93.17
2.51x
56.22
RESNET-56 & CIFAR-10
CONNECT
93.63
2.50x
53.20
93.53
DEPGRAPH
80.24
16.17x
98.27
CONNECT
83.12
17.21x
97.46
DEPGRAPH
65.89
8.12x
90.48
VGG-19 CIFAR-100
CONNECT
69.38
8.00
93.33
73.50
DEPGRAPH
57.48
16.10x
96. 14
CONNECT
62.56
16.07 x
97.51
4.2.2 One-shot Pruning LLMs
We adopt the structural definition of connectivity as proposed in CoNNect and incorporate it into the LLM- pruner framework (Ma al. 2023). In our LLM-pruner integration, we compute the connectivity score for each parameter group based its structural position and its contribution to information propagation within the transformer architecture. This metric is estimated using model parameter dependencies derived from a small calibration set, as described in Appendix C.4] This approach allows us to efficiently integrate CoNNect into large-scale models, capturing essential structural information without modifying the architecture or relying on activation- based analysis.
Now, we perform a one-shot pruning on LLaMA-7B (Touvron al. 2023) using our CoNNect integration in LLM-pruner. After pruning, the LLM fine-tuned with LoRA (Hu et al. 2022) to restore as much struc- tural capability possible under the current architecture. To assess the model performance,
conduct
zero-shot perplexity analysis, see Table 2 We first compare CoNNect to I,, T2, random, and vanilla LLM-Pruner's importance metrics with 25% of the parameter groups removed, thereby resulting in a 20% parameter reduction. All methods are equipped with the same group division and aggregation strategy. As presented in the upper half of Table 2 compared to vanilla LLM-Pruner, we have reduced the per- formance gap between the pruned model and the original model by 22.96% without fine-tuning, which is 29.15% when fine-tuning is applied. To ensure fairness, both models are fine-tuned on the Alpaca dataset for only 2 epochs with LoRA. Essentially, CoNNect enhances the LLM-Prueerss framework with extra consideration of connectivity, providing good results. The results differ significantly from those produced by randomly removing parameter groups, yet the grouping strategy helps prevent the harmful effects typically associated with random pruning. However, L2 regularization even results in incorrect pruning choices, which is consistent with the conclusions drawn in the previous two subsections. We show similar results for pruning 40% of the parameters through removing 50% of the parameter groups the lower half of Table 2 Please refer Appendix C.4 for detailed experimental settings. Morcover, we provide results for CoNNect I. 1333 please refer to Appendix D.4 for further details.
and
11

---

Published in Transactions on Machine Learning Research (09/2025)
Table 2: Zero-shot performance the compressed LLaMA-7B. High scores are better, except for WikiText2 and PTB (indicated by the downward arrow). The bold values indicate the best results. The average is calculated among seven classification accuracies. An asterisk denotes that performance normalization is not available. The evaluation is conducted following the prompting of LLM-pruner (Ma et al. 2023 For post-training, both models are fine-tuned the Alpaca dataset for only 2 epochs.
PRUNED MODEL
METROD
WIRTTxr2]
PTB]
BoolQ"
PIQA
HELLASWaG
WINOGRANDE"
ARC-E
ARC-c
OBQA
AVERAGE
RaTrO
LLAMA-7H
12.62
22.15
T3.15
77-48
73.01
67.09
52.57
41.47
42.40
L
179.72
911.75
50.15
61.26
43.26
52.49
36.15
26.38
31.00
13.03
La
580.15
1022.17
59.68
57.40
37.07
52.09
32.53
28.41
29.30
42.42
RAmo 20% wyO
RANDOM
22.54
46.21
70.46
06.51
41.46
31.91
37.20
49.02
LLM-PRRCES
19.09
34.23
57.13
75.08
66.83
59.75
50.13
36.35
39.90
55.01
CONNECT
18.91
38.25
61.65
75.63
67.78
61.56
50.38
36.95
56.39
L RANDOM LLM-PRINER
24.32
42.85
59.05
75.
65.51
61.56
47.10
37.37
39.20
55. 00
55.
Ramo 20% TUNE
19.25
32
51.31
73. 18
64.45
59.91
47.91
35.15
10.60
53. 0.5
76.88
37.03
57.83
CoNNECT
17.18
29.92
66.57
76. 82
69.42
64.72
63.24
39.16
41.40
58.76
L RANDOM
858.08
1014:22
53.73
51.31
26,90
50
28.16
26.37
30.80
38. 21
RaTrO 40% w// TUNE
13783.61
27814.06
12.69
52.01
28.29
51:46
27.36
25.85
29.80
36.78
100.42
133.36
40.00
57.
36,00
50.12
32.88
25.77
31.00
39
LLM PRUNER
48.09
105.21
58.90
64.74
47.58
$3 $3.20
37.75
29.41
35,00
46.08
CONNECT
46.43
95.08
60.95
67.30
50.04
52 00.
38.30
29.86
36.50
47.91
01
47.
Is L.LM-PRUNER
44.91
67.16
47.34
71.60
50.60
54.38
43..35
32.25
36.99
48.05
RaTro 40% w// TUNE
67.
55,
47.
27.62
48.28
59.97
71:38
56.21
$9.35
.44.53
32.44
36. 36.20
51.
CONNECT
32.85
52.33
5 Conclusions & Future Work
In this work, introduce novel regularizer, called CoNNect, that leverages network connectivity to promote NN sparsity. Theoretically, prove that CoNNect is well-behaved regularizer and aligns with the minimization of the Lo norm. Through numerical experiments, we have shown that CoNNect can be effectively applied as a regularizer during training and so outperforms standard L1 regularization. Moreover, we demonstrated how CoNNect can be applied competitively in a one-shot pruning framework for post- training pruning CNNs and LLMs, such as DepGraph (Fang ct al.) 2023), and LLM-pruner (Ma et al. 2023), showing improved results. Future work directions are ample, for example, one can extend CoNNect and make it suitable for the pruning of Recurrent Neural Networks (RNNs). Moreover, one push CoNNect towards the semi-structured pruning paradigm, for example, as in Sun et al. 2024).
References
Sajid Anwar, Kyuyeon Hwang, and Wonyong Sung. Structured pruning of deep convolutional neural net- works. ACM Journal on Emerging Technologies in Computing Systems (JETC), 13(3):1 18, 2017.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 2020.
Hongrong Cheng, Miao Zhang, and Javen Qinfeng Shi. A survey on deep neural network pruning: Taxon- omy, comparison, analysis, and recommendations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12):10558-1065 2024.
Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume. 1 (Long and Short Papers), pp. 2924 2936. 2019.
12

---

Published in Transactions on Machine Learning Research (09/2025)
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schocnick, and Oyvind Tafjord. Think you have solved question answering? Try ARC, the AT2 reasoning challenge. arXiv preprinl arXiv:1803.0544 05,57, 2018.
Subhayan De and Alireza Doostan. Neural network training using 11-regularization and bi-fidelity data. Journal of Compulational Physics, 458:111010 2022.
Xiaohan Ding, Guiguang Ding, Yuchen Guo, and Jungong Han. Centripetal sgd for pruning very deep convolutional networks with complicated structure. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4943 1943-4953, 2019.
Gongfan Fang, Xinyin Ma, Mingli Song, Michael Bi Mi, and Xinchao Wang. Depgraph: Towards any struc- tural pruning. In Proceedings of the IEEE/CVP Conference Computer Vision and Pattern Recognition (CVPR), pp. 16091-16101, 2023.
Gonglan Fang, Hongxu Yin, Sauray Muralidharan, Greg Heinrich, Jeff Pool, Jan Kautz, Pavlo Molchanov, and Xinchao Wang. Maskllm: Learnable semi-structured sparsity for large language models. Advances in Newral Information Processing Systems, 37:7736 7758, 2024.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pp. 10323 10337. PMLR, 2023.
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint 1rXiv:190.00957, 2019.
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Authony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, September 2021. URL httlhttpss://e org/ //rg/////0 10. 5281/zenodo 5377162
Masafumi Hagiwara. Removal of hidden units and weightss for back propagation networks. In Proceedings of 1993 International Conference on Neural Networks LJJCNN-93-Nayee Japan), volume 1, pp. 351 354. IEEE, 1993.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural networks. Advances in Neural Information Processing Systems, 28, 2015.
Babak Hassibi, David G Stork, and Gregory Wolff. Optimal brain surgeon and general network pruning. In IEEE International Conference on Newral Networks, pp.
IEEE, 1993.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Tn Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770 778, 2016.
Yang He and Lingao Xiao. Structured pruning for deep convolutional neural networks: A survey, IEEE Transactions on Pallern Analysis and Machine Intelligence, 46:2900-2991 2023.
Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In 2019 TEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4335 4344, 2019.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 1389 1397, 2017.
Geoffrey E Hinton. practical guide training restricted Boltzmann machines. In Neural Networks: Tricks of the Trade: Second Edition, pp. 699. Springer, 2012.
13

---

Published in Transactions on Machine Learning Research (09/2025)
Torsten Hoefter, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. Journal of Machine Learning Research, 22(241):1-122 124, 2021.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference Learning Represenlations, 2022.
Zehao Huang and Naiyan Wang. Data-driven sparse structure selection for deep neural networks. In Pro- ceedings of the European Conference on Computer Vision (ECCV), pp. 304-320, 2018.
Siddhant Jayakumar, Razvan Pascanu, Jack Rae, Simon Osindero, and Erich Flsen. Top-kast: Top-k always sparse training. Advances in Neural Information Processing Systems, 33:20744 20754, 2020.
Leo Katz. A new status index derived from sociometric analysis. Psychometrika, 18(1):39 43, 1953.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, 2017.
Chris Kolb, Tobias Weber, Bernd Bischl, and David Rügamer. Deep weight factorization: Sparse learning through the lens of artificial symmetries. In Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu (eds.), International Conference on Representation Leurning, volume 2025, pp. 97919 10199-9778, 2025.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in Neural Information Pro- cessing Systems, 2, 1989.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. SNIP: Single-shot network pruning based connection sensitivity. In International Conference Learning Representations, 2019.
Wenjing Li, Minghui Chu, and Junfei Qiao. A pruning feedforward small-world neural network based on katz centrality for noulinear system modeling. Neural Networks, 130:269 285, 2020.
llya Loshchilov, Frank Hutter, et al. Fixing weight decay regularization in adam. arXiv preprint arXiv:1711.05100 5, 2017.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. Lim-pruner: On the structural pruning of large language models. Advances in Neural Information Processing Systems, 36:21702-211 21720, 2023.
Mitch Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of English: The penn treebank. Computational Linguistics, 19(2):313 330, 1993.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2022.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical 2018.
Methods in Natural Language Processing, pp.
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and An- tonio Liotta. Scalable training artificial neural networks with adaptive sparse connectivity inspired by network science. Nature Communications, 9(1):2383, 2018.
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. In International Conference on Learning Representations, 2017.
Behnam Neyshabur, Russ R Salakhutdinov, and Nati Srebro. Path-sgd: Path-normalized optimization in deep neural networks. Advonces in Neural Information Processing Systems, 28, 2015.
14

---

Published in Transactions on Machine Learning Research (09/2025)
David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training. arXiv preprinl arXiv:2104.10350, 2021.
Ekachai Phaisangittisngul. An analysis of the regularization between L2 and dropout in single hidden layer neural network. In 2016 7th International Conference on Intelligent Systems, Modelling and Simulation (ISMS), pp. 174 179. IEEE, 2016.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge scale. Communscatioos of the ACM, 64(9))99-106, 2021.
Victor Sanh, Thomas Wolf, and Alexander Rush. Movement pruning: Adaptive sparsity by fine-tuning. Advances in Newral Information Processing Systems, 33:20378 20389, 2020.
Prithviraj Sen, Galileo Namala, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad. Collec- tive classification in network data. AT Magazine, 29(3):93-93, 2008.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015.
Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Koltor. A simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations, 2024.
Hidenori Tanaka, Daniel Kunin, Daniel L Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in Neural Information Processing Systems, 33: 6377-6389, 2020.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-follloing Ilaa model. Technical report, Stanford University, 2023. URL htttt::///rrftt stanf wdw/2023/03//111 alpaca. html Accessed: 2025-08-01.
Georg Thimm and Emile Fiesler. Evaluating pruming methods. In Proceedings of the International Symposium on Artificial Neural Networks, pp. 20-25, 1995.
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodooogy, 58(1):267 -288, 1996.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave; and Guillaune Lample. Llama: Open and efficient foundation language models. arXie preprint arXiv:2302. 13971, 2023.
Juan Miguel Valverde, Artem Shatillo, and Jussi Tohka. Sauron u-net: Simple automated redundancy elimination in medical image segmentation via filter pruning. Neurocomputing, 594:127817, 2024.
Artem Vysogorets and Julia Kempe. Connectivity matters: Neural network pruning through the lens of effective sparsity. Journal of Mochine Learning Research, 24(99):1 23, 2023.
Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations, 2020.
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. Advances in Neural Information Processing Systems, 29:2082 20900, 2016.
Chen Yang, Zhenghong Yang, Abdul Mateen Khattak, Liu Yang, Wenxin Zhang, Wanlin Gao, and Minjuan Wang. Structured pruning of convolutional neural networks via LI regularization. IEEE Access, 7:106385 106394, 2019.
Ming Yuan and Yi Lin. Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society Series B: Slatistical Methodology, 68(1):49 67, 2006.
15

---

Published in Transactions on Machine Learning Research (09/2025)
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? orXiv preprinl 1rX5::1905.0055 2019.
Guian Zhou and Jennie Si. Subset-based training and pruning of sigmoid neural networks. Neural Networks, 12(1):79 89, 1999.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 19-27, 2015.
Tao Zhuang, Zhixuan Zhang, Yuheng Huang, Xiaoyi 'Zeng Kai Shuang, and Xiang Li. Neuron-level struc- tured pruning using polarization regularizer. Advances in Neural Information Processing Systems, 33: 9865 9877, 2020.
Liu Ziyin and Zihao Wang. spred: Solving 4 penalty with sgd. In International Conference on Machine Learning, pp. 43407 43422. PMLR, 2023.
16

---

Published in Transactions on Machine Learning Research (09/2025)
Appendix
Table of Contents
A Proofs
18
A.1 Proof T Thoorem 3.1
18
A.2 Proof Lemma 3.2
18
B CoNNect Implementation Details
19
Experimental Settings
20
C.1
Experimental Settings for Section 4.1.1
20
C.2
Experimental Settings for Section 4.1.2
20
C,3
Experimental Settings for Section 4.2.1
20
C.4
Experimental Settings Section 4.2.2
................................
D
Ablation Studies
22
D.1
Impact of Initializations and Regularizer Strength in Section 3.3.1
22
D.2
Impact of Initializations and Regularizer Strength in Section 4.1.2
23
D.3 Deep Weight Factor Regularization in Numerical Example Section 1.1.2
25
D.4
Supplemental Results for LLaMA- 13B
25
Computational Improvement for LLMs
17

---

Published in Transactions on Machine Learning Research (09/2025)
A Proofs
A.1 Proof Theorem 3.1
Let Fim denote the set paths in the neural network that go from some input node i E Vi to the output node m VVs, where
7 = ((,,)), (1, m)) E TTi Timm
is a sequence of edges from the input layer to the output layer. Using that (W) is the sum of weights of paths from the input to the output layer (Neyshabur et al. 2015), we rewrite
K-1
K-1
W
(lo((W) (w) E
II (0())) EV me VEVE TET k=1
EE II
E(nO)EE W
VEVI mEVK TET k=1
where 7he refers to the kth edge in a sequence Then, to minimize R(W), i.e, maximize oo (W), we need to allocate all the mass to a single path from the input to the output, which means selecting a specific sequence of weights that maximizes the product along that path, effectively minimizing the contributions from all other paths.
To show the upper bound of [Vi |VK|+ K K-3 non-zero weights in W assume w.l.o.g. some W where single path i' has all mass in the network. It follows that slo*(W)) 1. Now, let W" denote a solution where some mass from the first weight Wi,, for (i,j) Tim is shifted to any other weight(s) Wi, (note that j is fixed), where E V1 connects to j E V2. It is easily seen that si (W") since
K-1
(ott(("))
Σ LEV
Σ
II
(((W"))
7ET J.m A=1
K-1
W
Σ TET
II k=1
(O(W')) = E LEV; E(ne)EE, W %el
-1 1.
(M,)EE,
W
LEV.
In words, (W) is indifferent in how many of the [Vi| inpul nodes connect to single node in the second layer. Note that a similar argument can be made for the weights connecting the K - -Ith layer with the Kth layer. It follows that the number of non-zero weights for W' is upper bounded by [V1 for the first layer, Vxl for layer K 1, and K 3 for the weights of the remaining layers. The resulting upper bound then Vi + VV + K
A.2 Proof Lemma 3.2
We prove this by induction using the necessary and sufficient system of equations for stationarity (W), see Equation (5). Assume any neural network of arbitrary size with K 2 layers. Note that for this specific any weight allocation will be stationary in stat (W)) Now, assume weight allocation such that a., a.j. for all i,j arg max a.k, since adding a layer VK+1 implies that this condition must hold to satisfy Equation 5 in the next step.
Now we add new layer of arbitrary size VK+1- In case VK+1 is the last layer, is sufficient to allocate (0(W))k, > O, for all E arg max a& to obtain a stationary point. In case the neural network is expanded with another layer VK+2 in a next step, we let (B(W)) > for i E arg max and j E arg max a.k. such that a. a.j, for all i, arg max d.k to satisfy Equation (5). Note that this immediately implies ()(W)), for all l = ........... K 2, 2, for all paths with positive path weight.
(r.W)),,es for all (i, (r, E arg (iji a.a..j. Hence, (0(W)) (O(W))
It remains to be shown that stationarity cannot be induced by the reparameterization O(W). To see this, we first observe that the normalization in Equation (2) separable for each layer A == 1,..., K - 1. Simply in- specting a single layer k note that Ve, J(0) # 0, where 0₂ (BL))) Morcover, let W; = (W.,) E and Vw, dg, is full rank (except W = 0) and thus preserves the non-zero property through the chain rule.
18

---

Published in Transactions on Machine Learning Research (09/2025)
B CoNNect Implementation Details
In this section, we outline how (W) can be efficiently computed using slightly modified forward pass of the neural network and a vector of ones as input. Below, we outline how different modules are treated in this modified forward pass. The ability to handle these modules enables the application of CoNNect across broad spectrum of neural network architectures.
This inchudes both dense (fully connected) layers and convolutional layers. The weights of these layers define the primary connections between nodes and normalize their weights via. Equation (2) The biases, however, merely shift activations (which we will exclude), and do not influence the connectivity structure and are therefore excluded.
Linear Layers:
BN Layers: Batch normalization layers apply standardization and scaling to the outputs of preceding layers. For the purposes of connectivity analysis, the standardization can be disregarded as does not alter the structure of connections, but rather rescales values. Thus, we consider BN layers as identity mappings with preserved connectivity.
Non-linear activation functions such ReLU, sigmoid, or tanh are ignored. These functions transform node outputs but do not influence the underlying connectivily. Ignoring them simplifies the analysis without affecting the structural representation.
Activation Functions:
Pooling Layers: Max-po max-ppoling layers are replaced with average pooling layers. This change ensures that all input connections are treated equally in the computation of connectivity, rather than prioritizing the strongest signal as in max-pooling.
Dropout layers are designed to randomly disable connections during training regularization method. Since they are stochastic and transient, they are ignored for connectivity analysis, as they do not represent fixed structural linkages in the network.
Dropout:
Identity Connections: Identity connections, such as skip connections in residual networks, are (generally) not parameterized and therefore can be ignored when optimizing the neural network's connectivity. Thus, we omit the identity connection in the forward pass.
19

---

Published in Transactions on Machine Learning Research (09/2025)
C
Experimental Settings
Platform: All experiments were performed on a single NVIDIA RTX4090 GPU with 24GB of memory.
C.1
Experimental Settings for Section 4.1.1
All models have been trained to solve Equation with cocfficients in Table 3 L(0. v) is the Binary Cross Entropy between largel and inpul probabilities, and ||W||,,1 is the often-applied L2 regularization (weight decay). All models were trained for 200 epochs using Adam with a learning rate of 0.01, cosine annealing scheduler, and batch size 256. After training, we pruned 96% of the weights in each layer using the pruning strategies discussed Section 3.3.1 i) magnitude pruning, and ii) SynFlow pruning. Finally, the model fine-tuned with the same hyperparameters but with decreased initial learning rate of 0.001 for 50 epochs.
Table Regularizer coefficients.
REGULARIZER
Aa
Ag
NONE L1 CONNECT
0
5 10
1 10
0
5 X 10
1 10
10
Remark: Synflow is traditionally introduced as pre-training pruming method, its data-agnostic nature makes it less effective in this context, given the presence of uninformative input nodes. Moreover, SynFlow is generally regarded as a global pruning strategy. However, we frequently observed layer collapse under this configuration. In contrast, applying a local pruning approach yielded significantly better results, particularly for models without regularization and L, regularization. We thus show the results using a local pruming approach.
C.2 Experimental Settings for Section 4.1.2
All models were trained for 300 epochs using Adam with learning rale of 0.005. We used a linear warmup of 10 epochs with start factor 0.01 and end factor 1. Subsequently, we used cosine annealing scheduler for the remaining 290 epochs. Fine-tuning is performed similarly, but with learning rate of 0.0005. Regularization coefficients used are shown in Table 4 and boldfaced parameters are best performing, thus shown in Figure The results for the remaining regularization coefficients are shown in figures and see Appendix D.2
Table 4: Regularizer coefficients used in GNN pruning. Boldfaced parameters are used in Figure
REGULARIZER
A
Aa
As
NONE L1
0
{10 8
10 41
{10
10 10
10-
10 4 {10-5
{10
CONNECT
[10 100
10
10-2)
10-4
C.3 Experimental Settings for Section 4.2.1
When using CoNNect for pre-trained model pruning, approximate connectivity by keeping all modules as they are to measure signal flow. Moreover, in our evaluation of connectivity, we use multiple inputs uniformly sampled between 0 and 1 instead of a single all-one for increased robust ness.
Datasct: We train ResNet-56 with CIFAR-10 (Krizhevsky) 2009), a datasct with 60,000 32x32 images with 10 different classes. Each class has 6,000 images. Moreover, we used CIFAR-106 [Kizhevssy 2009), a more challenging dataset consisting of 100 classes with 600 images per class, to train VGG-19.
20

---

Published in Transactions on Machine Learning Research (09/2025)
C.4 Experimental Settings for Section 4.2.2
In the current experiment, use 10 randomily selected samples from Bookcorpus (Zhu et al.
2015 to
be the calibration samples for establishing the dependency between parameters in the model and calculate the gradient for LLaMA-7B. To that end, we truncate cach sample to a sequence length of 128. We set the coefficient of CoNNect as 1 x 10a. During fine-tuning, we utilize Alpaca (Taori et al. 2023 which comprises approximately 50,000 samples, to recover the capacity of the pruned model, which requires just 2 hours on our platform (NVIDIA RTX4090 GPU).
To determine which groups to prune, compute importance scores for each weight in the model. CoNNect shares the same logic for computing the importance score from loss values as LLM--Pruer, but includes an additional connectivity term. The inputs used to evaluate connectivity are uniformly sampled between and the vocabulary size. Then, specifically for Lp pruning, we compute the importance of each group by computing the Lp norm and prune the groups with the lowest importance scores. For random pruming, there no ncod to compute importance scores for cach group-we simply randomly select certain groups for pruming. Moreover, we leave the first three layers and the final layer unchanged (similar to Ma et al. (202)), substantial changes to the parameters of these layers greatly influence the performance of the model. Finally, the discovered groups within each module are pruned according to a predetermined ratio. The pruning rate for the selected groups is higher than the pruning ratio for the parameters since some layers (e.g., the excluded layers) retain their parameters. For a total of 40% parameter removal, we must prune 50% of the groups specifically from the fourth to the thirtieth layer.
Datasets: To assess the model performance, we conduct zero-shot perplexity analysis on WikiText2 (Merty et al. 2022 and PTB (Mareus et al. 1993). and then follow Gao et. al. (2021) to test the model
with zero-shot classification tasks on common sense reasoning datasets: BoolQ { [Clark et al. 2019), PIQA
Biik et al.] 2020), HellaSwag (Zellers et al. 2019), WinoGrande (Sakaguchi et al. 2021). ARC-easy, ARC- challenge (Clark et al., 2018), OpenbookQA Mihaylov et al. 2018), where the model ranks the choices in these multiple-choice tasks.
21

---

Published in Transactions on Machine Learning Research (09/2025)
D Ablation Studies
D.1 Impact of Initializations and Regularizer Strength in Section 3.3.1
To show the robustness our results, we conduct an ablation study to: 1) analyze the impact of different initializations, and 2) the regularization strengths.
First, we present the results for 100 different initializations, where show the (aggregated) train and test loss in Figure 5 and the fine-tuned accuracies in Figure [ Roughly speaking, the final accuracy for each model can be categorized by the ability to find the network connecting the input nodes 1 and 2 to the output layer. If the fine-tuned accuracy is around 0.50, the algorithm was unable to connect node and node 2 to the output (see figures 3 3 and (b)) If the fine-tuned accuracy is around 0.75, the algorithm was able to connect node 1 or node 2 to the output. Finally, the algorithm preserved the edges connecting node 1 and node 2, it found the correct network and achieved an accuracy of more than 0.95 (sce Figure
As shown in Figure 5 CoNNect regularization via (W) is beneficial to both pruning strategies. Tt is noteworthy that SynFlow pruning does not offer much further improvement over connectivity regularization compared to simple magnitude pruning, although it seems to slightly decrease the frequency of layer collapse compared with magnitude pruning, see Table 5 This can be attributed to the fact that CoNNect regular- ization has already trained the network to use the correct paths to model the current problem, as shown in Figure clc.. It thus suffices to apply a simple maguitude pruning to identify these paths.
1.2
No Reg.
L1 Reg.
CoNNect Reg.
Accuracy
0.8
0.8
Loss
0.7
0.4
Reg.
0.6
L1 Reg.
CoNNect Reg.
0.5
100
150
200
250
150
250
Epoch
Epoch
(a) Loss values.
(b) Accuracies.
Figure 5: Learning curves for solving Equation are 95% confidence intervals.
SynFlow pruning happens at iteration 200. Bandwidths
50
50
Reg.
25
Frequency
50
Frequency
50
Reg.
25
25
50
50
Reg.
CoNNect Reg.
25
0.6
0.9
10
0.6
0.9
Accuracy
Accuracy
(a) Magnitude.
(b) SynFlow.
Figure 6: Fine-tuned accuracy after magnitude pruning and SynFlow pruning of regularized models.
Now we perform experiments with different values of X, see Table 0 for an overview. Specifically, increasing An by order of magnitude to 0.01 causes frequent occurrence of layer collapse, although it does increase
22

---

Published in Transactions on Machine Learning Research (09/2025)
Table 5: Layer collapse frequencies (out of 100 runs) per pruning method and regularization type.
Pruming Method
No Reg.
L1 Reg.
CoNNect Reg.
Magnitude
81
83
4
SynFlow
71
Table 6: Regularizer coelficients used for producing figures 7
a
respectively.
EXPERIMENT
EXPERIMENT
EXPERIMENT
EXPERIMENT
REGULARIZER
Au
Az
Às
Au
As
NONE
5x 10
30-3
5 10
5 10
L1
1x 10
0
5x 10-
1x
10
0
5 10-10-
1
10
x 10-2
x 10
x 10
CONNECT
xx 10
x 10
x 10
x 10-2
x 10
x 10
the performances for the cases without layer collapse, see Figure aa. Changing X2 by one order of magnitude to did not cause any specific change, arguing for the stability of CoNNect. Moreover, increasing X by one order of magnitude to 0.005 seems to improve the model performance overall, especially for the CoNNect regularized model, see Figure 7 Increasing À by another order of magnitude still shows very competitive results for CoNNect. Finally, we decrease \1 and A to 0.0005 and 0.05 respectively, and see that the regularizers become too weak leading the results to converge toward those of standard L2 regularization.
Reg.
Sing
Fng
Rrg
Froquency
Prequency
Proqueccy
Frequency
23
CoNNect Kog.
CoMftec Rog.
Colfict Roj.
CoWNect Rog
25
27
Accuracy
Accuracy
Accuracy
Accuracy
(a) Fine-tuned accuracy after magnitude and SynFlow pruning (Experiment 1).
(b) Fine-tuned accuracy after magnitude and SynFlow pruning (Experimentt 2).
No Tog.
No Teg.
No Tug.
Reg.
Prequency
Proquency
Frequency
CuWNect
CoMliict Rig.
Col fog.
Col/Noct lop.
10
Accuracy
Accuracy
Accuracy
Accuracy
(c) Fine-tuned accuracy after magnitude and SynFlow pruning (Experiment 3).
(d) Fine-tuned accuracy after magnitude and Syn pruning (Experiment 4).
Figure 7: Fine-tuned accuracy after magnitude and SynFlow pruning for different regularization settings.
D.2 Impact of Initializations and Regularizer Strength in Section 4.1.2
Ablation on the regularization coefficients for both Ln and CoNNect regularization are shown figures g and g Note that the cases with A and Ag correspond scttings where neither L nor CoNNect regularization is applied.
23

---

Published in Transactions on Machine Learning Research (09/2025)
-
0
-
=
-
A1 10
--
=1
=
0.7
0.7
0.6
0.6
Accurary
Accuracy
0.5
0.5
0.3
0.3
0.2
0.2
20
2'
2
2
2
2
2
2
2
Compression Ratio
Compression Ratio
(a)
Without fine-tuning (As 10 10--). 9).
(b)
With fine-tuning (As 10 10-3).
0.8
0.8
0.7
0.7
Accuracy
Accuracy
0.5
0.5
0.4
0.4
0.3
0.3
2
2
2
2
2
2
2
Compression Ratio
Compression Ratio
(c) Without fine-tuning (As = 10 )
(d) With fine-tuning (As = 10 10-4).
Figure 8: Accuracies of GNNs for given compression ratios under L regularization.
A==
A 10
10
10
t 10
0.8
0.8
0.7
0.7
0.6
Accuracy
Accuracy
0.5
0.4
0.4
0.3
0.3
20
22
2
25
2
2
22
24
29
2
Compression on Ratio
Compression Ratio
(a) Without fine-tuning (As = 10
(b) With fine-tuning (As = 10 10-
0.8
0.7
0.6
0.6
Accuracy
0.5
Accuracy
0.5
0.4
0.2
0.2
2
20
22
Compression
Compression Ratio
(c) Without fine-tuning (As 10 10-4).
(d) With fine-tuning (As 10-4).
Figure 9: Accuracies of GNNs for given compression ratios under CoNNect regularization.
24

---

Published in Transactions on Machine Learning Research (09/2025)
D.3 Deep Weight Factor Regularization in the Numerical Example in Section 4.1.2
In this section, present the results of applying Deep Weight Pactorization (DWF) to the numerical example from Section 4.1.2 (Kolb et al. 2025). DWF introduces an Lp/2 regularization by reparameterizing the network weights W using D22 factors. More formally, each parametorizod weight factorized as
(1) We Wij wis
(D)
This formulation allows Lp/2 regularization in the weight space W to be conveniently implemented through weight decay the individual factors.
Figure 10 reports the results, where we vary both the number of factors D and the strength of the weight decay A3. It can be seen that these results are not competitive with CoNNect, shown in Figure
A 10-3
As = 10
À = 10
0.7
0.7
0.6
0.6
Accuracy
Accuracy
0.5
0.5
0.3
0.3
0.2
0.2
20
2
22
2
2
2
2
2
Compression
Compression Ratio
(a)
Without fine-tuning (D 2).
(b)
With fine-tuning (D 2).
0.8
0.8
0.7
0.7
Accurscy
Accuracy
0.5
0.5
0.4
0.4
0.2
2
2a
2
2
2
2
2
Compression Ratio
Compression Ratio
(c) Without fine-tuning (D 3).
(d) With fine-tuning (D= =0:30 3).
Figure 10: Accuracios of GNNs for given compression ratios under DWF regularization.
D.4 Supplemental Results for LLaMA-13B
We further evaluate our performance on LLaMA-13B Touvron al. 2023 similar Section 4.2.2 where 20% and 40% of the parameters are pruned. We show these results in tables 7 and g We note that the LLaMA-13B model used in our evaluation is community-relessed yahma/ 1ataa/l1mmmmm1bb-ha checkpoint used in Ma et al. (2023) is no longer available. This may slightly affect the baseline performance. All models are evaluated using the same test benchmarks as presented Table 2 The results demonstrate that CoNNect almost always outperforms the bascline method across multiple benchmarks, regardless of whether fine-tuning applied. This observation is consistent with the findings in the main text, indicating the effectiveness and robustness of our methods on larger-scale models.
the original
25

---

Published in Transactions on Machine Learning Research (09/202)
Table 7: Zero-shot performance of the compressed LLaMA- 13B with 20% pruning ratio.
20% w/o CONNECT
20% w/ TUNE
DATASET
LLAMA-13B
L.M-PRRNER
L.M-PRUNER
CONNECT
WIKITEXT21
11.58
16.62
16.30
15.64
15.16
PTB|
44.56
60.91
59.03
59.96
58.80
BOOLQ"
68.53
64.43
69.05
65.69
72.63
PIQA
79.05
77.20
76.71
78.18
79.11
HELLASWAG
76.21
73.45
73.93
74.99
75.37
WINOGRANDE'
70.09
67.56
68.19
68.82
67.88
ARC-E
59.81
56.90
58.21
57.70
60.98
ARC-c
44.62
40.10
40.96
42.06
43.69
OBQA
42.20
41.40
41.60
43.60
43.40
AVERAGE
62.93
60.15
61.24
61.58
63.29
Table 8: Zero-shot performance of the compressed LLaMA-131 with 40% pruning ratio.
40% w/o TUNE
40% w/ TUNE
DATASET
LLAMA-13B
L.M--RRUNER
CONNECT
L.I.-PRUNER
CoNNECT
WIKTTEXT2/
11.58
35.18
32.41
22.49
21.85
PTBJ
44.56
120.19
106.90
78.21
76.10
BOOLQ"
68.53
62.05
62.11
62.17
62.48
PIQA
79.05
73.34
72.20
75.90
75.95
HELLASWAG
76.21
59.63
61.82
66.05
66.64
WINOGRANDE"
70.09
55.49
55.96
61.56
62.12
ARC-E
59.81
44.19
45.16
52.31
52.31
ARC-c
44.62
33.36
34.04
36.35
35.92
OBQA
42.20
38.60
39.20
39.80
41.40
AVERAGE
62.93
52.38
52.93
56.31
56.69
26

---

Published in Transactions on Machine Learning Research (09/2025)
E
Computational Improvement for LLMs
In Table we show the computational improvement of CoNNect integrated in LLM-pruner across different pruning ratios in terms of complexity. Indeed, speed-up close to the compression ratio.
Table 9: Computational complexity of CoNNect-pruned LLaMA-7B model for different. pruning ratios. Speed-up is the GMACs of the basc model divided by the GMACs of the pruned model.
PRUNING RATIO
PARAMETER COUNT
GPU MEMORY
COMP. COMPLEXITY (GMACs)
SPEED-UP (GMACs)
(MIB)
0%
6.7B
12892.6
425.1
20%
5.4B
10383.7
340.5
1.3x
40%
4.1B
7952.6
255.8
1.7x
27


Reviews:
[{"review_id": "2P6WhyurVw", "reviewer": "Reviewer_s6Vh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n- The manuscript is easy to follow.\n- The proposed regularizer is clearly defined.\n- This method permits to utilize unstructured pruning approaches without the issue of layer collapse, enabling their efficient use.\n- Experiments were conducted on a number of neural network architectures.\n\n\n# Weaknesses\n- Unintuitive regularizer. It is unclear why minimizing the layer-wise normalized weights (in essence, Eq. 2) would lead to a sparsity that respects the network's structure and would allow to do structure pruning.\n- Very few comparisons with current work.\n- Confusing/fuzzy categorization of Related work. Related work is divided into \"Regularization-based pruning\" and \"Stage-based pruning\": Regularization-based are methods that minimize a regularizer, and stage-based are those that explicitly remove the parameters. However, what if a method minimizes a regularizer and also explicitly removes the parameters?\n- No code is shared. I consider this an important weakness that, if addressed, could improve the impact of this work.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a regularization function that leads to sparse neural networks while keeping them connected to, ultimately, prune the networks while avoiding layer collapse (i.e., that a layer/neurons produce features that the following layer does not use). The regularizer is evaluated on an small MLP, on a graph neural network, and on pretrained CNN and LLM where it achieved higher performance than other methods.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "# Clarifications:\n- I have the impression that the issue that this regularizer aims to solve occurs only in unstructured pruning because, in structured pruning (filter or layer pruning), you can eliminate filters/layers directly without the issue of neurons/filters/layers producing features that won't be utilized. This is because, if you eliminate a filter in layer X, then, you also need to adjust layer X+1. Consider layer X being 32x1x3x3, and layer X+1 being 64x32x3x3; if you eliminate 10 filters from layer X, you will have that layer X is 22x1x3x3 and layer X+1 must be adjusted to 64x22x3x3. If my impression is correct (i.e., that this method is specifically for unstructured pruning in order to \"achieve\" structured pruning) then, I didn't find it too clear, and I suggest clarifying it.\n- In connection to my previous comment: doesn't \"Axiom 2\" hold in filter pruning? I can't see how, in filter pruning in CNNs, you would end up having this layer collapse issue.\n- After reading the Related work, it is unclear how redundancy elimination methods, such as [1,2,3] fit into this classification of previous work.\n- Could you, intuitively, indicate why minimizing the regularizer would lead the models to have a \"structured sparsity\" allowing structured pruning?\n\n[1]: Sauron U-Net: Simple automated redundancy elimination in medical image segmentation via filter pruning. Pattern Recognition 2024\n\n[2]: Centripetal sgd for pruning very deep convolutional networks with complicated structure. CVPR 2019\n\n[3]: Filter pruning via geometric median for deep convolutional neural networks acceleration. CVPR 2019\n\n# Small modifications/comments\n- \"Layer collapse\" is introduced in the abstract, but it's not defined. For the reader who doesn't know it, it would be beneficial to either omit this term and talk about what is it.\n- 1. Introduction \"Pruning weights based on absolute values\". What does this exactly refer to? Based on the L norms?\n- The equation above Eq. 2: Please indicate what's \"k\". I guess this is the layer.\n- Eq. 2: Isn't $W_{i,j}$ a single value (the edge connecting neuron $i$ with $j$)? If so, why doing $|.|$?\n- Below Eq. 3 \"[...] exclusively influences the learnable structure\". I think that rather than \"structure\", it is meant \"parameter\".\n- Below Eq. 3 \"sum of all (multiplicative) reparameterized weighted\". I think that rather than \"reparameterized, it is meant \"normalized\"?\n- Below Eq. 3. Typo \"paramterized\".\n- Below Eq. 3. \"Moreover, ...\" does this statement mean that one needs to replace all biases all the time?\n- Page 6. \"Also, our regularizer does not force..\" I found this sentence a bit convoluted.", "broader_impact_concerns": "This method (a strategy to improve pruning in deep learning networks) has no direct negative societal impact.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Jht1oAxSyg", "reviewer": "Reviewer_29EU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main strength of this paper is relative simplicity and the claims of evidence. This paper show that the proposed regularization works for several different models and in general it is very intuitive and efficient approach. \n\nThe weaknesses are several:\n\nFirst the authors claim that the other regularization methods such as $l_1$ or $l_2$ can result in network or layer collapse. However, during the evaluation of the proposed regularization not a single time such case was demonstrated. This can be seen on relatively similar results during the numerical evaluation. If the layer collapse would be occurring, wouldn't it be more visible or visible in at least one case?\n\nThe second issue is with the evaluation of the unstructured pruning. The authors show only very small toy results and do not investigate any further. Again, it would be very desirable to have an illustration of the advantages of the proposed regularizer with examples of layer collapse and real-world experimental data. \n\nMoreover, the authors did not mention multi step pruning or retraining after pruning. Is there a reason for this?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a regularization of the pruning methodology based on connectivity. The authors propose a method of limiting the removal or reducing to 0-weight such connections in a network that would result in network that is not connected, i.e. the input cannot be transmitted to output. The paper is well written, and the evaluation is performed on several different models and datasets. The proofs of the theorems seem to be also correct. \n\nIn more details, the authors use the standard optimization notation where the proposed method is introduced as a penalty for dropping certain weights. Then by comparing to other regularizations the authors show that for certain cases their proposed regularization improves lightly the result when other regularizes are used.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "More experiments on unstructured pruning. While the authors provided some experiments, for the present paper the experimental data is not enough. This is mainly due to the fact that the paper is of very applied nature and empirical data of the regularization should be well understood before propagated. While the theoretical results seem promising the authors should make a special case to in details evaluate major models of neural networks in order to observe its behavior.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1etUhQw5hx", "reviewer": "Reviewer_425e", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:** \n\n- The paper is well-written and easy to follow\n- The paper addresses an important problem in pruning about network connectivity\n- The paper introduces a novel regularizer and provides theoretical guarantees for its robustness\n- A good set of experiments are conducted to show the efficacy of the regularizer\n\n**Weaknesses:** \n\n- I would like to a discussion / set of experiments on the limitations of CoNNect Regularizer. Are there any network architectures / scenarios for which this regularizer would fail?\n- A comparison against Iterative Magnitude Pruning in the context of the Lottery Ticket Hypothesis would be interesting.\n- A bit more discussion on future directions would be helpful.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a novel regularizer called CoNNect which results in a sparse network while ensuring connectivity between input and output layers. Furthermore, the authors demonstrate via experiments that this regularizer helps improve the results of pruning strategies. \n\nThe authors state the following two axioms for a good pruning algorithm: \n\n1. Pruning should reduce the number of weights while minimizing the impact on accuracy and increasing computational efficiency \n2. Pruning should preserve network connectivity and the flow of information from input to output \n\nThe paper then highlights that while many algorithms satisfy the first axiom, the second axiom is often neglected leading to problems like layer collapse with the exception of SynFlow pruning technique which is designed to preserve network flow. \n\nThe authors then introduce their CoNNect regularizer which is inspired by Katz centrality and SynFlow  which satisfies both axioms and results in a sparse network where flow of information is preserved. The paper also provides theoretical guarantees to show the robustness of their regularizer.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I believe that the paper is strong enough to be accepted at TMLR as is, but I would still encourage the authors to address some of the comments mentioned in the weakness section.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)