Deep Weight Factorization: Sparse Learning Through the Lens of Artificial Symmetries
Authors: Chris Kolb
Tobias Weber
Bernd Bischl
David Rügamer
Venue: ICLR 2025 Poster (2025)
Keywords: Sparsity
Regularization
Neural Networks
Overparametrization
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose a deep weight factorization for sparse neural networks that enables smooth optimization of non-convex sparse regularization.

Abstract: Sparse regularization techniques are well-established in machine learning, yet their application in neural networks remains challenging due to the non-differentiability of penalties like the $L_1$ norm, which is incompatible with stochastic gradient descent. A promising alternative is shallow weight factorization, where weights are decomposed into two factors, allowing for smooth optimization of $L_1$-penalized neural networks by adding differentiable $L_2$ regularization to the factors. 
In this work, we introduce deep weight factorization, extending previous shallow approaches to more than two factors. We theoretically establish equivalence of our deep factorization with non-convex sparse regularization and analyze its impact on training dynamics and optimization. Due to the limitations posed by standard training practices, we propose a tailored initialization scheme and identify important learning rate requirements necessary for training factorized networks.
We demonstrate the effectiveness of our deep weight factorization through experiments on various architectures and datasets, consistently outperforming its shallow counterpart and widely used pruning methods.

Published as a conference paper at ICLR 2025
DEEP WEIGHT FACTORIZATION: SPARSE LEARNING THROUGH THE LENS OF ARTIFICIAL SYMMETRIES
Chris Kolb, Tobias Weber, Bernd Bischl, & David Rügamer
Department of Statistics, LMU Munich, Munich Munich Center for Machine Learning (MCML),, Munich {chris s.. kolb, tobias asiias... www..,.ebrrreee bernd. bischl, davidfeaattettttt..... wnw-muuncheee..deee
ABSTRACT
Sparse regularization techniques are well-established in machine learning, yet their application in neural networks remains challenging due to the non- differentiability of penalties like the L1 norm, which incompatible with stochas- tic gradient descent. A promising alternative is shallow weight factorization, where weights are decomposed into two factors, allowing for smooth optimiza- tion of -u-eeaalized neural networks by adding differentiable L.2 regularization to the factors. In this work, we introduce deep weight factorization, extending previous shallow approaches to more than two factors. We theoretically estab- lish equivalence of our deep factorization with non-convex sparse regularization and analyze its impact training dynamics and optimization. Due the limi- tations posed by standard training practices, we propose a tailored initialization scheme and identify important learning rate requirements necessary for training factorized networks. We demonstrate the effectiveness of our deep weight fac- torization through experiments on various architectures and datasets, consistently outperforming its shallow counterpart and widely used pruning methods.
INTRODUCTION
Making models sparse is contemporary challenge in deep learning, currently attracting lot of attention. Among the more prominent methods to achieve sparsity are model pruning meth- ods (Gale al. 2019 Blalock al. 2020) and regularization approaches sparsifying the model during training (Heefler et al. 2021). While in statistics and machine learning, sparse regular- ization approaches are well-established (see, e.g., Tian & Zhang 2022), the non-smoothness of sparsity penalties such as the L, norm impedes the optimization of neural networks when using classical stochastic gradient descent (SGD) optimization. based optimization while inducing L1 regularization is weight factorization. Originally proposed statistics for linear models (Hoff) (017). the idea of factorizing the weights w wi O wa to obtain differentiable L1 penalty on the product W1 @ W2 has recently been adopted also in deep learning (see. e.g., Ziyin & Wang 2023). This simple trick allows the integration of convex Lisbe sparsity into neural network training while promising di- rect applicability of familiar SGD. As shown in Fig. [ the obtained spar- sity of differentiable L1 is superior vanilla L1 regularization. This holds even after applying additional post-hoc pruning, demonstrating that the infe- rior sparsity performance of vanilla I, not just due to suboptimal threshold but also the incompatibility of SGD and non-smooth penalties.
possible solution that allows SGD-
sparsity (F-MNIST)
Additional post-hoc pruning
Test accuracy
ScOml narhine
0.6
0.6
0.4
0.4
Compression ratio Ip-pp
Compression ratio
Figure 1: Sparsity-accural tradeoff using vanilla Li penaliza- tion with SGD (blue) compared (deep) weight factorization. Mcans and std. deviations over 3 random seeds are shown.
Given the success of (shallow) weight factorization, we study deep weight factorization in this work, ie., factorizing w wi @ wp. D 2 2 (cf. Fig. 2 We investigate whether theoretical guarantees support the use of depth-D factorization, whether it is beneficial for sparsity, what implications its usage has training dynamics, and analyze other practices such initialization.

---

Published as a conference paper at ICLR 2025
① Deep Weight Factorization
Vanilla
Factorized
w wi
wp
Waa U
Initialize factors
Train with lz
Obtain Wi... Up
C wp
SGD
() Collapse factors
SGD
Theorem
[(w) + X||w|l
2/0
C(wi @ @ wn)
2d-1 ||
D
Figure 2: Overview of the proposed method (cf. Algorithm 2). Our approach proceeds by factorizing the neural network weights and running SGD the factors wa with weight decay. Post-training, the factors are collapsed again, with the resulting sparse solutions being minimizers of the non-smooth Tee/--regularized objective.
Our contributions In this work, we address the aforementioned challenges and close an important gap the current literature. We first theoretically show the equivalence of factorized neural net- works with sparse regularized optimization problems for depth D 2 2, allowing for differentiable non-convex sparsity regularization any neural network. We then discuss optimization strategies for these factorized networks including their initialization and appropriate learning rate schedules. We also analyze the training dynamics of such networks, showing a particularly interesting connec- tion between the evolution of weight norms, compression, accuracy, and generalization. Conducting experiments on a range of architectures and datasets, we further substantiate our theoretical findings and demonstrate that our proposed factorized networks usually outperform the recently proposed shallow factorization and yield competitive results to commonly used pruning methods.
2 BACKGROUND AND RELATED LITERATURE
2.1 NOTATION
Let ((x,, y: be the training data of independent samples (æ,, yr) E X and n, c E N. Let f(w, m) X XX TR denote network realization for any E R.. In general, we interested in minimizing ((.,-) IR x R
+ RJ denoting continuous per-sample loss. The Lg norm of a vector
w E IR is defined as ||wll, = (E |w.19) for > 0. Note that Lg regularizers are defined differently as |w|lg and that for q < only a non-convex quasi-norm is defined. For two vectors wi,Wy E RR, we use O to denote their plement-wiss multiplication. For an optimization problem minw L(w), we denote w == arg min L(w). Finally, the compression ratio (CR) is defined as the ratio of original to sparse model parameters.
2.2 DIFFERENTIABLE L2 REGULARIZATION
Weight factorizations were previously mostly studied for regularized linear models or as toy models for deep learning theory. We briefly illustrate this using the idea of differentiable lasso.
Differentiable lasso
The original lasso objective is defined as
min Lw,x(w) = (ys x. w) + A||w wEBV
(1)
where A 0 promotes sparsity via the L1 norm (Tbbshirani 1996). By factorizing w into wj and we such that w w1 w2, and replacing the -indeiffernntiabee L1 penalty with an Le penalty on = (w), W2), we can obtain differentiable formulation of the lasso (Hoff 2017):
min (w, = Σ (y:- x (wi C wz))² + (1wil1 + ||22 WiW
(2)
2

---

Published as a conference paper at ICLR 2025
The formulation Eq. is equivalent to Eq. in the sense that all minima of the non-convex objective in Eq. (2) are global and related to the unique lasso solution of Eq. (T) as W1 @ W = w.
Hoff 2017) proposes solving Eq. 2 via alternating ridge regression. However, this relies on the biconvexity of the problem and cannot be easily extended beyond linear models.
Differentiable L1 regularization in general neural networks Recently, Ziyin & Wang (2023) proposed applying a shallow factorization to arbitrary weights of a neural network. Coupled with weight decay, this allows obtaining differentiable formulation of the sparsity-inducing Lı penalty that can be optimized with simple SGD. Specifically, by factorizing the weights w of any neural network [w (W, .) as w w1 @ w2, and applying L2 regularization to the factors, the resulting optimization problem has the same minima as the L1 regularized vanilla network. The key insight for the equivalence with L1 regularization is that the factorization rescaling symmetry in the (unregularized) loss Lw.
w1 C w2 introduces
Definition 1 (Rescaling Symmetry). Let the parameters of loss function Lo( be partitioned as
= (W), w2, Po), with 8g denoting the remaining parameters. Then Lg( possesses a rescaling
symmetry w.r.t. arbitrary parameters w1, w2 belonging to 0 (WI, W2, 0o) if for any + 0:
Lo w2, 0o) (e Wi, c w2, 0o) 00
While previous works mainly studied rescaling symmetries naturally aris- ing in, e.g., homogeneous activation functions (Neyshabur et al.] 2015 Parhi & Nowak 2023), weight factorization constitutes an artificial sym- metry that is independent of L, and by extension also of f(.,)) -) and (..x.
Rescaling symmetry
0.75
This applicability to any parametric problem designates artificial symme- tries as a powerful tool for constrained learning Ziyin 2023 Chen et al. 2024). Intuitively, the additional L2 regularization enforces preference for min-norm factorizations (wi, wh) among all feasible factorizations of a given (Fig.B 3. At such min-norm factorization of w, the L2 penalty in Eq. 2 reduces to ||س| @ w 1 = |w/l1 effectively inducing I,2 regular- izatiin the collapsed parameter. This approach allows for implementing L1 regularization in general networks using GD without requiring special- ized algorithms handle non-differentiable regularization.
50.00
0.75
W1
Figure 3: Scalar rescal- ing symmetry and min- norm factorizations.
We refer to Appendix A for additional related methods and discussion. Appendix B provides some intuition why weight factorization with L2 regularization promotes sparse solutions based on Fig.
3 THEORETICAL RESULTS
Based on a given network specification of f(w, æ), we study its depth-D D factorization with 2 2, which we call Deep Weight Factorization (DWF) and is defined follows:
Definition 2 (Deep Weight Factorization). A depth- factorization with E N2 of an arbitrary neural network fw (w,,)) w E RP, is given by fw (w] and factors Wa = (Wi,d. (Wp.d:.....pp.)) E R", d E [D]. The original and factorized parameters are W1
Wp: x) with w (wi....,w) wp)
related through factorization depth is called shallow for D = 2 and otherwise deep.
WD w. where
denotes the collapsed parameter. Further,
In this work, we focus unstructured sparsity. This means all weights and biases in fw are factor- ized using DWF. In principle, however, the factorization can also be selectively applied to arbitrary subsets of the parameters w. Importantly, while DWF does not alter the expressive capacity of the underlying network fw. it drastically alters the optimization dynamics and enables sparse learning in conjunction with Le regularization weight decay. Therefore, focus lies examining the effects of L22 regularization and the behavior of SGD optimization factorized networks.
The regularized training loss with DWF and regularization strength X > 0 is defined to be
1 Σ l (yi, fw (wi .... x.) +
A
Cw.
wall
D
For a given w, applying DWF to the training objective introduces an infinite set of feasible factoriza- tions (wi,....WW)) w w) that leave the network output fw (w, ar) and loss invariant. Those
3

---

Published as a conference paper at ICLR 2025
factorizations, however, differ in their respective norms. While the norm of individual factors can grow arbitrarily large, there exist factorizations that minimize the Euclidean norm, or equivalentl, the factor L2 penalty. L2 regularization thus biases the optimization toward min-norm factoriza- tions. This regularization ensures that the parameter representation strives to be evenly distributed across factors. The following result formalizes the necessary optimality conditions for the factorized objective, identifying solution candidates as those that achieve minimal norm configuration.
Lemma (Necessary condition for solution and minimum La penalty). Let w = (Wi....,w)) E DD be local minimizer of (w,)(w). Then i) ارريا = = |w3 for all j E [p), and ü) the factor Le penalty reduces to D Ed= Wa 12 = - 2/DD*
2/D
Using the result of Lemma I we introduce the concept of factor misalignment to quantity the distance from balanced factorizations required for solutions. Specifically, the factor misalignment defined as Mw)) = D-1 Σd= ||al| - | 2 2 and captures the difference between the factor L2 penalty and that of a balanced minimum-none nnrm factorization of the same collapsed w. The misalignment satisfies (w) 2 0, with equality if and only if the factorization is balanced. This allows to restrict the search for potential solutions to balanced factorizations (w) = 0, as required by Lemma Lemma in Appendix C.niiix...eeeeee C.ddeecribes the remarkable implications of reaching zero misalignment for SGD dynamics, collapsing the dynamics to a constrained symmetry-inducee subspace in which the parameters remain for all future iterations (cf. Fig. 19 for dynamics of M (w)).
The results from Lemmas and 4 highlight the significance of factor misalignment for both the landscape of loss functions under DWF and the trajectories of SGD optimization. For balanced factorizations, the usual smooth L2 penalty remarkably takes the equivalent form of sparsity- inducing regularizer, with SGD dynamics being restricted to simpler symmetry-induced subspaces. Notably, both L2 regularization and SGD noise naturally drive the dynamics towards balance (Chen itt al. 2024). These observations motivate the following key result:
Theorem (Equivalence of optimization problems). The optimization problems
minweRs Lw.x(W) = 1 ET e (Yis fw (w, x;)) + A A|| 1
(4)
my f., == E E (Vi, Jw の Ow,, + E2 [w
(5)
have the same global and local minima with the respective minimizers related as w W O. Owp-
Practically speaking, instead of attempting to optimize the non-smooth problem in Eq. we can alternatively optimize the smooth problem in Eq. (5) as every local or global solution of the DWF model will yield a corresponding local or global solution in the original model space. Hence, this allows inducing sparsity in typical deep learning applications with SGD-optimization by a simple L2 regularization using Eq. (5). In contrast, the nin-differentiaabliiy in 44 will cause the optimization to oscillate and not provide the desired sparsity (see Section [5. and Fig. T.. The L2/D penalty in Eq. (4) becomes non-convex and increasingly closer to the T- penalty for 17'> 2, permitting more aggressive penalization of small weights than L, regularization (Frank Friedman, 1993).
While the theoretical equivalence derived in Theorem establishes correspondence of all minimizers and suggests a simple way to induce sparsity in arbitrary neural networks, the optimization of a DWF model is not straightforward and little is known about the learning dynamics of such model. We will hence study these two aspects in the following section.
OPTIMIZATION AND DYNAMICS OF DEEP FACTORIZED NETWORKS
Two crucial aspects of successfully training DWF models are their initialization and the learning rate when optimizing with SGD.
4.1 INITIALIZATION
Applying DWF to a neural network factorizes each parameter into a product of D factors. However, initializations for factorized neural networks not straightforward since the product distribution of random variables often drastically differs from the factor distribution, leading to pathological be- havior, especially for deep factorizations. To retain the properties of standard initializations, defined
4

---

Published as a conference paper at ICLR 2025
Factor initialization (depth=4) 4)
Product initialization (depth=4)
LeNet-300-100 -000 (F-MNIST)
10
Kaiming tactor)
Est. Kurtosis (4) faiming (factor] Va Natchtch PWF init tours)
Kaiming
Test accuracy
0.8
Density
DWF Init
Kaiming
(ours)
0.6
VarMatch
DWF Init tours)
Depth
2
Depth
Depth
1.0
0.5
0.0
0.5
1.0
0
0.50
0.25
0.00
0.25
0.50
10"
101
10
Factor
Product
Compression ratio
Figure 4: DWF initialization strategies. Left: factor densities with variance matching and truncation. Middle: product densities for D 4 illustrating kurtosis explosion without truncation. Right: for different initializations and D2 showing the failure of standard initialization.
curves
in Appendix C adjustments to the factor initializations need to be implemented. Our exposition focuses on the simplest case ve,") ~V(0 1/n where n is the number of input units to the 1-th layer (LeCun et al. 2002), but similar arguments can be made for other approaches. While Ziyin & Wang 22023) use standard initialization for shallow factorizations with good re- sults, this consistently fails for D > in our experiments and only works in few cases for D (cf. Figs. goss..4amm (2b). The following result shows that initializing a factorized neural network using a standard scheme leads to deteriorating initialization quality and vanishing activation variance:
Lemma 2 (Standard initializations in factorized networks). Consider a factorized neural network with L layers and factorization depth D 2 2, where w = w C @...@ww. w and the scalar factors wi j.a are initialized using a standard scheme. Then the collapsed weights w I1 W (L) -
(1)
as D grows, and ii) for any D 2 the variance of the activations vanishes both The and L.
Rectifying the failure of standard initializations in DWF Given standard initialization ~ P(w) with variance o² the first step is to correct the variance of the product w by initializing the factors wia that the variance of their product matches that of P(w)). This variance match- ing of wo and w is achieved by setting Var(wa) = Var(w)1/D and named VarMaich initialization here. However, only considering the variance overlooks the importance of higher-order moments for initialization in deep learning. For example, given a factor initialization wa ~ (N0, o²), we have [[[=]]] o2D, B[(=)'] 3D implying the kurtosis of w grows exponentially as Niw = 3 regardless of variance matching (cf. Fig. 4). In DWF with plain variance matching, observe a performance decline and the undesirable emergence of inactive weights (cf. Fig. 5a
Since variance matching alone does not yield satisfactory results for D 2, additionally propose a tailored interval truncation of the factor initialization outside of a certain absolute value range and name this approach DWF initialization (see also Algorithm []. This redistributes the accumulating probability mass away from 0 and prevents catastrophic initialization of dead weights. The trun- cation thresholds control the smallest and largest possible absolute values miin and mmax of w, defining the support of the product distribution. Setting the upper truncation threshold to (2rr )/DD to address large outliers and the lower threshold to e!/D) for some ε 0, successfully removes pathological product initializations in our experiments.
Together, the crucial ingredients for DWF initialization are corrections for both the vanishing variance of the product distribution and its concentration around zero
Remark 1. The factorized bias parameters should not be initialized to all zeros, as this corresponds to saddle point from which gradient descent cannot escape by symmetry (see Lemma 4)
4.2 LEARNING RATE
Another challenge in optimizing a depth-factorized model is the choice of learning rate (LR). As shown in Fig. [b] if the LR is chosen too small, the model cannot learn a sparse representation despite achieving the same generalization closely follows previous analyses of large LRs in neural network training dynamics: Nacson et al.
99.4% sparse model trained with large LR. This
Inactive or dead weights are collapsed weights consisting of factors wy with vanishingly small initial- ization values, resulting in s not changing during training.
5

---

Published as a conference paper at ICLR 2025
Init faillure w/o fruncation (CIFAR1O, x-
LR failure w/o large LR (MNIST, A=1e-3)
1.0
0.20
Accuracy
0.8
0.15
Accuracy
0.9
Tain
0.8
we trunc.
Sparsity
0.6
tranc
0.10
Smalll (0
96.5% acc.
0.4
Sparaly
3CC
0.05th
0.7-
Large (0.15) 99.4% sporse
sparse
0.2
0.00
0.0
20
100
120
140
45 Epoch
Epoch
(a) Acc. degradation and dead weights w/o truncation
(b) Sparsity only emerges with sufficiently large LR for LeNet-300- 100 and 1> 4.
despite À = 0 for ResNet-18 18 and D 10.
Figure 5: Failure modes when optimizing factorized neural networks.
2022) show that large LRs help transition to a sparsity-indueing regime in diagonal linear net- works. In more realistic scenarios, (Adriushchennoo et al. (2023) observe that piece-wise constant (step decay) LR schedule with large initial LR induces phased learning dynamics including a short initial learning phase, followed by period of sparse feature learning and loss stabilization, and sud- den generalization upon reduction of the LR. Particular to symmetries and SGD, [Chen commetry-induces structures through stochastic collapse. We conjecture that in DWF, the introduction of D-fold ar- tificial symmetries (cf. Definition D accelerates this phenomenon and thus additionally aids sparse learning. The first row of Fig. @lshows the training dynamics of deep factorized ResNets and demon- strates the requirement of large and small LR phases for DWF training. These training dynamics are further discussed in the following section. Additionally. Appendix E includes an ablation study on different LRs and factorization depths D, suggesting optimal sparsity-accuracy tradeoffs for initial LRs slightly below critical threshold where training becomes unstable (Fig.
demonstrate how large LRs help generalization by causing SGD to be attracted SGD,,Chen et al.] (2024)
Accuracy, Sparsity
1..
cosine
constant
stop decay
100
0.8-
Norm
0.6.
0.4.
0.2
20
0.0
Epoch
Epoch
Epoch
low sparsity
medium sparsity
high sparsity
Accuracy, Sparsky
11.0
0.8
150
0.6.
100
0.4-
0.2.
50
150 Epoch
200
250
150
200
250
Epoch
Epoch
acc.
Val. acc.
Sparsity
Norm
Figure 6: Factorized ResNet- 18 on CIFARIO with D = 4. Dashed lines indicate phase transitions. Top: Different LR schedules with same initial LR and X. Left: cosine LR learns sparse and generalizing solutions. Mid: const. large LR causes sparsification but generalization. Right: step-decay I.R displays sharply distinct sparsification and generalization phases in large and small LR phases. Bottom: For cosine LR, three distinct learning phases occur at all sparsity levels, with sharper contracted dynamics for high sparsity.
4.3 LEARNING DYNAMICS AND DELAYED GENERALIZATION
The learning dynamics of DWF with cosine annealing exhibit three distinct phases, characterized by changes in accuracy, sparsity, and L2 norm of the collapsed weights (Fig. [o] second row): In an initial phase, SGD learns easy-to-fit patterns without overfitting while the Le norm decreases. The reorganization phase is characterized by temporary drops in accuracy and an increase in weight norm, hinting at a period of representational restructuring to accommodate sparsity constraints. Sparsity emerges during
the end of this phase. The final mixed sparsification and gener- alization phase shows improvements in training and validation accuracy as sparsification continues decreasing rate. The mixed nature of the final phase, contrasting sharply separated sparsification

---

Published as a conference paper at ICLR 2025
and generalization with step decay, is owed to the gradual reduction in cosine annealing. Notably, with increasing regularization X, the dynamics contract, and the phases occur in closer succession. This phased behavior shows that the more contracted the reorganization phase is, the higher com- pression and the more severe delayed generalization will be. This is reminiscent of the "grokking"
phenomenon (Power et al. 2022) shown be tightly linked to L2 regularization (Liu al. 2023
4.4 IMPACT OF REGULARIZATION AND EVOLUTION OF LAYER- -IIE METRICS
To investigate dynamics in more detail, we analyze the effect of D and À on the sparsity and training trajectories (Fig. 7 Similar results for different architectures/datasels are included in Appendix
ResNet:18 (CIFAR1O, depth-2)
ResNet:18 (CIFAR10, depthh-)
ResNet-18 (CIFAR1O, depth-4)
Compression rato
10
10a
10a
100
102
10a
10
102
10:
10
104
10f
10a
109
Irain VVll acc.
0.8
0.8
0.8
0.2
0.0
0
50
100
150
200
250
0.0
50
100
150
200
250
0.0
50
100
150
200
250
Epoch
Epoch
Epoch
Figure 7: Impact of regularization A on compression (top), training, and validation accuracy (bottom) for factorized ResNet- 18 and D E (2, 3, 4]. For large À severely delayed generalization and extreme compression emerges simultancously. Colors indicate the same À in hoth rows.
As expected, increasing À leads to higher compression ratios across all depths. Moreover, greater D enables higher compression ratios for the same X. During the initial phase, the regularized training curves coincide with the unregularized trajectory until their departure at the onset of the reorgani- zation phase. This departure occurs earlier the stronger the regularization. For greater factorization depths, the same À values induce higher sparsity at the cost of reduced generalization performance, indicating a stronger regularizing effect] The relationship between sparsity, X, and the collapsed weight norm is further discussed in Appendix E.3] Appendix [F3 presents the layer-wise evolution of sparsity and weight norms, providing more detailed insights into the effects of DWF across the network topology for different architectures (e.g., Fig. [7a]. Two key observations can be made: the first and last layers exhibit less sparsity, owed to their increased importance for the prediction. For the intermediate layers, there is a general trend toward higher compression for later layers. Sec- ondly, the non-monotonic dynamies of the collapsed weight norm seem be almost entirely driven by the first few and the last layers, while the intermediate layers behave homogeneously. Finally, the evolution of factor misalignment and its relation the onset of sparsity discussed in Appendix F.44
5 PERFORMANCE EVALUATION
In this section, we evaluate the performance of DWF. In Appendices F and G we provide further results and details on the experimental setup, including hyperparameters and training protocols.
5.1 FAILURE OF VANILLA T OPTIMIZATION WITH SGD
The failure of SGD with vanilla Ln regularization to achieve inherent sparsity has been previously observed by Ziyin & Wang 2 Kolb et al. (2023). It is natural to ask whether this limitation is merely a benign optimization artifact or if it degrades the prunability of the regularized models. We, therefore, train LeNet-300-100 on MaSSion-MNIST with vanilla L1 regularization well with DWF and D = 2. 3. inducing differentiable L1 and non-convex L2/3 regularization.
Note that this does not imply worse performance in general, but different optimal for different D.
7

---

Published as a conference paper at ICLR 2025
Results The left plot in Fig. (page 1) shows the tradeoff between performance and inherent spar- sity (before pruning) for 100 logarithmically spaced À values, confirming prior findings on the limi- tations of vanilla L optimization. In contrast, differentiable L1 regularization using DWF achieves a compression ratio of about 350 at 80% test accuracy. In addition, our DWF network with D = 3 up to four times sparser than D = 2 at the same accuracy, underscoring the advantages of deeper factorizations. In the right plot of Fig. []] we subsequently apply post-hoc magnitude pruning to each of the models at increasing compression ratios (without fine-tuning) until reaching random chance performance and use the best- pest-eerfoning pruned model at each fixed compression ratio to obtain the pruning curves. Results indicate that differentiable sparse training with factorized networks provides better tradeoffs than vanilla LI, even after accounting for the issues producing sparsity when using SGD with vanilla L1- At 80% test accuracy, vanilla I1 plus pruning requires twice as many param- eters as its DWF counterpart, and three times many as DWF (D) 3). This suggests SGD with L1 struggles to find similarly well-prunable structures, while DWF yields much sparser models.
5.2
RUN TIMES
The perceptive reader might be concerned about the computational overhead induced by training deep factorized networks. Our experiments show this concern to be unwarranted, as the effect of the factorization depth rather unimportant compared to batch size for both time per sample and mem- ory cost. Appendix [.2] illusttats this for WRN-16-8 (Zagoruyko & Komodakis (216)) and VGG-19 (Simonyan & Zisserman 2014). For both models, the impact of factorization depth on computa- tion time and memory usage becomes negligible as batch size increases. These findings suggest that practitioners can leverage deeper factorized networks without incurring substantial additional computational costs, particularly typical batch sizes used in modern deep learning.
5.3 COMPRESSION BENCIMARK
We now evaluate DWF for factorization depths D E (2, 3,43 against various pruning methods con- cerning test accuracy vs. compression, as well the layer-wise allocation of the remaining weights. Architectures and datasets: Our experiments cover commonly used computer vision benchmarks: LeNet-300-1100 and LeNet-5 (LeCun et al.. 1998) on MNIST, Fashion-MNIST, and Kuzushiji- MNIST, VGG-16 and VGG-19 (Simonyan & Zisserman 2014 on CIFAR10 and CIFAR100 and ResNet-18 18 (He et al. 2016) on CIFARTÓ and Tiny ImageNet.
Methods: We compare our method against Global magnitude pruning (GMP) after training (Han) ct al. 2015), a simple pruning method that removes the smallest weights across all layers and is sur- prisingly competitive, especially at low sparsities (Gale et 2019 Frankle et al. (220);; Single-shot Network Pruning (SNIP) (Lee et al. 2019). a pruning at- at-initialization technique, showing competi- tive performance against other recent pruning methods (Wang et al. (020):: SynFlow (Tanaka al. 2020), considered a state-of-the-ar method for high sparsity regimes; Random pruning, serving as naive baseline that removes weights uniformly at random; shallow factorized network (7) 2) which is our variant of the spred algorithm (Ziyin & Wang 2023) with our tailored initialization. Tuning: For comparison methods, we use the established training configurations in Lee et al. (2019);
Wang et al. [2000)] Frankle et al. (2020) when available, and otherwise ensure comparability by us-
ing the same configuration for all methods. All models are trained with SGD and cosine learning rate annealing (Loshchilov & Hutter 2022). For our method, no post-hoc pruning or fine-tuning is required and all layers are regularized equally. Further details are given in Appendix G
Results Fig. g shows results for the fully-connected and convolutional LeNet-300-100 and LeNet- 5 architectures on MNIST. F-MNIST, and K-MNIST. Across all datasets and models, our proposed DWF consistently outperforms existing pruning techniques, particularly at higher compression ra- tios. For LeNet-300- 100 on MNIST, DWF with = 3, 4 remains within 5% of the dense perfor- mance even above compression ratio of 500, significantly surpassing other methods. On F-MNIST and K-MNIST, a similar performance gain is observed. LeNet-5 exhibits similar trends, with DWF maintaining high accuracy at compression ratios where other techniques, especially random pruning and SNIP, have collapsed. Notably, DWF sustains performance up to a compression ratio of 100 on K-MNIST, while competitors rapidly decline. SynFlow and GMP generally outperform random pruning and SNIP, but still fall short of DWF. When comparing shallow and deep factorizations, see that D > 2 retains performance and delays model collapse much longer than D = 2. The
8

---

Published as a conference paper at ICLR 2025
LeNet-300-100 (MNIST)
LeNoet300-110 (F-MNIST)
LeNet-300-100 (KMNIST)
1.5
Test sc.acuuraa
0.
2.4
10
10F
T0
10
10
10
10
10
10
1.0
LoNet-5 (MNIST)
LoNet-5 (F (F-MNIST)
LeNet-5 (K
Test accuragy
0.6
0.4
To
10
10
10
10
101
101
Compression ratio
Compression ratio
Compression ratio
Depth
Depth 3
Depth
Random
GMP
SNIP
Sysflow
Dense
Figure 8: Accuracy vs. sparsity tradcoffs for LeNet architectures MNIST and replacements of varying
difficulty. Lines depict median test accuracies and shaded areas the minimum over three random initializations.
results demonstrate that DWF offers substantial gains in compression capability. Further, the clear separation between DWF and other methods in the high-sparsity regime indicates that our approach captures aspects of the model's representational power that are missed by the other techniques. These findings underscore the potential of DWF, particularly under severe parameter constraints.
For larger architectures and more complex datasets (Fig. 9). DWF continues to demonstrate superior performance, albeit less pronounced. We observe that the D = factorization excels in the medium sparsity regime below a compression ratio of 100, while D > shows enhanced resilience to performance degradation and delayed model collapse at more extreme sparsity levels.
ResNet-18 (CIFAR10)
VGG-16 (CIFARIO)
Test accurac
0.6
0.4
0.4
10
10
10
10"
10"
10
VGG-19 (CIFAR100)
ResNet- 18 (Tiny Imagenet)
Test accuracy
0.8
0.6
0.6
Dapty Rundom
Synton
0.4
Darue
0.4
0.2
10
10
10"
10a
Compression ratio
Compression ratio
Figure Accuracy vs. sparsity for larger ResNet and VGG architectures CIFAR and Tiny ImageNet.
Table showcases the sparsest models achieved by each method while maintaining performance within 5% or 10% of the dense model accuracy. These tolerance levels suitable for testing the medium to high sparsity regimes we focus on. Other levels can be read from Fig. 9 Of the 10 presented scenarios, DWF with D 3, 4 achieves the highest compression in 9 cases, demonstrating the robustness of DWF in preserving model performance under extreme sparsity requirements. The best-ranked DWF model achieves and surpasses shallow factorization in all but one setting, albeit with smaller improvements ranging from 8% to 298%. For example, DWF with = 3 reaches a compression ratio of 1014 (1456) for VGG-19 on CIFAR 100 (ResNet-18 18 on CIFAR10) at 10% tolerance, improving by 8% (25%) over D and by 381% (102%) over the best pruning method SynFlow.
5 times the compression ratio of the best pruning method,
Allocation of layer-wise sparsity Finally, we investigate the reasons for model collapse in SNIP and GMP in the high sparsity regime by plotting the layer-wise remaining ratio (I/CR) for ResNet- 18 and VGG-16 on CIFARIO in the medium and extreme compression regimes, as shown in Fig. 10 At high compression, for both ResNet-18 and VGG-16, we observe that GMP and SNIP catastroph- ically prune entire layers. In contrast, SynFlow and DWF automatically learn adaptive layer- wise sparsity budgets which helps in avoiding such issues. While SynFlow does prune some layers en- tirely in ResNet-18 18, these correspond to skip connections that be removed without interrupting

---

Published as a conference paper at ICLR 2025
Table 1:
Compression ratios (sparsities) of sparsest model within Eace percentage points of the dense model test accuracy, Random pruming is left out clarity.
(1)
Euce
LeNe1-300-100
LeNet-5
ResNet-18
VGG-19
ResNet-11 Tiny ImageNet
I-MNIST
K-MNIST
CIFARIO
CIFARI00
Depth
5%
141 (99.29%)
52 (98.08%)
466 (99.79%)
484 (99.79%)
6988 34%)
10%
362 (99.72%)
78 (98.71%) %)
1169 (99.91%)
939 (99.89%)
99 (98,9999 99%)
Depth 3
5%
506 (99.80%)
75 (98.67%)
573 (99.83%)
440 (99.77%)
67 (98.51%)
10%
1422 (99. (99.93%)
134 (99. (99.25%)
1456 (99.93%)
1014 (99.90%)
161 (99.38%)
Depth 4
5%
486 79%)
75 (98. 67%)
445 (9%.78%)
215 (99.53%)
13 (92.39%)
10%
1442 (99.9399 93%)
139 (8%)
1161 (99.91%)
675 (99.85%)
113 (99.12%)
GMP
5%
156 (99.36%)
22 (95.37%)
211 (99.53%)
37 (97.27%)
60 (98.33%)
10%
235 (99.58%)
32 (96.84%)
484 (99.79%)
68 (98.52%)
133 (99.25%)
SNIP
5%
76 (98.69 69%)
17 10%)
140 (99.299))
28 (96.47%)
18 (94.34%)
10%
146 (99.32%)
24 (95.91%)
339 (99.70%)
42 (97.59%)
41 (97.56%)
Synflow
5%
141 (99.29%)
21 (95.23%)
210 (99.52%)
46 (97.81%)
24 (95.84%)
10%
(99.67%)
37 (97.30%)
721 (99.86%)
218 (99.54%)
(98.60%)
the synaptic flow. Comparing SynFlow and DWF, we observe that DWF produces higher sparsity in the first and last layers across all settings. This is particularly desirable property, as it leads to greater computational savings for a given overall sparsity level. Moreover, as opposed to remov- ing the skip connections, DWF allocates less sparsity to these structures, suggesting a qualitatively distinct underlying structure optimization mechanism.
ResNet-18,
ResNet-18,
100
10a
Remaining ratio (1/c)
10-1
10-1
10
10-1
10
7
13
15
17
10
19
VGG-16,
50
VGG-16,
1000
Remaining ratio (1/cr)
10-1
10-1
20-2
10"2.
Depth
Dopch
20-1
GMP
10-1.
10-3
SNIP
10-1
Syntow
10
11
13
15
5
11
13
15
Layer
Figure 10: Allocation of layer-wise sparsity for different methods. SNIP and GMP show catastrophic pruning of whole layers (collapse) for high sparsities, whereas DWF, like SynFlow, finds adaptive sparsity allocations.
CONCLUSION
This paper introduces deep weight factorization (DWF), extension of previously proposed dif- ferentiable J, regularization to induce sparsity in general neural networks. By factorizing weights not only into two, but D > 3 parts, that can be incorporated in any neural network. We identify practical training obstacles and propose tailored optimization strategies such depth-specific initialization and sparsity-promoting learn- ing rate scheme. We also characterize three distinct phases that describe the learning dynamics and (delayed) generalization behavior of DWF. Experiments demonstrate that DWF usually superior to shallow factorization and outperforms dominant pruning techniques.
method provably induces differentiable L2/D regularization
Limitations and future work In this work, we primarily focused on D E 3,43, While not incurring significant computational overhead (cf. Fig. 26). we found that increasing the factoriza- tion depth beyond four did not yield further sparsity improvements and introduced optimization challenges (cf. Fig. [4]. Additionally, limitation of our work is that exclusively focused on DWF approaches resulting in unstructured sparsity regularization. Hence, an interesting potential direction for future research is extend our factorization to structured sparsity problems.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
DR's research is funded by the Deutsche Forschungsgemeinnccaatt (DFG, German Research Foun- dation) 548823575.
REFERENCES
Maksym Andriushchenko, Aditya Vardhan Varre, Loucas Fillaud-Vivins and Nicolas Flammarion. Sgd with large step sizes learns sparse features. In International Conference on Machine Learn ing, pp. 903-925. PMLR, 2023.
Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. Advances in Neural Information Processing Systems, 32, 2019.
Francis Bach, Rodolphe Jenatton, Julien Mairal, Guillaume Obozinski, et al. Optimization with sparsity penalties. Foundations and Trends in Machine Learning. 4(1):1-106, 2012.
Kartikeya Bhardwaj,, Milos Milosavljevic, Liam O'Neil, Dibakar Gope, Ramon Matas, Alex Chalfin, Naveen Suda, Lingchuan Meng. and Danny Loh. Collapsible linear blocks for super-efficient super resolution. Proceedings of Machine Learning and Systems, 4:529-547 -0999547,250 2022.
Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state of neural network pruning? Proceedings of machine learning and systems, 2: 2:129-146, 2020.
Kevin Bui, Fredrick Park, Shuai Zhang, Yingyong Qi. and Jack Xin. Improving network slimming with nonconvex regularization. IEEE Access, 15292-115314, 2021.
Feng Chen, Daniel Kunin, Atsushi Yamamura, and Surya Ganguli. Stochastic collapse: How gra- dient noise attracts sgd dynamics towards simpler subnetworks. Advances in Neural Information Processing Systems, 36, 2024.
Hongrong Cheng, Miao Zhang, and Javen Qinfeng Shi. survey on deep neural network pruning: Taxonomy, comparison, analysis, and recommendations. IEEE Trunsactions on Pattern Analysis and Machine Intelligence, 2024.
Zhen Dai, Mina Karzand, and Nathan Srebro. Representation costs of linear neural networks: Anal- ysis and design. Advances Neural Information Processing Systems, 34, 2021.
Tristan Deleu and Yoshua Bengio. Structured sparsity inducing adaptive optimizers for deep learn- ing. arXiv preprint arXiv:2102.03886 2021.
Rayen Dhahri, Alexander Immer, Bertrand Charpentier, Stephan Günnnemann, and Vincent For- tuin. Shaving weights with occam's 's razor: Bayesian sparsification for neural networks using the marginal likelihood. In Sixth Symposium on Advances in Approximate Bayesian Inference-Non Archival Track, 2024.
Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International conference on machine learning. pp. 2943-2952. PMLR, 2020.
Jianqing Fan and Runze Li. Variable selection via nonconcave penalized likelihood and its oracle properties. Journal of the American Statistical Association, 96(456):1348-1360, 2001.
L. E. Frank and Jerome H Friedman. A statistical view of some chemometrics regression tools. Technometrics, 109-335 1993.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Pruning neural net- works at initialization: Why are we missing the mark? In International Conference on Learning Representations, 2020.
11

---

Published as a conference paper at ICLR 2025
Jerome Friedman, Trevor Hastie, and Rob Tibshirani. Regularization paths for generalized linear models via coordinate descent. Journal of Statistical Software, 33(1):1, 2010.
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv:1902.0999 095744 2019.
Daniel Gissin, Shai Shalev-Shwartz, and Amit Daniely, The implicit bias of depth: How incremental learning drives generalization. In International Conference on Learning Representations, 2019.
Patrick Glandorf, Timo Kaiser, and Bodo Rosenhahn. Hypersparse neural networks: Shifting ex- ploration exploitation through adaptive regularization. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pp. 1234-122 1243, 2023.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256. JMLR Workshop and Conference Proceedings, 2010.
Yves Grandvalet. Least absolute shrinkage is equivalent quadratic penalization. In ICANN 98: Proceedings of the 8th International Conference on Artificial Neural Networks, Skövde, Sweden, 2-4 September 1998 8, pp. 201- 2061. Springer, 1998.
Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nati Srebro. Implicit bias of gradient descent on linear convolutional networks. Advances in Neural Information Processing Systems, 31, 2018.
Shuxuan Guo, Jose M Alvarez, and Mathieu Salzmann. Expandnets: Linear over-varameee over-parameterization to train compact convolutional networks. Advances in Neural Information Processing Systems, 33:1298-1310, 2020.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances neural information processing systems, 28, 2015.
Trevor Hastie, Rahul Mazumder, Jason D Lee, and Reza Zadeh. Matrix completion and low-rank svd via fast alternating least squares. The Journal of Machine Learning Research, 16(1):3367-3402, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing huven-level performance on imagenet classification. In Proceedings of the IEEE International Conference Computer Vision, pp. 1026-1033 1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Yang He and Lingao Xiao. Structured pruning for deep convolutional neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence, 2023.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. In Proceedings of the IEEE international conference computer vision, pp. 1389-1397, 2017.
Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. Journal of Machine Learning Research, 22(241):1-124, 2021.
Peter Hoff. Lasso, fractional norm and structured sparse estimation using a hadamard product parametrization. Computational Statistics & Data Analysis, 115:186-198, 2017.
Yaohua Hu, Chong Li, Kaiwen Meng, Jing Qin, and Xiaoqi Yang. Group sparse optimization via Ep.9 regularization. The Journal of Machine Learning Research, 18(1):960-011,, 2017.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pp. 448-456. pmlr, 2015.
12

---

Published as a conference paper at ICLR 2025
Arthur Jacot. Implicit bias of large depth networks: notion of rank for nonlinear functions. In The Eleventh International Conference on Learning Representations, 2023.
Li Jing, Jure Zbontar, et al. Implicit rank-mininizing autoencoder. Advances in Neural Information Processing Systems, 33:14736-147483 2020.
LIU Junjie, XU Zhe, SHI Runbin, Ray CC CHEUNG, and KH Hayden. Dynamic sparse training: Find efficient sparse network from scratch with trainable masked layers. In 8th International Con- ference on Learning Representations (ICLR 2020). International Conference on Learning Repre- sentations, ICLR, 2020.
Chris Kolb, Christian L Müller, Bernd Bischl, and David Rügamer. Smoothing the edges: general framework for smooth optimization in sparse regularization using hadamard overparametrization. arXiv preprint arXiv:2307.0335 2023.
Daniel Kunin, Javier Sagastuy-Bres Brena, Surya Ganguli, Daniel LK Yamins, and Hidenori Tanaka. Neural mechanics: Symmetry and broken conservation laws in deep learning dynamics. In Inter- national Conference on Learning Representations, 2020.
Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateek Jain, Sham Kakade, and Ali Farhadi. Soft threshold weight reparameterization for learnable sparsity. In International Conference on Machine Learning, pp. 5544-5555 PMLR, 2020.
Thien Le and Stefanie Jegelka. Training invariances and the low-rank phenomenon: beyond linear networks. In International Conference on Learning Representations, 2022.
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):228--2324, 1998.
Yann LeCun, Léon Bottou, Genevieve B Orr, and Klaus-Robert Müller. Efficient backprop. In Neural networks: Tricks of the trade, pp. 9-50. Springer, 2002.
Lee, T Ajanthan, and P Torr. Snip: single-shot network pruning based on connection sensitivity. In International Conference on Learning Representations. Open Review, 2019.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In International Conference on Learning Representations, 2022.
Jianguuan Li, Thanh Nguyen, Chinmay Hegde, and Ka Wai Wong. Implicit sparse regularization: The impact of depth and early stopping. Advances in Neural Information Processing Systems, 34, 2021.
Yicheng Li and Qian Lin. Improving adaptivity via over- verrpaarametriization in sequence models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Zhiyuan Li, Kaifeng Lyu, and Sanjeev Arora. Reconciling modern deep learning with traditional optimization analyses: The intrinsic learning rate. Advances Neural Information Processing Systems, 33:14544 -4555-15555,,,, 2020.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. Proceedings of the IEEE 2017.
international conference computer vision, pp.
Ziming Liu, Eric Michaud, and Max Tegmark. Omnigrok: Grokking beyond algorithmic data. In The Eleventh International Conference on Learning Representations, 2023,
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In Interna- tional Conference on Learning Representations, 2022.
Christos Louizos, Max Welling, and Diederik P. Kingma. Learning sparse neural networks through ln regularization. In International Conference on Learning Representations, 2018.
13

---

Published as a conference paper at ICLR 2025
Rahul Mazumder, Trevor Hastie, and Robert Tibshirani. Spectral regularization algorithms for learn- ing large incomplete matrices. The Journal of Machine Learning Research, 11:2287-2322 2010.
Nicolai Meinshausen and Peter Bühlmann. High-ding ligeesimmenionnll graphs and variable selection with the lasso. The Annals of Statistics, 34(3):143-1162, 2006.
Mor Shpigel Nacson, Kavya Ravichandran, Nathan Srebro, and Daniel Soudry. Implicit bias of the step size in linear diagonal neural networks. In International Conference on Machine Learning, pp. 16270-16295, PMLR, 2022.
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. In search of the real inductive bias: On the role of implicit regularization in deep learning. In ICLR (Workshop), 2015.
Nadav Joseph Outmezguine and Noam Levi. Decoupled weight decay for any p norm. arXiv preprint arXiv:2404. 10824, 2024.
Wenqing Ouyang, Yuncheng Liu, Ting Kei Pong, and Hao Wang. Kurdyka-lojasievice exponent via hadamard parametrization. SIAM Journal on Optimization, 35(1):62-91, 2025.
Rahul Parhi and Robert D Nowak. Deep learning meets sparse regularization: A signal processing perspective. IEEE Signal Processing Magazine, 40(6):63- 74, 2023.
Scott Pesme, Loucas Hilaud-Vivien, and Nicolas Flammarion. Implicit bias of sgd for diagonal linear networks: provable benefit of stochasticity, Advances in Neural Information Processing Systems, 34:29218-292388 2021.
Iosif Pinelis. The exp-normal distribution is infinitely divisible. arXiv preprint arXiv: 193.0988 2018.
Clarice Poon and Gabriel Peyré. Smooth bilevel programming for sparse regularization. Advances in Neural Information Processing Systems, 34, 2021.
Clarice Poon and Gabriel Peyré. Smooth over-parameterizze solvers for non-smooth structured optimization. Mathematical Programming, pp. 1-56, 2023.
Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Gen- cralization beyond overfitting on small algorithmic datasets. arXiv preprint arX1v:2201.02177, 2022.
Pedro Savarese, Hugo Silva, and Michael Maire. Winning the lottery with continuous sparsification. Advances in neural information processing systems, 33:1 13:11380-111 13900 2020.
Simone Scardapane, Danilo Comminiello, Amir Hussain, and Aurelio Uncini. Group sparse regu- larization for deep neural networks. Neurocomputing, 241:81-89, 2017.
Jonathan Schwarz, Siddhant Jayakumar, Razvan Pascanu, Peter Latham, and Yee Teh. Powerpropa- gation: sparsity inducing weight reparameterisation. Advances Neural Information Process- ing Systems, 34, 2021.
Fanhua Shang, Yuanyuan Liu, Fanjie Shang, Hongying Liu, Lin Kong, and Licheng Jiao. unified scalable equivalent formulation for schatten quasi-norms. Mathematics, 8(8):1325, 2020.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint
2014.
Berfin Simsek, François Ged, Arthur Jacot, Francesco Spadaro, Clément Hongler, Wulfram Gerst- ner, and Johanni Brea. Geometry of the loss landscape in overparameterized neural networks: Symmetries and invariances. In International Conference on Machine Learning, pp. 9722-9732. PMLR, 2021.
Nathan Srebro, Jason D. M. Rennie, and Tommi Jaakkol.. Maximum-marn margin matrix factorization. In Proceedings of the 17th International Conference on Neural Information Processing Systems, NIPS 04, pp. 1329 1336, Cambridge, MA, USA, 2004. MIT Press.
14

---

Published as a conference paper at ICLR 2025
Hidenori Tanaka, Daniel Kunin, Daniel L Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in neural information pro- cessing systems, 33:6377 2020.
Yingjie Tian and Yuqi Zhang- A comprehensive survey regularization strategies in machine learning. Information Fusion, 80:146-166, 2022.
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1):267-288, 1996.
Ryan Tibshirani. Equivalences between sparse models and neural networks. Working Notes, 2021.
Tomas Vaskevicius, Varun Kanade, and Patrick Rebeschini. Implicit regularization for optimal sparse recovery. Advances in Neural Information Processing Systems, 32, 2019.
Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations, 2020.
Huan Wang, Can Qin, Yue Bai, and Yun Fu. Why the state of neural network pruning so con- fusing? on the fairness, comparison setup, and trainability in network pruning. arXiv preprint arXiv:2301.052/99 2023.
Sifan Wang, Hanwen Wang, Jacob Seidman, and Paris Perdikaris. Random weight factorization improves the training of continuous neural representations. arXiv preprint 1rXiv:2210.01274, 2022.
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In Advances in Neural Information Processing Systems, volume 29, 2016.
Blake Woodworth, Suriya Gunasekar, Jason D Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametrized models. In Conference Learning Theory, pp. 3635-663377 PMLR, 2020.
Zonghen Xu, Hai Zhang, Yao Wang. Xiang Yu Chang, and Yong Liang. L1/2 regularization. Science China Information Sciences, 53(6):1 159-1169, 2010.
Yang Yang, Yaxiong Yuan, Avraam Chatzimichailidis, Ruud JG van Sloun, Lei Lei, and Symeon Chatzinotas. Proxsgd: Training structured neural networks under regularization and constraints. In International Conference on Learning Representations (ICIR) 2020, 2020.
Sergey Zagoruyko. 92.45% on cifar-10 in torch.
cifar. html 2015. Torch Blog.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference 2016. British Machine Vision Association, 2016.
Cun-Hui Zhang, Nearly unbiased variable selection under minimax concave penalty. The Annals of Statistics, 38(2):894 942, 2010.
Cun-Hui Zhang and Jian Huang. The sparsity and bias of the lasso selection in high dimeeniiini linear regression. The Annals of Statistics, 16(4):1567-154 2008
Qiaozhe Zhang, Ruijie ZHANG, Jun Sun, and Yingzhuang Liu. How sparse can we prune deep network: A fundamental limit perspective. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Peng Zhao, Yun Yang, and Qiao-Chu He. High-dimensional linear regression via implicit regular- ization. Biometrika, 2022.
Xiao Zhou, Weizhong Zhang, Hang Xu, and Tong Zhang. Effecttvee sparsification of neural networks with global sparsity constraint. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3599-3608, 2021.
Liu Ziyin. Symmetry induces structure and constraint of learning. In Forty-first International Con- ference Machine Learning, 2023.
15

---

Published as a conference paper at ICLR 2025
Liu Ziyin and Zihao Wang. spred: Solving 11 penalty with sgd. In International Conference Machine Learning, pp. 43407-43422 PMLR, 2023.
Liu Ziyin, Botao Li, Tomer Galanti, and Masahito Ueda. The probabilistic stability of stochastic gradient descent. arXiv preprint arXiv:2303 2023.
16

---

Published as a conference paper at ICLR 2025
Appendices
Further related literature
18
Intuition for sparsity via L2 regularized weight factorization
19
Further results and missing proofs
21
Proof of Lemmal
21
Proof of Lemmal
................................
Proof of Theorem
22
Balanced factors and absorbing states in SGD optimization (Lemma
23
D Algorithms
25
D.I DWF initialization
25
D.2 DWF training
25
Details optimization
26
Learning rates in factorized networks
26
Ablation study on initializations
27
Relationship between sparsity, regularization and weight norms
27
Additional results and ablation studies
28
F11
Ablation study on the factorization depth D
28
Combined training and validation accuracy
28
Evolution of layer- wise compression and weight norms
28
Evolution of misalignment and onset of sparsity
................................
30
F55
Post-hoe pruning and tine-tuning
32
F66
Additional sparsity-accurace tradeofis
32
Additional benchmark results
32
Experimental details
34
G.1
Description of comparison methods
34
G.2
Details on architectures, datasets, and training hyperparameters
Other approaches to factor initialization
36
Root initialization and results
36
H.2
Exact Gaussian Factor Representation
36
r
Computational environment and runtime analysis
40
Computational environment
40
1.2
Runtime analysis
40
17

---

Published as a conference paper at ICLR 2025
A FURTHER RELATED LITERATURE
Convex and Non-convex sparse regularization Convex and non-convex regularization for sparsity-induinng effects has a long history as it is well-understood with strong theoretical un- derpinnings. Notable works in this direction include Tibshirani (1996;; Fan & Li (200T); MMin- shausen & Bühlmann (206);; Zhang & Huang (2088) for convex norm based sparse regularization and Friedman et al. (2010); Fan & Lij (2001) : Zhang (2010); Xu et al. (2010) studying its non- convex extensions. While mathematically rigorous, they see limited application in deep learning due to their non-differentiability exactly where sparsity is achieved, requiring the implementation of non-smooth optimization routines that are often inflexible and do not scale well. While some works exist on proximal-type optimization routines in deep learning (Yang et al.) 2020 Deleu & Bengio, 2021]. J. their popularity remains far behind pruning and other approaches, with these methods being rarely applied, used as comparisons, or given much attention in surveys (Hoefler et al., 2021 Gale et al. 2019). Nevertheless, the use of L1 or structured L2,1 regularization is ubiquitous in sparse deep learning (Han et al. 2015) He et al. 2017; Liu et al. 2017: Li et al. 2022). These sparsity regularization pruning methods (Cheng et al., (024) use sparse regularization shrink weights be-
fore applying a subsequent pruning step for actual sparsity. However, the conceptual framework
underlying these heuristic-based methods remains poorly understood.
Weight factorization without and with explicit regularization Weight factorizations, also stud- ied under the names diagonal linear networks (Woodworth al.] [2020). redundant parameterization
(Ziyin & Wang 2023), Hadamard product parameterization (Hoff 2017; Tibshirani |021] Kolb
ct al. 2023) in various contexts, can be traced back to Grandvaler (998) and was rediscovered both in statistics (Hoff 2017) and machine learning Neyshabur et al. (2015). Later, Tibshirani (2021);
Kolb et al. (2023) and Ziyin& & Wang (2023) used this approach to induce sparsity via Le regulariza- tion. Further works using weight factorization from the field of optimization include |Poon & Peyré 2021 2023). Ouyang et al. (025) show that the Kurdyka-Lojasievin
exponent at a second-order stationary point of factorized and L2 regularized objective can be inferred from its correspond- ing L1 penalized counterpart. These works are closest to ours in spirit, namely, factorizing the weights of existing problems achieve sparsity in the original weight space under L2 regulariza- tion. None of these works, however, studied deeper factorizations of neural network parameters. A closely related approach to incorporate the induced non-convex Lg regularization (g < 1) into DNN training was recently proposed by Outmezguine & Levi (2024), who base their method on the --trrcck (Bach et al. 2012) instead of the T.2 regularized weight factorization we study. This method re-parametrizes the regularization function instead of factorizing the network parameters, utilizing a different variational formulation of the Ly quasi-norm. Despite having the same effective Ly regular- ization, DWF incorporates additional symmetry-induccd sparsity- sparsity-promotnng promoting effects through weight factorization and the stochastic collapse phenomenon ZZyinn 2023, Chen et al.] 2024). Another branch of literature studies the representation cost of DNN architectures fw for specific function f, defined as R(J)- = mitlw:fa f [ [23. showing that the L2 cost of representing a linear function
using a diagonal linear network yields the T/2/D quasi-norm (Dai et al. 2021 Jacot 2023).
Apart from the previous works using explicit regularization, the implicit regularization effect of weight factorization was studied by various researchers, both in statistics and deep learning, includ- ing Neyshabuu et al. (2015); Gunasekar et al. (2018); Gissin et al. (2019); Vaskevicius et al. (2019); Woodworth et al. (2000); Pesme et al. (2021) 1): ; Zhao et al. (2022) ; et al. (2021). However, such approaches are usually impractical for real applications, requiring vanishing initializations or spe- cific loss functions. For example, the implicit bias of deep weight factorizations does not extend to non-convex regularizers under the squared loss (Nacson et al. 2022). Powerpropagation (Schwarz) et al. 202T) is a different sparsity-inducing weight transformation with implicit regularization ef- fects, proposed as a modular tool designed for use in practical applications.
Various papers have also studied the factorization of weights without explicit regularization in the context of implicit acceleration caused by the factors acting as adaptive learning rates. Notable examples include Arora et al. (2019): Wang et al. (2022) & Lin (2024).
Pruning in neural networks The landscape of pruning and sparse training methods is confusing due to the plethora of complex sparsification pipelines, incorporating numerous techniques at the same time, and an enormous amount of hyperparameters like pruning schedules learning rates at
18

---

Published as a conference paper at ICLR 2025
different stages. A fair comparison is further complicated by the lack of established, streamlined evaluation processes and opposing objectives in some methods. Pruning, arguably the most popular and widespread method, can be used in innumerable ways to sparsify neural networks, which makes comparisons among these particularly difficult (Wang et al.
2023). Existing pruning techniques use pruning at initialization (Lee et al.. [2019] Wang et al. 2020 Tanaka al. 2020), pruning after train- ing by, e.g., magnitude pruning (Han et al. 2015] et al. 2019). pruning during training, iterative
2020), or pruning and re-growth (Evci et 2020). Recent surveys can be pruning (Frankle al.
found in Blalock et al. (2020); Hoetler et al. (202T): Cheng et al. (2024). Prominent examples in- clude the lottery ticket hypothesis (Frankle & Carbin 2019). proposing that many networks contain equally performant, but much smaller subnetworks that can be found at initialization. A Bayesian pruning version is suggested in [Dahhri et al. (2024). Another approach related to pruning soft thresholding reparameterization (Kusupati et al.] (020), sparse training method incorporating soft sottthhrssooldng step into its network. Despite its success, however, it was shown to be outper- formed by the differentiable L1 approach, i.c., DWF with D 2 Ziyiin & Wang 2023). Recently, Zhang et al. (2024) established theoretical bounds on network prunability using convex geometry. showing that the fundamental one-shot pruning limit without sacrificing performance is determined by weight magnitudes and the sharpness of the loss landscape, providing a unified framework to help explain the effectiveness of magnitude-basee pruning. They empirically show that Lı regularized post-hoc magnitude pruning approximately matches the derived pruning limit before performance degrades significantly.
Sparsity-inducing regularization in neural networks Apart from pruning, the application of norm-based regularizers is also common in deep learning (Scardapane et al.) 2017: Wen al. 2016
Han et al., 2015 Bui et al. (021). This includes Lo-type reguarlization methods (Louizos et al. 2018 Zhou et al. [021 Savarese al., 2020). However, some of these, such as Louizos et al. (2018) were found to not work well due to the stochastic sampling in their training procedure. Other
approaches include adaptive regularization (Glandorf al. 2023) or dynamic masking (Junjie al.] 2020).
Sparsity based on structural constraints While we focus on unstructured sparsity in this paper, various approaches for structured pruning were proposed, including Wen et al.] (2016); Li et al. (2022); Bui et al.] (202T): Liu et al. (2017). For a recent survey, see He & Xiao (2023). A link also made in our paper is the connection between sparsity and symmetries. Using structures of symmetries can guide the sparsification of neural networks. Papers studying this link include the
works of Kunin et al. (2020) Simsek et al. (2021); ;
Le & Jegelka (2022), but also some various
recent work such Ziyin
2023); Ziyin et al. (2023); Chen et al. (2024).
Matrix factorization related induced regularization In [Srebro et al. (2004); Mazumder et al.) 2010)][Shang, et al. : [Hastie et al. (2015) different matrix factorization regularization schemes are proposed to. e.g., learn incomplete matrices (Mazumder et al., 2010 Hastie et al. 2015) for better generalization (Srebro et al.) 2004). There are also neural architectures implementing flavors of matrix factorization to achieve better performance or acceleration (Guo et al. 2020 Jing et al. 2020 Bhardwaj et al., 2022). Although not directly related to our factorization, we will briefly explain their idea to contrast it with our approach. For example, Guo et al.) (2020) note beneficial effect of applying T-2 regularization on the (matrix) factors in the form of [ + || as opposed the L2 regularizer || [ 22 proposed in Arora et al. (2019). This observation can be explained by the low-rank bias induced on the product matrix, whereas the second approach is simple Lg regularization on the product.
B INTUITION FOR SPARSITY VIA L2 REGULARIZED WEIGHT FACTORIZATION
Deep Weight Factorization introduces overparameterization by decomposing each original weight w multiplicatively into D > 2 factors wi,... .......,," WDD Without additional L2 regularization, this induces artificial rescaling symmetries (Definition D). resulting in infinitely many possible factorizations for each weight, all producing the same collapsed network and thus leaving the loss function unchanged. However, when Lg regularization is applied, it influences the choice among these factorizations by preferring those with minimal Fuclidean norm. With La regularization, only minimum-norm (bal-
19

---

Published as a conference paper at ICLR 2025
anced) factorizations can be optimal, otherwise, could always decrease the L2 penalty by pick- ing a more balanced factorization while leaving the unregularized loss unchanged (ef. Lemma D
To provide
geometric intuition using more concrete example, consider the simplest case of factorizing a scalar weight w R into two factors, w = W1 W2. as illustrated in Fig. B for {0, 0.25). The set of all possible factorizations {(w1, w2) R W1wz = w) is given by the points on the coordinate axes for w = O and forms a rectangular hyperbola in the (W), plane for non-zero (cf. Fig. 3 Among these, the factorizations with minimal L2 norm (i.e., minimal distance to the origin) are located at the vertices of the hyperbola. These minimum-norm factorizations are balanced, meaning the factors are equal in magnitude, as the vertices of rect- angular hyperbola always lie either on the diagonal We
or = -w1 Specifically, the two vertices of the resulting hyperbola are given by (Vlw). vww) and (-v [w] - -Vlw) for positive w, and (vlw, /[wl) and (-v V-w, v //wl) for negative w. Combined with the case w 0, the minimum-norm factorizations (wi, wx) for any w are obtained as
(Vlw, (w)) or (-Vlw,, -//l-v/w] (0,0)
0
(wi
(-Vlwl.
O.
At these points, the L2 penalty evaluates to 2/w/, effectively turning into Ln penalty the collapsed weight w scaled by factor of 2.
For deeper factorizations involving more than two factors the same line of reasoning applies, but visualizing the set of possible factorizations as in Fig. 3 for D = 2 becomes challenging. The minimum L.2 penalty at balanced factorizations reduces a non-convex pparsit--inducing T2/D penalty on the collapsed weight. This serves as lower bound of the L2 penalty for every fixed value of w. Once the factors reach this balanced state, which is an "absorbing state" under (S)GD, the optimization process locks in this configuration for all future iterations by symmetry (cf. Lemma Thus, the combination of DWF and L2 regularization induces sparsity in the collapsed weights by promoting balanced factorizations, at which the L2 penalty reduces to lower-degree quasi-norm penalty on w.
20

---

Published as a conference paper at ICLR 2025
C FURTHER RESULTS AND MISSING PROOFS
C.I PROOF OF LEMMA I
Proof. Let w = (w:..... WD) TR. Dy be a local minimizer of fw,)(w). As the factorization is applied independently to each parameter, it suffices to treat the scalar case: We will prove that |,,11| = ... = | for all j E [p].
The rescaling symmetries of DWF ensures that Lw.o (the factorized loss without regularization) is constant over all possible factorizations of a collapsed parameter w. However, the L2 regularization term AD-1 Ed=1 ||س/|| enforces preference for min-norm factorization. For each scalar weight indexed by j [p). consider its factors Wg.1,...,33. Applying the AM-GM inequality the L2 penalty of the DWF loss yields
D- E' w (Ia=1( L/D
W3.1 w... 2/D 1
Vj E
This shows the balancedness requirement for the minimizers of L (w),, the AM-GM inequality holds tight if and only if all terms are equal, i.e., |ر), = _... W|
Summing over the factorizations of weights yields the non-convex L2/D regularizer || the minimum L2 penalty for given collapsed weight w E IR..
2/D 1|2DD as
C.2 PROOF OF LEMMA 2
Definition 3 (Standard Weight Initialization). A standard weight initialization scheme for a neural network layer with Dm input units and Now output units is probability distribution with mean 0 and variance o², where x² og²
Here, g a gain factor depending on the activation function,
Buoak
e is a constant, and Doede either Din; Nout or their sum. Common examples include the Kaiming (o² 2
(He et al. 2015), Glorot (o²
Glooot & Bengio 2010), or LeCun initialization to Haa
2
(a Rin
LeeCn et 2002
Proof. Recall that using standard initialization (ef. Definition 3 each factor is initialized as j.dd ~
(1)
(0 of), where of 1/m (1) < 1 in the case of LeCun initialization (LeCun et al. 2002 For clarity, we assume the width nin to be constant across layers 1 E [L].
(I)
To prove the first statement, we note that B|w |01 = and Var [ (1)
= II Var (Wid) = 2D
Applying Chebyshev's inequality, we get for any O
Var ( (1)
02D
P((=])) - E [[= 2 c) = 1(
(8)
E2
22
Finally, have 0 < limp- P( P((=))) 2 E) Dimo 2D
0, and thus, by the squeeze
theorem: lim P( |( 2 =) 0. This shows that w
(1)
(1) P. 0 as D co.
For the second point, denote the pre-activation of neuron i in layer as
UK
!1) Wky
11)
II k.d Φ (1.
(9)
where w is the d-th scalar factor of the weight w/( associated with input i of neuron k in layer
(1)
1. The activation & y 1 is the activation function & applied to the pre-activations from layer l 1. To simplify calculations, we assume that the activation function is approximately linear
21

---

Published as a conference paper at ICLR 2025
around the origin, implying Var (o (b. 1)) so Var Vi ) and allowing us to ignore the gain
factor, as valid for, e.g., tanh activation. Using that the factors wk and activations & (v! (v(-))) are independent and identically distributed, respectively, the variance of yj (1)
given by:
Var
Min
Var
II
11) k.dd
Φ 1 11
Pie
Var (0:--11)
II d=1
Var (ar(w)))) (10)
Since the factors initialized with Var (wkid))
of
1
the variance of Vi is:
(1)
na
Din Var
Var (1 Var (1
(1) Uk
Var
niin
D
D-1
(11)
Yhin
in
In non-factorized layers, the Tinn in Eq. (IT) cancel out, resulting in equal activation variances across layers. In contrast, standard initializations in factorized layers do not account for the exponent D appearing in the variance of the collapsed weight (1) and thus result in a variance reduction in each subsequent layer as function of input units and factorization depth Applying the above
relationship recursively, we see that the variance at layer is
(
1 (D-1X(L-1)
Var(y Var(y
(12)
Fijn
To avoid reducing or amplifying the magnitudes of input signals exponentially, proper initialization (ar( to equal some constant, typically set unity (He et al. 2015). In factorized networks with D 2 2, however, standard initialization causes strong dependence Rins D, and L.
requires Var(y (1))
C.3 PROOF OF THEOREM
Before proving the theorem, we introduce some required notation. We define the inverse fac- torization function as : R Pp + R", w + W1 @ O WD w. and remark that it relate both objectives using the factor misalignment M (ww
smooth surjection. Using IC,
D-' D. W 11 (w) Lw, x (AC(W) + (M)),, where the misalignment M (w) 2 O attains zero if and only if w represents a balanced factorization. Further, let ((w,s) denote an open ball with radius e around E Rp and recall that K. is continuous at Vf 038 0: K(B(w, 5)) C B(K(w),
/ 22/D- Using Lemma the DWF objective [w.x(w) can be expressed as
Proof. First, we show that if w E arg min (w.)(w). then 3ه E arg argminwer". Lw x (w) such that K(@)) w and C (w) = (w,)(w).
(w.>(w) S 2 (w'). By surjectivity and the multiplicative structure of KC, we can pick a balanced factorization of w so that K(w) w and M (w) = O. Balanced factorizations leave the product sign invariant. Therefore L. (w) = (w.x(X(()))
Since w is a local minimizer of L (w), A > 0: Vw' E B(w, :
unique up to sign flip permutations that
(w)) By continuity
of KC, 3dg KCB((s, du)) S B(K(WW), Eu) B(w, su), i.e., all w' Blw, o() map to some w' E Ev). Then we obtain the following chain of inequalities
Yw' B((,,60) (w)) = Lw.x(w) S L Cw.x(((ww) s (w.x(K(w))) + M(w') = Lw,x(w'),
20
where the first equality holds because M and the subsequent inequality because w is local minimizer of C (w). This shows that à E arg min (w) with L (w)) = (w))
22

---

Published as a conference paper at ICLR 2025
For the other direction, assume that w arg meg (w:)(w), i.e., BED > 0: Yw' B(w, L I Lw.x(w)) By Lemmall [w) = O and therefore Lw, a. ((0) (w,(((())+++ Cw.x(wr) We prove that w K((w)) is local minimizer of L (w) by contradiction. Assume w is not a local minimizer of fw.x(w) then V > 0: 3w' E B(w, 6)= (ww)(w') < L
How-
ever, if s is balanced factorization of w, the following auxiliary result shows that for perturbed w' around w, there must also be a balanced factorization w' of w close to w:
Lemma 3. Let e E RD so that (()) B( = Ew's E B(w,E)) K(w') = w' and Mww')=0)
and let K(()) w. Then Ve 036 0: w'
Proof. We first consider the scalar case of balanced factorization ش, = (w,,1,.... ,W mapping to W E IR, i.e. (w2;) 0 and (W;)w;)))))) Wg. By Lemma | the magnitudes of the factors are equal: [ | 1// for all d E [D]. We now construct factors w/J for any w'J close to w such that (w)Ww) w/J M(w)) 0, and ||' / <e/Vp.
Let > be arbitrary. Since w + Iw, I/D is continuous at w, there exists 6j > such that Iw'- - 35 < dj implies ||w]| 1// | < e/(Dp)). h The factors w/ are defined as follows:
sign (3j.a) |w/1 1/D,
if if * 0, Ad (D),
w/S=
sign(w')) (w/!'/, i 0, d=
|w//'/
if ú = 0, d 2....... D.
This ensures the magnitudes are equal, so M(w)) 0, and that the product of the factors satisfies (w())) = w'j For w, * O, we can apply the sign pattern of W the w'ya by choosing 6, small enough. The resulting distance between w/J and W is then:
|w0// - 2اll2 VD. ||w]| p/ |0, E VD
since ||w/1//D |R,
<<((Dp). Extending to the vector case, let 5 min) For any w' B( 8), each component w' satisfies Iw'- Vty < 5j. Applying the scalar construction to Together, we get
each w'J obtain w' such that K(w') w' and Mww')
1/2
|| - w 0112
Σ 1w x
<E
Therefore, for any e >> 0, there exists 8 > 0 such that w' B(w, 8) implies the existence of w' E e) with K(ww') = w' and M Mww")= (wo)== O.
Choosing = Eo. then 36. > VW E B(w, on) Eü E B(w,.n): K(()) = w and ( 0. By assumption w is not minimizer of <w.)(w), hence Iw' E B(w, 5o) (w") < Lw. x (W). Let w/ E BB,, eo) the corresponding balanced factorization of w' constructed using Lemma with the properties K(w')= =w', M(w") 0 and thus (w,)(w") (w.)(w"). But then
Ew' E B(W, Lw.x(w') Lw.x(w') < Lw. (w) (w))
(13)
contradicting E arg min Lwe. (w). Therefore, if w is a local minimizer of Lw, (w=), then w- Ki((o) is a local minimizer of L..x(w) with L L., (Wr This finishes the proof. Li
C.4 BALANCED FACTORS AND ABSORBING STATES SGD OPTIMIZATION (LEMMA 4
Lemma 4 (Balanced factors are absorbing states in SGD). Consider the SGD iterates of a depth-D - w D at iteration N, where the j-th entry
factorized network with parameters (!!) (w (t)
of the collapsed weight vector =(!)) is w = jjll "....... ...D. Then, i) i = 0 and M (w = 0,
then w (t') 0 for all d E [D] and t' > t. Further, ii) M (0) implies (w) 0 for all t t.
23

---

Published as a conference paper at ICLR 2025
In other words, balanced factorization at 0 causes the SGD dynamics to "collapse" and the factors remain zero for all subsequent iterations, effectively reducing the expressiveness of the model.
Proof. Consider the SGD updates for the factors wa E R,, E [D], factorized network with L2 regularization. Let (w,)(w) denote the part of the loss function without regularization and assume a batch size of without loss of generality:
w (t+1) = =w?? - (Va 2D- (Aw
(14)
Using the chain rule, the SGD updates given by:
W1 (++1) w -n/(') (V ( (Okya (()) 2D- >
(15)
To show the collapse in the dynamics for balanced zero factorization, consider the scalar case such that M (w = O. Then w = for all
w y = with factorization w/") = fw 1D
j
d E [D], and the update becomes:
=0 -0 (.V''"(VVaaaa (Vo (1) - 2 = 3,d
(++1)
(16)
This holds for all subsequent iterations, proving w (t))
j.d 0 for all d [D] and 1' > Next we
show the more general case of SGD dynamics conserving balancedness, i.e., M Mw (0)
0, or
equivalently, (t) j.1
(e) j.nl == my Let Wj,a == (1) m) where s!t) j.d sign (w j.d
and
By = sign s (.)). We investigate the scalar updates:
(1+1) =
(1) _ -1 (IV (.w.cool((())))) (
20
2D 2-20-1100
(17)
Because 1/5
j.d we can factor out $j from all terms in the update. Hence, the resulting
(t)
magnitude at iteration + is:
w ((+1) |mg (0 ([ (m, p- slt) 2D 1m ())
(18)
Since the magnitude constant over d, it is shown that M (ww = 0 for all 11 >
This "stochastic collapse" (Chen et al. 2024) in the gradient dynamics is a recently investigated phenomenon where the noise in SGD dynamics drives iterates toward simpler invariant sets of the weight space that remain unchanged under SGD. The previous result (Lemma 4 about zero mis- alignment being absorbing state in DWF with SGD optimization exemplifies this collapse. How- ever, the dynamics that govern the collapse are poorly understood, including how it is determined when and to which simpler structure the model collapses, with unclear implications for generaliza- tion broad settings. The attractivity of these simpler structures associated with symmetries and high gradient noise levels and closely related the recently studied Type-II saddle points (Ziyin) et al. 2023), potentially helping to explain the benefits of large initial LRs, adaptively regularizing overly expressive networks to constrained substructures via stochastic collapse. While potentially positive effects generalization were shown, the research community is not yet certain about the broader consequences of this phenomenon.
24

---

Published as a conference paper at ICLR 2025
D ALGORITHMS
In the following, we provide the algorithms for the proposed initialization (Section 4.1 of DWF networks in Appendix D.r and how train these networks in Appendix D.2
D.1 DWF INITIALIZATION
Algorithm DWF Initialization with Variance- MatinnneMattchngg and Absolute Value Truncation
1: Imput:
Number and parameter size ny layers, factorization depth D, minimum absolute value e
3: Standard initializations (P(w N/(0, o ,
for to I. do
5: ai t- (Tw.))
1/D
6: (!) 2/D
miin
7: Wiit + min {1, (2ow.)
8:
for each weight w in do
for d 1 D do
10:
repeat
with NN(0,
12:
until wiin < | wman
13:
end for
14: end for
end
16: Output:
Initialized factors (1)) (w)dddd= for all weights E [mr] per layer and layers E [T]]
D.2 DWF TRAINING
Algorithm 2 Training Factorized Neural Networks
1: Input:
2:
Dataset D = ((v,, H-] network architecture A with T. layers and weights R
3: Factorization depth 1) 2 2,
Factor initialization method (DWF-Inis base initialization P,s), (Alg.
5: Training hyperparameters (T, |B], LRSchedule (0) 1,,
Einy (e.g.. float32 machine epsilon 1.19 10-7)
7: Deep Weight Factorization:
8: Factorize the weights w of as:
9: +- w1 .... wp and ohtain fo (w) from fw (w)
10: Initialize weights
[w]]
f- Owr-lnit inni(iiit...... D,E, E, standard init P)
12: for each training step E (O...., 1) do
13: Sample mini-hatch ( [( yb) [=] from D and compute gradient
14: Update using SGD:
15:
w
(++1) wi
(0) Vus (Fw,o(w(!)) AD-1 "2 11) v E [D]
[
Update LR:
17:
(1++))
LRSchedule(f + 1)
18:
end for
19: Post-training factor collapse:
20: Collapse factors to obtain weights for AA:
21:
s w (T) C... O (T)
22:
Apply numerical mach. epsilon threshold Etiny to remove approx. 0 weights:
23:
@j 0 if | Suny V [p]
24:
Transfer sparse weights + back
25:
Output:
26:
Sparse collapsed network parameters to W
25

---

Published as a conference paper at ICLR 2025
E DETAILS ON OPTIMIZATION
E.1 LEARNING RATES IN FACTORIZED NETWORKS
Ablation study on overall learning rate Our goal is to determine suitable LR ranges for achieving high sparsity with good generalization in factorized networks. Additionally, we investigate how deeper factorization affects LR requirements. We train factorized LeNet-300-100 with D (2,3, 4] and our DWF initialization on MNIST, using initial LRs ranging from 10 to 2. All models are trained using SGD with a cosine LR schedule. The results, displayed in Fig. [T] show excessively high LRs lead to unstable results, especially at higher compression ratios. Similarly, too small LRs result in poor or even no sparsification. Notably, models with greater 1) exhibit more robustness to LR variations, maintaining performance over wider range of compression ratios compared to shallower factorizations. Across all depths, selecting a large initial LR slightly below the edge where training becomes unstable yields the best overall results, balancing both effective training with high compression ratios. and providing evidence for the importance a large LR phase in DWF training.
LeNet-300-100 (MNIST, depth-2))
LeNet-300-100 (MNIST, depth-3)
LeNet-300-100 (MNIST, deoth=4)
1.0
1.0
accuracy
0.9
0.8
0.7
0.6 10"
10"
Compression ratio
Compression ratio
Compression ratio
Initial LR
L3e-33
02
12200
Lie0)
2.0+02
Figure 11: Sparsity-accurer tradeoff's for grid of learning rates, demonstrating the importance of appropri- ately large LRs for DWF. Left right shows factorization depths D E (2, 43.
Ablation on the stability of optimal LRs across the sparsity range In another ablation study, we investigate the impact of different sparsity requirements on the optimal initial LR. To do this, we train a LeNet-300-10 100 on MNIST for D E {2, 3, 4} on large number of LR and À combinations. For each D, we train all combinations of the learning rate v) and the regularization X, comprising different LRs between 10-3 and 1, and a grid of 20 À values logarithmically spaced between 10-6 and 10-1, We obtain the Pareto frontier for each D by removing all runs that are dominated by other runs in either test accuracy compression ratio. Fig. 12a shows the corresponding tradeoffs, with the color of the points indicating the optimal learning rate for the corresponding X. Confirming the importance of large LRs, the result further demonstrates that the range of optimal LRs remains a high level across sparsity requirements, except for a slight trend toward distinctly larger LRs for models with little regularization. This can be explained by the intricate relationship between LR and X, together forming the intrinsic LR. When À is reduced, this is compensated using a larger LR to
recover optimal performance (Li al. 2020
Pareto frontier over (LR)x(A) grid
LeNet-5 (F-MNIST)
5e 01
Test Accuraty
2e
accuracy
0.8
Kalming
1e:01 <
VarMatch
Se
0.6
0.8
Depth
2e-02
(ours)
2
0.4
Depth
0.7
5e:03
Depth
0.2
Depth
161
162
10'
1e:03
100
10a
'102
10a
Compression Ratio
Compression ratio
(a) Ablation optimal LRs at different amounts of sparsity using LeNet-300-1 100 on MNIST. Note that none of the smaller LRs are selected as optimal.
(b) Factor initializations and depths D. For D 2, standard initialization performs worse and becomes untrainable for 2.
Figure 12: Experiments on optimal LRs at different amounts of sparsity and different initialization approaches.
26

---

Published as a conference paper at ICLR 2025
LeNet-300-100 (MNIST)
LeNet-5 (K-MNIST)
ResNet-18 (CIFAR10)
10
Compression rabio
301
10.
Depth
Depth
101
207
10P
up3
101
10
Norm (callapted)
30*
10
10'
104
10%
107
101
10:
204
104
10:
201
109
10
101
10
10-1
10:0
1000
10-7
1930
104
1.8
1.8
Norm (collapted)
304
1.0
307
10a
107
107
107
109
191
104
109
10-4
10-1
10-1
101
10"
10
101
10
10
10"
10
10
10.
Compression ratio
Compression ratio
Compression ratio
Figure 13: Relationship between different regularization strengths and compression ratio (first row), regular-
ization strength and Le (second row), well compression ratio and La norm (third row) for different datasets (columns) and different factorization depths (colors).
E.2 ABLATION STUDY ON INITIALIZATIONS
In Fig. 12b we extend the experimental analysis of standard and corrected initialization schemes on performance and sparsity in factorized networks, complementing the experiment connected architecture (right plot of Fig. 4 by a convolutional LeNet-5 architecture. Similar to the results in Fig. 4 observe failure of standard initialization for D 2. For D 2, contrast- ing the results for LeNet-300-14 in Fig. 4 standard initialization indeed achieves some sparsity for LeNet-5 on F-MNIST. The attainable tradeoff, however, is vastly outperformed by using the two corrections in the DWF initialization (Algorithm)
fully-
E.3 RELATIONSHIP BETWEEN SPARSITY, REGULARIZATION AND WEIGHT NORMS
we present results on the relationship between sparsity (measured the CR), regulariza- tion induced by different À values, and the implicit L2 weight norms of the collapsed parameter From the first row, we see that increases in compression ratio for increasing values have a similar trend for all depths, starting to induce sparsity at approximately the same regularization strengths. For all datasets and regularization strengths, except for extremely large À values on ResNett18, the which in turn is sparser than the D model for given A. In the second row and for the smaller models, we see a short increase in the La norm with increasing followed by drop in La norm that finally goes zero at the point where the highest compression is achieved. Remarkably, the collapsed model L2 norm increases with À exactly up to the point where sparsity emerges. slightly different behavior can be seen for ResNet, where the collapsed norm seems to monotonically decrease for increasing À values (i.e., norms do increase first and then decrease). Finally, the third row indicates smaller T-2 norms the more compressed models become, again with deeper factorizations achieving higher compression ratios the same La norm just before model collapse.
Fig. 13
D 4 model always yields higher compression than D
27

---

Published as a conference paper at ICLR 2025
F ADDITIONAL RESULTS AND ABLATION STUDIES
F.1 ABLATION STUDY ON THE FACTORIZATION DEPTH D
In our experiments, we considered deeper factorizations up to a depth of D = 4. This cut-off is not chosen arbitrarily but follows empirical observations that non-convex Lg regularization achieves an optimal tradeoff between superior sparsity performance and difficulty of numerical optimization roughly q 0.5 (Hu et al. 2017). In ablation study, investigate if this also holds for the DWF approach. Fig. 14 displays the sparsity-accurant
curves attained by factorizations depths up to D = and three different LRs in the range that performed well for D 4. We the same hyperparameter configuration as described in Appendix G Results show that in all settings, deeper factorizations beyond 4 offer no improvements in generalization or sparsity, while their training becomes increasingly unstable.
1.0
Initial 0.1
1.0
Initial LR 0.15
1.0
Initial LR 0.2
Accuracy
0.9
Depth
0.9
0.9
0.8-
0.8
0.8-
0.7
0.7
0.7-
10a
10
10
10
10
10
10
10
10
10
10"
10
10"
10
10f
Compression
Compression Ratio
Compression Ratio
Figure 14: Factorization depths D 4 empirically do not improve performance but become unstable train.
Sparsity-accuracy accuracy curves for M-NNet300-100 on MNIST with increasing LRs shown from left right.
F.2 COMBINED TRAINING AND VALIDATION ACCURACY
Fig. contaiins the deferred training and compression trajectories over a range of values, as shown exemplarily for ResNet-18 CIFARIO the main text (Fig- 7 For improved clarity, display the running mean of the validation accuracy over three iterations. In addition, Fig. 15) illustrates the learning dynamics for much finer grid of À values in the top row provide clearer picture of how the training trajectories are affected by different values. Validation accuracies without moving average smoothing are displayed in the bottom row.
VGG-19 (CIFAR1OC, de 22pth=2)
VGG-19 (CIFRR1OO, depth-3)
VGG-19 (CIFARI depth-4)
Tain accuresy P0.
10-1
10-3
10-1
0.4
10
0.2
10
accuratt
9.4
30.2
50
100
150
200
250
50
100
150
200
250
50
100
150
200
250
Epoch
Epoch
Epoch
Figure 15: Impact of regularization
training (top) and validation accuracy (bottom) for VGG-19
CIFAR 100 and (2,3, 4). The top row shows the training curves the whole grid of values. Bottom row shows validation accuracies without running mean for selected X.
F.3 EVOLUTION LAYER-WISE COMPRESSION WEIGITT NORMS
This section provides detailed examination of the layer-wise dynamics regarding the evolution of sparsity, complementing the analysis in Section 4.4 Figure I7 illustrates the wiserwwise evolution of sparsity (top) and collapsed weight norm (bottom) for different architectures and datasets, using a factorization depth 3 and increasing regularization strength X. The plots reveal broadly consistent patterns across different architectures. For stronger regularization, observe more
28

---

Published as a conference paper at ICLR 2025
Lefeet-5 IKMNNIST, depthi
LeNee-5 (KMMSST,
(KMMNIST depth
LeNet 5 IKMNSTT depthi:4)
Comprrse
1e:0
Tann Vell arc.
0.2
0.8
0.0
0.6
Epoch
(a) Convolutional LeNet-5 on K-MNIST
VSG 16 IOIFRRO, depth=2)
VGG-16 (CIFAR10, (CIFAR10.
VGG-16 ICIPARLO, depth
104
2.0
107
30'
10.
2.5
0.8
0.8-
9.6
0.6
0.4
9.4
0.4
0.2
0.4
100
0.2
0.0
200
2.0
200
0.0
150
(b) VG on CIFARIO
WRN 16 (CPARLOO, de
WRN-16 ICIRARIOO, depth-3)
WAN-I6-8 ICIFARIDD, (CFFRIDD.Cepth..
10%
137
103
20P
104
104
10
201
307.
3.5
0.8
g.0.
5.8
Myuth
0.2
5.2
0.2
0.0
202
200
2.0
20
202
0.0
150
(c) WRN-16-8 on CIFAR100
Rushitt 26 (Tiyy Imagenet, depthm2)
BesHel-34 (Tiry Imagenet, desth=3)
Restar:34 (Tiny Imageret, depthidd
107
107
10"
Comarasaa
107
197
10'
301
101
30"
9.0.
5.8
5.4
0.
5.2
0.0
Epach
125
2.0
Epoch
100
125
0.0
25
Spoch
100
(d) ResNet-34 on Tiny ImageNet
Figure 16: Impact of regularization À on compression (top), training. and validation accuracy (bottom) for various architectures and datasets, using D E (2,3, 4).
rapid and pronounced onset of sparsity across all layers, Different layers exhibit varying rates of sparsification, with deeper layers generally achieving higher compression ratios more quickly than earlier layers. The layer-wise norm trajectories show characteristic pattern of initial increase, for the first layer, followed by a peak and gradual decrease. The deeper levels exhibit a simpler dynamic, showing initial short decline followed by low plateau. Stronger regularization leads to earlier
29

---

Published as a conference paper at ICLR 2025
peaking and faster decay of weight norms, corresponding to faster sparsification. Notably, the first layer exhibits distinct behavior (ef. Fig. [8]. often showing the lowest compression ratio and the highest peak in weight norm. These more complex dynamics indicate stronger feature learning in earlier layers closer to the input. Combined, this analysis provides insights into how DWF affects different parts of the network during training and how this process is mediated by regularization.
ResNet-18 (CIFAR1O, depth=3, A=Se-5)
ResNet-18 (CIFAR1D, depth=3 A=7e-5)
ResNet-18 (CIFAR1O, depth=3 X=1e-4)
10
Compression
to
103
10"
L2 Norm (collapsed)
20
100
200
250
100
150
250
100
150
250
Epoch
Epoch
Epoch
(a) ResNet-18 on CIFARIO
VGG-16 (CIPARIO, depth=3, Am-5-5)
VGG-16 ICIFARIO, depth A=7e-5)
VGG-16 (CIFARIO, depth 2m1e-4)
10F
102
10a
101
10a
Nerm (collapse)
80
(b) VGG-16 on CIFARIO 10
(CIFAR100, depth A-5e-5)
VGG-19 (CIFARIOD, depthi
(CIFAR100, depthi3,, x=1e-4)
Compression ratlo
10c
109
(callapzed]
150
100-
50
100
150
200
250
50
100
200
230
100
150
200
250
Epoch
Epoch
Epoch
VGG-19 CIFAR100
Figure 17: Layer-wise evolution sparsity (top) and collapsed weight norm (bottom) using D and increasing regularization À (left to right) for different architectures and datasets.
F.4 EVOLUTION OF MISALIGNMENT AND ONSET OF SPARSITY
We investigate the empirical dynamics of the factor misalignment MWw) and demonstrate that DWF ensures balanced factorizations for sufficiently large x. Our analysis reveals interesting connec- tion between the reduction of misalignment and the onset sparsity the learning dynamics, both the layer-wise and overall model levels.
30

---

Published as a conference paper at ICLR 2025
Resteeee B8 (OPARTO, destruz, 2-1e-4
600
Resket-1B (CIFALL,, depthaz, Aede-1
ResNet-18 (CIFARDD depthi 2, 7
0.0
2.00
0.02
6.0
0.0
Epo.ch
Epoch
Epoch
(a) ResNet-18 on CIFARIO (D = 2
6.10
VGG-16 (CIPARID, depth 20
VGG-16 (CIPARLO, depthe N:7e-1
V66-16 ICIPARIO, depth km
6.00
2.00
2.04
600
600
Epoch
Epoch
Epoch
(b) VGG-16 on CIFARIO (D = 2)
Figure 18: Evolution of the average laser-wise factor misalignment (dashed) together with layer-wise sparsity (solid) for ResNet-18 88 and VGG-16 on CIFAR10 and D 2. Increasing values of À shown from left right.
Figures 18a and 18b illustrate the layer-wise evolution of sparsity and the average misalignment per layer for depth-2 factorized ResNet-18 and VGG-16 trained on CIFARIO. The factor misalign- ment (w) is calculated at the layer level and normalized by the number of weights in each layer, providing granular view of misalignment evolution across the network.
VGG-19 (CIFAR1DD, 2epth-2)
VGG-19 19 (CIFAR1OD, depth
VGG-19 (CIFAR1DO, depth-44)
2,500,000
900,000
Misliggnent
2,000,000
800,000
1,000,000
50,000
500,000
$00,000
50
1.0
110
200
250
50
160
150
200
250
0
50
103
155
230
250
Epoch
Fpoch
Epoch
Figure 19: Evolution of factor misalignment (ws for VGG-19 on CIFARI00 with increasing X and factor- ization depths D E (2,3, 4} (left to right).
The results reveal clear relationship between the elimination of misalignment and sparsity emer- gence. The onset of sparsity coincides almost exactly with the elimination of average misalignment per layer, providing empirical evidence for the theoretical connection discussed in Section Larger values of À lead to faster reduction in misalignment and earlier onset of sparsity, demonstrating stronger regularization favors more balanced factorizations.
Two important observations emerge from these results. First, earlier layers broadly exhibit higher initial layer-wise misalignment but decrease a higher rate than later layers. Surprisingly, larger initial misalignment coincides with the most rapid and pronounced onset of sparsity the aver- age misalignment approaches zero. Second, the final layer (yellow) displays distinetly decoupled dynamics, with sparsity emerging within the first few epochs, opposed to the approximately si- multancous onset for the remaining layers.
We also explore if the onset of sparsity relates to the dynamics of different components of the regularized loss. Figure 20) shows the overall training loss Lw, (w) the data fit part (!), and the (non-collapsed) factor L2 penalty D (+) its minimal penalty and the excess penalty misalignment X (w)(),, Early in training, the L2 component strongly exceeds the data fit component. Since the data fit levels out much earlier than the L2 penalty, they intersect some point during training that both LR and influence. Notably, this point where the loss components are balanced coincides precisely with the onset of sparsity and the overall misalignment approaching zero
(+) ||3. The L2 penalty is further decomposed into described in Lemma 1
31

---

Published as a conference paper at ICLR 2025
Onset of sparsity (CIFAR10, depth=3)
Loss component
0.8
0.6
Sparsity
0.4
0.2
20
40
60
80
100
Epoch
pen.
pen: (misaligmment) Sparsity
Balanced
Data ntt (CE)
Min. pen.
Figure 20: Evolution loss components and sparsity for ResNet-18 with depth and 9 10-5
F.5 POST-HOC PRUNING AND FINE-TUNING
Since DWF operates distinctly from most sparsification methods, this offers potential for integration with other pruning techniques. To demonstrate this, we combined DWF with post-hoe pruning a ResNet-18 18 with = 2 factorization trained on CIFAR10. The setup used an initial learning rate of 0.27 and batch size of 256. Each model was trained across a range of À values obtain raw sparsity-accuracy tradcoff curve. These models were then further pruned along a sequence of compression ratios and fine-tuned for 50 epochs using SGD with an LR of 0.11. Fig. 21 presents the results of this experiment. Combining DWF with post-hoc pruning led to increased sparsity at certain accuracy levels up to three times while maintaining comparable accuracy. This demonstrates the potential for integrating DWF with existing pruning techniques.
ResNet-18 (CIFAR10, depth=2)
Ruw spanity PruF
0.90
Test accuracy
0.85
Jek 76
145 50.4
0.80
16
0.75
Jr1 75-7000
122 51
10a
10
20 50
Compression ratio
Figure 21: Additional post-hoc pruming and fine-tuning. ResNet-18 ist first trained with DWF and D 2. The models are post-hoc magnitude pruned and re-trained for another 50 epochs.
F.6 ADDITIONAL SPARSITY-ACCURACY TRADEOFFS
presents sparsity-accuracy tradeoffs for WRN-16-8, ResNet-18, and ResNet-34 on CI- FAR100 and Tiny ImageNet datasets, using factorization depths D (2,3, 4). Contrasting training protocol for section Section [53] we do not tune the LRs here and set them to fixed val- across datasets and architectures. The results show that DWF consistently produces range of sparsity-accuracy tradeoff's across different architectures and datasets without incurring model collapse. Deeper factorizations generally achieve higher accuracies at extreme sparsity levels.
Figure 22
F.7 ADDITIONAL BENCHMARK RESULTS
The following Table 2 shows test accuracies for different compression ratios different LeNet model specifications and different MNIST datasets. While GMP or SNIP sometimes perform best for 90% 95% sparsity. DWF models show the highest sparsity in all medium- and high-sparsie cases. In total, Synflow and SNIP each work best in case, GMP in 6 cases, D yields the highest sparsity in cases, D in 14 cases, and D 4 in 21 cases.
32

---

Published as a conference paper at ICLR 2025
ResNet-18 (CIFAR100)
(CIFAR100)
accuracy
Teyt: accurcyy
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
10
ResNet-18 (myy Imagenet)
ResNet-34 (Tiny Imagenet)
WRN-16-8 (Tiny Imagenet)
Test accuracy
0.6
0.4
0.4
0.4
0.2
Test
0.2
0.2
10a
10
10a
10
10
50
10
100
10a
10
20
10
307
Compression
Compression
Compression
Figure 22: Additional experiments applying DWF WRN-16-8 and ResNet-18. For these experiments, the
LRs were not tuned each selting to (0.2,0.5,007 0.7] for D E (2,3, 4} across models and datasets.
Test accuracy (%) for different compression ratios (columns), madels (rows), and datasets (lable sections).
Table 2:
Spanity
98%
99%
99.3%
73%
99.875%
98.9%
MNIST
Dease
99.26 + 0.03
Dopth
09.36 = 0.04
99.27 1: 0.06
0.06 99.12
99.02
98.33 + 0.09
16.88-2
40.23
10,00 0:00
10:00 = 0.00
10.00 1: 0.00
Depth
0.05
81.22
3.14
40.52
Depth
98.88
4,48
21.71
GMP
99.00
0.07
95.25
0.14
91.97
0.10
33.09
10.07
11.35
0.00
11.35
0.00
11:35
0.00
11:35 0.00
SNIP
08:92
0.21
95.63
0.24
91.29
1. 0.38
64.35
8.51
21.73
11.19
14.34
1 5.18
11:35
0.00
11.35 1: 0.00
Systow
99.00
0.04
95.68 + 0.09
98.18 0.25
96.71 0.33
91.97
2.82
74.37 + 7.80
56.00
2.68
43.08 + 2.34
Random
0.13
59.70
0000
0.00
0.00
0.00
E-MNIST
Daase
90.4 - 0.21
Depth 2
01.30
2: 0.13
90.78 1: 0.26
99:78 0.20
$8.06 0.18
34.68
$4.91
10.00
0.00
10.00
0.00
10.00
Dopth
90.83
0.23
90 + 0.16
90.10 2: 0.18
88.75 0:26
85.84
0.72
79.15
0.74
64.87
2.78
60.53 1: 3.17
0.03
90.00
1.41
0.04
54.1
25.57
SNIP
80.21
0.39
87.21
3.00
68.27
15:13
48.92
16.00
1000
0.00
10.00
0.00
10.00
0.00
1000 0:000
Systlow
89.77
= 0.13
89.16
0.11
61.21 + 0.12
$4.86 0.37
78.68
2.30
66.01 + 9.94
45.45 = 1.00
58.97 1.77
Random
89.28
0.26
86.29 + 0.11
46.32 9.87
15.00
6.67
10.00
0000
10,00 + 0:00
10.00 =: 0.00
10.00 + 0.00
K-MNIST
Dopth
95.45
0.24
94.56
90.52
0.30
81.88 0.15
10.00 0.00
10,00 0.00
10:00 11000
1000 0.00
Dopth 3
05.17 - 0.19
94.11 1: 0.15
93.08 0.10
88.68
0.59
78.02 2:022:::00 0.58
61.69 0.20
21:05
16.00
10.00 0.00
Dopth
94.72
0.19
94.41 0.22
92.91
0.21
$7.91
0.40
79.37
0.19
61.75 0.90
43,09 = 1.57
27.10 12.12
GMP
0.40)
1.64
Syatlow
6.51
Randon
89:52 = 8.60
82.18 0.79
52.65
7.40
2.52 11.20
9.47 0.92
10.00
0.00
10.00 = 1.00
10.00
LaNo1-W0-100
Dosse
0.05
Depth
0.10
97.30
96.33
96.54 0.27
91.15
10.00 0.00
10:00 0.00
10.00
Dopth
07.30 = 0.13
97.25 0.15
97.11 0.21
96.79 0.19
95.83 0.76
03.58 0.34
90.20 0.04
88.55 0,63
Dopth
07.31
0.10
97.22 0.12
97.08 0.15
96.11 + 0.15
96.05 0.25
94.27 0.26
90.62 0.40
88.49
GMP
0.36
Syattow
0.17
96.26
0.59
2.02
Randon
07:35 =: 0.08
95.83
79.32 7.61
38.01 10:45
14.64 2.00
11.35
0.00
11:35 11000
11:35
F-MNIST
0.40
Dopth
31.02
22.68
Depth
0.12
86.97
86.0
0.20
0.35
0.11
Dopth
87.68 = 0.15
87.53
$7.35
0.25
87.15
0.09
86.52
0.16
84.96
0.26
82:32
0.31
61.30 0.40
GMP
88.22
1.00
87.95 + 1
57.10
1.30
85.22
2.15
79.77
5.47
55.70
15.45
26.55
14994
17.29
1.08
83.70
75.64
5.99
66.57
0.57
73.35
29.92
10.16 0.27
0.00
0.00
K-MNIST
Dasse
91:44 = 0.34
0105)
90.74
0.63
0.00
86.00
Deepth
0.16
$6.00
70.16
1.78
GMP
00.43 = 0.37
88,00 0.43
83.18 0.34
75.34
0.61
$1.39 0.57
21.50 4.05
9.11 1.02
10.00 0.00
SNIP
88.51 = 0.49
85.15
79.96
1.03
68.30
0.79
47.25 + 2:15
25.12 +: 3.24
10.00 = 1000
1035 + 1.47
Synflow
88.52 2: 0.33
8600
$3.30 2:300-20000 0.45
77.39
0.30
65.65 1.66
$2.01 + 1.36
36.22 - 1.1
57.92 4: 2-2.0
Random
33

---

Published as a conference paper at ICLR 2025
G EXPERIMENTAL DETAILS
G.1 1 DESCRIPTION OF COMPARISON METHODS
In the following, we briefly describe the comparison methods used in our study, covering different approaches of network sparsification before or post-training.
SNIP (Single-shot Network Pruning): This method introduces the concept of connection sen- sitivity to quantify the impact of individual weights the network's loss function, given by 2 = =1g @ w(O) for layer 1 E [T., where g( is the loss gradient with respect to w(1), By computing this score for each weight at initialization, SNIP identifies and preserves the most crucial connections, enabling effective one-shot pruning before training. This approach has shown remark- able efficacy in maintaining network performance even at high sparsity levels (Lee et al. 2019).
SynFlow: As fata-independent pruning approach, SynFlow addresses the important issue of layer collapse in neural network pruning. It utilizes a layerwise conservation principle to ensure conser- vation of synaptic flow across the network, thereby maintaining high model capacity even under extreme compression ratios. SynFlow has demonstrated state-of-the-ar performance at very high sparsity levels, outperforming many data-driven approaches in scenarios where over 99% of param- eters are pruned (Tanaka et al. (2020).
Global Magnitude Pruning (GMP): This method based on the assumption that the weight mag- nitudes are good proxy for their importance in the network. Despite its heuristic nature, GMP has proven remarkably effective, especially at low sparsity levels. Its success has led numerous refinements and adaptations of pruning schedules and criteria, with its lasting popularity in both research and practice highlighting its robustness and efficacy (Han et al. 2013 Blalock et al. 2020 Frankle et al., 2020).
Random Pruning: Serving as a baseline method, random pruning uniformly removes weights or structures without considering their importance, thereby helping evaluate the effectiveness of more sophisticated pruning strategies.
G.2 DETAILS ON ARCHITECTURES, DATASETS, AND TRAINING HYPERPARAMETERS
Neural network architectures In the following, we briefly describe the neural architectures used in our experiments.
LeNer-300-100: This fully-connected network, designed for MNIST classification, con- sists of an input layer (784 units), two hidden layers (300 and 100 units respectively). and an output layer (10 units). All layers utilize ReLU activation functions. The architecture closely follows the original version proposed by (LeCun et al. 1989), adapted to incorpo- rate modern activations for improved performance.
LeNer-5 (LeCun et al., 1998 is a small but effective convolutional network with two con- volutional layers (6 and 16 filters, both 5x5), and three fully connected layers (120, 84, and 10 units). We use ReLU activations and add batch normalization (loffe & Szegedy 2015) and average pooling after each convolutional layer.
V6G-16 for CIFAR10/100 consists of 13 convolutional layers and fully connected layers (Simonyan & Zisserman 2014). The convolutional part is described by 2x(64 filters), 2x(128 filters), 3x(256 filters), 3x(512 filters), 3x(512 filters), with max pooling inserted after each group. All filter sizes are 3x3. Batch normalization is applied before each ReLU activation described by (Lee et al. 2019). VGG-19 extends VGG-16 by adding one more convolutional layer to each of the last three convolutional blocks, resulting in 19 layers in total. Following (Zagoruyko 2015), the two fully- fully-connected layers before the output are reduced to a single layer layer with 512 units compared to the ImageNet version.
R8sNer-18 is a popular residual network with 18 layers (He et al. 2016). In our implemen- tation, the architecture adapted following common practice for smaller image datasets (Tanaka et al. 2020). We modify the first convolutional layer to use 3x3 filters and remove the initial max pooling layer. The network consists of an initial convolutional layer, fol- lowed by 4 stages of basic blocks (2 blocks each), with filter sizes [64, 128, 256. 512]. Global average pooling is used before the fully connected output layer. Likewise, our ResNet-34 implementation is also adapted for smaller datasets. The architecture follows
34

---

Published as a conference paper at ICLR 2025
Table 3: Summary of datasets used in experiments.
Dataset
Training Samples
Test Samples
Classes
Input Features
MNIST
60,000
10,000
10
784 (28x28x 1)
F-MNIST
60,000
10,000
10
784 (28x28x1)
K-MNIST
60,000
10,000
10
784 (28x28x1)
CIFAR-10
50,000
10,000
10
3,072 (32x32x3)
CIFAR-100
50,000
10,000
100
3,072
(32x32x3)
Tiny ImageNet
100,000
10,000
200
12,288 (64x64x3)
similar pattern to ResNet-18 with more layers in each stage. As with ResNet--18, we use 3x3 filters in the first layer and omit the initial max pooling, appropriate for the image size of our experiments.
WideResNer a ResNet variant whose increased width compared to plain ResNets allows for better feature representations. In our experiments, we choose WRN-16-8, which is specifically suited for CIFAR-like tasks (Zagoruyko & Komodakis) 2016).
Datasets In our experimental evaluation, varying size and complexity, summarized in Table MNIST, Fashion-MNIST (F-MNIST), and KuzushijjMMNIST (K-MNIST) are grayscale image datasets, each containing 10 classes with images of 28x28 pixels. The original MNIST comprises handwritten digits. while F-MNIST contains images of clothing items, and K-MNIST has handwrit- ten Japanese characters. These datasets combine a range of classification tasks with similar input dimensions but varying levels of difficulty.
several standard image classification datasets of
CIFAR10 and CIFARIOK 100 contain 32x32x3 (color) images with 10 and 100 classes respectively. These datasets present more challenging classification tasks due to their higher resolution, color in- formation, and larger number of classes for CIFARI00.
Finally, Tiny ImageNet is a subset of the ImageNet dataset featuring 200 classes with 64x64x3 color images. This dataset is markedly more challenging and computationally intensive due to the rela- tively complex task with more and higher resolution images, as well as a larger number of classes. All datasets are split into training (50,000 60,000 samples) and test (10,000 samples) sets. We fur- ther apply standard data pre- pre-processing and augmentation techniques: For the three MNIST variants, we use pixel rescaling [0, 1]. The CIFAR and Tiny ImageNet images are normalized. For larger networks, we additionally employ data augmentation, including horizontal flips, width and height shifts (up to 12.5%), and rotations (up to 159). Table contains the combinations of architectures and datasets we conducted experiments on.
Training hyperparameters In our experiments, we use training hyperparameter confi gurations following broadly established standard settings (Simonyan & Zisserman 2014 He al.) 2015 Zagoruyko & Komodakis, (2016). as displayed in Table 4 For both LeNet-300-100 and LeNet-5, set the initial LR 0.15 and found it to perform well across datasets, with the exception of SeNet-300-100 on K-MNIST. Because established LRs were found to be suboptimal for DWF, we additionally select the best perforrrr LR (using small 10 6) from discrete grid between 0.05 and for each factorization depth, architecture, and dataset. For DWF, the sparsity level is controlled using a logarithmically spaced sequence of À parameters between 10-6 and 10-1 on which we train each model to obtain the sparsity-accurace tradeoff curves. For the comparison methods in Section we follow the implementation details provided in Frankle et al. (2000); [ee et al. (2019) if available. To make for fair comparison, also train the two LeNet architectures using the same LR of 0.15 and cosine decay. For the larger networks, we only adjust the LR schedule from step to cosine decay but use the prescribed initial LR. To obtain tradeoff curves for the respective pruning methods, we train each method on a sequence of 15 compression ratios between 10t and 10
Further details for DWF Although our method requires no post-hoc pruning, it is sensible to apply sufficiently small threshold to the final collapsed weights to account for numerical inaccu- racies which have no impact on performance. We set this threshold to float 32 mach. eps ss 1.19 x 10-7 Additionally, the DWE initialization (Algorithm I requires specification of the lower truncation threshold for the factor initializations, which we set to ømin = 3 x 10 3 for all our experiments (ef. left plot of Fig.
35

---

Published as a conference paper at ICLR 2025
Table 4: Training hyperparameters for different architectures and datasets. The LRs for the larger models
correspond to factorization depths 2, The comparison methods use standard Kaiming initialization.
LRs for the supplementary results on WRN-16-8 and ResNet-34 were not tuned.
Architecture
Dataset
Epochs
Batch
Optim.
Mom.
Init.
LR
Schedule
MINSTT
75
256
SGD
0.9
DRF-Imit
0.15
Cosine
LeNet-300-100
FANNST
75
256
SGD
0.9
DWF-Init
0.15
Cosine
M-NNIST
75
256
SGD
0.9
DWF-Init
0.4
Cosine
75
256
SGD
0.9
DWE-Imit
0.15
Cosine
LeNet-5
F-NNIST
75
256
SGD
0.9
DWF-Init
0.15
Cosine
MNNST
75
256
SGD
0.9
DWF-Init
0.15
Cosine
VGG-16
CIFAR-10
250
128
SGID
0.9
DWF-Init
(0.5,0.6.0.6)
Cosine
VGG-19
CIFAR-100
250
128
SGD
0.9
DMF-Init (0.3,0.6.0.6)
Cosine
CIBAR-10
250
128
SGD
0.9
DWF-Init (0.2.0.5.0.7)
Cosine
ResNet-18
CIFAR-100
250
128
SGD
0.9
DWF-Init (0.2.0.5.0.7)
Cosine
ImageNer
250
128
SGD
0.9
DWF-Init
(0.5.0.8...1
Cosine
CIFAR-10
250
128
SGD
0.9
DRF-Imit
(0.2,0.5.00.)
Cosine
WRN-16-8
CIFMR- 100
250
128
SGD
0.9
DNF-Init
(0.2,0.5.0.7)
Cosine
Tay InageNa
250
128
SGD
0.9
DNF-Init
(0.2.0.5.0.7)
Cosine
ResNet-34
CIFAR-ICO
250
128
SGD
0.9
DNF-Init (0.2.0.5.0.7)
Cosine
Thy ImageNer
150
128
SGD
0.9
DWF-Init (0.2.0.5.0.7)
Cosine
H OTHER APPROACHES TO FACTOR INITIALIZATION
H.1 ROOT INITIALIZATION AND RESULTS
An alternative option to obtain an initialization of factors Wa that recovers the distribution of the original weight is given in the following.
Definition& 4 (Root initialization). A root initialization of a depth- factorized weight w = w @ OD given by first drawing a single standard weight initialization (Definition for each entry of and assigning W1 +- sign(w) [w I/D and w...... WD w| element-wiss
Fig. 23 compares the root initialization against the vanilla initialization and the proposed DWF ini- tialization with and without truncation. While the root initialization yields satisfactory results im- proving upon vanilla initialization for D = 2, we observe that it behaves similarly to the VarMatch initialization for results for D = 4.
3, both outperformed compared to our DWF initialization and yields the worst
LeNet-300-100 (F-MNIST, depth-2) 1.0
LeNet-300-100 (F-MNIST, depth-3)
LeNet-300-100 (F-MNIST, depth-4) 4)
1.0
10
Test accuracy
0.6
0.6
0.6
Kaiming
VarMatth
Root
0.2
DWP mit
0.2
0.2
100
1o0 "1
107
10
10
103
16
10
107
100
Compression ratio (CR)
Compression (CR)
Compression ratio
Figure 23: Sparsity-accuracy tradeoffss for different depths (columns) and initializations (colors).
In Fig. 24 we further analyze the learning dynamics of DWF model with root initialization. The results demonstrate qualitatively similar learning dynamics to our proposed DWF initialization, sug- gesting them to be general feature of DWF and SGD optimization.
H.2 EXACT GAUSSIAN FACTOR REPRESENTATION
Although the density of random variables whose product is Gaussian is non-trivial, an exact ex- pression terms of compositions of transformations of Camma-distributed random variables can be derived. The expression is found by writing the (symmetric) standard Gaussian where is a Rademacher variable taking values 11 with equal probability and U := In /w|. Pinelis (2018) establish the infinite divisibility of the exp-normally distributed U by inspection of its char- acteristic function. This result can be readily exploited to obtain factor distributions such that their D-times product is zero-mean Gaussian with arbitrary variance og > 0:
-R.
36

---

Published as a conference paper at ICLR 2025
Sparsty,
Nestett11 Troot init, depthm4, cosine LR) ResiNet 18 froot init. depthm4, consi. LR)
ResNeet-1 18 (root init, depthhm4 step UR)
1800
ratio
1150
1500
0.
1206
Compression
10
L2 Norm
900
02
M/
150
Epoch
Epoch
Epoch
Training acc.
Val. acc.
Sparsity
Compression ratio
Norm (collapsed)
Figure 24:
Learning dynamics for the root initialization for different learning rate schedules (columns).
Lemma' (Gaussian Product Factor (GPF) Distribution). Let e Nt. Consider independent and identically (i..d.) distributed random variables {wald=) constructed as:
Wi 2 Ra . exp
(n In(2-ou) Go.d
Σ
20
G
1 In
(19)
2D
2k
2D
k=1
where Ra are i.i.d. Rademacher random variables (taking values =11 with probability 1)) and Gu.a are i..dd Gammal 1) variables for all & 2 O and E {l...., D). Then their product follows a zero-mean normal distribution:
D W d=1
IN0,00)
(20)
Proof. We first establish the result for o = 1 and then extend to arbitrary positive variances. Let ~ (O, 1). Given the symmetry of the normal distribution, we can represent w as w R e where R is a Rademacher random variable and U In |w] is the logarithm of its absolute value. The characteristic function of U is given by Φu (t) = E = E[ [wl]. Since w has an even density, we can simplify the expectation as:
(t) = 2 o it (x)dd
(21)
Substituting the standard normal density J (w(x)
1 x2112
V25"
2 (o
(t) 2 Jo
c- x2- 12
(22)
Substituting u 2²/2 implies /2u and dr V2u
and therefore
it -u du
du (1) = V /
(V2u)"
(23)
/2u
The term (V2u)" can be written as 2 22/2 giving using further simplification:
24//2
(t)
11/-1 ddu
(24)
V
The integral is the Gamma function with argument & 1tit.
(v((
2it/2 V
it
22t/2
I ((2+it)
(25)
2
r(
using the identity ((t) VT. We now apply Euler's product formula to both gamma functions:
(1
1
r(:)
1
(26)
1 +
Let 1tit and 20 = 1. Then:
2.
l itt
2 II (1+
(1) =
(1+
+
1-1tt
and
-
(2)
37

---

Published as a conference paper at ICLR 2025
Taking their ratio and substituting into expression for du (t):
1
2
(t) 2
1
26
(28)
I
+
2k
The exponent simplifies 1+it
2 2 giving:
@ (1) 2
(1+
1//2
1 1+
(29)
it
1-8 1+
2k
For the fraction in the product, multiply numerator and denominator by 2k:
1
26
2k +
1
(30)
2k
+ 1 + 2K+I
2
Writing the expression in exponential form:
DO
du (t) of In..
e
1n(1+
(31)
I+
+
2K+1
This characteristic function reveals that U can be expressed as:
U 2 In(2)
& 2k +1
In(1 + }
2 Eo
(32)
where & sid Exp(1) for all 2 0, since # ln2 is the characteristic function of the constant ln2
2
1iir is the characteristic function of EE., and for each k e lu(1+ }) is the characteristie function of 1 2 In(1 + t). Lastly, T+ 1+T
E For the factors Wa, we dis- is the characteristic function of 2x+1
tribute the components of U across dimensions using the property that the sum of independent Gamma( $. 1) random variables follows Exp(1):
In laal 2 12(
Gh.d
In(1 + x2 })
Go.d
Σ A=
(33)
2D
2
2D
where Gk.d Gamma( 1. 1). Taking the product:
II
Wa II
Rd
exp
Σnn In [wal]
(34)
Since Σdm Gn Gamma(1, Ex, the sum of logarithms recovers the distribution of U:
Σ In wal D U
(35)
Therefore, Ho wa distribution In(o 2D
2
A0 1). For arbitrary of > O, we modify the constant term the factor produce the desired product variance, yielding the result:
we Ra exp
Rid
1n(2 aū)
Go.d
Ih.d
2D In
1+ b)}
II d=1
D
wa 2 A( (36)
2D
2k
k=
Practical challenges Unfortunately, it is hard to efficiently implement sampling from the factor distribution due to the infinite sum and the series additional operations required to compute wa- Since the summands tend to 0 k oo, it seems practical truncate the infinite sum at rea- sonable value to approximate the true factor distribution. Fig. shoss the kernel density estimates of the factor distribution for D {2, 10, 20) as well the densities of the resulting product
38

---

Published as a conference paper at ICLR 2025
alongside an overlay of the ground truth Gaussian. To obtain the estimates, 1000 products sampled using the target standard deviation ow = 0.1. The plots in Fig. 25 show the approximation power computing only the first &e E {1,5} terms of the infinite sum. The results show that even the coarsest approximation using only a single summand k = yields product distributions that are sta- tistically indistinguishable from Gaussian using Kolmogorov-Smirat depth D. Expectedly, the approximation improves with the number of summands k.
test for any factorization
significant drawback the exact Gaussian factorization approach using Lemma over the pro- posed DWF initialization its impractical initialization complexity, rendering the approach unfeasi- ble practice. This is owed to the accumulating number of additional operations required for each scalar weight, such as drawing multiple random variables per single factor wa, computing several constants, summing or multiplying them, and exponentiaiion. For example, initializing a factorized ResNet- 18 with D using vanilla initialization takes z 1s, for the proposed DWF initialization 3s, but for the coarsely approximated exact Gaussian factor distribution with 1 around 180 minutes A-4000 GPU with 48GB RAM, despite yielding identical performance to the DWE init. The same network could not initialized within a time budget of six hours for depths D
Dodi
Retredeat 2000
16.00
Vilue
Danday
(a) Only including the first = 1 summands.
Tadaa 3y6 (nd)
Sactor dameyy IM D-3n
Fadar senett Dod)
Faccor dendryyta OH123)
Ficee danaty 0H201
30
Value
Roout
Podeet
(b) Only including the first 5 summands.
Figure 25: Approximation of a Gaussian product for different truncation values of the infinite sum in Lemmal K-S corresponds to the Halmogorov-Smimoff test for normality.
39

---

Published as a conference paper at ICLR 2025
1 COMPUTATIONAL ENVIRONMENT AND RUNTIME ANALYSIS
1.1 COMPUTATIONAL ENVIRONMENT
Large experiments on ResNet-18 and VGG-19 on datasets CIFARIO, CIFARIO0, and Tiny Ima- geNet were run on an A-100 GPU server with 32GB RAM and 16 CPU cores. Smaller experiments were conducted on single A-4000 GPU with 48GB RAM or CPU workstations.
1.2 RUNTIME ANALYSIS
Here, we investigate the computational overhead induced by DWF compared to vanilla training. We conducted experiments with WRN-16-8 on CIFAR1O and VGG-19 on CIFAR100 across various batch sizes. Each model was trained for 1000 iterations using SGD. We measured the average wall-clock time per sample and peak GPU memory utilization during training. All experiments were performed on single A-4000 GPU with 48GB RAM, repeated five times to report means and standard deviations. Our results, displayed in Fig. 26 and Fig. 27 show that the factorization depth D in the DWF model only has a minor impact on computational costs during training. For batch sizes of 256 or higher, both networks exhibit an indistinguishable time per sample comparable to vanilla training across all levels of D. At smaller batch sizes, we observe slight monotonic increase in runtime with greater D. For example, WRN-16-8 with a batch size of 128 and D 2 runs approximately 10% longer than vamilla training, while VGG-19 with a batch size of 64 and D shows the largest increase of about 80%. These findings demonstrate that DWF training under typical settings incurs only a small additional cost compared to standard training. This contrasts with many sparsification techniques like Iterative Magnitude Pruning (Frankle & Carbin, (019), which can lead to several-fold increases in training time due to multiple cycles of pruning and re-training.
WRN-16-8 (FRN-16-8CIIAAROO) (CIFAR10)
VGG-19 (CIFAR10))
Wallcleck time (mstoamppe)
Orgiral
0.8
0.6
16.6
8.4
0.2
1024
128
1024
Figure 26: Comparison of wall-clock time per sample for WRN-16-8 (left) and VGG-19 (right) on CI- 1AR10/100 across factorization depths D. Results indicate insignificant runtime overhead for DWF compared to vanilla training, particularly for larger batch sizes where runtime is identical.
GPU memory utilization is primarily dependent on batch size, with D having rather small effects in total. In conclusion, besides factorizing the weights into D factors, DWF incurs only a minor additional runtime and memory cost on commonly used convolutional architectures. The minimal increase, especially for typical batch sizes, suggests DWF can be readily integrated into existing training protocols without major changes computational overhead.
WRN-16-8 (CIFAR10)
VGG-19 (CIFAR100) 00)
Paak CIU usage (C10
10_
Iepth
Orignal
ligth
34
122
250
1004
04
120
225
2024
Figure 27: Peak GPU memory utilization for WRN-16-8 (left) and VGG-19 (right) depths D. The
results show that batch training is the dominant factor for memory usage, with only minimal impact of
40


Reviews:
[{"review_id": "eQlecMKMvR", "reviewer": "Reviewer_dKGU", "summary": "This paper introduces deep weight factorization, an extension of shallow weight factorization for neural network sparsification. The key idea is to decompose network weights into D factors (D≥2), allowing the use of standard $L_2$ regularization to achieve a similar optimization problem to that of the regularized $L_p$ loss, while avoiding smoothness problems. The authors study the performance of this algorithm as well as analyze the dynamics and representations learned during sparsification.", "questions": "As stated in the weaknesses section,  I would suggest the authors reframe the work as an exploratory study of the dynamics of sparsification in deep networks and learned representations, and submit to another venue, after conducting a more thorough literature survey.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 1, "strengths": "The paper has some strengths:\n\n1) The topic of sparsification during training is of great interest with growing model sizes.\n2) The paper is well-written and presented.\n3) The equivalence between the different regularization schemes is clearly explained and shown.\n4) Section 4 is particularly interesting to me, as I am not aware of other studies which attempt to analyze the sparsification dynamics, or attempt to interpret unstructured sparsification algorithms.\n5) The ideas presented in the paper are very good, but unfortunately suffer from a lack of novelty as I discuss in the weaknesses section.", "weaknesses": "The main weakness of this paper is that, unfortunately, the method proposed has already been implemented before with minimal difference, both in its current form based on Hadamard products [1] for linear networks and in a simpler completely equivalent form [2] incorporated directly to weight decay for any $p$-norm, including $2/d$. \n\nTherefore the paper offers no real novelty besides the analysis of the onset of sparsity, and since that is not the main focus of the paper it does not justify acceptance. I would suggest the authors reframe the work as an exploratory study of the dynamics of sparsification in deep networks, and submit to another venue.\n\nMinor weaknesses:\n\n1) The datasets and tasks are not studied at large scale, which is the most interesting setting in which sparsification should be performed. In particular, focusing either on small models or too simple data results in untrustworthy results when scaling up. Since this is not the first time this method is proposed, a potential avenue would be studying these types of $L_p$ methods in very large models, or for fine tuning.\n2) The baseline of random pruning is not a useful baseline, basic magnitude pruning during training should at least be explored.\n\n**References:**\n\n[1] *Representation Costs of Linear Neural Networks: Analysis and Design*, Zhen Dai, Mina Karzand, Nathan Srebro, NeurIPS 2021 (https://openreview.net/forum?id=3oQyjABdbC8)\n\n[2] *Decoupled Weight Decay for Any p Norm*, N. Outmezguine, N. Levi (https://doi.org/10.48550/arXiv.2404.10824)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6ABOzqWzk1", "reviewer": "Reviewer_h7VL", "summary": "This paper extends existing (shallow) weight factorization techniques to deep weight factorization(DWF), and demonstrates the equivalence of DWF  with the $L_\\frac{2}{D}$ (where $D$ denotes the depth of the factorization) regularization, thus potentially achieving higher sparsity compared to its shallow counterpart. The authors also provided enhanced strategies for  initialization and  learning rate to ensure the performance of DWF. Additionally, they analyzed the impact of DWF on training dynamics and optimization by empirical experiments, which show that DWF outperforms the shallow weight factorization and several existing pruning approaches.", "questions": "1. The equivalence of DWF to the $L_p$ regularization largely relies on the fact that balanced factorization (or zero factor misalignment) is optimal. Would you provide some intuitive explanation that regularization by the balanced DWF would imply (or encourage) the sparsity of weights?\n\n2. Whether the training for DWF could achieves a global minima, given that the weight factorization approach is  non-convex?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The author presented a generalization of the shallow weight factorization  to overcome the non-differentiability of $L_1$ regularization and proved its equivalence to the $L_\\frac{2}{D}$ regularization.\n2. The strategies like initialization and large learning rate ensure the application of DWF.\n3. The experiments show that DWF outperforms the original weight factorization and several pruning methods.1. The motivation of the paper is kind of vague. Why extending the shallow weight factorization in [1] to the deep one could potentially result in performance gain?  One  suggestion is: building on the equivalence of shallow weight factorization with the $L_1$ regularization, and taking into account that $L_p (p<1)$ regularization would potentially yield higher sparsity thanks to it is closer to the $L_0$ regularization compared to $L_1$ regularization, it is kind of natural to conjecture that deep weight factorization is equivalent to the $L_\\frac{2}{D}$ (where $D$ denotes the depth of the factorization) regularization.", "weaknesses": "1. The motivation of the paper is kind of vague. Why extending the shallow weight factorization in [1] to the deep one could potentially result in performance gain?  One  suggestion is: building on the equivalence of shallow weight factorization with the $L_1$ regularization, and taking into account that $L_p (p<1)$ regularization would potentially yield higher sparsity thanks to it is closer to the $L_0$ regularization compared to $L_1$ regularization, it is kind of natural to conjecture that deep weight factorization is equivalent to the $L_\\frac{2}{D}$ (where $D$ denotes the depth of the factorization) regularization.\n\n2. In Figure 1, does the different compression ratios of vanilla $L_1$ regularization relevant to different values of $\\lambda$? To my knowledge, achieving the highest test accuracy at various compression ratios often requires selecting different $\\lambda$ values. If the value of $\\lambda$  is fixed across different compression ratios, it would be beneficial to search $\\lambda$  at different compression ratios.\n\n3. Considering DWF is a pruning-before-training method, it is not much fair to only conduct comparison with those pruning-before-training methods, such as SNIP and SynFlow. It is suggested to add comparison with some high-performance pruning-after-training methods, such as [2] and [3] (pruning with $L_1$ regularization).\n\n[1] Liu Ziyin and Zihao Wang. spred: Solving l1 penalty with sgd. In International Conference on Machine Learning, pp. 43407–43422. PMLR, 2023.\n\n[2]Renda, A., Frankle, J., and Carbin, M. Comparing rewinding and fine-tuning in neural network pruning. arXiv preprint arXiv:2003.02389, 2020. 3, 4, 6, 9. \n\n[3]Zhang, Q., Zhang, R., Sun, J., & Liu, Y. (2023). How Sparse Can We Prune A Deep Network: A Fundamental Limit Viewpoint. *arXiv preprint arXiv:2306.05857*.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5dc83G71L9", "reviewer": "Reviewer_pmzc", "summary": "This paper is concerned with sparsity-inducing training of neural networks. The fundamental (pre-existing) technical tool is the transformation of a (non-differentiable) L1-penalized problem, into an “equivalent” (differentiable) L2-penalized problem via a Hadamard-product reparametrization of the weights. This paper introduces Deep Weight Factorization (DFW), an extension of the previous idea to reparametrizations including D \\ge 3 Hadamard factors, and shows an equivalence with non-convex sparse regularization. The authors present an analysis of the interplay between DFW and common optimization aspects such as the weight initialization and the choice of step size. Experimentally, the paper includes several comparisons against existing sparsity methods on vision benchmarks, favorably to DWF.", "questions": "The questions marked with * are most important.\n\n*Q1. I think the contributions of the work are sufficient for publication, but I would like to know the authors’ position on W1.\n\n*Q2. In Fig 7 (and related plots in App E), what is the step-size selection protocol? Is it tuned individually for each depth?\n\nQ3. For Sec 5.1, why is the magnitude pruning without fine-tuning protocol justified? Especially at high compression levels.\n\n*Q4. The paper mentions explicitly (L151) “in principle [...] the factorization can also be selectively applied to arbitrary subsets of the parameters w” in the context of unstructured sparsity (in implicit contrast with structured sparsity). Orthogonal to the structured vs unstructured discussion, what is the relationship between DWF and the type of layers involved? Do certain layers benefit or suffer disparately with different factorization depths?\n\nQ5A. I was pleasantly surprised to read the conclusions of the runtime analysis in Sec 5.2 and App I.2. Given (1) the claimed low computational overhead induced by DWF and (2) the change in optimization dynamics caused by DWF, what do the authors think could be the effects of this factorization idea on the training of non-regularized objectives?\n\n*Q5B. As I stated in W1, the paper focuses on relatively small-scale experiments. How do the authors think the experimental conclusions of their runtime analysis would transform at extremely high parameter counts (>1B)?\n\nQ5C. Along the same line of high parameter counts as in Q2B, do we need to “persistently hold” a (multi-factor) reparametrization during training? During the forward phase, it makes no difference to use the collapsed or not-collapsed parametrization, the only difference is for the gradient application. Would it be possible to hold an “ephemeral reparametrization”, sampled on demand during the parameter update? The idea would be to locally simulate the reparameterized GD dynamics, without requiring the persistent footprint of multiple model copies.\n\nQ6. I noticed the absence of optimization “momentum” (as in Nesterov/Polyak) in the experimental details (as presented in App G.2). I understand that this could be a consequence of the specific architectures and tasks considered in the submission. However, I would like the authors to expand on any potential consequences/challenges of using DWF with momentum-based optimizers.\n\n*Q7. The paper begins almost exclusively concerned with the case of the Lasso regularizer (q=1). Eventually, the theoretical connection presented in Thm 1 is established with a fractional regularized (q=2/D). Is this departure from the Lasso case “necessary”, or merely an artifact of the analysis/proof? In other words, if the problem I truly care about is the Lasso, can I still take advantage of the DWF-induced dynamics?\n\nMinor comments/suggestions (No response expected)\n* In Fig 7 (and more generally App E), the case “Depth=1” would be a useful baseline to understand how much of the improvement is afforded by DWF.\n* There are several instances in the paper where the authors mention that “Appendix X contains information on the relationship between Y and Z”, but do not highlight any core insight on the main paper. I believe the current appendices are quite valuable, and the submission could benefit from (very!) briefly summarizing said key discoveries in the main body.\n* Given the many appendices present, the authors may consider including a table of contents for the appendix to aid navigation.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1. The paper is well structured, and the presentation of the ideas is easy to follow.\n\nS2. While the idea of generalizing “weight factorization” to multiple factors is quite natural, the authors do a good job at motivating it, and providing theoretical and empirical insights on their proposed DWF technique.\n\nS3. The authors transparently explore the empirical uncertainties stemming from the change in parametrization. In particular, I appreciated the section on weight initialization: it identifies challenges with traditional initializations, discusses a potential explanation (based on the kurtosis of the distributions) and designs/prescribes a new initialization scheme that is more suitable for use in conjunction with DWF.\n\nS4. The problem of neural network sparsity is very relevant in the current large model context. The simple core principles of the DWF approach, along with the theoretical insights presented by the authors, make it an appealing technique for the deep learning community.", "weaknesses": "W1. The paper exclusively focuses on vision-related tasks, of relatively small scale. \n  * The submission does not provide results on ImageNet-scale vision tasks.\n  * The submission does not include any experiments on language or image generation.\n\nW2. Theorem 1 presents an equivalence between the DWF and “original” non-convex penalized problem, claiming that they “have the same global and local minima”. However, the current discussion does not explore whether the relative quality of a local DWF minimum is or not comparable to the quality of its related minimum for the original problem. In other words, let $\\hat{\\omega}_1 \\odot \\ldots \\odot \\hat{\\omega}_D = \\hat{w}$ be a local minimum of the DWF problem. How good of a local minimum is $\\hat{w}$ for the original problem?\n\nW3. I found the visual presentation of experimental results –in particular, Fig 6, and Fig 7 bottom– too crowded, making it difficult for the reader to extract the relevant information.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hALa9IqATD", "reviewer": "Reviewer_EGQD", "summary": "This paper extends earlier results on weight factorization and proposes a differentiable (though non-convex) regularized training objective for neural networks whose global and local optima coincide with those of a problem with the same training loss and with $\\frac{2}{D}$-regularization on the network weights. For $D \\geq 2$, the naive regularizer is sparsity-inducing but non-differentiable, and solutions to the latter problem fail to achieve competitive results at high compression ratios. The proposed solution is to factorize the network weights into a product of $D$ matrices and apply $L_2$ regularization to each factor independently; the authors call their factorization “deep weight factorization” (DWF). The paper’s key result, Theorem 1, conclusively supports the correctness of their method in theory – though there is no guarantee that one reaches equally desirable local minima in practice while solving each problem.\n\nThe authors then note that standard neural network weight initializations yield poor results for DWF and propose an alternative initialization that fixes the variance of the weights while avoiding weights that are too close or far from 0. They also study the impact of learning rate schedules on the trained networks’ sparsity-accuracy tradeoffs. They finally present empirical results to show that DWF outperforms shallow weight factorizations (albeit marginally for datasets such as CIFAR10/100 and Tiny Imagenet) and post-training pruning methods.", "questions": "- On lines 475-6, the authors claim that “For VGG-19 on CIFAR100 at 10% tolerance, DWF (D = 4) achieves a compression ratio of 1014, surpassing the runner-up (SynFlow at 218) by a factor of almost 5.” Perhaps I’m misreading the bottom-left plot in Figure 9, but don’t $D=2$ and D=3 both outperform $D=4$ at the 10% tolerance threshold (i.e. ~60% accuracy)? Furthermore, $D=2$ may be the best-performing method in this regime (though it is very close to $D=3$) – and this corresponds to shallow weight factorization, which is not an original contribution of this paper.\n- Why use LeNet architectures for the MNIST experiments? To my knowledge, these are essentially never used in practice. Is there a particular scientific reason for using a very simple architecture in these experiments?\n- The plots in Figures 6 and 7 are quite cluttered. Would the authors consider depicting curves for fewer values of $\\lambda$ in Figure 7? They could also be more selective with the information they present in Figure 6.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper develops a natural extension of existing shallow weight factorization methods to construct differentiable regularizers whose corresponding optima are equivalent to those obtained by non-convex and non-differentiable $L^{(2/D)}$ regularization. This is a valuable – and to my knowledge original – theoretical contribution to the literature.\n- The authors present extensive empirical results to study the impact of choices such as initializations and learning rates on their method’s performance. Doing this legwork to resolve critical pitfalls in the implementation of their method enables other researchers to use their work in practice, and substantially increases their method’s potential for impact.\n- The paper is generally well-written, and I was able to follow its exposition without too much trouble.", "weaknesses": "My primary critique of this paper is that DWF’s performance improvements over shallow weight factorization and Synflow (in terms of compression ratio achieved at acceptable cost to accuracy) seem marginal for CIFAR10/100 and Tiny Imagenet. DWF does improve over competing methods by large margins on toy problems such as MNIST and its variants – but the gap between its relative performance on MNIST and on somewhat more complex datasets such as CIFAR and Tiny Imagenet makes me wonder whether the claimed improvements would become even narrower once one leaves the realm of standard ML benchmarks and tackles real-world problems. In light of this and my first question below, the authors may consider tempering their claims regarding the empirical benefits of their method over shallow weight factorization and pruning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wA3dNVBduV", "reviewer": "Reviewer_ogCH", "summary": "Building upon the fact that regularizing neural networks with an L1 penalty is not adequate due to its non-smoothness and its limited compatibility with SGD, the authors investigate the application of the weight factorization framework to induce sparse weights upon training of deep learning models.\n\nThey extend the ideas of [Hoffn, 2017] and [Ziyin & Wang, 2023] to arbitrary factorization depths of neural network weights, and provide theoretically grounded equivalence of training deep factorized neural nets and sparse optimization problems (Theorem 1).\n\nThe authors then address the difficulties in optimizing their model with standard optimization practices and suggest more suited strategies. Furthermore, they analyze the training dynamics of these networks (namely the accuracy-sparsity interactions during training) and provide an empirical evaluation of the performances of their approach against other sparsification frameworks.", "questions": "Please refer to the Weaknesses section.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The presentation is very clear, with a nice balance of theoretical investigations coupled with empirical evaluations.\nI enjoyed reading this paper.\n\nI find Figure 2 to be especially clear and informative, offering a nice intuitive overview of the approach.\n\nI would also like to highlight the thoroughness of the appendix which provides appreciated details and references to related literature.\n\nOverall, factorizing the weights of a network to arbitrary depths to ensure the sparsity of the trained model upon collapsing the weights is an interesting idea that I find worth exploring.", "weaknesses": "I will provide here some suggestions on the presentation as well as technical questions.\n\n* Figure 1 displays accuracy as function of compression ratio (CR), but we don't know what that refers to at that point (it is introduced later in section 2.1). I would suggest defining CR in the caption of the figure or at least point to its definition.\n\n* I am not sure to understand what Figure 3 aims to show. Are the curves' colors referring to the value of $c$ in Definition 1 ?\n\n* I do understand the inevitability of a sparsity-accuracy tradeoff, and acknowledge the link made by the authors regarding the use of large LRs and generalization performance. However, I still fail to wrap my head around why the LR has such a significant impact on the sparsity of the collapsed weights, given that the authors previously show the equivalence between DWF and sparse regularization. At convergence, shouldn't sparsity be present no matter what ? I would greatly appreciate if you could perhaps provide me some more intuition on this.\n\n* I acknowledge the memory and computational advantages in using sparser neural networks, but I was wondering how relevant these advantages would be if one could instead just directly train a dense but smaller (with less parameters) network to begin with. If a large but sparse network can achieve competitive performance with its dense counterpart, wouldn't it be fair to assume that a small but dense network could also be competitive ? If so, what would be the point in sparsifying the large network ?\n\n* This is more of a research question, but there seems to be intricate interactions between accuracy and sparsity in DWF models. Do you plan on investigating these from a theoretical standpoint ? Perhaps it could be interesting to study the drops in accuracy observed in Figure 1 after a certain compression ratio is reached ?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)