Differentiable Sparsity via $D$-Gating: Simple and Versatile Structured Penalization
Authors: Chris Kolb
Laetitia Frost
Bernd Bischl
David Rügamer
Venue: NeurIPS 2025 spotlight ()
Keywords: Structured Sparsity
Overparametrization
Gating
Primary area: deep_learning

Abstract: Structured sparsity regularization offers a principled way to compact neural networks, but its non-differentiability breaks compatibility with conventional stochastic gradient descent and requires either specialized optimizers or additional post-hoc pruning without formal guarantees. In this work, we propose $D$-Gating, a fully differentiable structured overparameterization that splits each group of weights into a primary weight vector and multiple scalar gating factors. We prove that any local minimum under $D$-Gating is also a local minimum using non-smooth structured $L_{2,2/D}$ penalization, and further show that the $D$-Gating objective converges at least exponentially fast to the $L_{2,2/D}$–regularized loss in the gradient flow limit. Together, our results show that $D$-Gating is theoretically equivalent to solving the original group sparsity problem, yet induces distinct learning dynamics that evolve from a non-sparse regime into sparse optimization. We validate our theory across vision, language, and tabular tasks, where $D$-Gating consistently delivers strong performance–sparsity tradeoffs and outperforms both direct optimization of structured penalties and conventional pruning baselines.

Differentiable Sparsity via D-Gating: Simple and Versatile Structured Penalization
Chris Kolb!,2, Laetitia Frost', Bernd Bischl and David Rügamer
DDeparmment of Statistics, LMU Munich
Munich Center for Machine Learning (MCML)
Abstract
Structured sparsity regularization offers a principled way to compact neural ner- works, but its non-differentiabiiity breaks compatibility with conventional stochas- gradient descent and requires either specialized optimizers or additional post-hoc pruning without formal guarantees. In this work, propose D-Gating, a fully differentiable structured overparameterization that splits each group of weights into primary weight vector and multiple scalar gating factors. We prove that any local minimum under D-Gating is also a local minimum using non-smooth structured 22 penalization, and further show that the D-Gating objective converges least exponentially fast to the paappurregarrzzee loss in the gradient flow limit. Together, results show that D-Gating is theoretically equivalent to solving the original group sparsity problem, yet induces distinct learning dynamics that evolve from a non-sparse regime into sparse optimization. We validate theory across vision, language, and tabular tasks, where D-Gating consistently delivers strong performance-spee sparssty tradeoffs and outperforms both direct optimization of structured penalties and conventional pruning baselines.
Introduction
Sparsity in deep learning models has received considerable attention in recent years. On the one hand, unstructured sparsity methods remove individual weights to reduce parameter counts and achieve high compression ratios, but they produce irregular connectivity patterns that hard to accelerate on standard hardware. On the other hand, structured sparsity targets entire groups of parameters, such neurons [49, 62], convolutional filters [38, 40], or attention heads [12, 42, 53, 60, 71], yielding coarser sparsity patterns that translate directly into reductions in floating-poin operations and memory requirements on existing hardware, which results in more efficient deployment of large models [6, 14].
Beyond computational advantages, introducing sparsity can also improve generalization performance [10] and increase model interpretability [20]. Nevertheless, most popular sparsification techniques in deep learning are not based on the non-smooth L1 and L2,1 penalties widely used in classical statistics and machine learning [55, 56]. but rather constitute iterative pruming and retraining pipelines [3, 20, 34), whose main sparsification mechanism is defined by heuristic pruning criteria like parameter magnitudes. In these methods, the pruning step decoupled from training, making it difficult to characterize precisely what overall objective is optimized and to provide principled guarantees. Further, the decision space vast-pruning at initialization [10. 11. 18, 36, 54, 61], after training [38, 40, 41, 69], or sparsification during training [33, 46, 48], each with their own subtleties and tradeoff's [6. 14, making cumbersome for practitioners select a method that balances efficiency, accuracy, and theoretical soundness.
Author correspondence chris. kolb@stat. uni umi-muenchen.cr
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Direct GD L2 214 penalty
+ L2 D-Gating
0.50
0.50
Weight iterate
0.25
0.00
0.00
-0.25
-0.25
10-1
-0.50
W1
-0.50
10
W2
10-1
-0.75
160
1.0
200
0
50
100 Iteration
150
200
0
50
100
150
200
Iteration
Figure 1: Parameter trajectories for a twa-feature squared loss toy objective with non non-convex 22,2DD regulariza-
tion [(w) = (y 21 W1 22 w2) )² + >||w|22 / whose global minimizer is (wi, wā) = (0, 0). Left: Failure of direct gradient descent (GD) optimization to converge to 0 because of the non-differentiability at the origin. Right: D-Gated objective where w = w... I Td. converging smoothly to O.
Sparsity penalties Structured sparsity penalties such the 22,1 norm are, in theory, capable of climinating unimportant parameter groups, but in deep learning. they have mostly served as heuristics to steer post-hoe pruning rather than achieve exact sparsity [18, 40, 62]. Directly enforeing non- differentiable structured sparsity regularization requires solvers that can cope with its non-smooth nature; if this dinfffffreetiabblityy is ignored, optimization may oscillate or converge to dense, suboptimal solutions, as shown in Fig. 1. Replacing standard stochastic gradient descent (SGD) with specialized procedures such as proximal-type algorithms (e.g. [8, 23, 43]) introduces substantial complexity, demands non- standard hyperparameter configurations, and often the routines are adapted to specific use cases or model classes, thereby foregoing modularity. This renders such approaches cumbersome to implement and inhibits their adoption for large-scale deep learning.
With these obstacles in mind, we ask and positively answer the following main question:
Research question: Can we design modular structured sparsity regularization method integrable into any architecture, amenable to SGD, with theoretical guarantees and little practical overhead?
1.1 Related literature
Structured sparsity A range of methods has been proposed to induce block-wise zeros in neural networks, yet they often rely on either post-hoc pruning or non-standard optimizers. Early work applies the convex L2,1 penalty directly to weight groups but optimizes it with vanilla (sub-)gradient descent [49, 62], which fails to find sparse solutions and must be followed by an explicit pruming step [8]. To remedy the non-differentiability
zero. [8] proposes the use of a proximal algorithm, which we aim to avoid in favor of compatibility with SGD. [5] further generalizes the L2, regularizer to non-convex L2.g: 9 1, penalties using custom optimizer. Rather than penalizing weight structures directly, [4. 23, 40] introduce shared scaling factors for each group and impose sparsity on those factors instead of the whole parameter group, but still require careful tuning. Although these competitors can yield exact sparsity under certain settings, they either fall back pruning or abandon standard SGD, motivating our search for fully differentiable, compaaaat alternative.
Differentiable sparsity A possible solution incorporate sparsity-inducin penalties while retain- ing differentiability are approaches that split the parameters into multiple components and impose smooth L2 regularization sparsity penalty on the reconstructed parameters. This idea dates back to [15, 21] and has re- cently been adopted to incorporate differentiable Lı-type sparsity regularization neural networks [25, 30, 28, 29, 57, 72, 73]. In the case of matrix (instead of parameter) factorization using two factors, low-rank bias, given by the trace norm, is induced on the product [27, 52]. The implicit bias literature also investigates such parameter decompositions without L2 regularization, which can also induce sparsity under certain conditions -uch impractically small initialization scales [16, 58, 63, 70]. An extension to implicit group sparsity for linear models is presented by [39]. However, existing proposals either focus on unstructured sparsity, are constrained to only two factors, or do not constitute modular approaches applicable to arbitrary architectures. This leaves gap in the current literature on whether extensions to arbitrary structures are possible, how such an approach can be implemented in practice, and to what extent there are theoretical guarantees to back this method.
the factors, which be shown to induce the desired non-smooth

---

Training
734
7i,D-1
W)
1. Define groups (G....,)) Gr) Apply D-Gating D-1 Wy w/ 7M
Post-training
Collapse gates and reconstruct w, to get sparse solution
2. Obtuin compact architecture by removing groups
3. Train with L, penalty
Original network
D-Gated network
Compact network
Figure 2: Overview of differentiable D-Gating method for structured sparsity (cf. Algorithm 1). For simplicity, we show D D-Gating visually for a single fully- lully-coneected layer with input-wise grouping (colors). but our approach extends to arbitrary network components such convolutional filters attention heads. We proceed by applying D-Gating (red nodes and their connections) to neural network weight and running SGD on the gating parameters with weight decay, After training, the weights are collapsed again and the zero structures removed, with the resulting sparse minimizers also being minimizers of the non-smooth (e.2/-rregulari objective.
1.2 Our contributions
Inspired by prior work on differentiable sparse regularization, we propose a new approach called D-Gating, which constitutes structured sparsity-indccing penalty approach. It can be modularly incorporated in "arbiitray"" architectures and neither incurs a notable overhead additional parameters requires additional pruning steps, and is compatible with off-the-shelf SGD optimization. We further establish novel theoretical results that show the equivalence of our proposed differentiable regularization method and non-differentiable sparsity-inducing penalties (cf. Fig. 2), akin to what has been shown for approaches with unstructured sparsity penalties. Apart from thcoretically and practically studying the loss landscape and training dynamics of our approach, also validate our theory on an array of experiments to showcase its versatility in diverse deep learning applications.
2 Problem statement
In this paper, we propose a general structured sparsity approach for neural networks f(: w) that allows penalizing excessive network components without placing restrictions on the type of architecture or the position of the unit within the weight vector E R that is targeted with the regularization. Structures such as filters naturally arise in neural networks, yielding a partition G = (G ........ G,)
of subset of the indices [p] := {1, .1,....Pl pp) of w into J groups Wg, with elements Wj.g: 9 E 9,. For filter sparsity in convolutional neural networks, G would be all indices for weights in the convolutional layers, and each 9, the indices of weights of one of the J filters.
Given this structure, we seek to optimize general optimization problem
minimize WER*
Lw(() La (w) + \|| 2
2/12
(1)
for D 2 2, where the unregularized objective Lu = Ei= (( w)) is the sum of n observed loss contributions with loss é: x y - I& evaluated on independent data points {(x, y/ v.)]a Ja (X' x y)"" The regularization term in Eq. (1) constitutes a generalization of what is often referred to as the group lasso [67], L2,1 penalty, which is recovered for D 2. For D > 2, obtain the
more general and non-convex group penalty: [ 22,/ = =E Eyeg, |Wj.o 12 [22]. As this penalty is neither differentiable for D = nor D > 2. SGD optimization of (1) yields unfavorable optimization dynamics and does not achieve exact sparsity (cf. Fig. 1). We therefore either require specialized optimization routines or surrogate objective which induces the solution to (1). We choose the latter to minimize the overhead (cf. App. F.2) and changes to established training procedures.
3 Differentiable structured sparsity via D-Gating
To solve Eq. (1) while enabling practitioners use standard SGD optimizers, we derive fully differentiable method that implicitly tackles Eq. (1) by employing an overparameterized model and smooth surrogate penalty compatible with SGD optimization.
3

---

3.1 Model structure and gating variables
As we are interested in general method for structured sparsity where arbitrary subsets of network weights can be sparsified, we assume parametric learning model
f x Retp - R (x, w) + f(x; W):
(2)
with inputs x E and parameters w = (v, w). where w Rp contains the penalized parameters of interest and v E IR9 the remaining parameters. Further, w is a partitioned (structured) weight object comprising J groups, w = (w,)}=1 E -pP....PP.
Rp.
To convert the non-smooth optimization problem into smooth optimization problem, we subdivide w into two parts using the following gating operation:
Definition 1 (D-Catingg). Let RP and 9 (9 ...) be partition of the indices [pi of into groups Further let w E TRP be the primary weight of the same size as w, " E R one of D 1 vectors containing group-wise wise gating factors, and let 7 71 ② ... O 7D-1 denote the - 7j.a for j E [J]. For brevity, we collect the scaling factors in the matrix I = [YI...., YD-1 I E (xx(D1)), The D-Gating operation
element-wist product of the gating factors with entries 7
RP Jx((DD + RP, (w, + y0, decomposes w
Wj Io Wid ) 1=1
(wJ (
j=1
and we call w D-Gated if it is parametrized as yo.
Intuitively, the D-Gating operation splits each group weight w, into D factors: the vector wi. corresponding the original group weights, and the D 1 additional gating factors %j R, which are applied multiplicatively to all entries of wy.
3.2 Differentiable penalty
Given the gated formulation, we can now impose surrogate L2 regularization on (w, r), defined as
L(v,w, S.w.WDy I Lolv.w 70) AR(,II)
(4)
Σ (x, )))) + C E )3,1)
(5)
| + I1
where Lo(v, w) denotes the unregularized. differentiable loss function with per-sample loss E. In the following, will denote local minimizers of the D-Gating objective as
(v, a F) arg loc min www..D)eRotttttc.....
L(v, W.V.
(6)
While the function values of Eq. (4) provides a theoretical guarantee of the equivalence of both objectives with regards to their minima and shows that the difference between both objectives vanishing at least exponentially fast.
necessarily equal to those of Eq. (1), our section
4 Theoretical results
Because the presence of ungated parameters v is inconsequential for our analyses and all results directly carry over, they will be omitted from now on, and we assume for simplicity of exposition that all model parameters are D-Gated.
4.1 Stationarity condition and loss simplification
The following result establishes that stationary points of L(w, I' correspond to balanced D- Gating parameters. Otherwise, one could continuously perturb the D-Gating parameters toward more balanced configuration without altering the effective parameter w 70 while strictly decreasing the L2 penalty R(w, D).

---

Lemma (Balancedness at stationary points), Let (w.T) be D- D-Gating parameters satisfying w Wi II D-1 Iid for E [.]. (w, F) stationary point of the Le-regularized objective C(w, I)
with X > O, then the gating factors are group-wise balanced in the sense that
-5,00 = 11, 2 |
4j [J].
Notably, the loss evaluated at balanced parameters simplifies to reveal its sparsity- inducing nature: Corollary (Loss simplification balanced gating parameters). Lel (w, D') be balanced D-Galing parameters satisfying w = w I[ Yi,d and Eq. (7) j E [J]. The D-Gated objective L(w, I') in Eq. (5) then simplifies to Lo(w
= Lo(w)+ Iw, 1/ = Lo(w) + A ||
12/D =: Lw (8)
2,2/D
(w)())
4.2 Equivalence of optimization problems
The previous result is reassuring it demonstrates the equivalence of objectives at balanced gating parameters. It does, however, not guarantee that optimizing one objective provides a meaningful solution for the other objective. The following result establishes equivalence at the solution level.
Theorem (Equivalence of D-Gating and 2222/D regularization). The two optimization problems
minimize WERr DER",TER'X(D-.
((w,I) I) == Colw 70) + (1W IT
minimize WER
Lw(() == Lo(w) Al| w// 112.2/D
(10)
are equivalent in the sense that their local minima are identical. If (āo, I') E arg loc min L(w, F), then &s 40 = w E arg loc min L (w), and likewise, if w E arg loc min ew(w), then any
balanced gating representation ( () such that w =
4 is a local minimizer of f(w. I).
Specifically, there is bijective mapping between the local minimizers of L (w) and the equivalence class of local minimizers of C(w, T), resulting in the same effective parameter w.
4.3
Optimization dynamics
Under ubiquitous (S)GD-based optimization of the D-Gated objective in Eq. (5), as well as its theoretically simpler continuous- gradient flow (GF) limit with infinitesimal learning rate 77, can additionally establish results characterizing the evolution of parameter balancedness, i.e., quantify how fast the D-Gated objective converges the original 22,2 D regularized loss.
4.3.1 Evolution of imbalance and loss convergence for D-Gated models under GF dynamies
The grse continuous-time gradient flow dynamics for j E [J] are given by
ق = ww,f, Tid -.
(11)
The gradients with respect to the D-Gating parameters W, and the Ys. are, using the chain rule,
2A D
7g
2A
Vu, C = 7, Vw, Lo
ws.
w V Lo + D Ysa d E [D 1].
(12)
We define the wair imbalance I between any two group wise factors d + d' E [D] and show it vanishes exponentially time:
- (1x,)))))));;0 =
I (t)
(13)
- (t)? i 1.
Lemma 2 (Exponential decay of imbalance under continuous-time GF). Under the gradient flow dynamics Eqs. (11) and (12), the pair-wise imbalance Ij (t) (13) between two gating parameters d, d' of group j satisfies
d 4
I., = D I.S Vd + d',j [J].
4A
(14)
Solving the ODE shows I (t) decays exponentially for X 2 0: IJ (t) = II (0)e Bt
5

---

2 2-Gating
Gating
Gating
Loss canvergence
109
c(w.
10
10-12
50
100
150
200
50
100
150
200
50
100
150
200
Epoch
Epoch
Figure 3: Evolution of imbalance during SGD of neuron-wish D-Gated LeNet- 300- 100 for D E {2, (2,3,4, 3, (left to right). As predicted by our theory, the losses converge exponentially, with the rate increasing in and decreasing D.
This result further shows that for À = 0, a (t) conserved quantity [32, 74], i.c., imbalances decay with a rate. The difference of losses is determined by the difference of regularizers, termed misalignment M(w, D), and thus depends on the overall degree of balancedness.
L(w, r) Lw (w ) AM r = A(R(w, r - R (w
))))
(15)
| - || 1 1122/D)
2
(16)
Using this, the previous result can extended to show [(w, I) - Lw(w 7001 +
Lemma 3 (Convergence of D D-gated loss to 22,//D regularized loss under GF), Assume that the model parameters of Eq. (4) with effective weight w(t) w(t) 7 (t) in Eq. (3) evolve with time t according to the gradient flow in Eqs. (11) and (12). Then, the D-Gated loss [(w((t), (tt) in Eq. (9) converges the non-smooth 2 regularized loss (w))(t) in Eq. (10) least exponentially fast given an initialization dependenn constant C 2 0:
((w(I),r(I)) - (w()()) S Ce $1
(17)
Intuitively, this because balancedness D-Gating is precisely the condition required for C(w, I) to simplify to Lw (w) (cf. Corollary I), Hence, as the pair-wise imbalances vanish, the balancedness condition becomes increasingly true, and the two losses converge.
4.3.2 Evolution of imbalance for D-Gated models under (S)GD dynamics
For an analysis of the discrete-time evolution of imbalances, the dynamics becomes more convoluted, but we can establish geometric decay up first-order in 17, and find symmetry-induced absorbing SGD states [72] for balanced gating configurations.
Lemma 4 (Imbalance evolution under discrete-time GD). Consider the D-Gated objective Eq. (5). Then, under (SJGD, for any j E [J], (i) the pair-wise imbalances in Eq. (13) evolve as
((t+1) j,d.a' = (1 42p D ja, + i1 j.d.d' 7> 0, d, d' E [D)
(18)
with separate recond-order terms Dj for = and d, d' > 1. For sufficiently small or stationarity, the imbalance I((+1) (1-4An/D I exhibits discrete exponential decay.
j,d,d' (ii) Balancedness is conserved between any two scalar factors d. >> 1, i.e, I(1)
r(() id. = 1440 > f, and (ii), for balanced zero representations (w) ( (t) D-1 j,d,d'
0, holds
(w)) 12 Ov >t.
5 Numerical experiments
In the following, we empirically investigate the learning dynamics of our approach in Section 5.1 and then showcase various applications in Section 5.2 to demonstrate our method's 's modularity.
Learning dynamics and misalignment 5.1.1 Exponential decay of imbalance
5.1
We first validate our theoretical results on learning dynamics and loss convergence of D-Gating from Section 4.3. For this, we apply D-Gating with D E (2,3, 4} to LcNet-300-100 at the neuron level

---

1.00
Grout norms
Misalgnnent
60-L21
0.50
Group
2 Gatin unup Lnto
Crade
0.25
-2
-2 (log10)
(leg10)
A (llg10))
Figure 4: Regularization paths for sparse linear regression task using D-Gating. Left: Test RMSE A. The for 2-Gating and group lasso coincide beyond certain X, but oulperformed D-Gating with Middle: Group sparsity of 2-Gating coincides with group lasso solution. Dashed grey line indicates optimal for all models. Deeper gating yields sparser solutions. Right: Transition of 2 2-Gating to group lasso solution beyond certain coincides with zero imbalance attained after training. Direct optimization with GD yields notably different regularization paths far from the global minima. The dashed black line indicates misalignment the end of traininig. Means and 95% confidence intervals
simulations shown for the left two plots.
and train the model on the MNIST dataset using SGD. We use grid of À values and measure the loss convergence defined in Eq. (17). Fig. 3 visualizes the results and confirms our theoretical findings on the exponential decay of the loss difference. Appendix E.4 contains further results, e.g., for Adam.
5.1.2 D-Gating and misalignment for group lasso
To further validate the equivalence of optimization problems established in the previous section, we run a sparse linear regression where direct L2, rregularrre optimization more accessible due to the availability of specialized optimization routines. For this, simulate data as described in Appendix C.I with 40 feature groups of 5 features each, of which 7 are informative (with truly non-zero effects). We accelerated proximal gradient descent [50] directly optimize the original L2.1 L2..-penalized linear model and apply our approach for D E {2,3, 3, 4} over the same grid values. In addition, also perform direct GD optimization of the L 1 rregularizd linear model and compare all methods against an oracle (a linear model using only the signal variables). Results in Fig. confirm the established equivalence between the original and D-Gated objective for D 2, but also demonstrate the improvement in the performance-spariig tradeoff for D >
5.2 Modularity
Next, demonstrate the flexibility of method. To this end, study various types of structured sparsity problems that arise in neural networks. In these experiments, we focus on demonstrating the broad applicability of our method rather than exhaustive benchmark comparison. Our method
ISOLET
COIL20
ACTIVITY
1.0
0.9
0.8
2-Gating
0.8
0.7
3- Gating
Galing
Test
LassoNet
0.5
0.5
(DNN)
0.6
0.4
0.5
0.0
0.2
0.6
0.8
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Input sparsity FMNIST
MNIST
MADELON
1.0
0.9
0.8
Test accuracy
0.9
0.8
0.7
0.6
0.7
0.6
0.5
0.6
0.4
0.4
0.0
0.2
0.4
0.6
0.8
1.0
0.3
0.0
0.2
0.4
0.6
0.8
1.0
0.4
0.0
0.2
0.4
0.6
0.8
1.0
Input sparsity
Input sparsity
Input sparsity
Figure 5: Comparison of feature selection methods. Means and std. over 5 random initializations reported.

---

supports any form of structured sparsity in neural networks, enabling diverse applications. Selected use cases are shown below. Further results are presented in Appendix E.
5.2.1 Feature selection in non-linear models
We start by investigating input feature selection, i.e., by individually gating the first-layer weights outgoing from each imput feature. We follow the setup of [37] by running the proposed method, LassoNet, as well as HSIC [66] on six diverse datasets as done in [37]. We use the same LeNet-300- 100 architecture as a backbone for LassoNet, HSIC and our D-Gating approach (ef. Appendix C.2).
Fig. 5 depicts the comparison results, showing that 2-Gating is often inferior to LassoNet and HSIC. 3- and 4-Gating, however, dominate all other methods for almost all possible input sparsity configurations and, hence, are the favorable options among these methods for feature selection.
5.2.2 Filter sparsity in convolutional neural networks
In the next experiment, we investigate filter sparsification one of the most prominent applications of structured sparsity in neural networks. As comparison methods, we select three commonly used methods in the filter sparsity literature [20]: Global magnitude pruning. L2,1 L2,1- peenalizziinionn with naive optimization followed by magnitude pruning (MP) [62], and network slimming [40]. A more detailed description can be found in Appendix C.3. We run experiments on CIFAR- 10, CIFAR- 100, and SVHN, using a VGG-16 [51] and ResNet-18 model [19]. D-Gating is implemented by adding gating parameters on the filter level, which, given the size of these models, has negligible parameter overhead (see Table 5). As filter sparsity can be used to construct smaller model, potentially deployable on edge devices or similar, we also measure the theoretical speed-up of the sparsified model using floating-point operations (FLOPs). Similar to previous results, Fig. 6 unveils superior performance D > 2-Gating compared to gating with D 2. However, all gating approaches outperform the filter sparsity baselines despite our approach not requiring post-hoe pruning as the main sparsification mechanism. This is the case both terms of the accuracy-sparsii tradeoff provided by our method well the theoretical speed-up implied (ef. Table and Fig. 11).
VGG-16 (CIFAR-10)
VGG-16
ResNet-18 (CIFAR-100)
0.95
1.0
0.8
Test accuracy
0.90
2-6ating
3-Gating
0.85
4-Gating
Magritude pruning
0.7
0.4
0.80
Network slimming
0.6
0.0
Filter sparsity
Filber sparsity
Filter sparsity
Figure 6: Structured accuracy-sparsity tradeoff's of D-Gated neural networks and comparison methods.
Table I: Largest theoretical speed-up bise
FLOPs) within 5% / 10% of the max. test accuracy.
VGG-16
VGG-16 (SVHN)
ResNet-18 (CIFAR-I0M
Method
(CIFAR- 10)
D-Gating (D 2)
16.13 / 52.70
175.49 / 175.49
5.13 / 14.91
D D-Gating (D= 3)
11.17 / 25.51
266.66/ / 390.03
8.27 / 15.14
D-Gating (D) 4)
18.88 / 48.97
262.05 / 541.65 .665
12.76 / 41.04
L2,1 + Mag. pruning
6.97 / 6.97
18.88 18.88
3.35 / 14.01
Network Slimming
2
7.53 7.53
3.56
5.2.3 Structured sparsity in language modeling
Our next application considers the effect of D D-Gating in an attention-based language model [59]. For this, we use NanoGPT and apply D-Gating to the attention heads of all attention layers. A natural comparison is again the direct optimization of the 12, ipeeallyy i.e., without first transforming the objective into a differentiable one through D-Gating. To highlight the shortcomings this naive approach, also follow the direct optimization with an explicit pruning step. We train NP on
8

---

TinyShakespeare (details in Appendix C.4) and evaluate the model's validation accuracy as well as validation perplexity for different regularization strengths and hence levels of attention head sparsity.
Fig. 7 confirms our hypothesis that direct optimization does not result in structured sparsity. Notably, the min. and max. norms of the attention heads converge for large À under direct 22,1 penalization. In contrast, D-Gating shows the desired effect of increased sparsity for higher regularization and provides a smooth tradeoff between accuracy and sparsity. Even when combining direct optimization of the 22,1 regularized objective with additional post-hoc pruning, and taking, at each pruning ratio, the best performance over a grid of À values, we see that D-Gating achieves much higher head sparsity values before performance degrades significantly.
0.52
Accurassy (NarcGPT)
Perplexity (NanaCPI)
Reqularization path (NansGPT)
Attertion norms (NancGPT)
3.50
Vel perppeettty
0.8
Attentinn Veraatff
0.38
2-Gauing
33
2.55
J-deting
4-Getlog
22
2:Safin
4:Satin
0.56
Piuning
0.0
(2)
0.5
0.
3.0
0.5
3.5 3.0
1044
3--3
10-3
350
10-4
10-3
30d
18-3
sparsiy
Head
Figure 7: Structured carrsty-performance tradcoff's for NanoGPT trained on TinyShakespeare with D-Gated Value matrices in their attention heads and direct L2.1 regularization. Left: Val. accuracy vs. head sparsity. Middle: Val. perplexity vs. A. Right two: Effect of regularization À on head sparsity and cvolution of 2-norms for D E (2.3. 43. Dashed horizontal lines represent the vanilla performance.
5.2.4 Tree sparsity in neural frees
As final application, we investigate the sparsification of neural trees. More specifically, we propose novel modi- fication of Neural Oblivious Decision Ensembles (NODE)
500
0.990
[47]. a neural network-basee decision tree ensemble model, While there are multiple options to apply our approach within this architecture, we demonstrate the efficacy of D-Gating by inducing sparsity on the tree-level, i.e., using the different trees as groups. We test our approach on the Wine data set [7] using a range of X values for single tree-layer as suggested by [47]. The tree layer consists of 500 trees, and the weights corresponding to each tree are gated with D= to induce differentiable L2, group sparsity. This is already lower than the default hyperpa- rameter for the tree count of 2048 reported in [47], but raises the question whether 500 trees are in fact necessary to obtain reported performances. Fig. 8 shows the test root mean squared error (RMSE) and the number of active trees as functions of X. While the full non-sparse model reaches baseline RMSE values, we observe that it is possible to achieve a very similar performance by using less than 5 trees, e.g., for À = 3 10-3, This suggests that, at least for the Wine data set, a much simpler and notably less expensive configuration is sufficient.
Numbar trues
400
0.85
Test RAAE
300
Tree count
0.80
200
300
0.75
0.70
10
10-0
10a2
10a
(leg10)
Figure 8: Regularization paths for NODE on the Wine dataset with 2- 2-Gating on tree level. Means and standard deviations over four data splits are shown.
5.2.5 Further experiments and ablation studies
Additional experimental results are provided in Appendix F. Appendices E.5 and F.6 demonstrate the effectiveness of D-Gating beyond the model classes studied above. First, D-Gating is implemented for modal subnetwork sclection in late-fusion architectures, where it consistently succeeds in removing irrelevant data modalities and retaining only informative information.
Next, Appendix E.6 studies a variant of Neural Additive Models (NAMs) [1] with differentiable shape function sparsity, termed D-SNAMs. NAMs combine the inherent interpretabilityy of additive models with the expressivity of neural networks by processing each input independently through its own shape function subnetwork before summing the outputs. Here, D-Gating applied to the first-layer weights of each feature-specific subnetwork to enforce shape function sparsity, effectively removing uninformative inputs while maintaining the flexibility to model non-linear effects of informative features. Our differentiable approach outperforms competing methods in terms of

---

predictive performance, such sparse NAMs (SNAMs) [65], which are based on non-differentiable (group lasso) penalties, and does not require post-hoc pruning, In particular, we observe for D > i.e. non-convex induced regularization, that D-Gating produces increasingly sparse solutions while maintaining low generalization error, explainable by the the more aggressive sparsification capabilities of non-convex over convex sparsity penalties [22]. These experiments substantiate D-Gating promising approach for subnetwork sparsification, which amenable to differentiable optimization, whether in multi-modal settings for attaining shape function sparsity in neural additive modeling. Finally, Appendix E.2 includes further information and experiments on the negligible parameter, runtime, and memory overheads incurred by overparameteriiztion using D D-Gating, while Appendix E.3 contains ablation studies on performance and numerical stability with respect to the gating depth D, supporting the recommendation that D E {3, 4] typically yields the best tradeoff.
Discussion
In this paper, we introduce D-Gating, a differentiable structured sparsity method compatible with SGD and applicable arbitrary differentiable architectures, addressing limitations of non-differentiabl penalties in deep learning. We thereby positively answer our initial research question on whether it is possible to design modular structured sparsity routine, integrable into any architecture, amenable to SGD, with theoretical sparsity guarantees and little practical overhead.
Limitations and future work Due to the flexibility of D-Gating, our approach can provide structured sparsity penalties for arbitrary grouping structures. We systematically demonstrate this flexibility through a diverse set of applications in Section 5.2 and Appendix E. While our theoretical results guarantee equivalence to the original sparse but non-smooth optimization problem, future work could further explore the benefits of this formulation assess its performance when combined with sophisticated pruning and retraining schedules. Secondly, although the gradient flow limit admits clear analysis, it remains open question how discrete-time SGD with large learning rates scheduling impacts the learning dynamics. Finally, integrating D-Gating into the complex training pipelines of modern large-scale scale foundation models, where sparse training from scratch is often impractical, presents another promising direction.
Acknowledgments and Disclosure of Funding
We thank the four anonymous reviewers and the area chair for providing valuable feedback.
DR's and CK's research funded by the Deutsche Forschungsgemeinschatt (DFG, German Research Foundation) -44882377.
References
[1] Rishabh Agarwal, Levi Melnick, Nicholas Frosst, Xuezhou Zhang, Ben Lengerich, Rich Caruana, and Geoffrey E Hinton. Neural additive models: Interpretable machine learning with neural nets. Advances in neural information processing systems, 34:4699-4711 2021.
[2] Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra, Jorge Luis Reyes-Ortiz, et al. A public domain dataset for human activity recognition using smartphones. In Esann, volume 3, pages 3-4, 2013.
[3] Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state of neural network pruning? Proceedings of machine learning and systems, 2:129-146, 2020.
[4] Kevin Bui, Fredrick Park, Shuai Zhang, Yingyong Qi, and Jack Xin. Improving network slimming with nonconvex regularization. IEEE Access, 9:1 1:1122-111534,, 2021.
[5] Kevin Bui, Fredrick Park, Shuai Zhang, Yingyong Qi, and Jack Xin. Structured sparsity of convolutional neural networks via nonconvex sparse group regularization. Frontiers applied mathematics and statistics, 6:529564, 2021.
10

---

[6] Hongrong Cheng, Miao Zhang, and Javen Qinfeng Shi. A survey on deep neural network pruning: Taxonomy, comparison, analysis, and recommendations. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2024.
[7] P. Cortez, A. Cerdeira, F. Almeida, T. Matos, and J. Reis. Wine Quality. UCI Machine Learning Repository, 2009, DOI: http://doo.org/11.24442//////
[8] Tristan Deleu and Yoshua Bengio. Structured sparsity inducing adaptive optimizers for deep learning. arXiv preprint arXiv:2 102.0999,,,,,99,,, 2021.
[9] Mark Fanty and Ronald Cole. Spoken letter recognition. Advances in neural information processing systems, 3, 1990.
[10] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019.
[11] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Pruning neural networks initialization: Why are we missing the mark? In International Conference on Learning Representations, 2020.
[12] Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pages 10323-10337. PMLR, 2023.
[13] Konrad Gadzicki, Razieh Khamsehashari, and Christoph Zetzsche. Early vs late fusion in multimodal convolutional neural networks. In 2020 IEFE 23rd international conference on information fusion (FUSION), pages 1-6. IEEE, 2020.
[14] Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv: 0957444 2019.
[15] Yves Grandvalet. Least absolute shrinkage is equivalent quadratic penalization. ICANN 98: Proceedings of the 8th International Conference on Artificial Neural Networks, Skövde, Sweden, 2-4 September 1998 8, pages 201-206. Springer, 1998.
[16] Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nati Srebro. Implicit bias of gradient descent on linear convolutional networks. Advances in Neural Information Processing Systems, 31, 2018.
[17] Isabelle Guyon. Design of experiments of the nips 2003 variable selection benchmark. NIPS 2003 workshop feature extraction and feature selection, volume 253, page 40, 2003.
[18] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
[19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[20] Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. Journal of Machine Learning Research, 22(241):1-124, 2021.
[21] Peter D Hoff. Lasso, fractional norm and structured sparse estimation using a hadamard product parametrization. Computational Statistics & Data Analysis, 1 115:1864 186-198, 2017.
[22] Yaohua Hu, Chong Li, Kaiwen Meng, Jing Qin, and Xiaoqi Yang. Group sparse optimization via Epig regularization. The Journal of Machine Learning Research, 18(1):90-1011, 2017.
[23] Zehao Huang and Naiyan Wang. Data-driven sparse structure selection for deep neural networks. In Proceedings of the European conference on computer vision (ECCV), pages 304-320, 2018.
[24] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pages 448-456. pmlr, 2015,

---

[25] Tom Jacobs and Rebekka Burkholz. Mask in the mirror: Implicit sparsification. In International Conference on Machine Learning. PMLR, 2025.
[26] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 2014.
[27] Seijin Kobayashi, Yassir Akram, and Johannes Von Oswald. Weight decay induces low-rank attention layers. Advances Neural Information Processing Systems, 177:448-45116 2024.
[28] Chris Kolb, Bernd Bischl, and David Rügamer. Differentiable attention sparsity via structured d-gating. In ICLR Workshop on Sparsity in LLMs (SLLM): Deep Dive into Mixture of Experts, Quantization, Hardware, and Inference, 2025.
[29] Chris Kolb, Tobias Weber, Bernd Bischl, and David Rügamer. Deep weight factorization: Sparse learning through the lens of artificial symmetries. In The Thirteenth International Conference on Learning Representations, 2025.
[30] Chris Kolb, Christian I. Müller, Bernd Bischl, and David Rügamer. Smoothing the edges: a general framework for smooth optimization in sparse regularization using hadamard over- parametrization. To appear in: Machine Learning, 2026.
[3]] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[32] Daniel Kunin, Javier Sagastuy-Brent Surya Ganguli, Daniel LK Yamins, and Hidenori Tanaka, Neural mechanies: Symmetry and broken conservation laws in deep learning dynamics. arXiv preprint arXiv:2012.047228 2020.
[33] Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateek Jain, Sham Kakade, and Ali Farhadi. Soft threshold weight reparameterizztion for learnable sparsity. In International Conference on Machine Learning, pages 5544 5555. PMLR, 2020.
[34] Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989.
[35] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 86(11):2278-2324, 1998.
[36] N Lee, T Ajanthan, and P Torr. Snip; single-shot network pruning based on connection sensitivity. In International Conference on Learning Representations. Open Review, 2019.
[37] Ismael Lemhadri, Feng Ruan, and Rob Tibshirani. Lassonet: Neural networks with feature sparsity. In International Conference on Artificial Intelligence and Statistics, pages 10-18. PMLR, 2021.
[38] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In International Conference on Learning Representations, 2017.
[39] Jiangyuan Li, Thanh V Nguyen, Chinmay Hegde, and Raymond KW Wong, Implicit regulariza- tion for group sparsity. In The Eleventh International Conference on Learning Representations, 2023.
[40] Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming, In Proceedings of the IEEE international conference on computer vision, pages 2736-2744, 2017.
[41] Miao Lu, Xiaolong Luo, Tianlong Chen, Wuyang Chen, Dong Liu, and Zhangyang Wang. Learning pruning-friendly networks via frank-wolfe: One-shot, any-sparsity, and no retraining. In International Conference on Learning Representations, 2022.
[42] Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner: language models. Advances in neural information processing systems, 36:21702-21720, 2023,
On the structural pruning of large
12

---

[43] Morteza Mardani. Qingyun Sun, David Donoho, Vardan Papyan, Hatef Monajemi, Shreyas Vasanawala, and John Pauly. Neural proximal gradient descent for compressive imaging. Advances in Neural Information Processing Systems, 31, 2018.
[44] Sameer A Nene, Shree K Nayar, Hiroshi Murase, et al. Columbia object image library (coil-20). 1996.
[45] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng. et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, page 4. Granada, 2011.
[46] Alexandra Peste, Eugenia lofinova, Adrian Vladu, and Dan Alistarh. Aclde: Alternating compressed/decompresse training of deep neural networks. Advances in neural information processing systems, 34:8557-8570, 2021.
[47] Sergei Popov, Stanislav Morozov, and Artem Babenko. Neural oblivious decision ensembles for deep learning on tabular data. In International Conference on Learning Representations, 2020.
[48] Pedro Savarese, Hugo Silva, and Michael Maire. Winning the lottery with continuous sparsifi- cation. Advances neural information processing systems, 33:1 1380-11390, 2020.
[49] Simone Scardapane, Danilo Comminiello, Amir Hussain, and Aurelio Uncini Group sparse regularization for deep neural networks. Neurocomputing. 241:81 89, 2017.
[50] Noah Simon. Jerome Friedman, Trevor Hastie, and Robert Tibshirani. A pparse-group lasso. Journal of computational and graphical statistics, 22(2)23112245, 2013.
[51] K Simonyan and Zisserman. Very deep convolutional networks for large-scale image recogni- tion. In 3rd International Conference on Learning Representations (ICLR 2015). Computational and Biological Learning Society, 2015.
[52] Nathan Srebro. Learning with Matrix Factorizations. PhD thesis, Massachusetts Institute of Technology, 2004.
[53] Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations, 2023.
[54] Hidenori Tanaka, Daniel Kunin, Daniel L Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in neural information processing systems, 33:6377-6388) 2020.
[55] Yingjie Tian and Yuqi Zhang. A comprehensive survey regularization strategies in machine learning. Information Fusion, 80: 166. 2022.
[56] Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1):267-288, 1996.
[57] Ryan Tibshirani. Equivalences between sparse models and neural networks. Working Notes, 2021.
[58] Tomas Vaskevicius, Varun Kanade, and Patrick Rebeschini. Implicit regularization for optimal sparse recovery. Advances in Neural Information Processing Systems, 32, 2019.
[59] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.
[60] Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5797-5808, 2019.
[61] Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations, 2020.
13

---

[62] Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks, In Advances Neural Information Processing Systems, volume 29, 2016.
[63] Blake Woodworth, Suriya Gunasekar, Jason D Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametrized models. In Conference on Learning Theory, pages 3635-3677 PMLR, 2020.
[64] Han Xiao, Kashif Rasul, and Roland Vollgraf. mishion-mnist: novel image dataset for benchmarking machine learning algorithms. arXiv preprini arXiv: IrXiv:1708.077 7770007777777 2017.
[65] Shiyun Xu, Zhiqi Bu, Pratik Chaudhari, and Ian J Barnett. Sparse neural additive model: Inter- pretable deep learning with feature selection via group sparsity. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 343-359, Springer, 2023.
[66] Makoto Yamada, Wittawat Jitkrittum, Leonid Sigal, Eric P Xing, and Masashi Sugiyama. High- dimensional feature selection by feature-wise kernelized lasso. Neural computation, 26(1): 185-207, 2014.
[67] Ming Yuan and Yi Lin. Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society Series B: Statistical Methodology, 68(1):49-67, 2006.
[68] Sergey Zagoruyko. 92.45% on cifar- 10 in torch. http:://worcccccboooo/////////// cifar. .tmm, 2015. Torch Blog.
[69] Qiaozhe Zhang, Ruijie Zhang, Jun Sun, and Yingzhuang Liu. How sparse can we prune a deep network: A fundamental limit perspective. Advances in Neural Information Processing Systems, 17:91337-91377 2024.
[70] Peng Zhao, Yun Yang, and Qiao-Chu He. High-dimensional linear regression via implicit regularization. Biometrika, 2022.
[71] Haizhong Zheng, Xiaoyan Bai, Xueshen Liu, Zhuoqing Morley Mao, Beidi Chen, Fan Lai, and Atul Prakash. Learn to be efficient: Build structured sparsity in large language models. Advances in Neural Information Processing Systems, 37:101969-101999 2024.
[72] Liu Ziyin. Symmetry induces structure and constraint of learning. In Forty-first International Conference on Machine Learning. 2023.
[73] Liu Ziyin and Zihao Wang. spred: Solving 11 penalty with sgd. In International Conference on Machine Learning, pages 43407-43422 PMLR, 2023.
[74] Liu Ziyin, Mingze Wang, Hongchao Li, and Lei Wu. Parameter symmetry and noise equilibrium of stochastic gradient descent. Advances in Neural Information Processing Systems, 37:93874- 93906. 2024.
14

---

Appendices
Algorithm
16
B Missing proofs
16
B.1 Proof of Lemma
16
B.2 Proof of Corollary
17
B.3 Proof of Theorem
18
B.4 Proof of Lemma
19
B.5 Proof of Lemma
20
B.6 Proof of Lemma
20
Experimental details
22
C.I Details on group lasso simulation
23
C.2 Details on feature selection experiments
23
C.3 Details filter sparsity experiments
................................
23
C.4 Details on language modeling experiments
24
C.5 Details on Neural Oblivious Decision Ensembles
25
Computational Environment
25
Further experiments and ablation studies
25
E.1
Additional plots for filter sparsity in image classification
25
E,2 Overheads
26
E,3 Depth and instability ablation studies Imbalance decay and loss convergence for SGD and Adam
27
E,4
27
E.5 Subnetwork selection
29
E,6 Differentiable sparse neural additive models
15

---

A Algorithm
In Algorithm 1, provide the algorithm for sparse training using the proposed D D-Gating method. Not that it assumed here that all model parameters w are gated, which inconsequential for the procedure. If D-Gating is applied only model substructures, the remaining weights are initialized and updated as usual, and then copied over to the sparse architecture without modification.
Algorithm 1 D-Gating Network Training
1: Input:
2: Data D = {(x, hi) model architecture ff with weights w E Rº partitioned into J groups {9; 1=
3: Gating depth D
4: Training hyperparameters {T, [B), (1) JE
5: Threshold Etny
6: D-Gating parametrization:
7: Replace weights f by primary weights w (wy)}== and gating factors r (Ttdd))E(V) de 7 Wj w/ 1T Vidd
8: Effective weights
9: Initialize w Standard Standard-Init(w/ f), p 1-111-11 E L ER-ID-1) and 17
10: for 0 T do
Sample mini batch ((0)
D
12: Compute (+) (t) (~))(t) and mini-batch loss
Σ f(;; (x:: X (100001 ||
((w(() ((())=
D
(x.))
13:
Compute gradients (wL(w(() r VrL(w( (())
14:
Update
(1+1) (t)
11 (wl(w(t) r(t), p(+t)) -(t)) " (r r())
B
15: Set 1 t 17 (1+1)
16:
end for
17:
Collapse gates and reduce architecture:
18:
w (T)
())TT)
19:
Zero out removed weight groups: wiy + 0f || Etiny
20: Load sparse weights w into compact architecture f
21: Output: Sparse network parameters w and reduced architecture f
B Missing proofss
B.1 Proof of Lemma
Proof. We argue by contradiction. Suppose that for some group j the gating factors are not balanced; that there exist indices d, d' (D -1] such that, without loss of generality,
Ij.d "2
2
(19)
An analogous argument applies if one considers imbalance between ||ا; 11 and of the scalar factors. We now apply a first-order perturbation argument. Consider an infinitesimal perturbation parameter and define the perturbed factors as
F = Nj.a +) and 3/3,0' -i (1 (1-
(20)
Noting that
(1 5)2 ~ 1 + 28, the product of the perturbed factors satisfies
(1 (1= ~ 1 1-28; 28,
(21)
ig.d Fg.d 7s.,(1 + E) 73.00 (1 =E)
"idd"iid (1 51 s)
Miaaiia (1 22) ~ "j.a"
(22)
16

---

so that the effective parameter w wy [Ta=1 ?g.d remains unchanged to first order. On the other hand, the original L2 penalty for the two factors is given by
R(Tja: 7g.d") - 3..0
(23)
After perturbation, we have
(3r,a) ( 25), (5g,ar)² 15,0 - -25,,
(24)
so that the new penalty becomes
R (iidd (z,dr) ~ 12ja(1+22 33,0 (2))
= (15.0 + +22,0) + 28 (2..0 - (2.00)
(25)
Since T2 < 22 by Eq. (19), it follows that
28 15.d )3.d) < 0.
(26)
This strict decrease in the Le penalty contradicts the stationarity of (w, H) with respect to [(w, r). Therefore, the gating parameters must be balanced at any stationary point, which, together with the invariance of the effective parameter w, implies the result Eq. (7).
B.2 Proof of Corollary 1
Proof. We begin by group-wise separating the smooth surrogate penalty (5):
D-1
1/ + = (10) Σ rha).
(27)
j=1
Next, for each group j E [J], we apply the inequality of arithmetic and geometric means (AM-GM) the D non-negative terms ||w, 112,1,111 2 2.........
15.0-1
D-1 Σ 33
11
1 (11,)
2
(|w))
1 Mla 3D.
(28)
d=1
Since w, = W/ TTo-1 Yi.d, the right-hand side becomes
(110,112 -T-1 12,)) -ITT-= 2a ||,)||||2/DD = =1lo, -T 73.al2 =||w, 12,1 (29)
Summing over all J groups yields
1/ [
Nwwwl1. 2/D Σ 3=1
||
(30)
with equality holding if and only if | w =.. for each j E [J], i.e. exactly the balancedness condition (7). Under that condition, each group penalty attains its minimal value ||w, 112/ subject to the constraint that the effective parameter w, remains unchanged. Finally, gives
substituting back into L(we, D) = Lolw 7 + (I [|]
L(w,T)= = Lo(w) + A
2/D W. 2/D = Lo(w) + A||w
2.2/D 2 = Lw(w)),
(31)
completing the proof.
17

---

B.3
Proof of Theorem 1
Proof. The two objectives are
C) = Lo(w 70) + A D (1wl2 + 1T]
(32)
Lw(w) Lo(w)
Iw 1|w 2/D
(33)
For the first direction, suppose that w is a local minimizer of (w) Then there is é > 0 such that
(w) S (w) V E E)
(34)
where ((w, e) denotes an e-hall around IRP. By the multiplicative and surjective nature of the D-Gating overparameterization we can always choose a balanced representation (ws F) in the preimage of w, i.e.
w
1311 - =
=... 2
yj
(35)
Using Corollary 1, balancedness implies that the D-Gated objective simplifies to L(ws, D) fw (w)).
(36)
Further, by continuity of there is 8 >> 0 such that
(1 F) - (w, f222 < 6 - || 7 wl2 <.
(37)
Hence all (W. E (ww, I), map to some effective weight w = w yo E B( E). Moreover,
the gated objective C(w,I H) can be related to its non non-differentiable counterpart L (w) as [(w, r)
L (w 7 + *MW r), where M(w, I) == D D- ([[w] + T || y 11 2/D measures
the negatiega misalignment of the D-Gating variables, achieving if and only (w, r) is a
balanced representation of the effective weight w w 00. Then, using Eq. (34) and Eq. (36),
we obtain the following chain of inequalities
V(w, r) E ((( F),8) 8) L(w, I) Lw Lw(w 70) A.M... r Lw(w 70)
(38)
20
2 (w(()
L(ws, I')
showing (wo, f') a local minimizer of L(w,, I).
To show the reverse direction, assume (wo I') is a local minimizer of L(w, P). We can apply Lemma to establish the balancedness of the gating parameters (w, r) and further define the effective weight as e 00. Since (w, I') is balanced, MM@,, I') = 0 and thus L(w, I') -w((w, By local minimality of (ws, I'', there exists 8 > 0 such that
C(w,, F' < L(w, I)
(w, E ((( 6).
(39)
Define for each group j E [J] the continuous function ry (w;) = ||w, w' E R2 in neighborhood of w, we can construct corresponding balanced D-Gating parameters as follows:
/1//D on RP: Given any
2/a ( sign(fr.a)) r5 Red #0, 0, (r:(w)),
M22 D-1
ry(w,) ) # r,(w)) ) =
w//
(40)
3 0.
0,
Then w/) 111-111111 1/.5
D-1
2/D for all 1]. Let (w', I') denote the collection of group-wiid wise gating parameters, i.e.,
w/. and by construction, | = F.d = rr((w))² = |
j E [J] and d --11. (w',T") [[w))]= ( By Corollary
obtain L(w', I') - (w'). Continuity of r, and of the map w/ + w/, guarantees the existence of > 0 such that
||w - -wl2 < = I1(w", r') - (w. f')l2 < 6. Therefore, for every w' B),,s), the constructed D-Gating parameters (w', I') satisfy Eq. (39) and thus
(41)
VW E BBW,e) (w') L(w', I') > L( L (W).
It follows that w is local minimizer of Lw (w), completing the proof.
18

---

B.4 Proof of Lemma 2
Proof. We differentiate the pair-wise imbalance defined in Eq. (13) with respect to time and treat the two cases of d = 1 and d * separately. In the following we use 7 E IR to abbreviate I Yj.d-
Case 1: For 1, we have
I (1) 11,, T3.d ())6
Differentiating with respect to time yields
T = 2 (t) Ts (t) 23.0 (t).
(42)
Substituting the dynamies from Eq. (12) (with the appropriate negative signs) gives
Ji I (1) - 2j (o)())/w,, Co D 1w,
d
(43)
2), 1(
( ((
21 ww,Co Co
w/((1
D vo0(1))
Since
7/ Yia (t)
3/ (5.
the terms involving (t)t) Vw, Co cancel, leaving
4A D (1w,, (9.
41
dt (t)
D (t).a((t.
(44)
Case 2: For d + 1, we have
Ii (t) = (),((t) 75.a (1)2
Differentiating with respect to yields
d
di a. (t) = 2 (),(()) (t.a(t) - 2 Ya (t) fa.a (t).
(45)
Substituting the expressions from Eq. (12) for both (z,a(t) and Az.ar (t) and using the identity
7, = (j,((1)
(tid(t))
the cancellation of the terms involving (t) T V wr Lo follows analogously to the first case, yielding
dt I (1)
d
41
D (1)a(1)2= The (1)²)
-D (1)
41
(46)
Thus, in both cases, we have
4A dt I,a,
D I, (t),
so that the solution to the differential equation is
Iz, I. (0)e #
19

---

B.5
Proof of Lemma 3
Proof. For a fixed group j. the squared gating parameters are denoted as
a1 (t) = (+)) ad+1(t) = (s,a(t)², E [D-1].
(47)
the group-level regularizer is Ry(t) == Ed= au(t), and the effective group weight is wt( = wy (t) (t. = (t) 73 (t), so that by construction ||w (t) 2/D (T ad(t)
1/D
Note that by Lemma 2, for each group j E [J], the pair-wise imbalances (as in Eq. (13)) satisfy
Iz (t) = dj. (0)e #
(48)
The overall misalignment in group j. measuring the balancedness of the gating representation, is given by
M,(t) = 1x0) 2 0.
(49)
Let mt(t)= min (t(t) and m (t).). Then, by definition, the maximal (t). Since R,(t) is an arithmetic mean, it satisfies m,(t) I (t) I Mt) and hence is upper bounded by (t). Because the geometric mean is not smaller than its smallest component, it is lower bounded by my((). i.e.,
pair-wise wise imbalance for group [J] is (==x =
1/D
2 (t), so that we have
M,(t)= Rt) 1/ (w)(') S M,(())
mn(t) Imux. (t).
(50)
By Lemma 2, for any pair (d, d') in group j we have
laa(t) - az (t) S laa(0) - add (0) - Bt
(51)
so that Imaxx. (t) S Imax, (0)e #1, Hence, M₁(t) S Imax.; ;(0)e + for every group j. Defining Tmax((0 (0) max,e((1...... Imax. (0), we obtain M2(1) < mmax (0)e -ee--a
D for all j. Summing over
the overall misalignment is
M(t) Σ Mt(t) - J Imax (0)e -
(52)
2.2/D to the D-Gated objective L(w, I), we can express it as Lw (w) plus the misalignment M(w, I) 2 0, i.c.
By adding and subtracting Rw (w): = ||w|
2/D 1
C(w, I) = Lolw 0) + I) + A RR (w) - Rw (w)) = Lw 0 + AM(w. r)
(53)
As the difference between both losses is simply [(w((), I(t)) Lw (w(((()) = AM((), it follows that
[(w(t),r()) (ww(() S XJ (m)x 41 Ce -
(54)
where C is an initializationddeeeneen constant. We conclude that ast + co, ((w()), I(t)) converges least exponentially fast to L (w(t)), with their difference vanishing with rate increasing in À and decreasing in D.
B.6 Proof of Lemma 4
Proof. We first abbreviate the gradient of Lo w.r.t. w, as (t) = Vw, Co(w(t)). For clarity of exposition, w.l.o.g. consider full-batch gradient descent updates, given by
wfit()) = w - (( 9, +
2A
DD
(1))
(6+1)
(t) - n II
(T)T (6)
9, +
2x (t)
D ),,)), (55)
d' sdd
20

---

and pair- wise imbalances I We first inspect the scalar case [((+1) 3,1,d'
1,1,d'
12
(3),d² and 3.d,a (>,.,)) (1g.0) for d, > 1.
and start by expanding the squared norm of w/ (((+1)
( - ( ((1))) (1)
21
)
D
= | 1/ 2 ( S0
10
4d
D
11 + 1/
2A D
" (56)
Similarly, we expand the squared updated scalar
"j
(1+1) and separate the terms by their order in n
(1/,d' ) = (7
-217 2
II
3,d"
(t)
w
(C)
42
d' till
+ 1²
II d'dd
T
+
2X (1)
D )jd'
(57)
Subtracting (57) from (56) to get I((+1) the first-order terms 20( (0) (t)
9j from both
expansions cancel, since (1)) 3,dd"
(t)
-a o Vj and we obtain after re-grouping of terms:
(t)
[(++) = (1
41
D
I(t) d.l,d + +
(58)
where
3,1,d - (10))) (0) +
21
w) 10 - II
(t)T (2) 2A (1) 2
D )/d') )
"a w gj +
D
d!l sfd'
= IVV,,, ((w (0) )0 (aL(w()) )())
(59)
O7
is the second-order residual that reduces to the imbalance of gradients. 2 This further implies that close to stationarity of the D-Gated objective, 3,i,d' also vanishes and the discrete-time dynamics reduce to simple geometric decay.
For the scalar-scalar imbalances Z((++)), we already derived (3j,d' metrically defines (3j (, Subtracting both, the first-order terms cancel again, and we obtain
(1+1))2 in Eq. (57), which also sym-
I(++1) (1 3,d,d' D
4A n)
!0
41 hd.a + (1A(())))) - (A = (1 1)
f + 1 j.d.
(60)
where (A (ala J. w 9j + D /00) (BC(w(U.r()) (,(00)) This shows point i). T (0)
7j,0)
Similar to the vector-scalar case, (1) d'd,d' being a difference of squared partial derivatives implies that the term vanishes at stationarity, resulting in geometric decay.
For point ii), we assume balancedness, i.e., Iiid.d = 0, for any two scalar factors d, d' > 1. The D1 j.d,'' 12 ((4)))) )2) Therefore, we must show that pair-wise balancedness I 0 (with potentially non-zero factors Tj.d: Tj.a') implies A (t+1) j.d.d' for d, d' > 1. Abbreviating (7, (odg(0
imbalance under (S)GD evolves as Z.((t/) (1 4)
(t) (t) = -Ia "j.d": we factor the second-
order term as
A = (4 - = + A
- (.2)).
(61)
Note that this can be further simplified to A ),1.W' ((75 (w (17 )2 (10
"01 1|3 (w (OT 9, (003) +
2xa ((t)
Do b.l.ars showing that we can not fully factor (()) ,, out of (t)
d. Hence I(t)
1.1.d' = 0 does not generally
imply Z 2,
0 for sector-scalar imbalances where d > 1.
21

---

The sum term is ((2 (l1000) + (7; (()) " 10
9; B ( + +2,ar) ). and the difference term
(61) (l (7g old )(o) w (1) T 96 + B (.)) Expanding their product, we obtain:
A jidid'
= (17)10000 + (0) (0
( (w 00, +
4x2 (GH/0) (1g )²)
D
2X
(2i/ + (t)
(.d))) (7 -(1, (1910))(0)) w g
+
D
2x
(1) (1)
(170)0000) + (1 (()) w 90
(62)
D
The last two mixed terms cancel, which can be seen by summing them and factoring:
22
( 9j (0-[(155)) + +300))
(9) ) (nr
D
(2.90) 3/20) ) (1))) - (1)) [())
(0)
(63)
Now, expanding the bracket, all terms but 27d (7g ( (0) 27 y.d (7) o/a)
(t) 2/4
(1)
0 cancel, so that the remaining expression for the second-order residual is
j.daa = (((7)) ((12)))00)) (1,) (17510000)) ((0))) (w) 000h) 2
+ D2 +0(10)
412
(())
(64)
j.d.
Finally, using (2, (/000)1)
(1) -Ia Fza' we can simplify (1) = = (13 (oldd') ((0) =
(t)
(t)
A (1)) j,d,d'
= ((199 (0)) - (10 (00 () (TT )/0)²+ +
4X2
((()
D2
J,d,a'
((19)d)) -(199) Addd (01)) (w) (0) 9 )/0)²++
4X
(1t)
+
D2
j,l,d'
= =I/0 [ (wo 9j ) ((1) O'ddd 100/ +
4X2
(65)
D
This shows for scalar gates Yi Yi,d' with d, d' E [D] all terms in ((t+1) depend multiplicatively I(() on ". and hence _,,d,d = 0 implies I((42) j,d,d (1 1)) + 172 0 0. proving point ii).
For point iii), note that balanced zero representation (w)
0 of all vector-scalar for all d, d' E [D]. Since all terms in A (++1) (++1) ((()) it vanishes. Therefore, I (1 4x 12 0 0 for d, d' f 1. Analogously, A 3. = using the same argument. Hence j.1.d'
and scalar-scalar factor pairs satisfies Z multiplicative in either w or
j.d.d' are
( = 0. Beyond simply
enforcing balancedness between both vector and scalar gating factors, inspecting the updates for (e) ( 0 and 3/dd = 0 in Eq. (55) directly shows th (t-1) 0-1--0 = 0 and 3j.
(t+1) = 0-17-0 = 0.
By induction, the iterates of a balanced zero representation in group j E [J] are thus confined to
w (t))
1 > t under (S)GD dynamics. This completes the proof.
C Experimental details
In all our experiments, we evaluate D-Gating only for D) E (2,3, 4). resulting in induced I2211, 22.2/3- and 12,0055 regularization, following broadly established ranges for T.p.q group regularization [22]. Although an ever deeper D-Gating parametrization better approximates a group Lo penalty, the increase in non-convexity potentially causes numerical instabilities without necessarily improving performance beyond 4-Gating (cf. Appendix E.3). Nevertheless, in the majority of our experiments, there is marked distinction between convex group penalization (D 2) and the non-convex extension (D > 2). Throughout all experiments, w is initialized using a standard initialization scheme as for w, while the gating parameters Tia are initialized with unity, i.e., r(0) 1 see Algorithm 1.
22

---

Table 2: Summary of datasets used in feature selection experiments.
Dataset
Training Samples
Test Samples
Classes
Input Features
ISOLET [9]
6,328
1,559
26
617
COTL.00 [44]
1,152
288
20
400
ACTIVITY [2]
4,252
1,492
561
MNIST [34]
60,000
10,000
10
784
F-MNIST [64]
60,000
10,000
10
784
MADELON [17]
2,080
520
500
C.1 Details group lasso simulation
Simulation details For our group sparse linear regression set-up, we simulate 200 training (and 2000 test) samples with p = 200 features grouped into 40 groups of 5 features each. feature groups contain informative (non-zero) weights and 33 groups contain noise features only. We draw both feature values and the (informative) weights from (0, 1) and likewise add standard Gaussian additive noise to the simulated predictor to obtain the targets.
Training details We train the D-Gated models and direct L2,1 penalization using full-batch gradient descent for 1500 iterations using cosine learning rate schedule with initial learning rate 5 10 and 0.9 momentum. We consider parameter groups with norm smaller than 10-6 to be zero. For the specialized optimization routine to solve the Group Lasso efficiently, we use the accelerated proximal gradient method proposed by [50]. The oracle method shown in Fig. 4 is the efficient ordinary least squares estimator computed only on the truly non-zero coefficients. Within each simulation, all regularized methods are run grid of 30 À values spaced logarithmically between 10-5 and 15. The procedure is repeated over 10 simulations and the average results reported Fig.
C.2 Details on feature selection experiments
fully-connected LeNet-300-100 architecture [35] with two hidden layers (300 and 100 neurons) and ReLU activation functions and Kaiming Normal initialization. To obtain feature selection for non-linear models using our approach, we group the first-layer weights of the network by input feature and apply D-Gating. We train the network on all classification tasks and methods using SGD for 100 epochs with a batch size of 256 and cosine schedule with an initial learning rate of 0.1. All models are initialized using a Kaiming Normal scheme, with the scaling factors for D-Gating being initialized with umity (cf. Algorithm 1). Moreover, we consider parameter groups with 2-norm smaller than float32..m mach.ac..... mach.eps & 1.19 10-7 be Table describes the basic properties of the used datasets. For datasets which do not explicitly provide test set, we randomly assign 20% of the samples to the test set. Note that, to ensure a fair comparison D-Gating, LassoNet [37] is implemented without an additional debiasing step that retrains the model only on the selected features.
Architecture As backbone for all methods (D-Gating, LassoNet, HSIC),
C.3 Details on filter sparsity experiments
Architectures VGG-16 for CIFAR-10 and SVHN consists of 13 convolutional layers and 3 fully- connected layers [51]. The convolutional layers organized into five groups: two layers with 64 filters, two with 128 filters, layers with 256 filters, 3 layers with 512 filters, and another 3 layers with 512 filters. After each group, we apply 2 max-poolle max-pooling operation. All filters 3 3. We add batch normalization immediately before every ReLU activation. To reduce parameters compared the ImageNet version, follow [68] and replace the two dense layers before the output with single fully-connecter layer of 512 neurons.
ResNet- 18 is an 18-layer deep residual network introduced by [19]. For small-imase datasets, we follow, e.g., [54], by replacing the initial 7 x 7 convolution and the following max-poote max-pooling layer The model begins with that convolution, then proceeds through four stages of basic residual blocks, each stage consisting of two blocks, with filter counts of [64, 128, 256, 512]. Finally, global average pooling reduces the feature maps before the fully connected output layer. In all methods we only regularize the convolutional but not the fully-connected layers, either through
with a single
23

---

Table 3: Summary of datasets used in image classification experiments.
Dataset
Training Samples
Test Samples
Classes
Input Features
CIFAR- [31]
50,000
10,000
10 3,072 (32x32x3)
SVHN [45]
73,257
26,032
3,072 (32x32x3)
CIFAR-I00 100 [31]
50,000
10,000
100
3,072 (32x32x3)
D-Gating or by directly regularizing the filter weights using an 22,1 penalty [62, 49, 38] or the batch normalization scaling variables as in network slimming [40].
Data We conduct experiments on three standard image classification benchmark tasks described in Table 3. We use the train/test split provided by the datasets. Additionally, we use standard pre- processing and data augmentation techniques, namely normalizing the input images and employing horizontal flips, width or height shifts up to 12.5%, and up to 15 rotations.
Training Our training hyperparameter settings generally follow broadly established configurations for the respective tasks [51, 19, 68]. Because established learning rate settings were found to perform suboptimally, we scanned the interval [10- 3, 1] using small regularization strength À for each method and dataset separately. Table lists the hyperparameters for the filter sparsity experiments.
Table 4: Training hyperparameters for different architectures and image classification datasets. The learning rates correspond to D-Gating with D = {2, 3, 4} (sct) and the comparison methods (sccond value). The comparison
methods use standard Kaiming initialization.
Architecture
Dataset
Epochs
Barch size
Optim.
Mom.
Init.
LR 10.30040044) 0.T
Schedule
YGG-16
CIFAR tn
SYIN
200
128
SGD
SGD
0.9
0.9
Kaiming Normal
Kaiming Nonnal
(2e:3, 3e-31,
Cosine
Cosime
ResNet-18
CIAA--OG
200
128
SGD
0.9
Kaiming Normal
[0.2.0.3.044)).. 0.2
Cosine
In D-Gating, the sparsity is controlled by the regularization strength To obtain the accuracy- sparsity tradeoffs, we therefore train models for each D E {2, 4] along a grid of À values that spans the whole sparsity range. For the comparison methods, direct optimization of the sparsity-inducing L2,1 and L1 penalties does not result in sparse filters whose norm is below float32. mach. eps R 1.19 x 10-7. To achieve sparsity and construct the tradeoff curves, we also train these models along a grid of values but subsequently post-hoc prune each of these models along sequence pre- defined pruning ratios {0.1, 0.2. .... 0.9, 0.95, 0.98, 0.99}. The tradeoff curves are then constructed by taking the best performance over all values separately at each pruning ratio. This optimally reflects different regularization requirements at different sparsity ranges. In contrast to the original proposals of [62, 38, 40], we implement both direct L2, regularization with post-hoc filter pruning and network slimming (L1 regularization of the batch normalization scaling parameters) as a one-shot procedure without retraining after sparsification to ensure a fair comparison to D-Gating, which can be seen merely another optimization vehicle to optimize non-smooth structured sparsity penalties, making them comparable outside of a sophisticated pruning pipeline. Morcover, like direct optimization of these penalties, D-Gating can be arbitrarily combined with other pruning and retraining schemes.
C.4 Details on language modeling experiments
We choose variant of NanoGPT trained on the character-level TinyShakespeare dataset comprising 5, 458, 199 tokens and a vocabulary size of 91 different symbols. Deviating from the usual performance evaluation metrics for language models, the character-level granularity also allows for reporting the validation accuracy instead of only the perplexity. We set aside 10% of the training data for validation purposes.
Our NanoGPT implementation contains 10.8 mio. parameters and has 6 layers, 12 attention heads per multi-head attention layer, block size of 256, an embedding dimension of 384 and dropout with parameter 0.2 between the layers. We train all models on grid of logarithmically spaced À values between 10-1 and 102 for 3000 iterations using a batch size of 32 and Adam [26] with cosine schedule and initial learning rate 10-3 Since direct optimization of the L2,1 1 penalty does
24

---

V
Scaled
K
Dot- Pot-Produst
K
Det Product
Concal
Linsor
Astendon
Figure Schematic illustration of D- Gating overparameteriiatioon achieve structured head sparsity in multi-head attention layer. Left: original attention layer. Right: D-Gated value matrices sparsify the whole attention head. Red nodes indicate the location of the additional gating parameters.
not yield sparse solutions, defined as having a 2-norm smaller than 10 10-6, we further prune these models using structured global magnitude pruning to sparsity ratios {0.05, 0.2, ..... 0.9, 0.95), and report the best performance over all values for given sparsity ratio. Note that the D-Gated models incorporate
additional pruning.
Fig. 9 illustrates how we implement structured attention head penalization using D-Gating. Note that we only gate the value matrices of each head, which suffices for the whole attention head to be removed when sparsified.
C.5 Details on Neural Oblivious Decision Ensembles
We optimize the squared loss on the Wine quality prediction task (n 4, 898: p = 11) [7] using SGD with Nesterov momentum 0.9 and a cosine decay learning rate schedule with initial learning rate 0.4. The models are trained for 400 epochs without early stopping using batch size of 1024. Fig. 10 describes the application of D-Gating in NODEs for D = 2.
TIM FoATsi
621
Figure 10: Schematic illustration of D-Gating oxerparaameterization achieve structured tree sparsity in NODE model. Left: original NODE structure, Right: NODE with 2-gated tree structures (multiplied by shared factor).
D Computational Environment
All experiments were conducted either single NVIDIA RTX A6000 A4000 GPU with 48GB and 16GB of memory, respectively. Smaller experiments, e.g., for toy models, were carried out on standard CPU workstations. The total single-GPU runtime for all experiments is estimated to be around 500 hours.
E Further experiments and ablation studies E.1
Additional plots for filter sparsity in image classification
Besides the accuracy-sparsity tradeoff, the reduction of FLOPs through is of special interest for structured sparsification methods. Besides Table I, we show the full tradeoff curves for accuracy and theoretical speed-up, measured the ratio of FLOPSs of the original and sparse models, in Fig. 11.
Additionally, Fig. 12 shows the regularization paths for D-Gating with D E (2,3, 4) for all three image classification datasets experimented on. Complementing the conclusions from the tradcoff
25

---

10)
(CIFAR
0.94
0.8
Test accuracy
0.92
2-Galing
0.6
0.90
0.90
2-Gating
Sating
Magnitudc pruning
0.88
0.85
6a
Network slimiing
0.2
15
20
100
200
300
400
50
100
150
200
Theoretical speed-up LONG FLOPs)
Theoretical speed-up
bass FLOPs)
Theoretical speed-up fiporer FLOPs)
Figure 11: Test accuracy vs. theoretical speed-up (measured as the ratio of FLOPs) D-Gated convolutional neural networks and comparison methods.
curves in the main text, both plots corroborate our findings of superior tradeoffss for = {3,4 4} over D 2, as well as increased sparsity at the same regularization parameter X.
VGG-16 (CIFAR-10)
VGG-16 (SVHN)
ResNet-18 (CIFAR-100) OD)
2-Gating
Fiterpol sparsky
0.8
3-Gating
4-Gating
0.6
0.4
0.2
10 10
10
10
A (log10)
(lng10)
(log10)
Figure 12: Effect of regularization À on filter sparsity for D E (2,3,4)
E.2 Overheads
Parameter overhead To quantify the parameter overhead incurred by applying D-Gating in our experiments, we contrast the number of additional parameters with the models' parameters. The results are shown in Table 5, highlighting that larger architectures typically have fewer regularized groups (attention heads, convolutional filters) with large group sizes of thousands of parameters, suggesting that our approach scales better for larger than for small architectures by introducing relatively fewer additional parameters.
Table 5: Parameter count overhead for D-Gating. The linear model is the one from Fig. the input-sparse
LeNet-300- 10000-00,,000 the one from Fig. 5, VGG- 16 from Fig. 6, and NanoGPT from Fig. 7. The overhcad rate in the
worst case (D = 4 and the largest input dimension), is given by Aold params
feew parnns
1.
Model
Vanilla Params
Additional Params
Overhead rate (D Details
Lin. Mod.
200
40-(D (D-1)
6.0 10-1
40 groups of 5 features input_dār dám [400, 784] Sum of conv. filters in 13 layers layers, att. heads
I 100
a 266.000
input_ dim (D-1)
8.8 10-3
VGG-16
ss mio.
4,224 (D-1)
84 x 10
NanoGPT
10.5 mio.
72- (D 1) 2.0 10-6
Runtime and memory overhead Additionally, we measure the wall-clock runtime overhead of D-Gating for D e (2, 3,4, 4} against the vanilla models. Similarly, we record the peak GPU memory utilization during training.
The runtime results are shown in Fig. 13, showing slight to moderate increases between 0% and 30% for smaller batch sizes, with the difference becoming increasingly irrelevant for commonly used larger batch sizes as well as larger architectures.
Similarly. Fig. 14 shows the memory overhead over vanilla models, indicating indiscernible utilization for the vast majority of settings. The behavior of the fully-connected LeNet-300- exhibits high variance and less clear patterns compared to the larger models, potentially due to its small size. Generally, part of the difference runtime and memory overhead likely due to implementation efficiency reasons and not necessarily inherent difference.
26

---

LeNett-300-10 300-100
VGG-16
NonocIT
0.58
Van
0,015
0.030
0.2
0.005
0.1
0.000
125
206
512
1024
0.0
250
914
1024
0.00
15
Batch
size
Batcch size
Batch size
Figure 13: Wall-clock runtime for D E (2, 3,4) 4) and vanilla model. Means and std. over ten runs are shown.
LoNet-300-100
VGG-18
NanoEPT
Shak cro Litoattn 6
2.0
2.0
1.0
0.5
6.0
126
256
512
1024
110
256
512
1024
16
52
Figure 14: Peak memory use for D E (2,3, 4) and vanilla model. Means and std. over ten runs are shown.
E.3 Depth and instability ablation studies
To guide the selection of the gating depth and demonstrate that the full benefits of non-convex usually attained at 2/3 D-Gating [22], we further conduct depth ablation studies for the input-sparse LeNet-300-100 from the experiment in Fig. 5 on ISOLET, extend the results for the filter-sparse VGG-16 CIFAR-10 (Fig. 6), and analyze performance and instability for the group sparse regression task Fig. 4.
L2.9. 9 1, regularization
0.5, corresponding D 3, 4 for
Depth ablation (ISOLET)
Depth ablation (VGG-16 CIFAR10)
Test Test accuracy
0.9
Test accuracy
0.90
0.8
0.85
2-Galing
0.6
0.85
2.40
6:45
0.80
3-Gating
10.02ting
4-Galing
20-Galing
8-Gating
0.4
0.75
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Input sparsity
Filter sparsity
Figure 15: Effect increasing the gating depth beyond for the input-sparse LeNet-300-100 on ISOLET as well as the filter-sparse VGG-16 trained on CIFAR-10.
Fig. 15 shows the results for LeNet-300-100 and VGG-16 for increasing depth up to D = 20 and D respectively. Importantly, we observe no measurable improvement the cerformanceessprrsii tradeoff beyond D 4. while the instability increases slightly.
For the group sparse linear regression task shown in Fig. 16, the left panel illustrates the numerical instability caused by too large D, evidenced by the jagged regularization paths for D > 6. The tradeoff eurves in the right panel reveal that these instabilities degrade the performance at lower sparsity values for large D.
Taken together, the values of E 0.53, i.e.. D E {3,4), recommended in the literature are supported by our ablation studies, where deeper gating approaches show at best equivalent performance while increasing numerical instability.
E.4 Imbalance decay and loss convergence for SGD and Adam
The convergence of the D-Gating objective L(w,, T) to the non-smooth regularized objective L (w). shown in Fig. 3, is directly related to the convergence of the regularizers of both objectives, well
27

---

Regularization path
vs group sparsity
2-5ating
Test RMSE
4-Gating
Test RMSE
6-Giaiin
9-Gating
10.Gating
4
12 22-Gating
0.0
0.2
0.4
0.6
0.8
1.0
A (log10)
Group sparsity
Figure 16: Effect of increasing the gating depth 1 heyond the group-sparse linear regression task. The panel shows the regularization paths, and the right panel performance-sparn sparssit tradeoffs. The setup the same as in Fig. 4.
as the balancedness of the gating representations. To see this, note that
L(w,I) D) - -w((w) = Lolw 70) ) + 2(1101 + |fl|) - Lo(w) A//w ///l2///
2
= =(([)W)1 + |T ||
2/D
=X(R(w, r) Rw((w)
= =)M(wDT),
(66)
Hence, the losses converge if the regularizers converge, which, by the AM-GM incquality, happens and only if the gating representations are balanced. This relationship is verified in Fig. 17, showing the same experiment as Fig. 3, but plotting the convergence of regularizers instead of losses.
2-Gating
3-Gating
4-Gating
Klw. (w.fi-RR.k Rx(W)
10a
Misalignient
10
10-2
12
Epoch
Epoch
Epoch
Figure 17: Convergence of regularizers, i.e., evolution of misalignment, instead of loss convergence for SGD trained on neuron-sparse LeNet-3300166 100. Setting is the same as for Fig.
Another interesting question is to empirically investigate the evolution of misalignment for a common optimizer, for which the gradient flow and descent results Lemma and Lemma 4 do not apply, i.e., Adam [26]. Fig. 18 shows the evolution of misalignment for the same setting the previous Fig. 17, but optimized with Adam instead of SGD. Results show that the D- D-Gating objective also converges to the sparsity-inducing objective Lw (w) faster than SGD overall, albeit not at neat exponential rate. Moreover, the convergence speeds do not seem to depend as strongly on D as for SGD.
2-Gating (Adam)
3-Gating (Adam)
4-Gating (Adam)
Misaiggmnent
X(w. [T--)- K>(w)
10-1
10
10-12
100
Epoch
Epoch
Epoch
Figure 18: Convergence of regularizers with Adam instead of SGD trained on neuron-sparse LeNet-300-100.
Setting the same as for Fig.
28

---

E,5 Subnetwork selection
In an additional application, we investigate whether D-Gating can correctly select whole data modalities in a simple late-fusion multimodal network [13]. In these architectures, the modalities are processed independently in separate subnetworks before their latent representations are fused in the penultimate late-fusion layer. Fig. 19 visualizes such architecture as well as our D-Gating proposal for differentiable selection modalities. the D-Gating variant, the final-layer weights are grouped by modality and subsequently gated, effectively selecting out whole modality subnetworks. To control the contribution the modalities, we simulate a semi-synthetic dataset combining both
Figure 19: Schematic illustration of modality selection in late-fusion multimodal network using D D-Gating overparameterization. Left: original late-fusion architecture. Right: architecture including D-Gating of the late-fusion layer, where the red nodes indicate the location of the gating factors.
tabular and image information additively in predictor. To be precise, we simulate targets as == (1 - ax)
tabular
imuge + E =~ 1), (0,0.5,1 1}.
The image predictor is based on the MNIST dataset of handwritten images and is simply the (demeaned) face value of the digit contained in the image. The tabular predictor is given by s Ehs (xx), It ~ U[-1,1] 1], where the fi(() are fixed smooth non-linear shape functions. Both predictors are constructed be similarly distributed. We simulate observations from the above data Lata-generating process. In the late-fusion architecture, we use a small VGG-style subnetwork with 4 convolutional layers for the MNIST images and shallow subnetwork to model the non-linear effects fx(() of the tabular covariates epochs using SGD with 0.8 momentum, batch size of 32, and learning rate of 5 x 10-3, Fig. 20 shows the results for D E (2,3, 4) and E (0. 0.5, }. As expected, using the one standard error rule to select the regularization strength X, D-Gating can consistently select out the unimportant modalities, with the separation of important and unimportant modalities becoming clearer as D increases. Notably, for equal modality contributions (middle row), neither the validation-optimal Xopt nor dise selects out any of the modalities.
2000 training and test
The models are optimized for 300
E.6 Differentiable sparse neural additive models
In another application, we show that D-Gating can be seamlessly applied to achieve differentiable shape function sparsity in Neural Additive Models (NAMs) [i], which call D-SNAMs. In NAMs, each feature is processed independently in a separate subnetwork to learn a flexible shape function ponrpaametrrillly. Then, the learned functions are combined additively to obtain the final inherently interpretable predictor. Fig. 21 visualizes such an architecture as well as our D-Gating proposal for differentiable shape function selection in NAMs. In the D-Gating variant, all outgoing first-layer weights from each feature are grouped and subsequently gated, effectively selecting out shape functions altogether. A related approach we compare against, Sparse Neural Additive Models (SNAMs), imposes a non-differentiable T.2,1 group lasso penalty on the shape function weights and proceeds by direct optimization using SGID, i.e., its subgradient variant [65]. We further compare against direct optimization using SGD with an L1 or La penalty on the same weights.
We investigate the behavior of D-SNAM, i.e., D-Gating in NAMs, synthetic data simulated as follows. The sparse non-linear additive data-generating process (DGP) for the response given by
sin (w1) + (22) ) + 2 Is |ass Σラ 0 t e~ e~N(0,11 (0,
with five informative features T1, us and additional noise covariates .B. .200 drawn independently from (0 1). Thus, only the first five features contribute non-zero signal, while the
29

---

img (2-Gating)
(3-Gating)
img (4-Gating)
3.0
3.0
3.0
VAL. TLNSE
RMSS
Weightt narms
Vol. RASE
10
Weight names
RASE
10
2.0
10
2.0
2.0
30
Ats
10
1.5
10
10
10
10
10a
10
10
10
10
10
10
30
100
10"
50% tab, 50% Img (2-Gating)
50% tab, 50% img (3-Gtiing)
50% tab, 50% img (4-Gating)
1.8
10a
Wt narms
2.0
10t
Weight narms
100
Vil. RiSSE
10
10
3
1.5
1.5
10
10
10
10a
16
10"
100% tab, 0% img (2-Gating)
100% tab, 0% Img (3-Gating)
100% tab. 0% Img (4-Gating)
1.64
10a
Woight narms
10f
Valght narms
100
Vel RMSS
1.4
1.4
10-2
10
1.2
Vol.
1.2
10
10
30
1.0
Figure 20: Regularization path and weight norms in multi-modal late-fusion network for different modality contributions (rows) and D-Gating depth (2,3, 13 (columns). Norms below 10 are clipped. The solid vertical indicates validation-optimal Aopt and the dashed vertical bar Ai chosen by the Ise rule, i.e. the sparsest model that performs within one standard error of the lowest validation RMSE (10 splits).
Figure 21: Schematic illustration of differentiable shape function selection in Neural Additive Models using D-Gating overparameterization. Left: original NAM architecture. Right: differentiable sparse D-SNAM architecture including D-Gating for each feature, where the red nodes indicate the locations of the gating factors.
remaining 15 are uninformative noise. We draw Notal = 1000 independent samples from the DGP, and perform 5-fold cross-validation (CV) obtain mean performance and sparsity metrics, together with standard errors. The regularization strength varies on a grid between 10 and 10!.
The NAMs are defined to have identical shape function subnetworks with layers of size (100, 100, 64, 1), respectively, ReLU activations the hidden layers, and batch normalization [24] after each hidden layer. Optimization of the squared loss is performed for 1000 epochs using Adam optimizer with the default learning rate functions with weight norm smaller than 10-3 are considered zero.
and batch size 32 without carly stopping. Shape
Fig. 22 shows the results for E {2, 3,4) 4} as well the comparison methods SNAM and direct L1 and L2 regularization. Across settings. D D-Gating achieves differentiable shape function sparsity while SNAM not able shrink the weights sufficiently and demands post-training pruning step. Further, D-SNAM effectively removes irrelevant shape functions while retaining informative feature effects. For non-convex induced regularization using D > 2, the CV-optimal models are much sparser than both D = and the dense SNAM results. Moreover, compared to the direct optimization used the baselines, D-Gating with differentiable group sparsity achieves a markedly lower CV error for D {3, 4) (left plot in Fig. 22).
30

---

RMSE
RMSE sparsity
Sparsity
250
128
Shape fun sparsity
0.
RMSE
RMSE
02
10-1
(lag20)
Shape sparsity
(log10)
2-Gating
3-Gating
4-Gating
Diret
Direct
SNAM
Figure 22: Performance of sparse NAMs with differentiable D-Gating for D E {2, 4}. compared against L1. L2, and 12,1 (SNAM) baselines. Means and standard emros for 5-fold cross-validation reported. The annotated stars the right plot indicate the CV-optimal models each method and their CV error.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, we clearly state our claims in the abstract and contribution subsection. The claims are then further substantiated in the main sections using formal proof' validated using experiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We include specific subsection on the limitations of our proposed method in our discussion section and also discuss limitations of our theoretical analysis and experiments in the main sections.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution
31

---

is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faireess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: We provide complete proofs with the full set of assumptions in Appendix B, with the proven statements containing the full assumptions.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
We provide a detailed description of our experimental set-up for all exper- iments, including hyperparameter choices and their justification in Appendix C. All our experiments are done on well-known and openly accessible benchmark datasets. We also formalize our method in Algorithm 1 for reproducibility.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (c.g., in the case
32

---

of a large language model), releasing of a model checkpoint, or other means that are
appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (c.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: All used datasets are publicly available and the surce code for the experiments and baselines will be submitted in the supplementary materials. We aim to collect the code in a publicly accessible Git repository as soon as possible.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
Souuides/CodeSubiiie
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //ips.cc//pptt httpsscccppbbl////ceeeee wwwp:///gueeeeeeeee (CodeSubmissss wwwisoiiicciiiii for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide detailed description of our experimental setup, used hyperpa- rameters, and their justification Appendix C. Finer details can be found in the full code submitted in the supplementary material.
33

---

Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
We provide error bars and their description (e.g.. standard deviations, standard errors, or confidence intervals) for all experiments where multiple runs are not prohibitively expensive (e.g., NanoGPT).
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative етrог rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We specify our computational environment in Appendix and provide an estimate of the total runtime of the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
34

---

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics hptpr//ooooooooo/o//oooooooooooo
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
35

---

We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We reference all datasets and disclose the version of the used packages in the dependencies of the submitted code.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. wwt.coo//aaaaaa/o com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
36

---

According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our L.M policy hnttps://neurda: wwtp.ccc//oocnoeeecceee http://p....coooeeeeeeee//////// for what should or should not be described.
37


Reviews:
[{"review_id": "jhYjqzsx9L", "reviewer": "Reviewer_YPR6", "summary": "The paper proposed an SGD-compatible and  framework-agnostic approach for group-lasso-like objectives without abandoning standard optimizers. The framework is based on a drop-in re-parameterization that splits every weight group into a vector  and D−1 scalar gates. The claim is training the enlarged parameter set with ordinary weight-decay (an L2 penalty) is proved to be exactly equivalent—at the level of stationary points—to training the original network with the non-smooth group-lasso-like penalty.", "questions": "Does the surrogate–objective equivalence still hold when switching from vanilla SGD with embedded weight-decay to common decoupled optimizers such as AdamW? A formal analysis— or at least an empirical study—of whether D-Gating remains theoretically sound or practically stable under AdamW-style decay is missing.", "limitations": "• Scalable experimental results are required ( ImageNet, mid-size LLMs)\n\n•  The authors are encouraged to discuss (or provide empirical evidence) for stability and compute overhead when D>2.\n\n• All proofs use continuous-time gradient flow. The paper will be strengthen if it extends the theory to discrete SGD (possibly under a bounded-step condition).\n\n• Several more recent pruning baselines are missing, specially for transformers (e.g., LLM-Pruner, SparseGPT).", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n+ The authors prove that replacing each weight group w_j by a depth-D gated factorization converts plain L2 weight-decay into an exact surrogate for the non-differentiable L2,2/D  group norm.\n\n+ Four different forms of structured sparsity (feature selection, CNN filter pruning, Transformer-head pruning, tree selection in NODEs) are studied.\n\n\nWeaknesses:\n-\tEvaluation scope is limited to small and medium-scale benchmarks.\n\n-\tNumerical stability degrades beyond depth 4. Also, deeper gating could yield poorer optima.", "was_revised": true, "final_justification": "The", "rebuttal": "Dear Reviewer YPR6,\n\nWe thank you for your time and for your encouraging comments, as well as for the interesting questions you posed. Below, respond to the raised limitations and respond to the questions point-by-point.\n\n# Questions\n\n### **(Q1)**: Does the equivalence of objectives still hold when $L_2$ regularization is not hard-coded into the objective but a decoupled weight decay, as in AdamW, is applied?\n\nThank you for this question. Our proposed method turns a non-smooth objective with structured sparsity regularization into a fully differentiable surrogate with $L_2$ regularization. Therefore, the explicit $L_2$ penalty is necessary for our main equivalence result in terms of minima. This equivalence is baked into the objective structure and does not rely on a specific optimizer. Consequently, we may attempt to use (S)GD, Adam, or even AdamW, as long as the objective is the gated objective with $L_2$ penalty. \nYour question, if understood correctly, however, is rather if the equivalence holds if we additionally **replace** the $L_2$ penalty in our differentiable $D$-Gating objective by decoupled weight decay (WD) in AdamW, similar to how we can exchange explicit $L_2$ regularization using WD updates in (S)GD. As updates using decoupled WD do not correspond to updates under $L_2$ regularization, the straightforward answer is (likely) no. \nHowever, although no exact equivalence to an $L_{2,2/D}$ regularizer holds, it is plausible that sparsity-promoting effects also emerge for AdamW, but less cleanly. Supporting this, [1] introduce decoupled WD for $\\ell_q$ quasinorms and successfully train sparse DNNs with AdamW. How $L_2$ regularization and decoupled WD interact is a non-trivial question and an active area of research (see, e.g., [2]). In Appendix E, the authors of [3] discuss links between AdamW and $L_2$ regularization under certain simplifications, suggesting qualitatively similar dynamics to SGD/Adam could manifest.\n\n# Limitations:\n\n### **(L1)/(L4)**: no large-scale experiments/SOTA baselines.\n\nWhile we agree that large-scale experiments would strengthen our evaluation, we argue that our contribution has a methodological focus, and the experiments serve to showcase versatility. Given the large decision space {architecture types}x{data modalities}x{sparsity use cases}, we tried to best represent this in our experiments, spanning dense DNNs, CNNs, neural trees, and transformers; using numerical, image, text, and speech data, and successfully demonstrating $D$-Gating for input/filter/tree/attention sparsity. \n\nRegarding baselines, we agree that some are solid but dated methods, particularly for transformers where alternatives like SparseGPT exist, a method that efficiently solves a weight reconstruction problem post-training under sparsity constraints, and is thus fundamentally different from our method. Hence, we chose more directly comparable baselines, e.g., direct optimization of the non-smooth $L_{2,1}$ regularizer, to better illustrate our method’s advantage over the naive formulation, instead of trying to establish SOTA performance for every application.\n\n### **(L2)**: No overhead/stability/depth ablation experiments.\n\nWe agree that those experiments would improve our submission. Thus we included several additional experiments, measuring param./memory overhead, runtime, num. stability and ablations for large $D$.\nWe show that the **param./memory overhead is negligible** for all models, while **runtime increases slightly with $D$ for some models** but becomes irrelevant for larger batches. Large **$D>4$ does not improve performance but can become unstable**. We summarize the new results plots below as a table:\n\n------\n- **Parameter count overhead for $D$-Gating (factor=new params/old params-1)**\n\n| Model | Params| Add. Params | Factor ($D=4$)| Details|\n|------------------|--------------------|---------------------------|------------------|--------|\n| Lin. Mod. | 200 |40⋅(D-1) | $6.0 \\times 10^{-1}$  | 40 groups of 5 features    |\n| LeNet-300-100 | ≈266k | input dim⋅(D-1)  | $8.8 \\times 10^{-3}$  | Input dim $\\in [400, 784]$  |\n| VGG-16 | ≈15mio. | 4,224⋅(D-1) | $8.4 \\times 10^{-4}$   | Sum of conv. filters |\n| NanoGPT | ≈10.8mio.|72⋅(D-1)| $2 \\times 10^{-5}$ | 6 layers, 12 heads |\n------\n- We also measured the **memory overhead** of $D$-Gating, finding that the utilization over all settings are almost **indiscernible from the vanilla memory usage** (see **reply to ``dzXs``** for table). \n------\n- **Wall-clock runtime (training) for neuron, filter, and attention sparsity. Means and std. over ten runs are shown**\n\n| ms/sample| D | BS=128  | BS=256 | BS=512| BS=1024 |\n|----------------|---|----------------|----------------|----------------|----------------|\n|**LeNet-300-100**|1|0.0131 (0.0003)| 0.0072 (0.0002)| 0.0046 (0.0003)| 0.0030 (0.0003)|\n| | 2 | 0.0151 (0.0003)| 0.0082 (0.0002)| 0.0051 (0.0004)| 0.0032 (0.0000)|\n| | 3 | 0.0158 (0.0002)| 0.0093 (0.0008)| 0.0050 (0.0001)| 0.0046 (0.0011)|\n|   | 4 | 0.0170 (0.0015)| 0.0090 (0.0002)| 0.0052 (0.0001)| 0.0041 (0.0009)|\n|**VGG-16** | 1 | 0.3111 (0.0089)| 0.3054 (0.0126)| 0.1587 (0.0062)| 0.0796 (0.0038)|\n| | 2 | 0.3213 (0.0071)| 0.3066 (0.0066)| 0.1576 (0.0039)| 0.0835 (0.0027)|\n| | 3 | 0.3098 (0.0073)| 0.3019 (0.0013)| 0.1542 (0.0035)| 0.0804 (0.0016)|\n| | 4 | 0.3159 (0.0167)| 0.3057 (0.0084)| 0.1573 (0.0041)| 0.0796 (0.0021)|\n\n| ms/sample | D | BS = 16 | BS = 32 | BS = 64 |\n|--------|---|-----------|------|-------|\n|**NanoGPT**| 1 | 0.0508 (0.0012)| 0.0266 (0.0006)| 0.0174 (0.0004)|\n|  | 2 | 0.0603 (0.0011)| 0.0314 (0.0006)| 0.0185 (0.0002)|\n|| 3 | 0.0606 (0.0009)| 0.0314 (0.0007)| 0.0186 (0.0002)|\n| | 4 | 0.0617 (0.0017)| 0.0317 (0.0009)| 0.0187 (0.0002)|\n\n------\n- We further included **ablations** on how the gating depth $D$ influences performance and stability for an input-sparse LeNet-300-100, VGG-16, and a group-sparse linear model (contained in **reply to ``dzXs``** in \"Lin. Mod.\" table). In line with the sparsity regularization literature,  $L_{2,q}$ penalties are usually only helpful up to $q=0.5$ or $q=2/3$, corresponding to $D=3,4$. In our experiments, we observe **no performance gains for large $D$**.\n\n**Ablation on depth $D$ at various sparsity levels s (test acc.)**\n\n| LeNet (ISOLET) | s=40% | s=60% | s=80% | s=90% | s=92% | s=94% | s=96% | s=98% |\n|-------|-------|-------|-------|-------|-------|-------|-------|-------|\n| 2-Gating | 95.00%| 93.66%| 68.53%| 49.81%| 46.07%| 42.32%| 38.23%| 21.04%|\n| 3-Gating | 94.89%|**94.97%**|**94.81%**|**94.81%**|93.95%|92.56%|87.28%|70.57%|\n| 4-Gating |**95.09%**|94.59%|94.27%|93.30%|**93.40%**|**92.75%**|**89.57%**|74.57%|\n| 8-Gating | 94.75%|94.09%|93.42%|92.81%|92.49%|92.32%|87.90%|**76.39%**|\n|10-Gating | 94.54%|93.76%|92.99%|92.76%|92.64%|92.24%|88.76%|73.74%|\n|20-Gating | 94.24%|93.51%|92.78%|92.53%|92.11%|92.19%|86.28%|61.31%|\n\n| VGG16 | s=40% | s=60% | s=80% | s=90% | s=92% | s=94% | s=96% | s=98% |\n|----------|-------|-------|-------|-------|-------|-------|-------|-------|\n| 2-Gating | 93.11%| 92.96%| 90.74%| 88.24%| 86.54%| 84.20%| 78.62%| 47.54%|\n| 3-Gating |**93.58%**|**93.39%**|91.55%|88.14%|87.10%|84.57%|82.04%|73.55%|\n| 4-Gating |93.32%|93.36%|**92.38%**|89.67%|**88.55%**|**86.58%**|**82.55%**|**75.84%**|\n| 8-Gating |92.62%|92.78%|91.25%|**89.75%**|88.53%|86.11%|81.36%|67.62%|\n------\n- Due to character restrictions, further depth ablations and **instability results** for the linear model are contained in the **reply to ``dzXs``** (\"Lin. Mod.\" tables). We find that **large $D$ exhibits increased instability but no performance benefit**.\n------\n### **(L3)**: Proofs use only continuous-time gradient flow. The paper will be strengthen if it extends the theory to discrete SGD.\n\nWe agree that discrete-time results would strengthen our analysis beyond the existing (S)GD result in Lemma 4. We have revised the Lemma to derive the discrete-time imbalance dynamics and **prove exponential decay for sufficiently small steps**:\n\n> Consider the $D$-Gated objective Eq. (5). Then, under (S)GD with learning rate $\\eta>0$, (i) the pair-wise imbalances in Eq. (13) evolve as $\\mathcal{I}\\_{j,d,d'}^{(t+1)} = \\big(1 - \\frac{4\\lambda}{D} \\eta\\big) \\mathcal{I}\\_{j,d,d'}^{(t)} + \\eta^2 \\Delta\\_{j,d,d'}^{(t)}, \\\\:d \\neq d', d,d' \\in [D]$, with different second-order terms $\\Delta\\_{j,d,d'}$ for $\\mathcal{I}\\_{j,1,d'}$ and $\\mathcal{I}\\_{j,d,d'}$ with $d,d'>1$. In particular, for sufficiently small $\\eta$, $\\mathcal{I}_{j,d,d'}^{(t+1)} \\approx (1-4 \\lambda \\eta/D) \\cdot  \\mathcal{I}\\_{j,d,d'}^{(t)}$ exhibits discrete exponential decay. Moreover, (ii) for balanced zero gating parameters, $(\\boldsymbol{\\omega}_j^{(t)},\\{\\gamma\\_{j,d}^{(t)}\\}\\_{d=1}^{D-1})=\\boldsymbol{0}$, the future iterates are confined to $\\mathbf{w}_j^{(t')}=\\boldsymbol{0} \\, \\forall t'>t$, and (iii), balancedness, i.e.,  $\\mathcal{I}\\_{j,d,d'}^{(t)}=0$, is conserved $\\forall t'>t$ between any two scalar gates $\\gamma\\_{j,d}^{(t)},\\gamma\\_{j,d'}^{(t)}$.\n\n- Besides, we also include an experiment using Adam instead of SGD in Fig. 3, showing imbalance also decays to zero (even faster than SGD), although the decay is not exponential. This suggests that **Adam also leads to balancedness** and a sparsity-inducing objective. The feasibility of sparse DNN training using Adam is also demonstrated in our language experiment.\n\n------\n\nWe hope this adequately answers your questions and that the additional new results on the computational overhead, instability, and performance for larger depths, as well as our new theoretical discrete-time analysis of the imbalance evolution, further strengthen our submission in your eyes.\n\nReferences:\n\n[1]  Outmezguine, N.J., and Noam L.. \"Decoupled Weight Decay for Any p Norm.\" arXiv:2404.10824 (2024).\n\n[2] Xie, S., and Zhiyuan L.. \"Implicit Bias of AdamW: L∞-Norm Constrained Optimization.\" 41st ICML.\n\n[3] Kobayashi, S., Yassir A., and Johannes v.O.. \"Weight decay induces low-rank attention layers.\" NeurIPS 37 (2024)."}, {"review_id": "pDdFRta7nP", "reviewer": "Reviewer_dzXs", "summary": "The authors propose an approach to induce structured sparsity during training through D gating without requiring additional passes for pruning. The approach is compatible with SGD optimization and shows that the method is appliable on different taks including attention based language model, where sparsification is more challenging.", "questions": "So far i understood the method yields to overparametrization. Deeper gating based on the figures 5,6 delivers the best sparsity–accuracy trade-off, yet adds parameters and potentially slows convergence. Can you quantify the memory/computation overhead of increasing *D* and outline a guideline for choosing the gate depth in practice?", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "### Strengths\n1. **Zero-overhead, SGD-compatible formulation**  \n   D-Gating is implemented as a smooth regulariser; it requires no custom solvers, proximal steps, or pruning–retrain cycle, so it drops into vanilla SGD/AdamW pipelines with virtually no engineering overhead.\n\n2. **Task-agnostic applicability**  \n   The paper demonstrates the same gating mechanism on image classification (VGG-16, ResNet-18) *and* language modelling (NanoGPT), suggesting that the approach generalises across modalities.\n\n3. **Superior extreme-sparsity performance**  \n   With deeper gating ( *D = 4* ), the method outperforms L₂,₁ + magnitude pruning and plain magnitude pruning at the highest sparsity regimes for both CNNs and NanoGPT heads, where baselines usually collapse.\n\n4. **Solid theoretical grounding**  \n   The authors supply clear, formal proofs that the differentiable surrogate shares its local minima with the non-smooth L₂,₂∕ᴰ objective and that unbalanced gate factors vanish exponentially fast.\n\n5. **Meaningful proxy for real acceleration**  \n   Although no wall-clock inference is reported, FLOP estimates are provided to quantify *potential* speed-ups, giving readers a first-order sense of deployability.\n\n4. The paper is very well written and the appendix is structured to support the claims made.\n---\n\n### Weaknesses\n1. **Restricted model and dataset scope**  \n   Vision experiments stop at VGG-16/ResNet-18 on relatively small benchmarks; NLP results are limited to NanoGPT. Evaluating on a transformer vision model (e.g., ViT-S) or larger datasets would better establish robustness.\n\n2. **Baselines are not state of the art**  \n   Magnitude pruning and Network Slimming are solid but dated. More recent structured-sparsity methods (e.g., GMP, SparseGPT, DynaSparse) could provide a tougher comparative bar.\n\n3. **Theoretical speed-up unverified in practice**  \n   All acceleration claims are FLOP-based; without end-to-end latency or energy numbers on real hardware, the practical benefit remains speculative.\n\n4. **No ablation on gate depth vs. overhead**  \n   Deeper gating (*D ≥ 3*) yields higher sparsity, but the paper does not quantify the additional parameter/memory cost or convergence stability trade-off.", "was_revised": true, "final_justification": "Additional experiments showed that D gating add minimal to no overhead, even for the recommended values of 3 and 4", "rebuttal": "Dear Reviewer dzXs,\n\nThank you for your encouraging comments and the time and effort to evaluate our submission. In the following, we first respond to the question and then address the weaknesses.\n\n# Questions\n\n### **(Q1)**: Can you quantify the memory/computation overhead of increasing D?\n\nWe agree that those experiments would help in evaluating practical implementation considerations. To this end, we have included several additional experiments on **a)** the wall-clock runtime overhead during training for different models, batch sizes, and $D$, **b)** the GPU memory overhead for the same settings, and **c)** computed the parameter overhead in our models. Generally, the **parameter and memory overhead are essentially negligible** for any batch size, while the **wall-clock runtime increases slightly with $D$ for some models**. However, this difference becomes irrelevant for larger batch sizes.\n\n------\n\n**Wallclock runtime overhead (training) for neuron, filter, and attention sparsity. Means and std. over ten runs are shown.**\n\n| ms/sample| D | BS=128  | BS=256 | BS=512| BS=1024 |\n|----------------|---|----------------|----------------|----------------|----------------|\n|**LeNet-300-100**| 1 | 0.0131 (0.0003)| 0.0072 (0.0002)| 0.0046 (0.0003)| 0.0030 (0.0003)|\n| | 2 | 0.0151 (0.0003)| 0.0082 (0.0002)| 0.0051 (0.0004)| 0.0032 (0.0000)|\n| | 3 | 0.0158 (0.0002)| 0.0093 (0.0008)| 0.0050 (0.0001)| 0.0046 (0.0011)|\n|   | 4 | 0.0170 (0.0015)| 0.0090 (0.0002)| 0.0052 (0.0001)| 0.0041 (0.0009)|\n|**VGG-16**      | 1 | 0.3111 (0.0089)| 0.3054 (0.0126)| 0.1587 (0.0062)| 0.0796 (0.0038)|\n| | 2 | 0.3213 (0.0071)| 0.3066 (0.0066)| 0.1576 (0.0039)| 0.0835 (0.0027)|\n| | 3 | 0.3098 (0.0073)| 0.3019 (0.0013)| 0.1542 (0.0035)| 0.0804 (0.0016)|\n| | 4 | 0.3159 (0.0167)| 0.3057 (0.0084)| 0.1573 (0.0041)| 0.0796 (0.0021)|\n\n| ms/sample | D | BS = 16 | BS = 32 | BS = 64 |\n|-----------|---|----------------|----------------|----------------|\n|**NanoGPT**| 1 | 0.0508 (0.0012)| 0.0266 (0.0006)| 0.0174 (0.0004)|\n|  | 2 | 0.0603 (0.0011)| 0.0314 (0.0006)| 0.0185 (0.0002)|\n|| 3 | 0.0606 (0.0009)| 0.0314 (0.0007)| 0.0186 (0.0002)|\n| | 4 | 0.0617 (0.0017)| 0.0317 (0.0009)| 0.0187 (0.0002)|\n\n------\n\n**Peak memory utilization during training (GB). Means and std. over ten runs are shown.**\n\n|mem. (GB) |D|BS=128 |BS=256 |BS=512 |BS=1024 |\n|---------------|--|----------------|----------------|----------------|----------------|\n|**VGG-16**   |1|1.2449 (0.5116)|1.4715 (0.2340)|1.4967 (0.1974)|1.6416 (0.8027)|\n| |2|1.0844 (0.1329)|1.6644 (0.3408)|1.3656 (0.1662)|1.5073 (0.2974)|\n| |3|0.9857 (0.1211)|1.4344 (0.1882)|1.4172 (0.1496)|1.5120 (0.2128)|\n|  |4|0.9467 (0.1687)|1.3983 (0.1875)|1.4512 (0.2493)|1.3377 (0.1059)|\n\n|mem. (GB) |D|BS=16 |BS=32 |BS=64 |\n|---------|--|---------|----------|--------|\n|**NanoGPT**|1|1.4060 (0.0232)|2.6474 (0.0417)|5.1592 (0.0460)|\n| |2|1.4060 (0.0216)|2.6570 (0.0360)|5.1370 (0.0571)|\n| |3|1.4272 (0.0192)|2.6571 (0.0348)|5.1096 (0.0559)|\n| |4|1.4152 (0.0220)|2.6627 (0.0456)|5.0993 (0.0698)|\n\n------\n\n**Parameter Overhead (factor=new params/old params-1)**\n\n| Model | Params| Add. Params | Factor ($D=4$)| Details|\n|------------------|---------|-------------|------------|---------------------|\n| Lin. Mod. | 200 |40⋅(D-1) | $6.0 \\times 10^{-1}$  | 40 groups of 5 features    |\n| LeNet-300-100 | ≈266k | input dim⋅(D-1)  | $8.8 \\times 10^{-3}$  | Input dim $\\in [400, 784]$  |\n| VGG-16 | ≈15mio. | 4,224⋅(D-1) | $8.4 \\times 10^{-4}$   | Sum of conv. filters |\n| NanoGPT | ≈10.8mio.|72⋅(D-1)| $2 \\times 10^{-5}$ | 6 layers, 12 heads |\n\n------\n\n### **(Q2)**: Can you outline a guideline for choosing the gate depth in practice?\n\nIn the general non-convex regularization literature [1], it is established that $L_{2,0.5}$ or $L_{2,2/3}$ regularization ($D=3,4$) usually yields the best trade-off between performance and increased optimization issues as we approach group $L_0$ regularization. \n\nWe conducted ablation studies on $D$ for group sparse linear regression, an input-sparse LeNet-300-100 trained on ISOLET, and a filter-sparse VGG-16 trained on CIFAR-10. As expected, some settings showed increased **numerical instability for large $D$** (a jagged/“zig-zag” regularization path instead of a smooth one) and all settings confirmed that **$D=4$ essentially reaps the full benefits of non-convex regularization** while retaining stable training. The new plots are summarized as tables below.\n\n------\n\n**Ablation studies for gating depth $D$ at various sparsity levels s (either test acc. or RMSE)**\n\n| LeNet (ISOLET) | s=40% | s=60% | s=80% | s=90% | s=92% | s=94% | s=96% | s=98% |\n|----------|-------|-------|-------|-------|-------|-------|-------|-------|\n| 2-Gating | 95.00%| 93.66%| 68.53%| 49.81%| 46.07%| 42.32%| 38.23%| 21.04%|\n| 3-Gating | 94.89%|**94.97%**|**94.81%**|**94.81%**|93.95%|92.56%|87.28%|70.57%|\n| 4-Gating |**95.09%**|94.59%|94.27%|93.30%|**93.40%**|**92.75%**|**89.57%**|74.57%|\n| 8-Gating | 94.75%|94.09%|93.42%|92.81%|92.49%|92.32%|87.90%|**76.39%**|\n|10-Gating | 94.54%|93.76%|92.99%|92.76%|92.64%|92.24%|88.76%|73.74%|\n|20-Gating | 94.24%|93.51%|92.78%|92.53%|92.11%|92.19%|86.28%|61.31%|\n\n| VGG16 | s=40% | s=60% | s=80% | s=90% | s=92% | s=94% | s=96% | s=98% |\n|----------|-------|-------|-------|-------|-------|-------|-------|-------|\n| 2-Gating | 93.11%| 92.96%| 90.74%| 88.24%| 86.54%| 84.20%| 78.62%| 47.54%|\n| 3-Gating |**93.58%**|**93.39%**|91.55%|88.14%|87.10%|84.57%|82.04%|73.55%|\n| 4-Gating |93.32%|93.36%|**92.38%**|89.67%|**88.55%**|**86.58%**|**82.55%**|**75.84%**|\n| 8-Gating |92.62%|92.78%|91.25%|**89.75%**|88.53%|86.11%|81.36%|67.62%|\n\n| Lin. Mod. | s=40% | s=50% | s=60% | s=70% | s=80% | s=85% | s=92% | s=94% | s=96% |\n|-----------|-------|-------|-------|-------|-------|-------|-------|-------|-------|\n| 2-Gating |**1.184**|**1.190**|1.195|1.224|1.274|1.300|3.134|3.620|3.913|\n| 4-Gating |1.237|1.201|**1.165**|**1.129**|**1.101**|1.101|**1.816**|**2.358**|**2.938**|\n| 6-Gating |1.260|1.211|1.174|1.136|1.105|**1.082**|1.833|2.387|2.940|\n| 9-Gating |4.072|3.468|2.863|2.258|1.653|1.351|2.473|3.123|3.474|\n|10-Gating |4.209|3.791|3.494|3.237|3.036|2.945|2.842|3.235|3.600|\n|12-Gating |4.207|3.629|2.984|2.339|1.695|1.372|2.032|3.008|3.454|\n\n------\n\n**Depth ablation for numerical instability.** \n\nNote that the plot shows the jagged “zig-zag” regularization path for large $D$ much better. The table contains RMSE values, and bold values indicate instability.\n\n| Lin. Mod. | λ=0.001 | λ=0.01 | λ=0.03 | λ=0.07 | λ=0.1 | λ=0.2 | λ=0.3 | λ=0.4 | λ=0.6 | λ=0.8 | λ=1 |\n|----------------|---------|--------|--------|--------|-------|-------|-------|-------|-------|-------|-------|\n| 2-Gating  | 2.920   | 1.845  | 1.383  | 1.206  |1.172|1.198 | 1.312 | 1.462 | 1.798 | 2.153 | 2.510 |\n| 4-Gating | 2.700   | 1.839  | 1.333  | 1.101  |1.074|1.075 | 1.105 | 1.145 | 1.249 | 1.395 | 1.922 |\n| 6-Gating | 2.652   | 1.863  | 1.381  | 1.105  | 1.073  |1.057|1.071 | 1.089 | 1.135 | 1.197 | 1.280 |\n|**9-Gating** | 8.682   | 7.704  | 6.645  | 4.759  | 4.375  |**1.048**|2.819 | 2.824 |**1.086**|**div.**| 2.859 |\n|**10-Gating**| 8.535   | 7.773  | 6.821  | 4.870  | 4.418  | 3.686 | 3.173 |**2.853**|3.499 |**2.840**| 3.510 |\n|**12-Gating**| 2.569   |**1.980**|7.091  | 4.960  | 4.484  | 3.790 |**1.050**|**1.055**|3.496 | 3.500 | 3.505 |\n\n------\n\n# Weaknesses\n\n### **(W1)**: restricted model/data scope\n\nWhile we agree that large-scale LLM or ImageNet experiments would strengthen our empirical evaluation, we argue that given the decision space {architecture types}x{data modalities}x{sparsity use cases} is quite large, we tried to best represent this in our selection of experiments, especially given our contribution has a methodological focus and the experiments serve to showcase its versatility. Our selection spans fully-connected DNNs, CNNs, neural trees, and transformers, using numerical, image, text, and speech data of various sizes, and successfully demonstrates D-Gating for input sparsity, filter sparsity, neural tree sparsity,  and sparse attention heads.\n\n### **(W2)**: baselines are not SOTA\n\nWe agree that MP and NS are solid but dated methods, particularly for transformer-based architectures where alternatives like SparseGPT exist. However, our contribution is mainly methodological and proposes a differentiable formulation of $L_{2,2/D}$ regularization, with the experimental evaluation demonstrating its effectiveness for various architectures and data modalities. SparseGPT, for example, is a method that efficiently solves a weight reconstruction problem post-training under sparsity constraints, and is thus fundamentally different from our sparse training method. Therefore, we chose to select more directly comparable baselines for these experiments, i.e., direct optimization of the non-smooth $L_{2,1}$ regularizer (besides Network Slimming and LassoNet/HSIC for the feature selection experiments), to better illustrate our method’s advantage over the naive formulation, instead of trying to establish SOTA performance for every application.\n\n### **(W3)**: Theo. speed-up unverified in practice\n\nWe again agree with you that these experiments would be interesting. But as a submission that proposes mainly a methodological contribution and empirically validates it on a variety of applications, measuring acceleration other real hardware is an interesting avenue for future research.\n\n### **(W4)**: Missing ablation studies on runtime/memory/parameter overhead and instability)\n\nPlease see our new experiments listed above on all these items.\n\n------\n\nWe hope this adequately answers your questions and that the additional results on the computational/memory/parameter overhead, instability, and performance for larger $D$.\n\nReferences\n\n[1] Hu, Yaohua, et al. \"Group sparse optimization via $\\ell_{p,q}$ regularization.\" Journal of Machine Learning Research 18.30 (2017): 1-52."}, {"review_id": "UEmq3P18Dl", "reviewer": "Reviewer_vx9E", "summary": "This paper proposes a new way of imposing sparsity regularization to neural networks named as D-gating, achieved from combining a model re-parameterization trick with $L_2$ regularization. The authors argue that D-gating is modular and compatible with arbitrary architectures and show its equivalence with a generalized version of group Lasso regularization. Theoretically, the convergence speed along D-gating optimization trajectory is analyzed. Empirically, the authors conduct experiments in various downstream tasks  (feature selection, filter sparsity, language modeling and neural tree sparsity) an demonstrate the modularity and performance superiority of D-gating.", "questions": "1. I am not an expert in this field, but the experimental results seem to only compare D-gating with a few standard sparsity regularization methods such as Group Lasso or network-slimming. Can you include more in-depth comparisons with newer (structured and unstructured) methods such as [1,2]?\n2. Can you provide theoretical guarantees for convergence of SGD to sparse solutions (See the strengths and weaknesses section)?\n\nReferences:\n[1]. Lasby, Mike, et al. \"Dynamic sparse training with structured sparsity.\" arXiv preprint arXiv:2305.02299 (2023).\n[2]. Tai, Kai Sheng, Taipeng Tian, and Ser Nam Lim. \"Spartan: Differentiable sparsity via regularized transportation.\" Advances in Neural Information Processing Systems 35 (2022): 4189-4202.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Quality:\n\nI am not concerned about the soundness of theoretical proofs given their standard derivations. The empirical results also seem fine to me.\n\nClarity:\n\nThe general writing is fine. However, the technical notation system sometimes introduces unnecessary complications. For example, the D-gating framework, which is a direct re-parameterization introducing $D-1$ additional multiplicative parameters, is expressed in a complex notation system without much explanation. I suggest the authors simplify their technical writing and add more explanations to improve readability of the paper. \n\nSignificance:\n\nThe idea of achieving differentiable sparsity regularization via D-gating reparameterization seems interesting to me. The theoretical proofs are standard and derived from routine optimization analysis, but the authors also demonstrate the performance of their method with empirical validation.\n\nNevertheless, I think this paper has the following weaknesses:\n1. While the authors claim that one major advantage of D-gating is its modularity and compatibility with SGD, the theoretical analysis for SGD (lemma 4) relies on a strong assumption: a balanced zero representation can be achieved. The more important part \nquestion in optimization trajectory: how SGD reaches such a zero state, is unguaranteed.\n2. The authors claim the parameter overhead from the gating factors is negligible. However, the additional computational cost can be large with large $D$ or models with a very large number of small groups.\n\nOriginality:\n\nTo the best of my knowledge, the D-gating framework introduced in the paper is new and original,", "was_revised": true, "final_justification": "I have read the author rebuttal and other reviews. Since thorough experiments are provided to address my concern on effectiveness of the method, I have raised my score accordingly.", "rebuttal": "Dear Reviewer vx9E,\n\nThank you for your time and effort to evaluate our submission, as well as your interesting questions. In the following, we first respond to the weaknesses and then address the questions.\n\n# Weaknesses\n\n### **(W1)**: how SGD reaches such a zero (imbalance) state, is unguaranteed\n\nPlease see our response to question (Q2) below. Further, note that all our experiments also support the claim that SGD finds sparse solutions. E.g., in Fig. 4, we show that $D$-Gating finds exactly the same global minima as the Group Lasso for linear models.\n\n### **(W2)**: Additional computational cost can be large\n\nWe agree with you that the parameter count overhead could become considerable for extremely few weights, small groups, and many/all parameters being regularized. But in practice, the **param overhead is negligible**. Even for the simple linear regression with 200 features and 40 groups, $D$-Gating with $D=4$ adds only 120 new parameters to the model. In contrast, for our NanoGPT implementation, we use 6 layers with 12 heads, yielding 72 groups in total. For $D=4$ **this amounts to 216 additional parameters in a model with 10.8 mio. parameters**. To illustrate this better, we have included the following table in the Appendix:\n\n------\n\n**Paramever Overhead (factor=new params/old params-1)**\n\n| Model | Params| Add. Params | Factor ($D=4$)| Details|\n|-----|--------|------|------|-----|\n| Lin. Mod. | 200 |40⋅(D-1) | $6.0 \\times 10^{-1}$  | 40 groups of 5 features    |\n| LeNet-300-100 | ≈266k | input dim⋅(D-1)  | $8.8 \\times 10^{-3}$  | Input dim $\\in [400, 784]$  |\n| VGG-16 | ≈15mio. | 4,224⋅(D-1) | $8.4 \\times 10^{-4}$   | Sum of conv. filters |\n| NanoGPT | ≈10.8mio.|72⋅(D-1)| $2 \\times 10^{-5}$ | 6 layers, 12 heads |\n\n------\n\nIn addition, to verify that the computational overhead is not considerable, we measured the **wall-clock runtime** and **GPU memory overhead** for different models and batch sizes, summarized in the tables below. **Memory overhead is negligible** and **runtime slightly increases in $D$** but only for small batches.\n\n**Peak memory utilization during training (GB). Means and std. over ten runs are shown.**\n\n|mem. (GB) |D|BS=128 |BS=256 |BS=512 |BS=1024 |\n|------|--|------|------|------|--------|\n|**VGG-16**   |1|1.2449 (0.5116)|1.4715 (0.2340)|1.4967 (0.1974)|1.6416 (0.8027)|\n| |2|1.0844 (0.1329)|1.6644 (0.3408)|1.3656 (0.1662)|1.5073 (0.2974)|\n| |3|0.9857 (0.1211)|1.4344 (0.1882)|1.4172 (0.1496)|1.5120 (0.2128)|\n|  |4|0.9467 (0.1687)|1.3983 (0.1875)|1.4512 (0.2493)|1.3377 (0.1059)|\n\n|mem. (GB) |D|BS=16 |BS=32 |BS=64 |\n|-------|--|-------|------|------|\n|**NanoGPT**|1|1.4060 (0.0232)|2.6474 (0.0417)|5.1592 (0.0460)|\n| |2|1.4060 (0.0216)|2.6570 (0.0360)|5.1370 (0.0571)|\n| |3|1.4272 (0.0192)|2.6571 (0.0348)|5.1096 (0.0559)|\n| |4|1.4152 (0.0220)|2.6627 (0.0456)|5.0993 (0.0698)|\n\n------\n\n**Wallclock runtime overhead (training) for neuron, filter, and attention sparsity. Means and std. over ten runs are shown.**\n\n| ms/sample| D | BS=128  | BS=256 | BS=512| BS=1024 |\n|-----|---|------|-----|-----|--------|\n|**LeNet-300-100**| 1 | 0.0131 (0.0003)| 0.0072 (0.0002)| 0.0046 (0.0003)| 0.0030 (0.0003)|\n| | 2 | 0.0151 (0.0003)| 0.0082 (0.0002)| 0.0051 (0.0004)| 0.0032 (0.0000)|\n| | 3 | 0.0158 (0.0002)| 0.0093 (0.0008)| 0.0050 (0.0001)| 0.0046 (0.0011)|\n|   | 4 | 0.0170 (0.0015)| 0.0090 (0.0002)| 0.0052 (0.0001)| 0.0041 (0.0009)|\n|**VGG-16** | 1 | 0.3111 (0.0089)| 0.3054 (0.0126)| 0.1587 (0.0062)| 0.0796 (0.0038)|\n| | 2 | 0.3213 (0.0071)| 0.3066 (0.0066)| 0.1576 (0.0039)| 0.0835 (0.0027)|\n| | 3 | 0.3098 (0.0073)| 0.3019 (0.0013)| 0.1542 (0.0035)| 0.0804 (0.0016)|\n| | 4 | 0.3159 (0.0167)| 0.3057 (0.0084)| 0.1573 (0.0041)| 0.0796 (0.0021)|\n\n| ms/sample | D | BS = 16 | BS = 32 | BS = 64 |\n|-----|---|------|----|----|\n|**NanoGPT**| 1 | 0.0508 (0.0012)| 0.0266 (0.0006)| 0.0174 (0.0004)|\n|  | 2 | 0.0603 (0.0011)| 0.0314 (0.0006)| 0.0185 (0.0002)|\n|| 3 | 0.0606 (0.0009)| 0.0314 (0.0007)| 0.0186 (0.0002)|\n| | 4 | 0.0617 (0.0017)| 0.0317 (0.0009)| 0.0187 (0.0002)|\n\n------\n\n# Questions\n\n### **(Q1)**: no in-depth comparisons with SOTA methods\n\nWhile we agree that a comparison with sophisticated sparsification pipelines would be interesting, our contribution is mainly methodological, proposing a differentiable formulation of $L_{2,2/D}$ regularization, with the experimental evaluation demonstrating its effectiveness for various architectures and data modalities. For example, the method SRigL proposed in your reference [1] is a specialized approach for N:M-type structured sparsity with a constant fan-in constraint and based on pruning and re-growing connections, which is fundamentally different from our differentiable sparsity regularization approach. Therefore, we chose to opt for more directly comparable baselines in our experiments, i.e., direct optimization of the non-smooth $L_{2,1}$ regularizer, Network Slimming, or LassoNet, to better illustrate our method’s advantage over the naive formulation, instead of trying to establish SOTA performance for every application. Moreover, as shown in [2], the comparison set-up is massively complicated for sophisticated pruning and retraining/regrowing pipelines, further corroborating our use of directly related methods, albeit they may not be SOTA.\n\n### **(Q2)**: Can you provide theoretical guarantees for convergence of SGD to sparse solutions?\n\nWe thank you for this interesting question, which highlights a subtle difference between convergence of the objective to a solution and convergence of the objective $\\mathcal{L}(\\boldsymbol{\\omega}, \\boldsymbol{\\Gamma})$ to the non-smooth objective ${\\mathcal{L}}_{\\mathrm{w}}(\\mathrm{w})$. Whether a $D$-Gated network converges to a solution overall (i.e., to solutions with many zero groups) can not be answered simply, as it depends on many factors like the network architecture, data set, and optimization hyperparameters. It is similar to asking whether objectives with explicit Group Lasso regularization always converge, which can not be answered without specifying the concrete learning problem that is being regularized. A formal analysis of how $D$-Gating impacts overall convergence rates is beyond the scope of this paper, evidenced by the lengthy but still restricted derivations in [3], obtaining the Kurdyka-Łojasiewicz exponent for a much simpler (unstructured) overparameterization, from which a local convergence rate can be deducted.\nIn contrast, our theoretical results establish that a) all solutions of the $D$-Gating problem are solutions to the sparse regularized problem, and b) under gradient flow, the surrogate objective exponentially fast becomes the sparse regularized objective, i.e., the optimization of $\\mathcal{L}(\\boldsymbol{\\omega}, \\boldsymbol{\\Gamma})$ turns into $\\mathcal{L}\\_{\\mathbf{w}}(\\mathbf{w})$. \n\n### **Other comments**:\n\n#### **New result for discrete-time SGD dynamics**\nAdditionally, we have now extended our gradient flow results to discrete-time gradient descent, establishing (for sufficiently small step sizes) exponential decay of the imbalance similar to gradient flow, and thus $\\mathcal{L}(\\boldsymbol{\\omega}, \\boldsymbol{\\Gamma}) \\to {\\mathcal{L}}_{\\mathrm{w}}(\\mathrm{w})$. The novel Lemma reads:\n\n> Consider the $D$-Gated objective Eq. (5). Then, under (S)GD with learning rate $\\eta>0$, (i) the pair-wise imbalances in Eq. (13) evolve as $\\mathcal{I}\\_{j,d,d'}^{(t+1)} = \\big(1 - \\frac{4\\lambda}{D} \\eta\\big) \\mathcal{I}\\_{j,d,d'}^{(t)} + \\eta^2 \\Delta\\_{j,d,d'}^{(t)}, \\\\: d \\neq d', d,d' \\in [D]$, with different second-order terms $\\Delta\\_{j,d,d'}$ for $\\mathcal{I}\\_{j,1,d'}$ and $\\mathcal{I}\\_{j,d,d'}$ for $d,d'>1$. In particular, for sufficiently small $\\eta$, $\\mathcal{I}_{j,d,d'}^{(t+1)} \\approx (1-4 \\lambda \\eta/D) \\cdot  \\mathcal{I}\\_{j,d,d'}^{(t)}$ exhibits discrete exponential decay. Moreover, (ii) for balanced zero gating parameters, $(\\boldsymbol{\\omega}_j^{(t)},\\{\\gamma\\_{j,d}^{(t)}\\}\\_{d=1}^{D-1})=\\boldsymbol{0}$, the future iterates are confined to $\\mathbf{w}_j^{(t')}=\\boldsymbol{0} \\, \\forall \\,t'>t$, and (iii), balancedness, i.e.,  $\\mathcal{I}\\_{j,d,d'}^{(t)}=0$, is conserved $\\forall t'>t$ between any two scalar gates $\\gamma\\_{j,d}^{(t)},\\gamma\\_{j,d'}^{(t)}$.\n\n#### **Choice of notation** \nWe agree with you that it is not straightforward to read, but after much deliberation, we chose this notation to emphasize the “gating” operation using the triangle. The key issue is to find a representation for “group-wise element-wise multiplication”, a problem that other authors have also faced. E.g., [4] introduces an additional matrix $D$ that contains $0$s and $1$s to “broadcast” the group factors to the same dimension as the primary weight, yielding the expression $(D \\boldsymbol{\\gamma}^{\\odot} ) \\odot  \\boldsymbol{\\omega}$, which is arguably even more inconvenient, while [5] use the symbol $\\boldsymbol{\\omega }\\odot_{\\mathcal{G}} (\\boldsymbol{\\gamma}^{\\odot})$ for structured element-wise multiplication. But agreeing with your sentiment, we will think about how to best simplify the notation for a revised version, nevertheless.\n\n------\n\nWe thank you again and hope this adequately answers your questions. We also hope that the additional experimental results, as well as our new theoretical result for the discrete-time evolution of imbalances, strengthen our submission in your eyes.\n\nReferences\n\n[1] Lasby, M., et al. \"Dynamic Sparse Training with Structured Sparsity.\" The 12th ICLR.\n\n[2] Wang, H., et al. \"Why is the state of neural network pruning so confusing? on the fairness, comparison setup, and trainability in network pruning.\" arXiv:2301.05219 (2023).\n\n[3] Ouyang, W., et al. \"Kurdyka–Łojasiewicz exponent via Hadamard parametrization.\" SIAM Journal on Optimization 35.1 (2025): 62-91.\n\n[4] Li, Jiangyuan, et al. \"Implicit Regularization for Group Sparsity.\" 11th ICLR.\n\n[5] Poon, C., and Gabriel P. \"Smooth bilevel programming for sparse regularization.\" NeurIPS 34 (2021)"}, {"review_id": "LVXiQi7quq", "reviewer": "Reviewer_4J3D", "summary": "Group sparsity (the removal of entire filters, attention heads, and so on) is a hardware-friendly alternative to unstructured sparsity. This paper proposes a method called D-Gating, which induces group sparsity through a differentiable reparameterization. Instead of applying a non-differentiable L2-over-D group norm penalty directly, the authors introduce a smooth surrogate by multiplying each of each group’s weights by D differentiable (group-specific) gating parameters. This leads to an overparameterized but fully differentiable objective that encourages sparsity in a way equivalent to the original L2-over-D penalty. After training, the gates can be collapsed into the primary parameters, and groups with zeroed gates can be pruned. Theoretical results are developed for SGD, and small-scale experiments show more stable sparsity induction compared to magnitude pruning or direct optimization of the non-differentiable objective.", "questions": "Aside from any clarifications from the criticisms above, I'm just curious about the title: what is the key distinction between \"modular\" and \"versatile\" that is being implied here?", "limitations": "The paper could have benefited from a separate limitations section that really outlined the theoretical assumptions that may be violated in practice, e.g., in the very experiments used in the paper!  Moreover, as noted above, I felt the discussion of computational overheads was inadequate, and it should be clearly quantified, rather than simply saying \"negligible\" or not discussing it.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n\nThe method is simple and easy to implement, even without sample code. It is flexible and can be applied to arbitrary groupings, such as filters, neurons, or attention heads. This flexibility may allow it to be adapted to specific hardware constraints or architectural needs.\n\nAlthough the theoretical results are developed under SGD, the method appears to work with other optimizers in practice, as shown in the NanoGPT experiments using Adam.\n\nFigures 1 and 2 show a clear attempt to build reader intuition around the method and the gating structure.\n\nWeaknesses\n\nThe paper does not clearly quantify the overhead of the method. While it states that the parameter overhead is negligible, this should be more precise. For example, how many additional parameters are introduced in each experiment (calculable as D x J)? More importantly, what is the compute overhead? Each primary parameter appears to require D extra multiplications during forward and backward passes. How does this compare to the total FLOPs per token in standard models like Transformers? The authors could have provided FLOP estimates or at least wall-clock comparisons with and without D-Gating.\n\nThere is a mismatch between the theoretical and practical setups. The theory is developed for SGD, but the LLM experiments use Adam, which has different optimization dynamics. If the key convergence properties rely on SGD-specific behavior, the authors should either justify the use of Adam or empirically validate that the same convergence trends (or group balance statistics) hold under Adam.\n\nThe exposition is weaker than it should be. The key idea is simple, but the paper takes too long to explain it clearly. Even the abstract uses terms like \"overparameterization\" and \"structured sparsity\" without clarifying that the focus is on group sparsity applied to network components like heads or filters.\n\nFigure 2 may not be the clearest way to introduce the method. It shows input-wise grouping, which could mislead readers about the method’s generality (or, for me on a first read, how it actually operates). A different grouping example might have been more effective, or at least a caption that explains the meaning of the colors.\n\nThe paper loosely equates loss convergence with \"balancedness\" in ways that are not always intuitive or validated empirically. For example, Equation 7 defines balancedness as equal norms within each group, but what about balance across groups? Figure 3 claims to show imbalance decay, but it only plots loss convergence, not any norm-based imbalance metric. Since the paper equates balancedness with loss convergence theoretically, it would be helpful to empirically track both rather than rely on the theoretical connection.\n\nThe experiments are on a very small scale. For instance, the NanoGPT experiments use the Shakespeare dataset, which is tiny relative to modern LLMs. This suggests that D-Gating may introduce overheads that would make it impractical at larger scales (see points above). The paper also lacks strong baselines. For example, what happens if we just train a smaller dense model with fewer parameters? Would that reach the same loss with fewer total FLOPs? The benefit of D-Gating is supposed to be dynamic sparsity during training, but this should be framed in terms of loss vs. total FLOPs, as that is the key metric used in the LLM community.  Putting aside whether we can actually achieve high utilization with the FLOPs varying dynamically throughout training is already a simplication.\n\nFinally, the related work section could be strengthened by including a table that highlights how D-Gating compares to prior methods. Columns like \"compatible with SGD\", \"architecture-agnostic\", and \"fully differentiable\" would help readers see the method’s advantages more clearly.", "was_revised": true, "final_justification": "As noted below: I have read the other reviews and rebuttals, and the rebuttal to my own review.  I feel the responses help improve the clarity and quality of the work.  However, I believe we are still a few steps away from this method being used at large scales, e.g., for LLM pre-training.  Not least because the experiments in this paper are at such a small scale.  The authors say that their contribution is methodological, and the results serve only to \"showcase\" or \"demonstrate\" that it works, but as far as I can tell, there's nothing here to necessarily demonstrate that this method will provide consequential advantages at very large scales.", "rebuttal": "Dear Reviewer 4J3D,\n\nThank you for your time and effort to evaluate our submission, as well as your interesting comments and questions. In the following, we first respond to the weaknesses and then address the limitations and questions.\n\n# Weaknesses\n\n### **(W1)**: no overhead quantification\n\nWe agree that these experiments would corroborate the utility of our method in practical settings. To this end, we included several additional experiments on **a)** the wall-clock runtime during training for different models, batch sizes, and $D$, **b)** the GPU memory overhead for the same settings, and **c)** computed the parameter overhead in our models as you suggested. Generally, the **parameter and memory overhead is negligible** for any batch size, while the **wall-clock runtime increases slightly with $D$ for some models**, but this difference becomes irrelevant for larger batch sizes.\n\n------\n\n**Parameter Overhead (factor=new params/old params -1)**\n\n| Model | Params| Add. Params | Factor ($D=4$)| Structure|\n|--------|---------|------------|--------|------|\n| Lin. Mod. |200|40⋅(D-1) | $6.0 \\times 10^{-1}$  | 40 groups of 5 features    |\n| LeNet-300-100| ≈266k |input dim⋅(D-1)  | $8.8 \\times 10^{-3}$  | Input dim $\\in [400, 784]$  |\n| VGG-16| ≈15mio. |4,224⋅(D-1) | $8.4 \\times 10^{-4}$   | Sum of conv. filters |\n| NanoGPT| ≈10.8mio.|72⋅(D-1)| $2 \\times 10^{-5}$ | 6 layers, 12 heads |\n\n------\n\n**Peak memory utilization during training (GB). Means and std. over ten runs are shown.**\n\n|mem. (GB) |D|BS=128 |BS=256 |BS=512 |BS=1024 |\n|---------------|--|----------------|----------------|----------------|----------------|\n|**VGG-16** |1|1.2449 (0.5116)|1.4715 (0.2340)|1.4967 (0.1974)|1.6416 (0.8027)|\n| |2|1.0844 (0.1329)|1.6644 (0.3408)|1.3656 (0.1662)|1.5073 (0.2974)|\n| |3|0.9857 (0.1211)|1.4344 (0.1882)|1.4172 (0.1496)|1.5120 (0.2128)|\n|  |4|0.9467 (0.1687)|1.3983 (0.1875)|1.4512 (0.2493)|1.3377 (0.1059)|\n\n|mem. (GB) |D|BS=16 |BS=32 |BS=64 |\n|---------|--|------------------|------------------|------------------|\n|**NanoGPT**|1|1.4060 (0.0232)|2.6474 (0.0417)|5.1592 (0.0460)|\n| |2|1.4060 (0.0216)|2.6570 (0.0360)|5.1370 (0.0571)|\n| |3|1.4272 (0.0192)|2.6571 (0.0348)|5.1096 (0.0559)|\n| |4|1.4152 (0.0220)|2.6627 (0.0456)|5.0993 (0.0698)|\n\n------\n\n**Wall-clock runtime overhead (training) for neuron, filter, and attention sparsity. Means and std. over ten runs are shown.**\n\n| ms/sample| D | BS=128  | BS=256 | BS=512| BS=1024 |\n|----------------|---|----------------|----------------|----------------|----------------|\n|**LeNet-300-100**| 1| 0.0131 (0.0003)| 0.0072 (0.0002)| 0.0046 (0.0003)| 0.0030 (0.0003)|\n| | 2 | 0.0151 (0.0003)| 0.0082 (0.0002)| 0.0051 (0.0004)| 0.0032 (0.0000)|\n| | 3 | 0.0158 (0.0002)| 0.0093 (0.0008)| 0.0050 (0.0001)| 0.0046 (0.0011)|\n|  | 4 | 0.0170 (0.0015)| 0.0090 (0.0002)| 0.0052 (0.0001)| 0.0041 (0.0009)|\n|**VGG-16** | 1 | 0.3111 (0.0089)| 0.3054 (0.0126)| 0.1587 (0.0062)| 0.0796 (0.0038)|\n| | 2 | 0.3213 (0.0071)| 0.3066 (0.0066)| 0.1576 (0.0039)| 0.0835 (0.0027)|\n| | 3 | 0.3098 (0.0073)| 0.3019 (0.0013)| 0.1542 (0.0035)| 0.0804 (0.0016)|\n| | 4 | 0.3159 (0.0167)| 0.3057 (0.0084)| 0.1573 (0.0041)| 0.0796 (0.0021)|\n\n| ms/sample | D | BS = 16 | BS = 32 | BS = 64 |\n|--------|---|----------------|--------|-------------|\n|**NanoGPT**| 1 | 0.0508 (0.0012)| 0.0266 (0.0006)| 0.0174 (0.0004)|\n| | 2 | 0.0603 (0.0011)| 0.0314 (0.0006)| 0.0185 (0.0002)|\n|| 3 | 0.0606 (0.0009)| 0.0314 (0.0007)| 0.0186 (0.0002)|\n| | 4 | 0.0617 (0.0017)| 0.0317 (0.0009)| 0.0187 (0.0002)|\n\n------\n\n### **(W2)**: mismatch between theoretical and practical set-up\n\nThanks for pointing out this subtle point. Our main result, Theorem 1, establishes that our $D$-Gating objective is equivalent to the non-smooth sparsity-regularized base objective, irrespective of the optimizer. This equivalence in terms of the solution structure permits, in principle, using any optimizer to find those sparse solutions. Our convergence result, Lemma 4, quantifying how fast the $D$-Gated objective aligns with the non-smooth sparse objective, merely provides a clean upper bound for the rate under gradient flow. Hence, there is no mismatch between our theoretical and practical set-up, just that the convergence rate under gradient flow does not hold for, e.g., Adam.\nIntuitively, any common optimizer strives to find a balanced representation, since, at any point, the $D$-Gating parameters can be perturbed toward a more balanced solution, thus reducing the $L_2$ penalty, while leaving the unregularized loss constant. To corroborate this intuition, we repeated the experiment in Fig.3 using Adam, not SGD. We find that the imbalance also decays to zero (even faster than SGD), although the decay is not exponential. This indicates that although the rate may not be as cleanly described as for gradient flow, Adam also leads to balanced gating representations and thus a sparsity-inducing objective.\n\n### **(W3)**: use of words in exposition\n\nWe agree that “structured sparsity” can also encompass, e.g., N:M sparsity, not covered by our group sparse framework, although it is understood as group sparsity in certain communities. The phrase overparameterization was used as our $D$-Gating approach is inherently a (regularized) overparameterization. We will diligently go through our submission and reword it to emphasize that our approach is for group sparsity in a revised version.\n\n### **(W4)**: visualization of our method (Fig.2)\n\nThank you for pointing this out. We also spent quite some time deliberating the best way to visualize our approach in a simple way and decided on the presented input-sparse example with explicitly mentioning that this is only one possible use case in the caption. Note that our submission also includes more visualizations of $D$-Gating architectures in Figures 9, 10, and 13 of our Appendix. Yet, we will reconsider our example choice and try to come up with a better visualization for a revised version.\n\n### **(W5)**: Connection between loss convergence, norm regularizers, and balancedness\n\nFor clarity, we have added a version of Fig. 3 to the Appendix that shows the difference between the $L_{2}$ and the $L_{2,2/D}$ regularizer on the y-axis, showing qualitatively identical results. This is because the loss difference is simply the difference of both norm-based regularizers scaled by $\\lambda$. To see this, $\\mathcal{L}(\\boldsymbol{\\omega}, \\boldsymbol{\\Gamma}) - \\mathcal{L}\\_{\\mathbf{w}}(\\mathbf{w}) = \\mathcal{L}\\_0(\\cdot) -  \\mathcal{L}\\_0(\\cdot) + \\lambda \\big( (||\\boldsymbol{\\omega}|_2^2| + || \\boldsymbol{\\Gamma}||_F^2)/D - ||\\mathbf{w}||\\_{2,2/D}^{2/D} \\big)$. Hence, convergence of losses is virtually identical to convergence of the regularizers, which happens iff all parameters are balanced.\n\n### **(W6)**: The experiments are on a very small scale.[...] This suggests that D-Gating may introduce overheads that would make it impractical at larger scales. The paper also lacks strong baselines.\n\nRegarding the overheads, please see our new experiments described in **(W1)**, showing that they are small/negligible. \n\nRegarding the scale of experiments, while we agree that large-scale LLM experiments would strengthen our evaluation, we argue that our contribution has a methodological focus, and the experiments serve to showcase versatility. Given the decision space {architecture types}x{data modalities}x{sparsity use cases} is quite large, we tried to best represent this in our selection of experiments, spanning fully-connected DNNs, CNNs, neural trees, and transformers, using numerical, image, text, and speech data of various sizes, and successfully demonstrates $D$-Gating for input sparsity, filter sparsity, tree sparsity, and attention sparsity. \n\nRegarding baselines, we agree that they are solid but dated methods, particularly for transformers where alternatives like SparseGPT exist. However, our contribution is mainly methodological, proposing a differentiable formulation of $L_{2,2/D}$ regularization, with the experiments demonstrating its effectiveness for various architectures and data modalities. SparseGPT, for example, is a method that efficiently solves a weight reconstruction problem post-training under sparsity constraints, and is thus fundamentally different from our method. Hence, we selected more directly comparable baselines for our experiments, i.e., direct optimization of the non-smooth $L_{2,1}$ regularizer (besides Network Slimming and LassoNet/HSIC for the feature selection experiments), to better illustrate our method’s advantage over the naive formulation, instead of trying to establish SOTA performance for every application.\nMoreover, we note that most SOTA methods are multi-pass, pruning-and-regrowing, or post-training weight reconstruction strategies that make a fair comparison much harder (see [1]).\n\n# Limitations/Questions\n\n### **(Q1)**: key distinction between \"modular\" and \"versatile\" ?\n\nWe agree it can sound redundant and changed the title to “Differentiable Sparsity via $D$-Gating: Simple and Versatile Structured Penalization”.\n\n### **(L1)**: [...] theoretical assumptions that may be violated in practice, e.g., in the very experiments used in the paper!\n\nWe hope the previous discussion has cleared up this misunderstanding. The equivalence of the $D$-Gating objective holds for any optimizer, just the exponential rate we derived is for gradient flow specifically.\n\n### **(L2)**: Discussion of computational overheads was inadequate, and it should be clearly quantified, rather than simply saying \"negligible\"\n\nPlease see response to **(W1)**.\n\n------\n\nWe thank you again and hope this adequately answers your questions and concerns. We also hope that our additional experimental and theoretical results strengthen our submission in your eyes.\n\nReferences:\n\n[1] Wang, Huan, et al. \"Why is the state of neural network pruning so confusing? on the fairness, comparison setup, and trainability in network pruning.\" arXiv:2301.05219 (2023)."}]

Author rebuttal:

Meta-review: This paper proposes D-Gating, a reparametrization for structured sparsity which keeps differentiability intact while being equivalent to group penalties. Reviewers found the proposed method to be a simple and versatile algorithm with a solid theoretical grounding. Experimental validations covers diverse scenarios, with some limitations in scale and up-to-date baselines. The authors responded effectively with additional experiments, including analysis on computational overhead and stability under deeper gating. Overall, the manuscript is technically strong, novel, and potentially impactful. Thus I recommend acceptance.
Decision: Accept (spotlight)