Sparse Deep Learning: A New Framework Immune to Local Traps and Miscalibration
Authors: Yan Sun
Wenjun Xiong
Faming Liang
Venue: NeurIPS 2021 Poster (2021)
Keywords: Asymptotic Normality
Posterior Consistency
Prior Annealing
Structure Selection
Uncertainty Quantification

Abstract: Deep learning has powered recent successes of artificial intelligence (AI). However, the deep neural network, as the basic model of deep learning,  has suffered from issues such as local traps and miscalibration. In this paper, we provide a new framework for sparse deep learning, which has the above issues addressed in a coherent way. In particular, we lay down a theoretical foundation for sparse deep learning and propose prior annealing algorithms for learning sparse neural networks. The former has successfully tamed the sparse deep neural network into the framework of statistical modeling, enabling prediction uncertainty correctly quantified. The latter can be asymptotically guaranteed to converge to the global optimum, enabling the validity of the down-stream statistical inference. Numerical result indicates the superiority of the proposed method compared to the existing ones. 

Sparse Deep Learning: A New Framework Immune to Local Traps and Miscalibration
Yan Sun Purdue University West Lafayette, IN 47906 sun7480purdue. edu
Wenjun Xiong
Guangxi Normal University & Purdue University West Lafayette, IN 47906
xiong90@purdue.com edu
Faming Liang"
Purdue University
West Lafayette, IN 47906 fml i aaiiinngguuuueee. edu
Abstract
Deep learning has powered recent successes of artificial intelligence (Al). However, the deep neural network, as the basic model of deep learning. has suffered from issues such as local traps and miscalibration. framework for sparse deep learning, which has the above issues addressed in a coherent way. In particular, we lay down a theoretical foundation for sparse deep learning and propose prior annealing algorithms for learning sparse neural networks. The former has successfully tamed the sparse deep neural network into the framework of statistical modeling, enabling prediction uncertainty correctly quantified. The latter can be asymptotically guaranteed to converge to the global optimum, enabling the validity of the down-stream statistical inference. Numerical result indicates the superiority of the proposed method compared to the existing ones.
this paper, we provide a new
Keywords: Asymptotic Normality, Posterior Consistency, Prior Annealing, Struc- ture Selection, Uncertainty Quantification
Introduction
During the past decade, deep neural networks (DNNs) have achieved the state-of-the-art performance many machine learning tasks such as computer vision and natural language processing. However, the DNN suffers from training-prediction dilemma from the perspective of statistical inference: A small DNN model can be well calibrated. but tends to get trapped into a local optimum: on the other hand, over-parameterized DNN model can be easily trained to a global optimun (with zero training loss), but tends to be miscalibrated [14y. In consequence, it is often unclear whether DNN is guaranteed to have desired property after training instead of getting trapped into arbitrarily poor local minimum, or whether its decision/predicctonn reliable. This difficulty makes the trustworthiness of AI highly questionable.
To resolve this difficulty, researchers have attempted from two sides of the grainingppredittin dilemma. Towards understanding the optimization process of the DNN training, line of researches have been done. For example, [13] and (27) studied the training loss surface of over-parameterized DNNs. They showed that for fully connected DNN, almost all local minima are globally optimal, the width of one layer of the DÑN is no smaller than the training sample size and the network
To whom correspondence should be addressed: Faming Liang
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

structure from this layer on is pyramidal. Recently, I 8 35 and 36 explored the convergence theory of the gradient-based algorithms in training over-parameterizan DNNS. They showed that the gradient-based algorithms with random initialization can converge to global minima provided that the width of the DNN polynomial in training sample size.
To improve calibration of the DNN, different methods have been developed, see e.g., Monte Carlo dropout [22 and deep ensemble [18. However, these methods did not provide a rigorous study for the asymptotic distribution of the DNN prediction and thus could not correctly quantify its uncertainty,
Recently, researchers have attempted to address this issue with sparse deep learning. For example, for Bayesian sparse neural networks, [1]] [28] and [1 established the posterior consistency, and [33 further established the Bernstein-von Mises (BvM) theorem for linear and quadratic functionals.
The latter guarantees in theory that the Bayesian credible region has a faithful frequentist coverage.
However, since the theory by [3 does not cover the point evaluation functional, the uncertainty of the DNN prediction still cannot be correctly quantified. Moreover, their theory is developed with the spike-and-slab prior (i.e., each weight or bias of the DNN is subject to a spike-and-slad prior), whose discrete nature makes the resulting posterior distribution extremely hard to simulate. To facilitate computation, [1I employed mixture Gaussian prior. However, due to nonconvexity of the loss function of the DNN, direct MCMC simulation still cannot be guaranteed to converge to the right posterior distribution even with the mixture Gaussian prior.
In this paper, we provide a new framework for sparse deep learning, which successfully resolved the prdiing-prediction dilemma. In particular, we propose two prior annealing algorithms, one from the frequentist perspective and one from the Bayesian perspective, for learning sparse neural networks. The algorithms start with an over- wver-parameterized deep neural network and then have its structure gradually sparsified. We provide theoretical guarantee that the training procedures are immune to local traps, the resulting sparse structures are consistent, and the predicted values are asympiotically normally distributed. The latter enables the prediction uncertainty correctly quantified. Our contribution in this paper is two-fold:
We provide new framework for sparse deep learning, which is immune to local traps and miscalibration.
We lay down the theoretical foundation for how to make statistical inference with sparse deep neural networks.
The remaining part of the paper is organized follows. Section 2 lays down the theoretical foundation for sparse deep learning. Section 3 describes the proposed prior annealing algorithms. Section presents some numerical results. Section concludes the paper.
2 Theoretical Foundation for Sparse Deep Learning
As mentioned previously, sparse deep learning has received much attention as a promising way for addressing the miscalibration issue of the DNN. Theoretically, the approximation power of the Under the Bayesian setting, posterior consistency has been established in [19 28 31.. In particular, the work B1 has achieved important progress toward taming sparse DNNs into the framework of statistical modeling. They provide neural network approximation theory fundamentally different from the existing ones. In the existing theory, no data involved and small network can potentially achieve an arbitrarily small approximation error by allowing connection weights to take values in an unbounded space [24] In contrast, the theory by [31] links the network approximation error, the network size, and the bound of connection weights to the training sample size. They prove that for given training sample size n, sparse DNN of size O(n/ log(n)) ) has been large enough to approximate many types of functions, such affine functions and piccewise smooth functions, arbitrarily well as n 30. Moreover, they prove that the sparse DNN possesses many theoretical guarantees. For example, its structure is more interpretable, from which the relevant variables can be consistently identified for high- -igheeiinnsi nonlinear systems; and its generalization error bound is asymptotically optimal.
sparse DNN has been studied for various classes of functions 30 3
From the perspective of statistical inference, some gaps remain toward taming sparse DNNs into the framework of statistical modeling. This paper bridges the gap by establishing (i) asymptotic normality of the connection weights, and (ii) asymptotic normality of the prediction.

---

2.1 Posterior consistency and structure selection consistency
This subsection provides brief review of the sparse DNN theory developed in [31 and gives the conditions that we will use in the followed theoretical developments. Without loss of generality, we let Dn = (ax*), )ei.... denote a dataset n i.i.d d observations, where x(() ε R and y E R. Consider generalized linear model with the distribution of y given by
fwyl"" (æ)) = exp( (x))))) + (x)) + C(y)).
where 1 (2)) is a nonlinear function of æ and A(-). B()) and C(-) are appropriately defined functions. --1=2//0/00 C(y)
For example, for normal regression, we have A(x") = 14¹ /=/0², B(x")
-y² /20 2 - log(ooo²))/ 1/2, and 0² is constant. We approximate μ² (æ:) using fully connected DNN with H, hidden layers. Let Ln denote the number of hidden units at layer with LH.,
for the output layer and Lo = pn for the input layer. Let w E R X and b" R h {1,2, ..... }n} denote the weights and bias of layer h, and let sh R4xx1 + R denote a coordinate-ve wise and piecewise differentiable activation function of layer h. The DNN forms a nonlinear mapping
133 a0) tf [ +f² [wo + b'] .... b".,
i, where 3 (w, b) - { w/) by (,, fl..... Lnd.J E {l..... Ln-1 denotes the collection of all weights and biases, consisting of Kn = Emt (Ln-1 Ln LA) elements. For convenience, we treat bias as a special connection and call each element in B a connection weight. In order to represent the structure for sparse DNN, we introduce an indicator variable for each connection weight. Let y and 7/0" denote the indicator variables associated with w" and bah
b², respectively. Let y (r, .7k h E (1...., Hn). ). i,k E (1,..., La j E (1...., Ln-1)) }} which specifies the structure of the sparse DNN. With slight abuse of notation, we will also write uBB æ) as u(B, Y) to include the information of the network structure. We assume p²(x) can be well approximated by a parsimonious neural network with relevant variables, and call this parsimonious network as the true DNN model. More precisely, we define the true DNN model
(G', x²)= =
arg min (8.7)E9.. (1(( (x))) (2)))
hy|s
(2)
where 9n == g(Co, C1, Pns HR, L. I2..... I.,)) denotes the space of valid sparse networks satisfying condition A.2 (given below) for the given values of Hn, Pa. and LA 's, and wn some sequence converging to 0 as n + co. For any given DNN (B, 7), the error μ(B, y, x) μ (æ) can be generally decomposed as the network approximation error u18",, x²), network estimation error μ(B, 7. x) 1B "1""""""""""" x). The L2 norm of the former is bounded by wn.
p= (x) and the
and the order of the latter will be given in Lemma 2.1 For the sparse DNN, we make the following assumptions:
A. The input æ is bounded by entry-wisely, i.e. æ E R = [-1, 1| and the density of æ is bounded in its support Ω uniformly with respect to
A.2 The true sparse DNN model satisfies the following conditions:
A.2.1 The network structure satisfies: H, log +r, log L + s,, log Con 1-5 where 0 < E < 1 is a small constant, Tn = 1-* denotes the connectivity of y", I = maxigjsH.-114 Lj denotes the maximum hidden layer width, sn denotes the input dimension of 7
A.2.2 The network weights are polynomially bounded: || loo S En, where E,, = =n for some constant C₁ 0.
A.3 The activation function vb is Lipschitz continuous with Lipschitz constant of 1.
Refer to 31 for explanations and discussions on these assumptions. We let each connection weight and bias be subject to a mixture Gaussian prior, i.e.,
(3) where An E (0, 1) the mixture proportion, oi typically set to a very small number, while of is relatively large.
(O, by A, of + (1- N,, o w X, C. (0, (..) + +(1
3

---

Posterior Consistency Let P and E" denote the respective probability measure and expectation with respect to data D, Let d(pi.P2 P2) denote the Hellinger distance between two densities pi (x. v) and (2 v). Let (AA Pa) be the posterior probability of an event A.
Lemma 2.1. (Teorrm 2.1 of (37)) Suppose Assumptions A. A.1-A.3 hold. If the mixture Gaus- (Ip sian prior (3) satisfies the conditions: An 011 ( for some constant
0, / [Hn log + log L32/2 S FIn S n² for some constant a > 0, and Jo, S (n H,, Hx,,)>² }}. then there exists error se- quence & (= + O(S2) such that lim Cn 0 and lim ne xc, and the posterior
min { V, (n (n0/2/100/
(1)/HH)) Hn)""..
"*]].
distribution satisfies
Pd (idd 4en | 2 2 car? ]
-cach
2e 2e
(4)
FD."" v prdddpa,P. Pao) > 4en Dn] S 4e 2one
for sufficiently large n, where c denotes a constant, Su - Hn n log n. + I' log L + sn hoggp//// n,
Pie denoles the underlying true data distribution, and pa denotes the data distribution reconstructed by the Bayesian DNN based on its posterior samples.
Structure Selection Consistency The DNN is generally nonidentifiable due to the symmetry of network structure. For example, μ(33 7, x) can be invariant if one permutes certain hidden nodes or simultaneously changes the signs or scales of certain weights. As in [11, DNNs by e such that any possible DNN can be represented by one and only one DNN in θ via nodes permutation, sign changes, weight rescaling, etc. Let 3) E e be operator that maps any DNN to θ via appropriate weight transformations. To serve the purpose of structure selection in O, we consider the marginal inclusion posterior probability (MIPP) approach proposed in (0). For each connection, we define its MIPP by = =f En CT)((, ग (7/13, D..) ( for i = 1,2,.... K,. where ( is the indicator of connection i. The MIPP approach is to choose the connections whose MIPPs are greater than a threshold q. i.e., setting 54 = {i :x:q0 4: > ) 2 6 and define Ei(v(y"_s*)) ( D.) Dn) (3) which measures
define set of
G. = 1,2..... } as an estimator of >- O. Let A(e,,) = (33 d(ps. Pyx))
p(en)= maxisisK, JA(e.)o [eff((r7,3
the structure difference between the true and sampled models on A(e..). Then we have:
Lemma 2.2. (Theorem 2.2 of (31) If the conditions of Lemma holld and plEn) + 0 n + do and 6n - 0, then (i) mix, (gg- (())(>r>,-)))) 2/ 0; (ii) (sure screening) P(r, C caa) B
for any pre-specified q (0, 1); (iāi) (consistency) P(y, to.s) 3 1.
Lemma 2.2 implies consistency of variable selection for the true DNN model as defined in (
2.2 Asymptotic Normality of Connection Weights
In this section, we establish the asymptotic normality of the network parameters and predictions. Let (3) = E log(p1( (x, y:)) denote the log-likelihood function, and let r((3) denote the density of the mixture Gaussian prior (3) Let hi io. (3) denote the d-th order partial derivatives (3)
88, 83,, _ Let H, (33 denote the Hessian matrix of (B). Let (B) and (B) denote the (i,j)-th -tth component of (3) and (B), respectively. Let (B) and An, (3) denotes the maximum and minimum eigenvalue of the Hessian matrix Hn (3), respectively. Let BA,, = \ (1")(88)/ ) /,, (3)) and bx.n = =VTn VTnn //Tr//BB.ns where Tn is the connectivity of r For DNN parameterized by B, we define the weight truncation at the true model yr: (g ); = 33 for i E 7 and (G )4 0 otherwise.
For the mixture Gaussian prior (3). let Bs, (3¹) (B |g B < 8n, Vi E >-, 1,, B71 <
20 log(
X,, oo... "..,,..) Vi & Y'} }. We follow the definition of asymptotic normality in 4 and 33
Finn
Definition 2.1. Denote by dg the bounded Lipschitz metric for weak convergence and by on, the
mapping On 3 + vT(g(() 9-). We say that the posterior distribution of the functional g(3) is
asymptotically normal with the center 9r and variance G if d [ Dz tn NO0 G)) 0 in
Pt proprobability as n cc. We will write this more compactly as .[. D d= N(O,
Theorem 2.1 establishes the asymptotic normality of 2(3), where 2(9) denotes a transformation of (3 which invariant with respect to p(B,7 7, x) while minimizing [|((()

---

Theorem 2.1. Assume the conditions of Lemma 2.2 hold with p(en) = ol Km and Cj >
2 in
3
Condition A.2.2. For some ön s.t.
yn S 6n 2 Varn' ler A(cn, 5n) = { maxier 13, 3. >
dn. dips. Pv)) ) S Enj, where En, is the posterior contraction rate as defined in Lemma there exists some constants
2.1 Assume
and M > such that
C.I gr (2 ............ .k.....))...) is generic 1 Dn) + aas n - 00.
10y,
minjey
|8 > Co and nt (T((Gn) 5m) I
C.2 (1((8)) < M. (+))( < M, |1 < M. [ < M, [nr(3) < M for any i, E 7 $ Y and B E B25,, (3*).
C. sup (lEa(la"Tyyaaa 113 -1 1.2bx, ||||= ]3 S ( and Bxn. (1), where U from a neural network model parameterized by B.,,, and Es (6) denotes the mean of Z.
Eg.- (Z), Z denotes a random variable drawn
Then n. =(vn(((())) - 8²) Dm] ~ NN V) in P p'ooobbbbillity as n + 0o, where V = (Viz), and N.j = (7(()((0"))) if Ex" and O otherwise.
Condition C.I is essentially an identifiability condition, i.e., when is sufficiently large, the DNN weights cannot be too far away from the true weights if the DNN produces approximately the same distribution as the true data. Condition C.2 gives typical conditions on derivatives of the DNN. Condition C.3 ensures consistency of the MLE of 3 for the given structure 7 22
2.2.1 Asymptotic Normality of Prediction
Theorem 2.2 establishes asymptotic normality of the prediction n(3 æn) for a test data point Vo. which implies that a faithful prediction interval can be constructed for the learnt sparse neural network. Refer to Appendix A.4 for how to construct the prediction interval based on the theorem. 8² (31(3,00)
Let Uig (g (8, denote the d-th order partial derivative op., "P., "....."""''""""""
Theorem 2.2. Assume the conditions of Theorem 2.1 and the following condition hold: (B,(3" 20) < M, (Br,)(B, zo)] < M, (3,(3, xo B B28, (8*), where M is as defined in Theorem 2.1 Then x Vn(u(B, ar) -pl"" (r)) Dn) ~s
M hold for any i, E --, & r and
N( Σ), where E = V-r 26) H VV- 113" 2o) and H = E(- ((( is the Fisher information matrix.
The asymptotic normality for general smooth functional has been established in 4. For linear and quadratic functional of deep ReLU network with a spike-and-slab prior, the asymptotic normality has been established in [33]. The DNN prediction nrB, æo) can be viewed as point evaluation functional over the neural network function space. However, in general, this functional is not smooth
with respect to the locally asymptotic normal (LAN) norm. The results of and 33 are not directly applicable for the asymptotic normality of μ(3, zo).
3 Prior Annealing Algorithms for Sparse DNN Computation
As implied by Theorems 2.1 and 2.2] consistent estimator of (r* .13)) is essential for statistical inference of the sparse DNN. Toward this goal, 31 proved that the marginal inclusion probabilities qs 's can be estimated using Laplace approximation at the mode of the log- posttrrter,or, Based on this result, they proposed a multiple-run procedure. In each run, they first maximize the log-posterior by an optimization algorithm, such SGD or Adam; then sparsify the DNN structure by truncating the weights less than a threshold to zero, where the threshold is calculated from the prior (3) based on the Laplace approximation theory; and then refine the weights of the sparsified DNN by running an optimization algorithm for a few iterations. Finally, they select a sparse DNN model from those obtained in the multiple runs according to their Bayesian evidence or BIC values. The BIC is suggested when the size of the sparse DNN is large.
Although the multiple-run procedure works well for many problems, it is hard to justify that it will lead to a consistent estimator of the true model (7)))")) To tackle this issue, we propose two prior annealing algorithms,, one from the frequentist perspective and one from the Bayesian perspective.
5

---

3.1 Prior Annealing: Frequentist Computation
It has been shown in 27
13 that the loss of an over-parameterized DNN exhibits good properties:
(S*) For a fully connected DNN with an analytic activation function and convex loss function at the output layer, if the number of hidden units of one layer is larger than the number of training points and the network structure from this layer on is pyramidal, then almost all local minima are globally optimal.
Motivated by this result, we propose prior annealing algorithm, which immune local traps and aims to find a consistent estimate of (3*,, ,"") as defined in ( The detailed procedure of the algorithm is given in Algorithm
Algorithm Prior annealing: Frequentist
(Initial training) Train a DNN satisfying condition (S*) such that a global optimal solution
go = arg max (3) is reached, which can be accomplished using SGD or Adam [16].
(ii) (Prior annealing) Initialize 33 at Go and simulate from a sequence of distributions
((3 for = 1,2,...,77. m, where < (2) < 17 (2) < .............................. 5 m7(mm) = 1, Th = A,, o..) + (1-1,,,)(C (o)))))) and init (1) (2)
x ( Dn, 7, (k),, (k)) x (3)/ k
dun
0.n
Jun 2 .... Tu.r = end The simulation can done in an annealing manner using a stochastic gradient MCMC algorithm [34 6 22 26). After the stage has been reached, continue to run the simulated annealing algorithm by gradually decreasing the temperature
T to a very small value. Denote the resulting DNN by 3 = (B,, "t............. 3k.))
(iii) (Structure sparsification) For each connection 21,2,.... Kn set 5. 1 if [ >
and otherwise, where the threshold value of [ is ob- tained by solving (r, 113,) 0.5 based the mixture Gaussian prior in B1 Denote the yielded sparse DNN structure by 7.
20 Voī,n -ā. log A- Mi.. 0p.a
(iv) (Nonzero-weights refining) Refine the nonzero weights of the sparsified DNN by maximizing (B). Denote the resulting estimate by 34. which represents the MLE of g
For Algorithm the consistency of (5. 33y) as an estimator of (7", can be proved based on Theorem 3.4 of (27] for global convergence of 33. the property of simulated annealing (by choosing
an appropriate sequence of MJs and a cooling schedule of T), Theorem 2.2 for consistency of structure selection, Theorem 2.3 of (31) for consistency of structure sparsification, and Theorem 2.1 of [29 for consistency of MLE under the scenario of dimension diverging. Then we can construct the confidence intervals for neural network predictions using (7, Bq.) The detailed procedure is given in supplementary material.
Intuitively, the initial training phase can reach the global optimum of the likelihood function. In the prior annealing phase, as slowly add the effect of the prior, the landscape of the target distribution is gradually changed and the MCMC algorithm is likely to hit the region around the optimum of the target distribution. More explanations on the effect of the prior can be found in the supplementary material. In practice, let denote the step index, a simple implementation of the initial training and prior annealing phases of Algorithm can be given as follows: (i) for
0 t < Tı, run initial training;
inet and linearly increase 1)t by (ii) for Ti < t S T2. fix o.n = o,,
(t)
2
setting ( = I-T, 7:
(iii) for T3 < < T3. fix 77(1) = 1 and linearly decrease (0om)) by setting
(t) T--t (o + (0.n 7--72
(o 2 (iv) for > T3. fix (1 and d. 0 and t-T 7-
(t)
gradually decrease the temperature e.g., setting T: - for some constant

---

3.2 Prior Annealing: Bayesian Computation
For certain problems the size (or #nonzero elements) of 7 is large, calculation of the Fisher information matrix is difficult. In this case, the prediction uncertainty can be quantified via posterior simulations. The simulation can be started with DNN satisfying condition (S*) and performed using a SGMCMC algorithm [22] 26 with an annealed prior as defined in step (ii) of Algorithm | (For Bayesian approach, we may fix the temperature т = 1). The werrpaaaaeterized structure and annealed prior make the simulations immune to local traps.
To justify the Bayesian estimator for the prediction mean and variance, we study the deviation of the path averaging estimator + Σ ()((()) and the posterior mean f (()))((()) D,; 1f", o for some test function (( For simplicity, we will focus on SGLD with prior annealing. Our analysis
can be easily generalized to other SGMCMC algorithms [
For test function d(-), the difference between ((3) and (B Jan )dB can be char- acterized by the Poisson equation:
Cb((3) 6(B) / (33)T 17" , , )d.
where (-) is the solution of the Poisson equation and L is the infinitesimal generator of the Langevin diffusion. i.c. for the following Langevin diffusion d = v Log(T((3) 1f", Dd,t) + /2IdW, where I identity matrix and W is Brownian motion, we have
LE((3) == (B), V (o 77" )o,))) + (B)))
Let D'ub denote the kth-order derivatives of v. To control the perturbation of d(B), need the following assumption about the function (²())
Assumption 3.1. For ls E 2,, 3). D exists and there exists a function V, s.t. |Dbbbll 2 yo for some constant pk > In addition, smooth and the expectation of VP on ((t) is bounded for some p S 2 max: {pk), i.e. sup, ((V)((()) < co. Exe(o,1) (83 + (1- 3),) $ (²,,)) + (3,)).
In step & of the SGLD algorithm, the drift term is replaced by V log x log ( n, (7, (.)) where
D is used to represent the mini-bater data used in step Let L, be the corresponding infinitesimal generator. Let 6 = L. L. To quantify the effect of 6,, we introduce the following assumption: Assumption 3.2. (()) has bounded expectation and the expectation of log-prior Lipschitz continu-
ous with respect to J i.e. there exists some constant M s.J. sup, ((( S M < co. For all I,
E Eoog(( ( (g(1) |n, T 11,)) - -E log(( ( Ann 0on o1,))) M M o.n
(t1)
(2a)
(ta)
Then have the following theorem:
Theorem 3.1. Suppose the model sutisfy assumption 3.2 and a constant learning rate of is used. For test function ob(), if the solution of the Poisson equation eh(-) satisfy assumption 3.1 then
((3i)) - f ((B)) ,o )d3 =0
Et (1)( -7"1+
100.2 σùma))
+
T
(5)
where oūn is treated as fixed constant.
Theorem 3.1 shows that with prior annealing, the path averaging estimator can still be used for estimating the mean and variance of the prediction and constructing the confidence interval. The detailed procedure is given in supplementary material. For the case that a decaying learning rate is used, a similar theorem can be developed as in (5).
4 Numerical Experiments
This section illustrates the performance of the proposed method on synthetic and real data examples For the synthetic example, the frequentist algorithm is employed to construct prediction intervals.
'The code for running these experiments can be found in https://githut/ com..//yy//.o/////////aaaa..aa Sparse-Deet Shdep-Leerning NeNe Silew-Fraeewrkk-. Immuneto- Traps- and Miscaalbraatinn

---

Table I: Simulation Result: MSFE and MSPE were calculated by averaging 10 datasets, and their standard deviations were given in the parentheses.
Method
[S]
FSR
NSR
MSFE
MSPE
BNN_an _an_nel
5(0)
0
0
2.353(0.296)
2.428(0.297)
BNN Evidence
5(0)
0
0
2.372(0.093)
2.439(0. 132)
Spinn
10.7(3.874)
0.462
0
4.157(0.219)
4.488(0.350)
DNN
1.101e-5(1.1542-00
16.9226(0.322 (3.30)
Dropout
1.104(0.068)
13.183(0. 716)
BART50
16.5(1.222)
0.727
0.1
11.182(0.334)
12.097(0.360)
LASSO
566.8(4.844)
0.993
0.26
8.542(0.002)
9.496(0. 148)
SIS
467.2(11.776)
0.991
0.2
7.083(0.023)
10.1 (0.114(0.161)
The real data example involves a large network, so both the frequentist and Bayesian algorithms are employed along with comparisons with some existing network pruning methods.
4.1 Synthetic Example
We consider a high-dimensional nonlinear regression problem, which shows that our method can identify the sparse network structure and relevant features as well as produce prediction intervals with correct coverage rates. The datasets were generated as in [31]. where the explanatory variables
Ti ....... Ip were simulated by independently generating e, ........ . 2p., from NO, 1) and setting
T;
The response variable was generated from nonlinear regression model:
V2
5.z
y =
+5sin(zz3a) + 2ns Ozg + Oz'0000 +
1 x
where 6 N(00 1). Ten datasets were generated, each consisting of 10000 samples for training and 1000 samples for testing. This example was taken from [I. through which we show that the prior annealing method can achieve similar results with the multiple-run method proposed in [31.
We modeled the data by DNN of structure 2000-10000- 1000-10000-10010-- with tanh activation function. Here we intentionally made the network very wide in one hidden layer to satisfy the condition (S*). Algorithm was employed learn the model. The detailed setup for the experiments were given in the supplementary material. The variable selection performance were measured using the false
E1 (sY.S. where S is 20 [&AAS] and negative selection rate NSR = 10 [S| selection rate FSR = 11 [S.
the set of true variables, S, is the of selected variables from dataset and |S| the size of S;. The predictive performance is measured by mean square prediction eтror (MSPE) and mean square fitting error (MSFE). We compare our method with the multiple-run method (BNN evidence) well as other existing variable selection methods including Sparse input neural network(Spinn) Bayesian adaptive regression tree (BART) (2), linear model with lasso penalty (LASSO) B2) and sure independence screening with SCAD penalty (SI To demonstrate the importance of selecting correct variables, we also compare our method with two dense model with the same network structure: DNN trained with dropout( (ropout(Dropout) and DNN trained with no regularization(DNNY for these methods were given in the supplementary material as well. The results were summarized in Table I With single run, method BNN anneal achieves similar result with the multiple-run method. The latter trained the model for 10 times and selected the best one using Bayesian evidence. While for Spinn (with LASSO penalty), even with wee-parametrized structure, it performs worse than the sparse BNN model.
31 as
Detailed setups
To quantify the uncertainty of the prediction, we conducted 100 experiments over different training sets as generated previously. We constructed 95% prediction intervals over 1000 test points. Over the 1000 test points, the average coverage rate of the prediction intervals is 94.72% where (0.61%) denote the standard deviation. Figure shows prediction intervals constructed for 20 of the testing points. Refer the supplementary material for the detail of the computation.

---

Vilue
Prodicted
Figure I: Prediction intervals of 20 testing points, where the y-axis the response value, the x-axis is the index, and the blue point represents the true observation.
4.2 Real Data Example
As different type of applications of the proposed method, we conducted unstructured network pruning experiments on CIFARIO [ataset[[]]. Following the setup in [21]. we train the residual network with different networks size and pruned the network different sparsity levels. The detailed experimental setup can be found in the supplementary material.
We compared the proposed methods, BNN anneal (Algorithm D and BNN average (averaged over last 75 networks simulated by the Bayesian version of the prior annealing algorithm), with several state-of-the-an unstructured pruning methods, including Consistent Sparse Deep Learning (BNN BIC) [31]. Dynamic pruning with feedback (DPF) 121], Dynamic Sparse Reparameterization 25 and Sparse Momentum (SM) 77. The results of the baseline methods were taken from 21 and [31. The results of prediction accuracy for different models and target sparsity levels were summarized in Table Due to the threshold used in step (iii) of Algorithm []] it hard for our method to make the pruning ratio exactly the same as the targeted one. We intentionally make the pruning ratio smaller than the target ratio, while our method still achieve better test accuracy. Compared to BNN_BIC, the test accuracy is very close, but the result of BNN BIC is obtained by running the experiment 10 times while our method only run once. To further demonstrate that the proposed method result in better model calibration, we followed the setup of (23 and compared the proposed method with DPF on several metrics designed for model calibration, including negtive log likelihood (NLL), symmetrized, discretized KL distance between in and out of sample entropy distributions (JS-Distance). and expected calibration error (ECE). For JS-Distance, we used the test data of SVHN data set as out-of-distribution samples. The results were summarized in Table 3 As discussed in 23 14 a well calibrated model tends to have smaller NI.L,, larger JS- S-Dstttnce and smaller ECE, The comparison shows that the proposed method outperforms DPF in most cases. In addition to the network pruning method, we also train a dense model with the standard training set up. Compared to the dense model, the sparse network has worse accuracy, but it tends to outperform the dense network in terms of ECE and JS-Distance, which indicates that sparsification is also a useful way for improving calibration of the DNN.
(DSR)
Conclusion
This work, together with 31, has built solid theoretical foundation for sparse deep learning, which has successfully tamed the sparse deep neural network into the framework of statistical modeling, As implied by Lemma 2.1 Lemma 2.2] Theorem 2..1 and Theorem 2.2, the sparse DNN can be simply viewed as nonlinear statistical model which, like a traditional statistical model, possesses many nice properties such as posterior consistency, variable selection consistency, and asymptotic
'The Street View House Numbers (SVHN) Dataset: http: http://uuidl. stanford www.odddddu//osssuubrrr...

---

Table 2: ResNet network pruning results for CIFAR-10 data, which were calculated by averaging over 3 independent runs with the standard deviation reported in the parentheses.
ResNet-20
ResNet-32
Method
Pruning Ratio
Test Accuracy
Pruning Ratio
Test Accuracy
DNN dense
100%
92.93(0.04))
100%
93.76(0.02)
LaN_aaerage
19.85%(0.188))
92.53(0.08)
(0.9%))
93.12(0.09)
BNN_ BNN_anneal
19. (0.0% %)
92.30(0.16)
9.97% (0.0%)
92.63(0.09)
BNN BIC
19.67%().05%)
92.27(0.03)
90.0%()
92.74(0.07)
SM
20%
91.54(0.16)
10%
91.54(0.18)
DSR
20%
91.78(0.28)
10%
91.41(0.23)
DPF
20%
92.17(0.21)
10%
92.42(0.18)
BNN aaN_aveage
9.80%)).02%%
91.65(0.08)
4.77% (0.0%))
91.30(0.16)
BNN anneal
9..5%)0.03%%
91.28(0.11)
4.88% (0.02%)
91.17(0.08)
BNN _BC
9.55%(0.03%)
91.27(0.05)
4.78% (0.01 %)
91.21(0.01)
SM
10%
89.76(0.40)
5%
88.68(0.22)
DSR
10%
87.88(0.04)
5%
84.12(0.32)
DPF
10%
90.88(0.07)
5%
90.94(0.35)
Table 3: ResNet network pruning results for CIFAR-10 data, which were calculated by averaging over 3 independent runs with the standard deviation reported in the parentheses.
Method
Model
Pruning Ratio
NLL
JS IS-Distance
ECE
DNN dense
ResNet20
100%
0.2276(0.0021)
7.9118(0.9316)
0.02627(0.0005)
BNN_ -av_raeeeee
ResNet20
(0.0%%)
0,2528(0,0029)
9.9641(0.3069)
0.0113(0.000 00100)
BNN anneal
ResNet20
90.0%)
0.2618(0.0037)
10.1251
0.0175(0.000
DPF
ResNet20
0.2833(0.0004)
7.5712(0.4466)
0.0294(0.0009
BNN_ jaa_rverage
ResNet20
19. 19.85%(O.1)) 18%)
0.2323(0.00)
7.7007(0.5374)
0.0173(0.001)
BNN anneal
ResNet20
19.80% %)
0.2441(0.0042)
6.4435(0.2022
0.0233(0.002))
DPF
ResNet20
20%
0.2874(0.0029)
7.7329(0. 1400)
0.0391(0.0001)
DNN DNN_dense
ResNet32
100%
0.2042(0.0017)
6.7699(0.5253)
0.02613(0. (02613(0.00029))
BNN_ave __vverage
ResNet32
(0.08%)))
0.2116(0.0012)
(.4549(0.5456)
0.0132(0. 0001)
BNN anneal
ResNet32
9.90%().033%)
0.2218(0.0013)
8.5447(0. 1393)
0.0192(0. .0192(0.0000))
DPF
ResNet32
10%
0.2677(0.004))
7.8693(0. 1840)
0.0364(0.0015 00.005))
BNN_aver _av_racaee
ResNet32
4.7 4.77%
(..28)())0022))
7.01 2222)
0.0100(0.0002)
CON_aancce
ResNet32
(0.0%))
0.2676(0.0014)
6.8440(0.488
40149(0.0006)
DPF
ResNet32
5%
0.2921(0.0067)
6.3990(0.8384))
0.0276(0.0019)
normality. We have shown how the prediction uncertainty of the sparse DNN can be quantified based the asymptotic normality theory, and provided algorithms for training sparse DNNs with theoretical guarantees for its convergence to the global optimum. The latter ensures the validity of the down-stream statistical inference.
Acknowledgment
Funding in direct support of this work: NSF grant DMS-2015499 NIH grants R01-GM1 17597 and R01-GM126089, and Liang's startup fund Purdue University.
References
[1] Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. convergence theory for deep learning via over-paramettriizaion.. In ICML, 2019.
10

---

[2] Justin Bleich, Adam Kapelner, Edward I George, and Shane T Jensen. Variable selection for bart: an application gene regulation. The Annals of Applied Statistics, pages 1750-1781, 2014.
[3] Helmut Bölcskei, Philipp Grohs, Gitta Kutyniok, and Philiip Petersen, Optimal approximation with sparsely connected deep neural networks. CoRR, abs/ 711444, 2019.
[4] Ismaël Castillo, Judith Rousseau, et al. A bernttiin-vnn mises theorem for smooth functionals in semiparametric models. The Annals of Statistics, (5(6):2353-3386 2015.
[5] Changyou Chen, Nan Ding, and Lawrence Carin. On the convergence of stochastic gradi- ent meme algorithms with high-order integrators. In Proceedings of the 28th International Conference on Neural Information Processing Systems- -Voiume 2, pages 2278-22888 2786,286, 2015.
[6] Tianqi Chen, Emily Fox, and Carlos Guestrin. Stochastic gradient hamiltonian monte carlo. In International conference on machine learning, pages 1683-1691, 2014.
[7] Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. arXiv preprint arXivv 2019.
[8] Simon S. Du, Jason D. Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In ICMI., 2019.
[9] Jianqing Fan and Jinchi Lv. Sure independence screening for ultrahigh dimensional feature space. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 70(5):849-91,, 2008
[10] Charles Fefferman. Reconstructing neural net from its output. Revista Matemática Iberoamer- icana, 10(3):507-555 1994.
[11] Jean Feng and Noah Simon. Sparse-input neural networks for high-dimensional nonparametric regression and classification, arXiv preprint arXiv: 1.005592, 2017.
[12] Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the 33rd International Conference on International Conference on Machine Learning Volume 48, ICML'16, page 1050-1059. JorR.org, 2016.
[13] M. Gori and A. Tesi. On the problem of local minima in backpropagation. IEEE Transactions on Puttern Analysis and Machine Intelligence, 14(1):76-86, 1992.
[14] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML' 17, page 1321-1330. JMLR.org, 2017.
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[16] D.P. Kingma and J.I., Ba. Adam: a method for stochastic optimization. Conference on Learning Representations, 2015. 2015.
International
[17] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
[18] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Proceedings of the 31 International Conference on Neural Information Processing Systems, NIPS' 17, page 6405-6416, Red Hook, NY, USA, 2017. Curran Associates Inc.
[19] F. Liang, Q. Li, and L. Zhou. Bayesian neural networks for selection of drug sensitive genes. Journal of the American Statistical Association, 113(523) 2018
[20] Liang, Q. Song, and K. Yu. Bayesian subset modeling for high dimensional generalized linear models. Journal of the American Statistical Association, 108:589-600 606, 2013.
11

---

[21] Tao Lin, Sebastian U. Stich, Luis Barba, Daniil Dmitriev, and Martin Jaggi. Dynamic model pruning with feedback. In International Conference on Learning Representations, 2020.
[22] Yi-An Ma, Tianqi Chen, and Emily Fox. complete recipe for stochastic gradient meme. In Advances in Neural Information Processing Systems, pages 2917-2925, 2015.
[23] Wesley J Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson. A simple baseline for bayesian uncertainty in deep learning. In Advances in Neural Information Processing Systems, pages 13153-13164, 2019.
[24] Vitaly Maiorov and Allan Pinkus. Lower bounds for approximation by mlp neural networks. Neurocomputing, 25(1-3):8 1999.
[25] Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. International Conference on Machine Learning, pages 4646-4653 2019.
[26] Christopher Nemeth and Paul Fearnhead. Stochastic gradient markov chain monte carlo. arXiv preprint wxXiv:190.069900 2019.
[27] Quynh Nguyen and Matthias Hein. The loss surface of deep and wide neural networks. In ICML, 2017.
[28] Nicholas G. Polson and Veronika Ročková. Posterior concentration for sparse deep learning. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, page 938 9899999, Red Hook, NY, USA, 2018. Curran Associates Inc.
[29] S. Portnoy. Asymptotic behavior of likelihood methods for exponential families when the number of parameters tend to infinity. The Annals of Statistics, 16(1 (35)):356-66, 1988.
[30] Johannes Schmidt-Hieber. Nonparametric regression using deep neural networks with relu activation function. arXiv:1708.066. 066333, 2017.
[31] Y. Sun, Q Song, and F. Liang. Consistent sparse deep learning: Theory and computation. Journal of the American Statistical Association, page in press, 2021.
[32] Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society. Series B (Methodological), pages 267-288, 1996.
[33] Yuexi Wang and V. Rocková. Uncertainty quantification for sparse deep learning. In AISTATS, 2020.
[34] Max Welling and Yee W Tch. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference machine learning (ICML-11), pages 681-688, 2011.
[35] Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu. Gradient descent optimizes over- parameterized deep relu networks. Machine Learning, 109:467 492, 2020.
[36] Difan Zou and Quanquan Gu. An improved analysis of training over pee-pparameterized deep neural networks. In NuerIPS, 2019.
12


Reviews:
[{"review_id": "ySQab0mD1yY", "reviewer": "Reviewer_fjLT", "summary": "The paper proposes an approach to learning sparse deep neural networks (neural networks where most of the weights are set to zero) using annealing: starting by training a large overparameterized neural network to fit the training data perfectly, then adding weight penalties according to a prespecified schedule. A theoretical argument is provided to argue why this is reasonable, with two new theory proofs provided which extend previous work presented in Sun et al. 2021. Experiments on synthetic data show that the proposed algorithms perform feature selection better than other existing algorithms. Experiments on CIFAR10 show the algorithms are competitive in terms of the accuracy vs. pruning tradeoff.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is mostly clear and well-written, aside from some typos. However, its contribution is unclear to me. I'll address each contribution in turn:\n\nTheory: Proofs are provided for the asymptotic normality of the connection weights and asymptotic normality of the prediction. These appear to be novel contributions that build upon the previous work of Sun et al. 2021. I did not check the proofs carefully, but they appear to be non-trivial and novel contributions. However, I was unable to see their significance; the implications for the algorithms were unclear. \n\nAlgorithms: Two algorithms for training sparse DNNs are proposed. These algorithms seem reasonable, but I was left wondering about the theoretical and practical value. Line 230 says, “the MCMC algorithm is likely to hit the region around the optimum of the target distribution,” but if nothing concrete can be stated what is the advantage of starting from the global optimum and then annealing? Practically, this annealing procedure is more computationally intensive than training with standard regularization methods, and we are left with an empirical question of whether the annealing is more computationally-efficient than other annealing methods.\n\nExperiments: the experiments show that the annealing approach results in sparse networks that have competitive accuracy and calibration to other methods. However, I have concerns about the fairness of these experiments. As mentioned in Guo 2017, miscalibration is a result of overfitting and can largely be avoided by early stopping on the validation set NLL (not 01-loss). However, early stopping was not mentioned, and in the supplementary materials the authors say the benchmark comparison models were trained for a fixed number of epochs. In addition, the hyperparameters (lambda, sigma_0^end) for the annealing algorithms appear to be chosen in order to give good results, reducing my confidence in the fairness of these comparisons. Overall, I'm not convinced that this is a promising approach for training sparse DNNs in practice, if that was the goal. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately address this.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "yIZ7Hgt8FXy", "reviewer": "Reviewer_5FUc", "summary": "Overall, the work presents an interesting direction into NN pruning. However, the paper is quite hard to follow and overly complex in notation, with little effort to assist the reader. Further, the paper severely lacks useful devices, such as figures or in-depth description, to explain the methods to readers, making the work challenging to follow.  Last, some of the experimental evaluation is somewhat questionable, as specified in the details comments.  Though the direction seems useful, additional work is needed in communicating the results and in experiments in order to warrant acceptance.\n", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Per Section Comments\n———————————\n\nSection 2.1:\n- The paper is reasonable to follow up until the statement of assumptions (around line 106). However, at this point, why are these assumptions reasonable? It’s tricky to see what some of these assumptions mean (i.e., A.2.1 in particular) and why it might be reasonable to make other assumptions (i.e., A.1, this is rarely the case in practice, A.2.2 why we expect the polynomial bound to hold).  Even if it is the case that some of these assumptions may only be plausible, for the sake of the analysis, more context at this point would be highly useful for the reader.\n\nSection 3.1\n- Gradually decreasing the temperature: How is this specified?\n- It would be highly useful to provide some more intuition around algorithm 1.  Where does the constraint in (iii) come from, for example. Is \\sigma_0^init define anywhere?\n- A graphical figure here to explain the algorithm in more detail would be highly useful.\n\nExperiments:\n- Investigating the results provided in section 4.2, it seems like the proposed methods yield slightly test accuracy. Yet, the pruning ratio is slightly lower as well for these networks. Because the difference in accuracy is so slight and the pruning ratio changes slightly as well, can you account for these changes in evaluation? Looking over the results, it seems hard to justify that the method is better than baselines, given this dynamic, as the results are currently written.\n- Further, the proposed method (if I’m not mistaken) uses additional fine-tuning after the variable selection is applied (i.e., IV of alg. 1).  Do the other variable selection methods do additional fine-tuning? Some discussion about this would be good, because it seems difficult to compare this method (which allows further fine-tuning) with methods that do not.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "NA", "review_text": "", "extra_scores": {"ethical_concerns": "NA", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "aNrYGaSI71", "reviewer": "Reviewer_f5t3", "summary": "The submission addresses the issue of sparsifying trained deep neural networks in an optimal way, preserving performance and improving calibration. It builds on the formalism and results of Sun et al. (2021), proves additional results and proposes two prior-annealing algorithms, which they demonstrate on synthetic and real data, comparing favorably to existing sparsifying algorithms. ", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality\n----------------\nThis submission relies heavily on work by Sun et al. (2021), which is properly acknowledged, and expands on that work. The two prior-annealing algorithms seem novel, and the real-data, larger-scale evaluation (ResNets on CIFAR-10, following the set up of Lin et al., 2020 for meaningful comparison) make that work more accessible to current deep learning practitioners. \n\nQuality\n----------\nTo the best of my understanding, the theory is sound, and the theoretical part of the paper feels complete. The claims of performance and calibration against other sparsity-inducing methods are well supported by experiments, and conclusions drawn by the authors are faithful to the experimental results.\n\nSome of the broader claims may be over-stated, though (\"how to make statistical inference with deep neural networks\", \"tamed the powerful deep neural networks\", l.56-57), especially the ones that are not restricted to _sparse_ deep networks.\nIn order to support those claims, it would be better to have, for instance:\n- comparisons of test accuracy against non-pruned network (in Table 2)\n- comparisons of the measures of Table 3 (NLL, JS-Distance, ECE) against values obtained by algorithms aimed at improving calibration (e.g., temperature scaling, mixup training, focal loss training...) for regular, dense DNNs, rather than only considering sparsifying algorithms.\n\nClarity\n----------\nConsidering the complexity of the theoretical work, the article makes a good job at explaining the issues and results informally, giving a good intuition. The overall organization makes sense, and I appreciate the division between main paper and appendix.\n\nI believe a reader already well familiar with Sun et al. (2021) could implement the additional procedures described in this submission and reproduce the results.\n\nSignificance\n-----------------\n The perspective of taking a trained, well-performing DNN and turning it into a sparse, well-calibrated model, in a principled way, and with minimal loss of performance, is quite appealing, and I expect practitioners will want to use this method.\nThis submission adds additional theoretical guarantees compared to earlier work, making this method more solid.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors did mostly address the limitations of their work in the text.\nPotential societal impact were not discussed, but this is mostly theoretical work, so it seems fine to me.", "review_text": "", "extra_scores": {"ethical_concerns": "Minor point concerning citing existing assets: the SVHN dataset is mentioned and should be properly cited.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "9"}, "extra_text": {}}, {"review_id": "PZSXdzzAhf", "reviewer": "Reviewer_PeZg", "summary": "1. Theoretical contributions: the authors extend the theory established by Sun et al. [2021] and study the asymptotic behaviors of sparse deep learning.\n2. Methodological contributions: the authors propose prior annealing algorithms, one from the frequentist perspective and one from the Bayesian perspective, to implement the corresponding sparse neural nets. They show the performance of the prior annealing algorithms on one synthetic example and the CIFAR10 datasets.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors inherit the sparse DNN framework used in Sun et al. [2021], and they investigate the asymptotic normality of connection weights and predictions.  The theory established in the paper are around the true DNN model $\\mu^*(\\beta^*, \\gamma^*, x)$, which is a data-dependent $L^2$ projection of $\\mu^*(x)$ onto a pre-specified neural network space $\\mathcal{G}_n$. My intuition is that one would need prior knowledge on what an adequate network structure looks like, and do the pruning based on that, like the CIFAR10 example. It appears more as a refinement to already mature neural net models.\nAnd it is not clear how one can relate the theory to asymptotic behavior around $\\mu^*(x)$ since that will depend on the approximation error. Maybe the authors can elaborate more on that. In addition, how the noise $\\sigma^2$ could impact the model performance, is it assumed known?\n\nFor the two prior annealing algorithms, the results from the experiments look decent. But it is somewhat disconnected from the asymptotic theory and it is not obvious how to assess the uncertainty in prediction. It is not clear how the prediction intervals in Figure 1 are constructed, it seems like a variant of bootstrap. \n\nSome minor comments:\n1. The notation for log-likelihood function $l_n(\\beta)$ sometimes appear as $nl_n(\\beta)$. Line 126-127, the expression is not properly placed. Overall, the notations in the paper are really heavy and section 2.1 is directly borrowed from Sun et al. [2021].\n\n2. For the references, some conferences as NeurIPS and ICML have inconsistent formats.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I didn’t spot a discussion on limitations. Maybe the authors can discuss based on the reviews and provide the future direction of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed a framework for sparse deep learning, with the primary goals as addressing mis-calibration problems. The paper contains both theoretical analysis as well as some experiments to demonstrate the theoretical claims on Bayesian ResNets. 

This paper sits at the intersection of network pruning and uncertainty calibration so it might benefit both communities. However, the paper heavily assumes prior knowledge presented in Sun et al. (2021) which is a very recent JASA paper. I would expect this paper to be very difficult to read for deep learning practitioners interested in network pruning.

The paper stated that "This work, together with Sun et al. (2021), has built a solid theoretical foundation for sparse deep learning". Therefore in my view, acceptance decision critically relies on how significant the contribution of this paper is on top of Sun et al. (2021). 
- From my understanding after a brief read, the key theoretical result in this paper is the asymptotic consistency/normally estimate for the defined "ground truth" parameters. 
- While Sun et al. (2021) established posterior consistency and structure identification consistency, that is regarding the estimation of the data distribution, rather than the underlying parameters. 
- Therefore I think the two paper's theoretical results are different aspects of the same framework. This makes me think that in some sense this is a project split into two publications? 

Experimentally, the proposed approach and the approach of Sun et al. (2021) perform quite close in many of the experiments. 
- This also raises a practical question in that, to what extent it is important to ensure "consistency in parameter estimation"? 
- It is even more confusing that the "ground truth" parameters are defined using "universal approximation of neural network", so I would think "consistency in function estimation" might be more important here, which is already discussed in Sun et al. (2021) to certain extent.
Decision: Accept (Poster)