Asymptotic Properties for Bayesian Neural Network in Besov Space
Authors: Kyeongwon Lee
Jaeyong Lee
Venue: NeurIPS 2022 Accept (2022)
Keywords: Bayesian neural network
sparsity
deep learning
posterior consistency
optimal rate

Abstract: Neural networks have shown great predictive power when applied to unstructured data such as images and natural languages. The Bayesian neural network captures the uncertainty of prediction by computing the posterior distribution of the model parameters. In this paper, we show that the Bayesian neural network with spikeand-slab prior has posterior consistency with a near minimax optimal convergence rate when the true regression function belongs to the Besov space. The spikeand-slab prior is adaptive to the smoothness of the regression function and the posterior convergence rate does not change even when the smoothness of the regression function is unknown. We also consider the shrinkage prior, which is computationally more feasible than the spike-and-slab prior, and show that it has the same posterior convergence rate as the spike-and-slab prior.

Asymptotic Properties for Bayesian Neural Network in Besov Space
Kyeongwon Lee Department of Statistics Seoul National University
Jaeyong Lee Department of Statistics Seoul National University Seoul, Republic of Korea 08826 lee leejyccgmail. com
Seoul, Republic of Korea 08826 Skw1718@snt .c.
Abstract
Neural networks have shown great predictive power when applied unstructured data such as images and natural languages. The Bayesian neural network captures the uncertainty prediction by computing the posterior distribution of the model parameters. In this paper, we show that the Bayesian neural network with spike- and-slab prior has posterior consistency with near minimax optimal convergence rate when the true regression function belongs to the Besov space. The spike- and-slab prior adaptive to the smoothness of the regression function and the posterior convergence rate does not change even when the smoothness of the regression function unknown. We also consider the shrinkage prior, which is computationally more feasible than the andke-and-slab prior, and show that it has the same posterior convergence rate as the spike-and-slai prior.
Introduction
The neural network is machine learning technique that can handle complex structures in data by combining linear and non-linear transformations [Goodfellow et al. 2016). The neural network has shown great predictive power when applied to unstructured data ranging from image data to natural languages. The neural network automatically grasps the inherent structure of the data and makes end-to-end learning possible only with inputs and outputs without explicit intermediate steps, which is the main reason for the rapid development of numerous artificial intelligence applications.
The Bayesian neural network (BNN) provides the uncertainty of the prediction from the probability distribution of the model parameters. Before observing the data, a prior distribution that reflects prior beliefs about the model is put over the parameter space of the neural network model. As observations are added, posterior distribution is computed by updating the prior distribution via Bayes' rule.
Neal [1996), Williams (9966) showed that a shallow BNN which has only one infinite-width hidden layer converges to a Gaussian process regression model. This means that the shallow BNN can be regarded as finite approximation of the nonparametric Bayesian regression model.
Lee et al. 2017].
Matthews et al. 2018 extended this result to the deep BNN model.
However, it is difficult compute the exact posterior distribution of the BNN. Various methods have been proposed for BNN through approximate Bayesian computation algorithms. The methods based on the Markov chain Monte Carlo (MCMC) algorithm such as the Gibbs sampler [Geman and Geman 1984 Casella and George 1992] and Hamiltonian Monte Carlo (HMC) (Neal et al. 2011] are effective to approximate true posterior. By constructing Markov chain that has the desired distribution as its stationary distribution,
obtain sample of the desired distribution. Recently, scalable MCMC algorithms such as stochastic gradient Langevin dynamics (Welling and Teh [01 and stochastic gradient HMC (SGHMC, Chen al. 20144 were proposed. The variational inference (VI) is an approximation method that finds the closest variational distribution from the posterior
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

distribution. VI method changes the Bayesian inference problem to an optimization problem. Graves [201T] proposed to use of a Gaussian distribution as variational distribution for BNN and Kingma and Welling 2013) suggested the reparameterization trick for Gaussian variational distribution. Practically, VT methods for BNN are faster and more stable than MCMC methods. Blundell et al. [2015) suggested Bayes by Backprop algorithm for BNN via variational inference. There are also studies that approximate BNN with the dropout [Gal and Ghahramani 2016 or the Gaussian process [Lee et al. 2017
1.1 Related works
The performance of the neural network is justified by the universal approximation capability of the neural network. It is known that even a shallow neural network can approximate arbitrary continuous functions sufficiently precisely in TP-sense [Cybenko 1989. Hornik 1991], and as the network
deepens, the expressive power of the neural network gets stronger Delalleau and Bengio, 2011 1 Bianchini and llarselli] 2014 [Telgarsky] 2015 2016 Lee III| [999] showed that the BNN has the universal approximation property.
As a measure of the expressive power of a model, consider the range of function spaces that a model can express. The choice of the activation function can affect the expressive power of model, and
in this paper, we consider the Rectified Linear Unit (ReLU) (r) = max (0, x) [Nair and
Hinton 2010], which is the most popular activation function practice. Yarotsky| [2017] evaluated the approximation еттог of the neural network using the ReLU activation function (ReLU network)
for functions in the Hölder space. (Schmidt Hieber [2200] showed that the regularized least squared
estimator performed by the ReLU network in nonparametric regression problem converges to the true regression function with nearly minimax rate up to logarithmic factors on the Hölder space. Suzuki 2018) extended the results in |Schmidt-Hieber (2020) from the Hölder space to the Besov space which contains the Hölder and Sobolev spaces. Polson and Ročková (2018) proved a Bayesian version of the results in Sccmmidt-Hieber [2020]. They proved the Bayesian ReLU network with spike-and-slab prior has posterior consistency, i.e., the posterior probability is concentrated on the true parameter value as the number of observations increases to infinity, when the priors the parameters of the ReLU network are spike-and aadeee-nd-llabb priors. Albéreff-Abellaan (2020) showed that the results in Polson and Ročková (0118 are still valid when variational methods are applied.
All of the above results assume sparse structure the neural networks. Sparsity is often assumed in high- -igh-iinnnsionl models for asymptotic results. Castillo et al. (2015) showed theoretical properties including posterior consistency and rates of consistency of the nigh-dimensional linear regression model under the spike-and-slab prior. Song and Liang [2017) extended the results in Castillo et al. 20155) to the shrinkage priors which are more feasible than the spike-and-sla prior.
1.2 Main contribution
We show that the posterior of the Bayesian ReLU network with spike-and-slab prior with appropriate width, depth, and sparsity level is consistent with a near minimax rate. We extend the result of Suzuki (2188) to the Bayesian framework and the results of Polson and Ročková [2018] the Besov space. As in Polson and Ročková [2018), we show that the same convergence rate is achieved even when the smoothness parameter of the true regression function is not known. That is. the ReLU network has rich expressive power, and this ability is not constrained even without the smoothness information of the true function. We get a similar result for the shrinkage prior.
We try to narrow the gap between the study of the theoretical properties and the practical application of the Bayesian ReLU networks. For instance, Gaussian distribution is frequently considered as a prior distribution in practical Bayesian ReLU networks. In this case, the posterior distribution may not be consistent since the posterior becomes heavy tail distribution over the parameter space (Vladimirova et al. 2019]. Approaches to interpreting the Bayesian neural networks as Gaussian Process [Lee et al. 2017] avoid this problem by setting the variance of the prior distribution to be proportional to the width of the network. We suggest similar sufficient conditions for the posterior consistency of the Bayesian ReLU networks. Compared to precedent studies, our setting is closer to practical use. The results in Suzuki (2018) and Polson and Ročková (2018) require to solve the optimization problem under the lo constraint or compute posterior from spike-and-slab prior respectively. We suggest a BNN model, which is easy to infer and consistent with the (nearly) optimal convergence rate.

---

The rest of the paper is organized as follows. We set up the problem and introduce the necessary concepts in Section[] We present the obtained theoretical results Section and numerical examples in Section 4 A summary of the paper and discussion are given in Section
2
Preliminaries
2.1 Notation
The floor function [=] denotes the largest integer less than or equal to x R. the ceiling function [ denotes the smallest integer greater than or equal to æ E R, and ||a|l, is an mo-rr of a real vector
........ a. The u-th derivative of a d-dimensional function f is denoted by D with v E (0, ...... The covering number [e, A, d) is the minimal number of d-balls of radius >> necessary to cover set A. We denote the dirac-delta function at a E R by 6a. We use A S B as shorthand for the inequality CB for some constant C' and A B as shorthand for the m B).
A <
2.2 Model
Suppose that n input-output observations D. = 3:):3:)1111 C. [0, 1] x R are independent random sample from a regression model
y/ = (X() & (i = ,.))
(1)
where (5:),_1 is an i.i.d. sequence of Gaussian noises AO, o²) with known variance o² and fo is the true regression function belonging the space f
In frequentist statistics, the maximum likelihood estimation is used estimate the function fo. From the machine learning viewpoint, it equivalent to finding f that minimizes the square loss. Suzuki (2018) showed that fo can be estimated with nearly optimal rate in this setting.
In practice, sparse regularization term r(f) such as dropout and l1-reullarizer is also considered together with loss function. This is equivalent to estimating the MAP (maximum posteriori) in the Bayesian modeling. The posterior distribution gives not only predictive values but also the uncertainty of the prediction. Thus, in the Bayesian statistics, it is important to find sufficient conditions for prior distribution or regularizer which guarantees the consistency or optimal convergence rate of the posterior.
2.3 Function spaces
In this section, we introduce function spaces that statisticians have studied for the smoothness of functions. Let Ω 1] C I2d be the domain of functions we consider.
The L of function is defined as follows:
(aal)()) supzealf(x))
0<p <c, p=
| =
where f is a real-valued function defined on S2. The L space is the function space consisting of functions with bounded L"-norm and is denoted by LP (22) = {f IIfll < oo).
Let s > 0 be the smoothness parameter and m [s]. The s-Hölder norm of function f defined by
DO f(r) - f(y)
llfllo. = sup ||lll Sm
|| + sup sup [||]] =m eeE
v/
The s-Hölder space (9) is defined as a set functions with bounded s-Hölder norm. We call s the smoothness parameter of the s- Hölderr space. Let & e N and p < co. The Sobolev norm of function f is defined by
Ilflw == (Eilal, ((EEin) SalDD"If 1/p
1 S 80
Suppalll sk DD DlSll
00.
3

---

The Sobolev space (S))( is defined by the function space consisting of functions with bounded Sobolev norm.
Note the notion of the smoothness of a function is related to the differentiability of the function. The Besov space extends the concept of smoothness. Before defining the Besov space, define the r-th modulus of smoothness of the function f as follows [Gine and Nickl 2016]:
wr,p(f,)= = sup | St
(2)
A)(f(=)= = Eiz ((1))))-)) x xEΩ, + rh E
(3)
10
o.w.
For <...9 00, 8 8> 0, r= [s] + 1, define the Besov norm of a function f by
11 Pie = 1lll, + ( (t t)) at 4)1/ |suprsott "
q 00, 0c,
(4)
oC.
The Besov space B p, (92 is defined as the set of functions with finite Besov norms 2016). i.e.,
Gine and Nickl
B"
p.g (2) = {s f 11. < <oc
(2)
(5)
Note that the Besov spaces be defined without the differentiability and continuity of functions, and are more general than the Hölder and Sobolev spaces, which are subspaces of the Besov spaces. The Besov space may contain complicated functions including discontinuous function d/p 2 s. For instance, the Cantor function fi does not belong to any Sobolev space since it does not have weak derivative but belongs to a Besov space (Sawano, 2018 In fact, fi E Clog 2/ log (10, 1J) C plog 2/ log 3 ((0, 1)). Also.
[1/log(z/2), 0<
f2(x)
(6)
10,
x= 0
is not in any Hölder space. Note when < 2, the smoothness of the Besov space is inhomogeneous Suzuki 2018 Donoho and Johnstone [1994] suggested spatially variable functions including Blocks
fa(x) Σ K; 2,), K(x) (1 + sgn (z))
(xy) = [,0
0.400 0.44, 0.w.0...,,06,,,,.0 0.78, 0.81),
(h₂) = -4, -4,3, -,,,,42,,21,,,,3 4,,,4222,,,,,3333,,,,----111 -3.1,
where sgn(x) is function that extracts the sign of real number æ, and HeaviSine
fa()) = 4 sin(4 - sgn(22 x 0.3) - sgn
-ggn(.072 (0.72 - x).
(8)
Since fu, fa and fa are functions of bounded variation, in Bi,cc (10, 1]) We plot the functions fi. f2, fa and fa in Figure
Peetre 1976 Suzuki
2018
2.4 Sparse neural network
Define (deep) neural network space generated by a parameter space as follows:
( = fo(x) = (w(++)(() ((++1)) o o...........................oo.. . (w(l)) (W())+
(9)
0 = (1))),61) ,(((+)) (((+(1)) }).

---

Figure 1: Example functions in the Besov spaces
0.4
-1.2
-1.4
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
(a) (i()) on [0, 1]
(b) f2(x) on [0,
5-
4-
3-
2-
1-
0-
-1-
-2-
-6-
0.0
1.0
0.2
(c) fa(x) on [0, 1]
(d) fa(x) on (O, 1]
where Ç is activation function. In this paper, consider ((x) ReLU(()) and the following B-bounded and S-sparse parameter space [Suzuki 2018
OLL, W. =
W E P Xp: ,0(() E RR
Pt = W,l 1,2 ... L, Po = d = 1,
|1011- Σ (|wwc0/)
1+1
|1. + +11600| S.S
(10)
||0|| = =max
w(0)
v
800
S B
and denote S-sparse neural network space with B-bounded parameters ( W,S, B)) as @ (L, W. B). We can define the entire neural network model space as follows
~
W,
(11)
3 Main results
Let (B(() = ((0. If(x) F, Vr E (0, 1]1 be an uniformly bounded family of functions with bound F > 0. Consider F B (0, 11d) (UB(F). For each N, and f e F.
5

---

let p(() be the probability measure of the data D, = (X 26) with a density function pj with respect to the measure (()) and II be prior distribution of f. The posterior distribution is given by
(n)
_AP/ (m) (D)))
II(AD
(12)
P= (In,) ddl(f)
for A C θ Let
IlII,, = 1
1 Σ (U(x:)))) 11
1/2
be the norm on empirical measure P = Ei= dx. and
[ ( (f(X)) dPx
1/2
where Px is a marginal distribution of X. Suppose that Px has a bounded density function px (z) on [0, 1]d. Without loss of generality, we assume that 0 S px (2) S S 2.
Suppose that 0 < < oo, 0 < p, I co, w == d(1/p-1/2) 1/2]+ < 8 < oa and set v = (s (8- (2w) Assume that m E N satisfies < 8 < min(m, m + 1/p). Let N₂ = [nd/(20+6)), Wo = 6dm(m + 2) + 2d and
L, = L(NN), Wa = Wo:
(13)
S, (Ln N, + Bn ON,,)
where ((d.m) = (1 + 2de (20)
L(N.) = + 2[log2
gavm
+ 5l (d V m]], Tn
vm
N₂ w-s//d(e (v-) + (dd/p=>)) (log Na)-1 and E = (v-1 d (d/p-s) +)))
Note the theorems in this section are similar to the results in Suzuki (018 which studied the
asymptotic properties of the least squared estimator. They showed that for the least squared estimator
E(n f(x;)),
arg min
1.E. W. S.. Ba) i=1
(14)
ED. 11:00 fl (ILACE))
S n zंta (log n)3
where f is the clipping of defined by min -F)), F) for E UB(F) and Ep, [A] is an expectation of A with respect to the training data D,, Note the minimax convergence rate of Besov function fo in (1) has lower bound n 2-ス2 [Gine and Nickl 2016 Donoho and Johnstone Donoho and Johnstone [1998] also showed that the minimax rate of linear estimator has x-(1---1757
1998
lower bound, n 2 211y 172. for d = 1, > 1/p and 1 S p. S oc or 8 = = = 1. Thus, any linear estimator cannot achieve the minimax optimal rate on the Besov space for p < 2 and is dominated by the neural network model.
Spike-and-slab prior
In this subsection, extend Theorem 6.1 in Polson and Ročková 2018 They showed that the Bayesian ReLU networks are consistent with near minimax rates in the Hölder space. We extend the
function space from the Hölder space to the Besov space which contains both Hölder and Sobolev spaces. Assume the following snike-and-slab prior (0)
π T(, 11. L, W.S, B)
(, L, W. S, B) (1 71) (0,)))
1
(15)
( L, W)
x(L L.,) T W x $$ 5,)= (B= B,)) 1,
(16)
where W, B) U(0;; 1B BJ) and T (0 W. S)

---

Theorem 1. Assume model 1 and prior distribution (15) and (16. Suppose that 0 < F < 00, 0 < P.q S oo and d(1/p- 1/2) s. Then the posterior distribution concentrates at the true function with rate Cn n a/(+++))) (log n))/2. That is, IIffe (F) llfo foll,, Mn.n(DD) 0
p(a) fo pprobability as n + oo for any Mn + oc.
Proof. See appendix
As mentioned in Suzuki 2018). the condition d(1/p-1/23 < indicates that fo E Bp (lO, 114) is in (20, 1]4) and be discontinuous. When p oo. a (10, 1]4) C*0 1]4) and the result by Polson and Ročková 2018) is a special case of Theorem
Adaptive estimation
Theorem 2 beloo is an extension of Theorem 6.2 in Polson and Ročková [2018] to the Besov space. They showed that the Bayesian ReLU networks adapt to smoothness on the Hölder space under the spike-and-llad prior.
Theorem 2. Assume model 47). Let
(H) = (Hoggn)) 11, (H,,H = HN, S,HH,N) = (H), (H = N
(17)
Consider the following prior
_ 1v
A 1v [Z/lo/)) az (Z) Zl(cAx 1) for 7 = 1,2,
N
(18)
and prior distribution function space
15 of given Ln(H,), Wn (H,, Sn, N), Ba(Hu, N) on the
U
U (H,,(H)), W. W.(Hn,, N), Sn N), B, B,(Hn,
(19)
n=1 N=1
for any Hn + 00 and An > Suppose that F co, p.q S oo and d(1/p-11 1/2)+ <
< min m. m 1 + 1/p]. The posterior distribution concentrates at the true function with rate En =n 5/(6/ (2) (log n) 3/2 That is,
II (fe E ① NUB(F) llfe - foll, > M,,n(DD.) +
in p(1)
prrobability + oo for any M, + cc.
Proof. See appendix C.5
In Theorem
the prior distribution depends the smoothness parameters p. g, and s. Theorem shows that, as in Polson and Ročková 2018). even if the values of these parameters are unknown, the same posterior convergence rates can be obtained by considering the appropriate prior distribution.
Shrinkage prior
Asymptotic properties the spike-and-slas prior were obtained in Theorem I but its practical use for large neural network models is unsuitable due to the computational cost. The following prior distributions are computationally more feasible than the spike-and-slad prior.
The result to be introduced in this section shows that all the aforementioned results still hold even under the shrinkage prior. Assume the following prior
T
L, W,S,B) = II W,S,,,
(20)
J=1
where 7 7= (,, = L W, S, B) a symmetric density function and decreasing on t > 0.
7

---

Theorem 3. Assume model
4
prior distribution
16 and (0)). Suppose that 0 < F < 00. 0
P. S oo and d(1/p 1/2)+ s min m, m I+ 1/p). Let en n +/(2st)) (logn) 33/2 and
g(l) be function such that
On v 72Ln (B 11) "W' 1)
(21)
Un = | [amneal
g(t Wn Sn, B,)
satisfies
Sn
Sn Tn Un 2
T/n
(22)
Tn
log g( Bu W Sn. Ba) S (logn)?
(23)
continuous on [-Bn, Bn] and
[ B...
g(t|Ln, Wn, Sns Bn Bd) =o
(24)
where expl Hopp-Knnrt / for for some K, Ko > 4. The posterior distribution concentrates at the true function with a rate En. That is,
ll[fe 4 (uBB(F) life foll. > Mnen(BD) -
in p(a)
probbubiliy n - 0o for any Mn +
Proof. See appendix C.6
Note that condition 22 is continuous approximation for the spike part of (15). The next condition
(23)
means that the prior should have enough thick tail to sample true function. The last condition
24 restricts the thickness of the tail prevent function from being divergent.
Unlike the other priors, the shrinkage prior relatively straightforward to implement. For example, to implement the model in Theorem one consider the MCMC algorithm [Sun al.) [202)] which combines Gibbs sampler, the Metropolis-Hastinngs algorithm, and the SGHMC algorithm. The shrinkage prior avoids the posterior computation with varying dimensions, and enables feasible computation.
Note that the conditions sufficient conditions for the neural network model necessary to estimate the function in the worst case. In other words, we may infer true function fo sufficiently even for a prior distribution that satisfies only some conditions.
Example (Gaussian prior). Let en = n #/(2+t (logn) 3/22 and v be a density functions of the standard Gaussian distribution (0, 1).
( 5n
B²
g(t) = on
v
o²
2Kone?
satisfies (24).
Example 2 (Gaussian mixture prior). Let 6n n (log n 3/22, 1/ and Vi be density and inverse survival functions of the standard Gaussian distribution N0, 1) respectively.
t
1
g(t)
Tin
+ T2n
Tin
Tin
J2n
with
Sn
T2n
Tin 1 122
T
6n
B2
On - 72L, (Bn V ho 10 (W, + +1)an
22 <
2 2Kone?
and
On /
Tm
2 mm
v
Un 1)
oin
on/V
12n
1
v
On
Tin
02n
Tin
O2n
satisfies
22 and
24
8

---

Example (Relaxed spike- spike-and-See and-slab prior). Let en = n -/- (log [logn)3/ n)3/ v and y be density and inverse survival functions of the standard Gaussian distribution NO, 1) respectively.
(1)
s(1)
Fin
1
+ T2n (1; - -Bnr Bn)
Onn
0ln
with
Sn
n2n
Tin 1 12n
T,
On S 72Lm (Bn V 11 -WW, 1)
and
T2n
oin < <a./V an - (1 - -mi)
Ten an 2n Bn
an V
2
Bn
satisfies (22
23
and 24
Numerical examples
As the sample size increases, the complexity of the neural network obtain the theoretical optimality increases rapidly, which makes the results in the previous section limited for the practical application with large sample size. For instance, consider the estimation problems of Besov functions fI, f2, fa and fa with the Gaussian mixture prior with parameters as in Example 2
Bn 10.mmn((E.1) Ko 5, Th exp( Sesp--Kope / S.))
En
72 (Bn VI)L. (W, + 1)
B² 22 2(Ko 1)
Oin an/V
Tin 2 Tm
w-1
Qn
02n
Tables and 2 present the hyperparameters (model parameters) to estimate functions Ss. i == 1, 3,4 with theoretical optimality. These values affected by the complexity of the true regression function.
Table 1: Hyperparameters of BNN to estimate fi E Blor 2/ log 3 ((0, 1]).
L
W
0in
O2n
Tia
n2n
100
39
400
3.747 x 10 10-
8.443 x 10-1
8.719
10
1.280 x 10
1,000
43
1,100
1.505
10 10-234 234
7.597
10-1
9.534
10-1
4.651 x 10-2
10,000
45
2,950
1.508 x 10 283
7.954 x 10-1
9.826 x 10-1
1.733 x 10-2
Table 2: Hyperparameters of BNN to estimate f2, fa and fa E (),,
n
L
W
0in
02n
Tia
72n
100
41
250
8.669 x 10 172
6.779 x 10
7.957 x 10
2.042 x 10
1,000
43
500
1.259
x 10
10-205 205
5.028 x 10
8.977
10
1.022 x 10
10.000
47
1,100
1.963
10 10-256
4.894
10-1
9.535
10-1
4.642 x 10-2
In practice, models with smaller values L and W can estimate the true function with a sufficient precision. We present the numerical experiments with the four functions Appendix D
Conclusions
In this paper, we justify the theoretical properties of neural networks that are currently widely used. Specifically, we focused on the asymptotic properties of the Bayesian neural networks and extend the Hölder space results in Polson and Ročková 20188 to the Besov space. Theorem shows that the

---

Bayesian neural network with a spike-and-slab prior has consistent posterior with a near minimax rate the true Besov functions. Theorem eextends the result to the shrinkage prior. According to these theorems, the Bayesian neural networks have posterior consistency and (nearly) optimal rates by assuming proper prior. [Theorem 2] shows that even the smoothness of the true function is not known, the same optimality can be obtained by considering prior on the architecture of the neural networks.
The results in this paper are extensions of those in Suzuki suggested shrinkage prior for the Bayesian neural network as practical prior that can achieve the optimal rate while providing the uncertainty of the prediction, (Suzuki (2018) showed the theoretical properties of the neural networks in the Besov space via nonparametric regression based on B B-spline. We believe similar results can be obtained using the Bayesian LARK B-spline model [Park et al. 2021
2018 to the Bayesian neural networks. We
We obtained the theoretical results under the specific setting, fully connected networks with the ReLU activation function. As mentioned in |Section 4, the conditions to obtain the theoretical optimality is stringent for practical applications with large sample sizes. In [Therem 3 we mentioned the conditions that the shrinkage prior must satisfy and depend on the model parameters Ln; W., and B, For adaptive estimation like Theorem 2] it is a necessary to propose general shrinkage prior distribution that satisfies all conditions even when the parameters of the network are varying. The results of this study are just beginning and more theoretical research is needed for the Bayesian neural networks with industrial use, such as visual data analysis and natural language processing.
Acknowledgments
Kycongwon Lee and Jaeyong Lee were supported by the National Research Foundation of Korea (NRF) grant funded by the Korea governmen((MSIT) (No. 2018RIA2A3074973 and 2020RI44A1018207,
References
M. Bianchini and F. Scarselli. On the complexity of neural network classifiers: A comparison between shallow and deep architectures. IEEE transactions neural networks and learning systems, 25 55):1553-1565, 2014.
C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wierstra. Weight uncertainty in neural network. In International Conference on Machine Learning, pages 1613-1622. PMLR, 2015.
G. Casella and E. I. George. Explaining the gibbs sampler. The American Statistician, 4673):177-174, 1992.
I. Castillo, J. Schmidt-Hieber, and A. Van der Vaart. Bayesian linear regression with sparse priors. The Annals of Statistics, 43(5):1986 2018, 2015.
T. Chen, E. Fox, and C. Guestrin. Stochastic gradient hamiltonian monte carlo. In International conference on machine learning, pages 1683-1691. PMLR, 2014.
B.-E. Chérief-Abdellatif. Convergence rates of variational inference in sparse deep learning. In International Conference on Machine Learning, pages 1831-1842. PMLR, 2020.
G. Cybenko. Approximation by superpositions of sigmoidal function. Mathematics of control, signals and systems, 2(4):303-314, 1989.
O. Delalleau and Y. Bengio. Shallow vs. deep sum- cum-product networks. Advances in neural information processing systems, 24:666-674,
2011. 2011.
D. L. Donoho and I. M. Johnstone. Minimax estimation via wavelet shrinkage. The annals of Statistics, 26(3):879-92) 1998.
D. I.. Donoho and M. Johnstone. Ideal spatial adaptation by wavelet shrinkage. biometrika, 81(3): 425-455, 1994.
10

---

Y. Gal and Z. Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning. pages 1050-1059, PMLR, 2016.
Geman and D. Geman. Stochastic relaxation, gibbs distributions, and the bayesian restoration of images. IEEE Transactions pattern analysis and machine intelligence, (6):721-741, 1984.
E. Gine and R. Nickl. Mathematical Foundations of bridge University Press, 2016. ISBN 978-1-107-33786-22 doi: 10. 10.1017/CCB097811073336666 URL http::/ebookk: ctw.///roooo..ccmmm............ ottp://......r.///////a///////aa
Statistical Models, Cam-
I. Goodfellow, Y. Bengio, A. Courville, and Y. Bengio. Deep learning, volume 1. MIT press Cambridge, 2016.
A. Graves. Practical variational inference for neural networks. Advances in neural information processing systems, 24, 2011.
K. Horiik Approximation capabilities of multilayer feedforward networks. Neural networks, 4(2): 251-257, 1991.
D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
J. Lec, Y. Bahri, R. Novak, S. S. Schocnholz, Pennington, and J. Sohl-Dickstein. Deep neural networks gaussian processes. arXiv preprint arXiv: 2017.
H. K. H. Lee III. Model selection and model averaging for neural networks. PhD thesis, Carnegie Mellon University, 1999.
A. G. d. G. Matthews, M. Rowland, J. Hron, R. E. Turner, and Z. Ghahramani. Gaussian process behaviour wide deep neural networks. arXiv preprint 1 1221, 2018.
V. Nair and G. E. Hinton. Rectified linear units improve restricted boltzmann machines. In leml, 2010.
R. M. Neal. Priors for infinite networks. In Bayesian Learning for Neural Networks, pages 29-53. Springer, 1996.
R. M. Neal et al. Mome using hamiltonian dynamics. Handbook of markov chain monte carlo, 2(11):
2, 2011.
S. Park, H.-S. Oh, and Lee. Lévy adaptive b-spline regression via overcomplete systems. arXiv preprint arXiv:2101.1211 2021.
J. Peetre. New thoughts on Besov spaces. Number 1. Mathematics Department, Duke University, 1976.
N. G. Polson and Ročková. Posterior concentration for sparse deep learning. Advances Neural Information Processing Systems, 31, 2018.
Y. Sawano. Theory of Besov Spuces, volume 56 of Developments in Mathematics. Springer Singapore,
2018. ISBN 07799111388322 doi: 10. 1007/978 (00000/77-981-13-0000 18177898---------99- URL httthttt///nn springer
com/10 1000000007770 981 11088111111 13-066--9 0836-9
J. Schmidt-Hiebet Nonparametrie regression using deep neural networks with relu activation function. The Annals of Statistics, 48(4):1875-1897 2020.
Q. Song and F. Liang. Nearly optimal bayesian shrinkage for high dimensional regression. arXiv preprint arXiv:1712.08964 2017.
Y. Sun, Song, and F. Liang. Learning sparse deep neural networks with spike-and-se prior. Statistics & Probability Letters, I 180: 109246, 2022.
T. Suzuki. Adaptivity of deep relu network for learning in besov and mixed smooth besov spaces: optimal rate and curse of dimensionality. In International Conference on Learning Representations, 2018.

---

M. Telgarsky. Representation benefits of deep feedforward networks. arXiv:1509.08100 10011009.0111, 2015.
arXiv preprint
M. Telgarsky. Benefits depth in neural networks. In Conference on learning theory, pages 1517-1539. PMLR, 2016.
M. Vladimirova, J. Verbeek, P. Mesejo, and Arbel. Understanding priors in bayesian neural networks the unit level. In International Conference on Machine Learning, pages 6458-6467. PMLR, 2019.
M. Welling and Y. W. Teh. Bayesian learning via stochastic gradient langevin dynamics, In Proceedings of the 28th international conference on machine learning (ICML-I1), pages 681-688. Citeseer, 2011.
C. Williams. Computing with infinite networks. Advances in neural information processing systems, 9, 1996.
D. Yarotsky. Error bounds for approximations with deep relu networks. Neural Networks, 94: 103-114, 2017.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 5]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section 3)
(b) Did you include complete proofs of all theoretical results? [Yes] See Section 3 and Appendix C
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work existing assets, did you cite the creators? [Yes] See supplementary material.
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any assets either in the supplemental material as URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
12

---

(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "hdXKYNRtVYh", "reviewer": "Reviewer_Z4is", "summary": "This work extends the results of Polson and Rocková [2018] on the minimax contraction rate of Bayesian ReLU network from the Hölder space to the more general Besov space. Such result brings more freedom to the underlying functional form behind the data with similar minimax contraction rate guarantee. Beside the spike-and-slab prior, the result with shrinkage prior is also given but with more computational efficiency. ", "questions": "The underlying functions are usually assumed to be continues when using BNN for real-data modelling. Since the advantage of Besov space is its ability to include non-differentiable functions, can you find one or some real data that needs the functions from Besov space rather than Hölder space or even L^p space? \n\nCan you highlight the challenges on the proposed extensions? ", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strong point of this work is the ability to extend the functional form modelled by Bayesian ReLU network and its posterior contraction rate guarantee. The motivation is clear, and the idea is well presented.\n\nHowever, as the authors said, this is an extension of existing works Polson and Rocková [2018] with similar backgrounds, settings, and results, and the derivative looks a little straightforward, so the innovation is limited. The authors did not highlight the challenges or difficulties in how to extend the results from the Hölder space to the more general Besov space. \n\nAnother main weakness is the evaluation. Only one simple example is given to show the ability of designed prior for BNN regression. It would be better to compare with other options, like iid Gaussians, spike-and-slab, and the Gaussians priors with learnable hyperparameters. In practice, we normally learn the parameters of priors from the data as well rather than just fixing them while we using BNN for data modelling. Is there any effect from such learnable priors? \n\nThe underlying functions are usually assumed to be continues when using BNN for real-data modelling. Since the advantage of Besov space is its ability to include non-differentiable functions, can you find one or some real data that needs the functions from Besov space rather than Hölder space or even L^p space? \n\nSome minor problems include: the symbols should be explained, like “A” in (9), any conflict between “L_1” in Line 108 and “L_n” in (10), Line 158, “a expectation” and Line 185, “acontinuous”\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Y5z3ztdZWOM", "reviewer": "Reviewer_x58D", "summary": "The paper proposes the theoretical results for Bayesian neural networks in the Besov space. It extends the results from the previous work in Holder space into the Besov space. It shows that Bayesian neural networks with the spike and slab priors and further relaxes it to be just shrinkage priors converge to the true regression function in the Besov space. ", "questions": "My main concern with the paper is its clarity and I don’t have questions on that. \n\nThe other smaller concern is regarding the numerical example, so maybe the authors can answer questions about it, however, as this is not the major concern, the authors should be free to give priority to questions to other reviewers in their rebuttal if they would help to resolve some major concerns from the other reviewers.\n\n1. Wouldn’t MCMC methods with true posterior be more appropriate for the purpose of the numerical example? \n2. What can the authors say about the claims about practicality of the method?\n", "limitations": "There is no discussion on this, but since the work is theoretical it is not the major issue", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Update after rebuttal:** I would like to thank the reviewers for their incredible job during the rebuttal and by improving the submission very much. I believe that the clarity of the paper has drastically improved and therefore I am raising my score. Even though after the correction of the theory the numerical example does not satisfy the theory now I believe the submission provides an interesting step towards better theoretical guarantees of BNNs.\n\n=================================\n\n**Originality:** \n*Strengths:* The main results appear to be novel and an interesting extension of the previous results in Holder space into the Besov space. \n\n**Quality:** \n*Strengths:* I am sorry but I have to admit that I cannot assess the actual theoretical contribution of the paper as it is beyond my comfort level zone, but also because in contrast to some people who prefer formulae over word description I would prefer the latter to better understand the formulae. Please see comments on this below in Clarity section.\n\n*Weaknesses:* Though I appreciate that the main contribution of the paper is theoretical, the numerical example provided is not too satisfying. \n\n* The method is only used on 2 synthetic examples with analytical functions\n* The main theoretical results are about the posterior, but the numerical experiment uses the Bayes by Backdrop for inference, which is implementation of the variational inference, i.e. approximation of the true posterior. Wouldn’t MCMC methods with true posterior be more appropriate for the purpose of the numerical example? \n* There are several claims that the paper proposes a “practical” model, however, there is no empirical evidence on that. Some discussion on actual computational time used and how feasible the inference with the considered priors would be appreciated\n\n\n**Clarity:** \n*Strengths:* The text where available is mostly well written and easy to follow. The paper tries to tell the full story with a proper introduction rather than jumping straight into it as if a reader has just finished reading the previous work and doesn’t require any introduction - the approach used in many recent papers. \n\n*Weaknesses:* There is not so much text in the paper but a lot of formulae. I appreciate there are people who prefer formulae over words, but there are others who much better read words over formulae. Word description of formulae would be much appreciated. Also, the notation is not very careful which does not help further with readability of this lot of maths. \nIf the space is an issue, some general paragraphs from the introduction can easily be reduced or removed to save some space. Figures 2 and 3 can also lose the histogram part to save some space. (The histogram part can go into the appendix also)\n\n**Significance:** \n*Strengths:* The paper addresses a very important problem of finding theoretical guarantees for deep learning, Bayesian deep learning in particular in this case. It does seem to make a step further in extending those guarantees (by providing them in a bigger Besov space). \n\n**Summary:** Readability of the paper, namely the lack of words over maths, is the main issue with this submission in my opinion. So much so that it is difficult to assess the quality of the paper. It seems the paper has significant and original results but it should also be easier to understand these results while reading the paper. Moreover, the numerical example can be largely improved. \n\nSpecific comments/suggestions:\n1. Line 10. “In other words” - a bit inappropriate use here as it hasn’t been said before anything about practicality, only about asymptotic properties\n2. “Posterior consistency” is never properly defined in the paper\n3. Line 96, “u \\in {0, 1, …, d}”?\n4. D^u is used for u-th derivative, and then D_n is used for a dataset. It is a bit confusing though technically it is not overlapping notation. It is probably better to use another letter for a dataset\n5. Equation between lines 120 and 121. Both D^u and d^u is used. What is the difference?\n6. Eq. (7) - \\eta is not defined\n7. Line 152, \\delta is already used for the Dirac function\n8. Theorem 1, M_n is not defined, how does it depends on n?\n9. Line 204, N is already used, probably it is better to used \\mathcal{N} for the normal distribution \n10. Figures 2-3. Are dots training data points? Left column plots can used “x” and “y” labels for axis, also subplots without labels look too strange for me. It seems they may have “(a)”, “(b)”, … labels\n11. Line 218. “shrinkage prior” - “spike-and-slab prior” was meant?\n12. Lines 229-230, “Experiments … will also be of great help…” - not clear, is it suggested as future work? It would indeed be nice to have some comparison in the current submission.\n\n\nMinor:\n1. Line 21, “Before observing the data, A” - lower case for “A”\n2. Line 36, “are” -> “were”\n3. Line 69, “than THE spike-and-slab prior”\n4. Line 82. “GP” - acronym is not introduced\n5. Line 86, “to solve THE optimization problem”\n6. Line 88, “with THE (nearly) optimal convergence rate”\n7. Line 148, “Let” - lower case for “L”\n8. Line 211, “were” -> “are”", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "UU1Amylx77", "reviewer": "Reviewer_T5q1", "summary": "This paper present a proof of Bayesian posterior contraction rate in Besov spaces for Bayesian neural networks priors for the regression problem on $[0,1]^d$. More specifically, two priors are presented : spike and slab and shrinkage (the latter being easier to implement). Under the frequentist assumption that the true regression function $f_0$ belongs in a Besov space $B_{p,q }^s$, it is shown that the two priors achieve minimax rates of convergence (up to log terms) with a (non adaptive) choice of prior parameters (which includes a choice of depth, width & sparsity of the network). Moreover, it is shown that an adaptive rate of convergence over all smoothness index $s>0$ is possible for a spike and slab prior with an additional prior on the architecture. This work extend previous results for Bayesian posterior contraction in Hölder spaces and frequentist minimax theory in Besov for neural networks.", "questions": "I have two questions :\n\n- in appendix C1, proof of lemma 3 (35) : can you quickly justify how you lower bound the norm ?\n\n- in the proof of theorem 1 (and subsequently in the paper for the other analog proofs), it is not clear how the prior on the connectivity pattern (ie on $gamma$) appears in the proof of the prior thickness result (43)-(44)-(45), should there be a $\\binom{T_n}{S_n}$ factor somewhere or am I making a mistake ?", "limitations": "From my point of view the main limitation of the paper is that adaptive result is only proven in the case of the spika and slab prior, and that the numerical expriments are conducted on toy examples with known smoothness levels. It could be interesting to, at least, discuss why you are only presenting an adaptive result for the first prior.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper is a strong theoretical results closing a gap for posterior contraction of BNN in Besov spaces. Concerning the priors, it presents a new (computationally more efficient) shrinkage prior that is shown to lead to good posterior contraction rates, which is (to the best of my knownledge) a new result even for BNN in Hölder spaces.\nMoreover, I think that paper is overall wery clear and well written. However, ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept