Posterior Contraction for Sparse Neural Networks in Besov Spaces with Intrinsic Dimensionality
Authors: Kyeongwon Lee
Lizhen Lin
Jaewoo Park
Seonghyun Jeong
Venue: NeurIPS 2025 poster ()
Keywords: posterior contraction
Bayesian neural networks
Besov space
intrinsic dimensionality
Primary area: theory

Abstract: This work establishes that sparse Bayesian neural networks achieve optimal posterior contraction rates over anisotropic Besov spaces and their hierarchical compositions. These structures reflect the intrinsic dimensionality of the underlying function, thereby mitigating the curse of dimensionality. Our analysis shows that Bayesian neural networks equipped with either sparse or continuous shrinkage priors attain the optimal rates which are dependent on the intrinsic dimension of the true structures. Moreover, we show that these priors enable rate adaptation, allowing the posterior to contract at the optimal rate even when the smoothness level of the true function is unknown. The proposed framework accommodates a broad class of functions, including additive and multiplicative Besov functions as special cases.
These results advance the theoretical foundations of Bayesian neural networks and provide rigorous justification for their practical effectiveness in high-dimensional, structured estimation problems.

Posterior Contraction for Sparse Neural Networks in Besov Spaces with Intrinsic Dimensionality
Kyeongwon Lee'
Lizhen Lin' Jaewoo Park
Seonghyun Jeong"*
Department of Mathematics, University of Maryland
DDeparmeent of Statistics and Data Science, Yonsei University (kwlee, 1 Lizeen0 @umd edu
jyonn .c.
Abstract
This work establishes that sparse Bayesian neural networks achieve optimal pos- terior contraction rates over anisotropic Besov spaces and their hierarchical com- positions. These structures reflect the intrinsic dimensionality function, thereby mitigating the curse of dimensionality. Our analysis shows that Bayesian neural networks equipped with either sparse or continuous shrinkage priors attain the optimal rates which dependent the intrinsic dimension of the true structures. Moreover, we show that these priors enable rate adaptation, allowing the posterior to contract at the optimal rate when the smoothness level of the true function is unknown. The proposed framework accommodates broad class of functions, including additive and multiplicative Besov functions as special cases. These results advance the theoretical foundations of Bayesian neural networks and provide rigorous justification for their practical effectiveness in high- high-dimensional, structured estimation problems.
the underlying
Introduction
Neural networks (NNs) have been widely used to extract features from complex datasets, such as visual recognition and language modeling [U]. Due their approximation ability [2-22], NNs exhibit remarkable flexibility in representing complex multivariate functions. Given their network structure, NNs are trained by minimizing empirical risk defined through suitable loss function, which can be viewed as maximum likelihood estimation from a statistical perspective [X]] However, such fre- quentist approach may lead miscalibration and overeonfidence [X], particularly when the model is trained on
samples [[T]]. On the other hand, the Bayesian approach quantifies predictive uncertainty via the posterior distribution [IлI, [12] and can improve calibration and robust- ness in many cases [[33, 144) Another key advantage of the Bayesian perspective is that, unlike their frequentist counterparts, Bayesian neural networks (BNNs) can easily achieve the optimal rate of convergence without knowledge of the smoothness of the function. This property, known as rate adaptation, is inherently attained by the nature of Bayesian inference.
To capture the flexibility of NNs, it is desirable to consider function classes that are richer than the intuitively smooth ones, such as Hölder or Sobolev spaces. In particular, Besov spaces, which encompass non-smooth or even discontinuous functions. are well suited for this purpose. For exam- ple, images often exhibit local inhomogeneities and sharp edges naturally represented within Besov spaces [[5]]. Moreover, additional flexibility arises from accounting for intrinsic dimensionality. which helps explain the robustness of NNs to the curse of dimensionality. For instance, images frequently possess low intrinsic dimensionality [[6]]. Such an intrinsic dimensional structure can be implicitly modeled through anisotropy or composite function constructions. Although frequentist
"Coressponiing author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

approaches have extensively investigated theoretical properties in these complex settings [| [& Bayesian analyses remain largely limited to simpler cases IT9 22 This study aims fill this gap by establishing that BNNs achieve optimal posterior contraction and exhibit rate adaptation under the fully Bayesian paradigm in such complex scenarios.
1.1 Related works and our contribution
Rates of convergence provide fundamental means of assessing the quality of estimation procedures in both frequentist and Bayesian inference. While frequentist analyses focus on the convergence rates of estimators, Bayesian approaches examine the contraction rates of posterior distributions, which describe how quickly the posterior concentrates around the truth. In the deep neural network (DNN) literature, such notions of optimality have been investigated under a variety of settings from both frequentist and Bayesian perspectives.
Frequentist works. Under the frequentist paradigm, Schmidt- [8] showed that carefully constructed DNN estimator achieves the near-minimax optimal rate over Hölder classes, provided that the network architecture has sufficient depth and sparsity adapted to the underlying function complexity. Their result extends to composite structures, encompassing a wide range of structured function classes, including additive models. Suzuki [[Z]] further extended these results to Besov spaces, and Suzuki and Nitanda [[]] demonstrated that DNNs mitigate the curse of dimensionality by adapting to anisotropic smoothness and composite structures Besov spaces. In contrast to the aforementioned works that utilize sparse networks, Kohler and Langer [23] investigated dense NNs for composite Hölder spaces. Collectively, these frequentist results demonstrate that DNNs can achieve optimal convergence rates under appropriately specified conditions. However, attaining such optimality requires careful tuning of the network architecture to the underlying function complexity, and adte-adeetive results are currently not available.
Bayesian works. The Bayesian literature remains comparatively limited. A pioneering contribution by Polson and Ročková [24]] established that BNNs equipped with spike-and-slab priors achieve op- timal posterior contraction over Hölder classes. This work was extended Besov spaces by Lee and Lee [20]], who considered both spike-and-skee and-slab and continuous shrinkage priors, although the appli- cation of shrinkage priors remains largely limited. Kong and Kim [19]] obtained similar results using dense networks with non-sparse priors, further extending the theory to composite Hölder spaces. To the best of our knowledge, the only Bayesian study that considers anisotropic Besov spaces Egels and Castillo [22]]. However, that work primarily employs the fractional posterior approach [22]] 25), and the standard posterior under the fully Bayesian framework has been investigated only under strong restrictions. Morcover, they focused on Hölder classes in the composite setting, and, to knowledge, no Bayesian study has yet considered composite structures within anisotropic Besov spaces. Despite this limited scope, these Bayesian approaches achieve rate adaptation, attaining optimal posterior contraction without prior knowledge the smoothness level.
Our contribution. We establish that sparse BNNs achieve the near-minimax optimal rates over anisotropic and composite Besov spaces. This result demonstrates that BNNs adapt to intrinsic dimensional structures, thereby avoiding the curse of dimensionality. We show that sparse BNNs can accommodate a broader class of realistic functions that have not been fully addressed in earlier works. Compared to Egels and Castillo [22], we adopt the pure Bayesian framework, relying on the standard posterior distribution for inference. This choice aligns more closely with conventional Bayesian practice and facilitates broader acceptance within the Bayesian community. Furthermore, we show that sparse BNNs adapt to the underlying model complexity and attain the optimal rate for the target function without requiring oracle knowledge. As result, users are not required to specify the exact network architecture; rate adaptation is achieved through an appropriately designed prior distribution. A summary of related works and our contributions is provided in Table M
Preliminaries
2.1 Setup
Notation. For a E IR, let [a] and [a] denote the floor and ceiling functions, respectively. For E N, the notation [n] stands for the set (1, ... ,n). For a, b R. we write a and A b to denote max (a, by and min(a, b), respectively. For real vector ||ul| denotes the f norm for p E [1,o

---

Table 1: Summary of related works and this study. The abbreviations Iso., Aniso., and Ada. denote Isotropic. anisotropic, and adaptation, respectively.
Single Function
Composite Function
Study
Approach
Architecture
Function Space
Ada.
Function Space
Ada.
Schmidt- -eeeberrr [S]
Frequentist
Sparse
Iso. Hölder
Iso. Hölder
Kohler and Langer [24]]
Frequentist
Dense
Iso. Hölder
Iso. Hölder
Suzuki [[-]]
Frequentist
Sparse
Iso. Besov
X
Suzuki and Nitanda [
Frequentist
Sparse
Aniso. Besov
X
Aniso. Besov
X
Polson and Ročková [
Bayesian
Sparse
Iso. Hälder
Kong and Kim [0
Bayesian
Dense
Iso. Hölder
Iso. Hölder
Lee and Lee [
Bayesian
Sparse
Iso. Besov
Egels and Castillo
Bayesian" "
Dense
Aniso. Besov"*
Iso. Hölder
This work
Bayesian
Sparse
Aniso. Besov
Aniso. Besov
Egels and Castillo [12] primarily employ the fractional posterior circumvent issues model complexity.
** Egels and Castillo [12]] impose stronger restrictions on the smoothness parameter than other studies on anisotropic Besov spaces (800 Section 5 for details).
and ||e|| denotes the number of nonzero components. For measurable function ʃ [0, 1] - IR and measure 14, define [ (o..1 df) for p E (o). When μ is the Lebesgue measure, we write Ilfllzz for brevity. For = oo, we define [[ = ess sup [f(z)). We also define the supremum norm by | = su f(x))- The Dirac delta at zero is denoted by do, and the indicator function of set A is denoted by T(4). For sequences On and bn, we write an S bn and On Z bn mean that on Cbb, for some universal constant > O. If an bn S an. we write an x bn. Let = {f: [0, 1 + R; |l S 1} denote the family of uniformly bounded functions. For a normed space F, (F) denotes the unit ball of F.
Model. We consider a nonparametric regression problem with a d-dimensional input variable X; e [0, 1]dd and an output variable Y; R, for i E [m]. The observations D, {(x,, Y)) independent and identically distributed according to the model,
Y = fo())) Es. X Px ~~~(00,
ie [n].
where fo [0, 1jd R is the true regression function, oă distribution of X, We denote the joint distribution of D,, under this model by p(m)
the noise variance, and Px the
fo.00
Neural network. We denote the ReLU activation function by {()) and use the same notation for its
vectorized version. The parameter space of L-layered NNs with B-bounded and S-sparse weights is defined as
OL, = {0=
(0:=((), (W()) b())
W(++1) (((++)))) w(() E Bdddd 1, 0 E R
di = D, LE [4]. do = d. d = 1, 11011 S S, |101 S B).
o (W + 6()) : 10 R be the input-output mapping of an L-layered NN. We define the corresponding feedforward NN class as (LL D,, B) = {clip o fo 0 E O(L, S, B)}. where clip(x) min 1, max =x) clips its input to the interval []. We also define the unbounded sparse and unbounded dense pa- rameter spaces as O(L, D)
Let fo()) (w(L+1(() b((+1))
limg-xx O(L, D, B) and O D) lims O(L. D, S). respectively. By further taking union over the width, we define O(L) = U OL, The corresponding NN classes are defined analogously as ((L, D,S) lim 4 LL, (T.,, D) = lims (T., D, B), and (T.) = US (L., D).
2.2 Anisotropic Besov spaces
We first consider anisotropic Besov spaces, which accommodate varying smoothness across different directional components [2A 18]. Let s = ( sa) E Rd 0 < p.q S xx, and r maax 85] 1. For h E Rdd, the r-th order difference is defined as (z) = E (11-1)- + jh) v, x + rh E [o, 1jd, and otherwise. Given (L,...., la) E Rd the anisotropic modu- lus of smoothness is defined as (f,) SUPRER*/M,ISt, [A())) The anisotropic Besov
3

---

04
Figure 1: We illustrate two example functions, (x) = I({21 E [1/2, 1}}) + sin and
fa(2) = [1 1/2| + (22 1/2)², and their rotated counterparts fi and f2.
seminorm is then defined as Il.fll = llfll + I1
where
[E [2t Wr.p 1 (2 SUPR20 [2 (J: 2 k/ak)))
2-k//3))]]
1/4
if <
Il
if xc.
The anisotropic Besov space B2.g is the collection of all functions f E L. such that ||/ is finite. For anisotropic smoothness parameter 8, define the smallest smoothness largest smoothness as 3 = max; 8j, and the intrinsic smoothness (exponent of global smoothness) as 3 (Σj=1 sj )- In the special case where (su..... so) for some so 0, the space Bp.q reduces isotropic Besov space. Besov spaces generalize classical notions of differentiability and continuity and are more flexible than Hölder spaces, which can be viewed as particular subspaces of Besov spaces. For further discussion of the continuous embedding properties, see Remark [
minj 8j, the
well known that the minimax optimal rate of convergence over isotropic Besov classes with smoothness 8o Sb/ (2((t+)
[4 28 For anisotropic Besov spaces, the minimax rate is given by ==(/(2++ [29 GDJ, which can also be expressed ==((2=+""", where d' := 8/5. This form resembles the minimax rate over isotropic Besov spaces with smoothness & and dimension d', in which the anisotropic space is continuously embedded. In this sense, d' can be interpreted as an intrinsic dimension associated with anisotropic Besov spaces [18]. For example, the two functions
fi and fz in Figure II belong 1,00 for any 0 < 81 < and 82 81- The anisotropic
Besov space B((2)52) is continuously embedded in the isotropic Besov space with $1. By accounting for anisotropy, the rate exponent be improved from $2/(2s1 2) to 5/(28 + 1), where 3 $182/ ($1 + $2). The intrinsic dimension satisfies d' $1/3- $1 $2 + < 2, indicating a reduced effective dimension compared the ambient dimension d
2.3 Composite Besov spaces
Beyond anisotropic spaces, we consider composite function spaces in which functions are rep- resented as hierarchical compositions of simpler components. For a composite function ʃ [0, 1] + with composition depth H, each intermediate function fn maps between successive spaces, potentially involving dimension reduction through sparse connectivity. This com- positional structure naturally aligns with the layered architecture of NNs, where each layer progres- sively transforms its inputs into higher-level representations. Figure M1 presents examples in which rotated functions can be interpreted as compositions of Besov functions with affine transformations. Compositional function spaces offer useful framework for understanding the ability of NNs to approximate high-dimensional functions without suffering from the curse of dimensionality.
fH o.... fi
Let do (t(1) ...,,... ((H)) denote the effective dimensions, where each {(h satisfies
(d) dd!).... d(!)) be a sequence of dimensions with (0)) and ( 1. Let
{(h) < d(h-1),
Let 80 = (1(1)
....., ....,)(H) (HH) be sequence of anisotropic smoothness vectors, where each ((h) E (((a)) We define the composite anisotropic Besov space B Bde.to,so as the collection of composite functions [0, 1]d - IR such that the intermediate functions St. (Ju,1,..., (h,dfr)),,,,d(n))
f= SH (0,1] [0, 1j for h 1,.... H 1, and SH [0, 1]d(H-1)
- R satisfy the following condition: for each h and each E [d(h)], there exists subset This [d with Thul ((h)

---

(0
122
92
Σ (!2)
(
(()
(a) Additive structure
(b) Multiplicative structure
Figure 2: Illustration additive ((()== (x:) and multiplicative (f(x) = II. 9e (x()))
composite Besov functions. Each component function 9i depends single input dimension (L(1) = 1), although the ambient dimension d( = d may be much larger.
p.g and function fr.j E U(B:")) such that (x) fn,(a (z),))) where æ, denotes the subvector
of x restricted to the coordinates in I. Additive and multiplicative Besov spaces serve as simple examples of composite Besov spaces, as illustrated in Figure
Note that Suzuki and Nitanda [T]] did not impose effective low-dimensionality in their definition of with the composite Hölder spaces introduced by Schmidt-Hieher [8], and it is well suited for analyzing functions approximated by NNs, where dimension reduction may occur within layers. By incorporating anisotropy within each fr. the framework captures -imeensiiionll structures even when the ambient or latent dimensions high. This interplay between compositional structure and anisotropy enables efficient function representation.
composite Besov spaces. In contrast, our definition aligns
Main results
In this section, we present our main results on posterior contraction. Table summarizes the corre- sponding assumptions and theorems for the anisotropic and composite anisotropic Besov spaces.
3.1 Posterior contraction in anisotropic Besov spaces
We first establish thcoretical results for anisotropic Besov spaces. The following assumptions imposed.
(A1) The true regression function fo satisfies So U( UB for some < p.q & and
E R such that (1/p - 1/2)+ < 8.
(A2) The distribution Px has bounded density px such that ||x ooo S for some constant
R >> 0.
(A3) The true standard deviation an satisfies a < ag < o for some constants and o, and the prior for is supported [a, ø] with positive density throughout.
Assumption [A]] states that fu lies in a bounded Besov class. The condition (1/p - 1/2)+ <
aligns with those in other theoretical studies on Besov spaces [20 I1,, 18), whereas Egels and Castillo [[2]] impose stronger requirement 1/p < 3 to achieve sharper approximation result. Assumption [A2] is minor given that the dimension is fixed. Assumption AK) ensures that the prior for is supported on compact interval containing ag- We first show that the optimal rate is achieved by suitably specified spike-and and shrinkage priors, and then demonstrate that rate adaptation can be attained for both priors with slight modifications.
3.1.1 Spike-and-slab prior
Our results rely the fact that for every fo U(Bgg). there exists a NN approximator f & Din, Sin, Bi) achieving the optimal approximation error, where the network parameters satisfy
Iin x logn, Din = Nn. Sin = N₂ logn B x
(2)
5

---

Table 2: Summary of the main results. Along with the listed assumptions, all results also require
Assumptions [22] and [33] which are taken as common assumptions.
Anisotropic Besov
Composite Anisotropic Besov
Prior
Assumptions
Result
Assumptions
Result
Spike-and-
Theorem
A47
AS
Theorem
Shrinkage
Theorem
A4
A5
Theorem
Adaptive Spike-and-Slan
132
Theorem
A44
AS
Theorem
Adaptive Shrinkage
Theorem
A4
AS
Theorem 5T
with N,, 122++311) These network parameters depend on the unknown Besov parameters and p. See Lemma Bd and Remark BGG for details.
We place a spike-and-slab prior over @LLnn, Di, Sin) using the network parameters in (2). This implies that the network structure is determined based on the smoothness parameter and thus the procedure does not attain rate adaptation. A spike-an spike-and-slat prior for is given by
T( 7, L,,,
II j=1
(z (1-7)))60()))
<(> L,D,S) = (( e (0.1)", 11 = =S),
where Rsz. is the slab density for the nonzero components and = (L, D)J. The slab distribution is required to satisfy the following assumptions.
(B1) log Jups,, St (u) du S - -K,
n for any Kn - co.
(B2) log inf wSB (u) Z -llog
Assumption BT implies that RSL. has exponential tails on both sides. From a nonparametric Bayesian perspective, this condition is necessary to control the prior mass outside a chosen sieve. Assumption [B2] requires that Se to place sufficient mass around the NN approximator, which essential for ensuring adequate prior concentration in Kullback- independent n and bounded away from zero on [-B1, B1], then the assumption holds trivially.
neighborhood. If Æsz is
Example 3.1 (Uniform slab prior (20, 21]).. #s... the density of the uniform distribution
U--Cu, C.) for C, > B1, then TsL satisfies Assumptions [BT] [B2].
Example 3.2 (Gaussian slab prior). If #sL. is the density of zero-mean Gaussian distribution with fixed variance, then i.s satisfies Assumptions [BT] [B2]
Example 3 is simple but has the drawback that the prior depends B1. Example 3 this issue. We now present the posterior contraction result.
mitigates
hold. and that the prior distribution in (B) is placed over ((Lin) Dins Sta). Assume further that the slab density tsL satisfies Assumptions [BT] [BTH[B2] Then, the posterior distribution concentrates at the rate
Theorem 3.3 (Spike-and-slab prior). Suppose that Assumptions AT A3
Gn = -=/2((2++1) (log n/8/2, in the sense that
m((1o)) E (Lin Sin) x La, σ) - + +10-- -ool MM Da) -
in p(n)
fa.o probabiiity as n + do for any M, - sc.
Proof. See Appendix
The contraction rate 6n near-minimax rate and only depends on intrinsic smoothness of function. Suzuki and Nitanda [[S] showed that the empirical risk minimizer under the squared loss achieves the same rate e, Therefore, BNNs attain the same theoretical optimality as their frequentist counterpart for anisotropic Besov spaces.

---

Remark 3,4. The exponential tail condition in [BT] is imposed to effectively control model com- plexity, as measured by the metric entropy of suitably chosen sieve [3IL [2] If one instead uses a polynomial-tailed density, the entropy calculation can be bypassed using the Vapmik-Chervonenkii
dVm)-dimension technique [33]] In that case, the posterior contraction rate slightly degraded to 20-8/(251)) (log n)². See Appendix CD for further details.
3.1.2 Shrinkage prior
The main bottleneck in using spike-and-slab priors lies in the computational burden introduced by the point-mass component of the prior. To address this issue, we extend the results continu- ous shrinkage priors, which offer more computationally efficient alternative. These priors retain the oparsityppromoting nature of spike-and-slab priors while avoiding the complexity of variable- dimensional posterior inference. This leads to scalable and practical Bayesian procedures suitable for high-dimensional settings, where computational tractability essential.
Using the network parameters in (L)), we place prior over ( Din). That is, sparsity is not imposed explicitly but is instead induced implicitly through the use of shrinkage priors. We express a shrinkage prior as
T T(0 L) = II RSH (0,),
(4)
==1
where 7sH is density on I and T = 19(L, D)). Specifically, #Sи is assumed to satisfy the following assumptions.
(C1) log JujxK. (u)du S - Kn for any Kn + oc.
(C2) log DgginfuusB (u) Z -logn)²
(C3) log Suppan #sH #sH(u)dte (u)du CCallog n)² for sufficiently large constant CA 0, where a,
244n log
[C] imposes an additional constraint that serves as a continuous analogue of the spike component in spike-and- slab priors. This assumption is indeed restrictive: for rapidly decreasing an. it requires the prior to concentrate nearly all its mass on [-an, on). As a result, widely used shrinkage priors such as the horseshoe [[44]] do not satisfy Assumptions (CII-[C3) directly. Nevertheless, the assumptions can be satisfied by designing priors that place most of their mass near zero, as required in Assumption [e] while also satisfying the exponential tail decay in Assumption (CT), The following examples illus-
Assumptions CI serve similar roles Assumptions B B21
Assumption
trate such constructions, with verification given in Appendices CIll and C.L.
Example 3.5 (Relaxed spike-and-she [20]). Let 5Pk denote a sub- beibulib density with tail index
< k < [3S] satisfying Jiil> (u( du S CI CCp((_C for any > and some
constants C1, C2 > O. For Cn > B1, define #SH as
1
11 Tin
#sH (u) Tin
p/
+ T2n Uw; -u; Cu),
(5)
Oin
where Tin = 1 - (Coggn))) T2n = e 20' (Jogn)?
Jin = a, (2Ca(logn)))) and a sufficiently [
large CA > 0. Then, #sH satisfies Assumptions CCT
Example 3.6 (Relaxed spike-and-slab; Gaussian slab). If the uniform component in (5) is replaced
by the density of zero-mean Gaussian distribution with fixed variance, then सडम satisfies Assump- tions CTT-IOS)
sub- Weibull density SP corresponds sub-exponential density. For k = 1/2, it corresponds to sub-Gaussian density. In particular, if you. is chosen Gaussian density, the prior in Example B.f becomes a Gaussian mixture prior.
Remark 3.7. For
Note that the mixture weight Tin in (5) approaches 1. Although this may be practically undesirable, ensures that Assumption [C3] is satisfied. Similar constraints have also been adopted in the liter- ature sparse BNNs with shrinkage priors [36, 20J. We now formalize the contraction result for shrinkage priors. The following theorem shows that BNNs equipped with shrinkage priors achieve the mear-minimax rate.

---

Theorem 3.8 (Shrinkage prior). Suppose that Assumptions [AT] [A3] hold, and that the prior
distribution in (H) placed over ((Lin, Din). Assume further that the continuous density
7sH satisfies Assumptions [CTHIC3) Then, the posterior distribution concentrates at the rate 'n = 2on-87(22) -02-8)22++1))))) (log in the sense that
II (((1,o) (L Día) o) |ff SollCA(PX) + +002-
-og1 - Maan Dz)) -
in p(a)
f0.00 probabiiiiy + oo for any Mn -
Proof. See Appendix
3.1.3 Rate adaptation
In Section G and Section 1.22 the priors depend on the smoothness parameter 8, indicating that the procedures are rate-adaptive. By placing suitable priors the network parameters, the results can be extended to achieve rate adaptation. Therefore, the optimal contraction rate is achieved without knowing the characteristics of the true function.
Specifically, instead of the network parameters in (2), we consider the depth L. [Cr, log n for a sufficiently large Cr, > and the priors an and TT on the width D and sparsity S, respectively, given by
(D) & e -A (S()) xc e SA
(6)
for constants AD > 0 and Àg > 0.
Theorem 3.9 (Adaptation). Suppose that Assumptions [T) [33 hold, and that the prior distribu- tions satisfy either one of the following conditions:
(i) The priors TD and T in (6) are assigned to (D,S), S). respectively, and the prior in ( is placed
over O(L,, D, S) conditional on (D. The slab part isL satisfies Assumptions B BB2)
(ii The prior nn in (6) is assigned to D, and the prior in (H) is placed over @ D) conditional
on D. The continuous density #Sh satisfies Assumptions (CT
If Ct is sufficiently large that Ln 2 Liw then the posterior distribution concentrates the rate
En = n 3/ (log n) n)3/2, in the sense that
((( E(I)) o] IIf follx>(Px) + +102 -02-000 Mpcn Dr)) -
in pla)
foro, prrobability as + oo for any Mn + oc.
Proof. See Appendix
In both cases, the priors placed over ((Ln), and the procedures achieve rate adaptation once Cr. is chosen sufficiently large. Although the priors do not depend on the Besov parameters for the true function, the conditions 7.t. and 7.n still involve these unknown quantities. Nevertheless, the required conditions can be satisfied without explicit knowledge of them. Specifically, if Example 3 and Example 3 are defined with a sufficiently large constant C, such that C.2, B1. the conditions hold regardless of the value of B1, which depends on the Besov parameters. In contrast, Example 3, and Example B.G are already independent of these parameters. Therefore, rate adaptation can be readily achieved in practice.
3.2 Posterior contraction in composite Besov spaces
In this section, show that BNNs attain the minimax optimal rate over composite Besov classes. In other words, we demonstrate that BNNs outperform traditional statistical approaches by achieving the minimax optimal convergence rate. Following Suzuki and Nitanda [18], we assume that the true regression function satisfies one of the following two structural conditions.
(A4) The true regression function fo satisfies fo E Bo nUB for some 0 < p.g S co, do, to. and 8.. such that 3(1) > (1/p 1/2)+ and g(h) > 1/p for h = 2,.., H.
p.4
8

---

(A5)
The true regression function fo is defined as fo = fz fI, where fi = A A- with A e
Rdd xa and E
Rd such that d(l) v d( and f1()) E [0, 17 for all x E [0. 11d0 and
2) f2 U U( nuB with 0 p. S oc, and g(2) u such that 5(2) (1/p 1/2)+
Assumption [ corresponds to the composite Besov space defined in Section L We note that our assumptions are comparable to those in Suzuki and Nitanda [[8]. Assumption [55 defines nontrivial Besov class involving
affine transformation, which is specific case of
the general class p-q 1 However, it is not contained within Assumption [A4] owing to the
relaxed smoothness requirement 3(2), Under Assumption [A4), define (*) ghhh) /5(h),
(*() 5(h) I ((s()) (-((k) /p) 1}, h E [H] and h" arg mag 5*(h), We define the
intrinsic dimension as t" = = and the intrinsic smoothness 3 = g-(h"), For Assumption [A5] we adopt the same definitions with h- 2. Suzuki and Nitanda [TB]] obtained lower bound of the minimax risk that roughly matches =25122"++1 The following examples illustrate practically relevant functions that are covered by either Assumption [A4] or Assumption [A]]
Example 3.10 (Additive function: Figure Suppose fz) E gr where Ii E (B p.g. for so > (1/p-1/2)+. Then, f is covered by Assumption A4 in the form f E B (Idd for y(2) such that z(2) can be taken arbitrarily large, reflecting the additive structure.
Example 3.11 (Multiplicative function; Figure 21D. Suppose f(x I -1 where di U(B%%) p-g) for Su > case in Example 5.I0
This f is also covered by Assumption A4 similar to the additive
Example 3.12 (Rotation; Figure M.. Let R2 E RRd be the rotation matrix by angle Suppose
f= go ( (A +b), where g E U ( p. nuB, A = Rt/Vd and = - (1) ... ,12))². This
is covered by Assumption [A5)
Example 3.13 (Piecewise function). The indicator function of a hyper-rectangle lies in B if S
1/p for 1 < p 00 [37] For hyper-rectangles A; C [0, 1]d, suppose f(x) = Σ T(x E A.) (x).
0.00 for (1/p 1/2)+ & and 5g I 1/p. This f is covered by Assumption where 9; (B
using Examples SWEE and
We now establish posterior contraction properties for the composite Besov spaces. Similar to the anisotropic Besov case, our results rely on the existence of an optimal NN approximator E Φ(L2n, Dzn, S2n: B2) under Assumption (A4) and f E Φ( Dan, San, B3) sumption [A5] where
under As-
Lkn x logn, DEn N Skn x N logn, B₁ x k= 2,2,3
with N₁ 1/28 Therefore, the network parameters share the same asymptotic orders for k = 2, although their specific values differ. For further details, see Lemma B.4 and Remark B.G.
The posterior contraction results are formalized below.
Theorem 3.14 (Composite anisotropic Besov). Suppose that Assumptions A2
A3 hold, and that
either Assumption A4] or Assumption [A5] is satisfied as follows.
(i) If Assumption and 3.9 are satisfied with (Togss D2ns S2n) in place of (Ian: Dins Stn).
A4 holds, assume that the priors and conditions in one of Theorems
DK
B.8
(ii) If Assumption [A5] holds, assume that the priors and conditions in one of Theorems and B.g are satisfied with (Lon, Dan, San) in place of (Lin: Dins Sin).
Then, the posterior distribution concentrates at the rate en -3" 1(28" +1) (log m/3/22
Proof. See Appendix
The contraction rate in Theorem 6.14 matches that of Suzuki and Nitanda [18], indicating that BNNs
attain the same level of optimality. However, achieving the optimal rate Suzuki and Nitanda [[S] requires correctly specified network structure. In contrast, our Bayesian procedure offers clear
advantage by learning the unknown compositional depth, connectivity, and anisotropic smoothness directly from data, without requiring prior structural knowledge.

---

4 Discussion
In this paper, we have established that sparse BNNs achieve optimal posterior contraction rates over anisotropic Besov spaces and their hierarchical compositions. We show that BNNs equipped with either spike-and-slab or continuous shrinkage priors attain a near-minimat rate that depends only on the intrinsic dimension, thereby overcoming the curse of dimensionality. Furthermore, BNNs are shown to achieve rate adaptation over both anisotropic and composite Besov classes.
Practice meets theory. Our work provides rigorous theoretical foundation for the empirical suc- cess of NNs on complex real-world data. By establishing optimal rates over anisotropic Besov spaces, we shed light on why NNs perform well in high-dimensional settings where classical smoothness-boct
theory falls short. Although our analysis focuses on BNNs, which additionally offer rate adaptation and uncertainty quantification, the insights extend more broadly to understand- ing NN performance. The results provide actionable guidance: practitioners dealing with such data may benefit from employing sparse architectures with sparsity-inducing priors, potentially inspiring new algorithmic advances for efficient inference.
Extension. Our theoretical results are established under the Gaussian nonparametric regression model in (I)) These results are readily extended to other statistical models in which the Hellinger distance can be translated into an 1,2-type distance for the underlying function. As one such exten- sion, we present posterior contraction results for nonparametric binary classification. We also show that the same contraction rates derived for the L with respect to Px hold for the empirical -2-norm by applying empirical process theory. See Appendix A for details.
Future work. This study has several limitations that suggest promising directions for future re- search. While the spike-and-slab prior offers desirable theoretical properties, its practical implemen- tation is hampered by the point mass at zero, which introduces substantial computational challenges. Variational approximations may partially alleviate this burden [38, [9). Shrinkage priors address the issue differently, but our theoretical framework does not accommodate widely used choices such as the horseshoe prior [34]] Extending our results to incorporate such popular shrinkage priors is an important direction. Furthermore, natural direction for future work is to extend our theoretical framework to modern deep learning architectures widely used in practice. Recent developments in the statistical theory of convolutional NNs [MC, Aill and transformers [B 3Ky offer valuable founda-
tions for establishing posterior contraction results beyond fully connected architectures.
Acknowledgment
This research was supported by grants from the National Research Foundation of Korea (NRF) funded by the Korean government (MSIT) (2022RCCIC1007755, RS-2023-00211775, RS-2025- 00513129) and the National Science Foundation (NSF) 250311503119).
References
[1] Goodfellow, Y. Bengio, and A. Courville. Deep Learning. MIT Press, 2016.
[2] M. Bianchini and F. Scarselli. On the complexity of neural network classifiers: A comparison between shallow and deep architectures. IEEE Transactions on Neural Networks and Learning Systems, 255)) 2014.
[3] G. Cybenko. Approximation by superpositions of sigmoidal function. Mathematics of Con- Irol, Signals and Systems, 2(4):303-314, 1989.
[4] O. Delalleau and Y. Bengio. Shallow vs. deep sum-product networks. In Advances in Neural Information Processing Systems, volume 24, pages 666-674, 2011.
[5] K. Hornik. Approximation capabilities of multilayer feedforward networks. Neural Networks, 4(2):251-257, 1991.
[6] M. Telgarsky. Representation benefits of deep feedforward networks. 1rXivv1509.08101, 2015.
arXiv preprint
10

---

[7] M. Telgarsky. Benefits of depth in neural networks. In Proceedings the 29th Annual Confer- ence on Learning Theory, volume 49, pages 1517-1539, 2016.
[8] Schmidt-Hieber, Nonparametric renparaantrricegeeesiioooooonoo using deep neural networks with ReLU activation function. The Annals of Statistics, 48(4):1875-189) 2020.
[9] M. Minderer, J. Djolonga, R. Romijnders, Hubis, X. Zhai, N. Houlsby, D. Tran, and M. Lu- cic. Revisiting the calibration of modern neural networks. In Advances in Neural Information Processing Systems, volume 34, pages 15682-15694, 2021.
[10] M. Hein, M. Andriushchenko, and J. Bitterwolf. Why ReLU networks yield high-confidence predictions far away from the training data and how mitigate the problem. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 41-50, 2019.
[11] C. Blundell, J. Cornebise, K. Kavukeuoglu, and D. Wierstra. Weight uncertainty in neural network. In Proceedings of the 32nd International Conference Machine Learning, pages 1613-1622, 2015.
[12] Y. Gal and 7. Ghahramani. Dropout as Bayesian approximation: Representing model un- certainty in deep learning. In Proceedings of the 33rd International Conference on Machine Learning, pages 1050-1059, 2016.
[13] J. Arbel, K. Pitas, M. Vladimirova, and V. Fortuin. A primer Bayesian neural networks: Review and dehates. Statistical Science, 2024.
[14] E. Daxberger, E. Nalisnick, J. U. Allingham, J. Antoran, and J. M. Hernandez-Lobato. Bayesian deep learning via subnetwork inference. Proceedings of the 38th International Conference on Machine Learning, pages 25 2510-2521, 2021.
[15] D. L. Donoho and I. M. Johnstone. Adapting to unknown smoothness via wavelet shrinkage. Journal of the American Statistical Association, 120032) 1995.
[16] Pope, C. Zhu, A. Abdelkader, M. Goldblum, and T. Goldstein. The intrinsic dimension of images and its impact on learning. In Proceedings of the 9th International Conference on Learning Representations, 2021.
[17] T. Suzuki. Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: Optimal rate and curse of dimensionality. Proceedings of the 7th International Conference Learning Representations, 2019.
[18] T. Suzuki and A. Nitanda. Deep learning is adaptive to intrinsic dimensionality of model smoothness in anisotropic Besov space. In Advances in Neural Information Processing Sys- tems, volume 34, pages 1609-3621, 2021.
[19] I. Kong and Y. Kim. Posterior concentrations fully-connected Bayesian neural networks with general priors on the weights. Journal of Machine Learning Research, 26(94):1-60 2025.
[20] K. Lee and J. Lee. Asymptotic properties for Bayesian neural network in Besov space. In Advances in Neural Information Processing Systems, volume 35, pages 5641-5653, 2022.
[21] N. G. Polson and V. Ročková. Posterior concentration for sparse deep learning. In Advances in Neural Information Processing Systems, volume 31, 2018.
[22] Egels and I. Castillo. Posterior and variational inference for deep neural networks with heavy-tailed weights. Journal of Machine Leurning Research, 26(122): 2025.
[23] M. Kohler and S. Langer. On the rate of convergence of fully connected deep neural network regression estimates. The Annals of Statistics, 49(4):2231-2249, 2021.
[24] A. Bhattacharya, D. Pati, and Y. Yang. Bayesian fractional posteriors. The Annals of Statistics, 47(1):39-66, 2019.
[25] Y. Yang, D. Pati, and A. Bhattacharya, a-variational inference with statistical guarantees. The Annals of Statistics, 48(2):886 2020.

---

[26] S. M. Nikol skii. Approximation of Functions of Several Variables and Imbedding Theorems. Springer, 1975.
[27] D. L. Donoho and M. Johnstone. Minimax estimation via wavelet shrinkage. The Annals of Statistics, 26(3):879-99)) 92199991991111 1998.
[28] E. Giné and Nickl. Mathematical Foundations of Infinite-Dimensionnn Statistical Models. Cambridge University Press, 2016.
[29] M. Hoffiman and O. Lepski. Random rates in anisotropic regression. The Annals of Statistics, 30(2):325-4) 23-396, 2002.
[30] I. A. Ibragimov and R. Z. Has" minskii. Statistical Estimation: Asymptotic Theory. Springer, 2013.
[31] S. Ghosal and A. van der Vaart. Convergence rates of posterior distributions for noniid obser- vations. The Annals of Statistiics, 35(1):192-223, 2007.
[32] S. Ghosal, K. Ghosh, and A. der Vaart. Convergence rates of posterior distributions. The Annals of Statistics, 28(2):500-531 2000.
[33] P. L. Bartlett, N. Harvey, C. Liaw, and A. Mehrabian. Nearly-tight VC-dimension and pseudodi- mension bounds for piecewise linear neural networks. Journal of Machine Learning Research, 20(63):1-17, 2019.
[34] C. M. Carvalho, N. G. Polson, and J. G. Scott. The horseshoe estimator for sparse signals. Biometrika, 97(2):465-480 2010.
[35] M. Vladimirova, S. Girard, H. Nguyen, and J. Arbel. Sub-Weibull distributions: Generalizing sub-Gaussian and sub-exponential properties to heavier tailed distributions. Stat, 9(1)ee318 2020.
[36] Y. Sun, Q. Song, and F. Liang. Consistent sparse deep learning: Theory and computation. Journal of the American Statistical Association, 117(540):1881-1999 2022.
[37] W. Sickel. On the regularity of characteristic functions. In Anomalies in Partial Differential Equations, pages 395-441. Springer. 2021.
[38] B.-F. Chérief-Abdellati Convergence rates of variational inference in sparse deep learning. In Proceedings of the 37th International Conference on Machine Learning, pages 1831-1842. 2020.
[39] J. Bai, Q. Song, and G. Cheng. Efficient variational inference for sparse deep learning with theoretical guarantce. Advances in Neural Information Processing Systems, volume 33, pages 466-476, 2020.
[40] M. Kohler, A. Krzyžak, and B. Walter. On the rate of convergence of image classifiers based on convolutional neural networks. Annals of the Institute of Statistical Mathematics, 74(6): 1085-1108, 2022.
[41] Z. Fang and G. Cheng- Optimal convergence rates of deep convolutional neural networks: Additive ridge functions. Transactions Machine Learning Research, 2023.
[42] Kim, Nakamaki, and T. Suzuki. Transformers are minimax optimal nonparametric in- context learners. In Advances in Neural Information Processing Systems, volume 37, 2024.
[43] Y. Jiao, Y. Lai, D. Sun, Y. Wang, and B. Yan. Approximation bounds for transformer networks with application to regression. arXiv preprint 2rXiv:2500.12275, 2025.
[44] Y. Yang. Minimax nonparametric classification. I. rates of convergence. IEEE Transactions on Information Theory, 45(7):2271-2284 1999.
[45] A. B. Tsybakov. Optimal aggregation of classifiers in statistical learning. The Annals of Statis- tics, 3351):135-166, 2004.
12

---

[46] Y. Kim, I. Ohn, and D. Kim. Fast convergence rates of deep neural networks for classification. Neural Networks, 138:179-197, 2021.
[47] L. Györf, M. Kohler, A. Krzyzak, and H. Walk. Distribution-Fred Free Theory of Nonparametric Regression. Springer, 2006.
[48] I. Kong, D. Yang. J. Lee, I. Ohn, G. Baek, and Y. Kim. Masked Bayesian neural networks: Theoretical guarantee and its posterior inference. In Proceedings of the 40th International Conference Machine Learning, pages 17462-17491, 2023.
[49] Mallat, A Wavelet Tour of Signal Processing. Elsevier, 1999.
[50] S. Jeong and Ročková. The of BART: Minimax optimality over nonhomogeneous smoothness in high dimension. Journal of Machine Learning Research, 24(337): 2023.
[51] F. Xie and Y. Xu. Adaptive Bayesian nonparametric regression using kernel mixture of poly- nomials with application to partial linear models. Bayesian Analysis, 15(1):199-186 2020.
[52] S. Ghosal and A. van der Vaart. Fundamentals of Nonparametric Bayesian Inference. Cam- bridge University Press, 2017.
[53] A. der Vaart and J. A. Wellner. Weak Convergence and Empirical Processes. Springer, 1996.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state our main theoretical contributions on posterior contraction rates for Bayesian neural networks over anisotropic and composite Besov spaces.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss limitations of the work in Section 4, including computational challenges, theoretical constraints, and the necessity to extend our framework to modern DNN architectures. Furthermore, we provide future avenues for research based on these limitations.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
14

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: All theorems in the paper explicitly state assumptions, which are clearly la- beled throughout the manuscript. Each theoretical result is presented with comprehensive statements that reference these assumptions. Complete proofs of all theoretical results are provided the supplementary material. Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should he numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proof' can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [NA]
Justification: This paper focuses on the theoretical analysis of posterior contraction rates for Bayesian neural networks. As numerical experiments were conducted, this question is not applicable.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or the contribution it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, other means that are appropriate the research performed.
specific model and empirical evaluation.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature the contribution. For example
(a) the contribution primarily new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
15

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [NA]
Justification: The main contribution of work is to establish the theoretical properties of BNNs, making it purely mathematical; thus, this question is not applicable.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co)
for more details.
While we encourage the release of code and data, understand that this might be possible, so No is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
//nips.cc/ppp httppss.cc/pubIIc/gudddeeee//
www.rbbiiisioooooooo.o.. for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [NA]
Justification: Since the paper does not include numerical experiments, there is no need to specify implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [NA]
Justification: This is theoretical study centered on the mathematical analysis of posterior contraction rates. As no experiments were conducted, error bars and statistical significance analyses are not applicable.
16

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [NA]
Justification: Since no computational experiments were conducted, information regarding computer resources, such as processor types, memory requirements, execution times, is not applicable.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
Our research is fully in accordance with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
17

---

Answer: [NA]
This paper presents a purely theoretical analysis of the mathematical prop- erties of Bayesian neural networks. It has no direct societal applications impacts that would require discussion of positive or negative outcomes.
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
We not release any models, data, or assets that could potentially be mis- used.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards
allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: Our paper is purely theoretical and does not utilize existing code, data, or models that require licensing. All relevant literature and theoretical works are properly cited.
18

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, haperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the cense of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [NA]
Justification: Our paper is purely theoretical and does not release any assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our research is purely theoretical, focusing on mathematical analysis of Bayesian neural networks, and does not involve crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
19

---

Answer: [NA]
Justification: This research is purely theoretical and does not involve human subjects; there- fore, IRB approval is not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva-
lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: We did LLMs derive any core research findings mathematical results.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM for what should or should not be described.
20

---

A Supplement on posterior contraction
A.1 Nonparametrie regression
In this section, we present theorems that apply to general settings with independent and identically distributed (ID) data. For semi-metric space pA, we let N(e, A, denote the e-covering number. The following result builds on the foundational work of Ghosal et al. [62]] and Ghosal and van der Vaart [[]]]
Lemma A.l. Consider model (I) with fo E UB and do E o) constants 0 < S a. The prior for supported on [g. G. For C UB, define
A (J.0) (o IIf fo folla((X))
2 2.10- ool {
2
Suppose there exist subset Fr C F and sequence En + with ne + oo such that
log N (n Fni (1-1112(22)) 2 nen
(SI)
llog (T((A,)) S n
(S2)
II Fm σ) = of Cne
sup EE
(S3)
for a sufficiently large C > Then, the posterior satisfies
Fx [. Ilf foll2((xx) + 1022
- oğ1 1 > Mnen Dn) - 0
in p(a) fo.00 fo -probability as + oo for any Mn + oc.
Proof. See Appendix CA.
We focus on the case where II(F)) \ Fa ()) is invariant with respect to a, and we write simply as [[F) Fx)..
A.2
Nonparametric binary classification
Suppose we have a set of n input-output observations, where each pair an independent random sample from binary classification model with a d-dimensional input variable X, E [0. 1] and an output variable Y, {0, 1):
1 - X.) =
= 1- P = X,) = (w (X.), X Px, iE [n]]
(S4)
where u is the sigmoid function v/M (x) = (1 + M for large M> 0 and fo [0, 1j R denotes the true regression function. The scaling factor M is introduced because fo and the
prior support are restricted UB, whose elements have unit sup-norm. Let p(a) denote the joint distribution of D, with a regression function f.
Lemma A.2. Consider model (S4) with fo E EB. For F CUB, define
A = ffffFFlll follz2(Px) sc).
Suppose there exist subset Fn F and sequence en - with ne? + oc such that
log (Gn: (og.((nnFFri 1...111112AR (111111111)(()))) S n
log 11(A S ?
IIFF of e CnChe)
for sufficiently large C > 0. Then, the posterior satisfies
II I(s [ E F: a1 f) - (1bM o fo)ll > Mnen D.)) +
in p(m) -probabiliy + for any M, + 00.
21

---

Proof. See Appendix
HF
The following theorem shows that a plug-in classifier can achieve results analogous to those in Theorems B.3. G.N, [3. and B. for classification problems, under the same assumptions and notation as in Section The proof closely parallels that of the Gaussian case and is therefore omitted.
Theorem A.3 (Nonparametric classification). Consider model (Sa)) with the same prior specifica- tions and assumptions as in Theorem 33 (or in Theorems B.S. 1.g and B.T.),, except for Assump- tion [33] Then, the posterior distribution concentrates the rate Cn. n n2/// (1og
(log n)332, in
the sense that
m(s E + 11 f) (was o fo)llAA(PS) > Mann Da -
in pln) probbabiiiy - 0o for any sequence Mn. oo, where & denotes the corresponding NN model space defined in the referenced theorem.
can be shown that the minimax rate also attained for the excess risk for the misclassification error, defined as
E(fo. f) == LY # I(( f)( 2 1/2))] E * I(( fo) > 1/2))],
implying that optimal performance is achieved general nonparametric classification settings. is well known that [ the excess risk satisfies
E(fo f) s (v'ar
(vb o fo) fo)ll
This provides the posterior contraction rate with respect to the misclassification
follows.
Corollary A.4 (Nonparametric classification with respect to the misclassification error). Consider model (S4) with the same prior specifications and assumptions as in Theorem B.l (or in Theo- rems B.S. R and [.),,, except for Assumption [A3) Then, the posterior distribution concentrates
the rate cn - -/(22+1) (log n.) 3/2, such that
11ff E 2 E(fo.f) MnEn D +
in To -probability as + Do for any sequence M,. + co, where d denotes the corresponding NN model space defined in the referenced theorem.
Remark A.5. This result can be extended to achieve optimal convergence rates under boundary conditions, particularly under Tsybakov's 's margin condition [НН] as demonstrated in frequentist ap- proaches [А However, incorporating the likelihood corresponding this loss function poses substantial challenges, and we leave this extension for future work.
A.3 Empirical norm
The following lemma describes the relationship between expectations of empirical processes.
Lemma A.6 (Theorem 19.3 of Györfi et al. [17)). Let X, XI.... .n, E Rd be IID random vectors drawn from the distribution Px. Let K1. K₂ 2 be constants and let g be a class of functions
g Rdd + R such that
19(x)1 K1,
Blg(X)?1 K2E[g(X)].
For 0 < a < and > O, assume that
Vnavi- ave 288 max 2K1, v /2Kg
and that, for all *1,". In E Rd and for all 2 e/8.
Vna((1 - at)t
vi
1 "o(a)) s 16t -..1l11 1111112,00 du, log.N u, (U, geg:
96v2 max (onax(K1,2K0 [K,)K2}
ofl o0l- maxi (K122Kg)
1m
where I1gll,,0
n t (x,)l. Then,
[[g(X)] - 1 Σi 9(Xi)
nca (1 Q)
Pr sup geg
> a
< 60 exp
+ [xg
128 2304 max m.a K?,Kg)
K2}
22

---

For a function f [0. 7 R, we define the empirical Lo-no as Il
In-1 Zi=1 [f(x)) As corollary of Lemma A present the following result, adapted from Kong et al. [[S], which establishes relationship between the empirical norm |l-Il, and the population norm | [((x))))))
Lemma A.7 (Change of norm). Let N₂ = [221(28+2)
T 8/(28+12) (log n)3/2, and + be an Then,
NN model space Theorem (or Theorems and
1 - Jollip(Ps) || - sol
p(n) fa.00
n.Muc.
sup fE
60 exp
Moe + llf folli>(Px)
2
8 128. 2304 - 16
holds for sufficiently large constant Mo > 0 and sufficiently large n.
Proof. See Appendix
Using Lemma A we deduce that for any / E & and all sufficiently large n, the following inequal- ities hold with probability exp(-M ne for some Mi 0:
Mo n.
- - 2 f foll full2((Px) + |l - foll 2 - foll. + Moc
This yields a contraction rate with respect to the empirical norm.
Theorem A.8 (Contraction rate with respect to the empirical norm). Consider model (I), prior and the assumptions as in Theorem 3.3 (or in Theorems E.A. By and (.AA). Then, the posterior distribution concentrates the rate en -8/(28+1) (log (logn))/ n))32 in the sense that
[[(f.o) E 1 x [o Ilf folln + +10 og! > Mncn Dn) ++0
p(v) in fu.00 -probability as n + do for any sequence Mn + co, where & denotes the corresponding NN model space defined in the referenced theorem.
Proof. See Appendix
CE
B Properties of ReLU networks for Besov spaces
This section describes key properties of ReLU networks, highlighting their capacity to approxi- mate broad class of functions. In partieular, ReLU networks exhibit the universal approximation property: with sufficient depth or width, they can approximate any anisotropic Besov function on [0, 11d to arbitrary accuracy. We develop the theoretical foundations of this property and discuss its implications for both the design and theoretical understanding of deep learning models. Let (()) = (Jo E (-)) denote the NN model space without the clip function.
For the first step, note that ReLU NN can approximate a cardinal B-spline with arbitrary precision [IX. [ Define (xx) for [0, 1] and (z(z) = 0 otherwise. The cardinal B-spline of order m
is obtained recursively by convolution:
(m(x) = (v * thm 1mm (x) (xx)
for m > 0,
for m = 0,
where (f)g)(x)== == f(x ()g(()dl. For h: E 24 and j = (J.....,ja) E Zd, define
Mdm (x:) II tim mm (2 1 1)1 j&),
for E Rdd. Here, 8 (s 82) Rdd denotes the smoothness parameter, & controls the spatial resolution, and j determines the location at which the basis function is centered. A function f in an anisotropic Besov space can thus be approximated by a superposition of M (z). which closely related wavelet basis functions [49]] The following lemma establishes the approximation of the cardinal B-spline basis by ReLU activations.
23

---

Lemma B.1 (Approximation of B-spline basis by NNs; Lemma of Suzuki and Nitanda [
There exists constant c(d, m), depending only d and m, and an NN M (LLn, Do. So, Bp) with
3dvm
Lo = 3+2 log2
+5 V
ec(d, m)
Do = 6dm((m + 2) 2d,
S = Do
Bo 2(m 1)m
that satisfies (x) = 0 for all & [0, m + 1] and ||M M 1100 S efor all
For order m E N of the cardinal B-spline bases, define
J(k) = II JK),, J((k) = {-m, -m m, 1, 2(koT J3.
1=1
and define the quasi-norm (Besov sequence norm) of the sequence (EJ(() as
1/07 9 1/4
(an Des
2+ (E
Σ
lawel
(S5)
k=0
(EJ(K)
For p 00 or 9 = 00, the norm is modified the standard way.
As noted in Remark b anisotropic Besov spaces admit useful continuous embeddings.
Remark B.2 (Continuous embeddings anisotropic Besov spaces; Proposition of Suzuki and
Nitanda [I]). Let s = ($1, sa) E R Then, the following continuous embeddings hold on bounded domain Q c Rd.
si
sd so & N. then C0 (12) (l0 (12).
For <1 S S co, 0 0.00g co, :0, if3> (1/p1 - 1/p2) then B Br..g (22 ) B7 (9), Y
1 -(1/pi - 1/m2)++/ +/5.
For 0 < 0o and q: < 92 S xc, B p.q. (1) + (
If0 4.9 20 and > 1/p, B (2) + (I)).
The notation + denotes a continuous embedding from the space on the left into the space on the right. particular, if $ > 1/p. then (
(72( with y 1/(5)). In this case, every function B (52) continuous. Otherwise, the functions may display discontinuities or non-smooth behavior such as jumps or spikes.
Building on these embedding properties, the following lemma establishes the approximation of anisotropic Besov functions by cardinal B-spline basis, thereby extending the classical spline ap- proximation results for isotropic Besov spaces [177] to the anisotropic setting.
Lemma B.3 (Approximation of anisotropic Besov function by cardinal B-spline basis; Lemma of Suzuki and Nitanda [[X]). Suppose that < p.q.r < 0G, w == (1/p 1/r)+ < š, and v = (3 (G-w)/(2w)) Assume that 0 < min(m, - 11p/pp, where
N is the order of the cardinal B-spline bases. For any f E Bi and N 0, define K(N) such that (K()/s(
N and K (N) = [K(N)(1 + 1/v)]. Then, there exist Ex C ((k, j) S k S K (N),,J E J(k)) with
EN < N and fN such that
(N(x)
E (k.))EFx
ong Md. (x),
IN-NNlIE- SN II-ffflee
where the coefficient (ax.3) yields the following norm equivalence
|| ~ [I(an))ksloog..
(S6)
24

---

The following lemma extends Proposition 2 of Suzuki and Nitanda [ establishing that for any function fo in an anisotropic Besov space, there exist NNs that approximate fo with arbitrary accu- racy. In this version, the dependence of the hyperparameter B on N in the original proposition is removed.
Lemma B,4 (Approximation of anisotropic Besov functions by NNs). Suppose that O < P. S 00, w == (1/p - 1/r)+ < 8 and v = (8 -w)/(w), Assume that N E N is sufficiently large and m N satisfies 0 < 3 < min (m,
1 1/p). Then,
inf JoE (,, (0: B.) 115o //1 SN-
sup
with
3dvm cc(d, m)
L1 L.d = 3 loge
+5
[log2 (d (dVm)] m)].
= Di= N) NDo
(S7)
S (1d, N) L,DON,
Bu exp (1+ 11++-111/ V (1/p 3)+ log 1ogg
B1 (1dd,
3 Blogz (d V mm]
where Do Do(d, m) 6dm((m + 2) + 2d, Bo Bo(m)
2 1)m,
N-E-(L4U"\\11003) log(N)-), and constant c(d, m) does not depend on N.
Proof. See Appendix
Remark B.5. The key strength of Lemma B44 lies in that the approximation accuracy depends not on the ambient dimension d but on the intrinsic dimension. When NN models are employed, if the target function exhibits anisotropic smoothness, the approximation rate is governed by the intrinsic rather than the ambient dimension. This leads to minimax -optimal performance in estimation theory and illustrates how NNs can mitigate the curse of dimensionality. In the isotropic case where (so, 80: ...,,..,). Bp), we have 5 = 8p/d. recovering the classical results for ordinary Besov functions in Suzuki [[
Remark B.6. Note that the NN space (LL, D,
S B) is monotonically increasing with respect to L, D. and B. Thus, the parameters in (SZ)) can be replaced by
L C₁ log N, D = CDN, S CsN log
B₁ CB
for sufficiently large constants CL. CD. Cs, and CB-
NN models with ReLU activation permit redistribution of weights across layers without altering the output function. The following lemma formalizes this property, which will be used redistribute the magnitudes of the parameters.
Lemma B.7 (Re-scaling Lemma for ReLU NNs; Lemma A.1 1 of Kong and Kim [9). Let L 2 3 and D 2 3. For positive constants CI," CL+11 we define
1
W c,W(), 80h
II
Cy
for l E [L + 1]] and define Ò -W/(1) BO)).... ..... (L+11) ((L+1)), holds for every x.
M a = 1. fo(x) = Sa(x)
Remark B.8. The clipping function be implemented using simple ReLU network. Specifi- cally,
clip(z)) max -1} < + - ((x 1) 1,
where <(-) denotes the ReLU activation. Consequently, the clipped network space +b.) is contained in the unrestricted network space (--). Furthermore, to ensure that each partial block of composite Besov function produces outputs within [0, 1]. we apply
clip (](( (4)
{(x) Ç(x 1) max 0) maxf# 1.0).
With this construction, ReLU networks capable of approximating composite Besov functions.
25

---

The following lemma extends Theorem I of Suzuki and Nitanda [[8], which establishes an approxi- mation bound for any fo a composite anisotropic Besov space by NNs. Unlike their setting. which imposes no restriction on the input dimension of each component, we adopt the composite Hölder framework of Schmidt-Hieher [8] and assume that each layer map depends only on a small subset of its inputs. By combining this effective low-dimensionality constraint with anisotropic smoothness, we exploit both compositional structure and reduced intrinsic dimension to efficiently approximate functions in high ambient dimensions.
Lemma B.9. Suppose that z(1) (1/p- 1/r)+ for some r > and g(h) 1/p for all 2 Then, the approximation error over the composite anisotropic Besov space is bounded as
sup foE St,
inf
Il.uu fll N-
So.
where (*(M) 3(h) I ((s(H) -(s())((((k)) /p) A 13, 3" minne(N] g*(h) and
L2 Σ
Li [1(((h) m! p. (()) (())) .N) + 1]
D2 = max A21
D1 (t m (N) .N) v {(h+t)}
H
S2 = Σ
(1))
(S) (t (h), (h)
(h) p, r()), , s(h), .N) + 4),
h=1
B2 max B₁ (h) ml) p, (1)) s(h)).
A21
Here, (1) = r, (()) = 00 for 2 and mh)) (h) satisfies < g(h) min(m ml))) -1+1/pp
Proof. See Appendix C
Lemma B.10 (Theorem of Suzuki and Nitanda (1]).. Suppose that d()) S d and s(2) e
Bd g(2) (1/p- 1/r)+ for some 0. Consider the anisotropic Besov space that involves affine transformation
F == (20 f = A Ab, E (dd o0 b Rid
||A V 11b1l00 S Ca, f2 E Bo(
Then, the approximation error on the function space is bounded as
sup lea.
inf
|lfo -fller S -
where
L3 (d(d m (2) p.rr (2)) (2),,N) ,N),
D3 D, (d m (,,,,N
S3 (1((1), (2) ,p, s()) ,N),
Bg (C,d(), + 1) (1) m (2) pp. s())
for (2) N satisfying 0 ( minnmm (2) 1 1/p).
To apply Lemma A. we require results concerning the complexity of the model space and the
asymptotic properties of neural network models. By the Lipschitz continuity of clip,
[(elip fi)(2)) (clip [2)( (f(()) f2(x))
It follows that the covering number of the clipped network class can be bounded by that of the unrestricted class.
Lemma B.11 (Covering number; Lemma Suzuki [T].. For any > 0, D. and L 2 3.
log NogN((s, E(L D,S, S (S + 1) log (LBVV
26

---

To allow for relaxed sparsity with margin > O, we define
( D, B, = {o= (0,I( > a) E (
(S8)
where T (L, D)}, and denote the corresponding NN class by $(L, D, B, and (I, D,S, a)
Lemma B.12 (Lemma 5 of Lee and Lee [ For all (aL(BVI) D 3, and L23 3.
(o (L,
s (S 1)
The following lemma, introduced by Schmidt-Hieber [8] plays key role in the proofs of Lemma B.I and Lemma B.I2 It characterizes the distance between NN models in terms of their parameters.
Lemma B.13. For any and 0, 0r ( D,S, B) satisfying 10 0 1l 6
[ Sa loo S <el (L) 1)
Proof. See Appendix
C Proofs of the technical results
C.1 Proof of Lemma
Proof. We follow the proof Theorem 2 in Jeong and Ročková [B0]]. Since there exist upper and lower bounds for of and o², the rate for la² o is equivalent to that for la ool: hence we use the latter. For every (f1,01) (f2. 02) F x [o. define
(,,, 02), == lI f2 /2lia((x)) + loi - -0
By direct calculation, it straightforward verify that for some Cи 1,
(S9) where PH denotes the Hellinger distance and Pf.a is the density with f and a. Therefore, the Hellinger distance is equivalent to Pz up to constant factor and it suffices to establish the theorem with respect to the Hellinger distance, which admits an exponentially powerful test function.
CH (2((f.a)) (f2. 02)) S (p(Ps Pfz.0.) S CuPz (f2,
Using (S9), the Hellinger entropy can instead be estimated by replacing it with PL. Note that (1) $ holds if [I - f2ll S 5/2 and (a 02l S c/2. Hence, by (SI).
log log.V(En, Fr x la, (. S log N 2, 1| (1 log(en + log σ S ne
for all sufficiently large n. Using (S2). there exists C' 0 such that II(A..) 2 exp( exp(-C"nec.) Moreover, by Lemma B.2 of Xie and Xu [5Il],
(n)
(n) log
max
- fo.
log
P Pi.e
E Bio.o
f.o (n)) Poooro
C" ()(
(n)
Pio.
for some constant C" C²/(C
4) and all sufficiently large n. We obtain
(n) Pi.
max
Ef log
Pi.0 (m)
BBooo
log
(n)
Pio
Poo.0o
(ppi (), (fo, oo) S ( 2//"")
II( (AA./v)))))
exp(-ne2 /C")
exp( (-C 4)
for sufficiently large n. addition, (S3) implies
[(F u(F F. dd)ll((o?)) = ole Cne))
We get the desired result using Theorem 2.1 of Ghosal al. [2
27

---

C.2 Proof of Lemma
Proof. Let (2) (l o fi) (ehas (e))l for f1. f2 E F. The proof proceeds by first establishing the equivalence between the Hellinger distance pH and PL, and then deriving the contraction rate with respect to pn- For any f1 E F, let BMM o fi and 1/2 u/M o f2. Then,
(h 1 (Vm(s)- Vna(ai) (Vn(()) + vna(s)) Px (x(d))
+ 2 / (VI-m( m( vI - na(a)) 2
(VI- 1V1-m(6 (xx + vI- m2(x)) 2 Px(dx).
Since both 1/1 and W22 lie in 10, 8] for some d > O, there exists a constant Cи > 0 such that CH'Pe (1 f2) S PH (P(p), I CH,
Hence, it suffices to work with the Hellinger distance. Since the sigmoid function thas is M/4 Lipschitz, we obtain
M
(i, f2) Il 72ll 4 11.f1 f2)122(Px)-
Consequently, the Hellinger entropy is bounded by log. (en, pr.) S log(en; Fas (l-)) S
nee Let Pj
(n) denote the joint density under model (S24) with S. By Lemma 2.8 of Ghosal and
der Vaart [52]], for any measurable functions fi and f2.
(n)
(m)
27
P
Pf.
max
Ef
log
[m]
E
log
[m]
-
f/a
Pifa
The remainder of the proof follows the argument as in Lemma A.
C.3 Proof of Lemma
Proof. We prove the case corresponding to Theorem 3.3 the remaining cases follow by analogous
arguments. For Lemma Awf a 1/2, Mo K₃ = K₂ = 4, and
g = (9(x) = (f(x) - fo()))
})
where + (Liin Din Sin). The choice K1 K₂ justified because
[(x)l 4, B[g(X (X)]
for every g E G. Since ne, + 0o. we have
fhavl- aV2 2 288 max 2KI, V2Kg
for all sufficiently large n. Moreover, for any f1, f2 E 4,
[l(Ai - fo) (ʃ2 _ 4llfi f2lli.n
which implies
log No EG: Σ ((x;)² s 16t 111-11.m $161.1111..
s log 1og.N(w// (1/4, 11-11.))
1=1
Using Lemma B.Il we obtain for every u 2 2.
log.N uu/4, &b 11-11...))
S NoggN(/// 1 $.1.1.1.. 11.11
[oog Ln + Ln log((B, V 1)(,, + 1))) - loog u g
I (S, + 1)
N, n)) S nea
Therefore, for all t > M3
Vi
log.N u,
9 E G
1 Σ
g(xx)² < 161 11-111,, du S Vinez.
aft ma (Ky
Since V V8/Mg is sufficiently small, conclude the desired result.
28

---

C.4 Proof of Theorem A.8
Proof. Let En denote the event
|l - foll 11 - follin
en
sup FEB
Muc, + Ilf - -foll((PPX)
Then P(n) (E%) S exp( -Cne)) for some constant C > 0 by Lemma 10.00 any f E 2
On the event En, for
Mo Ilf S V 11 follz((22) + 2 c S 2 folle2(2x) +
Mo
2
We now express
uIlIf foll, + 10- obo > Mnen Dn) = =1((If - foll. + o - oāl > Mnen Da) I(E,) + +1022 -flla
oùl > Mnen Dn) ((E)).
For the first term, observe that
IIlIff folln + 10a 00. oo6l
Mn (n)) I(En)
S II IT(11 [I. (px)) + +10² 1 ob (M, C')cc// I(E,)
for some constant C' > 0. By Theorem 6 (or in Theorems HR C in ((n) fo.no -rrobability
and
B 40. this converges to 0
for any M,
For the second term, since p(n)
fa. (EA) 0, it
also converges 0 in p(a) fo.mo -probability + 0o. This proves the assertion.
C.5 Proof of Lemma
Proof. Using Lemma for K = K(N), K* = (K(L+v" and EN with Ex| S N, there exists fN such that
fx(z) = E
M (I),
(K.))EE
Il - fNll -
Since for every k and j.
Ma.a (x) -M,0
Ma.r (21 'Jan -i..... 21kea 2 xa ja),
(S10)
Lemma B. immples that there exists an NN approximator Mag satisfying | for any e > 0. By the definition of B-splines, it follows that
11 S e
kij
Σ
(M (x) + +0) < (m +1*(K"" +1).
(k.))EE
and the inequality N 2 (Kg/// 2 2kd we obtain
Moreover, using (S5). (G6),
laksl S 2 (1/p-8)+ $ 1(1/p
(S11)
We now define the approximator
Σ
a.. d.m.
(NJ)EEN
Combining the above bounds yields
[fx(x)
f(0)
Joneaal Maly" (x) - Maj (x))
(M
< Σ lakill (M (x) #0)
EN
S c(m ()*
S (elog(N) (L
= N-
29

---

Using the Lipschitz continuity of clip, we get the desired approximation bound:
Ilf (clip o flz = [[(clip f) - (clip fll & llf - fNll + IlfN flle. S SN-
Next, we verify that the network parameters of f are given as in (SZ).. Since the identity function (ar) be represented using the ReI.U function as id(x) ((x)) <(-x)), it follows that for any I. E N,
id = o......0 od = id E (L,22,4,1)
L times
Then, using Lo- = Lo() [log2 (d m)][s logg N,, we can rewrite fas
f(x)=
Σ oh (an, (er)))))))))) (x)).
(S12)
(k.j)EEE
which is also an NN approximator. Since Majj" requires the same depth, width, and sparsity as MM in Lemma B.I. each summand od (ax, N Md)(()) (x)) has depth Lo + Lo. width Do. and sparsity Lo + 4T.g +1. 1. Therefore, f expressed as in (ST22) requires depth Lo + Lo I L1, width N Do. and sparsity (L + 4h. + 1)N s St. To evaluate the magnitude of the network parameters, note from (SII) that the magnitudes of the parameters in the first layer of M are bounded by
2x* S CC1NN(+ for some constant Cn1 0. All remaining layers have the same magnitude those of M, namely Bo- In addition, by (SII), the magnitudes of the parameters in the first layer of L + id id (om,g() are bounded by C12 (1(++=)(((/pp)) for some constant C12 > Let and B12(N) (1t20/1t01110//--0000""/ B and define
Bu (N) C11N00
(M(M))Lott)"/CCnnNT" )/a)
1= 1,
(Ii(N)(oot))) /Bo
1= 2..., 1 + 1,
N(C:2 10/0--))
1=1,
C2r (ia(N)((o+t1) /Bo
1= 1 Lo +1
is easy to verify that I1 cu S and 11 c2r S for any and sufficiently large N. Therefore, by Lemma B.A. the rescaled h has parameter magnitudes bounded by (1(N))Lot))
B12(N)((a Since Lo(d, m, NN 2 Lo [loga v m)][slog,, N1 for sufficiently large N, we obtain
(Io+1)-1
Bu(()(lot))) v (o < Bo (ICI V (ta))
s BoNN(tu H(CL/////////
(1
v (1/p- 3) log 3 Blog (d V m)]
< Boexp
= =B+
which concludes f E (LI,, D1, B1).
C.6 Proof of Lemma RJ
Proof. Our proof is similar to the proof of Theorem 1 in Suzuki and Nitanda [[8]. Let fo = fo.H
- fo,1 Since g(h) > 1/p for h 2 2, by Lemma B.4, for each fo, there exists
fr,k E (t m (h)
p, oc, sc (h) (N),,N).N.... Dj (t( (M),
(1()), (h), (h)) ( B1 (1(0), mh) .p. 0o, (1)))
such that [lo, - Sh.llloo $ ( For 1, each fu,1. there exists
fI, E (d (1))) sp., (1), (N,, Dj (d))), m m, N),, S (d m pprr, (1) N). B, (1) ml))) .P, (1)))
30

---

such that 1o. Ji, S N- To ensure that each block produces outputs in [0, 1]. we apply the clipping function cl. as described Remark BR which adds one layer to each block. Let f = fn o fH-1--. o fi E (L2, D2, S2, B2) as in Schmidt-Hieber [S]. Then,
fl fllo-
| fo.H-1 ... fo,1 - fH fH-1 fill.r
H
f fo.H-1 fo, fn-1 ......f fi fo. fo. fo o fa.... f-llr
h=1
H
$
Σ
fo fo.n-1 for fo. fo.n-1 fo,++1 o fu
h=1
where r1 and rh 00 for h 2 2. By Remark B
fo. E -=""p_//pp"") j]1 = -(( = =ch,, h h 2,
where 2/2 = (2(h) (r(h) /p) A1 Therefore,
11 -o. for fo. fo. fo, full 2 Il fia Iall [ - Vos
S N g(h) 1 the
N N-
for h 2 2. For h = 1,
fi fill - 1
Il o fo. fo.1 - fo.H fo. ... fill 2 Il,
S N-
Therefore, 11fo- fl X maXhe[m N N- = N-33
C.7 Proof of Lemma B.K
Proof. For f E D, B) expressed f(x) (W(L)(())+ b(1)) o.... (W())()) (W)))))))) (
(W(Ix++ + b()), define
b(k o.... (W1))(()) + ((2) o (W + b()), (g))(() ((W(+)()) + ((5) ....o ( + b(++1)) (W(())x x ((0)),
(f)(() = Ç <o
for k 2.. L. and let A7+1 Af)((z) x. Then.
= = (/) (W(k). +D(0) A (f)
Using the definition of D,S, B),
(f)=)) < max W (-((k-1) |,, 1 + 116(8-111
$ DB [t))(()))))))))))))) B
<D + 1) (V) (
(B
where Aj: denotes the j-th row of matrix A. Similarly,
(4 - A (f)(2) < (BD) x2lloo
31

---

Fix > 0 and 0 E O(L,D,S, For any 0" (,,, satisfying ||0 - 10 < we obtain
(fo(z) (x))
Σ 1 (W(h) At(so)()) AT+1(fo-) (W (1y46)++bb00 +bb(") :
(BD) h
(H766) (W(1). 4 - (W(h) +b(()) AI (fo)
( (D +1]
k=1
I cLB
This proves the assertion.
C.8 Proof of Theorem
Proof. Let F (Linn Dins Sin). By Lemma A. suffices to show that there exists subset Fn CF such that
(a) log En. Iog.N(En: (111111122(Px)) S ne
(b) log [IA...) ) S ne?
(c) su [I Fr o) = 0 (
for sufficiently large C. Let F, L(Lnn Din: Sins Br), where B, n. To verify (c), observe that
[F [
[I : 10,1 > Bn Lin, Din, Sin) = (1 Un Smm
where Un Suu1>Bn #sL (u) (u)du. The direct calculation of the number of network parameters yields
T (LL D²
L+))
which implies
Ti == (L DIn)) x Di.
(S13)
Moreover, Un S kk B. for some ki > 0 by Assumption BTY Therefore, by Bernoullis inequality, obtain
Un) -1-1 S Sin S Tin Ba Tin o(e-Cnnt)
for sufficiently large C. By Lemma
log. N Frr Fn..lllleee)x)))) s log Fri l1.))
< (S + 1)
log Lin + Lin (Bn((Bn V 1)(D + 1)2) log 6n
2
S N₂ (logn)
$ n2
for sufficiently large n. Thus, (a) holds. Given Lemma B4 Remark B.G. and Assumption [A]] there exists fr = fo E @(Lnn Din. Sin, Bi) c Fn for θ O(LIn Dino Sin, B1) such that
Il foll s C1N < en/4,
(S14)
32

---

for some C1 0 and all sufficiently large n. Let y denote the index set of nonzero components
in 8, and let 0g denote the corresponding nonzero values. We define ( I.w. Pin:SSnn B1) C (Li,, in. Sin, Bi) as the subset of parameter space which only the components indexed by
are nonzero. The corresponding NN space is denoted by
(7) $b(;; Lin, Din: Sin: B1).
Using Assumption A3 we have
log log[I((o - Øol S cn/2) 2 log ca 2 -logn
and hence
2 |lf - L((x)) sn/2) loog
Given (S14), there exists a constant C₁ > 0 such that
m(JEF= 1| (olloll S scn/2) 2 ( 115 ( S Sen/4)
2 11 f Fn 11f - fullx S En/4)
1of-fylle
2 n(f @ Fn(()) :I ffuloo S sm//4)
Using Lemma B.1
n I(f (n(7): :1f fullo S Scn/4)
Il 0 RT Oge 0, 1,1001000 S &B2,110s - 0yll00 S 4(D,n + 1) Lin (Bi 1)
Sin
(Tin)
AsL(u))
2 tn inf
B,.
Sin
81n
ge (n(Din + 1i-
inf Brl
#sz #sz(u)
where tn = En/(2(DIn 1) Lin V 1)_ The last inequality follows from
Tin
Ti (Tin 1) ... (Tin Sin + +1) s Sin S (Din + 1) 11n
S
Sin!
Using
(Din + In Lin v1)mu-1
llog tn < log
2 (log n)²
2en
and Assumption B22
we obtain
log ((A,,))) x Sin log (n + 1)-Lan
inf B.]
+ log n # (u)(w)
Sin (logm)²
N (a n)
S ne?
which verifies (b).
C.9
Verification of Remark
Note that Kong and Kim [1 established posterior consistency over unbounded parameter spaces. Their analysis allows for greater flexibility prior specification by leveraging a complexity bound, and the result builds upon the earlier work of Kohler and Langer [23]. Motivated by their approach, we show that posterior concentration at the rate ne? N₂ (log n) remains valid (in contrast to the original rate ne, = N,, n) by employing the covering number bound stated below.
33

---

Lemma C.1 (Covering number for sparse unbounded NNs). For all é >> O, the following holds:
K
S Cv, LSlogg S log log.N(e, ( D, 11
+ log (Cv, LS log 5)
for some positive constants Cv, 0, Cv2 > 0 and K > O.
Proof. By Theorem 7 Bartlett et al. [3 the VC-dimension (,,) the class & D.
satisfies V (S.) S Cy L.5loog for some constant Cv > Then, by Theorem 2.6.4 of van der Vaart and Wellner [53]], it follows that
lee L(L, (H), S log Dogg(CVV&u.. (K/e*) V VuLL..)))
for some positive constants C" > 0 and K > 0. Therefore, the stated result follows.
Now are ready to prove the assertion in Remark 34
Proof. Let F (Lin, Din Sin). By Lemma A it suffices construct a Fn F such that the following conditions hold:
(a) log Ens Fr, (1-1l12((x)) S new
(b) log (A.) S ne.
0) (e-eeee)
(c) o IT
for some constant 0. Let Fr, D(Lin Dins Sin). Note that condition (c) is trivially satisfied. Using Lemma C.ll and Assumption [22 we obtain
log.V( (EN, F (lllaa()))( I
11-1l.2)
SIn log Sin log
N,, (1ogn)
S n
where the last line follows since Lin x log and Sin x N₂ logn. n. The remaining parts follow identically from the proof of Theorem 3.3
C.10 Verification of Example 4T
Assumption (22 holds trivially. To verify Assumption [CT]
observe that
[ J.KK,,K.J' (u) du S Jin Pr([Z] > Kn/omn s Ci exp (-C2K 10
1/1))
for all sufficiently large where Z is a random variable having density Pp. Since In is decreasing, ( (-n.KK #sh (u) du
log
- K1/K /
1/k S - KK]K S -Kn-
for k < 1. This verifies Assumption
CTT Next, observe that
[ [- (sH( du S Ten
(1 an C.
1in Pr([Z) > (2Ca(log (olooogn)))) 2)
< 2e 2Cog n)²
< Ca(lonn)
Hence, Assumption [ is satisfied.
34

---

C.11
Verification of Example B.6
Assumptions CCT and can be verified in the same manner as Example B.S. therefore
suffices verify Assumption (22 Note that Rsn (u) 2 #2n02 (((/02)) where denotes the density of the standard normal distribution and d2 a positive standardized deviation. Therefore,
B 2
inf log (s) log T2n + log(2Ta?) wc(- B.)
x (log n))
C.12 Proof of Theorem 3.8
Proof. Let (LInn, Din). By Lemma A.I. it suffices show that there exists subset F, F
such that
(a) log.V( En, Fa, (1-1122((x)) S ne?
(b) log [A(, $ ne,
[IF Fn σ) = ( CC'ne
(c) s
for a sufficiently large C². Let Fr = ((Lin Dini Sin, Bn: an) as defined in (SB), where B, = n. n). obtain (S15)
Since Bn 14-n 2 exp( Lin log n) and en/ /2/12 In + [r
ea/ v 1) 'Din + 1) [i 2 exp(-2L1,, log On
for all sufficiently large Hence, it is easy to show that (a) holds using Lemma B as in the proof of Theorem 3.3. Let U/2 Ju|sB. #sH (u) du and un Juul2a. #SH (u) Then,
Fn (0) I[(F Fr) IT[
Tin I(10,1 > an) Sin Lin Din
[I : 18,1 > Bn Lin Din)
(1 (1 ,,, )Tm) Pr(S> Sin),
where S ~ B(Tin un) is binomially distributed random variable. Since Un S e kk e Bn for some k > 0 by Assumption (CT) we have
1 - (1 - vn) TT I Unn S e eh Br Tin oe--Kane)
for sufficiently large K₂ 0. Let S1n/ Sin/(TTnun)) Using the multiplicative Chernoff bound of binomial distributions,
Pr(S > Sin) S exp((TTin Un log - (rn - 1)]) S exp(-Ci logr,.
for some C₁ 0. By SI together with Assumption
have log Tn 2 Kz(log for
sufficiently large constant K3 > 0. Therefore,
Pr(S Sin) S exp(-Kand Kznce
Next, as in the proof of Theorem B.3 there is constant C > 0 and fa E Φ(L Din, Sin: Bi) for i E @(Lin, Din. Sin, B1) such that
I 1oo-follzace foll S C1N= < cn/8
for all sufficiently large n. Let & denote the index set of nonzero components ô. Define the subset of the parameter space O(f; Lin Din, Sin; B1, an) C O[I Din, Sin; B1, a,) whose magnitudes exceed a, only at 9, and let (7) = 4(9; Lin D Dins Sin, B1 an) be the corresponding NN space. Then, by (S15) and Lemma B.K. there exists 0(7) E ( Lin, D Sin, B1, an) such that [| 166)) I c./8. Let f.. = Sā(4)- For all sufficiently large n, we obtain
n(fEFr foll I scn/2) 2 II f F. 1l fallea(px) I cn/4) n m 11 - S Smn/4)
35

---

By Lemma B13
II If ff - fullo S en/4)
cn/4
2 0g0 E an: an an Sin 110 1000 - &Bi,,18, - (Din - Lin V 11)4nn
Lin
Sim
2 (1 un) Tin tn
inf ,,
#sm(u))
where in = cn/12 + +1) Lin V 1). Therefore,
-log IT( (T(Ac.)) -Sin log in BB.1
inf #SH (u)
(Tin Sin) log(1 un) + log
$ Smn(lgn) Tin log(1 1 Sin/ / log
(S16)
Sln(log)) n)² + Tin (S Tin + o (Sin + log
$ Sin (log + Sin + o (Sin) + log n
S ne?
For the second inequality, we used the fact that
- loog(1 u.)) S log 1 e Ca(logn)))
-log SIn/Tin)
for sufficiently large n, as implied by Assumption (C3)
C.13 Proof of Theorem 5A
Proof. Let F = ((Ln). By Lemma A. it suffices to show that there exists subset F, F such
that
(a) log.N (en, (en (1-112((((()) S ne?
(b) log (l(A..)) S ne²
(c) suPoc(a,-] III Fn (0) [0)= (e
for a sufficiently large C" We first prove (i), the case corresponding to the spike-and-slab prior.
Define D, = CDNN, and Sn = Cs.Nn log for sufficiently large CD, C's 0. Define the sieve
F, U U (( D,S,Ba),
SS$.
where Bn = n. To verify the entropy bound, apply Lemma to obtain
2 2 (Bn v 11 1)2La
S+1
Fn ΣΣ
6n
DED. S55.
(S17)
Sa+1
D
(2
i vI) 1)2kn
En
Hence, (a) holds as in the proof of Theorem C such that
To verify (b), observe that there exists constant
min( Ts (S.)) 2 exp (m
Since Din S D, and Sin S... by Lemma B.4 and Remark B.G. there exists f. = Sa E (LIn, D., Sn B,) satisfying (TT)). Following the proof of Theorem 33 it can be shown that
log IT ( F |l - S en/2 / Dn, Sm) S ne?
36

---

Therefore,
log [l(A..)) - -log Fn 1-1f- - S Sc//2) log
$ - -log TD (Sn)
(S18)
log II f E Fm : 1lf follza(EX) & cn/2 Dr, S.) -log m
ne:
which verifies (b). Now, observe that
[I F o) = [(FFFF
S <TTD>> > D.)) + II(S > Sn)
Σ Σ xD(D)=s(S)II 3j 10,1> Ba Ln. D,S)
DSD, SgS..
First, we bound the tail probability of D as
II D Σ D(oo 2
Σ
DDog Dny z (oo I e e-Cine
D>
D>
for a sufficiently large Ci > 0, provided that Cp is large enough. Similarly, we obtain IT(S > 5,) S C-Cne for a sufficiently large C2 > O, provided that Cs is sufficiently large. Lastly, following the argument used in the proof of Theorem BA
Σ
Σ
*D(D)=s(()MI Aj 10,1> Bn i.n D.S)
DSD, S5S.
Σ
Σ (D(D) -.,) )
DSD, 858.
$ (1 Un) Sn
0 Cane? Canee))
for sufficiently large C3 0. Therefore, (c) holds.
To prove (ii), define the sieve
F, = U (LIn, D, Sin, Bay an)
DSD.
Using Lemma B. (a) is verified in (ST.). Moreover, following (STG) and (STS), obtain
- log I1(A...) < - log loggpp(D)) log 11 (f Fn : llf - follza(P5) S En/2 Ln, D.)) log n
$
n
2
which verifies (b). Lastly, we obtain
I(F l
=1((F) F,))
< <1(DD D..)
Tin I(10,1 an) Sin Lims D)
Σ (D) (T 10,0 By Lin, D) + D.D.
Following the calculation in the proof of Theorem S verify (c).
37

---

C.14 Proof of Theorem 4
Proof. We only prove the part corresponding to Theorem 33 under Assumption (A4). Other argu-
ments proceed similarly. Let F (Ln. Dzn: S2n) and Fr & L2n Dan S2n, B2). We verify
conditions (a) and (b) of Lemma A.I. Using Lemma B.III
log. N ((n.. Fn. (HllA())())))) s log len
log Lin Lzn ( 1) 1)3) log En
S 2n + 1)
$ (log
$ n
for sufficiently large n. The last inequalities hold because
Len logn, n, D2n Nas S2n N₂ logn
for fixed (()) Thus, (a) holds. Using Lemma B. and Assumption [T) there exists Sn. = fa Fr such that
Ilfn - follea(Px) I CiN S /
for some C1 0 and all sufficiently large n. Let Ý denote the index set of nonzero components Ó, and let 0g denote the corresponding nonzero values. We define @ L2n, Dzn, S2n, B2) C O (Lzn, D2n, S2n, (2)) the subset of parameter space in which only the components indexed by are nonzero. The corresponding NN space is denoted by
Fx(()) 2(5;; Lzn D2n: San B2).
Using Assumption A31 we have
log 1ogl - ool S En//2) 2 loge, 2 -log
and hence
II(A.)) 2 IIff E Fn |lf - ( log
cc/2)
Using the earlier result,
[ -ool:lllsssllleoo
( 11 falloo S Sen/4)
Using Lemma
f E Fu(f)) || falloo S eal4)
0 FT 2q0 0, 10100 S 11gg1000-0 -4lloo S 4(D2n + 1) Len BB 1)
San
T2n
t2n inf wel- Ba)
Asl(u)
S2n
ten (D2n + 1) 11-
inf
sss(u)
BEl
where lan en <n/12(D2n bm Lan B2 V 1)L2n-11 Therefore,
llo II(AA,.)
S2n log 12n (Dan + Tmn inf Tst.(w) log n nE( R2)
S Szn n)) S ne,
which verifies (b).
38


Reviews:
[{"review_id": "rajhHCz8wP", "reviewer": "Reviewer_Hrh9", "summary": "This paper presents a comprehensive, rigorous analysis of Bayesian sparse DNN. Compared to existing results, it considers a more general class of smooth functions and considers both spike-and-slab prior and shrinkage prior. It also proposes a prior to adapt to the unknown function smoothness. It provides an optimal posterior contraction rate.", "questions": "1. Line 181: ''.These network parameters depend on the Besov parameters $\\tilde{s}$ and $p$''. I fail to see the dependency on $p$.\n\n2. Prior (6): \"for some constants $\\lambda_N>0$ and $\\lambda_H > 0$.\" Going through the proof, it seems that the two $\\lambda$ can be arbitrary. Please clarify", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength: Along this line of research, I believe this work is most comprehensive. It combines the standard theory of Bayesian nonparametric convergence and the approximation theory of DNN. The theoretical analysis is rigorous and sound.\n\nWeakness: \n1. The scope of this submission is purely theoretical. Its impact on real applications is omitted. All results are asymptotic, and how to apply them to finite sample applications is unclear. All conditions required for convergence are sufficient conditions. Based on my personal experience, these conditions are over-sufficient for posterior consistency for practice.\n\n2. What is the additional theoretical insight obtained by this paper, beyond existing results? Or the paper only aims to generalize the existing theory/prior specification to Besov Spaces. \n\nOverall, it seems this paper is more suitable for a pure theory venue other than NeruIPS.", "was_revised": true, "final_justification": "I maintain my score. The paper delivers important theoretical contributions, but I don't think the author can address the weakness I pointed out: 1. lack of numerical justification 2. no additional insight on the practical implementation of sparse BNN beyond existing results.", "rebuttal": "We are grateful for the reviewer's thoughtful feedback. Our responses are outlined below.\n\n$\\textbf{W1: Scope of the paper}$ \n\nAs the reviewer pointed out, this study is purely theoretical in scope and does not directly consider real applications. This naturally follows from the theoretical nature of the work. Nonetheless, the results may offer meaningful practical insights. While Bayesian neural networks (BNNs) have demonstrated remarkable success in capturing complex patterns in real-world data, our theoretical understanding of their practical effectiveness remains limited. By incorporating anisotropic Besov spaces and their composite structure, this work significantly broadens the theoretical foundation of BNNs. For instance, images are often modeled as elements of Besov spaces due to their inhomogeneous and discontinuous characteristics [1-3]. As such, this work may help explain why BNNs perform well in such complex scenarios. We will elaborate further on the practical implications of the study in the final version, if accepted.\n\n$\\textbf{W2: Additional theoretical insights}$ \n\nWe believe that this study offers substantial theoretical insights that have not been addressed in prior work. The central theoretical contribution lies in the function classes that neural networks are designed to approximate, for which we consider anisotropic Besov spaces and their composite structures. To the best of our knowledge, no previous Bayesian studies have treated such rich function classes in the theoretical analysis of neural networks. Even the most recent work by [4] considers anisotropic Besov spaces but assumes H\\\"older-type regularity for the compositional structure. (This distinction was not clearly stated in the submitted version and will be clarified in the final version.) Accordingly, our results encompass more complex yet practically relevant functions, such as additive or multiplicative Besov functions. In this sense, our work advances the theoretical understanding of the function classes to which Bayesian neural networks can adapt. \n\n$\\textbf{Q1: Dependency on $p$}$ \n\nThe magnitude parameter $B_1$ depends explicitly on $p$, although this is not apparent in equation (2) owing to the use of proportionality notation for simplicity. As noted in the main text, the precise form of this dependence is given in Lemma B.4 of the supplementary material.\n\n$\\textbf{Q2: The constants $\\lambda_N$ and $\\lambda_H$}$\n\nWe thank the reviewer for pointing this out. It is correct that these constants can be arbitrary, and the prior in (6) can be applied for any positive values of $\\lambda_N$ and $\\lambda_H$. We will revise this in the final version.\n\n$\\textbf{References}$\n\n[1] Donoho, D. L., & Johnstone, I. M. (1995). Adapting to unknown smoothness via wavelet shrinkage. Journal of the American Statistical Association, 90(432), 1200-1224.  \n[2] Donoho, D. L. (2002). De-noising by soft-thresholding. IEEE Transactions on Information Theory, 41(3), 613-627.  \n[3] Meyer, Y. (2001). Oscillating patterns in image processing and nonlinear evolution equations: the fifteenth Dean Jacqueline B. Lewis memorial lectures (Vol. 22). American Mathematical Soc.  \n[4] Castillo, I., & Egels, P. (2024). Posterior and variational inference for deep neural networks with heavy-tailed weights. arXiv preprint arXiv:2406.03369."}, {"review_id": "mmSWQN4uHL", "reviewer": "Reviewer_JeeF", "summary": "This paper examines how fast the posterior distribution of Bayesian Neural Networks (BNNs) concentrates around the true posterior (\"posterior contraction\"). For BNNs with either sparse or continuous shrinkage prior, the key findings suggest that the optimal contraction rate can be attained, which is dependent on the intrinsic dimension of the true structures -- avoiding the curse of dimensionality. For a class of non-smooth function space called Besov space, the paper examines anisotropy, composite and adaptation. Contributions are centered around the theoretical findings: BNNs achieve optimal posterior contraction rate over Besov spaces.", "questions": "What would be the practical implication of the paper's findings? On Bayesian Deep Learning research, many of us are focusing on developing practical learning algorithms that can provide the promises of Bayesian methods: well-calibrated uncertainty, incorporation of domain knowledge through priors, etc. How does the findings on contraction rate over a particular type of function space manifest for many of on-going discussions within the community.", "limitations": "I am not from the learning theory side of the Bayesian Neural Networks research. I hope other reviewers can comment better.", "rating": 5, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- Differentiation to the existing work is well presented.\n\n- Several examples are presented that adds to the better understanding of the materials.\n\nWeakness:\n\n- The paper can be more kind to the readers, atleast in the introduction and abstract.\n\nFor example, general machine learning audience may not know the terms that are assumed in the paper, e.g., posterior contraction, sparsity inducing priors, Besov spaces, etc. \n\n- The choice of venue could improve.\n\nWe have COLT which is dedicated to theoretic research of machine learning. This paper I think also fits there.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments. Please find our responses below.\n\n$\\textbf{W1: Readability}$\n\nWe agree that the paper is technical due to the nature of the topic. Although our study is primarily intended for experienced readers, we will provide further elaboration where appropriate in the final version, if accepted.\n\n$\\textbf{W2: Venue choice}$\n\nWhile COLT could be a suitable alternative, we believe NeurIPS is one of the most appropriate venues given its broad scope and openness to theoretical contributions. In particular, NeurIPS has consistently published works on the statistical theory of neural networks [1-3], and our work aligns well with this line of research.\n\n\n$\\textbf{Q1: Practical implication}$\n\nThe findings of this paper help bridge the gap between theory and practice in Bayesian neural networks (BNNs). It is widely recognized that BNNs offer a compelling alternative to classical NN models by providing principled uncertainty quantification through the posterior distribution, which is often overlooked in traditional approaches. Although BNNs perform well across a wide range of functions, their theoretical analysis has largely been limited to relatively smooth function classes. By incorporating anisotropic Besov spaces and their composite structure, this work significantly broadens the theoretical understanding of BNNs. For example, images are often modeled as elements of Besov spaces due to their inhomogeneous and discontinuous nature [4-6], and our formulation offers additional flexibility through its treatment of anisotropy and compositional structure. Therefore, this work may offer insight into why BNNs perform well in image analysis, despite the discrepancy that our analysis is limited to feedforward network architectures. We will elaborate further on the practical implications of the study in the final version, if accepted.\n\n$\\textbf{References}$\n\n[1] Polson, N. G., & Ročková, V. (2018). Posterior concentration for sparse deep learning. Advances in Neural Information Processing Systems, 31.  \n[2] Suzuki, T., & Nitanda, A. (2021). Deep learning is adaptive to intrinsic dimensionality of model smoothness in anisotropic Besov space. Advances in Neural Information Processing Systems, 34.   \n[3] Lee, K., & Lee, J. (2022). Asymptotic properties for Bayesian neural network in Besov space. Advances in Neural Information Processing Systems, 35.  \n[4] Donoho, D. L., & Johnstone, I. M. (1995). Adapting to unknown smoothness via wavelet shrinkage. Journal of the American Statistical Association, 90(432), 1200-1224.  \n[5] Donoho, D. L. (2002). De-noising by soft-thresholding. IEEE Transactions on Information Theory, 41(3), 613-627.  \n[6] Meyer, Y. (2001). Oscillating patterns in image processing and nonlinear evolution equations: the fifteenth Dean Jacqueline B. Lewis memorial lectures (Vol. 22). American Mathematical Soc."}, {"review_id": "Lmo85K1mSC", "reviewer": "Reviewer_D5oT", "summary": "This paper presents a significant theoretical investigation into the properties of sparse Bayesian neural networks (BNNs) for nonparametric regression. The primary contribution is establishing that the posterior distribution of a BNN contracts around the true underlying function at a near-minimax optimal rate. The analysis extends existing theory to two highly complex and practically relevant function classes: anisotropic and composite Besov spaces. These classes model functions that exhibit direction-dependent smoothness or are formed through hierarchical compositions, respectively. Such structures are believed to possess an \"intrinsic dimensionality\" that is lower than the ambient data dimension, providing a formal framework for understanding how neural networks can mitigate the curse of dimensionality.   \n\nThe authors demonstrate these results for sparse ReLU networks using two distinct types of priors on the network weights: the theoretically well-understood but computationally demanding spike-and-slab prior, and a class of more tractable continuous shrinkage priors. A cornerstone of the work is the proof of rate adaptation. The authors show that by placing appropriate hyper-priors on the network's architectural parameters (width and sparsity), the BNN can automatically achieve the optimal contraction rate without requiring prior knowledge of the true function's smoothness level—a key advantage of the Bayesian paradigm over frequentist approaches that often require such oracle information for tuning. The results unify and extend previous work, most notably by providing the first analysis for these function spaces within a fully Bayesian framework using the standard posterior, in contrast to recent work that relied on a pseudo-Bayesian fractional posterior.", "questions": "1. On the Practicality and Interpretation of the Shrinkage Prior Results: Your work provides a crucial theoretical bridge for BNNs. However, there appears to be a significant tension regarding the continuous shrinkage priors. Assumption (C3) is instrumental for the proof of Theorem 3.8, but as you note, it excludes widely-used priors like the horseshoe and leads to constructions (Example 3.5) that may be \"practically undesirable\". Could you elaborate on this trade-off? Specifically, what is the key theoretical barrier to extending your results to priors with heavier tails (like the horseshoe), which are known to perform well in practice? Does the class of priors satisfying (C1)-(C3) offer tangible benefits over the true spike-and-slab prior beyond computational tractability, given how closely they must mimic its structure? My evaluation would improve if the authors can provide a clear roadmap for how to relax this assumption or can better articulate the unique value of the priors they analyze.\n\n\n2. Your work shows that a parametrically sparse BNN (one with few active weights) can optimally learn a structurally sparse composite function. My question is about the connection between these two types of sparsity. Is the sparsity imposed on the BNN's weights primarily a technical tool required for the theoretical proofs to work, or does it reflect a deeper assumption that the individual component functions ($f_{h,j}​$) within the true composite structure are themselves best represented by sparse networks?\n\n\n3. A Suggestion for Improved Clarity: The assumptions on the priors are complex and spread across several sections. To improve readability, have you considered adding a table in the main text that summarizes the key assumptions for each prior type (Spike-and-Slab, Shrinkage, and their adaptive versions), highlighting the dependencies (or lack thereof) on unknown quantities like $B_1$​ and $\\tilde{s}^*$? This could greatly help the reader navigate the dense theoretical conditions.", "limitations": "The \"Future work\" discussion in Section 4 is candid and accurate, correctly identifying the computational challenges of spike-and-slab priors, the theoretical gap between their assumptions and popular shrinkage priors like the horseshoe, and the need to extend the theory to more complex, modern network architectures like CNNs and Transformers. The work is purely theoretical, and as such, the authors correctly note in the checklist that discussions of direct societal impact, data are not applicable.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "This paper represents a substantial and high-quality contribution to the theoretical understanding of Bayesian deep learning. I would give more details from the following 4 perspectives\n\n$\\textbf{Quality}$:\n\nThe technical quality of this paper is high. The theoretical analysis is rigorous, comprehensive, and builds upon the foundational framework for Bayesian posterior contraction rates established by Ghosal, Ghosh, and van der Vaart .\nHowever, a significant weakness lies in the assumptions required for the continuous shrinkage priors. While these priors are motivated as a practical, computationally efficient alternative to spike-and-slab priors, the conditions imposed to secure the theoretical guarantees are highly restrictive. Specifically, Assumption (C3) requires that the prior probability mass outside an exponentially shrinking neighborhood of zero,\n$[−a_n​,a_n​]$ where $a_n​=e^{−L_{1n}​logn}$, must vanish at a rate of $o(1/[N_n​(logn)^2])$, which excludes popular, empirically successful priors like the horseshoe. The horseshoe prior is effective because it combines a sharp peak with heavy tails, allowing it to aggressively shrink noise coefficients while leaving large, true signals untouched. To satisfy the assumption, the authors construct a \"relaxed spike-and-slab\" prior (Example 3.5), but they concede this construction is \"practically undesirable\" as it forces the prior to behave almost entirely like a spike. Consequently, the theoretical results for shrinkage priors (Theorem 3.8) apply to a narrow class of priors not representative of those used in modern Bayesian practice, which limits the immediate practical implications of this part of the analysis.\n\n$\\textbf{Clarity}$\n\nThe paper is well-written and clear, especially given its technical density. The introduction provides an excellent motivation for the problem and situates the work within the existing literature. Table 1 is particularly effective, offering a concise and invaluable summary of related frequentist and Bayesian results, which immediately clarifies the paper's novel contributions. The definitions of anisotropic and composite Besov spaces are presented with care, and the illustrative figures (Figures 1 and 2) are helpful for building intuition.   \n\nA minor point of criticism regarding clarity pertains to the mechanism of rate adaptation. While Theorem 3.9 is a powerful result, the discussion of how it is achieved in practice could be more nuanced. The authors propose achieving rate adaptation with a \"sufficiently large\" constant. This \"sufficiently large constant\" is a common theoretical device, but it slightly obscures the fact that the adaptation is not entirely assumption-free; it relies on specific, robust choices of the slab prior. A more explicit discussion of this subtlety would further enhance the paper's clarity.\n\n\n$\\textbf{Significance}$\n\nThe significance of this work is profound. It addresses a fundamental question in machine learning: why do overparameterized models like neural networks generalize well in high dimensions. This paper provides a rigorous, Bayesian answer by showing that BNNs can adapt to the intrinsic, low-dimensional structure of the target function, as captured by anisotropic and composite Besov spaces.   \n\nThe paper makes a landmark contribution by being the first to establish these results for the standard Bayesian posterior. Prior frequentist work demonstrated that deep neural network estimators could achieve these optimal rates, but their construction required oracle knowledge of the function's smoothness to properly specify the network architecture. The most closely related Bayesian work by Castillo and Egels successfully analyzed these function spaces but did so using a fractional posterior, a technique that deviates from standard Bayesian inference and can be more difficult to interpret. By proving these results for the true posterior, this paper provides a much stronger theoretical justification for the practical use of BNNs and strengthens their standing within the Bayesian community. The results on rate adaptation are particularly impactful, as they demonstrate a key theoretical advantage of the Bayesian approach over its frequentist counterpart in this setting.   \n\n$\\textbf{Originality}$\n\nWhile the paper builds on existing lines of research in NN approximation theory and Bayesian nonparametrics, it is original since it synthesizes them in a novel way to tackle a problem that was previously unsolved. The primary original contributions are: (1) the extension of posterior contraction theory for BNNs to anisotropic and composite Besov spaces under a fully Bayesian framework; (2) the demonstration that this holds for both spike-and-slab and a (albeit restrictive) class of continuous shrinkage priors; and (3) the proof of automatic rate adaptation to unknown smoothness in these complex settings. These contributions collectively push the frontier of our theoretical understanding of Bayesian deep learning.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's thoughtful comments and detailed feedback. Our responses are provided below.\n\n$\\textbf{W1: Conditions for continuous shrinkage priors}$\n\nAs the reviewer pointed out and as noted in the text, we acknowledge that the required conditions for our shrinkage priors apply only to a limited class, excluding several widely used priors such as the horseshoe. We elaborate on this limitation in our response to Q1, addressing the reviewer's question.\n\n$\\textbf{W2: Clarity on rate adaptation}$\n\nThis is indeed a good point. As the reviewer noted, our adaptation results rely on specific constants and hold over a certain range of smoothness parameters, rather than being entirely assumption-free. To clarify further, if a Gaussian slab is used instead of a uniform slab, the adaptation becomes independent of the choice of priors. However, it still requires a properly calibrated depth parameter $\\tilde L_n$ with a sufficiently large constant $C_L$, so the reviewer's observation is accurate. While we briefly mentioned this limitation in the paragraph following Theorem 3.9, we will elaborate on it further in the final version to enhance clarity.\n\n$\\textbf{Q1: Practicality and interpretation of the shrinkage prior results}$\n\nAs the reviewer pointed out, the assumptions are instrumental but exclude several widely used shrinkage priors in practice. To elaborate on this trade-off, the main difficulty stems from the fact that the polynomial tail of the horseshoe prior renders the assumptions incompatible. Specifically, while (C1) and (C3) can be satisfied by an appropriately scaled horseshoe prior, such scaling is not suitable for (C2), and vice versa.\nThe heavy tail of the horseshoe necessitates a sufficiently large sieve to restrict the parameter space, effectively ignoring the prior mass outside the sieve. However, this enlargement results in overly loose entropy bounds relative to the target rate, making it infeasible to construct a global test function over the sieve using a union bound. In contrast, priors with exponential tails mitigate this issue, as demonstrated in our examples with relaxed spike-and-slab priors (Examples 3.5 and 3.6). \n\nNevertheless, we believe that priors satisfying Assumptions (C1)--(C3) offer advantages beyond computational convenience. In particular, they demonstrate that continuous priors can achieve the same contraction rate, allowing the point mass component to be removed. This, in turn, enables a theoretical understanding of optimization and algorithmic behavior through well-established results for gradient-based methods.\n\nWe plan to continue exploring the topic to address this fundamental limitation regarding shrinkage priors. While we do not yet have a clear roadmap, we have identified a few potential directions that may prove useful.\n\n* Tighter entropy bound:  Our current entropy calculation relies on previous results [1-2], but it is not entirely clear how tight these estimates are. If a sharper bound could be obtained, it might allow for a larger sieve that accommodates heavier-tailed priors. Although we have spent some time exploring this direction, we have not yet made meaningful progress. Nonetheless, we believe it is a question worth further contemplation.\n* Constructing a global test function: Our current theoretical approach follows standard results in testing-based posterior contraction theory, which requires a test function with exponentially small errors over a sieve, along with an appropriate prior that assigns negligible mass outside the sieve. To achieve this, we rely on the well-known fact that the Hellinger distance yields a well-behaved local test over small balls, which we then combine with an entropy bound. If we can directly construct a test function over the entire sieve, it may be possible to bypass the entropy calculation and thereby allow for a larger sieve.\n* Adaptation with heavy-tailed priors: There is a line of recent work on Bayesian adaptation using heavy-tailed priors [3-4]. Although these approaches are not directly applicable to our setting, they may offer valuable insights and suggest promising directions for future investigation. In particular, fractional posteriors always simplify the situation, as they only require sufficient prior mass on a Kullback-Leibler neighborhood. However, we are more inclined to focus on the usual posterior.\n\n$\\textbf{Q2: Sparse neural networks}$\n\nThe primary role of weight sparsity in the parameter space is to facilitate the derivation of suitable entropy bounds. Since every anisotropic Besov composite function can be approximated by a sparse neural network with optimal error (Lemma B.8), it is essential to control model complexity. In this context, the class of sparse neural networks provides a suitable approximation class that is not excessively complex. This suggests that weight sparsity primarily serves as a useful theoretical device and may not be strictly necessary to represent such Besov functions. However, this does not imply that weight sparsity is merely a technical artifact. To the best of our knowledge, it remains unclear whether anisotropic composite Besov functions can be approximated by smaller networks without sparsity. In the absence of such results, sparse neural networks remain an effective and possibly optimal representation strategy, as observed by recent studies on dense networks [5-6].\n\n \n$\\textbf{Q3: Improved clarity}$\n\nIn response to the reviewer's suggestion, we will include a table summarizing this information in the final version. While we are still considering the most effective format, the table will present the main results from Theorems 3.3, 3.8, 3.9, and 3.14, along with summaries of the priors (spike-and-slab, shrinkage) and the function spaces (anisotropic Besov, composite Besov).\n\n$\\textbf{References}$\n\n[1] Suzuki, T., & Nitanda, A. (2021). Deep learning is adaptive to intrinsic dimensionality of model smoothness in anisotropic Besov space. Advances in Neural Information Processing Systems, 34.  \n[2] Lee, K., & Lee, J. (2022). Asymptotic properties for Bayesian neural network in Besov space. Advances in Neural Information Processing Systems, 35.  \n[3] Agapiou, S., & Castillo, I. (2024). Heavy-tailed Bayesian nonparametric adaptation. The Annals of Statistics, 52(4), 1433-1459.  \n[4] Agapiou, S., Castillo, I., & Egels, P. (2025). Heavy-tailed and Horseshoe priors for regression and sparse Besov rates. arXiv preprint arXiv:2505.15543.  \n[5] Kong, I., & Kim, Y. (2024). Posterior concentrations of fully-connected Bayesian neural networks with general priors on the weights. arXiv preprint arXiv:2403.14225.  \n[6] Castillo, I., & Egels, P. (2024). Posterior and variational inference for deep neural networks with heavy-tailed weights. arXiv preprint arXiv:2406.03369."}, {"review_id": "CqbUDohi0Z", "reviewer": "Reviewer_1k7j", "summary": "This manuscript establishes that fully Bayesian sparse ReLU networks attain near‑minimax posterior contraction rates over both anisotropic and composite Besov spaces, with rates governed solely by the intrinsic (effective) dimension rather than the ambient dimension. Through careful approximation theory (Lemmas B.4, B.6) the authors show that suitably deep, sparse networks admit spike‑and‑slab or continuous shrinkage priors yielding a contraction rate\n$\\varepsilon_n = n^{-\\tilde s/(2\\tilde s + 1)} (\\log n)^{3/2},$\nwhere $\\tilde s$ is the intrinsic smoothness (Theorems 3.3, 3.8, 3.14). Importantly, they demonstrate rate‑adaptation by placing priors on depth, width, and sparsity (Theorem 3.9), so no oracle knowledge of smoothness or compositional structure is required .", "questions": "* How could the proposed theoretical framework scale up to modern foundational models (e.g., any changes in the validity of assumptions when dealing with large Bayesian neural networks).", "limitations": "* The authors identify the limitations and subsequent extensions in the future works section.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n* By working in anisotropic and composite Besov spaces, the paper covers a broad spectrum of functions—including non‑smooth, discontinuous, additive, and multiplicative structures—far beyond classical Hölder or Sobolev classes\n\n* The contraction rates depend on an intrinsic dimension $d^*$ , effectively mitigating the curse of dimensionality. This unifies and extends prior frequentist results (e.g., Suzuki & Nitanda) under a fully Bayesian framework \n\n**Weaknesses**\n\n* Although the paper is strong theoretically, there are no simulations or experiments accompany the theory. While the focus is theoretical, even small‑scale numerical illustrations would strengthen confidence that the asymptotic rates manifest in finite samples—and would help practitioners gauge constants and log‑factors.\n\n* All results assume a homoskedastic Gaussian noise model. Extensions to classification or heteroskedastic settings are only sketched (Appendix A), so applicability to common real‑world tasks (e.g., image classification) remains speculative\n\n* The analysis is confined to fully connected ReLU nets. Modern architectures (convolutional, residual, attention‑based) are unaddressed; Section 4’s “Future work” correctly flags this but provides no roadmap for how to generalize the key approximation and entropy bounds", "was_revised": true, "final_justification": "I thank the authors for their responses. While I am not an expert to fully follow the key theoretical analysis, the rebuttal by the authors addresses my doubts on the methods. I think the paper is of high quality and technically rigorous. I lean towards acceptance of the paper.", "rebuttal": "We acknowledge the reviewer's insightful observations and provide our responses below.\n\n$\\textbf{W1: Lack of simulations}$\n\nWe acknowledge that the paper does not include simulations or empirical experiments. This omission naturally stems from the theoretical focus of the work, which aims to investigate the theoretical properties of Bayesian neural networks (BNNs) rather than to propose a new algorithm. To complement this weakness and clarify the practical relevance, we will elaborate further on the practical implications of our results in the final version, emphasizing that the function classes considered in this study capture a range of practical scenarios.\n\n$\\textbf{W2: Extension to classification}$\n\nThe extension to binary classification is fully developed in the supplementary material. We will clarify this point in the introduction.\n\n$\\textbf{W3: Beyond fully connected ReLU networks}$\n\nSeveral studies have investigated the statistical theory of modern architectures, including convolutional networks [1-3], residual networks [4-5], and transformers [6-8]. These works may provide useful foundations for deriving posterior contraction results beyond fully connected networks. We will clarify their relevance in the future work section of the final version.\n\n$\\textbf{Q1: Scaling to modern foundational models}$\n\nOur theoretical results, consistent with related work in the literature, only require that the architectural complexity (such as depth and width) scale appropriately with certain constants as the number of observations increases. In principle, this means the underlying assumptions remain unchanged, and the results are applicable even to very large network models. However, modern foundational models are typically not simple feedforward networks, which introduces a gap between our theoretical framework and the architectural trends observed in large-scale models.\n\n$\\textbf{References}$\n\n\n[1] Kohler, M., Krzyżak, A., & Walter, B. (2022). On the rate of convergence of image classifiers based on convolutional neural networks. Annals of the Institute of Statistical Mathematics, 74(6), 1085-1108.  \n[2] Fang, Z., & Cheng, G. (2023). Optimal Convergence Rates of Deep Convolutional Neural Networks: Additive Ridge Functions. Transactions on machine learning research.  \n[3] Yang, Y., Feng, H., & Zhou, D. X. (2024). On the rates of convergence for learning with convolutional neural networks. arXiv preprint arXiv:2403.16459.  \n[4] Lai, J., Yu, Z., Tian, S., & Lin, Q. (2023). Generalization ability of wide residual networks. arXiv preprint arXiv:2305.18506.  \n[5] Zhang, Z., Zhang, K., Chen, M., Takeda, Y., Wang, M., Zhao, T., & Wang, Y. X. (2024). Nonparametric classification on low dimensional manifolds using overparameterized convolutional residual networks. Advances in Neural Information Processing Systems, 37.  \n[6] Kim, J., Nakamaki, T., & Suzuki, T. (2024). Transformers are minimax optimal nonparametric in-context learners. Advances in Neural Information Processing Systems, 37.  \n[7] Jiao, Y., Lai, Y., Sun, D., Wang, Y., & Yan, B. (2025). Approximation Bounds for Transformer Networks with Application to Regression. arXiv preprint arXiv:2504.12175.  \n[8] Liu, P., & Zhou, D. X. (2025). Generalization Analysis of Transformers in Distribution Regression. Neural Computation, 37(2), 260-293."}]

Author rebuttal:

Meta-review: This paper studies the posterior contraction rate of sparse Bayesian neural networks, showing that over anisotropic Besov spaces (and hierarchical versions) that they are optimal. This is a pure theory paper, and while some reviewers mention that it would be improved with some empirical validation, there is consensus that the theoretical insights are novel and timely to the NeurIPS audience, with one mentioning that the results may be profound. While some assumptions may be restrictive, the reviewers were largely satisfied after the rebuttal period, and there are various 'applied' takeaways from the theoretical results. For instance, it suggests that perhaps a large network with strong sparsity in the prior may be beneficial rather than a moderately sized, less sparse design on the same 'budget'. The priors enable rate adaptation, which sends another practical message in addition to theoretical novelty. The authors state they will take into account the constructive criticisms in a final camera ready version
Decision: Accept (poster)