Title: Posterior Contraction for Sparse Neural Networks in Besov Spaces with Intrinsic Dimensionality
Venue: NeurIPS 2025
Accepted: yes
Keywords: posterior contraction

Bayesian neural networks

Besov space

intrinsic dimensionality
Abstract: This work establishes that sparse Bayesian neural networks achieve optimal posterior contraction rates over anisotropic Besov spaces and their hierarchical compositions. These structures reflect the intrinsic dimensionality of the underlying function, thereby mitigating the curse of dimensionality. Our analysis shows that Bayesian neural networks equipped with either sparse or continuous shrinkage priors attain the optimal rates which are dependent on the intrinsic dimension of the true structures. Moreover, we show that these priors enable rate adaptation, allowing the posterior to contract at the optimal rate even when the smoothness level of the true function is unknown. The proposed framework accommodates a broad class of functions, including additive and multiplicative Besov functions as special cases.
These results advance the theoretical foundations of Bayesian neural networks and provide rigorous justification for their practical effectiveness in high-dimensional, structured estimation problems.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript establishes that fully Bayesian sparse ReLU networks attain near‑minimax posterior contraction rates over both anisotropic and composite Besov spaces, with rates governed solely by the intrinsic (effective) dimension rather than the ambient dimension. Through careful approximation theory (Lemmas B.4, B.6) the authors show that suitably deep, sparse networks admit spike‑and‑slab or continuous shrinkage priors yielding a contraction rate\n$\\varepsilon_n = n^{-\\tilde s/(2\\tilde s + 1)} (\\log n)^{3/2},$\nwhere $\\tilde s$ is the intrinsic smoothness (Theorems 3.3, 3.8, 3.14). Importantly, they demonstrate rate‑adaptation by placing priors on depth, width, and sparsity (Theorem 3.9), so no oracle knowledge of smoothness or compositional structure is required .\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n* By working in anisotropic and composite Besov spaces, the paper covers a broad spectrum of functions—including non‑smooth, discontinuous, additive, and multiplicative structures—far beyond classical Hölder or Sobolev classes\n\n* The contraction rates depend on an intrinsic dimension $d^*$ , effectively mitigating the curse of dimensionality. This unifies and extends prior frequentist results (e.g., Suzuki & Nitanda) under a fully Bayesian framework \n\n**Weaknesses**\n\n* Although the paper is strong theoretically, there are no simulations or experiments accompany the theory. While the focus is theoretical, even small‑scale numerical illustrations would strengthen confidence that the asymptotic rates manifest in finite samples—and would help practitioners gauge constants and log‑factors.\n\n* All results assume a homoskedastic Gaussian noise model. Extensions to classification or heteroskedastic settings are only sketched (Appendix A), so applicability to common real‑world tasks (e.g., image classification) remains speculative\n\n* The analysis is confined to fully connected ReLU nets. Modern architectures (convolutional, residual, attention‑based) are unaddressed; Section 4’s “Future work” correctly flags this but provides no roadmap for how to generalize the key approximation and entropy bounds\n\nQUESTIONS:\n* How could the proposed theoretical framework scale up to modern foundational models (e.g., any changes in the validity of assumptions when dealing with large Bayesian neural networks).\n\nLIMITATIONS:\n* The authors identify the limitations and subsequent extensions in the future works section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a significant theoretical investigation into the properties of sparse Bayesian neural networks (BNNs) for nonparametric regression. The primary contribution is establishing that the posterior distribution of a BNN contracts around the true underlying function at a near-minimax optimal rate. The analysis extends existing theory to two highly complex and practically relevant function classes: anisotropic and composite Besov spaces. These classes model functions that exhibit direction-dependent smoothness or are formed through hierarchical compositions, respectively. Such structures are believed to possess an \"intrinsic dimensionality\" that is lower than the ambient data dimension, providing a formal framework for understanding how neural networks can mitigate the curse of dimensionality.   \n\nThe authors demonstrate these results for sparse ReLU networks using two distinct types of priors on the network weights: the theoretically well-understood but computationally demanding spike-and-slab prior, and a class of more tractable continuous shrinkage priors. A cornerstone of the work is the proof of rate adaptation. The authors show that by placing appropriate hyper-priors on the network's architectural parameters (width and sparsity), the BNN can automatically achieve the optimal contraction rate without requiring prior knowledge of the true function's smoothness level—a key advantage of the Bayesian paradigm over frequentist approaches that often require such oracle information for tuning. The results unify and extend previous work, most notably by providing the first analysis for these function spaces within a fully Bayesian framework using the standard posterior, in contrast to recent work that relied on a pseudo-Bayesian fractional posterior.\n\nSTRENGTHS AND WEAKNESSES:\nThis paper represents a substantial and high-quality contribution to the theoretical understanding of Bayesian deep learning. I would give more details from the following 4 perspectives\n\n$\\textbf{Quality}$:\n\nThe technical quality of this paper is high. The theoretical analysis is rigorous, comprehensive, and builds upon the foundational framework for Bayesian posterior contraction rates established by Ghosal, Ghosh, and van der Vaart .\nHowever, a significant weakness lies in the assumptions required for the continuous shrinkage priors. While these priors are motivated as a practical, computationally efficient alternative to spike-and-slab priors, the conditions imposed to secure the theoretical guarantees are highly restrictive. Specifically, Assumption (C3) requires that the prior probability mass outside an exponentially shrinking neighborhood of zero,\n$[−a_n​,a_n​]$ where $a_n​=e^{−L_{1n}​logn}$, must vanish at a rate of $o(1/[N_n​(logn)^2])$, which excludes popular, empirically successful priors like the horseshoe. The horseshoe prior is effective because it combines a sharp peak with heavy tails, allowing it to aggressively shrink noise coefficients while leaving large, true signals untouched. To satisfy the assumption, the authors construct a \"relaxed spike-and-slab\" prior (Example 3.5), but they concede this construction is \"practically undesirable\" as it forces the prior to behave almost entirely like a spike. Consequently, the theoretical results for shrinkage priors (Theorem 3.8) apply to a narrow class of priors not representative of those used in modern Bayesian practice, which limits the immediate practical implications of this part of the analysis.\n\n$\\textbf{Clarity}$\n\nThe paper is well-written and clear, especially given its technical density. The introduction provides an excellent motivation for the problem and situates the work within the existing literature. Table 1 is particularly effective, offering a concise and invaluable summary of related frequentist and Bayesian results, which immediately clarifies the paper's novel contributions. The definitions of anisotropic and composite Besov spaces are presented with care, and the illustrative figures (Figures 1 and 2) are helpful for building intuition.   \n\nA minor point of criticism regarding clarity pertains to the mechanism of rate adaptation. While Theorem 3.9 is a powerful result, the discussion of how it is achieved in practice could be more nuanced. The authors propose achieving rate adaptation with a \"sufficiently large\" constant. This \"sufficiently large constant\" is a common theoretical device, but it slightly obscures the fact that the adaptation is not entirely assumption-free; it relies on specific, robust choices of the slab prior. A more explicit discussion of this subtlety would further enhance the paper's clarity.\n\n\n$\\textbf{Significance}$\n\nThe significance of this work is profound. It addresses a fundamental question in machine learning: why do overparameterized models like neural networks generalize well in high dimensions. This paper provides a rigorous, Bayesian answer by showing that BNNs can adapt to the intrinsic, low-dimensional structure of the target function, as captured by anisotropic and composite Besov spaces.   \n\nThe paper makes a landmark contribution by being the first to establish these results for the standard Bayesian posterior. Prior frequentist work demonstrated that deep neural network estimators could achieve these optimal rates, but their construction required oracle knowledge of the function's smoothness to properly specify the network architecture. The most closely related Bayesian work by Castillo and Egels successfully analyzed these function spaces but did so using a fractional posterior, a technique that deviates from standard Bayesian inference and can be more difficult to interpret. By proving these results for the true posterior, this paper provides a much stronger theoretical justification for the practical use of BNNs and strengthens their standing within the Bayesian community. The results on rate adaptation are particularly impactful, as they demonstrate a key theoretical advantage of the Bayesian approach over its frequentist counterpart in this setting.   \n\n$\\textbf{Originality}$\n\nWhile the paper builds on existing lines of research in NN approximation theory and Bayesian nonparametrics, it is original since it synthesizes them in a novel way to tackle a problem that was previously unsolved. The primary original contributions are: (1) the extension of posterior contraction theory for BNNs to anisotropic and composite Besov spaces under a fully Bayesian framework; (2) the demonstration that this holds for both spike-and-slab and a (albeit restrictive) class of continuous shrinkage priors; and (3) the proof of automatic rate adaptation to unknown smoothness in these complex settings. These contributions collectively push the frontier of our theoretical understanding of Bayesian deep learning.\n\nQUESTIONS:\n1. On the Practicality and Interpretation of the Shrinkage Prior Results: Your work provides a crucial theoretical bridge for BNNs. However, there appears to be a significant tension regarding the continuous shrinkage priors. Assumption (C3) is instrumental for the proof of Theorem 3.8, but as you note, it excludes widely-used priors like the horseshoe and leads to constructions (Example 3.5) that may be \"practically undesirable\". Could you elaborate on this trade-off? Specifically, what is the key theoretical barrier to extending your results to priors with heavier tails (like the horseshoe), which are known to perform well in practice? Does the class of priors satisfying (C1)-(C3) offer tangible benefits over the true spike-and-slab prior beyond computational tractability, given how closely they must mimic its structure? My evaluation would improve if the authors can provide a clear roadmap for how to relax this assumption or can better articulate the unique value of the priors they analyze.\n\n\n2. Your work shows that a parametrically sparse BNN (one with few active weights) can optimally learn a structurally sparse composite function. My question is about the connection between these two types of sparsity. Is the sparsity imposed on the BNN's weights primarily a technical tool required for the theoretical proofs to work, or does it reflect a deeper assumption that the individual component functions ($f_{h,j}​$) within the true composite structure are themselves best represented by sparse networks?\n\n\n3. A Suggestion for Improved Clarity: The assumptions on the priors are complex and spread across several sections. To improve readability, have you considered adding a table in the main text that summarizes the key assumptions for each prior type (Spike-and-Slab, Shrinkage, and their adaptive versions), highlighting the dependencies (or lack thereof) on unknown quantities like $B_1$​ and $\\tilde{s}^*$? This could greatly help the reader navigate the dense theoretical conditions.\n\nLIMITATIONS:\nThe \"Future work\" discussion in Section 4 is candid and accurate, correctly identifying the computational challenges of spike-and-slab priors, the theoretical gap between their assumptions and popular shrinkage priors like the horseshoe, and the need to extend the theory to more complex, modern network architectures like CNNs and Transformers. The work is purely theoretical, and as such, the authors correctly note in the checklist that discussions of direct societal impact, data are not applicable.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper examines how fast the posterior distribution of Bayesian Neural Networks (BNNs) concentrates around the true posterior (\"posterior contraction\"). For BNNs with either sparse or continuous shrinkage prior, the key findings suggest that the optimal contraction rate can be attained, which is dependent on the intrinsic dimension of the true structures -- avoiding the curse of dimensionality. For a class of non-smooth function space called Besov space, the paper examines anisotropy, composite and adaptation. Contributions are centered around the theoretical findings: BNNs achieve optimal posterior contraction rate over Besov spaces.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- Differentiation to the existing work is well presented.\n\n- Several examples are presented that adds to the better understanding of the materials.\n\nWeakness:\n\n- The paper can be more kind to the readers, atleast in the introduction and abstract.\n\nFor example, general machine learning audience may not know the terms that are assumed in the paper, e.g., posterior contraction, sparsity inducing priors, Besov spaces, etc. \n\n- The choice of venue could improve.\n\nWe have COLT which is dedicated to theoretic research of machine learning. This paper I think also fits there.\n\nQUESTIONS:\nWhat would be the practical implication of the paper's findings? On Bayesian Deep Learning research, many of us are focusing on developing practical learning algorithms that can provide the promises of Bayesian methods: well-calibrated uncertainty, incorporation of domain knowledge through priors, etc. How does the findings on contraction rate over a particular type of function space manifest for many of on-going discussions within the community.\n\nLIMITATIONS:\nI am not from the learning theory side of the Bayesian Neural Networks research. I hope other reviewers can comment better.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 1, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper presents a comprehensive, rigorous analysis of Bayesian sparse DNN. Compared to existing results, it considers a more general class of smooth functions and considers both spike-and-slab prior and shrinkage prior. It also proposes a prior to adapt to the unknown function smoothness. It provides an optimal posterior contraction rate.

STRENGTHS AND WEAKNESSES:
Strength: Along this line of research, I believe this work is most comprehensive. It combines the standard theory of Bayesian nonparametric convergence and the approximation theory of DNN. The theoretical analysis is rigorous and sound.

Weakness: 
1. The scope of this submission is purely theoretical. Its impact on real applications is omitted. All results are asymptotic, and how to apply them to finite sample applications is unclear. All conditions required for convergence are sufficient conditions. Based on my personal experience, these conditions are over-sufficient for posterior consistency for practice.

2. What is the additional theoretical insight obtained by this paper, beyond existing results? Or the paper only aims to generalize the existing theory/prior specification to Besov Spaces. 

Overall, it seems this paper is more suitable for a pure theory venue other than NeruIPS.

QUESTIONS:
1. Line 181: ''.These network parameters depend on the Besov parameters $\tilde{s}$ and $p$''. I fail to see the dependency on $p$.

2. Prior (6): "for some constants $\lambda_N>0$ and $\lambda_H > 0$." Going through the proof, it seems that the two $\lambda$ can be arbitrary. Please clarify

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-07