Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work extends the results of Polson and Rocková [2018] on the minimax contraction rate of Bayesian ReLU network from the Hölder space to the more general Besov space. Such result brings more freedom to the underlying functional form behind the data with similar minimax contraction rate guarantee. Beside the spike-and-slab prior, the result with shrinkage prior is also given but with more computational efficiency. 
strengths_and_weaknesses: The strong point of this work is the ability to extend the functional form modelled by Bayesian ReLU network and its posterior contraction rate guarantee. The motivation is clear, and the idea is well presented.

However, as the authors said, this is an extension of existing works Polson and Rocková [2018] with similar backgrounds, settings, and results, and the derivative looks a little straightforward, so the innovation is limited. The authors did not highlight the challenges or difficulties in how to extend the results from the Hölder space to the more general Besov space. 

Another main weakness is the evaluation. Only one simple example is given to show the ability of designed prior for BNN regression. It would be better to compare with other options, like iid Gaussians, spike-and-slab, and the Gaussians priors with learnable hyperparameters. In practice, we normally learn the parameters of priors from the data as well rather than just fixing them while we using BNN for data modelling. Is there any effect from such learnable priors? 

The underlying functions are usually assumed to be continues when using BNN for real-data modelling. Since the advantage of Besov space is its ability to include non-differentiable functions, can you find one or some real data that needs the functions from Besov space rather than Hölder space or even L^p space? 

Some minor problems include: the symbols should be explained, like “A” in (9), any conflict between “L_1” in Line 108 and “L_n” in (10), Line 158, “a expectation” and Line 185, “acontinuous”

questions: The underlying functions are usually assumed to be continues when using BNN for real-data modelling. Since the advantage of Besov space is its ability to include non-differentiable functions, can you find one or some real data that needs the functions from Besov space rather than Hölder space or even L^p space? 

Can you highlight the challenges on the proposed extensions? 
limitations: N/A
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper present a proof of Bayesian posterior contraction rate in Besov spaces for Bayesian neural networks priors for the regression problem on $[0,1]^d$. More specifically, two priors are presented : spike and slab and shrinkage (the latter being easier to implement). Under the frequentist assumption that the true regression function $f_0$ belongs in a Besov space $B_{p,q }^s$, it is shown that the two priors achieve minimax rates of convergence (up to log terms) with a (non adaptive) choice of prior parameters (which includes a choice of depth, width & sparsity of the network). Moreover, it is shown that an adaptive rate of convergence over all smoothness index $s>0$ is possible for a spike and slab prior with an additional prior on the architecture. This work extend previous results for Bayesian posterior contraction in Hölder spaces and frequentist minimax theory in Besov for neural networks.
strengths_and_weaknesses: This paper is a strong theoretical results closing a gap for posterior contraction of BNN in Besov spaces. Concerning the priors, it presents a new (computationally more efficient) shrinkage prior that is shown to lead to good posterior contraction rates, which is (to the best of my knownledge) a new result even for BNN in Hölder spaces.
Moreover, I think that paper is overall wery clear and well written. However, 
questions: I have two questions :

- in appendix C1, proof of lemma 3 (35) : can you quickly justify how you lower bound the norm ?

- in the proof of theorem 1 (and subsequently in the paper for the other analog proofs), it is not clear how the prior on the connectivity pattern (ie on $gamma$) appears in the proof of the prior thickness result (43)-(44)-(45), should there be a $\binom{T_n}{S_n}$ factor somewhere or am I making a mistake ?
limitations: From my point of view the main limitation of the paper is that adaptive result is only proven in the case of the spika and slab prior, and that the numerical expriments are conducted on toy examples with known smoothness levels. It could be interesting to, at least, discuss why you are only presenting an adaptive result for the first prior.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes the theoretical results for Bayesian neural networks in the Besov space. It extends the results from the previous work in Holder space into the Besov space. It shows that Bayesian neural networks with the spike and slab priors and further relaxes it to be just shrinkage priors converge to the true regression function in the Besov space. 
strengths_and_weaknesses: **Update after rebuttal:** I would like to thank the reviewers for their incredible job during the rebuttal and by improving the submission very much. I believe that the clarity of the paper has drastically improved and therefore I am raising my score. Even though after the correction of the theory the numerical example does not satisfy the theory now I believe the submission provides an interesting step towards better theoretical guarantees of BNNs.

=================================

**Originality:** 
*Strengths:* The main results appear to be novel and an interesting extension of the previous results in Holder space into the Besov space. 

**Quality:** 
*Strengths:* I am sorry but I have to admit that I cannot assess the actual theoretical contribution of the paper as it is beyond my comfort level zone, but also because in contrast to some people who prefer formulae over word description I would prefer the latter to better understand the formulae. Please see comments on this below in Clarity section.

*Weaknesses:* Though I appreciate that the main contribution of the paper is theoretical, the numerical example provided is not too satisfying. 

* The method is only used on 2 synthetic examples with analytical functions
* The main theoretical results are about the posterior, but the numerical experiment uses the Bayes by Backdrop for inference, which is implementation of the variational inference, i.e. approximation of the true posterior. Wouldn’t MCMC methods with true posterior be more appropriate for the purpose of the numerical example? 
* There are several claims that the paper proposes a “practical” model, however, there is no empirical evidence on that. Some discussion on actual computational time used and how feasible the inference with the considered priors would be appreciated


**Clarity:** 
*Strengths:* The text where available is mostly well written and easy to follow. The paper tries to tell the full story with a proper introduction rather than jumping straight into it as if a reader has just finished reading the previous work and doesn’t require any introduction - the approach used in many recent papers. 

*Weaknesses:* There is not so much text in the paper but a lot of formulae. I appreciate there are people who prefer formulae over words, but there are others who much better read words over formulae. Word description of formulae would be much appreciated. Also, the notation is not very careful which does not help further with readability of this lot of maths. 
If the space is an issue, some general paragraphs from the introduction can easily be reduced or removed to save some space. Figures 2 and 3 can also lose the histogram part to save some space. (The histogram part can go into the appendix also)

**Significance:** 
*Strengths:* The paper addresses a very important problem of finding theoretical guarantees for deep learning, Bayesian deep learning in particular in this case. It does seem to make a step further in extending those guarantees (by providing them in a bigger Besov space). 

**Summary:** Readability of the paper, namely the lack of words over maths, is the main issue with this submission in my opinion. So much so that it is difficult to assess the quality of the paper. It seems the paper has significant and original results but it should also be easier to understand these results while reading the paper. Moreover, the numerical example can be largely improved. 

Specific comments/suggestions:
1. Line 10. “In other words” - a bit inappropriate use here as it hasn’t been said before anything about practicality, only about asymptotic properties
2. “Posterior consistency” is never properly defined in the paper
3. Line 96, “u \in {0, 1, …, d}”?
4. D^u is used for u-th derivative, and then D_n is used for a dataset. It is a bit confusing though technically it is not overlapping notation. It is probably better to use another letter for a dataset
5. Equation between lines 120 and 121. Both D^u and d^u is used. What is the difference?
6. Eq. (7) - \eta is not defined
7. Line 152, \delta is already used for the Dirac function
8. Theorem 1, M_n is not defined, how does it depends on n?
9. Line 204, N is already used, probably it is better to used \mathcal{N} for the normal distribution 
10. Figures 2-3. Are dots training data points? Left column plots can used “x” and “y” labels for axis, also subplots without labels look too strange for me. It seems they may have “(a)”, “(b)”, … labels
11. Line 218. “shrinkage prior” - “spike-and-slab prior” was meant?
12. Lines 229-230, “Experiments … will also be of great help…” - not clear, is it suggested as future work? It would indeed be nice to have some comparison in the current submission.


Minor:
1. Line 21, “Before observing the data, A” - lower case for “A”
2. Line 36, “are” -> “were”
3. Line 69, “than THE spike-and-slab prior”
4. Line 82. “GP” - acronym is not introduced
5. Line 86, “to solve THE optimization problem”
6. Line 88, “with THE (nearly) optimal convergence rate”
7. Line 148, “Let” - lower case for “L”
8. Line 211, “were” -> “are”
questions: My main concern with the paper is its clarity and I don’t have questions on that. 

The other smaller concern is regarding the numerical example, so maybe the authors can answer questions about it, however, as this is not the major concern, the authors should be free to give priority to questions to other reviewers in their rebuttal if they would help to resolve some major concerns from the other reviewers.

1. Wouldn’t MCMC methods with true posterior be more appropriate for the purpose of the numerical example? 
2. What can the authors say about the claims about practicality of the method?

limitations: There is no discussion on this, but since the work is theoretical it is not the major issue
review_title: null

Meta-review: metareview: This work conducts novelty study and extends the results on asymptotic convergence of Bayesian ReLU networks from the Hölder space to the more general Besov space. The reviewers consider it "a strong theoretical results closing a gap for posterior contraction of BNN in Besov spaces". 

The authors' feedback addressed a few concerns in the initial reviews including the lack of clarity, the question on the technical challenges to extend to a Besov space, and an error in the proof.

During the author-reviewer discussion period, the authors also corrected a constant which determines the complexity of the neural network model. As a result, the condition of the theory became "harsh" (see authors' response to reviewer x58D), and the numerical results did not satisify the theory's requirement any more. The authors provided an updated version of the paper to include the change and moved the experiments to appendix. Nonetheless, the reviewers did not think that change decreased its theoretical value and still considered it above the threshold for acceptance due to its novelty.

The remaining concerns from the reviewer is lack of evaluations for the non-smoothness assumption and its usage in some real applications, also for the possible difference between a purposely designed fixed prior with a learnable prior.
confidence: Less certain
Decision: Accept