Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method to perform approximate Bayesian inference in function spaces with an emphasis on applications to Bayesian neural networks and gradient boosting. The proposed approach extends Stein variational gradient descent (Liu & Wang, 2016) to function spaces by applying it to minimise the KL divergence with respect to a posterior stochastic process. The approach is made tractable by approximations propagating the functional gradients to the parameter space of the models being learnt. Experiments learning BNNs and gradient boosting models on benchmark datasets are presented showing performance improvements with respect to baselines.
strength_and_weaknesses: ### Strengths
* Compared to previous work (Sun et al. 2019; Wang et al., 2019), the proposed approach more naturally translates SVGD to function spaces by starting its derivation from functional gradients.
* Theoretical results are provided on the validity of the functional gradient estimators.
* Experimental results show performance improvements against previous function-space variational approach FVBNN.

### Weaknesses
* Related work on SVGD could mention some recent work on other forms of SVGD, such as second-order methods, matrix-valued kernels, and its convergence analysis.
* Related work on BNNs only mentions variational inference approaches, while Markov chain Monte Carlo methods, such as stochastic gradient Hamiltonian Monte Carlo, have also shown some success on inference for BNNs, though usually in problems of low dimensionality.
* Background on gradient boosting could be expanded.
* What is most concerning to me is that practical performance gains seem marginal when considering the results in Table 1.They are mostly within the +/- 1 std. deviation margin.
* There is no discussion on the results in Table 1. For example, why did the BNN (using Bayes by backprop) baseline perform significantly better than the functional-gradient methods in a few of the benchmarks? And importantly, what would be a few reasons for the performance of the proposed SFVNN and the FVBNN baseline to be so similar in most cases?
* In the appendix, there were no baselines for the comparisons with SFVGB.

clarity,_quality,_novelty_and_reproducibility: The paper is mostly well written, despite a few typos. The contribution seems novel. There's no mention of whether code would be released for reproducibility purposes.

Minor clarity issues:
* $\mathcal{I}_b$ could have an explicit definition.
* The integral in the denominator in Eq. 7 should be over $\boldsymbol{\theta}$, not $\mathbf{x}y$.
* A few parts of the first paragraph in Sec. 2.1 are missing the space between the period and the beginning of the next sentence.
summary_of_the_review: The paper's contributions seem to be mostly on the formulation of a new method with theoretical support, but with only marginal practical performance gains in experimental evaluations, which cast doubts regarding its potential impact.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work studies function-space BNN inference, and formulated function-space variants of the Stein variational gradient descent algorithm.  The algorithm is evaluated on synthetic and tabular regression data.
strength_and_weaknesses: ### Strengths

This work attempts to provide a rigorous formulation of function-space, SVGD-like algorithms.  This is a sensible problem, as similar particle-based VI approaches have shown promises in the past, including for BNN inference where their function-space analogues have been studied in less rigorous ways.  A rigorous formulation would allow the principled use of such algorithms, and may also identify suboptimal constructions in past algorithms.

### Weaknesses

My main concerns are the following:

1. The current manuscript did not discuss previous work adequately.  It is not obvious from a first reading what efforts have been made to construct function-space SVGD algorithms: this work only cited Wang et al (2019), and neglected the works of D'Angelo et al (2021a; 2021b) which studied similar issues.  Moreover, there is no comparison between the update rules derived in this work and those in past works; and there is no empirical comparison either.  This leaves it very unclear how new derivations in this work have practical implications.

2. The presentation of the technical contents is quite confusing, which makes it difficult to understand or verify the claims.  Gradients were presented without specifying the inner product structure they are defined with.  The same gradient notation can be used to refer to both L2 and RKHS gradients (e.g., Eq. 3 and the discussion below).  Typos such as in Eq. 8 further complicated reading (Eq. 8 refers to the $\mathcal{H}$-Wasserstein gradient in a space of distribution over parameters, not "functions" which do not have to exist there).  It takes a lot of guesswork -- and familiarity with the background -- to understand what the authors really meant, and it should not have happened in a work that attempts to present a rigorous formulation for ideas that already exist (in part, and in various forms) in the past.

These two issues make it difficult to evaluate the manuscript in its present form, although I'm willing to go through it again once they are clarified.

3. A less important issue, which should nonetheless be made clear, is that all discussions in this work only applies to the full-batch training setting, where the sampled "measurement points" (Sun et al, 2019) always include the entire training set.  This is an inherent limitation (Burt et al, 2020) of the framework of Sun et al (2019), and the readers should be made aware about it.

### References

* Wang, Z., Ren, T., Zhu, J., & Zhang, B. (2018). Function Space Particle Optimization for Bayesian Neural Networks. In International Conference on Learning Representations.
* D'Angelo, Francesco, and Vincent Fortuin. "Repulsive deep ensembles are bayesian." Advances in Neural Information Processing Systems 34 (2021): 3451-3465.
* D'Angelo, Francesco, Vincent Fortuin, and Florian Wenzel. "On stein variational neural network ensembles." arXiv preprint arXiv:2106.10760 (2021).
* Burt, D. R., Ober, S. W., Garriga-Alonso, A., & van der Wilk, M. (2020). Understanding variational inference in function-space. arXiv preprint arXiv:2011.09421.
clarity,_quality,_novelty_and_reproducibility: See above for comments on clarity and novelty.
summary_of_the_review: While this work tackles an important problem and appears to have interesting contributions, the clarity issues make it difficult to judge its merits.

(My ratings below reflect the uncertainty due to the clarity issues, and I'm willing to update them after clarifications.)
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to perform Stein Variational Gradient Descent directly in the function space of neural networks or gradient-boosted models, thus offering two new methods. It derives theoretically how these approaches can be approximated in practice and shows empirically how they perform on simple benchmark regression tasks.
strength_and_weaknesses: Strengths:
- The problem of function-space inference is well-motivated.
- The paper is clearly written.
- The theoretical derivations seem to be correct.

Weaknesses:
- The related work is incompletely acknowledged.
- The experiments seem rather inconclusive.

Comments:
- The proposed SFVNN seems (at least superficially) very similar to [1] and to some lesser degree to [2]. Other functional BNN approaches that seem worth mentioning are [3] and [4].
- The paper relies a lot on the KL divergence estimator from Sun et al. However, [5] has shown that the KL divergence is generally infinite between many different function-space distributions and that the estimator is thus ill-defined. Could the authors comment on that?
- In the presented regression experiments, the proposed method only outperforms the baselines on a few datasets. Could the authors comment on why it doesn't work well on the others?
- Could the authors comment on the runtime of the proposed method? I would naively think that it would probably be slower than the baselines.
- Since NNs are not necessarily much better on these regression tasks than, e.g., GPs, I think an image classification task would be a more suited experiment to motivate the method, similar to e.g. [1,2].
- Since [2] have shown that their method works better than the functional SVGD [1], it seems like that might also outperform the proposed method and should be a baseline in the experiment.
- The notation is sometimes hard to parse (up to 4 stacked subscripts), so maybe that could be made lighter.

[1] https://arxiv.org/abs/2106.10760

[2] https://arxiv.org/abs/2106.11642

[3] https://arxiv.org/abs/2008.08400

[4] https://hudsonchen.github.io/papers/Tractable_Function_Space_Variational_Inference_in_Bayesian_Neural_Networks.pdf

[5] https://arxiv.org/abs/2011.09421
clarity,_quality,_novelty_and_reproducibility: The description of the method is clear and it is of high theoretical quality. However, the empirical quality seems limited and the novelty is questionable, especially since many closely related works are not discussed. The experiments seem reproducible.
summary_of_the_review: Overall, the method seems theoretically well-motivated, but the weak experiments and insufficient discussion of related work currently hinder me from recommending acceptance. If the authors could provide stronger experiments with relevant baselines on realistic data, I would be willing to change my assessment.

UPDATE: I have increased my score thanks to the changes made during the rebuttal.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors study general variational algorithms, which generalize Stein variational gradient descent (SVGD) in functional space. They generalize the functional space in minimizing the KL divergence step. They apply the algorithms in Bayesian neural networks and ensemble gradient boosting. The numerical experiments demonstrate the efficiency of their algorithms.  
strength_and_weaknesses: Weakness: Please do not use blue and red colors on the paper. 
clarity,_quality,_novelty_and_reproducibility: Quality: The paper presents an approach to generalize the Stein variational derivative. However, I did not see any motivations or analytical examples for the proposed methods. What is the quality of this method, at least in some simple examples? 
summary_of_the_review: This paper provides a method to approximate the Stein variational gradient in general settings. However, there is no analytical or motivation examples to support the proposed method.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper has received mixed reviews, with all 4 reviewers proposing a borderline decision (5-5-6-5). All reviewers agree that the paper's contribution for approximate Bayesian inference in function space is valuable. However, the reviewers have raised several concerns, including the clarity of the presentation, the absence of relevant recent work in the literature review, and the inconclusive results against Sun et al. To the best of my knowledge, the authors have swiftly addressed all these concerns during the discussion. For this reason, I believe the paper is now ready for publication.
justification_for_why_not_higher_score: This paper introduces a novel gradient descent algorithm in function space. The method is well-motivated and discussed. The experiments sufficiently support the claims of the authors. However, the research is also incremental, so I do not recommend a higher score. 
justification_for_why_not_lower_score: All initial concerns raised by the reviewers were addressed during the discussion.
Decision: Accept: poster