Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a new framework for conducting approximate Bayesian inference in the context of function-space optimization. The method described is fairly general and with good theoretical justifications to some of the most important claims here. To achieve their goal, the authors define a new approximate objective function with an alternative divergence to the regular KL  and propose the SPGs as a generalization over previous work. Even though there is some important information missing in the experimentation phase, the performance of the method seems overall much better than previous state-of-the-art models in many contexts. This needs to be addressed since it constitutes an important issue, but the method is promising and interesting. 



main_review: In this paper, the authors propose a new approach to inference in functional space, which has been an important issue of research for the past few years. This is done by introducing an alternative objective function with different properties to the ones of the regular functional-ELBO. The change in the objective function is effectively a newly proposed divergence (grid-functional KL divergence), which could be interpreted as the expected value of the KL divergence between functions sampled from two stochastic processes, evaluated at a random number of measurement locations. The key difference here is that the number of sampling points is a random variable instead of remaining fixed, proposing a grid approximation to the KL divergence. Combining this with the SPG, the authors extend on the formulation by previous work avoiding some of the restrictions of previous methods. 

The results presented in the paper show that the method behaves well and performs better overall than the rest of the methods compared it is being against in the experiments conducted. It represents a fairly general approach (although with certain important similarities to previous literature work) that could help further the comprehension and usage of function space optimization methods.  However, although the results are promising, there are some important experimental concerns that should be addressed in order to have a clearer picture of the capabilities of the proposed framework. 

# Pros

* There is considerable theoretical work being conducted to justify a lot of the properties mentioned of this new approach. Although a bit tough at times, the descriptions in the main text as well as in the appendix help a lot with the comprehension of the proposal.

* The framework constructed is general enough to be used with several methods. This flexibility could be interesting to extend the formulation with the usage of new methods. 

* The performance of the method seems to improve on the previous state-of-the-art, both when compared with some of the best methods in weight and function space optimization. 

* The paper, as well as the appendix material, is well written and easily comprehensible for the most part. Although some sections need a careful read, in most cases it is written in a very clear manner. 

# Cons

* If L^grid >= L^functional, then optimizing it does not mean we are also optimizing L^functional since it makes a weaker lower bound definition (since it is higher, maximizing it would not necessarily imply maximizing L^grid). Although the method seems to perform well, this point needs to be addressed more carefully due to its crucial importance in the development of the paper and the training of the model.  

* The experiments are conducted using the same prior, which is good as a reference. Since this approach also supports other priors, it would be interesting to see how other methods would perform in comparison given their respectively "best" prior (as described in their respective papers). For example, for f-BNN, the usage of a piece-wise implicit prior may not be optimal since that method is not able to train such type of priors, and thus a GP (or a sparse GP) could help improve its performance w.r.t. what is seen in the experiments. 

* Regarding the regression experiments, there is important information not being reported in the main text nor in the appendix. It would be very helpful to have a table with the RMSE (or some other alternative error metric) for completeness, while also having another alternative measure other than the log-likelihood to address the goodness-of-fit of the results (see [1]). 

* Through the paper there are some claims about the scalability of the method. However, the material provided lacks some kind of information on the convergence rate and time of this method in comparison to the other ones. To better support these claims, please provide a convergence study for the methods here. 




# Notes

* Does this formulation generalize over implicit stochastic processes (as described in the paper of Variational Implicit Processes), or is it a more restricted description form for the stochastic processes involved? Specifically, clarify this w.r.t. the description of the functions sampled from both stochastic processes in Proposition 2 and the implications made in the following paragraph (those very same properties could be assigned to implicit processes as well). 

* Also regarding the experiments, although an extensive comparison with other methods is conducted, in the case of weight-space optimization there are some methods that currently perform better than the ones present here, such as the one described in [2]. For a more comprehensive description of the performance, it would be convenient to include these approaches.  

* How flexible could be the resulting predictive distributions that the method is able to produce? Could you try to provide any tests to showcase this? As far as the formulation goes, the predictive distributions could be pretty general, but it would help if there was any experimental confirmation of this fact. 

* Are there any concerns with the problems that may arise due to using a sampling-based approach for the estimation of the KL? For f-BNN it could be seen that they would be somewhat affected by the dimensionality of the data precisely due to this fact. 

* Please provide the information on the computational resources employed to conduct the experiments.  

*Minor details:*

* Although it is comprehensible as is, maybe the notation in the equations of Proposition 2 could be a little more different so that there is a clearer distinction between the functions sampled from $q$ and $p$

* Double ":" in line 85

* The enumeration in the second paragraph of the introduction could appear differently so it helps reading the text, either by enclosing properly the (i) and (ii) or by separating each point with a different item in an enumerated list. 



##  References

[1] Gneiting, T., & Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477), 359-378.

[2] Santana, S. R., & Hernández-Lobato, D. (2020). Adversarial α-divergence minimization for Bayesian approximate inference. Neurocomputing.


 
limitations_and_societal_impact: The societal impact of research such as this one is mostly restricted to the community of approximate inference, especially to those interested in function space optimization. Therefore, I deem there is no potential negative societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper worked on the approximation for the posterior distribution, especially when we perform Bayesian inference in the space of functions. The authors provided a new Kullback Leibler divergence for stochastic processes. The number of the measurement points is a random variable. This new KL divergence is well defined for stochastic processes which are generated from different network architectures. Based on the new KL divergence, the authors developed a scalable variational inference using the VAE to generate approximate posterior stochastic processes.
main_review: Overall, I think the paper could be accepted, but requires a revision for the algorithm part in Section 3.2 to 3.4.

# Pros
-  This paper provides a new KL divergence, which is valid for different network architectures.
-  Numerical experiments, especially for the structured implicit priors show strong evidence that the proposed method is useful numerically.

# Cons
-  The theoretical part of the paper is hard to follow for a non-expert. Especially, in the proof of Proposition 4, I could not follow the discussion of the sequence of VAEs.
-  Algorithmic part is hard to follow at first since each detailed technique is presented one by one.

# Comments and Questions:
- How strong are assumptions in Proposition 2 ? In the proposition, the compactness of the supports of $p(\theta)$ and $q(\Gamma)$  are assumed. I thought these p and q are Gaussian distributions or Inverse Gamma distributions in standard Bayesian neural networks. Then, their supports are not compact, thus it violates the assumption.
- Is there any other example for the distribution of $n$ other than a geometry distribution ? And is there any reason that using a geometry distribution is important numerically ? 
- In numerical experiments, how did authors construct the compact space $\mathcal{T}$ for the input space from the data set ?
- Why should we distill the prior distribution $p(f)$ ? 
- Is it possible to compare the KL divergence defined in (2) and (3) numerically ? 
- In proposition 4, using VAE is justified regarding MMD distance. On the other hand, in variational inference, we are working on KL divergence, not MMD. Is it possible to justify using VAE based on KL divergence under some mild conditions?
- I think experiments for the structured implicit priors in Sec 5.1 strongly support the usefulness of the proposed method. I recommend using the real dataset for this task to support your proposed method. 



limitations_and_societal_impact: Yes, they are shown in the checklist.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose "Functional Variational Inference" (FVI), a novel variational approach to approximate Bayesian inference in function space. The approach is based on the "grid-functional KL divergence", a novel form of KL divergence between stochastic processes. The authors show that this is a proper KL-divergence and that it is well-defined in cases where previous work (fBNN) is not.

Furthermore, the authors propose "Stochastic Process Generators" (SPGs), a novel family of variational distributions which are more flexible in comparison to distributions used in previous work (VIP). Using SPGs, the authors derive an efficient approximation of the ELBO corresponding to the grid-functional KL divergence. The authors demonstrate that this approximation scales better than the one used in fBNN. 

The authors evaluate FVI on a range of experiments (interpolation with implicit priors, regression and classification with BNN priors, Thompson sampling). FVI tends to perform favourably in comparison to fBNN, VIP, and a range of other existing methods.
main_review: Originality: The proposed FVI is a novel variational approach in function space. The authors explain in which respect their work extends and improves upon prior approaches (fBNN and VIP). As described below (Sec. "Clarity"), the paper could be strengthened by providing a bit more intuition on how FVI remedies the weaknesses of prior work.

Quality: The proposed method appears to be formulated in a technical sound way (I did not check all of the proofs). The authors establish the well-definedness of FVI and provide both theoretical and empirical evidence that their method improves upon prior work. 

Nevertheless, there are some weaknesses, on which I ask the authors to elaborate:

- The authors use the Product of Experts (PoE) architecture for the encoder used to distill the prior into an SPG. While I think this is a valid approach, I do not understand the author's comments on the "Deep Set"-like approach used for the Neural Process (NP) family. Why should this method be inferior to the PoE approach, i.p., why should it be more "difficult in high-dimensional spaces" (l. 219). Also, what do the authors mean with "need to model all possible conditionals" (l. 222). Did the authors try the "Deep Set"-like approach for their method?
- The results for fBNN in Fig. 2 seem to be way worse than those shown by the original authors on a similar experiment, cf. Sun et al., "Functional Variational Bayesian Neural Networks" Fig. 3. Could the authors elaborate on this discrepancy? What do the authors mean with "for fairness" in l. 247?
- Why do the authors consider an additive white noise process $\nu$ in their definition of the SPG? Isn't noise already considered through the likelihood $p_\pi$?

Clarity: Overall, the writing style and paper organisation is good. Nevertheless, I think some points could be improved. As FVI is closely related to fBNN and VIP, I would like to see a bit more background on those methods and how FVI remedies their shortcomings. In particular, it would help the reader if the authors could provide some intuition for the following (I would suggest moving some of the more technical details to the appendix):

- What is the intuition behind using Eq. (3) instead of Eq. (2)? Why is it crucial to have a random number $n$ of measurement points in Eq. (3)? What is the influence of the sampling distribution $c$ in Eq. (3) and of the particular choice Eq. (6)? I am aware that the authors provide formal arguments to support these choices, but I think that their presentation lacks intuitive motivation for them.
- Why does it make sense to freeze the basis functions and decoder parameters after fitting the prior using an SPG? Is this merely a choice required to make the KL divergence between SPGs tractable? How does this choice relate to Proposition 4, i.e., how does this "parameter freezing" affect SPG's expressiveness?

Minor concerns/typos (do not influence my score):

- l. 26: missing space
- l. 28: F-BNN → f-BNN
- l. 56: to generated → to generate
- l. 68: supreme → supremum
- l. 88: exist → exists
- l. 97: $X_\mathcal D$ is undefined
- Eq. (8): the integral is missing a closing $\mathrm d h$. The same goes for all other appearances of this VAE integral.
- l. 126: Then, for $\forall$ → Then, $\forall$
- Text around Eq. (11) states twice that Eq. (11) is called $\tilde p_{SPG}$.
- ll. 140-142: $f_m^X$ is explained twice
- l. 157: extra comma
- Eq. (19): The likelihood misses a subscript $\pi$

Significance: The authors propose an interesting approach to an important problem. The results show improvements compared to previous work. Further, the method seems to be theoretically well-founded. I therefore think the proposed FVI can be a valuable contribution. However, in its current form, I judge the paper to be borderline because of the concerns I raised above. If the authors could elaborate and improve upon those points, I am willing to raise my score.

------------------------------------------------------------------------------
Update after reading the author's response:

I thank the authors for their response. I raise my score because the authors addressed my concerns and will clarify some of the points I raised in the updated manuscript.
limitations_and_societal_impact: No potential negative societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper presents an approach to probabilistic regression. The posterior distribution is estimated using a variational approach. A variational class of stochastic processes is presented, which is parameterized by neural networks - amongst other things by a product of experts variational autoencoder. For optimizing the variational distribution a variational bound and its sample approximation is derived. The proposed approach is evaluated on regression, classification and reinforcement learning tasks.
main_review: Overall, I enjoyed reading the paper and the results seem to be very competitive and impressive.  

Originality: The approach seems like a new approach, which combines and extends ideas from previous work, such as [44], [27]  and [50]. The techniques are as far as I could tell well known. The paper differentiates well from the related work, which is sufficiently discussed. 

Quality: As far as I could tell the submission is technically sound. The resulting algorithm is extensively evaluated empirically and some theoretical inside are given. 

Clarity: The paper is nicely written and organized. I think that it adequately informs the reader. 

Significance: The empirical results seem pretty impressive and could be of much use to the machine learning community. 


One thing I missed was:
- An experiment where the true posterior is a Gaussian process and the proposed approach is evaluated against it. In such an example the ground truth is not only given by samples from the posterior as in Sec. 5.1, but the whole posterior distribution can be evaluated in closed form. As such, one could evaluate how accurate the variational approach is.

Things I did not understand:
- Why is in eq. (12) the prior only fitted using regressors $\mathbf X_O \subseteq \mathbf X_{\mathcal{D}}$ from the data and not some arbitrary regressors $\mathbf X_O \subseteq \mathcal T^{\mathbb Z_+}$?
- Why is FBNN performing so poorly in Figure 2? The example seems extremely simple and Figure 3 in [44] suggest different results. If it is just because of the training time needed, I do not that such a comparison is fair. 

Remarks:

-Reference [44] is missing the venue.
limitations_and_societal_impact: N/A
review_title: null

Meta-review: This paper proposes a method for approximate Bayesian inference in function space using a novel form of the KL divergence between stochastic processes, the grid-functional KL divergence, where the number of measurement points is a random variable. This divergence has better properties than those used in previous work. All the reviewers recommend acceptance and I agree. 
Decision: Accept (Poster)