Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper considers improving uncertainty estimations for meta-learning. Previous work [1] has done this via using a hyper-prior over the prior parameters for meta-regularization. This hyper-prior, however, took the form of a simple Gaussian distribution over the prior parameters, which imposes smoothness regularization on the prior but may not be ideal for enforcing uncertainty in regions where no meta-training data is available. To remedy this, the authors propose a hyper-prior in the function space in the form of a stochastic process. They then show how the meta-learning algorithm can be defined so that it uses stochastic processes for the prior, hyper-prior and hyper-posterior. The experiments are conducted on a Bayesian optimization setup, where meta-training involves using a sequences of related BO problems corresponding to n target functions and evaluation involves using the meta-learned model to perform BO on a new target function. For example, one such setting is hyper-parameter optimization for machine learning models, where the training task functions correspond to training and testing the machine learning model on different datasets. Across different benchmarks in this setting, they show their proposed algorithm performs better than [1].

[1] Rothfuss et al. PACOH: Bayes-optimal meta-learning with PAC-guarantees. ICML 2021.
main_review: Originality: the paper is proposing a modification to [1] to improve its behavior and performance; however, I believe the proposed change is interesting and does seem to lead to better performance.

Quality: the paper seems technically sound. There was strong justification provided for the proposed method and the experimental results seem to show that it does provide the type of benefit in uncertainty estimation that was discussed in the motivation for the work.

Clarity: the paper is very well-written. Explains the proposed method and its justifications well and the experiments are thoroughly described.

Significance: I think Bayesian optimization is an interesting application of meta-learning and the experiments & benchmarks considered for meta-learning for Bayesian optimization in this work could be interesting to other researchers and help build more momentum in this area.

[1] Rothfuss et al. PACOH: Bayes-optimal meta-learning with PAC-guarantees. ICML 2021.
limitations_and_societal_impact: Not explicitly discussed but should not be different than existing work in this area of meta-learning.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper discusses a meta-learning method with priors defined on function spaces. Building on PACOH, a meta-learning framework with PAC-Bayesian bounds, the paper proposes F-PACOH where the hyper-priors and priors defining meta-learning models in function spaces. In original PACOH, the hyper-priors are set to be parametric distributions (usually Gaussians), and the authors argue that such parametric priors cannot effectively capture the functional structure of the tasks that we ultimately want to enforce for meta-learners. On the other hand, functional priors directly regularize functions of interest to be close to prior knowledge, e.g., standard Gaussian processes, so result in more reliable uncertainty estimates. 

The inference is done by optimizing the PAC-Bayesian bound, and the caveat here is the computation of KL-divergence between functional hyper-priors and hyper-posteriors. This can be approximated by the technique introduced in Sun et al., 2021, where a set of measurements points are sampled and the KL-divergence is approximated with the evaluations on those measurement points.

The paper mainly focuses on the application of the proposed method to Bayesian optimization. On several benchmarks, the proposed F-PACOH is demonstrated to quickly meta-learn and improves on similar BO tasks. 
main_review: Overall, the paper is clearly written and easy to follow. The motivation is well set, and the idea to use functional priors is a reasonable solution to the problem. The BO experiments, especially the sequential BO, are interesting. I'm no expert in BO literature, so my assessment may be wrong, but at least for me, these experiments seem novel.

I cannot say the proposed method is novel enough so that the paper should be accepted solely based on its novelty; the paper is definitely well-executed, but the core contribution is rather an incremental combination of well-established frameworks; PACOH and functional variational Bayesian neural nets (FBNNs). Roughly speaking, F-PACOH is an FBNN version of PACOH which uses the same inference techniques as the original FBNNs. The application to meta-learning is new, but it does not incur additional technical difficulty.

Although the framework is generic so virtually one can apply any functional hyper-priors and priors, the paper only presents the case when both of them and priors are set to be Gaussian processes (GPs). GP is easy to deal with due to its closed-form marginals. Have you actually tested with other types of priors or hyper-priors, for instance, implicit processes? In such a case, as noted in the paper, the gradient of KL-divergence should be approximated via SSGE or sliced score matching. Personally, I find the gradient estimates with such techniques unreliable, so wonder if the proposed method works well with them.

The measurement points should be sampled for the KL-divergence approximation. This might be trivial for 1D real line for instance, but what if the target domain is high-dimensional space such as images or sequences? Would the uniform sample method still work well? I guess this would be important because the proposed meta-learning will be more useful for such a high-dimensional setting for which the performance of a vanilla GP based BO is expected to degrade.

In the experiments, a straightforward baseline I can think of is, during meta-training gather all the training task points  $D_{1:T}$,
construct a posterior GP conditioned on $D_{1:T}$, and use it as a hyper-prior for subsequent tasks. I guess this can be a strong baseline despite having to store all the training task data $D_{1:T}$.

It would be good to directly assess the uncertainty estimates for BO tasks as well (with calibration error plots for instance).

How sample efficient the proposed method is? For instance, how are the performances of the algorithms w.r.t. different numbers of training tasks? As far as I know, NP requires quite many tasks to be reliably trained, even for simple GP regression tasks. Judging from the simulation results where F-PACOH is trained with only 10 tasks and still is able to reliably adapt to new tasks, I think F-PACOH to be much more sample-efficient than NP.

limitations_and_societal_impact: The authors discussed both limitation and potential negative societal impact of the proposed method.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work introdcues a new way to regularize priors in meta-learning through the use of stochastic processes as hyper-priors. The authors rely on sampling-based measurements to estiamte the KL divergence between the meta-learned prior and a stochastic process hyper-prior. 

Through experiments on toy datasets, the algorithm, F-PACOH, demonstrates principled uncertainty estimates. It provides higher levels of uncertainty far from training data, which is in contrast to previous methods for imposing meta-learning priors and hyper-priors. Finally, the authors apply their method to a Bayesian Optimization meta-learning problem and demonstrate decreased regret compared to other methods benchmarked. 
main_review: In my opinion, this paper is a strong combination of several existing ideas in meta-learning and stochastic processes. The application of KL divergence sampling-based estimates from Sun et al. in conjunction with the PACOH framework demonstrated principled uncertainty estimates and strong performance on Bayesian optimization tasks. The combination of these ideas yielded a method that has strong empirical justifications and theoretical motivations. 

Throughout, the authors were very clear in their presentation. One thing that may improve clarity is an appendix section (didn't have time to check those, so sorry if this already exists) on stochastic processes. I consulted Sun et al. for a brief review and that was helpful, but making the paper more self contained for someone coming from an adjacent area could be useful. Another point to that effect is that the authors keep referring to "regularization in the function space" and my brain keeps asking "the function space of <what>?" That could just be how one refers to function spaces but a few examples of writing "the function space of the prior" would have defintely helped me get the concept quicker. 

One error (I believe) is on Line 174. The authors missed the "KL" in the expression $$\mathbb{E}_{X_i}KL[q(h^{X_i}||\rho (h^{X_i}))]$$ 

In terms of experiments, I would be interested in more demonstrations that F-PACOH generates principled uncertainty estimates. The 1-D toy datasets did show that property but more empirical validation would be welcome. I'd be particularly interested in uncertainty estimates on out-of-distribution data that has undergone dataset shift (i.e. Ovadia et al.). 

Ultimately, my impression is that this paper represents a solid advance in (hyper-)priors for metalearning and combines several ideas into a useful new method. Given it's strong performance on the BO tasks, I could see other researchers building on these ideas. 

Update after responses: 
I think that a major limitation of the work is its restriction to lower dimensional settings. This will fundamentally limit its applications and is a larger issue than I originally thought. I still think this is a good paper but am slightly lowering my score because of this. 
limitations_and_societal_impact: The authors didn't spend much time on the limitations of their work, beyond addressing the cubic runtime of their methods. Some things that may or may not be limitations that could be addressed:
* Are you always able to get good sampling estimates for the KL-divergence and its gradient?
* Can this method scale in parallel at all? Can it handle large batches of evaluations?

Additionally, societal impacts weren't addressed. Could you talk about how BO can develop novel therapeutics through protein optimization, for instance?
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper observes that meta-learning algorithms typically underestimate epistemic uncertainty because of a failure of regularization to respect the structure of meta-learning.
They propose a meta-regularization step that builds on functional KL-divergence to a GP functional prior.
Their experiments in sequential BO suggest that this functional prior outperforms alternatives.
main_review: Thank you for this paper, which attacks an important problem for meta-learning: getting good uncertainty estimates that incorporate the 'meta-knowledge' as well as the task-knowledge in a principled way.
The general approach you take makes sense and feels like a promising strategy for dealing with this problem.

Nevertheless, I am not currently recommending acceptance.
I felt that the theoretical contribution is comparatively modest, as the paper mostly combines two previously established tools (adding the functional KL on top of a meta-learning formulation).
Given that, the paper needed to make some real progress on empirical demonstrations showing the usefulness of the approach.
Here, I would have preferred to either see good results on higher-dimensional spaces or some discussion/investigation of failures in high-dimensional spaces.
I would change my mind either if you clarified that I am undervaluing your theoretical contribution, or that the experimental support you offer has broader implications than I currently understand.

# Major comments

One major difficulty with Sun et al.'s approach is the estimation of this supremum over measurement sets.
The approach you take, similar to theirs, is to estimate the expectation over measurement sets (rather than the supremum over a sample).
To the best of my understanding, this might improve stability, but doesn't really match the derivations.
And the estimation of this term is quite difficult in more challenging data-sets.
But if I've misunderstood this, please correct me.

More generally, I worry that the approach of uniformly sampling data to fill in the off-distribution prior will only work in low-dimensional datasets.
This is not something that your experiments are in a position to evaluate.
In order to feel confident that this work is of relevance to the broader NeurIPS community, it would be good to examine settings with more than dozen or so dims (or to explicitly state the limitation if it is not possible).
(My current understanding is that the highest-dim experiment is xgboost with 10 dims, is that right?)
Here, it would be very valuable to push the limits of your approach into domains where it might be harder to get a functional approach to work.

## Originality
I have not seen the use of functional hyper-priors on this formulation of meta-learning before.
However if I have a slight reservation, it is that this paper represents a somewhat limited application to Sun et al.'s Functional VI to Rothfuss et al.'s PACOH.
It would be helpful to build further on these ideas, perhaps by analysing the strengths and limitations of the approach in different settings.

## Quality and Clarity
I felt that the writing could have been much more concise and that this would have improved the presentation of ideas.

I am slightly confused by how uncorrelated the ranking of the non-F-PACOH methods in Figure 3 is.
I think this either means that all the other methods are very sensitive to the dataset in their performance or that they might not be fully tuned to the task?

## Significance
This work could be significant in offering new strategies for dealing with uncertainty in meta-learning.
I suspect that the work will have much greater impact if it is able to apply to a broader range of data types.

# Minor remarks
Abstract could probably be cut significantly without losing meaning. E.g., the first sentence is probably not necessary and second could be "Meta-learning can improve accuracy when data are scarce, but existing methods have unreliable uncertainty which is often overconfident."
More generally, it felt like you could cut a lot of text in a way that would clarify the arguments and contributions.

Line 174 you are missing the KL divergence symbol inside your expectation.
limitations_and_societal_impact: I think the paper would have benefited from more discussion of the limitations of the functional KL estimation.
review_title: null

Meta-review: This paper uses PAC-Bayes meta-learning to do meta-learning using the functional KL.  They demonstrate this in a series of experiments where they meta-learn hyperpriors of Gaussian processes for the purpose of doing Bayesian optimization among related tasks.

The reviewers thought the paper was well written, technically strong and interesting.  The review scores were 6, 7, 4, 6.  The main concern here seems to be novelty.  The reviewer arguing for reject expressed that they thought it was a straightforward extension of functional KL and meta-learning ("The combination of functional-KL and meta-learning problem seem fairly marginal.”).  

One major criticism shared by all reviewers seems to be the scale of the experiments, in that they’re limited to low-dimensional problems due to limitations of the function space view.  One reviewer lowered their score from a 7 to a 6 as a result of this during the discussion period. It seems that the authors are aware of this concern and have promised to address this in their discussion.  From the authors: “We agree with the reviewers’ concern about high-dimensional data. For this reason, we have added a discussion of the method’s limitations to lower-dimensional data domains to the updated version of the paper.”

The majority vote among the reviewers was to accept the paper, citing that the paper is well written, technically well-executed and empirically strong (although limited to low-D problems). The application to meta-Bayesian optimization seems well motivated and practically useful.  Thus the recommendation is to accept the paper as a poster.  Hopefully the reviewers feedback can be incorporated into the camera ready version (including discussion of low-D limitations) to make the paper stronger.
Decision: Accept (Poster)