Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents a meta-learning framework to optimize controllers with safety constraints with bayesian optimization. To this end, the authors build on recent work for tuning gaussian process parameters and kernels with meta-learning. The authors test this approach on several bayesian optimization benchmarks and for tuning a controller for a linear robot system. The results demonstrate how optimization with meta-learning priors converges faster than previous paradigms while satisfying safety constraints.
strengths_and_weaknesses: Strengths:

1. The paper offers a strong technical analysis of different meta-learning paradigms for gaussian-processes
2. The paper runs thorough experiments to demonstrate the capabilities of this approach on several benchmarks.

Weaknesses:

1. The limitations section of this approach does not discuss algorithmic complexity or implications when applied to larger optimization problems.
2. Theorems and Lemmas are presented but not proven in the main paper, which forces the reader to look over appendices.
3. Several acronyms are used repeatedly over the paper and not explicitly defined. The presentation could be improved.
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The paper offers an interesting contribution and is written in a fashion that walks the unfamiliar reader through the key concepts that build this algorithm. The results are impressive and seem like a helpful contribution to the field of meta-learning for robotics. I recommend acceptance, but the authors can significantly improve the paper by improving the presentation and clarifying on the limitations of the approach.
issues: Section 1.
- F-PACOH is an acronym that is very important to this paper, yet the authors do not describe its meaning explicitly.
Section 4.2.
- The frontier search is hard to follow over long paragraphs and can, at times, be confusing to the reader. 
- the functions LARGESTMAXMINRECT() and BESTWORSTCASEQUERY() are not defined in the paper, is up to the reader to infer their meaning.
- It would be much more helpful if the authors described the functions LARGESTMAXMINRECT() and BESTWORSTCASEQUERY() under an equation or a separate algorithm environment.
- If Theorem 1 is not proven in the main paper it is very hard to assess its validity or impact. 
Section 6.1.
- Why is regret the best evaluation metric? It would be helpful to comment on this
Section 7.
- What are the limitations of this method when applied to larger problems?
- What kind of BO problems are hard to represent with the F-PACOH modeling?
- What are future lines of work where this algorithm could be impactful?
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: In this paper, the authors present an improved safe Bayesian optimization method by exploiting existing offline data for the meta-learning of priors. By introducing safety-compliant priors via empirical uncertainty metrics and a frontier search algorithm, the proposed framework provides a reliable uncertainty quantification which leads to more trustworthy safety guarantees in contrast to hand-tuned vanilla safe BO. A simulation and a robotic experiment show the benefit of the framework.
strengths_and_weaknesses: Pros:
+ The paper is very well written and clear in all its steps
+ The topic is of timely interest and important for the robotics community
+ The method allows to exploit eventually existing offline data.
+ The experiment/simulation are convincing

Cons:
- The footnote "the monotonicity of the calibration frequency in l is only an empirical heuristic that holds in almost all cases if ν is at least as big as the variance of the targets y in a dataset" seems to be quite important as the monotonicity is key for the success of the optimization. I would like to see some more discussion and evaluations here.

- The proposed framework seems to be computational expensive. Some more details on that would be of interest.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I really enjoyed reading this paper as it is well written and technical sound. The idea is interesting and can be definitely helpful in scenarios where offline-data is existing. However, I'm not sure if such a rich offline-data set is often available such that a major impact in robotics is questionable.
issues: Please address the cons mentioned above.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The authors present a method for meta-learning priors for safe bayesian optimization (BO) from data. The authors contribute a method for finding safety-compliant kernel hyper-parameters by developing an optimization problem which maximizes sharpness while maintaining calibration at all times. By exploiting monotonicity properties of the optimization problem, the authors develop an efficient search algorithm, “Frontier Search” to converge to optimal hyper-parameters. They contribute a safe BO algorithm “SaMBO”, which uses the frontier search in combination with F-PACOH and existing safe BO methods. The authors show that their method requires fewer iterations to converge as compared with data-free methods.
strengths_and_weaknesses: Strengths
- Section 3 problem statement and background is informative and clear. It introduces the background thoroughly and in an accessible way.
- The methods and contributions are well presented.
- The contribution is useful, and as far as I can tell, novel. The proposed method has the potential to significantly reduce computation time for safe BO without significantly complicating implementation. 
- Results (Fig 3, Fig 4) show clear improvement in number of iterations required in BO as compared with baseline, data-free, methods.

Weaknesses
- If I am understanding the examples correctly, you only compare SaMBO-G/S to the data-free baselines. It would be important to compare SaMBO to methods which can use the calibration data which SaMBO uses. 
- The authors make safe BO more efficient by spending computation time on the Frontier Search. It’s not immediately clear that the overall result is more efficient. Nor are there any results which discuss or compare overall efficiency (in terms of e.g. wall-clock time of the entire BO algorithm including the Frontier Search).
- Some assumptions are not entirely clear or well justified. E.g. the authors say that safe BO methods assume that correct kernel hyper-parameters are given (L52). However, their assumption that upper and lower bounds on the correct hyper-parameters (L167) is not well justified.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The challenge of hyper-parameter selection is ubiquitous and the authors present a practical, data-based, method for selecting kernel hyper-parameters. When combined with existing safe BO methods, the authors show their method requires fewer iterations to converge when compared with data-free methods. However, it is not entirely clear that the method has better computation time (i.e. in terms of wall-clock time) since no results for this are presented. I would also like to see a fairer comparison (i.e. to methods which also use data for calibration) such as a hand-tuned baseline or naive approach. However, as far as I am aware, no other method for this kind of safe kernel hyper-parameter selection exists. The methods are convincing and well presented. Therefore, I recommend weak acceptance.

---

**Update**: After reading the replies, the authors have clarified or improved upon all major issues and weaknesses. I have increased my recommendation to strong accept. 
issues: Content issues/questions: 
- Related work: I believe there are a number of related works in the area of PAC-bayes meta-learning regularization which could be added. E.g.: Pentane & Lampert NeurIPS 2015, Amit & Mier ICML 2018, Balaji et al. NeurIPS, 2018, Yin et al. ICLR 2020, Farid & Majumdar NeurIPS 2022. I think meta-regularization would be fair to mention as it relates to both uncertainty estimation and PAC-Bayes meta-learning which are already discussed in the related work.

- L137 What exactly about eq. (5) and (6) are different from [47, 48, 49, 50]? If the authors reference [47-50] purely to refer the reader to information about calibration and sharpness, it would be more clear to say “see e.g. [47, 48, 49, 50]”.

- L167: The assumption that the optimal (and even safe) kernel hyper-parameters have known upper and lower bounds is a key assumption used to prove Lemma 1 and Theorem 1. However, I do not see this assumption as well motivated/argued. Why can we assume that we known a region of safe hyper-parameter? If we choose a very large range to ensure the box includes the optimal solution, is that damaging to the sample efficiency of the Frontier Search?

- Lemma 1, Theorem 2: I would appreciate a proof outline/intuition in the body of the paper. At least add a reference to the proofs in the appendix so the reader can find them more easily.

- Section 6 Experiments: I think it is important to report computation of the proposed method SaMBO in comparison to methods which do not use the frontier search. The authors mention that Equations (6) and (7) are expensive. Since this is additional computation not usually performed in Safe BO methods, it is not clear to me that one is preferable in terms of total processing time. 

- L253: It is interesting that the method works with non-i.i.d. within-task data. Is there a significant improvement in performance if the method is provided with i.i.d data? If there is not a significant difference, this would be a major positive of the approach. 

- L265: Do the authors have any intuition as to why log-space of l and v work better? It would be interesting to see a comparison of log and linear search. 

- Figure 3: Which hyper-parameter selection do the authors use for GoOSE/SafeOpt? Can the hyper-parameters be tuned in any (potentially naive) way by manually looking at the reference data you give to SaMBO-G/S to provide a fairer comparison. 

- L302 “our framework makes safe BO free of hyper-parameters”. I believe $z_1^l, z_1^u, z_2^l, z_2^u$ (L167-L168) should be considered hyper-parameters in addition to the standard deviation of the Gaussian likelihood of the GP (L751). I agree that the proposed method is more practical in general, but it is not correct to say that SaMBO is free of hyper-parameters. 

- L302-L305: I believe the authors could expand on the discussion of the limitations. Can discuss possible time to compute Equations (5) and (6), are there cases where (5) and (6) are intractable? Similarly, are there cases where specifying l and v are straightforward, and therefore the frontier search methods is not necessary? Also, no longer needing to specify l and v moves the problem of hyper-parameter specification for Safe BO to other parameters — are there cases in which these other parameters are just as difficult to specify? 

Editorial issues: 
- Handful of minor typos: L10 and L37: “data-riven” -> “data-driven”, L51: “GoOSE by expanding” -> “GoOSE explores by expanding”, L234 “fist” -> “first 
- I am not certain the authors are using “c.f.” correctly. E.g. in L169 “Since both s(z) and c(z) are monotone we know that z* must lie on or directly above the constraint boundary c(z) = 1 (c.f. Lemma 2).”. I would interpret “(c.f. Lemma 2)” as “compared to Lemma 2”, but it seems that the authors mean “see Lemma 2”. Indeed, Lemma 2 states that “z* lies on or directly above the constraint boundary”. I believe the authors make this mistake on L194 and L762 as well.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents a meta-learning method for model selection in the safe BO setting. The method selects the hyperparameters of the target and constraint GP models such that the BO procedure respects constraints while avoiding too conservative hyperparameters.

This is achieved by the maximizing calibration and sharpness of the uncertainty estimates over a set of datasets collected from related tasks.

The best hyperparameters are employed in a meta-learning procedure (F-PACOH) to learn informative and safe priors for the GP models used in BO with the goal of improving efficiency. This approach mitigates the risk of overconfident priors when data are scarce.
Once the GP priors are meta-learned, they can be employed for efficient Safe BO. 

The approach demonstrates improved query efficiency with respect to baselines and respects constraints. Experiments are carried out in simulation in low-dimensions (2 benchmark functions and a PID tuning problem).

strengths_and_weaknesses: The problem of selecting hyperparameters for safe BO so as to ensure constraint compliance while at the same time avoiding excessively conservative exploration is a highly relevant one, since it can hamper actual applications of Safe BO to parameter tuning for safety-critical and costly-to-operate systems. This paper represents a significant step forward to tackle this problem, providing means to reliably estimate hyperparameters and informative GP priors in the case in which related task data are available.

The paper elegantly and coherently brings together tools and methods from model calibration and meta-learning, presenting them in a coherent way, and proposes a novel frontier search algorithm to select safe hyperparameters for GP priors, leveraging monotonicity of the calibration frequency (heuristically) and sharpness measures.

From the technical point of view, the presented method seems sound. The improvement in terms of query efficiency is shown both experimentally and in Theorem 1. 

The frontier search algorithm assumes monotonicity of the calibration frequency in the kernel length scale, which the authors state is a reasonable empirical heuristic in most cases. Still, it would be informative to elaborate more on possible edge cases and counterexamples.

A more in-depth discussion on the applicability of the method in real-world scenarios would also be beneficial. In particular, is it common to have access to offline data from similar tasks on a real system like a PID-controlled robot? How can we assess if the target task can be considered to belong to the same meta-distribution as the previously-collected data?

Experiments are carried out in low-dimensional parameter spaces, i.e., two 2-D benchmark functions and a 3-D PID gain tuning problem. The latter is not carried out on a physical system, and it is not clear how SaMBO would scale to higher dimensions. Yet, it represents a promising demonstration of the capabilities of SaMBO for many potential robotics and automation applications. The number of repetitions of the experiments appears adequate.

The quality of presentation is high. The paper is well-structured. Concepts are introduced at the right granularity and notation is not too heavy. Figures are informative and successful in clarifying the rationale of the frontier search algorithm and presenting the experimental results.

Few comments on potential improvements:
-	Consider harmonizing the notation in the figures and text, which appear to vary slightly in some cases (e.g., $Q_l$ vs. $Q^l$, define $f^{tar}$, $q^{tar}$)
-	I may be wrong, but, in Algorithm 1, wouldn’t the Prune function need to take as input the intersection between Q and z instead of the union?
-	The definition of average predictive std in Eq. (6) may be worth an additional short description in a way similar to Eq. (5) to better appreciate the conceptual difference

Others:

- L36: on which it falls back to $\rightarrow$  on which it falls back
- L37: data-riven $\rightarrow$ data-driven
- L51: GOOSE [10, 18] by expanding $\rightarrow$ GOOSE [10, 18] does so by expanding
- L60: non-i.i.d $\rightarrow$ non-i.i.d.
- L61: settings, however, $\rightarrow$ settings. However,
- L77: e.g., data of the same robotic platform under different conditions. $\rightarrow$  For example?
- L85: we ask the question how $\rightarrow$ we ask the question of how
- L93 where $\beta$ *is* the… *and* is set such…
- L100: w.r.t. $\rightarrow$  with respect to
- L114: Prior work *proposing* to *meta-learn* GP priors [44, 45, 32], *though*, fail…
- L117, missing a closing parenthesis in the inline expression
- Consider using the definition symbol in Eqq. (2), (3), (4), (5), (6)
- Add full stop after Eq. (4)
- Consider redefining “calib” by “avg-calib” in Eq. (5), since in the tect (L147) it is referred to as average calibration frequency and it is averaged in a similar way as “avg-std”
- L229 calibration- and sharpness-based
- L259 to measure a method’s
- L287 Note that, in case of the regret, the …
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Overall, the paper presents a novel method tackling a well-known relevant problem in Safe BO. To the best of my knowledge, relevant literature is exhaustively covered. Presentation quality is good. The method employs tools from calibration analysis and meta learning in an innovative way, and presents a novel frontier search method to safely select informative priors and improve query efficiency of BO procedures when data from similar tasks is available. 

The efficiency and safety of the method is demonstrated with two different BO algorithms on three low-dimensional tasks in simulation. Although not demonstrated on real hardware, the results seem promising and the method may be applied to a wide range of robot control and automation problems in which gains need be tuned under strict safety constraints. Scaling properties of this approach to higher dimensions remain to be investigated.

In my view, the presented method and results are timely and relevant to the robot learning community.


issues: Consider addressing the weaknesses listed above, i.e., 
- Comment upon scaling to higher-dimensional settings (or even show an experimental stress-test if time permits)
- Comment and provide examples of potential real-world scenarios in which similar tasks may typically be available or could be conveniently acquired, and how to quantify if a task is “similar enough” to another one for it to be employed for meta learning
- Discuss/show counter-examples and edge-cases for the monotonicity of the average calibration w.r.t. the kernel length scale
- Notation consistency between text and Algorithm environments  
- Add short description of avg-std
- Fix the listed typos
review_title: null

Meta-review: metareview: Please check the comments of the reviewers in detail.

### Strengths
- the paper is very well written and clear
- the topic is important for robotics
- promising experimental results (in simulation)

### Weaknesses
- evaluation in 2 2-dimensional functions and 1 simulated robot (3-dimensional problem): most problems in robotics are higher dimensional
- no experiment with the real system (simulation only)
- more discussion is needed about the computational cost of the approach
- no discussion about how much data is needed for the meta-training (e.g., plot the final performance vs the amount of prior data), and how often/why this data would be available

(minor remark: it would be useful to squeeze a picture of the robotic system in the main paper).

### Post-rebuttal review
I would like to thank the authors for their efforts in answering the comments and improving the paper. All reviewers agree that this is a high-quality paper.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Oral)