Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Optimization of an unknown function under unknown constraints is a problem which has been often addressed based on Gaussian process surrogate models in the past. Due to the unknown constraints, existing approaches typically start with a given set of safe parameters, which is subsequently expanded. In order to ensure the safety of the approach, this expansion must not lead to constraint violations. State-of-the-art methods determine evaluation points for the expansion of the safe set by finding the most uncertain point in discretized sets. However, this leads to additional hyperparameters and tends to scale poorly to high dimensional input domains. This paper aims to avoid the issues resulting from discretization by employing the information gain of the binary random variable describing the constraint violation at a given input point. Since this information gain cannot be computed exactly in practice, an efficient approximation is proposed. It is shown that determining evaluation points using the proposed criterion is guaranteed to achieve any upper bounds on the posterior GP variance in a finite number of evaluation points if the safe set is not expanded anymore. The practical strengths of the method are demonstrated in simulations with GP sample functions and control examples. In particular in high dimensional problems, a significantly better exploration of the true safe region can be observed.
strengths_and_weaknesses: The paper is on an interesting and relevant topic. The proposed ideas are novel to the best of my knowledge and nicely relate to entropy search and similar approaches in unconstrained Bayesian optimization. The paper combines theoretical analysis and demonstration of practical advantages well. In general, the paper is well-structured and the presentation of the method is excellent in my opinion. Merely in the supplemental material I have noticed that everything is proven for the approximation proposed by the authors, while it is stated that the analysis is performed for the mutual information itself. It took me some time to realize this discrepancy, such that I would recommend to clarify this. The derivation of the theoretical results looks quite straightforward, but the result is certainly interesting and fits well into the overall paper. However, it does not become clear if the scenario analyzed in Theorem 1 does actually ever occur. The reason for this is that it requires that the safe set needs to stop growing at some point in time, which could potentially never happen if the increase of the size of the safe set becomes smaller and smaller every iteration, but never becomes zero. This case is also not discussed in the paper, such that the significance of the theoretical results strongly diminishes in my opinion. I did not check the theoretical results in detail, but they seem to be correct to the best of my knowledge. The simulations are properly executed with sufficiently many random seeds including functions satisfying the assumptions of the approach and functions for which this is not clear. The effect of the dimensionality on the performance of the different approaches is illustrated nicely. I merely have a small technical comment regarding the statement in section 6 that the ISE approach is evaluated on samples from a GP because this would allow to test ISE under the assumptions of the theory. While this is true for the exploration, the usage of GP sample function poses the theoretical challenge that samples from a GP are well-known to almost surely have an unbounded RKHS norm. Therefore, the definition of the safe set based on (Chowdhary and Gopalan, 2017) does not work anymore because it requires a bound for the RKHS norm.


questions: Apart from my comments mentioned above, I only have minor suggestions/questions:
- Highlighting ISE using a bold line in Figs. 3 and 6 might be a good idea
- How is beta chosen in the simulations?
- Line 47: in $\rightarrow$ on
- Lines 231-232: exploration exploration
- Lines 262-263: the true safe for this problem
- The y axis label of Fig. 6 seems wrong. I think it should be the found optimum.
- Line 280: I think it should be referred to Fig. 5b instead of 6b here.

limitations: I think the limitations of the method and potential societal impact are generally well discussed. While the choice of the kernel function is highlighted as a potential challenge, it is not sufficiently discussed that the RKHS bound also needs to be known for the theory to hold. This bound is generally difficult to obtain in practice, which crucially limits practical safety guarantees. In my opinion, this should be discussed in the limitations section.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary:  The paper focuses on an important stage of GP safe exploration algorithms: safely expanding the size of the set of parameters believed to be safe, in a data-efficient manner.
To do this, the authors propose an information-theoretic approach inspired by other non-safety-constrained Bayesian Optimization (BO) literature.
Compared to standard SafeOpt (Sui et al. "Safe exploration for optimization with Gaussian processes." PMLR15), the method is shown to be more efficient at expanding the safe set.

strengths_and_weaknesses: Strengths:

The originality and contribution the paper makes is good:

* It adapts mutual-information-based methods to the safe set expansion setting, which no other work (that I was able to find) does. Usually information theoretic measures for BO simplify to just sampling at the parameters with maximum variance (Schillinger et al., 2017) or highest UCB, but in this case it is more interesting and the authors are able to come up with a function to optimise to find the maximum information gain about expanding the safe set.

* Their acquisition function simplification to make optimization tractable seems justified and well explained in the appendix.
This is not the only work to avoid the inclusion of a Lipschitz constant hyperparameter (Berkenkamp et al. "Safe controller optimization for quadrotors with Gaussian processes.", ICRA16) (they use hypothesised GP observations for safe set expansion) or to avoid SafeOpt-style discretisation (Schillinger et al., 2017) (this is another method that works in continuous parameter space). However, these two problems are solved here in an elegant way.
In particular, it avoids the basic heuristic/arbitrariness of the expander state definition in SafeOpt.

* I believe the authors have covered all the relevant related work – although not sure the paper experimentally compares to enough related work (see below).

* I believe the clarity of the paper is generally good until the experiments section.


Weaknesses:

* The acquisition function simplification should be more explained in the main body of the paper. Furthermore, it would be good to have further discussion on the approximation on the rest of the paper, e.g. Theorem 1.

* The evaluation with GP samples, shown in Figure 3, is confusing. The only alternative method comparison is standard StageOpt with differing Lipschitz (L) parameter values. The justification for the choice of plotted L-values isn’t clear, and furthermore there is a clear trend of decreasing L leading to better performance. From this graph, it would suggest that reducing L further <1 might result in it outperforming ISE.
For those with background knowledge, it is clear why it’s not possible to keep reducing L. Eventually, reducing L enough will result in StageOpt classifying unsafe states as safe and then failing the safety specification. However this section contains no discussion of whether/how any algorithms fail the safety specification, and only shows the first few % of safe set exploration. Evaluation should include plots on how many time s the safety spec is broken, as that is a crucial part of the problem

* I think evaluation might have benefited comparing to SafeOpt variants that do not require the Lipschitz constant, such as (Berkenkamp et al. "Safe controller optimization for quadrotors with Gaussian processes.", ICRA16). That would better differentiate which performance improvements are from more intelligent expansion behaviour, vs tuning of the Lipschitz constant.
Figure 6 does not have enough baseline comparison. Only one StageOpt plot is shown in 6 (a) and it is not clear which Lipschitz value was used for this. There is no StageOpt plot in 6 (b) and it is unclear why. It's also unclear why the plot only shows the first 5% of exploration.

* There should be some experiments on scalability, e.g. it would be nice to see a scalability plot on number of dimensions.



Other  points:

The last paragraph of related work should also mention Turchetta, Matteo, Felix Berkenkamp, and Andreas Krause. "Safe exploration in finite markov decision processes with gaussian processes." Advances in Neural Information Processing Systems 29 (2016).

“This process is made more efficient by SafeOpt, which restricts exploration to parameters that are close to the boundary of the current set of safe parameters (Sui et al., 2015)”: I think it is more accurate to say that StageOpt does this restriction (in its expansion phase). SafeOpt can also choose to sample potential maximisers in M_t, not just from the expander set G_t.

Clarity-wise, I don’t fully understand how Theorem 1 justifies the sentence “in practice it will first focus on reducing the uncertainty in areas of the safe set that are most informative about parameters whose classification is still uncertain (e.g. areas close to the boundary of the safe set), and only eventually turns to learning about the inside of the safe set”. If it doesn’t justify that sentence, the theorem doesn’t seem to actually prove anything about safe set expansion.


questions: why didn't the experimental section analyse the safety spec? 

In figure 2 (c), the left-most green evaluation cross is clearly under the orange safety bound line, which suggests that the agent has broken the safety specification in this run. Is this a formatting error or representative of an actual failed run?

Please clarify the point about Theorem 1 above.
limitations: Limitations on scalability are mentioned in text but not analysed empirically. Negative societal impacts are addressed with a boilerplate sentence in he end of the paper, bit unsatisfactory; probably better just not saying anything.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper provides a newer approach to explore and uncover the safe set of parameters. The exploration is principled and is guaranteed to be safe with high probability. The developments in the theoretical section provide closed-form approximations for the information-gain relevant quantities. The experiments show that their approach can provide the maximally safe set faster than existing methods. The novelty with respect to prior work is that the information gain criteria is used for bayesian optimization with an emphasis on safety. As an illustration of the algorithm, the GP prior is known beforehand with at least one point in the safe set. The next point is chosen to be in the region that is known to be safe such that decreasing the variance around this region will provide maximal information about the safety of any point beyond the safe set. Rather than the Lipschitz constant, the use of a smoothness prior does seem interesting and novel.
strengths_and_weaknesses: I believe this is overall a good paper as it is a principled approach to Bayesian optimization and safety. It is improving the Lipschitz constant based methods for sampling-based safety verification.






questions: From a theory perspective, [A] proves a finite-time regret bound while this paper only proves an asymptotic guarantee. The first type of bounds are stronger and provide more information about the rate of convergence. In practice, the method actually has an advantage but this is lost in the theory. What is proven here is that once we have found the maximal safe set, the posterior variances of the points inside the safe set will eventually become small. The exploration for uncovering the safe set is optimal at least assuming the prior is correct. Is it relevant to prove the number of samples it would take to uncover the safe set or let us say the regret wrt an algorithm that knows the optimal prior?

The comparison to previous related work is at a very high-level. It would help to provide a deeper theoretical comparison to the 2-3 most related papers. For example, can you place the theorem being proven in the context of the literature in this topic?

[A] Max-value Entropy Search for Efficient Bayesian Optimization - Wang and Jegelka
limitations: There is a great discussion on limitations of the work.
review_title: null

Meta-review: metareview: I went through the reviews and authors' responses. The reviewers are experienced in this field and provided good quality reviews. Scores are the same: weak accept. I also went through the paper and considered it reached the bar of NeurIPS.
confidence: Certain
Decision: Accept