Title: Transition Constrained Bayesian Optimization via Markov Decision Processes
Venue: NeurIPS 2024
Accepted: yes
Keywords: Bayesian Optimization

Transition Constrained

Markov Decision Process

Linear Bandits

Convex Reinforcement Learning
Abstract: Bayesian optimization is a methodology to optimize black-box functions. Traditionally, it focuses on the setting where you can arbitrarily query the search space. However, many real-life problems do not offer this flexibility; in particular, the search space of the next query may depend on previous ones. Example challenges arise in the physical sciences in the form of local movement constraints, required monotonicity in certain variables, and transitions influencing the accuracy of measurements. Altogether, such *transition constraints* necessitate a form of planning. This work extends classical Bayesian optimization via the framework of Markov Decision Processes. We iteratively solve a tractable linearization of our utility function using reinforcement learning to obtain a policy that plans ahead for the entire horizon. This is a parallel to the optimization of an *acquisition function in policy space*. The resulting policy is potentially history-dependent and non-Markovian. We showcase applications in chemical reactor optimization, informative path planning, machine calibration, and other synthetic examples.

Transition Constrained Bayesian Optimization via Markov Decision Processes

Abstract

Bayesian optimization is a methodology to optimize black-box functions. Traditionally, it focuses on the setting where you can arbitrarily query the search space. However, many real-life problems do not offer this flexibility; in particular, the search space of the next query may depend on previous ones. Example challenges arise in the physical sciences in the form of local movement constraints, required monotonicity in certain variables, and transitions influencing the accuracy of measurements. Altogether, such transition constraints necessitate a form of planning. This work extends classical Bayesian optimization via the framework of Markov Decision Processes. We iteratively solve a tractable linearization of our utility function using reinforcement learning to obtain a policy that plans ahead for the entire horizon. This is a parallel to the optimization of an acquisition function in policy space . The resulting policy is potentially history-dependent and non-Markovian. We showcase applications in chemical reactor optimization, informative path planning, machine calibration, and other synthetic examples.

1 Introduction

Many areas in the natural sciences and engineering deal with optimizing expensive black-box functions. Bayesian optimization (BayesOpt) [1-3], a method to optimize these problems using a probabilistic surrogate, has been successfully applied to a myriad of examples, e.g. hyper-parameter selection [4], robotics [5], battery design [6], laboratory equipment tuning [7], and drug discovery [8]. However, state-of-the-art algorithms are often ill-suited when physical sciences interact with potentially dynamic systems [9]. In such circumstances, real-life constraints limit our future decisions while depending on the prior state of our interaction with the system. This work focuses on transition constraints influencing future choices depending on the current state of the experiment. In other words, reaching certain parts of the decision space (search space) requires long-term planning in our optimization campaign. This effectively means we address a general sequential-decision problem akin to those studied in reinforcement learning (RL) or optimal control for the task of optimization. We assume the transition constraints are known a priori to the optimizer.

Applications with transition constraints include chemical reaction optimization [10-12], environmental monitoring [13-17], lake surveillance with drones [18-20], energy systems [21], vapor compression systems [22], electron-laser tuning [23] and seabed identification [24]. For example,

Robert M Lee

BASF SE

Ludwigshen, Germany

Figure 1 depicts an application in environmental monitoring where autonomous sensing vehicles must avoid obstacles (similar to Hitz et al. [18]). Our main focus application are transient flow reactors [25-27]. Such reactors allow efficient data collection by obtaining semi-continuous time-series data rather than a single measurement after reaching the steady state of the reactor. As we can only change the inputs of the reactor continuously and slowly to maintain quasi-steady-state operation, allowing arbitrary changes, as in conventional BayesOpt, would result in measurement sequences which are not possible due to physical limitations.

Problem Statement. More formally, we design an algorithm to identify the optimal configuration of a physical system governed by a black box function f , namely, x ⋆ = arg max x ∈X f ( x ) . The set X summarizes all possible system configurations, the so called search space . We assume that we can sequentially evaluate the unknown function at specific points x in the search space and obtain noisy observations, y = f ( x ) + ϵ ( x ) , where ϵ has a known Gaussian likelihood, which is possibly heteroscedastic. We assume that f can be modeled probabilistically using a Gaussian process prior that we introduce later. Importantly, the order of the evaluations is dictated by known , potentially stochastic, dynamics modeled by a Markov chain that limits our choices of x ∈ X .

BayesOpt with a Markov Decision Processes. The problem of maximizing an unknown function could be addressed by BayesOpt, which typically chooses to query f ( x ) by sequentially maximizing an acquisition function , u :

depending on all the past data at iteration t , X t . Eq. (1) arises as a greedy one-step approximation whose overall goal is to minimize e.g. cumulative regret, and assumes that any choice of point in the search space X is available. However, given transition constraints, we must traverse the search space according to the system dynamics. This work extends the BayesOpt framework and provides a method that constructs a potentially non-Markovian policy by myopically optimizing a utility as,

where U is the greedy utility of the policy π and X t encodes past trajectories through the search space. In the following sections, we will show how to tractably formulate the overall utility, how to greedily maximize it, and how to adapt it to admit policies depending on the full optimization history.

Contributions. We present a BayesOpt framework that tractably plans over the complete experimentation horizon and respects Markov transition constraints, building on active exploration in Markov chains [17]. Our key contributions include:

We identify a novel utility function for maximum identification as a function of policies, and greedily optimize it. The optimization is tractable, and does not scale exponentially in the policy horizon. In many cases, the problem is convex in the natural representation.

We provide exact solutions to the optimization problems using convex optimization for discrete Markov chains. For continuous Markov chains, we propose a reparameterization by viewing our problem as an instance of model predictive control (MPC) with a nonconvex objective. Interestingly, in both cases, the resulting policies are history-dependent (non-Markovian).

Weanalyze the scheme theoretically and empirically demonstrate its practicality on problems in physical systems, such as electron laser calibration and chemical reactor optimization.

2 Background

Because our contributions address experimental design of real-life systems by intersecting design of experiments, BayesOpt and RL, we review each of these of components. Refer to Figure 5 in Appendix A for a visual overview of how we selected the individual components for tractability of the entire problem.

Gaussian Processes To model the unknown function f , we use Gaussian processes (GPs) [28]. GPs are probabilistic models that capture nonlinear relationships and offer well-calibrated uncertainty estimates. Any finite marginal of a GP, e.g., for inputs ( x 1 , .., x p ) , the values { f ( x j ) } p j =1 , are normally distributed. We adopt a Bayesian approach and assume f is a sample from a GP prior with a known covariance kernel, k , and zero mean function, f ∼ GP(0 , k ) . Under these assumptions, the posterior of f , given a Gaussian likelihood of data, is a GP that is analytically tractable.

(a)

(c)

(b)

2.1 Maximum Identification: Experiment Design Goal

Classical BayesOpt is naturally myopic in its definition as a greedy one-step update (see (1)), but has the overall goal to minimize, e.g., the cumulative regret. Therefore u needs to chosen such that overall non-myopic goals can be achieved, usually defined as balancing an exploration-exploitation trade-off. In this paper we follow similar ideas; however, we do not focus on regret but instead on gathering information to maximize our chances to identify x ⋆ , the maximizer of f .

MaximumIdentification via Hypothesis testing. Maximumidentification can naturally be expressed as a multiple hypothesis testing problem, where we need to determine which of the elements in X is the maximizer. To do so, we require good estimates of the differences (or at least their signs) between individual queries f ( x i ) -f ( x j ) ; x i , x j ∈ X . For example, if f ( x i ) -f ( x j ) ≤ 0 , then x i cannot be a maximizer. Given the current evidence, the set of arms which we cannot rule out are all potential maximizers , Z ⊂ X . At termination we report our best guess for the maximizer as:

̸

Suppose we are in step t out of T , then let X t be the set of previous queries, we seek to identify new X new that when evaluated minimize the probability of returning a sub-optimal arm at the end. For a given function draw f , the probability of returning a wrong maximizer z = x ⋆ f is P ( µ T ( z ) -µ T ( x ⋆ f ) ≥ 0 | f ) . We can then consider the worst-case probability across potential maximizers, and taking expectation over f we obtain a utility through an asymptotic upper-bound on the log-probability, indeed for large T we obtain:

The expectation is on the current prior (posterior up to X t ), the kernel k is the posterior kernel given observations X t ∪ X new. Since we consider the probability of an error, it is more appropriate to talk about minimizing instead of 'maximizing the utility' but the treatment is analogous. Further, note the intuitive interpretation of the bound: the probability of an error will be minimized if the uncertainty is small or if the values of f ( z ) and f ( x ⋆ f ) are far apart. The non-trivial distribution of f ( x ⋆ ) [29] renders the utility intractable; therefore we employ a simple and tractable upper bound on the objective (3) which can be optimized by minimizing the uncertainty among all pairs in Z :

̸

Such objectives can be solved greedily in a similar way as acquisition functions in Eq. (1) by minimizing U over X new. Note that Fiez et al. [30] derive this objective for the same problem with linear bandits, albeit they consider the frequentist setting and (surprisingly) a different optimality criterion: minimizing T for a given failure rate. For their setting, the authors prove that it is an asymptotically optimal objective to follow. They do not consider any Markov chain structure. Derivation of the Bayesian utility and its upper bound in Eq.(4) can be found in Appendix C.1-C.2.

Utility with kernel embeddings. For illustrative purposes, consider a special case where the kernel k has a low rank due to existence of embeddings Φ( x ) ∈ R m , i.e., k ( x, y ) = Φ( x ) ⊤ Φ( y ) . Such embeddings can be, e.g., Nyström features [31] or Fourier features [32, 33]. While not necessary,

these formulations make the objectives considered in this work more tractable and easier to expose to the reader. With the finite rank assumption, the random function f becomes,

where θ are weights with a Gaussian prior. We can then rewrite the objective Eq. (4) as:

This reveals an essential observation that the utility depends only on the visited states; not their order. This suggests a vast simplification, where we do not to model whole trajectories, and Markov decision processes sufficiently describe our problem. Additionally, numerically, the objective involves the inversion of an m × m matrix instead of |X| × |X| (see Sec. 4). Appendix D.1 provides a utility without the finite rank-assumptions that is more involved symbolically and computationally.

2.2 Markov Decision Processes

To model the transition constraints, we use the versatile model of Markov Decision processes (MDPs). We assume an environment with state space X and action space A , where we interact with an unknown function f : X × A → R by rolling out a policy for H time-steps (horizon) and obtain a trajectory, τ = ( x 0 , a 0 , x 1 , a 1 , ..., x H -1 , a H -1 ) . From the trajectory, we obtain a sequence of noisy observations y ( τ ) := { y ( x 0 , a 0 ) , ..., y ( x H -1 , a H -1 ) } s.t. y ( x h ) = f ( x h , a h ) + ϵ ( x h , a h ) , where ϵ ( x h , a h ) is zero-mean Gaussian with known variance which is potentially state and action dependent. The trajectory is generated using a known transition operator P ( x h +1 | x h , a h ) . A Markov policy π ( a h | x h ) is a mapping that dictates the probability of action a h in state x h . Hence, the state-to-state transitions are P ( x h +1 , x h ) = ∑ a ∈A π h ( a | x h ) P ( x h +1 | x h , a ) . In fact, an equivalent description of any Markov policy π is the corresponding distribution giving us the probability of visiting a state-action pair under the policy, which we denote d π ∈ D , where

We will use this polytope to reformulate our optimization problem over trajectories. Any d ∈ D can be realized by a Markov policy π and vice-versa. We work with non-stationary policies, meaning the policies depend on horizon count h . The execution of deterministic trajectories is only possible for deterministic transitions. Otherwise, the resulting trajectories are random. In our setup, we repeat interactions T times (episodes) to obtain the final dataset of the form X T = { τ i } T i =1 .

2.3 Experiment Design in Markov Chains

Notice that the utility U in Eq. 6 depends on the states visited and hence states of the trajectory. In our notation, X t will now form a set of executed trajectories. With deterministic dynamics, we could optimize over trajectories, but this would lead to an exponential blowup (i.e. | X | H ). In fact, for stochastic transitions, we cannot pick the trajectories directly, so instead we work in the space of distributions. For a given policy, through sampling, we are able to create an empirical distribution of all the state-action pairs visited during policy executions, ˆ d π ( x, a ) , which assigns equal mass to each state-action visited during our trajectories. This allows us to focus on the expected utility over the randomness of the policy and the environment, namely,

This formulation stems from Mutný et al. [17] who try to tractably solve such objectives that arise in experiment design by performing planning in MDPs. They focus on learning linear operators of an unknown function, unlike identifying a maximum, as we do here. The key observation they make is that any policy π induces a distribution over the state-action visitations, d π . Therefore we can reformulate the problem of finding the optimal policy, into finding the optimal distribution over state-action visitations as: min d π ∈D U ( d π ) , and then construct policy π via marginalization. We refer to this optimization as the planning problem . The constraint D encodes the dynamics of the MDP.

2.4 Additional Related Works

The most relevant prior work to ours is exploration in reinforcement learning through the use of Markov decision processes as in Mutný et al. [17] and convex reinforcement learning of Hazan et al. [34], Zahavy et al. [35] which we will use to optimize the objective. Other related works are:

Pure exploration bandits objectives. Similar objectives to ours have been explored for BayesOpt. Li and Scarlett [36] use the G -allocation variant of our objective for batch BayesOpt, achieving good theoretical bounds. Zhang et al. [37] and recently Han et al. [38] take advantage of possible maximizer sets to train localized models, while Salgia et al. [39] show that considering adaptive maximization sets yields good regret bounds under random sampling. Contrary to them, motivation and derivation in terms of a Bayesian decision rule do not appear elsewhere according to our best knowledge. We also recognize that we can relax the objective and optimize it in the space of policies.

Optimizing over sequences. Previous work has focused on planning experimental sequences for minimizing switching costs [11, 21, 40, 41] however they are only able adhere to strict constraints under truncation heuristics [20, 22, 42]. Recently, Qing et al. [43] also tackle Bayesian optimization within dynamical systems, with the focus of optimizing initial conditions. Concurrent work of Che et al. [44] tackles a constrained variant of a similar problem using model predictive control with a different goal.

Regret vs Best-arm identification. Most algorithms in BayesOpt focus on regret minimization. This work focuses on maximizer identification directly, i.e., to identify the maximum after a certain number of iterations with the highest confidence. This branch of BayesOpt is mostly addressed in the bandit literature [45]. Our work builds upon prior works of Soare et al. [46], Yu et al. [47], and specifically upon the seminal approach of Fiez et al. [30] to design an optimal objective via hypothesis testing. Novel to our setting is the added difficulty of transition constraints necessitating planning.

Non-myopic Bayesian Optimization. Look-ahead BayesOpt [48-54] seeks to improve the greedy aspect of BayesOpt. Such works also use an MDP problem formulation, however, they define the state space to include all past observations (e.g. [55, 56]). This comes at the cost of simulating expensive integrals, and the complexity grows exponentially with the number of look-ahead steps (usually less than three steps). Our work follows a different path, we maintain the greedy approach to control computational efficiency (i.e. by optimizing over the space of Markovian policies), and maintain provable and state-of-art performance. Even though the optimal policy through non-myopic analysis is non-Markovian, in Sec. 4, we show that adaptive resampling iteratively approximates this non-myoptic optimal policies in a numerically tractable way via receeding horizon planning. In our experiments we comfortably plan for over a hundred steps.

3 Transition Constrained BayesOpt

This section introduces BayesOpt with transition constraints. We use MDPs to encode constraints. Namely, the Markov dynamics dictates which inputs we are allowed to query at time-step h + 1 given we previously queried state x h . This mean that the transition operator is P ( x h +1 | x h , a ) = 0 for any transition x h → x h +1 not allowed by the physical constraints.

Motivated by our practical experiments with chemical reactors, we distinguish two different types of feedback . With episodic feedback we can be split the optimization into episodes. At the end of each episode of length H , we obtain the whole set of noisy observations. On the other hand, instant feedback is the setting where we obtain a noisy observation immediately after querying the function. Asynchronous feedback describes a mix of the previous two, where we obtain observations with unspecific a delay.

3.1 Expected Utility for Maximizer Identification

In section 2.1 we introduced the utility for maximum identification. Using the same simplifying assumption (finite rank approximation of GPs in Sec. 2.1, Eq. (4)), we can show that the expected utility U can be rewritten in terms of the state-action distribution induced by X new:

where V ( d π ) = ( ∑ x,a ∈X×A d π ( x,a )Φ( x,a )Φ( x,a ) ⊤ σ 2 ( x,a ) + 1 TH I ) . The variable d π ( x, a ) is a state-action visitation, Φ( x ) are e.g. Nyström features of the GP. We prove that the function is additive in terms of state-action pairs in Lemma D.1 in Appendix D, a condition required for the expression as a function of state-action visitations [17]. Additionally, by rewriting the objective in this form, the dependence and convexity with respect to the state-action density d π becomes clear as it is only composition of a linear function with an inverse operator. Also, notice that the constraint set is a convex polytope. Therefore we are able to use convex optimization to solve the planning problem (see Sec. 4).

Input: Procedure for estimating sets of maximizers, initial point x 0 , initial set of maximizer candidates Z 0

Initialize the empirical state-action distribution ˆ d = 0

Set of potential maximizers Z . The definition of the objective requires the use of a set of maximizers. In the ideal case, we can say a particular input x , is not the optimum if there exists x ′ such that f ( x ′ ) > f ( x ) with high confidence. We formalize this using the GP credible sets (Bayesian confidence sets) and define:

where UCB and LCB correspond to the upper and lower confidence bounds of the GP surrogate with a user specified confidence level defined via the posterior GP with data up to X t .

3.2 Discrete vs Continuous MDPs.

Until this point, our formulation focused on discrete S and A for ease of exposition. However, the framework is compatible with continuous state-action spaces. The probabilistic reformulation of the objective in Eq. (7) is possible irrespective of whether X (or A ) is a discrete or continuous subset of R d . In fact, the convexity of the objective in the space of distributions is still maintained. The difference is that the visitations d are no longer probability mass functions but have to be expressed as probability density functions d c ( x, a ) . To recover probabilities in the definition of V , we need to

In the Eq. (8) we need to approximate a maximum over all input pairs in Z . While this can be enumerated in the discrete case without issues, it poses a non-trivial constrained optimization problem when X is continuous. As an alternative, we propose approximating the set Z using a finite approximation of size K which can be built using Thompson Sampling [57, 58] or through maximization of different UCBs for higher exploitation (see Appendix E.1). In Appendix E.5, we numerically benchmark reasonable choices of K , and show that the performance is not significantly affected by them.

3.3 General algorithm and Theory

The general algorithm combines the ideas introduced so far. We present it in Algorithm 1. Notice that apart from constructing the current utility, keeping track of the visited states and updating our GP model, an essential step is planning , where we need to find a policy that maximizes the utility. As this forms the core challenge of the algorithm, we devote Sec. 4 to it. In short, it solves a sequence of dynamic programming problems defined by the steps of the Frank-Wolfe algorithm. From a theoretical point of view, under the assumption of episodic feedback, the algorithm provably minimizes the utility as we show in Proposition C.1 in Appendix C.4.

4 Solving the planning problem

The planning problem, defined as min d π ∈D U ( d π ) , can be thought of as analogous to optimizing an acquisition function in traditional BayesOpt, with the added difficulty of doing it in the space of

policies. See the bottom half of Figure 5 in Appendix A for a breakdown of the different components of our solution. Following developments in Hazan et al. [34] and Mutný et al. [17], we use the classical Frank-Wolfe algorithm [59]. It proceeds by decomposing the problem into a series of linear optimization sub-problems. Each linearization results in a policy, and we build a mixture policy consisting of optimal policies for each linearization π mix ,n = { ( α i , π i ) } n i =1 , and α i step-sizes of Frank-Wolfe. Conveniently, after the linearization of U the subproblem on the polytope D corresponds to an RL problem with reward ∇U for which many efficient solvers exist. Namely, for a single mixture component we have,

Due to convexity, the state-action distribution follows the convex combination, d π mix ,n = ∑ n i =1 α i d π i . The optimization produces a Markovian policy due to the subproblem in Eq. (10) being optimized by one. We now detail how to construct a non-Markovian policies by adaptive resampling.

4.1 Adaptive Resampling: Non-Markovian policies.

A core contribution of our paper is receding horizon re-planning. This means that we keep track of the past states visited in the current and past trajectories and adjust the policy at every step h of the horizon H in each trajectory indexed by t . At h , we construct a Markov policy for a reward that depends on all past visited states. This makes the resulting policy history dependent. While in episode t and time-point h we follow a Markov policy for a single step, the overall policy is a history-dependent non-Markov policy.

We define the empirical state-action visitation distribution,

where δ x,a denotes a delta mass at state-action ( x, a ) . Instead of solving the objective U ( d ) as in Eq. (10), we seek to find a correction to the empirical distribution by minimizing,

We use the same Frank-Wolfe machinery to optimize this objective: d π t,h = arg min d π ∈ ˜ D U t,h ( d π ) . The distribution d π t,h represents the density of the policy to be deployed at trajectory t and horizon counter h . We now need to solve multiple ( n due to FW) RL problems at each horizon counter h . Despite this, for discrete MDPs, the sub-problem can be solved extremely efficiently to exactness using dynamic programming. As can be seen in Appendix B.4, our solving times are just a few seconds, even if planning for very long horizons. The resulting policy π can be found by marginalization π h ( a | x ) = d π,h ( x, a ) / ∑ a d π,h ( x, a ) , a basic property of MDPs [60].

4.2 Continuous MDPs: Model Predictive Control

With continuous search space, the sub-problem can be solved using continuous RL solvers. However, this can be difficult. The intractable part of the problem is that the distribution d π needs to be represented in a computable fashion. We represent the distribution by the sequence of actions taken { a h } T h =1 with the linear state-space model, x h +1 = Ax h + Ba h . While this formalism is not as general as it could be, it gives us a tractable sub-problem formulation common to control science scenario [61] that is practical for our experiments and captures a vast array of problems. The optimal set of actions is solved with the following problem, where we state it for the full horizon H :

such that || a h || ≤ a max , x h ∈ X , and x h +1 = Ax h + Ba h , where the known dynamics serves as constraints. Notice that instead of optimizing over the policy d π , we directly optimize over the parameterizations of the policy { a h } H h =1 . In fact, this formulation is reminiscent of the model predictive control (MPC) optimization problem. Conceptually, these are the same. The only caveat in our case is that unlike in MPC [62], our objective is non-convex and tends to focus on gathering information rather than stability. Due to the non-convexity in this parameterization, we need to solve it heuristically. We identify a number of useful heuristics to solve this problem in Appendix G.

1.0

0.05

回

1.0

Greedy-UCB

Best

MDP-EI

eactants

Potentialmaximizers

Regret

MDP-BO

Arm

0.75

Discardedstates

Identified

rea

Average

0.5

0.5

6

0.25

1(%)

atio

0.0

0.00

0.0

1

2

3

4

5

6

7

8

9

10

0

10

20

30

40

Episode

residence time (t)

5 Experiments

Sections 5.1 - 5.3 showcase real-world applications under physical transitions constraints, using the discrete version of the algorithm. Section 5.4 benchmarks against other algorithms in the continuous setting, where we consider the additive transition model of Section 4.2 with A = B = I . We include additional results in Appendix B. For each benchmark, we selected reasonable GP hyper-parameters and fixed them during the optimization. These are summarized in Appendix E.2. As we are interested maximizer identification, in discrete problems, we report the proportion of reruns that succeed at identifying the true maximum. For continuous benchmarks, we report inference regret at each iteration: Regret t = f ( x ∗ ) -f ( x µ,t ) , where x µ,t = arg max x ∈X µ t ( x ) . All statistics reported are over 25 different runs.

Baselines. We include a naive baseline that greedily optimizes the immediate reward to showcase a method with no planning (Greedy-UCB). Likewise, we create a baseline that replaces the gradient in Eq. (10) with Expected Improvement [63] (MDP-EI), a weak version of planning. In the continuous settings, we compare against truncated SnAKe (TrSnaKe) [42], which minimizes movement distance, and against local search region-constrained BayesOpt or LSR [22] for the same task. We compare two variants for approximating the set of maximizers, one using Thompson Sampling (MDP-BO-TS) and one using Upper Confidence Bound (MDP-BO-UCB).

5.1 Knorr pyrazole synthesis

1.0

0.037

Greedy-UCB

Best Arm Identified (%)

MDP-EI

Average Regret

MDP-B0

0.5

0.000

0.0

2

3

Episode

(a) Monitoring Lake Ypacarai.

1.0

0.739

MDP-BO

Best Arm Identified (%)

State-less BO

Average Regret

0.5

0.000

0.0

0

20

40

60

80

100

Iteration

(b) Free-electron laser tuning.

(f) SnAr 4D. (asynch)

3

3

Median Regret

1.5

MDP-BO-TS

Median Regret

Median Regret

MDP-BO-UCB

2

2

1.0

TrSnAKe

0.5

LSR

1

0.0

0

0

0

10

20

30

40

50

0

10

20

30

40

50

0

10

20

30

40

50

Iteration

Iteration

Iteration

1.5

Median Regret

0.6

Median Regret

3

0.4

2

1.0

0.2

1

0.5

0.0

0

0.0

20

30

40

50

60

70

80

90

100

20

30

40

50

60

70

80

90

100

20

30

40

50

60

70

80

90

100

Iteration

Iteration

Iteration

Our chemical reactor benchmark synthetizes Knorr pyrzole in a transient flow reactor. In this experiment, we can control the flow-rate (residence time) τ and ratio of reactants B in the reactor. We observe product concentration at discrete time intervals and we can also change inputs at these intervals. Our goal is to find the best parameters of the reaction subject to natural movement constraints on B , and τ . In addition, we assume decreasing the flow rate of a reactor can be easily achieved. However, increasing the flow rate can lead to inaccurate readings [64]. A lower flow rate leads to higher residence time, so we impose that τ must be non-decreasing.

The kernel. Schrecker et al. [27] indicate the reaction can be approximately represented by simple kinetics via a differential equation model. We use this information along with techniques for representing linear ODE as constraints in GP fitting [65, 66] to create an approximate ODE kernel k ode through the featurization:

where y ( i ) ( τ, B ) are equal to:

for i = 1 , 2 , where λ ( i ) 1 and λ ( i ) 2 are eigenvalues of the linearized ODE at different stationary points, γ 1 ( B ) = B , γ 2 ( B ) = 1 -B , and S ( x ) := (1 + e -α sig ( x -0 . 5) ) -1 is a sigmoid function. Appendix H holds the details and derivations which may be of independent interest. As the above kernel is only an approximation of the true ODE kernel, which itself is imperfect, we must account for the model mismatch. Therefore, we add a squared exponential term to the kernel to ensure a non-parametric correction, i.e.: k ( τ, B ) = α ode k ode ( τ, B ) + α rbf ( τ, B ) .

We report the examples of the trajectories in the search space in Figure 2. Notice that all satisfy the transition constraints. The paths are not space-filling and avoid sub-optimal areas because of our choice of non-isotropic kernel based on the ODE considerations. We run the experiment with episodic feedback, for 10 episodes of length 10 each, starting each episode with ( τ R , B ) = (0 , 0) . Figure 2 reports quantitative results and shows that the best-performing algorithm is MDP-BO.

5.2 Monitoring Lake Ypacarai

Samaniego et al. [20] investigated automatic monitoring of Lake Ypacarai, and Folch et al. [11] and Yang et al. [40] benchmarked different BayesOpt algorithms for the task of finding the largest contamination source in the lake. We introduce local transition constraints to this benchmark by creating the lake containing obstacles that limit movement (see Figure 12 in the Appendix). Such obstacles

in environmental monitoring may include islands or protected areas for animals. We add an initial and final state constraint with the goal of modeling that the boat has to finish at a maintenance port.

We focus on episodic feedback, where each episode consists of 50 iterations. Results can be seen in Figure 3a. MDP-EI struggles to identify the maximum contamination for the first few episodes. On the other hand, our method correctly identifies the maximum in approximately 50% of the runs by episode two and achieves better regret.

5.3 Free-electron laser: Transition-driven corruption

Apart from hard constraints, we can apply our framework to state-dependent BayesOpt problems involving transitions. For example, the magnitude of noise ϵ may depend on the transition. This occurs in systems observing equilibration constraints such as a free-electron laser [23]. Using the simplified simulator of this laser [67], we use our framework to model heteroscedastic noise depending on the difference between the current and next state, σ 2 ( x, x ′ ) = s (1 + w || x -x ′ || 2 ) . By choosing A = X , we rewrite the problem as σ ( s, a ) = s (1 + w || x -a || 2 ) . The larger the move, the more noisy the observation. This creates a problem, where the BayesOpt needs to balance between informative actions and movement, which can be directly implemented in the objective (8) via the matrix V ( d π ) = ∑ x,a ∈X d π ( x, a ) 1 σ 2 ( x,a ) Φ( x )Φ( x ) ⊤ + 1 TH I . Figure 3b reports the comparison between worst-case stateless BO and our algorithm. Our approach substantially improves performance.

5.4 Synthetic Benchmarks

We benchmark on a variety of classical BayesOpt problems while imposing local movement constraints and considering both immediate and asynchronous feedback (by introducing an observation delay of 25 iterations). We also include the chemistry SnAr benchmark, from Summit [68], which we treat as asynchronous as per Folch et al. [11]. Results are in Figure 4. In the synchronous setting, we found using the UCB maximizer criteria for MDP-BO yields the best results (c.f. Appendix for details of this variant). We also found that LSR performs very competitively on many benchmarks, frequently matching the performance of MDP-BO. In the asynchronous settings we achieved better results using MDP-BO with Thompson sampling. TrSnAKe baseline appears to be competitive in all synthetic benchmarks as well. However, MDP-BO is more robust having less variance in the chosen paths as seen in the quantiles. It is important to highlight that SnAKe and LSR are specialist heuristic algorithms for local box-constraints, and therefore it is not surprising they perform strongly. Our method can be applied to more general settings and therefore it is very encouraging that MDP-BO is able to match these SOTA algorithms in their specialist domain.

6 Conclusion

We considered transition-constrained BayesOpt problems arising in physical sciences, such as chemical reactor optimization, that require careful planning to reach any system configuration. Focusing on maximizer identification, we formulated the problem with transition constraints using the framework of Markov decision processes and constructed a tractable algorithm for provably and efficiently solving these problems using dynamic programming or model predictive control sub-routines. We showcased strong empirical performance in a large variety of problems with physical transitions, and achieve state-of-the-art results in classical BayesOpt benchmarks under local movement constraints. This work takes an important step towards the larger application of Bayesian Optimization to real-world problems. Further work could address the continuous variant of the framework to deal with more general transition dynamics, or explore the performance of new objective functions.

Acknowledgments

JPF is funded by EPSRC through the Modern Statistics and Statistical Machine Learning (StatML) CDT (grant no. EP/S023151/1) and by BASF SE, Ludwigshafen am Rhein. RM acknowledges support from the BASF / Royal Academy of Engineering Research Chair in Data-Driven Optimisation. This publication was created as part of NCCR Catalysis (grant number 180544), a National Centre of Competence in Research funded by the Swiss National Science Foundation, and was partially supported by the European Research Council (ERC) under the European Union's Horizon 2020 research and Innovation Program Grant agreement no. 815943. We would also like to thank Linden Schrecker, Ruby Sedgwick, and Daniel Lengyel for providing valuable feedback on the project.

References

Peter I Frazier. A tutorial on Bayesian optimization. arXiv preprint arXiv:1807.02811 , 2018.

Donald R Jones, Matthias Schonlau, and William J Welch. Efficient global optimization of expensive black-box functions. Journal of Global Optimization , 13(4):455-492, 1998.

Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P. Adams, and Nando de Freitas. Taking the human out of the loop: A review of Bayesian optimization. Proceedings of the IEEE , 104 (1):148-175, 2016.

Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren, editors. Automated Machine Learning Methods, Systems, Challenges . Springer, 2019.

Alonso Marco, Felix Berkenkamp, Philipp Hennig, Angela P Schoellig, Andreas Krause, Stefan Schaal, and Sebastian Trimpe. Virtual vs. real: Trading off simulations and physical experiments in reinforcement learning with Bayesian optimization. In 2017 IEEE International Conference on Robotics and Automation (ICRA) , pages 1557-1563. IEEE, 2017.

Jose Pablo Folch, Robert M Lee, Behrang Shafei, David Walz, Calvin Tsay, Mark van der Wilk, and Ruth Misener. Combining multi-fidelity modelling and asynchronous batch Bayesian optimization. Computers & Chemical Engineering , 172:108194, 2023.

Thomas M Dixon, Jeanine Williams, Maximilian Besenhard, Roger M Howard, James MacGregor, Philip Peach, Adam D Clayton, Nicholas J Warren, and Richard A Bourne. Operator-free HPLC automated method development guided by Bayesian optimization. Digital Discovery , 3 (8):1591-1601, 2024.

Jennifer Brennan, Lalit Jain, Sofia Garman, Ann E Donnelly, Erik Scott Wright, and Kevin Jamieson. Sample-efficient identification of high-dimensional antibiotic synergy with a normalized diagonal sampling design. PLOS Computational Biology , 18(7):e1010311, 2022.

Alexander Thebelt, Johannes Wiebe, Jan Kronqvist, Calvin Tsay, and Ruth Misener. Maximizing information from chemical engineering data sets: Applications to machine learning. Chemical Engineering Science , 252:117469, 2022.

Pratibha Vellanki, Santu Rana, Sunil Gupta, David Rubin, Alessandra Sutti, Thomas Dorin, Murray Height, Paul Sanders, and Svetha Venkatesh. Process-constrained batch Bayesian optimisation. In Advances in Neural Information Processing Systems . Curran Associates, Inc., 2017.

Jose Pablo Folch, Shiqiang Zhang, Robert Lee, Behrang Shafei, David Walz, Calvin Tsay, Mark van der Wilk, and Ruth Misener. SnAKe: Bayesian optimization with pathwise exploration. In Advances in Neural Information Processing Systems , volume 35, pages 35226-35239, 2022.

Sarah L Boyall, Holly Clarke, Thomas Dixon, Robert WM Davidson, Kevin Leslie, Graeme Clemens, Frans L Muller, Adam D Clayton, Richard A Bourne, and Thomas W Chamberlain. Automated optimization of a multistep, multiphase continuous flow process for pharmaceutical synthesis. ACS Sustainable Chemistry & Engineering , 2024.

Ilija Bogunovic, Jonathan Scarlett, Andreas Krause, and Volkan Cevher. Truncated variance reduction: A unified approach to Bayesian optimization and level-set estimation. Advances in Neural Information Processing Systems , 29, 2016.

Sigrid Passano Hellan, Christopher G Lucas, and Nigel H Goddard. Bayesian optimisation for active monitoring of air pollution. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 11908-11916, 2022.

Sigrid Passano Hellan, Christopher G Lucas, and Nigel H Goddard. Bayesian optimisation against climate change: Applications and benchmarks. Data-centric Machine Learning Research (DMLR) Workshop at the 40th International Conference on Machine Learning , 2023.

Clara Stoddart, Lauren Shrack, Richard Sserunjogi, Usman Abdul-Ganiy, Engineer Bainomugisha, Deo Okure, Ruth Misener, Jose Pablo Folch, and Ruby Sedgwick. Gaussian processes for monitoring air-quality in Kampala. NeurIPS 2023 Workshop: Tackling Climate Change with Machine Learning , 2023.

Mojmír Mutný, Tadeusz Janik, and Andreas Krause. Active Exploration via Experiment Design in Markov Chains. In Proceedings of The 26th International Conference on Artificial Intelligence and Statistics , volume 206, pages 7349-7374, 2023.

Gregory Hitz, Alkis Gotovos, Marie-Éve Garneau, Cédric Pradalier, Andreas Krause, Roland Y Siegwart, et al. Fully autonomous focused exploration for robotic environmental monitoring. In 2014 IEEE International Conference on Robotics and Automation (ICRA) , pages 2658-2664. IEEE, 2014.

Alkis Gotovos, Nathalie Casati, Gregory Hitz, and Andreas Krause. Active learning for level set estimation. In IJCAI 2013 , 2013.

Federico Peralta Samaniego, Daniel Gutiérrez Reina, Sergio L Toral Marín, Mario Arzamendia, and Derlis O Gregor. A Bayesian optimization approach for water resources monitoring through an autonomous surface vehicle: The Ypacarai lake case study. IEEE Access , 9:9163-9179, 2021.

Shyam Sundhar Ramesh, Pier Giuseppe Sessa, Andreas Krause, and Ilija Bogunovic. Movement penalized Bayesian optimization with application to wind energy systems. In Advances in Neural Information Processing Systems , volume 35, pages 27036-27048, 2022.

Joel A Paulson, Farshud Sorouifar, Christopher R Laughman, and Ankush Chakrabarty. LSRBO: Local search region constrained Bayesian optimization for performance optimization of vapor compression systems. In 2023 American Control Conference (ACC) , pages 576-582. IEEE, 2023.

Johannes Kirschner, Mojmír Mutný, Andreas Krause, Jaime Coello de Portugal, Nicole Hiller, and Jochem Snuverink. Tuning particle accelerators with safety constraints using Bayesian optimization. Phys. Rev. Accel. Beams , 25:062802, 2022.

Matthew Sullivan, John Gebbie, and John Lipor. Adaptive sampling for seabed identification from ambient acoustic noise. IEEE CAMSAP , 2023.

Sergey Mozharov, Alison Nordon, David Littlejohn, Charlotte Wiles, Paul Watts, Paul Dallin, and John M Girkin. Improved method for kinetic studies in microreactors using flow manipulation and noninvasive Raman spectrometry. Journal of the American Chemical Society , 133(10): 3601-3608, 2011.

Linden Schrecker, Joachim Dickhaut, Christian Holtze, Philipp Staehle, Andy Wieja, Klaus Hellgardt, and King Kuok Mimi Hii. An efficient multiparameter method for the collection of chemical reaction data via 'one-pot' transient flow. Reaction Chemistry & Engineering , 8(12): 3196-3202, 2023.

Linden Schrecker, Joachim Dickhaut, Christian Holtze, Philipp Staehle, Marcel Vranceanu, Klaus Hellgardt, and King Kuok Mimi Hii. Discovery of unexpectedly complex reaction pathways for the Knorr pyrazole synthesis via transient flow. Reaction Chemistry & Engineering , 8(1):41-46, 2023.

Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning) . The MIT Press, 2005.

Philipp Hennig and Christian J Schuler. Entropy search for information-efficient global optimization. Journal of Machine Learning Research , 13(6), 2012.

Tanner Fiez, Lalit Jain, Kevin G Jamieson, and Lillian Ratliff. Sequential experimental design for transductive linear bandits. Advances in neural information processing systems , 32, 2019.

Christopher Williams and Matthias Seeger. Using the Nyström method to speed up kernel machines. Advances in neural information processing systems , 13, 2000.

Mojmír Mutný and Andreas Krause. Efficient high dimensional Bayesian optimization with additivity and quadrature fourier features. Advances in Neural Information Processing Systems , 31, 2018.

Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in neural information processing systems , 20, 2007.

Elad Hazan, Sham Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy exploration. In International Conference on Machine Learning , pages 2681-2691. PMLR, 2019.

Tom Zahavy, Brendan O'Donoghue, Guillaume Desjardins, and Satinder Singh. Reward is enough for convex MDPs. Advances in Neural Information Processing Systems , 34:2574625759, 2021.

Zihan Li and Jonathan Scarlett. Gaussian process bandit optimization with few batches. In International Conference on Artificial Intelligence and Statistics , pages 92-107, 2022.

Fengxue Zhang, Jialin Song, James C Bowden, Alexander Ladd, Yisong Yue, Thomas Desautels, and Yuxin Chen. Learning regions of interest for Bayesian optimization with adaptive level-set estimation. In International Conference on Machine Learning , pages 41579-41595. PMLR, 2023.

Minbiao Han, Fengxue Zhang, and Yuxin Chen. No-regret learning of Nash equilibrium for black-box games via Gaussian processes. arXiv preprint arXiv:2405.08318 , 2024.

Sudeep Salgia, Sattar Vakili, and Qing Zhao. Random exploration in Bayesian optimization: Order-optimal regret and computational efficiency. In Forty-first International Conference on Machine Learning , 2024.

Adam X Yang, Laurence Aitchison, and Henry B Moss. MONGOOSE: Path-wise smooth Bayesian optimisation via meta-learning. arXiv preprint arXiv:2302.11533 , 2023.

Qiyuan Chen and Raed Al Kontar. The traveling bandit: A framework for Bayesian optimization with movement costs. arXiv preprint arXiv:2410.14533 , 2024.

Jose Pablo Folch, James Odgers, Shiqiang Zhang, Robert M Lee, Behrang Shafei, David Walz, Calvin Tsay, Mark van der Wilk, and Ruth Misener. Practical path-based Bayesian optimization. NeurIPS 2023 Workshop on Adaptive Experimental Design and Active Learning in the Real World , 2023.

Jixiang Qing, Becky D Langdon, Robert M Lee, Behrang Shafei, Mark van der Wilk, Calvin Tsay, and Ruth Misener. System-aware neural ODE processes for few-shot Bayesian optimization. arXiv preprint arXiv:2406.02352 , 2024.

Ethan Che, Jimmy Wang, and Hongseok Namkoong. Planning contextual adaptive experiments with model predictive control. NeurIPS 2023 Workshop on Adaptive Experimental Design and Active Learning in the Real World , 2023.

Jean-Yves Audibert and Sébastien Bubeck. Best arm identification in multi-armed bandits. In Conference on Learning Theory , pages 13-p, 2010.

Marta Soare, Alessandro Lazaric, and Rémi Munos. Best-arm identification in linear bandits. Advances in Neural Information Processing Systems , 27, 2014.

Kai Yu, Jinbo Bi, and Volker Tresp. Active learning via transductive experimental design. In Proceedings of the 23rd international conference on Machine learning , pages 1081-1088, 2006.

David Ginsbourger and Rodolphe Le Riche. Towards Gaussian process-based optimization with finite time horizon. In Advances in Model-Oriented Design and Analysis: Proceedings of the 9th International Workshop in Model-Oriented Design and Analysis , pages 89-96. Springer, 2010.

Roman Marchant, Fabio Ramos, Scott Sanner, et al. Sequential Bayesian optimisation for spatial-temporal monitoring. In UAI , pages 553-562, 2014.

Remi Lam, Karen Willcox, and David H Wolpert. Bayesian optimization with a finite budget: An approximate dynamic programming approach. Advances in Neural Information Processing Systems , 29, 2016.

Shali Jiang, Henry Chai, Javier Gonzalez, and Roman Garnett. Binoculars for efficient, nonmyopic sequential experimental design. In International Conference on Machine Learning , pages 4794-4803. PMLR, 2020.

Eric Hans Lee, David Eriksson, Valerio Perrone, and Matthias Seeger. A nonmyopic approach to cost-constrained Bayesian optimization. In Uncertainty in Artificial Intelligence , pages 568-577. PMLR, 2021.

Joel A Paulson, Farshud Sorouifar, and Ankush Chakrabarty. Efficient multi-step lookahead Bayesian optimization with local search constraints. In 2022 IEEE 61st Conference on Decision and Control (CDC) , pages 123-129. IEEE, 2022.

Mujin Cheon, Haeun Byun, and Jay H Lee. Reinforcement learning based multi-step look-ahead bayesian optimization. IFAC-PapersOnLine , 55(7):100-105, 2022.

Shali Jiang, Daniel Jiang, Maximilian Balandat, Brian Karrer, Jacob Gardner, and Roman Garnett. Efficient nonmyopic bayesian optimization via one-shot multi-step trees. Advances in Neural Information Processing Systems , 33:18039-18049, 2020.

Raul Astudillo, Daniel Jiang, Maximilian Balandat, Eytan Bakshy, and Peter Frazier. Multi-step budgeted bayesian optimization with unknown evaluation costs. Advances in Neural Information Processing Systems , 34:20197-20209, 2021.

William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika , 25(3-4):285-294, 1933.

Kirthevasan Kandasamy, Akshay Krishnamurthy, Jeff Schneider, and Barnabás Póczos. Parallelised Bayesian optimisation via Thompson sampling. In International Conference on Artificial Intelligence and Statistics , pages 133-142. PMLR, 2018.

Martin Jaggi. Revisiting Frank-Wolfe: Projection-Free Sparse Convex Optimization. In Sanjoy Dasgupta and David McAllester, editors, Proceedings of the 30th International Conference on Machine Learning , volume 28, pages 427-435. PMLR, 17-19 Jun 2013.

Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming . John Wiley Sons, 2014.

James Blake Rawlings, David Q Mayne, and Moritz Diehl. Model predictive control: theory, computation, and design , volume 2. Nob Hill Publishing, 2017.

Carlos E. García, David M. Prett, and Manfred Morari. Model predictive control: Theory and practice-A survey. Automatica , 25(3):335-348, 1989.

Vydu-nas R Šaltenis. One method of multiextremum optimization. Avtomatika i Vychislitel'naya Tekhnika (Automatic Control and Computer Sciences) , 5(3):33-38, 1971.

Linden Schrecker, Joachim Dickhaut, Christian Holtze, Philipp Staehle, Marcel Vranceanu, Andy Wieja, Klaus Hellgardt, and King Kuok Hii. A comparative study of transient flow rate steps and ramps for the efficient collection of kinetic data. Reaction Chemistry & Engineering , 2024.

Mojmír Mutný and Andreas Krause. Experimental design for linear functionals in reproducing kernel Hilbert spaces. Advances in Neural Information Processing Systems , 35:20175-20188, 2022.

Andreas Besginow and Markus Lange-Hegermann. Constraining Gaussian processes to systems of linear ordinary differential equations. Advances in Neural Information Processing Systems , 35:29386-29399, 2022.

Mojmír Mutný, Johannes Kirschner, and Andreas Krause. Experimental design for optimization of orthogonal projection pursuit models. In Proceedings of the 34th AAAI Conference on Artificial Intelligence (AAAI) , 2020.

Kobi Felton, Jan Rittig, and Alexei Lapkin. Summit: Benchmarking Machine Learning Methods for Reaction Optimisation. Chemistry Methods , February 2021.

Tianbao Yang, Yu-Feng Li, Mehrdad Mahdavi, Rong Jin, and Zhi-Hua Zhou. Nyström method vs random Fourier features: A theoretical and empirical comparison. Advances in neural information processing systems , 25, 2012.

Daniel Russo. Simple bayesian algorithms for best arm identification. In Conference on Learning Theory , pages 1417-1418. PMLR, 2016.

Niranjan Srinivas, Andreas Krause, Sham M Kakade, and Matthias Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. arXiv preprint arXiv:0912.3995 , 2009.

Felipe Cucker and Steve Smale. On the mathematical foundations of learning. Bulletin of the American mathematical society , 39(1):1-49, 2002.

Kathryn Chaloner and Isabella Verdinelli. Bayesian Experimental Design: A Review. Statist. Sci. , 10(3):273-304, 08 1995.

Friedrich Pukelsheim. Optimal Design of Experiments (Classics in Applied Mathematics) (Classics in Applied Mathematics, 50) . Society for Industrial and Applied Mathematics, Philadelphia, PA, USA, 2006. ISBN 0898716047.

Stephen Boyd and Lieven Vandenberghe. Convex optimization . Cambridge university press, 2004.

Max A Woodbury. Inverting modified matrices. In Memorandum Rept. 42, Statistical Research Group , page 4. Princeton Univ., 1950.

José Miguel Hernández-Lobato, Matthew W Hoffman, and Zoubin Ghahramani. Predictive entropy search for efficient global optimization of black-box functions. Advances in neural information processing systems , 27, 2014.

Ben Tu, Axel Gandy, Nikolas Kantas, and Behrang Shafei. Joint entropy search for multiobjective Bayesian optimization. Advances in Neural Information Processing Systems , 35: 9922-9938, 2022.

Carl Hvarfner, Frank Hutter, and Luigi Nardi. Joint entropy search for maximally-informed Bayesian optimization. Advances in Neural Information Processing Systems , 35:11494-11506, 2022.

Zi Wang and Stefanie Jegelka. Max-value entropy search for efficient Bayesian optimization. In International Conference on Machine Learning , pages 3627-3635. PMLR, 2017.

A Visual abstract of the algorithm

In Figure 5 we summarize how our algorithm creates non-Markovian policies for maximizer identification and the corresponding connections to other works in the literature.

Transition Constrained BO Problem Formulation

Objective Formulation

(a) Need to plan ahead

Look-ahead Utilities and

averaging out uncertainty

(intractable)

Hypothesis Testing

(b) Variance reduction

of a trajectory

Unconstrained Linear Bandits

(Soare et al. [46]) (frequentist)

Adaptive Objective with optimality

guarantees for Linear Bandits (Fiez

et al. [30]) (frequentist)

(c) Acquisition function

Acquisition function

optimization

Direct optimization in

the space of trajectories

(intractable)

(d) Relaxation to space of

state-action distributions

Algorithm

Mutný et al. [17],

Convergence to optimal

Markovian policy (Mutný

et al. [17])

e.g.

Dynamic Pro-

gramming

(f) Iterative Reinforcement

Learning sub-problem

Adaptive resampling

at every iteration

(g) Non-Markovian Policies

1.0

Greedy-UCB

BestA

MDP-EI

MDP-B0

0.8

t Arm Identified (%)

Log Regret

0.6

0.4

0.2

-12

0.0

100

200

300

400

500

Iteration

1.0

Greedy-UCB

MDP-EI

Average Log Regret

MDP-BO

0.8

Best Arm Identified

0.6

0.2

-12

0.0

10

2030

4050607080

90

100

Iteration

B Additional Empirical Results

B.1 Constrained Ypacarai

We also run the Ypacarai experiment with immediate feedback. To increase the difficulty, we used large observation noise, σ 2 = 0 . 01 . The results can be seen in Figure 6. The early performance of MDP-EI is much stronger, however, it gets overtaken by our algorithm from episode three onwards, and gives the worst result at the end, as it struggles to identify which of the two optima is the global one.

B.2 Knorr pyrazole synthesis

We also include results for the Knorr pyrazole synthesis with immediate feedback. In this case we observe very strong early performance from MDP-BO, but by the end MDP-EI is comparable. The greedy method performs very poorly.

B.3 Additional synthetic benchmarks

Finally, we also include additional results on more synthetic benchmarks for both synchronous and asynchronous feedback. The results are shown in Figures 8 and 9. The results back the conclusions in the main body. All benchmarks do well in 2-dimensions while highlighting further that MDPBO-UCB and LSR can be much stronger in the synchronous setting than Thompson Sampling planning-based approaches (with the one exception of the Levy function).

Table 1: Average acquisition function solving times for each practical benchmark. We give the solving times to the nearest second, and provide the size of the state-space, |S| , the maximum number of actions one can take from a specific state, |A ( S ) | , and the planning horizon. In all benchmarks we are able to solve the problem in a few seconds.

(a) Branin 2D. (asynch)

(a) Branin 2D.

(b) Michalewicz2D.

(c) Levy4D.

3

Median Regret

MDP-BO-TS

Median Regret

1.5

MDP-BO-TS

Median Regret

3

MDP-BO-TS

2

MDP-BO-UCB

MDP-BO-UCB

MDP-BO-UCB

TrSnAKe

1.0

TrSnAKe

2

TrSnAKe

1

0.5

1

0

0.0

20

30

40

50

60

70

80

90

100

20

30

40

50

60

70

80

90

100

20

30

40

50

60

70

80

90

100

Iteration

Iteration

Iteration

0.5

0.20

0.10

Median Regret

0.4

0.3

0.2

LSR

MDP-BO-TS

MDP-BO-UCB

TrSnAKe

Median Regret

0.15

0.10

LSR

TrSnAKe

MDP-BO-TS

MDP-BO-UCB

Regret

Median

0.08

0.06

0.04

TrSnAKe

LSR

MDP-BO-TS

MDP-BO-UCB

0.1

0.05

0.02

0.0

0.00

0.00

0

10

20

30

40

50

0

10

20

30

40

50

0

20

40

60

80

100

Iteration

Iteration

Iteration

B.4 Computational study

We include the average acquisition function solving time for each of the discrete problems. For the continuous case the running time was comparable to Truncated SnAKe [42] since most of the computational load was to create the set of maximizers using Thompson Sampling. The times were obtained in a simple 2015 MacBook Pro 2.5 GHz Quad-Core Intel Core i7. The bulk of the experiments was ran in parallel on a High Performance Computing cluser, equipped with AMD EPYC 7742 processors and 16GB of RAM.

B.5 Median plots for Ypacarai and reactor experiments

In Figures 10 and 11 we give the median and quantile plots for the Knorr pyrazole synthesis and the Ypacarai experiment, which were not included in the main paper to avoid cluttering the graphics.

(a) Episodic feedback results

(b) Immediate feedback results

Greedy-UCB

Greedy-UCB

0.07

MDP-EI

0.05

MDP-EI

0.06

MDP-BO

MDP-BO

0.04

0.05

t

ret

0.04

0.03

R

0.03

R

0.02

0.02

0.01

0.01

0.00

0.00

1

2

3

4

5

6

7

8

9

10

10

20

30

40

50

60

70

80

90

100

Episode

Iteration

C Utility function: Additional Info

We describe the utility function in complete detail using the kernelized variant that allows to extend the utility beyond the low-rank assumption in the main text.

C.1 Derivation of the Bayesian utility

Suppose that our decision rule is to report the best guess of the maximizer after the T steps as,

(a) Episodic feedback results

(b) Immediate feedback results

0.12

0.16

Greedy-UCB

Greedy-UCB

0.10

MDP-EI

0.14

MDP-EI

MDP-BO

MDP-BO

0.12

0.08

2

t

0.10

6

0.06

00.08

e

R

R

0.06

0.04

0.04

0.02

0.02

0.00

0.00

1

2

3

4

5

100

200

300

400

500

Episode

Iteration

We call this the selection the recommendation rule . We focus on this recommendation rule as this rule is interpretable to the facilitator of the analysis and experimenters. In this derivation we use that f = θ ⊤ Φ( x ) . More commonly, the notation ⟨ θ, Φ( x ) ⟩ is used, where the inner product is potentially infinite dimensional. We use use ⊤ notation for simplicity for both cases. Same is true for any other functional estimates, e.g., for the posterior mean estimate, we use µ t ( x ) = Φ( x ) ⊤ µ t . The inner product is in the reproducing kernel Hilbert space associated with the kernel k .

̸

Now, suppose there is a given f (we will take expectation over it later), then there is an x ∈ X achieving optimum value, denoted x ⋆ f (suppose unique for this development here). Hence, we would like to model the risk associated with predicting a fixed z = x ⋆ f , which is still in Z at time T . Suppose we are at time t , we develop the utility to gather additional data X new on top of the already acquired data X t . These should improve the discrepancy of the true answer, and the reported value the most.

Suppose there are two elements in Z simple = { z, x ⋆ f } . We will generalize to a composite hypothesis later. In two-element case, the probability of the error in incurred due to selecting z is:

. The randomness here is due to the observations y = f ( X new ) + ϵ that are used to fit the estimator µ T ( x ) . Namely due to ϵ ∼ N (0 , σ 2 ) . Given f (equivalently θ ), the distribution of our estimator (namely the posterior mean) is Gaussian. Hence, given f :

where V = ∑ T i =1 1 σ 2 Φ( x i )Φ( x i ) ⊤ is an operator on the reproducing kernel Hilbert space due to k as H → H , and I H the identity operator on the same space.

We can now bound the probability of making an error using a Gaussian tail bound inequality:

with the caveat that the inequality only holds when the a z is negative. However note that a z → f ( z ) -f ( x ∗ ) < 0 as T →∞ therefore it will hold once T is large enough. From this we can take logarithms and then the expectation across the randomness in the GP:

which is called the log Bayes' factor and is expected log failure rate for the set of potential maximizers Z simple . The expectation is over the posterior including the evaluations X t (or prior at the very beginning of the procedure). In fact, we can think of the posterior as being the new prior for the future

at any time point. Now assuming that Z has more than one additional element, we want to ensure the failure rate is small for all other failure modes, all other hypothesis. Technically this means, we have an alternate hypothesis, which is composite - composed of multiple point hypotheses. We take the worst-case perspective as its common with composite hypotheses. In expectation over the prior, we want to minimize:

For moderate to large T ≫ 0 , we can upper bound this objective via elegant argument to yield a very transparent objective:

where we have used an lower and upper bound on the a z and b z , respectively as follows:

In the last line we have used the same identity as in Eq. (25). We will explain how to eliminate the expectation in Section C.2

C.2 Upper-bounding the objective: Eliminating E f for large T .

The objective Eq. (15) is intractable due to the expectation of the prior and which involves expectation over the maximum f ( x ⋆ f ) , which is known to be very difficult to estimate. Interestingly, the denominator is independent of f if we adopt the worst-case perspective over the x ⋆ f , and hence the only dependence is through the set Z as well as the denominator. Given all current prior information, we can determine Z , and hence split the expectation. Let us now express

At any time point, we can upper-bound the denominator by the minimum as done by Fiez et al. [30]. Even if Z decreases, as we get more information, the worst-case bound is always proportional to the smallest gap gap( f ) between two arms in X . Hence, we can upper bound the objective as:

As the constant in front of the objective does not influence the optimization problem, we do not need to consider it when defining the utility. Furthermore, in order to minimise the probability of an error we can just minimise the variance in the denominator instead (since arg min x -g ( x ) is equivalent to arg min x 1 g ( x ) when g ( x ) > 0 ). However, the non-trivial distribution of f ( x ⋆ ) [29] renders the utility intractable; therefore we employ a simple and tractable upper bound on the objective by minimising the uncertainty among all pairs in Z :

̸

Surprisingly, this objective coincides with the objective from Fiez et al. [30] which has been derived as lower bound to the best-arm identification problem (maximum identification) with linear bandits. Their perspective is however slightly different as they try to minimize T for a fixed δ failure rate. Perhaps it should not be surprising that the dual variant, consider here, for fixed T and trying to minimize the failure rate leads to the same decision for large T when log( b z ) can be neglected.

C.3 Approximation of Gaussian Processes

Let us now briefly summarize the Nyström approximation [31, 69]. Given a kernel k ( · , · ) , and a data-set X , we can choose a sub-sample of the data ˆ x 1 , ..., ˆ x m . Using this sample, we can create a low r -rank approximation of the full kernel matrix

where K b = [ k ( x i , ˆ x j )] N × m , ˆ K = [ k (ˆ x i , ˆ x j )] m × m and K † denotes the pseudo-inverse operation. We can then define the Nyström features as:

where ˆ D r is the diagonal matrix of non-zero eigenvalues of ˆ K r and ˆ V r the corresponding matrix of eigenvectors. It follows that we obtain a finite-dimensional estimate of the GP:

where Φ( x ) = ( ϕ 1 ( x ) , . . . ϕ m ( x )) T , and θ are weights with a Gaussian prior.

C.4 Theory: convergence to the optimal policy

The fact that our objective is derived using Bayesian decision theory makes it well-rooted in theory. In addition to the derivation of Section C.1, we can prove that our scheme is able to converge in terms of the utility.

Notice that the set of potential maximizers is changing over time, and hence we add a time subscript to Z as Z t . Let us contemplate for a second what could the optimal policy. As the set of Z t is changing, we follow the line of work of started by Russo [70] and introduce an optimal algorithm that knows the true x ⋆ f for each possible realization of the prior f . In other words, its an algorithm that any time t , would follow:

where in the above ˆ d t ⊕ d represents the weighted sum as in the main Algorithm 1 that scales the distributions properly according to t and T , so to make the sum of them a valid distribution. Notice that in contrast to our objective, it does not take the maximum over z ′ ∈ Z , but fixes it to the value x ⋆ f that the hypothetical algorithm has privileged access to. To eliminate the cumbersome notation, we will refer to the objectives as U ( d |Z t , Z t ) as the objective used by our algorithm (real execution) and U ( d |Z t , { x ⋆ f } ) , as the objective that the privileged algorithm is optimizing which serves as theoretical baseline.

The visitation of d ⋆ t represents the best possible investment of the resources (of the size T -t ) to execute at time t had we known the x ⋆ f instead of only Z t . This is interpreted as if the modeler knows x ⋆ f , and sets up an optimal curriculum that is being shown to an observer in order to convince him/her of that x ⋆ f is the optimal value. He or she is using statistical testing to elucidate it from execution of the policy. Like the algorithm, the optimal policy changes along the optimization procedure due to changes in Z t . Hence, our goal is to show that we are closely tracking the performance of these optimal policies in time t , and eventually there is little difference between our sequence of executed policies (visitations) ˆ d t and the algorithm optimal d ⋆ t .

In order to prove the theorem formally, we need to assume that Z t is decreasing. The rate at which this set is decreasing determines the performance of the algorithm to a large extent. Namely, we assume that given two points in time, having the same empirical information ˆ d t . Given, f , suppose

As we gather information in our procedure the, { x ⋆ f } ⊂ Z t ⊆ Z t -1 , but the exact decrease depends on how Z t is constructed. We leave the particular choice for C t to make the above hold for future work. We conjecture that this is decreasing as C t ≈ γ t √ t , where γ t is the information gain due to Srinivas et al. [71]. We are now ready to state the formal theorem along with its assumptions.

Proposition C.1. Assuming episodic feedback, and suppose that for any Z ,

U is convex on D

B -locally Lipschitz continuous under || · || ∞ norm

locally smooth with constant L , i.e,

condition in ( ⋆ ) holds with Bayesian posterior inference,

we can show that the Algorithm 1 satisfied for the sequences of iterates { ˆ d t } T t =1 :

with probability 1 -δ on the sampling from the Markov chain. The randomness on the confidence set is captured by Assumption in Eq. ( ⋆ ) .

The previous proposition shows that as the budget of the experimental campaign T is increasing, we are increasingly converging to the optimal allocation of the experimental resources on average also on the objective that is unknown to us. In other words, our algorithm is becoming approximately optimal also under the privileged information setting representing the best possible algorithm. Despite having a limited understanding of potential maximizers at the beginning by following our procedure, we show that we are competitive to the best possible allocation of the resources. Now, we prove the Proposition. The proof is an extension of the Theorem 3 in [17]. Whether the objective satisfied the above conditions depends on the set X . Should the objective not satisfy smoothness, it can be easily extended by using the Nesterov smoothing technique as explained in the same priorly cited work.

Proof of Proposition C.1. The proof is based on the proof of Frank-Wolfe convergence that appears Appendix B.4 in Thm. 3. in Mutný et al. [17].

Let us start by notation. We will use the notation that U t is the privileged objective U ( d |Z t , { x f t } ) , while the original objective will be specified as U ( d |Z t , Z t ) .

First, what we follow in the algorithm:

The executed visitation is simply generated via sampling a trajectory from q t . Let us denote the empirical visiation of the trajectory as δ t ,

For the analysis, we also need the best greedy step for the unknown (privileged) objective U as

Let us start by considering the one step update:

We will now carefully insert and subtract two set of terms depending on the real objective so that we can bound them using ( ⋆ ):

Carrying on,

Now multiplying by t +1 both sides, and summing on 1 T -1 ∑ T -1 t =1 . Using the shorthand ρ t ( ˆ d t ) = U t ( ˆ d t ) -U t ( d ⋆ t ) we get:

First notice that 1 T -1 ∑ T -1 t =1 ϵ t ≤ B √ T log(1 /δ ) by Lemma in Mutný et al. [17] due to ϵ t being martingale difference sequence. The other term is the sum on 1 T -1 ∑ t =1 C t which appears in the main result. The sum on ∑ T -1 t =1 1 1+ t ≤ L log T T . The rest is eliminated by the reccurence of the terms, and using that U ( d |Z t , { x ⋆ f } ≤ U ( d |Z t -1 , { x ⋆ f } ) for any d . This is due to set Z t decreasing over time. We report the result in asymptotic notation as function of T and log(1 /δ ) .

D Objective reformulation and linearization

For the main objective we try to optimize over a subset of T trajectories X = { τ i ∈ X H } T i =1 . Let X H be the set of sequences of inputs τ = ( x 1 , ..., x H ) where they consist of states in the search space X . Furthermore, assume there exists, in the deterministic environment, a constraint such that x h +1 ∈ C ( x h ) for all h = 1 , ..., H -1 . Then we seek to find the set X ∗ , consisting of T trajectories (possibly repeated), such that we solve the constrained optimization problem:

We define the objective as:

Our goal is to show that optimization over sequences can be simplified to state-action visitations as in Mutný et al. [17]. For this, we require that the objective depends additively involving terms x, a separately. We formalize this in the next result. In order to prove the result, we utilize the theory of reproducing kernel Hilbert spaces [72].

Lemma D.1 (Additivity of Best-arm Objective) . Let X be a collection of t trajectories of length H . Assuming that f ∼ GP (0 , k ) . Assuming that k has Mercer decomposition as k ( x, y ) = ∑ k λ k ϕ k ( x ) ϕ k ( y ) .

Let d X be the visitation of the states-action in the trajectories in X , as d X = 1 TH ∑ T t =1 ∑ x,a ∈ τ t δ x,a , where the δ x,a represent delta function supported on x, a . Then optimization of the objective Eq. (23) can be rewritten as:

where V ( d ) = ∑ i ∑ x,a ∈ τ i d ( x, a )Φ( x )Φ( x ) ⊤ + I σ 2 / ( TH ) is a operator V ( d ) : H k →H k , the norm is RKHS norm, and Φ( z ) k = ϕ k ( z ) .

Proof. Notice that the posterior GP of any two points z, z ′ is ( f ( z ) , f ( z ′ )) = N (( µ ( z ) , µ ( z ′ )) , K z,z ′ ) , where K z,z ′ is posterior kernel (consult Rasmussen and Williams [28] for details) defined via a posterior kernel k X ( z, z ′ ) = k ( z, z ′ ) -k ( z, X )( K ( X , X ) + σ 2 I ) -1 k ( X , z ′ ) .

Utilizing k ( z, z ′ ) = Φ( z ) ⊤ Φ( z ) (RKHS inner product) with the Mercer decomposition we know that k t ( z ) = Φ( X ) ϕ ( z ) . Applying the matrix inversion lemma, the above can be written as using V = ∑ T t =1 ∑ x ∈ τ t Φ( x )Φ( x ) ⊤ + σ 2 I H .

Leading finally to:

Let us calculate Var [ f ( z ) -f ( z ′ ) | X ] . The variance does not depend on the mean. Hence,

to arrive at:

The symbol # counts the number of occurrences. Notice that we have been able to show that the objective decomposes over state-action visitations as d X decomposes over their visitations

Note that the objective equivalence does not imply that optimization problem in Eq. (23) is equivalent to finding,

In other words, optimization over trajectories and optimization over d π ∈ D is not equivalent. The latter is merely a continuous relaxation of discrete optimization problems to the space of Markov policies. It is in line with the classical relaxation approach addressed in experiment design literature with a rich history, e.g., Chaloner and Verdinelli [73]. For introductory texts on the topic, consider Pukelsheim [74] for the statistical perspective and Boyd and Vandenberghe [75] for the optimization perspective. However, as Mutný et al. [17] points out, reducing the relaxed objective does reduce the objective as Eq. (23) as well. In other words, by optimizing the relaxation with a larger budget of trajectories or horizons, we are able to decrease Eq. (23) as well.

For completeness, we state the auxiliary lemma. We make use of the Sherman-Morrison-Woodbury (SMW) formula, [76]:

Here we do the opposite, and invert an n × n matrix instead of a q × q one.

to show the following.

Note that instead of inverting n × n matrix, we can invert a q × q matrix.

Proof.

D.1 Objective formulation for general kernel methods

The previous discussion also allows us to write the objective in terms of the general kernel matrix instead of relying on finite dimensional embeddings. The modification is very similar and relies again on Sherman-Mirrison-Woodbury lemma.

We now work backwards from (26), and first write the objective in terms of features of arbitrarily large size. Using the shorthand, ˜ σ 2 = σ 2 /TH , let us define a diagonal matrix that describes the state-action distribution D = diag ( { d x : x ∈ X} ) of the size |X|×|X| , and Φ( X ) which corresponds to the unique (possibly infinite-dimensional) embeddings of each element in X , ordered in the same way as D .

If we then pre-multiply by Φ( z ) ⊤ and Φ( z ′ )

which allows us to calculate the objective for general kernel methods at the cost of an |X| × |X| inversion. Upon identifying the z, z ′ that maximize the above, we can use them in an optimization procedure. This holds irrespective of whether the state space is discrete or continuous. In continuous settings however, we again require a parametrization of the infinite dimensional probability distribution by some finite means such as claiming that D θ contains some finite dimensional simplicity. This is what we do with the linear system example in Section 4.2.

D.2 Linearizing the objective

To apply our method, we find ourselves having to frequently solve RL sub-problems where we try to maximize ∑ x,a d ( x, a ) ∇ F ( x, a ) . To approximately solve this problem in higher dimensions, it becomes very important to understand what the linearized functional looks like.

Remark D.4 . Assume the same black-box model as in Lemma D.1, and further assume that we have a mixture of policies π mix with density d π mix , such that there exists a set X mix satisfying d π mix = 1 N ∑ x ∈ X mix δ x for some integer N . Then:

Proof. To show this, we begin by defining:

In the definition above we dropped the constant pre-factors since they do not influence the maximizer of the gradient as they are related by a constant multiplicative factor.

It then follows, by applying Danskin's Theorem that:

E Implementation details and Ablation study

In this section we provide implementation details, and show some studies into the effects of specific hyper-parameters. We note that the implementation code will be made public after public review.

E.1 Approximating the set of maximizers using Batch BayesOpt

We give details of the two methods used for approximating the set of potential maximizers. In particular, we first focus on Thompson Sampling [58]:

where K is a new hyper-parameter influencing the accuracy of the approximation of Z . We found that the algorithm could be too exploratory in certain scenarios. Therefore, we also propose an alternative that encourages exploitation by guiding the maximization set using BayesOpt through the UCB acquisition function [71]:

where B = linspace(0, 2.5, K) which serves as scaling for the size of set Z ( UCB ) cont . Both cases reduce optimization over Z to enumeration as with discrete cases.

E.2 Benchmark Details

For all benchmarks, aside from the knorr pyrazole synthesis example, we use a standard squared exponential kernel for the surrogate Gaussian Process:

where σ 2 rbf is the prior variance of the kernel, and ℓ rbf the kernel. We fix the values of all the hyperparameters a priori and use the same for all algorithms. The hyper-parameters for each benchmarks are included in Table 2.

For the knorr pyrazole synthesis example, we further set α ode = 0 . 6 , α rbf = 0 . 001 , k 1 = 10 , k 2 = 874 , k 3 = 19200 , α sig = 5 . Recall we are using a finite dimensional estimate of a GP such that:

in this case we set a prior to the ODE weight such that ω ode ∼ N (0 . 6 , 0 . 0225) . This is incorporating two key pieces of prior knowledge that (a) the product concentration should be positive, and (b) we expect a maximum product concentration between 0.15 and 0.45.

The number of features for each experiment, M , is set to be M = |X| in the discrete cases and M = min ( 2 5+ d , 512 ) where d is the problem dimensionality.

In the case of Local Search Region BayesOpt (LSR) [22] we set the exploration hyper-parameter to be γ = 0 . 01 in all benchmarks.

E.3 Ypacarai Lake

Samaniego et al. [20] investigated the use of Bayesian Optimization for monitoring the lake quality in Lake Ypacarai in Paraguay. We extend the benchmark to include additional transition constraints, as well as initial and end-point constraints. These are all shown in Figure 12.

E.4 Free-electron Laser

We use the simulator from Mutný et al. [67] that optimizes quadrupole magnet orientations for our experiment with varying noise levels. We use a 2-dimensional variant of the simulator. We discretize the system on 10 × 10 grid and assume that the planning horizon H = 100 . The simulator itself is a GP fit with γ = 0 . 4 , hence we use this value. Then we make a choice that the noise variance is proportional to the change made as σ 2 ( x, a ) = s (1 + w || x -a || 2 ) , where s = 0 . 01 and w = 20 . Note that x ∈ [ -0 . 5 , 0 . 5] 2 in this modeling setup. This means that local steps are indeed very desired. We showcase the difference to classical BayesOpt, which uses the worst-case variance σ = sup x,a s (1 + w || x -a || 2 ) for modeling as it does not take into account the state in which the system is. We see that the absence of state modeling leads to a dramatic decrease in performance as indicated by much higher inference regret in Figure 3b.

initialstate

global optima

local optima

chosen trajectory

E.5 Ablation Study

E.5.1 Number of mixture components

We investigate the effect number of components used in the mixture policy when optimizing the Frank-Wolfe algorithm. We tested on the four real-world problem using N = 1 , 10 and 25 . In the Ypacari example (see Figure 14) we see very little difference in the results, while in the Knorr pyrazole synthesis (see Figure 13) we observe a much bigger difference. A single component gives a much stronger performance than multiple ones - we conjecture this is because the optimum is on the edge of the search space, and adding more components makes the policy stochastic and less likely to reach the boarder (given episodes are of length ten and ten right-steps are required to reach the boarder).

Overall, it seems the performance of a single component is better or at worst comparable as using multiple components. This is most likely due to the fact that we only follow the Markovian policies for a single time-step before recalculating, making the overall impact of mixture policies smaller. Based on this, we only present the single-component variant in the main paper.

(a) Episodic feedback results

(b) Immediate feedback results

1.0

0

1.0

0.037

MDP-BO (1)

MDP-BO (1)

MDP-BO (10)

MDP-BO (25)

Best Arm Identified (%)

Average Log Regret

MDP-BO (10)

MDP-BO (25)

0.8

Best Arm I

Average Regret

0.6

Identified

0.5

0.4

1(%)

0.2

0.000

0.0

-12

0.0

1

2

3

4

5

6

7

89

10

102030

405060708090100

Episode

Iteration

E.5.2 Size of batch for approximating the set of maximizers

We explore the effect of the number of maximizers, K , in the maximization sets Z ( TS ) cont and Z ( UCB ) cont . Overall we found the performance of the algorithm to be fairly robust to the size of the set in all benchmarks, with a higher K generally leading to a little less spread in the performance.

1.0

1.0

MDP-BO (1)

Best Arm Identified (%)

MDP-BO (1)

Best

Average Regret

0.033

MDP-BO (25)

MDP-BO (10)

MDP-BO (10)

MDP-BO (25)

0.8

t Arm Identified (%)

Log Regret

0.6

0.5

0.4

0.2

0.000

0.0

-12

0.0

1

2

3

5

100

200

300

400

500

Episode

Iteration

(a) Episodic feedback results

(b) Immediate feedback results

3.5

MDP-BO-TS (25)

3.0

MDP-BO-TS (50)

MDP-BO-TS (100)

2.5

egret

Re

2.0

Median

1.5

1.0

0.5

0.0

20

30

40

50

60

70

80

90

100

Iteration

(a) Hartmann6D (asynch).

(b) Hartmann3D.

(c) Levy4D.

3.0

0.10

3.0

MDP-BO-UCB (100)

MDP-BO-UCB (10)

MDP-BO-UCB (25)

2.5

MDP-BO-UCB (100)

MDP-BO-UCB (10)

MDP-BO-UCB (25)

0.08

MDP-BO-UCB (25)

MDP-BO-UCB (100)

MDP-BO-UCB (10)

Regret

2.5

Regret

2.0

Median Regret

0.06

Median

2.0

Median

1.5

0.04

1.5

1.0

1.0

0.02

0.5

0.5

0.0L

0.00

20

30

40

50

60

70

80

90

100

0

20

40

60

80

100

0

20

40

60

80

100

Iteration

Iteration

Iteration

F XY -allocation vs G -allocation

Our objective is motivated by hypothesis testing between different arms (options) z and z ′ . In particular,

One could maximize the information of the location of the optimum, as it has a Bayesian interpretation. This is at odds in frequentist setting, where such interpretation does not exists. Optimization of information about the maximum has been explored before, in particular via information-theoretic acquisition functions [29, 77-79]. However, good results (in terms of regret) have been achieved by focusing only on yet another surrogate to this, namely, the value of the maximum [80]. This is chiefly due to problem of dealing with the distribution of f ( x ⋆ ) . Defining a posterior value for f ( z ) is easy.

0.5

XY-allocation

G-allocation

0.4

Median Regret

0.3

0.2

0.1

0.0

0

20

40

60

80

100

Iteration

Using, this and the worst-case perspective, an alternative way to approximate the best-arm objective, could be:

What are we losing by not considering the differences? The original objective corresponds to the XY -allocation in the bandits literature. The modified objective will, in turn, correspond to the G -allocation, which has been argued can perform arbitrarily worse as it does not consider the differences, e.g. see Appendix A in Soare et al. [46]. We nonetheless implemented the algorithm with objective (33), and found the results to be as expected: performance was very similar in general , however in some cases not considering the differences led to much poorer performance. As an example, see Figure 17 for results on the synchronous Branin2D benchmark.

G Practical Planning for Continuous MDPs

From remark D.4 it becomes clear that for decreasing covariance functions, such as the squared exponential, ∇ F will consist of two modes around z ∗ and z ′ ∗ The sub-problem seems to find a sequence that maximizes the sum of gradients, therefore the optimal solution will try to reach one of the two modes as quickly as possible. For shorter time horizons, the path will reach whichever mode is closest, and for large enough horizons, the sum will be maximized by reaching the larger of the two modes.

Therefore we can approximately solve the problem by checking the value of the sub-problem objective in (13) for the shortest paths from x t -1 → z ∗ and x t -1 → z ′ ∗ , which are trivial to find under the constraints in (13). Note that the paths might not necessarily be optimal, as they may be improved by small perturbations, e.g., there might be a small deviation that allows us to visit the smaller mode on the way to the larger mode increasing the overall value of the sum of gradients, however, they give us a good and quick approximation.

H Kernel for ODE Knorr pyrazole synthesis

The kernel is based on the following ODE model, which is well known in the chemistry literature and given in [27].

and then:

Giving:

Our main goal is to optimize the product concentration of the reaction, which is given by y 1 . We do this by sequentially querying the reaction, where we select the residence time, and the initial conditions of the ODE, in the form y 0 = [0 , A, B, 0 , 0] , where A = 1 -B .

Due to the non-linearity in Eq. (34) we are unable to fit a GP to the process directly. Instead, we first linearize the ODE around two equilibrium points. The set of points of equilibrium are given by:

And the Jacobian of the system is:

Unfortunately, since the matrices are singular, we do not get theoretical results on the quality of the linearization. However, linearization is still possible, with the linear systems given by:

We focus on the first system for now. The matrix has the following eigenvalues:

Note that the three eigenvalues give us the corresponding solution based on their (linearly separable) eigenvectors:

where p i are constants. The behaviour of the ODE when this is not the case will depend on whether the remaining eigenvalues will be real or not. However, note:

and therefore all eigenvalues will always be real. Therefore we can write down the solution as:

where we ignore the case of repeated eigenvalues for simplicity (this is the case where b 2 1 k 2 1 + 2 b 1 c 1 k 1 k 2 + c 2 1 k 2 2 -2 k 3 ( b 1 k 1 -c 1 k 2 ) + k 2 3 is exactly equal to zero). We further note that the eigenvalues will be non-negative as:

therefore:

which means the solutions will always be a linear combination of exponentially decaying functions of time plus constants.

The eigenvectors have the closed form:

We are optimizing over initial set of conditions y 0 = [0 , A, B, 0 , 0] , so solving for the specific values of the constants gives:

Finally, since we are setting A = 1 -B and we are only optimizing the first component of ⃗ y we can obtain it in closed form:

0

2

4

6

8

10

Time

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

Yield

numerical

exact

The second ODE is very similar to the first, recall it depends has the following matrix:

The resulting ODE is symmetric to the alternate linearization giving the same solution:

with the only difference being the eigenvectors now are:

which in turn leads to solutions of the form:

where A = 1 -B . Note that we now have four different eigenvalues, which depend on the linearization points:

Giving solutions:

Due to the length of the derivation, we confirm that our analysis is correct by comparing the numerical solution of the ODE to the exact solution we found in Figure 18. Finally, we look at interpolating between the two solutions; so given the solutions y (1) ( t, B ) and y (2) ( t, B ) corresponding to the linearization with stationary point in S 1 and S 2 respectively, we consider a solution of the form:

where S ( x ) := (1 + e -α sig ( x -0 . 5) ) -1 is a sigmoid function centered at B = 0 . 5 and where we have introduced a new hyper-parameter α sig . Finally, given Eq. (37) we can obtain the kernel. In particular, we want (37) to be a feature we are predicting on; therefore the kernel is simply the (dot) product of the features therefore:

And because we know we are simply approximating the data we can simply correct the model by adding an Gaussian Process correction; giving us the final kernel:

where α ode and α rbf are parameters we can learn, e.g. using the marginal likelihood.

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: All claims in the abstract are backed up by and developed in the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss limitations in the conclusion: we are restricted to a specific parameterization in the continuous state-space case.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All proofs are in the Appendix, where we provide full assumptions and proofs, or cite the relevant results. Specifically see Appendix C - D.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We include all the method and computational details in G.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code will be made public upon acceptance, and an anonymized version is included for the review process.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We include all the method and computational details in G.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We provide regret quantiles in our experiments, reproduce on 25 random seeds and use a variety of different benchmarks.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: See section B.4.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Upon reading the guidelines, the paper seems to conform to every point in the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There are no broader impacts to discuss, outside of those already established by research into the area of design of experiments.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: There are no risks for misuse of our work.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All exisiting assests are cited and the corresponding licenses respected.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The only new asset will be the code implementation which will be published upon acceptance.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No crowd-sourcing or human subjected were used.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: No human subjects were used in this research.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores extending Bayesian optimization to incorporate transition constraints through state dynamics. The approach treats the problem as an optimal control problem. For planning purposes, known state dynamics model the transition constraints, while the unknown objective is represented using a Gaussian process. To address this, the paper introduces a tractable acquisition function—an upper bound on the maximum probability of selecting the wrong optimizer. This acquisition function serves as a cost metric in a planning scenario, which is solved using the Frank-Wolfe algorithm. The paper’s effectiveness is evaluated across various benchmark tasks.\n\nSTRENGTHS:\nThe paper presents a framework for optimizing challenging-to-evaluate black-box functions with transition constraints, which could have practical applications. Its overall structure is well-organized.\n\nThe proposed solution strategy for solving the planning problem using the Frank-Wolfe algorithm is novel and represents a valuable contribution to the research community. Additionally, considering both discrete and continuous state problems is commendable.\n\nWhile the experiments are fair, including a theoretical synthetic example would have further strengthened the paper.\n\nWEAKNESSES:\nGeneral Presentation: \n\nGenerally, I believe the paper’s presentation could be enhanced. Specifically, I am not very satisfied with how the generative model is derived and presented. In my opinion, the problem could be expressed as a stochastic optimal control with an unknown cost function. However, the paper takes a different approach, starting from Bayesian optimization, which typically does not incorporate control or dynamics. This choice leads to confusion in notation and extends the Bayesian optimization framework. For instance, the notation between sections 2.1 and 2.2 changes due to the introduction of the action variable a. Additionally, in Section 2.1, the authors discuss a set of arms, drawing from the bandit literature.\nWhat also confuses me is that the actions in the model are never formally introduced. Especially in Section 2.3, a formal definition of actions would have been helpful.\n\nBackground Section: \n\nIn the background section, it is not very obvious what the paper’s contribution is versus what constitutes background information. The citations are not very transparent.\n\nRelated Work (Section 2.4): \n\nI believe one of the most important related concepts, Bayesian Reinforcement Learning or Dual Control, is not discussed at all.\n\nAcquisition Function Analysis: \n\nAdditionally, I would have appreciated more analysis on why the specific acquisition function was chosen over others. How would the algorithm perform with a different acquisition function?\n\nQUESTIONS:\n- Is the utility function presented in the paper a contribution? Could the authors elaborate a bit on this?\n- I am a bit confused about the notation in line 132. Could the authors please define $\\tau_i$?\n - In line 136 and 199 is $S$ a switch in notation?\n-\tCould the authors elaborate a bit on line 209. What exactly is sampled using Thompson Sampling?\n-\tIn line 259, why is this specific contstraint of 0.5 chosen?\n-\tI am a bit confused about line 263. I do not think MPC is nescessarly convex. There is a lot of literature on non-linear MPC. Could the authors elaborate on this?\n\nLIMITATIONS:\n-", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new Bayesian optimization problem that finds the optimal policy to optimize a black-box function subject to transition constraints on the query. The method works for both discrete and continuous Markov chains. The paper empirically demonstrates several practical applications in physical systems, electron laser calibration, and chemical reactor optimization.\n\nSTRENGTHS:\nThe problem formulation is new, and the paper effectively motivates it using several practical applications. The solution is based on principled methods, with several experiments supporting the practicality of the problem.\n\nWEAKNESSES:\nThe paper may be improved by clarifying several questions below.\n\nQUESTIONS:\n1. Section 2 is the background but there are no citations in section 2.1. Is this a new solution proposed by the paper, or is it based on existing work?\n\n2. Line 575: Please explain why the proprotionality holds. Monotonicity does not imply proportionality (same ratio).\n\n3. Line 587: Please explain the two inequalities.\n\n4. Would the authors please clarify equation (7)? What does it mean by the expectation of a state $X_t$? What is $\\pi$ in $d_{\\pi}$? Z_t and GP posterior may change after every time step, so I have a hard time understand why U remains the same and the expectation is only taken over X_t in equation (7). Furthermore, as the utility depends on the time step (the GP posterior and Z_t), two visitations to the same state at different time steps have two different utilities. Then, how do we combine all visitations into $d_\\pi$ (lines 143-144).\n\n5. The pseudocode in Algorithm 1 is unclear. What is d_pi (there is no initialization of it)? Why does the update of $\\hat{d}\\_{t,h+1}(x)$ match equation (11)? Why are GP_{t+1} and Z_{t+1} only updated with X_{t,H} and Y_{t,H} (the observations from episode t, not all episodes from 1 to t)? What are X_{t,H} and Y_{t,H} (the time step is only until H-1).\n\n6. The policy depends on the time step, so pi_{t,0} is only deployed at the first time step. How is the state visitation frequency of such a policy estimated well if it is only deployed at the 1st time step of each episode?\n   \n7. Would the authors please clarify equation (12) using the correct subscript of d (d_{t,h})?\n   \n8. Some minor remarks: In equation (11), should j be from 1 to t-1? Line 122: y(xh) should be y(xh,ah).\n\nLIMITATIONS:\nThere are no potential negative societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n&nbsp;\n\nThe authors address the problem of transition-constrained Bayesian optimization, modeling the transition constraints as a Markov decision process. They take a novel approach in deriving their utility function for this problem setting based on maximum identification and bounding the probability of choosing an incorrect maximizer. While the method is well-described and has the potential to yield theoretical insight into the structure of transition-constrained Bayesian optimization, the empirical performance does not seem hugely compelling compared to existing methods such as SnAKe and LSR. As such, my recommendation is borderline with potential to increase my score if the authors can address the main criticism of the advantages of the method compared to competitor approaches.\n\n&nbsp;\n\nSTRENGTHS:\n&nbsp;\n\nThe authors derive a novel transition-constrained Bayesian optimization algorithm, leveraging the literature on maximum identification via hypothesis testing to derive their utility function.\n\n&nbsp;\n\nWEAKNESSES:\n&nbsp;\n\n__MAJOR POINTS__\n\n&nbsp;\n\n1. The main weakness of the work would appear to be the lack of compelling evidence for why the authors' method should be chosen over competitor methods such as LSR or SnAKe on practical problems. I think the paper would benefit substantially if the authors could highlight problem settings in which they expect their algorithm to perform better in relation to at least one practical performance metric.\n\n2. The code would benefit from a README describing how to reproduce the results of the paper.\n\n&nbsp;\n\n__MINOR POINTS__\n\n&nbsp;\n\n1. It would be great if the references appeared in numbered order.\n\n2. There is some missing capitalization in the references e.g. \"bayesian\" in place of \"Bayesian\".\n\n3. Line 90, typo, \"we seek to identify\".\n\n4. Line 566, typo, \"due to selecting\".\n\n5. Line 566/567, missing full stop at the end of the equation.\n\n6. On line 567 epsilon is homoscedastic but in the main paper the authors state that epsilon can be heteroscedastic?\n\n7. Line 584, typo, \"hypotheses\".\n\n8. Line 680, typo, extraneous \"be\".\n\n9. Line 680, in the notation should there be some kind of indication that X corresponds to T trajectories e.g. an indication of the dimensions of the space X corresponds to? The same on line 690.\n\n10. In Equation 4, the conditioning bar is difficult to read, could the authors introduce some more space to the left and right of the bar?\n\n11. In the Subsection on \"Utility with Embeddings\", the authors may wish to specify \"Kernel Embeddings\" so as to disambiguate from embeddings over the input space X.\n\n12. In the proof of statement 25, it might be worth giving the acronym for SMW, namely Sherman-Morrison-Woodbury, somewhere in the text.\n\n13. On line 136, have the authors defined S? This is the state space presumably.\n\n14. In the related work on look-ahead BayesOpt, it may also be worth mentioning [1].\n\n15. Line 179, typo, \"a mix of the previous two\".\n\n16. In Section 2.2, the variable d could be explained in more detail.\n\n17. When referencing Thompson sampling, it would be worth citing the original paper [2].\n\n18. In Section E.2, the expression for the RBF kernel implicitly assumes x is 1-dimensional. The authors could replace the squared distance with the squared 2-norm to generalize the expression.\n\n19. In Algorithm 1, would it be more descriptive to refer to Z_0 as the initial set of maximizer candidates rather than the initial set of maximizers?\n\n20. It would be worth citing [3] (as per the discussion the historical section by Garnett [4]) for the Expected Improvement acquisition function given that it is used.\n\n21. It would be great if the authors could provide the number of random trials over which the error bars were computed in Figure 4.\n\n22. Line 622, \"scales them properly\". What is \"them\"?\n\n23. Line 670, typo, \"using\".\n\n&nbsp;\n\n__REFERENCES__\n\n&nbsp;\n\n[1] Roman Marchant, Fabio Ramos, and Scott Sanner. Sequential Bayesian optimisation for spatial-temporal monitoring. In Proceedings of the International Conference on Uncertainty in Artificial Intelligence, 2014.\n\n[2] W. R. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3-4):285–294, 1933.\n\n[3] VR Saltenis (1971). One Method of Multiextremum Optimization. Avtomatika i Vychislitel’naya Tekhnika (Automatic Control and Computer Sciences) 5(3):33–38.\n\n[4] Garnett, R., 2023. Bayesian optimization. Cambridge University Press.\n\n&nbsp;\n\nQUESTIONS:\n&nbsp;\n\n1. In Equation 3, do the authors have any intuition as to why k(z, x_f*) / (f(z) - f(x_f*))^2 is related to the probability of returning an wrong maximizer? Beyond the derivation presented in Section C.1 of the appendix.\n\n2. In Equation 4, the authors state their objective as a maximization problem. However, in the text, the authors refer to minimizing the uncertainty among all pairs in \\mathcal{Z}. Could the authors explain this?\n\n3. In Section D.1 of the appendix, the authors give the formulation of the objective for general kernel matrices. The authors state that the objective may be computed by inverting a |\\Chi| x |\\Chi| matrix. How restrictive is this for the scale of problems that can be addressed with this formulation of the objective?\n\n4. In Section 5, what was the motivation for the decision to fix the GP hyperparameters?\n\n5. For Figures 2 and 3 why are there no error bars for the average regret curves?\n\n&nbsp;\n\nLIMITATIONS:\n&nbsp;\n\n1. The main limitation as described above would appear to be the lack of compelling evidence for problem settings under which the authors' algorithm outperforms other transition-constrained Bayesian optimization methods such as LSR and SnAKe.\n\n\n&nbsp;", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the best arm identification task in the transition-constrained setting where the MDP captures the transition. To solve the resulting planning problem defined as maximizing the acquisition propogated through the MDP, the paper studies using the RL solvers and heuritics in the discrete and continuous cases.\n\nSTRENGTHS:\n1. Valid scenario, well-motivated and well-illustrated algorithm.\n2. Comprehensive empirical evidence substantiating the effectiveness of the proposed algorithm, along with verification of parameter choices.\n\nWEAKNESSES:\n1. Although employing an RL solver, as admitted by the authors, the objective is non-convex and differs from conventional optimization in RL. This demands heuristics, which are not sufficiently discussed.\n\n2. In the experiments, the GP hyperparameters are fixed, which is questionable due to potential misspecification and manually introduced bias. A typical practice might include either a full Bayesian treatment of the GP hyperparameters or kernel learning. This is particularly relevant for the ODE kernel due to its higher complexity.\n\n3. The results in Figure 4 appear to lack statistical significance. Additionally, there is no clear preference among the variants of MDP-BO, resonating with the challenge in conventional BO.\n\n4. There is a series of works on applying BO to graphical structures. Although the MDP formulation is different, adding them to the discussion would enhance completeness as some design choices are conceptually related.\n\n***References***\n\n1. Kusakawa, Shunya, Shion Takeno, Yu Inatsu, Kentaro Kutsukake, Shogo Iwazaki, Takashi Nakano, Toru Ujihara, Masayuki Karasuyama, and Ichiro Takeuchi. \"Bayesian optimization for cascade-type multistage processes.\" Neural Computation 34, no. 12 (2022): 2408-243.\n2. Aglietti, Virginia, Xiaoyu Lu, Andrei Paleyes, and Javier González. \"Causal Bayesian optimization.\" In International Conference on Artificial Intelligence and Statistics, pp. 3155-3164. PMLR, 2020.\n3. Astudillo, Raul, and Peter Frazier. \"Bayesian optimization of function networks.\" Advances in Neural Information Processing Systems 34 (2021): 14463-14475.\n\nQUESTIONS:\n1. Could the author comment on the practice of limiting the extension of MDP-BO to either TS or UCB?\n2. One minor question: in figure 3(a), is the legend \"MDP-BO\" written as \"MDP-B0\"?\n3. The objective defined in Section 2.1 and the general BO framework presented in Algorithm 1, beyond the MDP-based planning solver, are commonly seen in previous works. Li and Scarlett (2022) studied the batched version of this formulation, Zhang et al. (2023) studied the high-dimensional treatment, and Han et al. (2024) studied the formulation in game theory, among others. There seems to be potential to generalize the proposed MDP-BO framework into broader applications. It would be a bonus point to additionally discuss the corresponding potentials of MDP-BO.\n\n4. Moreover, recent advancements by Salgia et al. (2024) suggest that random exploration could be sufficient on $Z_t$. I encourage the authors to explore this if it is of interest. \n\n***References***\n\n1. Li, Zihan, and Jonathan Scarlett. \"Gaussian process bandit optimization with few batches.\" In International Conference on Artificial Intelligence and Statistics, pp. 92-107. PMLR, 2022.\n2. Zhang, Fengxue, Jialin Song, James C. Bowden, Alexander Ladd, Yisong Yue, Thomas Desautels, and Yuxin Chen. \"Learning regions of interest for Bayesian optimization with adaptive level-set estimation.\" In International Conference on Machine Learning, pp. 41579-41595. PMLR, 2023.\n3. Han, Minbiao, Fengxue Zhang, and Yuxin Chen. \"No-Regret Learning of Nash Equilibrium for Black-Box Games via Gaussian Processes.\" In The 40th Conference on Uncertainty in Artificial Intelligence.\n4. Salgia, Sudeep, Sattar Vakili, and Qing Zhao. \"Random Exploration in Bayesian Optimization: Order-Optimal Regret and Computational Efficiency.\" In Forty-first International Conference on Machine Learning.\n\nLIMITATIONS:\nDiscussed above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}