SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhongxiang Dai\*1, Gregory Kang Ruey Lau\*1,2, Arun Verma1, Yao Shu1, Bryan Kian Hsiang Low1, Patrick Jaillet3
<sup>1</sup>Department of Computer Science, National University of Singapore
$^{2}$ CNRS@CREATE, 1 Create Way, #08-01 Create Tower, Singapore 138602
$^{3}$ Department of Electrical Engineering and Computer Science, MIT
dzx@nus.edu.sg, greglau@comp.nus.edu.sg, arun@comp.nus.edu.sg,
shuyao95@u.nus.edu, lowkh@comp.nus.edu.sg, jaillet@mit.edu
Kernelized bandits, also known as Bayesian optimization (BO), has been a prevalent method for optimizing complicated black-box reward functions. Various BO algorithms have been theoretically shown to enjoy upper bounds on their cumulative regret which are sub-linear in the number  $T$  of iterations, and a regret lower bound of  $\Omega(\sqrt{T})$  has been derived which represents the unavoidable regrets for any classical BO algorithm. Recent works on quantum bandits have shown that with the aid of quantum computing, it is possible to achieve tighter regret upper bounds better than their corresponding classical lower bounds. However, these works are restricted to either multi-armed or linear bandits, and are hence not able to solve sophisticated real-world problems with non-linear reward functions. To this end, we introduce the quantum-Gaussian process-upper confidence bound (Q-GP-UCB) algorithm. To the best of our knowledge, our Q-GP-UCB is the first BO algorithm able to achieve a regret upper bound of  $\mathcal{O}(\mathrm{poly}\log T)$ , which is significantly smaller than its regret lower bound of  $\Omega(\sqrt{T})$  in the classical setting. Moreover, thanks to our novel analysis of the confidence ellipsoid, our Q-GP-UCB with the linear kernel achieves a smaller regret than the quantum linear UCB algorithm from the previous work. We use simulations, as well as an experiment using a real quantum computer, to verify that the theoretical quantum speedup achieved by our Q-GP-UCB is also potentially relevant in practice.
Kernelized bandits [9], also named Bayesian optimization (BO) [10, 11, 16, 19, 32], has been an immensely popular method for various applications involving the optimization of complicated black-box reward functions. For example, BO has been extensively used to optimize the hyperparameters of machine learning (ML) models [12, 14, 22], the parameters of computationally expensive simulators [5], etc. In every iteration  $t = 1,\dots ,T$ , BO chooses an arm输入  $x_{t}$  and then queries the reward function  $f$  for a noisy observation  $y_{t} = f(x_{t}) + \zeta_{t}$  where  $\zeta_t$  is a sub-Gaussian noise. In addition to its impressive practical performance, BO is also equipped with solid theoretical guarantees. A number of commonly adopted BO algorithms have been theoretically shown to enjoy sub-linear upper bounds on their cumulative regret [9, 33], which ensures that they are guaranteed to be able to find the global optimum of the reward function as  $T$  (i.e., the total number of function queries) increases. On the other hand, a lower bound of  $\Omega (\sqrt{T})$  (ignoring additional log factors) on the cumulative regret of BO has been shown [31], which represents the fundamental limit of any BO algorithm (in the classical setting). In other words, no classical BO algorithm can achieve a cumulative regret smaller than  $\Omega (\sqrt{T})$ . This naturally begs the question: can we leverage more advanced technology to go beyond
the classical setting and hence break this fundamental limit of  $\Omega (\sqrt{T})$ ? In this work, we give an affirmative answer by showing that this can be achieved with the aid of quantum computing [34].
Quantum bandits, which incorporates quantum computing into bandit algorithms, has been studied by a number of recent works [8, 38, 40, 41]. Notably, the recent work of [38] has introduced quantum variants of classical algorithms for multi-armed bandits (MAB) and stochastic linear bandits (SLB). In the setting of quantum bandits adopted by [38], every query to the reward function  $f$  at the arm  $x_{t}$  (in the classical setting) is replaced by a chance to access a quantum oracle, which encodes the reward distribution for the arm  $x_{t}$ . For every selected arm  $x_{t}$ , [38] has proposed to adopt the quantum Monte Carlo (QMC) algorithm [4, 28] as a subroutine to obtain an accurate estimation of  $f(x_{t})$  in an efficient way, i.e., using a small number of queries to the quantum oracle (Lemma 1). As a result, [38] has shown that the regrets of the quantum algorithms for both MAB and SLB are significantly improved compared with their classical counterparts and are smaller than their classical lower bounds. However, both MAB and SLB fall short when optimizing complex real-world functions (e.g., optimizing the hyperparameters of ML models). This is because MAB is unable to exploit the correlations among the arms to model the reward function, and the assumption of a linear reward function adopted by SLB is usually too restrictive in practice. Therefore, designing a quantum bandit algorithm capable of optimizing sophisticated non-linear reward functions, which is a critical step towards practically useful quantum bandit algorithms, is still an open problem. In this work, we resolve this open problem by proposing the first algorithm for quantum BO.
Similar to [38], in our quantum BO problem setting, queries to the reward function in the classical setting are replaced by access to a quantum oracle encoding the reward distribution. As discussed in [38, 39], a quantum oracle is available when the learning environment is implemented by a quantum algorithm, which makes the setting of our quantum BO fairly general. For example, our quantum BO algorithm may be used to optimize the hyperparameters of quantum ML algorithms [3], such as quantum support vector machines (SVMs) [29], quantum neural networks (NNs) [1, 18], among others. It may also be used to optimize the parameters of simulators implemented on a quantum computer, or for applications involving quantum systems where the data produced is inherently quantum. Moreover, our quantum BO algorithm could also be applied to classical data and algorithms, because as discussed in [39], any classical computer program can be converted to a quantum circuit, allowing it to serve as a quantum oracle in our quantum BO algorithm.
In this work, we introduce the first quantum BO algorithm: quantum-Gaussian process-upper confidence bound (Q-GP-UCB). For every selected arm  $x_{s}$ , our Q-GP-UCB algorithm (Algo. 1) adopts the QMC subroutine (Lemma 1) to efficiently estimate the corresponding reward function value  $f(x_{s})$  to satisfy a target estimation error  $\epsilon_{s}$ . Every arm  $x_{s}$  is selected based on our weighted GP posterior distribution, in which every previously selected arm  $x_{s}$  is given a weight of  $1 / \epsilon_{s}^{2}$  which is inversely related to its estimation error. The theoretical analysis of our Q-GP-UCB is faced with non-trivial technical challenges, and our contributions can be summarized as follows:
- We analyze the growth rate of the weighted information gain (Sec. 5.1) which arises due to the use of our weighted GP regression (Sec. 4.1), and show its connection with the standard maximum information gain commonly used in the analysis of BO/kernelized bandits [9, 33]. Our analysis here may be of broader independent interest for future works adopting weighted GP regression.
- We derive a tight confidence ellipsoid which gives a guarantee on the concentration of the reward function around the weighted GP posterior mean (Sec. 5.3), and discuss the intuition behind our proof which corresponds to an interesting interpretation about our Q-GP-UCB algorithm. A particularly crucial and novel step in our analysis relies on the recognition to apply the concentration inequality for 1-sub-Gaussian noise (Sec. 5.3).
- We prove that our Q-GP-UCB achieves a regret upper bound of  $\mathcal{O}(\mathrm{poly}\log T)$  for the commonly used squared exponential (SE) kernel (Secs. 5.4 and 5.5), which is considerably smaller than the classical regret lower bound of  $\Omega (\sqrt{T})$  [31] and hence represents a significant quantum speedup.
- By using a similar proof technique to the one adopted for our tight confidence ellipsoid (Sec. 5.3), we improve the confidence ellipsoid for the quantum linear UCB (Q-LinUCB) algorithm [38]. This improved analysis leads to a tighter regret upper bound for  $Q$ -LinUCB, which matches the regret of our Q-GP-UCB with the linear kernel (Sec. 5.6).
- We use simulations implemented based on the Qiiski t package to verify the empirical improvement of our Q-GP-UCB over the classical GP-UCB and over Q-LinUCB, through a synthetic experiment and an experiment on automated machine learning (AutoML) (Sec. 6). Notably, we have also
performed an experiment using a real quantum computer, in which our Q-GP-UCB still achieves a consistent performance advantage (Fig. 4, App. J)
A number of recent works have studied bandit algorithms in the quantum setting. The works of [8] and [40] have focused on the problem of pure exploration in quantum bandits. [25] has studied a different problem setting where the learner aims to maximize some property of an unknown quantum state (i.e., the rewards) by sequentially selecting the observables (i.e., actions). The recent work of [38] has introduced algorithms for quantum multi-armed bandits (MAB) and stochastic linear bandits (SLB), and has shown that by incorporating the QMC subroutine into MAB and SLB, tight regret upper bounds can be achieved which are better than the classical lower bounds. More recently, the works of [23] and [41] have followed similar approaches to introduce quantum bandit algorithms for, respectively, stochastic convex bandits and bandits with heavy-tailed reward distributions. In addition to quantum bandits, some recent works have introduced quantum reinforcement learning (RL) algorithms. [39] has assumed access to a generative model of the environment and proved that their quantum RL algorithm achieves significantly better sample complexity over their classical counterparts. More recently, [17, 43] have removed the requirement for a generative model in quantum RL and achieved quantum speedup in terms of the regret. More comprehensive reviews of the related works on quantum RL can be found in recent surveys (e.g., [27]). The problem setting of our quantum BO (Sec. 3.2) is the same as many of these above-mentioned previous works on quantum bandits [38, 41] and quantum RL [17, 39, 43]. However, to the best of our knowledge, none of the existing works on quantum bandits (and quantum RL) is able to handle non-linear reward functions, which we resolve in this work. Over the years, extensive efforts [7, 24, 30, 36] have been made to design BO/kernelized bandit algorithms whose regret upper bounds are small enough to (nearly) match the classical regret lower bound of  $\Omega(\sqrt{T})$  (ignoring additional log factors). Our work provides an alternative route by proving that with the aid of quantum computing, it is possible to achieve a tight regret upper bound which is significantly smaller than the classical regret lower bound.
3.1 Classical Kernelized Bandits
A BO/kernelized bandit algorithm aims to maximize a reward function  $f: \mathcal{X} \to \mathbb{R}$  where  $\mathcal{X} \subset \mathbb{R}^d$ , i.e., it aims to find  $x^{*} \in \arg \max_{x \in \mathcal{X}} f(x)$ . Consistent with the previous works on BO/kernelized bandits [9, 37], we assume that  $f$  lies in the reproducing kernel Hilbert space (RKHS) associated with a kernel  $k: f \in H_k(\mathcal{X})$ . That is, we assume that  $\| f \|_{H_k} \leq B$  for  $B > 0$ , in which  $\| \cdot \|_{H_k}$  denotes the RKHS norm induced by the kernel  $k$ . Note that unlike previous works on linear bandits [2], our assumption here allows  $f$  to be non-linear w.r.t. the input  $x$ . In this work, we mainly focus on the widely used squared exponential (SE) kernel:  $k(x, x') \triangleq \exp(-\| x - x' \|_2^2 / (2l^2))$  in which  $l$  is the length scale. We also extend our results to the Matérn kernel in Sec. 5.7. Without loss of generality, we assume that  $k(\cdot, \cdot) \leq 1$ . In the following, we use  $[t]$  to denote  $\{1, \dots, t\}$  for simplicity.
In every iteration  $t \in [T]$  of BO, an input  $x_{t}$  is selected, after which a corresponding noisy observation  $y_{t} = f(x_{t}) + \zeta_{t}$  is collected where  $\zeta_t$  is a sub-Gaussian noise (e.g., bounded noise or Gaussian noise). The inputs  $x_{t}$ 's are sequentially selected by maximizing an acquisition function, which is calculated based on the Gaussian process (GP) posterior distribution. Specifically, after  $t$  iterations, we use the current history  $\mathcal{D}_t \triangleq \{(x_1,y_1),\ldots ,(x_t,y_t)\}$  to calculate the GP posterior predictive distribution at  $x \in \mathcal{X}$ :  $\mathcal{N}(\mu_t(x),\sigma_t^2 (x))$  where
$$
\mu_ {t} (x) \triangleq k _ {t} ^ {\top} (x) \left(K _ {t} + \lambda I\right) ^ {- 1} Y _ {t}, \quad \sigma_ {t} ^ {2} (x) \triangleq k (x, x) - k _ {t} ^ {\top} (x) \left(K _ {t} + \lambda I\right) ^ {- 1} k _ {t} (x), \tag {1}
$$
in which  $k_{t}(x) \triangleq [k(x_{\tau}, x)]_{\tau \in [t]}^{\top}$  and  $Y_{t} \triangleq [y_{x_{\tau}}]_{\tau \in [t]}^{\top}$  are column vectors,  $K_{t} \triangleq [k(x_{\tau}, x_{\tau'})]_{\tau, \tau' \in [t]}$  is the  $t \times t$  covariance matrix, and  $\lambda > 1$  is a regularization parameter. Based on (1), a commonly used acquisition function is GP-UCB [33], which selects the next query by:  $x_{t+1} = \arg \max_{x \in \mathcal{X}} \mu_t(x) + \xi_{t+1} \sigma_t(x)$  where  $\xi_{t+1}$  is a parameter carefully chosen to balance exploration and exploitation. The performance of a BO/kernelized bandit algorithm is usually theoretically analyzed by deriving an upper bound on its cumulative regret:  $R_{T} = \sum_{t=1}^{T} [f(x^{*}) - f(x_{t})]$ , and a tighter regret upper bound is an indicator of a better theoretical convergence.
3.2 Quantum Bandits/BO
A quantum state  $|x\rangle$  in Hilbert space  $\mathbb{C}^n$  can be expressed as a superposition of  $n$  basis states (e.g. qubits) via a vector  $\vec{x} = [x_1,\dots ,x_n]^\top$ , with  $|x_i|^2$  representing the probability of being in the  $i^{th}$  basis state. Given two quantum states  $|x\rangle \in \mathbb{C}^n$  and  $|y\rangle \in \mathbb{C}^m$ , we use  $|x\rangle |y\rangle = [x_1y_1,\dots ,x_ny_m]^\top$  to denote their tensor product. A quantum algorithm typically works by applying unitary operators to quantum states, and may have access to input data encoded in unitary operators called quantum oracles (examples have been discussed in Sec. 1). We defer a more detailed introduction to quantum computing to related works dedicated to these topics (e.g., [26]).
Our quantum BO setting follows that of the quantum bandits works in [38, 41]. In every iteration of quantum BO, after an input  $x$  is selected, instead of observing a noisy reward as in classical BO/bandits (Sec. 3.1), we get a chance to access a quantum unitary oracle  $\mathcal{O}_x$  and its inverse that encode the noisy reward distribution. Specifically, let  $P_x$  denote the reward distribution,  $\Omega_x$  denote the finite sample space of  $P_x$ , and  $y_x: \Omega_x \to \mathbb{R}$  denote the random reward associated with input  $x$ . Then,  $\mathcal{O}_x$  is formally defined as:
$$
\mathcal {O} _ {x}: | 0 \rangle \rightarrow \sum_ {\omega \in \Omega_ {x}} \sqrt {P _ {x} (\omega)} | \omega \rangle | y _ {x} (\omega) \rangle . \tag {2}
$$
Quantum mean estimation, which aims to estimate the mean of an unknown distribution with better sample efficiency than classical algorithms, has been a widely studied problem [4, 21]. Consistent with the works of [38, 41], we will make use of the following quantum Monte Carlo (QMC) algorithm:
Lemma 1 (Quantum Monte Carlo (QMC) [28]). Let  $y_{x} : \Omega_{x} \to \mathbb{R}$  denote a random variable,  $\Omega_{x}$  is equipped with probability measure  $P_{x}$ , and the quantum unitary oracle  $\mathcal{O}_{x}$  encodes  $P_{x}$  and  $y_{x}$ .
- Bounded Noise: If the noisy output observation satisfies  $y_{x} \in [0,1]$ , then there exists a constant  $C_1 > 1$  and a QMC algorithm  $QMC(\mathcal{O}_x, \epsilon, \delta)$  which returns an estimate  $\hat{y}_{x}$  of  $\mathbb{E}[y_x]$  such that  $\mathbb{P}(|\hat{y}_x - \mathbb{E}[y_x]| > \epsilon) \leq \delta$ , using at most  $\frac{C_1}{\epsilon} \log(1/\delta)$  queries to  $\mathcal{O}_x$  and its inverse.
- Noise with Bounded Variance: If the variance of  $y_{x}$  is  $\leq \sigma^{2}$ , then for  $\epsilon < 4\sigma$ , there exists a constant  $C_2 > 1$  and a QMC algorithm  $QMC(\mathcal{O}_x, \epsilon, \delta)$  which returns an estimate  $\hat{y}_{x}$  s.t.  $\mathbb{P}(|\hat{y}_{x} - \mathbb{E}[y_{x}]| > \epsilon) \leq \delta$ , using at most  $\frac{C_2\sigma}{\epsilon}\log_2^{3/2}\left(\frac{8\sigma}{\epsilon}\right)\log_2(\log_2\frac{8\sigma}{\epsilon})\log \frac{1}{\delta}$  queries to  $\mathcal{O}_x$  and its inverse.
The quantum query complexity of a quantum algorithm is usually measured by the number of queries to the quantum oracle [3, 38]. So, the sample complexities from Lemma 1 can be compared with that of classical Monte Carlo (MC) estimation. In the classical setting, it can be easily shown using concentration inequalities that MC estimation requires  $\widetilde{\mathcal{O}}(1/\epsilon^2)$  samples to reach a target mean estimation error of  $\epsilon$ . Therefore, the QMC algorithm (Lemma 1), which only needs  $\widetilde{\mathcal{O}}(1/\epsilon)$  samples, achieves a quadratic reduction in the required number of samples. This dramatic improvement is crucial for the quantum speedup in terms of regrets achieved by our algorithm (Sec. 5.4).
During our Q-GP-UCB algorithm, after every input  $x_{s}$  is selected, we will apply the QMC algorithm from Lemma 1 with the quantum oracle  $\mathcal{O}_{x_s}$  to obtain an estimation  $y_{s}$  of its reward  $f(x_{s})$  (line 6 of Algo. 1). Of note, the above-mentioned equivalence between a query to the quantum oracle and a sample for classical MC mean estimation implies that a query to the quantum oracle  $\mathcal{O}_x$  in quantum BO/bandits is equivalent to the pulling of an arm  $x$  in classical BO/bandits. Therefore, the cumulative regret  $R_{T}$  of our Q-GP-UCB algorithm is defined as the regret incurred after  $T$  queries to the quantum oracle, which makes it amenable to comparisons with the regrets of classical algorithms.
4 Quantum Bayesian Optimization
In this section, we first introduce weighted GP regression (Sec. 4.1) which will be used by our Q-GP-UCB algorithm to calculate the acquisition function for input selection, and then describe our Q-GP-UCB algorithm (Sec. 4.2) in detail.
4.1 Weighted GP Posterior Distribution
In contrast to standard GP-UCB [33] which uses the standard GP posterior distribution (1) to calculate the acquisition function, our Q-GP-UCB makes use of a weighted GP posterior distribution. That is, we assign a weight to every previous observation. Specifically, after stage  $s$  of our Q-GP-UCB (i.e., given  $\mathcal{D}_s\triangleq \{(x_\tau ,y_\tau)\}_{\tau \in [s]}$ , we define the weight matrix  $W_{s}\triangleq \mathrm{diag}(1 / \epsilon_{1}^{2},\ldots ,1 / \epsilon_{s}^{2})$ .  $W_{s}$  is
an  $s \times s$  diagonal matrix, in which the  $\tau^{\mathrm{th}}$  diagonal element represents the weight  $1 / \epsilon_{\tau}^{2}$  given to the  $\tau^{\mathrm{th}}$  observation  $(x_{\tau}, y_{\tau})$ . We will set  $\epsilon_{\tau} = \widetilde{\sigma}_{\tau - 1}(x_{\tau}) / \sqrt{\lambda}$  (Sec. 4.2), i.e.,  $\epsilon_{\tau}$  is calculated using the weighted GP posterior standard deviation (3) (conditioned on the first  $\tau - 1$  observations) at  $x_{\tau}$ .
Define  $K_{s} \triangleq [k(x_{\tau}, x_{\tau'})]_{\tau, \tau' \in [s]}$  which is the  $s \times s$ -dimensional covariance matrix given the first  $s$  observations, and define  $\widetilde{K}_{s} \triangleq W_{s}^{1/2} K_{s} W_{s}^{1/2}$  which is the weighted covariance matrix. Similarly, define  $k_{s}(x) \triangleq [k(x, x_{\tau})]_{\tau \in [s]}^{\top}$  and  $\widetilde{k}_{s}(x) \triangleq W_{s}^{1/2} k_{s}(x)$ . Denote the collection of output observations by  $Y_{s} \triangleq [y_{\tau}]_{\tau \in [s]}^{\top}$ , and define  $\widetilde{Y}_{s} \triangleq W_{s}^{1/2} Y_{s}$ . With these definitions, given  $\mathcal{D}_{s}$ , our weighted GP posterior distribution at an input  $x \in \mathcal{X}$  is a Gaussian distribution:  $\mathcal{N}(\widetilde{\mu}_{s}(x), \widetilde{\sigma}_{s}^{2}(x))$ , in which
$$
\widetilde {\mu} _ {s} (x) \triangleq \widetilde {k} _ {s} ^ {\top} (x) (\widetilde {K} _ {s} + \lambda I) ^ {- 1} \widetilde {Y} _ {s}, \quad \widetilde {\sigma} _ {s} ^ {2} (x) \triangleq k (x, x) - \widetilde {k} _ {s} ^ {\top} (x) (\widetilde {K} _ {s} + \lambda I) ^ {- 1} \widetilde {k} _ {s} (x). \tag {3}
$$
Note that the GP posterior mean  $\widetilde{\mu}_s$  above is equivalently the solution to the following weighted kernel ridge regression problem:  $\widetilde{\mu}_s = \arg \min_{f\in H_k(\mathcal{X})}\sum_{\tau = 1}^s\frac{1}{\epsilon_\tau^2} (y_\tau -f(x_\tau))^2 +\lambda \| f\| _H^2$ . We give a more detailed analysis of the weighted GP posterior (3) in App. A. Weighted GP regression has also been adopted by previous works on BO such as [15]. However, our choice of the weights, algorithmic design and theoretical analyses all require significantly novel treatments.
4.2 Q-GP-UCB Algorithm
Algorithm 1 Q-GP-UCB
1: for stage  $s = 1,2,\ldots$  do
2:  $x_{s} = \arg \max_{x\in \mathcal{X}}\widetilde{\mu}_{s - 1}(x) + \beta_{s}\widetilde{\sigma}_{s - 1}(x)$  (3).
3:  $\epsilon_{s} = \widetilde{\sigma}_{s - 1}(x_{s}) / \sqrt{\lambda}$ .
4: (a)  $N_{\epsilon_s} = \frac{C_1}{\epsilon_s}\log \left(\frac{2\overline{m}}{\delta}\right)$  (for bounded noise), or
(b)  $N_{\epsilon_s} = \frac{C_2\sigma}{\epsilon_s}\log_2^{3 / 2}\left(\frac{8\sigma}{\epsilon_s}\right)\log_2(\log_2\frac{8\sigma}{\epsilon_s})\log \frac{2\overline{m}}{\delta}$  (for noise with variance bounded by  $\sigma^2$
5: If  $\sum_{k=1}^{s} N_{\epsilon_k} > T$ , terminate the algorithm.
6: Run the QMC  $(\mathcal{O}_{x_s},\epsilon_s,\delta /(2\overline{m}))$  algorithm to query the quantum oracle of  $x_{s}$  for the next  $N_{\epsilon_s}$  rounds, and obtain  $y_{s}$  as an estimate of  $f(x_{s})$
7: Update the weighted GP posterior (3) using  $(x_{s},y_{s})$
Our Q-GP-UCB algorithm is presented in Algo. 1. Q-GP-UCB proceeds in stages. In stage  $s$ , we first select the next input  $x_{s}$  to query by maximizing the GP-UCB acquisition function calculated using the weighted GP posterior distribution (3) (line 2 of Algo. 1). Here  $\beta_{s} \triangleq B + \sqrt{2(\widetilde{\gamma}_{s - 1} + 1 + \log(2 / \delta))}$  (more details in Sec. 5.3), in which  $\delta \in (0, 2 / e]$  and  $\widetilde{\gamma}_{s - 1} \triangleq \frac{1}{2} \log (\det(I + \frac{1}{\lambda} \widetilde{K}_{s - 1}))$  is the weighted information gain (more details in Sec. 5.1). Next, we calculate  $\epsilon_{s} = \widetilde{\sigma}_{s - 1}(x_{s}) / \sqrt{\lambda}$  (line 3) and  $N_{\epsilon_{s}}$  (line 4), in which  $N_{\epsilon_{s}}$  depends on the type of noise and the value of  $\epsilon_{s}$ . Here  $\overline{m}$  is an upper bound on the total number  $m$  of stages which we will analyze in Sec. 5.2. Subsequently, unless the algorithm is terminated (line 5), we run the QMC algorithm (Lemma 1) to estimate  $f(x_{s})$  by querying the quantum oracle of  $x_{s}$  for  $N_{\epsilon_{s}}$  rounds (line 6). The QMC procedure returns an estimate  $y_{s}$  of the reward function value  $f(x_{s})$ , for which the estimation error is guaranteed to be bounded:  $|y_{s} - f(x_{s})| \leq \epsilon_{s}$  with probability of at least  $1 - \delta / (2\overline{m})$ . Lastly, we update the weighted GP posterior (3) using the newly collected input-output pair  $(x_{s}, y_{s})$ , as well as its weight  $1 / \epsilon_{s}^{2}$  (line 7).