SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\hat {\sigma} _ {n} ^ {2} \left(P _ {*}\right) = \hat {k} \left(P _ {*}, P _ {*}\right) - \hat {\mathbf {k}} _ {n} \left(P _ {*}\right) ^ {T} \left(\hat {\mathbf {K}} _ {n} + \sigma^ {2} \mathbf {I}\right) ^ {- 1} \hat {\mathbf {k}} _ {n} \left(P _ {*}\right), \tag {19}
$$
where  $\mathbf{y}_n\coloneqq [y_1,\dots ,y_n]^T,\hat{\mathbf{k}}_n(P_*)\coloneqq [\hat{k} (P_*,P_1),\dots ,\hat{k} (P_*,P_n)]^T$  and  $[\hat{\mathbf{K}}_n]_{ij} = \hat{k} (P_i,P_j)$
Now we restrict our Gaussian process in the subspace  $\mathcal{P}_{\mathcal{X}} = \{P_x,x\in \mathcal{X}\} \subset \mathcal{P}$ . We assume the observation  $y_{i} = f(x_{i}) + \zeta_{i}$  with the noise  $\zeta_{i}$ . The input-induced noise is defined as  $\Delta f_{p_{x_i}}\coloneqq$ $f(x_{i}) - \mathbb{E}_{P_{x_{i}}}[f] = f(x_{i}) - \hat{f} (P_{x_{i}})$ . Then the total noise is  $y_{i} - \mathbb{E}_{P_{x_{i}}}[f] = \zeta_{i} + \Delta f_{p_{x_{i}}}$ . To formulate the regret bounds, we introduce the information gain and estimated information gain given any  $\{P_t\}_{t = 1}^n\subset \mathcal{P}$ :
$$
\hat {I} \left(\mathbf {y} _ {n}; \hat {\mathbf {f}} _ {n} \mid \left\{P _ {t} \right\} _ {t = 1} ^ {n}\right) := \frac {1}{2} \ln \det  \left(\mathbf {I} + \sigma^ {- 2} \hat {\mathbf {K}} _ {n}\right), \tag {20}
$$
$$
\tilde {I} \left(\mathbf {y} _ {n}; \hat {\mathbf {f}} _ {n} \mid \left\{P _ {t} \right\} _ {t = 1} ^ {n}\right) := \frac {1}{2} \ln \det  \left(\mathbf {I} + \sigma^ {- 2} \tilde {\mathbf {K}} _ {n}\right), \tag {21}
$$
and the maximum information gain is defined as  $\hat{\gamma}_n \coloneqq \sup_{\mathcal{R} \in \mathcal{P}_{\mathcal{X}}; |\mathcal{R}| = n} \hat{I}(\mathbf{y}_n; \hat{\mathbf{f}}_n|\mathcal{R})$ . Here  $\hat{\mathbf{f}}_n \coloneqq [\hat{f}(p_1), \dots, \hat{f}(p_n)]^T$ .
We define the sub-Gaussian condition as follows:
Definition 1. For a given  $\sigma_{\xi} > 0$ , a real-valued random variable  $\xi$  is said to be  $\sigma_{\xi}$ -sub-Gaussian if:
$$
\forall \lambda \in \mathbb {R}, \mathbb {E} [ e ^ {\lambda \xi} ] \leq e ^ {\lambda^ {2} \sigma_ {\xi} ^ {2} / 2}
$$
Now we can state the lemma for bounding the uncertain-inputs regret of exact kernel evaluations, which is originally stated in Theorem 5 in [26].
Lemma 2. Let  $\delta \in (0,1)$ ,  $f\in \mathcal{H}_k$ , and the corresponding  $\| \hat{f}\|_{\hat{k}}\leq b$ . Suppose the observation noise  $\zeta_{i} = y_{i} - f(x_{i})$  is conditionally  $\sigma_{\zeta}$ -sub-Gaussian. Assume that both  $k$  and  $P_{x}$  satisfy the conditions for  $\Delta f_{P_x}$  to be  $\sigma_{E}$ -sub-Gaussian, for a given  $\sigma_E > 0$ . Then, we have the following results:
- The following holds for all  $x \in \mathcal{X}$  and  $t \geq 1$ :
$$
\left. \left| \hat {\mu} _ {n} \left(P _ {x}\right) - \hat {f} \left(P _ {x}\right) \right| \leq \left(b + \sqrt {\sigma_ {E} ^ {2} + \sigma_ {\zeta} ^ {2}} \sqrt {2 \left(\hat {I} \left(\mathbf {y} _ {n} ; \hat {\mathbf {f}} _ {n} \mid \{P _ {t} \} _ {t = 1} ^ {n}\right) + 1 + \ln (1 / \delta)\right)}\right) \hat {\sigma} _ {n} \left(P _ {x}\right) \right. \tag {22}
$$
- Running with upper confidence bound (UCB) acquisition function  $\alpha(x|\mathcal{D}_n) = \hat{\mu}_n(P_x) + \hat{\beta}_n\hat{\sigma}_n(P_x)$  where
$$
\hat {\beta} _ {n} = b + \sqrt {\sigma_ {E} ^ {2} + \sigma_ {\zeta} ^ {2}} \sqrt {2 \left(\hat {I} (\mathbf {y} _ {n} ; \hat {\mathbf {f}} _ {n} | \{P _ {t} \} _ {t = 1} ^ {n}) + 1 + \ln (1 / \delta)\right)},
$$
and set  $\sigma^2 = 1 + 2 / n$ , the uncertain-inputs cumulative regret satisfies:
$$
\hat {R} _ {n} \in O \left(\sqrt {n \hat {\gamma} _ {n}} \left(b + \sqrt {\hat {\gamma} _ {n} + \ln \left(1 / \delta\right)}\right)\right) \tag {23}
$$
Note that although the original theorem restricted to the case when  $\hat{k}(p,q) = \langle \psi_P, \psi_Q \rangle_k$ , the results can be easily generated to other kernels over  $\mathcal{P}$ , as long as its universal w.r.t  $C(\mathcal{P})$  given that  $\mathcal{X}$  is compact and the mean map  $\psi$  is injective [11, 22].
B.2 Error Estimates for Inexact Kernel Approximation
Now let us derive the inference under the introduction of inexact kernel estimations.
Theorem 3. Under the Assumption 1 for  $\varepsilon >0$ , let  $\tilde{\mu}_n,\tilde{\sigma}_n,\tilde{I} (\mathbf{y}_n;\hat{\mathbf{f}}_n|\{P_t\}_{t = 1}^n)$  as defined in (13),(14),(21) respectively, and  $\hat{\mu}_n,\hat{\sigma}_n,\hat{I} (\mathbf{y}_n;\hat{\mathbf{f}}_n|\{P_t\}_{t = 1}^n)$  as defined in (18),(19),(20). Assume  $\max_{x\in \mathcal{X}}f(x) = M$ , and assume the observation error  $\zeta_i = y_i - f(x_i)$  satisfies  $|\zeta_i| < A$  for all  $i$ . Then we have the following error bound holds with probability at least  $1 - n\varepsilon$ :
$$
\left| \hat {\mu} _ {n} \left(P _ {*}\right) - \tilde {\mu} _ {n} \left(P _ {*}\right) \right| <   \left(\frac {n}{\sigma^ {2}} + \frac {n ^ {2}}{\sigma^ {4}}\right) (M + A) e _ {\varepsilon} + O \left(e _ {\varepsilon} ^ {2}\right) \tag {24}
$$
$$
\left| \hat {\sigma} _ {n} ^ {2} \left(P _ {*}\right) - \tilde {\sigma} _ {n} ^ {2} \left(P _ {*}\right) \right| <   \left(1 + \frac {n}{\sigma^ {2}}\right) ^ {2} e _ {\varepsilon} + O \left(e _ {\varepsilon} ^ {2}\right) \tag {25}
$$
$$
\left| \tilde {I} \left(\mathbf {y} _ {n}; \hat {\mathbf {f}} _ {n} \mid \{P _ {t} \} _ {t = 1} ^ {n}\right) - \hat {I} \left(\mathbf {y} _ {n}; \hat {\mathbf {f}} _ {n} \mid \{P _ {t} \} _ {t = 1} ^ {n}\right) \right| <   \frac {n ^ {3 / 2}}{2 \sigma^ {2}} e _ {\varepsilon} + O \left(e _ {\varepsilon} ^ {2}\right) \tag {26}
$$
Proof. Denote  $e(P_{*}, Q) = \tilde{k}(P_{*}, Q) - \hat{k}(P_{*}, Q)$ ,  $\mathbf{e}_n(P_*) = [e(P_*, P_1), \dots, e(P_* P_n)]^T$ , and  $[\mathbf{E}_n]_{i,j} = e(P_i, P_j)$ . Now according to the matrix inverse perturbation expansion,
$$
\left(X + \delta X\right) ^ {- 1} = X ^ {- 1} - X ^ {- 1} \delta X X ^ {- 1} + O \left(\| \delta X \| ^ {2}\right),
$$
$$
(\hat {\mathbf {K}} _ {n} + \sigma^ {2} \mathbf {I} + \mathbf {E} _ {n}) ^ {- 1} = (\hat {\mathbf {K}} _ {n} + \sigma^ {2} \mathbf {I}) ^ {- 1} - (\hat {\mathbf {K}} _ {n} + \sigma^ {2} \mathbf {I}) ^ {- 1} \mathbf {E} _ {n} (\hat {\mathbf {K}} _ {n} + \sigma^ {2} \mathbf {I}) ^ {- 1} + O (\| \mathbf {E} _ {n} \| ^ {2}),
$$
$$
\begin{array}{l} \tilde {\mu} _ {n} \left(P _ {*}\right) = \left(\hat {\mathbf {k}} _ {n} \left(P _ {*}\right) + \mathbf {e} _ {n} \left(P _ {*}\right)\right) ^ {T} \left(\hat {\mathbf {K}} _ {n} + \sigma^ {2} \mathbf {I} + \mathbf {E} _ {n}\right) ^ {- 1} \mathbf {y} _ {n} \\ = \hat {\mu} _ {n} (P _ {*}) + \mathbf {e} _ {n} (P _ {*}) ^ {T} (\hat {\mathbf {K}} _ {n} + \sigma^ {2} \mathbf {I}) ^ {- 1} \mathbf {y} _ {n} - \hat {\mathbf {k}} _ {n} (P _ {*}) ^ {T} (\hat {\mathbf {K}} _ {n} + \sigma^ {2} \mathbf {I}) ^ {- 1} \mathbf {E} _ {n} (\hat {\mathbf {K}} _ {n} + \sigma^ {2} \mathbf {I}) ^ {- 1} \mathbf {y} _ {n} \\ + O \left(\| \mathbf {E} _ {n} \| ^ {2}\right) + O \left(\| \mathbf {e} _ {n} \left(P _ {*}\right)\right) \| \cdot \| \mathbf {E} _ {n} \|) \\ \end{array}
$$
$$
\begin{array}{l} \tilde {\sigma} _ {n} ^ {2} \left(P _ {*}\right) = \hat {\sigma} _ {n} ^ {2} \left(P _ {*}\right) + e \left(P _ {*}, P _ {*}\right) - \left(\hat {\mathbf {k}} _ {n} \left(P _ {*}\right) + \mathbf {e} _ {n} \left(P _ {*}\right)\right) ^ {T} \left(\hat {\mathbf {K}} _ {n} + \sigma^ {2} \mathbf {I} + \mathbf {E} _ {n}\right) ^ {- 1} \left(\hat {\mathbf {k}} _ {n} \left(P _ {*}\right) + \mathbf {e} _ {n} \left(P _ {*}\right)\right) \\ = \hat {\sigma} _ {n} ^ {2} \left(P _ {*}\right) + e \left(P _ {*}, P _ {*}\right) - 2 \mathbf {e} _ {n} (P) ^ {T} \left(\hat {\mathbf {K}} _ {n} + \sigma^ {2} \mathbf {I}\right) ^ {- 1} \hat {\mathbf {k}} _ {n} \left(P _ {*}\right) \\ + \hat {\mathbf {k}} _ {n} (P) ^ {T} (\hat {\mathbf {K}} _ {n} + \sigma^ {2} \mathbf {I}) ^ {- 1} \mathbf {E} _ {n} (\hat {\mathbf {K}} _ {n} + \sigma^ {2} \mathbf {I}) ^ {- 1} \hat {\mathbf {k}} _ {n} (P _ {*}) \\ + O \left(\left\| \mathbf {E} _ {n} \right\| ^ {2}\right) + O \left(\left\| \mathbf {e} _ {n} \right\| \cdot \left\| \mathbf {E} _ {n} \right\|\right) + O \left(\left\| \mathbf {e} _ {n} \right\| ^ {2} \cdot \left\| \mathbf {E} _ {n} \right\|\right) \\ \end{array}
$$
Notic that the following holds with a probability at least  $1 - n\varepsilon$ , according to the Assumption 1,
$$
| \mathbf {e} _ {n} (P _ {*}) ^ {T} (\hat {K} _ {n} + \sigma^ {2} \mathbf {I}) ^ {- 1} \mathbf {y} _ {n} | \leq \| \mathbf {e} _ {n} (P _ {*}) \| _ {2} \| (\hat {K} _ {n} + \sigma^ {2} \mathbf {I}) ^ {- 1} \| _ {2} \| \mathbf {y} _ {n} \| _ {2} \leq \frac {n}{\sigma^ {2}} (M + A) e _ {\varepsilon},
$$
$$
\begin{array}{l} \left| \hat {\mathbf {k}} _ {n} \left(P _ {*}\right) ^ {T} \left(\hat {\mathbf {K}} _ {n} + \sigma^ {2} \mathbf {I}\right) ^ {- 1} \mathbf {E} _ {n} \left(\hat {\mathbf {K}} _ {n} + \sigma^ {2} \mathbf {I}\right) ^ {- 1} \mathbf {y} _ {n} \right| \leq \| \hat {\mathbf {k}} _ {n} \left(P _ {*}\right) \| _ {2} \| \left(\hat {K} _ {n} + \sigma^ {2} \mathbf {I}\right) ^ {- 1} \| _ {2} ^ {2} \| \mathbf {E} _ {n} \| _ {2} \| \mathbf {y} _ {n} \| _ {2} \\ \leq \sqrt {n} \sigma^ {- 4} n e _ {\varepsilon} \sqrt {n} (M + A) = \frac {n ^ {2}}{\sigma^ {4}} (M + A), \\ \end{array}
$$
here we use the fact that  $\hat{K}_n$  semi-definite (which means  $\| (\hat{K}_n + \sigma^2 I)^{-1}\|_2 \leq \sigma^{-2}$ ),  $\hat{k}(P_*, P_*) \leq 1$ ,  $|y_i| \leq M + A$ . Combining these results, we have that
$$
| \tilde {\mu} _ {n} (P _ {*}) - \hat {\mu} _ {n} (P _ {*}) | <   (\frac {n}{\sigma^ {2}} + \frac {n ^ {2}}{\sigma^ {4}}) (M + A) e _ {\varepsilon} + O (e _ {\varepsilon} ^ {2}),
$$
holds with a probability at least  $1 - n\varepsilon$ .
Similarly, we can conduct the same estimation to  $\mathbf{e}_n(P)^T (\hat{\mathbf{K}}_n + \sigma^2\mathbf{I})^{-1}\hat{\mathbf{k}}_n(P_*)$  and  $\hat{\mathbf{k}}_n(P)^T (\hat{\mathbf{K}}_n + \sigma^2\mathbf{I})^{-1}\mathbf{E}_n(\hat{\mathbf{K}}_n + \sigma^2\mathbf{I})^{-1}\hat{\mathbf{k}}_n(P_*)$ , and get
$$
| \tilde {\sigma} _ {n} ^ {2} (P _ {*}) - \hat {\sigma} _ {n} ^ {2} (P _ {*}) | <   (1 + \frac {n}{\sigma^ {2}}) ^ {2} e _ {\varepsilon} + O (e _ {\varepsilon} ^ {2})
$$
It remains to estimate the error for estimating the information gain. Notice that, with a probability at least  $1 - n\varepsilon$ ,
$$
\begin{array}{l} \left| \tilde {I} (\mathbf {y} _ {n}; \hat {\mathbf {f}} _ {n} | \{p _ {t} \} _ {t = 1} ^ {n}) - \hat {I} (\mathbf {y} _ {n}; \hat {\mathbf {f}} _ {n} | \{p _ {t} \} _ {t = 1} ^ {n}) \right| = \left| \frac {1}{2} \log \frac {\det (\mathbf {I} + \sigma^ {- 2} \tilde {\mathbf {K}} _ {n})}{\det (\mathbf {I} + \sigma^ {- 2} \hat {\mathbf {K}} _ {n})} \right| \\ = \left| \frac {1}{2} \log \det  (\mathbf {I} - \left(\sigma^ {2} \mathbf {I} + \hat {\mathbf {K}} _ {n}\right) ^ {- 1} \mathbf {E} _ {n}) \right| \\ = \left| \frac {1}{2} \operatorname {T r} \left(\log \left(\mathbf {I} - \left(\sigma^ {2} \mathbf {I} + \hat {\mathbf {K}} _ {n}\right) ^ {- 1} \mathbf {E} _ {n}\right)\right) \right| \\ = \left| \frac {1}{2} \operatorname {T r} \left(- \left(\sigma^ {2} \mathbf {I} + \hat {\mathbf {K}} _ {n}\right) ^ {- 1} \mathbf {E} _ {n}\right) + O \left(\left\| \mathbf {E} _ {n} \right\| ^ {2}\right) \right| \\ \leq \frac {n ^ {3 / 2}}{2 \sigma^ {2}} e _ {\varepsilon} + O (\| \mathbf {E} _ {n} \| ^ {2}), \\ \end{array}
$$
here the second equation uses the fact that  $\operatorname{det}(AB^{-1}) = \operatorname{det}(A)\operatorname{det}(B)^{-1}$ , and the third and fourth equations use  $\log \operatorname{det}(I + A) = \operatorname{Tr}\log (I + A) = \operatorname{Tr}(A - \frac{A^2}{2} +\dots)$ . The last inequality follows from the fact
$$
\left. \operatorname {T r} \left(\sigma^ {2} \mathbf {I} + \hat {\mathbf {K}} _ {n}\right) ^ {- 1} \mathbf {E} _ {n}\right) \leq \left\| \left(\sigma^ {2} \mathbf {I} + \hat {\mathbf {K}} _ {n}\right) ^ {- 1} \right\| _ {F} \| \mathbf {E} _ {n} \| _ {F} \leq n ^ {3 / 2} \sigma^ {- 2} e _ {\varepsilon}
$$
and  $\tilde{\mathbf{K}}_n$  is semi-definite.
With the uncertainty bound given by Lemma 3, let us prove that under inexact kernel estimations, the posterior mean is concentrated around the unknown reward function  $\hat{f}$
Theorem 4. Under the former setting as in Theorem 3, with probability at least  $1 - \delta - n\varepsilon$ , let  $\sigma_{\nu} = \sqrt{\sigma_{\zeta}^{2} + \sigma_{E}^{2}}$ , taking  $\sigma = 1 + \frac{2}{n}$ , the following holds for all  $x \in \mathcal{X}$ :
$$
\left| \tilde {\mu} _ {n} \left(P _ {x}\right) - \hat {f} \left(P _ {x}\right) \right| \leq \beta_ {n} \tilde {\sigma} _ {n} \left(P _ {x}\right) + \beta_ {n} (1 + n) e _ {\varepsilon} ^ {1 / 2} + (n + n ^ {2}) (M + A) e _ {\varepsilon}, \tag {27}
$$
$$
w h e r e \beta_ {n} = \left(b + \sigma_ {\nu} \sqrt {2 (\hat {\gamma} _ {n} - \ln (\delta) + 1)}\right)
$$
Proof. According to Lemma 2, equation (22), we have
$$
\left| \hat {\mu} _ {n} (P _ {x}) - \hat {f} (P _ {x}) \right| \leq \hat {\beta} _ {n} \hat {\sigma} _ {n} (P _ {x})
$$
$$
\hat {\beta} _ {n} = b + \sigma_ {\nu} \sqrt {2 \left(\hat {I} (\mathbf {y} _ {n} ; \hat {\mathbf {f}} _ {n} | \{P _ {t} \} _ {t = 1} ^ {n}) + 1 + \ln (1 / \delta)\right)} \leq \beta_ {n}.
$$
$$
\left| \tilde {\mu} _ {n} \left(P _ {x}\right) - \hat {f} \left(P _ {x}\right) \right| \leq \left| \tilde {\mu} _ {n} \left(P _ {x}\right) - \hat {\mu} _ {n} \left(P _ {x}\right) \right| + \left| \hat {\mu} _ {n} \left(P _ {x}\right) - \hat {f} \left(P _ {x}\right) \right|, \tag {28}
$$
We also have (25), which means
$$
\hat {\sigma} _ {n} \left(P _ {x}\right) = \sqrt {\hat {\sigma} _ {n} \left(P _ {x}\right) ^ {2}} \leq \sqrt {\tilde {\sigma} _ {n} \left(P _ {x}\right) ^ {2} + (1 + n) ^ {2} e _ {\varepsilon}} \leq \tilde {\sigma} _ {n} \left(P _ {x}\right) + (1 + n) e _ {\varepsilon} ^ {1 / 2}, \tag {29}
$$
combining (24), (28) and (29), we finally get the result in (27).
B.3 Proofs for Theorem 1
Now we can prove our main theorem 1.
Proof of Theorem 1. Let  $x^{*}$  maximize  $\hat{f}(P_x)$  over  $\mathcal{X}$ . Observing that at each round  $n \geq 1$ , by the choice of  $x_{n}$  to maximize the acquisition function  $\tilde{\alpha}(x|\mathcal{D}_{n-1}) = \tilde{\mu}_{n-1}(P_x) + \beta_{n-1}\tilde{\sigma}_{n-1}(P_x)$ , we have
$$
\begin{array}{l} \tilde {r} _ {n} = \hat {f} (P _ {x ^ {*}}) - \hat {f} (P _ {x _ {n}}) \\ \leq \tilde {\mu} _ {n - 1} \left(P _ {x ^ {*}}\right) + \beta_ {n - 1} \tilde {\sigma} _ {n - 1} \left(P _ {x ^ {*}}\right) - \tilde {\mu} _ {n - 1} \left(P _ {x _ {n}}\right) + \beta_ {n - 1} \tilde {\sigma} _ {n - 1} \left(P _ {x _ {n}}\right) + 2 E r r (n - 1, e _ {\varepsilon}) \\ \leq 2 \beta_ {n - 1} \tilde {\sigma} _ {n - 1} \left(P _ {x _ {n}}\right) + 2 E r r (n - 1, e _ {\varepsilon}). \\ \end{array}
$$
Here we denote  $Err(n, e_{\varepsilon}) \coloneqq \left(\beta_n(1 + n) + \tilde{\sigma}_n(P_x)\sigma_\nu n^{3/4}\right)e_{\varepsilon}^{1/2} + \left(n + n^2\right)(M + A)e_{\varepsilon}$ . The second inequality follows from (27),
$$
\begin{array}{l} \hat {f} (P _ {x ^ {*}}) - \tilde {\mu} _ {n - 1} (P _ {x ^ {*}}) \leq \beta_ {n - 1} \tilde {\sigma} _ {n - 1} (P _ {x ^ {*}}) + E r r (n - 1, e _ {\varepsilon}) \\ \tilde {\mu} _ {n - 1} (P _ {x _ {n}}) - \hat {f} (P _ {x _ {n}}) \leq \beta_ {n - 1} \tilde {\sigma} _ {n - 1} (P _ {x _ {n}}) + E r r (n - 1, e _ {\varepsilon}), \\ \end{array}
$$
and the third inequality follows from the choice of  $x_{n}$ :
$$
\tilde {\mu} _ {n - 1} \left(P _ {x ^ {*}}\right) + \beta_ {n - 1} \tilde {\sigma} _ {n - 1} \left(P _ {x ^ {*}}\right) \leq \tilde {\mu} _ {n - 1} \left(P _ {x _ {n}}\right) + \beta_ {n - 1} \tilde {\sigma} _ {n - 1} \left(P _ {x _ {n}}\right).
$$
$$
\tilde {R} _ {n} = \sum_ {t = 1} ^ {n} \tilde {r} _ {t} \leq 2 \beta_ {n} \sum_ {t = 1} ^ {n} \tilde {\sigma} _ {t - 1} \left(P _ {x _ {t}}\right) + \sum_ {t = 1} ^ {T} E r r (t - 1, e _ {\varepsilon}).
$$
From Lemma 4 in [9], we have that
$$
\sum_ {t = 1} ^ {n} \tilde {\sigma} _ {t - 1} \left(P _ {x _ {t}}\right) \leq \sqrt {4 (n + 2) \ln \det  \left(I + \sigma^ {- 2} \tilde {K} _ {n}\right)} \leq \sqrt {4 (n + 2) \left(\hat {\gamma} _ {n} + \frac {n ^ {\frac {3}{2}}}{2} e _ {\varepsilon}\right)},
$$
$$
2 \beta_ {n} \sum_ {t = 1} ^ {n} \tilde {\sigma} _ {t - 1} \left(P _ {x _ {t}}\right) = O \left(\sqrt {n \hat {\gamma} _ {n}} + \sqrt {n \hat {\gamma} _ {n} (\hat {\gamma} _ {n} - \ln \delta)} + \sqrt {n ^ {\frac {5}{2}} e _ {\varepsilon}} + \sqrt {n ^ {\frac {5}{2}} (\hat {\gamma} _ {n} - \ln \delta) e _ {\varepsilon}}\right). \tag {30}
$$
On the other hand, notice that
$$
\sum_ {t = 1} ^ {n} E r r (t - 1, e _ {\varepsilon}) = O \left(n ^ {2} \sqrt {(\hat {\gamma} _ {n} - \ln \delta) e _ {\varepsilon}} + (n ^ {2} + n ^ {3}) e _ {\epsilon}\right), \tag {31}
$$
we find that the  $e_{\varepsilon}$  term in (30) can be controlled by in (31), thus we immediately get the result.
B.4 Proofs for Theorem 2
Proof. Define the square of the MMD distance between  $P_{x_1}, P_{x_2}$  as  $d_M(x_1, x_2)$ , we have
$$
\begin{array}{l} d _ {M} (x _ {1}, x _ {2}) \\ = \int_ {\mathbb {R} ^ {d}} k (x, x ^ {\prime}) P _ {x _ {1}} (x) P _ {x _ {1}} (x ^ {\prime}) \mathrm {d} x \mathrm {d} x ^ {\prime} + \int_ {\mathbb {R} ^ {d}} k (x, x ^ {\prime}) P _ {x _ {2}} (x) P _ {x _ {2}} (x ^ {\prime}) \mathrm {d} x \mathrm {d} x ^ {\prime} \\ - 2 \int_ {\mathbb {R} ^ {d}} k (x, x ^ {\prime}) P _ {x _ {1}} (x) P _ {x _ {2}} (x ^ {\prime}) \mathrm {d} x \mathrm {d} x ^ {\prime} \\ = \int_ {\mathbb {R} ^ {d}} (k (x - x _ {1}, x ^ {\prime} - x _ {1}) + k (x - x _ {2}, x ^ {\prime} - x _ {2}) - 2 k (x - x _ {1}, x ^ {\prime} - x _ {2})) P _ {0} (x) P _ {0} (x ^ {\prime}) \mathrm {d} x \mathrm {d} x ^ {\prime}. \\ \end{array}
$$
It is not hard to verify that  $d_{M}$  is shift invariant:  $d_{M}(x_{1}, x_{2}) = d_{M}(x_{1} - x_{2}, 0)$ , and  $d_{M}$  has  $r$ -th bounded derivatives, thus  $\hat{k}^{*}(x_{1}, x_{2}) \coloneqq \hat{k}(P_{x_{1}}, P_{x_{2}}) = \exp(-\alpha d_{M}(x_{1}, x_{2}))$  is shift invariant with  $r$ -th bounded derivatives. Then take  $\mu(x)$  as the Lebesgue measure over  $\mathcal{X}$ , according to Theorem 4, [17], the integral operator  $T_{k,\mu}: T_{k,\mu}f(x) = \int_{\mathcal{X}} K(x,y)f(y)d\mu(y)$  is a symmetric compact operator in  $L_{2}(\mathcal{X}, \mu)$ , and the spectrum of  $T_{k,\mu}$  satisfies
$$
\lambda_ {n} (T _ {k, \mu}) = O (n ^ {- 1 - r / d}).
$$
Then according to Theorem 5 in [30], we have  $\hat{\gamma}_n = O(n^{\frac{d(d + 1)}{r + d(d + 1)}}\log (n))$ , which finish the proof.
C Evaluation Details
C.1 Implementation
In our implementation of AIRBO, we design the kernel  $k$  used for MMD estimation to be a linear combination of multiple Rational Quadratic kernels as its long tail behavior circumvents the fast decay issue of kernel [6]:
$$
k \left(x, x ^ {\prime}\right) = \sum_ {a _ {i} \in \{0. 2, 0. 5, 1, 2, 5 \}} \left(1 + \frac {\left(x - x ^ {\prime}\right) ^ {2}}{2 a _ {i} l _ {i} ^ {2}}\right) ^ {- a _ {i}}, \tag {32}
$$
where  $l_{i}$  is a learnable lengthscale and  $a_{i}$  determines the relative weighting of large-scale and small-scale variations.
Depending on the form of input distributions, the sampling and sub-sampling sizes for Nyström MMD estimator are empirically selected via experiments. Moreover, as the input uncertainty is already modeled in the surrogate, we employ a classic UCB-based acquisition as Eq. 5 with  $\beta = 2.0$  and maximize it via an L-BFGS-B optimizer.
D More Experiments
D.1 Comparing with the Other Models
To compare the modeling performances with the other models, we design the input uncertainty to follow a step-changing Chi-squared distribution:  $P_{x} = \chi^{2}(g(x),\sigma = 0.01)$ , where  $g(x) = 0.5$  if  $x\in [0.0,0.6)$  and  $g(x) = 7.0$  when  $x\in [0.6,1.0]$ . Due to this sudden parameter change, the uncertainty at point  $x = 0.6$  is expected to be asymmetric: 1) on its left-hand side, as the Chi-squared distribution becomes quite lean and sharp with a small value of  $g(x) = 0.5$ , the distance from  $x = 0.6$  to its LHS points,  $x_{lhs}\in [0.0,0.6)$ , are relatively large, thus their covariances are small, resulting a fast-growing uncertainty. 2) Meanwhile, when  $x\in [0.6,1.0]$ , the  $g(x)$  suddenly increases to 7.0, rendering the input distribution a quite flat one with a long tail. Therefore, the distances between  $x = 0.6$  and its RHS points become relatively small, which leads to large covariances and small uncertainties for points in [0.6,1.0]. As a result, we expect to observe an asymmetric posterior uncertainty at  $x = 0.6$ .
Several surrogate models are employed in this comparison, including:
- MMDGP-nystrom(160/10) is our method with a sampling size  $m = 160$  and sub-sampling size  $h = 10$ . Its complexity is  $O(MNmh)$ , where  $M$  and  $N$  are the sizes of training and test samples (Note: all the models in this experiment use the same training and testing samples for a fair comparison).
- uGP(40) is the surrogate from [25], which employs an integral kernel with sampling size  $m = 40$ . Due to its  $O(MNm^2)$  complexity, we set the sampling size  $m = 40$  to ensure a similar complexity as ours.
- uGP(160) is also the surrogate from [25] but uses a much larger sampling size ( $m = 160$ ). Given the same training and testing samples, its complexity is 16 times higher than MMDGP-nystrom(160/10).
- skl is a robust GP surrogate equipped with a symmetric KL-based kernel, which is described in [20].
- ERBF [13] assumes the input uncertainty to be Gaussians and employs a close-form expected RBF kernel.
- GP utilizes a noisy Gaussian Process model with a learnable output noise level.
According to Figure 7a, our method, MMDGP-nystrom(160/10), can comprehensively quantify the sudden change of the input uncertainty, evidenced by its abrupt posterior change at  $x = 0.6$ . However, Figure 7b shows that uGP(40) with the same complexity fails to model the uncertainty correctly. We suspect this is because uGP requires much larger samples to stabilize its estimation of the integral kernel and thus can perform poorly with insufficient sample size, so we further evaluate the uGP(160) with a much larger sampling size  $m = 160$  in Figure 7c. It does successfully alleviate the issue but also results in a 16 times higher complexity. Apart from this, Figure 7d suggests the noisy GP model with a learnable output noise level is not aware of this uncertainty change at all as it treats the inputs as the exact values instead of random variables. Moreover, Figure 7e and 7f show that both the skl and ERBF fail in this case, this may be due to their misassumption of Gaussian input uncertainty.
D.2 Ablation Test for Nyström Approximation
In this experiment, we aim to examine the effect of Nyström approximation for optimization. To this end, we choose to optimize an RKHS function (Figure 4a) under a beta input distribution:  $P_{x} = \beta \alpha (\alpha = 0.4,\beta = 0.2,\sigma = 0.1)$ . Several amortized candidates include:
(a) MMD-GP with a Nystrom estimator, in which the sampling size  $m = 160$  and sub-sampling size  $h = 10$ .
(b) uGP model that uses an integral kernel [26] and a sampling size of  $m = 40$
(c) uGP with an integral kernel [26] and uses a much larger sampling size  $(m = 160)$ .
(d) Conventional noisy GP model.
(e) GP model with a symmetric KL-divergence kernel [20].
(f) Robust GP model with an expected RBF kernel [13]
Figure 7: Modeling performance with a step-changing Chi-squared distribution.
Figure 8: Ablation test for the Nyström approximation.
- MMDGP-nystrom is our method with Nystrom approximation, in which the sampling size  $m = 16$  and sub-sampling size  $h = 9$ . Its complexity is  $O(MNmh)$ , where  $M$  and  $N$  are the sizes of training and test samples respectively,  $m$  is the sampling size for MMD estimation, and  $h$  indicates the sub-sampling size during the Nystrom approximation.
- MMDGP-raw-S does not use the Nystrom approximation but employs an empirical MMD estimator. Due to its  $O(MNm^2)$  complexity, we set the sampling size  $m = 12$  to ensure a similar complexity as the MMDGP-nystrom.
- MMDGP-raw-L also uses an empirical MMD estimator, but with a larger sampling size  $(m = 16)$ .
-  $GP$  utilizes a vanilla GP with a learnable output noise level and optimizes with the upper-confidence-bound acquisition $^2$ .