SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
P (Z \leq \alpha) \leq \exp \left(- 2 n \left(p - \frac {\alpha}{n}\right) ^ {2}\right).
$$
For  $P(Z\leq \alpha)\leq \delta$
$$
\exp \left(- 2 n \left(p - \frac {\alpha}{n}\right) ^ {2}\right) = \delta
$$
3While we technically have  $\binom{T_i}{2}$  (overlapping) pairs, the squared distances between each such pair will be identically distributed but not independent. For example, if  $T_{i}\geq 3$  and we knew that  $\binom{T_i}{2}-1$  of the squared distances were equal to 0 (i.e., all the queries are exactly the same), the last squared distance must also be equal to 0.
$$
\alpha = n p - \sqrt {\frac {n}{2} \log \frac {1}{\delta}}
$$
$$
\alpha = \frac {1}{2} \left\lfloor \frac {T _ {i}}{2} \right\rfloor - \sqrt {\frac {1}{2} \left\lfloor \frac {T _ {i}}{2} \right\rfloor \log \frac {1}{\delta}}
$$
$$
\alpha \geq \frac {1}{4} (T _ {i} - 1) - \sqrt {\frac {1}{4} T _ {i} \log \frac {1}{\delta}}
$$
$$
\alpha \geq \left\lfloor \frac {1}{4} (T _ {i} - 1) - \sqrt {\frac {1}{4} T _ {i} \log \frac {1}{\delta}} \right\rfloor .
$$
Therefore, with probability more than  $1 - \delta$ ,  $Z > N_{i} \coloneqq \left\lfloor \frac{1}{4}(T_{i} - 1) - \sqrt{\frac{1}{4}T_{i}\log\frac{1}{\delta}} \right\rfloor$ , i.e., the number of non-overlapping pairs with squared distance greater than  $M_{i}$  is at least  $N_{i}$ , which completes the proof.
A.5 Proof of Theorem 4.4
Theorem 4.4. If the following assumptions hold:
1. The assumption of Lemma 4.3 holds;
2. The kernel  $k(\mathbf{x}, \mathbf{x}^{\prime})$  is an isotropic kernel (which only depends on distance and can be written as  $k(\| \mathbf{x} - \mathbf{x}^{\prime} \|)$ );
3. There exists an iteration  $r$  such that for all  $t \leq r, i_t \leq m - 1$  and for all  $t > r, i_t = m$ ;
then with probability at least  $1 - \delta$ , UCB-CVS (Alg. 1) incurs a cost-varying cumulative regret bounded by
$$
R _ {T} \leq c _ {m} \sum_ {t = 1} ^ {T} \epsilon_ {t} + \mathcal {O} \left(\left(B + \sqrt {\gamma_ {T} (\mathcal {X}) + \log \frac {2 m}{\delta}}\right) \left(c _ {m} \left(\sqrt {T \gamma_ {T} (\mathcal {X})} - \mathcal {L} + \log \frac {2 m}{\delta}\right) + \right. \right.
$$
$$
\left. \sum_ {i = 1} ^ {m - 1} c _ {i} \left(\sqrt {T _ {i} \gamma_ {T _ {i}} (\tilde {\mathcal {X}} _ {i})} + \log \frac {2 m}{\delta}\right)\right)
$$
by setting  $\beta_{t} = B + \sigma \sqrt{2(\gamma_{t - 1}(\mathcal{X}) + 1 + \log((2m) / \delta))}$ , where  $N_{i}$  and  $M_{i}$  are defined as in Lemma 4.3, and  $V_{i}$  and  $W$  are residual terms defined in (10).
Proof. We first construct a lower bound on the sum of posterior standard deviations of the queries up to iteration  $r$ , i.e., the queries that were chosen with any control set except the last.
$$
\begin{array}{l} \sum_ {t = 1} ^ {r} \sigma_ {t - 1} (\mathbf {x} _ {t}) \stackrel {(i)} {\geq} \sum_ {t = 1} ^ {r} \sigma_ {t - 1} ^ {2} (\mathbf {x} _ {t}) \\ = \lambda \sum_ {t = 1} ^ {r} \lambda^ {- 1} \sigma_ {t - 1} ^ {2} (\mathbf {x} _ {t}) \\ \stackrel {(i i)} {\geq} \lambda \sum_ {t = 1} ^ {r} \log (1 + \lambda^ {- 1} \sigma_ {t - 1} ^ {2} (\mathbf {x} _ {t})) \\ \stackrel {(i i i)} {=} \lambda \log \left| \mathbf {I} + \lambda^ {- 1} \mathbf {K} _ {r} \right| \\ = \lambda \log (\lambda^ {- r} | \lambda \mathbf {I} + \mathbf {K} _ {r} |) \\ = \lambda (- r \log \lambda + \log | \lambda \mathbf {I} + \mathbf {K} _ {r} |) \\ \geq \lambda (\log | \lambda \mathbf {I} + \mathbf {K} _ {r} | - 2) \tag {7} \\ \end{array}
$$
where  $(i)$  follows from the assumption that  $k(x,x) = 1$  which implies  $\sigma_{t - 1}(\mathbf{x})\leq 1$  for all  $\mathbf{x}\in \mathcal{X}$  and all  $t\geq 1$ ,  $(ii)$  follows since  $\log (1 + x)\leq x$  for all  $x > - 1$ ,  $(iii)$  follows from Lemma A.2, and  $(iv)$  follows from  $\lambda = 1 + \frac{2}{T}$  (Lemma A.4), noting that  $T\geq r$ , and taking  $\lim_{r\to \infty} - r\log \lambda$
From Lemma 4.3 with probability  $\delta /(2m)$ , there will be at least  $N_{i}$  pairs of queries chosen with control set  $i$  with squared distance at least  $M_{i}$ , where
$$
N _ {i} = \left\lfloor \frac {1}{4} (T _ {i} - 1) - \sqrt {\frac {1}{4} T _ {i} \log \frac {2 m}{\delta}} \right\rfloor
$$
$$
M _ {i} = \frac {4}{h _ {i} + 1} \sum_ {\ell \in \overline {{\mathcal {I}}} _ {i}} \mathbb {V} [ X _ {\ell} ]
$$
Gather these  $2\sum_{i=1}^{m-1}N_i$  queries in an ordered sequence  $\mathcal{S}$  and keep paired queries adjacent to each other. The sequence should be ordered such that, for any control sets  $i$  and  $j$ , if  $i < j$ , then queries chosen with  $i$  should appear in the sequence before queries chosen with  $j$ . Denote as  $\widetilde{\mathcal{T}}$  the ordered sequence of iterations at which each of these queries were chosen by the learner where the order corresponds to the order in  $\mathcal{S}$ . Using row and column swaps on  $\mathbf{K}_r$ , construct a new Gram matrix  $\mathbf{K}_s$  such that, for all  $j, \ell \leq 2\sum_{i=1}^{m-1}N_i$ ,
$$
\left[ \mathbf {K} _ {s} \right] _ {j \ell} = \left[ \mathbf {K} _ {r} \right] _ {\widetilde {\mathcal {T}} _ {j} \widetilde {\mathcal {T}} _ {\ell}}.
$$
In other words, we have simply reordered the underlying queries that result in the Gram matrix  $\mathbf{K}_r$  to produce a new Gram matrix  $\mathbf{K}_s$  such that the first  $2\sum_{i=1}^{m-1}N_i$  rows (and columns) correspond to the  $2N_i$  queries, and paired queries (that have at least  $M_i$  squared distance between them) are kept in adjacent rows (and columns). Note that
$$
\left[ \lambda \mathbf {I} + \mathbf {K} _ {r} \right] \tilde {\mathcal {T}} _ {j} \tilde {\mathcal {T}} _ {\ell} = \left[ \lambda \mathbf {I} + \mathbf {K} _ {s} \right] _ {j \ell}
$$
i.e., the same row and column swap operations on  $\lambda \mathbf{I} + \mathbf{K}_r$  result in  $\lambda \mathbf{I} + \mathbf{K}_s$ . Note that swapping the positions of two queries corresponds to a row swap and a column swap in the Gram matrix. We can thus conclude that
$$
\left| \lambda \mathbf {I} + \mathbf {K} _ {r} \right| = \left| \lambda \mathbf {I} + \mathbf {K} _ {s} \right| \tag {8}
$$
since determinants are invariant under an even number of row or column swaps.
Write  $|\lambda \mathbf{I} + \mathbf{K}_s|$  as
$$
| \lambda \mathbf {I} + \mathbf {K} _ {s} | = \left[ \begin{array}{c c} \mathbf {A} _ {1} & \mathbf {B} _ {1} \\ \mathbf {C} _ {1} & \mathbf {D} _ {1} \end{array} \right]
$$
where  $\mathbf{A}_1$  is a  $2\times 2$  matrix. Since  $\mathbf{A}_1$  is invertible,
$$
\left| \lambda \mathbf {I} + \mathbf {K} _ {s} \right| = \left| \mathbf {A} _ {1} \right| \left| \mathbf {D} _ {1} - \mathbf {C} _ {1} \mathbf {A} _ {1} ^ {- 1} \mathbf {B} _ {1} \right|
$$
where  $\mathbf{D}_1 - \mathbf{C}_1\mathbf{A}_1^{-1}\mathbf{B}_1$  is the Schur complement of  $\mathbf{A}_1$ . Observe that
$$
\begin{array}{l} \mathbf {D} _ {1} - \mathbf {C} _ {1} \mathbf {A} _ {1} ^ {- 1} \mathbf {B} _ {1} = \lambda \mathbf {I} + \mathbf {K} _ {s - 2} - \mathbf {k} _ {2, s - 2} ^ {\top} (\mathbf {K} _ {2} + \lambda \mathbf {I}) ^ {- 1} \mathbf {k} _ {2, s - 2} \\ = \lambda \mathbf {I} + \hat {\mathbf {K}} _ {s - 2} \\ \end{array}
$$
where  $\mathbf{K}_2$  and  $\mathbf{K}_{s - 2}$  are the prior covariance matrices of the first 2 queries and last  $r - 2$  queries respectively,  $\mathbf{k}_{2,s - 2}$  is the prior covariance between the first 2 queries and the last  $r - 2$  queries, and  $\hat{\mathbf{K}}_{s - 2}$  is the posterior covariance matrix of the last  $r - 2$  queries conditioned on observations at the first 2 queries. We can repeat this decomposition:
$$
\lambda \mathbf {I} + \hat {\mathbf {K}} _ {s - 2} = \left[ \begin{array}{c c} \mathbf {A} _ {2} & \mathbf {B} _ {2} \\ \mathbf {C} _ {2} & \mathbf {D} _ {2} \end{array} \right]
$$
$$
\left| \lambda \mathbf {I} + \hat {\mathbf {K}} _ {s - 2} \right| = | \mathbf {A} _ {2} | \left| \mathbf {D} _ {2} - \mathbf {C} _ {2} \mathbf {A} _ {2} ^ {- 1} \mathbf {B} _ {2} \right|
$$
$$
\mathbf {D} _ {2} - \mathbf {C} _ {2} \mathbf {A} _ {2} ^ {- 1} \mathbf {B} _ {2} = \lambda \mathbf {I} + \hat {\mathbf {K}} _ {s - 4}
$$
where  $\tilde{\mathbf{K}}_{s - 4}$  is the posterior covariance matrix of the last  $r - 4$  queries conditioned on observations at the first 4 queries, by the quotient property of the Schur complement [8]. Define  $N\coloneqq \sum_{i = 1}^{m - 1}N_{i}$ . Performing this decomposition  $N$  times yields
$$
\left| \lambda \mathbf {I} + \mathbf {K} _ {s} \right| = \prod_ {j = 1} ^ {N} \left| \mathbf {A} _ {j} \right| \left| \lambda \mathbf {I} + \hat {\mathbf {K}} _ {s - 2 N} \right|
$$
where each  $\mathbf{A}_j$  is the  $2\times 2$  posterior covariance matrix of a pair of queries chosen with some control set  $i$  that have least  $M_{i}$  squared distance between them conditioned on observations at the first  $2(j - 1)$  queries in the sequence, plus  $\lambda \mathbf{I}$ . From (7) and (8),
$$
\sum_ {t = 1} ^ {r} \sigma_ {t - 1} (\mathbf {x} _ {t}) \geq \lambda \left(\sum_ {j = 1} ^ {N} \log | \mathbf {A} _ {j} | + \log | \lambda \mathbf {I} + \hat {\mathbf {K}} _ {s - 2 N} | - 2\right). \tag {9}
$$
Let  $\hat{\mathbf{x}}_j$  and  $\hat{\mathbf{x}}_j'$  refer to the pair of queries associated with  $\mathbf{A}_j$ , and  $\tilde{k}_j$  to the posterior covariance function conditioned on observations at the first  $2(j - 1)$  queries in the sequence. Define  $\mathbf{k}_j$  and  $\mathbf{k}_j'$  as the  $\mathbb{R}^{2(j - 1)}$  vectors of the prior covariance between the first  $2(j - 1)$  queries in the sequence and  $\hat{\mathbf{x}}_j$  and  $\hat{\mathbf{x}}_j'$  respectively. Further define  $\mathbf{M}_j \coloneqq \mathbf{K}_{2(j - 1)} + \lambda \mathbf{I}$ . Use  $\langle \mathbf{u}, \mathbf{v} \rangle_{\mathbf{M}}$  to denote  $\mathbf{u}^\top \mathbf{M} \mathbf{v}$ , and  $\|\mathbf{u}\|_{\mathbf{M}}$  to denote  $\sqrt{\langle \mathbf{u}, \mathbf{u} \rangle_{\mathbf{M}}}$ . Each  $|\mathbf{A}_j|$  can be lower bounded as
$$
\begin{array}{l} \left| \mathbf {A} _ {j} \right| = (\tilde {k} _ {j} (\hat {\mathbf {x}} _ {j}, \hat {\mathbf {x}} _ {j}) + \lambda) (\tilde {k} _ {j} (\hat {\mathbf {x}} _ {j} ^ {\prime}, \hat {\mathbf {x}} _ {j} ^ {\prime}) + \lambda) - \tilde {k} _ {j} (\hat {\mathbf {x}} _ {j}, \hat {\mathbf {x}} _ {j} ^ {\prime}) ^ {2} \\ = \tilde {k} _ {j} (\hat {\mathbf {x}} _ {j}, \hat {\mathbf {x}} _ {j}) \tilde {k} _ {j} (\hat {\mathbf {x}} _ {j} ^ {\prime}, \hat {\mathbf {x}} _ {j} ^ {\prime}) + \lambda \tilde {k} _ {j} (\hat {\mathbf {x}} _ {j}, \hat {\mathbf {x}} _ {j}) + \lambda \tilde {k} _ {j} (\hat {\mathbf {x}} _ {j} ^ {\prime}, \hat {\mathbf {x}} _ {j} ^ {\prime}) + \lambda^ {2} - \tilde {k} _ {j} (\hat {\mathbf {x}} _ {j}, \hat {\mathbf {x}} _ {j} ^ {\prime}) ^ {2} \\ \stackrel {(i)} {=} \left(1 - \| \mathbf {k} _ {j} \| _ {\mathbf {M} _ {j} ^ {- 1}} ^ {2}\right) \left(1 - \left\| \mathbf {k} _ {j} ^ {\prime} \right\| _ {\mathbf {M} _ {j} ^ {- 1}} ^ {2}\right) + \lambda \left(1 - \| \mathbf {k} _ {j} \| _ {\mathbf {M} _ {j} ^ {- 1}} ^ {2}\right) + \lambda \left(1 - \left\| \mathbf {k} _ {j} ^ {\prime} \right\| _ {\mathbf {M} _ {j} ^ {- 1}} ^ {2}\right) + \lambda^ {2} \\ - \left(k \left(\hat {\mathbf {x}} _ {j}, \hat {\mathbf {x}} _ {j} ^ {\prime}\right) - \left\langle \mathbf {k} _ {j}, \mathbf {k} _ {j} ^ {\prime} \right\rangle_ {\mathbf {M} _ {j} ^ {- 1}}\right) ^ {2} \\ = \left(1 - \| \mathbf {k} _ {j} \| _ {\mathbf {M} _ {j} ^ {- 1}} ^ {2}\right) \left(1 - \left\| \mathbf {k} _ {j} ^ {\prime} \right\| _ {\mathbf {M} _ {j} ^ {- 1}} ^ {2}\right) + \lambda \left(1 - \| \mathbf {k} _ {j} \| _ {\mathbf {M} _ {j} ^ {- 1}} ^ {2}\right) + \lambda \left(1 - \left\| \mathbf {k} _ {j} ^ {\prime} \right\| _ {\mathbf {M} _ {j} ^ {- 1}} ^ {2}\right) + \lambda^ {2} \\ - k \left(\hat {\mathbf {x}} _ {j}, \hat {\mathbf {x}} _ {j} ^ {\prime}\right) ^ {2} + 2 k \left(\hat {\mathbf {x}} _ {j}, \hat {\mathbf {x}} _ {j} ^ {\prime}\right) \left\langle \mathbf {k} _ {j}, \mathbf {k} _ {j} ^ {\prime} \right\rangle_ {\mathbf {M} _ {j} ^ {- 1}} - \left\langle \mathbf {k} _ {j}, \mathbf {k} _ {j} ^ {\prime} \right\rangle_ {\mathbf {M} _ {j} ^ {- 1}} ^ {2} \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(i i)} {\geq} \left(1 - \| \mathbf {k} _ {j} \| _ {\mathbf {M} _ {j} ^ {- 1}} ^ {2}\right) \left(1 - \left\| \mathbf {k} _ {j} ^ {\prime} \right\| _ {\mathbf {M} _ {j} ^ {- 1}} ^ {2}\right) + \lambda \left(1 - \| \mathbf {k} _ {j} \| _ {\mathbf {M} _ {j} ^ {- 1}} ^ {2}\right) + \lambda \left(1 - \left\| \mathbf {k} _ {j} ^ {\prime} \right\| _ {\mathbf {M} _ {j} ^ {- 1}} ^ {2}\right) + \lambda^ {2} \\ - k \left(\hat {\mathbf {x}} _ {j}, \hat {\mathbf {x}} _ {j} ^ {\prime}\right) ^ {2} - 2 k \left(\hat {\mathbf {x}} _ {j}, \hat {\mathbf {x}} _ {j} ^ {\prime}\right) \left\| \mathbf {k} _ {j} \right\| _ {\mathbf {M} _ {j} ^ {- 1}} \left\| \mathbf {k} _ {j} ^ {\prime} \right\| _ {\mathbf {M} _ {j} ^ {- 1}} - \left\| \mathbf {k} _ {j} \right\| _ {\mathbf {M} _ {j} ^ {- 1}} ^ {2} \left\| \mathbf {k} _ {j} ^ {\prime} \right\| _ {\mathbf {M} _ {j} ^ {- 1}} ^ {2} \\ = 1 - \left\| \mathbf {k} _ {j} \right\| _ {\mathbf {M} _ {j} ^ {- 1}} ^ {2} - \left\| \mathbf {k} _ {j} ^ {\prime} \right\| _ {\mathbf {M} _ {j} ^ {- 1}} ^ {2} + \lambda \left(1 - \left\| \mathbf {k} _ {j} \right\| _ {\mathbf {M} _ {j} ^ {- 1}} ^ {2}\right) + \lambda \left(1 - \left\| \mathbf {k} _ {j} ^ {\prime} \right\| _ {\mathbf {M} _ {j} ^ {- 1}} ^ {2}\right) + \lambda^ {2} \\ - k \left(\hat {\mathbf {x}} _ {j}, \hat {\mathbf {x}} _ {j} ^ {\prime}\right) ^ {2} - 2 k \left(\hat {\mathbf {x}} _ {j}, \hat {\mathbf {x}} _ {j} ^ {\prime}\right) \left\| \mathbf {k} _ {j} \right\| _ {\mathbf {M} _ {j} ^ {- 1}} \left\| \mathbf {k} _ {j} ^ {\prime} \right\| _ {\mathbf {M} _ {j} ^ {- 1}} \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(i i i)} {\geq} 1 - \left\| \mathbf {k} _ {j} \right\| _ {\mathbf {M} _ {j} ^ {- 1}} ^ {2} - \left\| \mathbf {k} _ {j} ^ {\prime} \right\| _ {\mathbf {M} _ {j} ^ {- 1}} ^ {2} + \lambda \left(1 - \left\| \mathbf {k} _ {j} \right\| _ {\mathbf {M} _ {j} ^ {- 1}} ^ {2}\right) + \lambda \left(1 - \left\| \mathbf {k} _ {j} ^ {\prime} \right\| _ {\mathbf {M} _ {j} ^ {- 1}} ^ {2}\right) + \lambda^ {2} \\ - k (\hat {\mathbf {x}} _ {j}, \hat {\mathbf {x}} _ {j} ^ {\prime}) ^ {2} - 2 k (\hat {\mathbf {x}} _ {j}, \hat {\mathbf {x}} _ {j} ^ {\prime}) \\ = \lambda^ {2} - 1 + (\lambda + 1) \left(1 - \| \mathbf {k} _ {j} \| _ {\mathbf {M} _ {j} ^ {- 1}} ^ {2}\right) + (\lambda + 1) \left(1 - \left\| \mathbf {k} _ {j} ^ {\prime} \right\| _ {\mathbf {M} _ {j} ^ {- 1}} ^ {2}\right) - 2 k \left(\hat {\mathbf {x}} _ {j}, \hat {\mathbf {x}} _ {j} ^ {\prime}\right) - k \left(\hat {\mathbf {x}} _ {j}, \hat {\mathbf {x}} _ {j} ^ {\prime}\right) ^ {2} \\ = \lambda^ {2} - 1 + (\lambda + 1) \left(\tilde {k} _ {j} \left(\hat {\mathbf {x}} _ {j}, \hat {\mathbf {x}} _ {j}\right) + \tilde {k} _ {j} \left(\hat {\mathbf {x}} _ {j} ^ {\prime}, \hat {\mathbf {x}} _ {j} ^ {\prime}\right)\right) - 2 k \left(\hat {\mathbf {x}} _ {j}, \hat {\mathbf {x}} _ {j} ^ {\prime}\right) - k \left(\hat {\mathbf {x}} _ {j}, \hat {\mathbf {x}} _ {j} ^ {\prime}\right) ^ {2} \\ \end{array}
$$
where  $(i)$  follows from our assumption that  $k(\mathbf{x},\mathbf{x}) = 1$ ,  $(ii)$  follows from the Cauchy-Schwarz inequality, and  $(iii)$  follows since  $1 - \left\| \mathbf{k}_j\right\|_{\mathbf{M}_j^{-1}}^2\leq 1$  and  $1 - \left\| \mathbf{k}_j'\right\|_{\mathbf{M}_j^{-1}}^2\leq 1$ .
Define  $S_{i} \coloneqq \sum_{\ell = 1}^{i} N_{i}$  and  $\tilde{v}_{i} \coloneqq \min_{S_{i - 1} + 1 \leq j \leq S_{i}} \frac{1}{2} (\tilde{k}_{j}(\hat{\mathbf{x}}_{j}, \hat{\mathbf{x}}_{j}) + \tilde{k}_{j}(\hat{\mathbf{x}}_{j}', \hat{\mathbf{x}}_{j}'))$ . Substituting this result into (9),
$$
\begin{array}{l} \sum_ {t = 1} ^ {r} \sigma_ {t - 1} (\mathbf {x} _ {t}) \geq \lambda \left(\sum_ {j = 1} ^ {N} \log \left(\lambda^ {2} - 1 + (\lambda + 1) \left(\tilde {k} _ {j} (\hat {\mathbf {x}} _ {j}, \hat {\mathbf {x}} _ {j}) + \tilde {k} _ {j} (\hat {\mathbf {x}} _ {j} ^ {\prime}, \hat {\mathbf {x}} _ {j} ^ {\prime})\right) - 2 k (\hat {\mathbf {x}} _ {j}, \hat {\mathbf {x}} _ {j} ^ {\prime}) - k (\hat {\mathbf {x}} _ {j}, \hat {\mathbf {x}} _ {j} ^ {\prime}) ^ {2}\right) \right. \\ + \log \left| \lambda \mathbf {I} + \hat {\mathbf {K}} _ {s - 2 N} \right| - 2) \\ = \lambda \left(\sum_ {i = 1} ^ {m - 1} \sum_ {j = S _ {i - 1} + 1} ^ {S _ {i}} \log \left(\lambda^ {2} - 1 + (\lambda + 1) \left(\tilde {k} _ {j} (\hat {\mathbf {x}} _ {j}, \hat {\mathbf {x}} _ {j}) + \tilde {k} _ {j} (\hat {\mathbf {x}} _ {j} ^ {\prime}, \hat {\mathbf {x}} _ {j} ^ {\prime})\right) - 2 k (\hat {\mathbf {x}} _ {j}, \hat {\mathbf {x}} _ {j} ^ {\prime}) \right. \right. \\ \end{array}
$$
$$
\begin{array}{l} \left. \left. - k \left(\hat {\mathbf {x}} _ {j}, \hat {\mathbf {x}} _ {j} ^ {\prime}\right) ^ {2}\right) + \log \left| \lambda \mathbf {I} + \hat {\mathbf {K}} _ {s - 2 N} \right| - 2\right) \\ \geq \lambda \left(\sum_ {i = 1} ^ {m - 1} \sum_ {j = S _ {i - 1} + 1} ^ {S _ {i}} \log \left(\lambda^ {2} - 1 + 2 (\lambda + 1) \tilde {v} _ {i} - 2 k \left(\hat {\mathbf {x}} _ {j}, \hat {\mathbf {x}} _ {j} ^ {\prime}\right) - k \left(\hat {\mathbf {x}} _ {j}, \hat {\mathbf {x}} _ {j} ^ {\prime}\right) ^ {2}\right) \right. \\ + \log \left| \lambda \mathbf {I} + \hat {\mathbf {K}} _ {s - 2 N} \right| - 2) \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(i)} {\geq} \lambda \left(\sum_ {i = 1} ^ {m - 1} \sum_ {j = S _ {i - 1} + 1} ^ {S _ {i}} \log \left(\lambda^ {2} - 1 + 2 (\lambda + 1) \tilde {v} _ {i} - 2 k \left(\sqrt {M _ {i}}\right) - k \left(\sqrt {M _ {i}}\right) ^ {2}\right) \right. \\ + \log \left| \lambda \mathbf {I} + \hat {\mathbf {K}} _ {s - 2 N} \right| - 2) \\ \end{array}
$$
$$
\begin{array}{l} = \lambda \left(\sum_ {i = 1} ^ {m - 1} N _ {i} \log \left(\lambda^ {2} - 1 + 2 (\lambda + 1) \tilde {v} _ {i} - 2 k \left(\sqrt {M _ {i}}\right) - k \left(\sqrt {M _ {i}}\right) ^ {2}\right) \right. \\ + \log \left| \lambda \mathbf {I} + \hat {\mathbf {K}} _ {s - 2 N} \right| - 2) \\ \end{array}
$$
$$
\begin{array}{l} = \lambda \left(\sum_ {i = 1} ^ {m - 1} N _ {i} \log \left(V _ {i} - 2 k \left(\sqrt {M _ {i}}\right) - k \left(\sqrt {M _ {i}}\right) ^ {2}\right) + W\right) (10) \\ =: \mathcal {L} (11) \\ \end{array}
$$
where  $V_{i} \coloneqq \lambda^{2} - 1 + 2(\lambda + 1)\tilde{v}_{i}$  and  $W \coloneqq \log \left|\lambda \mathbf{I} + \hat{\mathbf{K}}_{s - 2N}\right| - 2$ , ( $i$ ) follows from our assumption that the kernel  $k$  is stationary and can be written in a single argument form as  $k(||\mathbf{x} - \mathbf{x}'||) = k(\mathbf{x},\mathbf{x}')$  and the fact that every pair of queries in  $S$  chosen with control set  $i$  has squared distance at least  $M_{i}$ . Starting from (4) in the proof of Theorem 4.1 except replacing the probabilities of all events with  $2m / \delta$ ,
$$
R _ {T} \leq \left(\sum_ {i = 1} ^ {m} c _ {i} \left(2 \beta_ {T}\right) \left(2 \sum_ {t \in \widetilde {T} _ {i}} \sigma_ {t - 1} (\mathbf {x} _ {t}) + 4 \log \frac {2 m}{\delta} + 8 \log (4) + 1\right)\right) + c _ {m} \sum_ {t = 1} ^ {T} \epsilon_ {t}
$$
$$
\begin{array}{l} = 2 \beta_ {T} \left(c _ {m} \left(2 \sum_ {t \in \tilde {T} _ {m}} \sigma_ {t - 1} (\mathbf {x} _ {t}) + 4 \log \frac {2 m}{\delta} + 8 \log (4) + 1\right) \right. \\ \left. + \sum_ {i = 1} ^ {m - 1} c _ {i} \left(2 \sum_ {t \in \widetilde {T} _ {i}} \sigma_ {t - 1} (\mathbf {x} _ {t}) + 4 \log \frac {2 m}{\delta} + 8 \log (4) + 1\right)\right) + c _ {m} \sum_ {t = 1} ^ {T} \epsilon_ {t} \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(i)} {\leq} 2 \beta_ {T} \left(c _ {m} \left(2 \sum_ {t \in \tilde {T} _ {m}} \sigma_ {t - 1} (\mathbf {x} _ {t}) + 4 \log \frac {2 m}{\delta} + 8 \log (4) + 1\right) \right. \\ \left. + \sum_ {i = 1} ^ {m - 1} c _ {i} \left(2 \sqrt {4 (T _ {i} + 2) \gamma_ {T _ {i}} (\tilde {\mathcal {X}} _ {i})} + 4 \log \frac {2 m}{\delta} + 8 \log (4) + 1\right)\right) + c _ {m} \sum_ {t = 1} ^ {T} \epsilon_ {t} \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(i i)} {\leq} 2 \beta_ {T} \left(c _ {m} \left(2 \sqrt {4 (T + 2) \gamma_ {T} (\mathcal {X})} - \sum_ {t = 1} ^ {r} \sigma_ {t - 1} (\mathbf {x} _ {t}) + 4 \log \frac {2 m}{\delta} + 8 \log (4) + 1\right) \right. \\ \left. + \sum_ {i = 1} ^ {m - 1} c _ {i} \left(2 \sqrt {4 (T _ {i} + 2) \gamma_ {T _ {i}} (\tilde {\mathcal {X}} _ {i})} + 4 \log \frac {2 m}{\delta} + 8 \log (4) + 1\right)\right) + c _ {m} \sum_ {t = 1} ^ {T} \epsilon_ {t} \\ \end{array}
$$
$$
\stackrel {(i i i)} {\leq} 2 \beta_ {T} \left(c _ {m} \left(2 \sqrt {4 (T + 2) \gamma_ {T} (\mathcal {X})} - \mathcal {L} + 4 \log \frac {2 m}{\delta} + 8 \log (4) + 1\right) \right.
$$
$$
\begin{array}{l} \left. + \sum_ {i = 1} ^ {m - 1} c _ {i} \left(2 \sqrt {4 (T _ {i} + 2) \gamma_ {T _ {i}} (\tilde {\mathcal {X}} _ {i})} + 4 \log \frac {2 m}{\delta} + 8 \log (4) + 1\right)\right) + c _ {m} \sum_ {t = 1} ^ {T} \epsilon_ {t} \\ = \mathcal {O} \left(\left(B + \sqrt {\gamma_ {T} (\mathcal {X}) + \log \frac {2 m}{\delta}}\right) \left(c _ {m} \left(\sqrt {T \gamma_ {T} (\mathcal {X})} - \mathcal {L} + \log \frac {2 m}{\delta}\right) \right. \right. \\ \left. \left. + \sum_ {i = 1} ^ {m - 1} c _ {i} \left(\sqrt {T _ {i} \gamma_ {T _ {i}} (\widetilde {\mathcal {X}} _ {i})} + \log \frac {2 m}{\delta}\right)\right)\right) + c _ {m} \sum_ {t = 1} ^ {T} \epsilon_ {t} \\ \end{array}
$$
where  $(i)$  follows from Lemma A.3,  $(ii)$  follows from Lemma A.3 again and the resulting inequality  $\sum_{t=1}^{r}\sigma_{t-1}(\mathbf{x}_t) + \sum_{t=r+1}^{T}\sigma_{t-1}(\mathbf{x}_t) \leq \sqrt{4(T+2)\gamma_T(\mathcal{X})}$ , and  $(iii)$  follows from substituting in (11). A union bound over the events of the single application of Lemma A.4,  $m$  applications of Lemma A.5, and  $m-1$  applications of Lemma 4.3 yields the desired  $1-\delta$  probability bound, which completes the proof.
A.6 Other Lemmas
Lemma A.2 ([7] Lemma 3). Let  $(\mathbf{x}_t)_{t=1}^T$  be a sequence of queries selected by some algorithm. Then, the mutual information  $I(\mathbf{y}_{1:T}; \mathbf{f}_{1:T})$  between the noisy observations  $\mathbf{y}_{1:T}$  and the function values  $\mathbf{f}_{1:T}$  at the queries is given by
$$
I \left(\mathbf {y} _ {1: T}; \mathbf {f} _ {1: T}\right) = \frac {1}{2} \log \left| \mathbf {I} + \lambda^ {- 1} \mathbf {K} _ {t} \right| = \frac {1}{2} \sum_ {t = 1} ^ {T} \log \left(1 + \lambda^ {- 1} \sigma_ {t - 1} ^ {2} (\mathbf {x} _ {t})\right).
$$
Lemma A.3 ([7] Lemma 4). Let  $(\mathbf{x}_t)_{t=1}^T$  be a sequence of queries selected by some algorithm. Then
$$
\sum_ {t = 1} ^ {T} \sigma_ {t - 1} (\mathbf {x} _ {t}) \leq \sqrt {4 (T + 2) \gamma_ {T} (\mathcal {X})}.
$$
Lemma A.4 ([7] Theorem 2). Let  $\beta_{t} \coloneqq B + \sigma \sqrt{2(\gamma_{t-1}(\mathcal{X}) + 1 + \log(1/\delta))}$  where  $B$  is the upper bound of the RKHS norm of  $f$ . With probability at least  $1 - \delta$ , for all  $\mathbf{x} \in \mathcal{X}$  and  $t \geq 1$ ,
$$
\left| \mu_ {t - 1} (\mathbf {x}) - f (\mathbf {x}) \right| \leq \beta_ {t} \sigma_ {t - 1} (\mathbf {x})
$$
where  $\mu_{t - 1}$  and  $\sigma_{t - 1}$  are defined in (1) with  $\lambda = 1 + \eta$  and  $\eta \coloneqq 2 / T$ .
Lemma A.5 ([16] Lemma 3). Let  $X_{t}$  be any non-negative stochastic process adapted to a filtration  $\{\mathcal{F}_t\}$ , and define  $m_t \coloneqq \mathbb{E}[X_t|\mathcal{F}_{t - 1}]$ . Further assume that  $X_{t} \leq b_{t}$  for a fixed, non-decreasing sequence  $(b_{t})_{t\geq 1}$ . If  $b_{T} \geq 1$ , with probability at least  $1 - \delta$ , for any  $T \geq 1$
$$
\sum_ {t = 1} ^ {T} m _ {t} \leq 2 \sum_ {t = 1} ^ {T} X _ {t} + 4 b _ {T} \log \frac {1}{\delta} + 8 b _ {T} \log (4 b _ {T}) + 1
$$
B Comparison to Naive Baselines