SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[33] S. Sussex, A. Makarova, and A. Krause. Model-based causal Bayesian optimization. In Proc. ICLR, 2023.
[34] P. M. Swamidass. Encyclopedia of production and manufacturing management. Springer Science & Business Media, 2000.
[35] K. Swersky, J. Snoek, and R. P. Adams. Multi-task Bayesian optimization. In Proc. NeurIPS, pages 2004–2012, 2013.
[36] S. Takeno, H. Fukuoka, Y. Tsukada, T. Koyama, M. Shiga, I. Takeuchi, and M. Karasuyama. Multi-fidelity Bayesian optimization with max-value entropy search and its parallelization. In Proc. ICML, pages 9334-9345, 2020.
[37] S. S. Tay, C. S. Foo, U. Daisuke, R. Leong, and B. K. H. Low. Efficient distributionally robust Bayesian optimization with worst-case sensitivity. In Proc. ICML, pages 21180-21204, 2022.
[38] S. Toscano-Palmerin and P. I. Frazier. Bayesian optimization with expensive integrands. SIAM Journal on Optimization, 32(2):417-444, 2022.
[39] R. Turner, D. Eriksson, M. McCourt, J. Kiili, E. Laaksonen, Z. Xu, and I. Guyon. Bayesian optimization is superior to random search for machine learning hyperparameter tuning: Analysis of the black-box optimization challenge 2020. In Proc. NeurIPS 2020 Competition and Demonstration Track, pages 3-26, 2021.
[40] P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weckesser, J. Bright, S. J. van der Walt, M. Brett, J. Wilson, K. J. Millman, N. Mayorov, A. R. J. Nelson, E. Jones, R. Kern, E. Larson, C. J. Carey, I. Polat, Y. Feng, E. W. Moore, J. VanderPlas, D. Laxalde, J. Perktold, R. Cirmrnan, I. Henriksen, E. A. Quintero, C. R. Harris, A. M. Archibald, A. H. Ribeiro, F. Pedregosa, P. van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-272, 2020.
[41] Z. Wang and S. Jegelka. Max-value entropy search for efficient Bayesian optimization. In Proc. ICML, pages 3627-3635, 2017.
[42] C. K. Williams and C. E. Rasmussen. Gaussian Processes for Machine Learning. MIT Press, Cambridge, MA, 2006.
A.1 Proof of Theorem 4.1
$$
R _ {T} \leq \mathcal {O} \left(\left(B + \sqrt {\gamma_ {T} (\mathcal {X}) + \log \frac {m + 1}{\delta}}\right) \left(\sum_ {i = 1} ^ {m} c _ {i} \left(\sqrt {T _ {i} \gamma_ {T _ {i}} (\tilde {\mathcal {X}} _ {i})} + \log \frac {m + 1}{\delta}\right)\right)\right) + c _ {m} \sum_ {t = 1} ^ {T} \epsilon_ {t}.
$$
$$
\begin{array}{l} R _ {T} := \sum_ {t = 1} ^ {T} c _ {i _ {t}} \left(\mathbb {E} \left[ f \left([ \mathbf {x} ^ {i ^ {*}}, \mathbf {X} ^ {- i ^ {*}} ]\right) \right] - \mathbb {E} \left[ f \left([ \mathbf {x} ^ {i _ {t}}, \mathbf {X} ^ {- i _ {t}} ]\right) \right]\right) \\ \leq \sum_ {t = 1} ^ {T} c _ {i _ {t}} \left(\mathbb {E} \left[ u _ {t - 1} \left([ \mathbf {x} ^ {i ^ {*}}, \mathbf {X} ^ {- i ^ {*}} ]\right) \right] - \mathbb {E} \left[ f \left([ \mathbf {x} ^ {i _ {t}}, \mathbf {X} ^ {- i _ {t}} ]\right) \right]\right) \\ \stackrel {(i)} {\leq} \sum_ {t = 1} ^ {T} c _ {i _ {t}} \left(\mathbb {E} \left[ u _ {t - 1} \left([ \mathbf {x} ^ {i _ {t}}, \mathbf {X} ^ {- i _ {t}} ]\right) \right] - \mathbb {E} \left[ f \left([ \mathbf {x} ^ {i _ {t}}, \mathbf {X} ^ {- i _ {t}} ]\right) \right] + \epsilon_ {t}\right) \\ \leq \sum_ {t = 1} ^ {T} c _ {i _ {t}} \left(\mathbb {E} \left[ u _ {t - 1} \left(\left[ \mathbf {x} ^ {i _ {t}}, \mathbf {X} ^ {- i _ {t}} \right]\right) \right] - \mathbb {E} \left[ f \left(\left[ \mathbf {x} ^ {i _ {t}}, \mathbf {X} ^ {- i _ {t}} \right]\right) \right]\right) + c _ {m} \sum_ {t = 1} ^ {T} \epsilon_ {t} (3) \\ = \sum_ {t = 1} ^ {T} c _ {i _ {t}} \left(\mathbb {E} \left[ u _ {t - 1} \left([ \mathbf {x} ^ {i _ {t}}, \mathbf {X} ^ {- i _ {t}} ]\right) - f \left([ \mathbf {x} ^ {i _ {t}}, \mathbf {X} ^ {- i _ {t}} ]\right) \right]\right) + c _ {m} \sum_ {t = 1} ^ {T} \epsilon_ {t} \\ = \left(\sum_ {i = 1} ^ {m} c _ {i} \sum_ {t \in \widetilde {T} _ {i}} \mathbb {E} \left[ u _ {t - 1} \left(\left[ \mathbf {x} ^ {i _ {t}}, \mathbf {X} ^ {- i _ {t}} \right]\right) - f \left(\left[ \mathbf {x} ^ {i _ {t}}, \mathbf {X} ^ {- i _ {t}} \right]\right) \right]\right) + c _ {m} \sum_ {t = 1} ^ {T} \epsilon_ {t} \\ \stackrel {(i i)} {\leq} \left(\sum_ {i = 1} ^ {m} c _ {i} \left(2 \beta_ {T}\right) \sum_ {t \in \widetilde {T} _ {i}} \mathbb {E} \left[ \sigma_ {t - 1} \left(\left[ \mathbf {x} ^ {i _ {t}}, \mathbf {X} ^ {- i _ {t}} \right]\right) \right]\right) + c _ {m} \sum_ {t = 1} ^ {T} \epsilon_ {t} \\ \leq \binom {i i i} {i = 1} c _ {i} (2 \beta_ {T}) \left(2 \sum_ {t \in \tilde {T} _ {i}} \sigma_ {t - 1} (\mathbf {x} _ {t}) + 4 \log \frac {m + 1}{\delta} + 8 \log (4) + 1\right) + c _ {m} \sum_ {t = 1} ^ {T} \epsilon_ {t} (4) \\ \stackrel {(i v)} {\leq} \left(\sum_ {i = 1} ^ {m} c _ {i} (2 \beta_ {T}) \left(2 \sqrt {4 (T _ {i} + 2) \gamma_ {T _ {i}} (\tilde {\mathcal {X}} _ {i})} + 4 \log \frac {m + 1}{\delta} + 8 \log (4) + 1\right)\right) + c _ {m} \sum_ {t = 1} ^ {T} \epsilon_ {t} \\ = \mathcal {O} \left(\sum_ {i = 1} ^ {m} c _ {i} \beta_ {T} \left(\sqrt {T _ {i} \gamma_ {T _ {i}} (\tilde {\mathscr {X}} _ {i})} + \log \frac {m + 1}{\delta}\right)\right) + c _ {m} \sum_ {t = 1} ^ {T} \epsilon_ {t} (5) \\ = \mathcal {O} \left(\left(B + \sqrt {\gamma_ {T} (\mathcal {X}) + \log \frac {m + 1}{\delta}}\right) \left(\sum_ {i = 1} ^ {m} c _ {i} \left(\sqrt {T _ {i} \gamma_ {T _ {i}} (\tilde {\mathcal {X}} _ {i})} + \log \frac {m + 1}{\delta}\right)\right)\right) + c _ {m} \sum_ {t = 1} ^ {T} \epsilon_ {t} \\ \end{array}
$$
where  $\widetilde{T}_i$  is the ordered sequence of iterations at which control set  $i$  is chosen,  $(i)$  follows from the algorithm's choice of  $\mathbf{x}^{it}$ ,  $(ii)$  follows from Lemma A.4 with probability  $\delta / (m + 1)$ ,  $(iii)$  follows from Lemma A.1 with probability  $\delta / (m + 1)$  applied once for each control set  $i$ , and  $(iv)$  follows from Lemma A.3 and the definition of  $\widetilde{\mathcal{X}}_i$  as the feasible set for control set  $i$ . A union bound over the  $m + 1$  events comprising the  $m$  applications of Lemma A.1 and single application of Lemma A.4 yields the desired  $1 - \delta$  probability bound.
A.2 Proof of Lemma A.1
Lemma A.1. Let  $k(\mathbf{x}, \mathbf{x}) = 1$  and let  $\widetilde{T}_i$  be the ordered sequence of iterations at which control set  $i$  is chosen by UCB-CVS. For any  $i \in [m]$ , with probability at least  $1 - \delta$ ,
$$
\sum_ {t \in \widetilde {T} _ {i}} \mathbb {E} \left[ \sigma_ {t - 1} \left(\left[ \mathbf {x} ^ {i _ {t}}, \mathbf {X} ^ {- i _ {t}} \right]\right) \right] \leq 2 \sum_ {t \in \widetilde {T} _ {i}} \sigma_ {t - 1} (\mathbf {x} _ {t}) + 4 \log \frac {1}{\delta} + 8 \log 4 + 1.
$$
Proof. For this proof, define a probability space  $(\Omega, \mathcal{F}, \mathcal{P})$  and a filtration  $\mathbb{F} = \{\mathcal{F}_t\}_{t=1}^{\infty}$ , where  $\mathcal{F}_t := \sigma(i_1, \mathbf{x}^{i_1}, \mathbf{x}^{-i_1}, y_1, i_2, \mathbf{x}^{i_2}, \mathbf{x}^{-i_2}, y_2, \dots, i_t, \mathbf{x}^{i_t}, \mathbf{x}^{-i_t}, y_t)$ , the sigma-algebra generated by all the random variables in the BO procedure known by the end of iteration  $t$ .
In advance of proving this result, it should be clarified that, in the main paper and all proofs excluding that of this lemma,  $\mathbb{E}\big[\sigma_{t - 1}([\mathbf{x}^{i_t},\mathbf{X}^{-i_t}])\big]$  denotes the quantity obtained by treating  $\sigma_{t - 1}$  and  $\mathbf{x}^{i_t}$  as deterministic and treating  $\mathbf{X}^{-i_t}$  as a random vector distributed according to the probability distribution  $\mathbb{P}^{-i_t}$ . This is for ease of exposition. In the following proof, however, when using the formalism of random processes, what was previously referred to as  $\mathbb{E}\big[\sigma_{t - 1}([\mathbf{x}^{i_t},\mathbf{X}^{-i_t}])\big]$  is actually  $\mathbb{E}[\sigma_{t - 1}(\mathbf{x}_t)\mid \mathcal{F}_{t - 1}]$ . The meaning is equivalent since  $\sigma_{t - 1}$ ,  $i_t$ , and  $\mathbf{x}^{i_t}$  are  $\mathbb{F}$ -predictable, and the only uncertainty about  $\sigma_{t - 1}(\mathbf{x}_t)$  arises from the lack of knowledge about  $\mathbf{x}^{-i_t}$ .
Now we begin the proof proper. Define  $m$  stochastic processes  $\{X_{t}^{(1)}\}_{t = 1}^{\infty}$ ,  $\{X_{t}^{(2)}\}_{t = 1}^{\infty}$ , ...,  $\{X_{t}^{(m)}\}_{t = 1}^{\infty}$ , where, using  $\mathbb{1}[A]$  to denote the indicator function that is equal to 1 when the event  $A$  is true and 0 otherwise,
$$
X _ {t} ^ {(i)} := \sigma_ {t - 1} (\mathbf {x} _ {t}) \cdot \mathbb {1} [ i _ {t} = i ].
$$
Since each  $X_{t}^{(i)}$  is  $\mathcal{F}_t$ -measurable, each stochastic process is adapted to  $\mathbb{F}$ . Next, define
$$
\begin{array}{l} m _ {t} ^ {(i)} := \mathbb {E} \left[ X _ {t} ^ {(i)} \mid \mathcal {F} _ {t - 1} \right] \\ = \mathbb {E} \left[ \sigma_ {t - 1} \left(\mathbf {x} _ {t}\right) \cdot \mathbb {1} \left[ i _ {t} = i \right] \mid \mathcal {F} _ {t - 1} \right] \\ = \mathbb {1} [ i _ {t} = i ] \mathbb {E} [ \sigma_ {t - 1} (\mathbf {x} _ {t}) \mid \mathcal {F} _ {t - 1} ] \\ \end{array}
$$
where the last equality uses the pull-through property since  $i_t$  is  $\mathcal{F}_{t-1}$  measurable. Using Lemma A.5 with  $b_t = 1$  since  $k(\mathbf{x}, \mathbf{x}) = 1$ , with probability at least  $1 - \delta$ , for any  $T \geq 1$ ,
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} m _ {t} ^ {(i)} \leq 2 \sum_ {t = 1} ^ {T} X _ {t} ^ {(i)} + 4 \log \frac {1}{\delta} + 8 \log 4 + 1 \\ \sum_ {t = 1} ^ {T} \mathbb {1} [ i _ {t} = i ] \mathbb {E} [ \sigma_ {t - 1} (\mathbf {x} _ {t}) \mid \mathcal {F} _ {t - 1} ] \leq 2 \sum_ {t = 1} ^ {T} \sigma_ {t - 1} (\mathbf {x} _ {t}) \cdot \mathbb {1} [ i _ {t} = i ] + 4 \log \frac {1}{\delta} + 8 \log 4 + 1 \\ \sum_ {t \in \tilde {T} _ {i}} \mathbb {E} [ \sigma_ {t - 1} (\mathbf {x} _ {t}) \mid \mathcal {F} _ {t - 1} ] \leq 2 \sum_ {t \in \tilde {T} _ {i}} \sigma_ {t - 1} (\mathbf {x} _ {t}) + 4 \log \frac {1}{\delta} + 8 \log 4 + 1 \\ \end{array}
$$
A.3 Proof of Proposition 4.2
Proposition 4.2. If there exists a  $\tilde{\epsilon} >0$  s.t.for all  $i\neq i^{*}$
$$
\epsilon_ {t} \leq \mathbb {E} \left[ f \left(\left[ \mathbf {x} ^ {i ^ {*}}, \mathbf {X} ^ {- i ^ {*}} \right]\right) \right] - \max  _ {\mathbf {x} ^ {i} \in \mathcal {X} ^ {i}} \mathbb {E} \left[ f \left(\left[ \mathbf {x} ^ {i}, \mathbf {X} ^ {- i} \right]\right) \right] - \tilde {\epsilon}
$$
eventually (i.e., the inequality holds for all  $t \geq q$  for some  $q \geq 1$ ), and  $\gamma_T(\mathcal{X}) < \mathcal{O}(\sqrt{T})$ , then, with probability at least  $1 - \delta$ ,  $\lim_{T \to \infty} T_i / T = 0$  for all  $i \neq i^*$  and UCB-CVS incurs a cost-varying cumulative regret that is sublinear in  $T$  by setting  $\beta_t = B + \sigma \sqrt{2(\gamma_{t-1}(\mathcal{X}) + 1 + \log((m+1)/\delta))}$ .
Proof. Define  $\mathbf{x}_t^i := \operatorname{argmax}_{\mathbf{x}^i \in \mathcal{X}^i} \mathbb{E}\left[u_{t-1}([\mathbf{x}^i, \mathbf{X}^{-i}])\right]$ , and  $j_t := \operatorname{argmax}_{i \in [m]} \max_{\mathbf{x}^i \in \mathcal{X}^i} \mathbb{E}\left[u_{t-1}([\mathbf{x}^i, \mathbf{X}^{-i}])\right]$ . Using  $\mathbb{1}[A]$  to denote the indicator function
that is equal to 1 when the event  $A$  is true and 0 otherwise,
$$
\begin{array}{l} T _ {i} \stackrel {(i)} {\leq} \sum_ {t = 1} ^ {T} \mathbb {1} \left[ \mathbb {E} \left[ u _ {t - 1} \left(\left[ \mathbf {x} _ {t} ^ {i}, \mathbf {X} ^ {- i} \right]\right) \right] + \epsilon_ {t} \geq \mathbb {E} \left[ u _ {t - 1} \left(\left[ \mathbf {x} _ {t} ^ {j _ {t}}, \mathbf {X} ^ {- j _ {t}} \right]\right) \right] \right] \\ \stackrel {(i i)} {\leq} \sum_ {t = 1} ^ {T} \mathbb {1} \left[ \mathbb {E} \left[ u _ {t - 1} \left(\left[ \mathbf {x} _ {t} ^ {i}, \mathbf {X} ^ {- i} \right]\right) \right] + \epsilon_ {t} \geq \mathbb {E} \left[ u _ {t - 1} \left(\left[ \mathbf {x} ^ {i ^ {*}}, \mathbf {X} ^ {- i ^ {*}} \right]\right) \right] \right] \\ \leq \sum_ {t = 1} ^ {T} \mathbb {1} \left[ \mathbb {E} \left[ u _ {t - 1} \left(\left[ \mathbf {x} _ {t} ^ {i}, \mathbf {X} ^ {- i} \right]\right) \right] + \epsilon_ {t} \geq \mathbb {E} \left[ f \left(\left[ \mathbf {x} ^ {i ^ {*}}, \mathbf {X} ^ {- i ^ {*}} \right]\right) \right] \right] \\ = \sum_ {t = 1} ^ {T} \mathbb {1} \left[ \mathbb {E} \left[ u _ {t - 1} \left(\left[ \mathbf {x} _ {t} ^ {i}, \mathbf {X} ^ {- i} \right]\right) \right] \geq \mathbb {E} \left[ f \left(\left[ \mathbf {x} ^ {i ^ {*}}, \mathbf {X} ^ {- i ^ {*}} \right]\right) \right] - \epsilon_ {t} \right] \\ \leq q - 1 + \sum_ {t = q} ^ {T} \mathbb {1} \left[ \mathbb {E} \left[ u _ {t - 1} \left(\left[ \mathbf {x} _ {t} ^ {i}, \mathbf {X} ^ {- i} \right]\right) \right] \geq \mathbb {E} \left[ f \left(\left[ \mathbf {x} ^ {i ^ {*}}, \mathbf {X} ^ {- i ^ {*}} \right]\right) \right] - \epsilon_ {t} \right] \\ \leq q - 1 + \sum_ {t = q} ^ {T} \mathbb {1} \left[ \mathbb {E} \left[ u _ {t - 1} \left(\left[ \mathbf {x} _ {t} ^ {i}, \mathbf {X} ^ {- i} \right]\right) \right] \geq \max  _ {\mathbf {x} ^ {i} \in \mathcal {X} ^ {i}} \mathbb {E} \left[ f \left(\left[ \mathbf {x} ^ {i}, \mathbf {x} ^ {- i} \right]\right) \right] + \tilde {\epsilon} \right] \\ \leq q - 1 + \sum_ {t = q} ^ {T} \mathbb {1} \left[ \mathbb {E} \left[ u _ {t - 1} \left(\left[ \mathbf {x} _ {t} ^ {i}, \mathbf {X} ^ {- i} \right]\right) \right] \geq \mathbb {E} \left[ f \left(\left[ \mathbf {x} _ {t} ^ {i}, \mathbf {X} ^ {- i} \right]\right) \right] + \tilde {\epsilon} \right] \\ = q - 1 + \sum_ {t = q} ^ {T} \mathbb {1} \left[ \mathbb {E} \left[ u _ {t - 1} \left(\left[ \mathbf {x} _ {t} ^ {i}, \mathbf {X} ^ {- i} \right]\right) \right] - \mathbb {E} \left[ f \left(\left[ \mathbf {x} _ {t} ^ {i}, \mathbf {X} ^ {- i} \right]\right) \right] \geq \tilde {\epsilon} \right] \\ \leq q - 1 + \frac {1}{\tilde {\epsilon}} \sum_ {t = q} ^ {T} \mathbb {E} \left[ u _ {t - 1} \left(\left[ \mathbf {x} _ {t} ^ {i}, \mathbf {X} ^ {- i} \right]\right) \right] - \mathbb {E} \left[ f \left(\left[ \mathbf {x} _ {t} ^ {i}, \mathbf {X} ^ {- i} \right]\right) \right] \\ \stackrel {(i i i)} {\leq} q - 1 + \mathcal {O} ^ {*} \left(\frac {1}{\tilde {\epsilon}} \sqrt {T - q + 1} \left(B \sqrt {\gamma_ {T - q + 1} (\mathcal {X})} + \gamma_ {T - q + 1} (\mathcal {X})\right)\right) \\ \end{array}
$$
where  $(i)$  follows since control set  $i$  is only played when the condition on the RHS is true,  $(ii)$  follows from the definitions of  $j_{t}$  and  $\mathbf{x}_t^{j_t}$ , and  $(iii)$  follows from the steps from (3) to (5) in the proof of Theorem 4.1, and  $\mathcal{O}^*$  denotes suppressing logarithmic factors. Now dividing both sides by  $T$  and taking the limit as  $T$  goes to infinity,
$$
\lim  _ {T \rightarrow \infty} \frac {T _ {i}}{T} \leq \lim  _ {T \rightarrow \infty} \frac {1}{T} \left(q - 1 + \mathcal {O} ^ {*} \left(\frac {1}{\tilde {\epsilon}} \sqrt {T - q + 1} \left(B \sqrt {\gamma_ {T - q + 1} (\mathcal {X})} + \gamma_ {T - q + 1} (\mathcal {X})\right)\right)\right)
$$
$= 0$
which follows from  $\gamma_T(\mathcal{X}) < \mathcal{O}(\sqrt{T})$  and completes the proof that, if the conditions in the proposition are fulfilled, suboptimal control sets will only be played a number of times that is sublinear in  $T$ . The proof that  $R_{T}$  will then also be sublinear in  $T$  is straightforward. Assuming without loss of generality that  $i^{*} = m$ ,
$$
\begin{array}{l} R _ {T} := \sum_ {t = 1} ^ {T} c _ {i _ {t}} \left(\mathbb {E} \left[ f \left(\left[ \mathbf {x} ^ {i ^ {*}}, \mathbf {X} ^ {- i ^ {*}} \right]\right) \right] - \mathbb {E} \left[ f \left(\left[ \mathbf {x} ^ {i _ {t}}, \mathbf {X} ^ {- i _ {t}} \right]\right) \right]\right) \\ = \sum_ {i = 1} ^ {m} \sum_ {t \in \widetilde {T} _ {i}} c _ {i} \left(\mathbb {E} \left[ f \left([ \mathbf {x} ^ {i ^ {*}}, \mathbf {X} ^ {- i ^ {*}} ]\right) \right] - \mathbb {E} \left[ f \left([ \mathbf {x} ^ {i t}, \mathbf {X} ^ {- i t} ]\right) \right]\right) \\ \leq \sum_ {i = 1} ^ {m - 1} T _ {i} c _ {i} \left(\mathbb {E} \left[ f \left(\left[ \mathbf {x} ^ {i ^ {*}}, \mathbf {X} ^ {- i ^ {*}} \right]\right) \right] - \min  _ {\mathbf {x} ^ {i} \in \mathcal {X} ^ {i}} \mathbb {E} \left[ f \left(\left[ \mathbf {x} ^ {i}, \mathbf {X} ^ {- i} \right]\right) \right]\right) \\ + c _ {m} \sum_ {t \in \widetilde {T} _ {m}} \left(\mathbb {E} \left[ f \left([ \mathbf {x} ^ {i ^ {*}}, \mathbf {X} ^ {- i ^ {*}} ]\right) \right] - \mathbb {E} \left[ f \left([ \mathbf {x} ^ {i _ {t}}, \mathbf {X} ^ {- i _ {t}} ]\right) \right]\right) \\ \end{array}
$$
$$
\begin{array}{l} = \sum_ {i = 1} ^ {m - 1} T _ {i} C _ {i} + c _ {m} \sum_ {t \in \widetilde {T} _ {m}} \left(\mathbb {E} \left[ f \left([ \mathbf {x} ^ {i ^ {*}}, \mathbf {X} ^ {- i ^ {*}} ]\right) \right] - \mathbb {E} \left[ f \left([ \mathbf {x} ^ {i _ {t}}, \mathbf {X} ^ {- i _ {t}} ]\right) \right]\right) \\ \stackrel {(i)} {\leq} \sum_ {i = 1} ^ {m - 1} T _ {i} C _ {i} + \mathcal {O} \left(c _ {m} \left(B + \sqrt {\gamma_ {T} (\mathcal {X}) + \log \frac {m + 1}{\delta}}\right) \left(\sqrt {T _ {m} \gamma_ {T _ {m}} (\widetilde {\mathcal {X}} _ {m})} + \log \frac {m + 1}{\delta}\right)\right) \\ \end{array}
$$
where  $\widetilde{T}_i$  is the ordered sequence of iterations at which control set  $i$  is chosen and  $C_i\coloneqq c_i\left(\mathbb{E}\big[f([{\bf x}^{i^*},{\bf X}^{-i^*}])\big] - \min_{\bf x^i\in \mathcal{X}^i}\mathbb{E}\big[f([{\bf x}^i,{\bf X}^{-i}])\big]\right)$ , and  $(i)$  follows from the steps in the proof of Theorem 4.1 but only for control set  $m$  and without accounting for the  $\epsilon$ -schedule. Since each  $T_{i}$  is sublinear in  $T$ , dividing both sides by  $T$ , using the fact that  $\gamma_T(\mathcal{X}) < \mathcal{O}(\sqrt{T})$ , and taking the limit as  $T\to \infty$  yields the desired result and completes the proof.
Lemma 4.3. Assume that, for each control set  $i$ , the random variable  $Y_{i} \coloneqq \left\| [0, \mathbf{X}_{1}^{-i}] - [0, \mathbf{X}_{2}^{-i}] \right\|^{2}$  has a median  $M_{i}$  such that  $\mathbb{E}[Y_{i}|Y_{i} > M_{i}] \leq h_{i}M_{i}$  for some  $h_{i} > 0$ , where  $\mathbf{X}_{1}^{-i}, \mathbf{X}_{2}^{-i} \sim \mathbb{P}^{-i}$ . With probability at least  $1 - \delta$ , there will be at least  $N_{i}$  non-overlapping pairs of queries  $\mathbf{x}$  and  $\mathbf{x}'$  chosen by UCB-CVS (Alg. 1) with control set  $i$  such that  $\| \mathbf{x} - \mathbf{x}' \|^{2} \geq M_{i}$ , where
$$
N _ {i} = \left\lfloor \frac {1}{4} (T _ {i} - 1) - \sqrt {\frac {1}{4} T _ {i} \log \frac {1}{\delta}} \right\rfloor ,
$$
$$
M _ {i} \geq \frac {4}{h _ {i} + 1} \sum_ {\ell \in \overline {{\mathcal {T}}} _ {i}} \mathbb {V} [ X _ {\ell} ].
$$
Proof. Consider two queries  $\mathbf{x} = [\mathbf{x}^i, \mathbf{x}^{-i}]$  and  $\mathbf{x}' = [\mathbf{x}'^i, \mathbf{x}'^{-i}]$  chosen with control set  $i$ . The learner only selects  $\mathbf{x}^i$  and  $\mathbf{x}'^i$  while  $\mathbf{x}^{-i}$  and  $\mathbf{x}'^{-i}$  are sampled from the environment. Before they are sampled, the queries may be considered themselves random vectors composed of one deterministic partial query and one random partial query. Denote these random vectors as  $\mathbf{X} = [\mathbf{x}^i, \mathbf{X}^{-i}]$  and  $\mathbf{X}' = [\mathbf{x}'^i, \mathbf{X}'^{-i}]$ .  $\|\mathbf{X} - \mathbf{X}'\|^2$  is therefore a random variable as well. Observe that
$$
\begin{array}{l} \left\| \mathbf {X} - \mathbf {X} ^ {\prime} \right\| ^ {2} = \sum_ {j \in \mathcal {I} _ {i}} (x _ {j} - x _ {j} ^ {\prime}) ^ {2} + \sum_ {\ell \in \overline {{\mathcal {I}}} _ {i}} (X _ {\ell} - X _ {\ell} ^ {\prime}) ^ {2} \\ \geq \sum_ {\ell \in \overline {{\mathcal {I}}} _ {i}} \left(X _ {\ell} - X _ {\ell} ^ {\prime}\right) ^ {2} \\ = \left\| \left[ \mathbf {0}, \mathbf {X} _ {1} ^ {- i} \right] - \left[ \mathbf {0}, \mathbf {X} _ {2} ^ {- i} \right] \right\| ^ {2} \\ = Y _ {i} ^ {\prime}. \\ \end{array}
$$
where  $Y_{i}^{\prime}$  is a random variable that is i.i.d. with  $Y_{i}$ . Therefore, any  $\| \mathbf{X} - \mathbf{X}^{\prime}\|^{2}$  can be treated as a random variable equal to some  $Y_{i}^{\prime}$  that is i.i.d. with  $Y_{i}$  plus some non-negative term. The rest of this proof will use lower bounds on random variables i.i.d. with  $Y_{i}$ , which will in turn imply lower bounds on  $\| \mathbf{X} - \mathbf{X}^{\prime}\|^{2}$ .
$$
\begin{array}{l} \mathbb {E} \left[ Y _ {i} \right] = \mathbb {E} \left[ \sum_ {\ell \in \overline {{\mathcal {I}}} _ {i}} \left(X _ {\ell} - X _ {\ell} ^ {\prime}\right) ^ {2} \right] \\ = \sum_ {\ell \in \overline {{\mathcal {I}}} _ {i}} \mathbb {E} \left[ \left(X _ {\ell} - \bar {X} _ {\ell} - \left(X _ {\ell} ^ {\prime} - \bar {X} _ {\ell}\right)\right) ^ {2} \right] \\ = \sum_ {\ell \in \overline {{\mathcal {I}}} _ {i}} \mathbb {E} \left[ \left(\left(X _ {\ell} - \bar {X} _ {\ell}\right) - \left(X _ {\ell} ^ {\prime} - \bar {X} _ {\ell} ^ {\prime}\right)\right) ^ {2} \right] \\ = \sum_ {\ell \in \overline {{\mathcal {I}}} _ {i}} \mathbb {E} \left[ \left(X _ {\ell} - \bar {X} _ {\ell}\right) ^ {2} - 2 \left(X _ {\ell} - \bar {X} _ {\ell}\right) \left(X _ {\ell} ^ {\prime} - \bar {X} _ {\ell} ^ {\prime}\right) + \left(X _ {\ell} ^ {\prime} - \bar {X} _ {\ell} ^ {\prime}\right) ^ {2} \right] \\ \end{array}
$$
$$
\begin{array}{l} = \sum_ {\ell \in \overline {{\mathcal {I}}} _ {i}} \mathbb {E} \left[ \left(X _ {\ell} - \bar {X} _ {\ell}\right) ^ {2} \right] - \mathbb {E} \left[ \left(X _ {\ell} - \bar {X} _ {\ell}\right) \left(X _ {\ell} ^ {\prime} - \bar {X} _ {\ell} ^ {\prime}\right) \right] + \mathbb {E} \left[ \left(X _ {\ell} ^ {\prime} - \bar {X} _ {\ell} ^ {\prime}\right) ^ {2} \right] \\ = \sum_ {\ell \in \overline {{\mathcal {I}}} _ {i}} \mathbb {E} \left[ \left(X _ {\ell} - \overline {{X}} _ {\ell}\right) ^ {2} \right] + \mathbb {E} \left[ \left(X _ {\ell} ^ {\prime} - \overline {{X}} _ {\ell} ^ {\prime}\right) ^ {2} \right] \\ = \sum_ {\ell \in \overline {{\mathcal {I}}} _ {i}} 2 \mathbb {V} \left[ X _ {\ell} \right]. \tag {6} \\ \end{array}
$$
We will now construct a lower bound for a median of  $Y_{i}$  denoted  $M_{i}$ .
$$
\begin{array}{l} \mathbb {E} [ Y _ {i} ] = \mathbb {E} [ Y _ {i} | Y _ {i} <   M _ {i} ] \cdot P (Y _ {i} <   M _ {i}) + \mathbb {E} [ Y _ {i} | Y _ {i} = M _ {i} ] \cdot P (Y _ {i} = M _ {i}) + \mathbb {E} [ Y _ {i} | Y _ {i} > M _ {i} ] \cdot P (Y _ {i} > M _ {i}) \\ \leq M _ {i} \cdot P \left(Y _ {i} \leq M _ {i}\right) + \mathbb {E} \left[ Y _ {i} \mid Y _ {i} > M _ {i} \right] \cdot P \left(Y _ {i} > M _ {i}\right) \\ \stackrel {(i)} {\leq} M _ {i} \cdot P \left(Y _ {i} \leq M _ {i}\right) + h _ {i} M _ {i} \cdot P \left(Y _ {i} > M _ {i}\right) \\ \stackrel {(i i)} {\leq} \frac {1}{2} M _ {i} + \frac {1}{2} \left(h _ {i} \cdot M _ {i}\right) \\ = \frac {h _ {i} + 1}{2} M _ {i} \\ \end{array}
$$
where  $(i)$  follows from our assumption on the median  $M_{i}$  and  $(ii)$  follows from the definition of a median:  $P(Y_{i} \leq M_{i}) \geq 1/2$ . Substituting in (6) completes our construction of the lower bound for  $M_{i}$ :
$$
\begin{array}{l} M _ {i} \geq \frac {2}{h _ {i} + 1} \mathbb {E} \left[ Y _ {i} \right] \\ \geq \frac {4}{h _ {i} + 1} \sum_ {\ell \in \overline {{\mathcal {I}}} _ {i}} \mathbb {V} [ X _ {\ell} ]. \\ \end{array}
$$
Now consider the  $\lfloor T_i / 2\rfloor$  non-overlapping pairs of queries chosen with control set  $i^3$ . Associate each pair with a random variable  $Y_{ij}$  such that we have  $\lfloor T_i / 2\rfloor$  i.i.d. random variables  $Y_{i1}, Y_{i2}, \ldots, Y_{i\lfloor T_i / 2\rfloor}$ . From the definition of a median,  $P(Y_i \geq M_i) \geq 1 / 2$ . Without loss of generality, assume the worst-case such that  $P(Y_i \geq M_i) = 1 / 2$ . We can now construct  $\lfloor T_i / 2\rfloor$  i.i.d. Bernoulli random variables  $Z_1, Z_2, \ldots, Z_n$ ,  $n = \lfloor T_i / 2\rfloor$ , with  $p = 1 / 2$  where a success ( $Z_j = 1$ ) corresponds to  $Y_{ij} \geq M_i$  and a failure ( $Z_j = 0$ ) corresponds to  $Y_{ij} < M_i$ . Further define the random variable  $Z := \sum_{j=1}^{n} Z_j$ .
ApplyingHoeffding'sinequality,
$$
P \left(\frac {1}{n} \sum_ {j = 1} ^ {n} \left(Z _ {j} - p\right) \leq - t\right) \leq \exp \left(- 2 n t ^ {2}\right)
$$
$$
P \left(\frac {1}{n} Z - p \leq - t\right) \leq \exp \left(- 2 n t ^ {2}\right)
$$
$$
P (Z \leq n (p - t)) \leq \exp (- 2 n t ^ {2}).
$$
Choosing  $t = p - \alpha / n$  for some constant  $\alpha$ ,