SlowGuess/NeurIPS_2023_OCR huggingface dataset:
While our method is not restricted to the form of the acquisition function, for convenience in the theoretical analysis, we follow the GP-UCB [52]. Given the GP predictive
Algorithm 1 DRBO with  $\varphi$ -divergence
1: Input: Max iteration  $T$ , initial data  $D_0$ ,  $\eta$
2: for  $t = 1, \dots, T$  do
3: Fit and estimate GP hyperparameter given  $D_{t-1}$
4: Select a next input  $\mathbf{x}_t = \arg \max \alpha (\mathbf{x})$
5:  $\chi^2$ -divergence:  $\alpha(\mathbf{x}) \coloneqq \alpha^{\chi^2}(\mathbf{x})$  from Eq. (4)
6: Total Variation:  $\alpha (\mathbf{x})\coloneqq \alpha^{TV}(\mathbf{x})$  from Eq. (5)
7: Observe a context  $c_t \sim q$
8: Evaluate the black-box  $y_{t} = f(\mathbf{x}_{t},c_{t}) + \eta_{t}$
9: Augment  $D_{t} = D_{t - 1}\cup (\mathbf{x}_{t},c_{t},y_{t})$
10: end for
mean and variance from Eqs. (7,8), we have the acquisition function for the  $\chi^2$  in Example 2 as follows:
$$
\alpha^ {\chi^ {2}} (\mathbf {x}) := \frac {1}{| C |} \sum_ {c} \left[ \mu_ {t} (\mathbf {x}, c) + \sqrt {\beta_ {t}} \sigma_ {t} (\mathbf {x}, c) \right] - \sqrt {\frac {\varepsilon_ {t}}{| C |} \sum_ {c} \left(\mu_ {t} (\mathbf {x} , c) - \bar {\mu_ {t}}\right) ^ {2}} \tag {4}
$$
where  $\beta_{t}$  is a explore-exploit hyperparameter defined in Srinivas et al. [52],  $\bar{\mu}_{t} = \frac{1}{|C|}\sum_{c}\mu_{t}(\mathbf{x},c)$  and  $c\sim q$  can be generated in a one dimensional space to approximate the expectation and the variance. In the experiment, we select  $q$  as the uniform distribution, but it is not restricted to. Similarly, an acquisition function for Total Variation in Example 3 is written as
$$
\alpha^ {T V} (\mathbf {x}) := \frac {1}{| C |} \sum_ {c} \left[ \mu_ {t} (\mathbf {x}, c) + \sqrt {\beta_ {t}} \sigma_ {t} (\mathbf {x}, c) \right] - \frac {\varepsilon_ {t}}{2} \left(\max  \mu_ {t} (\mathbf {x}, c) - \min  \mu_ {t} (\mathbf {x}, c)\right). \tag {5}
$$
We summarize all computational steps in Algorithm 1.
Computational Efficiency against MMD. We make an important remark that since we do not require our context space to be finite, our implementation scales only linearly with the number of context samples  $|C|$  drawing from  $q$ . This allows us to discretize our space and draw as many context samples as required while only paying a linear price. On the other hand, the MMD [26] at every iteration of  $t$  requires solving an  $|C|$ -dimensional constraint optimization problem that has no closed form solution. We refer to Section 5.2 for the empirical comparison.
4.2 Convergence Analysis
One of the main advantages of Kirschner et al. [26] is the choice of MMD makes the regret analysis simpler due to the nice structure and properties of MMD. In particular, the MMD is well-celebrated for a  $O(t^{-1/2})$  convergence where no such results exist for  $\varphi$ -divergences. However, using Theorem 1, we can show a regret bound for the Total Variation with a simple boundedness assumption and show how one can extend this result to other  $\varphi$ -divergences. We begin by defining the robust regret,  $R_T$ , with  $\varphi$ -divergence balls:
$$
R _ {T} (\varphi) = \sum_ {t = 1} ^ {T} \inf  _ {q \in B _ {\varphi} ^ {t}} \mathbb {E} _ {q (c)} [ f (\mathbf {x} _ {t} ^ {*}, c) ] - \inf  _ {q \in B _ {\varphi} ^ {t}} \mathbb {E} _ {q (c)} [ f (\mathbf {x} _ {t}, c) ], \tag {6}
$$
where  $\mathbf{x}_t^* = \arg \max_{\mathbf{x} \in \mathcal{X}} \inf_{q \in B_{\varepsilon, \varphi}^t} \mathbb{E}_{q(c)}[f(\mathbf{x}, c)]$ . We use  $\pmb{K}_t$  to denote the generated kernel matrix from dataset  $D_t = \{(\mathbf{x}_i, c_i)\}_{i=1}^t \subset \mathcal{X} \times \mathcal{C}$ . We now introduce a standard quantity in regret analysis in BO is the maximum information gain:  $\gamma_t = \max_{D \subset \mathcal{X} \times \mathcal{C}: |D| = t} \log \det(\mathbf{I}_t + \sigma_f^{-2}\pmb{K}_t)$  where  $\pmb{K}_t = [k([x_i, c_i], [x_j, c_j])]_{\forall i, j \leq t}$  is the covariance matrix and  $\sigma_f^2$  is the output noise variance.
Theorem 4 ( $\varphi$ -divergence Regret) Suppose the target function is bounded, meaning that  $M = \sup_{(\mathbf{x}, c) \in \mathcal{X} \times \mathcal{C}} |f(\mathbf{x}, c)| < \infty$  and suppose  $f$  has bounded RKHS norm with respect to  $k$ . For any lower semicontinuous convex  $\varphi : \mathbb{R} \to (-\infty, \infty]$  with  $\varphi(1) = 0$ , if there exists a monotonic invertible function  $\Gamma_{\varphi} : [0, \infty) \to \mathbb{R}$  such that  $\mathrm{TV}(p, q) \leq \Gamma_{\varphi}(\mathrm{D}_{\varphi}(p, q))$ , the following holds
$$
R _ {T} (\varphi) \leq \frac {\sqrt {8 T \beta_ {T} \gamma_ {T}}}{\log (1 + \sigma_ {f} ^ {- 2})} + \left(2 M + \sqrt {\beta_ {T}}\right) \sum_ {t = 1} ^ {T} \Gamma_ {\varphi} (\varepsilon_ {t}),
$$
with probability  $1 - \delta$ , where  $\beta_{t} = 2||f||_{k}^{2} + 300\gamma_{t}\ln^{3}(t / \delta)$ ,  $\gamma_{t}$  is the maximum information gain as defined above, and  $\sigma_{f}$  is the standard deviation of the output noise.
The full proof can be found in the Appendix Section 8. We first remark that with regularity assumptions on  $f$ , sublinear analytical bounds for  $\gamma_{T}$  are known for a range of kernels, e.g., given  $\mathcal{X} \times \mathcal{C} \subset \mathbb{R}^{d + 1}$  we have for the RBF kernel,  $\gamma_{T} \leq \mathcal{O}\left(\log (T)^{d + 2}\right)$  or for the Matérn kernel with  $\nu > 1$ ,  $\gamma_{T} \leq \mathcal{O}\left(T^{\frac{(d + 1)(d + 2)}{2\nu + (d + 1)(d + 2)}}(\log T)\right)$ . The second term in the bound is directly a consequence of DRO and by selecting  $\varepsilon_{t} = 0$ , it will vanish since any such  $\Gamma_{\varphi}$  will satisfy  $\Gamma_{\varphi}(0) = 0$ . To ensure sublinear regret, we can select  $\varepsilon_{t} = \Gamma_{\varphi}^{-1}\left(\frac{1}{\sqrt{t} + \sqrt{t + 1}}\right)$ , noting that the second term will reduce to  $\sum_{t = 1}^{T}\varepsilon_{t} \leq \sqrt{T}$ . Finally, we remark that the existence of  $\Gamma_{\varphi}$  is not so stringent since for a wide choices of  $\varphi$ , one can find inequalities between the Total Variation and  $D_{\varphi}$ , to which we refer the reader to Sason and Verdu [45]. For the examples discussed above, we can select  $\Gamma_{\varphi}(t) = t$  for the TV. For the  $\chi^2$  and KL cases, one can choose  $\Gamma_{\chi^2}(b) = 2\sqrt{\frac{b}{1 + b}}$  and  $\Gamma_{\mathrm{KL}}(b) = 1 - \exp (-b)$ .
Figure 3: Cumulative robust regret across algorithms. The results show that the proposed  $\chi^2$  and TV achieve the best performance across benchmark functions. Random and vanilla BO approaches perform poorly which do not take into account the robustness criteria. Best viewed in color.
Experimental setting. The experiments are repeated using 30 independent runs. We set  $|C| = 30$  which should be sufficient to draw  $c \stackrel{\mathrm{iid}}{\sim} q$  in one-dimensional space to compute Eqs. (4,5). We optimize the GP hyperparameter (e.g., learning rate) by maximizing the GP log marginal likelihood [43]. We will release the Python implementation code in the final version.
Baselines. We consider the following baselines for comparisons. Rand: we randomly select  $\mathbf{x}_t$  irrespective of  $c_t$ . BO: we follow the GP-UCB [52] to perform standard Bayesian optimization (ignoring the context  $c_t$ ). The selection at each iteration is  $\mathbf{x}_t = \mathrm{argmax}_{\mathbf{x}} \mu(\mathbf{x}) + \beta_t \sigma(\mathbf{x})$ . Stable-Opt: we consider the worst-case robust optimization presented in Bogunovic et al. [8]. The selection at each iteration  $\mathbf{x}_t = \mathrm{argmax}_{\mathbf{x}} \mathrm{argmin}_c \mu(\mathbf{x}, c) + \beta_t \sigma(\mathbf{x}, c)$ . DRBO MMD [26]: Since there is no official implementation available, we have tried our best to re-implement the algorithm.
We consider the popular benchmark functions<sup>3</sup> with different dimensions  $d$ . To create a context variable  $c$ , we pick the last dimension of these functions to be the context input while the remaining  $d - 1$  dimension becomes the input  $\mathbf{x}$ .
5.1 Ablation Studies
To gain understanding into how our framework works, we consider two popular settings below.
DRBO solution is different from stochastic solution. In Fig. 1a, the vanilla BO tends to converge greedily toward the stochastic solution (non-distributionally robust)  $\operatorname{argmax}_{\mathbf{x}} f(\mathbf{x}, \cdot)$ . Thus, BO keeps exploiting in the locality of  $\operatorname{argmax}_{\mathbf{x}} f(\mathbf{x}, \cdot)$  from iteration 15. On the other hand, all other DRBO methods will keep exploring to seek for the distributionally robust solutions. Using the high value of  $\varepsilon_t \in \{0.5, 1\}$  will result in the best performance.
DRBO solution is identical to stochastic solution. When the stochastic and robust solutions coincide at the same input  $\mathbf{x}^*$ , the solution of BO will be equivalent to the solution of DRBO methods. This is demonstrated by Fig. 1b. Both stochastic and robust approaches will quickly identify the optimal solution (see the  $\mathbf{x}_t$  selection). We learn empirically that setting  $\varepsilon_t \to 0$  will lead to the best performance. This is because the DRBO setting will become the standard BO.
The best choice of  $\varepsilon$  depends on the property of the underlying function, e.g., the gap between the stochastic and DRBO solutions. In practice, we may not be able to identify these scenarios in advance. Therefore, we can use the adaptive value of  $\varepsilon_{t}$  presented in Section 4.2. Using this adaptive setting, the performance is stable, as illustrated in the figures.
5.2 Computational efficiency
The key benefit of our framework is simplifying the existing intractable computation by providing the closed-form solution. Additional to improving the quality, we demonstrate this advantage in terms of computational complexity. Our main baseline for comparison is the MMD [26]. As shown in Fig. 2, our DRBO is consistently faster than the constraints linear programming approximation used for
<sup>3</sup>https://www.sfu.ca/ ssurjano/optimization.html
Figure 4: All divergences improve with larger  $|C|$ . However, MMD comes with the quadratic cost.
MMD. This gap is substantial in higher dimensions. In particular, as compared to Kirschner et al. [26], our DRBO is 5-times faster in  $5d$  and 10-times faster in  $6d$ .
5.3 Optimization performance comparison
We compare the algorithms in Fig. 3 using the robust (cumulative) regret defined in Eq. (6) which is commonly used in DRO literature [26, 33]. The random approach does not make any intelligent information in making decision, thus performs the worst. While BO performs better than random, it is still inferior comparing to other distributionally robust optimization approaches. The reason is that BO does not take into account the context information in making the decision. The StableOpt [8] performs relatively well that considers the worst scenarios in the subset of predefined context. This
Figure 2: We compare the computational cost across methods. Our proposed DRBO using  $\chi^2$  and TV take similar cost per iteration which is significantly lower than the DRBO MMD [26].
predefined subset can not cover all possible cases as opposed to the distributional robustness setting.
The MMD approach [26] needs to solve the inner adversary problem using linear programming with convex constraints, additional to the main optimization step. As a result, the performance of MMD is not as strong as our TV and  $\chi^2$ . Our proposed approach does not suffer this pathology and thus scale well in continuous and high dimensional settings of context input  $c$ .
Real-world functions. We consider the deterministic version of the robot pushing objective from Wang and Jegelka [60]. The goal is to find a good pre-image for pushing an object to a target location. The 3-dimensional function takes as input the robot location  $(r_x, r_y) \in [-5, 5]^2$  and pushing duration  $r_t \in [1, 30]$ . We follow Bogunovic et al. [8] to twist this problem in which there is uncertainty regarding the precise target location, so one seeks a set of input parameters that is robust against a number of different potential pushing duration which is a context.
We perform an experiment on Wind Power dataset [8] and vary the context dimensions  $|C| \in \{30,100,500\}$  in Fig. 4. When  $|C|$  enlarges, our DRBO  $\chi^2$ , TV and KL improves. However, the performances do not improve further when increasing  $|C|$  from 100 to 500. Similarly, MMD improves with  $|C|$ , but it comes with the quadratic cost w.r.t.  $|C|$ . Overall, our proposed DRBO still performs favourably in terms of optimization quality and computational cost than the MMD.
6 Conclusions, Limitations and Future works
In this work, we showed how one can study the DRBO formulation with respect to  $\varphi$ -divergences and derived a new algorithm that removes much of the computational burden, along with a sublinear regret bound. We compared the performance of our method against others, and showed that our results unveil a deeper connection between regularization and robustness, which serves useful conceptually.
Limitations and Future Works One of the limitations of our framework is in the choice of  $\varphi$ , for which we provide no guidance. For different applications, different choices of  $\varphi$  would prove to be more useful, the study of which we leave for future work.
We would like to thank the anonymous reviewers for providing feedback.
[1] Soroosh Shafieezadeh Abadeh, Peyman Mohajerin Mohajerin Esfahani, and Daniel Kuhn. Distributionally robust logistic regression. In Advances in Neural Information Processing Systems, pages 1576-1584, 2015. 3
[2] Amir Ahmadi-Javid. Entropic value-at-risk: A new coherent risk measure. Journal of Optimization Theory and Applications, 155(3):1105-1123, 2012. 3, 14
[3] Justin J Beland and Prasanth B Nair. Bayesian optimization under uncertainty. In NIPS BayesOpt 2017 workshop, 2017. 2
[4] Aharon Ben-Tal, Dick Den Hertog, Anja De Waegenaere, Bertrand Mellenberg, and Gijs Rennen. Robust solutions of optimization problems affected by uncertain probabilities. Management Science, 59(2):341-357, 2013. 4, 5
[5] M Bennouna and Bart PG Van Parys. Learning and decision-making with data: Optimal formulations and phase transitions. arXiv preprint arXiv:2109.06911, 2021. 4
[6] Jose Blanchet and Karthyek Murthy. Quantifying distributional model risk via optimal transport. Mathematics of Operations Research, 44(2):565-600, 2019. 2, 4
[7] Jose Blanchet, Yang Kang, and Karthyek Murthy. Robust Wasserstein profile inference and applications to machine learning. Journal of Applied Probability, 56(3):830-857, 2019. 2, 4
[8] Ilija Bogunovic, Jonathan Scarlett, Stefanie Jegelka, and Volkan Cevher. Adversarily robust optimization with Gaussian processes. In Conference on Neural Information Processing Systems (NIPS), number CONF, 2018. 8, 9
[9] Zac Cranko, Simon Kornblith, Zhan Shi, and Richard Nock. Lipschitz networks and distributional robustness. arXiv preprint arXiv:1809.01129, 2018. 2, 4
[10] Zac Cranko, Zhan Shi, Xinhua Zhang, Richard Nock, and Simon Kornblith. Generalised lipschitz regularisation equals distributional robustness. In International Conference on Machine Learning, pages 2178-2188. PMLR, 2021. 2, 3
[11] John Duchi and Hongseok Namkoong. Variance-based regularization with convex objectives. The Journal of Machine Learning Research, 20(1):2450-2504, 2019. 2, 3, 5, 6
[12] John C Duchi, Michael I Jordan, and Martin J Wainwright. Local privacy and statistical minimax rates. In FOCS, 2013. 2, 3, 4, 5
[13] John C Duchi, Peter W Glynn, and Hongseok Namkoong. Statistics of robust optimization: A generalized empirical likelihood approach. Mathematics of Operations Research, 46(3): 946-969, 2021. 2, 3, 4, 5
[14] KD Dvijotham, J Hayes, B Balle, Z Kolter, C Qin, A Gyorgy, K Xiao, S Gowal, and P Kohli. A framework for robustness certification of smoothed classifiers using f-divergences. In International Conference on Learning Representations, 2020. 5
[15] Ky Fan. Minimax theorems. Proceedings of the National Academy of Sciences of the United States of America, 39(1):42, 1953. 15
[16] Rui Gao, Xi Chen, and Anton J Kleywegt. Wasserstein distributional robustness and regularization in statistical learning. arXiv e-prints, pages arXiv-1712, 2017. 2
[17] Ziv Goldfeld, Kristjan Greenewald, Jonathan Niles-Weed, and Yury Polyanskiy. Convergence of smoothed empirical measures with applications to entropy estimation. IEEE Transactions on Information Theory, 66(7):4368-4391, 2020. 20
[53] Matthew Staib and Stefanie Jegelka. Distributionally robust optimization and generalization in kernel methods. In Advances in Neural Information Processing Systems, pages 9131-9141, 2019. 2, 4
[54] Sebastian Shenghong Tay, Chuan Sheng Foo, Urano Daisuke, Richalynn Leong, and Bryan Kian Hsiang Low. Efficient distributionally robust bayesian optimization with worst-case sensitivity. In International Conference on Machine Learning, pages 21180-21204. PMLR, 2022. 3, 4
[55] MC Tran, V Nguyen, Richard Bruce, DC Crockett, Federico Formenti, PA Phan, SJ Payne, and AD Farmery. Simulation-based optimisation to quantify heterogeneity of specific ventilation and perfusion in the lung by the inspired sinewave test. Scientific reports, 11(1):1-10, 2021. 1
[56] Tsuyoshi Ueno, Trevor David Rhone, Zhufeng Hou, Teruyasu Mizoguchi, and Koji Tsuda. Combo: an efficient Bayesian optimization library for materials science. *Materials discovery*, 4: 18-21, 2016. 1
[57] Nienke ER van Bueren, Thomas L Reed, Vu Nguyen, James G Sheffield, Sanne HG van der Ven, Michael A Osborne, Evelyn H Kroesbergen, and Roi Cohen Kadosh. Personalized brain stimulation for effective neurointervention across participants. PLOS Computational Biology, 17(9):e1008886, 2021. 1
[58] Cédric Villani. Optimal transport: old and new, volume 338. Springer Science & Business Media, 2008. 15
[59] Xingchen Wan, Cong Lu, Jack Parker-Holder, Philip J Ball, Vu Nguyen, Binxin Ru, and Michael Osborne. Bayesian generational population-based training. In International Conference on Automated Machine Learning, pages 14-1. PMLR, 2022. 1
[60] Zi Wang and Stefanie Jegelka. Max-value entropy search for efficient Bayesian optimization. In International Conference on Machine Learning, pages 3627-3635, 2017. 9
[61] Jingzhao Zhang, Aditya Krishna Menon, Andreas Veit, Srinadh Bhojanapalli, Sanjiv Kumar, and Suvrit Sra. Coping with label shift via distributionally robust optimisation. In International Conference on Learning Representations 2021. 19