SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Mikhail Khodak*
Carnegie Mellon University
khodak@cmu.edu	Ilya Osadchiy*
Technion - Israel Institute of Technology
osadchiy.ilya@gmail.com	Keegan Harris
CMU
Maria-Florina Balcan
CMU	Kfir Y. Levy
Technion	Ron Meir
Technion
We study online meta-learning with bandit feedback, with the goal of improving performance across multiple tasks if they are similar according to some natural similarity measure. As the first to target the adversarial online-within-online partial-information setting, we design meta-algorithms that combine outer learners to simultaneously tune the initialization and other hyperparameters of an inner learner for two important cases: multi-armed bandits (MAB) and bandit linear optimization (BLO). For MAB, the meta-learners initialize and set hyperparameters of the Tsallis-entropy generalization of Exp3, with the task-averaged regret improving if the entropy of the optima-in-hindsight is small. For BLO, we learn to initialize and tune online mirror descent (OMD) with self-concordant barrier regularizers, showing that task-averaged regret varies directly with an action space-dependent measure they induce. Our guarantees rely on proving that unregularized follow-the-leader combined with two levels of low-dimensional hyperparameter tuning is enough to learn a sequence of affine functions of non-Lipschitz and sometimes non-convex Bregman divergences bounding the regret of OMD.
Learning-to-learn [51] is an important area of research that studies how to improve the performance of a learning algorithm by meta-learning its parameters—e.g. initializations, step-sizes, and/or representations—across many similar tasks. The goal is to encode information from previous tasks in order to achieve better performance on future ones. Meta-learning has seen a great deal of experimental work [24, 49], practical impact [21, 29], and theoretical effort [11, 18, 22, 45, 20]. One important setting is online-within-online meta-learning [19, 31], where the learner performs a sequence of tasks, each of which has a sequence of rounds. Past work has studied the full-information setting, where the loss for every arm is revealed after each round. This assumption is not realistic in many applications, e.g. recommender systems and experimental design, where often partial or bandit feedback—only the loss of the action taken—is revealed. Such feedback can be stochastic, e.g. the losses are i.i.d. from some distribution, or adversarial, i.e. chosen by an adversary. We establish the first formal guarantees for online-within-online meta-learning with adversarial bandit feedback.
As with past full-information meta-learning results, our goal when faced with a sequence of bandit tasks will be to achieve low regret on average across them. Specifically, our task-averaged regret should (a) be no worse than that of algorithms for the single-task setting, e.g. if the tasks are not very similar, and should (b) be much better on tasks that are closely related, e.g. if the same small set of arms do well on all of them. We show that a natural way to achieve both is to initialize and tune online mirror descent (OMD), an algorithm associated with a strictly convex regularizer whose hyperparam
* Denotes equal contribution.
eters have a significant impact on performance. Our approach works because it can learn the best hyperparameters in hindsight across tasks, which will recover OMD's worst-case optimal performance if the tasks are dissimilar but will take advantage of more optimistic settings if they are related. As generalized distances, the regularizers also induce interpretable measures of similarity between tasks.
We design a meta-algorithm (Algorithm 1) for learning variants of OMD—specifically those with entropic or self-concordant regularizers—that are used for adversarial bandits. This meta-algorithm combines three full-information algorithms—follow-the-leader (FTL), exponentially weighted online optimization (EWOO), and multiplicative weights (MW)—to set the initialization, step-size, and regularizer-specific parameters, respectively. It works by optimizing a sequence of functions that each upper-bound the regret of OMD on a single task (Theorem 2.1), resulting in (a) interesting notions of task-similarity because these functions depend on generalized notions of distances (Bregman divergences) and (b) adaptivity, i.e not needing to know how similar the tasks are beforehand.
Our first application is to OMD with the Tsallis regularizer [3], a relative of Exp3 [6] that is optimal for adversarial MAB. We bound the task-averaged regret by the Tsallis entropy of the estimated optima-in-hindsight (Corollary 3.1), which we further extend to that of the true optima by assuming a gap between the best and second-best arms (Corollary 3.2). Both results are the first known consequences of the online learnability of Bregman divergences that are non-convex in their second arguments [31], while the latter is obtained by showing that the loss estimators of a modified algorithm identify the optimal arm w.h.p. As an example, our average  $m$ -round regret across  $T$  tasks under the gap assumption is
$$
o _ {T} (\operatorname {p o l y} (m)) + 2 \min  _ {\beta \in (0, 1 ]} \sqrt {H _ {\beta} d ^ {\beta} m / \beta} + o (\sqrt {m}) \tag {1}
$$
where  $d$  is the number of actions and  $H_{\beta}$  is the Tsallis entropy [52, 3] of the distribution of the optimal actions ( $\beta = 1$  recovers the Shannon entropy). This entropy is low if all tasks are usually solved by the same few arms, making it a natural task-similarity notion. For example, if only  $s \ll d$  of the arms are ever optimal then  $H_{\beta} = \mathcal{O}(s)$ , so using  $\beta = 1 / \log d$  in (1) yields an asymptotic task-averaged regret of  $\mathcal{O}(\sqrt{sm\log d})$ , dropping fast terms. For  $s = \mathcal{O}_d(1)$  this beats the minimax optimal rate of  $\Theta(\sqrt{dm})$  [5]. On the other hand, since  $H_{1/2} = \mathcal{O}(\sqrt{d})$ , the same bound recovers this rate in the worst-case of dissimilar tasks.
Lastly, we adapt our meta-algorithm to the adversarial BLO problem by setting the regularizer to be a self-concordant barrier function, as in Abernethy et al. [2]. Our bounds yield notions of task-similarity that depend on the constraints of the action space, e.g. over the sphere the measure is the closeness of the average of the estimated optima to the sphere's surface (Corollary 4.1). We also instantiate BLO on the bandit shortest-path problem (Corollary 4.2) [50, 30].
While we are the first to consider meta-learning under adversarial bandit feedback, many have studied meta-learning in various stochastic bandit settings [9, 34, 47, 48, 35, 13, 15, 41, 10]. The latter three study stochastic bandits under various task-generation assumptions, e.g. Azizi et al. [10] is in a batch-within-online setting where the optimal arms are adversarial. In contrast, we make no distributional assumptions either within or without. Apart from this difference, the results of Azizi et al. [10] are the ones our MAB results are most easily compared to, which we do in detail in Section 3. Notably, they assume that only  $s \ll d$  of the  $d$  arms are ever optimal across  $T$  tasks and show (roughly speaking)  $\tilde{\mathcal{O}}(\sqrt{sm})$  asymptotic regret; we instead focus on an entropic notion of task-similarity that achieves the same asymptotic regret when specialized to their  $s \ll d$ . However, avoiding their explicit assumption has certain advantages, e.g. robustness in the presence of  $o(T)$  outlier tasks (c.f. Section 3.3).
A setting that bears some similarity to online-within-online bandits is that of switching bandits [6], and more generally online learning with dynamic comparators [4, 27, 38, 7, 55]. In such problems, instead of using a static best arm as the comparator we use a piecewise constant sequence of arms, with a limited number of arm switches. The key difference between such work and ours is our assumption that task-boundaries are known; this makes the other setting more general. However, while e.g. Exp3.S [6] can indeed be applied to online meta-learning, guarantees derived
<sup>1</sup>We use  $\mathcal{O}_n(\cdot)$  (and  $o_n(\cdot))$  to denote terms with constant (and sub-constant) dependence on  $n$ .
from switching costs cannot improve upon just running Tsallis-INF on each task [39, Table 1]. Furthermore, these approaches usually quantify difficulty by the number of switches, whereas we focus on task-similarity. While there exists stochastic-setting work that measures difficulty using a notion of average change in distribution across rounds [53], it does not lead to improved performance if this average change is  $\Omega (T)$ , as is the case in e.g. the  $s$ -sparse setting discussed above.
There has been a variety of work on full-information online-within-online meta-learning [32, 12], including tuning OMD [31, 19]. Doing so for bandit algorithms has many additional challenges, including (1) their inherent and high-variance stochasticity, (2) the use of non-Lipschitz and even unbounded regularizers, and (3) the lack of access to task-optima in order to adapt to deterministic, algorithm-independent task-similarity measures. Theoretically our analysis draws on the average regret-upper-bound analysis (ARUBA) framework [31], which observes that OMD can be tuned by targeting its upper bounds, which are affine functions of Bregman divergences, and provide online learning tools for doing so. Our core structural result shows that the distance generating functions  $\psi_{\theta}$  of these Bregman divergences can be tuned without interfering with meta-learning the initialization and step-size; tuning  $\theta$  is critical for adapting to settings such as that of a small set of optimal arms in MAB. Doing so depends on several refinements of the original approach, including bounding the task-averaged-regret via the spectral norm of  $\nabla^2\psi_\theta$  and expressing the loss of the meta-comparator using only  $\psi_{\theta}$ , rather than via its Bregman divergence as in prior work. Finally, applying our structural result requires setting-specific analysis, e.g. to show regularity w.r.t.  $\theta$  or to obtain MAB guarantees in terms of the entropy of the true optimal arms. The latter is especially difficult, as Khodak et al. [31] define task-similarity via full information upper bounds, and involves applying tools from the best-arm-identification literature [1] to show that a constrained variant of Exp3 finds the optimal arm w.h.p.
2 Learning the regularizers of bandit algorithms
We consider the problem of meta-learning over bandit tasks  $t = 1,\dots ,T$  over some fixed set  $\mathcal{K}\subset \mathbb{R}^d$ , a (possibly improper) subset of which is the action space  $\mathcal{A}$ . On each round  $i = 1,\dots ,m$  of task  $t$  we play action  $\mathbf{x}_{t,i}\in \mathcal{A}$  and receive feedback  $\ell_{t,i}(\mathbf{x}_{t,i})$  for some function  $\ell_{t,i}:A\mapsto [-1,1]$ . Note that all functions we consider will be linear and so we will also write  $\ell_{t,i}(\mathbf{x}) = \langle \ell_{t,i},\mathbf{x}\rangle$ . Additionally, we assume the adversary is oblivious within-task, i.e. it chooses losses  $\ell_{t,1},\ldots ,\ell_{t,m}$  at time  $t$ . We will also denote  $\mathbf{x}(a)$  to be the  $a$ -th element of the vector  $\mathbf{x}\in \mathbb{R}^d$ ,  $\mathcal{K}^{\circ}$  to be the interior of  $\mathcal{K}$ ,  $\partial \mathcal{K}$  its boundary, and  $\triangle$  to be the simplex on  $d$  elements. Finally, note that all proofs can be found in the Appendix.
In online learning, the goal on a single task  $t$  is to play actions  $\mathbf{x}_{t,1},\ldots \mathbf{x}_{t,m}$  that minimize the regret  $\sum_{i = 1}^{m}\ell_{t,i}(\mathbf{x}_{t,i}) - \ell_{t,i}(\mathring{\mathbf{x}}_t)$ , where  $\mathbf{\tilde{x}}_t\in \arg \min_{\mathbf{x}\in \mathcal{K}}\sum_{i = 1}^{m}\ell_{t,i}(\mathbf{x})$ . Lifting this to the meta-learning setting, our goal as in past work [31, 19] will be to minimize the task-averaged regret:  $\frac{1}{T}\sum_{t = 1}^{T}\sum_{i = 1}^{m}\ell_{t,i}(\mathbf{x}_{i,t}) - \ell_{t,i}(\mathring{\mathbf{x}}_t)$ . In particular, we want to use multi-task data to improve average performance as the number of tasks  $T\to \infty$ . For example, we wish to attain a task-averaged regret bound of the form  $o_{T}(\mathrm{poly}(m)) + \tilde{\mathcal{O}} (V\sqrt{m}) + o(\sqrt{m})$ , where  $V\in \mathbb{R}_{\geq 0}$  is a measure of task-similarity that is small if the tasks are similar but still yields the worst-case single-task performance— $\mathcal{O}(\sqrt{dm})$  for MAB and  $\mathcal{O}(d\sqrt{m})$  for BLO—if they are not.
2.1 Online mirror descent as a base-learner
In meta-learning we are commonly interested in learning a within-task algorithm or base-learner, a parameterized method that we run on each task  $t$ . A popular approach is to learn the initialization and other parameters of a gradient-based method such as gradient descent [24, 44, 36]. If the task optima are close, the best initialization should perform well after only a few steps on a new task. We take a similar approach applied to online mirror descent, a generalization of gradient descent to non-Euclidean geometries [14]. Given a strictly convex regularizer  $\psi : \mathcal{K}^{\circ} \mapsto \mathbb{R}$ , step-size  $\eta > 0$  and initialization  $\mathbf{x}_{t,1} \in \mathcal{K}^{\circ}$ , OMD has the iteration
$$
\mathbf {x} _ {t, i + 1} = \underset {\mathbf {x} \in \mathcal {K} ^ {\circ}} {\arg \min } B (\mathbf {x} | | \mathbf {x} _ {t, 1}) + \eta \sum_ {j \leq i} \left\langle \nabla \ell_ {t, j} \left(\mathbf {x} _ {t, j}\right), \mathbf {x} \right\rangle \tag {2}
$$
where  $B(\mathbf{x}||\mathbf{y}) = \psi (\mathbf{x}) - \psi (\mathbf{y}) - \langle \nabla \psi (\mathbf{y}),\mathbf{x} - \mathbf{y}\rangle$  is the Bregman divergence of  $\psi$ . OMD recovers online gradient descent when  $\psi (\mathbf{x}) = \frac{1}{2}\| \mathbf{x}\| _2^2$ , in which case  $B(\mathbf{x}||\mathbf{y}) = \frac{1}{2}\| \mathbf{x} - \mathbf{y}\|_2^2$ ; another example is exponentiated gradient, for which  $\psi (\mathbf{p}) = \langle \mathbf{p},\log \mathbf{p}\rangle$  is the negative Shannon entropy on probability vectors  $\mathbf{p}\in \triangle$  and  $B$  is the KL-divergence [46]. An important property of  $B$  is that the sum over functions  $B(\mathbf{x}_t||\cdot)$  is minimized at the mean  $\bar{\mathbf{x}}$  of the points  $\mathbf{x}_1,\dots ,\mathbf{x}_T$ .
Algorithm 1: Tunes  $\mathrm{OMD}_{\eta, \theta}$  with regularizer  $\psi_{\theta}: \mathcal{K}^{\circ} \mapsto \mathbb{R}$  and step-size  $\eta > 0$ , which when run over loss estimators  $\hat{\ell}_{t,1}, \dots, \hat{\ell}_{t,m}$ , yielding task-optima  $\hat{\mathbf{x}}_t = \arg \min_{\mathbf{x} \in \mathcal{K}} \sum_{i=1}^{m} \langle \hat{\ell}_{t,i}, \mathbf{x} \rangle$ .
Input: compact set  $\mathcal{K}\subset \mathbb{R}^d$  , initialization  $\mathbf{x}_1\in \mathcal{K}$  , ordered subset  $\Theta_{k}\subset \mathbb{R}$  also used to index interval bounds  $\underline{\eta},\overline{\eta}\in \mathbb{R}_{\geq 0}^{k}$  and hyperparameters  $\alpha \in \mathbb{R}_{\geq 0}^{k}$  , scalar hyperparameters  $\rho >0$  and  $\lambda \geq 0$  , learners OMD,  $\theta :\mathcal{K}\mapsto \mathbb{R}^d$  projections  $\mathbf{c}_{\theta}:\mathcal{K}\mapsto \mathcal{K}_{\theta}$    
for  $\theta \in \Theta_k$  do   
 $\begin{array}{rlr}{\bf w}_1(\theta)\gets 1\mathrm{~and~}\eta_1(\theta)\leftarrow \frac{\underline{\eta}(\theta) + \overline{\eta}(\theta)}{2} & / / & \mathrm{initialize~MW~and~EWOO}\\ {\bf x}_t\gets \mathrm{OMD}_{\eta_t(\theta_t),\theta_t}({\bf c}_{\theta_t}({\bf x}_t)) & / / & \mathrm{run~bandit~OMD~within - task}\\ {\bf x}_{t + 1}\gets \frac{1}{t}\sum_{s = 1}^{t}\hat{\bf x}_s & / / & \mathrm{FTL~update~of~initialization}\\ {\bf for~}\theta \in \Theta_k\textbf{do}\\ {\bf \nabla}\eta_{t + 1}(\theta)\leftarrow \frac{\int_{\underline{\eta}(\theta)}^{\overline{\eta}(\theta)}v\exp\bigl (-\alpha(\theta)\sum_{s = 1}^{t}U_s^{(\rho)}({\bf x}_s,v,\theta)\bigr)dv}{\int_{\underline{\eta}(\theta)}^{\overline{\eta}(\theta)}\exp\bigl (-\alpha(\theta)\sum_{s = 1}^{t}U_s^{(\rho)}({\bf x}_s,v,\theta)\bigr)dv} & / / & \mathrm{EWOO~step - size~update}\\ {\bf w}_{t + 1}(\theta)\leftarrow {\bf w}_t(\theta) - \lambda U_t({\bf x}_t,\eta_t(\theta),\theta) / / & \mathrm{MW~update~of~tuning~parameter} \end{array}$
OMD on loss estimators  $\hat{\ell}_{t,i}$  constructed via partial feedback forms an important class of bandit methods [6, 2, 3]. Their regularizers  $\psi$  are often non-Lipschitz, e.g. the negative entropy, or even unbounded, e.g. the log-barrier. Thus full-information results for tuning OMD, e.g. by Khodak et al. [31] and Denevi et al. [19], do not suffice. We do adapt the former's approach of online learning a sequence  $U_{t}(\mathbf{x},\eta ,\theta)$  of affine functions of Bregman divergences from initializations  $\mathbf{x}$  to known points in  $\kappa$ . We are interested in them because the regret of OMD w.r.t. a comparator  $\mathbf{y}$  is bounded by  $B(\mathbf{y}||\mathbf{x}) / \eta +\mathcal{O}(\eta m)$  [46, 25]. In our case the comparator is based on the estimated optimum  $\hat{\mathbf{x}}_t\in \arg \min_{\mathbf{x}\in \mathcal{K}}\langle \hat{\ell}_t,\mathbf{x}\rangle$ , where  $\hat{\ell}_t = \sum_{i = 1}^m\hat{\ell}_{t,i}$ , resulting from running OMD on task  $t$  using initialization  $\mathbf{x}\in \kappa$  and hyperparameters  $\eta$  and  $\theta$ , which we denote  $\mathrm{ODM}_{\eta ,\theta}(\mathbf{x})$ . Unlike full-information meta-learning, we use a parameter  $\varepsilon >0$  to constrain this optimum to lie in a subset  $\kappa_{\varepsilon}\subset \kappa^{\circ}$ . Formally, we fix a point  $\mathbf{x}_1\in \kappa^\circ$  to be the "center"-e.g.  $\mathbf{x}_1 = \mathbf{1}_d / d$  when  $\kappa$  is the  $d$ -simplex  $\triangle$  and define the projection  $\mathbf{c}_{\varepsilon}(\mathbf{x}) = \mathbf{x}_1 + \frac{\mathbf{x} - \mathbf{x}_1}{1 + \varepsilon}$  mapping from  $\kappa$  to  $\kappa_{\varepsilon}$ . For example,  $\mathbf{c}_{\frac{\varepsilon}{1 - \varepsilon}}(\mathbf{x}) = (1 - \varepsilon)\mathbf{x} + \varepsilon \mathbf{1}_d / d$  on the simplex. This projection allows us to handle regularizers  $\psi$  that diverge near the boundary, but also introduces  $\varepsilon$ -dependent error terms. In the BLO case it also forces us to tune  $\varepsilon$  itself, as initializing too close to the boundary leads to unbounded regret while initializing too far away does not take advantage of task-similarity. Thus the general upper bounds of interest are the following functions of the initialization  $\mathbf{x}$ , the step-size  $\eta >0$ , and a third parameter  $\theta$  that is either  $\beta$  or  $\varepsilon$ , depending on the setting (MAB or BLO):
$$
U _ {t} (\mathbf {x}, \eta , \theta) = \frac {B _ {\theta} \left(\mathbf {c} _ {\theta} \left(\hat {\mathbf {x}} _ {t}\right) \right\| \mathbf {x})}{\eta} + \eta g (\theta) m + f (\theta) m \tag {3}
$$
Here  $B_{\theta}$  is the Bregman divergence of  $\psi_{\theta}$  while  $g(\theta)\geq 1$  and  $f(\theta)\geq 0$  are tunable constants. We overload  $\theta$  to be either  $\beta$  or  $\varepsilon$  for notational simplicity, as we will not tune them simultaneously; if  $\theta = \beta$  (for MAB) then  $c_{\theta}(\mathbf{x}) = \mathbf{x}_1 + \frac{\mathbf{x} - \mathbf{x}_1}{1 + \varepsilon}$  for fixed  $\varepsilon$ , while if  $\theta = \varepsilon$  (for BLO) then  $B_{\theta}$  is the Bregman divergence of a fixed  $\psi$ . The reason to optimize this sequence of upper bounds  $U_{t}$  is because they directly bound the task-averaged regret while being no worse than the worst-case single-task regret. Furthermore, an average over Bregman divergences is minimized at the average  $\hat{\mathbf{x}} = \frac{1}{T}\sum_{t = 1}^{T}\hat{\mathbf{x}}_t$ , where it attains the value  $\hat{V}_{\theta}^{2} = \frac{1}{T}\sum_{t = 1}^{T}\psi_{\theta}(\mathbf{c}_{\theta}(\hat{\mathbf{x}}_t)) - \psi_{\theta}(\mathbf{c}_{\theta}(\hat{\bar{\mathbf{x}}}))$  (c.f. Claim A.1). We will show that this quantity leads to intuitive and interpretable notions of task-similarity in all the applications we study.
2.2 A meta-algorithm for tuning bandit algorithms
To learn these functions  $U_{t}(\mathbf{x},\eta ,\theta)$  and thus to meta-learn  $\mathrm{OMD}_{\eta ,\theta}(\mathbf{x})$  our meta-algorithm sets  $\mathbf{x}$  to be the projection  $\mathbf{c}_{\theta}$  of the mean of the estimated optima—i.e. follow-the-leader (FTL) over the Bregman divergences in (3)—while simultaneously setting  $\eta$  via EWOO and  $\theta$  via discrete multiplicative weights (MW). We choose FTL, EWOO, and MW because each is well-suited to the way  $U_{t}$  depends on  $\mathbf{x}$ ,  $\eta$ , and  $\theta$ , respectively. First, the only effect of  $\mathbf{x}$  on  $U_{t}$  is via the Bregman
divergence  $B_{\theta}(\mathbf{c}_{\theta}(\hat{\mathbf{x}}_t)||\mathbf{x})$ , over which FTL attains logarithmic regret [31]. For  $\eta, U_t$  is exp-concave on  $\eta > 0$  so long as the first term is nonzero, but it is also non-Lipschitz; the EWOO algorithm is one of the few methods with logarithmic regret on exp-concave losses without a dependence on the Lipschitz constant [26], and we ensure the first term is nonzero by regularizing the upper bounds as follows for some  $\rho > 0$  and  $D_{\theta}^2 = \max_{\mathbf{x},\mathbf{y} \in \mathcal{K}_{\theta}} B_{\theta}(\mathbf{x}||\mathbf{y})$ :
$$
U _ {t} ^ {(\rho)} (\mathbf {x}, \eta , \theta) = \frac {B _ {\theta} \left(\mathbf {c} _ {\theta} \left(\hat {\mathbf {x}} _ {t}\right) \mid \mid \mathbf {x}\right) + \rho^ {2} D _ {\theta} ^ {2}}{\eta} + \eta g (\theta) m + f (\theta) m \tag {4}
$$
Note that this function is fully defined after obtaining  $\hat{\mathbf{x}}_t$  by running OMD on task  $t$ , which allows us to use full-information MW to tune  $\theta$  across the grid  $\Theta_k$ . Showing low regret w.r.t. any  $\theta \in \Theta \supset \Theta_k$  then just requires sufficiently large  $k$  and Lipschitzness of  $U_t$  w.r.t.  $\theta$ . Combining all three algorithms together thus yields the guarantee in Theorem 2.1, which is our main structural result. It implies a generic approach for obtaining meta-learning algorithms by (1) bounding the task-averaged regret by an average of functions of the form  $U_t$ , (2) applying the theorem to obtain a new bound  $o_T(1) + \min_{\theta, \eta} \frac{\hat{V}_\theta^2}{\eta} + \eta g(\theta)m + f(\theta)m$ , and (3) bounding the estimated task-similarity  $\hat{V}_\theta^2$  by an interpretable quantity. Crucially, since we can choose any  $\eta > 0$ , the asymptotic regret is always as good as the worst-case guarantee for running the base-learner separately on each task.
Theorem 2.1 (c.f. Thm. A.1). Suppose  $\mathbf{x}_1 = \arg \min_{\mathbf{x} \in \mathcal{K}} \psi_\theta(\mathbf{x}) \forall \theta$  and let  $D, M, F, S$  be maxima over  $\theta$  of  $D_\theta$ ,  $D_\theta \sqrt{g(\theta)m}$ ,  $f(\theta)$ , and  $\|\nabla^2 \psi_\theta\|_2$ , respectively. For each  $\rho \in (0,1)$  we can set  $\underline{\eta}, \overline{\eta}, \alpha$ , and  $\lambda$  s.t. the expected average of the losses  $U_t(\mathbf{c}_{\theta_t}(\mathbf{x}_t), \eta_t(\theta_t), \theta_t)$  of Algorithm 1 is at most
$$
\min  _ {\theta \in \Theta , \eta > 0} \frac {\mathbb {E} \hat {V} _ {\theta} ^ {2}}{\eta} + \eta g (\theta) m + f (\theta) m + \tilde {\mathcal {O}} \left(\frac {\frac {M}{\rho} + F m}{\sqrt {T}} + \frac {L _ {\eta}}{k} + \frac {M}{\rho^ {2} T} + \min  \left\{\frac {\rho^ {2} D ^ {2}}{\eta}, \rho M \right\} + \frac {S}{\eta T}\right) \tag {5}
$$
Here  $\hat{V}_{\theta}^{2} = \frac{1}{T}\sum_{t=1}^{T}\psi_{\theta}(\mathbf{c}_{\theta}(\hat{\mathbf{x}}_t)) - \psi_{\theta}(\mathbf{c}_{\theta}(\hat{\bar{\mathbf{x}}}))$  and  $L_{\eta}$  bounds the Lipschitz constant w.r.t.  $\theta$  at  $\hat{V}_{\theta}^{2}/\eta + \eta g(\theta)m + f(\theta)m$ . The same bound plus  $(M/\rho + Fm)\sqrt{\frac{1}{T}\log\frac{1}{\delta}}$  holds w.p.  $\geq 1 - \delta$ .