SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Bandit function-value (zeroth-order) feedback. Our setup concerns a bandit function-value feedback model, where the realized conversion and constraint functions in each period, namely  $f_{t}:\mathcal{X}\to \mathbb{R}_{\geq 0}$  and  $\pmb {g}_t:\mathcal{X}\rightarrow \mathbb{R}^K$  are never revealed to us, and we can only access their function values at our single lever decisions during the period. We also contrast our setup with a "two-point feedback" setup in online convex optimization literature, where in each period one can access function values twice (see [44] and references therein). Using the context of our setup, in a "two-point feedback" after nature samples  $f_{t}:\mathcal{X}\to \mathbb{R}_{\geq 0}$  and  $\pmb {g}_t:\mathcal{X}\to \mathbb{R}^K$  from  $\mathcal{P}_t$ , we can observe the values for two lever decisions  $\{\pmb{x}_t^{(1)},\pmb{x}_t^{(2)}\} \subseteq \mathcal{X}$ , namely  $(f_t(\pmb{x}_t^{(1)}),\pmb {g}_t(\pmb{x}_t^{(1)}))$  and  $(f_{t}(\pmb{x}_{t}^{(2)}),\pmb {g}_{t}(\pmb{x}_{t}^{(2)}))$ .
Universal algorithm for many worlds. Instead of developing separate algorithms that cope with each individual world described in Section 2.1, we aim to develop a single universal algorithm that performs well in every world without exploiting any knowledge on the realized input sequence  $\mathcal{P}_{1:T}$  or its structural properties.
The key component of our algorithm to handle the aforementioned challenges is the following Lagrangian function: for any  $f_{t}:\mathcal{X}\to \mathbb{R}_{\geq 0}$  and  $\pmb {g}_t:\mathcal{X}\rightarrow \mathbb{R}^K$  , let's define  $\mathcal{L}_t:\mathcal{X}\times \mathbb{R}_{\geq 0}^K\to \mathbb{R}$  such that
$$
\mathcal {L} _ {t} (\boldsymbol {x}, \boldsymbol {\lambda}) = f _ {t} (\boldsymbol {x}) + \boldsymbol {\lambda} ^ {\top} \boldsymbol {g} _ {t} (\boldsymbol {x}). \tag {4}
$$
Surrounding this Lagrangian function, our proposed algorithm is formally stated in Algorithm 1 with four key components described as follows.
1. Dual descent to decouple decisions across time. As our lever decisions over time should be somewhat intertwined with one another due to the presence of long term constraints, we decouple this cross-period dependency by lagrangifying the hindsight problem in Eq. (1) w.r.t. some benchmark dual variables. In each period  $t$ , we maintain an estimate of dual variables  $\lambda_{t} \in \mathbb{R}_{\geq 0}^{K}$  corresponding to each of our  $K$  constraints, and adopt a standard dual descent approach to dynamically adjust these estimates; see Eq. (9). In particular, after observing  $(f_{t}(\boldsymbol{x}_{t}), g_{t}(\boldsymbol{x}_{t}))$ , we update our dual variable by projecting each coordinate of  $\lambda_{t} - \eta \nabla_{\lambda} \mathcal{L}_{t}(\boldsymbol{x}_{t}, \lambda_{t}) = \lambda_{t} - \eta g_{t}(\boldsymbol{x}_{t})$  onto some interval  $[0, \frac{\bar{F}}{\beta}]$  where  $\eta > 0$  is the dual descent step size,  $\bar{F}$  is the maximum achievable conversion, and  $\beta > 0$  is some "safety buffer" to be defined later.
2. Primal ascent to handle bandit function-valued feedback in many worlds. Given a dual variable  $\lambda_{t}$  during period  $t$ , standard optimization frameworks suggest optimizing primal decisions (i.e., lever decisions) by setting  $\pmb{x}_{t} = \arg \max_{\pmb{x} \in \mathcal{X}} \mathbb{E}_{\mathcal{P}_{t}}[\mathcal{L}_{t}(\pmb{x}, \pmb{\lambda}_{t})]$ . However, in our setting this is not possible due to the bandit function-valued feedback structure: we do not know  $f_{t}(\cdot), g_{t}(\cdot)$  nor  $\mathcal{P}_{t}$ , and hence cannot optimize for  $\mathbb{E}_{\mathcal{P}_{t}}[\mathcal{L}_{t}(\cdot, \pmb{\lambda}_{t})]$ . To handle this, we view the primal decision problem as a bandit online convex optimization problem (BOCO) where the adversarial objective functions are  $\mathcal{L}_{t}(\cdot, \pmb{\lambda}_{t})$ , and run a BOCO algorithm to make primal lever decisions in each period.
Mathematically, we define the BOCO objective function  $h_t: \mathcal{X} \to \mathbb{R}$  as
$$
h _ {t} (\boldsymbol {x}) = \mathcal {L} _ {t} (\boldsymbol {x}, \boldsymbol {\lambda} _ {t}) = f _ {t} (\boldsymbol {x}) + \boldsymbol {\lambda} _ {t} ^ {\top} \boldsymbol {g} _ {t} (\boldsymbol {x}). \tag {5}
$$
Then, the BOCO algorithm constructs an estimate  $\nabla_t$  of  $\nabla h_t(\pmb{x})$  based on a random perturbation approach with perturbation parameter  $\rho > 0$  (see Eq. (7) and [25] for more details in random perturbations methods to estimate gradients). Next, the BOCO algorithm computes primal decision  $\pmb{x}_t$  by an ascent type update, i.e., projecting  $\pmb{x}_{t-1} + \gamma \pmb{\nabla}_t$  back onto  $\mathcal{X}$ , where  $\gamma > 0$  is the primal ascent step size; see Eq. (9). The loss we incur by running BOCO primal update instead of setting  $\arg \max_{\pmb{x} \in \mathcal{X}} \mathbb{E}_{\mathcal{P}_t}[h_t(\pmb{x}_t)]$  in the full information setting depends on the specific type of world we are in, i.e., the input sequence  $\mathcal{P}_{1:T}$ . We later show in Lemma 4.3 that input sequences in each world induce some comparator sequence  $\pmb{y}_1 \dots \pmb{y}_T \in \mathcal{X}$ , so the conversion loss due to BOCO primal ascent updates can be characterized as some BOCO "dynamic" regret  $\sum_{t \in [T]} h_t(\pmb{y}_t) - h_t(\pmb{x}_t^{BOCO})$ . Later in Lemma 4.6, we show that our BOCO algorithm achieves low dynamic regret against any comparator sequence  $\pmb{y}_1 \dots \pmb{y}_T \in \mathcal{X}$ .
3. Choosing primal ascent step sizes from expert advice for different worlds. Different worlds (and correspondingly different BOCO comparator sequences) may require different "optimal" BOCO primal ascent step sizes in order to achieve optimal regret guarantees. Since we are unaware of the world from which the input sequence  $\mathcal{P}_{1:T}$  is generated, on top of our primal ascent procedure we run a meta "expert" algorithm that allows us to adaptively approximate optimal step sizes in each world. In particular, we consider  $N$  experts, each corresponding to a primal ascent step size  $\gamma_{i} > 0$ ,
and produce independent lever decision based on primal ascent with her own step size; see Eq. (9). Then, we dynamically maintain exponential weights  $w_{t} \in \Delta_{N}$  in the  $N$ -dimensional simplex over  $N$  experts, based on their past performance measured by some surrogate loss function defined in Eq. (8). Finally, we set the weighted average primal decision over all experts as our ultimate primal decision; see Eq. (6). We later show in Lemma A.3 that such exponentially weighted average BOCO primal ascent decisions achieve low conversion loss compared to the "optimal" BOCO primal ascent expert (i.e., step size) in every world, without having to know which world we are in.
4. Constraint violation check to ensure constraint satisfaction. To ensure long-term constraint satisfaction, we maintain a constraint balance  $B_{k,t} = \sum_{\tau \in [t - 1]} g_{k,t}(\boldsymbol{x}_{\tau})$  for each constraint  $k \in [K]$  that keeps track of our deviation from satisfying the constraint. To make our problem tractable, we make the assumption that there is always a "safe action" with which we are always guaranteed to attain some small positive constraint balance. $^2$
Assumption 3.1 (Safe action). Assume there exists some  $\bar{\beta} > 0$  and an action  $\widetilde{\pmb{x}}_{\beta} \in \mathcal{X}$  such that for any  $(f, g) \in S$ , we have  $g_{k}(\widetilde{\pmb{x}}_{\beta}) > \bar{\beta}$  for all  $k \in [K]$ .
With the constraint balance, in each period before making a lever decision, we check the following: if by playing the safe action in all future rounds nearly violates constraints (step 2 in Algorithm 1), then we hard stop our algorithm and play the safe action in all subsequent rounds. We remark that we do not necessarily know the constraint function value lower bound  $\bar{\beta} > 0$  in Assumption 3.1. However, we show later in Theorem 4.1 that by considering any safety buffer  $\beta < \bar{\beta}$  (e.g., taking  $\beta = 1 / \log(T)$  for large enough  $T$ ), we are hard stopping more conservatively, and thus maintain constraint satisfaction. Note that our final regret scales with  $\frac{1}{\beta^2}$ ; see Lemma 4.6.
Input: Initial dual variable  $\lambda_{1} = 0$ , primal expert solutions  $\widetilde{\pmb{x}}_1^i = \mathbf{0}$  for any  $i\in [N]$ ; perturbation parameter  $\rho >0$  and  $\alpha \in (0,1)$ ; primal ascent step sizes for experts  $\{\gamma_1,\dots ,\gamma_N\} >0$ ; dual descent step size  $\eta$ ; learning rate of the meta-algorithm  $\epsilon$ ; initial expert weights  $w_{i,1} = 1$  for all  $i\in [N]$ ; safety buffer  $\beta >0$ .
1: Initialize constraint balance  $B_{k,\underline{1}} = 0$  for all  $k\in [K]$
2: while {for all  $k \in [K]$ ,  $B_{k,t} - G + \beta (T - t - 1) \geq 0$ } do
3: Compute exponentially weighted average forecaster:
$$
\tilde {\boldsymbol {x}} _ {t} = \frac {1}{\sum_ {i \in [ N ]} w _ {t} ^ {i}} \sum_ {i \in [ N ]} w _ {t} ^ {i} \tilde {\boldsymbol {x}} _ {t} ^ {i} \tag {6}
$$
4: Sample  $\pmb{u}_t\sim U(\mathbb{S})$  uniformly at random from the unit sphere.
5: Set  $\pmb{x}_t = \widetilde{\pmb{x}}_t + \rho \pmb{u}_t$  and observe  $f_{t}(\pmb{x}_{t})$  and  $\pmb{g}_t(\pmb{x}_t)$ . Update  $B_{k,t+1} = B_{k,t} + g_{k,t}(\pmb{x}_t)$ .
6: Construct gradient estimate for  $\nabla_{\pmb{x}}\mathcal{L}(\pmb{x}_t,\pmb{\lambda}_t)$
$$
\nabla_ {t} = \frac {d}{\rho} \left(f _ {t} \left(\boldsymbol {x} _ {t}\right) + \boldsymbol {\lambda} _ {t} ^ {\top} \boldsymbol {g} _ {t} \left(\boldsymbol {x} _ {t}\right)\right) \cdot \boldsymbol {u} _ {t} \tag {7}
$$
7: Update exponential weights for experts: Let  $\ell_t : \mathcal{X} \to \mathbb{R}$  be a surrogate loss function to measure the performance of each expert. Then we update expert weights by
$$
w _ {i, t + 1} = w _ {i, t} \exp \left(- \epsilon \ell_ {t} \left(\tilde {\boldsymbol {x}} _ {t} ^ {i}\right)\right) \quad \text {w h e r e} \quad \ell_ {t} (\boldsymbol {z}) = \boldsymbol {\nabla} _ {t} ^ {\top} \left(\tilde {\boldsymbol {x}} _ {t} - \boldsymbol {z}\right) \tag {8}
$$
8: Primal ascent per expert and dual descent:
$$
\tilde {\boldsymbol {x}} _ {t + 1} ^ {i} = \Pi_ {(1 - \alpha) \mathcal {X}} \left(\tilde {\boldsymbol {x}} _ {t} ^ {i} + \gamma_ {i} \boldsymbol {\nabla} _ {t}\right) \text {a n d} \boldsymbol {\lambda} _ {t + 1} = \Pi_ {[ 0, \frac {\bar {F}}{\beta} e ]} \left(\boldsymbol {\lambda} _ {t} - \eta \nabla_ {\boldsymbol {\lambda}} \boldsymbol {\mathcal {L}} _ {t} \left(\boldsymbol {x} _ {t}, \boldsymbol {\lambda} _ {t}\right)\right) _ {+} \tag {9}
$$
9: end while
10: Set stopping time  $\tau_{A} = t$ . For  $t = \tau_{A}\ldots T$  set safety option  $\pmb{x}_t = \widetilde{\pmb{x}}_\beta$ .
4 Performance analysis of our constrained BOCO algorithm
In this section, we analyze the performance of Algorithm 1 under input sequence  $\mathcal{P}_{1:T}$  which may be generated from any world described in Section 2.2. Our first result Lemma 4.1 shows that our algorithm maintains long-term constraint satisfaction almost surely for any input sequence.
2The existence of a safe action is not unnatural, and is studied in many literature; see [24, 21, 14]. For instance, let's recall our examples for constraint functions in Section 2.1 and assume we only have a long-term budget constraint. Then the corresponding safe action for the constraint function  $g_{k,t}(\pmb{x}_t) = B - x_{t,1}$  would be any  $\pmb{x}_t$  whose first entry  $x_{t,1} = 0$  so that we acquire positive constraint value  $B > 0$ .
Lemma 4.1 (Strict constraint satisfaction). Suppose Assumption 3.1 hold, and  $T$  is large enough so that the safety buffer  $\beta = \frac{1}{\log(T)} < \bar{\beta}$ , where  $\beta$  is defined in Assumption 3.1. Then, for any  $k \in [K]$ , we have  $\sum_{t \in [T]} g_{k,t}(\boldsymbol{x}_t) > 0$ .
The main result of this paper is the following Theorem 4.2, where we bound the regret of our proposed Algorithm 1 in all worlds specified in Section 2.2.
Theorem 4.2 (Bounding regret in many worlds). Let the safety buffer  $\beta = \frac{1}{\log(T)}$ , and the number of experts  $N = \max \left(1, \left\lceil -\log_2\left(K^{-\frac{1}{6}}(1 + DT)^{\frac{1}{2}}T^{-\frac{3}{4}}\right)\right\rceil + 1\right)$ , where  $K$  is the total number of constraints and  $D$  is diameter of the decision set  $\mathcal{X}$ . Choose the corresponding primal ascent step sizes for the  $N$  experts as  $\{\gamma_1 \dots \gamma_N\} = \{2^{-i}K^{-\frac{1}{6}}(1 + DT)^{\frac{1}{2}}T^{-\frac{3}{4}}: i = 0 \dots N - 1\}$ . Then by taking dual descent step size  $\eta = \frac{1}{\sqrt{KT}}$ , random perturbation parameter  $\rho = K^{\frac{1}{3}}T^{-\frac{1}{4}}$  and  $\alpha = \min \left(\frac{1}{2}, K^{\frac{1}{3}}T^{-\frac{1}{4}}\right)$ , and exponential weighted expert learning rate  $\epsilon = T^{-\frac{1}{2}}$ , we obtain the following bounds on  $\mathcal{R}_T$  in each input setting of interest: (1) Stochastic:  $\mathcal{R}_T \leq \mathcal{O}(T^{3/4})$ ; (2) Adversarial:  $\mathcal{R}_T \leq \left(1 - \frac{1}{\xi}\right)\mathrm{OPT}(\mathcal{P}_{1:T}) + \tilde{\mathcal{O}}\left(\sqrt{1 + P(\tilde{\mathbf{y}}_{1:T})} \cdot T^{\frac{3}{4}}\right)$ , where  $\xi = 1 - \frac{1}{\beta}\min_{(f,g) \in S} \min_{k \in [K], x \in \mathcal{X}} g_k(x) > 1$  under Assumption 2.1,  $\bar{\beta} > 0$  is defined in Assumption 3.1,  $\widetilde{\pmb{y}}_t = \arg \max_{\pmb{x} \in \mathcal{X}} f_t(\pmb{x}_t) + \pmb{\lambda}_t^\top \pmb{g}_t(\pmb{x}_t)$ , and  $P(\widetilde{\pmb{y}}_{1:T}) = \sum_{t \in [T-1]} \| \widetilde{\pmb{y}}_t - \widetilde{\pmb{y}}_{t+1} \|$ ; (3)  $\delta$ -corrupted:  $\mathcal{R}_T \leq \mathcal{O}(T^{3/4} + \delta)$ ; (4) Periodic:  $\mathcal{R}_T \leq \mathcal{O}(T^{3/4} + q\sqrt{T})$ ; (5) Ergodic:  $\mathcal{R}_T \leq \mathcal{O}(T^{3/4} + \kappa\sqrt{T})$ .
We also summarize our regret bounds in Table 1. Although it is always challenging to identify which input procedure the data currently comes from, Theorem 4.2 shows that the proposed Algorithm 1 achieves reasonable regret bound in five input settings without knowing which setting we are in. We remark that in the adversarial setting, our algorithm is  $(1 - \frac{1}{\xi})$ -competitive, namely, it can achieve at least a  $\frac{1}{\xi}$ -portion of the reward compared to OPT; later in Section 5 we comment that no sublinear regret is achievable. For the  $\delta$ -corrupted world, we corrupt  $\delta$  instances in the input sequence over time, so if  $\delta = 0$ , we recover the stochastic world and achieve diminishing  $\mathcal{O}(T^{3/4})$  regret; in the case we corrupt all samples, then we recover the adversarial case, and the regret becomes  $\mathcal{O}(T)$  as expected. In the periodic world, if period length  $q = 1$ , we recover the stochastic case.
Remark 4.1. We remark that our regret upper bound in the adversarial case depends on the dual variables  $\{\lambda_t\}_{t\in [T]}$  generated throughout the algorithm, which contrasts adversarial regret bounds in [44] that only depend on the total variance of the optimal sequence of 1. This is due to that fact that [44] handles a non-constrained online optimization problem, whereas we tackle a more difficult problem with long-term adversarial constraints that leads to more severe regret degradation as we ensure satisfaction of long-term adversarial constraints. We acknowledge that this bound should ideally be independent of dual variables, and we leave this for future work.
In the rest of this section, we present a proof scratch of Theorem 4.2, and the formal proof of the results can be found in Appendix A. The high-level idea of the proof is that we decompose total regret into three components as in Lemma 4.3 below, namely the conversion losses due to hard stopping, dual descent, and primal ascent, respectively. Then, we bound each component's regret.
Lemma 4.3 (Regret decomposition). Let  $\left(\lambda_t\in \mathbb{R}_{\geq 0}^K\right)_{t\in [T]}$  be the sequence of dual variables generated from Algorithm 1, and define  $h_t(\pmb {x}) = \mathcal{L}_t(\pmb {x},\pmb {\lambda}_t)$  where there Lagrangian function  $\mathcal{L}_t(\pmb {x},\pmb {\lambda})$  is defined in Eq. (4). Then we have
$$
\operatorname {O P T} \left(\mathcal {P} _ {1: T}\right) - \sum_ {t \in [ T ]} \mathbb {E} \left[ f _ {t} \left(\boldsymbol {x} _ {t}\right) \right] \leq \bar {F} (T - \tau_ {A}) + \sum_ {t \in [ \tau_ {A} ]} \boldsymbol {\lambda} _ {t} ^ {\top} \boldsymbol {g} _ {t} \left(\boldsymbol {x} _ {t}\right) + \mathcal {R} _ {\mathrm {B O C O}} (\tau_ {A}) \tag {10}
$$
where  $\mathcal{R}_{\mathrm{BOCO}}(\tau_A)$  admits the following bounds in each input setting: (1) Stochastic:  $\max_{\pmb{x} \in \mathcal{X}} \sum_{t \in [\tau_A]} h_t(\pmb{x}) - h_t(\pmb{x}_t)$ ; (2) Adversarial:  $\sum_{t \in [\tau_A]} h_t(\widetilde{\pmb{y}}_t) - h_t(\pmb{x}_t) + \left(1 - \frac{1}{\xi}\right) \mathrm{OPT}(\mathcal{P}_{1:T})$  where  $\widetilde{\pmb{y}}_t = \arg \max_{\pmb{x} \in \mathcal{X}} \mathbb{E}_{\mathcal{P}_t}[f_t(\pmb{x}) + \pmb{\lambda}_t^\top \pmb{g}_t(\pmb{x})]$ ; (3)  $\delta$ -corrupted:  $\max_{\pmb{x} \in \mathcal{X}} \sum_{t \in [\tau_A]} h_t(\pmb{x}) - h_t(\pmb{x}_t) + \mathcal{O}(\delta)$ ; (4) Periodic:  $\max_{\pmb{x} \in \mathcal{X}} \sum_{t \in [\tau_A]} h_t(\pmb{x}) - h_t(\pmb{x}_t) + \mathcal{O}(\eta qT)$ ; (5) Ergodic:  $\max_{\pmb{x} \in \mathcal{X}} \sum_{t \in [\tau_A]} h_t(\pmb{x}) - h_t(\pmb{x}_t) + \mathcal{O}(\eta \kappa T)$ .
Here, we remark that the first term  $\bar{F} (T - \tau_A)$  is the loss due to hard stopping to maintain long-term constraint satisfaction; the second term  $\sum_{t\in [\tau_A]}\pmb {\lambda}_t^\top \pmb {g}_t(\pmb {x}_t)$  is the loss due to dual descent; and the
final term  $\mathcal{R}_{\mathrm{BOCO}}(\tau_A)$  is the loss due to primal ascent. The first two components turn out to be identical in different settings, and the input model only affects the last term  $\mathcal{R}_{\mathrm{BOCO}}(\tau_A)$ . We next present our key results with which we use to bound first two components (dual descent and hard stop loss), and the third component (primal ascent loss), respectively.
Bounding dual descent and hard stop loss. Our strategy to bound the loss due to dual descent and hard stopping relies on bounding some cumulative "complementary slackness" induced from dual descent in the following Lemma 4.4. We note that the bound in Lemma 4.4 holds for any  $\lambda \in [\mathbf{0},\frac{\bar{F}}{\beta} e]$ , and thus by choosing appropriate  $\lambda$ , we can "internalize" the losses due to dual descent and hard stopping, as demonstrated in Lemma 4.5.
Lemma 4.4 (Bounding complementary slackness). For any  $\pmb{\lambda} \in [\mathbf{0}, \frac{\bar{F}}{\beta}\pmb{e}]$  and  $t \in [T]$ , we have  $\sum_{\tau \in [t]} (\pmb{\lambda}_{\tau} - \pmb{\lambda})^{\top} \pmb{g}_{\tau}(\pmb{x}_{\tau}) = \sum_{\tau \in [t]} (\pmb{\lambda}_{\tau} - \pmb{\lambda})^{\top} \nabla_{\pmb{\lambda}} \mathcal{L}_{\tau}(\pmb{x}_{\tau}, \pmb{\lambda}_{\tau}) \leq \frac{\eta}{2} t K \bar{G}^{2} + \frac{1}{2\eta} \| \pmb{\lambda} \|_{2}^{2}$
Lemma 4.5 (Internalizing dual descent and hard stop losses.). Given the stopping time  $\tau_{A} \in [T]$ , w.p. 1 we have  $\bar{F}(T - \tau_{A}) + \sum_{t \in [\tau_{A}]} \boldsymbol{\lambda}_{t}^{\top} \boldsymbol{g}_{t}(\boldsymbol{x}_{t}) \leq \frac{\eta}{2} T K \bar{G}^{2} + \frac{1}{2\eta} \left( \frac{\bar{F}}{\beta} \right)^{2} + \bar{F} + \frac{\bar{F}}{\beta} \bar{G}$ .
Bounding primal ascent loss (dynamic BOCO regret). We present our main result Lemma 4.6, which bounds the loss due to primal ascent for given any comparator sequence  $y_{1} \ldots y_{T} \in \mathcal{X}$ .
Lemma 4.6 (Bounding primal ascent (dynamic BOCO regret)). For any  $i \in [N]$ ,  $t \in [T]$  and any sequence  $\mathbf{y}_{1:t} \in \mathcal{X}^t$  we have  $\sum_{\tau \in [t]} h_{\tau}(\mathbf{y}_{\tau}) - h_{\tau}(\mathbf{x}_{\tau}) \leq \mathcal{O}\left(\frac{(\rho + \alpha)T}{\beta} + \frac{1 + P(\mathbf{y}_{1:T})}{\gamma_i} + \frac{\gamma_iKT}{\beta^2\rho^2} + T\epsilon + \frac{1}{\epsilon}\right)$ , where  $P(\mathbf{y}_{1:T}) = \sum_{t \in [T-1]} \| \mathbf{y}_t - \mathbf{y}_{t+1} \|$ , and parameters  $(\alpha, \gamma_i, \epsilon, \rho)$  are specified in Algorithm 1.
Proof sketch. We define the smoothed-versions of BOCO rewards  $h_t: \mathcal{X} \to \mathbb{R}$  as  $\hat{h}_t(\pmb{x}) = \mathbb{E}_{\pmb{v} \sim U(\mathbb{B})}[\mathcal{L}_t(\pmb{x} + \rho \pmb{v}, \pmb{\lambda}_t)]$ . Then, we decompose the primal ascent loss (BOCO dynamic regret)  $\sum_{\tau \in [t]} (h_\tau(\pmb{y}_\tau) - h_\tau(\pmb{x}_\tau))$  into 3 terms:
$$
\sum_ {\tau \in [ t ]} (h _ {\tau} (\boldsymbol {y} _ {\tau}) - \hat {h} _ {\tau} ((1 - \alpha) \boldsymbol {y} _ {\tau})) + \sum_ {\tau \in [ t ]} (\hat {h} _ {\tau} ((1 - \alpha) \boldsymbol {y} _ {\tau}) - \hat {h} _ {\tau} (\widetilde {\boldsymbol {x}} _ {\tau})) + \sum_ {\tau \in [ t ]} (\hat {h} _ {\tau} (\widetilde {\boldsymbol {x}} _ {\tau}) - h _ {\tau} (\boldsymbol {x} _ {\tau})),
$$
where we recall the  $\widetilde{\pmb{x}}_{\tau}$  is the un-perturbed version of our primal lever decision in Algorithm 1. The first and third summands can be directly be bounded via exploiting Lipschitz continuity properties for  $h_t$  (see Lemma A.1). The second summand can further be upper bounded (using Lemma A.2) by the surrogate loss difference between the primal decision weighted over all expert decisions and the comparator decision, namely  $\ell_{\tau}(\tilde{\pmb{x}}_{\tau}) - \ell_{\tau}((1 - \alpha)\pmb{y}_{\tau})$ . Then, we choose any expert  $i \in [N]$  corresponding to primal ascent step size  $\gamma_i > 0$  as an "intermediary" and decompose cumulative surrogate loss as  $\sum_{t \in [T]} (\ell_t(\tilde{\pmb{x}}_t) - \ell_t((1 - \alpha)\pmb{y}_t)) = \sum_{t \in [T]} (\ell_t(\tilde{\pmb{x}}_t) - \ell_t(\tilde{\pmb{x}}_t^i)) + \sum_{t \in [T]} (\ell_t(\tilde{\pmb{x}}_t^i) - \ell_t((1 - \alpha)\pmb{y}_t))$ . The first summand corresponds to the cumulative loss of our unperturbed primal decisions against any expert, and the second summand represents the cumulative loss of any expert w.r.t. the comparator sequence. Both summands are bounded in Lemma A.3.
We note that choosing different experts as the intermediary may yield different bounds. But since Lemma A.3 holds for any expert  $i \in [N]$ , we can consider the expert with the "optimal" primal ascent step size in each world, respectively.
5 Additional discussions
Lower bounds. A natural question to ask is whether regret bounds in Theorem 4.2 are optimal. Here, we look at lower bounds in "relaxed" problem settings, e.g., bandit online convex optimization with no-constraints [11, 44], or online constrained optimization under full-information feedback [8, 10]:
Stochastic	δ-corrupted	Adversarial	Periodic	Ergodic
Ω(√T) [19]	Ω(√T + δ) [10]	Ω((1 - 1/ξ)T) [8]	Ω(√qT)[10]	NA
These relaxed lower bounds are applicable to our setting (though they may not be the tightest bounds), and we can see that it may be possible to employ more complex machinery to attain better regret performances for our bandit convex online optimization setup with uncertain constraints. Yet, to identify an algorithm that can achieve the optimal regret in many worlds is an extremely challenging
open problem. For instance, even in the relaxed setting of bandit convex optimization with noconstraints, to the best of our knowledge there is no universal algorithm that achieves optimal regret in just the stochastic and adversarial worlds.
Other applications. Our proposed Algorithm 1 addresses a more general problem of online decision making with multi-dimensional decisions, bandit feedback and long-time uncertain constraints. That being said, our algorithm can also be applicable to other problem settings, for example: