SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Personalized recommendation and assortment in online retailing. Online retail platforms aim to optimize sales or revenue by recommending a limited assortment of products to customers. However, the likelihood of customer purchases given an assortment as well as their preferences are unknown, and different customer types may change over time due to evolving market trends. Platforms only observe bandit binary purchase decisions. Hence in our context, the decision set  $\mathcal{X}$  represents a probability simplex over  $d$  items. The online decisions  $\pmb{x}_t\in \mathcal{X}$  correspond to recommendation probabilities for each item. The function  $f_{t}(\pmb{x}_{t})$  reflects the revenue generated from customer purchases, while  $\pmb{g}_t(\pmb{x}_t)$  captures assortment capacity constraints, product inventory limits, or fairness considerations to ensure equal visibility for each product [18].
Real time posted pricing in E-commerce and cloud computing. In online E-commerce platforms such as Amazon and eBay, sellers set prices to sell various items to sequentially arriving customers [5, 34, 26]; and in cloud computing, operators of cloud services such as Alibaba Cloud (Alicloud) or Amazon Web Services (AWS) set prices for renting out different computing capacities (virtual machines, VMs etc.) upon customer requests [39]. The arrival of customer types as well as their demand may differ significantly over time (think about demand for sanitary products during pandemics outbursts, or computing resource demands during periodic business hours). Further, decision makers only get to observe (bandit) demand at the realized pricing decisions. Thereby, we can view  $\boldsymbol{x}_t \in \mathcal{X}$  as the vector of prices for various sold products,  $f_t(\boldsymbol{x}_t)$  as the generated revenue over a given period,  $g_t(\boldsymbol{x}_t)$  as real time product/resource capacity constraints, operating cost constraints, etc.
[1] Programmatic advertising trends, stats, & news. https://roirevolution.com/blog/programmatic-advertising-trends-stats-news/. Accessed: 2023-03-30.
[2] Programmatic digital display advertising in 2022: Ad spend, formats, and forecast. https://www.insiderintelligence.com/insights/programmatic-digital-display-ad-spending/. Accessed: 2023-03-30.
[3] Shipra Agrawal, Zizhuo Wang, and Yinyu Ye. A dynamic near-optimal algorithm for online linear programming. Operations Research, 62(4):876-890, 2014.
[4] Sanae Amani, Mahnoosh Alizadeh, and Christos Thrampoulidis. Linear stochastic bandits under safety constraints. Advances in Neural Information Processing Systems, 32, 2019.
[5] Moshe Babaioff, Shaddin Dughmi, Robert Kleinberg, and Aleksandrs Slivkins. Dynamic pricing with limited supply, 2015.
[6] Ashwinkumar Badanidiyuru, Robert Kleinberg, and Aleksandrs Slivkins. Bandits with knapsacks. Journal of the ACM (JACM), 65(3):1-55, 2018.
[7] Santiago Balseiro, Yuan Deng, Jieming Mao, Vahab Mirrokni, and Song Zuo. Robust auction design in the auto-bidding world. Advances in Neural Information Processing Systems, 34:17777-17788, 2021.
[8] Santiago Balseiro and Yonatan Gur. Learning in repeated auctions with budgets: Regret minimization and equilibrium. Management Science, 65(9):3952-3968, 2019.
[9] Santiago Balseiro, Anthony Kim, Mohammad Mahdian, and Vahab Mirrokni. Budget-management strategies in repeated auctions. Operations Research, 69(3):859-876, 2021.
[10] Santiago Balseiro, Haihao Lu, and Vahab Mirrokni. The best of many worlds: Dual mirror descent for online allocation problems. Operations Research, 71(1):101-119, 2023.
[11] Omar Besbes, Yonatan Gur, and Assaf Zeevi. Stochastic multi-armed-bandit problem with non-stationary rewards. Advances in Neural Information Processing Systems, 27, 2014.
[12] Mark Braverman, Jieming Mao, Jon Schneider, and Matt Weinberg. Selling to a no-regret buyer. In Proceedings of the 2018 ACM Conference on Economics and Computation, pages 523-538, 2018.
[13] Matteo Castiglioni, Andrea Celli, and Christian Kroer. Online learning with knapsacks: the best of both worlds. In International Conference on Machine Learning, pages 2767-2783. PMLR, 2022.
[14] Matteo Castiglioni, Andrea Celli, and Christian Kroer. Online bidding in repeated non-truthful auctions under budget and roi constraints. arXiv preprint arXiv:2302.01203, 2023.
[40] Wen Sun, Debadeepta Dey, and Ashish Kapoor. Safety-aware algorithms for adversarial contextual bandit. In International Conference on Machine Learning, pages 3280-3288. PMLR, 2017.
[41] Jonathan Weed, Vianney Perchet, and Philippe Rigollet. Online learning in repeated auctions. In Conference on Learning Theory, pages 1562-1583. PMLR, 2016.
[42] Scott Yang and Mehryar Mohri. Optimistic bandit convex optimization. Advances in Neural Information Processing Systems, 29, 2016.
[43] Jianjun Yuan and Andrew Lamperski. Online convex optimization for cumulative constraints. Advances in Neural Information Processing Systems, 31, 2018.
[44] Peng Zhao, Guanghui Wang, Lijun Zhang, and Zhi-Hua Zhou. Bandit convex optimization in non-stationary environments. The Journal of Machine Learning Research, 22(1):5562-5606, 2021.
Appendices for
A.1 Additional definitions for Section 4
Definition A.1 (Total variation between probability distributions). Consider two distributions  $\{\mathcal{P},\mathcal{P}^{\prime}\} \subseteq \Delta (\mathcal{S})$ . Then we define their total variation as  $\| \mathcal{P} - \mathcal{P}^{\prime}\|_{TV} = \frac{1}{2}\int_{\mathcal{S}}|\mathcal{P}(s) - \mathcal{P}^{\prime}(s)|ds$ .
We also define the smoothed version of  $h_t: \mathcal{X} \to \mathbb{R}$  (see Eq. (5)) for any  $t$  as follows:
$$
\hat {h} _ {t} (\boldsymbol {x}) = \mathbb {E} _ {\boldsymbol {v} \sim U (\mathbb {B})} \left[ \mathcal {L} _ {t} \left(\boldsymbol {x} + \rho \boldsymbol {v}, \boldsymbol {\lambda} _ {t}\right) \right] \tag {11}
$$
where we recall the Lagrangian function  $\mathcal{L}_t$  is defined in Eq. (4).
A.2 Additional lemmas for Section 4
Lemma A.1 (Lipschitz continuity). Let Assumption 2.1 hold, and we recall the definitions of  $h_t(\pmb{x})$  and  $\hat{h}_t(\pmb{x})$  from Eqs. (5) and (11), respectively, and recall  $\lambda_1 \ldots \lambda_T$  as the dual variables generated from Algorithm 1. Then for any  $\{\pmb{x}, \pmb{x}'\} \subseteq \mathcal{X}$ , we have  $|h_t(\pmb{x}) - h_t(\pmb{x}')| \leq (1 + K\frac{\bar{F}}{\beta})L \cdot \| \pmb{x} - \pmb{x}'\|$  and  $\left|h_t(\pmb{x}) - \hat{h}_t(\pmb{x})\right| \leq (1 + K\frac{\bar{F}}{\beta})L\rho$ .
Lemma A.2 (Bounding BOCO dynamic regret with surrogate loss). Recall the definition  $\hat{h}_t(\pmb{x}) = \mathbb{E}_{\pmb{v} \sim U(\mathbb{B})}[\mathcal{L}_t(\pmb{x} + \rho \pmb{v}, \pmb{\lambda}_t)]$ . Then,  $\hat{h}_t(\pmb{x})$  is concave. Further, for any  $\pmb{y} \in (1 - \alpha)\mathcal{X}$ , we have  $\hat{h}_t(\pmb{y}) - \hat{h}_t(\widetilde{\pmb{x}}_t) \leq \mathbb{E}_{\pmb{u}_t \sim U(\mathbb{S})}[\ell_t(\widetilde{\pmb{x}}_t) - \ell_t(\pmb{y})]$ , where  $\widetilde{\pmb{x}}_t$  is defined in Eq. (6), and the surrogate loss function  $\ell_t: \mathcal{X} \to \mathbb{R}$  is defined in Eq. (8).
Lemma A.3 (Bounding surrogate loss for each expert). Recall the definition of individual forecasters  $\widetilde{\pmb{x}}_t^i$  defined in Eq. (9), and the surrogate loss function  $\ell_t: \mathcal{X} \to \mathbb{R}$  defined in Eq. (8). Then for any  $i \in [N]$  and any sequence  $\pmb{y}_{1:T} \in \mathcal{X}^T$ , by defining  $P(\pmb{y}_{1:T}) = \sum_{t \in [T-1]} \| \pmb{y}_t - \pmb{y}_{t+1} \|$  (see Lemma 4.6), we have (i)  $\sum_{t \in [T]} (\ell_t(\widetilde{\pmb{x}}_t^i) - \ell_t((1 - \alpha)\pmb{y}_t)) \leq \mathcal{O}\left(\frac{1 + P(\pmb{y}_{1:T})}{\gamma_i} + \frac{\gamma_i}{\beta^2\rho^2} T\right)$  and (ii)  $\sum_{t \in [T]} (\ell_t(\widetilde{\pmb{x}}_t) - \ell_t(\widetilde{\pmb{x}}_t^i)) \leq \mathcal{O}(T\epsilon + \frac{1}{\epsilon})$ . where the constant  $\beta$  is specified in Algorithm 1. Here, recall  $D$  is the diameter of the decision set  $\mathcal{X}$ .
The proofs of Lemmas A.1, A.2,A.3 are shown in Appendices A.9, A.10, and A.11, respectively.
A.3 Proof for Lemma 4.1
Proof. For any  $k \in [K]$  we have
$$
\begin{array}{l} \sum_ {t \in [ T ]} g _ {k, t} (\boldsymbol {x} _ {t}) = \sum_ {t \in \left[ \tau_ {A} - 1 \right]} g _ {k, t} (\boldsymbol {x} _ {t}) + \sum_ {t = \tau_ {A}} ^ {T} g _ {k, t} (\boldsymbol {x} _ {t}) \stackrel {(a)} {\geq} \sum_ {t \in \left[ \tau_ {A} - 1 \right]} g _ {k, t} (\boldsymbol {x} _ {t}) + \bar {\beta} (T - \tau_ {A} + 1) \tag {12} \\ \geq \sum_ {t \in [ \tau_ {A} - 1 ]} g _ {k, t} (\boldsymbol {x} _ {t}) + \beta (T - \tau_ {A}) + \beta \stackrel {(b)} {\geq} \bar {G} + \beta > 0  , \\ \end{array}
$$
where in (a) we set  $\pmb{x}_t = \widetilde{\pmb{x}}_\beta$  for all  $t = \tau_A \dots T$  and  $g_{k,t}(\widetilde{\pmb{x}}_\beta) \geq \bar{\beta}$  for any  $k \in [K]$ ; (b) follows from the definition of the stopping time such that for any  $t' < \tau_A$  and  $k \in [K]$  we have  $\sum_{t \in [t']} g_{k,t}(\pmb{x}_t) - \bar{G} + \beta (T - t' - 1) \geq 0$ .
A.4 Proof for Lemma 4.4
Proof. It is easy to see  $\lambda_{t + 1} = \Pi_{[\mathbf{0},\frac{\bar{F}}{\beta}\mathbf{e}]}(\pmb {\lambda}_t - \eta \nabla_\pmb {\lambda}\mathcal{L}_t(\pmb {x}_t,\pmb {\lambda}_t))_+$  arg  $\min_{\pmb {\lambda}\in [\mathbf{0},\frac{\bar{F}}{\beta}\pmb {e}]}\nabla_{\pmb{\lambda}}\mathcal{L}_{t}(\pmb {x}_{t},\pmb {\lambda}_{t})^{\top}\pmb {\lambda} + \frac{1}{2\eta}\| \pmb {\lambda} - \pmb {\lambda}_{t}\|^{2}$ . By the first-order stationary condition
at  $\lambda_{t + 1}$ , we have for any  $\pmb {\lambda}\in [\pmb {0},\frac{\bar{F}}{\beta}\pmb {e}]$
$$
\left(\nabla_ {\boldsymbol {\lambda}} \mathcal {L} _ {t} (\boldsymbol {x} _ {t}, \boldsymbol {\lambda} _ {t}) + \frac {1}{\eta} (\boldsymbol {\lambda} _ {t + 1} - \boldsymbol {\lambda} _ {t})\right) ^ {\top} (\boldsymbol {\lambda} - \boldsymbol {\lambda} _ {t + 1}) \geq 0.
$$
Then for all  $\lambda \in \mathbb{R}_{\geq 0}^{K}$ , it follows that
$$
\begin{array}{l} \nabla_ {\boldsymbol {\lambda}} \mathcal {L} _ {t} \left(\boldsymbol {x} _ {t}, \boldsymbol {\lambda} _ {t}\right) ^ {\top} \left(\boldsymbol {\lambda} _ {t} - \boldsymbol {\lambda}\right) \\ = \nabla_ {\boldsymbol {\lambda}} \mathcal {L} _ {t} \left(\boldsymbol {x} _ {t}, \boldsymbol {\lambda} _ {t}\right) ^ {\top} \left(\boldsymbol {\lambda} _ {t} - \boldsymbol {\lambda} _ {t + 1}\right) + \nabla_ {\boldsymbol {\lambda}} \mathcal {L} _ {t} \left(\boldsymbol {x} _ {t}, \boldsymbol {\lambda} _ {t}\right) ^ {\top} \left(\boldsymbol {\lambda} _ {t + 1} - \boldsymbol {\lambda}\right) \\ \leq \nabla_ {\boldsymbol {\lambda}} \mathcal {L} _ {t} (\boldsymbol {x} _ {t}, \boldsymbol {\lambda} _ {t}) ^ {\top} (\boldsymbol {\lambda} _ {t} - \boldsymbol {\lambda} _ {t + 1}) + \frac {1}{\eta} (\boldsymbol {\lambda} _ {t + 1} - \boldsymbol {\lambda} _ {t}) ^ {\top} (\boldsymbol {\lambda} - \boldsymbol {\lambda} _ {t + 1}) \\ \leq \nabla_ {\boldsymbol {\lambda}} \mathcal {L} _ {t} (\boldsymbol {x} _ {t}, \boldsymbol {\lambda} _ {t}) ^ {\top} (\boldsymbol {\lambda} _ {t} - \boldsymbol {\lambda} _ {t + 1}) + \frac {1}{2 \eta} \| \boldsymbol {\lambda} - \boldsymbol {\lambda} _ {t} \| ^ {2} - \frac {1}{2 \eta} \| \boldsymbol {\lambda} - \boldsymbol {\lambda} _ {t + 1} \| ^ {2} - \frac {1}{2 \eta} \| \boldsymbol {\lambda} _ {t + 1} - \boldsymbol {\lambda} _ {t} \| ^ {2} \\ \leq \frac {\eta}{2} \| \nabla_ {\boldsymbol {\lambda}} \mathcal {L} _ {t} (\boldsymbol {x} _ {t}, \boldsymbol {\lambda} _ {t}) \| ^ {2} + \frac {1}{2 \eta} \| \boldsymbol {\lambda} - \boldsymbol {\lambda} _ {t} \| ^ {2} - \frac {1}{2 \eta} \| \boldsymbol {\lambda} - \boldsymbol {\lambda} _ {t + 1} \| ^ {2}. \\ \end{array}
$$
By a telescoping argument, we have
$$
\begin{array}{l} \sum_ {\tau \in [ t ]} \nabla_ {\pmb {\lambda}} \mathcal {L} _ {\tau} (\pmb {x} _ {\tau}, \pmb {\lambda} _ {\tau}) ^ {\top} (\pmb {\lambda} _ {\tau} - \pmb {\lambda}) \leq \frac {\eta}{2} \sum_ {\tau \in [ t ]} \| \nabla_ {\pmb {\lambda}} \mathcal {L} _ {\tau} (\pmb {x} _ {\tau}, \pmb {\lambda} _ {\tau}) \| ^ {2} + \frac {1}{2 \eta} \| \pmb {\lambda} - \pmb {\lambda} _ {1} \| ^ {2} \\ = \frac {\eta}{2} \sum_ {\tau \in [ t ]} \| \nabla_ {\boldsymbol {\lambda}} \mathcal {L} _ {\tau} \left(\boldsymbol {x} _ {\tau}, \boldsymbol {\lambda} _ {\tau}\right) \| ^ {2} + \frac {1}{2 \eta} \| \boldsymbol {\lambda} \| ^ {2}, \tag {13} \\ \end{array}
$$
where in the final equality we used  $\lambda_{1} = 0$ . Also,
$$
\left\| \nabla_ {\boldsymbol {\lambda}} \mathcal {L} _ {\tau} \left(\boldsymbol {x} _ {\tau}, \boldsymbol {\lambda} _ {\tau}\right) \right\| ^ {2} = \left\| \boldsymbol {g} _ {\tau} \left(\boldsymbol {x} _ {\tau}\right) \right\| ^ {2} \leq K \bar {G} ^ {2}. \tag {14}
$$
Hence, combining Eqs. (13) and (14), we get the desired bound.
A.5 Proof of Lemma 4.5
Proof. If  $\tau_A = T$ , taking  $\pmb{\lambda} = 0$  in Lemma 4.4 yields  $\sum_{t \in [T]} \pmb{\lambda}_t^\top \pmb{g}_t(\pmb{x}_t) \leq \frac{\eta}{2} TK\bar{G}^2$  and thus the desired inequality holds. If  $\tau_A < T$ , then there exists some  $k \in [K]$  such that  $\sum_{t \in [\tau_A]} g_{k,t}(\pmb{x}_t) - \bar{G} + \beta (T - \tau_A - 1) < 0$ , so by taking  $\pmb{\lambda} = \frac{\bar{F}}{\beta}\pmb{e}_k$  ( $\pmb{e}_k \in \mathbb{R}^K$  is the unit vector whose  $k$ th entry is 1) in Lemma 4.4 yields
$$
\begin{array}{l} \sum_ {t \in [ \tau_ {A} ]} \boldsymbol {\lambda} _ {t} ^ {\top} \boldsymbol {g} _ {t} (\boldsymbol {x} _ {t}) \leq \sum_ {t \in [ \tau_ {A} ]} \boldsymbol {\lambda} ^ {\top} \boldsymbol {g} _ {t} (\boldsymbol {x} _ {t}) + \frac {\eta}{2} T K \bar {G} ^ {2} + \frac {1}{2 \eta} \| \boldsymbol {\lambda} \| ^ {2} \\ = \frac {\bar {F}}{\beta} \sum_ {t \in [ \tau_ {A} ]} g _ {k, t} \left(\boldsymbol {x} _ {t}\right) + \frac {\eta}{2} T K \bar {G} ^ {2} + \frac {1}{2 \eta} \left(\frac {\bar {F}}{\beta}\right) ^ {2} \\ \leq - \frac {\bar {F}}{\beta} \cdot \beta (T - \tau_ {A} - 1) + \frac {\bar {F}}{\beta} \bar {G} + \frac {\eta}{2} T K \bar {G} ^ {2} + \frac {1}{2 \eta} \left(\frac {\bar {F}}{\beta}\right) ^ {2} \\ = - \bar {F} (T - \tau_ {A}) + \bar {F} + \frac {\bar {F}}{\beta} \bar {G} + \frac {\eta}{2} T K \bar {G} ^ {2} + \frac {1}{2 \eta} \left(\frac {\bar {F}}{\beta}\right) ^ {2}, \\ \end{array}
$$
A.6 Proof of Lemma 4.6
Proof. Recall the definition of  $h_t(x)$  in Eq. (5). Then, we have
$$
\begin{array}{l} \sum_ {\tau \in [ t ]} h _ {\tau} (\boldsymbol {y} _ {\tau}) - \sum_ {\tau \in [ t ]} h _ {\tau} (\boldsymbol {x} _ {\tau}) \\ = \sum_ {\tau \in [ t ]} \left(\underbrace {h _ {\tau} (\boldsymbol {y} _ {\tau}) - \hat {h} _ {\tau} ((1 - \alpha) \boldsymbol {y} _ {\tau})} _ {A} + \underbrace {\hat {h} _ {\tau} ((1 - \alpha) \boldsymbol {y} _ {\tau}) - \hat {h} _ {\tau} (\widetilde {\boldsymbol {x}} _ {\tau})} _ {B} + \underbrace {\hat {h} _ {\tau} (\widetilde {\boldsymbol {x}} _ {\tau}) - h _ {\tau} (\boldsymbol {x} _ {\tau})} _ {C}\right) \tag {15} \\ \end{array}
$$
Bounding  $A$
$$
\begin{array}{l} h _ {\tau} (\boldsymbol {y} _ {\tau}) - \hat {h} _ {\tau} ((1 - \alpha) \boldsymbol {y} _ {\tau}) = h _ {\tau} (\boldsymbol {y} _ {\tau}) - h _ {\tau} ((1 - \alpha) \boldsymbol {y} _ {\tau}) + h _ {\tau} ((1 - \alpha) \boldsymbol {y} _ {\tau}) - \hat {h} _ {\tau} ((1 - \alpha) \boldsymbol {y} _ {\tau}) \\ \stackrel {(a)} {\leq} \left(1 + K \frac {\bar {F}}{\beta}\right) L \alpha \| \boldsymbol {y} _ {\tau} \| + \left(1 + K \frac {\bar {F}}{\beta}\right) L \rho \tag {16} \\ \stackrel {(b)} {\leq} \left(1 + K \frac {\bar {F}}{\beta}\right) L \alpha D + \left(1 + K \frac {\bar {F}}{\beta}\right) L \rho , \\ \end{array}
$$
where (a) follows from Lemma A.1; (b) follows from  $\| \pmb{y}_{\tau}\| = \| \pmb{y}_{\tau} - \pmb{0}\| \leq D$  since we assumed  $\mathbf{0}\in \mathcal{X}$ .
Bounding  $B$
$$
\begin{array}{l} \sum_ {\tau \in [ t ]} \hat {h} _ {\tau} ((1 - \alpha) \boldsymbol {y} _ {\tau}) - \hat {h} _ {\tau} (\widetilde {\boldsymbol {x}} _ {\tau}) \\ \stackrel {(a)} {\leq} \sum_ {\tau \in [ t ]} \mathbb {E} _ {\boldsymbol {u} _ {\tau} \sim U (\mathbb {S})} \left[ \ell_ {\tau} \left(\widetilde {\boldsymbol {x}} _ {\tau}\right) - \ell_ {\tau} \left((1 - \alpha) \boldsymbol {y} _ {\tau}\right) \right] \tag {17} \\ = \sum_ {\tau \in [ t ]} \mathbb {E} _ {\boldsymbol {u} _ {\tau} \sim U (\mathbb {S})} \left[ \ell_ {\tau} \left(\widetilde {\boldsymbol {x}} _ {\tau}\right) - \ell_ {\tau} \left(\widetilde {\boldsymbol {x}} _ {\tau} ^ {i}\right) + \ell_ {\tau} \left(\widetilde {\boldsymbol {x}} _ {\tau} ^ {i}\right) - \ell_ {\tau} ((1 - \alpha) \boldsymbol {y} _ {\tau}) \right] \\ \stackrel {(b)} {\leq} \mathcal {O} \left(\frac {P (\boldsymbol {y} _ {1 : T})}{\gamma_ {i}} + \frac {\gamma_ {i} K _ {\beta} ^ {\bar {F}} T}{\rho^ {2}} + T \epsilon + \frac {1}{\epsilon}\right), \\ \end{array}
$$
where (a) follows from Lemma A.2 and (b) follows from Lemma A.3 (i) and (ii).
Bounding  $C$
$$
\begin{array}{l} \hat {h} _ {\tau} \left(\widetilde {\boldsymbol {x}} _ {\tau}\right) - h _ {\tau} \left(\boldsymbol {x} _ {\tau}\right) = \hat {h} _ {\tau} \left(\widetilde {\boldsymbol {x}} _ {\tau}\right) - h _ {\tau} \left(\widetilde {\boldsymbol {x}} _ {\tau}\right) + h _ {\tau} \left(\widetilde {\boldsymbol {x}} _ {\tau}\right) - h _ {\tau} \left(\boldsymbol {x} _ {\tau}\right) \\ \stackrel {(a)} {\leq} \left(1 + K \frac {\bar {F}}{\beta}\right) L \rho + \left(1 + K \frac {\bar {F}}{\beta}\right) L \cdot \| \widetilde {\boldsymbol {x}} _ {\tau} - \boldsymbol {x} _ {\tau} \| \\ \stackrel {(b)} {=} (1 + K \frac {\bar {F}}{\beta}) L \rho + (1 + K \frac {\bar {F}}{\beta}) L \cdot \| \rho \boldsymbol {u} _ {\tau} \| \tag {18} \\ \leq 2 \rho (1 + K \frac {\bar {F}}{\beta}) L, \\ \end{array}
$$
where (a) follows Lemma A.1; (b) is from the definition  $\pmb{x}_{\tau} = \widetilde{\pmb{x}}_{\tau} + \rho \pmb{u}_{\tau}$  in Algorithm 1.
A.7 Proof of Lemma 4.3
In this section, we provide upper bounds of the regret term under five different environments of input procedures: stochastic, adversarial,  $\delta$ -corrupted, periodic, and ergodic. The structure of the following proof might seem similar to [10]; see [10, Sections 3-5]. However, we note that the proof techniques are fundamentally different since our paper considers the bandit feedback environment while [10] makes sequential decisions after observations.
Stochastic.
Proof. In the stochastic regime, we have  $\mathcal{P} = \mathcal{P}_1 = \dots = \mathcal{P}_T$  for some  $\mathcal{P}$ , and therefore we can rewrite  $\mathrm{OPT}(\mathcal{P}_{1:T})$  in Eq. (1) as followed
$$
\mathrm {O P T} (\mathcal {P} _ {1: T}) = \max  _ {\boldsymbol {x} _ {1: T} \in \mathcal {X} ^ {T}} \sum_ {t \in [ T ]} F (\boldsymbol {x} _ {t}) \quad \text {s . t .} \sum_ {t \in [ T ]} G (\boldsymbol {x} _ {t}) \geq \mathbf {0}.
$$
where we defined  $F(\pmb{x}) = \mathbb{E}_{(f,\pmb{g})\sim \mathcal{P}}[f(\pmb{x})]$ , and  $G(\pmb{x}) = \mathbb{E}_{(f,\pmb{g})\sim \mathcal{P}}[\pmb{g}(\pmb{x})]$  for any  $\pmb{x} \in \mathcal{X}$ . Hence, for any  $\lambda \geq 0$  we have
$$
\begin{array}{l} \mathrm {O P T} \left(\mathcal {P} _ {1: T}\right) = \frac {T - \tau_ {A}}{T} \mathrm {O P T} \left(\mathcal {P} _ {1: T}\right) + \frac {\tau_ {A}}{T} \mathrm {O P T} \left(\mathcal {P} _ {1: T}\right) \\ \leq (T - \tau_ {A}) \bar {F} + \frac {\tau_ {A}}{T} \max _ {\boldsymbol {x} _ {1: T} \in \mathcal {X} ^ {T}} \sum_ {t \in [ T ]} \left(F (\boldsymbol {x} _ {t}) + \boldsymbol {\lambda} ^ {\top} \boldsymbol {G} (\boldsymbol {x} _ {t})\right) \\ = (T - \tau_ {A}) \bar {F} + \frac {\tau_ {A}}{T} \max  _ {\boldsymbol {x} \in \mathcal {X}} \sum_ {t \in [ T ]} \left(F (\boldsymbol {x}) + \boldsymbol {\lambda} ^ {\top} \boldsymbol {G} (\boldsymbol {x})\right) \\ = (T - \tau_ {A}) \bar {F} + \tau_ {A} \max  _ {\boldsymbol {x} \in \mathcal {X}} \left(F (\boldsymbol {x}) + \boldsymbol {\lambda} ^ {\top} G (\boldsymbol {x})\right), \\ \end{array}
$$
where in the inequality we applied Assumption 2.1 which states  $\sup_{\pmb{x} \in \mathcal{X}} |f(\pmb{x})| \leq \bar{F}$  for all  $(f, g) \in S$ . Choosing  $\lambda = \bar{\lambda}_{\tau_A} \coloneqq \frac{1}{\tau_A} \sum_{t \in [\tau_A]} \lambda_t$  we have
$$
\begin{array}{l} \operatorname {O P T} \left(\mathcal {P} _ {1: T}\right) \leq \mathbb {E} \left[ \left(T - \tau_ {A}\right) \bar {F} + \tau_ {A} \max  _ {\boldsymbol {x} \in \mathcal {X}} \left(F (\boldsymbol {x}) + \boldsymbol {\lambda} ^ {\top} \boldsymbol {G} (\boldsymbol {x})\right) \right] \\ \leq \mathbb {E} \left[ (T - \tau_ {A}) \bar {F} + \max  _ {\boldsymbol {x} \in \mathcal {X}} \sum_ {t \in [ \tau_ {A} ]} \left(F (\boldsymbol {x}) + \boldsymbol {\lambda} _ {t} ^ {\top} \boldsymbol {G} (\boldsymbol {x})\right) \right] \\ \stackrel {(a)} {\leq} \mathbb {E} \left[ (T - \tau_ {A}) \bar {F} + \max  _ {\boldsymbol {x} \in \mathcal {X}} \sum_ {t \in [ \tau_ {A} ]} \mathbb {E} \left[ f _ {t} (\boldsymbol {x}) + \boldsymbol {\lambda} _ {t} ^ {\top} \boldsymbol {g} _ {t} (\boldsymbol {x}) \mid \sigma \left(\mathcal {H} _ {t - 1}\right) \right] \right] \tag {20} \\ \stackrel {(b)} {=} \mathbb {E} \left[ (T - \tau_ {A}) \bar {F} + \max  _ {\boldsymbol {x} \in \mathcal {X}} \sum_ {t \in [ \tau_ {A} ]} \mathbb {E} \left[ h _ {t} (\boldsymbol {x}) \mid \sigma (\mathcal {H} _ {t - 1}) \right] \right] \\ \leq \mathbb {E} \Big [ (T - \tau_ {A}) \bar {F} + \max _ {\boldsymbol {x} \in \mathcal {X}} \sum_ {t \in [ \tau_ {A} ]} h _ {t} (\boldsymbol {x}) \Big ], \\ \end{array}
$$
where in (a) we used the fact that  $\lambda_{t}$  is  $\mathcal{H}_{t - 1}$ -measurable; in (b) we used definitions  $h_t(\pmb {x}) =$ $\mathcal{L}_t(\pmb {x};\pmb {\lambda}_t)$  and  $\mathcal{L}_t(\pmb {x};\pmb {\lambda}) = f_t(\pmb {x}) + \pmb{\lambda}^\top \pmb {g}_t(\pmb {x})$  in Eqs. (4) and (5) respectively.
$$
f _ {t} \left(\boldsymbol {x} _ {t}\right) = h _ {t} \left(\boldsymbol {x} _ {t}\right) - \boldsymbol {\lambda} _ {t} ^ {\top} \boldsymbol {g} _ {t} \left(\boldsymbol {x} _ {t}\right), \tag {21}
$$
so combining this with Eq. (20) we have
$$
\operatorname {O P T} \left(\mathcal {P} _ {1: T}\right) - \sum_ {t \in [ T ]} \mathbb {E} \left[ f _ {t} \left(\boldsymbol {x} _ {t}\right) \right] \leq \mathbb {E} \left[ \left(T - \tau_ {A}\right) \bar {F} + \max  _ {\boldsymbol {x} \in \mathcal {X}} \sum_ {t \in \left[ \tau_ {A} \right]} \left(h _ {t} (\boldsymbol {x}) - h _ {t} \left(\boldsymbol {x} _ {t}\right)\right) + \sum_ {t \in \tau_ {A}} \boldsymbol {\lambda} _ {t} ^ {\top} \boldsymbol {g} _ {t} \left(\boldsymbol {x} _ {t}\right) \right] \tag {22}
$$
where we also used the fact that  $f_{t}(\pmb {x})\geq 0$  for all  $t = \tau_A + 1\dots T$  and  $\pmb {x}\in \mathcal{X}$
Adversarial.
Proof. Recall the definition of  $\xi$  is Theorem 4.2:
$$
\xi = 1 - \frac {\min  _ {(f , g) \in \mathcal {S}} \min  _ {k \in [ K ] , \boldsymbol {x} \in \mathcal {X}} g _ {k} (\boldsymbol {x})}{\bar {\beta}} > 1. \tag {23}
$$
For any  $t\in [T]$  , define  $\widetilde{\pmb{y}}_t = \arg \max_{\pmb{x}}f_t(\pmb {x}) + \pmb{\lambda}_t^\top \pmb {g}_t(\pmb {x})$
By comparing to the safety action  $\pmb{x}_{\beta} \in \mathcal{X}$  which ensures  $g_{k}(\pmb{x}_{\beta}) \geq \bar{\beta}$  for any  $k \in [K]$  and  $(f, g) \in S$ , as well as the optimal hindsight action  $\pmb{x}_t^* \in \mathcal{X}$  (i.e.,  $\pmb{x}_1^* \dots \pmb{x}_T^*$  is the optimal decision sequence to OPT  $(\mathcal{P}_{1:T})$ ), we have
$$
f _ {t} \left(\widetilde {\boldsymbol {y}} _ {t}\right) + \boldsymbol {\lambda} _ {t} ^ {\top} \boldsymbol {g} _ {t} \left(\widetilde {\boldsymbol {y}} _ {t}\right) \geq f _ {t} \left(\boldsymbol {x} _ {\beta}\right) + \boldsymbol {\lambda} _ {t} ^ {\top} \boldsymbol {g} _ {t} \left(\boldsymbol {x} _ {\beta}\right) \geq \bar {\beta} \boldsymbol {\lambda} _ {t} ^ {\top} \boldsymbol {e} \tag {24}
$$
and  $f_{t}(\widetilde{\boldsymbol{y}}_{t}) + \boldsymbol{\lambda}_{t}^{\top}\boldsymbol{g}_{t}(\widetilde{\boldsymbol{y}}_{t})\geq f_{t}(\boldsymbol{x}_{t}^{*}) + \boldsymbol{\lambda}_{t}^{\top}\boldsymbol{g}_{t}(\boldsymbol{x}_{t}^{*}).$