SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \xi f _ {t} \left(\widetilde {\mathbf {y}} _ {t}\right) = f _ {t} \left(\widetilde {\mathbf {y}} _ {t}\right) + (\xi - 1) f _ {t} \left(\widetilde {\mathbf {y}} _ {t}\right) \\ \stackrel {(a)} {\geq} f _ {t} \left(\boldsymbol {x} _ {t} ^ {*}\right) + \boldsymbol {\lambda} _ {t} ^ {\top} \boldsymbol {g} _ {t} \left(\boldsymbol {x} _ {t} ^ {*}\right) - \boldsymbol {\lambda} _ {t} ^ {\top} \boldsymbol {g} _ {t} \left(\widetilde {\boldsymbol {y}} _ {t}\right) + (\xi - 1) \left(- \boldsymbol {\lambda} _ {t} ^ {\top} \boldsymbol {g} _ {t} \left(\widetilde {\boldsymbol {y}} _ {t}\right) + \bar {\beta} \boldsymbol {\lambda} _ {t} ^ {\top} \boldsymbol {e}\right) \\ = f _ {t} \left(\boldsymbol {x} _ {t} ^ {*}\right) + \boldsymbol {\lambda} _ {t} ^ {\top} \boldsymbol {g} _ {t} \left(\boldsymbol {x} _ {t} ^ {*}\right) - \xi \boldsymbol {\lambda} _ {t} ^ {\top} \boldsymbol {g} _ {t} \left(\widetilde {\boldsymbol {y}} _ {t}\right) + (\xi - 1) \bar {\beta} \boldsymbol {\lambda} _ {t} ^ {\top} \boldsymbol {e} \tag {25} \\ \stackrel {(b)} {\geq} f _ {t} \left(\boldsymbol {x} _ {t} ^ {*}\right) - \xi \boldsymbol {\lambda} _ {t} ^ {\top} \boldsymbol {g} _ {t} \left(\widetilde {\boldsymbol {y}} _ {t}\right) \\ \end{array}
$$
where (a) follows Eq. (24); in (b) we used the fact that  $g_{k,t}(\boldsymbol{x}_t^*) + (\xi - 1)\bar{\beta} \geq 0$  since we have  $\min_{(f,\boldsymbol{g}) \in \mathcal{S}} \min_{k \in [K], \boldsymbol{x} \in \mathcal{X}} (g_{k,t}(\boldsymbol{x}) + (\xi - 1)\bar{\beta}) \geq 0$  (see Eq. (23)). Hence we have
$$
\begin{array}{l} \operatorname {O P T} \left(\mathcal {P} _ {1: T}\right) - \sum_ {t \in [ T ]} \mathbb {E} \left[ f _ {t} \left(\boldsymbol {x} _ {t}\right) \right] \\ = \left(1 - \frac {1}{\xi}\right) \operatorname {O P T} \left(\mathcal {P} _ {1: T}\right) + \sum_ {t \in [ T ]} \mathbb {E} \left[ \frac {1}{\xi} f _ {t} \left(\boldsymbol {x} _ {t} ^ {*}\right) - f _ {t} \left(\boldsymbol {x} _ {t}\right) \right] \\ \leq \left(1 - \frac {1}{\xi}\right) \operatorname {O P T} \left(\mathcal {P} _ {1: T}\right) + \sum_ {t \in [ T ]} \mathbb {E} \left[ f _ {t} \left(\widetilde {\boldsymbol {y}} _ {t}\right) - f _ {t} \left(\boldsymbol {x} _ {t}\right) + \boldsymbol {\lambda} _ {t} ^ {\top} \boldsymbol {g} _ {t} \left(\widetilde {\boldsymbol {y}} _ {t}\right) \right] \tag {26} \\ \leq \left(1 - \frac {1}{\xi}\right) \mathrm {O P T} \left(\mathcal {P} _ {1: T}\right) + \mathbb {E} \left[ (T - \tau_ {A}) \bar {F} + \sum_ {t \in \tau_ {A}} \left(f _ {t} (\widetilde {\boldsymbol {y}} _ {t}) - f _ {t} (\boldsymbol {x} _ {t}) + \boldsymbol {\lambda} _ {t} ^ {\top} \boldsymbol {g} _ {t} (\widetilde {\boldsymbol {y}} _ {t})\right) \right], \\ \end{array}
$$
$\delta$  -corrupted.
Here, we will prove a more general  $\delta$ -corrupted model where the input distribution sequence  $\mathcal{P}_{1:T}$  satisfies the following:
$$
\sum_ {t \in [ T ]} \| \mathcal {P} _ {t} - \frac {1}{T} \sum_ {s \in [ T ]} \mathcal {P} _ {s} \| _ {T V} \leq \delta , \tag {27}
$$
where the total variation norm is defined in Definition A.1. In fact, the definition in Section 2.2 for the  $\delta$ -corrupted regime satisfies the above property: recall the definition in Section 2.2, there exists  $\mathcal{P} \in \Delta(\mathcal{S})$  as well as  $\delta \in \mathbb{N}$  periods  $\mathcal{T} = \{\tau_1 \dots \tau_\delta\} \subset [T]$  such that  $\mathcal{P}_t = \mathcal{P}$  for all  $t \notin \mathcal{T}$ , so for any  $t \notin \mathcal{T}$ , we have
$$
\begin{array}{l} \| \mathcal {P} _ {t} - \frac {1}{T} \sum_ {s \in [ T ]} \mathcal {P} _ {s} \| _ {T V} = \| \mathcal {P} - \frac {1}{T} \left(T \mathcal {P} + \sum_ {s \in \mathcal {T}} (\mathcal {P} _ {s} - \mathcal {P})\right) \| _ {T V} \\ = \| \frac {1}{T} \sum_ {s \in \mathcal {T}} (\mathcal {P} - \mathcal {P} _ {s}) \| _ {T V} \leq \frac {\delta}{2 T}. \\ \end{array}
$$
On the other hand, we have for any  $t \in \mathcal{T}$ ,  $\| \mathcal{P}_t - \frac{1}{T}\sum_{s \in [T]}\mathcal{P}_s\|_{TV} \leq \frac{1}{2}$ . After summing it up, we get
$$
\begin{array}{l} \sum_ {t \in [ T ]} \| \mathcal {P} _ {t} - \frac {1}{T} \sum_ {s \in [ T ]} \mathcal {P} _ {s} \| _ {T V} = \sum_ {t \in \mathcal {T}} \| \mathcal {P} _ {t} - \frac {1}{T} \sum_ {s \in [ T ]} \mathcal {P} _ {s} \| _ {T V} + \sum_ {t \notin \mathcal {T}} \| \mathcal {P} _ {t} - \frac {1}{T} \sum_ {s \in [ T ]} \mathcal {P} _ {s} \| _ {T V} \\ \leq \frac {\delta}{2} + (T - \delta) \frac {\delta}{2 T} \leq \delta , \\ \end{array}
$$
which coincides with our general definition of  $\delta$ -corruption in Eq. (27).
We now prove the  $\delta$ -corruption regime under the general definition in Eq. (27). Define  $\widetilde{\mathcal{P}} = \frac{1}{T}\sum_{s\in [T]}\mathcal{P}_s$ ,  $\widetilde{F} (\pmb {x}) = \mathbb{E}_{(f,\pmb {g})\sim \widetilde{\mathcal{P}}}[f(\pmb {x})]$ ,  $\widetilde{\pmb{G}} (\pmb {x}) = \mathbb{E}_{(f,\pmb {g})\sim \widetilde{\mathcal{P}}}[\pmb {g}(\pmb {x})]$ ,  $F_{t}(\pmb {x}) = \mathbb{E}_{(f,\pmb {g})\sim \mathcal{P}_{t}}[f(\pmb {x})]$  and  $\pmb {G}_t(\pmb {x}) = \mathbb{E}_{(f,\pmb {g})\sim \mathcal{P}_t}[\pmb {g}(\pmb {x})]$  for all  $t\in [T]$  and any  $x\in \mathcal{X}$ . Then for any  $\lambda \in [\mathbf{0},\bar{\underline{F}}\pmb {e}]$ , we have
$$
\begin{array}{l} \mathrm {O P T} \left(\mathcal {P} _ {1: T}\right) \leq \max  _ {\boldsymbol {x} _ {1: T} \in \mathcal {X} ^ {T}} \sum_ {t \in [ T ]} \left(F _ {t} (\boldsymbol {x} _ {t}) + \boldsymbol {\lambda} ^ {\top} \boldsymbol {G} _ {t} (\boldsymbol {x} _ {t})\right) \\ \leq \max  _ {\boldsymbol {x} _ {1: T} \in \mathcal {X} ^ {T}} \sum_ {t \in [ T ]} \left(\widetilde {F} \left(\boldsymbol {x} _ {t}\right) + \boldsymbol {\lambda} ^ {\top} \widetilde {\boldsymbol {G}} \left(\boldsymbol {x} _ {t}\right)\right) + (\bar {F} + \bar {G} K \frac {\bar {F}}{\beta}) \delta \tag {29} \\ = T \cdot \max  _ {\boldsymbol {x} \in \mathcal {X}} (\widetilde {F} (\boldsymbol {x}) + \boldsymbol {\lambda} ^ {\top} \widetilde {\boldsymbol {G}} (\boldsymbol {x})) + (\bar {F} + \bar {G} K \frac {\bar {F}}{\beta}) \delta , \\ \end{array}
$$
where the last inequality follows the definitions of  $(\widetilde{F},\widetilde{G})$ , Assumption 2.1, and the general definition of  $\delta$ -corruption in Eq. (27). After choosing  $\bar{\lambda} = \frac{1}{\tau_A}\sum_{t\in [\tau_A]}\lambda_t$ , similar to our proof in Eq. (20) for the stochastic case we have
$$
\begin{array}{l} \mathrm {O P T} (\mathcal {P} _ {1: T}) \\ = \mathbb {E} \left[ \frac {T - \tau_ {A}}{T} \mathrm {O P T} \left(\mathcal {P} _ {1: T}\right) + \frac {\tau_ {A}}{T} \mathrm {O P T} \left(\mathcal {P} _ {1: T}\right) \right] \\ \stackrel {(a)} {\leq} \mathbb {E} \left[ (T - \tau_ {A}) \bar {F} + \tau_ {A} \cdot \max  _ {\boldsymbol {x} \in \mathcal {X}} (\widetilde {F} (\boldsymbol {x}) + \boldsymbol {\lambda} ^ {\top} \widetilde {\boldsymbol {G}} (\boldsymbol {x})) + \frac {\tau_ {A}}{T} (\bar {F} + \bar {G} K \frac {\bar {F}}{\beta}) \delta \right] \\ = \mathbb {E} \left[ (T - \tau_ {A}) \bar {F} + \frac {\tau_ {A}}{T} (\bar {F} + \bar {G} K \frac {\bar {F}}{\beta}) \delta + \max  _ {\boldsymbol {x} \in \mathcal {X}} \left(\sum_ {t \in [ \tau_ {A} ]} \widetilde {F} (\boldsymbol {x}) + \boldsymbol {\lambda} _ {t} ^ {\top} \widetilde {\boldsymbol {G}} (\boldsymbol {x})\right) \right] \\ \stackrel {(b)} {\leq} \mathbb {E} \left[ (T - \tau_ {A}) \bar {F} + \left(1 + \frac {\tau_ {A}}{T}\right) (\bar {F} + \bar {G} K \frac {\bar {F}}{\beta}) \delta + \max  _ {\boldsymbol {x} \in \mathcal {X}} \sum_ {t \in [ \tau_ {A} ]} \left(F _ {t} (\boldsymbol {x}) + \boldsymbol {\lambda} _ {t} ^ {\top} \boldsymbol {G} _ {t} (\boldsymbol {x})\right) \right] \\ \leq \mathbb {E} \left[ (T - \tau_ {A}) \bar {F} + \left(1 + \frac {\tau_ {A}}{T}\right) (\bar {F} + \bar {G} K \frac {\bar {F}}{\beta}) \delta + \max  _ {\boldsymbol {x} \in \mathcal {X}} \sum_ {t \in [ \tau_ {A} ]} \mathbb {E} \left[ f _ {t} (\boldsymbol {x}) + \boldsymbol {\lambda} _ {t} ^ {\top} \boldsymbol {g} _ {t} (\boldsymbol {x}) \mid \sigma (\mathcal {H} _ {t - 1}) \right] \right] \\ \leq \mathbb {E} \left[ (T - \tau_ {A}) \bar {F} + 2 (\bar {F} + \bar {G} K \frac {\bar {F}}{\beta}) \delta + \max  _ {\boldsymbol {x} \in \mathcal {X}} \sum_ {t \in [ \tau_ {A} ]} h _ {t} (\boldsymbol {x}) \right], \tag {30} \\ \end{array}
$$
where (a) follows from Eq. (29); (b) follows from the definition of general  $\delta$ -corruption in Eq. (27). Finally, we complete the proof by using the definition  $f_{t}(\pmb{x}_{t}) = h_{t}(\pmb{x}_{t}) - \pmb{\lambda}_{t}^{\top}\pmb{g}_{t}(\pmb{x}_{t})$  and following the same argument as in Eq. (22) for the stochastic regime.
Periodic.
Recall in Section 2.2 that in the periodic regime, there exists cycle length  $q \in \mathbb{N}$  such that  $T = cq$  for some integer  $c \geq 2$  with  $\mathcal{P}_{1:T}$  as  $\mathcal{P}_{1:q} = \mathcal{P}_{q+1:2q} = \dots = \mathcal{P}_{(c-1)q+1:T}$ . For any  $t \in [T]$ , define  $c_t \in [c]$  such that  $(c_t - 1)q + 1 \leq t \leq c_tq$ . After denoting  $\widetilde{\mathcal{P}} = \frac{1}{q}\sum_{t \in [q]} \mathcal{P}_t$ , we define the mean deviation within a single cycle of length  $q$  as
$$
M D \left(\mathcal {P} _ {1: q}\right) = \sum_ {1 \leq t \leq q} \| \mathcal {P} _ {t} - \widetilde {\mathcal {P}} \| _ {T V} \quad \text {a n d} \quad \delta = c \cdot M D \left(\mathcal {P} _ {1: q}\right). \tag {31}
$$
We define  $\widetilde{F}(\pmb{x}) = \mathbb{E}_{(f,\pmb{g})\sim \widetilde{\mathcal{P}}}[f(\pmb{x})]$ ,  $\widetilde{\pmb{G}}(\pmb{x}) = \mathbb{E}_{(f,\pmb{g})\sim \widetilde{\mathcal{P}}}[\pmb{g}(\pmb{x})]$ ,  $F_{t}(\pmb{x}) = \mathbb{E}_{(f,\pmb{g})\sim \mathcal{P}_{t}}[f(\pmb{x})]$  and  $\pmb{G}_{t}(\pmb{x}) = \mathbb{E}_{(f,\pmb{g})\sim \mathcal{P}_{t}}[\pmb{g}(\pmb{x})]$  for all  $t \in [T]$  and any  $x \in \mathcal{X}$ . Then for any  $\lambda \in [0, \frac{\bar{F}}{\beta} e]$ , we have
$$
\begin{array}{l} \operatorname {O P T} \left(\mathcal {P} _ {1: T}\right) \leq \max  _ {\boldsymbol {x} _ {1: T} \in \mathcal {X} ^ {T}} \sum_ {t \in [ T ]} \left(F _ {t} \left(\boldsymbol {x} _ {t}\right) + \boldsymbol {\lambda} ^ {\top} \boldsymbol {G} _ {t} \left(\boldsymbol {x} _ {t}\right)\right) \\ = c \cdot \max  _ {\boldsymbol {x} _ {1: q} \in \mathcal {X} ^ {q}} \sum_ {t \in [ q ]} \left(F _ {t} (\boldsymbol {x} _ {t}) + \boldsymbol {\lambda} ^ {\top} \boldsymbol {G} _ {t} (\boldsymbol {x} _ {t})\right) \\ \leq c q \cdot \max  _ {\boldsymbol {x} \in \mathcal {X}} (\widetilde {F} (\boldsymbol {x}) + \boldsymbol {\lambda} ^ {\top} \widetilde {\boldsymbol {G}} (\boldsymbol {x})) + (\bar {F} + \bar {G} K \frac {\bar {F}}{\beta}) c \cdot M D (\mathcal {P} _ {1: q}) \\ \leq c q \cdot \max  _ {\boldsymbol {x} \in \mathcal {X}} (\widetilde {F} (\boldsymbol {x}) + \boldsymbol {\lambda} ^ {\top} \widetilde {G} (\boldsymbol {x})) + (\bar {F} + \bar {G} K \frac {\bar {F}}{\beta}) \delta , \\ \end{array}
$$
where the equality follows the nature of periodic setting and the last inequality follows the definitions of  $(\widetilde{F},\widetilde{G})$ , Assumption 2.1, and (31). After choosing  $\lambda = \sum_{\hat{c}\in [c_{\tau_A} - 1]}\frac{q}{\tau_A}\pmb{\lambda}_{(\hat{c} -1)q + 1}+$ $\frac{\tau_A - (c_{\tau_A} - 1)q}{\tau_A}\pmb{\lambda}_{(c_{\tau_A} - 1)q + 1}$ , we further have that
$$
\begin{array}{l} \mathrm {O P T} (\mathcal {P} _ {1: T}) \\ = \frac {T - \tau_ {A}}{T} \mathrm {O P T} \left(\mathcal {P} _ {1: T}\right) + \frac {\tau_ {A}}{T} \mathrm {O P T} \left(\mathcal {P} _ {1: T}\right) \\ \leq \left(T - \tau_ {A}\right) \bar {F} + \tau_ {A} \cdot \max  _ {\boldsymbol {x} \in \mathcal {X}} (\widetilde {F} (\boldsymbol {x}) + \boldsymbol {\lambda} ^ {\top} \widetilde {\boldsymbol {G}} (\boldsymbol {x})) + \frac {\tau_ {A}}{T} (\bar {F} + \bar {G} K \frac {\bar {F}}{\beta}) \delta \\ = (T - \tau_ {A}) \bar {F} + \max  _ {\boldsymbol {x} \in \mathcal {X}} \left(\tau_ {A} \widetilde {F} (\boldsymbol {x}) + \left(\sum_ {\hat {c} \in [ c _ {\tau_ {A}} - 1 ]} q \boldsymbol {\lambda} _ {(\hat {c} - 1) q + 1} + (\tau_ {A} - (c _ {\tau_ {A}} - 1) q) \boldsymbol {\lambda} _ {(c _ {\tau_ {A}} - 1) q + 1}\right) ^ {\top} \widetilde {\boldsymbol {G}} (\boldsymbol {x})\right) \\ + \frac {\tau_ {A}}{T} (\bar {F} + \bar {G} K \frac {\bar {F}}{\beta}) \delta \\ = (T - \tau_ {A}) \bar {F} + \max  _ {\boldsymbol {x} \in \mathcal {X}} \left(q \cdot \sum_ {\hat {c} \in [ c _ {\tau_ {A}} - 1 ]} \left(\widetilde {F} (\boldsymbol {x}) + \boldsymbol {\lambda} _ {(\hat {c} - 1) q + 1} ^ {\top} \widetilde {\boldsymbol {G}} (\boldsymbol {x})\right) \right. \\ \left. + \left(\tau_ {A} - \left(c _ {\tau_ {A}} - 1\right) q\right) \cdot \left(\widetilde {F} (\boldsymbol {x}) + \boldsymbol {\lambda} _ {\left(c _ {\tau_ {A}} - 1\right) q + 1} ^ {\top} \widetilde {\boldsymbol {G}} (\boldsymbol {x})\right)\right) + \frac {\tau_ {A}}{T} \left(\bar {F} + \bar {G} K \frac {\bar {F}}{\beta}\right) \delta \\ \leq (T - \tau_ {A}) \bar {F} + \max  _ {\boldsymbol {x} \in \mathcal {X}} \sum_ {t \in [ \tau_ {A} ]} \left(\widetilde {F} (\boldsymbol {x}) + \boldsymbol {\lambda} _ {t} ^ {\top} \widetilde {G} (\boldsymbol {x})\right) + \bar {G} \cdot \sum_ {t \in [ \tau_ {A} ]} \| \boldsymbol {\lambda} _ {t} - \boldsymbol {\lambda} _ {(c _ {t} - 1) q + 1} \| _ {1} + \frac {\tau_ {A}}{T} (\bar {F} + \bar {G} K \frac {\bar {F}}{\beta}) \delta . \\ \end{array}
$$
From Eq. (9) in Algorithm 1, we know that  $\| \pmb{\lambda}_{t + 1} - \pmb{\lambda}_t\| _1\leq \eta \bar{G} K$ , which further implies  $\| \pmb{\lambda}_{t + i} - \pmb{\lambda}_t\| _1\leq \eta \bar{G} Ki$  for any  $i\in [q - 1]$  and thus
$$
\sum_ {t \in [ \tau_ {A} ]} \| \boldsymbol {\lambda} _ {t} - \boldsymbol {\lambda} _ {(c _ {t} - 1) q + 1} \| _ {1} \leq c _ {\tau_ {A}} \eta \bar {G} K \sum_ {i \in [ q - 1 ]} i \leq \frac {1}{2} \bar {G} K \eta c _ {\tau_ {A}} q ^ {2}. \tag {32}
$$
After combining the two equations above, it follows that
$$
\begin{array}{l} \mathrm {O P T} (\mathcal {P} _ {1: T}) \\ \leq \mathbb {E} \left[ (T - \tau_ {A}) \bar {F} + \max  _ {\boldsymbol {x} \in \mathcal {X}} \sum_ {t \in [ \tau_ {A} ]} \left(\widetilde {F} (\boldsymbol {x}) + \boldsymbol {\lambda} _ {t} ^ {\top} \widetilde {G} (\boldsymbol {x})\right) + \frac {1}{2} \bar {G} ^ {2} K \eta c _ {\tau_ {A}} q ^ {2} + \frac {\tau_ {A}}{T} (\bar {F} + \bar {G} K \frac {\bar {F}}{\beta}) \delta \right] \\ \leq \mathbb {E} \left[ (T - \tau_ {A}) \bar {F} + \max  _ {\boldsymbol {x} \in \mathcal {X}} \sum_ {t \in [ \tau_ {A} ]} \left(F _ {t} (\boldsymbol {x}) + \boldsymbol {\lambda} _ {t} ^ {\top} \boldsymbol {G} _ {t} (\boldsymbol {x})\right) + \frac {1}{2} \bar {G} ^ {2} K \eta c _ {\tau_ {A}} q ^ {2} + 2 (\bar {F} + \bar {G} K \frac {\bar {F}}{\beta}) \delta \right] \\ \leq \mathbb {E} \left[ (T - \tau_ {A}) \bar {F} + 2 (\bar {F} + \bar {G} K \frac {\bar {F}}{\beta}) \delta + \frac {1}{2} \bar {G} ^ {2} K \eta q T + \max  _ {\boldsymbol {x} \in \mathcal {X}} \sum_ {t \in [ \tau_ {A} ]} \mathbb {E} \left[ h _ {t} (\boldsymbol {x}) \mid \sigma (\mathcal {H} _ {t - 1}) \right] \right] \\ \leq \mathbb {E} \left[ (T - \tau_ {A}) \bar {F} + 2 (\bar {F} + \bar {G} K \frac {\bar {F}}{\beta}) \delta + \frac {1}{2} \bar {G} ^ {2} K \eta q T + \max  _ {\boldsymbol {x} \in \mathcal {X}} \sum_ {t \in [ \tau_ {A} ]} h _ {t} (\boldsymbol {x}) \right], \\ \end{array}
$$
where the second last inequality follows from  $c_{\tau_A}q\leq cq = T$
Finally, we complete the proof by using the definition  $f_{t}(\pmb{x}_{t}) = h_{t}(\pmb{x}_{t}) - \pmb{\lambda}_{t}^{\top}\pmb{g}_{t}(\pmb{x}_{t})$  and following the same argument as in Eq. (22) for the stochastic regime.
Ergodic.
Consider some  $\kappa \geq \log (T)$ . Given the input distribution sequence  $\mathcal{P}_{1:T}$ , denote  $\mathcal{P}_{(t + \kappa)[t - 1]}$  as the conditional distribution of  $(f_{t + \kappa},\pmb{g}_{t + \kappa})$  conditioned on the  $\{(f_{\tau},\pmb{g}_{\tau})\}_{\tau \in [t]}$ . Then, in the ergodic regime, there exists a stationary distribution  $\widetilde{\mathcal{P}}\in \Delta (\mathcal{S})$  and absolute constant  $R > 0$  such that
$$
\sup  _ {\left\{\left(f _ {t}, \boldsymbol {g} _ {t}\right) \right\} _ {t \in [ T ]} \in \mathcal {S} ^ {T}} \sup  _ {t \in [ T - \kappa ]} \| \mathcal {P} _ {(t + \kappa) | [ t - 1 ]} - \widetilde {\mathcal {P}} \| _ {T V} \leq \delta := R \exp (- \kappa); \tag {33}
$$
see [10, Section 5.2] for further discussions. By defining  $\widetilde{F}(\boldsymbol{x}) = \mathbb{E}_{(f,\boldsymbol{g})\sim \widetilde{\mathcal{P}}}[f(\boldsymbol{x})]$ ,  $\widetilde{\boldsymbol{G}}(\boldsymbol{x}) = \mathbb{E}_{(f,\boldsymbol{g})\sim \widetilde{\mathcal{P}}}[\boldsymbol{g}(\boldsymbol{x})]$ ,  $\hat{F}_{t + \kappa}(\boldsymbol{x}) = \mathbb{E}_{(f,\boldsymbol{g})\sim \mathcal{P}_{(t + \kappa)[t - 1]}}[f(\boldsymbol{x})]$ ,  $\hat{\boldsymbol{G}}_{t + \kappa}(\boldsymbol{x}) = \mathbb{E}_{(f,\boldsymbol{g})\sim \mathcal{P}_{(t + \kappa)[t - 1]}}[\boldsymbol{g}(\boldsymbol{x})]$ ,  $F_{t}(\boldsymbol{x}) = \mathbb{E}_{(f,\boldsymbol{g})\sim \mathcal{P}_{t}}[f(\boldsymbol{x})]$  and  $\boldsymbol{G}_t(\boldsymbol{x}) = \mathbb{E}_{(f,\boldsymbol{g})\sim \mathcal{P}_t}[\boldsymbol{g}(\boldsymbol{x})]$  for all  $t\in [T]$  and any  $x\in \mathcal{X}$ , we know that for any  $\lambda \in [\mathbf{0},\frac{\bar{F}}{\beta}\boldsymbol{e}]$ , it follows that
$$
\begin{array}{l} \mathrm {O P T} (\mathcal {P} _ {1: T}) \\ \leq \max  _ {\boldsymbol {x} _ {1: T} \in \mathcal {X} ^ {T}} \mathbb {E} \left[ \sum_ {t \in [ T ]} \left(F _ {t} (\boldsymbol {x} _ {t}) + \boldsymbol {\lambda} ^ {\top} \boldsymbol {G} _ {t} (\boldsymbol {x} _ {t})\right) \right] \\ = \max  _ {\boldsymbol {x} _ {1: \kappa} \in \mathcal {X} ^ {\kappa}} \mathbb {E} \left[ \sum_ {t \in [ \kappa ]} \left(F _ {t} (\boldsymbol {x} _ {t}) + \boldsymbol {\lambda} ^ {\top} \boldsymbol {G} _ {t} (\boldsymbol {x} _ {t})\right) \right] + \max  _ {\boldsymbol {x} _ {\kappa + 1: T} \in \mathcal {X} ^ {T - \kappa}} \mathbb {E} \left[ \sum_ {t = 1} ^ {T - \kappa} \left(\hat {F} _ {t + \kappa} (\boldsymbol {x} _ {t + \kappa}) + \boldsymbol {\lambda} ^ {\top} \hat {\boldsymbol {G}} _ {t + \kappa} (\boldsymbol {x} _ {t + \kappa})\right) \right] \\ \leq (\bar {F} + \bar {G} K \frac {\bar {F}}{\beta}) \kappa + \max  _ {\boldsymbol {x} _ {\kappa + 1: T} \in \mathcal {X} ^ {T - \kappa}} \sum_ {t = 1} ^ {T - \kappa} (\widetilde {F} (\boldsymbol {x} _ {t + \kappa}) + \boldsymbol {\lambda} ^ {\top} \widetilde {\boldsymbol {G}} (\boldsymbol {x} _ {t + \kappa})) + (\bar {F} + \bar {G} K \frac {\bar {F}}{\beta}) \cdot (T - \kappa) \delta \\ \leq T \cdot \max  _ {\boldsymbol {x} \in \mathcal {X}} \left(\widetilde {F} (\boldsymbol {x}) + \boldsymbol {\lambda} ^ {\top} \widetilde {G} (\boldsymbol {x})\right) + \left(\bar {F} + \bar {G} K \frac {\bar {F}}{\beta}\right) \kappa + \left(\bar {F} + \bar {G} K \frac {\bar {F}}{\beta}\right) \cdot T \delta . \tag {34} \\ \end{array}
$$
By choosing  $\lambda = \frac{1}{\tau_A}\sum_{t\in [\tau_A]}\lambda_t$  , we further have
$$
\begin{array}{l} \mathrm {O P T} \left(\mathcal {P} _ {1: T}\right) \\ = \mathbb {E} \left[ \frac {T - \tau_ {A}}{T} \mathrm {O P T} \left(\mathcal {P} _ {1: T}\right) + \frac {\tau_ {A}}{T} \mathrm {O P T} \left(\mathcal {P} _ {1: T}\right) \right] \\ \leq \mathbb {E} \left[ (T - \tau_ {A}) \bar {F} + \tau_ {A} \cdot \max  _ {\boldsymbol {x} \in \mathcal {X}} \left(\widetilde {F} (\boldsymbol {x}) + \boldsymbol {\lambda} ^ {\top} \widetilde {\boldsymbol {G}} (\boldsymbol {x})\right) \right] + (\bar {F} + \bar {G} K \frac {\bar {F}}{\beta}) \kappa + (\bar {F} + \bar {G} K \frac {\bar {F}}{\beta}) \cdot T \delta \\ = \mathbb {E} \left[ (T - \tau_ {A}) \bar {F} + \max  _ {\boldsymbol {x} \in \mathcal {X}} \sum_ {t \in [ \tau_ {A} ]} \left(\widetilde {F} (\boldsymbol {x}) + \boldsymbol {\lambda} _ {t} ^ {\top} \widetilde {\boldsymbol {G}} (\boldsymbol {x})\right) \right] + (\bar {F} + \bar {G} K \frac {\bar {F}}{\beta}) \kappa + (\bar {F} + \bar {G} K \frac {\bar {F}}{\beta}) \cdot T \delta \\ \leq \mathbb {E} \left[ (T - \tau_ {A}) \bar {F} + \max  _ {\boldsymbol {x} \in \mathcal {X}} \sum_ {t \in [ \tau_ {A} ]} \left(\hat {F} _ {t + \kappa} (\boldsymbol {x}) + \boldsymbol {\lambda} _ {t} ^ {\top} \hat {\boldsymbol {G}} _ {t + \kappa} (\boldsymbol {x})\right) \right] + (\bar {F} + \bar {G} K \frac {\bar {F}}{\beta}) \kappa + 2 (\bar {F} + \bar {G} K \frac {\bar {F}}{\beta}) \cdot T \delta \\ = \mathbb {E} \left[ (T - \tau_ {A}) \bar {F} + \max  _ {\boldsymbol {x} \in \mathcal {X}} \mathbb {E} \sum_ {t \in [ \tau_ {A} ]} \left(\hat {F} _ {t + \kappa} (\boldsymbol {x}) + \boldsymbol {\lambda} _ {t + \kappa} ^ {\top} \hat {\boldsymbol {G}} _ {t + \kappa} (\boldsymbol {x}) + \left(\boldsymbol {\lambda} _ {t} - \boldsymbol {\lambda} _ {t + \kappa}\right) ^ {\top} \hat {\boldsymbol {G}} _ {t + \kappa} (\boldsymbol {x} _ {t})\right) \right] \\ + (\bar {F} + \bar {G} K \frac {\bar {F}}{\beta}) \kappa + 2 (\bar {F} + \bar {G} K \frac {\bar {F}}{\beta}) \cdot T \delta \\ \end{array}
$$
$$
\stackrel {(a)} {\leq} \mathbb {E} \left[ (T - \tau_ {A}) \bar {F} + \max  _ {\boldsymbol {x} \in \mathcal {X}} \sum_ {t \in [ \tau_ {A} ]} \left(\hat {F} _ {t + \kappa} (\boldsymbol {x}) + \boldsymbol {\lambda} _ {t + \kappa} ^ {\top} \hat {\boldsymbol {G}} _ {t + \kappa} (\boldsymbol {x} _ {t})\right) \right] + \kappa \eta T K \bar {G} ^ {2}
$$
$$
\begin{array}{l} + (\bar {F} + \bar {G} K \frac {\bar {F}}{\beta}) \kappa + 2 (\bar {F} + \bar {G} K \frac {\bar {F}}{\beta}) \cdot T \delta \\ \leq \mathbb {E} \left[ (T - \tau_ {A}) \bar {F} + \max  _ {\boldsymbol {x} \in \mathcal {X}} \sum_ {t \in [ \tau_ {A} - \kappa ]} \left(\hat {F} _ {t + \kappa} (\boldsymbol {x}) + \boldsymbol {\lambda} _ {t + \kappa} ^ {\top} \hat {\boldsymbol {G}} _ {t + \kappa} (\boldsymbol {x} _ {t})\right) \right] + \kappa \eta T K \bar {G} ^ {2} \\ + 2 (\bar {F} + \bar {G} K \frac {\bar {F}}{\beta}) \kappa + 2 (\bar {F} + \bar {K} \frac {\bar {F}}{\beta}) \cdot T \delta \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(b)} {\leq} \mathbb {E} \left[ (T - \tau_ {A}) \bar {F} + \max  _ {\boldsymbol {x} \in \mathcal {X}} \sum_ {t = \kappa + 1} ^ {\tau_ {A}} h _ {t} (\boldsymbol {x}) \right] + \kappa \eta T K \bar {G} ^ {2} + 2 (\bar {F} + \bar {G} K \frac {\bar {F}}{\beta}) \kappa + 2 (\bar {F} + \bar {G} K \frac {\bar {F}}{\beta}) \cdot T \delta \\ \leq \mathbb {E} \left[ (T - \tau_ {A}) \bar {F} + \max  _ {\boldsymbol {x} \in \mathcal {X}} \sum_ {t \in [ \tau_ {A} ]} h _ {t} (\boldsymbol {x}) \right] + \kappa \eta T K \bar {G} ^ {2} + 2 (\bar {F} + \bar {G} K \frac {\bar {F}}{\beta}) \kappa + 2 (\bar {F} + \bar {G} K \frac {\bar {F}}{\beta}) \cdot T \delta \\ \end{array}
$$
$$
\stackrel {(c)} {\leq} \mathbb {E} \Big [ (T - \tau_ {A}) \bar {F} + \max _ {\boldsymbol {x} \in \mathcal {X}} \sum_ {t \in [ \tau_ {A} ]} h _ {t} (\boldsymbol {x}) \Big ] + \kappa \eta T K \bar {G} ^ {2} + 2 (\bar {F} + \bar {G} K \frac {\bar {F}}{\beta}) \kappa + 2 R (\bar {F} + \bar {G} K \frac {\bar {F}}{\beta}),
$$
where in (a), from (9) in Algorithm 1, we know that  $\| \pmb{\lambda}_{t + 1} - \pmb{\lambda}_t\| _1\leq \eta \bar{G} K$  , which further implies  $\| \pmb{\lambda}_{t + \kappa} - \pmb{\lambda}_t\| _1\leq \kappa \eta \bar{G} K$  and thus
$$
\left(\boldsymbol {\lambda} _ {t} - \boldsymbol {\lambda} _ {t + \kappa}\right) ^ {\top} \hat {\boldsymbol {G}} _ {t + \kappa} \left(\boldsymbol {x} _ {t}\right) \leq \kappa \eta K \bar {G} ^ {2}. \tag {35}
$$
In (b), we used the fact that for any  $t \geq \kappa + 1$ , we have
$$
\begin{array}{l} \mathbb {E} \left[ \max  _ {\boldsymbol {x} \in \mathcal {X}} \sum_ {t \in \left[ \tau_ {A} - \kappa \right]} \left(\hat {F} _ {t + \kappa} (\boldsymbol {x}) + \boldsymbol {\lambda} _ {t + \kappa} ^ {\top} \hat {\boldsymbol {G}} _ {t + \kappa} (\boldsymbol {x} _ {t})\right) \right] \\ = \mathbb {E} \left[ \max  _ {\boldsymbol {x} \in \mathcal {X}} \sum_ {t \in \left[ \tau_ {A} - \kappa \right]} \mathbb {E} \left[ h _ {t + \kappa} (\boldsymbol {x}) \mid (f _ {\tau}, \boldsymbol {g} _ {\tau}) _ {\tau \in [ t - 1 ]} \right] \right] \tag {36} \\ \leq \mathbb {E} \left[ \max  _ {\boldsymbol {x} \in \mathcal {X}} \sum_ {t \in \left[ \tau_ {A} - \kappa \right]} h _ {t + \kappa} (\boldsymbol {x}) \right]. \\ \end{array}
$$