SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \| x _ {t} - \delta (x _ {t - p: t - 1}) \| ^ {2} - (1 + \lambda) \| x _ {t} ^ {\prime} - \delta (x _ {t - p: t - 1} ^ {\prime}) \| ^ {2} \\ \leq \left(1 + \frac {1}{\lambda}\right) \| \left(x _ {t} - x _ {t} ^ {\prime}\right) + \left(\delta \left(x _ {t - p: t - 1}\right) - \delta \left(x _ {t - p: t - 1} ^ {\prime}\right)\right) \| ^ {2} \\ \leq \left(1 + \frac {1}{\lambda}\right) \left(\| x _ {t} - x _ {t} ^ {\prime} \| + \| \delta \left(x _ {t - p: t - 1}\right) - \delta \left(x _ {t - p: t - 1} ^ {\prime}\right) \|\right) ^ {2} \\ \leq \left(1 + \frac {1}{\lambda}\right) \left(\| x _ {t} - x _ {t} ^ {\prime} \| + \sum_ {k = 1} ^ {p} L _ {k} \| x _ {t - k} - x _ {t - k} ^ {\prime} \|\right) ^ {2} \tag {11} \\ \leq (1 + \frac {1}{\lambda}) (1 + \sum_ {k = 1} ^ {p} L _ {k}) \left(\| x _ {t} - x _ {t} ^ {\prime} \| ^ {2} + \sum_ {k = 1} ^ {p} L _ {k} \| x _ {t - k} - x _ {t - k} ^ {\prime} \| ^ {2}\right) \\ \end{array}
$$
Summing up the switching costs of all time steps together, we have
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \| x _ {t} - \delta (x _ {t - p: t - 1}) \| ^ {2} - (1 + \lambda) \| x _ {t} ^ {\prime} - \delta (x _ {t - p: t - 1} ^ {\prime}) \| ^ {2} \\ \leq \left(1 + \frac {1}{\lambda}\right) \left(1 + \sum_ {k = 1} ^ {p} L _ {k}\right) \sum_ {t = 1} ^ {T} \left(\| x _ {t} - x _ {t} ^ {\prime} \| ^ {2} + \sum_ {k = 1} ^ {p} L _ {k} \| x _ {t - k} - x _ {t - k} ^ {\prime} \| ^ {2}\right) \tag {12} \\ \leq \left(1 + \frac {1}{\lambda}\right) \left(1 + \sum_ {k = 1} ^ {p} L _ {k}\right) \sum_ {t = 1} ^ {T} \left(1 + \sum_ {k = 1} ^ {p} L _ {k}\right) \left\| x _ {t} - x _ {t} ^ {\prime} \right\| ^ {2} \\ = (1 + \frac {1}{\lambda}) (1 + \sum_ {k = 1} ^ {p} L _ {k}) ^ {2} \sum_ {t = 1} ^ {T} \| x _ {t} - x _ {t} ^ {\prime} \| ^ {2} \\ \end{array}
$$
Substituting Eqn. (12) and Eqn. (10) into Eqn. (9), we finish the proof.
Now we propose Lemma C.4 based on these above lemmas, which ensures the feasibility of robustness constraint in Eqn. (1)
Lemma C.4. Let  $\pi$  be any expert algorithm for the SOCO problem with multi-step feedback delays and multi-step switching costs, for any  $\lambda \geq 0$  and  $\lambda \geq \lambda_0 \geq 0$ , the total cost by the projected actions  $x_{t}$  must satisfy  $\mathrm{cost}(x_{1:T}) \leq (1 + \lambda)\mathrm{cost}(x_{1:T}^{\pi})$
Proof. We prove by induction that the constraints in Eqn. (1) are satisfied for each  $t$ . For  $t = 1$ , since we assume the initial actions are the same ( $x_{-p + 1:0} = x_{-p + 1:0}^{\pi}$ ), it is obvious that  $x = x_1^\pi$  satisfies the robustness constraints Eqn. (1).
Then for any time step  $t \geq 2$ , suppose it holds at  $t - 1$  that
$$
\begin{array}{l} \sum_ {\tau \in \mathcal {A} _ {t - 1}} f (x _ {\tau}, y _ {\tau}) + \sum_ {\tau \in \mathcal {A} _ {t - 1} \cup \mathcal {B} _ {t - 1}} d (x _ {\tau}, x _ {\tau - p: \tau - 1}) + \sum_ {\tau \in \mathcal {B} _ {t - 1}} H (x _ {\tau}, x _ {\tau} ^ {\pi}) + G (x, x _ {t - p: t - 1}, x _ {t - p: t} ^ {\pi}) \\ \leq (1 + \lambda) \left(\sum_ {\tau \in \mathcal {A} _ {t - 1}} f \left(x _ {\tau} ^ {\pi}, y _ {\tau}\right) + \sum_ {\tau \in \mathcal {A} _ {t - 1} \cup \mathcal {B} _ {t - 1}} d \left(x _ {\tau} ^ {\pi}, x _ {\tau - p: \tau - 1} ^ {\pi}\right)\right) \tag {13} \\ \end{array}
$$
Now the robustness constraints Eqn. (1) is satisfied if we prove  $x_{t} = x_{t}^{\pi}$  satisfies the constraints in Eqn. (1) at time step  $t$ . Since for the sets  $\mathcal{A}$  and  $\mathcal{B}$ , we have
$$
\left(\mathcal {A} _ {t} \cup \mathcal {B} _ {t}\right) \backslash \left(\mathcal {A} _ {t - 1} \cup \mathcal {B} _ {t - 1}\right) = \{t \}, \quad \mathcal {A} _ {t - 1} \subseteq \mathcal {A} _ {t}, \tag {14}
$$
so it holds that
$$
\sum_ {\tau \in \mathcal {A} _ {t} \cup \mathcal {B} _ {t}} d \left(x _ {\tau}, x _ {\tau - p: \tau - 1}\right) - \sum_ {\tau \in \mathcal {A} _ {t - 1} \cup \mathcal {B} _ {t - 1}} d \left(x _ {\tau}, x _ {\tau - p: \tau - 1}\right) = d \left(x _ {t}, x _ {t - p: t - 1}\right) \tag {15}
$$
By Lemma C.1, we have
$$
\begin{array}{l} d \left(x _ {t} ^ {\pi}, x _ {t - p: t - 1}\right) - (1 + \lambda) d \left(x _ {t} ^ {\pi}, x _ {t - p: t - 1} ^ {\pi}\right) \\ \leq \frac {1}{2} \left(1 + \frac {1}{\lambda}\right) \| \delta \left(x _ {t - p: t - 1}\right) - \delta \left(x _ {t - p: t - 1} ^ {\pi}\right) \| ^ {2} \tag {16} \\ \leq \frac {1}{2} \left(1 + \frac {1}{\lambda}\right) \left(\sum_ {i = 1} ^ {p} L _ {i} \| x _ {t - i} - x _ {t - i} ^ {\pi} \|\right) ^ {2} \\ \end{array}
$$
Denote  $\alpha = 1 + \sum_{k=1}^{p} L_k$ . For the reservation cost, we have
$$
\begin{array}{l} G \left(x _ {t - 1}, x _ {t - p - 1: t - 2}, x _ {t - p - 1: t - 1} ^ {\pi}\right) - G \left(x _ {t} ^ {\pi}, x _ {t - p: t - 1}, x _ {t - p: t} ^ {\pi}\right) \\ = \frac {\alpha \left(1 + \frac {1}{\lambda_ {0}}\right)}{2} \left(\sum_ {k = 1} ^ {p} \sum_ {i = 0} ^ {p - k} L _ {k + i} \| x _ {t - i - 1} - x _ {t - i - 1} ^ {\pi} \| ^ {2} - \sum_ {k = 1} ^ {p} \sum_ {i = 1} ^ {p - k} L _ {k + i} \| x _ {t - i} - x _ {t - i} ^ {\pi} \| ^ {2}\right) \\ = \frac {\alpha \left(1 + \frac {1}{\lambda_ {0}}\right)}{2} \left(\sum_ {k = 0} ^ {p - 1} \sum_ {i = 1} ^ {p - k} L _ {k + i} \| x _ {t - i} - x _ {t - i} ^ {\pi} \| ^ {2} - \sum_ {k = 1} ^ {p} \sum_ {i = 1} ^ {p - k} L _ {k + i} \| x _ {t - i} - x _ {t - i} ^ {\pi} \| ^ {2}\right) \tag {17} \\ = \frac {\alpha \left(1 + \frac {1}{\lambda_ {0}}\right)}{2} \left(\sum_ {k = 0} ^ {p - 1} \sum_ {i = 1} ^ {p - k} L _ {k + i} \| x _ {t - i} - x _ {t - i} ^ {\pi} \| ^ {2} - \sum_ {k = 1} ^ {p - 1} \sum_ {i = 1} ^ {p - k} L _ {k + i} \| x _ {t - i} - x _ {t - i} ^ {\pi} \| ^ {2}\right) \\ = \frac {\alpha \left(1 + \frac {1}{\lambda_ {0}}\right)}{2} \sum_ {i = 1} ^ {p} L _ {i} \| x _ {t - i} - x _ {t - i} ^ {\pi} \| ^ {2} \\ \end{array}
$$
Continuing with Eqn. (17), we have
$$
\begin{array}{l} G (x _ {t - 1}, x _ {t - p - 1: t - 2}, x _ {t - p - 1: t - 1} ^ {\pi}) - G (x _ {t} ^ {\pi}, x _ {t - p: t - 1}, x _ {t - p: t} ^ {\pi}) = \frac {\alpha (1 + \frac {1}{\lambda_ {0}})}{2} \sum_ {i = 1} ^ {p} L _ {i} \| x _ {t - i} - x _ {t - i} ^ {\pi} \| ^ {2} \\ \geq \frac {\left(1 + \frac {1}{\lambda_ {0}}\right) \left(\sum_ {i = 1} ^ {p} L _ {i}\right) ^ {2}}{2} \sum_ {i = 1} ^ {p} \frac {L _ {i}}{\sum_ {i = 1} ^ {p} L _ {i}} \| x _ {t - i} - x _ {t - i} ^ {\pi} \| ^ {2} \\ \geq \frac {\left(1 + \frac {1}{\lambda_ {0}}\right) \left(\sum_ {i = 1} ^ {p} L _ {i}\right) ^ {2}}{2} \left(\sum_ {i = 1} ^ {p} \frac {L _ {i}}{\sum_ {i = 1} ^ {p} L _ {i}} \| x _ {t - i} - x _ {t - i} ^ {\pi} \|\right) ^ {2} \\ = \frac {1}{2} \left(1 + \frac {1}{\lambda_ {0}}\right) \left(\sum_ {i = 1} ^ {p} L _ {i} \| x _ {t - i} - x _ {t - i} ^ {\pi} \|\right) ^ {2} \geq \frac {1}{2} \left(1 + \frac {1}{\lambda}\right) \left(\sum_ {i = 1} ^ {p} L _ {i} \| x _ {t - i} - x _ {t - i} ^ {\pi} \|\right) ^ {2} \tag {18} \\ \end{array}
$$
where the second inequality holds by Jensen's inequality. Therefore, combining with (16), we have
$$
d \left(x _ {t} ^ {\pi}, x _ {t - p: t - 1}\right) + G \left(x _ {t} ^ {\pi}, x _ {t - p: t - 1}, x _ {t - p: t} ^ {\pi}\right) \leq G \left(x _ {t - 1}, x _ {t - p - 1: t - 2}, x _ {t - p - 1: t - 1} ^ {\pi}\right) + (1 + \lambda) d \left(x _ {t} ^ {\pi}, x _ {t - p: t - 1} ^ {\pi}\right) \tag {19}
$$
By Eqn. (19), we have
$$
\begin{array}{l} G \left(x _ {t} ^ {\pi}, x _ {t - p: t - 1}, x _ {t - p: t} ^ {\pi}\right) + \sum_ {\tau \in \mathcal {A} _ {t} \cup \mathcal {B} _ {t}} d \left(x _ {\tau}, x _ {\tau - p: \tau - 1}\right) - \sum_ {\tau \in \mathcal {A} _ {t - 1} \cup \mathcal {B} _ {t - 1}} d \left(x _ {\tau}, x _ {\tau - p: \tau - 1}\right) \\ \leq G \left(x _ {t - 1}, x _ {t - p - 1: t - 2}, x _ {t - p - 1: t - 1} ^ {\pi}\right) + (1 + \lambda) \left(\sum_ {\tau \in \mathcal {A} _ {t} \cup \mathcal {B} _ {t}} d \left(x _ {\tau} ^ {\pi}, x _ {\tau - p: \tau - 1} ^ {\pi}\right) - \sum_ {\tau \in \mathcal {A} _ {t - 1} \cup \mathcal {B} _ {t - 1}} d \left(x _ {\tau} ^ {\pi}, x _ {\tau - p: \tau - 1} ^ {\pi}\right)\right) \tag {20} \\ \end{array}
$$
Now we define a new set  $\mathcal{D}_t = \mathcal{A}_t\backslash \mathcal{A}_{t - 1}$ , which denotes the timestep set for the newly received context parameters at  $t$ .
Case 1: If  $t \in \mathcal{D}_t$ , then  $B_{t-1} \backslash B_t = \mathcal{D}_t \backslash \{t\}$ , then we have
$$
\begin{array}{l} \left(\sum_ {\tau \in \mathcal {A} _ {t}} f \left(x _ {\tau}, y _ {\tau}\right) + \sum_ {\tau \in \mathcal {B} _ {t}} H \left(x _ {\tau}, x _ {\tau} ^ {\pi}\right)\right) - \left(\sum_ {\tau \in \mathcal {A} _ {t - 1}} f \left(x _ {\tau}, y _ {\tau}\right) + \sum_ {\tau \in \mathcal {B} _ {t - 1}} H \left(x _ {\tau}, x _ {\tau} ^ {\pi}\right)\right) \\ = \sum_ {\tau \in \mathcal {D} _ {t}} f \left(x _ {\tau}, y _ {\tau}\right) - \sum_ {\tau \in \mathcal {D} _ {t} \backslash \{t \}} H \left(x _ {\tau}, x _ {\tau} ^ {\pi}\right) = f \left(x _ {t} ^ {\pi}, y _ {t}\right) + \sum_ {\tau \in \mathcal {D} _ {t} \backslash \{t \}} f \left(x _ {\tau}, y _ {\tau}\right) - \sum_ {\tau \in \mathcal {D} _ {t} \backslash \{t \}} H \left(x _ {\tau}, x _ {\tau} ^ {\pi}\right) \tag {21} \\ \end{array}
$$
Since hitting cost  $f(\cdot, y_t)$  is  $\beta_h$ -smooth, we have
$$
\begin{array}{l} \sum_ {\tau \in \mathcal {D} _ {t} \setminus \{t \}} f (x _ {\tau}, y _ {\tau}) - \sum_ {\tau \in \mathcal {D} _ {t} \setminus \{t \}} (1 + \lambda) f (x _ {\tau} ^ {\pi}, y _ {\tau}) \\ \leq \frac {\beta_ {h} \left(1 + \frac {1}{\lambda}\right)}{2} \sum_ {\tau \in \mathcal {D} _ {t} \backslash \{t \}} \| x _ {\tau} ^ {\pi} - x _ {\tau} \| ^ {2} \leq \sum_ {\tau \in \mathcal {D} _ {t} \backslash \{t \}} H \left(x _ {\tau}, x _ {\tau} ^ {\pi}\right) \tag {22} \\ \end{array}
$$
Substituting Eqn. (22) back to Eqn. (21), we have
$$
\begin{array}{l} \left(\sum_ {\tau \in \mathcal {A} _ {t}} f \left(x _ {\tau}, y _ {\tau}\right) + \sum_ {\tau \in \mathcal {B} _ {t}} H \left(x _ {\tau}, x _ {\tau} ^ {\pi}\right)\right) - \left(\sum_ {\tau \in \mathcal {A} _ {t - 1}} f \left(x _ {\tau}, y _ {\tau}\right) + \sum_ {\tau \in \mathcal {B} _ {t - 1}} H \left(x _ {\tau}, x _ {\tau} ^ {\pi}\right)\right) \tag {23} \\ \leq (1 + \lambda) \left(\sum_ {\tau \in \mathcal {A} _ {t}} f \left(x _ {\tau}, y _ {\tau}\right) - \sum_ {\tau \in \mathcal {A} _ {t - 1}} f \left(x _ {\tau}, y _ {\tau}\right)\right) \\ \end{array}
$$
Case 2: If  $t \notin \mathcal{D}_t$ , then  $(B_{t-1} \cup \{t\}) \backslash B_t = \mathcal{D}_t$  and we have
$$
\begin{array}{l} \left(\sum_ {\tau \in \mathcal {A} _ {t}} f \left(x _ {\tau}, y _ {\tau}\right) + \sum_ {\tau \in \mathcal {B} _ {t}} H \left(x _ {\tau}, x _ {\tau} ^ {\pi}\right)\right) - \left(\sum_ {\tau \in \mathcal {A} _ {t - 1}} f \left(x _ {\tau}, y _ {\tau}\right) + \sum_ {\tau \in \mathcal {B} _ {t - 1}} H \left(x _ {\tau}, x _ {\tau} ^ {\pi}\right)\right) \\ = \sum_ {\tau \in \mathcal {D} _ {t}} f \left(x _ {\tau}, y _ {\tau}\right) - \sum_ {\tau \in \mathcal {D} _ {t}} H \left(x _ {\tau}, x _ {\tau} ^ {\pi}\right) + H \left(x _ {t} ^ {\pi}, x _ {t} ^ {\pi}\right) \tag {24} \\ = \sum_ {\tau \in \mathcal {D} _ {t}} f (x _ {\tau}, y _ {\tau}) - \sum_ {\tau \in \mathcal {D} _ {t}} H (x _ {\tau}, x _ {\tau} ^ {\pi}) \\ \end{array}
$$
$$
\sum_ {\tau \in \mathcal {D} _ {t}} f \left(x _ {\tau}, y _ {\tau}\right) - \sum_ {\tau \in \mathcal {D} _ {t}} (1 + \lambda) f \left(x _ {\tau} ^ {\pi}, y _ {\tau}\right) \leq \frac {\beta_ {h} \left(1 + \frac {1}{\lambda}\right)}{2} \sum_ {\tau \in \mathcal {D} _ {t}} \left\| x _ {\tau} ^ {\pi} - x _ {\tau} \right\| ^ {2} \leq \sum_ {\tau \in \mathcal {D} _ {t}} H \left(x _ {\tau}, x _ {\tau} ^ {\pi}\right) \tag {25}
$$
Since  $\lambda \geq 0$ , we substitute Eqn. (25) back to Eqn. (24), we have the same conclusion as Eqn (23).
Adding Eqn. (13), Eqn. (20) and Eqn. (23) together, we can prove  $x = x_{t}^{\pi}$  satisfies the constraints in Eqn. (1). At time step  $T$ , we have
$$
\begin{array}{l} \sum_ {\tau \in \mathcal {A} _ {T}} f (x _ {\tau}, y _ {\tau}) + \sum_ {\tau \in \mathcal {A} _ {T} \cup \mathcal {B} _ {T}} d (x _ {\tau}, x _ {\tau - p: \tau - 1}) + \sum_ {\tau \in \mathcal {B} _ {T}} \left(f (x _ {\tau}, y _ {\tau}) - (1 + \lambda) f \left(x _ {\tau} ^ {\pi}, y _ {\tau}\right)\right) \\ \leq \sum_ {\tau \in \mathcal {A} _ {T}} f \left(x _ {\tau}, y _ {\tau}\right) + \sum_ {\tau \in \mathcal {A} _ {T} \cup \mathcal {B} _ {T}} d \left(x _ {\tau}, x _ {\tau - p: \tau - 1}\right) + \sum_ {\tau \in \mathcal {B} _ {T}} H \left(x _ {\tau}, x _ {\tau} ^ {\pi}\right) \tag {26} \\ \leq (1 + \lambda) \left(\sum_ {\tau \in \mathcal {A} _ {T}} f \left(x _ {\tau} ^ {\pi}, y _ {\tau}\right) + \sum_ {\tau \in \mathcal {A} _ {T} \cup \mathcal {B} _ {T}} d \left(x _ {\tau} ^ {\pi}, x _ {\tau - p: \tau - 1} ^ {\pi}\right)\right) \\ \end{array}
$$
$$
\sum_ {\tau \in \mathcal {A} _ {T} \cup \mathcal {B} _ {T}} \left(f \left(x _ {\tau}, y _ {\tau}\right) + d \left(x _ {\tau}, x _ {\tau - p: \tau - 1}\right)\right) \leq (1 + \lambda) \sum_ {\tau \in \mathcal {A} _ {T} \cup \mathcal {B} _ {T}} \left(f \left(x _ {\tau} ^ {\pi}, y _ {\tau}\right) + d \left(x _ {\tau}, x _ {\tau - p: \tau - 1}\right)\right) \tag {27}
$$
In the next lemma, we bound the difference between the projected action and the ML predictions.
Lemma C.5. Suppose hitting cost is  $\beta_{h}$ -smooth, given the expert policy  $\pi$ , ML predictions  $\tilde{x}_{1:T}$ , for any  $\lambda > 0$  and  $\lambda_1 > 0$ , the total distance between actual actions  $x_{1:T}$  and ML predictions  $\tilde{x}_{1:T}$  are bounded,
$$
\sum_ {i = 1} ^ {T} \| x _ {t} - \tilde {x} _ {t} \| ^ {2} \leq \sum_ {i = 1} ^ {T} \left(\left[ \| \tilde {x} _ {t} - x _ {t} ^ {\pi} \| - \sqrt {K \left(d \left(x _ {t} ^ {\pi} , x _ {t - p : t - 1} ^ {\pi}\right) + \sum_ {\tau \in \mathcal {D} _ {t}} f \left(x _ {\tau} ^ {\pi} , y _ {\tau}\right)\right)} \right] ^ {+}\right) ^ {2} \tag {28}
$$
where  $[\cdot ]^{+}$  is the ReLU function and  $K = \frac{2(\lambda - \lambda_0)}{\beta_h(1 + \frac{1}{\lambda_0}) + \alpha^2(1 + \frac{1}{\lambda_0})}$ ,  $\alpha = 1 + \sum_{i = 1}^{p}L_{i}$
Proof. Suppose we at  $t - 1$  have the following inequality:
$$
\begin{array}{l} \sum_ {\tau \in \mathcal {A} _ {t - 1}} f (x _ {\tau}, y _ {\tau}) + \sum_ {\tau \in \mathcal {A} _ {t - 1} \cup \mathcal {B} _ {t - 1}} d (x _ {\tau}, x _ {\tau - p: \tau - 1}) + \sum_ {\tau \in \mathcal {B} _ {t - 1}} H (x _ {\tau}, x _ {\tau} ^ {\pi}) + G (x, x _ {t - p: t - 1}, x _ {t - p: t} ^ {\pi}) \\ \leq (1 + \lambda) \left(\sum_ {\tau \in \mathcal {A} _ {t - 1}} f \left(x _ {\tau} ^ {\pi}, y _ {\tau}\right) + \sum_ {\tau \in \mathcal {A} _ {t - 1} \cup \mathcal {B} _ {t - 1}} d \left(x _ {\tau} ^ {\pi}, x _ {\tau - p: \tau - 1} ^ {\pi}\right)\right) \tag {29} \\ \end{array}
$$
Remember that  $\mathcal{D}_t = \mathcal{A}_t\backslash \mathcal{A}_{t - 1}$  is the set of the time steps for the newly received context parameters at  $t$ . The robustness constraint in Eqn. (1) is satisfied if  $x_{t}$  satisfies the following inequality.
$$
\begin{array}{l} \left(\sum_ {\tau \in \mathcal {D} _ {t}} f (x _ {\tau}, y _ {\tau}) + \sum_ {\tau \in \mathcal {B} _ {t}} H (x _ {\tau}, x _ {\tau} ^ {\pi}) - \sum_ {\tau \in \mathcal {B} _ {t - 1}} H (x _ {\tau}, x _ {\tau} ^ {\pi})\right) + d (x _ {t}, x _ {t - p: t - 1}) + G (x _ {t}, x _ {t - p: t - 1}, x _ {t - p: t} ^ {\pi}) \\ - G \left(x _ {t - 1}, x _ {t - p - 1: t - 2}, x _ {t - p - 1: t - 1} ^ {\pi}\right) \leq (1 + \lambda) \left(d \left(x _ {t} ^ {\pi}, x _ {t - p: t - 1} ^ {\pi}\right) + \sum_ {\tau \in \mathcal {D} _ {t}} f \left(x _ {\tau} ^ {\pi}, y _ {\tau}\right)\right) \tag {30} \\ \end{array}
$$
For the switching cost, we have
$$
\begin{array}{l} d (x, x _ {t - p: t - 1}) - (1 + \lambda_ {0}) d (x _ {t} ^ {\pi}, x _ {t - p: t - 1} ^ {\pi}) \\ \leq \frac {1}{2} \left(1 + \frac {1}{\lambda_ {0}}\right) \left(\| x - x _ {t} ^ {\pi} \| + \| \delta \left(x _ {t - p: t - 1}\right) - \delta \left(x _ {t - p: t - 1} ^ {\pi}\right) \|\right) ^ {2} \\ \leq \frac {1}{2} \left(1 + \frac {1}{\lambda_ {0}}\right) \left(\| x - x _ {t} ^ {\pi} \| + \sum_ {i = 1} ^ {p} L _ {i} \| x _ {t - i} - x _ {t - i} ^ {\pi} \|\right) ^ {2} \tag {31} \\ \leq \frac {\alpha \left(1 + \frac {1}{\lambda_ {0}}\right)}{2} \left(\| x - x _ {t} ^ {\pi} \| ^ {2} + \sum_ {i = 1} ^ {p} L _ {i} \| x _ {t - i} - x _ {t - i} ^ {\pi} \| ^ {2}\right) \\ \end{array}
$$
The first inequality comes from Lemma C.1, the second inequality comes from the  $L_{i}$ -Lipschitz assumption, and the third inequality is because  $\alpha \geq 1$ . Besides, from Eqn (17), we have
$$
G \left(x _ {t - 1}, x _ {t - p - 1: t - 2}, x _ {t - p - 1: t - 1} ^ {\pi}\right) - G \left(x _ {t} ^ {\pi}, x _ {t - p: t - 1}, x _ {t - p: t} ^ {\pi}\right) = \frac {\alpha \left(1 + \frac {1}{\lambda_ {0}}\right)}{2} \sum_ {i = 1} ^ {p} L _ {i} \| x _ {t - i} - x _ {t - i} ^ {\pi} \| ^ {2} \tag {32}
$$
$$
\begin{array}{l} G (x, x _ {t - p: t - 1}, x _ {t - p: t} ^ {\pi}) - G (x _ {t - 1}, x _ {t - p - 1: t - 2}, x _ {t - p - 1: t - 1} ^ {\pi}) \\ = G (x, x _ {t - p: t - 1}, x _ {t - p: t} ^ {\pi}) - G (x _ {t} ^ {\pi}, x _ {t - p: t - 1}, x _ {t - p: t} ^ {\pi}) + G (x _ {t} ^ {\pi}, x _ {t - p: t - 1}, x _ {t - p: t} ^ {\pi}) - G (x _ {t - 1}, x _ {t - p - 1: t - 2}, x _ {t - p - 1: t - 1} ^ {\pi}) \\ = G \left(x, x _ {t - p: t - 1}, x _ {t - p: t} ^ {\pi}\right) - G \left(x _ {t} ^ {\pi}, x _ {t - p: t - 1}, x _ {t - p: t} ^ {\pi}\right) - \frac {\alpha \left(1 + \frac {1}{\lambda_ {0}}\right)}{2} \sum_ {i = 1} ^ {p} L _ {i} \| x _ {t - i} - x _ {t - i} ^ {\pi} \| ^ {2}. \tag {33} \\ \end{array}
$$
Combining with inequality (31), we have
$$
\begin{array}{l} G (x _ {t}, x _ {t - p: t - 1}, x _ {t - p: t} ^ {\pi}) - G (x _ {t - 1}, x _ {t - p - 1: t - 2}, x _ {t - p - 1: t - 1} ^ {\pi}) + d (x _ {t}, x _ {t - p: t - 1}) - (1 + \lambda_ {0}) d (x _ {t} ^ {\pi}, x _ {t - p: t - 1} ^ {\pi}) \\ \leq G (x _ {t}, x _ {t - p: t - 1}, x _ {t - p: t} ^ {\pi}) - G (x _ {t} ^ {\pi}, x _ {t - p: t - 1}, x _ {t - p: t} ^ {\pi}) + \frac {\alpha (1 + \frac {1}{\lambda_ {0}})}{2} \| x _ {t} - x _ {t} ^ {\pi} \| ^ {2} \\ = \frac {\alpha \left(1 + \frac {1}{\lambda_ {0}}\right) \sum_ {k = 1} ^ {p} L _ {k}}{2} \| x _ {t} - x _ {t} ^ {\pi} \| ^ {2} + \frac {\alpha \left(1 + \frac {1}{\lambda_ {0}}\right)}{2} \sum_ {k = 1} ^ {p} \| x _ {t} - x _ {t} ^ {\pi} \| ^ {2} \\ = \frac {\alpha^ {2} \left(1 + \frac {1}{\lambda_ {0}}\right)}{2} \| x _ {t} - x _ {t} ^ {\pi} \| ^ {2} \tag {34} \\ \end{array}
$$
Substituting Eqn. (34) back to Eqn. (30), we have
$$
\begin{array}{l} \sum_ {\tau \in \mathcal {D} _ {t}} (f (x _ {\tau}, y _ {\tau}) - (1 + \lambda_ {0}) f (x _ {\tau} ^ {\pi}, y _ {\tau})) + \sum_ {\tau \in \mathcal {B} _ {t}} H (x _ {\tau}, x _ {\tau} ^ {\pi}) - \sum_ {\tau \in \mathcal {B} _ {t - 1}} H (x _ {\tau}, x _ {\tau} ^ {\pi}) \\ + \frac {\alpha^ {2} \left(1 + \frac {1}{\lambda_ {0}}\right)}{2} \| x - x _ {t} ^ {\pi} \| ^ {2} \leq (\lambda - \lambda_ {0}) \left(d \left(x _ {t} ^ {\pi}, x _ {t - p: t - 1} ^ {\pi}\right) + \sum_ {\tau \in \mathcal {D} _ {t}} f \left(x _ {\tau} ^ {\pi}, y _ {\tau}\right)\right) \tag {35} \\ \end{array}
$$
Case 1: If  $t \in \mathcal{D}_t$ , then  $B_{t-1} \backslash B_t = \mathcal{D}_t \backslash \{t\}$ , then Eqn.(35) becomes
$$
\begin{array}{l} f (x _ {t}, y _ {t}) - (1 + \lambda_ {0}) f (x _ {t} ^ {\pi}, y _ {t}) + \frac {\alpha^ {2} (1 + \frac {1}{\lambda_ {0}})}{2} \| x - x _ {t} ^ {\pi} \| ^ {2} \\ + \sum_ {\tau \in \mathcal {D} _ {t} \backslash \{t \}} f \left(x _ {\tau}, y _ {\tau}\right) - \left(1 + \lambda_ {0}\right) f \left(x _ {\tau} ^ {\pi}, y _ {\tau}\right) - H \left(x _ {\tau}, x _ {\tau} ^ {\pi}\right) \leq \left(\lambda - \lambda_ {0}\right) \left(d \left(x _ {t} ^ {\pi}, x _ {t - p: t - 1} ^ {\pi}\right) + \sum_ {\tau \in \mathcal {D} _ {t}} f \left(x _ {\tau} ^ {\pi}, y _ {\tau}\right)\right) \tag {36} \\ \end{array}
$$
Since hitting cost is  $\beta_{h}$ -smooth, the sufficient condition for Eqn. (35) becomes
$$
\frac {(\beta_ {h} + \alpha^ {2}) (1 + \frac {1}{\lambda_ {0}})}{2} \| x - x _ {t} ^ {\pi} \| ^ {2} \leq (\lambda - \lambda_ {0}) \left(d \left(x _ {t} ^ {\pi}, x _ {t - p: t - 1} ^ {\pi}\right) + \sum_ {\tau \in \mathcal {D} _ {t}} f \left(x _ {\tau} ^ {\pi}, y _ {\tau}\right)\right) \tag {37}
$$
Since the hitting cost is non-negative, the sufficient condition can be further simplified, which is
$$
\frac {\left(\beta_ {h} + \alpha^ {2}\right) \left(1 + \frac {1}{\lambda_ {0}}\right)}{2} \| x - x _ {t} ^ {\pi} \| ^ {2} \leq (\lambda - \lambda_ {0}) \left(f \left(x _ {t} ^ {\pi}, y _ {t}\right) + d \left(x _ {t} ^ {\pi}, x _ {t - p: t - 1} ^ {\pi}\right)\right) \tag {38}
$$
Case 2: If  $t \notin \mathcal{D}_t$ , then  $(B_{t-1} \cup \{t\}) \backslash B_t = \mathcal{D}_t$ , then Eqn.(35) becomes
$$
\begin{array}{l} \frac {\alpha^ {2} \left(1 + \frac {1}{\lambda_ {0}}\right)}{2} \| x - x _ {t} ^ {\pi} \| ^ {2} + H \left(x, x _ {t} ^ {\pi}\right) + \sum_ {\tau \in \mathcal {D} _ {t}} \left(f \left(x _ {\tau}, y _ {\tau}\right) - \left(1 + \lambda_ {0}\right) f \left(x _ {\tau} ^ {\pi}, y _ {\tau}\right) - H \left(x _ {\tau}, x _ {\tau} ^ {\pi}\right)\right) \tag {39} \\ \leq \left(\lambda - \lambda_ {0}\right) \left(d \left(x _ {t} ^ {\pi}, x _ {t - p: t - 1} ^ {\pi}\right) + \sum_ {\tau \in \mathcal {D} _ {t}} f \left(x _ {\tau} ^ {\pi}, y _ {\tau}\right)\right) \\ \end{array}
$$
Since hitting cost is  $\beta_{h}$ -smooth, the sufficient condition for Eqn. (39) becomes
$$
\frac {(\beta_ {h} + \alpha^ {2}) (1 + \frac {1}{\lambda_ {0}})}{2} \| x - x _ {t} ^ {\pi} \| ^ {2} \leq (\lambda - \lambda_ {0}) \left(d \left(x _ {t} ^ {\pi}, x _ {t - p: t - 1} ^ {\pi}\right) + \sum_ {\tau \in \mathcal {D} _ {t}} f \left(x _ {\tau} ^ {\pi}, y _ {\tau}\right)\right) \tag {40}
$$
$$
K = \frac {2 (\lambda - \lambda_ {0})}{(\beta_ {h} + \alpha^ {2}) (1 + \frac {1}{\lambda_ {0}})}
$$
At time step  $t$ , if  $x_{t}^{\prime}$  is the solution to this alternative optimization problem
$$
x _ {t} ^ {\prime} = \arg \min  _ {x} \frac {1}{2} \| x - \tilde {x} _ {t} \| ^ {2}
$$
$$
s. t. \quad \| x - x _ {t} ^ {\pi} \| ^ {2} \leq K \left(d \left(x _ {t} ^ {\pi}, x _ {t - p: t - 1} ^ {\pi}\right) + \sum_ {\tau \in \mathcal {D} _ {t}} f \left(x _ {\tau} ^ {\pi}, y _ {\tau}\right)\right) \tag {41}
$$
The solution to this problem can be calculated as
$$
x _ {t} ^ {\prime} = \theta x _ {t} ^ {\pi} + (1 - \theta) \tilde {x} _ {t}
$$
$$
\theta = \left[ 1 - \frac {\sqrt {K \left(d \left(x _ {t} ^ {\pi} , x _ {t - p : t - 1} ^ {\pi}\right) + \sum_ {\tau \in \mathcal {D} _ {t}} f \left(x _ {\tau} ^ {\pi} , y _ {\tau}\right)\right)}}{\| \tilde {x} _ {t} - x _ {t} ^ {\pi} \|} \right] ^ {+}. \tag {42}
$$
Then  $\| x_t' - \tilde{x}_t\| = \left[\|\tilde{x}_t - x_t^\pi\| - \sqrt{K\left(d(x_t^\pi, x_{t-p:t-1}^\pi) + \sum_{\tau \in \mathcal{D}_t} f(x_\tau^\pi, y_\tau)\right)}\right]^+$ . Since  $x_t'$  also satisfies the original robustness constraint, we have  $\| x_t - \tilde{x}_t\| \leq \| x_t' - \tilde{x}_t\|$  and we finish the proof.
Proof of Theorem 4.1
Now summing up the distance through 1 to T, we have
$$
\sum_ {i = 1} ^ {T} \left\| x _ {t} - \tilde {x} _ {t} \right\| ^ {2} \leq \sum_ {i = 1} ^ {T} \left(\left[ \left\| \tilde {x} _ {t} - x _ {t} ^ {\pi} \right\| - \sqrt {K \left(d \left(x _ {t} ^ {\pi} , x _ {t - p : t - 1} ^ {\pi}\right) + \sum_ {\tau \in \mathcal {D} _ {t}} f \left(x _ {\tau} ^ {\pi} , y _ {\tau}\right)\right)} \right] ^ {+}\right) ^ {2} \tag {43}
$$
Based on Lemma C.3 we have  $\forall \lambda_2 > 0$
$$
\operatorname {c o s t} \left(x _ {1: T}\right) - \left(1 + \lambda_ {2}\right) \operatorname {c o s t} \left(\tilde {x} _ {1: T}\right) \leq \frac {\beta + \alpha^ {2}}{2} \left(1 + \frac {1}{\lambda_ {2}}\right) \sum_ {i = 1} ^ {T} \| x _ {t} - \tilde {x} _ {t} \| ^ {2}. \tag {44}
$$
Suppose the offline optimal action sequence is  $x_{1:T}^*$ , the optimal cost is  $\mathrm{cost}(x_{1:T}^*)$ . Then we divide both sides of Eqn. (44) by  $\mathrm{cost}(x_{1:T}^*)$ , and get  $\forall \lambda_2 > 0$