SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\operatorname {R e g r e t} (T) := \sum_ {t = 1} ^ {T} \left(V ^ {\star} \left(s _ {t}\right) - V ^ {\pi_ {t}} \left(s _ {t}\right)\right), \tag {13}
$$
where  $\pi_t$  is the stationary policy that the algorithm uses to take action  $a_t$  at step  $t$ .
The non-stationary regret metric considered in [12] is
$$
\operatorname {R e g r e t} _ {\mathrm {N S}} (T) := \sum_ {t = 1} ^ {T} \left(V ^ {\star} \left(s _ {t}\right) - V ^ {\{\pi_ {j} \} _ {j = t} ^ {\infty}} \left(s _ {t}\right)\right). \tag {14}
$$
Here,  $V^{\{\pi_j\}_{j = t}^\infty}(s) \coloneqq \mathbb{E}\left[\sum_{i = 0}^\infty \gamma^i r(s_i,\pi_{t + i}(s_i))\mid s_0 = s\right]$ , which is the expected cumulative future reward of the non-stationary algorithm starting from time  $t$ .
In fact, similar difference can also be observed in the literature focused on sample complexity of exploration. Notably, the metric in [49] compares the optimal value function  $V^{\star}(s_t)$  against  $V^{\pi_t}(s_t)$ , while the metric in works such as [35; 8] compares against  $V^{\{\pi_j\}_{j = t}^{\infty}(s_t)}$ .
There is no formal equivalence between the two regret metrics in general, despite the intuition we have provided in Remark 1 that  $\mathrm{Regret}_{\mathrm{NS}}(T)$  should be smaller than  $\mathrm{Regret}(T)$  for any improving algorithm. However, we can show that the specific algorithm Q-SlowSwitch-Adv (Algorithm 1) achieves  $\widetilde{O} (\sqrt{\frac{SAT}{(1 - \gamma)^3}})$  regret under the regret metric defined in [12], which matches the lower bound in [12] and is thus optimal for this metric as well.
Before the analysis, let us first define a notation. Let  $f$  and  $g$  be two real-valued functions that take  $\mathcal{X} := (S, A, \gamma, T, \frac{1}{\delta})$  as arguments. If there exists a universal constant  $C > 0$  such that  $f(\mathcal{X}) \leq C g(\mathcal{X})$  for any instantiation of  $\mathcal{X}$ , we can denote this with the notation  $f(\mathcal{X}) \lesssim g(\mathcal{X})$ .
$$
\mathcal {T} = \{1 \leq t \leq T: \pi_ {t} \neq \pi_ {t + 1} \}, \tag {15}
$$
which is the set of time indices that the execution policy switches in the following step, and
$$
\mathcal {T} _ {H} = \{1 \leq t \leq T: t + h \in \mathcal {T} \text {f o r s o m e} 0 \leq h \leq H \}, \tag {16}
$$
which is the set of time indices that the execution policy switches in any of the following  $H$  steps.
First, notice
$$
\begin{array}{l} \left| \sum_ {t \notin \mathcal {T} _ {H}} \left(V ^ {\{\pi_ {j} \} _ {j = t} ^ {\infty}} (s _ {t}) - \sum_ {i = 0} ^ {H} \gamma^ {i} r (s _ {t + i}, a _ {t + i})\right) \right| \\ \leq \left| \sum_ {t \notin \mathcal {T} _ {H}} \left(\mathbb {E} \Big [ \sum_ {i = 0} ^ {H} \gamma^ {i} r (s _ {t + i}, a _ {t + i}) \Big ] - \sum_ {i = 0} ^ {H} \gamma^ {i} r (s _ {t + i}, a _ {t + i})\right) \right| + \frac {\gamma^ {H} T}{1 - \gamma} \\ \leq \sum_ {k = 1} ^ {H} \left| \sum_ {t = j H + k \notin \mathcal {T} _ {H}} \left(\mathbb {E} \left[ \sum_ {i = 0} ^ {H} \gamma^ {i} r (s _ {t + i}, a _ {t + i}) \right] - \sum_ {i = 0} ^ {H} \gamma^ {i} r (s _ {t + i}, a _ {t + i})\right) \right| + \frac {1}{T} \\ \lesssim \sqrt {\frac {T \log^ {3} T}{(1 - \gamma) ^ {3}}}, \tag {17} \\ \end{array}
$$
where the second line holds when  $H \gtrsim \frac{\log T}{1 - \gamma}$ , and the last line makes use of the Azuma-Hoeffding inequality.
$$
\left| \sum_ {t \notin \mathcal {T} _ {H}} \left(V ^ {\pi_ {t}} \left(s _ {t}\right) - \sum_ {i = 0} ^ {H} \gamma^ {i} r \left(s _ {t + i}, a _ {t + i}\right)\right) \right| \lesssim \sqrt {\frac {T \log^ {3} T}{(1 - \gamma) ^ {3}}} \tag {18}
$$
since  $\pi_t = \pi_{t + i}$  for  $0\leq i\leq H$  and  $t\notin \mathcal{T}_H$
Putting (17) and (18) together leads to
$$
\begin{array}{l} \left| \operatorname {R e g r e t} (T) - \operatorname {R e g r e t} _ {\mathrm {N S}} (T) \right| = \left| \sum_ {t = 1} ^ {T} \left(V _ {t} ^ {\pi_ {t}} \left(s _ {t}\right) - V ^ {\{\pi_ {j} \} _ {j = t} ^ {\infty}} \left(s _ {t}\right)\right) \right| \\ \lesssim \sqrt {\frac {T \log^ {3} T}{(1 - \gamma) ^ {3}}} + \frac {| \mathcal {T} | \log T}{(1 - \gamma) ^ {2}} \tag {19} \\ \end{array}
$$
by noticing that  $\left|\sum_{t\in \mathcal{T}_H}\big(V^{\pi_t}(s_t) - V^{\{\pi_j\}_{j = t}^\infty}(s_t)\big)\right|\lesssim \frac{|\mathcal{T}_H|}{1 - \gamma}\lesssim \frac{|\mathcal{T}| \log T}{(1 - \gamma)^2}.$
Since our algorithm is low-switching and  $|\mathcal{T}|\leq \widetilde{O} (\frac{SA}{(1 - \gamma)^{9 / 2}} +\frac{(SA)^{3 / 4}T^{1 / 4}}{(1 - \gamma)^{5 / 4}})$ , the difference between the two regret metrics (19) is dominated by the upper bound on the regret itself. Thus, given our result from Theorem 1 that  $\mathrm{Regret}(T)\leq \widetilde{O} (\sqrt{\frac{SAT}{(1 - \gamma)^3}})$ , we can conclude  $\mathrm{Regret}_{\mathrm{NS}}(T)\leq \widetilde{O} (\sqrt{\frac{SAT}{(1 - \gamma)^3}})$  for our algorithm Q-SlowSwitch-Adv. In fact, this conversion holds as long as the algorithm's switching cost is dominated by the regret itself, e.g., when the switching cost is  $o(\sqrt{T})$ .