SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Then, the meta-algorithm chooses the base-learner by measuring the quality of each base-learner. In our approach, we choose the best base-learner at the last episode, that is,
$$
\pi_ {k, h} (\cdot \mid s) = \pi_ {k, h} ^ {i _ {k - 1, h} ^ {*}} (\cdot \mid s) \text {w i t h} i _ {k - 1, h} ^ {*} (s) = \arg \max  _ {i \in [ N ]} \langle Q _ {k - 1, h} (s, \cdot), \pi_ {k - 1, h} ^ {i} (\cdot \mid s) \rangle . \tag {14}
$$
The details are summarized in Algorithm 2 of Appendix A and the guarantee is as follows.
Theorem 5. Set  $\gamma = 1 / (KH)$ , step size pool  $\mathcal{H} = \{\eta_i = (2^i / H) \sqrt{(\log A) / K} \mid i \in [N]\}$  with  $N = \left\lfloor \frac{1}{2} \log \left( \frac{K}{\log A} \right) \right\rfloor$ . Algorithm 2 ensures
$$
\mathrm {D - R e g r e t} (K) \leq \widetilde {\mathcal {O}} \Big (\sqrt {d H ^ {4} K + d ^ {2} H ^ {3} K} + \sqrt {H ^ {4} (K + P _ {T}) (1 + P _ {T}) + H ^ {2} S _ {T} ^ {2}} \Big),
$$
where  $P_{T} = \sum_{k=1}^{K}\sum_{h=1}^{H}\|\pi_{k,h}^{c} - \pi_{k-1,h}^{c}\|_{1,\infty}$  is the path length of the compared policies,  $S_{T} = \sum_{k=1}^{K}\sum_{h=1}^{H}\mathbb{E}_{\pi_{k}^{c}}\mathbb{1}[i_{k,h}^{*}(s_{h}) \neq i_{k-1,h}^{*}(s_{h})]$  is the expected switching number of best base-learner.
Combining it with Theorem 3, we discuss the optimality of our result. We consider two regimes.
- Small  $S_T$ : when  $S_T \leq \max \{d\sqrt{HK}, H\sqrt{(K + P_T)(1 + P_T)}\}$ , the term  $S_T$  can be subsumed by other terms. In this case, the upper bound in Theorem 5 is entirely the same as that in Theorem 3. This implies we maintain the same guarantees without  $P_T$  as algorithmic input.
- Large  $S_T$ : when  $S_T > \max \{d\sqrt{HK}, H\sqrt{(K + P_T)(1 + P_T)}\}$ , our result looses a factor of  $HS_T$  compared with the result in Theorem 3 for the known  $P_T$  setting.
By the above discussion, our result can be optimal in terms of  $K$  and  $P_T$  under certain regimes when  $P_T$  is unknown. In comparison, the regret bounds achieved via the restart mechanism [35, 36] remain sub-optimal across all regimes even  $P_T$  is known. Note that we introduce the notation  $S_T$  in the regret analysis, which also reflects the degree of environmental non-stationarity to some extent. Consider the following two examples: (i) in the stationary environment,  $S_T$  could be relatively small as the best base-learner would seldom change, and (ii) in the piecewise-stationary environment,  $S_T$  would align with the frequency of environmental changes. Indeed, given that  $S_T$  is a data-dependent quantity, its inclusion in the regret analysis is not ideal. Deriving bounds that rely exclusively on problem-dependent quantities, like  $P_T$ , remains a significant open challenge.
In this work, we study the dynamic regret of adversarial linear mixture MDPs with the unknown transition. For the case when  $P_T$  is known, we propose a novel policy optimization algorithm that incorporates a fixed-share mechanism without the need for restarts. We show it enjoys a dynamic regret of  $\widetilde{\mathcal{O}}\big(\sqrt{d^2H^3K} + \sqrt{H^4(K + P_T)(1 + P_T)}\big)$ , strictly improving the previously best-known result of Zhong et al. [36] for the same setting and Fei et al. [35] when specialized to tabular MDPs. We also establish an  $\Omega\big(\sqrt{d^2H^3K} + \sqrt{HK(H + P_T)}\big)$  lower bound, indicating that our algorithm is optimal regarding  $d$ ,  $K$  and  $P_T$  and can be optimal in terms of  $H$  under certain regimes. Moreover, we explore the more complex scenario where  $P_T$  is unknown. We show this setting presents unique challenges that distinguish online MDPs from conventional online convex optimization. We introduce a novel two-layer algorithm and show its dynamic regret guarantee is attractive under certain regimes.
There are several important future works to investigate. First, how to remove the dependence on the switching number  $S_{T}$  is an important open question. Moreover, we focus on the full-information feedback in this work, it remains an open problem to extend the results to the bandit feedback.
This research was supported by National Key R&D Program of China (2022ZD0114800) and NSFC (62206125, 61921006). Peng Zhao was supported in part by the Xiaomi Foundation.
[1] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing Atari with deep reinforcement learning. ArXiv preprint, 1312.5602, 2013.
[2] David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Vedavyas Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy P. Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of Go with deep neural networks and tree search. Nature, pages 484-489, 2016.
[3] John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning (ICML), pages 1889-1897, 2015.
[4] Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In Proceedings of the 33nd International Conference on Machine Learning (ICML), pages 1329-1338, 2016.
[5] Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. Deep reinforcement learning for dialogue generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1192-1202, 2016.
[6] William Yang Wang, Jiwei Li, and Xiaodong He. Deep reinforcement learning for NLP. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL), pages 19-21, 2018.
[7] Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley, 1994.
[8] Eyal Even-Dar, Sham. M. Kakade, and Yishay Mansour. Online Markov decision processes. Mathematics of Operations Research, pages 726-736, 2009.
[9] Jia Yuan Yu, Shie Mannor, and Nahum Shimkin. Markov decision processes with arbitrary reward processes. Mathematics of Operations Research, pages 737-757, 2009.
[10] Gergely Neu, András György, Csaba Szepesvári, and András Antos. Online Markov decision processes under bandit feedback. In Advances in Neural Information Processing Systems 24 (NIPS), pages 1804–1812, 2010.
[11] Gergely Neu, András György, and Csaba Szepesvári. The online loop-free stochastic shortest-path problem. In Proceedings of 23rd Conference on Learning Theory (COLT), pages 231-243, 2010.
[12] Alexander Zimin and Gergely Neu. Online learning in episodic Markovian decision processes by relative entropy policy search. In Advances in Neural Information Processing Systems 26 (NIPS), pages 1583-1591, 2013.
[13] Aviv Rosenberg and Yishay Mansour. Online convex optimization in adversarial Markov decision processes. In Proceedings of the 36th International Conference on Machine Learning (ICML), pages 5478-5486, 2019.
[14] Chi Jin, Tiancheng Jin, Haipeng Luo, Suvrit Sra, and Tiancheng Yu. Learning adversarial Markov decision processes with bandit feedback and unknown transition. In Proceedings of the 37th International Conference on Machine Learning (ICML), pages 4860-4869, 2020.
[48] Wang Chi Cheung, David Simchi-Levi, and Ruihao Zhu. Reinforcement learning for nonstationary Markov decision processes: The blessing of (more) optimism. Proceedings of the 37th International Conference on Machine Learning (ICML), pages 1843-1854, 2020.
[49] Chen-Yu Wei and Haipeng Luo. Non-stationary reinforcement learning without prior knowledge: An optimal black-box approach. In Proceedings of the 34th Conference on Learning Theory (COLT), pages 4300–4354, 2021.
[50] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ArXiv preprint, 1707.06347, 2017.
[51] Yasin Abbasi-Yadkori, David Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. In Advances in Neural Information Processing Systems 24 (NIPS), pages 2312-2320, 2011.
[52] Lijun Zhang, Shiyin Lu, and Zhi-Hua Zhou. Adaptive online learning in dynamic environments. In Advances in Neural Information Processing Systems 31 (NeurIPS), pages 1330-1340, 2018.
[53] Peng Zhao and Lijun Zhang. Improved analysis for dynamic regret of strongly convex and smooth functions. In Proceedings of the 3rd Conference on Learning for Dynamics and Control (L4DC), pages 48-59, 2021.
[54] Liyu Chen, Haipeng Luo, and Chen-Yu Wei. Impossible tuning made possible: A new expert algorithm and its applications. In Proceedings of the 34th Conference On Learning Theory (COLT), pages 1216-1259, 2021.
[55] Chi Jin, Zeyuan Allen-Zhu, Sébastien Bubeck, and Michael I. Jordan. Is Q-learning provably efficient? In Advances in Neural Information Processing Systems 31 (NeurIPS), pages 4868-4878, 2018.
A Algorithm with Unknown  $P_{T}$
In this part, we present the POWERS-FixShare-OnE algorithm in Algorithm 2 for the case when  $P_T$  is unknown. The algorithm is based on the POWERS-FixShare algorithm and we further employ an online ensemble structure to eliminate the algorithmic dependence on  $P_T$ . In Line 5, each base-learner updates her policy with a specific step size  $\eta_i$  and the meta-learner selects the best policy among the base-learners in Line 6. In the policy evaluation phase (Line 11 - 21), we use the same estimator as in Algorithm 1 to estimate the parameter  $\theta_h^*$  and construct the confidence set.
Algorithm 2 POWERS-FixShare-OnE
Input: step size pool  $\mathcal{H}$ , exploration parameter  $\gamma$  and regularization parameter  $\lambda$ .  
1: Initialize  $\{\pi_{0,h}^{i}(\cdot |s)\}_{h = 1}^{H}, \forall i\in [N], s\in S$  as uniform distribution on  $\mathcal{A}$ ,  $\{p_{0,h}(s)\}_{h = 1}^{H}, \forall s\in S$  as uniform distribution on  $N$  and set  $\{Q_{0,h}(\cdot ,\cdot)\}_{h = 1}^{H}$  as zero function.  
2: for  $k = 1,2,\dots ,K$  do  
3: Receive the initial state  $s_{k,1}$ .  
4: for  $h = 1,2,\dots ,H$  do  
5: For all  $h\in [H], s\in S$ , each base-learner  $\mathcal{B}_i, \forall i\in [N]$  updates policy by  $\pi_{k,h}^{i,\prime}(\cdot |s)\propto \pi_{k - 1,h}^{i}(\cdot |s)\exp \left(\eta_{i}Q_{k - 1,h}(s,\cdot)\right), \pi_{k,h}^{i}(\cdot |s) = (1 - \gamma)\pi_{k,h}^{i,\prime}(\cdot |s) + \gamma \pi^u (\cdot |s)$ .  
6: Set  $\pi_{k,h}(\cdot |s) = \pi_{k,h}^{i_{k - 1,h}}(\cdot |s)$  with  $i_{k - 1,h}^{*}(s) = \arg \max_{i\in [N]}\langle Q_{k - 1,h}(s,\cdot),\pi_{k - 1,h}^{i}(\cdot |s)\rangle$ .  
7: Take the action following  $a_{k,h}\sim \pi_{k,h}(\cdot |s_{k,h})$  and transit to the next state  $s_k,h + 1$ .  
8: Obtain reward  $r_{k,h}(s_{k,h},a_{k,h})$  and observe the reward function  $r_k,h(\cdot ,\cdot)$ .  
9: end for  
10: Initialize  $V_{k,H + 1}(\cdot)$  as a zero function.  
11: for  $h = H,H - 1,\dots ,1$  do  
12: Set  $Q_{k,h}(\cdot ,\cdot)\gets \left[r_{k,h}(\cdot ,\cdot) + \langle \widehat{\theta}_{k,h},\phi_{V_{k,h + 1}}(\cdot ,\cdot)\rangle +\widehat{\beta}_k\| \widehat{\Sigma}_{k,h}^{-1 / 2}\phi_{V_{k,h + 1}}(\cdot ,\cdot)\| _2\right]_{[0,H - h + 1]}$ .  
13: Set  $V_{k,h}(\cdot)\gets \mathbb{E}_{a\sim \pi_{k,h}(\cdot |\cdot)}[Q_{k,h}(\cdot ,\cdot)]$ .  
14: Set the estimated variance  $[\bar{\nabla}_{k,h}V_{k,h + 1}](s_{k,h},a_{k,h})$  as in (11), bonus  $E_{k,h}$  as in (21).  
15:  $\bar{\sigma}_{k,h}\gets \sqrt{\max\left\{H^2 / d,[\bar{\nabla}_{k,h}V_{k,h + 1}] (s_{k,h},a_{k,h}) + E_{k,h}\right\}}$ .  
16:  $\widehat{\Sigma}_{k + 1,h}\gets \widehat{\Sigma}_{k,h} + \bar{\sigma}_{k,h}^{-2}\phi_{V_{k,h + 1}}(s_{k,h},a_{k,h})\phi_{V_{k,h + 1}}(s_{k,h},a_{k,h})^\top$ .  
17:  $\widehat{b}_{k + 1,h}\gets \widehat{b}_{k,h} + \bar{\sigma}_{k,h}^{-2}\phi_{V_{k,h + 1}}(s_{k,h},a_{k,h}) V_{k,h + 1}(s_{h + 1}^k)$ .  
18:  $\widetilde{\Sigma}_{k + 1,h}\gets \widetilde{\Sigma}_{k,h} + \phi_{V_{k,h + 1}^2}(s_{k,h},a_{k,h})\phi_{V_{k,h + 1}^2}(s_{k,h},a_{k,h})^\top$ .  
19:  $\widetilde{b}_{k + 1,h}\gets \widetilde{b}_{k,h} + \phi_{V_{k,h + 1}^2}(s_{k,h},a_{k,h}) V_{k,h + 1}^2 (s_h^k)$ .  
20:  $\widehat{\theta}_{k + 1,h}\gets \widehat{\Sigma}_{k + 1,h}^{-1}\widehat{b}_{k + 1,h},\widetilde{\theta}_{k + 1,h}\gets \widetilde{\Sigma}_{k + 1,h}^{-1}\widetilde{b}_{k + 1,h}$ .  
21: end for  
22: end for
B Recovering Tabular Case
In this part, we show the result of our algorithm when specialized to the tabular case. Note in the linear case, we adopt the weighted ridge regression to estimate the parameter  $\theta_h^*$ , that is, we construct the estimator  $\widehat{\theta}_{k,h}$  by solving the following weighted ridge regression problem:
In the tabular case, we simply set  $\bar{\sigma}_{j,h} = 1$ , and compute  $\phi_{V_{k,h+1}}$  by taking the sample mean of  $\{V_{k,h+1}(s_{j,h+1})\}_{j \in [k-1]}$ . That is, we set it as
$$
\phi_ {V _ {j, h + 1}} \left(s _ {j, h}, a _ {j, h}\right) = \sum_ {s ^ {\prime} \in \mathcal {S}} \frac {N _ {k} \left(s _ {j , h} , a _ {j , h} , s ^ {\prime}\right)}{N _ {k} \left(s _ {j , h} , a _ {j , h}\right) + \lambda} \cdot V _ {j, h + 1} \left(s ^ {\prime}\right),
$$
of each  $(s,a)$ , where  $N_{k}$  counts the number of times each tuple  $(s,a,s)$  or  $(s,a)$  has been visited up to episode  $k$ . Then, we construct the estimator  $\widehat{\theta}_{k,h}$  by solving the linear regression problem:
$$
\widehat {\theta} _ {k, h} = \underset {\theta \in \mathbb {R} ^ {d}} {\arg \min} \sum_ {j = 1} ^ {k - 1} \left[ \left\langle \phi_ {V _ {j, h + 1}} (s _ {j, h}, a _ {j, h}), \theta \right\rangle - V _ {j, h + 1} (s _ {j, h + 1}) \right] ^ {2} + \lambda \| \theta \| _ {2} ^ {2}.
$$
Then, the confidence set  $\widehat{C}_{k,h}$  is constructed as follows:
$$
\widehat {\mathcal {C}} _ {k, h} = \left\{\theta \mid \| \widehat {\Sigma} _ {k, h} ^ {1 / 2} (\theta - \widehat {\theta} _ {k, h}) \| _ {2} \leq \beta \right\}.
$$
The remaining part of the algorithm is the same as the linear case. The following theorem shows the result of our algorithm in the tabular case.
Theorem 6. Set  $\gamma = 1 / (KH)$ , step size pool  $\mathcal{H} = \{\eta_i = (2^i / H) \sqrt{(\log A) / K} \mid i \in [N]\}$  with  $N = \left\lfloor \frac{1}{2} \log \left( \frac{K}{\log A} \right) \right\rfloor$  and  $\beta = H \sqrt{S \log(dKH/\delta)}$ , with probability at least  $1 - \delta$ , it holds
$$
\mathrm {D} \text {- R e g r e t} (K) \leq \widetilde {\mathcal {O}} \Big (\sqrt {H ^ {4} S ^ {2} A K} + \sqrt {H ^ {4} (K + P _ {T}) (1 + P _ {T})} \Big),
$$
where  $P_{T} = \sum_{k = 1}^{K}\sum_{h = 1}^{H}\| \pi_{k,h}^{c} - \pi_{k - 1,h}^{c}\|_{1,\infty}$  is the path length of the compared policy sequence.
Proof. By Lemma 5, we decompose the dynamic regret as follows.
$$
\begin{array}{l} \mathrm {D} - \operatorname {R e g r e t} (K) \leq \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi_ {k} ^ {c}} \left[ \left\langle Q _ {k, h} \left(s _ {h}, \cdot\right), \pi_ {k, h} ^ {c} (\cdot \mid s _ {h}) - \pi_ {k, h} (\cdot \mid s _ {h}) \right\rangle \right] + \mathcal {M} _ {K, H} \\ + \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \left(\mathbb {E} _ {\pi_ {k} ^ {c}} \left[ \iota_ {k, h} \left(s _ {h}, a _ {h}\right) \right] - \iota_ {k, h} \left(s _ {k, h}, a _ {k, h}\right)\right), \\ \end{array}
$$
Note that the policy evaluation phase is the same as the one in Fei et al. [35, Algorithm 3], by the estimator bound in Fei et al. [35, Lemmas 7 and 9], we have
$$
\sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi_ {k} ^ {c}} \left[ \iota_ {k, h} \left(s _ {h}, a _ {h}\right) \right] \leq 0, \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} - \iota_ {k, h} \left(s _ {k, h}, a _ {k, h}\right) \leq \mathcal {O} \left(\sqrt {H ^ {4} S ^ {2} A T \log^ {2} (S A K H / \delta)}\right). \tag {15}
$$
The remaining proof is the same as the proof of Theorem 3 in Appendix C.7.
In this section, we provide the proof of the results in this paper.
Proof. By the definition of dynamic regret, we have
$$
\begin{array}{l} \mathrm {D} \text {- R e g r e t} (K) \\ = \sum_ {k = 1} ^ {K} V _ {k, 1} ^ {\pi_ {k} ^ {c}} (s _ {k, 1}) - \sum_ {k = 1} ^ {K} V _ {k, 1} ^ {\pi_ {k}} (s _ {k, 1}) \\ = \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi_ {k} ^ {c}} \left[ \left\langle Q _ {k, h} ^ {\pi_ {k}} (s _ {h}, \cdot), \pi_ {k, h} ^ {c} (\cdot | s _ {h}) - \pi_ {k, h} (\cdot | s _ {h}) \right\rangle \mid s _ {1} = s _ {k, 1} \right] \\ \leq \frac {\eta K H ^ {3}}{2} + \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi_ {k} ^ {c}} \left[ D _ {\mathrm {K L}} \left(\pi_ {k, h} ^ {c} (\cdot | s _ {h}) \| \pi_ {k, h} (\cdot | s _ {h})\right) - D _ {\mathrm {K L}} \left(\pi_ {k, h} ^ {c} (\cdot | s _ {h}) \| \pi_ {k + 1, h} (\cdot | s _ {h})\right) \right], \\ \end{array}
$$
where the equality holds by the performance difference lemma in Lemma 13 and the inequality holds by the one-step descent guarantee in Lemma 14. This finishes the proof.
Proof. Note the update rule is
$$
\pi_ {k, h} ^ {\prime} (\cdot \mid s) \propto \pi_ {k - 1, h} (\cdot \mid s) \exp (\eta \cdot Q _ {k - 1, h} ^ {\pi_ {k - 1}} (s, \cdot)), \pi_ {k, h} (\cdot \mid s) = (1 - \gamma) \pi_ {k, h} ^ {\prime} (\cdot \mid s) + \gamma \pi^ {u} (\cdot \mid s)
$$
By Lemma 14, for any  $s_h \in S, h \in [H], k \in [K]$ , we have
$$
\begin{array}{l} \left\langle Q _ {k, h} ^ {\pi_ {k}} \left(s _ {h}, \cdot\right), \pi_ {k, h} ^ {c} \left(\cdot \mid s _ {h}\right) - \pi_ {k, h} \left(\cdot \mid s _ {h}\right) \right\rangle \\ \leq \frac {\eta H ^ {2}}{2} + \frac {1}{\eta} \left(D _ {\mathrm {K L}} \left(\pi_ {k, h} ^ {c} (\cdot | s _ {h}) \| \pi_ {k, h} (\cdot | s _ {h})\right) - D _ {\mathrm {K L}} \left(\pi_ {k, h} ^ {c} (\cdot | s _ {h}) \| \pi_ {k + 1, h} ^ {\prime} (\cdot | s _ {h})\right)\right) \\ = \frac {\eta H ^ {2}}{2} + \frac {1}{\eta} \sum_ {a \in \mathcal {A}} \left(\pi_ {k, h} ^ {c} (a \mid s _ {h}) \log \frac {1}{\pi_ {k , h} (a \mid s _ {h})} - \pi_ {k, h} ^ {c} (a \mid s _ {h}) \log \frac {1}{\pi_ {k + 1 , h} ^ {\prime} (a \mid s _ {h})}\right), \tag {16} \\ \end{array}
$$
where the equality holds by the definition of KL divergence. We decompose the last term as follows:
$$
\begin{array}{l} \sum_ {a \in \mathcal {A}} \left(\pi_ {k, h} ^ {c} (a \mid s _ {h}) \log \frac {1}{\pi_ {k , h} (a \mid s _ {h})} - \pi_ {k, h} ^ {c} (a \mid s _ {h}) \log \frac {1}{\pi_ {k + 1 , h} ^ {\prime} (a \mid s _ {h})}\right) \\ = \sum_ {a \in \mathcal {A}} \left(\pi_ {k, h} ^ {c} (a \mid s _ {h}) \log \frac {1}{\pi_ {k , h} (a \mid s _ {h})} - \pi_ {k - 1, h} ^ {c} (a \mid s _ {h}) \log \frac {1}{\pi_ {k , h} ^ {\prime} (a \mid s _ {h})}\right) \tag {17} \\ + \sum_ {a \in \mathcal {A}} \left(\pi_ {k - 1, h} ^ {c} (a \mid s _ {h}) \log \frac {1}{\pi_ {k , h} ^ {\prime} (a \mid s _ {h})} - \pi_ {k, h} ^ {c} (a \mid s _ {h}) \log \frac {1}{\pi_ {k + 1 , h} ^ {\prime} (a \mid s _ {h})}\right). \\ \end{array}
$$
For the first term, we have
$$
\begin{array}{l} \sum_ {a \in \mathcal {A}} \left(\pi_ {k, h} ^ {c} (a \mid s _ {h}) \log \frac {1}{\pi_ {k , h} (a \mid s _ {h})} - \pi_ {k - 1, h} ^ {c} (a \mid s _ {h}) \log \frac {1}{\pi_ {k , h} ^ {\prime} (a \mid s _ {h})}\right) \\ = \sum_ {a \in \mathcal {A}} \left(\left(\pi_ {k, h} ^ {c} (a \mid s _ {h}) - \pi_ {k - 1, h} ^ {c} (a \mid s _ {h})\right) \log \frac {1}{\pi_ {k , h} (a \mid s _ {h})} + \pi_ {k - 1, h} ^ {c} (a \mid s _ {h}) \log \frac {\pi_ {k , h} ^ {\prime} (a \mid s _ {h})}{\pi_ {k , h} (a \mid s _ {h})}\right). \\ \end{array}
$$
By the update rule, we have  $1 \leq 1 / \pi_{k,h}(a \mid s_h) \leq A / \gamma$  and  $\pi_{k,h}'(a \mid s_h) / \pi_{k,h}(a \mid s_h) \leq 1 / (1 - \gamma)$ . Therefore, we have
$$
\begin{array}{l} \sum_ {a \in \mathcal {A}} \left(\left(\pi_ {k, h} ^ {c} (a \mid s _ {h}) - \pi_ {k - 1, h} ^ {c} (a \mid s _ {h})\right) \log \frac {1}{\pi_ {k , h} (a \mid s _ {h})} + \pi_ {k - 1, h} ^ {c} (a \mid s _ {h}) \log \frac {\pi_ {k , h} ^ {\prime} (a \mid s _ {h})}{\pi_ {k , h} (a \mid s _ {h})}\right) \\ \leq \left\| \pi_ {k, h} ^ {c} (\cdot \mid s _ {h}) - \pi_ {k - 1, h} ^ {c} (\cdot \mid s _ {h}) \right\| _ {1} \cdot \log \frac {A}{\gamma} + \log \frac {1}{1 - \gamma}. \tag {18} \\ \end{array}
$$
Then, the dynamic regret is bounded as follows:
D-Regret  $(K)$
$$
\begin{array}{l} = \sum_ {k = 1} ^ {K} V _ {k, 1} ^ {\pi_ {k} ^ {c}} \left(s _ {k, 1}\right) - \sum_ {k = 1} ^ {K} V _ {k, 1} ^ {\pi_ {k}} \left(s _ {k, 1}\right) \\ = \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi_ {k} ^ {c}} \left[ \left\langle Q _ {k, h} ^ {\pi_ {k}} (s _ {h}, \cdot), \pi_ {k, h} ^ {c} (\cdot | s _ {h}) - \pi_ {k, h} (\cdot | s _ {h}) \right\rangle \mid s _ {1} = s _ {k, 1} \right] \\ \leq \frac {\eta K H ^ {3}}{2} + \frac {1}{\eta} \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \sum_ {a \in \mathcal {A}} \mathbb {E} _ {\pi_ {k} ^ {c}} \left[ \pi_ {k, h} ^ {c} (a \mid s _ {h}) \log \frac {1}{\pi_ {k , h} (a \mid s _ {h})} - \pi_ {k, h} ^ {c} (a \mid s _ {h}) \log \frac {1}{\pi_ {k + 1 , h} ^ {\prime} (a \mid s _ {h})} \right] \\ \leq \frac {\eta K H ^ {3}}{2} + \frac {1}{\eta} \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi_ {k} ^ {c}} \left[ \left\| \pi_ {k, h} ^ {c} (\cdot | s _ {h}) - \pi_ {k - 1, h} ^ {c} (\cdot | s _ {h}) \right\| _ {1} \cdot \log \frac {A}{\gamma} + \log \frac {1}{1 - \gamma} \right] \\ + \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi_ {k} ^ {c}} \left[ \sum_ {a \in \mathcal {A}} \left(\pi_ {k - 1, h} ^ {c} (a \mid s _ {h}) \log \frac {1}{\pi_ {k , h} ^ {\prime} (a \mid s _ {h})} - \pi_ {k, h} ^ {c} (a \mid s _ {h}) \log \frac {1}{\pi_ {k + 1 , h} ^ {\prime} (a \mid s _ {h})}\right) \right] \\ \end{array}
$$
$$
\begin{array}{l} \leq \frac {\eta K H ^ {3}}{2} + \frac {1}{\eta} \left(P _ {T} \log \frac {A}{\gamma} + K H \log \frac {1}{1 - \gamma}\right) \\ + \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi_ {k} ^ {c}} \left[ \sum_ {a \in \mathcal {A}} \left(\pi_ {k - 1, h} ^ {c} (a \mid s _ {h}) \log \frac {1}{\pi_ {k , h} ^ {\prime} (a \mid s _ {h})} - \pi_ {k, h} ^ {c} (a \mid s _ {h}) \log \frac {1}{\pi_ {k + 1 , h} ^ {\prime} (a \mid s _ {h})}\right) \right], \\ \end{array}
$$
where the first inequality holds by (16), the second inequality is due to (17) and (18), and the last inequality holds by the definition of  $P_{T} = \sum_{k=1}^{K} \sum_{h=1}^{H} \left\| \pi_{k,h}^{c} - \pi_{k-1,h}^{c} \right\|_{1,\infty}$ . It ends the proof.
Proof. Lemma 3 is a simplified version of Chen et al. [54, Lemma 17] and we prove it in our notations below for self-containedness. It is easy to verify that the update rule  $p'(a) \propto p(a) \cdot \exp(\eta \cdot Q(s, a))$  is equivalent to the update
$$
p^{\prime} = \operatorname *{arg  max}_{p^{\prime}\in \Delta (\mathcal{A}}\eta \langle p^{\prime},Q(s,\cdot)\rangle +D_{\mathrm{KL}}(p^{\prime}\| p).
$$
Thus, by the KKT condition, we have for some  $\lambda$  and  $\mu(a) \geq 0$ , such that
$$
Q (s, a) - \frac {1}{\eta} \log \frac {p ^ {\prime} (a)}{p (a)} + \lambda + \mu (a) = 0, \text {a n d} \mu (a) p ^ {\prime} (a) = 0, \forall a \in \mathcal {A}.
$$
The above equations give the closed-form solution  $p'(a) = p(a)\exp(\eta(Q(s,a) + \lambda + \mu(a)))$ . First, we prove that for all  $a \in \mathcal{A}$  we have  $\mu(a) = 0$ . Indeed, when  $\mu(a) \neq 0$ , by  $\mu(a)p'(a) = 0$ , we have  $p'(a) = p(a)\exp(\eta(Q(s,a) + \lambda + \mu(a))) = 0$ , which contradicts with  $p(a) > 0$ .
Then, we now separately discuss two cases.
Case 1:  $\min_{a\in \mathcal{A}}Q(s,a)\neq \max_{a\in \mathcal{A}}Q(s,a)$ . In this case, we first show that  $\min_{a\in \mathcal{A}} - Q(s,a)\leq \lambda \leq \max_{a\in \mathcal{A}} - Q(s,a)$ . We prove it by contradiction: If  $\lambda \geq \max_{a\in \mathcal{A}} - Q(s,a)$ , then we have
$$
\sum_ {a \in \mathcal {A}} p ^ {\prime} (a) = \sum_ {a \in \mathcal {A}} p (a) \exp (\eta (Q (s, a) + \lambda + \mu (a))) \geq \sum_ {a \in \mathcal {A}} p (a) \geq 1.
$$