SlowGuess/NeurIPS_2023_OCR huggingface dataset:
6  $\begin{array}{r}\big{\lfloor}\mathrm{for~every}((s,a),q)\in \mathcal{D}\mathrm{d}\boldsymbol {Q}^{\mathrm{lazy}}(s,a)\leftarrow q. \end{array}$  # Update execution policy with the buffer
7 Function update-q-reference():
8  $[\mu^{\mathrm{ref}},\sigma^{\mathrm{ref}}](s_t,a_t)\gets \mathbf{update - moments}();$  9  $[\delta^{\mathrm{R}},B^{\mathrm{R}}](s_t,a_t)\gets \mathbf{update - reference - bonus});$  10  $b^{\mathrm{R}}\gets B^{\mathrm{R}}(s_t,a_t) + (1 - \eta_n)\frac{\delta^{\mathrm{R}}(s_t,a_t)}{\eta_n} +cb_{\frac{\iota^2}{n^{3 / 4}(1 - \gamma)^2}};$  11  $Q^{\mathrm{R}}(s_t,a_t)\gets$ $(1 - \eta_{n})Q^{\mathrm{R}}(s_{t},a_{t}) + \eta_{n}\left(r(s_{t},a_{t}) + \gamma \left(V(s_{t + 1}) - V^{\mathrm{R}}(s_{t + 1}) + \mu^{\mathrm{ref}}(s_{t},a_{t})\right) + b^{\mathrm{R}}\right).$
12 Function update-moments():
13  $\mu^{\mathrm{ref}}(s_t,a_t)\gets (1 - \frac{1}{n})\mu^{\mathrm{ref}}(s_t,a_t) + \frac{1}{n} V^{\mathrm{R}}(s_{t + 1});$  # Running mean of the reference   
14  $\sigma^{\mathrm{ref}}(s_t,a_t)\gets (1 - \frac{1}{n})\sigma^{\mathrm{ref}}(s_t,a_t) + \frac{1}{n}\left(V^{\mathrm{R}}(s_{t + 1})\right)^2;$  # Running  $2^{\mathrm{nd}}$  moment of the reference   
15  $\mu^{\mathrm{adv}}(s_t,a_t)\gets (1 - \eta_n)\mu^{\mathrm{adv}}(s_t,a_t) + \eta_n\left(V(s_{t + 1}) - V^{\mathrm{R}}(s_{t + 1})\right);$  # Running mean of the advantage
16  $\begin{array}{r}\sigma^{\mathrm{adv}}(s_t,a_t)\gets (1 - \eta_n)\sigma^{\mathrm{adv}}(s_t,a_t) + \eta_n\left(V(s_{t + 1}) - V^{\mathrm{R}}(s_{t + 1})\right)^2. \end{array}$  # Running  $2^{\mathrm{nd}}$  moment of the advantage
17 Function update-reference-bonus():
18  $B^{\mathrm{next}}(s_t,a_t)\gets$ $c_{b}\sqrt{\frac{\tau}{n}}\left(\sqrt{\sigma^{\mathrm{ref}}(s_{t},a_{t}) - (\mu^{\mathrm{ref}}(s_{t},a_{t}))^{2}} +\frac{1}{\sqrt{1 - \gamma}}\sqrt{\sigma^{\mathrm{adv}}(s_{t},a_{t}) - (\mu^{\mathrm{adv}}(s_{t},a_{t}))^{2}}\right);$    
19  $\delta^{\mathrm{R}}(s_t,a_t)\gets B^{\mathrm{next}}(s_t,a_t) - B^{\mathrm{R}}(s_t,a_t);$    
20  $B^{\mathrm{R}}(s_t,a_t)\gets B^{\mathrm{next}}(s_t,a_t).$
3.1 Review: Q-Learning with UCB and reference advantage
First, we make a brief review of the Q-learning with UCB method proposed in [15], referred to as UCB-Q hereafter, and its variance-reduced variant UCB-Q-Advantage, later introduced in [48]. The Q-function updates in Q-SlowSwitch-Adv are inspired by these two methods.
Q-learning with UCB The original Q-learning [41; 40] is a fixed-point iteration based on a stochastic approximation of the Bellman optimality equation (2). It uses a greedy policy with respect to its estimate of  $Q^{\star}$ , whose update rule can be summarized as:
$$
Q (s, a) \leftarrow (1 - \eta) Q (s, a) + \eta \left(r (s, a) + \gamma \widehat {P} _ {s, a} V\right). \tag {5}
$$
Above,  $Q$  (resp.  $V$ ) is the running estimate of  $Q^{\star}$  (resp.  $V^{\star}$ );  $\eta \in (0,1]$  is the (possibly varying) learning rate;  $\widehat{P}_{s,a}V$  is a stochastic approximation of  $P_{s,a}V$  (cf. (4)). Commonly,  $V(s')$  is used for  $\widehat{P}_{s,a}V$  in (5) as an unbiased estimate of  $P_{s,a}V$ , when a sample of state transition from  $(s,a)$ , namely  $(s,a,s')$ , is available.
However, as [15] point out, using (5) naively suffers from great regret suboptimality, for it rules out the state-action pairs with high value but few observations. To promote the exploration of such state-action pairs, UCB-Q appends (5) with an exploration bonus. Its update rule can be written as:
$$
Q ^ {\mathrm {U C B}} (s, a) \leftarrow (1 - \eta) Q ^ {\mathrm {U C B}} (s, a) + \eta \left(r (s, a) + \gamma \widehat {P} _ {s, a} V + b\right). \tag {6}
$$
To encourage exploration, the bonus  $b \geq 0$  is designed to maintain an upper confidence bound (UCB) on  $(\widehat{P}_{s,a} - P_{s,a})V$ , which in turn keeps  $Q^{\mathrm{UCB}}(s,a)$  as an "optimistic" overestimate of  $Q^{\star}(s,a)$ .
Q-learning with UCB and reference advantage The regret guarantee for UCB-Q is still shy of being optimal. In order to attain optimality, one can turn to the celebrated idea of variance reduction [16; 23; 34; 37], which decomposes the stochastic approximation target into two parts: a low-variance reference estimated with batches of samples and a low-magnitude advantage estimated with every new sample. In this spirit, [48] introduce UCB-Q-Advantage based on UCB-Q and a reference-advantage decomposition. Specifically, given a reference  $V^{\mathrm{R}}$  that is maintained as an approximation for  $V^{\star}$ , the update rule of UCB-Q-Advantage reads:
$$
Q ^ {\mathrm {R}} (s, a) \leftarrow (1 - \eta) Q ^ {\mathrm {R}} (s, a) + \eta \left(r (s, a) + \gamma \left(\widehat {P} _ {s, a} \left(V - V ^ {\mathrm {R}}\right) + \widehat {P V ^ {\mathrm {R}}} (s, a)\right) + b ^ {\mathrm {R}}\right). \tag {7}
$$
This idea of (7) is used in subroutine update-q-reference() of our algorithm. Let us discuss it in more details:
- Given a transition sample  $(s, a, s')$ , we can take  $V(s') - V^{\mathrm{R}}(s')$  as an unbiased estimate of the advantage  $P_{s,a}(V - V^{\mathrm{R}})$ . The magnitude of  $V - V^{\mathrm{R}}$  is small when  $V$  and  $V^{\mathrm{R}}$  are close. This engenders smaller stochastic volatility, compared to  $\widehat{P}_{s,a}V$  in (6) from UCB-Q.
- The reference estimate  $\widehat{PV^{\mathrm{R}}}$  is a stochastic approximation of  $PV^{\mathrm{R}}$ . In our algorithm, the auxiliary estimate  $\mu^{\mathrm{ref}}$  (cf. Line 13 of Algorithm 2) is used as the estimate for  $PV^{\mathrm{R}}$ . Specifically,  $\mu^{\mathrm{ref}}(s,a)$  is a running mean of  $P_{s,a}V^{\mathrm{R}}$  based on the samples from all past visitations of  $(s,a)$ . In contrast to the advantage, which is computed every time a new sample arrives, the reference is computed with a batch of samples and thus more stable. In sacrifice, the reference is only updated intermittently and not as up-to-date as the advantage.
The exploration bonus  $b^{\mathrm{R}}$  is computed from the auxiliary estimates in Line 8 and 9 to serve as an upper confidence bound on the aggregation of the aforementioned reference and advantage. Specifically, for each  $(s,a)$ ,  $\mu^{\mathrm{ref}}(s,a)$  and  $\sigma^{\mathrm{ref}}(s,a)$  are the running mean and 2nd moment of the reference  $[PV^{\mathrm{R}}](s,a)$  respectively;  $\mu^{\mathrm{adv}}(s,a)$  and  $\sigma^{\mathrm{adv}}(s,a)$  are the running mean and 2nd moment of the advantage  $[P(V - V^{\mathrm{R}})](s,a)$  respectively;  $B^{\mathrm{R}}(s,a)$  combines the empirical standard deviations of the reference and the advantage;  $\delta^{\mathrm{R}}(s,a)$  is the temporal difference between  $B^{\mathrm{R}}(s,a)$  and its previous value.  $b^{\mathrm{R}}(s,a)$  can be computed from these estimates as a temporally-weighted average of  $B^{\mathrm{R}}(s,a)$ . Thanks to the low variability of the reference  $PV^{\mathrm{R}}$ , we can obtain a more accurate, milder overestimation in the upper confidence bound for faster overall convergence.
3.2 Review: Early settlement of reference value
Given the optimistic overestimates  $Q^{\mathrm{UCB}}$  and  $Q^{\mathrm{R}}$ , it is natural to design an update rule of our Q-function estimate as the minimum of the estimate itself and these two overestimates (Line 9 of Algorithm 1). This makes our Q-function estimate monotonically decrease without violating the optimistic principle  $Q \geq Q^{\star}$ , which effectively enables us to lessen the overestimation in  $Q$  over time until it converges to  $Q^{\star}$ . In fact, this is precisely the update rule in UCB-Q-Advantage [48]. Nevertheless, we need to equip our algorithm with additional features to strive for regret optimality.
[23] introduced a way to update the reference with higher sample efficiency in the finite-horizon setting. As they noted, it is critical to update the reference  $V^{\mathrm{R}}$  in a smart fashion so as to balance the tradeoff between its synchronization with  $V$  and the volatility that results from too many stochastic updates. Concretely, the reference  $V^{\mathrm{R}}$  needs to be updated in a timely manner from  $V$  so that the magnitude of  $\widehat{P}_{s,a}(V - V^{\mathrm{R}})$  can be kept low as desired, but the updates cannot be too frequent either, because the stochasticity or variance in  $\widehat{PV^{\mathrm{R}}}(s,a)$  could be as high as that in  $\widehat{P}_{s,a}V$  of (6) and thus lead to suboptimality if it is not carefully controlled. To resolve this dilemma, we can update  $V^{\mathrm{R}}$  until it becomes sufficiently close to  $V^{\star}$  and fix its value thereafter.
To this end, we maintain a "pessimistic" underestimate  $Q^{\mathrm{LCB}}$  (resp.  $V^{\mathrm{LCB}}$ ) of  $Q^{\star}$  (resp.  $V^{\star}$ ) in the algorithm, which are computed from the lower confidence bound for  $Q^{\star}$  (resp.  $V^{\star}$ ). This can provide us with an upper bound on  $V^{\mathrm{R}} - V^{\star}$ , which will be used to determine when the update of the reference  $V^{\mathrm{R}}$  should be stopped.
In particular, the if-else block in Line 23 to 26 is designed to keep the reference  $V^{\mathrm{R}}$  synchronized with  $V$  for each state  $s$  respectively and terminate the update once
$$
V (s) \leq V ^ {\mathrm {L C B}} + 3 \leq V ^ {\star} + 3. \tag {8}
$$
This can guarantee  $|V - V^{\mathrm{R}}| \leq 6$  throughout the execution of the algorithm. As a result, the standard deviation of  $\widehat{P}_{s,a}(V - V^{\mathrm{R}})$  is guaranteed to be  $O(1)$ , which can be  $O\left(\frac{1}{1 - \gamma}\right)$  times smaller than the standard deviation of  $\widehat{P}_{s,a}V$  in (2). This can lead to smaller  $\frac{1}{1 - \gamma}$  factor in the final regret guarantee.
3.3 Adaptive low-switching greedy policy
Although these aforementioned designs from the finite-horizon literature help increase the accuracy of our estimate  $Q$ , they are still insufficient to attain regret optimality in the infinite-horizon setting. Since data collection takes place over a single trajectory with no reset, drastic changes in the execution policy can inflict a long-lasting volatility on the future trajectory and slow down the convergence. This is precisely the difficulty of the infinite-horizon setting over the finite-horizon one. The need to control the trajectory variability motivates us to design a novel adaptive switching technique.
Recall in UCB-Q and UCB-Q-Avantage, the execution policy is greedy with respect to the estimate  $Q$ , i.e.,  $\pi_t(s_t) = \arg \max_a Q(s_t, a)$ . Every time  $Q$  gets updated, what the algorithm effectively does is to make an estimate of  $Q^{\pi_t}$  with the samples generated by  $\pi_t$ . Such  $Q^{\pi_t}$  is only estimated and updated once before the execution policy is switched to  $\pi_{t+1}$ . This seems insufficient from a stochastic fixed-point iteration perspective, so we seek to update it more and learn each  $Q^{\pi_t}$  better before switching to a new policy.
To tackle this issue, we make the execution policy  $\pi_t$  greedy to  $Q^{\mathrm{lazy}}$ , which is updated lazily yet adaptively in Q-SlowSwitch-Adv. Specifically, for every  $(s,a)$ , we use  $\theta(s,a)$  (cf. Line 12 in Algorithm 2) to keep track of the cumulative difference between the current  $Q(s,a)$  and  $Q_M(s,a)$ , the latter of which is defined to be the value of  $Q(s,a)$  last time  $Q^{\mathrm{lazy}}$  is updated immediately after visiting  $(s,a)$ . Whenever  $\theta(s,a)$  exceeds  $\frac{1}{1 - \gamma}$ , indicating  $Q^{\mathrm{lazy}}(s,a)$  and the execution policy has become outdated with respect to the current  $Q(s,a)$ , we reset  $\theta(s,a)$  and set  $u^{\mathrm{switch}}$  to True, which will direct the algorithm to update the entire function  $Q^{\mathrm{lazy}}$  in the following step. update-q-lazy() updates  $Q^{\mathrm{lazy}}$  with the samples from  $\mathcal{D}$ , which is a dictionary that serves as a buffer to store all the new sample transitions and their latest estimates since the last update of  $Q^{\mathrm{lazy}}$ .
In contrast, conventional low-switching algorithms update the execution policy on a predetermined, exponentially phased schedule [5; 48]. While trajectory stability is attained with these algorithms, as time goes on, it takes them exponentially longer to update policy, making them oblivious to recent large updates in the estimated Q-function. This would lead to suboptimal regret in the infinite-horizon setting, as continual choices of suboptimal actions will keep a lasting effect on future trajectory in the absence of trajectory reset. This issue is overcome in our algorithm by ignoring minor changes in function  $Q$  yet still being adaptive to substantial changes in any state-action pair.
Our model-free algorithm Q-SlowSwitch-Adv can achieve optimal regret with short burn-in time. Its theoretical guarantee can be summarized in the following theorem.
Theorem 1. Choose any  $\delta \in (0,1)$ . Suppose  $\iota = \log \frac{SAT}{\delta}$  and  $c_{b}$  is chosen to be a sufficiently large universal constant in Algorithm 1. Then with probability at least  $1 - \delta$ , Algorithm 1 achieves
$$
\operatorname {R e g r e t} (T) \leq C _ {0} \left(\sqrt {\frac {S A T \iota^ {3}}{(1 - \gamma) ^ {3}}} + \frac {S A \iota^ {\frac {7}{2}}}{(1 - \gamma) ^ {8}}\right) \tag {9}
$$
for an absolute constant  $C_0 > 0$
To prove this theorem, we need to use a recursive error decomposition scheme different from the existing work. The stationary nature of the infinite-horizon setting gives rise to several error terms unique to the infinite-horizon setting, and our novel switching technique is crucial at controlling them optimally. The proof is provided in the full version [14]. Now let us highlight a few key properties of our algorithm.
Optimal regret with low burn-in. Q-SlowSwitch-Adv achieves optimal regret modulo some logarithmic factor as soon as the sample size  $T$  exceeds
$$
T \geq \frac {S A}{\operatorname {p o l y} (1 - \gamma)}. \tag {10}
$$
This burn-in threshold is significantly lower than the  $\frac{S^3A^2}{\mathrm{poly}(1 - \gamma)}$  threshold in [12] when  $SA \gg \frac{1}{1 - \gamma}$ . In other words, in the regime of (10), the regret of Q-SlowSwitch-Adv is guaranteed to satisfy
$$
\operatorname {R e g r e t} (T) \leq \widetilde {O} \left(\sqrt {\frac {S A T}{(1 - \gamma) ^ {3}}}\right), \tag {11}
$$
which matches the lower bound in Table 1.
Sample complexity. As a corollary of Theorem 1, it can be seen that Q-SlowSwitch-Adv attains  $\epsilon$ -average regret (i.e.  $\frac{1}{T}\mathrm{Regret}(T)\leq \epsilon$  for any fixed  $T$ ) with sample complexity
$$
\widetilde {O} \left(\frac {S A}{(1 - \gamma) ^ {3} \epsilon^ {2}}\right). \tag {12}
$$
This is lower than the sample complexity of the model-free algorithm in [24], which is  $\widetilde{O}\left(\frac{SA}{(1 - \gamma)^5\epsilon^2}\right)$ . Moreover, (12) holds true for any desired accuracy  $\epsilon \in \left(0,\frac{(1 - \gamma)^{13}}{SA}\right]$ . This is a broader range than the ones in [12; 49], which involve higher order of  $S$  and  $A$  and only allow their algorithms to attain their respective optimal sample complexity in the high-precision regime.
Space complexity. Q-SlowSwitch-Adv is a model-free algorithm that keeps a few estimates of the Q-function during execution, so its memory cost is as low as  $O(SA)$ . This is not improvable in the tabular setting, since it requires  $O(SA)$  units of space to store the optimal policy. In contrast, the model-based UCBVI- $\gamma$  in [12] stores an estimate of the probability transition kernel and thus incurs a higher memory cost of  $O(S^2 A)$ .
Computational complexity. The computational cost of Q-SlowSwitch-Adv is only  $O(T)$ . This is on the same order as reading samples along the  $T$ -length executed trajectory and is thus unimprovable. In comparison, our algorithm has a considerably lower computational cost than the one in [12], which requires  $O(S^2 AT)$  operations overall.
This work has introduced a model-free algorithm that achieves optimal regret in infinite-horizon discounted MDPs, which reduces the space and computational complexity requirement for regret optimality in the existing work. It also achieves optimal sample efficiency with a short burn-in time compared to other algorithms, including [12; 49]. Moreover, our algorithm has demonstrated the importance of switching policies slowly in infinite-horizon MDPs and introduced a novel technique might be of additional interest to future work. While our burn-in threshold is considerably reduced with respect to the order of  $S$  and  $A$ , it still has nontrivial suboptimality in the effective horizon  $\frac{1}{1 - \gamma}$ , which is a price we pay for using the reference-advantage technique. It is open for future work to investigate how to further improve the effective horizon factors in the burn-in cost.
The authors are very grateful to Yuxin Chen for helpful advice as well as suggesting the direction. X. Ji was supported in part by the NSF grants 1907661 and 2014279.
[1] Alekh Agarwal, Sham Kakade, and Lin F Yang. Model-based reinforcement learning with a generative model is minimax optimal. In Conference on Learning Theory, pages 67-83. PMLR, 2020.
[38] Mengdi Wang. Randomized linear programming solves the discounted markov decision problem in nearly-linear (sometimes sublinear) running time. arXiv preprint arXiv:1704.01869, 2017.
[39] Tianhao Wang, Dongruo Zhou, and Quanquan Gu. Provably efficient reinforcement learning with linear function approximation under adaptivity constraints. Advances in Neural Information Processing Systems, 34:13524-13536, 2021.
[40] C. J. C. H. Watkins. Learning from Delayed Rewards. PhD thesis, King's College, Oxford, 1989.
[41] Christopher J. C. H. Watkins and Peter Dayan. Q-learning. Machine Learning, 8(3):279-292, 1992. doi: 10.1007/BF00992698.
[42] Lin Xiao and Tong Zhang. A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization, 24(4):2057-2075, 2014.
[43] Tengyu Xu, Zhe Wang, Yi Zhou, and Yingbin Liang. Reanalysis of variance reduced temporal difference learning. arXiv preprint arXiv:2001.01898, 2020.
[44] Yuling Yan, Gen Li, Yuxin Chen, and Jianqing Fan. The efficacy of pessimism in asynchronous q-learning. arXiv preprint arXiv:2203.07368, 2022.
[45] Kunhe Yang, Lin Yang, and Simon Du. Q-learning with logarithmic regret. In International Conference on Artificial Intelligence and Statistics, pages 1576–1584. PMLR, 2021.
[46] Ming Yin, Yu Bai, and Yu-Xiang Wang. Near-optimal offline reinforcement learning via double variance reduction. Advances in neural information processing systems, 34:7677-7688, 2021.
[47] Junyu Zhang, Chengzhuo Ni, Csaba Szepesvari, Mengdi Wang, et al. On the convergence and sample efficiency of variance-reduced policy gradient method. Advances in Neural Information Processing Systems, 34:2228-2240, 2021.
[48] Zihan Zhang, Yuan Zhou, and Xiangyang Ji. Almost optimal model-free reinforcement learning via reference-advantage decomposition. Advances in Neural Information Processing Systems, 33:15198–15207, 2020.
[49] Zihan Zhang, Yuan Zhou, and Xiangyang Ji. Model-free reinforcement learning: from clipped pseudo-regret to sample complexity. In International Conference on Machine Learning, pages 12653-12662. PMLR, 2021.
[50] Zihan Zhang, Yuhang Jiang, Yuan Zhou, and Xiangyang Ji. Near-optimal regret bounds for multi-batch reinforcement learning. arXiv preprint arXiv:2210.08238, 2022.
[51] Dongruo Zhou, Jiafan He, and Quanquan Gu. Provably efficient reinforcement learning for discounted mdps with feature mapping. In International Conference on Machine Learning, pages 12793-12802. PMLR, 2021.
In this section, we take a moment to discuss the related work beyond those in Table 1.
Regret analysis for online episodic RL In the online episodic setting, regret is the predominant choice of metric for demonstrating the sample efficiency of a method [45; 28; 13]. [4] was the first to introduce a model-based method that can achieve near-optimal regret guarantee, but the model-based nature of their method induces a high space complexity and burn-in time. On the other hand, model-free methods are proposed in [15; 5], which are motivated by Q-learning and thus enjoy a low space complexity. However, these methods cannot guarantee optimal regret. It was not until [48] that proposed the first model-free method with optimal regret guarantee UCB-Q-Advantage, but it incurs a large burn-in time of  $S^6 A^4 H^{28}$ , where  $H$  is the horizon of the episodic MDP. In addition, [26] proposed UCB-M-Q, a Q-learning variant with momentum, which can achieve optimal regret with low burn-in time, but it requires the storage of all momentum bias and thus incurs high memory cost. Recently, [23] propose a Q-learning variant with variance reduction that achieves optimal regret with  $O(SAH)$  space complexity and  $SApoly(H)$  burn-in threshold at the same time. Table 1 in [23] provides a more detailed comparison of related work from the online episodic RL literature.
Sample complexity for infinite-horizon RL In the infinite-horizon setting, there exist other sample efficiency metrics besides sample complexity of exploration. Initially, [19] considered the sample complexity needed to find an  $\epsilon$ -approximate optimal policy. The same definition is also considered in [38; 34; 33; 21]. Later, [37] studied the sample complexity needed to find an  $\epsilon$ -approximate optimal Q-function. Note that all of these works assume the generative setting. Indeed, a limitation of these aforementioned sample complexity definitions is that they only measure the performance of the final output policy and do not reflect the online regret during learning. Thus, existing works that study the online setting consider sample complexity of exploration and cumulative regret instead.
Variance reduction in RL The idea of variance reduction was first introduced to accelerate stochastic finite-sum optimization by [16], which is followed by a rich literature [42; 27; 11]. Later, for better sample efficiency in RL, it is applied to policy gradient methods [25; 47; 29] as well as value-based methods in various problems including generative setting RL [33; 34; 37], policy evaluation [9; 43], asynchronous Q-learning [22; 44] and offline RL [46; 32].
Low-switching algorithms Since our algorithm includes a novel feature that switches the execution policy slowly, we make a review of the low-switching approaches in RL. The idea of changing the execution policy infrequently during learning was first introduced by [3] as an approach to minimize regret in the multi-armed bandit problem. [5] adapted this idea to tabular RL and formalized the switching cost as a secondary metric that an algorithm can minimize. To reduce the number of policy switches and thus the switching cost, their algorithm updates the policy in geometrically longer intervals. Similar techniques can be found in [48], whose algorithm can achieve regret optimality while maintaining low switching cost. Later, [10; 39] introduced a new low-switching approach in linear MDPs by switching policies when the estimated covariance matrix gets a significant update. All these methods guarantee a  $O(\log T)$  switching cost. The switching cost guarantee was later improved to  $O(\log \log T)$  by the algorithms proposed in [31; 50].
B A Discussion about Different Regret Metrics
As we have discussed earlier, multiple metrics exist in the existing literature for evaluating the online performance of an RL algorithm in the infinite-horizon setting. In fact, [12] and [24] both use a respectively different regret metric from the one in this work and [18]. An argument showing the equivalence between the regret metric in [24] and the one in [12] is discussed in Appendix A.2 of [12]. For this reason and the fact that the regret guarantee in [24] is not minimax-optimal, we focus on the relation between our theoretical guarantee in Theorem 1 and the regret metric from [12] in this section.
Recall that the goal of RL is to learn the optimal policy through online interactions with the environment (and there always exists a stationary optimal policy), so the regret metric we consider in this
work is