SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xiang Ji *
Gen Li
A crucial problem in reinforcement learning is learning the optimal policy. We study this in tabular infinite-horizon discounted Markov decision processes under the online setting. The existing algorithms either fail to achieve regret optimality or have to incur a high memory and computational cost. In addition, existing optimal algorithms all require a long burn-in time in order to achieve optimal sample efficiency, i.e., their optimality is not guaranteed unless sample size surpasses a high threshold. We address both open problems by introducing a model-free algorithm that employs variance reduction and a novel technique that switches the execution policy in a slow-yet-adaptive manner. This is the first regret-optimal model-free algorithm in the discounted setting, with the additional benefit of a low burn-in time.
In reinforcement learning (RL), a crucial task is to find the optimal policy that maximizes its expected cumulative reward in any given environment with unknown dynamics. An immense body of literature is dedicated to finding algorithms that solve this task with as few samples as possible, which is the prime goal under this task. Ideally, one hopes to find an algorithm with a theoretical guarantee of optimal sample efficiency. At the same time, this task might be accompanied with additional requirements such as low space complexity and computational cost, as it is common that the state and action spaces exhibit high dimensions in modern applications. The combination of these various goals and requirements presents an important yet challenging problem in algorithm design.
The task of searching for optimal policy has been well-studied by existing work in the generative setting [33; 34; 21; 1]. This fundamental setting allows the freedom of querying samples at any state-action pair. In contrast, it is more realistic but difficult to consider the same task in the online setting, in which samples can only be collected along trajectories generated from executing a policy in the unknown Markov decision process (MDP). Solving this task with optimal sample efficiency requires a careful balance between exploration and exploitation, especially when coupled with other goals such as memory and computational efficiency.
MDPs can be divided into two types: the episodic finite-horizon MDPs and the infinite-horizon MDPs. Although these two types of MDPs can be approached in similar ways under the generative setting, there is a clear dichotomy between them in the online setting. In an episodic MDP, sample trajectories are only defined in fixed-length episodes, so samples are collected in episodes, and a reset to an arbitrary initial state occurs at the end of every online episode. Its transition kernel is usually assumed to be non-stationary over time. In contrast, the transition kernel of an infinite-horizon MDP stays stationary over time, and the online sample collection process amounts to drawing a single infinitely long sample trajectory with no reset. These differences render most optimal algorithms for episodic
*Department of Electrical and Computer Engineering, Princeton University, Princeton, NJ 08544, USA.
$^{\dagger}$ Department of Statistics, The Chinese University of Hong Kong, Hong Kong, China.
MDPs suboptimal when applied to infinite-horizon MDPs. Without reset and non-stationarity, the high dependency between consecutive trajectory steps in the infinite-horizon setting presents a new challenge over the episodic setting. In this work, we consider the infinite-horizon discounted MDPs, which is widely used in practice but still has some fundamental questions unanswered in theory.
1.1 Sample Efficiency in Infinite-Horizon MDPs
To evaluate the sample efficiency of online RL algorithms, a natural and widely-accepted metric is the cumulative regret. It captures the performance difference between the optimal policy and the learned policy of an algorithm over its online interactions with a given MDP. The notion of cumulative regret was first introduced in the bandit literature and later adopted in the RL literature [2; 15]. It is profusely used in the online episodic RL literature. Such works aim to prove regret guarantees for algorithms and provide analyses that characterize such regret guarantees in terms of all problem parameters such as state space, action space and sample size in a non-asymptotic fashion. A cumulative regret guarantee can also suggest the sample complexity needed to reach a certain level of average regret.
In the online infinite-horizon setting, many works study a different metric called the sample complexity of exploration, first introduced in [17]. In essence, given a target accuracy level  $\epsilon$ , this metric characterizes the total number of  $\epsilon$ -suboptimal steps committed by an algorithm over an infinitely-long trajectory in the MDP. While this is indicative of the sample efficiency of an algorithm, the focus of this metric is very different from that of cumulative regret, as it only reflects the total number of failures but does not distinguish their sizes. As [24; 12] point out, even an optimal guarantee on the sample complexity of exploration can only be converted to a very suboptimal guarantee on the cumulative regret. To obtain a more quantitative characterization of the total volume of failures in the regime of finite samples, some works have turned to studying cumulative regret guarantees for algorithms.
It was not until recently that some works [24; 51; 12; 18] begin to research into the problem of cumulative regret minimization in infinite-horizon discounted MDPs. Among them, [51] focus on linear MDPs while others study tabular MDPs. In this work, we study the regret minimization problem in the tabular case. Hereafter and throughout, we denote the size of the state space, the size of the action space and the discount factor of the problem MDP with  $S$ ,  $A$  and  $\gamma$ , respectively, and let  $T$  denote the sample size.
1.2 Model-Based and Model-Free Methods
Since modern RL applications are often large-scale, algorithms with low space complexity and computational complexity are much desired. This renders the distinction between model-based algorithms and model-free algorithms particularly important. The procedure of a model-based method includes a model estimation stage that involves estimating the transition kernel and a subsequent planning stage that searches the optimal policy in the learned model. Thus,  $O(S^2 A)$  space is required to store the estimated model. This is unfavorable when the state space is large and a memory constraint is present. Additionally, updating the transition kernel estimate brings a large computational burden. In comparison, model-free methods do not learn the entire model and thus can run with  $o(S^2 A)$  space. Notably, most value-based methods such as Q-learning only require storage of an estimated Q-function, which can take as little as  $O(SA)$  memory. In the infinite-horizon discounted setting, although UCBVI-γ in [12] can achieve optimal regret, its model-based nature exacts a  $O(S^2 A)$  memory and computational cost; conversely, the algorithms in [24] and [18] are model-free but have suboptimal regret guarantee.
1.3 Burn-in Cost in Regret-Optimal RL
Naturally, one aims to develop algorithms that find the optimal policy with the fewest number of samples. In regards to regret, this motivates numerous works to work towards algorithms with minimax-optimal cumulative regret. However, the job is not done once such an algorithm is found. As can be seen in the episodic RL literature, algorithms that achieve optimal regret as sample size  $T$  tends towards infinity can still have different performance in the regime when  $T$  is limited. Specifically, for every existing algorithm, there exists a certain sample size threshold such that regret is suboptimal before  $T$  exceeds it. Such threshold is commonly referred to as the initial burn-in time of the algorithm. Therefore, it is of great interest to find an algorithm with low burn-in time so that it
Algorithm	Sample complexity of exploration	Cumulative Regret	Range of T with optimal regret	Space complexity
Delayed Q-learning [35]	SA/(1-γ)8ε4	S1/5A1/5T4/5(1-γ)9/5	never	SA
R-Max [7]	SA2/(1-γ)6ε3	S1/2A1/4T3/4(1-γ)7/4	never	S2A
UCB-Q [8]	SA/(1-γ)7ε2	S1/3A1/3T2/3(1-γ)8/3	never	SA
MoRmax [36]	SA/(1-γ)6ε2	S1/3A1/3T2/3(1-γ)7/3	never	S2A
UCRL [20]	SA2/(1-γ)3ε2	S2/3A1/3T2/3(1-γ)4/3	never	S2A
UCB-multistage [49]	SA/(1-γ)11/2ε2	S1/3A1/3T2/3(1-γ)13/6	never	SA
UCB-multistage-adv1 [49]	SA/(1-γ)3ε2	S1/3A1/3T2/3(1-γ)4/3	never	SA
MAIN2 [18]	N/A	K√(S4+S2A2/T(1-γ)8)	never	SA
Double Q-learning [24]	N/A	√SAT/(1-γ)5	never	SA
UCBVI-γ3 [12]	N/A	√SAT/(1-γ)3	[S3A2/(1-γ)4,∞)	S2A
Q-SlowSwitch-Adv (This work)	N/A	√SAT/(1-γ)3	[SA/(1-γ)13,∞)	SA
Lower bound [20]; [12]	SA/(1-γ)3ε2	√SAT/(1-γ)3	N/A	N/A
Table 1: A comparison between our results and existing work in the online infinite-horizon discounted setting. Logarithmic factors are omitted for clearer presentation. The second column shows the sample complexity when the target accuracy  $\epsilon$  is sufficiently small. The third column shows the regret when sample size  $T$  is sufficiently large (beyond the burn-in period). The algorithms in the first seven rows only have sample complexity results in their original works; their regret bounds are derived from their respective sample complexity bounds and presented in this table for completeness. Details about the conversions can be found in [12]. The fourth column lists the sample size range in which regret optimality can be attained, which shows the burn-in time. We would like to point out that the results in [12; 24] are under slightly different regret definitions from the regret definition in [18] and this work. In fact, their regret metric can be more lenient, and our algorithm can also achieve  $\widetilde{O} \left( \sqrt{\frac{SAT}{(1 - \gamma)^3}} \right)$  optimal regret under it. This is further discussed in Remark 1 and Appendix B.
can still attain optimal regret in the sample-starved regime. Such effort has been made by [21; 1] in the generative setting and by [23; 26] in the online episodic setting. Yet, this important issue has not been addressed in the infinite-horizon setting, as optimal algorithms all suffer long burn-in times.
Specifically, while UCBVI-  $\gamma$  in [12] achieves a state-of-the-art regret guarantee of  $\widetilde{O}\left(\sqrt{\frac{SAT}{(1 - \gamma)^3}}\right)$ , which they prove minimax-optimal, their theory does not guarantee optimality unless the samples size  $T$  becomes as large as
$$
T \geq \frac {S ^ {3} A ^ {2}}{(1 - \gamma) ^ {4}}.
$$
${}^{1}$  UCB-multistage-adv achieves optimal sample complexity only in the high-accuracy regime when  $\epsilon  \leq$ ${S}^{-2}{A}^{-2}{\left( 1 - \gamma \right) }^{14}$  . This is similar to a burn-in threshold in that the optimal guarantee cannot be achieved unless in a specific range.
2The regret analysis for MAIN assumes an ergodicity parameter  $\kappa$ .
${}^{3}$  UCBVI-  $\gamma$  achieves optimal regret for  $T \geq  \frac{{S}^{3}{A}^{2}}{{\left( 1 - \gamma \right) }^{4}}$  only if the MDP satisfies  ${SA} \geq  \frac{1}{1 - \gamma }$  .
This threshold can be prohibitively large when  $S$  and  $A$  are huge, which is true in most applications. Thus, this makes reducing these factors in the burn-in cost particularly important. For instance, a 5-by-5 tic-tac-toe has a state space of size  $3^{25}$ . While this is a manageable number in modern machine learning, any higher power of it may cause computational difficulties; in contrast, the horizon of this game is much smaller—no more than 25. Since no lower bound precludes regret optimality for  $T \geq \frac{SA}{(1 - \gamma)^4}$ , one might hope to design an algorithm with smaller  $S$  and  $A$  factors in the burn-in cost so that it can achieve optimality even in the sample-starved regime.
1.4 Summary of Contributions
While it is encouraging to see recent works have shown that in the discounted setting, model-free methods can provide nearly optimal guarantees on sample complexity of exploration and that model-based methods can provide nearly optimal finite-sample regret guarantees, there still lacks a model-free approach that can attain regret optimality. In the orthogonal direction, there is still a vacancy for algorithms that can attain optimal regret for a broader sample size range, i.e., with fewer samples than  $\frac{S^3A^2}{\mathrm{poly}(1 - \gamma)}$ .
In fact, we can summarize these two lingering theoretical questions as follows:
Is there an algorithm that can achieve minimax regret optimality with low space complexity and computational complexity in the infinite-horizon discounted setting, even when sample size is limited?
We answer this question affirmatively with a new algorithm Q-SlowSwitch-Adv, which uses variance reduction and a novel adaptive switching technique. It is the first model-free algorithm that achieves optimal regret in the infinite-horizon discounted setting. This result can be summarized as follows:
Theorem (informal). For any sample size  $T \geq \frac{SA}{\mathrm{poly}(1 - \gamma)}$ , Q-SlowSwitch-Adv is guaranteed to achieve near-optimal cumulative regret  $\widetilde{O}\left(\sqrt{\frac{SAT}{(1 - \gamma)^3}}\right)$  with space complexity  $O(SA)$  and computational complexity  $O(T)$ .
A formal theorem is presented in Section 4; its proof can be found in the full version [14]. We also provide a complete summary of related prior results in Table 1. A discussion about the additional related work is deferred to Appendix A.
Let us specify the problem we aim to study in this section. Throughout this paper, we let  $\Delta(\mathcal{X})$  denote the probability simplex over any set  $\mathcal{X}$ . We also introduce the notation  $[m] := \{1, 2, \dots, m\}$  for a positive integer  $m$ .
2.1 Infinite-Horizon Discounted Markov Decision Process
We consider an infinite-horizon discounted Markov decision process (MDP) represented with  $(S, \mathcal{A}, \gamma, P, r)$ . Notably, we consider a tabular one, in which  $S := \{1, 2, \dots, S\}$  denotes the state space with size  $S$  and  $\mathcal{A} := \{1, 2, \dots, A\}$  denotes the action space with size  $A$ .  $P: S \times \mathcal{A} \to \Delta(S)$  denotes the probability transition kernel in that  $P(\cdot | s, a) \in \Delta(S)$  is the transition probability vector from state  $s \in S$  when action  $a \in \mathcal{A}$  is taken.  $r: S \times \mathcal{A} \to [0, 1]$  denotes the reward function, which is assumed to be deterministic in this work. Specifically,  $r(s, a)$  is the immediate reward for taking action  $a \in \mathcal{A}$  at state  $s \in S$ . Lastly,  $\gamma$  denotes the discount factor for the reward, which makes  $\frac{1}{1 - \gamma}$  the effective horizon.
A (stationary) policy  $\pi : S \to \Delta(\mathcal{A})$  specifies a rule for action selection in that  $\pi(\cdot|s) \in \Delta(\mathcal{A})$  is the action selection probability vector at state  $s \in S$ . We overload this notation by letting  $\pi(s)$  denote the action policy  $\pi$  takes at state  $s$ . Given a policy  $\pi$ , the Q-function of  $\pi$  is defined as
$$
Q ^ {\pi} (s, a) := \mathbb {E} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} r \left(s _ {t}, a _ {t}\right) \mid s _ {0} = s, a _ {0} = a \right],
$$
in which  $s_{t+1} \sim P(\cdot | s_t, a_t)$  for  $t \geq 0$  and  $a_t \sim \pi(\cdot | s_t)$  for  $t \geq 1$ . Moreover, the value function of  $\pi$  is defined as
$$
V ^ {\pi} (s) := \mathbb {E} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} r (s _ {t}, a _ {t}) \Big | s _ {0} = s \right],
$$
in which  $s_{t + 1} \sim P(\cdot |s_t, a_t)$  and  $a_{t} \sim \pi (\cdot |s_{t})$  for  $t \geq 0$ . The Q-function and value function satisfy an equation, called the Bellman equation [6]:
$$
Q ^ {\pi} (s, a) = r (s, a) + \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} \left[ V ^ {\pi} \left(s ^ {\prime}\right) \right]. \tag {1}
$$
A policy  $\pi^{\star}$  is called an optimal policy if it maximizes the value function for all states simultaneously. The optimal value function and optimal Q-function can be defined as
$$
V ^ {\star} (s) := \max  _ {\pi} V ^ {\pi} (s) = V ^ {\pi^ {\star}} (s)
$$
$$
Q ^ {\star} (s, a) := \max  _ {\pi} Q ^ {\pi} (s, a) = Q ^ {\pi^ {\star}} (s, a),
$$
which satisfy
$$
V ^ {\star} (s) = V ^ {\pi^ {*}} (s) \quad \text {a n d} \quad Q ^ {\star} (s, a) = Q ^ {\pi^ {*}} (s, a)
$$
for any optimal policy  $\pi^{\star}$ . The optimal policy always exists and satisfies the Bellman optimality equation [30]:
$$
\begin{array}{l} Q ^ {\pi^ {\star}} (s, a) = r (s, a) + \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} \left[ \max  _ {a ^ {\prime} \in \mathcal {A}} Q ^ {\pi^ {\star}} \left(s ^ {\prime}, a ^ {\prime}\right) \right] \\ = r (s, a) + \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} \left[ V ^ {\star} \left(s ^ {\prime}\right) \right]. \tag {2} \\ \end{array}
$$
2.2 Online Learning in an Infinite-Horizon MDP
We consider the online (single-trajectory) setting, in which the agent is permitted to execute a total of  $T$  steps sequentially in the MDP. More specifically, the agent starts from an arbitrary (and possibly adversarial) initial state  $s_1$ . At each step  $t \in [T]$ , the agent at state  $s_t$  computes policy  $\pi_t$ , takes action  $a_t$  based on  $\pi_t(\cdot | s_t)$ , receives reward  $r(s_t, a_t)$ , and transitions to state  $s_{t+1}$  in the following step. At the end of execution, the agent generates a trajectory  $(s_1, a_1, r_1, s_2, a_2, r_2, \dots, s_T, a_T, r_T)$ , which amounts to  $T$  samples.
2.3 Problem: Regret Minimization
As a standard metric to evaluate the performance of the aforementioned agent over a finite number of  $T$  steps, the cumulative regret with respect to the sequence of stationary policies  $\{\pi_t\}_{t=1}^T$  learned by the algorithm is defined as follows:
$$
\operatorname {R e g r e t} (T) := \sum_ {t = 1} ^ {T} \left(V ^ {\star} \left(s _ {t}\right) - V ^ {\pi_ {t}} \left(s _ {t}\right)\right). \tag {3}
$$
Verbally, the regret measures the cumulative suboptimality between the optimal policy and the execution policy  $\pi_t$  at each step throughout the  $T$ -step online interaction process. Naturally, one aims to minimize this regret by finding an algorithm whose regret scales optimally in  $T$ . This would require a strategic balance between exploration and exploitation, which can be difficult when sample size  $T$  is small.
Remark 1. In the infinite-horizon setting, many prior works [12; 24] consider slightly different regret definitions with respect to non-stationary policies. Specifically, at each  $s_t$  along the trajectory, this different regret metric compares the optimal value function  $V^{\star}(s_t)$  against the expected cumulative reward of running the non-stationary policy  $\{\pi_k\}_{k = t}^{\infty}$  starting from  $s_t$ . By doing this, it is effectively evaluating the cumulative reward difference between the stationary optimal policy and a non-stationary algorithm. While there exists no formal conversion between the regret defined in this way and the one in (3), it is expected to be smaller and thus more easily controlled than (3), because the execution policy  $\pi_t$  improves over time. In addition, we can show our algorithm also achieves the same level of
Algorithm 1: Q-SlowSwitch-Adv
Initialize:  $\forall (s,a),Q^{\mathrm{azy}}(s,a),Q(s,a),Q^{\mathrm{UCB}}(s,a),Q^{\mathrm{R}}(s,a),Q_M(s,a)\gets \frac{1}{1 - \gamma};$ $N(s,a)\gets 0;V(s),V^{\mathrm{R}}(s)\gets \frac{1}{1 - \gamma};Q^{\mathrm{LCB}}(s,a),\theta (s,a)\gets 0;\mathcal{D}\gets \mathrm{dict}();$ $\mu^{\mathrm{ref}}(s,a),\sigma^{\mathrm{ref}}(s,a),\mu^{\mathrm{adv}}(s,a),\sigma^{\mathrm{adv}}(s,a),B^{\mathrm{R}}(s,a),\delta^{\mathrm{R}}(s,a)\gets 0;u^{\mathrm{switch}}\gets \mathrm{False};$ $u^{\mathrm{ref}}(s)\gets \mathrm{True};H = \lceil \frac{2}{1 - \gamma}\rceil ;\iota = \log \frac{SAT}{\delta}.$    
for  $t = 1,\dots ,T$  do Take action  $a_{t} = \pi_{t}(s_{t}) = \arg \max_{a}Q^{\mathrm{azy}}(s_{t},a)$  , and draw  $s_{t + 1}\sim P(\cdot |s_t,a_t)$ $N(s_{t},a_{t})\gets N(s_{t},a_{t}) + 1;n\gets N(s_{t},a_{t});\quad \#$  Update the counter  $\eta_{n}\leftarrow \frac{H + 1}{H + n};\quad \#$  Update the learning rate  $Q^{\mathrm{UCB}}(s_t,a_t)\gets$  update-q-ucb(); # Compute the UCB. See Algorithm 2  $Q^{\mathrm{LCB}}(s_t,a_t)\gets$  update-q-lcb(); #Compute the LCB. See Algorithm 2  $Q^{\mathrm{R}}(s_t,a_t)\gets$  update-q-reference(); #Compute the reference value. See Algorithm 2  $Q(s_{t},a_{t})\gets \min \{Q^{\mathrm{R}}(s_{t},a_{t}),Q^{\mathrm{UCB}}(s_{t},a_{t}),Q(s_{t},a_{t})\} ;$ $V(s_{t})\gets \max_{a}Q(s_{t},a);$ $V^{\mathrm{LCB}}(s_t)\gets \max \{\max_a Q^{\mathrm{LCB}}(s_t,a),V^{\mathrm{LCB}}(s_t)\} ;$ $\theta (s_t,a_t)\gets \theta (s_t,a_t) + Q_M(s_t,a_t) - Q(s_t,a_t).$  # Track the staleness of current policy if  $u^{\mathrm{switch}} = \mathrm{True}$  then  $Q^{\mathrm{azy}}\gets$  update-q-lazy(); #Update execution policy's Q-function (switch policy)  $\mathcal{D}\gets \mathrm{dict};$  #Reset the buffer  $u^{\mathrm{switch}}\gets$  False;  $\mathcal{D}[(s_t,a_t)]\gets Q(s_t,a_t).$  #Add the new transition and  $Q$  entry to the buffer ""Switch policy when the staleness tracker is large""if  $\theta (s_t,a_t) > \frac{1}{1 - \gamma}$  then  $Q_{M}(s_{t},a_{t})\gets Q(s_{t},a_{t});$  # Save the current  $Q$  entry for staleness determination later  $u^{\mathrm{switch}}\gets \mathrm{True};$  # Signal to switch policy at the following step  $\theta (s_t,a_t)\gets 0;\quad \#$  Reset the staleness tracker if  $V(s_{t}) - V^{\mathrm{LCB}}(s_{t}) > 3$  then  $V^{\mathrm{R}}(s_t)\gets V(s_t),u^{\mathrm{ref}}(s_t) = \mathrm{True};$  else if  $u^{\mathrm{ref}}(s_t) = \mathrm{True}$  then  $\begin{array}{rl}{\bigvee^{R}(s_{t})\leftarrow V(s_{t}),u^{\mathrm{ref}}(s_{t}) = \mathrm{True};} & {}\\ {\bigvee^{R}(s_{t})\leftarrow V(s_{t}),u^{\mathrm{ref}}(s_{t}) = \mathrm{False}.} & {}\\ {\bigvee^{R}(s_{t})\leftarrow V(s_{t}),u^{\mathrm{ref}}(s_{t}) = \mathrm{False}.} & {}\\ {\bigvee^{R}(s_{t})\leftarrow V(s_{t}),u^{\mathrm{ref}}(s_{t}) = \mathrm{False}.} & {}\\ {\bigvee^{R}(s_{t})\left\langle V(s_{t}),u^{\mathrm{ref}}(s_{t}) = \right\rangle .} & {}\\ {\bigvee^{R}(s_{t})\left\langle V(s_{t}),u^{\mathrm{ref}}(s_{t}) = \right\rangle .} & {}\\ {\bigvee^{R}(s_{t})\left\langle V(s_{t}),u^{\mathrm{ref}}(s_{t}) = \right\rangle .} & {}\\ {\bigvee^{R}(s_{t})}\end{array}$  # Update the reference only on certain conditions
regret under this different definition with an analysis specifically tailored to our algorithm, which is deferred to Appendix B. Furthermore, since the transition kernel in the infinite-horizon setting is invariant over time and the optimal policy itself is also stationary, it is more natural to compare the optimal policy to a stationary policy, e.g., the policy  $\pi_t$  deployed by the algorithm at each step, as in (3). Before this work, this has also been recently studied in [49; 18].
Notation. Given any vector  $x \in \mathbb{R}^{SA}$  that represents a function  $x: S \times \mathcal{A} \to \mathbb{R}$ , we use  $x(s, a)$  to denote the entry corresponding to the state-action pair  $(s, a)$ . We also denote the probability transition vector at  $(s, a)$  with
$$
P _ {s, a} = P (\cdot \mid s, a) \in \mathbb {R} ^ {1 \times S}, \tag {4}
$$
that is, given any  $V \in \mathbb{R}^S$ ,  $P_{s,a}V = \mathbb{E}_{s' \sim P(\cdot|s,a)}[V(s')]$ . For two vectors  $x, y \in \mathbb{R}^{SA}$ , we override the notation  $x \leq y$  to mean that  $x(s,a) \leq y(s,a)$  in every dimension  $(s,a)$ .
In this section, we present our algorithm Q-SlowSwitch-Adv and some relevant discussion.
Algorithm 2: Auxiliary functions
1 Function update-q-ucb(): 2  $\begin{array}{r}Q^{\mathrm{UCB}}(s_t,a_t)\gets (1 - \eta_n)Q^{\mathrm{UCB}}(s_t,a_t) + \eta_n\Big(r(s_t,a_t) + \gamma V(s_{t + 1}) + c_b\sqrt{\frac{\iota}{(1 - \gamma)^3n}}\Big). \end{array}$
3 Function update-q-lcb(): 4  $\begin{array}{r}Q^{\mathrm{LCB}}(s_t,a_t)\gets (1 - \eta_n)Q^{\mathrm{LCB}}(s_t,a_t) + \eta_n\Big(r(s_t,a_t) + \gamma V^{\mathrm{LCB}}(s_{t + 1}) - c_b\sqrt{\frac{\iota}{(1 - \gamma)^3n}}\Big). \end{array}$
5 Function update-q-lazy():