SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Daniil Tiapkin $^{1,2}$  Denis Belomestny $^{3,2}$  Daniele Calandriello $^{4}$  Éric Moulines $^{1,5}$   
Remi Munos $^{4}$  Alexey Naumov $^{2}$  Pierre Perrault $^{6}$  Michal Valko $^{4}$  Pierre Ménard $^{7}$ $^{1}$ CMAP, École Polytechnique  $^{2}$ HSE University  $^{3}$ Duisburg-Essen University  
 $^{4}$ Google DeepMind  $^{5}$ Mohamed Bin Zayed University of AI, UAE  $^{6}$ IDEMIA  $^{7}$ ENS Lyon {daniil.tiapkin,eric.moulines}@polytechnique.edu  
denis.belomestny@uni-due.de  
{dcalandriello,munos,valkom}@google.com anaumov@hse.ru  
pierre.perrault@outlook.com pierre.menard@ens-lyon.fr
In this paper, we introduce Randomized Q-learning (RandQL), a novel randomized model-free algorithm for regret minimization in episodic Markov Decision Processes (MDPs). To the best of our knowledge, RandQL is the first tractable model-free posterior sampling-based algorithm. We analyze the performance of RandQL in both tabular and non-tabular metric space settings. In tabular MDPs, RandQL achieves a regret bound of order  $\widetilde{\mathcal{O}} (\sqrt{H^5SAT})$ , where  $H$  is the planning horizon,  $S$  is the number of states,  $A$  is the number of actions, and  $T$  is the number of episodes. For a metric state-action space, RandQL enjoys a regret bound of order  $\widetilde{\mathcal{O}} (H^{5 / 2}T^{(d_z + 1) / (d_z + 2)})$ , where  $d_{z}$  denotes the zooming dimension. Notably, RandQL achieves optimistic exploration without using bonuses, relying instead on a novel idea of learning rate randomization. Our empirical study shows that RandQL outperforms existing approaches on baseline exploration environments.
In reinforcement learning (RL, Sutton and Barto 1998), an agent learns to interact with an unknown environment by acting, observing the next state, and receiving a reward. The agent's goal is to maximize the sum of the collected rewards. To achieve this, the agent can choose to use model-based or model-free algorithms. In model-based algorithms, the agent builds a model of the environment by inferring the reward function and the transition kernel that produces the next state. The agent then plans in this model to find the optimal policy. In contrast, model-free algorithms directly learn the optimal policy, which is the mapping of a state to an optimal action, or equivalently, the optimal Q-values, which are the mapping of a state-action pair to the expected return of an optimal policy starting by taking the given action at the given state.
Although empirical evidence suggests that model-based algorithms are more sample efficient than model-free algorithms [Deisenroth and Rasmussen, 2011, Schulman et al., 2015]; model-free approaches offer several advantages. These include smaller time and space complexity, the absence of a need to learn an explicit model, and often simpler algorithms. As a result, most of the recent breakthroughs in deep RL, such as those reported by Mnih et al. [2013], Schulman et al. [2015, 2017], Haarnoja et al. [2018], have been based on model-free algorithms, with a few notable exceptions, such as Schrittwieser et al. [2020], Hessel et al. [2021]. Many of these model-free algorithms [Mnih et al., 2013, Van Hasselt et al., 2016, Lillicrap et al., 2016] are rooted in the well-known Q-learning algorithm of Watkins and Dayan [1992]. Q-learning is an off-policy learning technique where the agent follows a behavioral policy while simultaneously incrementally learning the optimal Q-values
by combining asynchronous dynamic programming and stochastic approximation. Until recently, little was known about the sample complexity of Q-learning in the setting where the agent has no access to a simulator allowing to sample an arbitrary state-action pair. In this work, we consider such challenging setting where the environment is modelled by an episodic Markov Decision Process (MDP) of horizon  $H$ . After  $T$  episodes, the performance of an agent is measured through regret which is the difference between the cumulative reward the agent could have obtained by acting optimally and what the agent really obtained during the interaction with the MDP.
This framework poses the famous exploration-exploitation dilemma where the agent must balance the need to try new state-action pairs to learn an optimal policy against exploiting the current observations to collect the rewards. One effective approach to resolving this dilemma is to adopt the principle of optimism in the face of uncertainty. In finite MDPs, this principle has been successfully implemented in the model-based algorithm using bonuses [Jaksch et al., 2010, Azar et al., 2017, Fruit et al., 2018, Dann et al., 2017, Zanette and Brunskill, 2019]. Specifically, the upper confidence bounds (UCBs) on the optimal Q-value are built by adding bonuses and then used for planning. Building on this approach, Jin et al. [2018] proposed the OptQL algorithm, which applies a similar bonus-based technique to Q-learning, achieving efficient exploration. Recently, Zhang et al. [2020] introduced a simple modification to OptQL that achieves optimal sample complexity, making it competitive with model-based algorithms.
Another class of methods for optimistic exploration is Bayesian-based approaches. An iconic example among this class is the posterior sampling for reinforcement learning (PSRL,Strens 2000, Osband et al. 2013) algorithm. This model-based algorithm maintains a surrogate Bayesian model of the MDP, for instance, a Dirichlet posterior on the transition probability distribution if the rewards are known. At each episode, a new MDP is sampled (i.e., a transition probability for each state-action pair) according to the posterior distribution of the Bayesian model. Then, the agent plans in this sampled MDP and uses the resulting policy to interact with the environment. Notably, an optimistic variant of PSRL, named optimistic posterior sampling for reinforcement learning (OPSRL, Agrawal and Jia, 2017, Tiapkin et al., 2022a) also enjoys an optimal sample complexity [Tiapkin et al., 2022a]. The random least square value iteration (RLSVI, Osband et al. [2013]) is another well-known model-based algorithm that leverages a Bayesian-based technique for exploration. Precisely, RLSVI directly sets a Gaussian prior on the optimal Q-values and then updates the associated posterior trough value iteration in a model [Osband et al., 2013, Russo, 2019]. A close variant of RLSVI proposed by Xiong et al. [2022], using a more sophisticated prior/posterior couple, is also proven to be near-optimal.
It is noteworthy that Bayesian-based exploration techniques have shown superior empirical performance compared to bonus-based exploration, at least in the tabular setting [Osband et al., 2013, Osband and Van Roy, 2017]. Furthermore, these techniques have also been successfully applied to the deep RL setting [Osband et al., 2016, Azizzadenesheli et al., 2018, Fortunato et al., 2018, Li et al., 2022, Sasso et al., 2023]. Finally, Bayesian methods allow for the incorporation of apriori information into exploration (e.g. by giving more weight to important states). However, most of the theoretical studies on Bayesian-based exploration have focused on model-based algorithms, raising the natural question of whether the PSRL approach can be extended to a provably efficient model-free algorithm that matches the good empirical performance of its model-based counterparts. Recently, Dann et al. [2021] proposed a model-free posterior sampling algorithm for structured MDPs, however, it is not computationally tractable. Therefore, a provably tractable model-free posterior sampling algorithm has remained a challenge.
In this paper, we aim to resolve this challenge. We propose the randomized Q-learning (RandQL) algorithm that achieves exploration without bonuses, relying instead on a novel idea of learning rate randomization. RandQL is a tractable model-free algorithm that updates an ensemble of Q-values via Q-learning with Beta distributed step-sizes. If tuned appropriately, the noise introduced by the random learning rates is similar to the one obtained by sampling from the posterior of the PSRL algorithm. Thus, one can see the ensemble of Q-values as posterior samples from the same induced posterior on the optimal Q-values as in PSRL. Then, RandQL chooses among these samples in the same optimistic fashion as OPSRL. We prove that for tabular MDPs, a staged version [Zhang et al., 2020] of RandQL, named Staged-RandQL enjoys the same regret bound as the OptQL algorithm, that is,  $\widetilde{\mathcal{O}} (\sqrt{H^5SAT})$  where  $S$  is the number of states and  $A$  the number of actions. Furthermore, we extend Staged-RandQL beyond the tabular setting into the Net-Staged-RandQL algorithm to deal with metric state-action spaces [Domingues et al., 2021c, Sinclair et al., 2019]. Net-Staged-RandQL operates similarly to Staged-RandQL but over a fixed discretization of the state-action space and
uses a specific prior tuning to handle the effect of discretization. We prove that Net-Staged-RandQL enjoys a regret bound of order  $\widetilde{\mathcal{O}}(H^{5/2}T^{(d_c+1)/(d_c+2)})$ , where  $d_c$  denotes the covering dimension. This rate is of the same order as the one of Adaptive-QL by Sinclair et al. [2019, 2023], an adaptation of OptQL to metric state-action space and has a better dependence on the budget  $T$  than one of the model-based kernel algorithms such that Kernel-UCBVI by Domingues et al. [2021c]. We also explain how to adapt Net-Staged-RandQL and its analysis to work with an adaptive discretization as by Sinclair et al. [2019, 2023]. Finally, we provide preliminary experiments to illustrate the good performance of RandQL against several baselines in finite and continuous environments.
We highlight our main contributions:
- The RandQL algorithm, a new tractable (provably efficient) model-free Q-learning adaptation of the PSRL algorithm that explores through randomization of the learning rates.
- A regret bound of order  $\widetilde{\mathcal{O}}(\sqrt{H^5SAT})$  for a staged version of the RandQL algorithm in finite MDPs where  $S$  is the number of states and  $A$  the number of actions,  $H$  the horizon and  $T$  the budget.
- A regret bound of order  $\tilde{\mathcal{O}}(H^{5/2}T^{(d_c + 1) / (d_c + 2)})$  for an adaptation of RandQL to metric spaces where  $d_c$  denotes the covering dimension.
- Adaptive version of metric space extension of RandQL algorithm that achieves a regret bound of order  $\widetilde{\mathcal{O}}(H^{5/2}T^{(d_z + 1) / (d_z + 2)})$ , where  $d_z$  is a zooming dimension.
- Experiments in finite and continuous MDPs that show that RandQL is competitive with model-based and model-free baselines while keeping a low time-complexity.
We consider an episodic MDP  $(\mathcal{S},\mathcal{A},H,\{p_h\}_{h\in [H]},\{r_h\}_{h\in [H]})$ , where  $\mathcal{S}$  is the set of states,  $\mathcal{A}$  is the set of actions,  $H$  is the number of steps in one episode,  $p_h(s'|s,a)$  is the probability transition from state  $s$  to state  $s'$  upon taking action  $a$  at step  $h$ , and  $r_h(s,a)\in [0,1]$  is the bounded deterministic reward received after taking the action  $a$  in state  $s$  at step  $h$ . Note that we consider the general case of rewards and transition functions that are possibly non-stationary, i.e., that are allowed to depend on the decision step  $h$  in the episode.
Policy & value functions A deterministic policy  $\pi$  is a collection of functions  $\pi_h: S \to \mathcal{A}$  for all  $h \in [H]$ , where every  $\pi_h$  maps each state to a single action. The value functions of  $\pi$ , denoted by  $V_h^\pi$ , as well as the optimal value functions, denoted by  $V_h^\star$  are given by the Bellman and the optimal Bellman equations,
$$
Q _ {h} ^ {\pi} (s, a) = r _ {h} (s, a) + p _ {h} V _ {h + 1} ^ {\pi} (s, a) \quad V _ {h} ^ {\pi} (s) = \pi_ {h} Q _ {h} ^ {\pi} (s)
$$
$$
Q _ {h} ^ {\star} (s, a) = r _ {h} (s, a) + p _ {h} V _ {h + 1} ^ {\star} (s, a) \quad \quad \quad \quad \quad \quad V _ {h} ^ {\star} (s) = \max  _ {a} Q _ {h} ^ {\star} (s, a),
$$
where by definition,  $V_{H + 1}^{\star} \triangleq V_{H + 1}^{\pi} \triangleq 0$ . Furthermore,  $p_h f > (s,a) \triangleq \mathbb{E}_{s' \sim p_h(\cdot | s,a)}[f(s')]$  denotes the expectation operator with respect to the transition probabilities  $p_h$  and  $\pi_h g(s) \triangleq g(s,\pi_h(s))$  denotes the composition with the policy  $\pi$  at step  $h$ .
Learning problem The agent, to which the transitions are unknown (the rewards are assumed to be known<sup>1</sup> for simplicity), interacts with the environment during  $T$  episodes of length  $H$ , with a fixed initial state  $s_1$ . Before each episode  $t$  the agent selects a policy  $\pi^t$  based only on the past observed transitions up to episode  $t - 1$ . At each step  $h \in [H]$  in episode  $t$ , the agent observes a state  $s_h^t \in S$ , takes an action  $\pi_h^t(s_h^t) = a_h^t \in \mathcal{A}$  and makes a transition to a new state  $s_{h+1}^t$  according to the probability distribution  $p_h(s_h^t, a_h^t)$  and receives a deterministic reward  $r_h(s_h^t, a_h^t)$ .
Regret The quality of an agent is measured through its regret, that is the difference between what it could obtain (in expectation) by acting optimally and what it really gets,
$$
\mathfrak {R} ^ {T} \triangleq \sum_ {t = 1} ^ {T} V _ {1} ^ {\star} (s _ {1}) - V _ {1} ^ {\pi^ {t}} (s _ {1}).
$$
<sup>1</sup>Our work can be extended without too much difficulty to the case of random rewards.
As explained by Fiechter [1994] if the first state is sampled randomly as  $s_1 \sim p$ , we can simply add an artificial first state  $s_{1'}$  such that for any action  $a$ , the transition probability is defined as the distribution  $p_{1'}(s_{1'}, a) \triangleq p$ .
Additional notation For  $N \in \mathbb{N}_{++}$ , we define the set  $[N] \triangleq \{1, \dots, N\}$ . We denote the uniform distribution over this set by  $\mathrm{Unif}[N]$ . We define the beta distribution with parameters  $\alpha, \beta$  as  $\mathrm{Beta}(\alpha, \beta)$ . Appendix A references all the notation used.
3 Randomized Q-learning for Tabular Environments
In this section we assume that the state space  $S$  is finite of size  $S$  as well as the action space  $\mathcal{A}$  of size  $A$ . We first provide some intuitions for RandQL algorithm.
3.1 Concept
The main idea of RandQL is to perform the usual Q-learning updates but instead of adding bonuses to the targets as OptQL to drive exploration, RandQL injects noise into the updates of the Q-values through noisy learning rates. Precisely, for  $J \in \mathbb{N}$ , we maintain an ensemble of size  $J$  of Q-values $^3$  ( $\overline{Q}^{n,j}$ ) $_{j \in [J]}$  updated with random independent Beta-distributed step-sizes ( $w_{n,j}$ ) $_{j \in [J]}$  where  $w_{n,j} \sim \mathrm{Beta}(H,n)$ . Then, policy Q-values  $\overline{Q}^n$  are obtained by taking the maximum among the Q-values of the ensemble
$$
\begin{array}{l} \overline {{Q}} _ {h} ^ {n + 1, j} (s, a) = \left(1 - w _ {n, j}\right) \overline {{Q}} _ {h} ^ {n, j} (s, a) + w _ {n, j} \left[ r _ {h} (s, a) + \bar {V} _ {h + 1} ^ {n} \left(s _ {h + 1} ^ {n}\right) \right] \\ \overline {{Q}} _ {h} ^ {n + 1} (s, a) = \max _ {j \in [ J ]} \overline {{Q}} _ {h} ^ {n + 1, j} (s, a), \quad \overline {{V}} _ {h} ^ {n + 1} (s) = \max _ {a \in \mathcal {A}} \overline {{Q}} _ {h} ^ {n + 1} (s, a), \\ \end{array}
$$
where  $s_{h+1}^n$  stands for the next (in time) state after  $n$ -th visitation of  $(s, a)$  at step  $h$ .
Note that the policy Q-values  $\overline{Q}^n$  are designed to be upper confidence bound on the optimal Q-values. The policy used to interact with the environment is greedy with respect to the policy Q-values  $\pi_h^n (s)\in \arg \max_a\overline{Q}_h^n (s,a)$ . We provide a formal description of RandQL in Appendix B.
Connection with  $0\mathrm{pt}\mathsf{QL}$  We observe that the learning rates of RandQL are in expectation of the same order  $\mathbb{E}[w_{n,j}] = H / (n + H)$  as the ones used by the 0ptQL algorithm. Thus, we can view our randomized Q-learning as a noisy version of the 0ptQL algorithm that doesn't use bonuses.
Connection with PSRL If we unfold the recursive formula above we can express the Q-values  $\overline{Q}^{n + 1,j}$  as a weighted sum
$$
\overline {{Q}} _ {h} ^ {n + 1, j} (s, a) = W _ {n, j} ^ {0} \overline {{Q}} _ {h} ^ {1, j} (s, a) + \sum_ {k = 1} ^ {n} W _ {n, j} ^ {k} \left[ r _ {h} (s, a) + \overline {{V}} _ {h + 1} ^ {k} \left(s _ {h + 1} ^ {k}\right) \right],
$$
where we define  $W_{n,j}^{0} = \prod_{\ell = 0}^{n - 1}(1 - w_{\ell ,j})$  and  $W_{n,j}^{k} = w_{k - 1,j}\prod_{\ell = k}^{n - 1}(1 - w_{\ell ,j})$ .
To compare, we can unfold the corresponding formula for PSRL algorithm using the aggregation properties of the Dirichlet distribution (see e.g. Section 4 of Tiapkin et al. [2022b] or Appendix C)
$$
\bar {Q} _ {h} ^ {n + 1} (s, a) = \widetilde {W} _ {n} ^ {0} \bar {Q} _ {h} ^ {1} (s, a) + \sum_ {k = 1} ^ {n} \widetilde {W} _ {n} ^ {k} \left[ r _ {h} (s, a) + \bar {V} _ {h + 1} ^ {n + 1} \left(s _ {h + 1} ^ {k}\right) \right], \tag {1}
$$
where weights  $(\widetilde{W}_n^0,\dots ,\widetilde{W}_n^n)$  follows Dirichlet distribution  $\mathrm{Dir}(n_0,1,\ldots ,1)$  and  $n_0$  is a weight for the prior distribution. In particular, one can represent these weights as partial products of other weights  $w_{n}\sim \mathrm{Beta}(1,n + n_{0})$ . If we use (1) to construct a model-free algorithm, this would require recomputing the targets  $r_h(s,a) + \overline{V}^{n + 1}(s_{h + 1}^k)$  in each iteration. To make algorithm more efficient and model-free, we approximate  $\overline{V}^{n + 1}$  by  $\overline{V}^k$ , and, as a result, obtain RandQL algorithm with weight distribution  $w_{n,j}\sim \mathrm{Beta}(1,n + n_0)$ .
Note that in expectation this algorithm is equivalent to OptQL with the uniform step-sizes which are known to be sub-optimal due to a high bias (see discussion in Section 3 of [Jin et al., 2018]). There are two known ways to overcome this sub-optimality for Q-learning: to introduce more aggressive
<sup>3</sup>We index the quantities by  $n$  in this section where  $n$  is the number of times the state-action pair  $(s, a)$  is visited. In particular this is different from the global time  $t$  since, in our setting, all the state- action pair are not visited at each episode. See Section 3.2 and Appendix B precise notations.
learning rates  $w_{n,j} \sim \mathrm{Beta}(H, n + n_0)$  leading to RandQL algorithm, or to use stage-dependent framework by Bai et al. [2019], Zhang et al. [2020] resulting in Staged-RandQL algorithm.
The aforementioned transition from PSRL to RandQL is similar to the transition from UCBVI [Azar et al., 2017] to Q-learning. To make UCBVI model-free, one has to keep old targets in Q-values. This, however, introduces a bias that could be eliminated either by more aggressive step-size [Jin et al., 2018] or by splitting on stages [Bai et al., 2019]. Our algorithms (RandQL and Staged-RandQL) perform the similar tricks for PSRL and thus could be viewed as model-free versions of it. Additionally, RandQL shares some similarities with the OPSRL algorithm [Agrawal and Jia, 2017, Tiapkin et al., 2022a] in the way of introducing optimism (taking maximum over  $J$  independent ensembles of Q-values). Let us also mention a close connection to the theory of Dirichlet processes in the proof of optimism for the case of metric spaces (see Remark 1 in Appendix E.4).
Prior As remarked above, in expectation, RandQL has a learning rate of the same order as OptQL. In particular, it implies that the first  $(1 - 1 / H)$  fraction of the target will be forgotten exponentially fast in the estimation of the Q-values, see Jin et al. [2018], Menard et al. [2021]. Thus we need to re-inject prior targets, as explained in Appendix B, in order to not forget too quickly the prior and thus replicate the same exploration mechanism as in the PSRL algorithm.
3.2 Algorithm
In this section, following Bai et al. [2019], Zhang et al. [2020], we present the Staged-RandQL algorithm a scheduled version of RandQL that is simpler to analyse. The main idea is that instead of using a carefully tuned learning rate to keep only the last  $1 / H$  fraction of the targets we split the learning of the Q-values in stages of exponentially increasing size with growth rate of order  $1 + 1 / H$ . At a given stage, the estimate of the Q-value relies only on the targets within this stage and resets at the beginning of the next stage. Notice that the two procedures are almost equivalent. A detail description of Staged-RandQL is provided in Algorithm 1.
Counts and stages Let  $n_h^t(s, a) \triangleq \sum_{i=1}^{t-1} \mathbb{1}\{(s_h^i, a_h^i) = (s, a)\}$  be the number of visits of state-action pair  $(s, a)$  at step  $h$  before episode  $t$ . We say that a triple  $(s, a, h)$  belongs to the  $k$ -th stage at the beginning of episode  $t$  if  $n_h^t(s, a) \in [\sum_{i=0}^{k-1} e_i, \sum_{i=0}^k e_i)$ . Here  $e_k = \lfloor (1 + 1 / H)^k \cdot H \rfloor$  is the length of the stage  $k \geq 0$  and, by convention,  $e_{-1} = 0$ . Let  $\widetilde{n}_h^t(s, a) \triangleq n_h^t(s, a) - \sum_{i=0}^{k-1} e_i$  be the number of visits of state-action pair  $(s, a)$  at step  $h$  during the current stage  $k$ .
Temporary Q-values At the beginning of a stage, let say time  $t$ , we initialize  $J$  temporary Q-values as  $\widetilde{Q}_h^{t,j}(s,a) = r_h(s,a) + r_0(H - h - 1)$  for  $j \in [J]$  and  $r_0$  some pseudo-reward. Then as long as  $(s_h^t, a_h^t, h)$  remains within a stage we update recursively the temporary Q-values
$$
\widetilde {Q} _ {h} ^ {t + 1, j} (s, a) = \left\{ \begin{array}{l l} (1 - w _ {j, \widetilde {n}}) \widetilde {Q} _ {h} ^ {t, j} (s, a) + w _ {j, \widetilde {n}} [ r _ {h} (s, a) + \overline {{V}} _ {h + 1} ^ {t} (s _ {h + 1} ^ {t}) ], & (s, a) = (s _ {h} ^ {t}, a _ {h} ^ {t}) \\ \widetilde {Q} _ {h} ^ {t, j} (s, a) & \text {o t h e r w i s e ,} \end{array} \right.
$$
where  $\widetilde{n} = \widetilde{n}_h^t (s,a)$  is the number of visits,  $w_{j,\widetilde{n}}$  is a sequence of i.i.d. random variables  $w_{j,\widetilde{n}}\sim$ $\mathrm{Beta}(1 / \kappa ,(\widetilde{n} +n_0) / \kappa)$  with  $\kappa >0$  being some posterior inflation coefficient and  $n_0$  a number of pseudo-transitions.
Policy Q-values Next we define the policy Q-values that is updated at the end of a stage. Let say for state-action pair  $(s,a)$  at step  $h$  an stage ends at time  $t$ . This policy Q-values is then given by the maximum of temporary Q-values  $\overline{Q}_h^{t + 1} = \max_{j\in [J]}\widetilde{Q}_h^{t + 1,j}(s,a)$ . Then the policy Q-values is constant within a stage. The value used to define the targets is  $\overline{V}_h^{t + 1}(s) = \max_{a\in \mathcal{A}}\overline{Q}_h^{t + 1}(s,a)$ . The policy used to interact with the environment is greedy with respect to the policy Q-values  $\pi_h^{t + 1}(s)\in \arg \max_{a\in \mathcal{A}}\overline{Q}_h^{t + 1}(s,a)$  (we break ties arbitrarily).
3.3 Regret bound
We fix  $\delta \in (0,1)$  and the number of posterior samples  $J \triangleq \lceil c_J \cdot \log(2SAHT / \delta) \rceil$ , where  $c_J = 1 / \log(2 / (1 + \Phi(1)))$  and  $\Phi(\cdot)$  is the cumulative distribution function (CDF) of a normal distribution. Note that  $J$  has a logarithmic dependence on  $S, A, H, T$ , and  $1 / \delta$ .
We now state the regret bound of Staged-RandQL with a full proof in Appendix D.