SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm 1 Tabular Staged-RandQL
1: Input: inflation coefficient  $\kappa$ ,  $J$  ensemble size, number of prior transitions  $n_0$ , prior reward  $r_0$ .  
2: Initialize:  $\overline{V}_h(s) = \overline{Q}_h(s,a) = \widetilde{Q}_h^j (s,a) = r(s,a) + r_0(H - h - 1)$ , initialize counters  $\widetilde{n}_h(s,a) = 0$  for  $j,h,s,a\in [J]\times [H]\times S\times A$  and stage  $q_{h}(s,a) = 0$ .  
3: for  $t\in [T]$  do  
4: for  $h\in [H]$  do  
5: Play  $a_{h}\in \arg \max_{a}\overline{Q}_{h}(s_{h},a)$ .  
6: Observe reward and next state  $s_{h + 1}\sim p_h(s_h,a_h)$ .  
7: Sample learning rates  $w_{j}\sim \mathrm{Beta}(1 / \kappa ,(\widetilde{n} +n_{0}) / \kappa)$  for  $\widetilde{n} = \widetilde{n}_h(s_h,a_h)$ .  
8: Update temporary  $Q$ -values for all  $j\in [J]$ $\widetilde{Q}_h^j (s_h,a_h)\coloneqq (1 - w_j)\widetilde{Q}_h^j (s_h,a_h) + w_j(r_h(s_h,a_h) + \overline{V}_{h + 1}(s_{h + 1}))$ .  
9: Update counter  $\widetilde{n}_h(s_h,a_h)\coloneqq \widetilde{n}_h(s_h,a_h) + 1$   
10: if  $\widetilde{n}_h(s_h,a_h) = \lfloor (1 + 1 / H)^q H\rfloor$  for  $q = q_{h}(s_{h},a_{h})$  being the current stage then  
11: Update policy  $Q$ -values  $\overline{Q}_h(s_h,a_h)\coloneqq \max_{j\in [J]}\widetilde{Q}_h^j (s_h,a_h)$ .  
12: Update value function  $\overline{V}_h(s_h)\coloneqq \max_{a\in A}\overline{Q}_h(s_h,a)$   
13: Reset temporary  $Q$ -values  $\widetilde{Q}_h^j (s_h,a_h)\coloneqq r_h(s_h,a_h) + r_0(H - h - 1)$ .  
14: Reset counter  $\widetilde{n}_h(s_h,a_h)\coloneqq 0$  and change stage  $q_{h}(s_{h},a_{h})\coloneqq q_{h}(s_{h},a_{h}) + 1$ .  
15: end if  
16: end for  
17: end for
Theorem 1. Consider a parameter  $\delta \in (0,1)$ . Let  $\kappa \triangleq 2(\log(8SAH / \delta) + 3\log(\mathrm{e}\pi(2T + 1))$ ,  $n_0 \triangleq \lceil \kappa(c_0 + \log_{17/16}(T)) \rceil$ ,  $r_0 \triangleq 2$ , where  $c_0$  is an absolute constant defined in (5); see Appendix D.3. Then for Staged-RandQL, with probability at least  $1 - \delta$ ,
$$
\mathfrak {R} ^ {T} = \widetilde {\mathcal {O}} \left(\sqrt {H ^ {5} S A T} + H ^ {3} S A\right).
$$
Discussion The regret bound of Theorem 1 coincides (up to a logarithmic factor) with the bound of the OptQL algorithm with Hoeffding-type bonuses from Jin et al. [2018]. Up to a  $H$  factor, our regret matches the information-theoretic lower bound  $\Omega (\sqrt{H^3SAT})$  [Jin et al., 2018, Domingues et al., 2021b]. This bound could be achieved (up to logarithmic terms) in model-free algorithms by using Bernstein-type bonuses and variance reduction [Zhang et al., 2020]. We keep these refinements for future research as the main focus of our paper is on the novel randomization technique and its use to construct computationally tractable model-free algorithms.
Computational complexity Staged-RandQL is a model-free algorithm, and thus gets the  $\widetilde{\mathcal{O}}(HSA)$  space complexity as  $0\mathrm{ptQL}$ , recall that we set  $J = \widetilde{\mathcal{O}}(1)$ . The per-episode time-complexity is also similar and of order  $\widetilde{\mathcal{O}}(H)$ .
4 Randomized Q-learning for Metric Spaces
In this section we present a way to extend RandQL to general state-action spaces. We start from the simplest approach with predefined  $\varepsilon$ -net type discretization of the state-action space  $S \times \mathcal{A}$  (see Song and Sun 2019), and then discuss an adaptive version of the algorithm, similar to one presented by Sinclair et al. [2019].
To pose the first assumption, we start from a general definition of covering numbers.
Definition 1 (Covering number and covering dimension). Let  $(M,\rho)$  be a metric space. A set  $\mathcal{M}$  of open balls of radius  $\varepsilon$  is called an  $\varepsilon$ -cover of  $M$  if  $M\subseteq \bigcup_{B\in \mathcal{M}}B$ . The cardinality of the minimal  $\varepsilon$ -cover is called covering number  $N_{\varepsilon}$  of  $(M,\rho)$ . We denote the corresponding minimal  $\varepsilon$ -covering by  $\mathcal{N}_{\varepsilon}$ . A metric space  $(M,\rho)$  has a covering dimension  $d_{c}$  if  $\forall \varepsilon >0:N_{\varepsilon}\leq C_{N}\varepsilon^{-d_{c}}$ , where  $C_N$  is a constant.
The last definition extends the definition of dimension beyond vector spaces. For example, is case of  $M = [0,1]^d$  the covering dimension of  $M$  is equal to  $d$ . For more details and examples see e.g. Vershynin [2018, Section 4.2].
Next we are ready to introduce the first assumption.
Assumption 1 (Metric Assumption). Spaces  $S$  and  $\mathcal{A}$  are separable compact metric spaces with the corresponding metrics  $\rho_{S}$  and  $\rho_{\mathcal{A}}$ . The joint space  $S \times \mathcal{A}$  endowed with a product metric  $\rho$  that satisfies  $\rho((s, a), (s', a')) \leq \rho_{S}(s, s') + \rho_{\mathcal{A}}(a, a')$ . Moreover, the diameter of  $S \times \mathcal{A}$  is bounded by  $d_{\max}$ , and  $S \times \mathcal{A}$  has covering dimension  $d_c$  with a constant  $C_N$ .
This assumption is, for example, satisfied for the finite state and action spaces endowed with discrete metrics  $\rho_{\mathcal{S}}(s,s^{\prime}) = \mathbb{1}\{s\neq s^{\prime}\}$ ,  $\rho_{\mathcal{A}}(a,a^{\prime}) = \mathbb{1}\{a\neq a^{\prime}\}$  with  $d_c = 0$ ,  $C_N = SA$  and  $S$  and  $A$  being the cardinalities of the state and action spaces respectively. The above assumption also holds in the case  $S\subseteq [0,1]^{d_S}$  and  $\mathcal{A}\subseteq [0,1]^{d_{\mathcal{A}}}$  with  $d_{c} = d_{\mathcal{S}} + d_{\mathcal{A}}$ .
The next two assumptions describe the regularity conditions of transition kernel and rewards.
Assumption 2 (Reparametrization Assumption). The Markov transition kernel could be represented as an iterated random function. In other words, there exists a measurable space  $(\Xi, \mathcal{F}_{\Xi})$  and a measurable function  $F_{h} \colon (\mathcal{S} \times \mathcal{A}) \times \Xi \to \mathcal{S}$ , such that  $s_{h+1} \sim p_{h}(s_{h}, a_{h}) \iff s_{h+1} = F_{h}(s_{h}, a_{h}, \xi_{h})$  for a sequence of independent random variables  $\{\xi_{h}\}_{h \in [H]}$ .
This assumption is naturally satisfied for a large family of probabilistic model, see Kingma and Welling [2014]. Moreover, it has been utilized by the RL community both in theory [Ye and Zhou, 2015] and practice [Heess et al., 2015, Liu et al., 2018]. Essentially, this assumption holds for Markov transition kernels over a separable metric space, see Theorem 1.3.6 by Douc et al. [2018]. However, the function  $F_{h}$  could be ill-behaved. To avoid this behaviour, we need the following assumption.
Assumption 3 (Lipschitz Assumption). The function  $F_{h}(\cdot, \xi_{h})$  is  $L_{F}$ -Lipschitz in the first argument for almost every value of  $\xi_{h}$ . Additionally, the reward function  $r_{h} \colon S \times \mathcal{A} \to [0,1]$  is  $L_{r}$ -Lipschitz.
This assumption is commonly used in studies of the Markov processes corresponding to iterated random functions, see Diaconis and Freedman [1999], Ghosh and Marecek [2022]. Moreover, this assumption holds for many cases of interest. As main example, it trivially holds in tabular and Lipschitz continuous deterministic MDPs [Ni et al., 2019]. Notably, this observation demonstrates that Assumption 3 does not necessitate Lipschitz continuity of the transition kernels in total variation distance, since deterministic Lipschitz MDPs are not continuous in that sense. Additionally, incorporation of an additive noise to deterministic Lipschitz MDPs will lead to Assumption 3 with  $L_{F} = 1$ .
Furthermore, it is possible to show that Assumption 3 implies other assumptions stated in the literature. For example, it implies that the transition kernel is Lipschitz continuous in 1-Wasserstein metric, and that  $Q^{\star}$  and  $V^{\star}$  are both Lipschitz continuous.
Lemma 1. Let Assumption 1,2,3 hold. Then the transition kernels  $p_h(s,a)$  are  $L_{F}$ -Lipschitz continuous in 1-Wasserstein distance
$$
\mathcal {W} _ {1} \left(p _ {h} (s, a), p _ {h} \left(s ^ {\prime}, a ^ {\prime}\right)\right) \leq L _ {F} \cdot \rho \left(\left(s, a\right), \left(s ^ {\prime}, a ^ {\prime}\right)\right),
$$
where 1-Wasserstein distance between two probability measures on the metric space  $(M,\rho)$  is defined as  $\mathcal{W}_1(\nu ,\eta) = \sup_{f\text{is} 1 - Lipschitz}\int_M f\mathrm{d}\nu -\int_M f\mathrm{d}\eta .$
Lemma 2. Let Assumption 1,2,3 hold. Then  $Q_h^\star$  and  $V_h^\star$  are Lipschitz continuous with Lipschitz constant  $L_{V,h} \leq \sum_{h' = h}^{H} L_F^{h' - h} L_r$ .
The proof of these lemmas is postponed to Appendix E. For a more detailed exposition on 1-Wasserstein distance we refer to the book by Peyre and Cuturi [2019]. The first assumption was studied by Domingues et al. [2021c], Sinclair et al. [2023] in the setting of model-based algorithms in metric spaces. We are not aware of any natural examples of MDPs with a compact state-action space where the transition kernels are Lipschitz in  $\mathcal{W}_1$  but fail to satisfy Assumption 3.
In this section, following Song and Sun [2019], we present Net-Staged-RandQL algorithm that combines a simple non-adaptive discretization and an idea of stages by Bai et al. [2019], Zhang et al. [2020].
We assume that we have an access to all Lipschitz constants  $L_{r}, L_{F}, L_{V} \triangleq L_{V,1}$ . Additionally, we have access to the oracle that computes  $\varepsilon$ -cover  $\mathcal{N}_{\varepsilon}$  of the space  $\mathcal{S} \times \mathcal{A}$  for any predefined  $\varepsilon > 0^4$ .
Counts and stages Let  $n_h^t(B) \triangleq \sum_{i=1}^{t-1} \mathbb{1}\{(s_h^i, a_h^i) \in B\}$  be the number of visits of the ball  $B \in \mathcal{N}_{\varepsilon}$  at step  $h$  before episode  $t$ . Let  $e_k = \lfloor (1 + 1 / H)^k \cdot H \rfloor$  be the length of the stage  $k \geq 0$  and, by convention,  $e_{-1} = 0$ . We say that  $(B, h)$  belongs to the  $k$ -th stage at the beginning of episode  $t$  if  $n_h^t(B) \in [\sum_{i=0}^{k-1} e_i, \sum_{i=0}^k e_i)$ . Let  $\widetilde{n}_h^t(B) \triangleq n_h^t(s, a) - \sum_{i=0}^{k-1} e_i$  be the number of visits of the ball  $B$  at step  $h$  during the current stage  $k$ .
Temporary Q-values At the beginning of a stage, let say time  $t$ , we initialize  $J$  temporary Q-values as  $\widetilde{Q}_h^{t,j}(B) = r_0H$  for  $j \in [J]$  and  $r_0$  some pseudo-reward. Then within a stage  $k$  we update recursively the temporary Q-values
$$
\widetilde {Q} _ {h} ^ {t + 1, j} (B) = \left\{ \begin{array}{l l} (1 - w _ {j, \widetilde {n}}) \widetilde {Q} _ {h} ^ {t, j} (B) + w _ {j, \widetilde {n}} [ r _ {h} (s _ {h} ^ {t}, a _ {h} ^ {t}) + \overline {{V}} _ {h + 1} ^ {t} (s _ {h + 1} ^ {t}) ], & (s, a) = (s _ {h} ^ {t}, a _ {h} ^ {t}) \\ \widetilde {Q} _ {h} ^ {1, j} (B) & \text {o t h e r w i s e ,} \end{array} \right.
$$
where  $\widetilde{n} = \widetilde{n}_h^t(B)$  is the number of visits,  $w_{j,\widetilde{n}}$  is a sequence of i.i.d random variables  $w_{j,\widetilde{n}} \sim \mathrm{Beta}(1/\kappa, (\widetilde{n} + n_0(k)) / \kappa)$  with  $\kappa > 0$  some posterior inflation coefficient and  $n_0(k)$  a number of pseudo-transitions. The important difference between tabular and metric settings is the dependence on the pseudo-count  $n_0(k)$  on  $k$  in the latter case, since here the prior is used to eliminate the approximation error.
Policy Q-values Next, we define the policy Q-values that are updated at the end of a stage. Let us fix a ball  $B$  at step  $h$  and suppose that the currents stage ends at time  $t$ . Then the policy Q-values are given by the maximum of the temporary Q-values  $\overline{Q}_h^{t+1}(B) = \max_{j \in [J]} \widetilde{Q}_h^{t+1,j}(B)$ . The policy Q-values are constant within a stage. The value used to define the targets is computed on-flight using the formula  $\overline{V}_h^t(s) = \max_{a \in \mathcal{A}} \overline{Q}_h^t(\psi_\varepsilon(s,a))$ , where  $\psi_\varepsilon \colon S \times \mathcal{A} \to \mathcal{N}_\varepsilon$  is a quantization map, that assigns each state-action pair  $(s,a)$  to a ball  $B \ni (s,a)$ . The policy used to interact with the environment is greedy with respect to the policy Q-values and also computed on-flight  $\pi_h^t(s) \in \arg \max_{a \in \mathcal{A}} \overline{Q}_h^t(\psi_\varepsilon(s,a))$  (we break ties arbitrarily).
A detail description of Net-Staged-RandQL is provided in Algorithm 4 in Appendix E.2.
4.3 Regret Bound
We fix  $\delta \in (0,1)$ , the discretization level  $\varepsilon > 0$  and the number of posterior samples
$$
J \triangleq \left[ \tilde {c} _ {J} \cdot \left(\log \left(2 C _ {N} H T / \delta\right) + d _ {c} \log (1 / \varepsilon)\right) \right],
$$
where  $\tilde{c}_J = 1 / \log (4 / (3 + \Phi (1)))$  and  $\Phi (\cdot)$  is the cumulative distribution function (CDF) of a normal distribution. Note that  $J$  has a logarithmic dependence on  $H,T,1 / \varepsilon$  and  $1 / \delta$ . For the regret-optimal discretization level  $\varepsilon = T^{-1 / (d_c + 2)}$ , the number  $J$  is almost independent of  $d_{c}$ . Let us note that the role of prior in metric spaces is much higher than in the tabular setting. Another important difference is dependence of the prior count on the stage index. In particular, we have
$$
n _ {0} (k) = \left[ \widetilde {n} _ {0} + \kappa + \frac {\varepsilon L}{H - 1} \cdot \left(e _ {k} + \widetilde {n} _ {0} + \kappa\right) \right], \quad \widetilde {n} _ {0} = \left(c _ {0} + 1 + \log_ {1 7 / 1 6} (T)\right) \cdot \kappa
$$
where  $c_{0}$  is an absolute constant defined in (5) (see Appendix D.3),  $\kappa$  is the posterior inflation coefficient and  $L = L_{r} + (1 + L_{F})L_{V}$  is a constant. We now state the regret bound of Net-Staged-RandQL with a full proof being postponed to Appendix E.
Theorem 2. Suppose that  $N_{\varepsilon} \leq C_N \varepsilon^{-d_c}$  for all  $\varepsilon > 0$  and some constant  $C_N > 0$ . Consider a parameter  $\delta \in (0,1)$  and take an optimal level of discretization  $\varepsilon = T^{-1/(d_c + 2)}$ . Let  $\kappa \triangleq 2(\log(8HC_N/\delta) + d_c \log(1/\varepsilon) + 3 \log(\mathrm{e}\pi(2T + 1)))$ ,  $r_0 \triangleq 2$ . Then it holds for Net-Staged-RandQL, with probability at least  $1 - \delta$ ,
$$
\mathfrak {R} ^ {T} = \widetilde {\mathcal {O}} \bigg (H ^ {5 / 2} C _ {N} ^ {1 / 2} T ^ {\frac {d _ {c} + 1}{d _ {c} + 2}} + H ^ {3} C _ {N} T ^ {\frac {d _ {c}}{d _ {c} + 2}} + L T ^ {\frac {d _ {c} + 1}{d _ {c} + 2}} \bigg).
$$
We can restore the regret bound in the tabular setting by letting  $d_{c} = 0$  and  $C_N = SA$ , where  $S$  is the cardinality of the state-space, and  $A$  is the cardinality of the action-space.
Remark that the simple greedy algorithm can generate  $\varepsilon$ -cover of size  $N_{\varepsilon /2}$ , that will not affect the asymptotic behavior of our regret bounds, see Song and Sun [2019].
Discussion From the point of view of instance-independent bounds, our algorithm achieves the same result as Net-QL [Song and Sun, 2019] and Adaptive-QL [Sinclair et al., 2019], that matches the lower bound  $\Omega (HT^{\frac{d_c + 1}{d_c + 2}})$  by Sinclair et al. [2023] in dependence on budget  $T$  and covering dimension  $d_{c}$ . Notably, as discussed by Sinclair et al. [2023], the model-based algorithm such as Kernel-UCBVI [Domingues et al., 2021c] does not achieve optimal dependence in  $T$  due to hardness of the transition estimation problem.
Computational complexity For a fixed level of discretization  $\varepsilon$ , our algorithm has a space complexity of order  $\widetilde{\mathcal{O}}(HN_{\varepsilon})$ . Assuming that the computation of a quantization map  $\psi_{\varepsilon}$  has  $\widetilde{\mathcal{O}}(1)$  time complexity, we achieve a per-episode time complexity of  $\widetilde{\mathcal{O}}(HA)$  for a finite action space and  $\mathcal{O}(HN_{\varepsilon})$  for an infinite action space in the worst case due to computation of  $\arg \max_{a \in \mathcal{A}} \overline{Q}_h(\psi_{\varepsilon}(s,a))$ . However, this can be improved to  $\widetilde{\mathcal{O}}(H)$  if we consider adaptive discretization [Sinclair et al., 2019].
Adaptive discretization Additionally, we propose a way to combine RandQL with adaptive discretization by Cao and Krishnamurthy [2020], Sinclair et al. [2023]. This combination results in two algorithms: Adaptive-RandQL and Adaptive-Staged-RandQL. The second one could achieve the instance-dependent regret bound that scales with a zooming dimension, the instance-dependent measure of dimension. We will follow Sinclair et al. [2023] in the exposition of the required notation.
Definition 2. For any  $(s,a)\in S\times \mathcal{A}$ , the stage-dependent sub-optimality gap is defined as  $\mathrm{gap}_h(s,a) = V_h^\star (s) - Q_h^\star (s,a)$ .
This quantity is widely used in the theoretical instance-dependent analysis of reinforcement learning and contextual bandit algorithms.
Definition 3. The near-optimal set of  $\mathcal{S} \times \mathcal{A}$  for a given value  $\varepsilon$  defined as  $Z_h^\varepsilon = \{(s, a) \in \mathcal{S} \times \mathcal{A} | \operatorname{gap}_h(s, a) \leq (H + 1)\varepsilon\}$ .
The main insight of this definition is that essentially we are interested in a detailed discretization of the near-optimal set  $Z_h^\varepsilon$  for small  $\varepsilon$ , whereas all other state-action pairs could be discretized in a more rough manner. Interestingly enough,  $Z_h^\varepsilon$  could be a lower dimensional manifold, leading to the following definition.
Definition 4. The step- $h$  zooming dimension  $d_{z,h}$  with a constant  $C_{N,h}$  and a scaling factor  $\rho >0$  is given by
$$
d _ {z, h} = \inf  \left\{d > 0: \forall \varepsilon > 0 N _ {\varepsilon} \left(Z _ {h} ^ {\rho \cdot \varepsilon}\right) \leq C _ {N, h} \varepsilon^ {- d} \right\}.
$$
Under some additional structural assumptions on  $Q_h^\star$ , it is possible to show that the zooming dimension could be significantly smaller than the covering dimension, see, e.g., Lemma 2.8 in Sinclair et al. [2023]. However, at the same time, it has been shown that  $d_{z,h} \geq d_S - 1$ , where  $d_S$  is a covering dimension of the state space. Thus, the zooming dimension allows adaptation to a rich action space but not a rich state space.
Given this definition, it is possible to define a definite adaptive algorithm Adaptive-Staged-RandQL that attains the following regret guarantees
Theorem 3. Consider a parameter  $\delta \in (0,1)$ . For a value  $\kappa$  that depends on  $T$ ,  $d_c$  and  $\delta$ , for Adaptive-Staged-RandQL the following holds with probability at least  $1 - \delta$ ,
$$
\mathfrak {R} ^ {T} = \widetilde {\mathcal {O}} \left(H ^ {3} + H ^ {3 / 2} \sum_ {h = 1} ^ {H} T ^ {\frac {d _ {z , h} + 1}{d _ {z , h} + 2}}\right),
$$
where  $d_{z,h}$  is the step- $h$  zooming dimension and we ignore all multiplicative factors in the covering dimension  $d_c$ ,  $\log(C_N)$ , and Lipschitz constants.
We refer to Appendix F to a formal statement and a proof.
In this section we present the experiments we conducted for tabular environments using rlberry library [Domingues et al., 2021a]. We also provide experiments in non-tabular environment in Appendix I.
Environment We use a grid-world environment with 100 states  $(i,j) \in [10] \times [10]$  and 4 actions (left, right, up and down). The horizon is set to  $H = 50$ . When taking an action, the agent moves in the corresponding direction with probability  $1 - \varepsilon$ , and moves to a neighbor state at random with probability  $\varepsilon = 0.2$ . The agent starts at position  $(1,1)$ . The reward equals to 1 at the state  $(10,10)$  and is zero elsewhere.
Variations of randomized Q-learning For the tabular experiment we use the RandQL algorithm, described in Appendix B as it is the version of randomized Q-learning that is the closest to the baseline OptQL. Note that, we compare the different versions of randomized Q-learning in Appendix B.
Baselines We compare RandQL algorithm to the following baselines: (i) OptQL [Jin et al., 2018] (ii) UCBVI [Azar et al., 2017] (iii) Greedy-UCBVI, a version of UCBVI using real-time dynamic programming [Efroni et al., 2019] (iv) PSRL [Osband et al., 2013] and (v) RLSVI [Russo, 2019]. For the hyper-parameters used for these baselines refer to Appendix I.
Figure 1: Regret curves of RandQL and baselines in a grid-world environment for  $H = 50$  and transition noise  $\varepsilon = 0.2$ . The average is over 4 seeds.
Results Figure 1 shows the result of the experiments. Overall, we see that Redis outperforms OptQL algorithm on tabular environment, but still degrades in comparison to model-based approaches, that is usual for model-free algorithms in tabular environments. Indeed, using a model and backward induction allows new information to be more quickly propagated. But as counterpart, Redis has a better time-complexity and space-complexity than model-based algorithm, see Table 2 in Appendix I.
This paper introduced the RandQL algorithm, a new model-free algorithm that achieves exploration without bonuses. It utilizes a novel idea of learning rate randomization, resulting in provable sample efficiency with regret of order  $\widetilde{\mathcal{O}} (\sqrt{H^5SAT})$  in the tabular case. We also extend RandQL to the case of metric state-action space by using proper discretization techniques. The proposed algorithms inherit the good empirical performance of model-based Bayesian algorithm such that PSRL while keeping the small space and time complexity of model-free algorithm. Our result rises following interesting open questions for a further research.
Optimal rate for RandQL We conjecture that RandQL could get optimal regret in the tabular setting if coupled with variance reductions techniques as used by Zhang et al. [2020]. However, obtaining such improvements is not straightforward due to the intricate statistical dependencies involved in the analysis of RandQL.
Beyond one-step learning We observe a large gap in the experiments between Q-learning type algorithm that do one-step planning and e.g. UCBVI algorithm that does full planning or Greedy-UCBVI that does one-step planning with full back-up (expectation under transition of the model) for all actions. Therefore, it would interesting to study also algorithms that range between these two extremes [Efroni et al., 2018, 2019].
The work of D. Tiapkin, A. Naumov, and D. Belomestny were supported by the grant for research centers in the field of AI provided by the Analytical Center for the Government of the Russian Federation (ACRF) in accordance with the agreement on the provision of subsidies (identifier of the agreement 000000D730321P5Q0002) and the agreement with HSE University No. 70-2021-00139. E. Moulines received support from the grant ANR-19-CHIA-002 SCAI and parts of his work has been done under the auspices of Lagrange Center for maths and computing. P. Ménard acknowledges the Chaire SeqALO (ANR-20-CHIA-0020-01). This research was supported in part through computational resources of HPC facilities at HSE University.