SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $C'(s') = \max \left(4, 16\gamma^2\varphi^2\operatorname{Var}_{s',\pi^{*}(s')}[V^{\star}]\right)$ .  $U_{1}(\omega)$  resembles the term in Theorem 4.1 (note that we do not know whether  $U_{1}$  is a valid upper bound for  $T_{\varepsilon}$ ). For the second question, our numerical experiments (presented below) suggest that  $U(\omega)$  is a tighter upper bound than  $U_{0}(\omega)$ .
4.2 Example on Tabular MDPs
In Figure 1, we compare the characteristic time upper bounds obtained in the previous subsection. These upper bounds correspond to the allocations  $\omega^{\star}$ ,  $\omega_0^{\star}$ , and  $\omega_1^{\star}$  obtained by minimizing, over  $\Delta (S\times A)^{1}$ ,  $U(\omega)$ ,  $U_{0}(\omega)$ , and  $U_{1}(\omega)$ , respectively. We evaluated these characteristic times on various MDPs: (1) a random MDP (see Sec. A in the appendix); (2) the RiverSwim environment
Results are similar when we account for the navigation constraints. We omit these results for simplicity.
Figure 1: Comparison of the upper bounds (2) and (3) for different sizes of  $S$  and  $\gamma = 0.95$ . We evaluated different allocations using  $U_0(\omega)$  and  $U(\omega)$ . The allocations are:  $\omega_0^\star$  (the optimal allocation in (2),  $\omega^\star$  (the optimal allocation in (3) and  $\omega_1^\star$  (the optimal allocation in (4) by setting  $k = 1$  uniformly across states and actions). For the random MDP we show the median value across 30 runs.
[47]; (3) the Forked RiverSwim, a novel environment where the agent needs to constantly explore two different states to learn the optimal policy (compared to the RiverSwim environment, the sample complexity is higher; refer to Appendix A for a complete description).
We note that across all plots, the optimal allocation  $\omega_0^\star$  has a quite large characteristic time (black cross). Instead, the optimal allocation  $\omega^{\star}$  (blue circle) computed using our new upper bound (3) achieves a lower characteristic time. When we evaluate  $\omega_0^\star$  on the new bound (3) (orange star), we observe similar characteristic times.
Finally, to verify that we can indeed choose  $k = 1$  uniformly across states and actions, we evaluated the characteristic time  $\omega_1^\star$  computed using (4) (green triangle). Our results indicate that the performance is not different from those obtained with  $\omega^{\star}$ , suggesting that the quantities of interest (gaps and variances) are enough to learn an efficient exploration allocation. We investigate the choice of  $k$  in more detail in Appendix A.
5 Model-Free Active Exploration Algorithms
In this section we present MF-BPI, a model-free exploration algorithm that leverages the optimal allocations obtained through the previously derived upper bound of the sample complexity lower bound. We first present an upper bound  $\tilde{U} (\omega)$  of  $U(\omega)$ , so that it is possible to derive a closed form solution of the optimal allocation (an idea previously proposed in [28]).
Proposition 5.1. Assume that  $\phi$  has a unique optimal policy  $\pi^{\star}$ . For all  $\omega \in \Delta(S \times A)$ , we have:
$$
U (\omega) \leq \tilde {U} (\omega) := \max  _ {s, a \neq \pi^ {\star} (s)} \frac {H (s , a)}{\omega (s , a)} + \frac {H}{\min  _ {s ^ {\prime}} \omega \left(s ^ {\prime} , \pi^ {\star} \left(s ^ {\prime}\right)\right)},
$$
with  $H(s, a) \coloneqq \frac{2 + 8\varphi^2M_{sa}^k[V^\star]^{2^{1 - k}}}{\Delta(s, a)^2}$  and  $H \coloneqq \frac{\max_{s'} C(s') (1 + \gamma)^2}{\Delta_{\min}^2 (1 - \gamma)^2}$ . The minimizer  $\tilde{\omega}^\star \coloneqq \arg \inf_{\omega} \tilde{U}(\omega)$  satisfies  $\tilde{\omega}^\star(s, a) \propto H(s, a)$  for  $a \neq \pi^\star(s)$  and  $\tilde{\omega}^\star(s, \pi^\star(s)) \propto \sqrt{H \sum_{s, a \neq \pi^\star(s)} H(s, a) / |S|}$  otherwise.
In the MF-BPI algorithm, we estimate the gaps  $\Delta(s, a)$  and  $M_{sa}^{k}[V^{\star}]$  for a fixed small value of  $k$  (we later explain how to do this in a model-free manner.) and compute the corresponding allocation  $\tilde{\omega}^{\star}$ . This allocation drives the exploration under MF-BPI. Using this design approach, we face two issues:
(1) Uniform  $k$  and regularization. It is impractical to estimate  $M_{sa}^{k}[V^{\star}]$  for multiple values of  $k$ . Instead, we fix a small value of  $k$  (e.g.,  $k = 1$  or  $k = 2$ ) for all state-action pairs (refer to the previous section for a discussion on this choice). Then, to avoid excessively small values of the gaps in the denominator, we regularize the allocation  $\tilde{\omega}^{\star}$  by replacing, in the expression of  $H(s,a)$  (resp.  $H_{\mathrm{min}}$ ),  $\Delta(s,a)$  (resp.  $\Delta_{\mathrm{min}}$ ) by  $(\Delta(s,a) + \lambda)$  (resp.  $(\Delta_{\mathrm{min}} + \lambda)$ ) for some  $\lambda > 0$ .
(2) Handling parametric uncertainty via bootstrapping. The quantities  $\Delta(s, a)$  and  $M_{sa}^{k}[V^{\star}]$  required to compute  $\tilde{\omega}^{\star}$  remain unknown during training, and we adopt the Certainty Equivalence principle, substituting the current estimates of these quantities to compute the exploration strategy.
Algorithm 1 Boostrapped MF-BPI (Boostrapped Model Free Best Policy Identification)
Require: Parameters  $(\lambda ,k,p)$  ; ensemble size  $B$  ; learning rates  $\{(\alpha_{t},\beta_{t})\}_{t}$    
1: Initialize  $Q_{1,b}(s,a)\sim \mathcal{U}([0,1 / (1 - \gamma)])$  and  $M_{1,b}(s,a)\sim \mathcal{U}([0,1 / (1 - \gamma)^{2^k}])$  for all  $(s,a)\in$ $S\times A$  and  $b\in [B]$    
2: for  $t = 0,1,2,\ldots$  do   
3: Bootstrap a sample  $(\hat{Q}_t,\hat{M}_t)$  from the ensemble, and compute the allocation  $\omega^{(t)}$  using Proposition 5.1. Sample  $a_{t}\sim \omega^{(t)}(s_{t},\cdot)$  ; observe  $(r_t,s_{t + 1})\sim q(\cdot |s_t,a_t)\otimes P(\cdot |s_t,a_t)$    
4: for  $b = 1,\dots,B$  do   
5: With probability  $p$  , using the experience  $(s_t,a_t,r_t,s_{t + 1})$  , update  $Q_{t,b}$  and  $M_{t,b}$  using Equations (5) and (6).   
6: end for   
7: end for
By doing so, we are inherently introducing parametric uncertainty into these terms that is not taken into account by the allocation  $\tilde{\omega}^{\star}$ . To deal with this uncertainty, the traditional method, as used e.g. in [28, 29]), involves using  $\epsilon$ -soft exploration policies to guarantee that all state-action pairs are visited infinitely often. This ensures that the estimation errors vanish as time grows large. In practice, we find this type of forced exploration inefficient. In MF-BPI, we opt for a bootstrapping approach to manage parametric uncertainties, which can augment the traditional forced exploration step, leading to more principled exploration.
5.1 Exploration in tabular MDPs.
The pseudo-code of MF-BPI for tabular MDPs is presented in Algorithm 1. In round  $t$ , MF-BPI explores the MDP using the allocation  $\omega^{(t)}$  estimating  $\tilde{\omega}^{\star}$ . To compute this allocation, we use Proposition 5.1 and need (i) the sub-optimality gaps  $\Delta(s,a)$ , which can be easily derived from the  $Q$ -function; (ii) the  $2^k$ -th moment  $M_{sa}^k[V^{\star}]$ , which can always be learnt by means of stochastic approximation. In fact, for any Markovian policy  $\pi$  and pair  $(s,a)$  we have  $M_{sa}^k[V_\phi^\pi] = \frac{1}{\gamma^{2^k}} \mathbb{E}_{s' \sim P(\cdot|s,a)} [\delta^\pi(s,a,s')^{2^k}]$ , where  $\delta^\pi(s,a,s') = r(s,a) + \gamma \mathbb{E}_{a' \sim \pi(\cdot|s')} [Q^\pi(s',a')] - Q^\pi(s,a)$  is a variant of the TD-error. MF-BPI then uses an asynchronous two-timescale stochastic approximation algorithm to learn  $Q^{\star}$  and  $M_{sa}^k[V^{\star}]$ ,
$$
Q _ {t + 1} \left(s _ {t}, a _ {t}\right) = Q _ {t} \left(s _ {t}, a _ {t}\right) + \alpha_ {t} \left(s _ {t}, a _ {t}\right) \left(r _ {t} + \gamma \max  _ {a} Q _ {t} \left(s _ {t + 1}, a\right) - Q _ {t} \left(s _ {t}, a _ {t}\right)\right), \tag {5}
$$
$$
M _ {t + 1} \left(s _ {t}, a _ {t}\right) = M _ {t} \left(s _ {t}, a _ {t}\right) + \beta_ {t} \left(s _ {t}, a _ {t}\right) \left(\left(\delta_ {t} ^ {\prime} / \gamma\right) ^ {2 ^ {k}} - M _ {t} \left(s _ {t}, a _ {t}\right)\right), \tag {6}
$$
where  $\delta_t' = r_t + \gamma \max_a Q_{t+1}(s_{t+1}, a) - Q_{t+1}(s_t, a_t)$ , and  $\{(\alpha_t, \beta_t)\}_{t \geq 0}$  are learning rates satisfying  $\sum_{t \geq 0} \alpha_t(s, a) = \sum_{t \geq 0} \beta_t(s, a) = \infty$ ,  $\sum_{t \geq 0} (\alpha_t(s, a)^2 + \beta_t(s, a)^2) \leq \infty$ , and  $\frac{\alpha_t(s, a)}{\beta_t(s, a)} \to 0$ .
MF-BPI uses bootstrapping to handle parametric uncertainty. We maintain an ensemble of  $(Q,M)$ -values, with  $B$  members, from which we sample  $(\hat{Q}_t,\hat{M}_t)$  at time  $t$ . This sample is generated by sampling a uniform random variable  $\xi \sim \mathcal{U}([0,1])$  and, for each  $(s,a)$  set  $\hat{Q}_t(s,a) = \mathrm{Quantile}_{\xi}(Q_{t,1}(s,a),\ldots ,Q_{t,B}(s,a))$  (assuming a linear interpolation). This method is akin to sampling from the parametric uncertainty distribution (we perform the same operation also to compute  $\hat{M}_t$ ). This sample is used to compute the allocation  $\omega^{(t)}$  using Proposition 5.1 by setting  $\Delta_t(s,a) = \max_{a'}\hat{Q}_t(s,a') - \hat{Q}_t(s,a)$ ,  $\pi_t^\star (s) = \arg \max_a\hat{Q}_t(s,a)$  and  $\Delta_{\min ,t} = \min_{s,a\neq \pi_t^\star (s)}\Delta_t(s,a)$ . Note that, the allocation  $\omega^{(t)}$  can be mixed with a uniform policy, to guarantee asymptotic convergence of the estimates. Upon observing an experience, with probability  $p$ , MF-BPI updates a member of the ensemble using this new experience.  $p$  tunes the rate at which the models are updated, similar to sampling with replacement, speeding up the learning process. Selecting a high value for  $p$  compromises the estimation of the parametric uncertainty, whereas choosing a low value may slow down the learning process.
Exploration without bootstrapping? To illustrate the need for our bootstrapping approach, we tried to use the allocation  $\omega^{(t)}$  mixed with a uniform allocation. In Figure 2, we show the results on Riverswim-like environments with 5 states. While forced exploration ensures infinite visits to all
Figure 2: Forced exploration example with 5 states. We explore according to  $\omega^{(t)}(s_t,a) = (1 - \epsilon_t)\frac{\tilde{\omega}_t^\star(s_t,a)}{\sum_{a'}\tilde{\omega}_t^\star(s_t,a')} + \epsilon_t\frac{1}{|A|}$ , mixing the estimate of the allocation  $\tilde{\omega}^{\star}$  from Proposition 5.1 with a uniform policy, with  $\epsilon_t = \max (10^{-3},1 / N_t(s_t))$  where  $N_{t}(s)$  indicates the number of times the agent visited state  $s$  up to time  $t$ . Shade indicates  $95\%$  confidence interval.
state-action pairs, this guarantee only holds asymptotically. As a result, the allocation mainly focuses on the current MDP estimate, neglecting other plausible MDPs that could produce the same data. This makes the forced exploration approach too sluggish for effective convergence, suggesting its inadequacy for rapid policy learning. These results highlight the need to account for the uncertainty in  $Q$ ,  $M$  when computing the allocation.
5.2 Extension to Deep Reinforcement Learning
To extend bootstrapped MF-BPI to continuous MDPs, we propose DBMF-BPI (see Algorithm 2, or Appendix B). DBMF-BPI uses the mechanism of prior networks from BSP [39](bootstrapping with additive prior) to account for uncertainty that does not originate from the observed data. As before, we keep an ensemble  $\{Q_{\theta_1},\dots ,Q_{\theta_B}\}$  of  $Q$  -values (with their target networks) and an ensemble  $\{M_{\tau_1},\ldots ,M_{\tau_B}\}$  of  $M$  -values, as well as their prior networks. We use the same procedure as in the tabular case to compute  $(\hat{Q}_t,\hat{M}_t)$  at time  $t$ , except that we sample  $\xi \sim \mathcal{U}([0,1])$  every  $T_{s}\propto (1 - \gamma)^{-1}$  training steps (or at the end of an episode) to make the training procedure more stable. The quantity  $\hat{Q}_t$  is used to compute  $\pi_t^\star (s_t)$  and  $\Delta_t(s_t,a)$ . We estimate  $\Delta_{\mathrm{min},t}$  via stochastic approximation, with the minimum gap from the last batch of transitions sampled from the replay buffer serving as a target. To derive the exploration strategy, we compute  $H_{t}(s_{t},a) = \frac{2 + 8\varphi^{2}\hat{M}_{t}(s_{t},a)^{2^{1 - k}}}{(\Delta_{t}(s_{t},a) + \lambda)^{2}}$  and  $H_{t} = \frac{4(1 + \gamma)^{2}\max(1,4\gamma^{2}\varphi^{2}\hat{M}_{t}(s_{t},\pi_{t}^{\star}(s_{t}))^{2^{1 - k}})}{(\Delta_{\mathrm{min},t} + \lambda)^{2}(1 - \gamma)^{2}}$ . Next, we set the allocation  $\omega_o^{(t)}$  as follows:  $\omega_o^{(t)}(s_t,a) = H_t(s_t,a)$  if  $a\neq \pi_t^\star (s_t)$  and  $\omega_o^{(t)}(s_t,a) = \sqrt{H_t\sum_{a\neq\pi_t^\star(s_t)}H_t(s_t,a)}$  otherwise. Finally, we obtain an  $\epsilon_{t}$ -soft exploration policy  $\omega^{(t)}(s_t,\cdot)$  by mixing  $\omega_o^{(t)}(s_t,\cdot) / \sum_a\omega_o^{(t)}(s_t,a)$  with a uniform distribution (using an exploration parameter  $\epsilon_t$ ).
Algorithm 2 DBMF-BPI (Deep Bootstrapped Model Free BPI)
Require: Parameters  $(\lambda, k)$ ; ensemble size  $B$ ; exploration rate  $\{\epsilon_t\}_t$ ; estimate  $\Delta_{\min,0}$ ; mask probability  $p$ .
1: Initialize replay buffer  $\mathcal{D}$ , networks  $Q_{\theta_b}$ ,  $M_{\tau_b}$  and targets  $Q_{\theta_b'}$  for all  $b \in [B]$ .
2: for  $t = 0,1,2,\ldots ,$  do
3: Sampling step.
4: Compute allocation  $\omega^{(t)}\gets$  ComputeAllocation  $(s_t,\{Q_{\theta_b},M_{\tau_b}\}_{b\in [B]},\Delta_{\min ,t},\gamma ,\lambda ,k,\epsilon_t)$
5: Sample  $a_{t}\sim \omega^{(t)}(s_{t},\cdot)$  and observe  $(r_t,s_{t + 1})\sim q(\cdot |s_t,a_t)\otimes P(\cdot |s_t,a_t)$
6: Add transition  $z_{t} = (s_{t}, a_{t}, r_{t}, s_{t + 1})$  to the replay buffer  $\mathcal{D}$ .
7: Training step.
8: Sample a batch  $\mathcal{B}$  from  $\mathcal{D}$ , and with probability  $p$  add the  $i^{th}$  experience in  $\mathcal{B}$  to a sub-batch  $\mathcal{B}_b$ ,  $\forall b \in [B]$ . Update the  $(Q, M)$ -values of the  $b^{th}$  member in the ensemble using  $\mathcal{B}_b$ :  $\{Q_{\theta_b}, Q_{\theta_b'}, M_{\tau_b}\}_{b \in [B]} \gets \text{Training}(\{\mathcal{B}_b, Q_{\theta_b}, Q_{\theta_b'}, M_{\tau_b}\}_{b \in [B]})$ .
9: Update estimate  $\Delta_{\min ,t + 1}\gets \mathrm{EstimateMinimumGap}(\Delta_{\min ,t},\mathcal{B},\{Q_{\theta_b}\}_{b\in [B]})$
10: end for
We evaluate the performance of MF-BPI on benchmark problems and compare it against state-of-the-art methods (details can be found in Appendix A).
Tabular MDPs. In the tabular case, we compared various algorithms on the Riverside and Forked Riverside environments. We evaluate MF-BPI with (1) bootstrapping and with (2) the forced exploration step using an  $\epsilon$ -soft exploration policy, MDP-NAS [29], PSRL [36] and Q-UCB [19, 56]. For MDP-NAS, the model of the MDP was initialized in an optimistic way (with additive smoothing).
In both environments, we varied the size of the state space. In Figure 3, we show  $1 - \frac{\|V^{\star} - V^{\pi_T}\|_{\infty}}{\|V^{\star}\|_{\infty}}$ , a performance measure for the estimated policy  $\pi_T^\star$  after  $T = |S|\times 10^{4}$  steps with  $\gamma = 0.99$ . Results (the higher the better) indicate that bootstrapped MF-BPI can compete with model-based and model-free algorithms on hard-exploration problems, without resorting to expensive model-based procedures. Details of the experiments, including the initialization of the algorithms, are provided in Appendix A.
Figure 3: Evaluation of the estimated optimal policy  $\pi_T^\star$  after  $T$  steps for MF-BPI, Q-UCB, MDP-NAS and PSRL. Results are averaged across 10 seeds and lines indicate  $95\%$  confidence intervals.
Deep RL. In environments with continuous state space, we compared DBMF-BPI with BSP [40, 39] (Bootstrapped DQN with randomized priors) and IDS [33] (Information-Directed Sampling). We also evaluated DBMF-BPI against BSP2, a variant of BSP that uses the same masking mechanism as DBMF-BPI for updating the ensemble. These methods were tested on challenging exploration problems from the DeepMind behavior suite [41] with varying levels of difficulty: (1) a stochastic version of DeepSea and (2) the Cartpole swingup problem. The DeepSea problem includes a  $5\%$  probability of the agent slipping, i.e., that an incorrect action is executed, which increases the aleatoric variance.
The results for the Cartpole swingup problem are depicted in Figure 4 for various difficulty levels  $k$  (see also Appendix A.5 for more details), demonstrating the ability of DBMF-BPI to quickly learn an efficient policy. While BSP generally performs well, there is a notable difference in performance when compared to DBMF-BPI. For a fair comparison, we used the same network initialization across all methods, except for IDS. Untuned, IDS performed poorly; proper initialization improved its performance, but results remained unsatisfactory. In Figure 5, we present two exploration metrics
Figure 4: Cartpole swingup problem. On the left: total upright time at a difficulty level of  $k = 10$ . On the right: total upright time after 200 episodes for different difficulties  $k$ . To observe a positive reward, the pole's angle must satisfy  $\cos(\theta) > k/20$ , and the cart's position should satisfy  $|x| \leq 1 - k/20$ . Bars and shaded areas indicate 95% confidence intervals.
Figure 5: Exploration in Cartpole swingup for  $k = 5$ . On the left, we show the entropy of visitation frequency for the state space  $(x,\dot{x},\theta ,\dot{\theta})$  during training. On the right, we show a measure of the dispersion of the most recent visits; smaller values indicate that the agent is less explorative as  $t$  increases.
for difficulty  $k = 5$ . The frequency of visits measures the uniformity and dispersion of visits across the state space, while the second metric evaluates the recency of visits to different regions, capturing how frequently the methods keep visiting previously visited states (a smaller value indicates that the agent tends to concentrate on a specific region of the state space). For detailed analysis, please refer to appendix A.
Figure 6: Slipping DeepSea problem. On the left: total number of successful episodes (i.e., that the agent managed to reach the final reward) for a grid with  $30^2$  input features. On the right: standard deviation of  $t_{\mathrm{visit}}$  at the last episode, depicting how much each agent explored (the lower the better).
For the slipping DeepSea problem, results are depicted in Fig. 6 (see also Appendix A.4 for more details). Besides the number of successful episodes, we also display the standard deviation of  $(t_{\mathrm{visit}})_{ij}$  across all cells  $(i,j)$ , where  $(t_{\mathrm{visit}})_{ij}$  indicates the last timestep  $t$  that a cell  $(i,j)$  was visited (normalized by  $NT$ , the product of the grid size, and the number of episodes). The right plot shows  $\mathrm{std}(t_{\mathrm{visit}})$  for different problem sizes, highlighting the good exploration properties of DBMF-BPI. Additional details and exploration metrics can be found in Appendix A.
In this work, we studied the problem of exploration in Reinforcement Learning and presented MF-BPI, a model-free solution for both tabular and continuous state-space MDPs. To derive this method, we established a novel approximation of the instance-specific lower bound necessary for identifying nearly-optimal policies. Importantly, this approximation depends only on quantities learnable via stochastic approximation, paving the way towards model-free methods. Numerical results on hard-exploration problems highlighted the effectiveness of our approach for learning efficient policies over state-of-the-art methods.
This research was supported by the Swedish Foundation for Strategic Research through the CLAS project (grant RIT17-0046) and partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation. The authors would also like to thank the anonymous reviewers for their valuable and insightful feedback. On a personal note, Alessio Russo wishes to personally thank Damianos Tranos, Yassir Jedra, Daniele Foffano, and Letizia Orsini for their invaluable assistance in reviewing the manuscript.
[1] Peter Atkins, Peter William Atkins, and Julio de Paula. Atkins' physical chemistry. Oxford university press, 2014.
[2] Peter Auer. Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research, 3(Nov):397-422, 2002.
[3] Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47:235-256, 2002.
[4] Peter Auer, Thomas Jaksch, and Ronald Ortner. Near-optimal regret bounds for reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS), 21, 2008.
[5] Kamyar Azizzadenesheli, Emma Brunskill, and Animashree Anandkumar. Efficient exploration through bayesian deep q-networks. In 2018 Information Theory and Applications Workshop (ITA), pages 1-9. IEEE, 2018.
[6] Chenjia Bai, Lingxiao Wang, Lei Han, Jianye Hao, Animesh Garg, Peng Liu, and Zhaoran Wang. Principled exploration via optimistic bootstrapping and backward induction. In International Conference on Machine Learning, pages 577-587. PMLR, 2021.
[7] Andrew G Barto. Intrinsic motivation and reinforcement learning. Intrinsically Motivated Learning in Natural and Artificial Systems, Springer, pages 17-47, 2013.
[8] Marc G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In International Conference on Machine Learning, pages 449-458. PMLR, 2017.
[9] Ronen I Brafman and Moshe Tennenholtz. R-max-a general polynomial time algorithm for near-optimal reinforcement learning. Journal of Machine Learning Research, 3(Oct):213-231, 2002.
[10] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2018.
[11] Richard Y Chen, Szymon Sidor, Pieter Abbeel, and John Schulman. Ucb exploration via q-ensembles. arXiv preprint arXiv:1706.01502, 2017.
[12] William R Clements, Bastien Van Delft, Benoit-Marie Robaglia, Reda Bahi Slaoui, and Sébastien Toth. Estimating risk and uncertainty in deep reinforcement learning. arXiv preprint arXiv:1905.09638, 2019.
[13] Will Dabney, Mark Rowland, Marc Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[14] Richard Dearden, Nir Friedman, and Stuart Russell. Bayesian q-learning. In Proceedings of the 15th National Conference on Artificial Intelligence, American Association for Artificial Intelligence, July, 1998, pages 761-768. AAAI Press, 1998.
[15] Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, et al. Noisy networks for exploration. arXiv preprint arXiv:1706.10295, 2017.