SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The second one is a chain environment described by Osband et al. [2016] with  $L = 15$  states and 2 actions (left or right). The horizon is equal to 30, the probability of moving into wrong direction is equal to 0.1. The agent starts in the leftmost state with reward 0.05, also the largest reward is equal to 1 is the rightmost state.
Variations of randomized Q-learning First we compare the different variations of randomized Q-learning on grid-world environment. Precisely we consider:
- RandQL a randomized version of OptQL, detailed in Appendix B.
- Staged-RandQL a staged version of RandQL, described in Section 3.2.
- Sampled-RandQL a version of RandQL which samples one Q-value function in the ensemble to act, described in Appendix B.
For these algorithms we used the same parameters: posterior inflation  $\kappa = 1.0$ ,  $n_0 = 1 / S$  prior sample (same as PSRL, see below), ensemble size  $J = 10$ . We use a similar ensemble size as the one used for the experiments with OPSRL by Tiapkin et al. [2022a]. For Staged-RandQL we use stage of sizes  $\left((1 + 1 / H)^k\right)_{k\geq 1}$  without the  $H$  factor, in order to have several epochs per state-action pair even for few episodes.
The comparison is presented in Figure 2. We observe that Redis and Sampled-RandQL behave similarly with slightly better performance for Sampled-RandQL. This is coherent with the experiment on the comparison between OPSRL and PSRL Tiapkin et al. [2022a] where the optimistic version performs worst than the fully randomized algorithm. We also note that even with the aggressive stage schedule, Staged-RandQL needs more episode to converge. We conclude that despite that stage simplifies the analysis, it artificially slows down the learning in practice.
To ease the comparison with the baselines, for the rest of the experiments we only use RandQL because of its similarity with OptQL.
Baselines We compare RandQL algorithm to the following baselines:
- OptQL [Jin et al., 2018] a model-free optimistic Q-learning.
Figure 3: Regret curves of RandQL and several baselines in (left) a grid-world environment with 100 states and 4 actions for  $H = 50$  an transitions noise 0.2, and (right) in a chain environment of length  $L = 15$ , 2 actions for  $H = 30$  with transition noise 0.1: smaller is better. We show average and error bars over 4 seeds.
- UCBVI [Azar et al., 2017] a model-based optimistic dynamic programming.
Greedy-UCBVI [Efroni et al., 2019] optimistic real-time dynamic programming.
- PSRL [Osband et al., 2013] model-based posterior sampling.
- RLSVI [Russo, 2019] model-based randomized dynamic programming.
The selection of parameters can have a significant impact on the empirical regrets of an algorithm. For example, adjusting the multiplicative constants in the bonus of UCBVI or the scale of the noise in RLSVI can result in vastly different regrets. To ensure a fair comparison between algorithms, we have made the following parameter choices:
- For bonus-based algorithm, UCBVI, OptQL we use simplified bonuses from an idealized Hoeffding inequality of the form
$$
\beta_ {h} ^ {t} (s, a) \triangleq \min  \left(\sqrt {\frac {1}{n _ {h} ^ {t} (s , a)}} + \frac {H - h + 1}{n _ {h} ^ {t} (s , a)}, H - h + 1\right). \tag {25}
$$
As explained by Ménard et al. [2021], this bonus does not necessarily result in a true upper-confidence bound on the optimal Q-value. However, it is a valid upper-confidence bound for  $n_h^t (s,a) = 0$  which is important in order to discover new state-action pairs.
- For RLSVI we use the variance of Gaussian noise equal to simplified Hoeffding bonuses described above in (25).
- For PSRL, we use a Dirichlet prior on the transition probability distribution with parameter  $(1 / S,\dots ,1 / S)$  and for the rewards a Beta prior with parameter (1, 1). Note that since the reward  $r$  is not necessarily in  $\{0,1\}$  we just sample a new randomized reward  $r^{\prime}\sim \mathrm{Ber}(r)$  accordingly to a Bernoulli distribution of parameter  $r$ , to update the posterior, see Agrawal and Goyal [2013].
Results Figure 3 shows the result of the experiments. Overall, we see that  $\mathsf{RandQL}$  outperforms OptQL algorithm on tabular environment, but still degrades in comparison to model-based approaches, that is usual for model-free algorithms in tabular environments. Indeed, as explained by Menard et al. [2021], using a model and backward induction allows new information to be more quickly propagated. For example UCBVI needs only one episode to propagate information about the last step  $h = H$  to the first step  $h = 1$  whereas OptQL or  $\mathsf{RandQL}$  need at least  $H$  episodes. But as counterpart,  $\mathsf{RandQL}$  has a better time-complexity and space- complexity than model based algorithm, see Table 2.
I.2 Non-tabular experiments
The second experiment was performed on a set of two dimensional continuous environments [Domingues et al., 2021a] with levels of increasing exploration difficulty.
Algorithm	Time-complexity (per episode)	Space complexity
UCBVI [Azar et al., 2017]
PSRL [Osband et al., 2013]	\(\widetilde{\mathcal{O}}(HS^2A)\)	\(\widetilde{\mathcal{O}}(HS^2A)\)
RLSVI [Russo, 2019]
Greedy-UCBVI [Efroni et al., 2019]	\(\widetilde{\mathcal{O}}(HSA)\)
OptQL [Jin et al., 2018]	\(\widetilde{\mathcal{O}}(H)\)	\(\widetilde{\mathcal{O}}(HSA)\)
RandQL (this paper)	\(\widetilde{\mathcal{O}}(H)\)	\(\widetilde{\mathcal{O}}(HSA)\)
Table 2: Time- and space- complexity of several tabular algorithms.
Environment We use a ball environment with the 2-dimensional unit Euclidean ball as state-space  $\mathcal{S} = \{s\in \mathbb{R}^2,\| s\| _2\leq 1\}$  and of horizon  $H = 30$  . The action space is a list of 2-dimensional vectors  $\mathcal{A} = \{[0.0,0.0],[-0.05,0.0],[0.05,0.0],[0.0,0.05],[0.0, - 0.05]\}$  that can be associated with the action of staying at the same place, moving left, right, up or down. Given a state  $s_h$  and an action  $a_{h}$  the next state is
$$
s _ {h + 1} = \operatorname {p r o j} _ {\mathcal {S}} \left(s _ {h} + a _ {h} + \sigma z _ {h}\right)
$$
where  $z_{h} \sim \mathcal{N}([0,0], I_{2})$  is some independent Gaussian noise with zero mean and identity covariance matrix and  $\mathrm{proj}_B$  is the euclidean projection on the unit ball  $S$ . The initial position  $s_1 = \sigma_1z_1$  with  $z_{1} \sim \mathcal{N}([0,0], I_{2})$  and  $\sigma_{1} = 0.001$ , is sampled at random from a Gaussian distribution. The reward function independent of the action and the step
$$
r _ {h} (s, a) = \max  (0, 1 - \| s - s ^ {\prime} \| / c)
$$
where  $s' = [0.5, 0.5] \in S$  is the reward center and  $c > 0$  is some smoothness parameter. We distinguish 3 levels by increasing exploration difficulty:
- Level 1, dense reward and small noise. The smoothness parameter is  $c = 0.5 \cdot \sqrt{2} \approx 0.71$  and the transition standard deviation is  $\sigma = 0.01$ .
- Level 2, sparse reward and small noise. The smoothness parameter is  $c = 0.2$  and the transition standard deviation is  $\sigma = 0.01$ .
- Level 3, sparse reward and large noise. The smoothness parameter is  $c = 0.2$  and the transition standard deviation is  $\sigma = 0.025$ .
RandQL algorithm Among the different versions of RandQL for continuous state-action space, see Section 4, we pick the Adaptive-RandQL algorithm, described in Appendix F, as it is the closest version to the Adaptive-QL algorithm. It combines the RandQL algorithm and adaptive discretization. For Adaptive-RandQL we used an ensemble of size  $J = 10 \approx \log (T)$ ,  $\kappa = 10 \approx \log (T)$  and a prior number of samples of  $n_0 = 0.33$ . Note that we increased the number of prior samples in comparison to the tabular case as explained in Section 4.
Baselines We compare Adaptive-RandQL algorithm to the following baselines:
- Adaptive-QL [Sinclair et al., 2019, 2023], an adaptation of OptQL algorithm to continuous state-space thanks to adaptive discretization;
- Kernel-UCBVI [Domingues et al., 2021c], a kernel-based version of the UCBVI algorithm;
- DQN [Mnih et al., 2013], a deep RL algorithm;
- BootDQN [Osband and Van Roy, 2015], a deep RL algorithm with an additional exploration given by bootstrapping several Q-networks;
For Adaptive-QL and Kernel-UCBVI baselines we employ the same simplified bonuses (25) used for the tabular experiments. For Kernel-UCBVI we used Gaussian kernel of bandwidth 0.025 and the representative states technique, with 300 representative states, described by Domingues et al. [2021c].
For DQN and BootDQN we use as network a 2-layer multilayer perceptron (MLP) with hidden layer size equals to 64. For exploration, DQN utilizes  $\varepsilon$ -greedy exploration with coefficient annealing from 1.0 to 0.1 during the first 10,000 steps. For BootDQN we use ensemble of 10 heads and do not use  $\varepsilon$ -greedy exploration.
Figure 4: Cumulative rewards (higher is better) of Adaptive-RandQL and several baselines in ball environments with increasing exploration difficulty: Upper Left displays Level 1, Upper Right shows Level 2, Down shows Level 3. We show average and error bars over 4 seeds.
Algorithm	Episode time (second)
Adaptive-RandQL	5.780e-02
Adaptive-QL	4.213e-02
Kernel-UCBVI	1.523e-01
Table 3: Average time of one episode in second (averaged over 20000 episodes).
Results Figure 4 shows the results of non-tabular experiments. Overall, we see that Adaptive-RandQL outperforms Adaptive-QL in all environments, especially in the sparse reward setting. However, we see that model-based algorithm is much more sample efficient than model-free algorithm, as it was shown by Domingues et al. [2021c]. This is connected to low dimension of the presented environment, where the difference in theoretical regret bounds is not so large. However, this performance come at the price of 3-times larger time complexity, see Table 3.
Regarding the comparison to neural-network based algorithms, we see that approaches based on adaptive discretization always outperforms DQN and BootDQN on an environment with non-sparse rewards. We connect this phenomenon to the fact that neural network algorithms are solving two problems at the same time: exploration and optimization, whereas discretization-based approaches solve only exploration problem.
In the setup of sparse rewards it turns out that neural network-based approaches are competitive with Adaptive-QL and Adaptive-RandQL. Notably, DQN shows itself as the worst one, whereas Adaptive-RandQL and BootDQN show similar performance, additionally justifying exploration effect of ensemble learning and randomized exploration.