SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Assumptions 3-4 hold in many models of interest; see Appendix E. As average cost optimality is our design criterion, we need to ensure the existence of solutions to ACOE when  $\Pi$  is the set of all policies, or Poisson equation when  $\Pi$  is a subset of all policies. We discuss these two cases separately. Case 1:  $\Pi$  is the set of all policies. For any parameter  $\theta \in \Theta$ , the MDP  $(\mathcal{X},\mathcal{A},c,P_{\theta})$  is said to satisfy the ACOE if there exists a constant  $J(\theta)$  and a unique function  $v(\cdot;\theta): \mathcal{X} \to \mathbb{R}$  such that
$$
J (\theta) + v (\boldsymbol {x}; \theta) = \min  _ {a \in \mathcal {A}} \left\{c (\boldsymbol {x}, a) + \sum_ {\boldsymbol {y} \in \mathcal {X}} P _ {\theta} (\boldsymbol {y} | \boldsymbol {x}, a) v (\boldsymbol {y}; \theta) \right\} \text {w i t h} v (0 ^ {d}; \theta) = 0.
$$
From [13] if the following conditions hold, ACOE has a solution,  $J_{\theta}$  is the optimal infinite-horizon average cost, and there is an optimal stationary policy with ACOE becoming (5): (i) for every  $(\pmb{x},a)$  and  $z \geq 0$ , cost function  $c(\pmb{x},a) \geq z$  outside a finite subset of  $\mathcal{X}$ ; (ii) there is a stationary policy with an irreducible and aperiodic Markov process with finite average cost; and (iii) from every  $(\pmb{x},a)$  transition to a finite number of states is possible. From Assumptions 1-3, the above conditions hold. Case 2:  $\Pi$  is a proper subset of all policies. Here, we posit that for every  $\theta \in \Theta$  and its best in-class policy  $\pi_{\theta}^{*}$ , there exists a constant  $J(\theta)$ , the average cost of  $\pi_{\theta}^{*}$ , and a function  $v(\cdot;\theta): \mathcal{X} \to \mathbb{R}$  with
$$
J (\theta) + v (\boldsymbol {x}; \theta) = c \left(\boldsymbol {x}, \pi_ {\theta} ^ {*} (\boldsymbol {x})\right) + \sum_ {\boldsymbol {y} \in \mathcal {X}} P _ {\theta} \left(\boldsymbol {y} \mid \boldsymbol {x}, \pi_ {\theta} ^ {*} (\boldsymbol {x})\right) v (\boldsymbol {y}; \theta). \tag {5}
$$
This holds by the solution of the Poisson equation with the appropriate forcing function. For a Markov process  $X$  on the space  $\mathcal{X}$  with transition kernel  $P$  and cost function  $\bar{c}(\cdot)$ , a solution to the Poisson equation [41] is a scalar  $J$  and function  $v(\cdot): \mathcal{X} \mapsto \mathbb{R}$  such that  $J + v = \bar{c} + Pv$ , where  $v(z) = 0$  for some  $z \in \mathcal{X}$ . In our setting using [41, Sections 9.6-9.8], for a model governed by  $\theta \in \Theta$  following policy  $\pi_{\theta}^{*}$ , we show a solution to the Poisson equation exists and is given by  $v^{\pi_{\theta}^{*}}(0^{d}) = 0$ , and
$$
J (\theta) = \bar {C} ^ {\pi_ {\theta} ^ {*}} \left(0 ^ {d}\right) / \mathbb {E} _ {0 ^ {d}} ^ {\pi_ {\theta} ^ {*}} \left[ \tau_ {0 ^ {d}} \right] \text {a n d} v ^ {\pi_ {\theta} ^ {*}} (\boldsymbol {x}) = \bar {C} ^ {\pi_ {\theta} ^ {*}} (\boldsymbol {x}) - J (\theta) \mathbb {E} _ {\boldsymbol {x}} ^ {\pi_ {\theta} ^ {*}} \left[ \tau_ {0 ^ {d}} \right], \quad \forall \boldsymbol {x} \in \mathcal {X}, \tag {6}
$$
where  $\bar{C}^{\pi_{\theta}^{*}}(\pmb{x}) = \mathbb{E}_{\pmb{x}}^{\pi_{\theta}^{*}}\left[\sum_{i=0}^{\tau_{0}d-1}c(\pmb{X}(i),\pi_{\theta}^{*}(\pmb{X}(i)))\right]$ , and expectation is over trajectories of Markov chain  $\pmb{X}$  with transition kernel  $P_{\theta}^{\pi_{\theta}^{*}}$  starting in state  $\pmb{x}$ . In Appendix A.3, we present related definitions and show that from Assumptions 3-4, the requirements for the existence and finiteness of the solutions to Poisson equation are satisfied. Finally, we assume  $\sup_{\theta \in \Theta}J(\theta)$  is finite, which typically holds as a result of the boundedness assumptions stated in Assumptions 3 or 4, along with Assumption 1.
Algorithm 1 Thompson Sampling with Dynamically-sized Episodes (TSDE)
1: Input:  $\nu_{0}$   
2: Initialization:  $X(1) = 0^{d}, t \gets 1$   
3: for episodes  $k = 1, 2, \ldots$  do  
4:  $t_{k} \gets t$   
5: Generate  $\theta_{k} \sim \nu_{t_{k}}$   
6: while  $t \leq t_{k} + \tilde{T}_{k-1}$  and  $N_{t}(\boldsymbol{x}, a) \leq 2 N_{t_{k}}(\boldsymbol{x}, a)$  for all  $(\boldsymbol{x}, a) \in \mathcal{X} \times \mathcal{A}$  do  
7: Apply action  $A(t) = \pi_{\theta_{k}}^{*}(\boldsymbol{X}(t))$   
8:  $N_{t}(\boldsymbol{X}(t), A(t)) \gets N_{t}(\boldsymbol{X}(t), A(t)) + 1$   
9: Observe new state  $\boldsymbol{X}(t+1)$   
10: Update  $\nu_{t+1}$  according to (7)  
11:  $t \gets t+1$   
12: end while  
13:  $\tilde{T}_{k} \gets t - t_{k}$   
14: while  $\boldsymbol{X}(t) \neq 0^{d}$  do  
15: Apply action  $A(t) = \pi_{\theta_{k}}^{*}(\boldsymbol{X}(t))$   
16: Observe new state  $\boldsymbol{X}(t+1)$   
17: end while  
18:  $T_{k} \gets t - t_{k}$   
19: end for
Remark 2. In Assumption 4 we can use any other policy  $\pi_{\theta_2}$  such that the Markov process obtained from MDP  $(\mathcal{X},\mathcal{A},c,P_{\theta_1})$  by following policy  $\pi_{\theta_2}$  is irreducible and polynomially ergodic via the Foster-Lyapunov criteria with the uniformity discussed. Irreducibility is important as the policy will be used at times when the state is not known in advance, specifically at Steps 14-17 in Algorithm 1.
Assumption 5. We assume that  $J^{*} \coloneqq \sup_{\theta \in \Theta} J(\theta) < +\infty$ .
3 Thompson sampling based learning algorithm
We will use the learning algorithm Thompson sampling with dynamically-sized episodes from [49] to learn the unknown parameter  $\theta^{*} \in \Theta$  and the corresponding policy,  $\pi_{\theta^{*}}^{*}$ , but suitably modify it for our countable state-space setting. Consider the prior distribution  $\nu_{0} = \nu$  defined on  $\Theta$  from which  $\theta^{*}$  is sampled. At each time  $t \in \mathbb{N}$ , the posterior distribution  $\nu_{t}$  is updated according to Bayes' rule as
$$
\nu_ {t + 1} (d \theta) = \frac {\mathbb {P} _ {\theta} (\boldsymbol {X} (t + 1) \mid \boldsymbol {X} (t) , A (t)) \nu_ {t} (d \theta)}{\int_ {\theta^ {\prime} \in \Theta} \mathbb {P} _ {\theta^ {\prime}} (\boldsymbol {X} (t + 1) \mid \boldsymbol {X} (t) , A (t)) \nu_ {t} (d \theta^ {\prime})}, \tag {7}
$$
and the posterior estimate  $\theta_{t + 1}$ , if generated, is from the posterior distribution  $\nu_{t + 1}$ . The modified Thompson-sampling with dynamically-sized episodes algorithm (TSDE) is presented in Algorithm 1. The TSDE algorithm operates in episodes: at the beginning of episode  $k$ , parameter  $\theta_{k}$  is sampled from the posterior distribution  $\nu_{t_k}$  and during episode  $k$ , actions are generated from the stationary policy according to  $\theta_{k}$ , i.e.,  $\pi_{\theta_k}^*$ . Let  $t_k$  be the time the  $k$ -th episode begins. Define  $\tilde{t}_{k + 1}$  as the first time after  $t_k$  that the conditions of Line 6 of Algorithm 1 is triggered and  $t_{k + 1}$  as the first time at or after  $\tilde{t}_{k + 1}$  where state  $0^{d}$  is visited; for the last episode started before or at  $T$ , we ensure that  $t_k$  and  $\tilde{t}_k$  are less than or equal  $T + 1$ . Explicitly,  $t_1 = 1$  and for  $k > 1$ ,  $t_k = \min \{t \geq \tilde{t}_k : X(t) = 0^d \text{ or } t > T\}$ . Let  $T_k = t_{k + 1} - t_k$  be the length of the  $k$ -th episode and set  $\tilde{T}_k = \tilde{t}_{k + 1} - t_k$  with the convention  $\tilde{T}_0 = 1$ . For any state-action pair  $(x, a)$ , we define  $N_1(x, a) = 0$  and for  $t > 1$ ,
$$
N _ {t} (\boldsymbol {x}, a) = \left| \left\{t _ {k} \leq i <   \tilde {t} _ {k + 1} \leq t \text {f o r s o m e} k \geq 1: (\boldsymbol {X} (i), A (i)) = (\boldsymbol {x}, a) \right\} \right|.
$$
Notice that for all state-action pairs  $(\pmb{x}, a)$  and  $\tilde{t}_{k+1} \leq t \leq t_{k+1}$ , we have  $N_t(\pmb{x}, a) = N_{\tilde{t}_{k+1}}(\pmb{x}, a)$ . We denote  $K_T$  as the number of episodes started by or at time  $T$ , or  $K_T = \max \{k : t_k \leq T\}$ . The length of episode  $k < K_T$  is not fixed and is determined according to two stopping criteria: (1)  $t > t_k + \tilde{T}_{k-1}$ , (2)  $N_t(\pmb{x}, a) > 2N_{t_k}(\pmb{x}, a)$  for some state-action pair  $(\pmb{x}, a)$ . After either criterion is met, the system will still follow policy  $\pi_{\theta_k}^*$  until the first time at which state  $0^d$  is visited; see Line 14 and Figure 1. We use this settling period to  $0^d$  because the system state can be arbitrary when the
Figure 1: MDP evolution in episode  $k < K_{T}$ .
first stopping criterion is met. As the countable state-space setting precludes a simple union-bound argument to overcome this uncertainty (as in the literature for finite state settings), we let the system reach the special state  $0^d$ . Another (essentially equivalent) option is to wait until the state hits the finite set  $C_*^g$  or  $C_*^p$  and then use a union bound argument for all states in either set. For analytical convenience, we only use the state samples observed before arrival  $\tilde{t}_{k+1}$  to update the posterior distribution. The posterior update is halted during the settling period to  $0^d$  as we have no control on the states visited during it, despite it being finite in duration (by our assumptions).
4 Regret analysis of Algorithm 1
The performance of any learning policy  $\pi_L$  is evaluated using the metric of expected regret compared to the optimal expected average cost of true parameter  $\theta^*$ , namely,  $J(\theta^*)$ . In this section, we evaluate the performance of Algorithm 1 and derive an upper bound for  $R(T, \pi_{TSDE})$ , its expected regret up to time  $T$ . In Section 2, we argued that at time  $t$  in episode  $k$  ( $t_k \leq t < t_{k+1}$ ), there exist a constant  $J(\theta_k)$  and a unique function  $v(\cdot; \theta_k): \mathcal{X} \to \mathbb{R}$  such that  $v(0^d; \theta_k) = 0$  and
$$
J \left(\theta_ {k}\right) + v (\boldsymbol {X} (t); \theta_ {k}) = c \left(\boldsymbol {X} (t), \pi_ {\theta_ {k}} ^ {*} (\boldsymbol {X} (t))\right) + \sum_ {\boldsymbol {y} \in \mathcal {X}} P _ {\theta_ {k}} (\boldsymbol {y} | \boldsymbol {X} (t), \pi_ {\theta_ {k}} ^ {*} (\boldsymbol {X} (t))) v (\boldsymbol {y}; \theta_ {k}), \tag {8}
$$
in which  $\pi_{\theta_k}^*$  is the optimal or best-in-class policy (depending on the context) according to parameter  $\theta_{k}$  and  $J(\theta_{k})$  is the average cost for the Markov process obtained from MDP  $(\mathcal{X},\mathcal{A},c,P_{\theta_k})$  by following  $\pi_{\theta_k}^*$ . We derive a bound for the expected regret  $R(T,\pi_{TSDE})$  following the proof steps of [49] while extending it to the countable state-space setting of our problem. Using (8), the regret is decomposed into three terms and each term is bounded separately:
$$
R (T, \pi_ {T S D E}) = \mathbb {E} \left[ \sum_ {k = 1} ^ {K _ {T}} \sum_ {t = t _ {k}} ^ {t _ {k + 1} - 1} c (\boldsymbol {X} (t), \pi_ {\theta_ {k}} ^ {*} (\boldsymbol {X} (t))) \right] - T \mathbb {E} \left[ J \left(\boldsymbol {\theta} ^ {*}\right) \right] = R _ {0} + R _ {1} + R _ {2}, \tag {9}
$$
$$
\text {w i t h} R _ {0} = \mathbb {E} \left[ \sum_ {k = 1} ^ {K _ {T}} T _ {k} J \left(\theta_ {k}\right) \right] - T \mathbb {E} \left[ J \left(\boldsymbol {\theta} ^ {*}\right) \right], \tag {10}
$$
$$
R _ {1} = \mathbb {E} \left[ \sum_ {k = 1} ^ {K _ {T}} \sum_ {t = t _ {k}} ^ {t _ {k + 1} - 1} \left[ v (\boldsymbol {X} (t); \theta_ {k}) - v (\boldsymbol {X} (t + 1); \theta_ {k}) \right] \right], \tag {11}
$$
$$
R _ {2} = \mathbb {E} \left[ \sum_ {k = 1} ^ {K _ {T}} \sum_ {t = t _ {k}} ^ {t _ {k + 1} - 1} \left[ v (\boldsymbol {X} (t + 1); \theta_ {k}) - \sum_ {\boldsymbol {y} \in \mathcal {X}} P _ {\theta_ {k}} (\boldsymbol {y} | \boldsymbol {X} (t), \pi_ {\theta_ {k}} ^ {*} (\boldsymbol {X} (t))) v (\boldsymbol {y}; \theta_ {k}) \right] \right]. \tag {12}
$$
Before bounding the above regret terms, we address the complexities arising from the countable state-space setting. Firstly, we need to study the maximum state (with respect to the  $\ell_{\infty}$ -norm) visited up to time  $T$  in the MDP  $(\mathcal{X},\mathcal{A},c,P_{\theta^{*}})$  following Algorithm 1; we denote this maximum state by  $M_{\theta^{*}}^{T}$ . In Appendix C, we derive upper bounds on the moments of hitting times of state  $0^{d}$  and utilize this to bound the moments of random variable  $M_{\theta^{*}}^{T}$ , which then lets us study the number of episodes  $K_{T}$  by time  $T$ . Another challenge in analyzing the regret is that the relative value function  $v(\pmb{x};\theta)$  is unlikely to be bounded in the countable state-space setting. Hence, in (13) and (14), we find bounds for the relative value function in terms of hitting time  $\tau_{0^d}$  from the initial state  $\pmb{x}$ . Based on these results, we provide an upper bound for the regret of Algorithm 1 in Theorem 1.
Maximum state norm under polynomial and geometric ergodicity. Here we state the results that characterize the maximum  $l_{\infty}$ -norm of the state vector achieved up until and including time  $T$ , and
the resulting bounds on the number of episodes executed until time  $T$ . Owing to space constraints the details (including formal statements) are presented in Appendix B. The results are listed as below: (i) In Lemma 6, we bound the moments of the maximum length of recurrence times of state  $0^d$ , using the ergodicity assumptions 3 and 4. This, along with the skip-free property, allows us to prove that the  $p$ -th moment of  $\max_{1\leq i\leq T}\tau_{0^d}^{(i)}$  and  $M_{\pmb{\theta}^*}^T$  are both of order  $O(\log^p T)$ .
(ii) In Lemma 7, we find an upper bound for the number of episodes in which the second stopping criterion is met or there exists a state-action pair for which  $N_{t}(\pmb{x},a)$  has increased more than twice. (iii) In Lemma 8, we bound the total number of episodes  $K_{T}$  by time  $T$  by bounding the number of episodes triggered by the first stopping criterion, using the fact that in such episodes,  $\tilde{T}_k = \tilde{T}_{k - 1} + 1$ . Moreover, to account for the settling time of each episode, we use geometric ergodicity and Lemma 6. It follows that the expected value of the number of episodes  $K_{T}$  is of the order  $\tilde{O} (\sqrt{h^d|\mathcal{A}|T})$ .
Regret analysis. Next, we bound regret terms  $R_0$ ,  $R_1$  and  $R_2$  using the approach of [49] along with additional arguments to extend their result to a countably infinite state-space. We consider the relative value function  $v(\pmb{x};\theta)$  of policy  $\pi_{\theta}^{*}$  introduced for the optimal policy in ACOE or for the best in-class policy in the Poisson equation. In either of these cases, policy  $\pi_{\theta}^{*}$  satisfies (5), which is the corresponding Poisson equation with forcing function  $c(\pmb{x},\pi_{\theta}^{*}(\pmb{x}))$  in a Markov chain with transition matrix  $P_{\theta}^{\pi_{\theta}^{*}}$ . In (6), we presented the solution  $(J,v)$  to the Poisson equation, which yields the following upper bound for the relative value function, as argued in Appendix A.3:
$$
v (\boldsymbol {x}; \theta) \leq \bar {C} ^ {\pi_ {\theta} ^ {*}} (\boldsymbol {x}) \leq \mathbb {E} _ {\boldsymbol {x}} ^ {\pi_ {\theta} ^ {*}} \left[ K d \left(\| \boldsymbol {x} \| _ {\infty} + h \tau_ {0 ^ {d}}\right) ^ {r} \tau_ {0 ^ {d}} \right]. \tag {13}
$$
We can similarly lower bound the relative value function using Assumption 5 as
$$
v (\boldsymbol {x}; \theta) \geq - J (\theta) \mathbb {E} _ {\boldsymbol {x}} ^ {\pi_ {\theta} ^ {*}} \left[ \tau_ {0 ^ {d}} \right] \geq - J ^ {*} \mathbb {E} _ {\boldsymbol {x}} ^ {\pi_ {\theta} ^ {*}} \left[ \tau_ {0 ^ {d}} \right]. \tag {14}
$$
From Assumption 3, all moments of  $\tau_{0^d}$  and thus, the derived bounds are finite. Also, in Lemma 10 we bound the moments of  $\tau_{0^d}$  of order  $i \leq r + 1$  using the polynomial Lyapunov function  $V_{\theta_1,\theta_2}^p$ , which is then used to bound the expected regret. We next bound the first regret term  $R_0$  from the first stopping criterion in terms of the number of episodes  $K_T$  and the settling time of each episode  $k$ .
Lemma 1. The first regret term  $R_0$  satisfies  $R_0 \leq J^* \mathbb{E}[K_T(\max_{1 \leq i \leq T} \tau_{0^d}^{(i)} + 1)]$ .
Proof of Lemma 1 is given in Appendix B.4. From Lemma 6, all moments of  $\max_{1\leq i\leq T}\tau_{0^d}^{(i)}$  are bounded by a polylogarithmic function. Furthermore, as a result of Lemma 8, expected value of the number of episodes  $K_{T}$  is of the order  $\tilde{O} (\sqrt{h^d|\mathcal{A}|T})$ , which leads to a  $\tilde{O} (\sqrt{h^d|\mathcal{A}|T})$  regret term  $R_0$ . Next, an upper bound on  $R_{1}$  defined in (11) is derived. In the proof of Lemma 2 we argue that as the relative value function is equal to 0 at all time instances  $t_k$  for  $k\le K_{T}$ , the only term that contributes to the regret is the value function at the end of time horizon  $T$ . We use the lower bound derived in (14) to show that the second regret term  $R_{1}$  is  $\tilde{O}(1)$ ; the proof is given in Appendix B.5.
Lemma 2. The second regret term  $R_{1}$  satisfies  $R_{1} \leq c_{2}\mathbb{E}[(M_{\pmb{\theta}^{*}}^{T})^{r_{*}^{p}}] + c_{3}$ , where  $c_{2} = J^{*}2^{r_{*}^{p}}s_{*}^{p}(\beta_{*}^{p})^{-1}$  and  $c_{3} = J^{*}(\beta_{*}^{p})^{-1}\big(s_{*}^{p}(2h)^{r_{*}^{p}} + b_{*}^{p}(K_{*})^{-1}\big)$ .
From Lemma 6,  $\mathbb{E}[(M_{\theta^*}^T)^{r_*^p}]$  is  $O(\log^{r_*^p}T)$ ; hence,  $R_1$  is upper bounded by a polylogarithmic function of the order  $r_*^p$ . Finally, in Lemma 3, we derive an upper bound for the third regret term  $R_2$  defined in (12) using the bound derived for the relative value function in (13). To bound  $R_2$ , we characterize it in terms of the difference between the empirical and true unknown transition kernel and following the concentration method used in [64, 10, 49, 7, 8], we argue that with high probability the total variation distance between the two distributions is small; for proof, see Appendix B.6.
Lemma 3. For problem-dependent constant  $c_{p_3}$  and polynomial  $Q(T) = c_{p_3}(Th)^{r + r_*^p} / 48$ , we have
$$
R_{2}\leq (\log (hT + h) + 1)^{d} + c_{p_{3}}\sqrt{|\mathcal{A}|T}\log_{2}\left(2|\mathcal{A}|T^{2}Q(T)\right)\mathbb{E}\left[ \left(M_{\boldsymbol{\theta}^{*}}^{T} + h\right)^{d + r + r_{*}^{p}}\Bigl(\max_{1\leq i\leq T}\tau_{0^{d}}^{(i)}\Bigr)\right].
$$
The above Lemma results in a  $\tilde{O}(KrdJ^{*}h^{d+2r + r_{*}^{p}}\sqrt{|\mathcal{A}|T})$  regret term as a result of Lemma 6, where  $h$  is the skip-free parameter defined in Assumption 2,  $d$  is the dimension of the state-space,  $K$  and  $r$  are the cost function parameters defined in Assumption 1,  $J^{*}$  is the supremum on the optimal cost,  $r_{*}^{p}$  is defined in Assumption 4, and where  $\tilde{O}$  hides logarithmic factors in problem parameters one of which is  $\log^{d+r + r_{*}^{p} + 2}(T)$ . For simplicity, we have not included the Lyapunov
functions related parameters in the regret. Finally, from Lemmas 1, 2, 3, along with the Cauchy-Schwarz inequality, we conclude that the regret of Algorithm  $1$ $R(T,\pi_{TSDE})(= R_0 + R_1 + R_2)$  is  $\tilde{O} (KrdJ^{*}h^{d + 2r + r^{p}}\sqrt{|\mathcal{A}|T})$ ; for brevity, we will state that regret is of the order  $\tilde{O} (dh^d\sqrt{|\mathcal{A}|T})$ .
Theorem 1. Under Assumptions 1-5, the regret of Algorithm 1,  $R(T,\pi_{TSDE})$ , is  $\tilde{O} (dh^{d}\sqrt{|\mathcal{A}|T})$ .
Theorem 1 can be extended to the problem of finding the best policy within a sub-class of policies in set  $\Pi$ , which may or may not contain the optimal policy. In Section 2, we stated that Assumptions 3 and 4 hold for policies in  $\Pi$  and we used this to argue that the Poisson equation has a solution given in (6). As a result, repeating the same arguments as in Theorem 1 with the modification that  $\pi_{\theta}^{*}$  is the best in-class policy of the MDP governed by parameter  $\theta$ , yields the following corollary.
Corollary 1. Under Assumptions 1 through 5, the regret of Algorithm 1 when using the best in-class policy is  $\tilde{O}(dh^d\sqrt{|\mathcal{A}|T})$ .
Requirement of an optimal policy oracle. To implement our algorithm, we need to find the optimal policy for each model sampled by the algorithm—optimal policy for Theorem 1 and optimal policy within policy class  $\Pi$  for Corollary 1. In the finite state-space setting, [49] provides a schedule of  $\epsilon$  values and selects  $\epsilon$ -optimal policies to obtain  $\tilde{O}(\sqrt{T})$  regret guarantees. The issue with extending the analysis of [49] to the countable state-space setting is that we need to ensure (uniform) ergodicity for the chosen  $\epsilon$ -optimal policies. Another issue is that, to the best of our knowledge, there isn't a general structural characterization of all  $\epsilon$ -optimal stationary policies for countable state-space MDPs or even a characterization of the policy within this set that is selected by any computational procedure in the literature; current results only discuss characterization of the stationary optimal policy. In the absence of such results, stability assumptions with the same uniformity across models as in our submission will be needed, which are likely too strong to be useful. However, if we could verify the stability requirements of Assumptions 3 and 4 for a subset of policies, the optimal oracle is not needed, and instead, by choosing approximately optimal policies within this subset, we can follow the same proof steps as [49] to guarantee regret performance similar to Corollary 1 (without knowledge of model parameters). Thus, in Theorem 2 we extend the previous regret guarantees to the algorithm employing  $\epsilon$ -optimal policy; proof is given in Appendix B.8.
Theorem 2. Consider a non-negative sequence  $\{\epsilon_k\}_{k=1}^{\infty}$  such that for every  $k \in \mathbb{N}$ ,  $\epsilon_k$  is bounded above by  $\frac{1}{k+1}$  and an  $\epsilon_k$ -optimal policy satisfying Assumptions 3 and 4 is given. The regret incurred by Algorithm 1 while using the  $\epsilon_k$ -optimal policy during any episode  $k$  is  $\tilde{O}(dh^d \sqrt{|\mathcal{A}|T})$ .
5 Evaluation and Conclusion: Application of Algorithm 1 to queueing models
Next, we present an evaluation of our algorithm. We study two different queueing models shown in Figure 2, each with Poisson arrivals at rate  $\lambda$ , and two heterogeneous servers with exponentially distributed services times with unknown service rate vector  $\pmb{\theta}^{*} = (\theta_{1}^{*},\theta_{2}^{*})$ . Vector  $\pmb{\theta}^{*}$  is sampled from the prior distribution  $\nu$  defined on the space  $\Theta$  given as  $\Theta = \left\{(\theta_1,\theta_2)\in \mathbb{R}_+^2:\frac{\lambda}{\theta_1 + \theta_2}\leq \frac{1 - \delta}{1 + \delta},1\leq \frac{\theta_1}{\theta_2}\leq R\right\}$ , for fixed  $R\geq 1$  and  $\delta \in (0,0.5)$ . The first condition ensures the stability of the queueing models, while the second guarantees the compactness of the parameter space of the parameterized policies. In both systems, the goal of the dispatcher is to minimize the expected sojourn time of jobs, which by Little's law [52] is equivalent to minimizing the average number of jobs in the system. After verifying Assumptions 1-5 in Appendix E for the cost function  $c(\pmb {x}) = \| \pmb {x}\| _1$ , Theorem 1 yields a Bayesian regret of order  $\tilde{O} (\sqrt{|\mathcal{A}|T})$  for Algorithm 1.