SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Model 1. Two-server queueing system with a common buffer. We consider the continuous-time queueing system of Figure 2a, where the countable state space is  $\mathcal{X} = \{\pmb{x} = (x_0,x_1,x_2)\in$ $\mathbb{Z}_{+}\times \{0,1\}^{2}\}$ , where  $x_0$  is the queue length, and  $x_{i}, i = 1,2$  equal 1 if server  $i$  is busy. The action space is  $\mathcal{A} = \{h,b,1,2\}$ , where  $h$  means no action,  $b$  sends a job to both servers, and  $i = 1,2$  assigns a job to server  $i$ . In [38], it is shown that by uniformization [39] and sampling the continuous-time Markov process at rate  $\lambda +\theta_1^* +\theta_2^*$ , a discrete-time Markov chain is obtained, which converts the original continuous-time problem to an equivalent discrete-time problem where we need to minimize  $\lim \sup_{T\to \infty}T^{-1}\sum_{t = 0}^{T - 1}\| X(t)\| _1$ . Further, [38] shows that the optimal policy is a threshold policy  $\pi_{t(\pmb{\theta}^{*})}$  with optimal finite threshold  $t(\pmb{\theta}^{*})\in \mathbb{N}$ : always assign a job to the faster (first) server when free, and to the second server if it is free and  $\| x\| _1 > t(\pmb{\theta}^*)$ , and take no action otherwise. In
(a) Queueing system with a common buffer.
(b) Two parallel queues.
Figure 2: Two-server queueing systems with heterogeneous service rates.
(a) Queueing system of Figure 2a.
(b) Queueing system of Figure 2b.
Figure 3: Regret performance for  $\lambda = 0.3, 0.5, 0.7$ . Shaded region shows the  $\pm \sigma$  area of mean regret.
Appendix E.1, we argue that the discrete-time Markov process governed by  $\theta \in \Theta$  and following threshold policy  $\pi_t$  for any threshold  $t$  belonging to a compact set satisfies Assumptions 1-5.
Model 2. Two heterogeneous parallel queues. We consider the continuous-time queueing system of Figure 2b with countable state space  $\mathcal{X} = \{\pmb{x} = (x_1, x_2) \in \mathbb{Z}_+^2\}$ , where  $x_i$  is the number of jobs in the server-queue pair  $i$ . The action space is  $\mathcal{A} = \{1, 2\}$ , where action  $i$  sends the arrival to queue  $i$ . We obtain the discrete-time MDP by sampling the queueing system at the arrivals, and then aim to find the average cost minimizing policy within the class  $\Pi = \{\pi_\omega; \omega \in [(c_R R)^{-1}, c_R R]\}$ ,  $c_R \geq 1$ . Policy  $\pi_\omega: \mathcal{X} \to \mathcal{A}$  routes arrivals based on the weighted queue lengths:  $\pi_\omega(\pmb{x}) = \arg \min \left(1 + x_1, \omega(1 + x_2)\right)$  with ties broken for 1. Even with the transition kernel fully specified (by the values of arrival and service rates), the optimal policy in  $\Pi$  is not known except when  $\theta_1 = \theta_2$  where the optimal value is  $\omega = 1$ , and so, to learn it, we will use Proximal Policy Optimization for countable state-space MDPs [18]. Note that [18] requires full model knowledge, which holds in our scheme as we use parameters sampled from the posterior for choosing the policy at the beginning of each episode. In Appendix E.2, we argue that the discrete-time Markov process governed by parameter  $\theta \in \Theta$  and following policy  $\pi_\omega$  for  $\omega \in [(c_R R)^{-1}, c_R R]$  satisfies Assumptions 1-5.
Next, we report the numerical results of Algorithm 1 in the two queueing models of Figure 2 and calculate regret using (2). The regret is averaged over 2000 simulation runs and plotted against the number of transitions in the sampled discrete-time Markov process. Figure 3 shows the behavior of the regret of the two queuing models for three different arrival rates and service rates distributed according to a Dirichlet prior over  $[0.5, 1.9]^2$ . We observe that the regret is sub-linear in time and grows as the arrival rate increases. For the queuing model of Figure 2a, the minimum average cost  $J(\theta)$  and optimal policy  $\pi_{\theta}^{*}$  are known explicitly [38] for every  $\theta \in \Theta$ , which are used in Algorithm 1 and for regret calculation. Conversely, for the second queueing model,  $J(\theta)$  and  $\pi_{\theta}^{*}$  are not known. The PPO algorithm [18] is used to empirically find both the optimal weight and the policy's average cost. Additional details of the simulations and more plots are presented in Appendix G.
Conclusions and future work. We studied the problem of learning optimal policies in countable state-space MDPs governed by unknown parameters. We proposed a learning policy based on Thompson sampling and established finite-time performance guarantees on the Bayesian regret. We highlighted the practicality of our proposed algorithm by considering two different queuing models and showing that our algorithm can be applied to develop optimal control policies. For future work we plan two directions to explore: to generalize our algorithm to consider polices that might not all be stabilizing, and also to simplify the algorithm using ideas from [61, 57].
Disclosure of Funding
SA's research was supported by NSF via grants ECCS2038416, CCF2008130, and CNS1955777, and a grant from General Dynamics via MIDAS at the University of Michigan, Ann Arbor. VS's research is supported in part by NSF via grants ECCS2038416, CCF2008130, CNS1955777, and CMMI2240981.
[58] Leandros Tassiulas and Anthony Ephremides. Jointly optimal routing and scheduling in packet ratio networks. IEEE Transactions on Information Theory, 38(1):165-168, 1992.
[59] Leandros Tassiulas and Anthony Ephremides. Dynamic server allocation to parallel queues with randomly varying connectivity. IEEE Transactions on Information Theory, 39(2):466-478, 1993.
[60] Georgios Theocharous, Zheng Wen, Yasin Abbasi-Yadkori, and Nikos Vlassis. Posterior sampling for large scale reinforcement learning. arXiv preprint arXiv:1711.07979, 2017.
[61] Georgios Theocharous, Zheng Wen, Yasin Abbasi Yadjkori, and Nikos Vlassis. Scalar posterior sampling with applications. Advances in Neural Information Processing Systems, 31, 2018.
[62] William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3-4):285-294, 1933.
[63] Neil Walton and Kuang Xu. Learning and information in stochastic networks and queues. In Tutorials in Operations Research: Emerging Optimization Methods and Modeling Techniques with Applications, pages 161-198. INFORMS, 2021.
[64] Tsachy Weissman, Erik Ordentlich, Gadiel Seroussi, Sergio Verdu, and Marcelo J Weinberger. Inequalities for the  $L_{1}$  deviation of the empirical distribution. Hewlett-Packard Labs, Tech. Rep, 2003.
[65] Zixian Yang, R Srikant, and Lei Ying. Learning while scheduling in multi-server systems with unknown statistics: Maxweight with discounted UCB. In International Conference on Artificial Intelligence and Statistics, pages 4275-4312. PMLR, 2023.
[66] Andrea Zanette, David Brandfonbrener, Emma Brunskill, Matteo Pirotta, and Alessandro Lazaric. Frequentist regret bounds for randomized least-squares value iteration. In International Conference on Artificial Intelligence and Statistics, pages 1954-1964. PMLR, 2020.
A Proofs related to problem formulation
A.1 Ergodicity definitions
Suppose that Markov process  $X$  on  $\mathcal{X}$  with transition kernel  $P$  is irreducible, aperiodic and positive recurrent with stationary distribution  $\mu$  and let  $f:\mathcal{X}\mapsto [1,\infty)$  be a measurable function such that  $\mu (f)\coloneqq \mathbb{E}_{\mu}[f(Y)] < + \infty$  with  $Y\sim \mu$ . We are interested in conditions under which for a sequence of positive numbers  $\rho \coloneqq (\rho (n))_{n\geq 0}$ ,
$$
\lim  _ {n \rightarrow \infty} \rho (n) \| P ^ {n} (\boldsymbol {x}, \cdot) - \mu (\cdot) \| _ {f} = 0, \quad \forall \boldsymbol {x} \in \mathcal {X}, \tag {15}
$$
where for a signed measure  $\tilde{\mu}$  on  $\mathcal{X}$ ,  $\| \tilde{\mu} \|_f \coloneqq \sup_{|g| \leq f} |\tilde{\mu}(g)|$ . The sequence  $\rho$  is interpreted as the rate function, and three different notions of ergodicity are distinguished based on the following rate functions:  $\rho(n) \equiv 1$ ,  $\rho(n) = \zeta^n$  for  $\zeta > 1$ , and  $\rho(n) = n^{\zeta - 1}$  for  $\zeta \geq 1$ . Further, for each rate function  $\rho$ , we state the Foster-Lyapunov characterization of ergodicity of the Markov process  $X$ , which provides sufficient conditions for (15) to hold.
1. If  $\rho(n) \equiv 1$  for all  $n \geq 0$ , the Markov process  $X$  satisfying (15) is said to be  $f$ -ergodic. From [43], for an irreducible and aperiodic chain,  $f$ -ergodicity is equivalent to the existence of a function  $V: \mathcal{X} \mapsto [0, \infty)$ , a finite set  $C$ , and positive constant  $b$  such that
$$
\Delta V \leq - f + b \mathbb {I} _ {C}, \tag {16}
$$
where  $\Delta V\coloneqq PV - V$  with  $PV(\pmb {x})\coloneqq \sum_{\pmb{x}^{\prime}\in \mathcal{X}}P(\pmb {x},\pmb{x}^{\prime})V(\pmb{x}^{\prime})$ . The drift condition (16) implies positive recurrence of the Markov process, existence of a unique stationary distribution  $\mu$ , and  $\mu (f)\leq b < + \infty$  ([43], Theorem 14.3.7).
2. If  $\rho(n) = \zeta^n$  for some  $\zeta > 1$ , the Markov process  $X$  satisfying (15) is said to be  $f$ -geometrically ergodic. From [43], for an irreducible and aperiodic chain,  $f$ -geometric ergodicity is equivalent to the existence of a function  $V: \mathcal{X} \mapsto [1, \infty)$ , a finite set  $C$ , a constant  $\gamma \in (0, 1)$  and positive constant  $b$  such that
$$
\Delta V \leq - (1 - \gamma) V + b \mathbb {I} _ {C}. \tag {17}
$$
The drift condition (17) implies positive recurrence of the Markov process, existence of a unique stationary distribution  $\mu$ , and  $\mu(V) \leq \frac{b}{1 - \gamma} < +\infty$  ([43], Theorem 14.3.7). Moreover, if  $f(\cdot) \equiv 1$  in (15), then the Markov process  $\mathbf{X}$  is called geometrically ergodic.
3. If  $\rho(n) = n^{\zeta - 1}$  for some  $\zeta \geq 1$ , the Markov process  $X$  satisfying (15) is said to be  $f$ -polynomially ergodic. From [43, 29], for an irreducible and aperiodic chain, the existence of a function  $V: \mathcal{X} \mapsto [1, \infty)$ , a finite set  $C$ , a constant  $\alpha \in [0, 1)$ , and positive constants  $c$  and  $b$  such that
$$
\Delta V \leq - c V ^ {\alpha} + b \mathbb {I} _ {C} \tag {18}
$$
implies  $V_{\zeta}$ -polynomial ergodicity of  $X$  at rate  $\rho(n) = n^{\zeta - 1}$  for all  $\zeta \in [1, 1 / (1 - \alpha)]$  with  $V_{\zeta} = V^{1 - \zeta(1 - \alpha)}$ . The drift condition (18) implies positive recurrence of the Markov process, existence of a unique stationary distribution  $\mu$ , and  $\mu(V^{\alpha}) \leq \frac{b}{c} < +\infty$ .
A.2 Lemma 4
Lemma 4. For any state  $x \neq 0^d$ , there exists constants  $\kappa > 1$  and  $c_1$  such that the following holds for the hitting time of state  $0^d$ ,  $\tau_{0^d}$ ,
$$
\mathbb {E} _ {\boldsymbol {x}} \left[ \kappa^ {\tau_ {0 d}} \right] \leq c _ {1} V ^ {g} (\boldsymbol {x}).
$$
Proof. We define  $\tilde{V} \coloneqq \sum_{n=0}^{\infty} P^{n} V^{g}$  where  $P^{n}$  is the  $n$ -step taboo probability [43] defined as
$$
{ } _ { A } P _ { \boldsymbol { x } B } ^ { n } = \mathbb { P } _ { \boldsymbol { x } } \left( \boldsymbol { X } _ { n } \in B , \tau _ { A } > n \right) ,
$$
for  $A, B \subseteq \mathcal{X}$ , and  $\tau_A$  is the first hitting time of set  $A$ . We also let  ${}_A P_{\boldsymbol{x}B}^0 = \mathbb{I}_B(\boldsymbol{x})$ . We have
$$
\begin{array}{l} { } _ { 0 ^ { d } } P \tilde { V } ( \boldsymbol { x } ) = \sum _ { \boldsymbol { y } \neq 0 ^ { d } } P _ { \boldsymbol { x y } } \tilde { V } ( \boldsymbol { y } ) = \sum _ { n = 0 } ^ { \infty } \sum _ { \boldsymbol { y } , \boldsymbol { z } \neq 0 ^ { d } } P _ { \boldsymbol { x y }   0 ^ { d } } P _ { \boldsymbol { y z } } ^ { n } V ^ { g } ( \boldsymbol { z } ) \\ = \sum_ {n = 0} ^ {\infty} \sum P _ {\boldsymbol {x z}} ^ {n + 1} V ^ {g} (\boldsymbol {z}) = \tilde {V} (\boldsymbol {x}) - V ^ {g} (\boldsymbol {x}). \\ \end{array}
$$
In Appendix D.3, we argue that there exists  $\tilde{b}^g > 1$  such that  $\tilde{V}(\boldsymbol{y}) \leq \tilde{b}^g V^g(\boldsymbol{y})$  for all  $\boldsymbol{y} \in \mathcal{X}$ , which leads to
$$
{ } _ { 0 ^ { d } } P \tilde { V } = \tilde { V } - V ^ { g } \leq \tilde { V } - \frac { 1 } { \tilde { b } ^ { g } } \tilde { V } = \left( 1 - \frac { 1 } { \tilde { b } ^ { g } } \right) \tilde { V } . \tag {19}
$$
Define Lyapunov function
$$
\tilde {V} ^ {g} (\boldsymbol {x}) = \left\{ \begin{array}{l l} (1 + 2 \tilde {b} ^ {g}) \tilde {V} (\boldsymbol {x}), & \text {i f} \boldsymbol {x} \neq 0 ^ {d}, \\ 1 + \left(2 \tilde {b} ^ {g}\right) ^ {- 1}, & \text {i f} \boldsymbol {x} = 0 ^ {d}. \end{array} \right.
$$
From the above equation and (19), we get
$$
\begin{array}{l} P \tilde {V} ^ {g} (\boldsymbol {x}) = \sum_ {\boldsymbol {y} \neq 0 ^ {d}} P _ {\boldsymbol {x y}} \tilde {V} ^ {g} (\boldsymbol {y}) + P _ {\boldsymbol {x 0} ^ {d}} \tilde {V} ^ {g} (0 ^ {d}) \\ = \sum_ {\boldsymbol {y} \neq 0 ^ {d}} P _ {\boldsymbol {x y}} (1 + 2 \tilde {b} ^ {g}) \tilde {V} (\boldsymbol {y}) + P _ {\boldsymbol {x 0} ^ {d}} \left(1 + \frac {1}{2 \tilde {b} ^ {g}}\right) \\ \leq \left(1 - \frac {1}{\tilde {b} ^ {g}}\right) (1 + 2 \tilde {b} ^ {g}) \tilde {V} (\boldsymbol {x}) + 1 + \frac {1}{2 \tilde {b} ^ {g}} \\ \leq \left(1 - \frac {1}{\tilde {b} ^ {g}}\right) (1 + 2 \tilde {b} ^ {g}) \tilde {V} (\boldsymbol {x}) + \left(1 + \frac {1}{2 \tilde {b} ^ {g}}\right) \tilde {V} (\boldsymbol {x}) \\ = \left(1 - \frac {1}{2 \tilde {b} ^ {g}}\right) (1 + 2 \tilde {b} ^ {g}) \tilde {V} (\boldsymbol {x}). \\ \end{array}
$$
$$
P \tilde {V} ^ {g} (\boldsymbol {x}) \leq \left(1 - \frac {1}{2 \tilde {b} ^ {g}}\right) \tilde {V} ^ {g} (\boldsymbol {x}) + \left(1 - \frac {1}{2 \tilde {b} ^ {g}}\right) (1 + 2 \tilde {b} ^ {g}) \tilde {V} (0 ^ {d}) \mathbb {I} _ {0 ^ {d}} (\boldsymbol {x}), \quad \boldsymbol {x} \in \mathcal {X}.
$$
To find an upper bound for  $\mathbb{E}_{\pmb{x}}[\kappa^{\tau_{0d}}]$ , we apply [43, Theorem 15.2.5], which is a generalization of Lemma 12. For any  $1 \leq \kappa \leq \frac{2\tilde{b}^{g}}{2\tilde{b}^{g} - 1}$ , there exists  $\epsilon > 0$  such that
$$
\mathbb {E} _ {\boldsymbol {x}} \left[ \sum_ {i = 0} ^ {\tau_ {0} d - 1} \tilde {V} ^ {g} \left(\boldsymbol {X} _ {i}\right) \kappa^ {i} \right] \leq \epsilon^ {- 1} \kappa^ {- 1} \tilde {V} ^ {g} (\boldsymbol {x}).
$$
As  $\tilde{V}^g (\pmb {y})\geq 1$  for all  $\pmb {y}\in \mathcal{X}$  , we have
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {x}} \left[ \kappa^ {\tau_ {0} d} \right] \leq \kappa \mathbb {E} _ {\boldsymbol {x}} \left[ \sum_ {i = 0} ^ {\tau_ {0} d - 1} \tilde {V} ^ {g} \left(\boldsymbol {X} _ {i}\right) \kappa^ {i} \right] \leq \epsilon^ {- 1} \tilde {V} ^ {g} (\boldsymbol {x}), \\ = \epsilon^ {- 1} \left(1 + 2 \tilde {b} ^ {g}\right) \tilde {V} (\boldsymbol {x}) \leq \tilde {b} ^ {g} \epsilon^ {- 1} \left(1 + 2 \tilde {b} ^ {g}\right) V ^ {g} (\boldsymbol {x}), \\ \end{array}
$$
and the claim holds for any  $\kappa \in [1, \frac{2\tilde{b}^g}{2\tilde{b}^g - 1}]$  and  $c_1 = \tilde{b}^g\epsilon^{-1}\left(1 + 2\tilde{b}^g\right)$ .
A.3 Poisson equation
For an irreducible Markov process on the countably-infinite space  $\mathcal{X}$  with time-homogeneous transition kernel  $P$  and cost function  $\bar{c}(\cdot)$ , a solution pair to the Poisson equation [41] is a scalar  $J$  and function  $v(\cdot): \mathcal{X} \mapsto \mathbb{R}$  such that  $J + v = \bar{c} + Pv$ , where  $v(z) = 0$  for some  $z \in \mathcal{X}$ . If the Markov process is also positive recurrent and  $\mathbb{E}_x\left[\sum_{i=0}^{\tau_y - 1} |\bar{c}(X(i))|\right] < \infty$ , where  $\tau_y$  is the first hitting time of some state  $y \in \mathcal{X}$ , then solution pair  $(J, v)$  given as
$$
J = \frac {\mathbb {E} _ {\boldsymbol {y}} \left[ \sum_ {i = 0} ^ {\tau_ {\boldsymbol {y}} - 1} | \bar {c} (\boldsymbol {X} (i)) | \right]}{\mathbb {E} _ {\boldsymbol {y}} [ \tau_ {\boldsymbol {y}} ]} \text {a n d} v (\boldsymbol {x}) = \mathbb {E} _ {\boldsymbol {y}} \left[ \sum_ {i = 0} ^ {\tau_ {\boldsymbol {x}} - 1} | \bar {c} (\boldsymbol {X} (i)) | \right] - J \mathbb {E} _ {\boldsymbol {x}} [ \tau_ {\boldsymbol {y}} ], \quad \forall \boldsymbol {x} \in \mathcal {X},
$$
is a solution to the Poisson equation  $J + v = \bar{c} + Pv$  with  $v(z) = 0$  [41, Theorem 9.5].
Lemma 5. Consider Markov Decision Processes  $(\mathcal{X},\mathcal{A},c,P_{\theta})$  governed by parameter  $\theta \in \Theta$  following the best-in-class policy  $\pi_{\theta}^{*}$ . Then the pair  $(J(\theta),v^{\pi_{\theta}^{*}})$  given as
$$
J (\theta) := \frac {\bar {C} ^ {\pi_ {\theta} ^ {*}} (0 ^ {d})}{\mathbb {E} _ {0 ^ {d}} ^ {\pi_ {\theta} ^ {*}} [ \tau_ {0 ^ {d}} ]} a n d v ^ {\pi_ {\theta} ^ {*}} (\boldsymbol {x}) = \bar {C} ^ {\pi_ {\theta} ^ {*}} (\boldsymbol {x}) - J (\theta) \mathbb {E} _ {x} ^ {\pi_ {\theta} ^ {*}} [ \tau_ {0 ^ {d}} ], \quad \forall \boldsymbol {x} \in \mathcal {X},
$$
is a solution to the Poisson equation  $v + J = c + P_{\theta}^{\pi_{\theta}^{*}}v$ , where  $v^{\pi_{\theta}^{*}}(0^{d}) = 0$  and  $\bar{C}^{\pi_{\theta}^{*}}(\pmb{x}) = \mathbb{E}_{\pmb{x}}^{\pi_{\theta}^{*}}\Big[\sum_{i=0}^{\tau_{0}d-1}c(\pmb{X}(i),\pi_{\theta}^{*}(\pmb{X}(i)))\Big]$ .
Proof. From [41, Theorem 9.5], a solution pair to the Poisson equation exists if  $\mathbb{E}_{\boldsymbol{x}}^{\pi_{\theta}^{*}}[\tau_{0^d}]$  and  $\bar{C}^{\pi_{\theta}^{*}}(\boldsymbol{x})$  are finite for all  $\boldsymbol{x}\in \mathcal{X}$ . The former follows from positive recurrence assumed in Assumption 3 and for the latter, from Assumptions 1 and 2,
$$
\begin{array}{l} \bar {C} ^ {\pi_ {\theta} ^ {*}} (\boldsymbol {x}) = \mathbb {E} _ {\boldsymbol {x}} ^ {\pi_ {\theta} ^ {*}} \Big [ \sum_ {i = 0} ^ {\tau_ {0} d - 1} c (\boldsymbol {X} (i), \pi_ {\theta} ^ {*} (\boldsymbol {X} (i))) \Big ] \leq \mathbb {E} _ {\boldsymbol {x}} ^ {\pi_ {\theta} ^ {*}} \Big [ \sum_ {i = 0} ^ {\tau_ {0} d - 1} \sum_ {j = 1} ^ {d} K (X _ {j} (i)) ^ {r} \Big ] \\ \leq \mathbb {E} _ {\boldsymbol {x}} ^ {\pi_ {\theta} ^ {*}} \left[ \sum_ {i = 0} ^ {\tau_ {0} d - 1} K d \left(\| \boldsymbol {x} \| _ {\infty} + h i\right) ^ {r} \right] \leq \mathbb {E} _ {\boldsymbol {x}} ^ {\pi_ {\theta} ^ {*}} \left[ K d \left(\| \boldsymbol {x} \| _ {\infty} + h \tau_ {0 ^ {d}}\right) ^ {r} \tau_ {0 ^ {d}} \right], \\ \end{array}
$$
which is finite from geometric ergodicity (Assumption 3) and the discussion following that.
B Proofs of regret analysis
In this section, we state the proofs related to regret analysis of Section 4. We first note a key property of Thompson sampling from [49], which states that for any episode  $k$ , measurable function  $f$ , and  $\mathcal{H}_{t_k}$  -measurable random variable  $Y$ , we have
$$
\mathbb {E} \left[ f \left(\theta_ {k}, Y\right) \right] = \mathbb {E} \left[ f \left(\boldsymbol {\theta} ^ {*}, Y\right) \right], \tag {20}
$$
where  $\mathcal{H}_t\coloneqq \sigma (X(1),\ldots ,X(t),A(1),\ldots ,A(t - 1))$  for all  $t\in \mathbb{N}$ . We start with deriving upper bounds on the hitting times of state  $0^{d}$  using the ergodicity conditions of Assumptions 3 and 4. Previous works [25, 27, 29] have already established bounds on hitting times in geometrically and polynomially ergodic chains in terms of their corresponding Lyapunov function. However, our objective is to provide a precise characterization of all constants included in these bounds in terms of the constants of the drift equations 3 and 4. This characterization allows us to derive uniform bounds across the model class. In Appendix C.1, using the polynomial Lyapunov function provided in Assumption 4, we establish upper bounds on the  $i$ -th moment of hitting time of state  $0^{d}$  from any state  $\pmb {x}\in \mathcal{X}$  and for  $1\leq i\leq r + 1$ . Importantly, the derived bound is polynomial in terms of any component of the state  $x_{i}$ . Additionally, in Appendix C.2, we characterize the tail probabilities of the return time to state  $0^{d}$  starting from  $0^{d}$  in terms of the geometric Lyapunov function of Assumption 3. The derived tail bounds will be used in Lemma 6 to derive upper bounds for all moments of hitting times in the model class. These bounds, along with the skip-free behavior of the model, allow us to study the maximum state (with respect to  $\ell_{\infty}$ -norm) achieved up to time  $T$  in MDP  $(\mathcal{X},\mathcal{A},c,P_{\theta^*})$  following Algorithm 1 as follows.
Lemma 6. For  $p \in \mathbb{N}$ , the  $p$ -th moment of  $\max_{1 \leq i \leq T} \tau_{0^d}^{(i)}$  and  $M_{\theta^*}^T$ , that is the maximum  $\ell_{\infty}$ -norm of the state vector achieved up until and including time  $T$  is  $O(\log^p T)$ .
In the proof of Lemma 6 given in Appendix B.1, we make use of geometric ergodicity of the chain and the fact that hitting times have geometric tails to find an upper bound for moments of  $M_{\theta^*}^T$ . Using this, we aim to bound the number of episodes started before or at  $T$ , denoted by  $K_T$ . We first find an upper bound for the number of episodes in which the second stopping criterion is met or there exists a state-action pair for which  $N_t(\pmb{x},a)$  has increased more than twice. In the following lemma, we bound the number of such episodes, which we denote by  $K_M$ , in terms of random variable  $M_{\theta^*}^T$  and other problem-dependent constants. Proof of Lemma 7 is given in Appendix B.2.
Lemma 7. The number of episodes triggered by the second stopping criterion and started before or at time  $T$ , denoted by  $K_{M}$ , satisfies  $K_{M} \leq 2|\mathcal{A}|(M_{\boldsymbol{\theta}^{*}}^{T} + 1)^{d}\log_{2}T$  a.s.
We next bound the total number of episodes  $K_{T}$  by bounding the number of episodes triggered by the first stopping criterion, using the fact that in such episodes,  $\tilde{T}_k = \tilde{T}_{k - 1} + 1$ . Moreover, to address the settling time of each episode  $k$ , shown by  $E_{k} = T_{k} - \tilde{T}_{k}$ , we use the geometric ergodicity property and Lemma 6. Finally, the proof of Lemma 8 is given in Appendix B.3.
Lemma 8. The number of episodes started by  $T$  satisfies  $K_{T} \leq 2\sqrt{|\mathcal{A}|(M_{\boldsymbol{\theta}^{*}}^{T} + 1)^{d}T\log_{2}T}$  a.s.
From Lemma 8, the upper bound given in Lemma 6 for moments of  $M_{\theta^*}^T$ , and Cauchy-Schwarz inequality, it follows that the expected value of the number of episodes  $K_T$  is of the order  $\tilde{O}(\sqrt{h^d |\mathcal{A}| T})$ . This term has a crucial role in determining the overall order of the total regret up to time  $T$ . In the rest of this section, we present a detailed proof of the lemmas and other results used to prove Theorem 1.
Remark 3. The skip-free to the right property in Assumption 2 yields a polynomially-sized subset of the underlying state-space that can be explored as a function of  $T$ . This polynomially-sized subset can be viewed as the effective finite-size of the system in the worst-case, and then, directly applying finite-state problem bounds [49] would result in a regret of order  $\tilde{O}(T^{d+0.5})$ ; since  $d \geq 1$ , such a coarse bound is not helpful even for asserting asymptotic optimality! However, to achieve a regret of  $\tilde{O}(\sqrt{T})$ , it is essential to carefully understand and characterize the distribution of  $M_{\theta^*}^T$  and then its moments, as demonstrated in Lemma 6.