SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We now upper-bound  $\mathbb{P}_{0^d}(\tau_{0^d} > n)$  for all  $n\geq 1$  in an inductive manner, starting with  $\mathbb{P}_{0^d}(\tau_{0^d} > 1)$ . As a part of showing this, for every  $\pmb {x}\neq 0^{d}$  we argue that for all  $n\geq 1$
$$
\mathbb {P} _ {\boldsymbol {x}} \left(\tau_ {0 ^ {d}} > n\right) \leq \tilde {V} ^ {g} (\boldsymbol {x}) \left(1 - \frac {1}{\tilde {b} ^ {g}}\right) ^ {n}. \tag {51}
$$
First note that
$$
\tilde {V} ^ {g} (\boldsymbol {x}) \geq V ^ {g} (\boldsymbol {x}) \geq 1. \tag {52}
$$
$$
\begin{array}{l} \mathbb {P} _ {\boldsymbol {x}} \left(\tau_ {0 ^ {d}} > 1\right) = \sum P _ {\boldsymbol {x y}} \leq \sum P _ {\boldsymbol {x y}} \tilde {V} ^ {g} (\boldsymbol {y}) \\ = \sum P _ {\boldsymbol {x y}} \sum_ {n = 0} ^ {\infty} \sum P _ {\boldsymbol {y z}} ^ {n} V ^ {g} (\boldsymbol {z}) = \sum_ {\boldsymbol {z} \in \mathcal {X}} \sum_ {n = 1} ^ {\infty} _ {0 ^ {d}} P _ {\boldsymbol {x z}} ^ {n} V ^ {g} (\boldsymbol {z}). \tag {53} \\ \end{array}
$$
We now apply the bound in (50) to get
$$
\mathbb {P} _ {\boldsymbol {x}} \left(\tau_ {0 ^ {d}} > 1\right) \leq \sum_ {\boldsymbol {z} \in \mathcal {X}} \sum_ {n = 1} ^ {\infty} P _ {\boldsymbol {x z}} ^ {n} V ^ {g} (\boldsymbol {z}) = \tilde {V} ^ {g} (\boldsymbol {x}) - V ^ {g} (\boldsymbol {x}) \leq \tilde {V} ^ {g} (\boldsymbol {x}) \left(1 - \frac {1}{\tilde {b} ^ {g}}\right). \tag {54}
$$
With the base of induction established, we assume the statement in (51) is true for  $n$ , and show that it continues to hold for  $n + 1$  as follows:
$$
\begin{array}{l} \mathbb {P} _ {\boldsymbol {x}} \left(\tau_ {0 ^ {d}} > n + 1\right) = \sum_ {\boldsymbol {y} \neq 0 ^ {d}} P _ {\boldsymbol {x y}} \mathbb {P} _ {\boldsymbol {y}} \left(\tau_ {0 ^ {d}} > n\right) \\ \leq \left(1 - \frac {1}{\bar {b} ^ {g}}\right) ^ {n} \sum_ {\boldsymbol {y} \neq 0 ^ {d}} P _ {\boldsymbol {x y}} \tilde {V} ^ {g} (\boldsymbol {y}) \\ \leq \tilde {V} ^ {g} (\boldsymbol {x}) \left(1 - \frac {1}{\tilde {b} ^ {g}}\right) ^ {n + 1}, \\ \end{array}
$$
where the final inequality uses the same arguments as in (53) and (54).
Finally, using the tail probabilities of hitting time of state  $0^d$  from any state  $\pmb{x} \neq 0^d$ , we bound the tail probability of the return time to state  $0^d$  (starting from  $0^d$ ) as follows
$$
\begin{array}{l} \mathbb {P} _ {0 ^ {d}} \left(\tau_ {0 ^ {d}} > n + 1\right) = \sum_ {\boldsymbol {x} \neq 0 ^ {d}} P _ {0 \boldsymbol {x}} \mathbb {P} _ {\boldsymbol {x}} \left(\tau_ {0 ^ {d}} > n\right) \leq \left(1 - \frac {1}{\tilde {b} ^ {g}}\right) ^ {n} \sum_ {\boldsymbol {x} \neq 0 ^ {d}} P _ {0 \boldsymbol {x}} \tilde {V} ^ {g} (\boldsymbol {x}) \\ \leq \tilde {b} ^ {g} \left(1 - \frac {1}{\tilde {b} ^ {g}}\right) ^ {n} \sum_ {\boldsymbol {x} \neq 0 ^ {d}} P _ {0 \boldsymbol {x}} V ^ {g} (\boldsymbol {x}) \leq b ^ {g} \tilde {b} ^ {g} \left(1 - \frac {1}{\tilde {b} ^ {g}}\right) ^ {n}, \\ \end{array}
$$
where the final inequality follows from the definition of  $b^g$ , and we have
$$
\tilde {\gamma} ^ {g} = 1 - \frac {1}{\tilde {b} ^ {g}}, \text {a n d} c ^ {g} = \frac {b ^ {g} (\tilde {b} ^ {g}) ^ {2}}{\tilde {b} ^ {g} - 1},
$$
and the proof is complete.
E Queueing model examples
E.1 Model 1: Two-server queueing system with a common buffer
We consider a continuous-time queueing system with two heterogeneous servers with unknown service rate vector  $\theta^{*} = (\theta_{1}^{*},\theta_{2}^{*})$  and a common infinite buffer, shown in Figure 2a. Arrivals to the system are according to a Poisson process with rate  $\lambda$  and service times are exponentially distributed
with parameter  $\theta_{i}^{*}$ , depending on the assigned server. The service rate vector  $\pmb{\theta}^{*}$  is sampled from the prior distribution  $\nu_{0}$  defined on the space  $\Theta$  given as
$$
\Theta = \left\{\left(\theta_ {1}, \theta_ {2}\right) \in \mathbb {R} _ {+} ^ {2}: \frac {\lambda}{\theta_ {1} + \theta_ {2}} \leq \frac {1 - \delta}{1 + \delta}, 1 \leq \frac {\theta_ {1}}{\theta_ {2}} \leq R \right\}, \tag {55}
$$
for fixed  $\delta \in (0,0.5)$  and  $R \geq 1$ . Note that for any  $(\theta_1,\theta_2) \in \Theta$ , we have  $\theta_1 \geq \theta_2$  and the stability requirement  $\lambda < \theta_1 + \theta_2$  holds. The countable state space  $\mathcal{X}$  is defined as  $\mathcal{X} = \{\pmb{x} = (x_0,x_1,x_2): x_0 \in \mathbb{N} \cup \{0\}, x_1,x_2 \in \{0,1\}\}$ , in which  $x_0$  is the length of the queue, and  $x_i, i = 1,2$  is equal to 1 if server  $i$  is busy serving a job. At each time instance  $r \in \mathbb{R}_+$ , the dispatcher can assign jobs from the (non-empty) buffer to an available server. Thus, the action space  $\mathcal{A}$  is equal to
$$
\mathcal {A} = \{h, b, 1, 2 \},
$$
where  $h$  indicates no action,  $b$  sends a job to both of the servers, and  $i = 1,2$  assigns a job to server  $i$ . The goal of the dispatcher is to minimize the expected sojourn time of customers, which by Little's law [52] is equivalent to minimizing the average number of customers in the system, or
$$
\inf  _ {\pi \in \Pi} \lim  _ {T \rightarrow \infty} \sup  _ {0} \frac {1}{T} \int_ {0} ^ {T} \| \boldsymbol {X} (r) \| _ {1} d r, \tag {56}
$$
where  $X(r)$  is the state of the system at time  $r \in \mathbb{R}_+$ , immediately after the arrival/departure and just before the action is taken. In [38], it is argued that from uniformization [39] and sampling the continuous-time Markov process at a rate of  $\lambda + \theta_1^* + \theta_2^*$ , a discrete-time Markov chain is obtained, which converts the original continuous-time problem shown in (56) to an equivalent discrete-time problem as below:
$$
\inf  _ {\pi \in \Pi} \lim  _ {T \rightarrow \infty} \frac {1}{T} \int_ {0} ^ {T} \| \boldsymbol {X} (r) \| _ {1} d r = \inf  _ {\pi \in \Pi} \lim  _ {T \rightarrow \infty} \frac {1}{T} \sum_ {i = 0} ^ {T - 1} \| \boldsymbol {X} (i) \| _ {1}. \tag {57}
$$
To obtain a uniform sampling rate of  $\lambda +\theta_1^* +\theta_2^*$ , the continuous-time system is sampled at arrivals, real and dummy customer departures. In [38], it is further shown that the optimal policy that achieves the infimum in (57) is a threshold policy  $\pi_t$  with the optimal finite threshold  $t(\theta)\in \mathbb{N}$ , with the policy defined as below:
$$
\pi_ {t} (\boldsymbol {x}) = \left\{ \begin{array}{l} \text {h i f} \{x _ {0} = 0 \} \text {o r} \{\| \boldsymbol {x} \| _ {1} \leq t, x _ {1} = 1 \} \text {o r} \{x _ {1} = x _ {2} = 1 \} \\ 1 \text {i f} \{x _ {0} \geq 1, x _ {1} = 0 \} \\ 2 \text {i f} \{x _ {0} \geq 1, \| \boldsymbol {x} \| _ {1} \geq t + 1, x _ {1} = 1, x _ {2} = 0 \}; \end{array} \right.
$$
note that action  $b$  is not used. Policy  $\pi_t$  assigns a job to the faster (first) server whenever there is a job waiting in the queue and the first server is available. In contrast,  $\pi_t$  dispatches a job to the second server only if the number of jobs in the system are greater than threshold  $t$  and the second server is available. If neither of these conditions hold, no action or  $h$  is taken. Consequently, we can restrict the set of all policies  $\Pi$  in (57) to the set  $\Pi_t$ , which is the set of all possible threshold policies corresponding to some  $t \in \mathbb{N}$ .
In the rest of this subsection, our aim is to show that Assumptions 1-5 are satisfied for the discrete-time Markov process obtained by uniformization of the described queueing system and hence, conclude that Algorithm 1 can be used to learn the unknown service rate vector  $\theta^{*}$  with the expected regret of order  $\tilde{O} (\sqrt{T})$ .
Assumption 1. Cost function is given as  $c(\pmb{x}, a) = \| \pmb{x} \|_1$ , which satisfies Assumption 1 with  $f_c(\pmb{x}) = x_0 + x_1 + x_2$  and  $K = r = 1$ .
Assumption 2. For any state-action pair  $(\pmb{x}, a)$  and  $\theta \in \Theta$ , we have  $P_{\theta}(A(\pmb{x}); \pmb{x}, a) = 0$  where  $A(\pmb{x}) = \{\pmb{y} \in \mathcal{X} : \| \pmb{y} \|_1 - \| \pmb{x} \|_1 > 1\}$ ; thus, Assumption 2 holds with  $h = 1$ .
Assumption 3. Consider a queueing system with parameter  $\theta$  following threshold policy  $\pi_t$  for some  $t\in \mathbb{N}$ . The uniformized discrete-time Markov chain is irreducible and aperiodic on a subset of state space given as  $\mathcal{X}_t = \mathcal{X}\setminus (\{(i,0,0):i\geq \min (t,2)\} \cup \{(0,1,1)\})$ . In [38], it is proved that for every  $t$ , the chain consists of a single positive recurrent class and the corresponding average number of customers, depicted by  $J^{t}(\theta)$ , is calculated. Moreover, it is shown that for every  $\theta \in \Theta$  the optimal threshold  $t(\theta)$  can be numerically found as the smallest  $i\in \mathbb{N}$  for which  $J^{i}(\theta) < J^{i + 1}(\theta)$ . Define the set  $T^{*}$  as the set of all optimal thresholds corresponding to at least one  $\theta \in \Theta$ , or
$$
T ^ {*} = \{t: t = t (\theta) \text {f o r} \theta \in \Theta \}.
$$
Remark 7. There is a discrepancy between the class of MDPs defined in this section and in Section 2, as in the former the MDPs are not irreducible in the whole state space  $\mathcal{X}$ . Specifically, for every Markov process generated by a queueing system with parameter  $\theta$  following threshold policy  $\pi_t$ , irreducibility holds on  $\mathcal{X}_t \subset \mathcal{X}$ . Nevertheless, the results of Section 4 are valid as starting from state  $(0,0)$ , the visited states are positive recurrent; see Remark 4.
In the following proposition, we verify the geometric ergodicity of the discrete-time chain governed by any parameter  $\theta \in \Theta$  and obtained by following any threshold policy  $\pi_t$  for  $t \in T^*$ ; proof is given in Appendix F.1.
Proposition 1. The discrete-time Markov process obtained from the queueing system governed by parameter  $\theta = (\theta_{1},\theta_{2})\in \Theta$  and following threshold policy  $\pi_t$  for some  $t\in T^{*}$  is geometrically ergodic. Equivalently, the following holds
$$
\Delta V _ {\theta , t} ^ {g} (\boldsymbol {x}) \leq - \left(1 - \gamma_ {\theta , t} ^ {g}\right) V _ {\theta , t} ^ {g} (\boldsymbol {x}) + b _ {\theta , t} ^ {g} \mathbb {I} _ {C _ {\theta , t} ^ {g}} (\boldsymbol {x}), \quad \boldsymbol {x} \in \mathcal {X} _ {t},
$$
$$
V _ {\theta , t} ^ {g} (\boldsymbol {x}) = \exp (- \log (1 - \delta) \| \boldsymbol {x} \| _ {1}),
$$
$$
C _ {\theta , t} ^ {g} = \left\{\left(x _ {0}, x _ {1}, 0\right): x _ {0} <   t \right\} \cup \{(0, 0, 1) \}, \tag {58}
$$
$$
b _ {\theta , t} ^ {g} = \max  _ {\boldsymbol {x} \in C _ {\theta , t} ^ {g}} \exp (- \log (1 - \delta) (\| \boldsymbol {x} \| _ {1} + 1)), \tag {59}
$$
$$
\gamma_ {\theta , t} ^ {g} = \frac {1}{2} - \frac {1}{2 \left(\theta_ {1} + \theta_ {2} + \lambda\right)} \left(\left(\theta_ {1} + \theta_ {2}\right) (1 - \delta) + \lambda (1 - \delta) ^ {- 1}\right). \tag {60}
$$
Having described all the terms explicitly, we verify the rest of the conditions of Assumption 3, which lead to uniform (over model class) upper-bounds on the moments of hitting time to  $0^d$  as follows:
1. From (60),  $\sup_{\theta \in \Theta, t \in T^*} \gamma_{\theta, t}^g \leq 1/2 < 1$ .
2. From (58), we can see that state  $(0,0)$  belongs to  $C_{\theta ,t}^{g}$  for all  $\theta \in \Theta$  and  $t\in T^{*}$ . In order for  $C_*^g = \cup_{\theta \in \Theta ,t\in T^*}C_{\theta ,t}^g$  to be a finite set, the supremum of the optimal threshold  $t(\theta)$  over  $\Theta$  should be finite. In [37] with service rate vector  $(\theta_{1},\theta_{2})$ , it is shown that the optimal threshold is bounded above by  $\sqrt{2}\theta_1 / \theta_2$ , which further gives
$$
t (\theta) \leq \sqrt {2} \frac {\theta_ {1}}{\theta_ {2}} \leq \sqrt {2} R. \tag {61}
$$
Thus,  $\sup_{\theta \in \Theta} t(\theta) \leq \sqrt{2} R$ , which is finite. To confirm a uniform upper bound for  $b_{\theta,t}^{g}$ , we note that from (59),
$$
\sup_{\theta \in \Theta ,t\in T^{*}}b_{\theta ,t}^{g} = \frac{2 - \delta}{1 - \delta}\max_{x\in C_{*}^{g}}\exp (-\log (1 - \delta)\| \boldsymbol {x}\|_{1}),
$$
which is finite as  $|C_{*}^{g}| < \infty$ .
Assumption 4. To find an upper bound on the second moment of hitting times, we verify Assumption 4 and show that there exists a finite set  $C_{\theta,t}^p$ , constants  $\beta_{\theta,t}^p$ ,  $b_{\theta,t}^p > 0$ ,  $r / (r + 1) \leq \alpha_{\theta,t}^p < 1$ , and a function  $V_{\theta,t}^p : \mathcal{X}_t \to [1, +\infty)$  satisfying
$$
\Delta V _ {\theta , t} ^ {p} (\boldsymbol {x}) \leq - \beta_ {\theta , t} ^ {p} \left(V _ {\theta , t} ^ {p} (\boldsymbol {x})\right) ^ {\alpha_ {\theta , t} ^ {p}} + b _ {\theta , t} ^ {p} \mathbb {I} _ {C _ {\theta , t} ^ {p}} (\boldsymbol {x}), \quad \boldsymbol {x} \in \mathcal {X} _ {t}. \tag {62}
$$
Proposition 2. The discrete-time Markov process obtained from the queueing system governed by parameter  $\theta = (\theta_{1},\theta_{2})\in \Theta$  and following threshold policy  $\pi_t$  for some  $t\in T^{*}$  is polynomially
ergodic. This is true because (62) holds for
$$
V _ {\theta , t} ^ {p} (\boldsymbol {x}) = \left\| \boldsymbol {x} \right\| _ {1} ^ {2}, \tag {63}
$$
$$
C _ {\theta , t} ^ {p} = \left\{\left(x _ {0}, x _ {1}, 0\right): x _ {0} <   t \right\} \cup \left\{\left(x _ {0}, x _ {1}, x _ {2}\right): x _ {0} <   \frac {2 \lambda}{\theta_ {1} + \theta_ {2} - \lambda}, x _ {1} + x _ {2} \geq 1 \right\}, \tag {64}
$$
$$
b _ {\theta , t} ^ {p} = \max  _ {\boldsymbol {x} \in C _ {\theta , t} ^ {p}} \left(\left\| \boldsymbol {x} \right\| _ {1} + 1\right) ^ {2}), \tag {65}
$$
$$
\beta_ {\theta , t} ^ {p} = 1 - \frac {2 \lambda}{\theta_ {1} + \theta_ {2} + \lambda}, \tag {66}
$$
$$
\alpha_ {\theta , t} ^ {p} = \frac {1}{2}. \tag {67}
$$
Proof of Proposition 2 is given in Appendix F.2. We define the normalized rates as  $\tilde{\lambda} = \frac{\lambda}{\lambda + \theta_1 + \theta_2}$  and  $\tilde{\theta}_i = \frac{\theta_i}{\lambda + \theta_1 + \theta_2}$ , for  $i = 1, 2$ . From the choice of parameter space  $\Theta$ , we have  $\tilde{\lambda} \leq 0.5 - 0.5\delta$ ,  $\tilde{\theta}_1 + \tilde{\theta}_2 \geq 0.5 + 0.5\delta$ , and  $\tilde{\theta}_1 \geq 0.25 + 0.25\delta$ . We verify the remaining conditions of Assumption 4 as follows:
1. From (63), the first condition holds with  $r_*^p = 2$  and  $s_*^p = 2$ .
2. From (64), we can see that state  $(0,0)$  belongs to  $C_{\theta ,t}^{p}$  for all  $\theta \in \Theta$  and  $t\in T^{*}$ . Furthermore,
$$
\sup _ {\theta \in \Theta , t \in T ^ {*}} \frac {2 \lambda}{\theta_ {1} + \theta_ {2} - \lambda} \leq \frac {1 - \delta}{\delta},
$$
which follows from the stability condition  $\tilde{\lambda} \leq 0.5 - 0.5\delta$ . Thus, from the definition of  $C_{\theta,t}^p$  in (64), and the fact that  $\sup_{\theta \in \Theta} t(\theta) \leq \sqrt{2} R$  as argued in (61),  $C_*^p = \cup_{\theta \in \Theta, t \in T^*} C_{\theta,t}^p$  is a finite set. We also note that  $\sup_{\theta \in \Theta, t \in T^*} b_{\theta,t}^p$  is finite as  $|C_*^p| < \infty$ . It remains to show that  $\inf_{\theta \in \Theta, t \in T^*} \beta_{\theta,t}^p$  is positive, which is equivalent to verifying that  $\sup_{\theta \in \Theta, t \in T^*} \tilde{\lambda} < 1/2$ , which follows from the stability condition  $\tilde{\lambda} \leq 0.5 - 0.5\delta$ .
3. We need to show that  $K_{\theta, t}(\pmb{x}) \coloneqq \sum_{n=0}^{\infty} 2^{-n-2} (P_{\theta}^{t})^{n}(\pmb{x}, 0^{d})$  is strictly bounded away from zero. We notice that from any non-zero state  $\pmb{x}$ , the queueing system hits  $0^{d}$  in  $\| \pmb{x} \|_{1}$  transitions only if all transitions are real departures. Hence,
$$
\begin{array}{l} K _ {\theta , t} (\boldsymbol {x}) \geq 2 ^ {- \| \boldsymbol {x} \| _ {1} - 2} \left(P _ {\theta} ^ {t}\right) ^ {\| \boldsymbol {x} \| _ {1}} \left(\boldsymbol {x}, 0 ^ {d}\right) \\ \geq 2 ^ {- \| \boldsymbol {x} \| _ {1} - 2} \left(\tilde {\theta} _ {1}\right) ^ {\| \boldsymbol {x} \| _ {1}} \left(\tilde {\theta} _ {2}\right) ^ {\| \boldsymbol {x} \| _ {1}} \\ \geq 2 ^ {- \| \boldsymbol {x} \| _ {1} - 2} R ^ {- \| \boldsymbol {x} \| _ {1}} \left(\tilde {\theta} _ {1}\right) ^ {2 \| \boldsymbol {x} \| _ {1}} \\ \geq 2 ^ {- \| \boldsymbol {x} \| _ {1} - 2} R ^ {- \| \boldsymbol {x} \| _ {1}} \left(\frac {1}{4} + \frac {\delta}{4}\right) ^ {2 \| \boldsymbol {x} \| _ {1}}, \\ \end{array}
$$
where the third and fourth inequalities follow from the definition of  $\Theta$  in (55). Thus, the infimum of  $K_{\theta ,t}(\pmb {x})$  over the finite set  $C_*^p$  and sets  $\Theta$  and  $T^{*}$  is strictly greater than zero.
Assumption 5. We finally verify Assumption 5, which asserts that  $\sup_{\theta \in \Theta}J(\theta)$  is finite. We have
$$
J (\theta) = \mathbb {E} _ {\boldsymbol {X} \sim \mu_ {\theta , t (\theta)}} [ c (\boldsymbol {X}) ] = \mathbb {E} _ {\boldsymbol {X} \sim \mu_ {\theta , t (\theta)}} [ \| \boldsymbol {X} \| _ {1} ] = \mathbb {E} _ {\boldsymbol {X} \sim \mu_ {\theta , t (\theta)}} \left[ \sqrt {V _ {\theta , t (\theta)} ^ {p} (\boldsymbol {X})} \right],
$$
where  $\mu_{\theta ,t(\theta)}$  is the stationary distribution of the discrete-time process governed by parameter  $\theta$  and following the optimal policy according to  $\theta$ . From (62) and [43, Theorem 14.3.7],
$$
\mu_ {\theta , t (\theta)} \left(\sqrt {V _ {\theta , t (\theta)} ^ {p} (\boldsymbol {X})}\right) \leq \frac {b _ {*} ^ {p}}{\beta_ {*} ^ {p}},
$$
which is finite from the previously verified assumption. Consequently,
$$
\sup  _ {\theta \in \Theta} J (\theta) \leq \frac {b _ {*} ^ {p}}{\beta_ {*} ^ {p}} <   \infty .
$$
E.2 Model 2: Two heterogeneous parallel queues
We consider two parallel queues with infinite buffers, each with its own single server, and unknown service rate vector  $\pmb{\theta}^{*} = (\theta_{1}^{*},\theta_{2}^{*})$ , shown in Figure 2b. The service rate vector  $\pmb{\theta}^{*}$  is sampled from the prior distribution  $\nu_{0}$  defined on the space  $\Theta$  given as
$$
\Theta = \left\{\left(\theta_ {1}, \theta_ {2}\right) \in \mathbb {R} _ {+} ^ {2}: \frac {\lambda}{\theta_ {1} + \theta_ {2}} \leq \frac {1 - \delta}{1 + \delta}, 1 \leq \frac {\theta_ {1}}{\theta_ {2}} \leq R \right\}, \tag {68}
$$
for fixed  $\delta \in (0, 0.5)$  and  $R \geq 1$ , which ensures the stability of the queueing system. Consider the discrete-time MDP  $(\mathcal{X}, \mathcal{A}, P_{\theta^{*}}, c)$  obtained by sampling the queueing system at the Poisson arrival sequence. The countably infinite state space  $\mathcal{X}$  is defined as below
$$
\mathcal {X} = \left\{\boldsymbol {x} = \left(x _ {1}, x _ {2}\right): x _ {i} \in \mathbb {N} \cup \{0 \} \right\},
$$
where the state of the system is the number of jobs in the server-queue pair  $i$  just before an arrival. Furthermore, the action space  $\mathcal{A}$  is equal to
$$
\mathcal {A} = \{1, 2 \},
$$
where action  $i \in \mathcal{A}$  indicates the arrival dispatched to queue  $i$ . The unbounded cost function  $c: \mathcal{X} \times \mathcal{A} \to \mathbb{N} \cup \{0\}$  is defined as the total number of jobs in the queueing system, i.e.,  $c(\boldsymbol{x}, a) = \| \boldsymbol{x} \|_1$ . For every  $\omega \in \mathbb{R}_+$ , we define policy  $\pi_{\omega}: \mathcal{X} \to \mathcal{A}$ , which routes the arrival according to the weighted queue lengths, as
$$
\pi_ {\omega} (\boldsymbol {x}) = \arg \min  \left(1 + x _ {1}, \omega (1 + x _ {2})\right),
$$
where the tie is broken in favor of the first server. We also define policy class  $\tilde{\Pi}$  as the set of policies  $\pi_{\omega}$  such that  $\omega$  belongs to a compact interval; in other words,
$$
\tilde {\Pi} = \left\{\pi_ {\omega}; \omega \in \left[ \frac {1}{c _ {R} R}, c _ {R} R \right] \right\},
$$
where  $R$  is defined in (68) and  $c_{R} \geq 1$ . We aim to minimize the infinite-horizon average cost in the policy class  $\tilde{\Pi}$ , that is,
$$
J (\theta) = \inf  _ {\pi \in \tilde {\Pi}} \lim  _ {T \rightarrow \infty} \frac {1}{T} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} c (X (t), A (t)) \right], \tag {69}
$$
where  $\mathbf{X}(t) = (X_{1}(t), X_{2}(t))$  is the occupancy vector of the queueing system just before arrival  $t$ . Even with the controlled Markov process transition kernel fully-specified (by the values of the arrival rate and the two service rates), the optimal policy<sup>1</sup> that satisfies (69) in policy class  $\tilde{\Pi}$  is not known except when  $\theta_{1} = \theta_{2}$  where the optimal value is  $\omega = 1$ , and so, to learn it, we will use Proximal Policy Optimization for countable state-space controlled Markov processes as developed in [18]. Note that [18] requires full knowledge of the controlled Markov process, which holds in our learning scheme since we use the parameters sampled from the posterior for determining the policy at the beginning of each episode. Furthermore, for each policy in the set of applicable policies  $\tilde{\Pi}$ , [18] also requires that the resulting Markov process be geometrically ergodic, which we will establish below.
Proposition 3. The discrete-time Markov process obtained from the queueing system governed by parameter  $\theta = (\theta_{1},\theta_{2})\in \Theta$  and following policy  $\pi_{\omega}\in \tilde{\Pi}$  is geometrically ergodic. Equivalently, the following holds
$$
\Delta V _ {\theta , \omega} ^ {g} (\boldsymbol {x}) \leq - \left(1 - \gamma_ {\theta , \omega} ^ {g}\right) V _ {\theta , \omega} ^ {g} (\boldsymbol {x}) + b _ {\theta , \omega} ^ {g} \mathbb {I} _ {C _ {\theta , \omega} ^ {g}} (\boldsymbol {x}), \quad \boldsymbol {x} \in \mathcal {X}, \tag {70}
$$
<sup>1</sup>When  $\theta_{1} = \theta_{2}$ , then the policy with  $\omega = 1$  (Join-the-Shortest-Queue) is the optimal policy [19] for the underlying MDP.
$$
V _ {\theta , \omega} ^ {g} (\pmb {x}) = \frac {\omega}{\omega + 1} \exp \left(a _ {\theta , \omega} ^ {g} \frac {x _ {1} + 1}{\omega}\right) + \frac {1}{\omega + 1} \exp \left(a _ {\theta , \omega} ^ {g} (x _ {2} + 1)\right),
$$
$$
a _ {\theta , \omega} ^ {g} = \min  \left(\omega \log (1 + \delta), \log (1 + \delta), \omega \log \frac {1 - 0 . 5 \delta}{1 - \delta}, \log \frac {1 - 0 . 5 \delta}{1 - \delta}, \frac {\delta (1 - \delta^ {2})}{4 c _ {R} R (1 - 0 . 5 \delta)}\right), \tag {71}
$$
$$
C _ {\theta , \omega} ^ {g} = \left\{\left(x _ {1}, x _ {2}\right) \in \mathcal {X}: x _ {i} \leq \max  \left(x _ {i, \theta , \omega} ^ {g _ {j}}, 0\right), i, j = 1, 2 \right\}, \tag {72}
$$
$$
b _ {\theta , \omega} ^ {g} = \max  _ {\boldsymbol {x} \in C _ {\theta , \omega} ^ {g}} \left(\frac {2 \omega}{\omega + 1} \exp \left(a _ {\theta , \omega} ^ {g} \frac {x _ {1} + 2}{\omega}\right) + \frac {2}{\omega + 1} \exp \left(a _ {\theta , \omega} ^ {g} \left(x _ {2} + 2\right)\right)\right), \tag {73}
$$
$$
\gamma_ {\theta , \omega} ^ {g} = \frac {1}{2} + \frac {1}{2} \max  \left(\zeta_ {1, \theta , \omega}, \zeta_ {2, \theta , \omega}, \frac {\zeta_ {1 , \theta , \omega} \omega}{1 + \omega} \exp \left(\frac {a _ {\theta , \omega} ^ {g}}{\omega}\right) + \frac {\zeta_ {2 , \theta , \omega}}{1 + \omega}, \frac {\zeta_ {1 , \theta , \omega} \omega}{1 + \omega} + \frac {\zeta_ {2 , \theta , \omega}}{1 + \omega} \exp (a _ {\theta , \omega} ^ {g})\right), \tag {74}
$$
and problem-dependent constants  $x_{i,\theta ,\omega}^{g_j}$  and  $\zeta_{i,\theta ,\omega}$  for  $i,j = 1,2$
Proof of Proposition 3 is given in Appendix F.3. In the rest of this subsection, our aim is to show that Assumptions 1-5 are satisfied for the discrete-time MDP and conclude that Algorithm 1 can be used to learn the unknown service rate vector  $\theta^{*}$  with expected regret of order  $\tilde{O} (\sqrt{T})$ .
Assumption 2. For any state-action pair  $(\pmb{x},a)$  and  $\theta \in \Theta$ , we have  $P_{\theta}(A(\pmb{x});\pmb{x},a) = 0$  where  $A(\pmb{x}) = \{\pmb{y}\in \mathcal{X}:\| \pmb{y}\| _1 - \| \pmb {x}\| _1 > 1\}$ ; thus, the MDP is skip-free to the right with  $h = 1$ . Moreover, from any  $(\pmb {x},a)$ , the finite set  $\{\pmb {y}\in \mathcal{X}:\| \pmb {y}\| _1\leq \| \pmb {x}\| _1 + 1\}$  is only accessible in one step; thus, Assumption 2 holds.