SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} x _ {2} \left(- \frac {2 d _ {2}}{1 - d _ {2}} + \sqrt {\omega + 1} \beta_ {\theta , \omega} ^ {p}\right) + \sqrt {\omega + 1} \beta_ {\theta , \omega} ^ {p} + \frac {\left(1 - d _ {2} ^ {x _ {2}}\right) \left(d _ {2} + d _ {2} ^ {2}\right)}{\left(1 - d _ {2}\right) ^ {2}} \\ \leq x _ {2} \left(- \frac {2 d _ {2}}{1 - d _ {2}} + \sqrt {\omega + 1} \beta_ {\theta , \omega} ^ {p}\right) + \frac {d _ {2} + d _ {2} ^ {2}}{1 - d _ {2}} x _ {2} + \sqrt {\omega + 1} \beta_ {\theta , \omega} ^ {p} \\ = x _ {2} \left(- \frac {2 d _ {2}}{1 - d _ {2}} + \frac {d _ {2} + d _ {2} ^ {2}}{1 - d _ {2}} + \sqrt {\omega + 1} \beta_ {\theta , \omega} ^ {p}\right) + \sqrt {\omega + 1} \beta_ {\theta , \omega} ^ {p} \\ = x _ {2} \left(- d _ {2} + \sqrt {\omega + 1} \beta_ {\theta , \omega} ^ {p}\right) + \sqrt {\omega + 1} \beta_ {\theta , \omega} ^ {p}, \\ \end{array}
$$
where the second line follows from (103). Taking  $\beta_{\theta ,\omega}^{p}\leq d_{2} / 2\sqrt{\omega + 1}$ , we get
$$
x _ {2} \left(- \frac {2 d _ {2}}{1 - d _ {2}} + \sqrt {\omega + 1} \beta_ {\theta , \omega} ^ {p}\right) + \sqrt {\omega + 1} \beta_ {\theta , \omega} ^ {p} + \frac {\left(1 - d _ {2} ^ {x _ {2}}\right) \left(d _ {2} + d _ {2} ^ {2}\right)}{\left(1 - d _ {2}\right) ^ {2}} \leq - \frac {d _ {2}}{2} x _ {2} + \frac {d _ {2}}{2},
$$
which is non-positive for  $x_{2} \geq 1$ . Finally, when  $0.8 \leq d_{1} < 1$ ,  $x_{1}, x_{2} > 0$ , and  $x_{1} + 1 \leq \omega (x_{2} + 1)$ , (76) holds for  $\beta_{\theta, \omega}^{p} \leq d_{2} / 2\sqrt{\omega + 1}$ .
$\mathbf{Two.} d_{1} < 0.8:$  Taking  $\beta_{\theta, \omega}^{p} \leq \frac{d_{2}}{\sqrt{\omega + 1}(1 - d_{2})}$ , we note that the coefficient of  $x_{2}$  in (105) is negative. Thus, from  $x_{1} + 1 \leq \omega(x_{2} + 1)$ , (104) and (105),
$$
\begin{array}{l} \Delta V _ {\theta , \omega} ^ {p} (\pmb {x}) + \sqrt {\omega + 1} \beta_ {\theta , \omega} ^ {p} (x _ {2} + 1) \\ \leq \frac {x _ {1} + 1}{\omega} \left(2 - \frac {2 d _ {1}}{1 - d _ {1}}\right) - \frac {1}{\omega} + \frac {\left(1 - d _ {1} ^ {x _ {1} + 1}\right) \left(d _ {1} + d _ {1} ^ {2}\right)}{\omega \left(1 - d _ {1}\right) ^ {2}} \\ + \frac {x _ {1} + 1}{\omega} \left(- \frac {2 d _ {2}}{1 - d _ {2}} + \sqrt {\omega + 1} \beta_ {\theta , \omega} ^ {p}\right) + \frac {2 d _ {2}}{1 - d _ {2}} + \frac {\left(1 - d _ {2} ^ {x _ {2}}\right) \left(d _ {2} + d _ {2} ^ {2}\right)}{\left(1 - d _ {2}\right) ^ {2}} \\ <   \frac {x _ {1} + 1}{\omega} \left(2 - \frac {2 d _ {1}}{1 - d _ {1}} - \frac {2 d _ {2}}{1 - d _ {2}} + \sqrt {\omega + 1} \beta_ {\theta , \omega} ^ {p}\right) + \frac {2 d _ {2}}{1 - d _ {2}} + \frac {d _ {1} + d _ {1} ^ {2}}{\omega (1 - d _ {1}) ^ {2}} + \frac {d _ {2} + d _ {2} ^ {2}}{(1 - d _ {2}) ^ {2}}. \tag {106} \\ \end{array}
$$
As  $d_{i} = \tilde{\theta}_{i} / (\tilde{\theta}_{i} + \tilde{\lambda})$  in terms of the normalized rates, we get
$$
2 - \frac {2 d _ {1}}{1 - d _ {1}} - \frac {2 d _ {2}}{1 - d _ {2}} = 2 - \frac {2 \tilde {\theta} _ {1}}{\tilde {\lambda}} - \frac {2 \tilde {\theta} _ {1}}{\tilde {\lambda}} = \frac {- 2 (\tilde {\theta} _ {1} + \tilde {\theta} _ {2} - \tilde {\lambda})}{\tilde {\lambda}},
$$
which is negative from the stability condition. For  $\beta_{\theta ,\omega}^{p}\leq \frac{\tilde{\theta}_{1} + \tilde{\theta}_{2} - \tilde{\lambda}}{\tilde{\lambda}\sqrt{\omega + 1}}$  , from (106) we get
$$
\begin{array}{l} \Delta V _ {\theta , \omega} ^ {p} (\pmb {x}) + \sqrt {\omega + 1} \beta_ {\theta , \omega} ^ {p} (x _ {2} + 1) \\ <   \frac {- (\tilde {\theta} _ {1} + \tilde {\theta} _ {2} - \tilde {\lambda})}{\omega \tilde {\lambda}} (x _ {1} + 1) + \frac {2 d _ {2}}{1 - d _ {2}} + \frac {d _ {1} + d _ {1} ^ {2}}{\omega (1 - d _ {1}) ^ {2}} + \frac {d _ {2} + d _ {2} ^ {2}}{(1 - d _ {2}) ^ {2}} \\ = \frac {- (\tilde {\theta} _ {1} + \tilde {\theta} _ {2} - \tilde {\lambda})}{\omega \tilde {\lambda}} (x _ {1} + 1) + \frac {2 \tilde {\theta} _ {2}}{\tilde {\lambda}} + \frac {\tilde {\theta} _ {1} (2 \tilde {\theta} _ {1} + \tilde {\lambda})}{\omega \tilde {\lambda} ^ {2}} + \frac {\tilde {\theta} _ {2} (2 \tilde {\theta} _ {2} + \tilde {\lambda})}{\tilde {\lambda} ^ {2}}, \\ \end{array}
$$
which is non-positive for
$$
x _ {1} + 1 \geq \frac {\tilde {\theta} _ {1} (2 \tilde {\theta} _ {1} + \tilde {\lambda}) + \omega \tilde {\theta} _ {2} (2 \tilde {\theta} _ {2} + 3 \tilde {\lambda})}{\tilde {\lambda} (\tilde {\theta} _ {1} + \tilde {\theta} _ {2} - \tilde {\lambda})}.
$$
As  $d_1 < 0.8$ , we can see that  $\tilde{\lambda} > \tilde{\theta}_1 / 4$ ; thus,
$$
\frac {\tilde {\theta} _ {1} \left(2 \tilde {\theta} _ {1} + \tilde {\lambda}\right) + \omega \tilde {\theta} _ {2} \left(2 \tilde {\theta} _ {2} + 3 \tilde {\lambda}\right)}{\tilde {\lambda} \left(\tilde {\theta} _ {1} + \tilde {\theta} _ {2} - \tilde {\lambda}\right)} <   \frac {4 \tilde {\theta} _ {1} \left(2 \tilde {\theta} _ {1} + \tilde {\lambda}\right) + 4 \omega \tilde {\theta} _ {2} \left(2 \tilde {\theta} _ {2} + 3 \tilde {\lambda}\right)}{\tilde {\theta} _ {1} \left(\tilde {\theta} _ {1} + \tilde {\theta} _ {2} - \tilde {\lambda}\right)} <   \frac {4 c _ {R} R (1 + 2 \tilde {\lambda})}{\delta} \leq 4 c _ {R} R,
$$
where we have used the fact that  $\tilde{\theta}_1 \geq \tilde{\theta}_2$ ,  $\omega \leq c_R R$ ,  $\tilde{\theta}_1 + \tilde{\theta}_2 - \tilde{\lambda} \geq \delta$ , and  $\tilde{\lambda} \leq 0.5 - 0.5\delta$  and it suffices for  $x_1$  to be greater than or equal to  $4c_R R$ . For  $x_1 < 4c_R R$ , (104) can be upper bounded as
$$
\frac {8 c _ {R} R}{\omega} + \frac {1 - 3 d _ {1}}{\omega (1 - d _ {1})} + \frac {d _ {1} + d _ {1} ^ {2}}{\omega (1 - d _ {1}) ^ {2}} \leq \frac {8 c _ {R} R}{\omega} + \frac {2}{\omega (1 - d _ {1}) ^ {2}} <   \frac {8 c _ {R} R + 5 0}{\omega},
$$
where in the last inequality we have used  $d_1 < 0.8$ . From (105) and taking  $\beta_{\theta, \omega}^p \leq d_2 / 2\sqrt{\omega + 1}$ ,
$$
\begin{array}{l} \Delta V _ {\theta , \omega} ^ {p} (\boldsymbol {x}) + \sqrt {\omega + 1} \beta_ {\theta , \omega} ^ {p} (x _ {2} + 1) \\ \leq \frac {8 c _ {R} R + 5 0}{\omega} + \left(- \frac {2 d _ {2}}{1 - d _ {2}} + \frac {d _ {2}}{2}\right) (x _ {2} + 1) + \frac {2 d _ {2}}{1 - d _ {2}} + \frac {\left(1 - d _ {2} ^ {x _ {2}}\right) \left(d _ {2} + d _ {2} ^ {2}\right)}{\left(1 - d _ {2}\right) ^ {2}} \\ \leq \left(- \frac {2 d _ {2}}{1 - d _ {2}} + \frac {d _ {2}}{2} + \frac {d _ {2} + d _ {2} ^ {2}}{1 - d _ {2}}\right) x _ {2} + \frac {d _ {2}}{2} + \frac {8 c _ {R} R + 5 0}{\omega} \\ = - \frac {d _ {2}}{2} x _ {2} + \frac {d _ {2}}{2} + \frac {8 c _ {R} R + 5 0}{\omega}, \\ \end{array}
$$
which is negative for
$$
x _ {2} \geq 1 + \frac {1 6 c _ {R} R + 1 0 0}{\omega d _ {2}}.
$$
Finally, when  $x_{1} + 1 \leq \omega(x_{2} + 1)$  and  $x_{1}, x_{2} > 0$ , (76) holds for  $\beta_{\theta, \omega}^{p} \leq \frac{1}{\sqrt{\omega + 1}} \min \left( \frac{\tilde{\theta}_{2}}{2(\tilde{\theta}_{2} + \tilde{\lambda})}, \tilde{\theta}_{1} + \tilde{\theta}_{2} - \tilde{\lambda} \right)$ ,  $x_{1} \geq 4c_{R}R$ , and  $x_{2} \geq 1 + \frac{16c_{R}R + 100}{\omega d_{2}}$ . Repeating the same arguments when  $x_{1}, x_{2} > 0$  and  $x_{1} + 1 > \omega(x_{2} + 1)$ , (76) holds for  $\beta_{\theta, \omega}^{p} \leq \frac{1}{\sqrt{\omega + 1}} \min \left( \frac{\tilde{\theta}_{1}}{2(\tilde{\theta}_{1} + \tilde{\lambda})}, \tilde{\theta}_{1} + \tilde{\theta}_{2} - \tilde{\lambda} \right)$ ,  $x_{1} \geq 1 + \frac{\omega(16c_{R}R^{2} + 100)}{d_{1}}$ , and  $x_{2} \geq 4c_{R}R^{2}$ . Finally, (76) holds with
$$
V _ {\theta , \omega} ^ {p} (\pmb {x}) = \frac {x _ {1} ^ {2}}{\omega} + x _ {2} ^ {2},
$$
$$
C _ {\theta , \omega} ^ {p} = \left\{\left(x _ {1}, x _ {2}\right) \in \mathcal {X}: x _ {i} \leq \left(1 6 c _ {R} ^ {2} R ^ {3 - i} + 1 0 1 c _ {R} R\right) \frac {\lambda + \theta_ {i}}{\theta_ {i}}, i = 1, 2 \right\},
$$
$$
\beta_ {\theta , \omega} ^ {p} = \min  \left(\frac {\tilde {\theta} _ {2}}{2 (\tilde {\theta} _ {2} + \tilde {\lambda}) \sqrt {\omega + 1}}, \frac {\tilde {\theta} _ {1} + \tilde {\theta} _ {2} - \tilde {\lambda}}{\sqrt {\omega + 1}}, \frac {\tilde {\theta} _ {2}}{2 (\tilde {\theta} _ {2} + \tilde {\lambda})}, \frac {\tilde {\theta} _ {1}}{2 (\tilde {\theta} _ {1} + \tilde {\lambda}) \sqrt {\omega}}\right),
$$
$$
b _ {\theta , \omega} ^ {p} = (\beta_ {\theta , \omega} ^ {p} + 1) \max  _ {\boldsymbol {x} \in C _ {\theta , \omega} ^ {p}} \left(\frac {(x _ {1} + 1) ^ {2}}{\omega} + (x _ {2} + 1) ^ {2}\right),
$$
$$
\alpha_ {\theta , \omega} ^ {p} = \frac {1}{2},
$$
where the fourth line holds since  $PV_{\theta, \omega}^p(\boldsymbol{x}) \leq V_{\theta, \omega}^p(\boldsymbol{y})$  for  $\boldsymbol{y} = (y_1, y_2)$  such that  $y_i = x_i + 1$  for  $i = 1, 2$ .
G Numerical results
G.1 Comparison of Algorithm 1 with other learning algorithms
We first note that due to the countably infinite state-space setting of our problem, we are unable to directly compare our algorithm to other learning algorithms proposed in the literature. One potential candidate algorithm uses the reward biased maximum likelihood estimation (RBMLE) [33, 34, 11, 42], which estimates the unknown model parameter with the likelihood perturbed a vanishing bias towards parameters with a larger long-term average reward (i.e., optimal value). This scheme also uses the principle of "optimism in the face of uncertainty" in how it perturbs the maximum likelihood estimate. The naive version of the RMBLE algorithm does not apply to our examples due the following key assumption: over all parameters (and the control policies used for them), the transition probabilities are assumed to be mutually absolutely continuous; this is critical for the proofs and also allows the use of log-likelihood functions for computations. Similarly, naive use of the algorithms in [36] and [24] is not possible, again due to a similar absolutely continuity assumption which is critical for the proofs. Our posterior computations avoid such issues as the true parameter always has non-zero mass during the execution of the algorithm: episode  $k$  always starts in state  $0^d$  which is positive recurrent for the Markov chain with true parameter  $\theta^*$  and policy used  $\pi_{\theta_k}^*$ . The RBMLE algorithm has yet another issue in that it requires knowledge of the optimal value function, and hence, for our examples, it may only apply to Model 1 for which the value function is known analytically. Finally, whereas we do get to observe inter-arrival times for both model, we never directly observe completed service times owing to the sampling employed, and this precludes the direct use of Upper-Confidence-Bound based parameter estimation followed by certainty equivalent control algorithms. Owing to these issues, at this point in time, we're unable to perform empirical comparisons of Algorithm 1 to other candidate algorithms with theoretical performance guarantees in a countable state setting.
As discussed in the previous paragraph, learning algorithms with theoretical performance guarantees are established in the finite state setting. One such algorithm is the certainty equivalence control with forcing, which is proposed and discussed in detail in [4]. To assess the finite-time performance of our algorithm, in Figure 4, we compare the performance of our proposed learning algorithm, denoted as TSDE, with the algorithm introduced in [4], referred to as AgrawalTeneketzis. Reference [4] proposes a certainty equivalence control law with forced exploration, which operates in episodes with increasing lengths and a priori fixed sequences of forcing times. Specifically, at the beginning of each episode, all possible stationary control laws are explored for one recurrence interval of state  $(0,0)$ . Subsequently, based on this exploration, an empirical estimate of the average collected reward
(a) Model 1: Queueing system of Figure 2a.
(b) Model 2: Queueing system of Figure 2b.
Figure 4: Comparison of the regret performance of Algorithm 1 (referred to as TSDE) with the algorithm proposed by [4] (denoted as AgrawalTeneketzis) for the queueing models of Figure 2.
(b) Model 2: Queueing system of Figure 2b.
Figure 5: Total variation distance between the posterior and real distribution for  $\lambda = 0.3, 0.5, 0.7$ . The y axis is plotted on a logarithmic scale to display the differences clearly.
is formed, and the control law resulting in the maximum average reward is implemented for the remainder of the episode. The length of the episodes are determined according to sequence  $\{a_i\}_{i=0}^{\infty}$  defined as following:
$$
a _ {0} = 0,
$$
$$
a _ {i} = \sum_ {k = 1} ^ {i} b _ {k} + i p, \qquad \text {f o r} i \geq 1,
$$
where  $p$  is the number of possible stationary control laws and  $b_{i} = \left\lfloor \exp \left(i^{\frac{1}{1 + \delta}}\right)\right\rfloor$  for any  $\delta > 0$ . Specifically, episode  $i$  terminates after completing additional  $a_{i} - a_{i-1}$  recurrence intervals to state  $(0,0)$ . Both algorithms are implemented in the two queueing systems of Figure 2, where the arrival rate is  $\lambda = 0.5$  and service rates are distributed according to a Dirichlet prior over  $[0.5,1.9]^{2}$ . In Figures 4a and 4b, we set  $\delta = 3.5$  and  $\delta = 3$ , respectively. Moreover, in Figure 4b, the goal is to find the optimal weight  $w$  in the set  $\{1.5,2,2.5,3,3.5\}$ . The results in Figure 4 show that both algorithms exhibit a sublinear regret performance. Specifically, Algorithm 1, TSDE, achieves an  $\tilde{O}(\sqrt{T})$  as predicted in our theoretical results of Theorem 1 and Corollary 1. Furthermore, in both queueing models, our proposed algorithm consistently outperforms the algorithm presented in [4] in terms of regret order.
(b) Model 2: Queueing system of Figure 2b.
Figure 6: Optimal policy parameters for different service rate vectors in the two exemplary queuing systems in Model 1 and Model 2 with  $\lambda = 0.5$ .
Figure 7: Estimated average cost of Model 2 for three different service rate vectors.
G.2 Model 1: Two-server queueing system with a common buffer.
Figure 3b illustrates the behavior of the regret of Model 1 for three different arrival rate values and averaged over 2000 simulation runs. In these simulations, the parameter space is selected as
$$
\Theta = \left\{\left(\theta_ {1}, \theta_ {2}\right) \in [ 0. 5, 0. 6, \dots , 1. 9 ] ^ {2}: \lambda <   \theta_ {1} + \theta_ {2}, \theta_ {2} <   \theta_ {1} \right\},
$$
which results in a prior size of 105. As depicted in Figure 3a, the regret has a sub-linear behavior and increases with the arrival rate. The total variation distance between the posterior and real distribution, a point-mass on the random  $\theta^{*}$ , are plotted in Figure 5a. As expected, the distance diminishes towards 0, indicating the learning of the true parameter. As mentioned in Appendix E.1, the optimal policy minimizing the average number of jobs in a system with parameter  $\theta$ , is a threshold policy  $\pi_{t(\theta)}$  with optimal finite threshold  $t(\theta) \in \mathbb{N}$ , which can be numerically determined as the smallest  $i \in \mathbb{N}$  for which  $J^{i}(\theta) < J^{i+1}(\theta)$ , calculated in [38]. We compute the optimal threshold  $t(\theta)$  for every  $\theta \in \Theta$  and present the results in Figure 6a. We can see that the threshold increases as the ratio of the service rates grows. Specifically, this is why in Appendix G, we imposed conditions on  $\Theta$  to ensure that the ratio between the service rates is both upper and lower bounded.
G.3 Model 2: Two heterogeneous parallel queues
Figure 3b illustrates the behavior of the regret of Model 2 for three different arrival rate values and averaged over 2000 simulation runs. We note that the regret is sub-linear and increases with higher arrival rates. In these simulations, the parameter space is selected as
$$
\Theta = \left\{\left(\theta_ {1}, \theta_ {2}\right) \in [ 0. 5, 0. 7, \dots , 1. 9 ] ^ {2}: \lambda <   \theta_ {1} + \theta_ {2}, \theta_ {2} <   \theta_ {1} \right\},
$$
which results in a prior size of 28. As discussed earlier, our goal is to find the average cost minimizing policy within the class of policies  $\Pi = \{\pi_{\omega}; \omega \in [(c_R R)^{-1}, c_R R]\}$ ,  $c_R \geq 1$ , where  $\pi_{\omega}(\pmb{x}) =$
arg min  $(1 + x_{1},\omega (1 + x_{2}))$  with ties broken for 1. As discussed before, even with the transition kernel fully specified (by the values of arrival and service rates), the optimal policy in  $\Pi$  is not known except when  $\theta_{1} = \theta_{2}$  where the optimal value is  $\omega = 1$ , and so, to learn it, we will use Proximal Policy Optimization with approximating martingale-process (AMP) method for countable state-space MDPs [18]. We run the algorithm for 200 policy iterations, using 20 actors for each iteration. We take the state  $(0,0)$  as a regeneration state and simulate 1500 independent regenerative cycles per actor in each algorithm iteration. To approximate the value function, we employ a fully connected feed-forward neural network with one hidden layer consisting of  $10\times 10$  units and ReLU activation functions. The AMP method is also employed for variance reduction in value function estimation. The optimal  $\omega$  for every  $\theta \in \Theta$  is shown in Figure 6b, indicating that  $\omega$  increases as the ratio of the service rates grows. Therefore, it is necessary to ensure that the ratio between the service rates is bounded from above and below. Furthermore, to evaluate the regret numerically, the value of  $J(\theta)$  is required for every  $\theta \in \Theta$ , which is not known. Thus, after finding the optimal  $\omega$  using the PPO algorithm, we perform a separate simulation to approximate the optimal average cost. In Figure 7, we plot the estimated average cost for three different service rate vectors, demonstrating that the optimal average cost decreases as the service rates increase. In Figure 5b we also depict the total variation distance between the posterior and real distribution, which is a point-mass on the random  $\theta^{*}$ , and observe that the distance is converging to zero.