Title: Bayesian Learning of Optimal Policies in Markov Decision Processes with Countably Infinite State-Space
Venue: NeurIPS 2023
Accepted: yes
Keywords: Thompson Sampling

Reinforcement Learning

Queueing theory
Abstract: Models of many real-life applications, such as queueing models of communication networks or computing systems, have a countably infinite state-space. Algorithmic and learning procedures that have been developed to produce optimal policies mainly focus on finite state settings, and do not directly apply to these models. To overcome this lacuna, in this work we study the problem of optimal control of a family of discrete-time countable state-space Markov Decision Processes (MDPs) governed by an unknown parameter $\theta\in\Theta$, 
 and defined on a countably-infinite state-space $\mathcal X=\mathbb{Z}_+^d$, with finite action space $\mathcal A$, and an unbounded cost function. We take a Bayesian perspective with the random unknown parameter $\boldsymbol{\theta}^*$ generated via a given fixed prior distribution on $\Theta$. To optimally control the unknown MDP, we propose an algorithm based on Thompson sampling with dynamically-sized episodes: at the beginning of each episode, the posterior distribution formed via Bayes' rule is used to produce a parameter estimate, which then decides the policy applied during the episode. To ensure the stability of the Markov chain obtained by following the policy chosen for each parameter, we impose ergodicity assumptions. From this condition and using the solution of the average cost Bellman equation, we establish an $\tilde O(dh^d\sqrt{|\mathcal A|T})$ upper bound on the Bayesian regret of our algorithm, where $T$ is the time-horizon. Finally, to elucidate the applicability of our algorithm, we consider two different queueing models with unknown dynamics, and show that our algorithm can be applied to develop approximately optimal control algorithms.

Bayesian Learning of Optimal Policies in Markov Decision Processes with Countably Infinite State-Space

Abstract

Models of many real-life applications, such as queueing models of communication networks or computing systems, have a countably infinite state-space. Algorithmic and learning procedures that have been developed to produce optimal policies mainly focus on finite state settings, and do not directly apply to these models. To overcome this lacuna, in this work we study the problem of optimal control of a family of discrete-time countable state-space Markov Decision Processes (MDPs) governed by an unknown parameter θ ∈ Θ , and defined on a countably-infinite state-space X = Z d + , with finite action space A , and an unbounded cost function. We take a Bayesian perspective with the random unknown parameter θ ∗ generated via a given fixed prior distribution on Θ . To optimally control the unknown MDP, we propose an algorithm based on Thompson sampling with dynamically-sized episodes: at the beginning of each episode, the posterior distribution formed via Bayes' rule is used to produce a parameter estimate, which then decides the policy applied during the episode. To ensure the stability of the Markov chain obtained by following the policy chosen for each parameter, we impose ergodicity assumptions. From this condition and using the solution of the average cost Bellman equation, we establish an ˜ O ( dh d √ |A| T ) upper bound on the Bayesian regret of our algorithm, where T is the time-horizon. Finally, to elucidate the applicability of our algorithm, we consider two different queueing models with unknown dynamics, and show that our algorithm can be applied to develop approximately optimal control algorithms.

1 Introduction

Many real-life applications, such as communication networks, supply chains, and computing systems, are modeled using queueing models with countably infinite state-space. In the existing analysis of these systems, the models are assumed to be known, but despite this, developing optimal control schemes is hard, with only a few examples worked out [35, 9, 54]. However, knowing the model, algorithmic procedures exist to produce approximately optimal policies [35] (such as value iteration and linear programming). Given the success of data-driven optimal control design, in particular Reinforcement Learning (RL), we explore the use of such methods for the countable state-space controlled Markov processes. However, current RL methods that focus on finite-state settings do not apply to the mentioned queueing models. With the model unknown, our goal is to develop a meta-learning scheme that is RL-based but obtains good performance by utilizing algorithms developed when models are known. Specifically, we study the problem of optimal control of a family of discrete-time countable state-space MDPs governed by an unknown parameter θ from a general space Θ with each MDP evolving on the countable state-space X = Z d + and finite action space A . The cost function is unbounded and polynomially dependent on the state, following the examples of minimizing waiting times in queueing systems. Taking a Bayesian view, we assume the model is governed by an unknown parameter θ ∗ ∈ Θ generated from a fixed and known prior distribution. We aim to learn a policy π that minimizes the optimal infinite-horizon average cost over a given class of policies Π with low Bayesian regret with respect to the (parameter-dependent) optimal policy in Π .

To avoid many technical difficulties in countably infinite state-space settings, it is crucial to establish certain assumptions regarding the class of models from which the unknown system is drawn; some examples are: i) the number of deterministic stationary policies is not finite; and ii) in average cost optimal control problems, without stability/ergodicity assumptions, an optimal policy may not exist [40], and when it exists, it may not be stationary or deterministic [20]. With these in mind, we assume that for any state-action pair, the transition kernels in the model class are categorical and skip-free to the right, i.e., with finite support with a bound depending on the state only in an additive manner; both are common features of queueing models where an increase in state is due to arrivals. A second set of assumptions ensure stability by assuming that the Markov chains obtained by using different policies in Π are geometrically ergodic with uniformity across Θ . From these assumptions, moments on hitting times are derived in terms of Lyapunov functions for polynomial ergodicity. These assumptions also yield a solution to the average cost optimality equation (ACOE) [9].

Contributions: To optimally control the unknown MDP, we propose an algorithm based on Thompson sampling with dynamically-sized episodes; posterior sampling is used based on its broad applicability and computational efficiency [46, 47]. At the beginning of each episode, a posterior distribution is formed using Bayes' rule, and an estimate is realized from this distribution which then decides the policy used throughout the episode. To evaluate the performance of our proposed algorithm, we use the metric of Bayesian regret, which compares the expected total cost achieved by a learning policy π L until time horizon T with the policy achieving the optimal infinite-horizon average cost in the policy class Π . We consider regret guarantees in three different settings as follows:

2. In Corollary 1, where class Π is a subset of all stationary policies and where we know the best policy within this subset for each parameter via an oracle, we prove an ˜ O ( dh d √ |A| T ) upper bound on the Bayesian regret of our proposed algorithm, relative to the best-in-class policy.

1. In Theorem 1, for Π being the set of all policies and assuming that we have oracle access to the optimal policy for each parameter, we establish an ˜ O ( dh d √ |A| T ) upper bound on the Bayesian regret of this algorithm compared to the optimal policy.

3. In Theorem 2, we explore a scenario where we have access to an approximately optimal policy, rather than the optimal policy in set Π (which are all assumed to be stationary policies). When the approximately optimal policies satisfy Assumptions 3-4, we prove an ˜ O ( dh d √ |A| T ) regret bound, relative to the optimal policy in set Π .

Finally, to provide examples of our framework for developing approximately optimal control algorithms for stochastic systems with unknown dynamics, we study two different queueing models that meet our technical conditions. The first example is a continuous-time queueing system with two heterogeneous servers with unknown service rates and a common infinite buffer with the decision being the use of the slower server. Here, the optimal policy that minimizes the average waiting time is a threshold policy [38] which yields a queue-length after which the slower server is always used. The second model is a two-server queueing system, each with separate infinite buffers, to one of which a dispatcher routes an incoming arrival. Here, the optimal policy minimizing the waiting time is a switching-curve [26] with the specifics unknown for general parameter values, so we find the best policy within a commonly used set of switching-curve policies (Max-Weight policies [58, 59]), and assign the arrival to the queue with minimum weighted queue-length. For both models, we verify our assumptions for the class of optimal/best-in-class policies corresponding to different service rates and conclude that our proposed algorithm can be used to learn the optimal/best-in-class policy.

Related Work: Thompson sampling [62], or posterior sampling, has been applied to RL in many contexts of unknown MDPs [55, 45] and partially observed MDPs [28]; see tutorials [22, 50] for a comprehensive survey. It has been used in the parametric learning context [6] to minimize either Bayesian [46, 47, 49, 1, 60, 61] or frequentist [5, 23] regret. The bulk of the literature, including [5, 23, 49], analyzes finite-state and finite-action models but with different parameterizations such that a general dependence of the models on the parameters is allowed. The work in [61] studies general state-space MDPs but with a scalar parameterization with a Lipschitz dependence of the underlying models. Our problem formulation specifically considers countable state-space models with the models related via ergodicity, which we believe is a natural choice. Our focus on parametric learning is also connected to older work in adaptive control [3, 24] which studies asymptotically optimal learning for general parameter settings but with either a finite or countably infinite number of policies. Learning-based asymptotically optimal control in queues has a long history [36, 35] but recently there is increased work that also characterizes finite-time regret performance with respect to a well-known good policy or the optimal policy; see [63] for a survey. A series of work has studied

learning with Max-Weight policies to get stability and linear regret [44, 30] or just stability [65]. A recent related work [18] considers learning optimal paramterized policies in queueing networks when the MDP is known. In a finite or countable state-space setting of specific queueing models where the parameters can be estimated, many works [2, 17, 53, 32, 31, 14, 21, 16] have used forced exploration type schemes to obtain either regret that is constant or scaling logarithmically in the time-horizon.

Another line of work studies the problem of learning the optimal policy in an undiscounted finitehorizon MDP with a bounded reward function. Reference [66] uses a Thompson sampling-based learning algorithm with linear value function approximation to study an MDP with a bounded reward function in a finite-horizon setting. Reference [15] considers an episodic finite-horizon MDP with known bounded rewards but unknown transition kernels modeled using linearly parameterized exponential families. A maximum likelihood (ML) based algorithm coupled with exploration done by constructing high probability confidence sets around the ML estimate is used to learn the unknown parameters. In another work, [48] extends the problem setting of [15] to an episodic finite-horizon MDP with unknown rewards and transitions modeled using parametric bilinear exponential families. To learn the unknown parameters, they use a ML based algorithm with exploration done with explicit perturbation. We note that all mentioned works consider a finite-horizon problem. In contrast, our work considers an average cost problem, an infinite-horizon setting, and provides finite-time performance guarantees. In addition, these works focus on an MDP with a bounded reward function. Our focus, however, is learning in MDPs with unbounded rewards with the goal of covering practical queueing examples. We note that the parameterization of transitions used in [48, 15] can be used within our framework. However, similar to our work, additional stability assumptions are necessary to guarantee asymptotic learning and sub-linear regret. Another issue with exponential transition families is that they do not allow for 0 entries, which limits their applicability in queueing models.

In another work, [51] studies discounted MDPs with unknown dynamics, and unbounded state-space, but with bounded rewards, and learns an online policy that satisfies a specific notion of stability. It is also assumed that a Lyapunov function ensuring stability for the optimal policy exists. We note that [51] ignores optimality and focuses on finding a stable policy, which contrasts with our work that evaluates performance relative to the optimal policy. Secondly, [51] considers a discounted reward problem, essentially a finite-time horizon problem. Average cost problems, such as ours, are infinite-time horizon problems, so connections to discounted problems can only be made in the limit of the discount parameter going to 1 . Moreover, [51] considers a bounded reward function, simplifying their analysis but not practical for many queueing examples. Further, the assumption of a stable optimal policy with a Lyapunov function (as in [51]) is highly restrictive for bounded reward settings with discounting. Additionally, average cost problems with bounded costs need strong state-independent recurrence conditions for the existence of (stationary) optimal solutions, which many queueing examples don't satisfy; see [12]. Further complications can also arise with bounded costs: e.g., [20] shows that a stationary average cost optimal policy may not exist.

2 Problem formulation

We consider a family of discrete-time Markov Decision Processes (MDPs) governed by parameter θ ∈ Θ with the MDP for parameter θ described by ( X , A , c, P θ ) . For exposition purposes, we assume that all the MDPs are on (a common) countably infinite state-space X = Z d + . We denote the finite action space by A , the transition kernel by P θ : X ×A → ∆( X ) , and the cost function by c : X ×A → R + . As mentioned earlier, we will take a Bayesian view of the problem and assume that the model is generated using an unknown parameter θ ∗ ∈ Θ , which is generated from a given fixed prior distribution ν ( · ) on Θ . Our goal is to find a policy π : X → A that tries to achieve Bayesian optimal performance in policy class Π , i.e., minimizes the expected regret with θ ∗ chosen from the prior distribution ν ( · ) . For each value θ ∈ Θ , the minimum infinite-horizon average cost is defined as

where we optimize over a given class of policies Π and X ( t ) = ( X 1 ( t ) , . . . , X d ( t )) ∈ X and A ( t ) ∈ A are the state and action at t ∈ N . Typically, we set this class to be all (causal) policies, but it is also possible to consider Π to be a proper subset of all policies as we will explore in our results. For a learning policy π L that aims to select the optimal control without model knowledge but with

knowledge of Θ and the prior ν , the Bayesian regret until time horizon T ≥ 2 is defined as

where the expectation is taken over θ ∗ ∼ ν and the dynamics induced by π L . Owing to underlying challenges in countable state-space MDPs, we require the below assumptions on the cost function.

Assumption 1. The cost function c : X ×A → R + is assumed to satisfy the following two conditions:

For every number z ≥ 0 and action a ∈ A , c ( x , a ) ≥ z outside a finite subset of X .

The cost function is upper-bounded by a multivariate polynomial f c : Z d + → R + which is increasing in every component on x ∈ Z d + and has maximum degree of r ( ≥ 1) in any dimension. We can assume that f c ( x ) = K ∑ d i =1 ( x i ) r for some K > 0 , where x = ( x 1 , . . . , x d ) .

Thus, the cost function increases without bound (in the state) at a polynomial rate. This assumption is common in practice-holding costs in queueing models are polynomial in the state components. To avoid technical issues the infinite state-space setting also necessitates some assumptions on the class from which the unknown model is drawn. For instance, irreducibility of Markov chains on such state-spaces does not ensure positive recurrence (and ergodicity). Moreover, for average cost optimal control problems, without stability even the existence of an optimal policy is not guaranteed, and we need more conditions. The following assumption ensures a skip-free behaviour for transitions, which holds in many queueing models, where an increase in state corresponds to (new) arrivals.

Assumption 2. From any state-action pair ( x , a ) , the transition is to a finite number of states. We also assume that all transition kernels are skip-free to the right: for some h ≥ 1 which is independent of θ ∈ Θ and ( x , a ) ∈ X ×A , we have P θ ( x ′ ; x , a ) = 0 for all x ′ ∈ { ˜ x ∈ Z d + : ∥ ˜ x ∥ 1 > ∥ x ∥ 1 + h } .

Learning necessitates some commonalities within the class of models so that using a policy well-suited to one model provides information on other models too. For us, these are in the form of constraints on the transition kernels of the models and stability assumptions. As simple union bound arguments don't work in the countably infinite state-space setting, we will use the stability assumptions instead. In our setting, we consider a class of models, each with a policy being well-suited to at least one model in the class, and use the set of policies to search within. Using a reduced set of policies is necessary as the number of deterministic stationary policies is infinite. To learn correctly while restricting attention to this subset policy class, requires some regularity assumptions when a policy well-suited to one model is tried on a different model. Our ergodicity assumptions are one convenient choice; see Appendix A.1 for details. These assumptions let us characterize the distributions of the first passage times of the Markov processes via stability conditions; see Lemmas 10 and 11.

Assumption 3. For any MDP ( X , A , c, P θ ) with parameter θ ∈ Θ , there exists a unique optimal policy π ∗ θ that minimizes the infinite-horizon average cost within the class of policies Π . Furthermore, for any θ 1 , θ 2 ∈ Θ , the Markov process with transition kernel P π ∗ θ 2 θ 1 obtained from the MDP ( X , A , c, P θ 1 ) by following policy π ∗ θ 2 is irreducible, aperiodic, and geometrically ergodic with geometric ergodicity coefficient γ g θ 1 ,θ 2 ∈ (0 , 1) and stationary distribution µ θ 1 ,θ 2 . This is equivalent to the existence of finite set C g θ 1 ,θ 2 and Lyapunov function V g θ 1 ,θ 2 : X → [1 , + ∞ ) satisfying

Then, we have the following assumptions relating all the models in Θ :

The geometric ergodicity coefficient is uniformly bounded below 1 : γ g ∗ := sup θ 1 ,θ 2 ∈ Θ γ g θ 1 ,θ 2 < 1 .

We assume that { 0 d } ⊆ ∩ θ 1 ,θ 2 ∈ Θ C g θ 1 ,θ 2 and C g ∗ = ∪ θ 1 ,θ 2 ∈ Θ C g θ 1 ,θ 2 is a finite set. We further assume that b g ∗ := sup θ 1 ,θ 2 b g θ 1 ,θ 2 < + ∞ .

Remark 1. The uniqueness of the optimal policy is not essential for the validity of our results, provided that all optimal policies satisfy our assumptions. When this condition is not met, we need to select an optimal policy that is geometrically ergodic for all θ ∈ Θ . This issue can be avoided by using a smaller subset of policies for which ergodicity can be shown, such as Max-Weight policies.

̸

Geometric ergodicity implies that all moments of the hitting time of state 0 d , say τ 0 d , from any initial state x = 0 d are finite as E x [ κ τ 0 d ] ≤ c 1 V g ( x ) (for specific κ > 1 and c 1 ), and so, E x [ τ k 0 d ] ≤ c 1 V g ( x ) k ! / log k ( κ ) < + ∞ for all k ∈ N ; see Appendix A.2. Function V g is typically exponential in some norm of the state and yields an exponential bound for moments of hitting times, and a poor regret bound. To improve the regret bound, we need a different drift equation with function V p with polynomial dependence on a norm of the state that bounds certain polynomial moments of τ 0 d .

Assumption 4. Given θ 1 , θ 2 ∈ Θ , Markov process obtained from MDP ( X , A , c, P θ 1 ) by following policy π ∗ θ 2 is polynomially ergodic through the Foster-Lyapunov criteria: there exists a finite set C p θ 1 ,θ 2 , constants β p θ 1 ,θ 2 , b p θ 1 ,θ 2 > 0 , α p θ 1 ,θ 2 ∈ [ r r +1 , 1) , and function V p θ 1 ,θ 2 : X → [1 , + ∞ ) satisfying

Then, we have the following assumptions relating all the models in Θ :

V p θ 1 ,θ 2 is a polynomial with positive coefficients, maximum degree (in any dimension) r p θ 1 ,θ 2 , and sum of coefficients s p θ 1 ,θ 2 . We assume r p ∗ = sup θ 1 ,θ 2 r p θ 1 ,θ 2 < ∞ and s p ∗ = sup θ 1 ,θ 2 s p θ 1 ,θ 2 < ∞ .

We assume that { 0 d } ⊆ ∩ θ 1 ,θ 2 ∈ Θ C p θ 1 ,θ 2 and C p ∗ = ∪ θ 1 ,θ 2 ∈ Θ C p θ 1 ,θ 2 is a finite set. We further assume that β p ∗ := inf θ 1 ,θ 2 β p θ 1 ,θ 2 > 0 and b p ∗ := sup θ 1 ,θ 2 b p θ 1 ,θ 2 < ∞ .

Let K θ 1 ,θ 2 ( x ) := ∑ ∞ n =0 2 -n -2 ( P π ∗ θ 2 θ 1 ) n ( x , 0 d ) , which is positive for any pair θ 1 , θ 2 ∈ Θ by irreducibility. We assume that it is strictly positive in Θ : K ∗ := inf θ 1 ,θ 2 min x ∈ C p ∗ K θ 1 ,θ 2 ( x ) > 0 .

Assumptions 3-4 hold in many models of interest; see Appendix E. As average cost optimality is our design criterion, we need to ensure the existence of solutions to ACOE when Π is the set of all policies, or Poisson equation when Π is a subset of all policies. We discuss these two cases separately. Case 1: Π is the set of all policies. For any parameter θ ∈ Θ , the MDP ( X , A , c, P θ ) is said to satisfy the ACOE if there exists a constant J ( θ ) and a unique function v ( · ; θ ) : X → R such that

From [13] if the following conditions hold, ACOE has a solution, J θ is the optimal infinite-horizon average cost, and there is an optimal stationary policy with ACOE becoming (5): (i) for every ( x , a ) and z ≥ 0 , cost function c ( x , a ) ≥ z outside a finite subset of X ; (ii) there is a stationary policy with an irreducible and aperiodic Markov process with finite average cost; and (iii) from every ( x , a ) transition to a finite number of states is possible. From Assumptions 1-3, the above conditions hold. Case 2: Π is a proper subset of all policies. Here, we posit that for every θ ∈ Θ and its best in-class policy π ∗ θ , there exists a constant J ( θ ) , the average cost of π ∗ θ , and a function v ( · ; θ ) : X → R with

This holds by the solution of the Poisson equation with the appropriate forcing function. For a Markov process X on the space X with transition kernel P and cost function ¯ c ( · ) , a solution to the Poisson equation [41] is a scalar J and function v ( · ) : X ↦→ R such that J + v = ¯ c + Pv , where v ( z ) = 0 for some z ∈ X . In our setting using [41, Sections 9.6-9.8], for a model governed by θ ∈ Θ following policy π ∗ θ , we show a solution to the Poisson equation exists and is given by v π ∗ θ (0 d ) = 0 , and

∗

θ

where

¯

C

π

(

x

) =

E

π

∗

θ

x

[ ∑

-

τ

i

0

d

=0

c

(

X

∗

(

i

)

, π

∗

θ

(

X

(

i

)))

]

,

and expectation is over trajectories of Markov chain X with transition kernel P π θ θ starting in state x . In Appendix A.3, we present related definitions and show that from Assumptions 3-4, the requirements for the existence and finiteness of the solutions to Poisson equation are satisfied. Finally, we assume sup θ ∈ Θ J ( θ ) is finite, which typically holds as a result of the boundedness assumptions stated in Asumptions 3 or 4, along with Assumption 1.

1

̸

Assumption 5. We assume that J ∗ := sup θ ∈ Θ J ( θ ) < + ∞ .

3 Thompson sampling based learning algorithm

We will use the learning algorithm Thompson sampling with dynamically-sized episodes from [49] to learn the unknown parameter θ ∗ ∈ Θ and the corresponding policy, π ∗ θ ∗ , but suitably modify it for our countable state-space setting. Consider the prior distribution ν 0 = ν defined on Θ from which θ ∗ is sampled. At each time t ∈ N , the posterior distribution ν t is updated according to Bayes' rule as

and the posterior estimate θ t +1 , if generated, is from the posterior distribution ν t +1 . The modified Thompson-sampling with dynamically-sized episodes algorithm (TSDE) is presented in Algorithm 1. The TSDE algorithm operates in episodes: at the beginning of episode k , parameter θ k is sampled from the posterior distribution ν t k and during episode k , actions are generated from the stationary policy according to θ k , i.e., π ∗ θ k . Let t k be the time the k -th episode begins. Define ˜ t k +1 as the first time after t k that the conditions of Line 6 of Algorithm 1 is triggered and t k +1 as the first time at or after ˜ t k +1 where state 0 d is visited; for the last episode started before or at T , we ensure that t k and ˜ t k are less than or equal T +1 . Explicitly, t 1 = 1 and for k > 1 , t k = min { t ≥ ˜ t k : X ( t ) = 0 d or t > T } . Let T k = t k +1 -t k be the length of the k -th episode and set ˜ T k = ˜ t k +1 -t k with the convention ˜ T 0 = 1 . For any state-action pair ( x , a ) , we define N 1 ( x , a ) = 0 and for t > 1 ,

Notice that for all state-action pairs ( x , a ) and ˜ t k +1 ≤ t ≤ t k +1 , we have N t ( x , a ) = N ˜ t k +1 ( x , a ) . We denote K T as the number of episodes started by or at time T , or K T = max { k : t k ≤ T } . The length of episode k < K T is not fixed and is determined according to two stopping criteria: (1) t > t k + ˜ T k -1 , (2) N t ( x , a ) > 2 N t k ( x , a ) for some state-action pair ( x , a ) . After either criterion is met, the system will still follow policy π ∗ θ k until the first time at which state 0 d is visited; see Line 14 and Figure 1. We use this settling period to 0 d because the system state can be arbitrary when the

first stopping criterion is met. As the countable state-space setting precludes a simple union-bound argument to overcome this uncertainty (as in the literature for finite state settings), we let the system reach the special state 0 d . Another (essentially equivalent) option is to wait until the state hits the finite set C g ∗ or C p ∗ and then use a union bound argument for all states in either set. For analytical convenience, we only use the state samples observed before arrival ˜ t k +1 to update the posterior distribution. The posterior update is halted during the settling period to 0 d as we have no control on the states visited during it, despite it being finite in duration (by our assumptions).

4 Regret analysis of Algorithm 1

The performance of any learning policy π L is evaluated using the metric of expected regret compared to the optimal expected average cost of true parameter θ ∗ , namely, J ( θ ∗ ) . In this section, we evaluate the performance of Algorithm 1 and derive an upper bound for R ( T, π TSDE ) , its expected regret up to time T . In Section 2, we argued that at time t in episode k ( t k ≤ t < t k +1 ), there exist a constant J ( θ k ) and a unique function v ( · ; θ k ) : X → R such that v ( 0 d ; θ k ) = 0 and

in which π ∗ θ k is the optimal or best-in-class policy (depending on the context) according to parameter θ k and J ( θ k ) is the average cost for the Markov process obtained from MDP ( X , A , c, P θ k ) by following π ∗ θ k . We derive a bound for the expected regret R ( T, π TSDE ) following the proof steps of [49] while extending it to the countable state-space setting of our problem. Using (8), the regret is decomposed into three terms and each term is bounded separately:

Before bounding the above regret terms, we address the complexities arising from the countable state-space setting. Firstly, we need to study the maximum state (with respect to the ℓ ∞ -norm) visited up to time T in the MDP ( X , A , c, P θ ∗ ) following Algorithm 1; we denote this maximum state by M T θ ∗ . In Appendix C, we derive upper bounds on the moments of hitting times of state 0 d and utilize this to bound the moments of random variable M T θ ∗ , which then lets us study the number of episodes K T by time T . Another challenge in analyzing the regret is that the relative value function v ( x ; θ ) is unlikely to be bounded in the countable state-space setting. Hence, in (13) and (14), we find bounds for the relative value function in terms of hitting time τ 0 d from the initial state x . Based on these results, we provide an upper bound for the regret of Algorithm 1 in Theorem 1.

Maximum state norm under polynomial and geometric ergodicity. Here we state the results that characterize the maximum l ∞ -norm of the state vector achieved up until and including time T , and

the resulting bounds on the number of episodes executed until time T . Owing to space constraints the details (including formal statements) are presented in Appendix B. The results are listed as below: (i) In Lemma 6, we bound the moments of the maximum length of recurrence times of state 0 d , using the ergodicity assumptions 3 and 4. This, along with the skip-free property, allows us to prove that the p -th moment of max 1 ≤ i ≤ T τ ( i ) 0 d and M T θ ∗ are both of order O (log p T ) .

(ii) In Lemma 7, we find an upper bound for the number of episodes in which the second stopping criterion is met or there exists a state-action pair for which N t ( x , a ) has increased more than twice. (iii) In Lemma 8, we bound the total number of episodes K T by time T by bounding the number of episodes triggered by the first stopping criterion, using the fact that in such episodes, ˜ T k = ˜ T k -1 +1 . Moreover, to account for the settling time of each episode, we use geometric ergodicity and Lemma 6. It follows that the expected value of the number of episodes K T is of the order ˜ O ( √ h d |A| T ) .

Regret analysis. Next, we bound regret terms R 0 , R 1 and R 2 using the approach of [49] along with additional arguments to extend their result to a countably infinite state-space. We consider the relative value function v ( x ; θ ) of policy π ∗ θ introduced for the optimal policy in ACOE or for the best in-class policy in the Poisson equation. In either of these cases, policy π ∗ θ satisfies (5), which is the corresponding Poisson equation with forcing function c ( x , π ∗ θ ( x )) in a Markov chain with transition matrix P π ∗ θ θ . In (6), we presented the solution ( J, v ) to the Poisson equation, which yields the following upper bound for the relative value function, as argued in Appendix A.3:

We can similarly lower bound the relative value function using Assumption 5 as

From Assumption 3, all moments of τ 0 d and thus, the derived bounds are finite. Also, in Lemma 10 we bound the moments of τ 0 d of order i ≤ r +1 using the polynomial Lyapunov function V p θ 1 ,θ 2 , which is then used to bound the expected regret. We next bound the first regret term R 0 from the first stopping criterion in terms of the number of episodes K T and the settling time of each episode k .

Lemma 1. The first regret term R 0 satisfies R 0 ≤ J ∗ E [ K T (max 1 ≤ i ≤ T τ ( i ) 0 d +1)] .

Proof of Lemma 1 is given in Appendix B.4. From Lemma 6, all moments of max 1 ≤ i ≤ T τ ( i ) 0 d are bounded by a polylogarithmic function. Futhermore, as a result of Lemma 8, expected value of the number of episodes K T is of the order ˜ O ( √ h d |A| T ) , which leads to a ˜ O ( √ h d |A| T ) regret term R 0 . Next, an upper bound on R 1 defined in (11) is derived. In the proof of Lemma 2 we argue that as the relative value function is equal to 0 at all time instances t k for k ≤ K T , the only term that contributes to the regret is the value function at the end of time horizon T . We use the lower bound derived in (14) to show that the second regret term R 1 is ˜ O (1) ; the proof is given in Appendix B.5.

Lemma2. The second regret term R 1 satisfies R 1 ≤ c 2 E [( M T θ ∗ ) r p ∗ ]+ c 3 , where c 2 = J ∗ 2 r p ∗ s p ∗ ( β p ∗ ) -1 and c 3 = J ∗ ( β p ∗ ) -1 ( s p ∗ (2 h ) r p ∗ + b p ∗ ( K ∗ ) -1 ) .

From Lemma 6, E [( M T θ ∗ ) r p ∗ ] is O (log r p ∗ T ) ; hence, R 1 is upper bounded by a polylogarithmic function of the order r p ∗ . Finally, in Lemma 3, we derive an upper bound for the third regret term R 2 defined in (12) using the bound derived for the relative value function in (13). To bound R 2 , we characterize it in terms of the difference between the empirical and true unknown transition kernel and following the concentration method used in [64, 10, 49, 7, 8], we argue that with high probability the total variation distance between the two distributions is small; for proof, see Appendix B.6.

Lemma 3. For problem-dependent constant c p 3 and polynomial Q ( T ) = c p 3 ( Th ) r + r p ∗ / 48 , we have

The above Lemma results in a ˜ O ( KrdJ ∗ h d +2 r + r p ∗ √ |A| T ) regret term as a result of Lemma 6, where h is the skip-free parameter defined in Assumption 2, d is the dimension of the state-space, K and r are the cost function parameters defined in Assumption 1, J ∗ is the supremum on the optimal cost, r p ∗ is defined in Assumption 4, and where ˜ O hides logarithmic factors in problem parameters one of which is log d + r + r p ∗ +2 ( T ) . For simplicity, we have not included the Lyapunov

functions related parameters in the regret. Finally, from Lemmas 1, 2, 3, along with the CauchySchwarz inequality, we conclude that the regret of Algorithm 1 R ( T, π TSDE )(= R 0 + R 1 + R 2 ) is ˜ O ( KrdJ ∗ h d +2 r + r p ∗ √ |A| T ) ; for brevity, we will state that regret is of the order ˜ O ( dh d √ |A| T ) .

Theorem 1. Under Assumptions 1-5, the regret of Algorithm 1, R ( T, π TSDE ) , is ˜ O ( dh d √ |A| T ) .

Theorem 1 can be extended to the problem of finding the best policy within a sub-class of policies in set Π , which may or may not contain the optimal policy. In Section 2, we stated that Assumptions 3 and 4 hold for policies in Π and we used this to argue that the Poisson equation has a solution given in (6). As a result, repeating the same arguments as in Theorem 1 with the modification that π ∗ θ is the best in-class policy of the MDP governed by parameter θ , yields the following corollary.

Corollary 1. Under Assumptions 1 through 5, the regret of Algorithm 1 when using the best in-class policy is ˜ O ( dh d √ |A| T ) .

Requirement of an optimal policy oracle. To implement our algorithm, we need to find the optimal policy for each model sampled by the algorithm-optimal policy for Theorem 1 and optimal policy within policy class Π for Corollary 1. In the finite state-space setting, [49] provides a schedule of ϵ values and selects ϵ -optimal policies to obtain ˜ O ( √ T ) regret guarantees. The issue with extending the analysis of [49] to the countable state-space setting is that we need to ensure (uniform) ergodicity for the chosen ϵ -optimal policies. Another issue is that, to the best of our knowledge, there isn't a general structural characterization of all ϵ -optimal stationary policies for countable state-space MDPs or even a characterization of the policy within this set that is selected by any computational procedure in the literature; current results only discuss characterization of the stationary optimal policy. In the absence of such results, stability assumptions with the same uniformity across models as in our submission will be needed, which are likely too strong to be useful. However, if we could verify the stability requirements of Assumptions 3 and 4 for a subset of policies, the optimal oracle is not needed, and instead, by choosing approximately optimal policies within this subset, we can follow the same proof steps as [49] to guarantee regret performance similar to Corollary 1 (without knowledge of model parameters). Thus, in Theorem 2 we extend the previous regret guarantees to the algorithm employing ϵ -optimal policy; proof is given in Appendix B.8.

Theorem 2. Consider a non-negative sequence { ϵ k } ∞ k =1 such that for every k ∈ N , ϵ k is bounded above by 1 k +1 and an ϵ k -optimal policy satisfying Assumptions 3 and 4 is given. The regret incurred by Algorithm 1 while using the ϵ k -optimal policy during any episode k is ˜ O ( dh d √ |A| T ) .

5 Evaluation and Conclusion: Application of Algorithm 1 to queueing models

Next, we present an evaluation of our algorithm. We study two different queueing models shown in Figure 2, each with Poisson arrivals at rate λ , and two heterogeneous servers with exponentially distributed services times with unknown service rate vector θ ∗ = ( θ ∗ 1 , θ ∗ 2 ) . Vector θ ∗ is sampled from the prior distribution ν defined on the space Θ given as Θ = { ( θ 1 , θ 2 ) ∈ R 2 + : λ θ 1 + θ 2 ≤ 1 -δ 1+ δ , 1 ≤ θ 1 θ 2 ≤ R } , for fixed R ≥ 1 and δ ∈ (0 , 0 . 5) . The first condition ensures the stability of the queueing models, while the second guarantees the compactness of the parameter space of the parameterized policies. In both systems, the goal of the dispatcher is to minimize the expected sojourn time of jobs, which by Little's law [52] is equivalent to minimizing the average number of jobs in the system. After verifying Assumptions 1-5 in Appendix E for the cost function c ( x ) = ∥ x ∥ 1 , Theorem 1 yields a Bayesian regret of order ˜ O ( √ |A| T ) for Algorithm 1.

Model 1. Two-server queueing system with a common buffer. We consider the continuous-time queueing system of Figure 2a, where the countable state space is X = { x = ( x 0 , x 1 , x 2 ) ∈ Z + ×{ 0 , 1 } 2 } , where x 0 is the queue length, and x i , i = 1 , 2 equal 1 if server i is busy. The action space is A = { h, b, 1 , 2 } , where h means no action, b sends a job to both servers, and i = 1 , 2 assigns a job to server i . In [38], it is shown that by uniformization [39] and sampling the continuous-time Markov process at rate λ + θ ∗ 1 + θ ∗ 2 , a discrete-time Markov chain is obtained, which converts the original continuous-time problem to an equivalent discrete-time problem where we need to minimize lim sup T →∞ T -1 ∑ T -1 t =0 ∥ X ( t ) ∥ 1 . Further, [38] shows that the optimal policy is a threshold policy π t ( θ ∗ ) with optimal finite threshold t ( θ ∗ ) ∈ N : always assign a job to the faster (first) server when free, and to the second server if it is free and ∥ x ∥ 1 > t ( θ ∗ ) , and take no action otherwise. In

θ

∗

1

θ

∗

2

λ

Infinite queue

(a) Queueing system with a common buffer.

θ

∗

1

θ

∗

2

λ

Infinite queue

(b) Two parallel queues.

(a) Queueing system of Figure 2a.

(b) Queueing system of Figure 2b.

50

入=0.3

40

入=0.5

入=0.7

Regret

30

20

10

0

1

2

3

4

5

T

1e4

200

入=0.3

入=0.5

150

入=0.7

gret

100

50

o+

0.0

0.5

1.0

1.5

2.0

2.5

T

1e5

Appendix E.1, we argue that the discrete-time Markov process governed by θ ∈ Θ and following threshold policy π t for any threshold t belonging to a compact set satisfies Assumptions 1-5.

Model 2. Two heterogeneous parallel queues. We consider the continuous-time queueing system of Figure 2b with countable state space X = { x = ( x 1 , x 2 ) ∈ Z 2 + } , where x i is the number of jobs in the server-queue pair i . The action space is A = { 1 , 2 } , where action i sends the arrival to queue i . We obtain the discrete-time MDP by sampling the queueing system at the arrivals, and then aim to find the average cost minimizing policy within the class Π = { π ω ; ω ∈ [( c R R ) -1 , c R R ] } , c R ≥ 1 . Policy π ω : X → A routes arrivals based on the weighted queue lengths: π ω ( x ) = arg min (1 + x 1 , ω (1 + x 2 )) with ties broken for 1 . Even with the transition kernel fully specified (by the values of arrival and service rates), the optimal policy in Π is not known except when θ 1 = θ 2 where the optimal value is ω = 1 , and so, to learn it, we will use Proximal Policy Optimization for countable state-space MDPs [18]. Note that [18] requires full model knowledge, which holds in our scheme as we use parameters sampled from the posterior for choosing the policy at the beginning of each episode. In Appendix E.2, we argue that the discrete-time Markov process governed by parameter θ ∈ Θ and following policy π ω for ω ∈ [( c R R ) -1 , c R R ] satisfies Assumptions 1-5.

Next, we report the numerical results of Algorithm 1 in the two queueing models of Figure 2 and calculate regret using (2). The regret is averaged over 2000 simulation runs and plotted against the number of transitions in the sampled discrete-time Markov process. Figure 3 shows the behavior of the regret of the two queuing models for three different arrival rates and service rates distributed according to a Dirichlet prior over [0 . 5 , 1 . 9] 2 . We observe that the regret is sub-linear in time and grows as the arrival rate increases. For the queueing model of Figure 2a, the minimum average cost J ( θ ) and optimal policy π ∗ θ are known explicitly [38] for every θ ∈ Θ , which are used in Algorithm 1 and for regret calculation. Conversely, for the second queueing model, J ( θ ) and π ∗ θ are not known. The PPO algorithm [18] is used to empirically find both the optimal weight and the policy's average cost. Additional details of the simulations and more plots are presented in Appendix G.

Conclusions and future work. We studied the problem of learning optimal policies in countable state-space MDPs governed by unknown parameters. We proposed a learning policy based on Thompson sampling and established finite-time performance guarantees on the Bayesian regret. We highlighted the practicality of our proposed algorithm by considering two different queuing models and showing that our algorithm can be applied to develop optimal control policies. For future work we plan two directions to explore: to generalize our algorithm to consider polices that might not all be stabilizing, and also to simplify the algorithm using ideas from [61, 57].

Disclosure of Funding

SA's research was supported by NSF via grants ECCS2038416, CCF2008130, and CNS1955777, and a grant from General Dynamics via MIDAS at the University of Michigan, Ann Arbor. VS's research is supported in part by NSF via grants ECCS2038416, CCF2008130, CNS1955777, and CMMI2240981.

References

Yasin Abbasi-Yadkori and Csaba Szepesvari. Bayesian optimal control of smoothly parameterized systems: The lazy posterior sampling algorithm. arXiv preprint arXiv:1406.3926 , 2014.

Saghar Adler, Mehrdad Moharrami, and Vijay Subramanian. Learning a discrete set of optimal allocation rules in queueing systems with unknown service rates. arXiv preprint arXiv:2202.02419 , 2022.

R. Agrawal, D. Teneketzis, and V. Anantharam. Asymptotically efficient adaptive allocation schemes for controlled Markov chains: Finite parameter space. IEEE Transactions on Automatic Control , 34(12):1249-1259, 1989.

Rajeev Agrawal and Demosthenis Teneketzis. Certainty equivalence control with forcing: Revisited. Systems & Control Letters , 13(5):405-412, 1989.

Shipra Agrawal and Randy Jia. Optimistic posterior sampling for reinforcement learning: Worst-case regret bounds. Advances in Neural Information Processing Systems , 30, 2017.

Shipra Agrawal and Randy Jia. Learning in structured MDPs with convex cost functions: Improved regret bounds for inventory management. In Proceedings of the 2019 ACM Conference on Economics and Computation , pages 743-744, 2019.

Nima Akbarzadeh and Aditya Mahajan. On learning Whittle index policy for restless bandits with scalable regret. arXiv preprint arXiv:2202.03463 , 2022.

Nima Akbarzadeh and Aditya Mahajan. On learning Whittle index policy for restless bandits with scalable regret. IEEE Transactions on Control of Network Systems , 2023.

Aristotle Arapostathis, Vivek S Borkar, Emmanuel Fernández-Gaucherand, Mrinal K Ghosh, and Steven I Marcus. Discrete-time controlled Markov processes with average cost criterion: A survey. SIAM Journal on Control and Optimization , 31(2):282-344, 1993.

Peter Auer, Thomas Jaksch, and Ronald Ortner. Near-optimal regret bounds for reinforcement learning. Advances in Neural Information Processing Systems , 21, 2008.

V S Borkar. The Kumar-Becker-Lin scheme revisited. Journal of Optimization Theory and Applications , 66:289-309, 1990.

Rolando Cavazos-Cadena. Necessary conditions for the optimality equation in average-reward Markov decision processes. Applied Mathematics and Optimization , 19(1):97-112, 1989.

Rolando Cavazos-Cadena. Weak conditions for the existence of optimal stationary policies in average Markov decision chains with unbounded costs. Kybernetika , 25(3):145-156, 1989.

Tuhinangshu Choudhury, Gauri Joshi, Weina Wang, and Sanjay Shakkottai. Job dispatching policies for queueing systems with unknown service rates. In Proceedings of the Twenty-Second International Symposium on Theory, Algorithmic Foundations, and Protocol Design for Mobile Networks and Mobile Computing , MobiHoc '21, page 181-190, New York, NY, USA, 2021. Association for Computing Machinery.

Sayak Ray Chowdhury, Aditya Gopalan, and Odalric-Ambrym Maillard. Reinforcement learning in parametric MDPs with exponential families. In International Conference on Artificial Intelligence and Statistics , pages 1855-1863. PMLR, 2021.

Asaf Cohen, Vijay Subramanian, and Yili Zhang. Learning-based optimal admission control in a single-server queuing system. Stochastic Systems , 2024.

Asaf Cohen, Vijay G. Subramanian, and Yili Zhang. Learning-based optimal admission control in a single server queuing system. https://arxiv.org/abs/2212.11316, 2022.

Jim G Dai and Mark Gluzman. Queueing network controls via deep reinforcement learning. Stochastic Systems , 12(1):30-67, 2022.

Anthony Ephremides, Pravin Varaiya, and Jean Walrand. A simple dynamic routing problem. IEEE Transactions on Automatic Control , 25(4):690-693, 1980.

Lloyd Fisher and Sheldon M Ross. An example in denumerable decision processes. The Annals of Mathematical Statistics , 39(2):674-675, 1968.

Daniel Freund, Thodoris Lykouris, and Wentao Weng. Efficient decentralized multi-agent learning in asymmetric queuing systems. In Conference on Learning Theory , pages 4080-4084. PMLR, 2022.

Mohammad Ghavamzadeh, Shie Mannor, Joelle Pineau, and Aviv Tamar. Bayesian reinforcement learning: A survey. Foundations and Trends® in Machine Learning , 8(5-6):359-483, 2015.

Aditya Gopalan and Shie Mannor. Thompson sampling for learning parameterized Markov decision processes. In Conference on Learning Theory , pages 861-898. PMLR, 2015.

Todd L Graves and Tze-Leung Lai. Asymptotically efficient adaptive choice of control laws incontrolled Markov chains. SIAM Journal on Control and Optimization , 35(3):715-743, 1997.

Bruce Hajek. Hitting-time and occupation-time bounds implied by drift analysis with applications. Advances in Applied Probability , 14(3):502-525, 1982.

Bruce Hajek. Optimal control of two interacting service stations. IEEE Transactions on Automatic Control , 29(6):491-499, 1984.

Arie Hordijk and Flora Spieksma. On ergodicity and recurrence properties of a Markov chain by an application to an open Jackson network. Advances in Applied Probability , 24(2):343-376, 1992.

Mehdi Jafarnia Jahromi, Rahul Jain, and Ashutosh Nayyar. Online learning for unknown partially observable MDPs. In International Conference on Artificial Intelligence and Statistics , pages 1712-1732. PMLR, 2022.

Soren F Jarner and Gareth O Roberts. Polynomial convergence rates of Markov chains. The Annals of Applied Probability , 12(1):224-247, 2002.

Subhashini Krishnasamy, PT Akhil, Ari Arapostathis, Rajesh Sundaresan, and Sanjay Shakkottai. Augmenting Max-Weight with explicit learning for wireless scheduling with switching costs. IEEE/ACM Transactions on Networking , 26(6):2501-2514, 2018.

Subhashini Krishnasamy, Ari Arapostathis, Ramesh Johari, and Sanjay Shakkottai. On learning the c µ rule in single and parallel server networks. https://arxiv.org/abs/1802.06723, 2018.

Subhashini Krishnasamy, Rajat Sen, Ramesh Johari, and Sanjay Shakkottai. Learning unknown service rates in queues: A multiarmed bandit approach. Operations Research , 69(1):315-330, 2021.

P R Kumar and A Becker. A new family of optimal adaptive controllers for Markov chains. IEEE Transactions on Automatic Control , 27(1):137-146, 1982.

P R Kumar and Woei Lin. Optimal adaptive controllers for unknown Markov chains. IEEE Transactions on Automatic Control , 27(4):765-774, 1982.

P R Kumar and Pravin Varaiya. Stochastic systems: Estimation, identification, and adaptive control . SIAM, 2015.

Tze-Leung Lai and Sidney Yakowitz. Machine learning and nonparametric bandit theory. IEEE Transactions on Automatic Control , 40(7):1199-1209, 1995.

Ronald Larsen. Control of multiple exponential servers with application to computer systems . PhD thesis, University of Maryland, 1981.

Woei Lin and P R Kumar. Optimal control of a queueing system with two heterogeneous servers. IEEE Transactions on Automatic Control , 29(8):696-703, 1984.

Steven A Lippman. Applying a new device in the optimization of exponential queuing systems. Operations Research , 23(4):687-710, 1975.

Ashok P. Maitra. Dynamic programming for countable state systems . PhD thesis, University of California, Berkeley, 1963.

Armand M Makowski and Adam Shwartz. The Poisson equation for countable Markov chains: Probabilistic methods and interpretations. Handbook of Markov Decision Processes: Methods and Applications , pages 269-303, 2002.

Akshay Mete, Rahul Singh, Xi Liu, and P R Kumar. Reward biased maximum likelihood estimation for reinforcement learning. In Learning for Dynamics and Control , pages 815-827. PMLR, 2021.

Sean P Meyn and Richard L Tweedie. Markov chains and stochastic stability . Springer Science &Business Media, 2012.

Michael J Neely, Scott T Rager, and Thomas F La Porta. Max-Weight learning algorithms for scheduling in unknown environments. IEEE Transactions on Automatic Control , 57(5):11791191, 2012.

Pedro A Ortega and Daniel A Braun. A minimum relative entropy principle for learning and acting. Journal of Artificial Intelligence Research , 38:475-511, 2010.

Ian Osband, Daniel Russo, and Benjamin Van Roy. (More) Efficient reinforcement learning via posterior sampling. Advances in Neural Information Processing Systems , 26, 2013.

Ian Osband and Benjamin Van Roy. Why is posterior sampling better than optimism for reinforcement learning? In International Conference on Machine Learning , pages 2701-2710. PMLR, 2017.

Reda Ouhamma, Debabrota Basu, and Odalric Maillard. Bilinear exponential family of MDPs: Frequentist regret bound with tractable exploration and planning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 9336-9344, 2023.

Yi Ouyang, Mukul Gagrani, Ashutosh Nayyar, and Rahul Jain. Learning unknown Markov decision processes: A Thompson sampling approach. Advances in Neural Information Processing Systems , 30, 2017.

Daniel J Russo, Benjamin Van Roy, Abbas Kazerouni, Ian Osband, and Zheng Wen. A tutorial on Thompson sampling. Foundations and Trends® in Machine Learning , 11(1):1-96, 2018.

Devavrat Shah, Qiaomin Xie, and Zhi Xu. Stable reinforcement learning with unbounded state space. arXiv preprint arXiv:2006.04353 , 2020.

R. Srikant and Lei Ying. Communication networks: An optimization, control, and stochastic networks perspective . Cambridge University Press, 2013.

Thomas Stahlbuhk, Brooke Shrader, and Eytan Modiano. Learning algorithms for minimizing queue length regret. IEEE Transactions on Information Theory , 67(3):1759-1781, 2021.

Shaler Stidham and Richard Weber. A survey of Markov decision models for control of networks of queues. Queueing Systems , 13:291-314, 1993.

Malcolm Strens. A Bayesian framework for reinforcement learning. In ICML , volume 2000, pages 943-950, 2000.

Wojciech Szpankowski and Vernon Rego. Yet another application of a binomial recurrence order statistics. Computing , 43(4):401-410, 1990.

Dengwang Tang, Rahul Jain, Botao Hao, and Zheng Wen. Efficient online learning with offline datasets for infinite horizon MDPs: A Bayesian approach. arXiv preprint arXiv:2310.11531 , 2023.

Leandros Tassiulas and Anthony Ephremides. Jointly optimal routing and scheduling in packet ratio networks. IEEE Transactions on Information Theory , 38(1):165-168, 1992.

Leandros Tassiulas and Anthony Ephremides. Dynamic server allocation to parallel queues with randomly varying connectivity. IEEE Transactions on Information Theory , 39(2):466-478, 1993.

Georgios Theocharous, Zheng Wen, Yasin Abbasi-Yadkori, and Nikos Vlassis. Posterior sampling for large scale reinforcement learning. arXiv preprint arXiv:1711.07979 , 2017.

Georgios Theocharous, Zheng Wen, Yasin Abbasi Yadkori, and Nikos Vlassis. Scalar posterior sampling with applications. Advances in Neural Information Processing Systems , 31, 2018.

William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika , 25(3-4):285-294, 1933.

Neil Walton and Kuang Xu. Learning and information in stochastic networks and queues. In Tutorials in Operations Research: Emerging Optimization Methods and Modeling Techniques with Applications , pages 161-198. INFORMS, 2021.

Tsachy Weissman, Erik Ordentlich, Gadiel Seroussi, Sergio Verdu, and Marcelo J Weinberger. Inequalities for the L 1 deviation of the empirical distribution. Hewlett-Packard Labs, Tech. Rep , 2003.

Zixian Yang, R Srikant, and Lei Ying. Learning while scheduling in multi-server systems with unknown statistics: Maxweight with discounted UCB. In International Conference on Artificial Intelligence and Statistics , pages 4275-4312. PMLR, 2023.

Andrea Zanette, David Brandfonbrener, Emma Brunskill, Matteo Pirotta, and Alessandro Lazaric. Frequentist regret bounds for randomized least-squares value iteration. In International Conference on Artificial Intelligence and Statistics , pages 1954-1964. PMLR, 2020.

A Proofs related to problem formulation

A.1 Ergodicity definitions

Suppose that Markov process X on X with transition kernel P is irreducible, aperiodic and positive recurrent with stationary distribution µ and let f : X ↦→ [1 , ∞ ) be a measurable function such that µ ( f ) := E µ [ f ( Y )] < + ∞ with Y ∼ µ . We are interested in conditions under which for a sequence of positive numbers ρ := ( ρ ( n )) n ≥ 0 , where for a signed measure ˜ µ on X , ∥ ˜ µ ∥ f := sup | g |≤ f | ˜ µ ( g ) | . The sequence ρ is interpreted as the rate function, and three different notions of ergodicity are distinguished based on the following rate functions: ρ ( n ) ≡ 1 , ρ ( n ) = ζ n for ζ > 1 , and ρ ( n ) = n ζ -1 for ζ ≥ 1 . Further, for each rate function ρ , we state the Foster-Lyapunov characterization of ergodicity of the Markov process X , which provides sufficient conditions for (15) to hold.

If ρ ( n ) ≡ 1 for all n ≥ 0 , the Markov process X satisfying (15) is said to be f -ergodic . From [43], for an irreducible and aperiodic chain, f -ergodicity is equivalent to the existence of a function V : X ↦→ [0 , ∞ ) , a finite set C , and positive constant b such that

where ∆ V := PV -V with PV ( x ) := ∑ x ′ ∈X P ( x , x ′ ) V ( x ′ ) . The drift condition (16) implies positive recurrence of the Markov process, existence of a unique stationary distribution µ , and µ ( f ) ≤ b < + ∞ ([43], Theorem 14.3.7).

If ρ ( n ) = ζ n for some ζ > 1 , the Markov process X satisfying (15) is said to be f -geometrically ergodic . From [43], for an irreducible and aperiodic chain, f -geometric ergodicity is equivalent to the existence of a function V : X ↦→ [1 , ∞ ) , a finite set C , a constant γ ∈ (0 , 1) and positive constant b such that

The drift condition (17) implies positive recurrence of the Markov process, existence of a unique stationary distribution µ , and µ ( V ) ≤ b 1 -γ < + ∞ ([43], Theorem 14.3.7). Moreover, if f ( · ) ≡ 1 in (15), then the Markov process X is called geometrically ergodic .

If ρ ( n ) = n ζ -1 for some ζ ≥ 1 , the Markov process X satisfying (15) is said to be f -polynomially ergodic . From [43, 29], for an irreducible and aperiodic chain, the existence of a function V : X ↦→ [1 , ∞ ) , a finite set C , a constant α ∈ [0 , 1) , and positive constants c and b such that

implies V ζ -polynomial ergodicity of X at rate ρ ( n ) = n ζ -1 for all ζ ∈ [1 , 1 / (1 -α )] with V ζ = V 1 -ζ (1 -α ) . The drift condition (18) implies positive recurrence of the Markov process, existence of a unique stationary distribution µ , and µ ( V α ) ≤ b c < + ∞ .

A.2 Lemma 4

Lemma 4. For any state x = 0 d , there exists constants κ > 1 and c 1 such that the following holds for the hitting time of state 0 d , τ 0 d ,

Proof. We define ˜ V := ∑ ∞ n =0 0 d P n V g where 0 d P n is the n

-step taboo probability [43] defined as

for A,B ⊆ X , and τ A is the first hitting time of set A . We also let A P 0 x B = I B ( x ) . We have

̸

̸

̸

̸

In Appendix D.3, we argue that there exists ˜ b g > 1 such that ˜ V ( y ) ≤ ˜ b g V g ( y ) for all y ∈ X , which leads to

Define Lyapunov function

̸

̸

From the above equation and (19), we get

̸

Thus,

To find an upper bound for E x [ κ τ 0 d ] , we apply [43, Theorem 15.2.5], which is a generalization of Lemma 12. For any 1 ≤ κ ≤ 2 ˜ b g 2 ˜ b g -1 , there exists ϵ > 0 such that

As ˜ V g ( y ) ≥ 1 for all y ∈ X , we have

and the claim holds for any κ ∈ [1 , 2 ˜ b g 2 ˜ b g -1 ] and c 1 = ˜ b g ϵ -1 ( 1 + 2 ˜ b g ) .

A.3 Poisson equation

For an irreducible Markov process on the countably-infinite space X with time-homogeneous transition kernel P and cost function ¯ c ( · ) , a solution pair to the Poisson equation [41] is a scalar J and function v ( · ) : X ↦→ R such that J + v = ¯ c + Pv , where v ( z ) = 0 for some z ∈ X . If the Markov process is also positive recurrent and E x [ ∑ τ y -1 i =0 | ¯ c ( X ( i )) | ] < ∞ , where τ y is the first hitting time of some state y ∈ X , then solution pair ( J, v ) given as

is a solution to the Poisson equation J + v = ¯ c + Pv with v ( z ) = 0 [41, Theorem 9.5].

Lemma 5. Consider Markov Decision Processes ( X , A , c, P θ ) governed by parameter θ ∈ Θ following the best-in-class policy π ∗ θ . Then the pair ( J ( θ ) , v π ∗ θ ) given as

is a solution to the Poisson equation v + J = c + P π ∗ θ θ v , where v π ∗ θ (0 d ) = 0 and ¯ C π ∗ θ ( x ) = E π ∗ θ x [ ∑ τ 0 d -1 i =0 c ( X ( i ) , π ∗ θ ( X ( i ))) ] .

Proof. From [41, Theorem 9.5], a solution pair to the Poisson equation exists if E π ∗ θ x [ τ 0 d ] and ¯ C π ∗ θ ( x ) are finite for all x ∈ X . The former follows from positive recurrence assumed in Assumption 3 and for the latter, from Assumptions 1 and 2,

which is finite from geometric ergodicity (Assumption 3) and the discussion following that.

B Proofs of regret analysis

In this section, we state the proofs related to regret analysis of Section 4. We first note a key property of Thompson sampling from [49], which states that for any episode k , measurable function f , and H t k -measurable random variable Y , we have

where H t := σ ( X (1) , . . . , X ( t ) , A (1) , . . . , A ( t -1)) for all t ∈ N . We start with deriving upper bounds on the hitting times of state 0 d using the ergodicity conditions of Assumptions 3 and 4. Previous works [25, 27, 29] have already established bounds on hitting times in geometrically and polynomially ergodic chains in terms of their corresponding Lyapunov function. However, our objective is to provide a precise characterization of all constants included in these bounds in terms of the constants of the drift equations 3 and 4. This characterization allows us to derive uniform bounds across the model class. In Appendix C.1, using the polynomial Lyapunov function provided in Assumption 4, we establish upper bounds on the i -th moment of hitting time of state 0 d from any state x ∈ X and for 1 ≤ i ≤ r +1 . Importantly, the derived bound is polynomial in terms of any component of the state x i . Additionally, in Appendix C.2, we characterize the tail probabilities of the return time to state 0 d starting from 0 d in terms of the geometric Lyapunov function of Assumption 3. The derived tail bounds will be used in Lemma 6 to derive upper bounds for all moments of hitting times in the model class. These bounds, along with the skip-free behavior of the model, allow us to study the maximum state (with respect to ℓ ∞ -norm) achieved up to time T in MDP ( X , A , c, P θ ∗ ) following Algorithm 1 as follows.

Lemma 6. For p ∈ N , the p -th moment of max 1 ≤ i ≤ T τ ( i ) 0 d and M T θ ∗ , that is the maximum ℓ ∞ -norm of the state vector achieved up until and including time T is O (log p T ) .

In the proof of Lemma 6 given in Appendix B.1, we make use of geometric ergodicity of the chain and the fact that hitting times have geometric tails to find an upper bound for moments of M T θ ∗ . Using this, we aim to bound the number of episodes started before or at T , denoted by K T . We first find an upper bound for the number of episodes in which the second stopping criterion is met or there exists a state-action pair for which N t ( x , a ) has increased more than twice. In the following lemma, we bound the number of such episodes, which we denote by K M , in terms of random variable M T θ ∗ and other problem-dependent constants. Proof of Lemma 7 is given in Appendix B.2.

Lemma 7. The number of episodes triggered by the second stopping criterion and started before or at time T , denoted by K M , satisfies K M ≤ 2 |A| ( M T θ ∗ +1) d log 2 T a.s.

We next bound the total number of episodes K T by bounding the number of episodes triggered by the first stopping criterion, using the fact that in such episodes, ˜ T k = ˜ T k -1 +1 . Moreover, to address the settling time of each episode k , shown by E k = T k -˜ T k , we use the geometric ergodicity property and Lemma 6. Finally, the proof of Lemma 8 is given in Appendix B.3.

From Lemma 8, the upper bound given in Lemma 6 for moments of M T θ ∗ , and Cauchy-Schwarz inequality, it follows that the expected value of the number of episodes K T is of the order ˜ O ( √ h d |A| T ) . This term has a crucial role in determining the overall order of the total regret up to time T . In the rest of this section, we present a detailed proof of the lemmas and other results used to prove Theorem 1.

Remark 3. The skip-free to the right property in Assumption 2 yields a polynomially-sized subset of the underlying state-space that can be explored as a function of T . This polynomially-sized subset can be viewed as the effective finite-size of the system in the worst-case, and then, directly applying finite-state problem bounds [49] would result in a regret of order ˜ O ( T d +0 . 5 ) ; since d ≥ 1 , such a coarse bound is not helpful even for asserting asymptotic optimality! However, to achieve a regret of ˜ O ( √ T ) , it is essential to carefully understand and characterize the distribution of M T θ ∗ and then its moments, as demonstrated in Lemma 6.

Remark 4. The derived regret bound can be extended to a larger class of MDPs which consist of transient states in addition to the single irreducible class. Specifically, for any θ 1 , θ 2 ∈ Θ , the Markov process with transition kernel P π θ 2 θ 1 obtained from the MDP ( X , A , c, P θ 1 ) by following policy π ∗ θ 2 has a single irreducible class I θ 1 ,θ 2 and a set of transient states T θ 1 ,θ 2 . Furthermore, Assumptions 3 and 4 hold for the single irreducible class. The reasoning behind the proof remains true in this case using the following argument: each episode k starts at 0 d which is in the irreducible set for the chosen policy π ∗ θ k , hence, throughout the episode the algorithm remains in the irreducible set that is positive recurrent and never visits any transient states. In other words, episodes starting and ending at 0 d with a fixed episode dependent policy implies that reachable set of 0 d is all that can be explored, which is positive recurrent by our assumptions. As a result, we can restrict our proof derivations to the subset that is reachable from 0 d in each episode and follow the same analysis. The Lyapunov function based bounds apply to the positive recurrent states, and hence, restricting attention to states reachable from 0 d within each episode, we can use these bounds for our assessment of regret using norms of the state. Thereafter, the coarse bounds on the norms of the state can be applied as carried out in our proof.

∗

Remark 5. By problem-dependent parameters, we refer to the parameters that characterize the complexity or size of the model class Θ . These parameters are not just a function of the size of the statespace and diameter of the MDP (as mentioned in the literature on finite-size problems[5, 23, 49]), as stability needs to be accounted for in the countable state-space setting. The dependence is, thus, more complex and requires the inclusion of stability parameters, such as Lyapunov functions, petite sets, and ergodicity coefficients that are discussed in Assumptions 1-4.

Remark 6. In the subsequent sections, several equalities and inequalities in the proofs are between random variables and hold almost surely (a.s.). Throughout the remainder, we will omit the explicit mention of a.s., but any such statement should be interpreted in this context.

B.1 Proof of Lemma 6

Proof. Let { α i } i ≥ 0 be the sequence of hitting times of state 0 d starting from 0 d (set α 0 = 0 ). Define τ ( i ) 0 d as the length of the i -th recurrence time of state 0 d for i ∈ N , i.e., τ ( i ) 0 d = α i -α i -1 . For simplicity, we take τ 0 d = τ (1) 0 d . Each such recurrence time is generated using policy π ∗ θ i that is determined using the algorithm in operation in an MDP governed by parameter θ ∗ . Furthermore, { τ ( i ) 0 d } i ∈ N are independent with length at least 1 , but they need not be identically distributed. The time T can be in the middle of one of these recurrence times, hence the current recurrence interval count is N ( T ) = inf { n : ∑ n i =1 τ ( i ) 0 d ≥ T } . Note that the lower bound of 1 on every τ ( i ) 0 d says that N ( T ) ≤ T a.s. Further, from the skip-free to the right property, the most any component of state can increase in during recurrence time τ ( i ) 0 d is hτ ( i ) 0 d . Hence, the most any component of the state (and also the ∥ · ∥ ∞ norm of the state) can increase is given by h max i =1 ,...,T τ ( i ) 0 d where the random variables

are independent with geometrically decaying tails with a worst case rate of

see Lemma 11. From Lemma 10, we have

and we define ˜ γ g ∗ := 1 -( ˜ b g ∗ ) -1 . From the definition of b g θ 1 ,θ 2 in Assumption 3, b g θ 1 ,θ 2 is greater than or equal to 2 . Thus, ˜ b g θ 1 ,θ 2 ≥ 7 and we have

and as a result of Lemma 11,

We upper bound E [ M T θ ∗ ] using the independence of { τ ( i ) 0 d } i ∈ N and the above equation,

where n 0 is the smallest n ≥ 0 such that c g ∗ ( γ g ∗ ) n < 1 . By Reimann sum approximation, we get

where the last inequality follows from ∑ T n =1 n -1 ≤ log T +1 and thus E [ M T θ ∗ ] is O ( h log T ) . We now extend the result to moments of order greater than one. From (22), for 1 ≤ i ≤ T ,

For n ≥ n 0 , let t = n -n 0 ≥ 0 and Y i = max( τ ( i ) 0 d -n 0 , 0) to get

which means random variables { Y i } T i =1 are stochastically dominated by independent and identically distributed geometric random variables with parameter 1 -γ g ∗ . Furthermore, [56] argues that the p -th moment of the maximum of T independent and identically distributed geometric random variables is O (log p T ) . Thus, the p -th moment of max 1 ≤ i ≤ T Y i is O (log p T ) and

which gives

Since the right-hand side of the above equation is O ( h p log p T ) , the claim is proved.

B.2 Proof of Lemma 7

Proof. Let K M ( x , a ) be the number of episodes k such that 1 ≤ k ≤ K T and in which the number of visits to the state-action pair ( x , a ) is increased more than twice at episode k , or

As for every episode in the above set the number of visits to ( x, a ) doubles,

and we can upper bound K M as follows

This completes the proof.

B.3 Proof of Lemma 8

Proof. We define macro episodes with start times t n k , k = 1 , 2 , . . . , K M + 1 where t n 1 = t 1 , t n KM +1 = T +1 (which is equivalent to n K M +1 = K T +1 ), and for 1 < k < K M +1

which are episodes wherein the second stopping criterion is triggered. Any episode (except for the last episode) in a macro episode must be triggered by the first stopping criterion; equivalently, ˜ T j = ˜ T j -1 +1 for all j = n k , n k +1 , . . . , n k +1 -2 . For 1 ≤ k ≤ K M , let T k M = ∑ n k +1 -1 j = n k T j be the length of the k -th macro episode. We have

Consequently, n k +1 -n k ≤ √ 2 T k M for all 1 ≤ k ≤ K M . From this, we obtain

Using the above equation and the fact that ∑ K M k =1 T k M = T we get

Finally, from Lemma 7 we get

This completes the proof.

B.4 Proof of Lemma 1

Proof. Let E k = T k -˜ T k ≥ 0 be the settling time needed to return to state 0 d after a stopping criterion is realized in episode k . We have

We first simplify the first term in the above summation. From the monotone convergence theorem,

Note that the first stopping criterion of Algorithm 1 ensures that ˜ T k ≤ ˜ T k -1 +1 at all episodes k ≥ 1 . Hence

Since I { t k ≤ T } ( ˜ T k -1 +1) is measurable with respect to H t k , by (20) we get

Therefore,

Thus,

For the second term in (23), from Assumption 5

Substitutinh (24) and (25) in (23), we get

B.5 Proof of Lemma 2

Proof. We note that the state of the MDP is equal to 0 d at the beginning of all episodes and the relative value function v ( x ; θ ) is equal to 0 at x = 0 d for all θ . Thus,

From the lower bound derived for the relative value function in (14),

where the second inequality follows from (21) in the proof of Lemma 6. We also note that ∥ X ( T + 1) ∥ ∞ ≤ M T θ ∗ + h . Thus,

From the inequality ( a + b ) r ≤ 2 r ( a r + b r ) , we have

B.6 Proof of Lemma 3

Proof. Let Z ( t ) = ( X ( t ) , π ∗ θ k ( X ( t )) ) be the state-action pair at t k ≤ t < t k +1 . R 2 can be upper bounded as

and for any tuple ( x , a, y ) , we define N 1 ( x , a, y ) = 0 and for t > 1 , N t ( x , a, y ) = |{ t k ≤ i < ˜ t k +1 ≤ t for some k ≥ 1 : ( X ( i ) , A ( i ) , X ( i +1)) = ( x , a, y ) }| . Thus, from (26) and defining random variable v M = max 1 ≤ k ≤ K T x ∞ M T ∗ | v ( x ; θ k ) | ,

∥ ∥ ≤ θ

We define set B k as the set of parameters θ for which the transition kernel P θ ( ·| z ) is close to the empirical transition kernel P ˆ θ k ( ·| z ) at episode k for every state-action pair z = ( x , a ) ∈ X ×A , or

where β k ( z ) = √ 14 ∏ d i =1 ( x i + h ) max(1 ,N t k ( z )) log ( 2 |A| T ˜ δ ) for x = ( x 1 , . . . , x d ) and some 0 < ˜ δ < 1 , which will be determined later. We simplify the ℓ 1 -difference of the real and empirical transition kernels as follows

Similarly, we have

Substituting in (27), we get

We first find an upper bound for v M = max 1 ≤ k ≤ K T ∥ x ∥ ∞ ≤ M T θ ∗ | v ( x ; θ k ) | using the bounds derived in (13) and (14). From (13),

where the second line follows from the inequality ( a + b ) r ≤ 2 r ( a r + b r ) , the fifth line from Lemma 10, and the last line from Assumption 4 and (21). We further have

where using the definition of b η j θ 1 ,θ 2 in (38),

We also define

We next find a lower bound for v ( x ; θ k ) using (14) as follows:

Combining (29) and the above equation, we get a uniform upper bound for | v ( x ; θ k ) | over Θ , which we use to upper bound v M = max 1 ≤ k ≤ K T ∥ x ∥ ∞ ≤ M T θ ∗ | v ( x ; θ k ) | as below

where the constant terms are defined as

A deterministic upper bound on v M can also be found from the above equation. Noting that from Assumption 2, until time T only states with each component less than or equal to hT are visited, we have where Q ( T ) is a polynomial defined as above. Using the bounds derived for v M , we bound R 2 starting with the first term on the right-hand side of (28). We have

where the last inequality follows from (20) and the fact that set B k is H t k -measurable. To further simplify the first term in (28), we find an upper bound for P { θ ∗ / ∈ B k } using [64]. For a fixed z = ( x , a ) and n independent samples of the distribution P θ ∗ ( . | z ) , the L 1 -deviation of the true distribution P θ ∗ ( . | z ) and empirical distribution at the end of episode k , P ˆ θ k ( . | z ) , is bounded in [10] as

Therefore,

and

The probability that at episode k ≤ T , the true parameter θ ∗ does not belong to the confidence set B k can be bounded using the above and union bound as

In the summation in the above equation, we have simplified the expression by summing over x i ≤ hT instead of considering the more detailed summation over x i ≤ M T θ ∗ . However, this simplification does not affect the final evaluation of regret, as this term is not dominant and only contributes to a logarithmic term in the regret bound. Substituting in (31),

We now upper bound the second term in (28). From (30),

To bound the regret term resulting from the summation of β k ( Z ( t )) , we note that from the second stopping criterion, N t ( Z ( t )) ≤ 2 N t k ( Z ( t )) for all t k ≤ t < t k +1 and

The first summation can be simplified as

where the second inequality is due to the following arguments,

For the second term in (34), we get

where E k = T k -˜ T k , and K T is bounded from Lemma 8. Thus ∑ K T k =1 ∑ t k +1 -1 t = t k β k ( Z ( t )) is bounded as

Substituting the above bound in (33),

where c p 3 := 48 c p 2 . Finally, from the above equation, (32), and (28),

By choosing ˜ δ = 1 TQ ( T ) , we get R 2

B.7 Proof of Theorem 1

Proof. Lemmas 1, 2, and 3 along with Cauchy-Schwarz inequality showed that the regret terms R 0 and R 2 are of the order ˜ O ( KrdJ ∗ h d +2 r + r p ∗ √ |A| T ) and the term R 1 is ˜ O ( J ∗ ( h ) r p ∗ ) . Therefore, from R ( T, π TSDE ) = R 0 + R 1 + R 2 , the regret of Algorithm 1, R ( T, π TSDE ) , is ˜ O ( KrdJ ∗ h d +2 r + r p ∗ √ |A| T ) .

B.8 Requirement of an optimal policy oracle.

To implement our algorithm, we need to find the optimal policy for each model sampled by the algorithm-optimal policy for Theorem 1 and optimal policy within policy class Π for Corollary 1; this has also been used in past work [23, 24, 36]. In the finite state-space setting, [49] provides a schedule of ϵ values and selects ϵ -optimal policies to obtain ˜ O ( √ T ) regret guarantees. The issue with extending the analysis of [49] to the countable state-space setting is that we need to ensure (uniform) ergodicity for the chosen ϵ -optimal policies; the lim sup or lim inf of the time-average expected reward (used to define the average cost problem) being finite doesn't imply ergodicity. In other words, we must formulate (and verify) ergodicity assumptions for a potentially large set of close-to-optimal algorithms whose structure is undetermined. Another issue is that, to the best of our knowledge, there isn't a general structural characterization of all ϵ -optimal stationary policies for countable state-space MDPs or even a characterization of the policy within this set that is selected by any computational procedure in the literature; current results only discuss existence and characterization of the stationary optimal policy. In the absence of such results, stability assumptions with the same uniformity across models as in our submission will be needed, which are likely too strong to be useful.

If we could verify the stability requirements of Assumptions 3 and 4 for a subset of policies, the optimal oracle is not needed, and instead, by choosing approximately optimal policies within this subset, we can follow the same proof steps as [49] to guarantee regret performance similar to Corollary 1 (without knowledge of model parameters). To theoretically analyze the performance of the algorithm that follows an approximately optimal policy rather than the optimal one, we assume that for a specific sequence of { ϵ k } ∞ k =1 , an ϵ k -optimal policy is given, which is defined below.

Definition 1. Policy π ∈ Π is called an ϵ -optimal policy if for every θ ∈ Θ ,

where π ∗ θ is the optimal policy in the policy class Π corresponding to parameter θ and v ( . ; θ ) is the solution to Poisson equation (5) .

Given ϵ -optimal policies that satisfy Assumptions 3 and 4, in Theorem 2 we extend the regret guarantees of Corollary 1 to the algorithm employing ϵ -optimal policy, instead of the best-in-class policy, and show that the same regret upper bounds continue to apply.

Theorem 3. Consider a non-negative sequence { ϵ k } ∞ k =1 such that for every k ∈ N , ϵ k is bounded above by 1 k +1 and an ϵ k -optimal policy satisfying Assumptions 3 and 4 is given. The regret incurred by Algorithm 1 while using the ϵ k -optimal policy during any episode k is ˜ O ( dh d √ |A| T ) .

Proof. For the ϵ k -optimal policy used in episode k , shown by π ϵ k , we have

Thus,

We assumed that given ϵ -optimal policies satisfy Assumptions 3 and 4. As a result, we can utilize the proof of Theorem 1 to deduce that the term R 0 + R 1 + R 2 is of the order ˜ O ( dh d √ |A| T ) . Moreover, we can simplify the term E [ ∑ K T k =1 T k ϵ k ] as below:

From the second stopping condition of Algorithm 1, we have ˜ T k ≤ ˜ T k -1 +1 ≤ . . . ≤ k +1 and

where we have used the assumption that ϵ k ≤ 1 k +1 . For the second term of (35), from (25)

where in the last inequality we have used ∑ n i =1 1 n ≤ 1 + log( n ) . Finally, as a result of Lemma 6 and Lemma 8, the result follows.

C Bounds on hitting times under polynomial and geometric ergodicity

C.1 Polynomial upper bounds for the moments of hitting time of state 0 d

For any θ 1 , θ 2 ∈ Θ , consider the Markov process with transition kernel P π ∗ θ 2 θ 1 obtained from the MDP ( X , A , c, P θ 1 ) by following policy π ∗ θ 2 . [29, Lemma 3.5] establishes that if the process is polynomially ergodic, equivalently satisfies (4), then for every 0 < η ≤ 1 , there exists constants β η θ 1 ,θ 2 , b η θ 1 ,θ 2 > 0 such that the following holds:

where for η ∈ (0 , 1) , ˜ β p θ 1 ,θ 2 := min( β p θ 1 ,θ 2 , 1) and

and for η = 1 , β η θ 1 ,θ 2 = β p θ 1 ,θ 2 and b η θ 1 ,θ 2 = b p θ 1 ,θ 2 . Consequently, the following result is immediate from the proof of [29, Theorem 3.6]; for completeness, we provide the proof in Appendix D.1.

Lemma 9. Suppose a finite set C p θ 1 ,θ 2 , constants β p θ 1 ,θ 2 , b p θ 1 ,θ 2 > 0 , r/ ( r +1) ≤ α p θ 1 ,θ 2 < 1 , and a function V p θ 1 ,θ 2 : X → [1 , + ∞ ) exist such that (4) holds. Then, there exist a sequence of non-negative functions V i θ 1 ,θ 2 : X → [1 , + ∞ ) for i = 0 , . . . , r + 1 that satisfy the following system of drift equations for finite sets C i θ 1 ,θ 2 , constants b i θ 1 ,θ 2 ≥ 0 and β i θ 1 ,θ 2 > 0 :

Notice that r is the maximum degree of the cost function c defined in Assumption 1. Following the proof and approach of [29] and using the set of equations (39), we can find an upper-bound for E x [ τ i 0 d ] for i = 1 , . . . , r + 1 in Lemma 10. In order to establish upper bounds for the first r + 1 moments of τ 0 d , it is crucial to choose the value of α p θ 1 ,θ 2 greater than or equal to r r +1 , as demonstrated in the proof of Lemma 10 in Appendix D.2

Lemma 10. For i = 1 , . . . , r +1 , and for all x ∈ X

,

Based on Lemma 10, we impose the conditions of Assumption 4 to obtain uniform (over model class) and polynomial (in norm of the state) upper-bounds on the moments of hitting times to 0 d . Moreover, these conditions lead to a uniform characterization of parameters of Lemma 10 over all models in our class.

C.2 Distribution of return times to state 0 d

For any θ 1 , θ 2 ∈ Θ , consider the Markov process with transition kernel P π ∗ θ 2 θ 1 obtained from the MDP ( X , A , c, P θ 1 ) by following policy π ∗ θ 2 . In the following lemma, we show that the tail probabilities of the return times to the common state 0 d , again τ 0 d , converge geometrically fast to 0 , and characterize the convergence parameters in terms of the constants given in Assumption 3. Explicitly, we show

for problem and policy dependent constants c g θ 1 ,θ 2 and ˜ γ g θ 1 ,θ 2 . We will follow the method outlined in [27] with the goal to identify problem dependent parameters that will be relevant to our results. Proof of the following lemma is given in Appendix D.3 and follows the methodology of [27].

Lemma 11. For every θ 1 , θ 2 ∈ Θ in the Markov process obtained from the Markov decision process ( X , A , c, P θ 1 ) following policy π ∗ θ 2 , the return time to state 0 starting from state 0 satisfies the following:

where with

Based on Lemma 11, it is necessary to impose the conditions in Assumption 3 to obtain uniform tail probability bounds on τ 0 d for all model parameters and policy choices in Θ . Moreover, these conditions lead to a uniform characterization of c g θ 1 ,θ 2 and ˜ γ g θ 1 ,θ 2 over Θ . Furthermore, as a result of

Lemma 10 and uniformity conditions of Assumption 4, E π θ 2 u [ τ 0 d ] has a uniform bound over Θ and C g θ 1 ,θ 2 \ { 0 d } , which can be characterized in terms of the polynomial Lyapunov function.

∗

D Proofs of hitting time bounds

D.1 Proof of Lemma 9

Proof. In the proof, to avoid cumbersome notation we will drop the indices θ 1 , θ 2 . Following the proof of Theorem 3.6 in [29], we choose η i = 1 -( i -1)(1 -α p ) for i = 1 , . . . , r +1 and note that as α p ∈ [ r r +1 , 1) , we have η i ∈ [ 1 r +1 , 1] . As a result, we can apply (37) to each η i to get

Thus, the system of drift equations (39) hold for

where β η i and b η i are defined in (38).

D.2 Proof of Lemma 10

The proof of Lemma 10 uses the following lemma.

Lemma 12 (Proposition 11.3.2, [43]) . Suppose for nonnegative functions f , g , and V on the state space X and every k ∈ Z + , the following holds:

Then, for any initial condition x and stopping time τ

Proof of Lemma 10. Following [29], the proof uses an induction argument. We will use the notation of Lemma 9 for simplicity. Similarly, in this proof we will also denote ϕ p θ 1 ,θ 2 ( i ) as ϕ ( i ) , K θ 1 ,θ 2 ( · ) as K ( · ) , and V i , b i , β i , C i as V i , b i , β i , C i .

From irreducibility, for all x ∈ X , K ( x ) is positive and finite. Considering the system of drift equations found in Lemma 9, C i = C p is a finite set for all i = 1 , . . . , r +1 . Thus, min y ∈ C i K ( y ) is strictly positive. For all x ∈ X and i = 1 , . . . , r +1 , we have

We set α C p := (min y ∈ C i K ( y )) -1 = (min y ∈ C p K ( y )) -1 . From Lemma 9, for j = 1 and x ∈ X ∆ V 0 ( x ) ≤ -β 1 V 1 ( x ) + b 1 I C 1 ( x ) .

By applying Lemma 12, for all x ∈ X we get

Using (40) and (41), followed by noting that

we get

As V 1 ( x ) ≥ 1 , this gives us a bound on E x [ τ 0 d ] as follows:

Assume for i ≥ 1 , by the induction assumption we have

Set j = i +1 in (39), which yields

Define Z k = k i V i ( X k ) . From the above equation, we have

By applying Lemma 12 to the above equation, we get

where the second inequality follows from (40) and the induction hypothesis (42). Thereafter, from (42) (by using integral lower bound after using V i ≥ 1 ), we have

Substituting in (43), we get

This completes the proof.

D.3 Proof of Lemma 11

Proof. In the proof, to avoid cumbersome notation we will drop the indices θ 1 , θ 2 . Based on Assumption 3, there exists a finite set C g , constants b g , γ g ∈ (0 , 1) , and a function V g : X → [1 , + ∞ ) satisfying

For n ≥ 1 , define the n -step taboo probabilities [43] as

where A,B ⊆ X , and τ A is the first hitting time of set A . We also let A P 0 x B = I B ( x ) and ˜ V g = ∑ ∞ n =0 0 d P n V g . Applying the last exit decomposition on C g \ { 0 d } for all x ∈ X , we obtain

where we break up the trajectories starting at state x and reaching state y while avoiding state 0 d into two: ones that never visit the set C g , and the others that visit C g \ { 0 d } up until time m but not afterwards and exit C g \ { 0 d } at time m .

We first bound Term 1 in (46) by finding an upper bound for the probability term ∑ ∞ m =1 0 d P m xz using the first entrance decomposition on C g \ { 0 d } while noting that z ∈ C g \ { 0 d } :

where the third line follows from the fact that ∑ ∞ l =0 ∑ v / ∈ C g C g P l xv P vu is the probability of entrance to C g through u ∈ C g \ { 0 } , so it is less than 1 . Irreducibility and positive recurrence combined with | C g | < ∞ imply that max u ∈ C g \{ 0 d } E u [ τ 0 d ] < ∞ , which shows ∑ ∞ m =0 0 d P m xz is finite. Next,

by induction we prove that for n ≥ 1 and z ∈ C g \ { 0 d } we have

For n = 1 , we have using Assumption 3 that

Assuming that (48) holds for n , for n +1 we have

so (48) is shown. We collect these bounds later on for our result on Term 2 .

We now simplify the summation in (45). Similar to previous arguments, we will use induction for n ≥ 1 and show for all x ∈ X

For n = 1 , we have

Assuming that (49) holds for n , for n +1 we have

where the first and second inequalities follow from the definition of taboo probabilities and (44). Thus, (49) is proved. Lastly, for Term 2 in (46), we note

From the above equation, (47), (48), and (49), we bound ˜ V g ( x ) as follows:

where the last line is due to V g ( x ) ≥ 1 . Taking

we have shown that

̸

We now upper-bound P 0 d ( τ 0 d > n ) for all n ≥ 1 in an inductive manner, starting with P 0 d ( τ 0 d > 1) . As a part of showing this, for every x = 0 d we argue that for all n ≥ 1

First note that

Thus,

We now apply the bound in (50) to get

With the base of induction established, we assume the statement in (51) is true for n , and show that it continues to hold for n +1 as follows:

̸

̸

where the final inequality uses the same arguments as in (53) and (54).

Finally, using the tail probabilities of hitting time of state 0 d from any state x = 0 d , we bound the tail probability of the return time to state 0 d (starting from 0 d ) as follows

̸

̸

̸

̸

where the final inequality follows from the definition of b g , and we have

and the proof is complete.

E Queueing model examples

E.1 Model 1: Two-server queueing system with a common buffer

We consider a continuous-time queueing system with two heterogeneous servers with unknown service rate vector θ ∗ = ( θ ∗ 1 , θ ∗ 2 ) and a common infinite buffer, shown in Figure 2a. Arrivals to the system are according to a Poisson process with rate λ and service times are exponentially distributed

with parameter θ ∗ i , depending on the assigned server. The service rate vector θ ∗ is sampled from the prior distribution ν 0 defined on the space Θ given as

for fixed δ ∈ (0 , 0 . 5) and R ≥ 1 . Note that for any ( θ 1 , θ 2 ) ∈ Θ , we have θ 1 ≥ θ 2 and the stability requirement λ < θ 1 + θ 2 holds. The countable state space X is defined as X = { x = ( x 0 , x 1 , x 2 ) : x 0 ∈ N ∪ { 0 } , x 1 , x 2 ∈ { 0 , 1 }} , in which x 0 is the length of the queue, and x i , i = 1 , 2 is equal to 1 if server i is busy serving a job. At each time instance r ∈ R + , the dispatcher can assign jobs from the (non-empty) buffer to an available server. Thus, the action space A is equal to

where h indicates no action, b sends a job to both of the servers, and i = 1 , 2 assigns a job to server i . The goal of the dispatcher is to minimize the expected sojourn time of customers, which by Little's law [52] is equivalent to minimizing the average number of customers in the system, or

where X ( r ) is the state of the system at time r ∈ R + , immediately after the arrival/departure and just before the action is taken. In [38], it is argued that from uniformization [39] and sampling the continuous-time Markov process at a rate of λ + θ ∗ 1 + θ ∗ 2 , a discrete-time Markov chain is obtained, which converts the original continuous-time problem shown in (56) to an equivalent discrete-time problem as below:

To obtain a uniform sampling rate of λ + θ ∗ 1 + θ ∗ 2 , the continuous-time system is sampled at arrivals, real and dummy customer departures. In [38], it is further shown that the optimal policy that achieves the infimum in (57) is a threshold policy π t with the optimal finite threshold t ( θ ) ∈ N , with the policy defined as below:

note that action b is not used. Policy π t assigns a job to the faster (first) server whenever there is a job waiting in the queue and the first server is available. In contrast, π t dispatches a job to the second server only if the number of jobs in the system are greater than threshold t and the second server is available. If neither of these conditions hold, no action or h is taken. Consequently, we can restrict the set of all policies Π in (57) to the set Π t , which is the set of all possible threshold policies corresponding to some t ∈ N .

In the rest of this subsection, our aim is to show that Assumptions 1-5 are satisfied for the discrete-time Markov process obtained by uniformization of the described queueing system and hence, conclude that Algorithm 1 can be used to learn the unknown service rate vector θ ∗ with the expected regret of order ˜ O ( √ T ) .

Assumption 1. Cost function is given as c ( x , a ) = ∥ x ∥ 1 , which satisfies Assumption 1 with f c ( x ) = x 0 + x 1 + x 2 and K = r = 1 .

Assumption 2. For any state-action pair ( x , a ) and θ ∈ Θ , we have P θ ( A ( x ); x , a ) = 0 where A ( x ) = { y ∈ X : |∥ y ∥ 1 -∥ x ∥ 1 | > 1 } ; thus, Assumption 2 holds with h = 1 .

Assumption 3. Consider a queueing system with parameter θ following threshold policy π t for some t ∈ N . The uniformized discrete-time Markov chain is irreducible and aperiodic on a subset of state space given as X t = X \ ( { ( i, 0 , 0) : i ≥ min( t, 2) } ∪ { (0 , 1 , 1) } ) . In [38], it is proved that for every t , the chain consists of a single positive recurrent class and the corresponding average number of customers, depicted by J t ( θ ) , is calculated. Moreover, it is shown that for every θ ∈ Θ the optimal threshold t ( θ ) can be numerically found as the smallest i ∈ N for which J i ( θ ) < J i +1 ( θ ) . Define the set T ∗ as the set of all optimal thresholds corresponding to at least one θ ∈ Θ , or

In the following proposition, we verify the geometric ergodicity of the discrete-time chain governed by any parameter θ ∈ Θ and obtained by following any threshold policy π t for t ∈ T ∗ ; proof is given in Appendix F.1.

Proposition 1. The discrete-time Markov process obtained from the queueing system governed by parameter θ = ( θ 1 , θ 2 ) ∈ Θ and following threshold policy π t for some t ∈ T ∗ is geometrically ergodic. Equivalently, the following holds

for

Having described all the terms explicitly, we verify the rest of the conditions of Assumption 3, which lead to uniform (over model class) upper-bounds on the moments of hitting time to 0 d as follows:

From (60), sup θ ∈ Θ ,t ∈ T ∗ γ g θ,t ≤ 1 / 2 < 1 .

From (58), we can see that state (0 , 0) belongs to C g θ,t for all θ ∈ Θ and t ∈ T ∗ . In order for C g ∗ = ∪ θ ∈ Θ ,t ∈ T ∗ C g θ,t to be a finite set, the supremum of the optimal threshold t ( θ ) over Θ should be finite. In [37] with service rate vector ( θ 1 , θ 2 ) , it is shown that the optimal threshold is bounded above by √ 2 θ 1 /θ 2 , which further gives

Thus, sup θ ∈ Θ t ( θ ) ≤ √ 2 R , which is finite. To confirm a uniform upper bound for b g θ,t , we note that from (59),

which is finite as | C g ∗ | < ∞ .

Assumption 4. To find an upper bound on the second moment of hitting times, we verify Assumption 4 and show that there exists a finite set C p θ,t , constants β p θ,t , b p θ,t > 0 , r/ ( r +1) ≤ α p θ,t < 1 , and a function V p θ,t : X t → [1 , + ∞ ) satisfying

Proposition 2. The discrete-time Markov process obtained from the queueing system governed by parameter θ = ( θ 1 , θ 2 ) ∈ Θ and following threshold policy π t for some t ∈ T ∗ is polynomially

ergodic. This is true because (62) holds for

Proof of Proposition 2 is given in Appendix F.2. We define the normalized rates as ˜ λ = λ λ + θ 1 + θ 2 and ˜ θ i = θ i λ + θ 1 + θ 2 , for i = 1 , 2 . From the choice of parameter space Θ , we have ˜ λ ≤ 0 . 5 -0 . 5 δ , ˜ θ 1 + ˜ θ 2 ≥ 0 . 5 + 0 . 5 δ , and ˜ θ 1 ≥ 0 . 25 + 0 . 25 δ . We verify the remaining conditions of Assumption 4 as follows:

From (63), the first condition holds with r p ∗ = 2 and s p ∗ = 2 .

From (64), we can see that state (0 , 0) belongs to C p θ,t for all θ ∈ Θ and t ∈ T ∗ . Furthermore,

which follows from the stability condition ˜ λ ≤ 0 . 5 -0 . 5 δ . Thus, from the definition of C p θ,t in (64), and the fact that sup θ ∈ Θ t ( θ ) ≤ √ 2 R as argued in in (61), C p ∗ = ∪ θ ∈ Θ ,t ∈ T ∗ C p θ,t is a finite set. We also note that sup θ ∈ Θ ,t ∈ T ∗ b p θ,t is finite as | C p ∗ | < ∞ . It remains to show that inf θ ∈ Θ ,t ∈ T ∗ β p θ,t is positive, which is equivalent to verifying that sup θ ∈ Θ ,t ∈ T ∗ ˜ λ < 1 / 2 , which follows from the stability condition ˜ λ ≤ 0 . 5 -0 . 5 δ .

We need to show that K θ,t ( x ) := ∑ ∞ n =0 2 -n -2 ( P t θ ) n ( x , 0 d ) is strictly bounded away from zero. We notice that from any non-zero state x , the queueing system hits 0 d in ∥ x ∥ 1 transitions only if all transitions are real departures. Hence,

where the third and fourth inequalities follow from the definition of Θ in (55). Thus, the infimum of K θ,t ( x ) over the finite set C p ∗ and sets Θ and T ∗ is strictly greater than zero.

Assumption 5. We finally verify Assumption 5, which asserts that sup θ ∈ Θ J ( θ ) is finite. We have

where µ θ,t ( θ ) is the stationary distribution of the discrete-time process governed by parameter θ and following the optimal policy according to θ . From (62) and [43, Theorem 14.3.7],

which is finite from the the previously verified assumption. Consequently,

E.2 Model 2: Two heterogeneous parallel queues

We consider two parallel queues with infinite buffers, each with its own single server, and unknown service rate vector θ ∗ = ( θ ∗ 1 , θ ∗ 2 ) , shown in Figure 2b. The service rate vector θ ∗ is sampled from the prior distribution ν 0 defined on the space Θ given as

for fixed δ ∈ (0 , 0 . 5) and R ≥ 1 , which ensures the stability of the queueing system. Consider the discrete-time MDP ( X , A , P θ ∗ , c ) obtained by sampling the queueing system at the Poisson arrival sequence. The countably infinite state space X is defined as below

where the state of the system is the number of jobs in the server-queue pair i just before an arrival. Furthermore, the action space A is equal to

where action i ∈ A indicates the arrival dispatched to queue i . The unbounded cost function c : X ×A → N ∪{ 0 } is defined as the total number of jobs in the queueing system, i.e., c ( x , a ) = ∥ x ∥ 1 . For every ω ∈ R + , we define policy π ω : X → A , which routes the arrival according to the weighted queue lengths, as

where the tie is broken in favor of the first server. We also define policy class ˜ Π as the set of policies π ω such that ω belongs to a compact interval; in other words,

where R is defined in (68) and c R ≥ 1 . We aim to minimize the infinite-horizon average cost in the policy class ˜ Π , that is,

where X ( t ) = ( X 1 ( t ) , X 2 ( t )) is the occupancy vector of the queueing system just before arrival t . Even with the controlled Markov process transition kernel fully-specified (by the values of the arrival rate and the two service rates), the optimal policy 1 that satisfies (69) in policy class ˜ Π is not known except when θ 1 = θ 2 where the optimal value is ω = 1 , and so, to learn it, we will use Proximal Policy Optimization for countable state-space controlled Markov processes as developed in [18]. Note that [18] requires full knowledge of the controlled Markov process, which holds in our learning scheme since we use the parameters sampled from the posterior for determining the policy at the beginning of each episode. Furthermore, for each policy in the set of applicable policies ˜ Π , [18] also requires that the resulting Markov process be geometrically ergodic, which we will establish below.

Proposition 3. The discrete-time Markov process obtained from the queueing system governed by parameter θ = ( θ 1 , θ 2 ) ∈ Θ and following policy π ω ∈ ˜ Π is geometrically ergodic. Equivalently, the following holds

for

and problem-dependent constants x g j i,θ,ω and ζ i,θ,ω for i, j = 1 , 2 .

Proof of Proposition 3 is given in Appendix F.3. In the rest of this subsection, our aim is to show that Assumptions 1-5 are satisfied for the discrete-time MDP and conclude that Algorithm 1 can be used to learn the unknown service rate vector θ ∗ with expected regret of order ˜ O ( √ T ) .

Assumption 1. Cost function is given as c ( x , a ) = ∥ x ∥ 1 , which satisfies Assumption 1 with f c ( x ) = x 0 + x 1 + x 2 and K = r = 1 .

Assumption 2. For any state-action pair ( x , a ) and θ ∈ Θ , we have P θ ( A ( x ); x , a ) = 0 where A ( x ) = { y ∈ X : ∥ y ∥ 1 -∥ x ∥ 1 > 1 } ; thus, the MDP is skip-free to the right with h = 1 . Moreover, from any ( x , a ) , the finite set { y ∈ X : ∥ y ∥ 1 ≤ ∥ x ∥ 1 + 1 } is only accessible in one step; thus, Assumption 2 holds.

Assumption 3. In Proposition 3, we verified the geometric ergodicity of the discrete-time chain governed by parameter θ = ( θ 1 , θ 2 ) ∈ Θ and following policy π ω ∈ ˜ Π and thus, it only remains to verify the uniform model conditions. We define the normalized rates as ˜ λ = λ λ + θ 1 + θ 2 and ˜ θ i = θ i λ + θ 1 + θ 2 , for i = 1 , 2 . From the choice of parameter space Θ , we have ˜ λ ≤ 0 . 5 -0 . 5 δ , ˜ θ 1 + ˜ θ 2 ≥ 0 . 5 + 0 . 5 δ , and ˜ θ 1 ≥ 0 . 25 + 0 . 25 δ .

1. We first argue that ζ 1 ,θ,ω is bounded away from 1 as follows

where the first line follows from the definition of ζ 1 ,θ,ω in Appendix F.3, the second line from (71) and the definition of policy class ˜ Π . As a g θ,ω does not depend on θ , sup θ ∈ Θ ,ω ∈ [ 1 c R R ,c R R ] ζ 1 ,θ,ω < 1 . Furthermore, by similar arguments it can be shown that ζ 2 ,θ,ω is bounded away from 1 . We next argue that ζ 1 ,θ,ω ω 1+ ω exp ( a g θ,ω ω ) + ζ 2 ,θ,ω 1+ ω is bounded away from 1 using an upper bound found in

where ζ 3 = (1 + δ ) -1 , ζ 4 = 1 -0 . 5 δ 1 -δ , and we have used the arguments of Appendix F.3 and the definition of Θ . Using a similar argument, we can show that ζ 1 ,θ,ω ω 1+ ω + ζ 2 ,θ,ω 1+ ω exp ( a g θ,ω ) is bounded away from one, and finally, we conclude that sup θ ∈ Θ ,ω ∈ [ 1 c R R ,c R R ] γ g θ,ω < 1 .

From (72), we can see that state (0 , 0) belongs to C g θ,ω for all θ ∈ Θ and ω ∈ [ 1 c R R , c R R ] . In order for C g ∗ to be a finite set, the supremum of x g j i,θ,ω over Θ and ˜ Π should be finite. From the definition of x g 1 1 ,θ,ω in Appendix F.3,

and we can derive a lower bound for the denominator from (75). Similarly, we can show that sup θ ∈ Θ ,ω ∈ [ 1 c R R ,c R R ] x g 2 2 ,θ,ω is finite. We next find a uniform upper bound for x g 1 2 ,θ,ω from Appendix F.3, x g 1 2 ,θ,ω

which is uniformly bounded as γ g θ,ω is unformly bounded away from 1 and the second line follows from (74) and the fact that γ g θ,ω -ζ 2 ,θ,ω ≥ 1 -γ g θ,ω . Arguments verifying the finiteness of the supremum of x g 2 1 ,θ,ω follow similarly, and we conclude that | C g ∗ | < ∞ . To confirm a uniform upper bound for b g θ,ω , we note that from (73),

which is finite as a g θ,c R R is independent of the choice of θ and | C g ∗ | < ∞ .

Assumption 4. We next verify Assumption 4 and show that there exists a finite set C p θ,ω , constants β p θ,ω , b p θ,ω > 0 , r/ ( r +1) ≤ α p θ,ω < 1 , and a function V p θ,ω : X → [1 , + ∞ ) satisfying

Proposition 4. The discrete-time Markov process obtained from the queueing system governed by parameter θ = ( θ 1 , θ 2 ) ∈ Θ and following policy π ω ∈ ˜ Π is polynomially ergodic. This follow because (76) holds for

Proof of Proposition 4 is given in Appendix F.4. Next, we verify the remaining conditions of Assumption 4.

From (77) and the fact that ω ∈ [ 1 c R R , c R R ] , the first condition holds with r p ∗ = 2 and s p ∗ = sup θ ∈ Θ ,ω ∈ [ 1 c R R ,c R R ] s θ,ω = c R R +1 .

From (78), state (0 , 0) belongs to C p θ,ω for all θ ∈ Θ and ω ∈ [ 1 c R R , c R R ] . Furthermore, for i = 1 , 2 ,

which follows from the fact that θ 1 ≤ Rθ 2 and ˜ θ 1 ≥ 0 . 25 + 0 . 25 δ . Thus, from the definition of C p θ,ω in (78), C p ∗ = ∪ θ ∈ Θ ,ω ∈ [ 1 c R R ,c R R ] C p θ,ω is a finite set. We next verify that the infimum of β p θ,ω , found in (79), is positive. In (82), we showed that infimum of λ + θ i θ i over Θ is lower bounded by 1+ δ 4 . From this, the fact that ω belongs to a compact set, and θ 1 + θ 2 + λ ≥ δ , it follows that inf θ ∈ Θ ,ω ∈ [ 1 c R R ,c R R ] β p θ,ω > 0 . Furthermore, it is easy to see that β p θ,ω ≤ √ c R R . Hence, from (80),

which is finite as | C p ∗ | < ∞ .

We need to show that K θ,ω ( x ) := ∑ ∞ n =0 2 -n -2 ( P π ω θ ) n ( x , 0 d ) is strictly bounded away from zero. We show this using the fact that from any state x , the queueing system hits (0 , 0) in one step with positive probability. Take x i,θ,ω = max x ∈ C θ,ω x i for i = 1 , 2 . We have

The infimum in the right-hand side of the above equation is attained for the minimum normalized service rates possible for each server, or ˜ θ 1 = 1+ δ 4 and ˜ θ 2 = 1+ δ 4 R . Therefore, the infimum of K θ,ω ( x ) over the finite set C p ∗ , Θ , and interval [ 1 c R R , c R R ] is strictly greater than zero.

Assumption 5. We finally verify that sup θ ∈ Θ J ( θ ) is finite. We first note that for x = ( x 1 , x 2 ) ,

From the above equation,

where µ θ,ω ∗ ( θ ) is the stationary distribution of the discrete-time process governed by parameter θ and following the best in-class policy according to θ , shown by π ω ∗ ( θ ) . From [43], Theorem 14.3.7,

which is finite from the the previous verified assumption. Thus,

F Proofs related to the queueing model examples

F.1 Proof of Proposition 1

Proof. We define the normalized rates as

for i = 1 , 2 . From the choice of parameter space Θ , we have ˜ λ ≤ 0 . 5 -0 . 5 δ , θ 1 + θ 2 ≥ 0 . 5 + 0 . 5 δ , and θ 1 ≥ 0 . 25 + 0 . 25 δ . To prove geometric ergodicity, from the discussions of Section 2, it suffices to show that there exists a finite set C g θ,t , constants b g θ,t > 0 , γ g θ,t ∈ (0 , 1) , and a function V g θ,t : X t → [1 , + ∞ ) satisfying

Take V g θ,t ( x ) = exp( a g θ,t ∥ x ∥ 1 ) for some a g θ,t > 0 . For i ≥ 1 and x = ( i, 1 , 1) ,

where P t θ is the corresponding transition kernel. Thus,

Take ˜ a θ,t = exp( a g θ,t ) . We need to find ˜ a θ,t > 1 and 0 < γ g θ,t < 1 such that

Take ˜ a θ,t = (1 -δ ) -1 > 1 and

We need to have ˜ γ θ,t < 1 which follows from the stability condition ˜ λ ≤ 0 . 5 -0 . 5 δ as below:

We now verify (85):

where the last line follows from ˜ λ ≤ 0 . 5 -0 . 5 δ < (1 -δ ) / (2 -δ ) .

For x = ( i, 0 , 1) and i ≥ 1 , we have

and

which results in the same conditions as previously discussed. When x = ( i, 1 , 0) and i ≥ t also same argument holds.

Finally, (84) holds for

where the last line holds because PV g θ,t ( x ) ≤ V g θ,t ( y ) for y such that ∥ y ∥ 1 = ∥ x ∥ 1 +1 .

F.2 Proof of Proposition 2

Proof. In order to show polynomially ergodicity, we will verify (62). We define V p θ,t ( x ) = ∥ x ∥ 2 1 and α p θ,t = 1 / 2 , which is equal to r/ ( r +1) for r = 1 ; r is defined in Assumption 1. For x = ( i, 0 , 1) and i ≥ 1 ,

in which ˜ λ , ˜ θ 1 , and ˜ θ 2 are the normalized rates defined in (83). Thus,

For β p θ,t = 1 -2 ˜ λ , the right-hand side of above equation is non-positive for i ≥ 2 ˜ λ 1 -2 ˜ λ . For x = ( i, 1 , 0) and i ≥ t ,

Thus,

which is also non-positive under the same conditions as the previous case. For i ≥ 1 and x = ( i, 1 , 1) ,

Thus,

which is non-positive under the same conditions as the first case. Finally, (62) holds for

where the last line holds because PV p θ,t ( x ) ≤ V p θ,t ( y ) for y such that ∥ y ∥ 1 = ∥ x ∥ 1 +1 .

F.3 Proof of Proposition 3

Proof. To show geometric ergodicity of the chain that follows π ω , we verify (70). Take a g θ,ω > 0 and

First, we find PV g θ,ω ( x ) for the function defined above. We have

where X (2) = ( X 1 (2) , X 2 (2)) is the state of the system at the second arrival, starting from state x . To find the above expectations, we first find the corresponding transition probabilities. If the number of departures from server i during a fixed interval with length t is less than the total number of jobs in the queue of that server, the number of departures follows a Poisson distribution with parameter θ i t . Let P (( x 1 , x 2 ) → ( x ′ 1 , X )) be the probability of transitioning from a system with x i jobs in server-queue pair i (just after the assignment of the arrival) to a queueing system with x ′ 1 jobs in the first server-queue pair (just before the upcoming arrival). For 1 ≤ x ′ 1 ≤ x 1 , we have

and

Assume 1 + x 1 ≤ ω (1 + x 2 ) , which results in the new arrival being assigned to the first server. For the first term in (87), we have

Similarly, for the second term in (87), we have

To satisfy (70), for some 0 < γ g θ,ω < 1 and all but finitely many x , the following should hold, PV g θ,ω ( x ) ≤ γ g θ,ω V g θ,ω ( x ) , or from (86) and (87),

Notice that

From (90) and (91), it suffices to have

Define

Simplifying (92), we need the following to hold

As ζ i,θ,ω < 1 , there exists γ g θ,ω such that

From the assumption 1 + x 1 ≤ ω (1 + x 2 ) and the above equation, (93) can be further simplified as

For the above to hold outside a finite set, we need to have

Note that ζ 3 < 1 and ζ 4 > 1 . Defining function f ( y ) := 1+ ζ 4 y -exp( y ) , we note that for y ≤ log ζ 4 , f ( y ) > 0 , where log ζ 4 is the maximizer of f ( y ) . Similarly, taking g ( y ) := 1 -ζ 3 y -exp( -y ) , for y ≤ -log ζ 3 , g ( y ) > 0 , where -log ζ 3 is the maximizer of g ( y ) . Thus, we conclude that for a g θ,ω ≤ min( -ω log ζ 3 , -log ζ 3 , ω log ζ 4 ) ,

To guarantee the existence of 0 < γ g θ,ω < 1 that satisfies (95), we need to ensure the left-hand side of (95) is strictly less than 1. Using the bounds found in (97) and (98) and the definition of ζ 1 ,θ,ω and ζ 2 ,θ,ω , we simplify (95) to get

Define which is equivalent to

To make sure there exists a g θ,ω > 0 that satisfies (99), the right-hand side of (99) needs to be positive, which follows as below:

where ˜ λ , ˜ θ 1 , and ˜ θ 2 are the normalized rates defined in (83) and we have used the stability condition ˜ λ ≤ 0 . 5 -0 . 5 δ . We further simplify the left-hand side of (99) as

From the above equation and (100), a g θ,ω needs to satisfy

Finally, we take a g θ,ω as

After finding an appropriate a g θ,ω , we can choose 0 < γ g θ,ω < 1 such that (95) holds or

Moreover, from (94) a lower bound x g 1 1 ,θ,ω for x 1 is derived; In other words,(94) holds for x 1 > x g 1 1 ,θ,ω . From (93), we can find the corresponding x g 1 2 ,θ,ω and take x g 1 θ,ω = ( x g 1 1 ,θ,ω , x g 1 2 ,θ,ω ) . By repeating the same arguments when 1 + x 1 < ω (1 + x 2 ) , we finally conclude that

for

F.4 Proof of Proposition 4

Proof. Define V p θ,ω ( x ) = x 2 1 ω + x 2 2 , and α p θ,ω = 1 / 2 . Assume that x 1 = 0 and x 2 > (1 -ω ) /ω ; which means the new job will be assigned to the first server. The transition probabilities of the discrete-time chain sampled at Poisson arrivals is given in (88) and (89), and we calculate PV p θ,ω ( x ) as

We define d i := θ i / ( θ i + λ ) for i = 1 , 2 and

From (101),

Outside a finite set, we need the above equation to be non-positive; which is equivalent to

Thus,

By taking β p θ,ω ≤ d 2 / 2 , it suffices for the following to be non-positive,

which holds for x 2 ≥ 2 c R R/d 2 . Thus, for x 1 = 0 and x 2 ≥ max(2 c R R ( λ + θ 2 ) /θ 2 , (1 -ω ) /ω ) = 2 c R R ( λ + θ 2 ) /θ 2 , (76) holds. The case of x 2 = 0 and non-zero x 1 follows same arguments and (76) holds for β p θ,ω ≤ d 1 / 2 √ ω , x 2 = 0 , and x 1 ≥ max(2 c R R ( λ + θ 1 ) /θ 1 , ω -1) = 2 c R R ( λ + θ 1 ) /θ 1 . We now consider the case of x 1 , x 2 > 0 and x 1 +1 ≤ ω ( x 2 +1) , and note that

Hence, it suffices to find finite set C p θ,ω , constants b p θ,ω and β p θ,ω > 0 , such that the following holds for V p θ,ω ( x ) = x 2 1 ω + x 2 2 ,

As x 1 +1 ≤ ω ( x 2 + 1) , the new arrival is assigned to the first queue and we find ∆ V p θ,ω ( x ) + √ ω +1 β p θ,ω ( x 2 +1) using the same calculations as (102).

We next consider two different cases based on the value of d 1 and analyze them separately.

One. 0 . 8 ≤ d 1 < 1 : We first notice that the coefficient of x 1 in (104) is negative, as d 1 > 1 / 2 . For x 1 ≥ 1 , (104) is equal to

where the third line follows from (103), and the last line from the fact that when 0 . 8 ≤ d 1 < 1 , both terms -d 3 1 -2 d 2 1 -2 d 1 +2 and -d 2 1 -3 d 1 +3 are negative. Next, we notice that (105) is equal to

where the second line follows from (103). Taking β p θ,ω ≤ d 2 / 2 √ ω +1 , we get

which is non-positive for x 2 ≥ 1 . Finally, when 0 . 8 ≤ d 1 < 1 , x 1 , x 2 > 0 , and x 1 +1 ≤ ω ( x 2 +1) , (76) holds for β p θ,ω ≤ d 2 / 2 √ ω +1 .

Two. d 1 < 0 . 8 : Taking β p θ,ω ≤ d 2 √ ω +1(1 -d 2 ) , we note that the coefficient of x 2 in (105) is negative. Thus, from x 1 +1 ≤ ω ( x 2 +1) , (104) and (105),

As d i = ˜ θ i / ( ˜ θ i + ˜ λ ) in terms of the normalized rates, we get

which is negative from the stability condition. For β p θ,ω ≤ ˜ θ 1 +˜ θ 2 -˜ λ ˜ λ √ ω +1 , from (106) we get

which is non-positive for

As d 1 < 0 . 8 , we can see that ˜ λ > ˜ θ 1 / 4 ; thus,

where we have used the fact that ˜ θ 1 ≥ ˜ θ 2 , ω ≤ c R R , ˜ θ 1 + ˜ θ 2 -˜ λ ≥ δ , and ˜ λ ≤ 0 . 5 -0 . 5 δ and it suffices for x 1 to be greater than or equal to 4 c R R . For x 1 < 4 c R R , (104) can be upper bounded as

where in the last inequality we have used d 1 < 0 . 8 . From (105) and taking β p θ,ω ≤ d 2 / 2 √ ω +1 ,

which is negative for

Finally, when x 1 +1 ≤ ω ( x 2 + 1) and x 1 , x 2 > 0 , (76) holds for β p θ,ω ≤ 1 √ ω +1 min ( ˜ θ 2 2( ˜ θ 2 +˜ λ ) , ˜ θ 1 + ˜ θ 2 -˜ λ ) , x 1 ≥ 4 c R R , and x 2 ≥ 1 + 16 c R R +100 ωd 2 . Repeating the same arguments when x 1 , x 2 > 0 and x 1 +1 > ω ( x 2 + 1) , (76) holds for β p θ,ω ≤ 1 √ ω +1 min ( ˜ θ 1 2( ˜ θ 1 +˜ λ ) , ˜ θ 1 + ˜ θ 2 -˜ λ ) , x 1 ≥ 1 + ω (16 c R R 2 +100) d 1 , and x 2 ≥ 4 c R R 2 . Finally, (76) holds with

where the fourth line holds since PV p θ,ω ( x ) ≤ V p θ,ω ( y ) for y = ( y 1 , y 2 ) such that y i = x i +1 for i = 1 , 2 .

G Numerical results

G.1 Comparison of Algorithm 1 with other learning algorithms

We first note that due to the countably infinite state-space setting of our problem, we are unable to directly compare our algorithm to other learning algorithms proposed in the literature. One potential candidate algorithm uses the reward biased maximum likelihood estimation (RBMLE) [33, 34, 11, 42], which estimates the unknown model parameter with the likelihood perturbed a vanishing bias towards parameters with a larger long-term average reward (i.e., optimal value). This scheme also uses the principle of 'optimism in the face of uncertainty' in how it perturbs the maximum likelihood estimate. The naive version of the RMBLE algorithm does not apply to our examples due the following key assumption: over all parameters (and the control policies used for them), the transition probabilities are assumed to be mutually absolutely continuous; this is critical for the proofs and also allows the use of log-likelihood functions for computations. Similarly, naive use of the algorithms in [36] and [24] is not possible, again due to a similar absolutely continuity assumption which is critical for the proofs. Our posterior computations avoid such issues as the true parameter always has non-zero mass during the execution of the algorithm: episode k always starts in state 0 d which is positive recurrent for the Markov chain with true parameter θ ∗ and policy used π ∗ θ k . The RBMLE algorithm has yet another issue in that it requires knowledge of the optimal value function, and hence, for our examples, it may only apply to Model 1 for which the value function is known analytically. Finally, whereas we do get to observe inter-arrival times for both model, we never directly observe completed service times owing to the sampling employed, and this precludes the direct use of Upper-Confidence-Bound based parameter estimation followed by certainty equivalent control algorithms. Owing to these issues, at this point in time, we're unable to perform empirical comparisons of Algorithm 1 to other candidate algorithms with theoretical performance guarantees in a countable state setting.

As discussed in the previous paragraph, learning algorithms with theoretical performance guarantees are established in the finite state setting. One such algorithm is the certainty equivalence control with forcing, which is proposed and discussed in detail in [4]. To assess the finite-time performance of our algorithm, in Figure 4, we compare the performance of our proposed learning algorithm, denoted as TSDE, with the algorithm introduced in [4], referred to as AgrawalTeneketzis. Reference [4] proposes a certainty equivalence control law with forced exploration, which operates in episodes with increasing lengths and a priori fixed sequences of forcing times. Specifically, at the beginning of each episode, all possible stationary control laws are explored for one recurrence interval of state (0 , 0) . Subsequently, based on this exploration, an empirical estimate of the average collected reward

(a) Model 1: Queueing system of Figure 2a.

(b) Model 2: Queueing system of Figure 2b.

102

egret

101

TSDE

AgrawalTeneketzis

100

一等-

0.12T0.5

0.023T0.95

102

103

104

T

103

Regret

102

101

TSDE

AgrawalTeneketzis

一等-

0.35T0.5

100

0.15T0.8

102

103

104

105

T

(a) Model 1: Queueing system of Figure 2a.

(b) Model 2: Queueing system of Figure 2b.

1.0

ance

一

入=0.3

二

istar

0.7

入=0.5

入=0.7

Total variation

0.4

0.1

0

1

2

3

4

5

T

1e4

distance

0.5

入=0.3

入=0.5

入=0.7

Total variation c

0.1

0.0

0.5

1.0

1.5

2.0

2.5

T

1e5

is formed, and the control law resulting in the maximum average reward is implemented for the remainder of the episode. The length of the episodes are determined according to sequence { a i } ∞ i =0 defined as following:

where p is the number of possible stationary control laws and b i = ⌊ exp ( i 1 1+ δ )⌋ for any δ > 0 . Specifically, episode i terminates after completing additional a i -a i -1 recurrence intervals to state (0 , 0) . Both algorithms are implemented in the two queueing systems of Figure 2, where the arrival rate is λ = 0 . 5 and service rates are distributed according to a Dirichlet prior over [0 . 5 , 1 . 9] 2 . In Figures 4a and 4b, we set δ = 3 . 5 and δ = 3 , respectively. Moreover, in Figure 4b, the goal is to find the optimal weight w in the set { 1 . 5 , 2 , 2 . 5 , 3 , 3 . 5 } . The results in Figure 4 show that both algorithms exhibit a sublinear regret performance. Specifically, Algorithm 1, TSDE, achieves an ˜ O ( √ T ) as predicted in our theoretical results of Theorem 1 and Corollary 1. Furthermore, in both queueing models, our proposed algorithm consistently outperforms the algorithm presented in [4] in terms of regret order.

2t(θ)

1.8

1.4

0.6

1.0

1.0

θ1

02

1.4

1.8

0.6

(a) Model 1: Queueing system of Figure 2a.

3.8

3.2

m

2.6

2.0

1.4

1.8

1.4

0.6

1.0

1.0

θ1

1.4

1.8

0.6

(b) Model 2: Queueing system of Figure 2b.

Figure 6: Optimal policy parameters for different service rate vectors in the two exemplary queuing systems in Model 1 and Model 2 with λ = 0 . 5 .

0.6

Average cost

0.4

0.2

01=1.1,02 =0.5

01=1.3,02 =0.5

01=1.5,02 =0.5

0.0

0.0

2.5

5.0

7.5

T

1e4

G.2 Model 1: Two-server queueing system with a common buffer.

Figure 3b illustrates the behavior of the regret of Model 1 for three different arrival rate values and averaged over 2000 simulation runs. In these simulations, the parameter space is selected as

which results in a prior size of 105 . As depicted in Figure 3a, the regret has a sub-linear behavior and increases with the arrival rate. The total variation distance between the posterior and real distribution, a point-mass on the random θ ∗ , are plotted in Figure 5a. As expected, the distance diminishes towards 0, indicating the learning of the true parameter. As mentioned in Appendix E.1, the optimal policy minimizing the average number of jobs in a system with parameter θ , is a threshold policy π t ( θ ) with optimal finite threshold t ( θ ) ∈ N , which can be numerically determined as the smallest i ∈ N for which J i ( θ ) < J i +1 ( θ ) , calculated in [38]. We compute the optimal threshold t ( θ ) for every θ ∈ Θ and present the results in Figure 6a. We can see that the threshold increases as the ratio of the service rates grows. Specifically, this is why in Appendix G, we imposed conditions on Θ to ensure that the ratio between the service rates is both upper and lower bounded.

G.3 Model 2: Two heterogeneous parallel queues

Figure 3b illustrates the behavior of the regret of Model 2 for three different arrival rate values and averaged over 2000 simulation runs. We note that the regret is sub-linear and increases with higher arrival rates. In these simulations, the parameter space is selected as

which results in a prior size of 28 . As discussed earlier, our goal is to find the average cost minimizing policy within the class of policies Π = { π ω ; ω ∈ [( c R R ) -1 , c R R ] } , c R ≥ 1 , where π ω ( x ) =

arg min (1 + x 1 , ω (1 + x 2 )) with ties broken for 1 . As discussed before, even with the transition kernel fully specified (by the values of arrival and service rates), the optimal policy in Π is not known except when θ 1 = θ 2 where the optimal value is ω = 1 , and so, to learn it, we will use Proximal Policy Optimization with approximating martingale-process (AMP) method for countable state-space MDPs [18]. We run the algorithm for 200 policy iterations, using 20 actors for each iteration. We take the state (0 , 0) as a regeneration state and simulate 1500 independent regenerative cycles per actor in each algorithm iteration. To approximate the value function, we employ a fully connected feed-forward neural network with one hidden layer consisting of 10 × 10 units and ReLU activation functions. The AMP method is also employed for variance reduction in value function estimation. The optimal ω for every θ ∈ Θ is shown in Figure 6b, indicating that ω increases as the ratio of the service rates grows. Therefore, it is necessary to ensure that the ratio between the service rates is bounded from above and below. Furthermore, to evaluate the regret numerically, the value of J ( θ ) is required for every θ ∈ Θ , which is not known. Thus, after finding the optimal ω using the PPO algorithm, we perform a separate simulation to approximate the optimal average cost. In Figure 7, we plot the estimated average cost for three different service rate vectors, demonstrating that the optimal average cost decreases as the service rates increase. In Figure 5b we also depict the total variation distance between the posterior and real distribution, which is a point-mass on the random θ ∗ , and observe that the distance is converging to zero.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on an online learning setting for Markov Decision Processes (MDPs) with a countably infinite number of states. It adopts a Bayesian learning perspective, assuming that the parameters of the MDP follow a prior distribution over a known parameter space. The paper proposes a Thompson-sampling-like approach to solve the MDP in an online fashion. This approach assumes access to an optimal policy oracle, where the parameters of the MDP are provided as inputs, and it also relies on specific assumptions about the features of the parameter space.\n\nSTRENGTHS:\nThe model investigated in this paper exhibits a high degree of generality, and the results presented contribute significantly to the field of theoretical reinforcement learning by offering near-optimal algorithms for MDPs without a bounded state space. The inherent complexity of the problem necessitates intricate proofs, and although I haven't examined the complete proof in detail, I have confidence in the correctness of the underlying intuitions. The proof combines Lyapunov analysis with the proof presented in [38] for Bayesian learning in an MDP with a bounded state space, thus offering a potentially valuable contribution for future research. Additionally, the simulations conducted in the paper, which demonstrate the scaling of the algorithm's regret, are appreciated for providing empirical evidence supporting the algorithm's performance.\n\nWEAKNESSES:\nThe results of the paper rely on a set of assumptions that may be difficult to verify. Of particular concern is Assumption 3, which assumes stability of the optimal algorithm under one set of parameters for the MDP, even when considering another set of parameters. Establishing this property for more general systems can be challenging and requires careful calibration of the parameter space and policy space.\n\nThe algorithm presented in the paper is heavily dependent on access to an oracle capable of solving the optimal policy, which itself is a complex problem for general queueing systems. This reliance on an oracle can limit the practical applicability of the algorithm.\n\nThe algorithm necessitates returning to state 0 (line 14) at the end of each episode. This requirement could result in an exponential dependence on the maximum queue length, potentially rendering the algorithm less relevant for practical implementation. As a result, the claim of practicality made in line 345 may not be adequately supported.\n\nIn relation to the previous point, the paper obscures many constants within the theoretical results. These constants, associated with the system's dimension and ergodicity, could play a crucial role in determining practical performance and should be given more attention and consideration.\n\nQUESTIONS:\nIs there a heuristic for designing the parameter space and policy space to ensure that Assumption 3 holds for a queueing system? What would happen to the algorithm if this assumption fails? It would be beneficial to test the algorithm in a more general system, such as the ones described in [41, 52].\n\nHow crucial is the requirement for an optimal policy oracle? Would the analysis still hold true if the policy space is restricted to, for example, the MaxWeight policy? It would be highly valuable if the results were presented in a form such as \"the total queue lengths of this algorithm minus that of the MaxWeight policy is less than or equal to \\sqrt{T}\". This would also eliminate the need for the optimal oracle.\n\nCould you discuss the dependence of the regret on the system size? Why does the dependence on T for the regret hold more significance than these constants?\n\nPlease maintain consistency by using either \"queueing\" or \"queuing\" throughout the paper, but not both.\n\nIn line 122, after \"ergodicity)\", there should be a period.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study Bayesian learning of the problem of optimal control\nof a family of discrete-time countable state-space MDPs governed by an unknown parameter $\\theta$ from a general parameter space $\\Theta$ with each MDP evolving on a common countably-infinite state space $X$ and finite action space $A$.\nAs the setting is Bayesian, they assume that the model is governed by an unknown\nparameter $\\theta_\\star \\in \\Theta $ generated from a fixed and known prior distribution. \nThe learning goal: Bayesian regret minimization where the value function is the infinite-horizon average cost, and the regret is measured with respect to best policy in $\\Pi$.\nThey prove a $\\sqrt{TA}$ regret bound, up to poly-logarithmic factors, but the dependency in the complexity of the function class is unclear to me. \n\nDisclaimer: I am not much familiar with this area of RL literature and hence might not understand the results correctly. Also, I could not fully verify the correctness of the presented results.\nI gave that assessment  as the paper was very hard to follow for an unfamiliar reader.\n\nSTRENGTHS:\n1.\tI think the results are a nice contribution to the Bayesian  RL community. \n2.\tThe two examples presented contribute to the richness of the paper. \n3.\tThe presented bounds seem reasonable.\n4.\tThe adaption of Thompson sampling to countable state spaces might be useful in other RL settings.\n\nWEAKNESSES:\n1.\tAbstract is quite long and parts of it seems like copy-paste of the introduction.\n2.\tWriting requires improvement: (1) lightening the contribution compared to existing literature, (2) notation is hard to follow and makes proof reading hard, (3) complexity of $\\Pi$ or the dimension $d$ should appear in the presented bounds, even if the dependency on them is logarithmic, those parameters are not negligible. \n3.\tNeed to mention the dependency in complexity of the function class is in the regret bound. Is it logarithmic for finite $\\Pi$?  Is the regret dependent on the covering number? I would be happy if the authors could clarify that point.\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nN/A.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-01"}

comments: SUMMARY:
This paper present an adaptation of TSDE to parametric MDPs with unbounded state space.

The regret is sqrt{T} which is good but that is  under strong ergodic assumptions and lower order terms can harm the behavior of the algorithm for small values of T.

STRENGTHS:
- The paper is sound technically. (I quickly checked the proofs)

- The problem of learning unbounded MDP is a natural and important question to address.

WEAKNESSES:
1. The strong conditions (especially  Ass 4) are not  necessary and sufficient conditions for existence of an optimal policy, nor for the existence of a solution of the Bellman equation.

2. The queueing examples are not really convincing: The optimal service rates can be computed efficiently by just estimating the arrival rate and solving the optimal control problem for the expected sojourn time. Also, numerically,  the growing rate of the regret gets worse as the arrival rate increases.

QUESTIONS:
1. The assumption on the unicity of the optimal policy is not related with all the other assumptions and is not discussed. This is actually a strong assumption restricting the class of MDPs that can be learned. This deserves some discussion why it is needed here.


2. The authors should discuss the practical aspect of their approach. In particular, can one check whether all the assumptions are satisfied while the MDP is unknown?

3. The Q-learning approach seems more natural than a model based approach under unbounded space spaces, where only the visited states are used for learning. Several papers have already taken this option successfully and are not discussed here (see for example "Stable reinforcement learning with unbounded state space" by  D Shah, Q Xie, Z Xu )

LIMITATIONS:
I cannot see any limitations
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider the average reward Markov decision process framework with countable state spaces. The considered objective is to perform closed-loop optimal control for a family of MDPs parameterized in a compact space; this is a particularly interesting setting as the cost function is not assumed to be bounded. The authors propose a Thompson Sampling based algorithm and analyze its regret under suitable assumptions. They are able to show a finite time $\\sqrt{|\\mathcal{A}| T}$ bayesian regret bound. The authors also show the practical significance of their algorithm by an empirical application to queuing models.\n\nSTRENGTHS:\n- This paper is very relevant to the community as it opens the way towards studying RL in continuous spaces without the assumptions of bounded reward functions. Therefore tightening the connection of reinforcement learning theory and optimal control.\n\n- Although certain assumptions are somehow stringent, it is nice that the paper is able to convert an infinite horizon setting to a somewhat episodic setting. The latter usually allow for simpler analyses.\n\n- The application to a queuing model is also appreciated, a nice change from the standard RL benchmarks. \n\n- I would like to clarify that I did not go through the theoretical proofs and can therefore not comment on their correctness apart from the general intuition that such results should be possible given appropriate assumptions on stability and dynamics.\n\nWEAKNESSES:\n- The assumptions seem somewhat stringent. For example, the finite support of transitions can be challenged even in the specific example of queuing models when large amounts of arrivals are possible. More importantly, assumption 3 seems quite limiting, can the authors elaborate on what this implies for stability? I am not very knowledgeable in optimal control but it seems that you are assuming stability of all policies? \n\n- The paper is sometimes very dense and not straightforward to follow. For example, lines 173 to 192 requires prior knowledge of several papers and definitions, e.g. Poisson equation’s link to the problem at hand/ forcing function and other similar passages in the text. I would advise the authors to add the relevant definitions and Lemmas at least in the appendix to make the paper self contained.\n\n- The paper fails to cite many RL works in the continuous spaces setting. Namely, the entire line of function approximation in RL, for example: 1) “Frequentist Regret Bounds for Randomized Least-Squares Value Iteration” by Zanette et al. which provides a algorithm based on Thompson Sampling as well, and many other works (see references therein) in the model-free paradigm 2) “Bilinear Exponential Family of MDPs: Frequentist Regret Bound with Tractable Exploration and Planning” which also provides similar algorithms for MDPs that seem to include the queuing models presented here, see references therein for model-based approaches to RL in continuous spaces.\n\nQUESTIONS:\n- In line 203, where is $\\pi_{\\theta_k}^*$ defined? How is this implemented in the examples? \n\n- From your understanding of your assumptions, is the stability ensured by default? Or does Thompson sampling just ensure it by some magic in the proof? If the latter, and since you have a bound on the maximum $\\ell_\\infty$ norm of the states, is it not more straightforward to use existing RL algorithms? I guess what I’m asking is how necessary the Lyapunov-type analysis is in this case?\n\n- Why aren’t there comparisons with other RL algorithms in the experiments? especially since you implement PPO in the second model for your experiment, isn’t it possible to compare directly against it and other deep RL methods?\n\nLIMITATIONS:\nThe proposed work and algorithms is theoretical and does direct societal impacts. The possible theoretical limitations are detailed to some extent in the paper and to be addressed in future work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-24"}