SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Saghar Adler  
University of Michigan
Vijay Subramanian  
University of Michigan
Models of many real-life applications, such as queueing models of communication networks or computing systems, have a countably infinite state-space. Algorithmic and learning procedures that have been developed to produce optimal policies mainly focus on finite state settings, and do not directly apply to these models. To overcome this lacuna, in this work we study the problem of optimal control of a family of discrete-time countable state-space Markov Decision Processes (MDPs) governed by an unknown parameter  $\theta \in \Theta$ , and defined on a countably-infinite state-space  $\mathcal{X} = \mathbb{Z}_+^d$ , with finite action space  $\mathcal{A}$ , and an unbounded cost function. We take a Bayesian perspective with the random unknown parameter  $\theta^*$  generated via a given fixed prior distribution on  $\Theta$ . To optimally control the unknown MDP, we propose an algorithm based on Thompson sampling with dynamically-sized episodes: at the beginning of each episode, the posterior distribution formed via Bayes' rule is used to produce a parameter estimate, which then decides the policy applied during the episode. To ensure the stability of the Markov chain obtained by following the policy chosen for each parameter, we impose ergodicity assumptions. From this condition and using the solution of the average cost Bellman equation, we establish an  $\tilde{O}(dh^d \sqrt{|\mathcal{A}|T})$  upper bound on the Bayesian regret of our algorithm, where  $T$  is the time-horizon. Finally, to elucidate the applicability of our algorithm, we consider two different queueing models with unknown dynamics, and show that our algorithm can be applied to develop approximately optimal control algorithms.
Many real-life applications, such as communication networks, supply chains, and computing systems, are modeled using queueing models with countably infinite state-space. In the existing analysis of these systems, the models are assumed to be known, but despite this, developing optimal control schemes is hard, with only a few examples worked out [35, 9, 54]. However, knowing the model, algorithmic procedures exist to produce approximately optimal policies [35] (such as value iteration and linear programming). Given the success of data-driven optimal control design, in particular Reinforcement Learning (RL), we explore the use of such methods for the countable state-space controlled Markov processes. However, current RL methods that focus on finite-state settings do not apply to the mentioned queueing models. With the model unknown, our goal is to develop a meta-learning scheme that is RL-based but obtains good performance by utilizing algorithms developed when models are known. Specifically, we study the problem of optimal control of a family of discrete-time countable state-space MDPs governed by an unknown parameter  $\theta$  from a general space  $\Theta$  with each MDP evolving on the countable state-space  $\mathcal{X} = \mathbb{Z}_{+}^{d}$  and finite action space  $\mathcal{A}$ . The cost function is unbounded and polynomially dependent on the state, following the examples of minimizing waiting times in queueing systems. Taking a Bayesian view, we assume the model is governed by an unknown parameter  $\theta^{*} \in \Theta$  generated from a fixed and known prior distribution. We aim to learn a policy  $\pi$  that minimizes the optimal infinite-horizon average cost over a given class of policies  $\Pi$  with low Bayesian regret with respect to the (parameter-dependent) optimal policy in  $\Pi$ .
To avoid many technical difficulties in countably infinite state-space settings, it is crucial to establish certain assumptions regarding the class of models from which the unknown system is drawn; some examples are: i) the number of deterministic stationary policies is not finite; and ii) in average cost optimal control problems, without stability/ergodicity assumptions, an optimal policy may not exist [40], and when it exists, it may not be stationary or deterministic [20]. With these in mind, we assume that for any state-action pair, the transition kernels in the model class are categorical and skip-free to the right, i.e., with finite support with a bound depending on the state only in an additive manner; both are common features of queueing models where an increase in state is due to arrivals. A second set of assumptions ensure stability by assuming that the Markov chains obtained by using different policies in  $\Pi$  are geometrically ergodic with uniformity across  $\Theta$ . From these assumptions, moments on hitting times are derived in terms of Lyapunov functions for polynomial ergodicity. These assumptions also yield a solution to the average cost optimality equation (ACOE) [9].
Contributions: To optimally control the unknown MDP, we propose an algorithm based on Thompson sampling with dynamically-sized episodes; posterior sampling is used based on its broad applicability and computational efficiency [46, 47]. At the beginning of each episode, a posterior distribution is formed using Bayes' rule, and an estimate is realized from this distribution which then decides the policy used throughout the episode. To evaluate the performance of our proposed algorithm, we use the metric of Bayesian regret, which compares the expected total cost achieved by a learning policy  $\pi_L$  until time horizon  $T$  with the policy achieving the optimal infinite-horizon average cost in the policy class  $\Pi$ . We consider regret guarantees in three different settings as follows:
1. In Theorem 1, for  $\Pi$  being the set of all policies and assuming that we have oracle access to the optimal policy for each parameter, we establish an  $\tilde{O}(dh^d\sqrt{|\mathcal{A}|T})$  upper bound on the Bayesian regret of this algorithm compared to the optimal policy.
2. In Corollary 1, where class  $\Pi$  is a subset of all stationary policies and where we know the best policy within this subset for each parameter via an oracle, we prove an  $\tilde{O}(dh^d\sqrt{|\mathcal{A}|T})$  upper bound on the Bayesian regret of our proposed algorithm, relative to the best-in-class policy.
3. In Theorem 2, we explore a scenario where we have access to an approximately optimal policy, rather than the optimal policy in set  $\Pi$  (which are all assumed to be stationary policies). When the approximately optimal policies satisfy Assumptions 3-4, we prove an  $\tilde{O}(dh^d\sqrt{|\mathcal{A}|T})$  regret bound, relative to the optimal policy in set  $\Pi$ .
Finally, to provide examples of our framework for developing approximately optimal control algorithms for stochastic systems with unknown dynamics, we study two different queueing models that meet our technical conditions. The first example is a continuous-time queueing system with two heterogeneous servers with unknown service rates and a common infinite buffer with the decision being the use of the slower server. Here, the optimal policy that minimizes the average waiting time is a threshold policy [38] which yields a queue-length after which the slower server is always used. The second model is a two-server queueing system, each with separate infinite buffers, to one of which a dispatcher routes an incoming arrival. Here, the optimal policy minimizing the waiting time is a switching-curve [26] with the specifics unknown for general parameter values, so we find the best policy within a commonly used set of switching-curve policies (Max-Weight policies [58, 59]), and assign the arrival to the queue with minimum weighted queue-length. For both models, we verify our assumptions for the class of optimal/best-in-class policies corresponding to different service rates and conclude that our proposed algorithm can be used to learn the optimal/best-in-class policy.
Related Work: Thompson sampling [62], or posterior sampling, has been applied to RL in many contexts of unknown MDPs [55, 45] and partially observed MDPs [28]; see tutorials [22, 50] for a comprehensive survey. It has been used in the parametric learning context [6] to minimize either Bayesian [46, 47, 49, 1, 60, 61] or frequentist [5, 23] regret. The bulk of the literature, including [5, 23, 49], analyzes finite-state and finite-action models but with different parameterizations such that a general dependence of the models on the parameters is allowed. The work in [61] studies general state-space MDPs but with a scalar parameterization with a Lipschitz dependence of the underlying models. Our problem formulation specifically considers countable state-space models with the models related via ergodicity, which we believe is a natural choice. Our focus on parametric learning is also connected to older work in adaptive control [3, 24] which studies asymptotically optimal learning for general parameter settings but with either a finite or countably infinite number of policies. Learning-based asymptotically optimal control in queues has a long history [36, 35] but recently there is increased work that also characterizes finite-time regret performance with respect to a well-known good policy or the optimal policy; see [63] for a survey. A series of work has studied
learning with Max-Weight policies to get stability and linear regret [44, 30] or just stability [65]. A recent related work [18] considers learning optimal paramterized policies in queueing networks when the MDP is known. In a finite or countable state-space setting of specific queueing models where the parameters can be estimated, many works [2, 17, 53, 32, 31, 14, 21, 16] have used forced exploration type schemes to obtain either regret that is constant or scaling logarithmically in the time-horizon.
Another line of work studies the problem of learning the optimal policy in an undiscounted finite-horizon MDP with a bounded reward function. Reference [66] uses a Thompson sampling-based learning algorithm with linear value function approximation to study an MDP with a bounded reward function in a finite-horizon setting. Reference [15] considers an episodic finite-horizon MDP with known bounded rewards but unknown transition kernels modeled using linearly parameterized exponential families. A maximum likelihood (ML) based algorithm coupled with exploration done by constructing high probability confidence sets around the ML estimate is used to learn the unknown parameters. In another work, [48] extends the problem setting of [15] to an episodic finite-horizon MDP with unknown rewards and transitions modeled using parametric bilinear exponential families. To learn the unknown parameters, they use a ML based algorithm with exploration done with explicit perturbation. We note that all mentioned works consider a finite-horizon problem. In contrast, our work considers an average cost problem, an infinite-horizon setting, and provides finite-time performance guarantees. In addition, these works focus on an MDP with a bounded reward function. Our focus, however, is learning in MDPs with unbounded rewards with the goal of covering practical queueing examples. We note that the parameterization of transitions used in [48, 15] can be used within our framework. However, similar to our work, additional stability assumptions are necessary to guarantee asymptotic learning and sub-linear regret. Another issue with exponential transition families is that they do not allow for 0 entries, which limits their applicability in queueing models.
In another work, [51] studies discounted MDPs with unknown dynamics, and unbounded state-space, but with bounded rewards, and learns an online policy that satisfies a specific notion of stability. It is also assumed that a Lyapunov function ensuring stability for the optimal policy exists. We note that [51] ignores optimality and focuses on finding a stable policy, which contrasts with our work that evaluates performance relative to the optimal policy. Secondly, [51] considers a discounted reward problem, essentially a finite-time horizon problem. Average cost problems, such as ours, are infinite-time horizon problems, so connections to discounted problems can only be made in the limit of the discount parameter going to 1. Moreover, [51] considers a bounded reward function, simplifying their analysis but not practical for many queueing examples. Further, the assumption of a stable optimal policy with a Lyapunov function (as in [51]) is highly restrictive for bounded reward settings with discounting. Additionally, average cost problems with bounded costs need strong state-independent recurrence conditions for the existence of (stationary) optimal solutions, which many queueing examples don't satisfy; see [12]. Further complications can also arise with bounded costs: e.g., [20] shows that a stationary average cost optimal policy may not exist.
We consider a family of discrete-time Markov Decision Processes (MDPs) governed by parameter  $\theta \in \Theta$  with the MDP for parameter  $\theta$  described by  $(\mathcal{X},\mathcal{A},c,P_{\theta})$ . For exposition purposes, we assume that all the MDPs are on (a common) countably infinite state-space  $\mathcal{X} = \mathbb{Z}_+^d$ . We denote the finite action space by  $\mathcal{A}$ , the transition kernel by  $P_{\theta}:\mathcal{X}\times \mathcal{A}\to \Delta (\mathcal{X})$ , and the cost function by  $c:\mathcal{X}\times \mathcal{A}\to \mathbb{R}_{+}$ . As mentioned earlier, we will take a Bayesian view of the problem and assume that the model is generated using an unknown parameter  $\pmb{\theta}^{*}\in \Theta$ , which is generated from a given fixed prior distribution  $\nu (\cdot)$  on  $\Theta$ . Our goal is to find a policy  $\pi :\mathcal{X}\rightarrow \mathcal{A}$  that tries to achieve Bayesian optimal performance in policy class II, i.e., minimizes the expected regret with  $\pmb{\theta}^{*}$  chosen from the prior distribution  $\nu (\cdot)$ . For each value  $\theta \in \Theta$ , the minimum infinite-horizon average cost is defined as
$$
J (\theta) = \inf  _ {\pi \in \Pi} \lim  _ {T \rightarrow \infty} \sup  _ {T \rightarrow \infty} \frac {1}{T} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} c (\boldsymbol {X} (t), A (t)) \right], \tag {1}
$$
where we optimize over a given class of policies  $\Pi$  and  $\pmb{X}(t) = (X_{1}(t),\dots,X_{d}(t))\in \mathcal{X}$  and  $A(t)\in \mathcal{A}$  are the state and action at  $t\in \mathbb{N}$ . Typically, we set this class to be all (causal) policies, but it is also possible to consider  $\Pi$  to be a proper subset of all policies as we will explore in our results. For a learning policy  $\pi_L$  that aims to select the optimal control without model knowledge but with
knowledge of  $\Theta$  and the prior  $\nu$ , the Bayesian regret until time horizon  $T \geq 2$  is defined as
$$
R (T, \pi_ {L}) = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left[ c (\boldsymbol {X} (t), A (t)) - J \left(\boldsymbol {\theta} ^ {*}\right) \right] \right], \tag {2}
$$
where the expectation is taken over  $\theta^{*} \sim \nu$  and the dynamics induced by  $\pi_{L}$ . Owing to underlying challenges in countable state-space MDPs, we require the below assumptions on the cost function.
Assumption 1. The cost function  $c: \mathcal{X} \times \mathcal{A} \to \mathbb{R}_+$  is assumed to satisfy the following two conditions:
1. For every number  $z \geq 0$  and action  $a \in \mathcal{A}$ ,  $c(\pmb{x}, a) \geq z$  outside a finite subset of  $\mathcal{X}$ .
2. The cost function is upper-bounded by a multivariate polynomial  $f_{c}:\mathbb{Z}_{+}^{d}\to \mathbb{R}_{+}$  which is increasing in every component on  $\pmb {x}\in \mathbb{Z}_+^d$  and has maximum degree of  $r$  ( $\geq 1$ ) in any dimension. We can assume that  $f_{c}(\pmb {x}) = K\sum_{i = 1}^{d}(x_{i})^{r}$  for some  $K > 0$ , where  $\pmb {x} = (x_{1},\dots ,x_{d})$ .
Thus, the cost function increases without bound (in the state) at a polynomial rate. This assumption is common in practice—holding costs in queueing models are polynomial in the state components. To avoid technical issues the infinite state-space setting also necessitates some assumptions on the class from which the unknown model is drawn. For instance, irreducibility of Markov chains on such state-spaces does not ensure positive recurrence (and ergodicity). Moreover, for average cost optimal control problems, without stability even the existence of an optimal policy is not guaranteed, and we need more conditions. The following assumption ensures a skip-free behaviour for transitions, which holds in many queueing models, where an increase in state corresponds to (new) arrivals.
Assumption 2. From any state-action pair  $(\pmb{x}, a)$ , the transition is to a finite number of states. We also assume that all transition kernels are skip-free to the right: for some  $h \geq 1$  which is independent of  $\theta \in \Theta$  and  $(\pmb{x}, a) \in \mathcal{X} \times \mathcal{A}$ , we have  $P_{\theta}(\pmb{x}'; \pmb{x}, a) = 0$  for all  $\pmb{x}' \in \{\tilde{\pmb{x}} \in \mathbb{Z}_+^d : \| \tilde{\pmb{x}} \|_1 > \| \pmb{x} \|_1 + h\}$ .
Learning necessitates some commonalities within the class of models so that using a policy well-suited to one model provides information on other models too. For us, these are in the form of constraints on the transition kernels of the models and stability assumptions. As simple union bound arguments don't work in the countably infinite state-space setting, we will use the stability assumptions instead. In our setting, we consider a class of models, each with a policy being well-suited to at least one model in the class, and use the set of policies to search within. Using a reduced set of policies is necessary as the number of deterministic stationary policies is infinite. To learn correctly while restricting attention to this subset policy class, requires some regularity assumptions when a policy well-suited to one model is tried on a different model. Our ergodicity assumptions are one convenient choice; see Appendix A.1 for details. These assumptions let us characterize the distributions of the first passage times of the Markov processes via stability conditions; see Lemmas 10 and 11.
Assumption 3. For any MDP  $(\mathcal{X},\mathcal{A},c,P_{\theta})$  with parameter  $\theta \in \Theta$ , there exists a unique optimal policy  $\pi_{\theta}^{*}$  that minimizes the infinite-horizon average cost within the class of policies  $\Pi$ . Furthermore, for any  $\theta_{1},\theta_{2}\in \Theta$ , the Markov process with transition kernel  $P_{\theta_1}^{\pi_{\theta_2}^*}$  obtained from the MDP  $(\mathcal{X},\mathcal{A},c,P_{\theta_1})$  by following policy  $\pi_{\theta_2}^*$  is irreducible, aperiodic, and geometrically ergodic with geometric ergodicity coefficient  $\gamma_{\theta_1,\theta_2}^g\in (0,1)$  and stationary distribution  $\mu_{\theta_1,\theta_2}$ . This is equivalent to the existence of finite set  $C_{\theta_1,\theta_2}^g$  and Lyapunov function  $V_{\theta_1,\theta_2}^g:\mathcal{X}\to [1, + \infty)$  satisfying
$\Delta V_{\theta_1,\theta_2}^g (\pmb {x})\leq -\big(1 - \gamma_{\theta_1,\theta_2}^g\big)V_{\theta_1,\theta_2}^g (\pmb {x}),$ $\pmb {x}\in \mathcal{X}\setminus C_{\theta_1,\theta_2}^g$  and  $P_{\theta_1}^{\pi_\theta_2^*}V_{\theta_1,\theta_2}^g (\pmb {x}) <   + \infty$ $\pmb {x}\in C_{\theta_1,\theta_2}^g$  where  $\Delta V_{\theta_1,\theta_2}^g (\pmb {x}):= P_{\theta_1}^{\pi_\theta_2^*}V_{\theta_1,\theta_2}^g (\pmb {x}) - V_{\theta_1,\theta_2}^g (\pmb {x}).$  Setting  $b_{\theta_1,\theta_2}^g := \max_{\pmb {x}\in C_{\theta_1,\theta_2}^g}P_{\theta_1}^{\pi_\theta_2^*}V_{\theta_1,\theta_2}^g (\pmb {x})+$ $V_{\theta_1,\theta_2}^g (\pmb {x})$  yields
$$
\Delta V _ {\theta_ {1}, \theta_ {2}} ^ {g} (\boldsymbol {x}) \leq - \left(1 - \gamma_ {\theta_ {1}, \theta_ {2}} ^ {g}\right) V _ {\theta_ {1}, \theta_ {2}} ^ {g} (\boldsymbol {x}) + b _ {\theta_ {1}, \theta_ {2}} ^ {g} \mathbb {I} _ {C _ {\theta_ {1}, \theta_ {2}} ^ {g}} (\boldsymbol {x}), \quad \boldsymbol {x} \in \mathcal {X}. \tag {3}
$$
Then, we have the following assumptions relating all the models in  $\Theta$ :
1. The geometric ergodicity coefficient is uniformly bounded below 1:  $\gamma_{*}^{g}\coloneqq \sup_{\theta_{1},\theta_{2}\in \Theta}\gamma_{\theta_{1},\theta_{2}}^{g} < 1$
2. We assume that  $\{0^d\} \subseteq \cap_{\theta_1,\theta_2\in \Theta}C_{\theta_1,\theta_2}^g$  and  $C_*^g = \cup_{\theta_1,\theta_2\in \Theta}C_{\theta_1,\theta_2}^g$  is a finite set. We further assume that  $b_{*}^{g}\coloneqq \sup_{\theta_{1},\theta_{2}}b_{\theta_{1},\theta_{2}}^{g} < + \infty$
Remark 1. The uniqueness of the optimal policy is not essential for the validity of our results, provided that all optimal policies satisfy our assumptions. When this condition is not met, we need to select an optimal policy that is geometrically ergodic for all  $\theta \in \Theta$ . This issue can be avoided by using a smaller subset of policies for which ergodicity can be shown, such as Max-Weight policies.
Geometric ergodicity implies that all moments of the hitting time of state  $0^d$ , say  $\tau_{0^d}$ , from any initial state  $\pmb{x} \neq 0^d$  are finite as  $\mathbb{E}_{\pmb{x}}[\kappa^{\tau_{0^d}}] \leq c_1 V^g(\pmb{x})$  (for specific  $\kappa > 1$  and  $c_1$ ), and so,  $\mathbb{E}_{\pmb{x}}[\tau_{0^d}^k] \leq c_1 V^g(\pmb{x}) k! / \log^k(\kappa) < +\infty$  for all  $k \in \mathbb{N}$ ; see Appendix A.2. Function  $V^g$  is typically exponential in some norm of the state and yields an exponential bound for moments of hitting times, and a poor regret bound. To improve the regret bound, we need a different drift equation with function  $V^p$  with polynomial dependence on a norm of the state that bounds certain polynomial moments of  $\tau_{0^d}$ .
Assumption 4. Given  $\theta_{1},\theta_{2}\in \Theta$ , Markov process obtained from MDP  $(\mathcal{X},\mathcal{A},c,P_{\theta_1})$  by following policy  $\pi_{\theta_2}^*$  is polynomially ergodic through the Foster-Lyapunov criteria: there exists a finite set  $C_{\theta_1,\theta_2}^p$ , constants  $\beta_{\theta_1,\theta_2}^p,b_{\theta_1,\theta_2}^p >0,\alpha_{\theta_1,\theta_2}^p\in [r_{r + 1},1)$ , and function  $V_{\theta_1,\theta_2}^p:\mathcal{X}\to [1, + \infty)$  satisfying
$$
\Delta V _ {\theta_ {1}, \theta_ {2}} ^ {p} (\boldsymbol {x}) \leq - \beta_ {\theta_ {1}, \theta_ {2}} ^ {p} \left(V _ {\theta_ {1}, \theta_ {2}} ^ {p} (\boldsymbol {x})\right) ^ {\alpha_ {\theta_ {1}, \theta_ {2}} ^ {p}} + b _ {\theta_ {1}, \theta_ {2}} ^ {p} \mathbb {I} _ {C _ {\theta_ {1}, \theta_ {2}} ^ {p}} (\boldsymbol {x}), \quad \boldsymbol {x} \in \mathcal {X}. \tag {4}
$$
1.  $V_{\theta_1, \theta_2}^p$  is a polynomial with positive coefficients, maximum degree (in any dimension)  $r_{\theta_1, \theta_2}^p$ , and sum of coefficients  $s_{\theta_1, \theta_2}^p$ . We assume  $r_*^p = \sup_{\theta_1, \theta_2} r_{\theta_1, \theta_2}^p < \infty$  and  $s_*^p = \sup_{\theta_1, \theta_2} s_{\theta_1, \theta_2}^p < \infty$ .
2. We assume that  $\{0^d\} \subseteq \cap_{\theta_1,\theta_2\in \Theta}C_{\theta_1,\theta_2}^p$  and  $C_*^p = \cup_{\theta_1,\theta_2\in \Theta}C_{\theta_1,\theta_2}^p$  is a finite set. We further assume that  $\beta_{*}^{p}\coloneqq \inf_{\theta_{1},\theta_{2}}\beta_{\theta_{1},\theta_{2}}^{p} > 0$  and  $b_{*}^{p}\coloneqq \sup_{\theta_{1},\theta_{2}}b_{\theta_{1},\theta_{2}}^{p} < \infty$
3. Let  $K_{\theta_1,\theta_2}(\pmb{x}) \coloneqq \sum_{n=0}^{\infty} 2^{-n-2}\left(P_{\theta_1}^{\pi_{\theta_2}^*}\right)^n(\pmb{x},0^d)$ , which is positive for any pair  $\theta_1,\theta_2 \in \Theta$  by irreducibility. We assume that it is strictly positive in  $\Theta$ :  $K_* \coloneqq \inf_{\theta_1,\theta_2} \min_{\pmb{x} \in C_*^p} K_{\theta_1,\theta_2}(\pmb{x}) > 0$ .