SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Daniel Freund  
MIT
Cambridge, MA 02139 dfreund@mit.edu
Thodoris Lykouris  
MIT
Cambridge, MA 02139 lykouris@mit.edu
Wentao Weng  
MIT
Cambridge, MA 02139  
wweng@mit.edu
Queueing systems are widely applicable stochastic models with use cases in communication networks, healthcare, service systems, etc. Although their optimal control has been extensively studied, most existing approaches assume perfect knowledge of the system parameters. Of course, this assumption rarely holds in practice where there is parameter uncertainty, thus motivating a recent line of work on bandit learning for queueing systems. This nascent stream of research focuses on the asymptotic performance of the proposed algorithms.
In this paper, we argue that an asymptotic metric, which focuses on late-stage performance, is insufficient to capture the intrinsic statistical complexity of learning in queueing systems which typically occurs in the early stage. Instead, we propose the Cost of Learning in Queueing (CLQ), a new metric that quantifies the maximum increase in time-averaged queue length caused by parameter uncertainty. We characterize the CLQ of a single-queue multi-server system, and then extend these results to multi-queue multi-server systems and networks of queues. In establishing our results, we propose a unified analysis framework for CLQ that bridges Lyapunov and bandit analysis, provides guarantees for a wide range of algorithms, and could be of independent interest.<sup>1</sup>
Queueing systems are widely used stochastic models that capture congestion when services are limited. These models have two main components: jobs and servers. Jobs wait in queues and have different types. Servers differ in capabilities and speed. For example, in content moderation of online platforms [28], jobs are user posts with types defined by contents, languages and suspected violation types; servers are human reviewers who decide whether a post is benign or harmful. Moreover, job types can change over time upon receiving service. For instance, in a hospital, patients and doctors can be modeled as jobs and servers. A patient in the queue for emergent care can become a patient in the queue for surgery after seeing a doctor at the emergency department [3]. That is, queues can form a network due to jobs transitioning in types. Queueing systems also find applications in other domains such as call centers [16], communication networks [36] and computer systems [17].
The single-queue multi-server model is a simple example to illustrate the dynamics and decisions in queueing systems. In this model, there is one queue and  $\mathrm{K}$  servers operating in discrete periods. In each period, a new job arrives with probability  $\lambda$ . Servers have different service rates  $\mu_1, \ldots, \mu_K$ . The decision maker (DM) selects a server to serve the first job in the queue if there is any. If server  $j$  is selected, the first job in the queue leaves with probability  $\mu_j$ . The DM aims to minimize the average wait of each job, which is equivalent to minimizing the queue length. The optimal policy thus selects the server with the highest service rate  $\mu^{\star} = \max_{j} \mu_j$ ; the usual regime of interest is one where the system is stabilizable, i.e.,  $\mu^{\star} > \lambda$ , which ensures that the queue length does not grow to
<sup>1</sup>A full version of this paper [12] can be found at https://arxiv.org/abs/2308.07817v2.
infinity under an optimal policy. Of course, this policy requires perfect knowledge of the service rates. Under parameter uncertainty, the DM must balance the trade-off between exploring a server with an uncertain rate or exploiting a server with the highest observed service rate.
A recent stream of work studies efficient learning algorithms for queueing systems. First proposed by [41], and later used by [24, 38], queueing regret is a common metric to evaluate learning efficiency in queueing systems. In the single-queue multi-server model, let  $Q(T,\pi)$  and  $Q^{\star}(T)$  be the number of jobs in period  $T$  under a policy  $\pi$  and under the optimal policy respectively. Queueing regret is defined as either the last-iterate difference in expected queue length, i.e.,  $\mathbb{E}\left[Q(T,\pi) - Q^{\star}(T)\right]$  [41, 24], or the time-average version  $\frac{1}{T}\mathbb{E}\left[\sum_{t = 1}^{T}Q(t,\pi) - Q^{\star}(t)\right]$  [38]. In the stabilizable case  $(\lambda < \mu^{\star})$ , the goal is to bound its scaling relative to the time horizon; examples include  $o(T)$  [41],  $\tilde{O}(1/T)$  [24], and  $O(1/T)$  [38].
In this paper, we argue that an asymptotic metric for per-period queue length does not capture the statistical complexity of learning in queueing systems. This is because, in a queueing system, learning happens in initial periods while queueing regret focuses only on late periods. Whereas cumulative regret in multi-armed bandits is non-decreasing in  $T$ , the difference in queue lengths between a learning policy and the benchmark optimal policy eventually decreases since the policy eventually learns the parameters (see Figure 1). This leads to the two main questions of this paper:
1. What metric characterizes the statistical complexity of learning in queueing systems?
2. What are efficient learning algorithms for queueing systems?
Our work studies these questions in general queueing systems that go beyond the single-queue multi-server model and can capture settings such as the hospital and content moderation examples.
Figure 1: Expected per-period and time-averaged queue lengths of UCB and Q-UCB [24] in a singlequeue setting with  $K = 5$ ,  $\lambda = 0.45$ ,  $\mu = (0.045, 0.35, 0.35, 0.35, 0.55)$ ; results are averaged over 50 runs. The difference between both algorithms' queue lengths is indistinguishable asymptotically (left figure) though they clearly differ in their learning efficiency for early periods as illustrated by Cost of Learning in Queueing, the metric that our work introduces (right figure).
Cost of learning in queueing. Tackling the first question, we propose the Cost of Learning in Queueing (CLQ) to capture the efficiency of a learning algorithm in queueing systems. The CLQ of a learning policy is defined as the maximum difference of its time-averaged queue length and that of any other policy (with knowledge of parameters) over the entire horizon (see Fig 1 on the right). In contrast to queueing regret, CLQ is 1) a finite-time metric that captures the learning efficiency in early periods and 2) focused on time-averaged queue length instead of per-period queue length. This is favorable as for any periods  $1,\ldots ,T$ , the time-averaged queue length is related to the average wait time by Little's Law. The formal definition of CLQ can be found in Section 3.
Lower bound of CLQ (Theorem 1). To characterize the statistical complexity of learning in queueing systems, we consider the simplest non-trivial stabilizable setting that involves one queue and  $K$  servers. It is known that the queue length scales as  $O(1 / \varepsilon)$  under the optimal policy, where  $\varepsilon = \mu^{\star} - \lambda$  is the traffic slackness of the system. Fixing  $\varepsilon$  and the number of servers  $K$ , we establish
a worst-case lower bound  $\Omega\left(\frac{K}{\varepsilon}\right)$  of CLQ. That is, for any  $\varepsilon, K$  and a fixed policy, there always exists a setting of arrival and service rates, such that the CLQ of this policy is at least  $\Omega\left(\frac{K}{\varepsilon}\right)$ . Combined with the  $O(1 / \varepsilon)$  optimal queue length, this lower bound result shows that the effect of learning dominates the performance of queueing systems when  $K$  is large (as it may increase the maximum time-averaged queue length by a factor of  $K$ ). This is shown in Figure 1 (right) where the peak of time-averaged queue lengths of the optimal policy with knowledge of parameters is much lower than that of the other two policies (our Algorithm 1 and Q-UCB [24]).
An efficient algorithm for single-queue multi-server systems (Theorem 2). Given the above lower bound, we show that the Upper Confidence Bound (UCB) algorithm attains an optimal CLQ up to a logarithmic factor in the single-queue multi-server setting. Our analysis is motivated by Fig. 1 (right) where the time-averaged queue length initially increases and then decreases. Based on this pattern, we divide the horizon into an initial learning stage and a later regenerate stage.
In the learning stage, the queue length increases similar to the multi-armed bandit regret. We formalize this observation by coupling the queue under a policy  $\pi$  with a nearly-optimal queue and show that their difference is captured by the satisficing regret of  $\pi$ . Satisficing regret resembles the classical multi-armed bandit regret but disregards the loss of choosing a nearly optimal server (see Eq. (6)); this concept is studied from a Bayesian perspective in multi-armed bandits [32]. Nevertheless, our result in the learning stage is not sufficient as the satisficing regret eventually goes to infinity.
In the regenerate stage, queue lengths decrease as the policy has learned the parameters sufficiently well; the queue then behaves similarly as under the optimal policy and stabilizes the system. To capture this observation, we use Lyapunov analysis and show that the time-averaged queue length for the initial  $T$  periods scales as the optimal queue length, but with an additional term depending on the second moment of satisficing regret divided by  $T$ . Hence, as  $T$  increases, the impact of learning gradually disappears. Combining the results in the learning and regenerate stages, we obtain a tight CLQ bound of UCB for the single-queue multi-server setting.
Efficient algorithms for queueing networks (Theorem 3) We next generalize the above result to queueing networks that include multiple queues, multiple servers, and transitions of served jobs from servers to queues. For this setting, we build on the celebrated BACKPRESSURE policy that stabilizes a queueing network with knowledge of system parameters [39]. We propose BACKPRESSURE-UCB as a new algorithm to transform BACKPRESSURE into a learning algorithm with appropriate estimates for system parameters and show that its cost of learning scales near-optimally as  $\tilde{O}(1 / \varepsilon)$  with respect to the traffic slackness  $\varepsilon$  (Definition 2). This result extends our framework for single-queue multi-server settings through a coupling approach that reduces the loss incurred by learning in a high-dimensional vector of queue-lengths to a scalar-valued potential function. To the best of our knowledge, this is the first efficient learning algorithm for general queueing networks (see related work for a discussion).
Related work A recent line of work studies online learning in queueing systems [40]. To capture uncertainty in services, [41] studies a single-queue setting in which the DM selects a mode of service in each period and the job service time varies between modes (the dependence is a priori unknown and revealed to the DM after the service). The metric of interest is the queueing regret, i.e., the difference of queue length between an algorithm and the optimal policy, for which the authors show a sublinear bound. [24] considers the same single-queue multi-server setting as ours and show that a forced exploration algorithm achieves a queueing regret scaling of  $\tilde{O}(1/T)$  (under strong structural assumptions this result extends to multiple queues). [38] shows that by probing servers when the queue is idle, it is possible to give an algorithm with queueing regret converging as  $O(1/T)$ . However, with respect to the traffic slackness  $\varepsilon \to 0^{+}$ , both bounds yield suboptimal CLQ: [24] gives at least  $O(1/\varepsilon^2)$  and [38] gives at least  $O(1/\varepsilon^4)$  (see Appendices A.1, A.2 in [12]). In the analysis of [24], forced exploration is used for low adaptive regret, i.e., regret over any interval [19]; no such guarantee is known for adaptive exploration. But as noted by our Figure 1 and [24, Figure 2], an adaptive exploration algorithm like UCB has a better early-stage performance than Q-UCB. Using our framework in Section 4, we show that UCB indeed has a near-optimal CLQ that scales as  $O\left(\frac{K}{\varepsilon} \ln \frac{K}{\varepsilon}\right)$ . Our framework also allows us to show that Q-UCB enjoys a CLQ scaling as  $O\left(\frac{K}{\varepsilon} \ln \frac{K}{\varepsilon} + \ln^3 \frac{K}{\varepsilon}\right)$  (Appendix A.1 in [12]). This improves the guarantee implied by [24] and
shows the inefficiency due to forced exploration is about  $O(\ln^3 \frac{K}{\varepsilon})$  and that Q-UCB has both strong transient (CLQ) and asymptotic (queueing regret) performance.
Focusing on the scaling of queueing regret, [23] and [43] study the scheduling in multi-queue settings (with [43] also considering job abandonment), [9, 14] study learning for a load balancing model, [8] studies pricing and capacity sizing for a single-queue single-server model with unknown parameters. For more general settings, [1] designs a Bayesian learning algorithm for Markov Decision Processes with countable state spaces (of which queueing systems are special cases) where parameters are sampled from a known prior over a restricted parameter space; in contrast, our paper does not assume any prior of the unknown parameters. The main difference between all of these works and ours is that we focus on how the maximum time-averaged queue lengths scales with respect to system parameters (traffic slackness and number of queues and servers), not on how the queue lengths scale as time grows. Apart from the stochastic learning setting we focus on, there are works that tackle adversarial learning in queueing systems [21, 26]; these require substantially different algorithms and analyses.
Going beyond queueing regret, there are papers focusing on finite-time queue length guarantees. In a multi-queue multi-server setting, it is known that the MaxWeight algorithm has a polynomial queue length for stabilizable systems. However, it requires knowledge of system parameters. For a joint scheduling and utility maximization problem, [30] combines MaxWeight with forced exploration to handle parameter uncertainty. By selecting a suitable window for sample collection, their guarantee corresponds to a CLQ bound of at least  $O(K^4 / \varepsilon^3)$  for our single-queue setting (see Appendix A.3 in [12]). [37] studies a multi-queue multi-server setting and propose a frame-based learning algorithm based on MaxWeight. They focus on a greedy approximation which has polynomial queue lengths when the system is stabilizable with twice of the arrival rates. [42] considers a non-stationary setting and shows that combining MaxWeight with discounted UCB estimation leads to stability and polynomial queue length that scales as  $\tilde{O}(1 / \varepsilon^3)$  (Appendix A.4 in [12]). There is also a line of work studying decentralized learning in multi-queue multi-server settings. [15] assumes queues are selfish and derives conditions under which a no-regret learning algorithm is stable; this is generalized to queueing networks in which queues and servers form a directed acyclic graph by [13]. [33] allows collaborative agents and gives an algorithm with maximum stability, although the queue length scales exponentially in the number of servers. [11] designs a decentralized learning version of MaxWeight and shows that the algorithm always stabilizes the system with polynomial queue lengths  $\tilde{O}(1 / \varepsilon^3)$  (Appendix A.5 in [12]). In contrast to the above, our work shows for the centralized setting that MaxWeight with UCB achieves the near-optimal time-averaged queue length guarantee of  $\tilde{O}(1 / \varepsilon)$ .
Our paper extends the ability of online learning to general single-class queueing networks [7]. The literature considers different complications that arise in these settings, including jobs of different classes and servers that give service simultaneously to different jobs [39, 10, 7]. For the class of networks we consider, it is known that BACKPRESSURE can stabilize the system with knowledge of system parameters [39]. Noted in [7], one potential drawback of BACKPRESSURE is its need of full knowledge of job transition probabilities. In this regard, our paper contributes to the literature by proposing the first BACKPRESSURE-based algorithm that stabilizes queueing networks without knowledge of system parameters.
Moving beyond our focus on uncertainties in services, an orthogonal line of work studies uncertainties in job types. [2] considers a single server setting where an arriving job belongs to one of two types; but the true type is unknown and is learned by services. They devise a policy that optimizes a linear function of the numbers of correctly identified jobs and the waiting time. [6] studies a similar setting with two types of servers where jobs can route from one server to the others. They focus on the impact on stability due to job type uncertainties. [29, 34] consider multiple job types and server types. Viewing Bayesian updates as job type transitions, they use queueing networks to model the job learning process and give stable algorithms based on BACKPRESSURE. [22, 20] consider online matchings between jobs with unknown payoffs and servers where the goal is to maximize the total payoffs subject to stability. As noted in [29, 34, 22], one key assumption of this line of work is the perfect knowledge of server types and transition probability. Our result thus serves as a step to consider both server uncertainties and job uncertainties, at least in a context without payoffs.
Concurrently to our work, [31] proposes a frame-based MaxWeight algorithm with sliding-window UCB for scheduling in a general multi-queue multi-server system with non-stationary service rates. With a suitable frame size (depending on the traffic slackness), they show stability of the algorithm and obtain a queue length bound of  $\tilde{O}(1/\varepsilon^3)$  in the stationary setting (Appendix A.6 in [12]).
We consider a sequential learning setting where a decision maker (DM) repeatedly schedules jobs to a set of servers of unknown quality over discrete time periods  $t = 1,2,\ldots$ . For any  $T$ , we refer to the initial  $T$  periods as the time horizon  $T$ . To ease exposition, we first describe the simpler setting where there is only one job type (queue) and subsequently extend our approach to a general setting with multiple queues that interact through a network structure.
Single-queue multi-server system. A single-queue multi-server system is specified by a tuple  $(\mathcal{K},\lambda ,\mu)$ . There is one queue of jobs and a set of servers  $\mathcal{K}$  with  $|\mathcal{K}| = K$ . The arrival rate of jobs is  $\lambda$ , that is, in each period there is a probability  $\lambda$  that a new job arrives to the queue. The service rate of a server  $k\in \mathcal{K}$ , that is, the probability it successfully serves the job it is scheduled to work on, is  $\mu_{k}$ . Let  $Q(t)$  be the number of jobs at the start of period  $t$ . Initially there is no job and  $Q(1) = 0$ .
Figure 2 summarizes the events that occur in each period  $t$ . If there is no job in the queue, i.e.,  $Q(t) = 0$ , then the DM selects no server; to ease notation, they select the null server  $J(t) = \bot$ .<sup>2</sup> Otherwise, the DM selects a server  $J(t) \in \mathcal{K}$  and requests service for the first job in the queue. The
Figure 2: Flowchart for a single-queue multi-server system.
service request is successful with probability  $\mu_{J(t)}$  and the job then leaves the system; otherwise, it remains in the queue. At the end of the period, a new job arrives with probability  $\lambda$ . We assume that arrival and service events are independent. Let  $A(t)$  and  $\{S_k(t)\}_{k\in \mathcal{K}}$  be a set of independent Bernoulli random variables such that  $\mathbb{E}[A(t)] = \lambda$  and  $\mathbb{E}[S_k(t)] = \mu_k$  for  $k\in \mathcal{K}$ ; for the null server,  $\mu_{\perp} = S_{\perp}(t) = 0$  for all  $t$ . The queue length dynamics are thus given by
$$
Q (t + 1) = Q (t) - S _ {J (t)} (t) + A (t).
$$
A non-anticipatory policy  $\pi$  for the DM maps for every period  $t$  the historical observations until  $t$ , i.e.,  $(A(\tau), S_{J(\tau)}(\tau))_{\tau < t}$ , to a server  $J(t) \in \mathcal{K} \cup \{\bot\}$ . We define  $Q(t, \pi)$  as the queue length in period  $t$  under policy  $\pi$ . The DM's goal is to select a non-anticipatory policy  $\pi$  such that for any time horizon  $T \geq 1$ , the expected time-averaged queue length  $\frac{1}{T} \sum_{t < T} \mathbb{E}[Q(t, \pi)]$  is as small as possible.
When service rates are known, the policy  $\pi^{\star}$  selecting the server with the highest service rate  $\mu^{\star}$  in every period (unless the queue is empty) minimizes the expected time-averaged queue length for any time horizon [24, 38]. If  $\lambda \geq \mu^{\star}$ , even under  $\pi^{\star}$ , the expected time-averaged queue length goes to infinity as the time horizon increases. We thus assume  $\lambda < \mu^{\star}$ , in which case, the system is stabilizable, i.e., the expected time-averaged queue length under  $\pi^{\star}$  is bounded by a constant over the entire time horizon. We next define the traffic slackness of this system:
Definition 1. A single-queue multi-server system has a traffic slackness  $\varepsilon \in (0,1]$  if  $\lambda +\varepsilon \leq \mu^{\star}$ .
A larger traffic slackness implies that a system is easier to stabilize. It is known that the policy  $\pi^{\star}$  obtains an expected time-averaged queue length of the order of  $\frac{1}{\varepsilon}$  [36].
Queueing network. A queueing network extends the above case by having multiple queues and probabilistic job transitions after service completion; our model here resembles the one in [7]. A queueing network is defined by a tuple  $(\mathcal{N},\mathcal{K},\Lambda ,\mu ,\mathcal{A},\Sigma ,\mathcal{B},\mathcal{D},\boldsymbol {P})$  , where  $\pmb {\beta} = \{\mathcal{B}_n\}_{n\in \mathcal{N}}$ $\pmb {\mathcal{D}} = \{\mathcal{D}_k\}_{k\in \mathcal{K}}$  and  $P = (p_{k,n})_{k\in \mathcal{K},n\in \mathcal{N}\cup \{\bot \}}$  such that  $\sum_{n\in \mathcal{N}\cup \{\bot \}}p_{k,n} = 1,\forall k.$  In contrast to the single-queue case, there is now a set of queues  $\mathcal{N}$  with cardinality  $N$  and a virtual queue  $\perp$  to which jobs transition once they leave the system. Each queue  $n\in \mathcal{N}$  has a set of servers  $\mathcal{B}_n$  , each of which
${}^{2}$  Compared to common assumptions in the literature, e.g., [38, 11, 42], this makes for a more challenging setting as algorithms cannot learn service rates by querying servers in periods when they have no jobs.
belongs to a single queue. As before, the service rate of server  $k$  is  $\mu_{k}$ . The set  $\mathcal{D}_k$  contains the destination queues of server  $k$  (and can include the virtual queue  $\bot$ ).
In each period  $t$ , the DM selects a set of servers to schedule jobs to and, if a service request from queue  $n$  to server  $k$  is successful, a job from queue  $n$  transitions to a queue  $n' \in \mathcal{D}_k$  with probability  $p_{k,n'}$  (this implies  $p_{k,n'} = 0$  if  $n' \notin \mathcal{D}_k$ ). The selected set of servers comes from a set of feasible schedules  $\Sigma \subseteq \{0,1\}^{\mathcal{K}}$ , which captures interference between servers. We require that for any queue, the number of selected servers is no larger than the number of jobs in this queue. Formally, letting  $\sigma_k = 1$  if schedule  $\sigma \in \Sigma$  selects server  $k$  and denoting  $Q(t) = (Q_n(t))_{n \in \mathcal{N}}$  as the queue length vector at the beginning of period  $t$ , the set of feasible schedules in this period is
$$
\boldsymbol {\Sigma} _ {t} = \{\boldsymbol {\sigma} \in \boldsymbol {\Sigma}: \sum_ {k \in \mathcal {B} _ {n}} \sigma_ {k} \leq Q _ {n} (t), \forall n \in \mathcal {N} \}
$$