Title: Quantifying the Cost of Learning in Queueing Systems
Venue: NeurIPS 2023
Accepted: yes
Keywords: bandits

learning

queueing systems

optimal control
Abstract: Queueing systems are widely applicable stochastic models with use cases in communication networks, healthcare, service systems, etc. 
Although their optimal control has been extensively studied, most existing approaches assume perfect knowledge of the system parameters. Of course, this assumption rarely holds in practice where there is parameter uncertainty, thus motivating a recent line of work on bandit learning for queueing systems. This nascent stream of research focuses on the asymptotic performance of the proposed algorithms. 

In this paper, we argue that an asymptotic metric, which focuses on late-stage performance, is insufficient to capture the intrinsic statistical complexity of learning in queueing systems which typically occurs in the early stage. Instead, we propose the *Cost of Learning in Queueing (CLQ)*, a new metric that quantifies the maximum increase in time-averaged queue length caused by parameter uncertainty.
We characterize the CLQ of a single-queue multi-server system, and then extend these results to multi-queue multi-server systems and networks of queues. In establishing our results, we propose a unified analysis framework for CLQ that bridges Lyapunov and bandit analysis, provides guarantees for a wide range of algorithms, and could be of independent interest.

Quantifying the Cost of Learning in Queueing Systems

Abstract

Queueing systems are widely applicable stochastic models with use cases in communication networks, healthcare, service systems, etc. Although their optimal control has been extensively studied, most existing approaches assume perfect knowledge of the system parameters. Of course, this assumption rarely holds in practice where there is parameter uncertainty, thus motivating a recent line of work on bandit learning for queueing systems. This nascent stream of research focuses on the asymptotic performance of the proposed algorithms.

In this paper, we argue that an asymptotic metric, which focuses on late-stage performance, is insufficient to capture the intrinsic statistical complexity of learning in queueing systems which typically occurs in the early stage. Instead, we propose the Cost of Learning in Queueing (CLQ) , a new metric that quantifies the maximum increase in time-averaged queue length caused by parameter uncertainty. We characterize the CLQ of a single-queue multi-server system, and then extend these results to multi-queue multi-server systems and networks of queues. In establishing our results, we propose a unified analysis framework for CLQ that bridges Lyapunov and bandit analysis, provides guarantees for a wide range of algorithms, and could be of independent interest. 1

1 Introduction

Queueing systems are widely used stochastic models that capture congestion when services are limited. These models have two main components: jobs and servers. Jobs wait in queues and have different types. Servers differ in capabilities and speed. For example, in content moderation of online platforms [28], jobs are user posts with types defined by contents, languages and suspected violation types; servers are human reviewers who decide whether a post is benign or harmful. Moreover, job types can change over time upon receiving service. For instance, in a hospital, patients and doctors can be modeled as jobs and servers. A patient in the queue for emergent care can become a patient in the queue for surgery after seeing a doctor at the emergency department [3]. That is, queues can form a network due to jobs transitioning in types. Queueing systems also find applications in other domains such as call centers [16], communication networks [36] and computer systems [17].

The single-queue multi-server model is a simple example to illustrate the dynamics and decisions in queueing systems. In this model, there is one queue and K servers operating in discrete periods. In each period, a new job arrives with probability λ . Servers have different service rates µ 1 , . . . , µ K . The decision maker (DM) selects a server to serve the first job in the queue if there is any. If server j is selected, the first job in the queue leaves with probability µ j . The DM aims to minimize the average wait of each job, which is equivalent to minimizing the queue length. The optimal policy thus selects the server with the highest service rate µ ⋆ = max j µ j ; the usual regime of interest is one where the system is stabilizable , i.e., µ ⋆ > λ , which ensures that the queue length does not grow to

infinity under an optimal policy. Of course, this policy requires perfect knowledge of the service rates. Under parameter uncertainty, the DM must balance the trade-off between exploring a server with an uncertain rate or exploiting a server with the highest observed service rate.

A recent stream of work studies efficient learning algorithms for queueing systems. First proposed by [41], and later used by [24, 38], queueing regret is a common metric to evaluate learning efficiency in queueing systems. In the single-queue multi-server model, let Q ( T, π ) and Q ⋆ ( T ) be the number of jobs in period T under a policy π and under the optimal policy respectively. Queueing regret is defined as either the last-iterate difference in expected queue length, i.e., E [ Q ( T, π ) -Q ⋆ ( T )] [41, 24], or the time-average version 1 T E [ ∑ T t =1 Q ( t, π ) -Q ⋆ ( t ) ] [38]. In the stabilizable case ( λ < µ ⋆ ), the goal is to bound its scaling relative to the time horizon; examples include o ( T ) [41], ˜ O (1 /T ) [24], and O (1 /T ) [38].

In this paper, we argue that an asymptotic metric for per-period queue length does not capture the statistical complexity of learning in queueing systems. This is because, in a queueing system, learning happens in initial periods while queueing regret focuses only on late periods. Whereas cumulative regret in multi-armed bandits is non-decreasing in T , the difference in queue lengths between a learning policy and the benchmark optimal policy eventually decreases since the policy eventually learns the parameters (see Figure 1). This leads to the two main questions of this paper:

What metric characterizes the statistical complexity of learning in queueing systems?

What are efficient learning algorithms for queueing systems?

Our work studies these questions in general queueing systems that go beyond the single-queue multi-server model and can capture settings such as the hospital and content moderation examples.

10 0

10 1

10 2

10 3

10 4

10 5

Periods

t

0

20

40

60

80

100

120

Queueing regret

10 0

10 1

10 2

10 3

10 4

10 5

Periods

t

0

20

40

60

80

100

120

Time-averaged queue length

Cost of Learning in Queueing

UCB

Benchmark

Q-UCB

Cost of learning in queueing. Tackling the first question, we propose the Cost of Learning in Queueing (CLQ) to capture the efficiency of a learning algorithm in queueing systems. The CLQ of a learning policy is defined as the maximum difference of its time-averaged queue length and that of any other policy (with knowledge of parameters) over the entire horizon (see Fig 1 on the right). In contrast to queueing regret, CLQ is 1) a finite-time metric that captures the learning efficiency in early periods and 2) focused on time-averaged queue length instead of per-period queue length. This is favorable as for any periods 1 , . . . , T , the time-averaged queue length is related to the average wait time by Little's Law. The formal definition of CLQ can be found in Section 3.

Lower bound of CLQ (Theorem 1). To characterize the statistical complexity of learning in queueing systems, we consider the simplest non-trivial stabilizable setting that involves one queue and K servers. It is known that the queue length scales as O (1 /ε ) under the optimal policy, where ε = µ ⋆ -λ is the traffic slackness of the system. Fixing ε and the number of servers K , we establish

a worst-case lower bound Ω( K ε ) of CLQ. That is, for any ε, K and a fixed policy, there always exists a setting of arrival and service rates, such that the CLQ of this policy is at least Ω( K ε ) . Combined with the O (1 /ε ) optimal queue length, this lower bound result shows that the effect of learning dominates the performance of queueing systems when K is large (as it may increase the maximum time-averaged queue length by a factor of K ). This is shown in Figure 1 (right) where the peak of time-averaged queue lengths of the optimal policy with knowledge of parameters is much lower than that of the other two policies (our Algorithm 1 and Q-UCB [24]).

An efficient algorithm for single-queue multi-server systems (Theorem 2). Given the above lower bound, we show that the Upper Confidence Bound (UCB) algorithm attains an optimal CLQ up to a logarithmic factor in the single-queue multi-server setting. Our analysis is motivated by Fig. 1 (right) where the time-averaged queue length initially increases and then decreases. Based on this pattern, we divide the horizon into an initial learning stage and a later regenerate stage.

In the learning stage, the queue length increases similar to the multi-armed bandit regret. We formalize this observation by coupling the queue under a policy π with a nearly-optimal queue and show that their difference is captured by the satisficing regret of π . Satisficing regret resembles the classical multi-armed bandit regret but disregards the loss of choosing a nearly optimal server (see Eq. (6)); this concept is studied from a Bayesian perspective in multi-armed bandits [32]. Nevertheless, our result in the learning stage is not sufficient as the satisficing regret eventually goes to infinity.

In the regenerate stage, queue lengths decrease as the policy has learned the parameters sufficiently well; the queue then behaves similarly as under the optimal policy and stabilizes the system. To capture this observation, we use Lyapunov analysis and show that the time-averaged queue length for the initial T periods scales as the optimal queue length, but with an additional term depending on the second moment of satisficing regret divided by T . Hence, as T increases, the impact of learning gradually disappears. Combining the results in the learning and regenerate stages, we obtain a tight CLQ bound of UCB for the single-queue multi-server setting.

Efficient algorithms for queueing networks (Theorem 3) We next generalize the above result to queueing networks that include multiple queues, multiple servers, and transitions of served jobs from servers to queues. For this setting, we build on the celebrated BACKPRESSURE policy that stabilizes a queueing network with knowledge of system parameters [39]. We propose BACKPRESSURE-UCB as a new algorithm to transform BACKPRESSURE into a learning algorithm with appropriate estimates for system parameters and show that its cost of learning scales near-optimally as ˜ O (1 /ε ) with respect to the traffic slackness ε (Definition 2). This result extends our framework for single-queue multi-server settings through a coupling approach that reduces the loss incurred by learning in a high-dimensional vector of queue-lengths to a scalar-valued potential function. To the best of our knowledge, this is the first efficient learning algorithm for general queueing networks (see related work for a discussion).

Related work A recent line of work studies online learning in queueing systems [40]. To capture uncertainty in services, [41] studies a single-queue setting in which the DM selects a mode of service in each period and the job service time varies between modes (the dependence is a priori unknown and revealed to the DM after the service). The metric of interest is the queueing regret , i.e., the difference of queue length between an algorithm and the optimal policy, for which the authors show a sublinear bound. [24] considers the same single-queue multi-server setting as ours and show that a forced exploration algorithm achieves a queueing regret scaling of ˜ O (1 /T ) (under strong structural assumptions this result extends to multiple queues). [38] shows that by probing servers when the queue is idle, it is possible to give an algorithm with queueing regret converging as O (1 /T ) . However, with respect to the traffic slackness ε → 0 + , both bounds yield suboptimal CLQ: [24] gives at least O (1 /ε 2 ) and [38] gives at least O (1 /ε 4 ) (see Appendices A.1, A.2 in [12]). In the analysis of [24], forced exploration is used for low adaptive regret, i.e., regret over any interval [19]; no such guarantee is known for adaptive exploration. But as noted by our Figure 1 and [24, Figure 2], an adaptive exploration algorithm like UCB has a better early-stage performance than Q-UCB. Using our framework in Section 4, we show that UCB indeed has a near-optimal CLQ that scales as O ( K ε ln K ε ) . Our framework also allows us to show that Q-UCB enjoys a CLQ scaling as O ( K ε ln K ε +ln 3 K ε ) (Appendix A.1 in [12]). This improves the guarantee implied by [24] and

shows the inefficiency due to forced exploration is about O (ln 3 K ε ) and that Q-UCB has both strong transient (CLQ) and asymptotic (queueing regret) performance.

Focusing on the scaling of queueing regret, [23] and [43] study the scheduling in multi-queue settings (with [43] also considering job abandonment), [9, 14] study learning for a load balancing model, [8] studies pricing and capacity sizing for a single-queue single-server model with unknown parameters. For more general settings, [1] designs a Bayesian learning algorithm for Markov Decision Processes with countable state spaces (of which queueing systems are special cases) where parameters are sampled from a known prior over a restricted parameter space; in contrast, our paper does not assume any prior of the unknown parameters. The main difference between all of these works and ours is that we focus on how the maximum time-averaged queue lengths scales with respect to system parameters (traffic slackness and number of queues and servers), not on how the queue lengths scale as time grows. Apart from the stochastic learning setting we focus on, there are works that tackle adversarial learning in queueing systems [21, 26]; these require substantially different algorithms and analyses.

Going beyond queueing regret, there are papers focusing on finite-time queue length guarantees. In a multi-queue multi-server setting, it is known that the MaxWeight algorithm has a polynomial queue length for stabilizable systems. However, it requires knowledge of system parameters. For a joint scheduling and utility maximization problem, [30] combines MaxWeight with forced exploration to handle parameter uncertainty. By selecting a suitable window for sample collection, their guarantee corresponds to a CLQ bound of at least O ( K 4 /ε 3 ) for our single-queue setting (see Appendix A.3 in [12]). [37] studies a multi-queue multi-server setting and propose a frame-based learning algorithm based on MaxWeight. They focus on a greedy approximation which has polynomial queue lengths when the system is stabilizable with twice of the arrival rates. [42] considers a non-stationary setting and shows that combining MaxWeight with discounted UCB estimation leads to stability and polynomial queue length that scales as ˜ O (1 /ε 3 ) (Appendix A.4 in [12]). There is also a line of work studying decentralized learning in multi-queue multi-server settings. [15] assumes queues are selfish and derives conditions under which a no-regret learning algorithm is stable; this is generalized to queueing networks in which queues and servers form a directed acyclic graph by [13]. [33] allows collaborative agents and gives an algorithm with maximum stability, although the queue length scales exponentially in the number of servers. [11] designs a decentralized learning version of MaxWeight and shows that the algorithm always stabilizes the system with polynomial queue lengths ˜ O (1 /ε 3 ) (Appendix A.5 in [12]). In contrast to the above, our work shows for the centralized setting that MaxWeight with UCB achieves the near-optimal time-averaged queue length guarantee of ˜ O (1 /ε ) .

Our paper extends the ability of online learning to general single-class queueing networks [7]. The literature considers different complications that arise in these settings, including jobs of different classes and servers that give service simultaneously to different jobs [39, 10, 7]. For the class of networks we consider, it is known that BACKPRESSURE can stabilize the system with knowledge of system parameters [39]. Noted in [7], one potential drawback of BACKPRESSURE is its need of full knowledge of job transition probabilities. In this regard, our paper contributes to the literature by proposing the first BACKPRESSURE-based algorithm that stabilizes queueing networks without knowledge of system parameters.

Moving beyond our focus on uncertainties in services, an orthogonal line of work studies uncertainties in job types. [2] considers a single server setting where an arriving job belongs to one of two types; but the true type is unknown and is learned by services. They devise a policy that optimizes a linear function of the numbers of correctly identified jobs and the waiting time. [6] studies a similar setting with two types of servers where jobs can route from one server to the others. They focus on the impact on stability due to job type uncertainties. [29, 34] consider multiple job types and server types. Viewing Bayesian updates as job type transitions, they use queueing networks to model the job learning process and give stable algorithms based on BACKPRESSURE. [22, 20] consider online matchings between jobs with unknown payoffs and servers where the goal is to maximize the total payoffs subject to stability. As noted in [29, 34, 22], one key assumption of this line of work is the perfect knowledge of server types and transition probability. Our result thus serves as a step to consider both server uncertainties and job uncertainties, at least in a context without payoffs.

Concurrently to our work, [31] proposes a frame-based MaxWeight algorithm with sliding-window UCB for scheduling in a general multi-queue multi-server system with non-stationary service rates. With a suitable frame size (depending on the traffic slackness), they show stability of the algorithm and obtain a queue length bound of ˜ O (1 /ε 3 ) in the stationary setting (Appendix A.6 in [12]).

2 Model

We consider a sequential learning setting where a decision maker (DM) repeatedly schedules jobs to a set of servers of unknown quality over discrete time periods t = 1 , 2 , . . . . For any T , we refer to the initial T periods as the time horizon T . To ease exposition, we first describe the simpler setting where there is only one job type (queue) and subsequently extend our approach to a general setting with multiple queues that interact through a network structure.

Single-queue multi-server system. A single-queue multi-server system is specified by a tuple ( K , λ, µ ) . There is one queue of jobs and a set of servers K with |K| = K . The arrival rate of jobs is λ , that is, in each period there is a probability λ that a new job arrives to the queue. The service rate of a server k ∈ K , that is, the probability it successfully serves the job it is scheduled to work on, is µ k . Let Q ( t ) be the number of jobs at the start of period t . Initially there is no job and Q (1) = 0 .

Figure 2 summarizes the events that occur in each period t . If there is no job in the queue, i.e., Q ( t ) = 0 , then the DM selects no server; to ease notation, they select the null server J ( t ) = ⊥ . 2 Otherwise, the DM selects a server J ( t ) ∈ K and requests service for the first job in the queue. The

Start of Period

t

Queue:

Q

(

t

)

>

0

DMSelects Server

J

(

t

)

Service Job from Queue

Job Leaves System

(Prob.

µ

J

(

t

)

)

New Job Arrival (Prob.

λ

)

Start of Period

t

+1

Yes

No

Successful

Unsuccessful

service request is successful with probability µ J ( t ) and the job then leaves the system; otherwise, it remains in the queue. At the end of the period, a new job arrives with probability λ . We assume that arrival and service events are independent. Let A ( t ) and { S k ( t ) } k ∈K be a set of independent Bernoulli random variables such that E [ A ( t )] = λ and E [ S k ( t )] = µ k for k ∈ K ; for the null server, µ ⊥ = S ⊥ ( t ) = 0 for all t . The queue length dynamics are thus given by

Anon-anticipatory policy π for the DM maps for every period t the historical observations until t , i.e., ( A ( τ ) , S J ( τ ) ( τ )) τ<t , to a server J ( t ) ∈ K ∪ {⊥} . We define Q ( t, π ) as the queue length in period t under policy π . The DM's goal is to select a non-anticipatory policy π such that for any time horizon T ≥ 1 , the expected time-averaged queue length 1 T ∑ t ≤ T E [ Q ( t, π )] is as small as possible.

When service rates are known, the policy π ⋆ selecting the server with the highest service rate µ ⋆ in every period (unless the queue is empty) minimizes the expected time-averaged queue length for any time horizon [24, 38]. If λ ≥ µ ∗ , even under π ⋆ , the expected time-averaged queue length goes to infinity as the time horizon increases. We thus assume λ < µ ⋆ , in which case, the system is stabilizable , i.e., the expected time-averaged queue length under π ⋆ is bounded by a constant over the entire time horizon. We next define the traffic slackness of this system:

Definition 1. A single-queue multi-server system has a traffic slackness ε ∈ (0 , 1] if λ + ε ≤ µ ⋆ .

A larger traffic slackness implies that a system is easier to stabilize. It is known that the policy π ⋆ obtains an expected time-averaged queue length of the order of 1 ε [36].

Queueing network. A queueing network extends the above case by having multiple queues and probabilistic job transitions after service completion; our model here resembles the one in [7]. A queueing network is defined by a tuple ( N , K , Λ , µ , A , Σ , B , D , P ) , where B = {B n } n ∈N , D = {D k } k ∈K and P = ( p k,n ) k ∈K ,n ∈N∪{⊥} such that ∑ n ∈N∪{⊥} p k,n = 1 , ∀ k . In contrast to the single-queue case, there is now a set of queues N with cardinality N and a virtual queue ⊥ to which jobs transition once they leave the system. Each queue n ∈ N has a set of servers B n , each of which

belongs to a single queue. As before, the service rate of server k is µ k . The set D k contains the destination queues of server k (and can include the virtual queue ⊥ ).

In each period t , the DM selects a set of servers to schedule jobs to and, if a service request from queue n to server k is successful, a job from queue n transitions to a queue n ′ ∈ D k with probability p k,n ′ (this implies p k,n ′ = 0 if n ′ ̸∈ D k ). The selected set of servers comes from a set of feasible schedules Σ ⊆ { 0 , 1 } K , which captures interference between servers. We require that for any queue, the number of selected servers is no larger than the number of jobs in this queue. 3 Formally, letting σ k = 1 if schedule σ ∈ Σ selects server k and denoting Q ( t ) = ( Q n ( t )) n ∈N as the queue length vector at the beginning of period t , the set of feasible schedules in this period is

and the DM's decision in period t is to select a schedule σ ( t ) ∈ Σ t . Following [39], we assume that any subset of a feasible schedule is still feasible, i.e., if σ ∈ Σ and σ ′ k ≤ σ k ∀ k ∈ K , then σ ′ ∈ Σ .

We now formalize the arrival and service dynamics in every period, which are captured by the independent random variables { A ( t ) , { S k ( t ) } k ∈K } t . The arrival vector A ( t ) = { A n ( t ) } n ∈N consists of (possibly correlated) random variables A n ( t ) taking value in A ⊆ { 0 , 1 } N ; we denote its distribution by Λ and let E [ A n ( t )] = λ n ( Λ ) with λ = ( λ n ( Λ )) n ∈N . The service S k ( t ) for each server k ∈ K is a Bernoulli random variable indicating whether the selected service request was successful. 4 To formalize the job transition, let L k ( t ) = ( L k,n ( t )) n ∈N∪{⊥} be a random vector over { 0 , 1 } N∪{⊥} for server k independent of other randomness such that P { L k,n ( t ) = 1 } = p k,n and ∑ n ∈N∪{⊥} L k,n ( t ) = 1 . The queueing dynamic is given by

We assume that the DM has knowledge of which policies are allowed, i.e., they know Σ and B , but has no prior knowledge of the rates λ , µ , P and the set D . In period t , the observed history is the set ( { A n ( τ ) } n ∈N , { S k ( τ ) L k,n ( τ ) } n ∈ ¯ N ,k ∈K : σ k ( τ )=1 ) τ<t that includes transition information on top of arrivals and services. Note that a job transition is only observed when the server is selected and the service is successful. Similar to before, a non-anticipatory policy π maps an observed history to a feasible schedule; we let Q n ( t, π ) be the length of queue n in period t under this policy.

Unlike the single-queue case, it is usually difficult to find the optimal policy for a queueing network even with known system parameters. Fortunately, if the system is stabilizable, i.e., lim t →∞ 1 T ∑ t ≤ T E [ ∥ Q ( t ) ∥ 1 ] < ∞ under some scheduling policy, then the arrival rate vector must be within the capacity region of the servers [39]. Formally, let Φ = { ϕ ∈ [0 , 1] Σ : ∑ σ ∈ Σ ϕ σ = 1 } be the probability simplex over Σ . A distribution ϕ in Φ can be viewed as the frequency of a policy using each schedule σ ∈ Σ , and the effective service rate queue n can get is given by µ net n ( ϕ ) = ∑ σ ∈ Σ ϕ σ (∑ k ∈B n σ k µ k -∑ k ′ ∈K σ k ′ µ k ′ p k ′ ,n ) ; this includes both job inflow and outflow. We denote the effective service rate vector for a schedule distribution ϕ by µ net ( ϕ ) . Then, the capacity region is S ( µ , Σ , B , P ) = { µ net ( ϕ ): ϕ ∈ Φ } . For a queueing network to be stabilizable, we must have λ ( Λ ) ∈ S ( µ , Σ , B , P ) [39]. As in the single-queue case, we further assume that the system has a positive traffic slackness and let 1 denote a vector of 1 s with suitable dimension.

Definition 2. A queueing network has traffic slackness ε ∈ (0 , 1] if λ ( Λ ) + ε 1 ∈ S ( µ , Σ , B , P ) .

We also study a special case of queueing networks, multi-queue multi-server systems, where jobs immediately leave after a successful service, i.e., D k = {⊥} for all k ∈ K ; this extends the models in [11, 42] mentioned above. Since the transition probability matrix P is trivial ( p k, ⊥ = 1 , ∀ k ), we denote the capacity region of a multi-queue multi-server system by S ( µ , Σ , B ) .

3 Main results: the statistical complexity of learning in queueing

This section presents our main results on the statistical complexity of learning in queueing systems. We first define the Cost of Learning in Queueing , or CLQ as a shorthand, a metric capturing this complexity and provide a lower bound for the single-queue multi-server setting. Motivated by this, we design an efficient algorithm for the single-queue multi-server setting with a matching CLQ and then extend this to the multi-queue multi-server and queueing network systems.

3.1 Cost of Learning in Queueing

We first consider learning in the single-queue multi-server setting. Previous works on learning in queueing systems focus on the queueing regret E [ Q ( T, π ) -Q ( T, π ⋆ )] in the asymptotic regime of T →∞ . The starting point of our work stems from the observation that an asymptotic metric, which measures performance in late periods, cannot capture the complexity of learning as learning happens in initial periods (recall the left of Figure 1). In addition, a guarantee on per-period queue length cannot easily translate to the service experience (or wait time) of jobs.

Motivated by the above insufficiency of queueing regret, we define the Cost of Learning in Queueing (or CLQ) as the maximum increase in expected time-averaged queue lengths under policy π compared with the optimal policy. Specifically, we define the single-queue CLQ as:

As shown in Figure 1 (right), CLQ is a finite-time metric and explicitly takes into account how fast learning occurs in the initial periods. In addition, a bound on the maximum increase in time-averaged queue length translates approximately (via Little's Law [27]) to a bound on the increase in average job wait times.

Given that the traffic slackness measures the difficulty of stabilizing a system, we also consider the worst-case cost of learning in queueing over all pairs of ( λ, µ ) with a fixed traffic slackness ε . In a slight abuse of notation, we overload CLQ single to also denote this worst-case value, i.e.,

Our goal is to design a policy π , without knowledge of the arrival rate, the service rates, and the traffic slackness, that achieves low worst-case cost of learning in a single-queue multi-server system.

We can extend the definition of CLQ to the multi-queue multi-server and the queueing network settings. Since the optimal policy is difficult to design, we instead define CLQ for a policy π by comparing it with any non-anticipatory policy (which makes decisions only based on the history):

As in the single-queue setting, we can define the worst-case cost of learning for a fixed structure A , Σ , B , D and a traffic slackness ε as the supremum across all arrival, service, and transition rates with this traffic slackness. With the same slight abuse of notation as before, we denote these quantities by CLQ multi ( A , Σ , B , ε, π ) and CLQ net ( A , Σ , B , D , ε, π ) .

3.2 Lower bound on the cost of learning in queueing

Our first result establishes a lower bound on CLQ single ( K,ε,π ) . In particular, for any feasible policy π , we show a lower bound of Ω( K ε ) for sufficiently large K . With known parameters, the optimal time-averaged queue length is of the order of 1 / ε . Hence, our result shows that the cost of learning is non-negligible in queueing systems when there are many servers. For fixed K and ε , our lower bound considers the cost of learning of the worst-case setting and is instance-independent.

Theorem 1. For K ≥ 2 14 , ε ∈ (0 , 0 . 25] and feasible policy π , we have CLQ single ( K,ε,π ) ≥ K 2 14 ε .

√

Although our proof is based on the distribution-free lower bound Ω( KT ) for classical multi-armed bandits [5], this result does not apply directly to our setting. In particular, suppose the queue in our system is never empty. Then the accumulated loss in service of a policy is exactly the regret in bandits and the lower bound implies that any feasible policy serves at least Ω( √ KT ) jobs fewer than the optimal policy in the first T periods. However, due to the traffic slackness, the queue does get empty under the optimal policy, and in periods when this occurs, the optimal policy also does not receive service. As a result, the queue length of a learning policy could be lower than Θ( √ KT ) despite the loss of service compared with the optimal policy.

We next discuss the intuition (formal proof in Appendix C.1 of [12]). Fixing K and ε , suppose the gap in service rates between the optimal server and others is 2 ε . Then for any t in a time horizon T = O ( K ε 2 ) , the number of arrivals in the first t periods is around λt and the potential service of the optimal server is around ( λ + ε ) t . By the multi-armed bandit lower bound, the total service of a policy is at most around λt + εt -√ Kt ≤ λt since t ≤ T = O ( K ε 2 ) . Then the combined service rate of servers chosen in the first T periods, i.e., ∑ t ≤ T µ J ( t ) , is strictly bounded from above by the total arrival rate λt . A carefully constructed example shows that the number of unserved jobs is around εt for every t ≤ T and thus the time-averaged queue length for the horizon T is of the order of εT = O ( K ε ) .

Remark 1. In [24, Proposition 3], the authors established an instance-dependent lower bound on E [ Q ( t, π ) -Q ⋆ ( t )] . The implied CLQ lower bound is weaker than ours ( ˜ Ω( K ) , see Appendix C.2 in [12]) and is constrained to α -consistent policy π (see definition 1 in [24]) whereas ours does not.

3.3 Upper bound on the cost of learning in queueing

Motivated by the lower bound, we propose efficient algorithms with a focus on heavy-traffic optimality, i.e., ensuring CLQ = ˜ O ( 1 / ε ) as ε → 0 + . The rationale is that stabilizing the system with unknown parameters is more difficult when the traffic slackness is lower as an efficient algorithm must strive to learn parameters more accurately. We establish below that the classical upper confidence bound policy (UCB, see Algorithm 1) achieves near-optimal CLQ = ˜ O ( K ε ) for any K and ε in the single-queue multi-server setting with no prior information of any system parameters.

The proof of the theorem (Section 4) bridges Lyapunov and bandit analysis, makes an interesting connection to satisficing regret , and is a main technical contribution of our work.

Wenext extend our approach to the queueing network setting. Fixing the system structure A , Σ , B , D , we define M A = max A ∈ A ∑ n ∈N A n and M Σ , B = max σ ∈ Σ ∑ n ∈N ∑ k ∈B n σ k , to be the maximum number of new job arrivals and the maximum number of selected servers per period. Further, for queueing networks, we also define the quantity M D = ∑ k ∈K |D k | 2 , related to the number of queues each server may see its jobs transition to. The following result shows that the worst-case cost of learning of our algorithm BACKPRESSURE-UCB (BP-UCB as a short-hand, [12, Algorithm 3]) has optimal dependence on 1 ε . The proof of this Theorem is provided in [12, Section 6].

Theorem 3. For any A , Σ , B , D and traffic slackness ε ∈ (0 , 1] , we have

Our proof builds on the special case of multi-queue multi-server systems, for which we provide Algorithm MAXWEIGHT-UCB with a corresponding performance guarantee (see [12, Section 5]).

4 Proof for the single-queue multi-server setting (Theorem 2)

In this section, we bound the CLQ of UCB for the single-queue multi-server setting (Theorem 2). In each period t , when the queue is non-empty, UCB selects a server with the highest upper confidence bound estimation ¯ µ k ( t ) = min ( 1 , ˆ µ k ( t ) + √ 2 ln( t ) C k ( t ) ) where ˆ µ k ( t ) is the sample mean of services and C k ( t ) is the number of times server k is selected in the first t -1 periods.

̸

We establish a framework to upper bound CLQ single for any policy by considering separately the initial learning stage and the later regenerate stage. The two stages are separated by a parameter T 1 that appears in our analysis: intuitively, during the learning stage ( t < T 1 ), the loss in total service of a policy compared with the optimal server's outweighs the slackness ε of the system (Definition 1), i.e., ∑ t τ =1 µ ⋆ -µ J ( τ ) > tε and thus the queue length grows linearly with respect to the left-hand side. After the learning stage ( t > T 1 ), when ∑ t τ =1 µ ⋆ -µ J ( τ ) < tε , the queue regenerates to a constant length independent of t . To prove the ˜ O ( K ε ) bound on CLQ single , we couple the queue with an 'auxiliary' queue where the DM always chooses a nearly optimal server in the learning stage. Then we utilize a Lyapunov analysis to bound the queue length during the regenerate stage.

The framework establishes a connection between CLQ single ( λ, µ , π ) and the satisficing regret defined as follows. For any horizon T , the satisficing regret SaR single ( π, T ) is the total service rate gap between the optimal server and the server selected by π except for the periods where the gap is less than ε 2 or the queue length is zero. That is, the selected server is satisficing as long as its service rate is nearly optimal or the queue is empty. To formally define it, we denote max( x, 0) by x + and define the satisficing regret of a policy π over the first T periods by

We use the satisficing regret notation because our motivation is similar to that in multi-armed bandits [32], initially considered for an infinite horizon and a Bayesian setting. In multi-armed bandits, optimal bounds on regret ∑ T t =1 ( µ ⋆ -µ J ( t ) ) are either instance-dependent O ( ∑ k : µ k <µ ⋆ ( ln T µ ⋆ -µ k )) [4] or instance-independent O ( √ KT ) [5]. However, both are futile to establish a ˜ O ( K ε ) bound for CLQ single ( K,ε ) : The first bound depends on the minimum gap (which can be infinitesimal), whereas the second is insufficient as we explain in the discussion after Lemma 4.2.

We circumvent these obstacles by connecting the time-averaged queue length of the system with the satisficing regret of the policy via Lemma 4.1 (for the learning stage) and Lemma 4.2 (for the regenerate stage). Lemma 4.1 explicitly bounds the expected queue length through the expected satisficing regret; this is useful during the learning stage but does not give a strong bound for the regenerate stage. Lemma 4.2 gives a bound that depends on SaR single ( π,T ) 2 T , and is particularly useful during the latter regenerate stage. We then show that the satisficing regret of UCB is O ( K ln T ε ) (Lemma 4.3). Combining these results, we establish a tight bound for the cost of learning of UCB.

Formally, Lemma 4.1 shows that the expected queue length under π in period t is at most that under a nearly optimal policy plus the expected satisficing regret up to that time.

Lemma 4.1 is established by coupling the queue with an auxiliary queue that always selects a nearly optimal server. However, it cannot provide a useful bound on the cost of learning. For large T , it is known that E [ SaR single ( π, T ) ] must grow with a rate of at least O ( log T ε ) [25]. Hence, Lemma 4.1 only meaningfully bounds the queue length for small T (learning stage). For large T (regenerate stage), we instead have the following bound (Lemma 4.2).

Lemma 4.2. For any policy π and horizon T , we have ∑ T t =1 E [ Q ( t )] T ≤ 4 ε + 8 ε 2 · E [ SaR single ( π,T ) 2 ] T .

This lemma shows that the impact of learning, reflected by E [ SaR single ( π, T ) 2 ] , decays at a rate of 1 T . Therefore, as long as SaR single ( π, T ) 2 is of a smaller order than T , the impact of learning eventually disappears. This also explains why the instance-independent O ( √ KT ) regret bound for multi-armed bandits is insufficient for our analysis: the second moment of the regret scales linearly with the horizon and does not allow us to show a decreasing impact of learning on queue lengths.

Lemma 4.2 suffices to show stability ( lim T →∞ ∑ T t =1 E [ Q ( t )] T < ∞ ), but gives a suboptimal bound for small T . Specifically, when SaR single ( π, T ) 2 ⪆ T , this bound is of the order of Ω( 1 ε 2 ) . 5 We thus need both Lemma 4.1 and Lemma 4.2 to establish a tight bound on the cost of learning in queues.

The following result bounds the first and second moments of the satisficing regret of UCB.

Lemma 4.3. For any horizon T , we have

We next offer a proof sketch of Theorem 2. The theorem and lemmas are proven in [12, Section 4].

Proof sketch of Theorem 2. Fix any pair of λ, µ with max k ∈K µ k = λ + ε . Let T 1 = ⌊ ( 2 12 K 2 / ε 4 ) 2 ⌋ . We bound CLQ single ( λ, µ , UCB ) by considering T ≤ T 1 (learning stage) and T ≥ T 1 (regenerate stage) separately. For T ≤ T 1 , we have

where we use Lemmas 4.1 and 4.3 (i). For T > T 1 , we have

where we use Lemmas 4.2 and 4.3 (ii).

Remark 2. Although the CLQ metric is focused on the entire horizon, our analysis extends to bounding the maximum expected time-averaged queue lengths in the later horizon, which is formalized as max T ≥ T 1 ∑ t ≤ T E [ Q ( t )] T for any T 1 . In particular, for T 1 ≥ ( 2 12 K 2 ε 4 ) 2 , Lemma 4.2 shows that max T ≥ T 1 ∑ t ≤ T E [ Q ( t )] T ≤ 5 ε ; UCB thus enjoys the optimal asymptotic queue length scaling of O ( 1 ε ) .

5 Conclusions

Motivated by the observation that queueing regret does not capture the complexity of learning which tends to occur in the initial stages, we propose an alternative metric (CLQ) to encapsulate the statistical complexity of learning in queueing systems. For a single-queue multi-server system with K servers and a traffic slackness ε , we derive a lower bound Ω( K ε ) on CLQ, thus establishing that learning incurs a non-negligible increase in queue lengths. We then show that the classical UCB algorithm has a matching upper bound of ˜ O ( K ε ) . Finally, we extend our result to multi-queue multi-sever systems and general queueing networks by providing algorithms, MAXWEIGHT-UCB and BACKPRESSURE-UCB, whose CLQ has a near optimal ˜ O (1 /ε ) dependence on traffic slackness.

Having introduced a metric that captures the complexity of learning in queueing systems, our work can serve as a starting point for interesting extensions that can help shed further light on the area. In particular, future research may focus on beyond worst case guarantees for CLQ, non-stationary settings, improved bounds using contextual information, etc.

References

Saghar Adler and Vijay Subramanian. Bayesian learning of optimal policies in markov decision processes with countably infinite state-space. arXiv preprint arXiv:2306.02574 , 2023.

Saed Alizamir, Francis De Véricourt, and Peng Sun. Diagnostic accuracy under congestion. Management Science , 59(1):157-171, 2013.

Mor Armony, Shlomo Israelit, Avishai Mandelbaum, Yariv N Marmor, Yulia Tseytlin, and Galit B Yom-Tov. On patient flow in hospitals: A data-based queueing-science perspective. Stochastic systems , 5(1):146-194, 2015.

Peter Auer, Nicolò Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Mach. Learn. , 47(2-3):235-256, 2002.

Peter Auer, Nicolò Cesa-Bianchi, Yoav Freund, and Robert E. Schapire. The nonstochastic multiarmed bandit problem. SIAM J. Comput. , 32(1):48-77, 2002.

Kostas Bimpikis and Mihalis G Markakis. Learning and hierarchies in service systems. Management Science , 65(3):1268-1285, 2019.

Maury Bramson, Bernardo D'Auria, and Neil Walton. Stability and instability of the maxweight policy. Math. Oper. Res. , 46(4):1611-1638, 2021.

Xinyun Chen, Yunan Liu, and Guiyu Hong. An online learning approach to dynamic pricing and capacity sizing in service systems. Operations Research , 2023.

Tuhinangshu Choudhury, Gauri Joshi, Weina Wang, and Sanjay Shakkottai. Job dispatching policies for queueing systems with unknown service rates. In Proceedings of the Twenty-second International Symposium on Theory, Algorithmic Foundations, and Protocol Design for Mobile Networks and Mobile Computing , pages 181-190, 2021.

Jim G. Dai and Wuqin Lin. Maximum pressure policies in stochastic processing networks. Oper. Res. , 53(2):197-218, 2005.

Daniel Freund, Thodoris Lykouris, and Wentao Weng. Efficient decentralized multi-agent learning in asymmetric bipartite queueing systems. Operations Research , 2023.

Daniel Freund, Thodoris Lykouris, and Wentao Weng. Quantifying the cost of learning in queueing systems. arXiv preprint arXiv:2308.07817v2 , 2023.

Hu Fu, Qun Hu, and Jia'nan Lin. Stability of decentralized queueing networks beyond complete bipartite cases. In Kristoffer Arnsfelt Hansen, Tracy Xiao Liu, and Azarakhsh Malekian, editors, Web and Internet Economics - 18th International Conference, WINE 2022, Troy, NY, USA, December 12-15, 2022, Proceedings , volume 13778 of Lecture Notes in Computer Science , pages 96-114. Springer, 2022.

Xinzhe Fu and Eytan Modiano. Optimal routing to parallel servers with unknown utilities-multi-armed bandit with queues. IEEE/ACM Transactions on Networking , 2022.

Jason Gaitonde and Éva Tardos. The price of anarchy of strategic queuing systems. Journal of the ACM , 2023.

Noah Gans, Ger Koole, and Avishai Mandelbaum. Telephone call centers: Tutorial, review, and research prospects. Manuf. Serv. Oper. Manag. , 5(2):79-141, 2003.

Mor Harchol-Balter. Performance modeling and design of computer systems: queueing theory in action . Cambridge University Press, 2013.

Refael Hassin and Moshe Haviv. To queue or not to queue: Equilibrium behavior in queueing systems , volume 59. Springer Science & Business Media, 2003.

Elad Hazan and C. Seshadhri. Efficient learning algorithms for changing environments. In Andrea Pohoreckyj Danyluk, Léon Bottou, and Michael L. Littman, editors, Proceedings of the 26th Annual International Conference on Machine Learning, ICML 2009, Montreal, Quebec, Canada, June 14-18, 2009 , volume 382 of ACM International Conference Proceeding Series , pages 393-400. ACM, 2009.

Wei-Kang Hsu, Jiaming Xu, Xiaojun Lin, and Mark R Bell. Integrated online learning and adaptive control in queueing systems with uncertain payoffs. Operations Research , 70(2):11661181, 2022.

Jiatai Huang, Leana Golubchik, and Longbo Huang. Queue scheduling with adversarial bandit learning. arXiv preprint arXiv:2303.01745 , 2023.

Ramesh Johari, Vijay Kamble, and Yash Kanoria. Matching while learning. Operations Research , 69(2):655-681, 2021.

Subhashini Krishnasamy, Ari Arapostathis, Ramesh Johari, and Sanjay Shakkottai. On learning the c µ rule in single and parallel server networks. arXiv preprint arXiv:1802.06723 , 2018.

Subhashini Krishnasamy, Rajat Sen, Ramesh Johari, and Sanjay Shakkottai. Learning unknown service rates in queues: A multiarmed bandit approach. Oper. Res. , 69(1):315-330, 2021.

TL Lai and Herbert Robbins. Asymptotically efficient adaptive allocation rules. Advances in Applied Mathematics , 6(1):4-22, 1985.

Qingkai Liang and Eytan Modiano. Minimizing queue length regret under adversarial network models. Proceedings of the ACM on Measurement and Analysis of Computing Systems , 2(1):132, 2018.

John DC Little. A proof for the queuing formula: L= λ w. Operations research , 9(3):383-387, 1961.

Rahul Makhijani, Parikshit Shah, Vashist Avadhanula, Caner Gocmen, Nicolás E. Stier Moses, and Julián Mestre. QUEST: queue simulation for content moderation at scale. CoRR , abs/2103.16816, 2021.

Laurent Massoulié and Kuang Xu. On the capacity of information processing systems. Operations Research , 66(2):568-586, 2018.

Michael J. Neely, Scott Rager, and Thomas F. La Porta. Max weight learning algorithms for scheduling in unknown environments. IEEE Trans. Autom. Control. , 57(5):1179-1191, 2012.

Quang Minh Nguyen and Eytan Modiano. Learning to schedule in non-stationary wireless networks with unknown statistics. arXiv preprint arXiv:2308.02734 , 2023.

Daniel Russo and Benjamin Van Roy. Satisficing in time-sensitive bandit learning. Mathematics of Operations Research , 47(4):2815-2839, 2022.

Flore Sentenac, Etienne Boursier, and Vianney Perchet. Decentralized learning in online queuing systems. pages 18501-18512, 2021.

Virag Shah, Lennart Gulikers, Laurent Massoulié, and Milan Vojnovic. Adaptive matching for expert systems with uncertain task types. Oper. Res. , 68(5):1403-1424, 2020.

Simrita Singh, Itai Gurvich, and Jan A Van Mieghem. Feature-based priority queuing. Available at SSRN 3731865 , 2022.

R. Srikant and Lei Ying. Communication networks: an optimization, control, and stochastic networks perspective . Cambridge University Press, 2014.

Thomas Stahlbuhk, Brooke Shrader, and Eytan H. Modiano. Learning algorithms for scheduling in wireless networks with unknown channel statistics. Ad Hoc Networks , 85:131-144, 2019.

Thomas Stahlbuhk, Brooke Shrader, and Eytan H. Modiano. Learning algorithms for minimizing queue length regret. IEEE Trans. Inf. Theory , 67(3):1759-1781, 2021.

Leandros Tassiulas. Stability properties of constrained queueing systems and scheduling policies for maximum throughput in multihop radio networks. IEEE TRANSACTIONS ON AUTOMATIC CONTROL , 31(12), 1992.

Neil Walton and Kuang Xu. Learning and information in stochastic networks and queues. In Tutorials in Operations Research: Emerging Optimization Methods and Modeling Techniques with Applications , pages 161-198. INFORMS, 2021.

Neil S. Walton. Two queues with non-stochastic arrivals. Oper. Res. Lett. , 42(1):53-57, 2014.

Zixian Yang, R Srikant, and Lei Ying. Learning while scheduling in multi-server systems with unknown statistics: Maxweight with discounted ucb. In International Conference on Artificial Intelligence and Statistics , pages 4275-4312. PMLR, 2023.

Yueyang Zhong, John R Birge, and Amy Ward. Learning the scheduling policy in time-varying multiclass many server queues with abandonment. Available at SSRN , 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors introduce a new regret metric, called Cost of Learning in Queueing (CLQ), to quantify the rate at which an optimal scheduling policy can be learned to minimize the time average queue lengths. The authors derive a lower bound to CLQ and show that an UCB-based policy comes close to achieving the lower bound. They also extend their policy to multiple queues in a network setting by combining the UCB policy with the celebrated Backpressure policy.\n\nSTRENGTHS:\n1. The dependence of the regret bound with respect to the slack parameter (epsilon) is optimal.\n2. Bounding the CLQ metric in terms of satisficing regret is intriguing.\n\nWEAKNESSES:\n1.\tThe paper [34] considers an unnormalized version of the same metric proposed in this paper. However, the results presented in this paper give a potentially weaker bound (linearly increasing) than the result in [34]. To be more specific, Theorem 2 proves a constant (let’s denote it by $c$) upper bound to the CLQ metric. Plugging this upper bound into Eq (3) yields the following linear bound on the queue length regret:\n\n   $\\sum_{t=1}^T \\mathbb{E}\\left[(Q(t, \\pi)) -  Q(t, \\pi^*) \\right]\\leq cT, ~ \\forall T \\geq 1 $\n\nBut it is already known from [34, Theorem 2, 3] that there exist simple dynamic policies under which the queue-length regret (i.e., the LHS of the above equation) can be bounded by a constant. Clearly, the CLQ metric fails to capture this strong result and paints an overly pessimistic picture. It is also not clear if the result presented in this paper strictly and quantitatively improves upon [34], even for smaller horizon lengths. \n\n2.\tThe proposed algorithm directly uses UCB, and its analysis does not present any new technical insights.\n\nQUESTIONS:\n1.\tPlease respond to Comment 1 above by arguing why Theorem 2 gives a stronger bound than [34].\n2.\tSince the proposed UCB-based policy directly estimates the mean service rate, it might not work in the non-stationary setting, although the classic Max-weight (or Backpressure) might be able to stabilize the queues. Can the authors shed light on this aspect?\n\nLIMITATIONS:\nThis is a theoretical work and does not seem to have any potentially negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose a new metric to quantify the cost of learning in queueing networks. This notion is required to capture the differences between holding costs in queues and, say costs accumulated in a bandit setting; the latter having a monotonicity property (in expectation). The authors then derive a worst-case lower bound for this metric, and propose UCB based algorithms that come `close' to the lower bound in the order sense.\n\nSTRENGTHS:\n1. The CLQ metric proposed here is meaningful, and quite appropriate for queueing systems. Prior work uses a standard regret metric, which may not be very appropriate given that queue tend to regenerate.\n\n2. The UCB-based analysis appears quite novel; the authors have to bound the CLQ differently in the initial learning phase and subsequently; this issue does not arise in standard UCB analyses for bandits.\n\n3. The analysis extends to queueing networks.\n\nWEAKNESSES:\nNo significant weaknesses here. I would have liked to see some more exposition in certain places, including a description of the algorithms, and a comparison between the lower and upper bounds derived here. But I can see that the authors have done the best they could to tell the story within the space constraints.\n\nI do have some (minor) suggestions though.\n\n1. I think it is worth highlighting around Theorem 1 that the bound derived is not instance-dependent, but worst-case in nature.\n\n2. The following sentence on Line 270 on Page 7 \"Therefore, in expectation, the queue under .... never empties.\" was unclear to me.\n\nQUESTIONS:\nThis is not a serious issue for me, but I did not get what the authors meant in saying that the guarantee on time averaged queue lengths translates to one on wait times via Little's Law. This statement seems somewhat vague. Can CLQ be formally be related to a sub-optimality on average wait times?\n\nLIMITATIONS:\nNot applicable.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies a problem that involves both learning with queueing.\nIn the simple setting, there is a single queue served by multiple\nservers. However, the service rate at each server is unknown and needs\nto be learned. Intuitively, the combination of the learning policy and\nthe scheduling policy will impact the queue length dynamics of the\nsystem.  Prior work mostly focuses on the queue length performance in\nthe late stage of the system. However, due to the nature of queueing\nsystems, this late-stage performance is relatively invariant to the\nlearning policy, and thus the late-stage metric does not adequately\ncapture the impact of learning. In contrast, the first contribution of\nthis paper is to propose a new metric, called CLQ (Cost-of-Learning).\nCLQ takes the difference between the time-averaged queue length of a\ngiven policy and that of the optimal, and then takes the maximum over time. \nThis maximization over the entire time-horizon allows CLQ to capture the\nearly-stage dynamics of the system, where the impact of learning is more\nobvious.  Then, the authors provide both lower bounds and achievable\nbounds for this new metrics, which differ only by a logarithmic factor\nin the number of servers.  Finally, the results are also extended to\nmore general multi-queue multi-server systems.\n\nSTRENGTHS:\n1. The new notion of CLQ captures the impact of learning more accurately\nthan existing studies, by including the early-stage dynamics of the\nqueue. This is a significant contribution. \n\n2. The lower-bound and achievable-bound are nice and differ only by a\nlogarithmic factor.\n\n3. The results are extended to general multi-queue multi-server systems. \n\n4. The proof idea based on satisfying regret is also very interesting.\n\nWEAKNESSES:\nI don't find major weaknesses.\n\nQUESTIONS:\n1. The work in [39] also provides any-time queueing bounds. I wonder if\nthey can be translated into an achieveable result for CLQ. Can the\nauthors comment on how the any-time bound from [39] may compare with the\nachieveable bound in this paper?\n\n2. The lower bound involves a very large $K > 2^14 \\approx 16000$. Can\nthe authors comment on what will happen when $K$ is not this large? \n\nPost rebuttal phase:\n\nThe reviewer wishes to thank the authors for their response, which clarifies the relation to [39].\n\nLIMITATIONS:\nI do not find discussions on limitations or potential negative societal\nimpact.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider online queuing systems in a discrete time setting. They study settings with single class queue and multi-class queues, and they propose to consider a metric CLQ that serves as a conservative measure on how the queue length(s) could grow across every time point in a horizon. The authors propose a natural UCB algorithm, and demonstrate that it has a near optimal CLQ in a single queue setting. The authors also derive similar bounds in a queueing network setting.\n\nSTRENGTHS:\n- The analysis is quite novel. In addition, the definition of the CLQ and its analysis is new to the best of my knowledge. In particular, the consideration of the satisficing regret is quite interesting. \n- The authors achieve a nearly tight result in the case of one queue. Overall, the technical results are solid.\n- The notion of CLQ seems adequate as an alternate metric for online queueing systems, but I still have some question (see Question 2 in ``Question'')\n\nWEAKNESSES:\n1. The discussions on queueing networks could be benefited from more details. While the authors stated that they will provide additional examples on settings modeled by their queueing networks model in Appendix A, Appendix A does not seem to contain much details. To overcome this weakness, the authors should provide a detail account on how \n \n  - the model on the bipartite queueing system in [11] (Line 187), \n  - the model on the multi-server system in [39] (Line 190),\n\nare modeled by the queueing network formulation used by the authors. In particular, it will be crucial for the authors to highlight what are the individual element in the instance tuple (Line 170) in these models. While there could be quite a fair bit of details, I believe that the authors could include them in the Appendix A so that useful details are provided without violating the page limit.\n\n2. There is no simulation to showcase the results. \n\n3. The notion of CLQ seems not to tell us the long term behavior of a policy, since it is taking a worst case over all horizon lengths $1, 2, ....$ (See Question 2)\n\nQUESTIONS:\n- Can I confirm if $\\{B_n\\}_{n\\in{\\cal N}}$ is a partition on ${\\cal K}$, since the authors say that each server in $B_n$ belongs to one queue?\n\n- While I find the CLQ notion interesting, I still find that it might not have told us the whole story on who the queue length fluctuate under a policy. Let's stick with the single queue model for our discussion sake. For example, let say we have a policy $\\pi$ that achieve a CLQ of $\\text{constant} \\times K/\\epsilon$, which is optimal, within a constant factor. Knowing that $\\mathbf{E}[Q(t, \\pi^*)] = O(1/\\epsilon)$ for a large $t$ (I think you can show this by a similar logic to how we derive the average queue length of a stable M/M/1), achieving an optimal CLQ stated above actually does not prevent us from having $\\mathbf{E}[Q(t, \\pi)] = O(K/\\epsilon)$ for all sufficiently large $t$. In this way, achieving an optimal CLQ does not necessarily mean that we converge to the optimal queue length when $t$ grows. \n\nA priori, this might not be a critical issue since the authors aspire to investigate a short time horizon regime and not only a long one, and IMHO I regard CLQ as a conservative measure that tells us how long the queue length could be over all possible horizon lengths.  But I am wondering if the authors could also consider a worst case over the horizon lengths in $\\{\\tau, \\tau+1, \\ldots \\}$ instead of $\\{1, 2, \\ldots \\}$, for an arbitrary $\\tau$? More precisely, do the authors' analysis inform us of a bound on (3) with the range of maximum $\\{1, 2, \\ldots \\}$ replaced with $\\{\\tau, \\tau+1, \\ldots \\}$ for and arbitrary $\\tau$?\n\n- What is the difference between the two definitions (5), (6)?\n\nLIMITATIONS:\nThere is no potential societal impact to my best knowledge. This is a theoretical paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-16"}