SlowGuess/NeurIPS_2023_OCR huggingface dataset:
and the DM's decision in period  $t$  is to select a schedule  $\sigma(t) \in \Sigma_t$ . Following [39], we assume that any subset of a feasible schedule is still feasible, i.e., if  $\sigma \in \Sigma$  and  $\sigma_k' \leq \sigma_k \forall k \in \mathcal{K}$ , then  $\sigma' \in \Sigma$ .
We now formalize the arrival and service dynamics in every period, which are captured by the independent random variables  $\{\pmb{A}(t),\{S_k(t)\}_{k\in \mathcal{K}}\}_t$ . The arrival vector  $\pmb{A}(t) = \{A_{n}(t)\}_{n\in \mathcal{N}}$  consists of (possibly correlated) random variables  $A_{n}(t)$  taking value in  $\mathcal{A}\subseteq \{0,1\}^{\mathcal{N}}$ ; we denote its distribution by  $\Lambda$  and let  $\mathbb{E}[A_n(t)] = \lambda_n(\Lambda)$  with  $\lambda = (\lambda_{n}(\Lambda))_{n\in \mathcal{N}}$ . The service  $S_{k}(t)$  for each server  $k\in \mathcal{K}$  is a Bernoulli random variable indicating whether the selected service request was successful. To formalize the job transition, let  $L_{k}(t) = (L_{k,n}(t))_{n\in \mathcal{N}\cup \{\bot \}}$  be a random vector over  $\{0,1\}^{\mathcal{N}\cup \{\bot \}}$  for server  $k$  independent of other randomness such that  $\mathbb{P}\{L_{k,n}(t) = 1\} = p_{k,n}$  and  $\sum_{n\in \mathcal{N}\cup \{\bot \}}L_{k,n}(t) = 1$ . The queueing dynamic is given by
$$
Q _ {n} (t + 1) = Q _ {n} (t) - \sum_ {k \in \mathcal {B} _ {n}} \sigma_ {k} (t) S _ {k} (t) + A _ {n} (t) + \sum_ {k ^ {\prime} \in \mathcal {K}} \sigma_ {k ^ {\prime}} (t) S _ {k ^ {\prime}} (t) L _ {k ^ {\prime}, n} (t). \tag {1}
$$
We assume that the DM has knowledge of which policies are allowed, i.e., they know  $\pmb{\Sigma}$  and  $\pmb{B}$ , but has no prior knowledge of the rates  $\lambda, \mu, P$  and the set  $\pmb{D}$ . In period  $t$ , the observed history is the set  $\left(\{A_{n}(\tau)\}_{n \in \mathcal{N}}, \{S_{k}(\tau)L_{k,n}(\tau)\}_{n \in \bar{\mathcal{N}}, k \in \mathcal{K}}: \sigma_{k}(\tau) = 1\right)_{\tau < t}$  that includes transition information on top of arrivals and services. Note that a job transition is only observed when the server is selected and the service is successful. Similar to before, a non-anticipatory policy  $\pi$  maps an observed history to a feasible schedule; we let  $Q_{n}(t, \pi)$  be the length of queue  $n$  in period  $t$  under this policy.
Unlike the single-queue case, it is usually difficult to find the optimal policy for a queueing network even with known system parameters. Fortunately, if the system is stabilizable, i.e.,  $\lim_{t\to \infty}\frac{1}{T}\sum_{t\leq T}\mathbb{E}[\| Q(t)\| _1] < \infty$  under some scheduling policy, then the arrival rate vector must be within the capacity region of the servers [39]. Formally, let  $\Phi = \{\phi \in [0,1]^{\Sigma}\colon \sum_{\sigma \in \Sigma}\phi_{\sigma} = 1\}$  be the probability simplex over  $\pmb{\Sigma}$ . A distribution  $\phi$  in  $\Phi$  can be viewed as the frequency of a policy using each schedule  $\sigma \in \Sigma$ , and the effective service rate queue  $n$  can get is given by  $\mu_n^{\mathrm{net}}(\phi) = \sum_{\sigma \in \Sigma}\phi_\sigma \left(\sum_{k\in \mathcal{B}_n}\sigma_k\mu_k - \sum_{k'\in \mathcal{K}}\sigma_{k'}\mu_{k'}p_{k',n}\right)$ ; this includes both job inflow and outflow. We denote the effective service rate vector for a schedule distribution  $\phi$  by  $\mu^{\mathrm{net}}(\phi)$ . Then, the capacity region is  $S(\mu ,\Sigma ,\mathcal{B},P) = \{\mu^{\mathrm{net}}(\phi)\colon \phi \in \Phi \}$ . For a queueing network to be stabilizable, we must have  $\lambda (\Lambda)\in S(\mu ,\Sigma ,\mathcal{B},P)$  [39]. As in the single-queue case, we further assume that the system has a positive traffic slackness and let 1 denote a vector of 1s with suitable dimension.
Definition 2. A queueing network has traffic slackness  $\varepsilon \in (0,1]$  if  $\lambda (\Lambda) + \varepsilon \mathbf{1}\in S(\pmb {\mu},\pmb {\Sigma},\pmb {\mathcal{B}},\pmb {P})$
We also study a special case of queueing networks, multi-queue multi-server systems, where jobs immediately leave after a successful service, i.e.,  $\mathcal{D}_k = \{\bot\}$  for all  $k \in \mathcal{K}$ ; this extends the models in [11, 42] mentioned above. Since the transition probability matrix  $\pmb{P}$  is trivial  $(p_{k,\perp} = 1, \forall k)$ , we denote the capacity region of a multi-queue multi-server system by  $\mathcal{S}(\boldsymbol{\mu}, \boldsymbol{\Sigma}, \boldsymbol{\mathcal{B}})$ .
3Though this reflects the feature from the single-queue setting, that  $Q(t) = 0 \Rightarrow J(t) = \bot$ , it maintains the flexibility to have a queue that has multiple jobs served in a single period.
This formulation captures settings where arrivals are independent of the history, such as the example of each queue having independent arrivals (e.g., the bipartite queueing model in [11]) and the example of feature-based queues [35] where jobs have features; each type of feature has one queue; at most one job arrives among all queues in each period. Our formulation cannot capture state-dependent arrivals such as queues with balking [18].
3 Main results: the statistical complexity of learning in queueing
This section presents our main results on the statistical complexity of learning in queueing systems. We first define the Cost of Learning in Queueing, or CLQ as a shorthand, a metric capturing this complexity and provide a lower bound for the single-queue multi-server setting. Motivated by this, we design an efficient algorithm for the single-queue multi-server setting with a matching CLQ and then extend this to the multi-queue multi-server and queueing network systems.
3.1 Cost of Learning in Queueing
We first consider learning in the single-queue multi-server setting. Previous works on learning in queueing systems focus on the queueing regret  $\mathbb{E}\left[Q(T,\pi) - Q(T,\pi^{\star})\right]$  in the asymptotic regime of  $T\to \infty$ . The starting point of our work stems from the observation that an asymptotic metric, which measures performance in late periods, cannot capture the complexity of learning as learning happens in initial periods (recall the left of Figure 1). In addition, a guarantee on per-period queue length cannot easily translate to the service experience (or wait time) of jobs.
Motivated by the above insufficiency of queueing regret, we define the Cost of Learning in Queueing (or CLQ) as the maximum increase in expected time-averaged queue lengths under policy  $\pi$  compared with the optimal policy. Specifically, we define the single-queue CLQ as:
$$
\operatorname {C L Q} ^ {\text {s i n g l e}} (\lambda , \boldsymbol {\mu}, \pi) = \max  _ {T \geq 1} \frac {\sum_ {t = 1} ^ {T} \mathbb {E} [ Q (t , \pi) - Q (t , \pi^ {\star}) ]}{T}. \tag {2}
$$
As shown in Figure 1 (right), CLQ is a finite-time metric and explicitly takes into account how fast learning occurs in the initial periods. In addition, a bound on the maximum increase in time-averaged queue length translates approximately (via Little's Law [27]) to a bound on the increase in average job wait times.
Given that the traffic slackness measures the difficulty of stabilizing a system, we also consider the worst-case cost of learning in queueing over all pairs of  $(\lambda, \mu)$  with a fixed traffic slackness  $\varepsilon$ . In a slight abuse of notation, we overload  $\mathrm{CLQ}^{\mathrm{single}}$  to also denote this worst-case value, i.e.,
$$
\operatorname {C L Q} ^ {\text {s i n g l e}} (K, \varepsilon , \pi) = \sup  _ {\lambda \in [ 0, 1), \boldsymbol {\mu} \in [ 0, 1 ] ^ {K}: \lambda + \varepsilon \leq \max  _ {k} \mu_ {k}} \operatorname {C L Q} ^ {\text {s i n g l e}} (\lambda , \boldsymbol {\mu}, \pi). \tag {3}
$$
Our goal is to design a policy  $\pi$ , without knowledge of the arrival rate, the service rates, and the traffic slackness, that achieves low worst-case cost of learning in a single-queue multi-server system.
We can extend the definition of CLQ to the multi-queue multi-server and the queueing network settings. Since the optimal policy is difficult to design, we instead define CLQ for a policy  $\pi$  by comparing it with any non-anticipatory policy (which makes decisions only based on the history):
$$
\operatorname {C L Q} ^ {\text {m u l t i}} (\boldsymbol {\Lambda}, \boldsymbol {\mu}, \boldsymbol {\Sigma}, \boldsymbol {\mathcal {B}}, \pi) = \max  _ {\text {n o n - a n t i c i p a t o r y}} \max  _ {\pi^ {\prime}} \max  _ {T \geq 1} \frac {\sum_ {t = 1} ^ {T} \sum_ {n \in \mathcal {N}} \mathbb {E} \left[ Q _ {n} (t , \pi) - Q _ {n} \left(t , \pi^ {\prime}\right) \right]}{T}. \tag {4}
$$
$$
\operatorname {C L Q} ^ {\text {n e t}} (\boldsymbol {\Lambda}, \boldsymbol {\mu}, \boldsymbol {\Sigma}, \boldsymbol {\mathcal {B}}, \boldsymbol {\mathcal {D}}, \boldsymbol {P}, \pi) = \max  _ {\text {n o n - a n t i c i p a t o r y}} \max  _ {\pi^ {\prime}} \max  _ {T \geq 1} \frac {\sum_ {t = 1} ^ {T} \sum_ {n \in \mathcal {N}} \mathbb {E} \left[ Q _ {n} (t , \pi) - Q _ {n} \left(t , \pi^ {\prime}\right) \right]}{T} \tag {5}
$$
As in the single-queue setting, we can define the worst-case cost of learning for a fixed structure  $\mathcal{A}, \Sigma, \mathcal{B}, \mathcal{D}$  and a traffic slackness  $\varepsilon$  as the supremum across all arrival, service, and transition rates with this traffic slackness. With the same slight abuse of notation as before, we denote these quantities by  $\mathrm{CLQ}^{\mathrm{multi}}(\mathcal{A}, \Sigma, \mathcal{B}, \varepsilon, \pi)$  and  $\mathrm{CLQ}^{\mathrm{net}}(\mathcal{A}, \Sigma, \mathcal{B}, \mathcal{D}, \varepsilon, \pi)$ .
3.2 Lower bound on the cost of learning in queueing
Our first result establishes a lower bound on  $\mathrm{CLQ}^{\mathrm{single}}(K,\varepsilon ,\pi)$ . In particular, for any feasible policy  $\pi$ , we show a lower bound of  $\Omega \left(\frac{K}{\varepsilon}\right)$  for sufficiently large  $K$ . With known parameters, the optimal time-averaged queue length is of the order of  $1 / \varepsilon$ . Hence, our result shows that the cost of learning is non-negligible in queueing systems when there are many servers. For fixed  $K$  and  $\varepsilon$ , our lower bound considers the cost of learning of the worst-case setting and is instance-independent.
Theorem 1. For  $K \geq 2^{14}, \varepsilon \in (0, 0.25]$  and feasible policy  $\pi$ , we have  $\mathrm{CLQ}^{\mathrm{single}}(K, \varepsilon, \pi) \geq \frac{K}{2^{14}\varepsilon}$ .
Although our proof is based on the distribution-free lower bound  $\Omega(\sqrt{KT})$  for classical multi-armed bandits [5], this result does not apply directly to our setting. In particular, suppose the queue in our system is never empty. Then the accumulated loss in service of a policy is exactly the regret in bandits and the lower bound implies that any feasible policy serves at least  $\Omega(\sqrt{KT})$  jobs fewer than the optimal policy in the first  $T$  periods. However, due to the traffic slackness, the queue does get empty under the optimal policy, and in periods when this occurs, the optimal policy also does not receive service. As a result, the queue length of a learning policy could be lower than  $\Theta(\sqrt{KT})$  despite the loss of service compared with the optimal policy.
We next discuss the intuition (formal proof in Appendix C.1 of [12]). Fixing  $K$  and  $\varepsilon$ , suppose the gap in service rates between the optimal server and others is  $2\varepsilon$ . Then for any  $t$  in a time horizon  $T = O\left(\frac{K}{\varepsilon^2}\right)$ , the number of arrivals in the first  $t$  periods is around  $\lambda t$  and the potential service of the optimal server is around  $(\lambda + \varepsilon)t$ . By the multi-armed bandit lower bound, the total service of a policy is at most around  $\lambda t + \varepsilon t - \sqrt{K} t \leq \lambda t$  since  $t \leq T = O\left(\frac{K}{\varepsilon^2}\right)$ . Then the combined service rate of servers chosen in the first  $T$  periods, i.e.,  $\sum_{t \leq T} \mu_{J(t)}$ , is strictly bounded from above by the total arrival rate  $\lambda t$ . A carefully constructed example shows that the number of unserved jobs is around  $\varepsilon t$  for every  $t \leq T$  and thus the time-averaged queue length for the horizon  $T$  is of the order of  $\varepsilon T = O\left(\frac{K}{\varepsilon}\right)$ .
Remark 1. In [24, Proposition 3], the authors established an instance-dependent lower bound on  $\mathbb{E}[Q(t,\pi) - Q^{\star}(t)]$ . The implied CLQ lower bound is weaker than ours  $(\tilde{\Omega} (K)$ , see Appendix C.2 in [12]) and is constrained to  $\alpha$  consistent policy  $\pi$  (see definition 1 in [24]) whereas ours does not.
3.3 Upper bound on the cost of learning in queueing
Motivated by the lower bound, we propose efficient algorithms with a focus on heavy-traffic optimality, i.e., ensuring  $\mathrm{CLQ} = \tilde{O}(1 / \varepsilon)$  as  $\varepsilon \to 0^{+}$ . The rationale is that stabilizing the system with unknown parameters is more difficult when the traffic slackness is lower as an efficient algorithm must strive to learn parameters more accurately. We establish below that the classical upper confidence bound policy (UCB, see Algorithm 1) achieves near-optimal  $\mathrm{CLQ} = \tilde{O}\left(\frac{K}{\varepsilon}\right)$  for any  $K$  and  $\varepsilon$  in the single-queue multi-server setting with no prior information of any system parameters.
Theorem 2. For any  $K \geq 1, \varepsilon \in (0,1]$ ,  $\mathrm{CLQ}^{\mathrm{single}}(K,\varepsilon,\mathrm{UCB}) \leq \frac{323K + 64K(\ln K + 2\ln 1 / \varepsilon)}{\varepsilon}$ .
The proof of the theorem (Section 4) bridges Lyapunov and bandit analysis, makes an interesting connection to satisficing regret, and is a main technical contribution of our work.
We next extend our approach to the queueing network setting. Fixing the system structure  $\mathcal{A}, \Sigma, \mathcal{B}, \mathcal{D}$ , we define  $M_{\mathcal{A}} = \max_{\mathcal{A} \in \mathcal{A}} \sum_{n \in \mathcal{N}} A_n$  and  $M_{\Sigma, \mathcal{B}} = \max_{\sigma \in \Sigma} \sum_{n \in \mathcal{N}} \sum_{k \in \mathcal{B}_n} \sigma_k$ , to be the maximum number of new job arrivals and the maximum number of selected servers per period. Further, for queueing networks, we also define the quantity  $M_{\mathcal{D}} = \sum_{k \in \mathcal{K}} |\mathcal{D}_k|^2$ , related to the number of queues each server may see its jobs transition to. The following result shows that the worst-case cost of learning of our algorithm BACKPRESSURE-UCB (BP-UCB as a short-hand, [12, Algorithm 3]) has optimal dependence on  $\frac{1}{\varepsilon}$ . The proof of this Theorem is provided in [12, Section 6].
Theorem 3. For any  $\mathcal{A},\Sigma ,\mathcal{B},\mathcal{D}$  and traffic slackness  $\varepsilon \in (0,1]$ , we have
$$
\operatorname {C L Q} ^ {\text {n e t}} (\mathcal {A}, \Sigma , \mathcal {B}, \mathcal {D}, \varepsilon , \text {B P - U C B}) \leq \frac {\sqrt {N} \left(3 2 M _ {\mathcal {A}} + 2 ^ {1 2} M _ {\mathcal {D}} M _ {\Sigma , \mathcal {B}} ^ {2} \left(1 + \ln \left(M _ {\mathcal {A}} M _ {\mathcal {D}} M _ {\Sigma , \mathcal {B}} / \varepsilon\right)\right)\right)}{\varepsilon}.
$$
Our proof builds on the special case of multi-queue multi-server systems, for which we provide Algorithm MAXWEIGHT-UCB with a corresponding performance guarantee (see [12, Section 5]).
4 Proof for the single-queue multi-server setting (Theorem 2)
In this section, we bound the CLQ of UCB for the single-queue multi-server setting (Theorem 2). In each period  $t$ , when the queue is non-empty, UCB selects a server with the highest upper confidence bound estimation  $\bar{\mu}_k(t) = \min \left(1, \hat{\mu}_k(t) + \sqrt{\frac{2\ln(t)}{C_k(t)}}\right)$  where  $\hat{\mu}_k(t)$  is the sample mean of services and  $C_k(t)$  is the number of times server  $k$  is selected in the first  $t - 1$  periods.
Algorithm 1: UCB for a single-queue multi-server system
Sample mean  $\hat{\mu}_k(1)\gets 0$  , number of samples  $C_k(1)\gets 0$  for  $k\in \mathcal{K}\cup \{\bot \}$  , queue  $Q(1)\leftarrow 0$  for  $t = 1\dots$  do  $\bar{\mu}_k(t) = \min \left(1,\hat{\mu}_k(t) + \sqrt{\frac{2\ln(t)}{C_k(t)}}\right),\forall k\in \mathcal{K}$  if  $Q(t) > 0$  then  $J(t)\gets \arg \max_{k}\bar{\mu}_{k}(t)$  ; else  $J(t)\gets \bot$  /\* Update queue length & estimates based on  $S_{J(t)}(t)$ $A(t)$  , and  $J(t)$ $\ast /$ $Q(t + 1)\gets Q(t) - S_{J(t)}(t) + A(t)$ $C_{J(t)}(t + 1)\gets C_{J(t)}(t) + 1,\quad \hat{\mu}_{J(t)}(t + 1)\gets \frac{C_{J(t)}(t)\hat{\mu}_{J(t)}(t) + S_{J(t)}(t)}{C_{J(t)}(t + 1)}$  for  $k\neq J(t)$  set  $C_k(t + 1)\gets C_k(t),\quad \hat{\mu}_k(t + 1)\gets \hat{\mu}_k(t)$
We establish a framework to upper bound  $\mathrm{CLQ}^{\mathrm{single}}$  for any policy by considering separately the initial learning stage and the later regenerate stage. The two stages are separated by a parameter  $T_{1}$  that appears in our analysis: intuitively, during the learning stage ( $t < T_{1}$ ), the loss in total service of a policy compared with the optimal server's outweighs the slackness  $\varepsilon$  of the system (Definition 1), i.e.,  $\sum_{\tau=1}^{t} \mu^{\star} - \mu_{J(\tau)} > t\varepsilon$  and thus the queue length grows linearly with respect to the left-hand side. After the learning stage ( $t > T_{1}$ ), when  $\sum_{\tau=1}^{t} \mu^{\star} - \mu_{J(\tau)} < t\varepsilon$ , the queue regenerates to a constant length independent of  $t$ . To prove the  $\tilde{O}\left(\frac{K}{\varepsilon}\right)$  bound on  $\mathrm{CLQ}^{\mathrm{single}}$ , we couple the queue with an "auxiliary" queue where the DM always chooses a nearly optimal server in the learning stage. Then we utilize a Lyapunov analysis to bound the queue length during the regenerate stage.
The framework establishes a connection between  $\mathrm{CLQ}^{\mathrm{single}}(\lambda, \mu, \pi)$  and the satisficing regret defined as follows. For any horizon  $T$ , the satisficing regret  $\mathrm{SaR}^{\mathrm{single}}(\pi, T)$  is the total service rate gap between the optimal server and the server selected by  $\pi$  except for the periods where the gap is less than  $\frac{\varepsilon}{2}$  or the queue length is zero. That is, the selected server is satisficing as long as its service rate is nearly optimal or the queue is empty. To formally define it, we denote  $\max(x, 0)$  by  $x^{+}$  and define the satisficing regret of a policy  $\pi$  over the first  $T$  periods by
$$
\operatorname {S a R} ^ {\text {s i n g l e}} (\pi , T) = \sum_ {t = 1} ^ {T} \left(\mu^ {\star} - \mu_ {J (t)} - \frac {\varepsilon}{2}\right) ^ {+} \mathbb {1} (Q (t) \geq 1) \tag {6}
$$
We use the satisficing regret notation because our motivation is similar to that in multi-armed bandits [32], initially considered for an infinite horizon and a Bayesian setting. In multi-armed bandits, optimal bounds on regret  $\sum_{t=1}^{T} (\mu^{\star} - \mu_{J(t)})$  are either instance-dependent  $\mathcal{O}\left(\sum_{k: \mu_k < \mu^{\star}} \left(\frac{\ln T}{\mu^{\star} - \mu_k}\right)\right)$  [4] or instance-independent  $O(\sqrt{KT})$  [5]. However, both are futile to establish a  $\tilde{\mathcal{O}}\left(\frac{K}{\varepsilon}\right)$  bound for  $\mathrm{CLQ}^{\mathrm{single}}(K, \varepsilon)$ : The first bound depends on the minimum gap (which can be infinitesimal), whereas the second is insufficient as we explain in the discussion after Lemma 4.2.
We circumvent these obstacles by connecting the time-averaged queue length of the system with the satisficing regret of the policy via Lemma 4.1 (for the learning stage) and Lemma 4.2 (for the regenerate stage). Lemma 4.1 explicitly bounds the expected queue length through the expected satisficing regret; this is useful during the learning stage but does not give a strong bound for the regenerate stage. Lemma 4.2 gives a bound that depends on  $\frac{\mathrm{SaR}^{\mathrm{single}}(\pi,T)^2}{T}$ , and is particularly useful during the latter regenerate stage. We then show that the satisficing regret of UCB is  $O\left(\frac{K\ln T}{\varepsilon}\right)$  (Lemma 4.3). Combining these results, we establish a tight bound for the cost of learning of UCB.
Formally, Lemma 4.1 shows that the expected queue length under  $\pi$  in period  $t$  is at most that under a nearly optimal policy plus the expected satisficing regret up to that time.
Lemma 4.1. For any policy  $\pi$  and horizon  $T$ , we have  $\frac{\sum_{t=1}^{T} \mathbb{E}[Q(t)]}{T} \leq \frac{3}{\varepsilon} + \mathbb{E}\left[\mathrm{SaR}^{\mathrm{single}}(\pi, T)\right]$ .
Lemma 4.1 is established by coupling the queue with an auxiliary queue that always selects a nearly optimal server. However, it cannot provide a useful bound on the cost of learning. For large  $T$ , it is known that  $\mathbb{E}\left[\mathrm{SaR}^{\mathrm{single}}(\pi ,T)\right]$  must grow with a rate of at least  $\mathcal{O}\left(\frac{\log T}{\varepsilon}\right)$  [25]. Hence, Lemma 4.1 only meaningfully bounds the queue length for small  $T$  (learning stage). For large  $T$  (regenerate stage), we instead have the following bound (Lemma 4.2).
Lemma 4.2. For any policy  $\pi$  and horizon  $T$ , we have  $\frac{\sum_{t=1}^{T} \mathbb{E}[Q(t)]}{T} \leq \frac{4}{\varepsilon} + \frac{8}{\varepsilon^2} \cdot \frac{\mathbb{E}\left[\mathrm{SaR}^{\mathrm{single}}(\pi, T)^2\right]}{T}$ .
This lemma shows that the impact of learning, reflected by  $\mathbb{E}\left[\mathrm{SaR}^{\mathrm{single}}(\pi, T)^2\right]$ , decays at a rate of  $\frac{1}{T}$ . Therefore, as long as  $\mathrm{SaR}^{\mathrm{single}}(\pi, T)^2$  is of a smaller order than  $T$ , the impact of learning eventually disappears. This also explains why the instance-independent  $O(\sqrt{KT})$  regret bound for multi-armed bandits is insufficient for our analysis: the second moment of the regret scales linearly with the horizon and does not allow us to show a decreasing impact of learning on queue lengths.
Lemma 4.2 suffices to show stability  $(\lim_{T\to \infty}\frac{\sum_{t = 1}^{T}\mathbb{E}[Q(t)]}{T} < \infty)$ , but gives a suboptimal bound for small  $T$ . Specifically, when  $\mathrm{SaR}^{\mathrm{single}}(\pi ,T)^2\gtrsim T$ , this bound is of the order of  $\Omega (\frac{1}{\varepsilon^2})$ . We thus need both Lemma 4.1 and Lemma 4.2 to establish a tight bound on the cost of learning in queues.
The following result bounds the first and second moments of the satisficing regret of UCB.
Lemma 4.3. For any horizon  $T$ , we have
$$
(i) \mathbb {E} \left[ \operatorname {S a R} ^ {\text {s i n g l e}} (\mathbf {U C B}, T) \right] \leq \frac {1 6 K (\ln T + 2)}{\varepsilon}, (i i) \mathbb {E} \left[ \operatorname {S a R} ^ {\text {s i n g l e}} (\mathbf {U C B}, T) ^ {2} \right] \leq \frac {2 ^ {9} K ^ {2} (\ln T + 2) ^ {2}}{\varepsilon^ {2}}.
$$
We next offer a proof sketch of Theorem 2. The theorem and lemmas are proven in [12, Section 4].
Proof sketch of Theorem 2. Fix any pair of  $\lambda, \mu$  with  $\max_{k \in \mathcal{K}} \mu_k = \lambda + \varepsilon$ . Let  $T_1 = \left\lfloor \left(2^{12} K^2 / \varepsilon^4\right)^2 \right\rfloor$ . We bound  $\mathrm{CLQ}^{\mathrm{single}}(\lambda, \mu, \mathrm{UCB})$  by considering  $T \leq T_1$  (learning stage) and  $T \geq T_1$  (regenerate stage) separately. For  $T \leq T_1$ , we have
$$
\frac {1}{T} \sum_ {t \leq T} \mathbb {E} \left[ Q (t) \right] \leq \frac {3}{\varepsilon} + \mathbb {E} \left[ \mathrm {S a R} ^ {\mathrm {s i n g l e}} (\mathbf {U C B}, T) \right] \leq \frac {3 2 3 K + 6 4 K (\ln K + 2 \ln^ {1 / \varepsilon})}{\varepsilon},
$$
where we use Lemmas 4.1 and 4.3 (i). For  $T > T_{1}$ , we have
$$
\frac {1}{T} \sum_ {t \leq T} \mathbb {E} [ Q (t) ] \leq \frac {4}{\varepsilon} + \frac {8}{\varepsilon^ {2}} \frac {\mathbb {E} \left[ \operatorname {S a R} ^ {\text {s i n g l e}} (\mathbf {U C B} , T) ^ {2} \right]}{T} \leq \frac {3 2 3 K + 6 4 K (\ln K + 2 \ln^ {1 / \varepsilon})}{\varepsilon}, \tag {7}
$$
where we use Lemmas 4.2 and 4.3 (ii).