SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Remark 2. Although the CLQ metric is focused on the entire horizon, our analysis extends to bounding the maximum expected time-averaged queue lengths in the later horizon, which is formalized as  $\max_{T\geq T_1}\frac{\sum_{t\leq T}\mathbb{E}[Q(t)]}{T}$  for any  $T_{1}$ . In particular, for  $T_{1}\geq \left(\frac{2^{12}K^{2}}{\varepsilon^{4}}\right)^{2}$ , Lemma 4.2 shows that  $\max_{T\geq T_1}\frac{\sum_{t\leq T}\mathbb{E}[Q(t)]}{T}\leq \frac{5}{\varepsilon}$ ; UCB thus enjoys the optimal asymptotic queue length scaling of  $O(\frac{1}{\varepsilon})$ .
Motivated by the observation that queueing regret does not capture the complexity of learning which tends to occur in the initial stages, we propose an alternative metric (CLQ) to encapsulate the statistical complexity of learning in queueing systems. For a single-queue multi-server system with  $K$  servers and a traffic slackness  $\varepsilon$ , we derive a lower bound  $\Omega \left(\frac{K}{\varepsilon}\right)$  on CLQ, thus establishing that learning incurs a non-negligible increase in queue lengths. We then show that the classical UCB algorithm has a matching upper bound of  $\tilde{O}\left(\frac{K}{\varepsilon}\right)$ . Finally, we extend our result to multi-queue multi-sever systems and general queueing networks by providing algorithms, MAXWEIGHT-UCB and BACKPRESSURE-UCB, whose CLQ has a near optimal  $\tilde{O}(1 / \varepsilon)$  dependence on traffic slackness.
Having introduced a metric that captures the complexity of learning in queueing systems, our work can serve as a starting point for interesting extensions that can help shed further light on the area. In particular, future research may focus on beyond worst case guarantees for CLQ, non-stationary settings, improved bounds using contextual information, etc.
This is suboptimal as long as in the second term of Lemma 4.2 we have an exponent greater than 1 for  $1 / \varepsilon$ .
[1] Saghar Adler and Vijay Subramanian. Bayesian learning of optimal policies in markov decision processes with countably infinite state-space. arXiv preprint arXiv:2306.02574, 2023.
[2] Saed Alizamir, Francis De Vercourt, and Peng Sun. Diagnostic accuracy under congestion. Management Science, 59(1):157-171, 2013.
[3] Mor Armony, Shlomo Israelit, Avishai Mandelbaum, Yariv N Marmor, Yulia Tseytlin, and Galit B Yom-Tov. On patient flow in hospitals: A data-based queueing-science perspective. Stochastic systems, 5(1):146-194, 2015.
[4] Peter Auer, Nicolò Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Mach. Learn., 47(2-3):235-256, 2002.
[5] Peter Auer, Nicolò Cesa-Bianchi, Yoav Freund, and Robert E. Schapire. The nonstochastic multiarmed bandit problem. SIAM J. Comput., 32(1):48-77, 2002.
[6] Kostas Bimpikis and Mihalis G Markakis. Learning and hierarchies in service systems. Management Science, 65(3):1268-1285, 2019.
[7] Maury Bramson, Bernardo D'Auria, and Neil Walton. Stability and instability of the maxweight policy. Math. Oper. Res., 46(4):1611-1638, 2021.
[8] Xinyun Chen, Yunan Liu, and Guiyu Hong. An online learning approach to dynamic pricing and capacity sizing in service systems. Operations Research, 2023.
[9] Tuhinangshu Choudhury, Gauri Joshi, Weina Wang, and Sanjay Shakkottai. Job dispatching policies for queueing systems with unknown service rates. In Proceedings of the Twenty-second International Symposium on Theory, Algorithmic Foundations, and Protocol Design for Mobile Networks and Mobile Computing, pages 181-190, 2021.
[10] Jim G. Dai and Wuqin Lin. Maximum pressure policies in stochastic processing networks. Oper. Res., 53(2):197-218, 2005.
[11] Daniel Freund, Thodoris Lykouris, and Wentao Weng. Efficient decentralized multi-agent learning in asymmetric bipartite queueing systems. Operations Research, 2023.
[12] Daniel Freund, Thodoris Lykouris, and Wentao Weng. Quantifying the cost of learning in queueing systems. arXiv preprint arXiv:2308.07817v2, 2023.
[13] Hu Fu, Qun Hu, and Jia'nan Lin. Stability of decentralized queueing networks beyond complete bipartite cases. In Kristoffer Arnsfelt Hansen, Tracy Xiao Liu, and Azarakhsh Malekian, editors, Web and Internet Economics - 18th International Conference, WINE 2022, Troy, NY, USA, December 12-15, 2022, Proceedings, volume 13778 of Lecture Notes in Computer Science, pages 96-114. Springer, 2022.
[14] Xinzhe Fu and Eytan Modiano. Optimal routing to parallel servers with unknown utilities—multi-armed bandit with queues. IEEE/ACM Transactions on Networking, 2022.
[15] Jason Gaitonde and Éva Tardos. The price of anarchy of strategic queuing systems. Journal of the ACM, 2023.
[16] Noah Gans, Ger Koole, and Avishai Mandelbaum. Telephone call centers: Tutorial, review, and research prospects. Manuf. Serv. Oper. Manag., 5(2):79-141, 2003.
[17] Mor Harchol-Balter. Performance modeling and design of computer systems: queueing theory in action. Cambridge University Press, 2013.
[18] Refael Hassin and Moshe Haviv. To queue or not to queue: Equilibrium behavior in queueing systems, volume 59. Springer Science & Business Media, 2003.
[39] Leandros Tassiulas. Stability properties of constrained queueing systems and scheduling policies for maximum throughput in multihop radio networks. IEEE TRANSACTIONS ON AUTOMATIC CONTROL, 31(12), 1992.
[40] Neil Walton and Kuang Xu. Learning and information in stochastic networks and queues. In Tutorials in Operations Research: Emerging Optimization Methods and Modeling Techniques with Applications, pages 161-198. INFORMS, 2021.
[41] Neil S. Walton. Two queues with non-stochastic arrivals. Oper. Res. Lett., 42(1):53-57, 2014.
[42] Zixian Yang, R Srikant, and Lei Ying. Learning while scheduling in multi-server systems with unknown statistics: Maxweight with discounted ucb. In International Conference on Artificial Intelligence and Statistics, pages 4275-4312. PMLR, 2023.
[43] Yueyang Zhong, John R Birge, and Amy Ward. Learning the scheduling policy in time-varying multiclass many server queues with abandonment. Available at SSRN, 2022.