[19] Anthony Ephremides, Pravin Varaiya, and Jean Walrand. A simple dynamic routing problem. IEEE Transactions on Automatic Control, 25(4):690-693, 1980.
[20] Lloyd Fisher and Sheldon M Ross. An example in denumerable decision processes. The Annals of Mathematical Statistics, 39(2):674-675, 1968.
[21] Daniel Freund, Thodoris Lykouris, and Wentao Weng. Efficient decentralized multi-agent learning in asymmetric queuing systems. In Conference on Learning Theory, pages 4080-4084. PMLR, 2022.
[22] Mohammad Ghavamzadeh, Shie Mannor, Joelle Pineau, and Aviv Tamar. Bayesian reinforcement learning: A survey. Foundations and Trends® in Machine Learning, 8(5-6):359–483, 2015.
[23] Aditya Gopalan and Shie Mannor. Thompson sampling for learning parameterized Markov decision processes. In Conference on Learning Theory, pages 861-898. PMLR, 2015.
[24] Todd L Graves and Tze-Leung Lai. Asymptotically efficient adaptive choice of control laws incontrolled Markov chains. SIAM Journal on Control and Optimization, 35(3):715-743, 1997.
[25] Bruce Hajek. Hitting-time and occupation-time bounds implied by drift analysis with applications. Advances in Applied Probability, 14(3):502-525, 1982.
[26] Bruce Hajek. Optimal control of two interacting service stations. IEEE Transactions on Automatic Control, 29(6):491-499, 1984.
[27] Arie Hordijk and Flora Spieksma. On ergodicity and recurrence properties of a Markov chain by an application to an open Jackson network. Advances in Applied Probability, 24(2):343-376, 1992.
[28] Mehdi Jafarnia Jahromi, Rahul Jain, and Ashutosh Nayyar. Online learning for unknown partially observable MDPs. In International Conference on Artificial Intelligence and Statistics, pages 1712-1732. PMLR, 2022.
[29] Soren F Jarner and Gareth O Roberts. Polynomial convergence rates of Markov chains. The Annals of Applied Probability, 12(1):224-247, 2002.
[30] Subhashini Krishnasamy, PT Akhil, Ari Arapostathis, Rajesh Sundaresan, and Sanjay Shakkottai. Augmenting Max-Weight with explicit learning for wireless scheduling with switching costs. IEEE/ACM Transactions on Networking, 26(6):2501-2514, 2018.
[31] Subhashini Krishnasamy, Ari Arapostathis, Ramesh Johari, and Sanjay Shakkottai. On learning the  $c\mu$  rule in single and parallel server networks. https://arxiv.org/abs/1802.06723, 2018.
[32] Subhashini Krishnasamy, Rajat Sen, Ramesh Johari, and Sanjay Shakkottai. Learning unknown service rates in queues: A multiarmed bandit approach. Operations Research, 69(1):315-330, 2021.
[33] PR Kumar and A Becker. A new family of optimal adaptive controllers for Markov chains. IEEE Transactions on Automatic Control, 27(1):137-146, 1982.
[34] P R Kumar and Woei Lin. Optimal adaptive controllers for unknown Markov chains. IEEE Transactions on Automatic Control, 27(4):765-774, 1982.
[35] P R Kumar and Pravin Varaiya. Stochastic systems: Estimation, identification, and adaptive control. SIAM, 2015.
[36] Tze-Leung Lai and Sidney Yakowitz. Machine learning and nonparametric bandit theory. IEEE Transactions on Automatic Control, 40(7):1199-1209, 1995.
[37] Ronald Larsen. Control of multiple exponential servers with application to computer systems. PhD thesis, University of Maryland, 1981.
[38] Woei Lin and P R Kumar. Optimal control of a queueing system with two heterogeneous servers. IEEE Transactions on Automatic Control, 29(8):696-703, 1984.