[36] Aditya Mate, Andrew Perrault, and Milind Tambe. Risk-Aware Interventions in Public Health: Planning with Restless Multi-Armed Bandits. In Proc.of AAMAS, 2021.
[37] Francisco S Melo, Sean P Meyn, and M Isabel Ribeiro. An analysis of reinforcement learning with function approximation. In Proceedings of the 25th international conference on Machine learning, pages 664-671, 2008.
[38] Rahul Meshram, Aditya Gopalan, and D Manjunath. Optimal recommendation to users that react: Online learning for a class of pomdps. In 2016 IEEE 55th Conference on Decision and Control (CDC), pages 7210-7215. IEEE, 2016.
[39] Abdelkader Mokkadem and Mariane Pelletier. Convergence rate and averaging of nonlinear two-time-scale stochastic approximation algorithms. The Annals of Applied Probability, 16(3):1671-1702, 2006.
[40] Khaled Nakhleh, Santosh Ganji, Ping-Chun Hsieh, I Hou, Srinivas Shakkottai, et al. Neurwin: Neural whittle index network for restless bandits via deep rl. Advances in Neural Information Processing Systems, 34, 2021.
[41] Khaled Nakhleh, I Hou, et al. Deeptop: Deep threshold-optimal policy for mdps and rmabs. arXiv preprint arXiv:2209.08646, 2022.
[42] Ronald Ortner, Daniil Ryabko, Peter Auer, and Rémi Munos. Regret Bounds for Restless Markov Bandits. In Proc. of Algorithmic Learning Theory, 2012.
[43] Tejas Pagare, Vivek Borkar, and Konstantin Avrachenkov. Full gradient deep reinforcement learning for average-reward criterion. arXiv preprint arXiv:2304.03729, 2023.
[44] Christos H Papadimitriou and John N Tsitsiklis. The Complexity of Optimal Queueing Network Control. In Proc. of IEEE Conference on Structure in Complexity Theory, 1994.
[45] Martin L Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, 1994.
[46] Guannan Qu and Adam Wierman. Finite-Time Analysis of Asynchronous Stochastic Approximation and  $Q$ -Learning. In Proc. of COLT, 2020.
[47] Hiteshi Sharma, Mehdi Jafarnia-Jahromi, and Rahul Jain. Approximate relative value learning for average-reward continuous state mdps. In Uncertainty in Artificial Intelligence, pages 956-964. PMLR, 2020.
[48] Rayadurgam Srikant and Lei Ying. Finite-time error bounds for linear stochastic approximation and td learning. In Conference on Learning Theory, pages 2803-2830. PMLR, 2019.
[49] Wesley Suttle, Kaiqing Zhang, Zhuoran Yang, Ji Liu, and David Kraemer. Reinforcement Learning for Cost-Aware Markov Decision Processes. In Proc. of ICML, 2021.
[50] Cem Tekin and Mingyan Liu. Online Learning of Rested and Restless Bandits. IEEE Transactions on Information Theory, 58(8):5588-5611, 2012.
[51] John N Tsitsiklis and Benjamin Van Roy. Average Cost Temporal-Difference Learning. Automatica, 35(11):1799-1808, 1999.
[52] Yi Wan, Abhishek Naik, and Richard S Sutton. Learning and Planning in Average-Reward Markov Decision Processes. In Proc. of ICML, 2021.
[53] Siwei Wang, Longbo Huang, and John Lui. Restless-UCB, an Efficient and Low-complexity Algorithm for Online Restless Bandits. In Proc. of NeurIPS, 2020.
[54] Richard R Weber and Gideon Weiss. On An Index Policy for Restless Bandits. Journal of applied probability, pages 637-648, 1990.
[55] Chen-Yu Wei, Mehdi Jafarnia Jahromi, Haipeng Luo, Hiteshi Sharma, and Rahul Jain. Model-free Reinforcement Learning in Infinite-Horizon Average-Reward Markov Decision Processes. In Proc. of ICML, 2020.