Subhojyoti Mukherjee and Odalric-Ambrym Maillard. Distribution-dependent and time-uniform bounds for piecewise i.i.d bandits. Reinforcement Learning for Real Life (RL4RealLife) Workshop in the 36th International Conference on Mearning Learning, 2019.
Ronald Ortner, Pratik Gajane, and Peter Auer. Variational regret bounds for reinforcement learning. In Proceedings of The 35th Uncertainty in Artificial Intelligence Conference, volume 115 of Proceedings of Machine Learning Research, pages 81-90. PMLR, 22-25 Jul 2020.
Vianney Perchet and Philippe Rigollet. The multi-armed bandit problem with covariates. The Annals of Statistics, 41(2):693-721, 2013.
Yury Polyanskiy and Yihong Wu. Information Theory: From Coding to Learning. Cambridge University Press, 2022.
Wei Qian and Yuhong Yang. Kernel estimation and model combination in a bandit problem with covariates. Journal of Machine Learning Research, 17(149):1-37, 2016a.
Wei Qian and Yuhong Yang. Randomized allocation with arm elimination in a bandit problem with covariates. Electronic Journal of Statistics, 10(1):242 - 270, 2016b.
Henry Reeve, Joe Mellor, and Gavin Brown. The k-nearest neighbour ucb algorithm for multi-armed bandits with covariates. In Proceedings of Algorithmic Learning Theory, volume 83 of Proceedings of Machine Learning Research, pages 725–752. PMLR, 07–09 Apr 2018.
Phillipe Rigollet and Assaf Zeevi. Nonparametric bandits with covariates.  $COLT$ , 2010.
Jyotirmoy Sarkar. One-armed bandit problems with covariates. The Annals of Statistics, pages 1978-2002, 1991.
David Simchi-Levi and Yunzong Xu. Bypassing the monster: a faster and simpler optimal algorithm for contextual bandits under realizability. Mathematics of Operations Research, 47(3):1904-1931, 2021.
Aleksandrs Slivkins. Contextual bandits with similarity information. The Journal of Machine Learning Research, 15(1):2533-2568, 2014.
Joe Suk and Samory Kpotufe. Tracking most significant arm switches in bandits. In Proceedings of Thirty Fifth Conference on Learning Theory, volume 178 of Proceedings of Machine Learning Research, pages 2160-2182. PMLR, 02-05 Jul 2022.
Joseph Suk and Samory Kpotufe. Self-tuning bandits over unknown covariate-shifts. International Conference on Algorithmic Learning Theory, 2021.
Ahmed Touati and Pascal Vincent. Efficient learning in non-stationary linear Markov decision processes. arXiv preprint arXiv:2010.12870, 2020.
Chen-Yu Wei and Haipeng Luo. Non-stationary reinforcement learning without prior knowledge: An optimal black-box approach. Proceedings of the 32nd International Conference on Learning Theory, 2021.
Chen-Yu Wei, Christoph Dann, and Julian Zimmert. A model selection approach for corruption robust reinforcement learning. In International Conference on Algorithmic Learning Theory, pages 1043-1096. PMLR, 2022.
Lai Wei and Vailbav Srivatsva. On abruptly-changing and slowly-varying multiarmed bandit problems. Annual American Control Conference (ACC), 2018.
Michael Woodrofe. A one-armed bandit problem with a concomitant variable. Journal of the American Statistical Association, 74(368):799-806, 1979.
Qingyun Wu, Naveen Iyer, and Hongning Wang. Learning contextual bandits in a non-stationary environment. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pages 495-504, 2018.