[45] M. H. A. Davis. *Markov models and optimization*. Monographs on statistics and applied probability. Chapman & Hall, London ; New York, 1st ed edition, 1993.

[46] Benoîte de Saporta, François Dufour, and Huilong Zhang. *Numerical methods of simulation and optimization of piecewise deterministic Markov processes: application to reliability*. ISTE, Ltd. John Wiley & Sons, London Hoboken, NJ, 2016.

[47] Benoîte de Saporta and Huilong Zhang. Predictive maintenance for the heated hold-up tank. *Reliability Engineering & System Safety*, 115:82-90, July 2013.

[48] Benoîte de Saporta, F Dufour, H Zhang, and C Elegbede. Optimal stopping for the predictive maintenance of a structure subject to corrosion. *Proceedings of the Institution of Mechanical Engineers, Part O: Journal of Risk and Reliability*, 226(2):169-181, April 2012.

[49] Benoîte de Saporta and François Dufour. Numerical method for impulse control of piecewise deterministic Markov processes. *Automatica*, 48(5):779-793, May 2012.

[50] Benoîte de Saporta, François Dufour, and Alizée Geeraert. Optimal strategies for impulse control of piecewise deterministic Markov processes. *Automatica*, 77:219-229, March 2017.

[51] Benoîte de Saporta, François Dufour, and Karen Gonzalez. Numerical method for optimal stopping of piecewise deterministic Markov processes. *The Annals of Applied Probability*, 20(5), October 2010.

[52] Benoîte de Saporta, Aymar Thierry d'Argenlieu, Régis Sabbadin, and Alice Cleynen. A Monte-Carlo planning strategy for medical follow-up optimization: Illustration on multiple myeloma data. *PLOS ONE*, 19(12):e0315661, December 2024.

[53] Richard Dearden, Nicholas Meuleau, Richard Washington, and Zhengzhu Feng. Dynamic programming for structured continuous markov decision problems. In *Twentieth Conference on Uncertainty in Artificial Intelligence (UAI-04)*, 2004.

[54] M. P. Deisenroth, G. Neumann, and J. Peters. A survey on policy search for robotics. *Foundations and Trends in Robotics*, 2(1-2):1-142, 2013.

[55] Pierre Del Moral. Nonlinear filtering: Interacting particle resolution. *Comptes Rendus de l'Académie des Sciences - Series I - Mathematics*, 325(6):653-658, September 1997.

[56] M. A. H. Dempster. Optimal control of piecewise deterministic Markov processes. *Applied stochastic analysis*, 5:303-325, 1991.

[57] M. A. H. Dempster and J. J. Ye. Impulse control of piecewise deterministic Markov processes. *The Annals of Applied Probability*, 5(2):399-423, 1995.

[58] M. A. H. Dempster and J. J. Ye. Generalized Bellman-Hamilton-Jacobi optimality conditions for a control problem with a boundary condition. *Applied Mathematics & Optimization*, 33(3):211-225, May 1996.

[59] M.A.H. Dempster and J.J. Ye. Necessary and sufficient optimality conditions for control of piecewise deterministic markov processes. *Stochastics and Stochastic Reports*, 40(3-4):125-145, September 1992.

[60] Marie Doumic, Marc Hoffmann, Nathalie Krell, and Lydia Robert. Statistical estimation of a growth-fragmentation model observed on a genealogical tree. *Bernoulli*, 21(3), August 2015.

[61] Michael O'Gordon Duff. *Optimal learning: Computational procedures for Bayes-adaptive Markov decision processes*. PhD Thesis, UMassAmherst, January 2002.

[62] F. Dufour, M. Horiguchi, and A. B. Piunovskiy. Optimal impulsive control of piecewise deterministic Markov processes. *Stochastics*, 88(7):1073-1098, October 2016.

[63] Paul Fearnhead, Joris Bierkens, Murray Pollock, and Gareth O. Roberts. Piecewise Deterministic Markov Processes for Continuous-Time Monte Carlo. *Statistical Science*, 33(3), August 2018.
[64] Joaquin Fontbona, Hélène Guérin, and Florent Malrieu. Quantitative Estimates for the Long-Time Behavior of an Ergodic Variant of the Telegraph Process. *Advances in Applied Probability*, 44(4):977-994, December 2012.

[65] Lothar Forwick, Manfred Schäl, and Michael Schmitz. Piecewise Deterministic Markov Control Processes with Feedback Controls and Unbounded Costs. *Acta Applicandae Mathematicae*, 82(3):239-267, July 2004.

[66] Coralie Fritsch, Jérôme Harmand, and Fabien Campillo. A modeling approach of the chemostat. *Ecological Modelling*, 299:1-13, March 2015.

[67] Ninon Fétique. Long-time behaviour of generalised Zig-Zag process, 2017.

[68] Dariusz Gatarek. Optimality conditions for impulsive control of piecewise-deterministic processes. *Mathematics of Control, Signals, and Systems*, 5(2):217-232, June 1992.

[69] Daniel T. Gillespie. Exact stochastic simulation of coupled chemical reactions. *The Journal of Physical Chemistry*, 81(25):2340-2361, 1977.

[70] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. *Deep Learning*. Cambridge: MIT press, 2016.

[71] Divya Grover. *Sample Efficient Bayesian Reinforcement Learning*. PhD thesis, CHALMERS UNIVERSITY OF TECHNOLOGY, 2020.

[72] Arthur Guez, David Silver, and Peter Dayan. Efficient Bayes-Adaptive Reinforcement Learning using Sample-Based Search. In *Advances in Neural Information Processing Systems*, 2012.

[73] U.S. Gugerui. Optimal stopping of a piecewise-deterministic markov process. *Stochastics*, 19(4):221-236, December 1986.

[74] Xianping Guo, Onésimo Hernández-Lerma, Xianping Guo, and Onésimo Hernández-Lerma. *Continuous-time Markov decision processes*. Springer, 2009.

[75] A. Génadot and M. Thieullen. Multiscale Piecewise Deterministic Markov Process in infinite dimension: central limit theorem and Langevin approximation. *ESAIM: Probability and Statistics*, 18:541-569, 2014.

[76] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering Diverse Domains through World Models, April 2024. arXiv:2301.04104 [cs, stat].

[77] Hado Hasselt. Double Q-learning. In *Advances in Neural Information Processing Systems*, volume 23. Curran Associates, Inc., 2010.

[78] Hado van Hasselt, Arthur Guez, and David Silver. Deep Reinforcement Learning with Double Q-Learning. *Proceedings of the AAAI Conference on Artificial Intelligence*, 30(1), March 2016. Number: 1.

[79] Ulysse Herbach, Arnaud Bonnaffoux, Thibault Espinasse, and Olivier Gandrillon. Inferring gene regulatory networks from single-cell data: a mechanistic approach. *BMC Systems Biology*, 11(1):105, December 2017.

[80] Onésimo Hernández-Lerma and Jean Bernard Lasserre. *Discrete-Time Markov Control Processes*. Springer New York, New York, NY, 1996.

[81] Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: combining improvements in deep reinforcement learning. In *Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence*, AAAI'18/IAAI'18/EAAI'18, pages 3215-3222, New Orleans, Louisiana, USA, 2018. AAAI Press.

[82] Karl Hinderer, Ulrich Rieder, and Michael Stieglitz. *Dynamic Optimization*. Universitext. Springer International Publishing, Cham, 2016.
[83] Jesse Hoey, Pascal Poupart, Axel von Bertoldi, Tammy Craig, Craig Boutilier, and Alex Mihailidis. Automated handwashing assistance for persons with dementia using video and a partially observable Markov decision process. *Comput. Vision Image Understanding*, 114(5):503–519, May 2010. Publisher: Academic Press.

[84] Ronald A. Howard. *Dynamic programming Markov processes*. Cambridge: MIT Press, 1962.

[85] Martin Jacobsen. *Point Process Theory and Applications*. Probability and its Applications. Birkhäuser-Verlag, Boston, 2006.

[86] L.P. Kaelbling, M.L. Littman, and A.R. Cassandra. Planning and acting in partially observable stochastic domains. *Artificial Intelligence*, 101(1-2):99–134, 1998.

[87] Sean K. Keneally, Matthew J. Robbins, and Brian J. Lunday. A markov decision process model for the optimal dispatch of military medical evacuation assets. *Health Care Manag. Sci.*, 19(2):111–129, June 2016. Publisher: See full text options at Springer _eprint: 25223847.

[88] Ludvig Killingberg and Helge Langseth. Bayesian Exploration in Deep Reinforcement Learning. *he 2023 symposium of the Norwegian AI Society*, 2023.

[89] L. Kocsis and C. Szepesvari. Bandit based Monte-Carlo Planning. In Springer, editor, *European Conference on Machine Learning*, pages 282–293, 2006.

[90] Richard Kohar. *Bayes-Adaptive Semi-Markov Decision Processes: A Pathway to Optimal Learning in Sequential Decision Making with Time under Uncertainty*. PhD thesis, Royal Military College of Canada, 2023.

[91] Nathalie Krell. Statistical estimation of jump rates for a piecewise deterministic Markov processes with deterministic increasing motion and jump mechanism. *ESAIM: Probability and Statistics*, 20:196–216, 2016.

[92] Nathalie Krell and Émeline Schmisser. Nonparametric estimation of jump rates for a specific class of piecewise deterministic Markov processes. *Bernoulli*, 27(4), November 2021.

[93] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. *Nature*, 521(7553):436–444, May 2015. Publisher: Nature Publishing Group.

[94] Vincent Lemaire, Michèle Thieullen, and Nicolas Thomas. Thinning and multilevel Monte Carlo methods for piecewise deterministic (Markov) processes with an application to a stochastic Morris-Lecar model. *Advances in Applied Probability*, 52(1):138–172, March 2020.

[95] Suzanne M. Lenhart. Viscosity solutions associated with impulse control problems for piecewise-deterministic processes. *International Journal of Mathematics and Mathematical Sciences*, 12(1):145–157, January 1989.

[96] Esther Levin, R. Pieraccini, and Wieland Eckert. Using Markov decision process for learning dialogue strategies. In *Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP'98 (Cat. No. 98CH36181)*, pages 201–204 vol.1, June 1998.

[97] Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph E. Gonzalez, Michael I. Jordan, and Ion Stoica. RLlib: Abstractions for Distributed Reinforcement Learning. In *International conference on machine learning*, pages 3053–3062. PMLR, 2018.

[98] Fan-Ming Luo, Tian Xu, Hang Lai, Xiong-Hui Chen, Weinan Zhang, and Yang Yu. A survey on model-based reinforcement learning. *Science China Information Sciences*, 67(2):121101, 2024.

[99] Peter Stone, Matthew Hausknecht. Deep Recurrent Q-Learning for Partially Observable MDPs.

[100] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing Atari with Deep Reinforcement Learning, December 2013. arXiv:1312.5602 [cs].

[101] Joelle Pineau Mohammad Ghavamzadeh, Shie Mannor and Aviv Tamar. Bayesian Reinforcement Learning: A Survey. *Foundations and Trends in Machine Learning*, 8, 2015.
[102] Athina Monemvassitis, Arnaud Guillin, and Manon Michel. PDMP Characterisation of Event-Chain Monte Carlo Algorithms for Particle Systems. *Journal of Statistical Physics*, 190(3):66, March 2023.

[103] Chloé Pasin, François Dufour, Laura Villain, Huilong Zhang, and Rodolphe Thiébaut. Control-ling IL-7 Injections in HIV-Infected Patients. *Bulletin of Mathematical Biology*, 80(9):2349–2377, September 2018.

[104] Pascal Poupart, Nikos Vlassis, Jesse Hoey, and Kevin Regan. An analytic solution to discrete Bayesian reinforcement learning. In *Proceedings of the 23rd international conference on Machine learning*, ICML '06, pages 697–704, New York, NY, USA, 2006. Association for Computing Machinery.

[105] Adrien Prodhomme and Edouard Strickler. Large population asymptotics for a multitype stochastic SIS epidemic model in randomly switching environment. *The Annals of Applied Probability*, 34(3), June 2024.

[106] Martin L. Puterman. *Markov Decision Processes: Discrete Stochastic Dynamic Programming*. Wiley Series in Probability and Statistics, John Wiley & Sons, April 1994.

[107] Martin Riedler, Michèle Thieullen, and Gilles Wainrib. Limit theorems for infinite-dimensional piecewise deterministic Markov processes. Applications to stochastic excitable membrane models. *Electronic Journal of Probability*, 17(none), January 2012.

[108] Martin G. Riedler. Almost sure convergence of numerical approximations for Piecewise Determin-istic Markov Processes. *Journal of Computational and Applied Mathematics*, 239:50–71, 2013.

[109] Martin G. Riedler and Michèle Thieullen. Spatio-temporal hybrid (PDMP) models: Central limit theorem and Langevin approximation for global fluctuations. Application to electrophysiology. *Bernoulli*, 21(2), May 2015.

[110] Stéphane Ross, Joelle Pineau, Brahim Chaib-draa, and Pierre Kreitmann. A Bayesian Approach for Learning and Planning in Partially Observable Markov Decision Processes. *The Journal of Machine Learning Research*, 12(null):1729–1770, July 2011.

[111] Moumita Roy, Chandreyee Chowdhury, and Nauman Aslam. Designing Transmission Strategies for Enhancing Communications in Medical IoT Using Markov Decision Process. *Sensors (Basel)*., 18(12):4450., December 2018. Publisher: See full text options at MDPI _eprint: 30558343.

[112] Gavin A Rummery and Mahesan Niranjan. *On-line Q-learning using connectionist systems*, volume 37. University of Cambridge, Department of Engineering Cambridge, UK, 1994.

[113] Andrew J. Schaefer, Matthew D. Bailey, Steven M. Shechter, and Mark S. Roberts. Modeling Medical Treatment Using Markov Decision Processes. In Margaret L. Brandeau, François Sainfort, and William P. Pierskalla, editors, *Operations Research and Health Care: A Handbook of Methods and Applications*, pages 593–612. Springer US, Boston, MA, 2004.

[114] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms, August 2017. arXiv:1707.06347 [cs].

[115] Manfred Schäl. On piecewise deterministic Markov control processes: Control of jumps and of risk processes in insurance. *Insurance: Mathematics and Economics*, 22(1):75–91, May 1998.

[116] Yoav Shoham, Rob Powers, and Trond Grenager. If multi-agent learning is the answer, what is the question? *Artif. Intell.*, 171(7):365–377, 2007.

[117] D. Silver, G. Lever, N. Heess, T. Dęgris, D. Wierstra, and M. Riedmiller. Deterministic policy gradient algorithms. In *International Conference on Machine Learning (ICML)*, pages 387–395, 2014.

[118] David Silver and Joel Veness. Monte-Carlo planning in large POMDPs. *Advances in neural information processing systems*, 23, 2010.
[119] Satinder Singh, Tommi Jaakkola, Michael L. Littman, and Csaba Szepesvári. Convergence Results for Single-Step On-Policy Reinforcement-Learning Algorithms. *Machine Learning*, 38(3):287–308, March 2000.

[120] Matthijs T. J. Spaan and Nikos Vlassis. Perseus: randomized point-based value iteration for POMDPs. *J. Artif. Int. Res.*, 24(1):195–220, 2005.

[121] Alexander L Strehl, Lihong Li, Eric Wiewiora, John Langford, and Michael L Littman. Pac model-free reinforcement learning. In *Proceedings of the 23rd international conference on Machine learning*, pages 881–888, 2006.

[122] Malcolm J. A. Strens. A Bayesian Framework for Reinforcement Learning. In *Proceedings of the Seventeenth International Conference on Machine Learning*, ICML '00, pages 943–950, San Francisco, CA, USA, 2000. Morgan Kaufmann Publishers Inc.

[123] R. S. Sutton, D. A. McAllester, S. P. Singh, and Y. Mansour. Policy gradient methods for reinforcement learning with function approximation. In *Neural Information Processing Systems 12*, pages 1057–1063, 1999.

[124] Richard S. Sutton. Learning to predict by the methods of temporal differences. *Machine Learning*, 3(1):9–44, August 1988.

[125] Richard S. Sutton and Andrew G. Barto. *Reinforcement Learning: An Introduction*. A Bradford Book, Cambridge, MA, USA, October 2018.

[126] John N Tsitsiklis. On the convergence of optimistic policy iteration. *Journal of Machine Learning Research*, 3(Jul):59–72, 2002.

[127] R. Veltz. A new twist for the simulation of hybrid systems using the true jump method, 2015. arXiv preprint arXiv:1504.06873.

[128] Tao Wang, Daniel Lizotte, Michael Bowling, and Dale Schuurmans. Bayesian sparse sampling for on-line reward optimization | Proceedings of the 22nd international conference on Machine learning. In *Proceedings of the 22nd international conference on Machine learning*, pages 956–963. Association for Computing Machinery, 2005.

[129] Ziyu Wang, Tom Schaul, Matteo Hessel, Hado van Hasselt, Marc Lanctot, and Nando de Freitas. Dueling Network Architectures for Deep Reinforcement Learning, April 2016. arXiv:1511.06581 [cs].

[130] Christopher J. C. H. Watkins. *Learning from Delayed Rewards*. PhD thesis, University of Cambridge, 1989.

[131] Christopher J. C. H. Watkins and Peter Dayan. Q-learning. *Machine Learning*, 8(3):279–292, May 1992.

[132] Steve Young, Milica Gašić, Blaise Thomson, and Jason D Williams. POMDP-Based Statistical Spoken Dialog Systems: A Review, 2013.

[133] Kaiyu Zheng and Stefanie Tellex. pomdpPY: A framework to build and solve pomdp problems, 2020.

[134] K. J. Åström. Optimal control of Markov processes with incomplete state information. *Journal of Mathematical Analysis and Applications*, 10(1):174–205, 1965.

[135] Maciej Świechowski, Konrad Godlewski, Bartosz Sawicki, and Jacek Mańdziuk. Monte Carlo Tree Search: a review of recent modifications and applications. *Artificial Intelligence Review*, 56(3):2497–2562, March 2023.