[48] Mohammadhosein Hasanbeig, Natasha Yogananda Jeppu, Alessandro Abate, Tom Melham, and Daniel Kroening. DeepSynth: Automata synthesis for automatic task segmentation in deep reinforcement learning. In AAAI, pages 7647-7656, 2021.
[49] Hippolyte Bourel, Anders Jonsson, Odalric-Ambrym Maillard, and Mohammad Sadegh Talebi. Exploration in reward machines with low regret. In International Conference on Artificial Intelligence and Statistics, pages 4114-4146. PMLR, 2023.
[50] Jie Fu and Ufuk Topcu. Probably approximately correct MDP learning and control with temporal logic constraints. In RSS, 2014.
[51] Mohammadhosein Hasanbeig, Alessandro Abate, and Daniel Kroening. Cautious reinforcement learning with logical constraints. In AAMAS, pages 483-491, 2020.
[52] Lewis Hammond, Alessandro Abate, Julian Gutierrez, and Michael J. Wooldridge. Multi-agent reinforcement learning with temporal logic specifications. In AAMAS, pages 583-592, 2021.
[53] Alper Kamil Bozkurt, Yu Wang, Michael M. Zavlanos, and Miroslav Pajic. Control synthesis from linear temporal logic specifications using model-free reinforcement learning. In ICRA, pages 10349-10355, 2020.
[54] Ernst Moritz Hahn, Mateo Perez, Sven Schewe, Fabio Somenzi, Ashutosh Trivedi, and Dominik Wojtczak. Omega-regular objectives in model-free reinforcement learning. In TACAS, pages 395–412, 2019.
[55] Sultan Javed Majeed and Marcus Hutter. On Q-learning convergence for non-Markov decision processes. In IJCAI, pages 2546-2552, 2018.
[56] Dana Ron, Yoram Singer, and Naftali Tishby. On the learnability and usage of acyclic probabilistic finite automata. J. Comput. Syst. Sci., 56(2):133-152, 1998.
[57] Alexander Clark and Franck Thollard. PAC-learnability of probabilistic deterministic finite state automata. J. Mach. Learn. Res., 5:473–497, 2004.
[58] Nick Palmer and Paul W. Goldberg. PAC-learnability of probabilistic deterministic finite state automata in terms of variation distance. Theor. Comput. Sci., 387(1):18-31, 2007.
[59] Borja Balle. Learning Finite-State Machines: Statistical and Algorithmic Aspects. PhD thesis, Universitat Politècnica de Catalunya, 2013.
[60] Borja Balle, Jorge Castro, and Ricard Gavalda. Adaptively learning probabilistic deterministic automata from data streams. Machine Learning, 96(1):99-127, 2014.
[61] Edward Forrest Moore. Gedanken-experiments on sequential machines. Automata studies, 34:129-153, 1956.
[62] John E. Hopcroft, Rajeev Motwani, and Jeffrey D. Ullman. Introduction to Automata Theory, Languages, and Computation. Addison-Wesley Longman Publishing Co., Inc., 2006.
[63] Jeffrey Shallit. A second course in formal languages and automata theory. Cambridge University Press, 2008.
[64] Ronen I. Brafman, Giuseppe De Giacomo, and Fabio Patrizi. LTLf/LDLf non-markovian rewards. In AAAI, pages 1771-1778, 2018.
[65] Maor Gaon and Ronen I. Brafman. Reinforcement learning with non-markovian rewards. In AAAI, pages 3980-3987, 2020.
[66] Andrew Kachites McCallum. Reinforcement Learning with Selective Perception and Hidden State. PhD thesis, University of Rochester, 1996.
[67] Jorma Rissanen. A universal data compression system. IEEE Transactions on information theory, 29(5): 656-664, 1983.
[68] Dana Ron, Yoram Singer, and Naftali Tishby. The power of amnesia: Learning probabilistic automata with variable memory length. Mach. Learn., 25(2-3):117-149, 1996.
[69] Marcus Hutter. Extreme state aggregation beyond Markov decision processes. Theor. Comp. Sci., pages 73-91, 2016.
[70] Andreas Maurer and Massimiliano Pontil. Empirical Bernstein bounds and sample-variance penalization. In  $COLT$ , 2009.
[71] Christoph Dann, Tor Lattimore, and Emma Brunskill. Unifying PAC and regret: Uniform PAC bounds for episodic reinforcement learning. In NeurIPS, pages 5713-5723, 2017.
[72] Thomas M. Cover and Joy A. Thomas. Elements of Information Theory (2. Ed.). Wiley, 2006.
[73] Balázs Szörenyi. Characterizing statistical query learning: Simplified notions and proofs. In ALT, pages 186-200, 2009.