[18] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015.
[19] E. N. Lorenz. The Essence of Chaos. University of Washington Press, 1995.
[20] S. W. McDonald, C. Grebogia, E. Ott, and J. A. Yorke. Fractal basin boundaries. Physica, 17D:125-153, 1985.
[21] L. Metz, C. D. Freeman, S. S. Schoenholz, and T. Kachman. Gradients are not all you need. 2021.
[22] J. Millán, D. Posenato, and E. Dedieu. Continuous-action Q-learning. Machine Learning, 49:247-265, 2002.
[23] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, and et al. Human-level control through deep reinforcement learning. Nature, 518:529-533, 2015.
[24] P. Parmas, C. E. Rasmussen, J. Peters, and K. Doya. PIPPS: Flexible model-based policy search robust to the curse of chaos. Proceedings of the 35th International Conference on Machine Learning, pages 4065-4074, 2018.
[25] R. Pascanu, T. Mikolov, and Y. Bengio. On the difficulty of training recurrent neural networks. Proceedings of the 30th International Conference on Machine Learning, pages 1310-1318, 2013.
[26] D. Preiss. Geometry of measures in  $\mathbb{R}^n$ : Distribution, rectifiability, and densities. Annals of Mathematics, 125(3):537-643, 1987.
[27] J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz. Trust region policy optimization. Proceedings of the 32nd International Conference on Machine Learning, pages 1889–1897, 2015.
[28] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
[29] D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller. Deterministic policy gradient algorithms. Proceedings of the 31st International Conference on Machine Learning, pages 387-395, 2014.
[30] D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, Y. Chen, T. Lillicrap, F. Hui, L. Sifre, G. Driessche, T. Graepel, and D. Hassabis. Mastering the game of Go without human knowledge. Nature, 550:354-359, 2017.
[31] H. Suh, M. Simchowitz, K. Zhang, and R. Tedrake. Do differentiable simulators give better policy gradients? Proceedings of the 39th International Conference on Machine Learning, 162:20668-20696, 2022.
[32] R. S. Sutton and A. Barto. Reinforcement Learning: an Introduction. MIT Press, 1998.
[33] R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in Neural Information Processing Systems 12, pages 1057-1063, 1999.
[34] P. Thomas and E. Brunskill. Data-efficient off-policy policy evaluation for reinforcement learning. Proceedings of The 33rd International Conference on Machine Learning, pages 2139-2148, 2016.
[35] J. N. Tsitsiklis and B. V. Roy. Analysis of temporal-difference learning with function approximation. Advances in Neural Information Processing Systems 9, pages 1075-1081, 1996.
[36] S. S. Vallender. Calculation of the Wasserstein distance between probability distributions on the line. Theory of Probability and its Applications, 18(4):784-786, 1974.