[23] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
[24] Kamil Ciosek and Shimon Whiteson. Expected policy gradients. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[25] Chunjie Luo, Jianfeng Zhan, Xiaohe Xue, Lei Wang, Rui Ren, and Qiang Yang. Cosine normalization: Using cosine similarity instead of dot product in neural networks. In International Conference on Artificial Neural Networks, pages 382-391. Springer, 2018.
[26] Paul Bratley, Bennet L Fox, and Linus E Schrage. A guide to simulation. Springer Science & Business Media, 2011.
[27] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016.
[28] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pages 5026-5033. IEEE, 2012.
[29] Ignasi Clavera, Violet Fu, and Pieter Abbeel. Model-augmented actor-critic: Backpropagating through paths. arXiv preprint arXiv:2005.08068, 2020.
[30] Shimon Whiteson. Expected policy gradients for reinforcement learning. Journal of Machine Learning Research, 21, 2020.
[31] Harm Van Seijen, Hado Van Hasselt, Shimon Whiteson, and Marco Wiering. A theoretical and empirical analysis of expected sarsa. In 2009 *ieee symposium on adaptive dynamic programming and reinforcement learning*, pages 177–184. IEEE, 2009.
[32] Kavosh Asadi, Cameron Allen, Melrose Roderick, Abdel-rahman Mohamed, George Konidaris, Michael Littman, and Brown University Amazon. Mean actor critic. stat, 1050:1, 2017.
[33] Matthew Fellows, Kamil Ciosek, and Shimon Whiteson. Fourier policy gradients. In International Conference on Machine Learning, pages 1486-1495. PMLR, 2018.
[34] Benjamin Petit, Loren Amdahl-Culleton, Yao Liu, Jimmy Smith, and Pierre-Luc Bacon. All-action policy gradient methods: A numerical integration approach. arXiv preprint arXiv:1910.09093, 2019.
[35] Maxim Kaledin, Alexander Golubev, and Denis Belomestny. Variance reduction for policy-gradient methods via empirical variance minimization. arXiv preprint arXiv:2206.06827, 2022.
[36] Arushi Jain, Gandharv Patil, Ayush Jain, Khimya Khetarpal, and Doina Precup. Variance penalized on-policy and off-policy actor-critic. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 7899-7907, 2021.
[37] Aviv Tamar, Dotan Di Castro, and Shie Mannor. Temporal difference methods for the variance of the reward to go. In International Conference on Machine Learning, pages 495-503. PMLR, 2013.
[38] Matthew J Sobel. The variance of discounted markov decision processes. Journal of Applied Probability, 19(4):794-802, 1982.
[39] Nicolas Heess, Gregory Wayne, David Silver, Timothy Lillicrap, Tom Erez, and Yuval Tassa. Learning continuous control policies by stochastic value gradients. Advances in neural information processing systems, 28, 2015.
[40] Brandon Amos, Samuel Stanton, Denis Yarats, and Andrew Gordon Wilson. On the model-based stochastic value gradient for continuous reinforcement learning. In Learning for Dynamics and Control, pages 6-20. PMLR, 2021.