[1] Anna Harutyunyan, Will Dabney, Thomas Mesnard, Mohammad Gheshlaghi Azar, Bilal Piot, Nicolas Heess, Hado P van Hasselt, Gregory Wayne, Satinder Singh, Doina Precup, and Remi Munos. Hindsight Credit Assignment. In H. Wallach, H. Larochelle, A. Beygelzimer, F Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 12488-12497. Curran Associates, Inc., 2019.
[2] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning, second edition: An Introduction. MIT Press, November 2018.
[3] Thomas Mesnard, Theophane Weber, Fabio Viola, Shantanu Thakoor, Alaa Saade, Anna Harutyunyan, Will Dabney, Tom Stepleton, Nicolas Heess, Arthur Guez, Marcus Hutter, Lars Buesing, and Rémi Munos. Counterfactual Credit Assignment in Model-Free Reinforcement Learning. In Proceedings of the 38th International Conference on Machine Learning, PMLR 139, 2021.
[4] Chia-Chun Hung, Timothy Lillicrap, Josh Abramson, Yan Wu, Mehdi Mirza, Federico Carnevale, Arun Ahuja, and Greg Wayne. Optimizing agent behavior over long time scales by transporting value. Nature Communications, 10(1):5223, November 2019. Number: 1 Publisher: Nature Publishing Group.
[5] Jose A. Arjona-Medina, Michael Gillhofer, Michael Widrich, Thomas Unterthiner, Johannes Brandstetter, and Sepp Hochreiter. RUDDER: Return Decomposition for Delayed Rewards. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d\text{text}quotesingle Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 13566-13577. Curran Associates, Inc., 2019.
[6] Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3):229-256, May 1992.
[7] Evan Greensmith, Peter L. Bartlett, and Jonathan Baxter. Variance Reduction Techniques for Gradient Estimates in Reinforcement Learning. Journal of Machine Learning Research, 5(Nov): 1471-1530, 2004.
[8] Théophane Weber, Nicolas Heess, Lars Buesing, and David Silver. Credit Assignment Techniques in Stochastic Computation Graphs. In Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, pages 2650–2660. PMLR, April 2019. ISSN: 2640-3498.
[9] Jonathan Baxter and Peter L. Bartlett. Infinite-Horizon Policy-Gradient Estimation. Journal of Artificial Intelligence Research, 15:319–350, November 2001.
[10] Philip Thomas. Bias in Natural Actor-Critic Algorithms. In Proceedings of the 31st International Conference on Machine Learning, pages 441-448. PMLR, January 2014. ISSN: 1938-7228.
[11] Sham Kakade. Optimizing Average Reward Using Discounted Rewards. In David Helmbold and Bob Williamson, editors, Computational Learning Theory, Lecture Notes in Computer Science, pages 605-615, Berlin, Heidelberg, 2001. Springer.
[12] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-Dimensional Continuous Control Using Generalized Advantage Estimation. In International Conference on Learning Representations. arXiv, 2016. arXiv:1506.02438 [cs].
[13] Peter Marbach and John Tsitsiklis. Approximate Gradient Methods in Policy-Space Optimization of Markov Reward Processes. Discrete Event Dynamic Systems, page 38, 2003.