[44] Ioana Bica, Ahmed M. Alaa, James Jordon, and Mihaela van der Schaar. Estimating Counterfactual Treatment Outcomes over Time Through Adversarily Balanced Representations. In International Conference on Learning Representations. arXiv, February 2020. arXiv:2002.04083 [cs, stat].
[45] Pushi Zhang, Li Zhao, Guoqing Liu, Jiang Bian, Minlie Huang, Tao Qin, and Tie-Yan Liu. Independence-aware Advantage Estimation. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, pages 3349-3355, Montreal, Canada, August 2021. International Joint Conferences on Artificial Intelligence Organization.
[46] Kenny Young. Variance Reduced Advantage Estimation with  $\$ \backslash$ delta $\$$  Hindsight Credit Assignment. arXiv:1911.08362 [cs], September 2020. arXiv:1911.08362.
[47] Michel Ma and Bacon Pierre-Luc. Counterfactual Policy Evaluation and the Conditional Monte Carlo Method. In Offline Reinforcement Learning Workshop, NeurIPS, 2020.
[48] Paul Bratley, Bennett L. Fox, and Linus E. Schrage. A Guide to Simulation. Springer, New York, NY, 1987.
[49] J. M. Hammersley. Conditional Monte Carlo. Journal of the ACM, 3(2):73-76, April 1956.
[50] Dilip Arumugam, Peter Henderson, and Pierre-Luc Bacon. An Information-Theoretic Perspective on Credit Assignment in Reinforcement Learning. arXiv:2103.06224 [cs, math], March 2021. arXiv: 2103.06224.
[51] Kenny Young. Hindsight Network Credit Assignment: Efficient Credit Assignment in Networks of Discrete Stochastic Units. Proceedings of the AAAI Conference on Artificial Intelligence, 36 (8):8919-8926, June 2022. Number: 8.
[52] Shixiang Gu, Timothy Lillicrap, Zoubin Ghahramani, Richard E. Turner, and Sergey Levine. Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic. 2017.
[53] Philip S. Thomas and Emma Brunskill. Policy Gradient Methods for Reinforcement Learning with Function Approximation and Action-Dependent Baselines, June 2017. arXiv:1706.06643 [cs].
[54] Hao Liu*, Yihao Feng*, Yi Mao, Dengyong Zhou, Jian Peng, and Qiang Liu. Action-dependent Control Variates for Policy Optimization via Stein Identity. February 2022.
[55] Cathy Wu, Aravind Rajeswaran, Yan Duan, Vikash Kumar, Alexandre M. Bayen, Sham Kakade, Igor Mordatch, and Pieter Abbeel. Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselines. February 2022.
[56] George Tucker, Surya Bhupatiraju, Shixiang Gu, Richard Turner, Zoubin Ghahramani, and Sergey Levine. The Mirage of Action-Dependent Baselines in Reinforcement Learning. In Proceedings of the 35th International Conference on Machine Learning, pages 5015–5024. PMLR, July 2018. ISSN: 2640-3498.
[57] Chris Nota, Philip Thomas, and Bruno C. Da Silva. Posterior Value Functions: Hindsight Baselines for Policy Gradient Methods. In Proceedings of the 38th International Conference on Machine Learning, pages 8238-8247. PMLR, July 2021. ISSN: 2640-3498.
[58] Arthur Guez, Fabio Viola, Theophane Weber, Lars Buesing, Steven Kapturowski, Doina Precup, David Silver, and Nicolas Heess. Value-driven Hindsight Modelling. Advances in Neural Information Processing Systems, 33, 2020.
[59] David Venuto, Elaine Lau, Doina Precup, and Ofir Nachum. Policy Gradients Incorporating the Future. January 2022.
[60] Jiawei Huang and Nan Jiang. From Importance Sampling to Doubly Robust Policy Gradient. In Proceedings of the 37th International Conference on Machine Learning, pages 4434-4443. PMLR, November 2020. ISSN: 2640-3498.
[61] Andrew Y. Ng, Daishi Harada, and Stuart J. Russell. Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping. In Proceedings of the Sixteenth International Conference on Machine Learning, ICML '99, pages 278-287, San Francisco, CA, USA, June 1999. Morgan Kaufmann Publishers Inc.