SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[5] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897. PMLR, 2015.
[6] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
[7] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference on machine learning, pages 387-395. PMLR, 2014.
[8] Claas Voelcker, Victor Liao, Animesh Garg, and Amir-massoud Farahmand. Value gradient weighted model-based reinforcement learning. arXiv preprint arXiv:2204.01464, 2022.
[9] Andrew G Barto, Richard S Sutton, and Charles W Anderson. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE transactions on systems, man, and cybernetics, pages 834-846, 1983.
[10] Vijay Konda and John Tsitsiklis. Actor-critic algorithms. Advances in neural information processing systems, 12, 1999.
[11] Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, 3(1):9-44, 1988.
[12] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[13] Michael Fairbank and Eduardo Alonso. The local optimality of reinforcement learning by value gradients, and its relationship to policy gradient learning. arXiv preprint arXiv:1101.0428, 2011.
[14] Leslie Pack Kaelbling, Michael L Littman, and Andrew W Moore. Reinforcement learning: A survey. Journal of artificial intelligence research, 4:237-285, 1996.
[15] Richard S Sutton. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In Machine learning proceedings 1990, pages 216-224. Elsevier, 1990.
[16] Shixiang Gu, Timothy Lillicrap, Ilya Sutskever, and Sergey Levine. Continuous deep q-learning with model-based acceleration. In International conference on machine learning, pages 2829-2838. PMLR, 2016.
[17] Vladimir Feinberg, Alvin Wan, Ion Stoica, Michael I Jordan, Joseph E Gonzalez, and Sergey Levine. Model-based value estimation for efficient model-free reinforcement learning. arXiv preprint arXiv:1803.00101, 2018.
[18] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. Advances in Neural Information Processing Systems, 32, 2019.
[19] Pierluca D'Oro and Wojciech Jaskowski. How to learn a useful critic? model-based action-gradient-estimator policy optimization. Advances in Neural Information Processing Systems, 33:313–324, 2020.
[20] Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, and Honglak Lee. Sample-efficient reinforcement learning with stochastic ensemble value expansion. Advances in neural information processing systems, 31, 2018.
[21] John Tsitsiklis and Benjamin Van Roy. Analysis of temporal-difference learning with function approximation. Advances in neural information processing systems, 9, 1996.
[22] Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8:279-292, 1992.
[41] David Balduzzi and Muhammad Ghifary. Compatible value gradients for reinforcement learning of continuous deep policies. arXiv preprint arXiv:1509.03005, 2015.
[42] Michael Fairbank and Eduardo Alonso. Value-gradient learning. In The 2012 international joint conference on neural networks (ijcnn), pages 1-8. IEEE, 2012.
[43] Yunhao Tang, Michal Valko, and Rémi Munos. Taylor expansion policy optimization. In International Conference on Machine Learning, pages 9397-9406. PMLR, 2020.
[44] Shiau Hong Lim and Ilyas Malik. Distributional reinforcement learning for risk-sensitive policies. Advances in Neural Information Processing Systems, 35:30977–30989, 2022.
[45] LA Prashanth, Michael C Fu, et al. Risk-sensitive reinforcement learning via policy gradient search. Foundations and Trends® in Machine Learning, 15(5):537-693, 2022.
[46] Ralph Neuneier and Oliver Mihatsch. Risk sensitive reinforcement learning. Advances in Neural Information Processing Systems, 11, 1998.
[47] Yun Shen, Michael J Tobia, Tobias Sommer, and Klaus Obermayer. Risk-sensitive reinforcement learning. Neural computation, 26(7):1298-1328, 2014.
[48] Atilim Gunes Baydin, Barak A Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. Automatic differentiation in machine learning: a survey. Journal of Marchine Learning Research, 18:1-43, 2018.
[49] Wojciech M Czarnecki, Simon Osindero, Max Jaderberg, Grzegorz Swirszcz, and Razvan Pascanu. Sobolev training for neural networks. Advances in Neural Information Processing Systems, 30, 2017.
[50] Harris Drucker and Yann Le Cun. Improving generalization performance using double backpropagation. IEEE transactions on neural networks, 3(6):991-997, 1992.
A First-order Taylor expansion
A.1 Action expansion update proof
Transposing the scalar inner product,  $\pmb{\xi}_{\mathrm{a}}^{T}\nabla_{\mathbf{a}}Q_{\theta}(\mathbf{s},\pmb{\mu}_{\mathrm{a}})$ , in the first-order Taylor expansion in the main text (Eq. 10) gives,
$$
\Delta_ {\mathrm {T a}} \theta (\mathbf {s}, \boldsymbol {\mu} _ {\mathrm {a}}) = \operatorname {E} _ {\boldsymbol {\xi} _ {\mathrm {a}}} \left[ \left(\delta_ {\theta} (\mathbf {s}, \boldsymbol {\mu} _ {\mathrm {a}}) + \boldsymbol {\xi} _ {\mathrm {a}} ^ {T} \nabla_ {\mathbf {a}} \delta_ {\theta} (\mathbf {s}, \boldsymbol {\mu} _ {\mathrm {a}})\right) \nabla_ {\theta} \left(Q _ {\theta} (\mathbf {s}, \boldsymbol {\mu} _ {\mathrm {a}}) + \left(\nabla_ {\mathbf {a}} Q _ {\theta} (\mathbf {s}, \boldsymbol {\mu} _ {\mathrm {a}})\right) ^ {T} \boldsymbol {\xi} _ {\mathrm {a}}\right) \mid \mathbf {s}, \boldsymbol {\mu} _ {\mathrm {a}} \right] \tag {25}
$$
As  $\nabla_{\theta}$  is a linear operator, and the action noise,  $\xi_{\mathrm{a}}$  does not depend on the policy parameters,  $\theta$
$$
\Delta_ {\mathrm {T a}} \theta (\mathbf {s}, \boldsymbol {\mu} _ {\mathrm {a}}) = \mathrm {E} \left[ \left(\delta_ {\theta} (\mathbf {s}, \boldsymbol {\mu} _ {\mathrm {a}}) + \boldsymbol {\xi} _ {\mathrm {a}} ^ {T} \nabla_ {\mathbf {a}} \delta_ {\theta} (\mathbf {s}, \boldsymbol {\mu} _ {\mathrm {a}})\right) \left(\nabla_ {\theta} Q _ {\theta} (\mathbf {s}, \boldsymbol {\mu} _ {\mathrm {a}}) + \nabla_ {\theta , \mathbf {a}} ^ {2} Q _ {\theta} (\mathbf {s}, \boldsymbol {\mu} _ {\mathrm {a}}) \boldsymbol {\xi} _ {\mathrm {a}}\right) | \mathbf {s}, \boldsymbol {\mu} _ {\mathrm {a}} \right] \tag {26}
$$
where  $\nabla_{\mathbf{a},\theta}^2 Q_\theta (\mathbf{s},\pmb {\mu}_\mathrm{a})$  is a matrix of second derivatives. The expectation of  $\xi_{\mathrm{a}}$  is zero, so the terms linear in  $\xi_{\mathrm{a}}$  are zero, leading to,
$$
\Delta_ {\mathrm {T a}} \theta (\mathbf {s}, \boldsymbol {\mu} _ {\mathrm {a}}) = \delta_ {\theta} (\mathbf {s}, \boldsymbol {\mu} _ {\mathrm {a}}) \nabla_ {\theta} Q _ {\theta} (\mathbf {s}, \boldsymbol {\mu} _ {\mathrm {a}}) + \mathrm {E} _ {\boldsymbol {\xi} _ {\mathrm {a}}} \left[ \left(\boldsymbol {\xi} _ {\mathrm {a}} ^ {T} \nabla_ {\mathbf {a}} \delta_ {\theta} (\mathbf {s}, \boldsymbol {\mu} _ {\mathrm {a}})\right) \left(\nabla_ {\theta , \mathbf {a}} ^ {2} Q _ {\theta} (\mathbf {s}, \boldsymbol {\mu} _ {\mathrm {a}}) \boldsymbol {\xi} _ {\mathrm {a}} ^ {T}\right) | \mathbf {s}, \boldsymbol {\mu} _ {\mathrm {a}} \right] \tag {27}
$$
Swapping the order of the terms in the expectation (which is valid, as  $\left(\pmb{\xi}_{\mathrm{a}}^{T}\nabla_{\mathbf{a}}\delta_{\theta}(\mathbf{s},\pmb{\mu}_{\mathrm{a}})\right)$  is a scalar),
$$
\Delta_ {\mathrm {T a}} \theta (\mathbf {s}, \boldsymbol {\mu} _ {\mathrm {a}}) = \delta_ {\theta} (\mathbf {s}, \boldsymbol {\mu} _ {\mathrm {a}}) \nabla_ {\theta} Q _ {\theta} (\mathbf {s}, \boldsymbol {\mu} _ {\mathrm {a}}) + \operatorname {E} _ {\boldsymbol {\xi} _ {\mathrm {a}}} \left[ \left(\nabla_ {\theta , \mathbf {a}} ^ {2} Q _ {\theta} (\mathbf {s}, \boldsymbol {\mu} _ {\mathrm {a}}) \boldsymbol {\xi} _ {\mathrm {a}}\right) \left(\boldsymbol {\xi} _ {\mathrm {a}} ^ {T} \nabla_ {\mathbf {a}} \delta_ {\theta} (\mathbf {s}, \boldsymbol {\mu} _ {\mathrm {a}})\right) | \mathbf {s}, \boldsymbol {\mu} _ {\mathrm {a}} \right] \tag {28}
$$
We can then move the terms independent of  $\xi_{\mathrm{a}}$  out of the expectation
$$
\Delta_ {\mathrm {T a}} \theta (\mathbf {s}, \boldsymbol {\mu} _ {\mathrm {a}}) = \delta_ {\theta} (\mathbf {s}, \boldsymbol {\mu} _ {\mathrm {a}}) \nabla_ {\theta} Q _ {\theta} (\mathbf {s}, \boldsymbol {\mu} _ {\mathrm {a}}) + \nabla_ {\theta , \mathbf {a}} ^ {2} Q _ {\theta} (\mathbf {s}, \boldsymbol {\mu} _ {\mathrm {a}}) \operatorname {E} _ {\boldsymbol {\xi} _ {\mathrm {a}}} \left[ \boldsymbol {\xi} _ {\mathrm {a}} \boldsymbol {\xi} _ {\mathrm {a}} ^ {T} \right] \nabla_ {\mathbf {a}} \delta_ {\theta} (\mathbf {s}, \boldsymbol {\mu} _ {\mathrm {a}}) \tag {29}
$$
Finally, we know  $\operatorname{E}\left[\pmb{\xi}_{\mathrm{a}}\pmb{\xi}_{\mathrm{a}}^{T}\right] = \pmb{\Sigma}_{\mathrm{a}},$
$$
\Delta_ {\mathrm {T a}} \theta (\mathbf {s}, \boldsymbol {\mu} _ {\mathrm {a}}) = \boldsymbol {\mu} _ {\mathrm {a}}) \nabla_ {\theta} Q _ {\theta} (\mathbf {s}, \boldsymbol {\mu} _ {\mathrm {a}}) + \nabla_ {\theta , \mathbf {a}} ^ {2} Q _ {\theta} (\mathbf {s}, \boldsymbol {\mu} _ {\mathrm {a}}) \boldsymbol {\Sigma} _ {\mathrm {a}} \nabla_ {\mathbf {a}} \delta_ {\theta} (\mathbf {s}, \boldsymbol {\mu} _ {\mathrm {a}}) \tag {30}
$$
If we assume the action covariance is isotropic,  $\Sigma_{\mathrm{a}} = \lambda_{\mathrm{a}}\mathbf{I}$ , we get the (1st-order) Taylor TD-update for the action expansion,
$$
\Delta_ {\mathrm {T a}} \theta (\mathbf {s}, \boldsymbol {\mu} _ {\mathrm {a}}) = \delta_ {\theta} (\mathbf {s}, \boldsymbol {\mu} _ {\mathrm {a}}) \nabla_ {\theta} Q _ {\theta} (\mathbf {s}, \boldsymbol {\mu} _ {\mathrm {a}}) + \lambda_ {\mathrm {a}} \nabla_ {\theta , \mathbf {a}} ^ {2} Q _ {\theta} (\mathbf {s}, \boldsymbol {\mu} _ {\mathrm {a}}) \nabla_ {\mathbf {a}} \delta_ {\theta} (\mathbf {s}, \boldsymbol {\mu} _ {\mathrm {a}}) \tag {31}
$$
A.2 State expansion update proof
Applying the first-order Taylor expansion to Eq. (16), then following the approach from the previous section,
$$
\begin{array}{l} \Delta_ {\mathrm {T a}} \theta (\boldsymbol {\mu} _ {\mathrm {s}}, \mathbf {a}) = \operatorname {E} _ {\boldsymbol {\xi} _ {\mathrm {s}}} \left[ \left(\delta_ {\theta} \left(\boldsymbol {\mu} _ {\mathrm {s}}, \mathbf {a}\right) + \boldsymbol {\xi} _ {\mathrm {s}} ^ {T} \nabla_ {\mathbf {s}} \delta_ {\theta} \left(\boldsymbol {\mu} _ {\mathrm {s}}, \mathbf {a}\right)\right) \nabla_ {\theta} \left(Q \left(\boldsymbol {\mu} _ {\mathrm {s}}, \mathbf {a}\right) + \left(\nabla_ {\mathbf {s}} Q \left(\boldsymbol {\mu} _ {\mathrm {s}}, \mathbf {a}\right)\right) ^ {T} \boldsymbol {\xi} _ {\mathrm {s}}\right) | \boldsymbol {\mu} _ {\mathrm {s}}, \mathbf {a} \right] (32) \\ = \operatorname {E} _ {\boldsymbol {\xi} _ {\mathrm {s}}} \left[ \left(\delta_ {\theta} \left(\boldsymbol {\mu} _ {\mathrm {s}}, \mathbf {a}\right) + \boldsymbol {\xi} _ {\mathrm {s}} ^ {T} \nabla_ {\mathbf {s}} \delta_ {\theta} \left(\boldsymbol {\mu} _ {\mathrm {s}}, \mathbf {a}\right)\right) \left(\nabla_ {\theta} Q _ {\theta} \left(\boldsymbol {\mu} _ {\mathrm {s}}, \mathbf {a}\right) + \nabla_ {\theta , \mathbf {s}} ^ {2} Q _ {\theta} \left(\boldsymbol {\mu} _ {\mathrm {s}}, \mathbf {a}\right) \boldsymbol {\xi} _ {\mathrm {s}}\right) | \boldsymbol {\mu} _ {\mathrm {s}}, \mathbf {a} \right] (33) \\ = \delta_ {\theta} \left(\boldsymbol {\mu} _ {\mathrm {s}}, \mathbf {a}\right) \nabla_ {\theta} Q _ {\theta} \left(\boldsymbol {\mu} _ {\mathrm {s}}, \mathbf {a}\right) + \operatorname {E} _ {\boldsymbol {\xi} _ {\mathrm {s}}} \left[ \left(\nabla_ {\theta , \mathbf {s}} ^ {2} Q _ {\theta} \left(\boldsymbol {\mu} _ {\mathrm {s}}, \mathbf {a}\right) \boldsymbol {\xi} _ {\mathrm {s}}\right) \left(\boldsymbol {\xi} _ {\mathrm {s}} ^ {T} \nabla_ {\mathbf {s}} \delta_ {\theta} \left(\boldsymbol {\mu} _ {\mathrm {s}}, \mathbf {a}\right)\right) | \boldsymbol {\mu} _ {\mathrm {s}}, \mathbf {a} \right] (34) \\ = \delta_ {\theta} \left(\boldsymbol {\mu} _ {\mathrm {s}}, \mathbf {a}\right) \nabla_ {\theta} Q _ {\theta} \left(\boldsymbol {\mu} _ {\mathrm {s}}, \mathbf {a}\right) + \nabla_ {\theta , \mathbf {s}} ^ {2} Q _ {\theta} \left(\boldsymbol {\mu} _ {\mathrm {s}}, \mathbf {a}\right) \operatorname {E} _ {\boldsymbol {\xi} _ {\mathrm {s}}} \left[ \boldsymbol {\xi} _ {\mathrm {s}} \boldsymbol {\xi} _ {\mathrm {s}} ^ {T} \right] \nabla_ {\mathbf {s}} \delta_ {\theta} \left(\boldsymbol {\mu} _ {\mathrm {s}}, \mathbf {a}\right) (35) \\ \end{array}
$$
Finally, we know  $\operatorname{E}\left[\pmb{\xi}_{\mathrm{s}}\pmb{\xi}_{\mathrm{s}}^{T}\right] = \pmb{\Sigma}_{\mathrm{s}}$
$$
\Delta_ {\mathrm {T a}} \theta (\boldsymbol {\mu} _ {\mathrm {s}}, \mathbf {a}) = \delta_ {\theta} (\boldsymbol {\mu} _ {\mathrm {s}}, \mathbf {a}) \nabla_ {\theta} Q _ {\theta} (\boldsymbol {\mu} _ {\mathrm {s}}, \mathbf {a}) + \nabla_ {\theta , \mathbf {s}} ^ {2} Q _ {\theta} (\boldsymbol {\mu} _ {\mathrm {s}}, \mathbf {a}) \boldsymbol {\Sigma} _ {\mathrm {s}} \nabla_ {\mathbf {s}} \delta_ {\theta} (\boldsymbol {\mu} _ {\mathrm {s}}, \mathbf {a}) \tag {36}
$$
If we assume the state covariance is isotropic,  $\Sigma_{\mathrm{s}} = \lambda_{\mathrm{s}}\mathbf{I}$ , we get the (1st-order) Taylor TD-update for the state expansion,
$$
\Delta_ {\mathrm {T a}} \theta (\boldsymbol {\mu} _ {\mathrm {s}}, \mathbf {a}) = \delta_ {\theta} (\boldsymbol {\mu} _ {\mathrm {s}}, \mathbf {a}) \nabla_ {\theta} Q _ {\theta} (\boldsymbol {\mu} _ {\mathrm {s}}, \mathbf {a}) + \lambda_ {s} \nabla_ {\theta , \mathbf {s}} ^ {2} Q _ {\theta} (\boldsymbol {\mu} _ {\mathrm {s}}, \mathbf {a}) \nabla_ {\mathbf {s}} \delta_ {\theta} (\boldsymbol {\mu} _ {\mathrm {s}}, \mathbf {a}) \tag {37}
$$
B Proof of stable learning for Taylor TD with linear function approximation (with a fixed policy)
The Taylor TD update for the action expansion can be written as (an equivalent proof can be derived for the state expansion):
$$
\Delta_ {\mathrm {T a}} \theta (\mathbf {s}, \boldsymbol {\mu} _ {\mathrm {a}}) = \delta_ {\theta} (\mathbf {s}, \boldsymbol {\mu} _ {\mathrm {a}}) \nabla_ {\theta} Q _ {\theta} (\mathbf {s}, \boldsymbol {\mu} _ {\mathrm {a}}) + \lambda_ {\mathrm {a}} \nabla_ {\theta , \boldsymbol {\mu} _ {\mathrm {a}}} ^ {2} Q _ {\theta} (\mathbf {s}, \boldsymbol {\mu} _ {\mathrm {a}}) \nabla_ {\mathbf {a}} \delta_ {\theta} (\mathbf {s}, \boldsymbol {\mu} _ {\mathrm {a}}) \tag {38}
$$
This update is composed of two quantities, the standard TD update (i.e., first term in the sum) plus the extra term induced by the Taylor expansion (i.e., second term in the sum). For the purposes of this proof, we consider linear function approximation,
$$
Q _ {\theta} \left(\mathbf {s}, \boldsymbol {\mu} _ {\mathrm {a}}\right) = \theta^ {T} \mathbf {x} \quad Q _ {\theta} \left(\mathbf {s} ^ {\prime}, \mathbf {a} ^ {\prime}\right) = \theta^ {T} \mathbf {x} ^ {\prime} \tag {39}
$$
$$
\mathbf {x} = \phi (\mathbf {s}, \boldsymbol {\mu} _ {\mathrm {a}}) \in \mathbb {R} ^ {N} \quad \mathbf {x} ^ {\prime} = \phi \left(\mathbf {s} ^ {\prime}, \mathbf {a} ^ {\prime}\right) \in \mathbb {R} ^ {N} \tag {40}
$$
and where  $\mathbf{x}$  and  $\mathbf{x}'$  are feature-vectors of length  $N$ . We can re-write each of the two terms in the Taylor TD update in terms of this linear function approximation. The first term, corresponding to a standard TD-update can be written,
$$
\begin{array}{l} \delta_ {\boldsymbol {\theta}} (\mathbf {s}, \mathbf {a}) \nabla_ {\boldsymbol {\theta}} Q _ {\boldsymbol {\theta}} (\mathbf {s}, \mathbf {a}) = (r + \gamma \theta^ {T} \mathbf {x} ^ {\prime} - \theta^ {T} \mathbf {x}) \mathbf {x} \\ = r \mathbf {x} - \mathbf {x} \left(\mathbf {x} - \gamma \mathbf {x} ^ {\prime}\right) ^ {T} \theta . \tag {41} \\ \end{array}
$$
The second term, which is the new term introduced by Taylor-TD methods, can be written,
$$
\begin{array}{l} \nabla_ {\theta , \mathbf {a}} ^ {2} Q _ {\theta} (\mathbf {s}, \mathbf {a}) \nabla_ {\mathbf {a}} \delta_ {\theta} (\mathbf {s}, \mathbf {a}) = \nabla_ {\theta , \mathbf {a}} \left(\mathbf {x} ^ {T} \theta\right) \left(\nabla_ {\mathbf {a}} r + \gamma \nabla_ {\mathbf {a}} \left(\mathbf {x} ^ {\prime T} \theta\right) - \nabla_ {\mathbf {a}} \left(\mathbf {x} ^ {T} \theta\right)\right) \\ = \left(\nabla_ {\mathbf {a}} \mathbf {x}\right) ^ {T} \left(\nabla_ {\mathbf {a}} r + \gamma \left(\nabla_ {\mathbf {a}} \mathbf {x} ^ {\prime}\right) \theta - \left(\nabla_ {\mathbf {a}} \mathbf {x}\right) \theta\right) \\ = (\nabla_ {\mathbf {a}} \mathbf {x}) ^ {T} \nabla_ {\mathbf {a}} r + \gamma (\nabla_ {\mathbf {a}} \mathbf {x}) ^ {T} \nabla_ {\mathbf {a}} \mathbf {x} ^ {\prime} \theta - (\nabla_ {\mathbf {a}} \mathbf {x}) ^ {T} \nabla_ {\mathbf {a}} \mathbf {x} \theta \\ = \left(\nabla_ {\mathbf {a}} \mathbf {x}\right) ^ {T} \nabla_ {\mathbf {a}} r - \left(\nabla_ {\mathbf {a}} \mathbf {x}\right) ^ {T} \left(\nabla_ {\mathbf {a}} \mathbf {x} - \gamma \nabla_ {\mathbf {a}} \mathbf {x} ^ {\prime}\right) \theta \tag {42} \\ \end{array}
$$
Here,  $\nabla_{\mathbf{a}}\mathbf{x}\in \mathbb{R}^{A\times N}$ , is a matrix, while  $\nabla_{\mathbf{a}}r\in \mathbb{R}^{A}$  is a vector.
Putting the two terms together Eq. 41 & 42 and factorising the terms multiplying  $\theta_{t}$ , we can write the expected next weight vector as:
$$
\operatorname {E} \left[ \theta_ {t + 1} \mid \theta_ {t} \right] = \theta_ {t} + \eta \Delta \theta = (\mathbf {I} - \eta (\mathbf {A} + \lambda_ {\mathrm {a}} \tilde {\mathbf {A}})) \theta_ {t} + \eta \mathbf {u} \tag {43}
$$
$$
\mathbf {u} = \operatorname {E} \left[ r \mathbf {x} + \left(\nabla_ {\mathbf {a}} \mathbf {x}\right) ^ {T} \nabla_ {a} r \right] \tag {44}
$$
$$
\mathbf {A} = \operatorname {E} \left[ \mathbf {x} \left(\mathbf {x} - \gamma \mathbf {x} ^ {\prime}\right) ^ {T} \right] \in \mathbb {R} ^ {N \times N} \tag {45}
$$
$$
\tilde {\mathbf {A}} = \operatorname {E} \left[ \left(\nabla_ {\mathbf {a}} \mathbf {x}\right) ^ {T} \left(\nabla_ {\mathbf {a}} \mathbf {x} - \gamma \nabla_ {\mathbf {a}} \mathbf {x} ^ {\prime}\right) \right] \in \mathbb {R} ^ {N \times N} \tag {46}
$$
Since only  $\mathbf{A}$  and  $\tilde{\mathbf{A}}$  multiplies  $\theta$ , these two quantities exclusively are important for guaranteeing stable learning. If  $\mathbf{A}$  is positive definite and  $\tilde{\mathbf{A}}$  is positive-semi-definite, then  $(\mathbf{A} + \lambda_{\mathrm{a}}\tilde{\mathbf{A}})$  is positive-definite, and for sufficiently small  $\eta$ , the magnitude of the eigenvalues of  $\mathbf{I} - \eta (\mathbf{A} + \lambda_{\mathrm{a}}\tilde{\mathbf{A}})$  are all smaller than 1, in which case the system is stable. Crucially, the term  $\mathbf{A}$  is the same as traditional TD-learning so [11] provides a proof that  $\mathbf{A}$  is always positive definite [see also 12]. Thus, all we have to prove is that  $\tilde{\mathbf{A}}$  is positive semi-definite. In order to prove that  $\tilde{\mathbf{A}}$  is positive semi-definite, we require an (very reasonable) assumption, that the timestep  $\Delta t$  is small. Specifically, if we take,
$$
\mathbf {s} ^ {\prime} = \Delta t f (\mathbf {s}, \mathbf {a}) + \mathbf {s} \tag {47}
$$
Then the  $\nabla_{\mathbf{a}}\mathbf{x}^{\prime}$  terms must be small. Specifically, applying the chain rule, the full derivative is (expressed here for 1d case,  $\nabla_{\mathbf{a}}\mathbf{x}^{\prime} = \frac{\partial x^{\prime}}{\partial a}$ , for simplicity),
$$
\frac {\partial x ^ {\prime}}{\partial a} = \frac {\partial x ^ {\prime}}{\partial a ^ {\prime}} \frac {\partial a ^ {\prime}}{\partial s ^ {\prime}} \frac {\partial s ^ {\prime}}{\partial a} + \frac {\partial x ^ {\prime}}{\partial s ^ {\prime}} \frac {\partial s ^ {\prime}}{\partial a} \frac {\partial x ^ {\prime}}{\partial a} = \left(\frac {\partial x ^ {\prime}}{\partial a ^ {\prime}} \frac {\partial a ^ {\prime}}{\partial s ^ {\prime}} + \frac {\partial x ^ {\prime}}{\partial s ^ {\prime}}\right) \frac {\partial s ^ {\prime}}{\partial a} \tag {48}
$$
This is proportional to  $\Delta t$ , as
$$
\frac {\partial s ^ {\prime}}{\partial a} = \Delta t \frac {\partial f (s , a)}{\partial a} \tag {49}
$$
The key test for positive semi-definiteness is that for all vectors  $\mathbf{b}$
$$
0 \leq \mathbf {b} ^ {T} \tilde {\mathbf {A}} \mathbf {b}. \tag {50}
$$
Substituting the definition of  $\tilde{\mathbf{A}}$  from Eq. (46),
$$
0 \leq \mathbf {b} ^ {T} E \left[ \left(\nabla_ {\mathbf {a}} \mathbf {x}\right) ^ {T} \nabla_ {\mathbf {a}} \mathbf {x} \right] \mathbf {b} - \gamma \mathbf {b} ^ {T} E \left[ \left(\nabla_ {\mathbf {a}} \mathbf {x}\right) ^ {T} \nabla_ {\mathbf {a}} \mathbf {x} ^ {\prime} \right] \mathbf {b}. \tag {51}
$$
Putting the b's inside the expectation,
$$
0 \leq E \left[ \left(\nabla_ {\mathbf {a}} \mathbf {x} \mathbf {b}\right) ^ {T} \left(\nabla_ {\mathbf {a}} \mathbf {x} \mathbf {b}\right) \right] - \gamma E \left[ \left(\nabla_ {\mathbf {a}} \mathbf {x} \mathbf {b}\right) ^ {T} \left(\nabla_ {\mathbf {a}} \mathbf {x} ^ {\prime} \mathbf {b}\right) \right] \tag {52}
$$
Now,  $(\nabla_{\mathbf{a}}\mathbf{x}\mathbf{b})\in \mathbb{R}^{1\times A}$  is a length-  $A$  row-vector, and the terms inside the expectations are inner products of two length  $A$  vectors. We can write these vector inner products as sums,
$$
0 \leq \sum_ {i = 1} ^ {A} E \left[ \left(\nabla_ {a _ {i}} \mathbf {x b}\right) ^ {T} \left(\nabla_ {a _ {i}} \mathbf {x b}\right) \right] - \gamma E \left[ \left(\nabla_ {a _ {i}} \mathbf {x b}\right) ^ {T} \left(\nabla_ {a _ {i}} \mathbf {x} ^ {\prime} \mathbf {b}\right) \right] \tag {53}
$$
where  $a_{i}$  is a particular element of the action-vector,  $\mathbf{a}$ , so  $\nabla_{a_i}\mathbf{x} \in \mathbb{R}^N$  is a length- $\mathbf{N}$  vector, and
$$
\nabla_ {\mathbf {a}} \mathbf {x} = \left( \begin{array}{c} \nabla_ {a _ {1}} \mathbf {x} \\ \nabla_ {a _ {2}} \mathbf {x} \\ \vdots \\ \nabla_ {a _ {A}} \mathbf {x} \end{array} \right) \tag {54}
$$
Critically, the overall inequality in Eq. (52) holds if a similar inequality holds for every term in the sum in Eq. (53),
$$
0 \leq E \left[ \left(\nabla_ {a _ {i}} \mathbf {x b}\right) ^ {T} \left(\nabla_ {a _ {i}} \mathbf {x b}\right) \right] - \gamma E \left[ \left(\nabla_ {a _ {i}} \mathbf {x b}\right) ^ {T} \left(\nabla_ {a _ {i}} \mathbf {x} ^ {\prime} \mathbf {b}\right) \right] \tag {55}
$$
As  $\nabla_{a_i}\mathbf{x}\mathbf{b}$  and  $\nabla_{a_i}\mathbf{x}'\mathbf{b}$  are scalars, we can write,
$$
0 \leq E \left[ \left(\nabla_ {a _ {i}} \mathbf {x b}\right) ^ {2} \right] - \gamma E \left[ \left(\nabla_ {a _ {i}} \mathbf {x b}\right) \left(\nabla_ {a _ {i}} \mathbf {x} ^ {\prime} \mathbf {b}\right) \right] \tag {56}
$$
There are now two cases. If  $0 = E[(\nabla_{a_i}\mathbf{x}\mathbf{b})^2]$ , we must have that  $0 = (\nabla_{a_i}\mathbf{x}\mathbf{b})$  always (except at a set of measure zero). Thus, the second term must also be zero (except at a set of measure zero), i.e.  $0 = E[(\nabla_{a_i}\mathbf{x}\mathbf{b})(\nabla_{a_i}\mathbf{x}'\mathbf{b})]$  and the inequality holds. Alternatively, if  $E[(\nabla_{a_i}\mathbf{x}\mathbf{b})^2]$  is non-zero, it must be positive, in that case, the second term can also be non-zero, but it scales with  $\Delta t$ , so we can always choose a  $\Delta t$ , small enough to ensure that the Eq. (56) holds, in which case  $\tilde{\mathbf{A}}$  is positive semi-definite.
C Using the chain rule to expand the gradient of Taylor TD
A (learned) differentiable model of the transitions and rewards is needed to compute the gradient terms  $\nabla_{\mathbf{a}}\delta_{\theta}(\mathbf{s},\mathbf{a})$  and  $\nabla_{\mathbf{s}}\delta_{\theta}(\mathbf{s},\mathbf{a})$  in Taylor TD. This is because the TD target,  $\delta_{\theta} = r(\mathbf{s},\mathbf{a}) + \gamma Q_{\theta}(\mathbf{s}',\mathbf{a}')$ , comprises the reward and the Q-value at the next time step, both of which depend on  $\mathbf{s}$  and  $\mathbf{a}$ . The full gradients for the action expansion can be written: