SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Case 2: Overparameterized Regime Alternatively, if the feature dimension exceeds the total number of states, then the fixed point equation for TD is underspecified. However, throughout TD learning  $\mathbf{w}_{TD} \in \operatorname{span}\{\psi(s)\}_{s \in S}$  so we can instead consider the decomposition  $\mathbf{w}_V = \sum_s \alpha(s) \psi(s)$ , where  $\alpha \in \mathbb{R}^{|S|}$  satisfies
$$
\operatorname {d i a g} (\boldsymbol {p}) (\boldsymbol {I} - \gamma \boldsymbol {\Pi}) \boldsymbol {K} \alpha = \operatorname {d i a g} (\boldsymbol {p}) \boldsymbol {R} \tag {A.4}
$$
where  $\pmb{K} \in \mathbb{R}^{|S| \times |S|}$  is the kernel computed with features  $K(s, s') = \psi(s) \cdot \psi(s')$ . The solution to the above equation is unique and the learned value function  $\hat{\pmb{V}} = \pmb{\Psi}^\top \pmb{w}_{TD} = \pmb{K} \pmb{K}^{-1} (\pmb{I} - \gamma \pmb{\Pi})^{-1} \pmb{R} = (\pmb{I} - \gamma \pmb{\Pi})^{-1} \pmb{R} = \pmb{V}$ . Therefore, in the over-parameterized limit, the irreducible value error for TD learning is zero. This limit was considered dynamically in the infinite batch (vanishing SGD noise) setting by [44].
B Derivation of Learning Curves
In this section, we now consider the dynamics of TD learning when  $B$  random episodes are sampled at a time. In this calculation, the finite batch of episodes leads to non-negligible SGD effects which can cause undesirable plateaus in TD dynamics.
B.1 Field Theory Derivation
In this section we use a Gaussian field theory formalism to compute the learning curve in the high dimensional asymptotic limit  $N, B \to \infty$  with  $B / N = \alpha$ . The episode length  $T$  is treated as  $\mathcal{O}(1)$ . While this paper focuses on the online setting, where fresh trajectories  $\{\tau_n^\mu\}$  are sampled at each iteration  $n$ , this model can be straightforwardly extended to the case where a fixed number of experience trajectories  $\{\tau^\mu\}$  are replayed repeatedly during TD learning. We leave the experience
replay dynamic mean field theory calculation for future work. The starting point of our analysis is tracking the moment generating function for the iterate dynamics
$$
Z \left[ \left\{\boldsymbol {j} _ {n} \right\} \right] = \mathbb {E} _ {\left\{\boldsymbol {w} _ {n} \right\}, \left\{s _ {n} ^ {\mu} (t) \right\}} \exp \left(i \sum_ {n = 0} ^ {\infty} \boldsymbol {j} _ {n} \cdot \boldsymbol {w} _ {n}\right). \tag {B.1}
$$
To compute this object over random draws of training trajectories, we express the joint average over  $\pmb{w}_n$ ,  $\{s_n^\mu(t)\}$  into conditional averages over  $\pmb{w}_n$ ,  $\{\Delta_n^\mu(t)\}$  or  $\{\psi_n^\mu(t)\}$ . To simplify the computation, in this section, we will compute the learning curve for mean zero features  $\pmb{\mu}(s) = 0$  and
$$
\begin{array}{l} Z = \mathbb {E} _ {\left\{\boldsymbol {\psi} _ {n} ^ {\mu} (t) \right\}} \int \prod_ {n} d \boldsymbol {w} _ {n} \delta \left(\boldsymbol {w} _ {n + 1} - \boldsymbol {w} _ {n} - \frac {\eta}{\sqrt {B T}} \sum_ {\mu t} \Delta_ {n} ^ {\mu} (t) \boldsymbol {\psi} _ {n} ^ {\mu} (t)\right) \exp \left(i \sum_ {n = 0} ^ {\infty} \boldsymbol {j} _ {n} \cdot \boldsymbol {w} _ {n}\right) \\ \times \int_ {t \mu n} d \Delta_ {n} ^ {\mu} (t) \delta \left(\Delta_ {n} ^ {\mu} (t) - \frac {1}{\sqrt {N}} \left(\boldsymbol {w} _ {R} - \boldsymbol {w} _ {n}\right) \cdot \boldsymbol {\psi} _ {n} ^ {\mu} (t) - \frac {\gamma}{\sqrt {N}} \boldsymbol {w} _ {n} \cdot \boldsymbol {\psi} _ {n} ^ {\mu} (t + 1)\right) \tag {B.2} \\ \end{array}
$$
Expressing the Dirac-delta function as a Fourier integral  $\delta (z) = \int \frac{d\hat{z}}{2\pi}\exp (i\hat{z} z)$  for each of our constraints. Under the Gaussian equivalence ansatz, we can easily average over Gaussian  $\psi$  to obtain
$$
\begin{array}{l} Z = \int \mathcal {D} \Delta \hat {\mathcal {D}} \boldsymbol {\mathcal {D}} \boldsymbol {w} \mathcal {D} \hat {\boldsymbol {w}} \exp \left(- \frac {\eta^ {2}}{2 B T ^ {2}} \sum_ {n \mu} \sum_ {t t ^ {\prime}} \Delta_ {n} ^ {\mu} (t) \Delta_ {n} ^ {\mu} (t ^ {\prime}) \hat {\boldsymbol {w}} _ {n} ^ {\top} \boldsymbol {\Sigma} (t, t ^ {\prime}) \hat {\boldsymbol {w}} _ {n}\right) \\ \exp \left(i \sum_ {n} \hat {\boldsymbol {w}} _ {n} \cdot (\boldsymbol {w} _ {n + 1} - \boldsymbol {w} _ {n})\right) \\ \exp \left(- \frac {1}{2 N} \sum_ {n \mu t t ^ {\prime}} \left[ \left(\boldsymbol {w} _ {R} - \boldsymbol {w} _ {n}\right) \hat {\Delta} _ {n} ^ {\mu} (t) \right] \boldsymbol {\Sigma} (t, t ^ {\prime}) \left[ \left(\boldsymbol {w} _ {R} - \boldsymbol {w} _ {n}\right) \hat {\Delta} _ {n} ^ {\mu} (t ^ {\prime}) \right]\right) \\ \exp \left(- \frac {\gamma^ {2}}{2 N} \sum_ {n \mu t t ^ {\prime}} \hat {\Delta} _ {n} ^ {\mu} (t - 1) \hat {\Delta} _ {n} ^ {\mu} (t ^ {\prime} - 1) \boldsymbol {w} _ {n} ^ {\top} \boldsymbol {\Sigma} (t, t ^ {\prime}) \boldsymbol {w} _ {n}\right) \\ \exp \left(- \frac {\gamma}{N} \sum_ {n \mu t t ^ {\prime}} \hat {\Delta} _ {n} ^ {\mu} (t - 1) \hat {\Delta} _ {n} ^ {\mu} (t ^ {\prime}) \boldsymbol {w} _ {n} ^ {\top} \boldsymbol {\Sigma} (t, t ^ {\prime}) (\boldsymbol {w} _ {R} - \boldsymbol {w} _ {n})\right) \\ \exp \left(- \frac {\eta}{\sqrt {N B T}} \sum_ {n \mu t t ^ {\prime}} \left[ \hat {\Delta} _ {n} ^ {\mu} (t) \left(\boldsymbol {w} _ {R} - \boldsymbol {w} _ {n}\right) + \gamma \hat {\Delta} _ {n} ^ {\mu} (t - 1) \boldsymbol {w} _ {n} \right] ^ {\top} \boldsymbol {\Sigma} (t, t ^ {\prime}) \hat {\boldsymbol {w}} _ {n} \Delta_ {n} ^ {\mu} \left(t ^ {\prime}\right)\right) \\ \exp \left(i \sum_ {n \mu t} \hat {\Delta} _ {n} ^ {\mu} (t) \Delta_ {n} ^ {\mu} (t) + i \sum_ {n} \boldsymbol {j} _ {n} \cdot \boldsymbol {w} _ {n}\right) \tag {B.3} \\ \end{array}
$$
where we adopted the shorthand  $\mathcal{D}\Delta = \prod_{\mu ,n,t}d\Delta_n^\mu (t)$  for the measure for the collection of variables  $\{\Delta_n^\mu (t)\}$ . Likewise one should interpret  $\mathcal{D}\boldsymbol {w} = \prod_{n}d\boldsymbol{w}_{n}$ . To analyze the high dimensional limit of the above moment generating function, we introduce order parameters for the theory
$$
\begin{array}{l} Q _ {n} (t, t ^ {\prime}) = \frac {1}{B} \sum_ {\mu = 1} ^ {B} \Delta_ {n} ^ {\mu} (t) \Delta_ {n} ^ {\mu} (t ^ {\prime}), C _ {n} (t, t ^ {\prime}) = \frac {1}{N} \boldsymbol {w} _ {n} ^ {\top} \boldsymbol {\Sigma} (t, t ^ {\prime}) \boldsymbol {w} _ {n} \\ C _ {n} ^ {R} (t, t ^ {\prime}) = \frac {1}{N} \boldsymbol {w} _ {R} \boldsymbol {\Sigma} (t, t ^ {\prime}) \boldsymbol {w} _ {n}, D _ {n} (t, t ^ {\prime}) = - \frac {i}{N} \hat {\boldsymbol {w}} _ {n} ^ {\top} \boldsymbol {\Sigma} (t, t ^ {\prime}) \boldsymbol {w} _ {n}, D _ {n} ^ {R} (t, t ^ {\prime}) = - \frac {i}{N} \hat {\boldsymbol {w}} _ {n} ^ {\top} \boldsymbol {\Sigma} (t, t ^ {\prime}) \boldsymbol {w} _ {R} \tag {B.4} \\ \end{array}
$$
For each of these order parameters, we enforce the definition of the order parameter using the Fourier representation of a Dirac-delta function
$$
\begin{array}{l} 1 = B \int d Q _ {n} (t, t ^ {\prime}) \delta \left(B Q _ {n} (t, t ^ {\prime}) - \sum_ {\mu} \Delta_ {n} ^ {\mu} (t) \Delta_ {n} ^ {\mu} (t ^ {\prime})\right) \\ = B \int \frac {d Q _ {n} \left(t , t ^ {\prime}\right) d \hat {Q} _ {n} \left(t , t ^ {\prime}\right)}{4 \pi i} \exp \left(\frac {B}{2} \hat {Q} _ {n} \left(t, t ^ {\prime}\right) Q _ {n} \left(t, t ^ {\prime}\right) - \frac {1}{2} \sum_ {\mu} \Delta_ {n} ^ {\mu} (t) \Delta_ {n} ^ {\mu} \left(t ^ {\prime}\right) \hat {Q} _ {n} \left(t, t ^ {\prime}\right)\right). \tag {B.5} \\ \end{array}
$$
Repeating this procedure for all order parameters  $q = \{Q, \hat{Q}, C, \hat{C}, C^R, \hat{C}^R, D, \hat{D}, D^R, \hat{D}^R\}$  and disregarding irrelevant prefactors, we have the following formula for the moment generating function
$$
Z \propto \int \mathcal {D} q \exp \left(\frac {N}{2} S [ q ]\right) \tag {B.6}
$$
where the action  $S$  has the form
$$
S = \sum_ {n} \sum_ {t t ^ {\prime}} \left[ \alpha Q _ {n} (t, t ^ {\prime}) \hat {Q} _ {n} (t, t ^ {\prime}) + C _ {n} (t, t ^ {\prime}) \hat {C} _ {n} (t, t ^ {\prime}) + C _ {n} ^ {R} (t, t ^ {\prime}) \hat {C} _ {n} ^ {R} (t, t ^ {\prime}) \right]
$$
$$
- 2 \sum_ {n} \sum_ {t t ^ {\prime}} \left[ D _ {n} (t, t ^ {\prime}) \hat {D} _ {n} (t, t ^ {\prime}) + D _ {n} ^ {R} (t, t ^ {\prime}) \hat {D} _ {n} ^ {R} (t, t ^ {\prime}) \right] + \frac {2}{N} \ln \mathcal {Z} _ {w} + 2 \alpha \ln \mathcal {Z} _ {\Delta}
$$
$$
\begin{array}{l} \mathcal {Z} _ {w} = \int \mathcal {D} \boldsymbol {w} \mathcal {D} \hat {\boldsymbol {w}} \exp \left(- \frac {\eta^ {2}}{2 T ^ {2}} \sum_ {n t t ^ {\prime}} Q _ {n} (t, t ^ {\prime}) \hat {\boldsymbol {w}} _ {n} ^ {\top} \boldsymbol {\Sigma} (t, t ^ {\prime}) \hat {\boldsymbol {w}} _ {n} + i \sum_ {n} \hat {\boldsymbol {w}} _ {n} \cdot (\boldsymbol {w} _ {n + 1} - \boldsymbol {w} _ {n})\right) \\ \exp \left(- \frac {1}{2} \sum_ {n t t ^ {\prime}} \hat {C} _ {n} (t, t ^ {\prime}) \boldsymbol {w} _ {n} ^ {\top} \boldsymbol {\Sigma} (t, t ^ {\prime}) \boldsymbol {w} _ {n} - \frac {1}{2} \hat {C} _ {n} ^ {R} (t, t ^ {\prime}) \boldsymbol {w} _ {R} ^ {\top} \boldsymbol {\Sigma} (t, t ^ {\prime}) \boldsymbol {w} _ {n}\right) \\ \exp \left(- i \sum_ {n t t ^ {\prime}} \hat {D} _ {n} (t, t ^ {\prime}) \hat {\pmb {w}} _ {n} ^ {\top} \pmb {\Sigma} (t, t ^ {\prime}) \pmb {w} _ {n} - i \sum_ {n t t ^ {\prime}} \hat {D} _ {n} ^ {R} (t, t ^ {\prime}) \hat {\pmb {w}} _ {n} ^ {\top} \pmb {\Sigma} (t, t ^ {\prime}) \pmb {w} _ {R}\right) \\ \end{array}
$$
$$
\begin{array}{l} \mathcal {Z} _ {\Delta} = \int \mathcal {D} \Delta \mathcal {D} \hat {\Delta} \exp \left(- \frac {1}{2} \sum_ {n t t ^ {\prime}} \hat {Q} _ {n} (t, t ^ {\prime}) \Delta_ {n} (t) \Delta_ {n} (t ^ {\prime}) + i \sum_ {n t} \hat {\Delta} _ {n} (t) \Delta_ {n} (t)\right) \\ \exp \left(- \frac {1}{2} \sum_ {n t t ^ {\prime}} \hat {\Delta} _ {n} (t) \hat {\Delta} _ {n} \left(t ^ {\prime}\right) \left[ \frac {1}{N} \boldsymbol {w} _ {R} ^ {\top} \boldsymbol {\Sigma} \left(t, t ^ {\prime}\right) \boldsymbol {w} _ {R} + C (t, t ^ {\prime}) \right]\right) \\ \exp \left(\frac {1}{2} \sum_ {n t t ^ {\prime}} \hat {\Delta} _ {n} (t) \hat {\Delta} _ {n} \left(t ^ {\prime}\right) \left[ C ^ {R} \left(t, t ^ {\prime}\right) + C ^ {R} \left(t ^ {\prime}, t\right) \right]\right) \\ \exp \left(- \gamma \sum_ {t, t ^ {\prime}} \hat {\Delta} _ {n} (t) \hat {\Delta} _ {n} (t ^ {\prime} - 1) C _ {n} ^ {R} (t, t ^ {\prime})\right) \\ \exp \left(- \frac {\gamma^ {2}}{2} \sum_ {t, t ^ {\prime}} \hat {\Delta} _ {n} (t - 1) \hat {\Delta} _ {n} \left(t ^ {\prime} - 1\right) C _ {n} \left(t, t ^ {\prime}\right)\right) \\ \exp \left(- \frac {\eta i}{\sqrt {\alpha} T} \sum_ {n t, t ^ {\prime}} \hat {\Delta} _ {n} (t) \left[ D _ {n} ^ {R} \left(t ^ {\prime}, t\right) - D _ {n} \left(t ^ {\prime}, t\right) + \gamma D _ {n} \left(t ^ {\prime}, t + 1\right) \right] \Delta_ {n} \left(t ^ {\prime}\right)\right) \tag {B.7} \\ \end{array}
$$
The function  $\mathcal{Z}$  has the interpretation of an effective partition function conditional on order parameters  $q$ . To study the  $N \to \infty$  limit, we use the steepest descent method and analyze the saddle point
$\frac{\partial S}{\partial q} = 0$  . These saddle point equations give
$$
\frac {\partial S}{\partial \hat {Q} _ {n} (t , t ^ {\prime})} = \alpha Q _ {n} (t, t ^ {\prime}) - \alpha \left<   \Delta_ {n} (t) \Delta_ {n} (t ^ {\prime}) \right> = 0
$$
$$
\frac {\partial S}{\partial Q _ {n} (t , t ^ {\prime})} = \alpha \hat {Q} _ {n} (t, t ^ {\prime}) - \frac {\eta^ {2}}{T ^ {2} N} \left<   \hat {\pmb {w}} _ {n} ^ {\top} \pmb {\Sigma} (t, t ^ {\prime}) \hat {\pmb {w}} _ {n} \right> = 0
$$
$$
\frac {\partial S}{\partial \hat {C} _ {n} (t , t ^ {\prime})} = C _ {n} (t, t ^ {\prime}) - \frac {1}{N} \left\langle \boldsymbol {w} _ {n} ^ {\top} \boldsymbol {\Sigma} (t, t ^ {\prime}) \boldsymbol {w} _ {n} \right\rangle = 0
$$
$$
\frac {\partial S}{\partial C _ {n} (t , t ^ {\prime})} = \hat {C} _ {n} (t, t ^ {\prime}) - \alpha \left\langle \hat {\Delta} _ {n} (t) \hat {\Delta} _ {n} (t ^ {\prime}) + \gamma^ {2} \hat {\Delta} _ {n} (t - 1) \hat {\Delta} _ {n} (t ^ {\prime} - 1) \right\rangle = 0
$$
$$
\frac {\partial S}{\partial \hat {C} _ {n} ^ {R} (t , t ^ {\prime})} = C _ {n} ^ {R} (t, t ^ {\prime}) - \frac {1}{N} \left\langle \boldsymbol {w} _ {R} ^ {\top} \boldsymbol {\Sigma} (t, t ^ {\prime}) \boldsymbol {w} _ {n} \right\rangle = 0
$$
$$
\frac {\partial S}{\partial C _ {n} (t , t ^ {\prime})} = \hat {C} _ {n} (t, t ^ {\prime}) - \alpha \left\langle \hat {\Delta} _ {n} (t) \hat {\Delta} _ {n} (t ^ {\prime}) + \gamma \hat {\Delta} _ {n} (t) \hat {\Delta} _ {n} (t ^ {\prime} - 1) \right\rangle = 0
$$
$$
\frac {\partial S}{\partial \hat {D} _ {n} (t , t ^ {\prime})} = - 2 D _ {n} (t, t ^ {\prime}) - \frac {2 i}{N} \left\langle \hat {\boldsymbol {w}} _ {n} ^ {\top} \boldsymbol {\Sigma} (t, t ^ {\prime}) \boldsymbol {w} _ {n} \right\rangle = 0
$$
$$
\frac {\partial S}{\partial \hat {D} _ {n} ^ {R} (t , t ^ {\prime})} = - 2 D _ {n} ^ {R} (t, t ^ {\prime}) - \frac {2 i}{N} \left\langle \hat {\boldsymbol {w}} _ {n} ^ {\top} \boldsymbol {\Sigma} (t, t ^ {\prime}) \boldsymbol {w} _ {n} \right\rangle = 0
$$
$$
\frac {\partial S}{\partial D _ {n} (t , t ^ {\prime})} = - 2 \hat {D} _ {n} (t, t ^ {\prime}) - \frac {2 \alpha \eta i}{\sqrt {\alpha} T} \left<   \gamma \hat {\Delta} _ {n} (t - 1) \Delta_ {n} (t ^ {\prime}) - \hat {\Delta} _ {n} (t) \Delta_ {n} (t ^ {\prime}) \right> = 0
$$
$$
\frac {\partial S}{\partial D _ {n} ^ {R} \left(t , t ^ {\prime}\right)} = - 2 \hat {D} _ {n} ^ {R} \left(t, t ^ {\prime}\right) - \frac {2 \alpha \eta i}{\sqrt {\alpha} T} \left\langle \hat {\Delta} _ {n} (t) \Delta_ {n} \left(t ^ {\prime}\right) \right\rangle = 0 \tag {B.8}
$$
The brackets  $\langle \rangle$  denote averaging over the stochastic processes defined by moment generating functions  $\mathcal{Z}_{\Delta}$ ,  $\mathcal{Z}_w$ . After these saddle point equations are solved the order parameters  $q$  are treated as non-random and a Hubbard-Stratonovich transformation is employed. For example,
$$
\exp \left(- \frac {1}{2} \hat {\boldsymbol {w}} _ {n} \left[ \frac {\eta^ {2}}{T ^ {2}} \sum_ {t t ^ {\prime}} Q _ {n} \left(t, t ^ {\prime}\right) \boldsymbol {\Sigma} \left(t, t ^ {\prime}\right) \right] \hat {\boldsymbol {w}} _ {n}\right) = \mathbb {E} _ {\boldsymbol {u} _ {n} ^ {w}} \exp \left(i \sum_ {n} \boldsymbol {u} _ {n} ^ {w} \cdot \hat {\boldsymbol {w}} _ {n}\right) \tag {B.9}
$$
where the average is over  $\pmb{u}_n^w\sim \mathcal{N}\left(0,\eta^2 T^{-2}\sum_{tt'}Q_n(t,t')\pmb {\Sigma}(t,t')\right)$ . After introducing these Hubbard fields  $\pmb{u}_n^w$  and  $u_{n}^{\Delta}(t)$ , we can perform the integrals over  $\hat{\pmb{w}}_n$  and  $\hat{\Delta}_n(t)$  which collapse to Dirac-Delta functions. The resulting identities of the delta functions define the following stochastic processes on  $\pmb{w}_n$  and  $u_{n}^{\Delta}$
$$
\boldsymbol {w} _ {n + 1} = \boldsymbol {w} _ {n} + \boldsymbol {u} _ {n} ^ {w} + \sum_ {t t ^ {\prime}} \hat {D} _ {n} ^ {R} (t, t ^ {\prime}) \boldsymbol {\Sigma} (t, t ^ {\prime}) \boldsymbol {w} _ {R} + \sum_ {t, t ^ {\prime}} \hat {D} _ {n} (t, t ^ {\prime}) \boldsymbol {\Sigma} (t, t ^ {\prime}) \boldsymbol {w} _ {n}
$$
$$
\Delta_ {n} (t) = u _ {n} ^ {\Delta} (t) + \frac {\eta}{\sqrt {\alpha} T} \sum_ {t t ^ {\prime}} \left[ D _ {n} ^ {R} \left(t, t ^ {\prime}\right) - D _ {n} \left(t, t ^ {\prime}\right) - \gamma D _ {n} \left(t ^ {\prime}, t + 1\right) \right] \Delta_ {n} \left(t ^ {\prime}\right). \tag {B.10}
$$
Using a similar trick, we can show that for any observable depending on  $\pmb{w}_n$  or  $\{\Delta_n(t)\}$  that
$$
\begin{array}{l} - i \left\langle \hat {\boldsymbol {w}} _ {n} O (\boldsymbol {w} _ {n}) \right\rangle = \left\langle \frac {\partial}{\partial \boldsymbol {u} _ {n}} O (\boldsymbol {w} _ {n}) \right\rangle \\ - i \left\langle \hat {\Delta} _ {n} (t) O \left(\left\{\Delta_ {n} \left(t ^ {\prime}\right) \right\}\right) \right\rangle = \left\langle \frac {\partial}{\partial u _ {n} ^ {\Delta} (t)} O \left(\left\{\Delta_ {n} \left(t ^ {\prime}\right) \right\}\right) \right\rangle \tag {B.11} \\ \end{array}
$$
Since  $\pmb{w}_n$  is independent. This can be used to conclude
$$
D _ {n} \left(t, t ^ {\prime}\right) = 0, D _ {n} ^ {R} \left(t, t ^ {\prime}\right) = 0 \tag {B.12}
$$
which implies that  $\Delta_n(t) = u_n^\Delta(t)$ . Consequently the response functions have trivial structure
$$
\hat {D} _ {n} (t) = - \frac {\eta \sqrt {\alpha}}{T} [ \delta (t - t ^ {\prime}) - \gamma \delta (t - 1 - t ^ {\prime}) ], \hat {D} _ {n} ^ {R} (t, t ^ {\prime}) = \frac {\sqrt {\alpha} \eta}{T} \delta (t - t ^ {\prime}). \tag {B.13}
$$
We therefore obtain a stochastic process of the form
$$
\boldsymbol {w} _ {n + 1} = \boldsymbol {w} _ {n} + \boldsymbol {u} _ {n} ^ {w} + \frac {\eta \sqrt {\alpha}}{T} \sum_ {t} \boldsymbol {\Sigma} (t, t) \boldsymbol {w} _ {R} - \frac {\eta \sqrt {\alpha}}{T} \sum_ {t} [ \boldsymbol {\Sigma} (t, t) - \gamma \boldsymbol {\Sigma} (t, t + 1) ] \boldsymbol {w} _ {n}
$$
$$
\boldsymbol {u} _ {n} \sim \mathcal {N} \left(0, \frac {\eta^ {2}}{T ^ {2}} \sum_ {t t ^ {\prime}} Q _ {n} (t, t ^ {\prime}) \boldsymbol {\Sigma} (t, t ^ {\prime})\right), \{\Delta_ {n} (t) \} \sim \mathcal {N} (0, \boldsymbol {Q} _ {n})
$$
$$
Q _ {n} (t, t ^ {\prime}) = \left\langle \Delta_ {n} (t) \Delta_ {n} (t ^ {\prime}) \right\rangle = \frac {1}{N} \boldsymbol {w} _ {R} \boldsymbol {\Sigma} (t, t ^ {\prime}) \boldsymbol {w} _ {R} - C ^ {R} (t, t ^ {\prime}) - C ^ {R} (t ^ {\prime}, t) + C (t, t ^ {\prime})
$$
$$
C _ {n} (t, t ^ {\prime}) = \frac {1}{N} \left\langle \boldsymbol {w} _ {n} ^ {\top} \boldsymbol {\Sigma} (t, t ^ {\prime}) \boldsymbol {w} _ {n} \right\rangle , C _ {n} ^ {R} (t, t ^ {\prime}) = \frac {1}{N} \left\langle \boldsymbol {w} _ {R} ^ {\top} \boldsymbol {\Sigma} (t, t ^ {\prime}) \boldsymbol {w} _ {n} \right\rangle
$$
These are the final equations defining the stochastic evolution of  $\pmb{w}_n$  and  $\Delta_n(t)$ .
B.2 Simplifying the Saddle Point Equations
Using the above saddle point equations, we see that the variables  $\{\Delta_n(t)\}$  and  $\{\pmb{w}_n\}$  will be Gaussian random variables. It thus suffices to track their mean and covariance. The  $\{\Delta_n(t)\}$  variables have zero mean and covariance given by the  $Q_{n}(t,t^{\prime})$  function. The  $\{\pmb {w}_n\}$  variables have the following mean evolution
$$
\begin{array}{l} \langle \boldsymbol {w} _ {n + 1} \rangle = \langle \boldsymbol {w} _ {n} \rangle + \eta \sqrt {\alpha} \left[ \bar {\boldsymbol {\Sigma}} \boldsymbol {w} _ {R} - \left[ \bar {\boldsymbol {\Sigma}} - \gamma \bar {\boldsymbol {\Sigma}} _ {+} \right] \langle \boldsymbol {w} _ {n} \rangle \right] \\ = \left\langle \boldsymbol {w} _ {n} \right\rangle + \eta \sqrt {\alpha} \left[ \bar {\boldsymbol {\Sigma}} - \gamma \bar {\boldsymbol {\Sigma}} _ {+} \right] \left[ \boldsymbol {w} _ {T D} - \left\langle \boldsymbol {w} _ {n} \right\rangle \right] \tag {B.14} \\ \end{array}
$$
where  $\pmb{w}_{TD} = \left[\bar{\pmb{\Sigma}} -\gamma \bar{\pmb{\Sigma}}_{+}\right]^{-1}\bar{\pmb{\Sigma}}\pmb{w}_{R}$  is the fixed point of the TD dynamics. We next compute  $M_{n} = \left\langle (\pmb{w}_{n} - \pmb{w}_{TD})(\pmb{w}_{n} - \pmb{w}_{TD})^{\top}\right\rangle$  which admits the recursion
$$
\boldsymbol {M} _ {n + 1} = \left(\boldsymbol {I} - \eta \sqrt {\alpha} \left[ \bar {\boldsymbol {\Sigma}} - \gamma \bar {\boldsymbol {\Sigma}} _ {+} \right]\right) \boldsymbol {M} _ {n} \left(\boldsymbol {I} - \eta \sqrt {\alpha} \left[ \bar {\boldsymbol {\Sigma}} - \gamma \bar {\boldsymbol {\Sigma}} _ {+} \right]\right) + \frac {\eta^ {2}}{T ^ {2}} \sum_ {t t ^ {\prime}} Q _ {n} (t, t ^ {\prime}) \boldsymbol {\Sigma} (t, t ^ {\prime}) \tag {B.15}
$$
To obtain our formulas which hold for finite batch size, we rescale the learning rate by  $\eta \rightarrow \eta / \sqrt{\alpha}$  giving the following evolution
$$
\begin{array}{l} \langle \boldsymbol {w} _ {n + 1} \rangle = \langle \boldsymbol {w} _ {n} \rangle + \eta [ \bar {\boldsymbol {\Sigma}} - \gamma \bar {\boldsymbol {\Sigma}} _ {+} ] [ \boldsymbol {w} _ {T D} - \langle \boldsymbol {w} _ {n} \rangle ] \\ \boldsymbol {M} _ {n + 1} = \left(\boldsymbol {I} - \eta \left[ \bar {\boldsymbol {\Sigma}} - \gamma \bar {\boldsymbol {\Sigma}} _ {+} \right]\right) \boldsymbol {M} _ {n} \left(\boldsymbol {I} - \eta \left[ \bar {\boldsymbol {\Sigma}} - \gamma \bar {\boldsymbol {\Sigma}} _ {+} \right]\right) ^ {\top} + \frac {\eta^ {2}}{T ^ {2} \alpha^ {2}} \sum_ {t t ^ {\prime}} Q _ {n} (t, t ^ {\prime}) \boldsymbol {\Sigma} (t, t ^ {\prime}) \tag {B.16} \\ \end{array}
$$
After this rescaling, we see that the mean evolution for  $\boldsymbol{w}_n$  is independent of  $\alpha$  but that the variance picks up an additive term on each step on the order of  $\mathcal{O}(\eta^2\alpha^{-2})$  which vanishes in the infinite batch limit  $B / N \to \infty$ . The error for value learning can be obtained from  $M_n$  with  $\mathcal{L}_n = \frac{1}{N}\mathrm{Tr}M_n\bar{\Sigma}$ . Lastly, we note that we can express the formula for  $Q_{n}(t,t^{\prime})$  entirely in terms of  $M_n$  and  $\langle \boldsymbol{w}_n\rangle$ . This
givesthelengthyexpression
$$
\begin{array}{l} Q _ {n} (t, t ^ {\prime}) = \frac {1}{N} \left\langle \left(\boldsymbol {w} _ {R} - \boldsymbol {w} _ {n}\right) ^ {\top} \boldsymbol {\Sigma} (t, t ^ {\prime}) \left(\boldsymbol {w} _ {R} - \boldsymbol {w} _ {n}\right) \right\rangle + \frac {\gamma}{N} \left\langle \left(\boldsymbol {w} _ {R} - \boldsymbol {w} _ {n}\right) ^ {\top} \boldsymbol {\Sigma} (t, t ^ {\prime} + 1) \boldsymbol {w} _ {n} \right\rangle \\ + \frac {\gamma}{N} \left\langle \boldsymbol {w} _ {n} ^ {\top} \boldsymbol {\Sigma} (t + 1, t ^ {\prime}) (\boldsymbol {w} _ {R} - \boldsymbol {w} _ {n}) \right\rangle + \frac {\gamma^ {2}}{N} \left\langle \boldsymbol {w} _ {n} ^ {\top} \boldsymbol {\Sigma} (t + 1, t ^ {\prime} + 1) \boldsymbol {w} _ {n} \right\rangle \\ = \frac {1}{N} \operatorname {T r} M _ {n} \boldsymbol {\Sigma} \left(t, t ^ {\prime}\right) + \frac {1}{N} \left(\boldsymbol {w} _ {T D} - \langle \boldsymbol {w} _ {n} \rangle\right) \left[ \boldsymbol {\Sigma} \left(t, t ^ {\prime}\right) + \boldsymbol {\Sigma} \left(t ^ {\prime}, t\right) \right] \left(\boldsymbol {w} _ {R} - \boldsymbol {w} _ {T D}\right) \\ + \frac {1}{N} \left(\pmb {w} _ {R} - \pmb {w} _ {T D}\right) ^ {\top} \pmb {\Sigma} (t, t ^ {\prime}) \left(\pmb {w} _ {R} - \pmb {w} _ {T D}\right) \\ - \frac {\gamma}{N} \operatorname {T r} M _ {n} [ \boldsymbol {\Sigma} (t, t ^ {\prime} + 1) + \boldsymbol {\Sigma} (t + 1, t ^ {\prime}) ] \\ + \frac {\gamma}{N} \left(\boldsymbol {w} _ {T D} - \langle \boldsymbol {w} _ {n} \rangle\right) \left[ \boldsymbol {\Sigma} (t, t ^ {\prime} + 1) + \boldsymbol {\Sigma} (t + 1, t ^ {\prime}) \right] \boldsymbol {w} _ {T D} \\ + \frac {\gamma}{N} \left(\boldsymbol {w} _ {R} - \boldsymbol {w} _ {T D}\right) ^ {\top} \left[ \boldsymbol {\Sigma} \left(t, t ^ {\prime} + 1\right) + \boldsymbol {\Sigma} \left(t + 1, t ^ {\prime}\right) \right] \left\langle \boldsymbol {w} _ {n} \right\rangle \\ + \frac {\gamma^ {2}}{N} \operatorname {T r} M _ {n} \boldsymbol {\Sigma} (t + 1, t ^ {\prime} + 1) + \frac {2 \gamma^ {2}}{N} \left(\langle \boldsymbol {w} _ {n} \rangle - \boldsymbol {w} _ {T D}\right) \boldsymbol {\Sigma} (t + 1, t ^ {\prime} + 1) \boldsymbol {w} _ {T D} \\ + \frac {\gamma^ {2}}{N} \boldsymbol {w} _ {T D} ^ {\top} \boldsymbol {\Sigma} (t + 1, t ^ {\prime} + 1) \boldsymbol {w} _ {T D} \tag {B.17} \\ \end{array}
$$
B.3 Final Result
Below we state in compact form the full final result for our TD learning curves. The below equations give the evolution of the first and second moments of  $\boldsymbol{w}_n$  obtained from the mean-field density of the previous section. Concretely, these moments obey dynamics
$$
\langle \boldsymbol {w} _ {n + 1} \rangle = \langle \boldsymbol {w} _ {n} \rangle + \eta [ \bar {\boldsymbol {\Sigma}} - \gamma \bar {\boldsymbol {\Sigma}} _ {+} ] [ \boldsymbol {w} _ {V} - \langle \boldsymbol {w} _ {n} \rangle ]
$$