SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\boldsymbol {M} _ {n + 1} = \left[ \boldsymbol {I} - \eta \bar {\boldsymbol {\Sigma}} + \eta \gamma \bar {\boldsymbol {\Sigma}} _ {+} \right] \boldsymbol {M} _ {n} \left[ \boldsymbol {I} - \eta \bar {\boldsymbol {\Sigma}} + \eta \gamma \bar {\boldsymbol {\Sigma}} _ {+} \right] ^ {\top} + \frac {\eta^ {2}}{\alpha^ {2} T ^ {2}} \sum_ {t t ^ {\prime}} Q _ {n} (t, t ^ {\prime}) \boldsymbol {\Sigma} (t, t ^ {\prime})
$$
$$
\begin{array}{l} Q _ {n} (t, t ^ {\prime}) = \frac {1}{N} \left\langle \left(\boldsymbol {w} _ {R} - \boldsymbol {w} _ {n}\right) ^ {\top} \boldsymbol {\Sigma} (t, t ^ {\prime}) \left(\boldsymbol {w} _ {R} - \boldsymbol {w} _ {n}\right) \right\rangle + \frac {\gamma}{N} \left\langle \left(\boldsymbol {w} _ {R} - \boldsymbol {w} _ {n}\right) ^ {\top} \boldsymbol {\Sigma} (t, t ^ {\prime} + 1) \boldsymbol {w} _ {n} \right\rangle \\ + \frac {\gamma}{N} \left\langle \boldsymbol {w} _ {n} ^ {\top} \boldsymbol {\Sigma} \left(t + 1, t ^ {\prime}\right) \left(\boldsymbol {w} _ {R} - \boldsymbol {w} _ {n}\right) \right\rangle + \frac {\gamma^ {2}}{N} \left\langle \boldsymbol {w} _ {n} ^ {\top} \boldsymbol {\Sigma} \left(t + 1, t ^ {\prime} + 1\right) \boldsymbol {w} _ {n} \right\rangle . \tag {B.18} \\ \end{array}
$$
These equations can be solved iteratively for  $\bar{w}_n, M_n, Q_n$ . Finite dimensional versions of this result can be obtained by replacing  $\alpha$  with  $B / N$  as written in the main text. The value estimation error is
$$
\mathcal {L} _ {n} = \frac {1}{N} \operatorname {T r} M _ {n} \bar {\boldsymbol {\Sigma}}. \tag {B.19}
$$
B.4 Non-Zero Mean Feature
We can also simply modify the DMFT equations if the mean feature is nonvanishing  $\pmb{\mu}(s) \neq 0$ . In this case, when averaging over all possible trajectories through state space, there is a mean feature vector at each episodic time  $\pmb{\mu}(t)$ . The above equations are exact for non-zero mean features if  $\pmb{\Sigma}(t, t')$  is regarded as the (non-centered) correlation matrix  $\langle \pmb{\psi}(t) \pmb{\psi}(t') \rangle$ .
B.5 Action Dependent Rewards
B.5.1 Expected Q-Learning Reduces to Previous Model
In the case where we consider using features that depend on both states and actions  $\psi(s, a)$  then we can use expected value learning to identify the expected value of a state-action pair under policy  $\pi$
$$
V (s, a) = R (s, a) + \gamma \left\langle V \left(s ^ {\prime}, a ^ {\prime}\right) \right\rangle_ {s ^ {\prime}, a ^ {\prime} | s, a} \tag {B.20}
$$
This  $V$  function quantifies the expected reward associated with taking action  $a$  when in state  $s$  and subsequently following policy  $\pi$ . This problem is structurally identical to the state dependent case
by recognizing that state action pairs  $(s, a)$  act as new states  $\tilde{s}$ . As before, the policy defines the probability distribution over transitions on  $\tilde{s}$ . We can thus use Equation (4) to calculate the learning curve for this problem.
B.5.2 Action Dependence Generates Target Noise in State Dependent Value Learning
In the case where the rewards depend on both state and action  $R(s, a)$  but features only depend on state  $\psi(s)$ , we need a slight modification of our theory which models the reward at each state as a mean value (over actions) plus a noise. For each state, we decompose the reward function into mean and fluctuation
$$
R (s, a) = \bar {R} (s) + \epsilon (s, a), \bar {R} (s) = \mathbb {E} _ {a \sim \pi (a | s)} R (s, a) \tag {B.21}
$$
The function  $\bar{R}(s)$  can again be decomposed into the basis of features  $\psi(s)$ . However, we need to consider the correlation structure of  $\epsilon(s, a)$ .
$$
\begin{array}{l} \mathbb {E} _ {\tau} \epsilon (s _ {t}, a _ {t}) \epsilon (s _ {t ^ {\prime}}, a _ {t ^ {\prime}}) = \mathbb {E} _ {s _ {t}} \mathbb {E} _ {a _ {t} | s _ {t}} \epsilon (s _ {t}, a _ {t}) \left[ \mathbb {E} _ {s _ {t ^ {\prime}} | s _ {t}, a _ {t}} \left(\mathbb {E} _ {a _ {t ^ {\prime}} | s _ {t ^ {\prime}}} \epsilon (s _ {t ^ {\prime}}, a _ {t ^ {\prime}})\right) \right] \\ = \delta_ {t, t ^ {\prime}} \operatorname {V a r} _ {a \mid s _ {t}} R \left(s _ {t}, a\right). \tag {B.22} \\ \end{array}
$$
The above average vanishes for  $t \neq t'$  since  $\epsilon(s_{t'}, a_{t'})$  is zero mean over  $a|s$ . We introduce the notation  $\sigma_t^2 = \operatorname{Var}_{a|s_t} R(s_t, a)$ . Thus, we effectively have a model where our TD errors obey
$$
\Delta (t) = \bar {R} \left(s _ {t}\right) + \epsilon \left(s _ {t}, a _ {t}\right) + \gamma \hat {V} \left(s _ {t}\right) - \hat {V} \left(s _ {t}\right) \tag {B.23}
$$
The addition of this term leads to a simple modification of our  $Q(t,t)$  function
$$
\begin{array}{l} Q _ {n} (t, t ^ {\prime}) = \frac {1}{N} \left\langle \left(\boldsymbol {w} _ {R} - \boldsymbol {w} _ {n}\right) ^ {\top} \boldsymbol {\Sigma} (t, t ^ {\prime}) \left(\boldsymbol {w} _ {R} - \boldsymbol {w} _ {n}\right) \right\rangle + \frac {\gamma}{N} \left\langle \left(\boldsymbol {w} _ {R} - \boldsymbol {w} _ {n}\right) ^ {\top} \boldsymbol {\Sigma} (t, t ^ {\prime} + 1) \boldsymbol {w} _ {n} \right\rangle \\ + \frac {\gamma}{N} \left\langle \pmb {w} _ {n} ^ {\top} \pmb {\Sigma} (t + 1, t ^ {\prime}) (\pmb {w} _ {R} - \pmb {w} _ {n}) \right\rangle + \frac {\gamma^ {2}}{N} \left\langle \pmb {w} _ {n} ^ {\top} \pmb {\Sigma} (t + 1, t ^ {\prime} + 1) \pmb {w} _ {n} \right\rangle \\ + \delta_ {t, t ^ {\prime}} \sigma_ {t} ^ {2}. \tag {B.24} \\ \end{array}
$$
This change to the  $Q_{n}(t,t^{\prime})$  correlation function alters the dynamics of  $M_{n}$ . Lastly, our population risk for the value estimation takes the form
$$
\mathcal {L} _ {n} = \frac {1}{N} \operatorname {T r} M _ {n} \bar {\Sigma} + \frac {1}{T} \sum_ {t} \sigma_ {t} ^ {2} \tag {B.25}
$$
where  $\frac{1}{T}\sum_{t}\sigma_{t}^{2}$  exactly quantifies the variance in rewards unexplained by state-dependent features.
B.6 Tracking Iterate Moments with Direct Recurrence Relation
In this section we give a direct calculation of the first two moments of  $\pmb{w}$  over the collection of randomly sampled features  $\{\psi_n^\mu (t)\}$  and show which terms are disregarded in the proportional limit examined in the main text.
Letting  $A = \bar{\Sigma} -\gamma \bar{\Sigma}_{+}$ , we note that the average evolution of  $\pmb{w}$  has the form
$$
\langle \boldsymbol {w} _ {n + 1} \rangle = (\boldsymbol {\Sigma} - \gamma \boldsymbol {\Sigma} _ {+}) \left(\boldsymbol {w} _ {T D} - \langle \boldsymbol {w} _ {n} \rangle\right) \tag {B.26}
$$
Thus, if we disregarded fluctuations in  $\mathbf{w}_n$  due to SGD, the model will converge to the correct fixed point. Next, we look at  $M_{n} = \langle (\pmb{w}_{n} - \pmb{w}_{TD})(\pmb{w}_{n} - \pmb{w}_{TD})\rangle$ . Under the Gaussian equivalence ansatz, we have
$$
\begin{array}{l} \boldsymbol {M} _ {n + 1} = \boldsymbol {M} _ {n} - \eta \boldsymbol {A} \boldsymbol {M} _ {n} - \eta \boldsymbol {M} _ {n} \boldsymbol {A} ^ {\top} + \frac {\eta^ {2}}{T ^ {2} B ^ {2}} \sum_ {\mu \nu t t ^ {\prime}} \langle \Delta_ {n} ^ {\mu} (t) \Delta_ {n} ^ {\nu} (t ^ {\prime}) \psi_ {n} ^ {\mu} (t) \psi_ {n} ^ {\nu} (t ^ {\prime}) \rangle \\ = (\pmb {I} - \eta \pmb {A}) \pmb {M} _ {n} (\pmb {I} - \eta \pmb {A}) ^ {\top} - \frac {\eta^ {2}}{B} \pmb {A} \pmb {M} _ {n} \pmb {A} ^ {\top} + \frac {\eta^ {2}}{T ^ {2} B} \sum_ {t t ^ {\prime}} \left\langle \Delta_ {n} (t) \Delta_ {n} (t ^ {\prime}) \psi (t) \psi (t ^ {\prime}) ^ {\top} \right\rangle \\ = (\boldsymbol {I} - \eta \boldsymbol {A}) \boldsymbol {M} _ {n} (\boldsymbol {I} - \eta \boldsymbol {A}) ^ {\top} + \frac {\eta^ {2}}{T ^ {2} B} \sum_ {t t ^ {\prime}} Q _ {n} (t, t ^ {\prime}) \boldsymbol {\Sigma} (t, t ^ {\prime}) \\ + \frac {\eta^ {2}}{T ^ {2} B} \sum_ {t t ^ {\prime}} \left\langle \Delta_ {n} \left(t ^ {\prime}\right) \psi (t) \right\rangle \left\langle \Delta_ {n} (t) \psi \left(t ^ {\prime}\right) ^ {\top} \right\rangle \tag {B.27} \\ \end{array}
$$
The mean field theory derived from saddle point integration consists of the first two terms in the final expression. Therefore mean field theory disregards the last term which computes cross time correlations of RPEs with features, effectively making the approximation
$$
\frac {\eta^ {2}}{T ^ {2} B} \sum_ {t t ^ {\prime}} \left\langle \Delta_ {n} \left(t ^ {\prime}\right) \psi (t) \right\rangle \left\langle \Delta_ {n} (t) \psi \left(t ^ {\prime}\right) ^ {\top} \right\rangle \approx 0. \tag {B.28}
$$
After making this approximation, we recover the learning curve obtained in the previous Section B.3. We show in our experiments that dropping this term does not significantly alter the learning curves.
B.7 Scaling of Asymptotic Fixed Points
To identify fixed points in the value error dynamics, we can seek non-vanishing fixed points for the weight error covariance  $M = \langle (\pmb{w} - \pmb{w}_{TD})(\pmb{w} - \pmb{w}_{TD}) \rangle$ . We note that  $\langle \pmb{w} \rangle \sim \pmb{w}_{TD}$  asymptotically. Again, letting  $A = \bar{\Sigma} - \gamma \bar{\Sigma}_+$ , we obtain the following fixed point condition for  $M$  under these assumptions
$$
\boldsymbol {A} \boldsymbol {M} + \boldsymbol {M} \boldsymbol {A} ^ {\top} - \eta \boldsymbol {A} \boldsymbol {M} \boldsymbol {A} ^ {\top} = \frac {\eta}{B T ^ {2}} \sum_ {t t ^ {\prime}} Q (t, t ^ {\prime}) \boldsymbol {\Sigma} (t, t ^ {\prime})
$$
$$
\begin{array}{l} Q (t, t ^ {\prime}) = \operatorname {T r} M \boldsymbol {\Sigma} (t, t ^ {\prime}) - \gamma \operatorname {T r} M [ \boldsymbol {\Sigma} (t, t ^ {\prime} + 1) + \boldsymbol {\Sigma} (t + 1, t ^ {\prime}) ] + \gamma^ {2} \operatorname {T r} M \boldsymbol {\Sigma} (t + 1, t ^ {\prime} + 1) \\ + \gamma^ {2} \boldsymbol {w} _ {T D} ^ {\top} \bar {\boldsymbol {\Sigma}} ^ {- 1} \bar {\boldsymbol {\Sigma}} _ {+} \boldsymbol {\Sigma} (t, t ^ {\prime}) \bar {\boldsymbol {\Sigma}} _ {+} \bar {\boldsymbol {\Sigma}} ^ {- 1} \boldsymbol {w} _ {T D} + \gamma^ {2} \boldsymbol {w} _ {T D} ^ {\top} \boldsymbol {\Sigma} (t + 1, t ^ {\prime} + 1) \boldsymbol {w} _ {T D} \\ + \gamma^ {2} \boldsymbol {w} _ {T D} \bar {\boldsymbol {\Sigma}} ^ {- 1} \bar {\boldsymbol {\Sigma}} _ {+} [ \boldsymbol {\Sigma} (t, t ^ {\prime} + 1) + \boldsymbol {\Sigma} (t + 1, t ^ {\prime}) ] \boldsymbol {w} _ {T D}. \tag {B.29} \\ \end{array}
$$
Where we used the formula for  $Q_{n}(t,t^{\prime})$  from Appendix B.6, evaluated at  $\langle \pmb {w}\rangle = \pmb{w}_{TD}$  and used the fact that  $\pmb {w}_R = \pmb {w}_{TD} - \gamma \bar{\pmb{\Sigma}}^{-1}\bar{\pmb{\Sigma}}_+ \pmb {w}_{TD}$ . The solution  $M = 0$  is a valid fixed point for  $M$  in the  $\eta \rightarrow 0$  and  $B\to \infty$  limits because the constant terms on the right-hand side vanish. Similarly, if  $\gamma = 0$  (which corresponds to the standard supervised learning case), the right hand side is linear in  $M$ , allowing  $M = 0$  to be a valid fixed point.
However, for finite  $B$  and non-zero  $\eta$  and  $\gamma$ , there exists a solution to the above fixed point equation. For small  $\frac{\eta\gamma^2}{B}$ , we can deduce that  $M$  must satisfy a self-consistent asymptotic scaling of the form
$$
M = \mathcal {O} \left(\frac {\eta \gamma^ {2}}{B}\right) \tag {B.30}
$$
implying an asymptotic value error scaling of  $\mathcal{L} \sim \mathrm{Tr}M\bar{\Sigma} \sim \mathcal{O}\left(\frac{\eta\gamma^2}{B}\right)$ . These scalings are examined in Figure 3 where experiments obey the expected behavior.
C Reward Shaping
In this section, we consider the role of reward shaping on the dynamics of TD learning. As discussed in the main text, we consider potential based shaping with potential function decomposable in the features  $\phi(s) = \pmb{w}_{\phi} \cdot \pmb{\psi}(s)$ . We first describe the change to the average weight evolution  $\langle \pmb{w}_n \rangle$  and then describe the dynamics of the correlations. In potential based shaping, the TD errors take the form
$$
\Delta (t) = R (s (t)) + \phi (s (t)) - \gamma \phi (s (t + 1)) + \gamma \hat {V} (s (t + 1)) - \hat {V} (s (t)) \tag {C.1}
$$
Computing from the DMFT equations the evolution of  $\langle \pmb{w}_n\rangle$  we have
$$
\begin{array}{l} \langle \boldsymbol {w} _ {n + 1} \rangle = \langle \boldsymbol {w} _ {n} \rangle + \eta \bar {\boldsymbol {\Sigma}} (\boldsymbol {w} _ {R} + \boldsymbol {w} _ {\phi} - \langle \boldsymbol {w} _ {n} \rangle) + \gamma \eta \bar {\boldsymbol {\Sigma}} _ {+} (\langle \boldsymbol {w} _ {n} \rangle - \boldsymbol {w} _ {\phi}) \\ = \left\langle \boldsymbol {w} _ {n} \right\rangle - \eta \boldsymbol {A} \left[ \boldsymbol {w} _ {T D} + \boldsymbol {w} _ {\phi} - \left\langle \boldsymbol {w} _ {n} \right\rangle \right]. \tag {C.2} \\ \end{array}
$$
We see that including the reward shaping function  $\phi$  offsets the fixed point of the algorithm to be  $\pmb{w}_{TD} + \pmb{w}_{\phi}$ . This occurs precisely because the potential-based reward shaping generates an additive correction to the target value function by  $\phi(s)$  [64]. When we predict value at evaluation, we use the reshifted value  $\hat{V}(s) - \phi(s)$ . The natural quantity to track at the level of the mean field equations is the adapted version of  $M_{n}$
$$
\boldsymbol {M} _ {n} = \left\langle \left(\boldsymbol {w} _ {n} - \boldsymbol {w} _ {T D} - \boldsymbol {w} _ {\phi}\right) \left(\boldsymbol {w} _ {n} - \boldsymbol {w} _ {T D} - \boldsymbol {w} _ {\phi}\right) ^ {\top} \right\rangle . \tag {C.3}
$$
This correlation matrix has dynamics
$$
\boldsymbol {M} _ {n + 1} = (\boldsymbol {I} - \eta \boldsymbol {A}) \boldsymbol {M} _ {n} (\boldsymbol {I} - \eta \boldsymbol {A}) ^ {\top} + \frac {\eta^ {2}}{B T ^ {2}} \sum_ {t t ^ {\prime}} Q _ {n} (t, t ^ {\prime}) \boldsymbol {\Sigma} (t, t ^ {\prime}) \tag {C.4}
$$
and the TD-error correlations  $Q_{n}(t,t^{\prime})$  have the form
$$
\begin{array}{l} Q _ {n} (t, t ^ {\prime}) = \left\langle \left(\boldsymbol {w} _ {R} + \boldsymbol {w} _ {\phi} - \boldsymbol {w} _ {n}\right) ^ {\top} \boldsymbol {\Sigma} (t, t ^ {\prime}) \left(\boldsymbol {w} _ {R} + \boldsymbol {w} _ {\phi} - \boldsymbol {w} _ {n}\right) \right\rangle \\ + \gamma \left\langle \left(\boldsymbol {w} - \boldsymbol {w} _ {\phi}\right) ^ {\top} \left[ \boldsymbol {\Sigma} \left(t, t ^ {\prime}\right) + \boldsymbol {\Sigma} \left(t ^ {\prime}, t\right) \right] \left(\boldsymbol {w} _ {R} + \boldsymbol {w} _ {\phi} - \boldsymbol {w} _ {n}\right) \right\rangle \\ + \gamma^ {2} \left\langle \left(\boldsymbol {w} _ {n} - \boldsymbol {w} _ {\phi}\right) ^ {\top} \boldsymbol {\Sigma} \left(t + 1, t ^ {\prime} + 1\right) \left(\boldsymbol {w} _ {n} - \boldsymbol {w} _ {\phi}\right) \right\rangle \tag {C.5} \\ \end{array}
$$
The value estimation error is again  $\mathcal{L}_n = \mathrm{Tr}M_n\bar{\Sigma}$ . We see that the two primary ways that reward shaping alters the loss dynamics is
- A change in the initial condition for  $M_{n}$  to be  $M_0 = (\pmb{w}_{TD} + \pmb{w}_{\phi})(\pmb{w}_{TD} + \pmb{w}_{\phi})^\top$
- A change in the TD error covariance term  $Q_{n}(t,t^{\prime})$
Both effects can generate significant changes in the dynamics and plateaus of the model.
D Non-Gaussian Features
The full non-asymptotic theory (no assumptions on  $N, B$  large) of TD dynamics with linear function approximation closes under the fourth moments of the features. In this setting we do not incorporate explicit factors of  $N$  in the definition of the value estimator  $\hat{V}(t) = \boldsymbol{w} \cdot \boldsymbol{\psi}(t)$ . As before, we track the update to the  $M$  matrix
$$
\begin{array}{l} \boldsymbol {M} _ {n + 1} = \boldsymbol {M} _ {n} - \eta \boldsymbol {A} \boldsymbol {M} _ {n} - \eta \boldsymbol {M} _ {n} \boldsymbol {A} ^ {\top} + \frac {\eta^ {2} (B - 1)}{B} \boldsymbol {A} \boldsymbol {M} _ {n} \boldsymbol {A} ^ {\top} \\ + \frac {\eta^ {2}}{B} \sum_ {t t ^ {\prime}} \left\langle \Delta_ {n} (t) \Delta_ {n} \left(t ^ {\prime}\right) \psi (t) \psi \left(t ^ {\prime}\right) ^ {\top} \right\rangle \tag {D.1} \\ \end{array}
$$
To calculate the last term, we introduce the following tensor of fourth moments
$$
\kappa_ {i j k l} ^ {4} \left(t _ {1}, t _ {2}, t _ {3}, t _ {4}\right) = \left\langle \psi_ {i} \left(t _ {1}\right) \psi_ {j} \left(t _ {2}\right) \psi_ {k} \left(t _ {3}\right) \psi_ {l} \left(t _ {4}\right) \right\rangle \tag {D.2}
$$
In the Gaussian case, this reduces to an expression involving only the correlations. For example, if the features are zero mean, then we can use Wick's theorem to obtain the decomposition
$$
\kappa_ {i j k l} ^ {4, G a u s s} \left(t _ {1}, t _ {2}, t _ {3}, t _ {4}\right) = \Sigma_ {i j} \left(t _ {1}, t _ {2}\right) \Sigma_ {k l} \left(t _ {3}, t _ {4}\right) + \Sigma_ {i k} \left(t _ {1}, t _ {3}\right) \Sigma_ {j l} \left(t _ {2}, t _ {4}\right) + \Sigma_ {i j} \left(t _ {1}, t _ {2}\right) \Sigma_ {k l} \left(t _ {3}, t _ {4}\right) \tag {D.3}
$$
Now, using the fact that  $\Delta_n(t) = (\pmb{w}_R - \pmb{w}_n) \cdot \pmb{\psi}(t) + \gamma \pmb{w}_n \cdot \pmb{\psi}(t + 1)$ , we find
$$
\begin{array}{l} \left\langle \Delta_ {n} (t) \Delta_ {n} \left(t ^ {\prime}\right) \psi_ {i} (t) \psi_ {j} \left(t ^ {\prime}\right) ^ {\top} \right\rangle \\ = \left\langle \psi_ {i} (t) \psi_ {j} \left(t ^ {\prime}\right) \boldsymbol {\psi} (t) ^ {\top} \left(\boldsymbol {w} _ {R} - \boldsymbol {w} _ {n}\right) \left(\boldsymbol {w} _ {R} - \boldsymbol {w} _ {n}\right) ^ {\top} \boldsymbol {\psi} \left(t ^ {\prime}\right) \right\rangle \\ + \gamma \left\langle \psi_ {i} (t) \psi_ {j} \left(t ^ {\prime}\right) \boldsymbol {\psi} (t) ^ {\top} \left(\boldsymbol {w} _ {R} - \boldsymbol {w} _ {n}\right) \boldsymbol {w} _ {n} ^ {\top} \boldsymbol {\psi} \left(t ^ {\prime} + 1\right) \right\rangle \\ + \gamma \left\langle \psi_ {i} (t) \psi_ {j} \left(t ^ {\prime}\right) \psi (t + 1) ^ {\top} \boldsymbol {w} _ {n} \left(\boldsymbol {w} _ {R} - \boldsymbol {w} _ {n}\right) ^ {\top} \boldsymbol {\psi} \left(t ^ {\prime}\right) \right\rangle \\ + \gamma^ {2} \left\langle \psi_ {i} (t) \psi_ {j} \left(t ^ {\prime}\right) \psi (t + 1) ^ {\top} \boldsymbol {w} _ {n} \boldsymbol {w} _ {n} ^ {\top} \psi \left(t ^ {\prime} + 1\right) \right\rangle \tag {D.4} \\ \end{array}
$$
Putting this all together, we find the following recurrence for  $M_{n}$  in the non-Gaussian case
$$
\begin{array}{l} \boldsymbol {M} _ {n + 1} = \boldsymbol {M} _ {n} - \eta \boldsymbol {A} \boldsymbol {M} _ {n} - \eta \boldsymbol {M} _ {n} \boldsymbol {A} ^ {\top} + \frac {\eta^ {2} (B - 1)}{B} \boldsymbol {A} \boldsymbol {M} _ {n} \boldsymbol {A} ^ {\top} \\ + \frac {\eta^ {2}}{B T ^ {2}} \sum_ {t, t ^ {\prime}} \operatorname {T r} \boldsymbol {\kappa} (t, t ^ {\prime}, t, t ^ {\prime}) \left\langle (\boldsymbol {w} _ {R} - \boldsymbol {w} _ {n}) (\boldsymbol {w} _ {R} - \boldsymbol {w} _ {n}) ^ {\top} \right\rangle \\ + \frac {\gamma \eta^ {2} T ^ {2}}{B} \sum_ {t, t ^ {\prime}} \operatorname {T r} \kappa (t, t ^ {\prime}, t, t ^ {\prime} + 1) \left\langle \boldsymbol {w} _ {n} \left(\boldsymbol {w} _ {R} - \boldsymbol {w} _ {n}\right) ^ {\top} \right\rangle \\ + \frac {\gamma \eta^ {2}}{B T ^ {2}} \sum_ {t, t ^ {\prime}} \operatorname {T r} \kappa (t, t ^ {\prime}, t + 1, t ^ {\prime}) \left\langle (\boldsymbol {w} _ {R} - \boldsymbol {w} _ {n}) \boldsymbol {w} _ {n} ^ {\top} \right\rangle \\ + \frac {\gamma^ {2} \eta^ {2}}{B T ^ {2}} \sum_ {t, t ^ {\prime}} \operatorname {T r} \kappa (t, t ^ {\prime}, t + 1, t ^ {\prime} + 1) \left\langle \boldsymbol {w} _ {n} \boldsymbol {w} _ {n} ^ {\top} \right\rangle \tag {D.5} \\ \end{array}
$$
where all traces are taken against the last two time and feature indices. The averages over the weights close in terms of the average  $\langle \pmb{w}_n\rangle$  and the covariance  $M_{n}$ . This equation is exact for any feature distribution, but requires significant computational resources to evaluate and is less illuminating than the mean field limit analyzed in the previous sections.
D.1 Breakdown of Gaussian Theory in Low Dimension
In this section, we discuss the breakdown of Gaussian theory at low dimension  $N$ . In Figure D.1 we provide an example where the non-Gaussian distributions exhibit noticeably different learning curves than the Gaussian approximate theory (dashed black) and Gaussian samples with matching covariance (dashed color). We use the features
(a)  $B = 5$
(b)  $N = 100$
Figure D.1: The Gaussian theory can break down for non-Gaussian features in low dimension  $N$ . Illustration of the possible gap between Gaussian and non-Gaussian performance in the power law features of Figure 3 and defined in Appendix G. (a) The learning curves for batchsize  $B = 5$  and varying dimension  $N$ . As  $N$  increases the gap between the non-Gaussian experiment (solid) and the Gaussian theory (black dashed) decreases.
D.2 A Simple Solveable Example
We next examine a very simple case where we can exactly characterize the gap between the non-Gaussian and Gaussian distributions. In this section, we examine the special case of  $T = 1$  and look at features which are independent  $p(\psi) = \prod_{i=1}^{N} p(\psi_i)$  (form a factor distribution). In this case, we obtain the following exact learning curve
$$
\mathcal {L} _ {n} = \left[ (1 - \eta) ^ {2} + \frac {\eta^ {2} (N + 1 + \kappa)}{B} \right] ^ {n}, \kappa = \left\langle \psi^ {4} \right\rangle - 3 \left\langle \psi^ {2} \right\rangle^ {2} \tag {D.6}
$$
which holds for any  $N, B$ . We note that in the limit where  $N, B \to \infty$  with  $B / N = \alpha$ , we see that the dependence on  $\kappa$  disappears and we arrive at the universal behavior
$$
\mathcal {L} _ {n} \sim \left[ (1 - \eta) ^ {2} + \frac {\eta^ {2}}{\alpha} \right] ^ {n}, N, B \rightarrow \infty \tag {D.7}
$$
For example, we can consider vectors on the hypercube where  $\psi_{k}\in \{\pm 1\}$  with equal probability for  $k\in \{1,\dots,N\}$  for the non-Gaussian distribution and compare to the Gaussian with identical covariance  $\psi \sim \mathcal{N}(0,I)$ .
$$
\mathcal {L} _ {n} = \left\{ \begin{array}{l l} {\left[ (1 - \eta) ^ {2} + \frac {\eta^ {2} (N + 1)}{B} \right] ^ {n}} & \text {G a u s s i a n} \psi \\ {\left[ (1 - \eta) ^ {2} + \frac {\eta^ {2} (N - 1)}{B} \right] ^ {n}} & \text {H y p e r c u b e} \psi \end{array} \right. \tag {D.8}
$$
The reason for the discrepancy between the Gaussian and Bernoulli/Hypercube loss curves is exactly the negative kurtosis of the hypercube features
$$
\kappa_ {\mathrm {G a u s s}} = 0
$$
$$
\kappa_ {\text {B e r n o u l l i}} - 3 \Sigma_ {\text {B e r n o u l l i}} ^ {2} = \left\langle \psi^ {4} \right\rangle - 3 \left\langle \psi^ {2} \right\rangle^ {2} = - 2 \tag {D.9}
$$
An example of this result for low and high dimensions  $N$  with  $B = \alpha N$  with  $\alpha = 0.1$  is provided in Figure D.2. In low dimension ( $N = 10$ ) the Bernoulli/Hypercube feature has noticeably different dynamics than the Gaussian features. In the proportional limit  $N, B \to \infty$  with  $\alpha = B / N$  these learning curves are identical and all have the form  $\mathcal{L}_n \sim \left[(1 - \eta)^2 + \frac{\eta^2}{\alpha}\right]^n$ .
(a) Low Dimension
(b) High Dimension
Figure D.2: A simple explicitly solvable case shows how non-Gaussian corrections appear at finite size but disappear in the proportional limit where  $N, B \to \infty$  with  $\alpha = B / N = \mathcal{O}(1)$ .
E Tests on Other Feature Distributions
In this section, we include additional tests of our theory on alternative features with the same random walk policy as in Figure 1.
F Plateau Scaling in MountainCar-v0 Environment
We verified the results of the theory on the environment MountainCar-v0. First, we train a policy with tabular  $\epsilon$ -greedy Q-Learning ( $\epsilon = 0.1$ ,  $\gamma = 0.99$ ,  $\eta = 0.01$ ) to learn policy  $\pi$ . The position and velocity are discretized into 42 and 28 states, respectively. The learned policy  $\pi$  is not optimal but consistently reaches goal within 350 timesteps. Therefore, each episode is set to have a length of 350 timesteps. Next, we take  $\pi$  and evaluate it with TD learning.