SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Since MountainCar-v0 is a continuous environment, there is no closed solution to the ground truth of the value function. To estimate the ground truth value function, we ran TD learning with a small learning rate for 10M batches ( $\eta = 0.01$ ,  $B = 1$ ,  $\gamma = 0.99$ ) to obtain  $V^{\pi} \approx \hat{V}_{10M}$ .
(a) Polynomial features
(b) Fourier Features
Figure E.1: In this Figure, we simulate the same random walk policy on the 2D grid world but use other types of features beyond RBF place cells. Learning dynamics are still accurately described by our theory. (c) The polynomial basis over states with random powers is constructed as  $\psi_i(s_1,s_2) = s_1^{c_{i,1}}s_2^{c_{i,2}}$  where  $c_{i,1},c_{i,2}$  are chosen at random from  $\{0,1,\dots,k\}$  where  $k$  is the degree. (f) Fourier features with spectral power density  $\frac{1}{1 + \sigma^2(k_1^2 + k_2^2)}$ , where  $\sigma$  is the scale/bandwidth.
G Numerical methods and additional details
The code to generate the Figures is provided in the Supplementary Material as a Jupyter Notebook at the following Github repository https://github.com/Pehlevan-Group/TD-RL-dynamics. Here, we briefly highlight some of the parameter choices.
For Figures 3 and 4 we use diagonally decoupled, but temporally correlated power law features with  $\Sigma_{k\ell}(t,t^{\prime}) = \delta_{k\ell}k^{-1.2}\exp \left(-|t - t^{\prime}| / \tau_{k}\right)$  with  $\tau_{k} = \frac{10}{k + 1}$  and  $w_{k}^{R} = k^{-1.1}$  for  $k\in [N]$  with  $N = 300$ . This type of feature structure is especially easy to evaluate the theoretical learning curves for. Unless otherwise stated, these figures used  $\gamma = 0.9$  and batch size  $B = 10$ .
For the 2D MDP grid world, we defined a discrete set of states on a  $17 \times 17$  grid. The agent starts in the middle position and follows a random diffusion policy where each possible movement (up, down, left, right) is taken with equal probability. The features were generated as bell-shaped place cells (shown). We computed  $\Sigma(t, t')$  for the theory by sampling 5000 random draws of length  $T = 50$ . The Gaussian learning curve is obtained with TD learning with  $\psi_G \sim \mathcal{N}(0, \Sigma)$ .
Numerical experiments were performed on a NVIDIA SMX4-A100-80GB GPU using JAX to vectorize repetitive aspects of the experiments. With the exception of the MountainCar-v0 simulations, the numerical experiments (both preliminary experiments and those presented in the paper) took around 1 hour of compute time.
(a)  $V^{*}$  estimated with tabular  $\epsilon$  -greedy Q-Learning
(b)  $V^{\pi}$  TD Converged Value Function
(c) Scaling of value error with learning rate
(d) Scaling of value error with batch size
Figure F.1: Simulation in a MountainCar-v0 environment. (a) Value function learned by Tabular Q-Learning that approximates the value function of an optimal policy. (b) An example value function of a policy  $(V^{\pi})$  learned by TD learning. Notice that the value function does not equate to that in (a) due to the policy  $\pi$  not reaching all states in the environment. (c-d) Linear scaling of convergence value error with the learning rate and the inverse of batch size. Target value function is the same across both experiments. Each dot represents a different seed. A total of 10 seeds were used. (c) Convergence value errors were computed by averaging the 100k batches before batch 10M. (d) Convergence value errors were computed by averaging the 100k batches before batch 1M.