Title: Loss Dynamics of Temporal Difference Reinforcement Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Reinforcement Learning

Statistical Mechanics

Stochastic Gradient Descent
Abstract: Reinforcement learning has been successful across several applications in which agents have to learn to act in environments with sparse feedback. However, despite this empirical success there is still a lack of theoretical understanding of how the parameters of reinforcement learning models and the features used to represent states interact to control the dynamics of learning. In this work, we use concepts from statistical physics, to study the typical case learning curves for temporal difference learning of a value function with linear function approximators. Our theory is derived under a Gaussian equivalence hypothesis where averages over the random trajectories are replaced with temporally correlated Gaussian feature averages and we validate our assumptions on small scale Markov Decision Processes. We find that the stochastic semi-gradient noise due to subsampling the space of possible episodes leads to significant plateaus in the value error, unlike in traditional gradient descent dynamics. We study how learning dynamics and plateaus depend on feature structure, learning rate, discount factor, and reward function. We then analyze how strategies like learning rate annealing and reward shaping can favorably alter learning dynamics and plateaus. To conclude, our work introduces new tools to open a new direction towards developing a theory of learning dynamics in reinforcement learning.

Loss Dynamics of Temporal Difference Reinforcement Learning

Abstract

Reinforcement learning has been successful across several applications in which agents have to learn to act in environments with sparse feedback. However, despite this empirical success there is still a lack of theoretical understanding of how the parameters of reinforcement learning models and the features used to represent states interact to control the dynamics of learning. In this work, we use concepts from statistical physics, to study the typical case learning curves for temporal difference learning of a value function with linear function approximators. Our theory is derived under a Gaussian equivalence hypothesis where averages over the random trajectories are replaced with temporally correlated Gaussian feature averages and we validate our assumptions on small scale Markov Decision Processes. We find that the stochastic semi-gradient noise due to subsampling the space of possible episodes leads to significant plateaus in the value error, unlike in traditional gradient descent dynamics. We study how learning dynamics and plateaus depend on feature structure, learning rate, discount factor, and reward function. We then analyze how strategies like learning rate annealing and reward shaping can favorably alter learning dynamics and plateaus. To conclude, our work introduces new tools to open a new direction towards developing a theory of learning dynamics in reinforcement learning.

1 Introduction

Reinforcement learning (RL) is a general paradigm which allows agents to learn from experience the relative value of states in their environment and to take actions that maximize long term rewards [1]. RL algorithms have been successfully applied in a number of real world scenarios such as strategic games like backgammon and Go, autonomous vehicles, and fine tuning language models [2-7].

Despite these empirical successes, a theoretical understanding of the learning dynamics and inductive biases of RL algorithms is currently lacking [8]. A large fraction of the theoretical work has focused on proving convergence and deriving bounds both in the asymptotic [9-14] and non-asymptotic [15-17] limits, but do not provide a full picture of the evolution of the learning dynamics.

A desired feature of a candidate theory is to characterize the influence of function approximation to RL dynamics and its performance. Early versions of RL operated in a tabular setting, similar to dynamic programming [18], where all the states in the environment could be mapped one-to-one to a specific value and policy. In large and complex environments, it is not possible to enumerate all the states in the environment necessitating the use of function approximation for the target value and policy functions. Indeed, the recent success of many RL algorithms relies on deep reinforcement

learning architectures that combine an RL architecture with deep neural networks to build effective value estimators and policy networks [19].

One difficulty in analysing these algorithms compared to supervised learning settings is that the distribution of the data received at each time-step is not stationary. This non-stationarity arises from two principal sources: First, whether in an episodic or continuous setting, states visited within a learning trajectory are dependent on the recent past. Trajectories might be randomly sampled but points within a trajectory are correlated. Second, when the policy is updated it also changes the distribution of future visited states.

Here, we will focus on the first form of non-stationarity when learning a value function in the context of policy evaluation [1] using a classical RL algorithm, temporal difference (TD) learning [20]. We develop a theory of learning dynamics for RL in this setting in a high dimensional asymptotic limit with a focus on understanding the role of linear function approximation from a set of nonlinear and static features. In particular, we leverage ideas from recent work in application of statistical physics to machine learning theory to perform an average over the possible sequences of features encountered during learning. Our contributions are as follows:

We introduce concepts from statistical physics, including a path integral approach to describe dynamics [21-25] and the Gaussian equivalence assumption [26-29], to derive a theory of learning dynamics in TD learning (§3) in an online setting. We provide an analytical formula for the typical case learning curve for TD learning.

We show that our theory predicts scaling of the learning convergence speed and performance plateaus with parameters of the problem including task-feature alignment [30], learning rate, discount factor or batch size (§4 and §5). Task-feature alignment is a metric that quantifies how features allow fast or slow learning for a given task.

We show our theory can be used to understand and guide design principles when choosing metaparameters. Specifically, we show that we can use our theory to infer optimal schedules of learning rate annealing and the effects of reward shaping (§5 and §6).

2 Problem Setup and Related Works

2.1 Problem Setup

We consider a set of states denoted by s , possibly continuous, and a fixed policy π which generates a distribution over actions given the state. The state dynamics are defined by a distribution p ( τ ) over trajectories through state space τ = { s 1 , s 2 , ..., s T } . Note that state transitions do not have to be Markovian, but each trajectory is i.i.d. sampled from p ( τ ) . We consider trajectories of length T . Each state is represented by an N -dimensional feature vector ψ ( s ) ∈ R N , so that trajectory generates a collection of feature vectors { ψ ( s t ) } T t =1 . The rewards are generated by a reward function R ( s ) which depends on the state. (In general, the features and rewards can depend on action as well: transition dynamics are still fixed as the policy is fixed, but variance over rewards at a given state may need to be modeled, see Appendix B.5).

At any time, we are interested in characterizing the value function associated with a state, which measures the expected discounted sum of future rewards when starting in state s 0

We use linear function approximation to learn the value function ˆ V ( s ) = ψ ( s ) · w . Similar to kernel learning [31], the features ψ should be high dimensional so that they can express a large set of possible value functions.

We study TD learning dynamics given this setup. At each step of the TD iteration, we sample a batch of B independent trajectories from the distribution and compute the TD update

We operate in a online batch regime as the trajectories in each batch are resampled at each iteration. This is distinct from an offline setting where the batches would be resampled from a finite-sized buffer [1]. Convergence considerations for infinite-batch online TD learning width different types of features ψ are outlined in Appendix A. The specific form for the TD-error ∆ µ n ( t ) depends on the precise variant of TD learning that is used. Here, we will focus on TD(0) but our approach can be extended to other TD learning rules and definitions of the return function. We see that the iterates w n will form a stochastic process as each sequence of states in an episode { s µ n ( t ) } are drawn randomly from p ( τ ) . In general, we allow the learning rate η n to depend on iteration, an important point we will revisit later. The distribution of features { ψ ( s µ n ( t )) } over random trajectories τ is in general quite complicated, depending on the details of the state transitions and the nonlinear feature maps, which motivates the following question:

Question: How can the stochastic dynamics of temporal difference learning be characterized for complicated trajectory distributions p ( τ ) and feature maps ψ ( s ) ?

To address this question, in this work, we provide an analysis of TD learning that explicitly models the statistics of stochastic semi-gradient updates to w n . Our framework is based on a Gaussian equivalence ansatz for TD learning and high dimensional mean field theory which predicts the statistics of TD errors ∆ µ n ( t ) and the weight iterates w n . The theory reveals a rich set of phenomena including plateaus unique to SGD noise in TD learning which can be ameliorated with learning rate annealing.

2.2 Related Works

The dynamics of TD learning have been notoriously difficult to analyse. Unlike supervised learning settings, sampled states are correlated across a trajectory and the algorithms involve bootstrapping: using estimates of the value function for future states in the temporal difference update [1]. Some prior works study the least-square TD learning rule, which solves, at each step n of the algorithm, a linear system for the instantaneous best fit to n samples [32-34]. Alternatively, many works focus on the online SGD version of TD learning, where incremental updates are made to the parameters at each step, using fresh samples. This is the setting of our work. The focus of this literature has initially been to prove convergence and bounds on asymptotic behavior [11-14, 35]. More recently, progress has been made in deriving bounds in the non-asymptotic regime. Initial work assumed that data samples were i.i.d. [15-17, 36] and recent work has extended those approaches to Markovian noise [15, 37-39]. The majority of these proofs use the ODE-like method for stochastic approximation [11, 40], which corresponds to a limit of the stochastic semi-gradient dynamics where the effects of mini-batch noise are neglected. This is also known as the 'mean-path' dynamics of TD learning and will correspond to the infinite batch limit of our theory. Furthermore, many of these methods require the use of iterative averaging of the learned value function, whereas we study the final iterate convergence. The approach we take here differs from many of these results as our goal is not to provide bounds on worst-case behavior but instead to provide a full description of the dynamics of the typical case scenario during learning.

Our approach also highlights the importance of the structure of the representations in controlling the dynamics of learning. This had been long been recognized in reinforcement learning and previous works proposed to improve feature representations to improve algorithmic performance [41-43]. This line of work has shown the importance of the relative smoothness of the representations and target functions in the ODE limit of TD dynamics [43, 44]. Similarly, several methods have been proposed to empirically learn a better shaping function [45, 46]. In policy learning it has also been recognized that using a gradient aligned to the statistics of the tasks, such as the natural gradient [47] can greatly speed up convergence [48]. Our work does not explore such feature learning per se but could be used as a diagnostic tool to analyse how representations impact learning speed.

We adopt the perspective of statistical physics, by working with a simplified feature distribution which captures the learning dynamics and solving the theory in a high-dimensional limit [49-51]. We derive TD reinforcement learning curves from a mean field theory formalism which is exact for infinite dimensional features and batch size. Similar calculations for supervised learning on Gaussian data have been shown to provide an accurate description of high dimensional dynamics [52-54]. Further, even when data is not actually Gaussian, several algorithms, such as kernel or random-features regression, exhibit universality in their loss behavior, enabling analysis of the learning curve with a simpler Gaussian proxy [26-28, 30, 55]. We exploit this idea in the TD learning setting to some success. We note that Gaussian equivalence or universality is not a panacea, and in many cases the Gaussian proxy can fail to capture important machine learning phenomena [27, 56, 57].

3 Theoretical Results for Online TD Learning

3.1 Computation of Learning Curves

We develop a dynamical mean field theory (DMFT) formalism can be utilized to compute the learning curves. We provide the full derivation of the DMFT in Appendix B. This computation consists of tracking the moment generating function for the iterates w n over the trajectories of randomly sampled features { ψ n µ ( t ) } T t =1 . In an appropriate high dimensional asymptotic limit, the results of our theory can be summarized as the following proposition.

Proposition 3.1. Let N,B →∞ with B/N = O (1) and episode length T = O (1) . Let the ground truth reward function be R ( s ) = w R · ψ ( s ) and value function V ( s ) = w TD · ψ ( s ) in the basis of our features. Define matrices

and assume that the features are such that matrix A is of extensive rank in N . Then the typical value estimation error L n = 〈 ( V ( s ) -ˆ V n ( s ) ) 2 〉 s after n steps has the form

where α = B/N and Q n ( t, t ′ ) = 〈 ∆ n ( t )∆ n ( t ′ ) 〉 is the correlation of randomly sampled TD-errors at episodic times t, t ′ and iteration n . The average over weights 〈〉 denotes a Gaussian average whose moments are related to M n . The correlation function Q n ( t, t ′ ) depends on M n and the average weights 〈 w n 〉 ; we provide its full formula in Appendix B.3, equation (B.17) .

Proof. The full derivation is in Appendix B. At a high level, we track the moment generating function of the iterates w n over random draws of features { ψ µ n ( t ) } , Z [ { j n } ] = E { ψ µ n ( t ) } exp( i ∑ n j n · w n ) ∝ ∫ D q exp ( N 2 S [ q, { j n } ] ) where S is a O (1) action and q are a set of order parameters of the theory which include the following overlaps C n ( t, t ′ ) = 1 N w glyph[latticetop] n Σ ( t, t ′ ) w n and Q n ( t, t ′ ) = 1 B ∑ B µ =1 ∆ µ n ( t )∆ µ n ( t ′ ) . In this high dimension N,B →∞ limit with B/N = O (1) and episode length T = O (1) , the order parameters can be obtained from saddle point integration, which requires solving ∂S ∂q = 0 . This procedure results in a deterministic learning curve given in equations (4),(5),(6) even though the realization of sampled states are disordered. The TD-error variables ∆ n ( t ) become mean zero Gaussians and the { w n } also follow a Gaussian distribution with mean and variance determined by the order parameters.

Before we explore the predictions of this theory, we first make a few remarks about this result. Remark 1 . Though the theory is technically derived for large batch size B , we will show that it provides an accurate description of the loss trajectory even for batches as small as B = 1 . An alternative formulation in terms of recursive averaging reveals transparently which approximations lead to the same result as the mean field theory (Appendix B.6).

Remark 2 . The case where the reward function and/or the value function are inexpressible by the features ψ can also be handled within this framework. In this case, the unlearnable components of the value function act as additional noise which limits performance [29]. These can also be handled by our theory, see Appendix A.

Remark 3 . The limit where γ = 0 recovers known results in online supervised learning with stochastic gradient methods [29, 58, 59]. In this limit, the dynamics will converge to zero loss provided the model features are sufficiently rich to represent the true value function.

Remark 4 . The TD learner with perfect coverage (infinite batch size) at each step will converge to the ground truth w TD = ( ¯ Σ -γ ¯ Σ + ) -1 ¯ Σ w R (see Appendix A).

Remark 5 . M n is equivalently defined as M n = 〈 ( w -w TD )( w -w TD ) glyph[latticetop] 〉 { τ µ n ′ } n ′ <n , which measures deviation from the fixed point of gradient flow (vanishing learning rate) dynamics w TD over random sets of sampled episodes (Appendix B).

3.2 Gaussian Approximation

The theory presented in Section 3.1 relies on an approximation of the feature distribution as Gaussian. Similar approximations have been successfully utilized in high dimensional regression problems even when the true features are non-Gaussian [26-29]. We note that an exact, non-asymptotic theory for non-Gaussian features can be provided which closes under knowledge of the fourth cumulants of the features as we show in Appendix D, though this theory is especially cumbersome to analyze or evaluate compared to the theory of Section 3.1. Concretely, Proposition 3.1 relies on the following.

Gaussian Feature Assumption. The learning curves for a TD learner with high dimensional features { ψ ( s t ) } T t =1 over random τ are well approximated by the learning curves of a TD learner trained with Gaussian features ψ G ∼ N ( µ , Σ + µµ glyph[latticetop] ) with matching mean and correlations

where averages are taken over sequences of states { s ( t ) } ∼ p ( τ ) .

(a) 2D Exploration

(b) Place Cell/RBF Features

ψ

(

s

t

)

 

 

⋃√˜√√

 

 

 

∧∐⌈√˜(〉√√}√

⊕〈∑

⋂[˜}√glyph[arrowbt]

3

2

1

0

1

2

3

r

0.0

0.2

0.4

0.6

0.8

1.0

i

(

r

r

i

)

BW = 0.25

BW = 0.5

BW = 0.75

BW = 1.0

(d) Place Cell Bandwidths

 

 

⋃√˜√√

 

 

∧∐⌈√˜(〉√√}√

∨((/

∨((/

∨((/

∨((/

(e) Large batch

B

= 30

 

 

⋃√˜√√

 

 

∧∐⌈√˜(〉√√}√

∨((/

∨((/

∨((/

∨((/

(f) Small batch

B

= 3

⊔∐√√√]∐{

One interpretation of this ansatz is that the dependence of the learning curve on higher order cumulants of the features is negligible in high dimensional feature spaces under the square loss. This

approximation has been shown to provide an accurate description on realistic supervised learning settings with non-Gaussian data with the square loss in prior works [26, 27, 29, 30, 55, 58]. As shown in these works, for standard supervised learning, even highly non-Gaussian features { ψ ( s t ) } have least squares learning curves which are only sensitive to the first two cumulants of the distribution. We do not aim to provide a rigorous proof of this ansatz for TD learning but instead compute the learning curve implied by this assumption and compare to experiments on simple Markov Decision Processes (MDPs). The benefit of this hypothesis in the RL setting is that it abstracts away details of transitions in the state space and instead deals with the correlations of sampled features through time.

To illustrate an example of the Gaussian Equivalence idea, in Figure 1, we consider an MDP which is defined by diffusion through a 2-dimensional (2D) state space (Figure 1(a)). We choose the features ψ ( s ) to be a collection of localized 2D Radial Basis Function (RBF) bumps which tile the 2D space, similarly to the 'place cell' neurons found in the mammalian hippocampus [60, 61] (Figure 1(b)). The feature map is parameterized by the bandwidth of individual 'place cells'. In Figure 1(c), we show the value error learning curve as a function of the number of steps n (blue) and compare the value estimation error of the MDP with a Gaussian distribution for ψ ( t ) with matching first and second moments (orange). Lastly, we plot the theoretical prediction of our theory (described in Section 3), which is computed under the Gaussian equivalence ansatz (black dashed). We see a remarkable match of the three curves. The equivalence can be used to predict the speed of TD learning for different features, such as place cells with varying bandwidth as we illustrated in Figure 1 (d)-(f). In Figure 1 (e) and (f), we plot the loss trajectories for a single run of TD for each feature set. We observe that bandwidth affects both the learning dynamics and the asymptotic error with an optimal bandwidth at any step. One of our goals will be to elucidate the role of feature quality in learning dynamics. While the large batch dynamics are approximately self-averaging, as shown by the fact that single runs of TD learning coincide with our theoretical typical case theory curves, there is significant semi-gradient variance in the value error at small batch sizes. While we expect Gaussian equivalence to hold for high dimensional features, in low dimensions non-Gaussian effects can significantly alter the learning curves as we show in Appendix D.1. However, for high dimensional features, the equivalence holds for many other feature distributions such as polynomial and fourier features (Appendix E).

4 Spectral Perspective on Hard Reward Functions

R

1(

s

)

R

2(

s

)

(a) Sparse

R

1

or Dense

R

2

Rewards

0

50

100

150

200

250

300

350

k

0.0

0.2

0.4

0.6

0.8

1.0

C

(

k

)

R

1(

s

)

R

2(

s

)

(b) Code-Task Alignment

 

 

⋃√˜√√

 

 

∧∐⌈√˜(〉√√}√

(c)

B

= 20

Learning Curves

Our theory can provide some insights into the structure of tasks which can be learned easily and which require more sampled trajectories to estimate based on spectral decompositions of the feature covariances. We note that similar spectral arguments have been given in the ODE-limit [44] and are intimately related to the source conditions used in recent work to identify power-law rates in the large batch regime [39].

To build our argument, we diagonalize the matrix A = ¯ Σ -γ ¯ Σ + , obtaining Au k = λ k u k , noting that eigenvalues λ k can be complex. We then expand the TD solution in this basis w TD = ∑ k w k u k .

The theory predicts that, the average learned weights will be 〈 w n 〉 = ∑ k | 1 -ηλ k | n e iθ k n w k u k , where | · | is complex modulus and θ k = Arg (1 -ηλ k ) . We can therefore order the modes by their convergence timescales | 1 -ηλ k | . Given this ordering of timescales, we can order the modes k from those with smallest to largest timescales. Given this ordering, we see that tasks can be learned efficiently are those with most of the norm of w k in the modes with small timescales. We quantify how well aligned a task is to a given feature representation by computing a cumulative power distribution for the target weights C ( k ) = ∑ glyph[lscript]<k w 2 glyph[lscript] ∑ glyph[lscript] w 2 glyph[lscript] . If this quantity rises rapidly with k then the task can be learned from a small number of samples [30].

We consider again, the setting of Figure 1, the 2D exploration MDP but now contrast two different reward functions. In Figure 2 we show that this spectral decomposition can account for the gaps in loss for a place cell code in learning a sparse or dense reward function (Figure 2(a)). As expected the cumulative power rises more rapidly for the dense reward function R 2 ( s ) (Figure 2(b)). As a consequence, the value error converges to zero more rapidly than for the sparse rewards.

5 Stochastic Semi-Gradient Learning Plateaus and Annealing Strategies

 

 

√√˜√√

 

 

 

∧∐⌈√˜(〉√√}√

((

((

((

((

((

10 0

10 1

10 2

steps

10

5

10

4

10

3

10

2

10

1

10 0

Value Error

=0.1

=0.25

=0.5

=0.8

=0.9

(b) Discount Factor

10 0

10 1

10 2

steps

10

3

10

2

10

1

10 0

Value Error

=0.01

=0.02

=0.05

=0.1

(c) Fixed Learning Rate

10 0

10 1

10 2

steps

10

3

10

2

10

1

10 0

Value Error

: 0.00

: 0.20

: 0.75

: 1.25

0.0

0.5

1.0

1.5

2.0

10

3

10

2

10

1

Value Error

n

=50

n

=100

n

=200

n

=500

(e) Optimal Annealing

The stochastic noise from TD learning has striking qualitative differences from SGD noise in the standard supervised case. In standard supervised learning (such as γ = 0 version of this theory), the stochastic gradient noise does not prevent the model from fitting the target function with zero error provided the features are sufficiently rich to represent the target function. However, this is not the case in TD learning, where the predicted value ˆ V ( s ) is bootstrapped using the model's weights w n at each iteration n . This leads to asymptotic plateaus in learning curves. Our theory can predict these plateaus and their scaling whose proof is given in Appendix B.7.

Proposition 5.1. Our theoretical learning curves exhibit a fixed point for the value error dynamics for finite B and non-zero η and γ . For small ηγ 2 B , we deduce that M satisfies a self-consistent

𝜃

(1 + 𝛽)𝑤

!"

𝑤

!"

cos 𝜃

+𝑤

#

sin(𝜃)

𝑤

!"

(a) Geometry of Reward Shaping

10 0

10 1

10 2

Steps

10

4

10

3

10

2

10

1

10 0

Value Error

= -0.8

= -0.5

= 0.0

= 0.5

= 1.0

(b) Scale-based Shaping

10 0

10 1

10 2

Steps

10

2

10

1

10 0

Value Error

= 0

=

/4

=

/2

(c) Rotation-based Shaping

asymptotic scaling of the form M = O ( ηγ 2 B ) impliying an asymptotic value error scaling of L ∼ 1 N Tr M ¯ Σ ∼ O ( ηγ 2 B ) .

In Figure 3, we demonstrate that our theory predicts the plateaus and their scaling as a function of finite batch size B (Figure 3(a)), non-zero discount factor γ > 0 (Figure 3(b)) and non-negligible learning rate (Figure 3(c)).

A strategy used in the literature to increase rates of convergence and improve asymptotic behavior is adaptation of the learning learning through an annealing schedule [1, 16, 62, 63]. To overcome this plateau in the loss, we consider annealing the learning rate η n with iteration n . In Figure 3(d), we show the effect of annealing the learning rate as a power law η n = η 0 n -χ for some non-negative exponent χ . For χ = 0 the learning rate is constant and a fixed plateau is reached. For small nonzero χ , such as χ = 0 . 2 , the value error is, after an initial transient, always near its instantaneous fixed point plateau so the loss scales linearly with the learning rate, giving the asymptotic rate L n ∼ O ( n -χ ) . For large χ , the learning rate decreases very quickly and the plateau is never reached. Our approach can be used to find an optimal annealing exponent χ and in Figure 3(e), we show that the optimal annealing exponent balances these effects and is well predicted by our theory.

6 Reward Shaping

Another strategy to improve the learning dynamics in reinforcement learning algorithms is reward shaping [64]. In standard supervised learning, the goal is to directly approximate the target objective given a cost function. However, in reinforcement learning, the objective is not to estimate rewards at each state directly but the discounted sum of future rewards, the value function. Importantly, many different reward schedules can lead to identical value functions. Reward shaping exploits this symmetry to speed up learning by altering the structure of TD updates and SGD noise. Here, we provide a theoretical description of the changes in the learning dynamics due to reward shaping which suggests they can be understood through a change of the alignment between the original rewards and the reshaped rewards in the space of the features used to represent the states.

The original ideas around reward shaping were inspired by work in experimental psychology and were closer to what is now studied as curriculum learning [65-67]. Reward shaping as currently used in reinforcement learning directly changes the reward function by adding a potential-based shaping function F such that F ( s t , a, s t +1 ) = γφ ( s t +1 ) -φ ( s t ) [64]. In each step of the algorithm we feed

the following reshaped rewards ˜ R to the TD learner

We note that this transformation simply offsets the target value function by φ ( s ) as the series above telescopes with a cancellation of φ ( s t ) between the t -1 and t -th terms [64] (see Appendix C). However, the dynamics of TD learning with these reshaped rewards ˜ R is quite distinct from the dynamics with original rewards R . Here, we study the case where we can express φ ( s ) as a linear function of our features: φ ( s ) = ψ ( s ) · w φ . This leads to a change in the dynamics for M n and 〈 w n 〉 that we describe in the Appendix C.

In Figure 4, we illustrate the possible benefits of reward shaping. We explore two types of reward shaping. First, a scale based reward shaping where w φ is parallel to the target TD weights w TD . This merely changes the overall scale of the weights needed to converge in the dynamics, leading to similar timescales and an identical plateau for TD learning as we show in Figure 4 (b). On the other hand, reward shaping which rotates the fixed point of the TD dynamics into directions of higher feature variance can improve timescales of convergence. In Figure 4 (c), we show an example where we vary the angle θ of the shaped-TD fixed point (see also Appendix C).

7 TD Learning Plateaus in More Realistic Settings

In this section, we test if some of the phenomena observed in our theory and experiments also hold in more realistic settings. We perform TD learning with Fourier features to evaluate a pre-trained policy on MountainCar-v0. As expected, we see that the value error plateaus to an error level determined by both the learning rate (Figure 5a) and batch size (Figure 5b) due to semigradient noise.

10 0

10 1

10 2

10 3

10 4

10 5

10 6

10 7

Batch

10 1

10 2

10 3

10 4

10 5

10 6

10 7

Value Error

=0.01

=0.02

=0.05

=0.1

=0.2

(a) Varying Learning Rates

10 0

10 1

10 2

10 3

10 4

10 5

10 6

Batch

10 1

10 2

10 3

10 4

10 5

10 6

10 7

Value Error

B

=1

B

=2

B

=4

B

=8

(b) Varying Batch Sizes

We show that the plateaus obey the predicted scalings of O ( ηB -1 ) in Appendix F.

8 Discussion

Our work presents a new approach using concepts from statistical physics to derive average-case learning curve for policy evaluation in TD-learning. However, it is only a first step towards a new theory of learning dynamics in reinforcement learning.

One major limitation of the present work is that it concerns linear function approximation where the features representing states/actions are fixed throughout learning. This limit can apply to neural networks in the 'lazy' regime of training [68, 69], however it cannot account for neural networks that adapt their internal representations to the structure of the reward function. This differs from the setting of most practical algorithms, including in deep reinforcement learning, that specifically adapt their representations.

Our theory provides a description of learning dynamics through a set of iterative equations (Proposition 3.1). In Figure 1 we evaluate these dynamics for a simple MDP but although the predicted dynamics present an excellent fit to the empirical simulations, the iterative equations can be difficult to interpret and computationally expensive to evaluate in a larger network and more realistic tasks. Nevertheless, our equations can be used to derive some scaling between key parameters of the algorithm for example by studying their fixed points as in Proposition 5.1.

Here, we considered the simplest form of temporal difference learning, batched online TD(0). In future work, it will be important to further characterize the behavior for online TD(0) with batch size B = 1 and to expand our approach to TD( λ ) and other return distributions. Similarly, expanding our theory to the offline setting, in which the buffer of resampled trajectories would be of finite size, could provide an understanding of how the interactions between parameters govern convergence and divergence [1, 70-72].

Another limitation of our work is that we only considered the setting of policy evaluation with a fixed policy. The goal of an RL agent is to learn how to act in the work and not merely to represent the value is its states. Unlike in supervised learning, the changes in the value function affect the policy but in many of RL algorithms, for example in actor-critic architecture, there is a separation of the policy evaluation (critic) and the policy learning (actor) [73, 74]. Such algorithms estimate the value associated with state/action pairs under a given policy and then use this information to make beneficial updates to the policy, usually with the value and policy functions approximated by separate neural networks. In this paper, we only treated the first part of this process. Recently, a related approach has been used to analyse the dynamics of policy learning in an 'RL perceptron" setup [75]. A full theory of reinforcement learning combining policy evaluation and policy learning remains difficult due to the interaction between the two processes, but combining these approaches would be fruitful. One promising direction is in settings where the timescales of the two processes are different [76], such as when policy learning occurring at a much slower rate which is often the case in practice.

Beyond developing a theory of learning dynamics in reinforcement learning, the approach could be used in neuroscience to understand how neural representation of space or value can shape the learning dynamics at the behavioral level. Ideas from reinforcement learning have been extremely influential to understand phenomena observed in neuroscience and have been mapped directly onto specific brain circuits [77-79]. The place cells of the hippocampus [60] exhibit localized tuning as the example in Figure 1 and together with grid cells in enthorinal cortex are thought to be crucial for navigation in spatial and cognitive spaces and their tuning is shaped by experience [61, 79-81]. Our theory specifically link the structure of representations, policy and reward to learning rates, which can all be experimentally measured simultaneously and could shed on light on how the spectral properties of representations govern learning and navigation [79, 82], similarly to how the mean field theories we have used here can explain learning of sensory features [83]. Future work could straightforwardly extend this DMFT formalism to deal with replay of sampled experiences during TD learning [84] at the cost of tracking correlations of weight updates across iterations of the algorithm [52].

To summarize, our work provide a new promising direction towards a theory of learning dynamics in reinforcement learning in artificial and biological agents.

Acknowledgments and Disclosure of Funding

BB is supported by a Google PhD Fellowship. CP and BB were supported by NSF grant DMS2134157. CP is further supported by NSF CAREER Award IIS-2239780, and a Sloan Research Fellowship. PM was supported by NIH grant 5R01DC017311 to Venkatesh Murthy and Naoshige Uchida. HK was supported by the Harvard College Research Program. This work has been made possible in part by a gift from the Chan Zuckerberg Initiative Foundation to establish the Kempner Institute for the Study of Natural and Artificial Intelligence. We thank Jacob Zavatone-Veth for useful discussions and comments on this manuscript.

References

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Gerald Tesauro. Temporal difference learning and td-gammon. Communications of the ACM , 38(3):58-69, 1995.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature , 518(7540):529-533, 2015.

David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature , 529(7587):484-489, 2016.

Peter R Wurman, Samuel Barrett, Kenta Kawamoto, James MacGlashan, Kaushik Subramanian, Thomas J Walsh, Roberto Capobianco, Alisa Devlic, Franziska Eckert, Florian Fuchs, et al. Outracing champion gran turismo drivers with deep reinforcement learning. Nature , 602(7896): 223-228, 2022.

B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent Transportation Systems , 23(6):4909-4926, 2021.

Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593 , 2019.

Matteo Hessel, Hado van Hasselt, Joseph Modayil, and David Silver. On inductive biases in deep reinforcement learning. arXiv preprint arXiv:1907.02908 , 2019.

Peter Dayan. The convergence of td () for general. Machine learning , 8(3):341-362, 1992.

Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning , 8:279-292, 1992.

JN Tsitsiklis and B Vanroy. An analysis of temporal-difference learning with function approximation. IEEE Transactions on Automatic Control , 42(5):674-690, 1997.

Geoffrey J Gordon. Reinforcement learning with function approximation converges to a region. Advances in neural information processing systems , 13, 2000.

Michael Kearns and Satinder Singh. Near-optimal reinforcement learning in polynomial time. Machine learning , 49:209-232, 2002.

Peter Auer, Thomas Jaksch, and Ronald Ortner. Near-optimal regret bounds for reinforcement learning. Advances in neural information processing systems , 21, 2008.

Jalaj Bhandari, Daniel Russo, and Raghav Singal. A finite time analysis of temporal difference learning with linear function approximation, 2018. URL https://arxiv.org/abs/1806. 02450 .

Gal Dalal, Balázs Szörényi, Gugan Thoppe, and Shie Mannor. Finite sample analyses for td (0) with function approximation. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018.

Chandrashekar Lakshminarayanan and Csaba Szepesvari. Linear stochastic approximation: How far does constant step-size and iterate averaging go? In Amos Storkey and Fernando Perez-Cruz, editors, Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics , volume 84 of Proceedings of Machine Learning Research , pages 1347-1355. PMLR, 09-11 Apr 2018. URL https://proceedings.mlr.press/v84/lakshminarayanan18a. html .

Richard E Bellman. Dynamic programming . Princeton university press, 2010.

Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. Deep reinforcement learning: A brief survey. IEEE Signal Processing Magazine , 34(6):26-38, 2017.

Richard Stuart Sutton. Temporal credit assignment in reinforcement learning . University of Massachusetts Amherst, 1984.

Paul Cecil Martin, ED Siggia, and HA Rose. Statistical dynamics of classical systems. Physical Review A , 8(1):423, 1973.

A Crisanti and H Sompolinsky. Path integral approach to random neural networks. Physical Review E , 98(6):062120, 2018.

Moritz Helias and David Dahmen. Statistical field theory for neural networks , volume 970. Springer, 2020.

Blake Bordelon and Cengiz Pehlevan. Self-consistent dynamical field theory of kernel evolution in wide neural networks. arXiv preprint arXiv:2205.09653 , 2022.

Blake Bordelon and Cengiz Pehlevan. The influence of learning rule on representation dynamics in wide neural networks. arXiv preprint arXiv:2210.02157 , 2022.

Blake Bordelon, Abdulkadir Canatar, and Cengiz Pehlevan. Spectrum dependent learning curves in kernel regression and wide neural networks. In International Conference on Machine Learning , pages 1024-1034. PMLR, 2020.

Bruno Loureiro, Cedric Gerbelot, Hugo Cui, Sebastian Goldt, Florent Krzakala, Marc Mezard, and Lenka Zdeborová. Learning curves of generic features maps for realistic datasets with a teacher-student model. Advances in Neural Information Processing Systems , 34:18137-18151, 2021.

Hong Hu and Yue M Lu. Universality laws for high-dimensional learning with random features. IEEE Transactions on Information Theory , 2022.

Blake Bordelon and Cengiz Pehlevan. Learning curves for SGD on structured features. In International Conference on Learning Representations , 2022. URL https://openreview. net/forum?id=WPI2vbkAl3Q .

Abdulkadir Canatar, Blake Bordelon, and Cengiz Pehlevan. Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks. Nature communications , 12(1):2914, 2021.

Bernhard Schölkopf, Alexander J Smola, Francis Bach, et al. Learning with kernels: support vector machines, regularization, optimization, and beyond . MIT press, 2002.

Manel Tagorti and Bruno Scherrer. On the rate of convergence and error bounds for lstd ( λ ). In International Conference on Machine Learning , pages 1521-1529. PMLR, 2015.

Yangchen Pan, Adam White, and Martha White. Accelerated gradient temporal difference learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 31, 2017.

Alborz Geramifard, Michael Bowling, Martin Zinkevich, and Richard S Sutton. ilstd: Eligibility traces and convergence analysis. Advances in Neural Information Processing Systems , 19, 2006.

Fernando J Pineda. Mean-field theory for batched td ( λ ). Neural computation , 9(7):1403-1419, 1997.

Gandharv Patil, LA Prashanth, Dheeraj Nagaraj, and Doina Precup. Finite time analysis of temporal difference learning with linear function approximation: Tail averaging and regularisation. In International Conference on Artificial Intelligence and Statistics , pages 5438-5448. PMLR, 2023.

Rayadurgam Srikant and Lei Ying. Finite-time error bounds for linear stochastic approximation andtd learning. In Conference on Learning Theory , pages 2803-2830. PMLR, 2019.

LAPrashanth, Nathaniel Korda, and Rémi Munos. Concentration bounds for temporal difference learning with linear function approximation: the case of batch data and uniform sampling. Machine Learning , 110:559-618, 2021.

Eloïse Berthier, Ziad Kobeissi, and Francis Bach. A non-asymptotic analysis of non-parametric temporal-difference learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 7599-7613. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/ 32246544c237164c365c0527b677a79a-Paper-Conference.pdf .

Vivek S Borkar and Sean P Meyn. The ode method for convergence of stochastic approximation and reinforcement learning. SIAM Journal on Control and Optimization , 38(2):447-469, 2000.

Ishai Menache, Shie Mannor, and Nahum Shimkin. Basis function adaptation in temporal difference reinforcement learning. Annals of Operations Research , 134(1):215-238, 2005.

Sridhar Mahadevan and Mauro Maggioni. Proto-value functions: A laplacian framework for learning representation and control in markov decision processes. Journal of Machine Learning Research , 8(10), 2007.

Marc Bellemare, Will Dabney, Robert Dadashi, Adrien Ali Taiga, Pablo Samuel Castro, Nicolas Le Roux, Dale Schuurmans, Tor Lattimore, and Clare Lyle. A geometric perspective on optimal representations for reinforcement learning. Advances in neural information processing systems , 32, 2019.

Clare Lyle, Mark Rowland, Will Dabney, Marta Kwiatkowska, and Yarin Gal. Learning dynamics and generalization in deep reinforcement learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 14560-14581. PMLR, 17-23 Jul 2022. URL https://proceedings.mlr.press/v162/lyle22a.html .

Yujing Hu, Weixun Wang, Hangtian Jia, Yixiang Wang, Yingfeng Chen, Jianye Hao, Feng Wu, and Changjie Fan. Learning to utilize shaping rewards: A new approach of reward shaping. Advances in Neural Information Processing Systems , 33:15931-15941, 2020.

Haosheng Zou, Tongzheng Ren, Dong Yan, Hang Su, and Jun Zhu. Learning task-distribution reward shaping with meta-learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 11210-11218, 2021.

Shun-Ichi Amari. Natural gradient works efficiently in learning. Neural computation , 10(2): 251-276, 1998.

Sham M Kakade. A natural policy gradient. Advances in neural information processing systems , 14, 2001.

Hyunjune Sebastian Seung, Haim Sompolinsky, and Naftali Tishby. Statistical mechanics of learning from examples. Physical review A , 45(8):6056, 1992.

Andreas Engel and Christian Van den Broeck. Statistical mechanics of learning . Cambridge University Press, 2001.

Yasaman Bahri, Jonathan Kadmon, Jeffrey Pennington, Sam S Schoenholz, Jascha SohlDickstein, and Surya Ganguli. Statistical mechanics of deep learning. Annual Review of Condensed Matter Physics , 11:501-528, 2020.

Francesca Mignacco, Florent Krzakala, Pierfrancesco Urbani, and Lenka Zdeborová. Dynamical mean-field theory for stochastic gradient descent in gaussian mixture classification. Advances in Neural Information Processing Systems , 33:9540-9550, 2020.

Cedric Gerbelot, Emanuele Troiani, Francesca Mignacco, Florent Krzakala, and Lenka Zdeborova. Rigorous dynamical mean field theory for stochastic gradient descent methods. arXiv preprint arXiv:2210.06591 , 2022.

Michael Celentano, Chen Cheng, and Andrea Montanari. The high-dimensional asymptotics of first order methods with random data. arXiv preprint arXiv:2112.07572 , 2021.

James B Simon, Madeline Dickens, Dhruva Karkada, and Michael Deweese. The eigenlearning framework: A conservation law perspective on kernel ridge regression and wide neural networks. Transactions on Machine Learning Research , 2023. ISSN 2835-8856. URL https://openreview.net/forum?id=FDbQGCAViI .

Maria Refinetti, Alessandro Ingrosso, and Sebastian Goldt. Neural networks trained with sgd learn distributions of increasing complexity. In International Conference on Machine Learning , pages 28843-28863. PMLR, 2023.

Alessandro Ingrosso and Sebastian Goldt. Data-driven emergence of convolutional structure in neural networks. Proceedings of the National Academy of Sciences , 119(40):e2201854119, 2022.

Aditya Vardhan Varre, Loucas Pillaud-Vivien, and Nicolas Flammarion. Last iterate convergence of sgd for least-squares in the interpolation regime. Advances in Neural Information Processing Systems , 34:21581-21591, 2021.

Maksim Velikanov, Denis Kuznedelev, and Dmitry Yarotsky. A view of mini-batch SGD via generating functions: conditions of convergence, phase transitions, benefit from negative momenta. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=bzaPGEllsjE .

John O'Keefe. Place units in the hippocampus of the freely moving rat. Experimental neurology , 51(1):78-109, 1976.

Edvard I Moser, Emilio Kropff, and May-Britt Moser. Place cells, grid cells, and the brain's spatial representation system. Annu. Rev. Neurosci. , 31:69-89, 2008.

Robert A Jacobs. Increased rates of convergence through learning rate adaptation. Neural networks , 1(4):295-307, 1988.

William Dabney and Andrew Barto. Adaptive step-size for online temporal difference learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 26, pages 872-878, 2012.

Andrew Y Ng, Daishi Harada, and Stuart J Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Proceedings of the Sixteenth International Conference on Machine Learning , pages 278-287, 1999.

Burrhus Frederic Skinner. Science and human behavior . Number 92904. Simon and Schuster, 1965.

Vijaykumar Gullapalli and Andrew G Barto. Shaping as a method for accelerating reinforcement learning. In Proceedings of the 1992 IEEE international symposium on intelligent control , pages 554-559. IEEE, 1992.

Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning , pages 41-48, 2009.

Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. Advances in neural information processing systems , 32, 2019.

Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31, pages 8571-8580. Curran Associates, Inc., 2018. URL https://proceedings. neurips.cc/paper/2018/file/5a4be1fa34e62bb8a6ec6b91d2462f5a-Paper.pdf .

Hado Van Hasselt, Yotam Doron, Florian Strub, Matteo Hessel, Nicolas Sonnerat, and Joseph Modayil. Deep reinforcement learning and the deadly triad. arXiv preprint arXiv:1812.02648 , 2018.

Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 , 2020.

Juan Perdomo, Akshay Krishnamurthy, Peter L Bartlett, and Sham Kakade. A sharp characterization of linear estimators for offline policy evaluation. Journal of machine learning research , 2023.

Vijay Konda and John Tsitsiklis. Actor-critic algorithms. Advances in neural information processing systems , 12, 1999.

Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning , pages 1928-1937. PMLR, 2016.

Nishil Patel, Sebastian Lee, Stefano Sarao Mannelli, Sebastian Goldt, and Andrew M Saxe. The rl perceptron: Dynamics of policy learning in high dimensions. In ICLR 2023 Workshop on Physics for Machine Learning , 2023.

Vijay R Konda and John N Tsitsiklis. Convergence rate of linear two-time-scale stochastic approximation. Annals of Applied Probability , pages 796-819, 2004.

Wolfram Schultz, Peter Dayan, and P Read Montague. A neural substrate of prediction and reward. Science , 275(5306):1593-1599, 1997.

Kenji Doya. Modulators of decision making. Nature neuroscience , 11(4):410-416, 2008.

Timothy EJ Behrens, Timothy H Muller, James CR Whittington, Shirley Mark, Alon B Baram, Kimberly L Stachenfeld, and Zeb Kurth-Nelson. What is a cognitive map? organizing knowledge for flexible behavior. Neuron , 100(2):490-509, 2018.

Kimberly L Stachenfeld, Matthew M Botvinick, and Samuel J Gershman. The hippocampus as a predictive map. Nature neuroscience , 20(11):1643-1653, 2017.

Marielena Sosa and Lisa M Giocomo. Navigating for reward. Nature Reviews Neuroscience , 22 (8):472-487, 2021.

Daniel C McNamee, Kimberly L Stachenfeld, Matthew M Botvinick, and Samuel J Gershman. Flexible modulation of sequence generation in the entorhinal-hippocampal system. Nature neuroscience , 24(6):851-862, 2021.

Blake Bordelon and Cengiz Pehlevan. Population codes enable learning from few examples by shaping inductive bias. Elife , 11:e78606, 2022.

William Fedus, Prajit Ramachandran, Rishabh Agarwal, Yoshua Bengio, Hugo Larochelle, Mark Rowland, and Will Dabney. Revisiting fundamentals of experience replay. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 3061-3071. PMLR, 13-18 Jul 2020. URL https://proceedings.mlr.press/v119/fedus20a.html .

Appendix

A General Convergence Considerations for MDPs in Finite State Space

In this section, we will discuss the infinite batch limit and compare the value function obtained with TD to the ground truth value function. We will, for simplicity, consider in this section a Markov reward process with transition matrix p ( s t +1 = s ′ | s t = s ) = Π( s, s ′ ) . The general theory described in the main text does not only apply to MDPs, but the convergence analysis for MDPs is much more straightforward so we describe it here. In this case, the ground truth value function satisfies

which gives the vector equation V = ( I -γ Π ) -1 R for V , R ∈ R |S| . Suppose the limiting distribution over states is p ∈ R |S| which has entries p ( s ) = 1 T ∑ T t =1 p ( s t = s ) . The fixed point of TD dynamics is

We now consider the two possible cases for this fixed point condition.

Case 1: Underparameterized Regime First, if the feature dimension N is smaller than the size of the state space |S| and the features are maximal rank, then the TD learning fixed point is

In this case, the value function is not learned perfectly, as can be seen by computing ˆ V = Ψ glyph[latticetop] w TD and comparing to the ground truth V = ( I -γ Π ) -1 R . In this case, we would say that TD learning has an irreducible value error due to capturing only a N dimensional projection of the value function.

Case 2: Overparameterized Regime Alternatively, if the feature dimension exceeds the total number of states, then the fixed point equation for TD is underspecified. However, throughout TD learning w TD ∈ span { ψ ( s ) } s ∈S so we can instead consider the decompostion w V = ∑ s α ( s ) ψ ( s ) , where α ∈ R |S| satisfies

where K ∈ R |S|×|S| is the kernel computed with features K ( s, s ′ ) = ψ ( s ) · ψ ( s ′ ) . The solution to the above equation is unique and the learned value function ˆ V = Ψ glyph[latticetop] w TD = KK -1 ( I -γ Π ) -1 R = ( I -γ Π ) -1 R = V . Therefore, in the over-parameterized limit, the irreducible value error for TD learning is zero. This limit was considered dynamically in the infinite batch (vanishing SGD noise) setting by [44].

B Derivation of Learning Curves

In this section, we now consider the dynamics of TD learning when B random episodes are sampled at a time. In this calculation, the finite batch of episodes leads to non-negligible SGD effects which can cause undesirable plateaus in TD dynamics.

B.1 Field Theory Derivation

In this section we use a Gaussian field theory formalism to compute the learning curve in the high dimensional asymptotic limit N,B → ∞ with B/N = α . The episode length T is treated as O (1) . While this paper focuses on the online setting, where fresh trajectories { τ µ n } are sampled at each iteration n , this model can be straightforwardly extended to the case where a fixed number of experience trajectories { τ µ } are replayed repeatedly during TD learning. We leave the experience

replay dynamic mean field theory calculation for future work. The starting point of our analysis is tracking the moment generating function for the iterate dynamics

To compute this object over random draws of training trajectories, we express the joint average over w n , { s µ n ( t ) } into conditional averages over w n , { ∆ µ n ( t ) }|{ ψ µ n ( t ) } . To simplify the computation, in this section, we will compute the learning curve for mean zero features µ ( s ) = 0 and

Expressing the Dirac-delta function as a Fourier integral δ ( z ) = ∫ d ˆ z 2 π exp( i ˆ zz ) for each of our constraints. Under the Gaussian equivalence ansatz , we can easily average over Gaussian ψ to obtain

where we adopted the shorthand D ∆ = ∏ µ,n,t d ∆ µ n ( t ) for the measure for the collection of variables { ∆ µ n ( t ) } . Likewise one should interpret D w = ∏ n d w n . To analyze the high dimensional limit of the above moment generating function, we introduce order parameters for the theory

For each of these order parameters, we enforce the definition of the order parameter using the Fourier representation of a Dirac-delta function

Repeating this procedure for all order parameters q = { Q, ˆ Q,C, ˆ C,C R , ˆ C R , D, ˆ D,D R , ˆ D R } and disregarding irrelevant prefactors, we have the following formula for the moment generating function

where the action S has the form

The function Z has the interpretation of an effective partition function conditional on order parameters q . To study the N → ∞ limit, we use the steepest descent method and analyze the saddle point

∂S ∂q = 0 . These saddle point equations give

The brackets 〈〉 denote averaging over the stochastic processes defined by moment generating functions Z ∆ , Z w . After these saddle point equations are solved the order parameters q are treated as non-random and a Hubbard-Stratonovich transformation is employed. For example,

where the average is over u w n ∼ N ( 0 , η 2 T -2 ∑ tt ′ Q n ( t, t ′ ) Σ ( t, t ′ ) ) . After introducing these Hubbard fields u w n and u ∆ n ( t ) , we can perform the integrals over ˆ w n and ˆ ∆ n ( t ) which collapse to Dirac-Delta functions. The resulting identities of the delta functions define the following stochastic processes on w n and u ∆ n

Using a similar trick, we can show that for any observable depending on w n or { ∆ n ( t ) } that

Since w n is independent. This can be used to conclude

which implies that ∆ n ( t ) = u ∆ n ( t ) . Consequently the response functions have trivial structure

We therefore obtain a stochastic process of the form

These are the final equations defining the stochastic evolution of w n and ∆ n ( t ) .

B.2 Simplifying the Saddle Point Equations

Using the above saddle point equations, we see that the variables { ∆ n ( t ) } and { w n } will be Gaussian random variables. It thus suffices to track their mean and covariance. The { ∆ n ( t ) } variables have zero mean and covariance given by the Q n ( t, t ′ ) function. The { w n } variables have the following mean evolution

where w TD = [ ¯ Σ -γ ¯ Σ + ] -1 ¯ Σ w R is the fixed point of the TD dynamics. We next compute M n = 〈 ( w n -w TD ) ( w n -w TD ) glyph[latticetop] 〉 which admits the recursion

To obtain our formulas which hold for finite batch size, we rescale the learning rate by η → η/ √ α giving the following evolution

After this rescaling, we see that the mean evolution for w n is independent of α but that the variance picks up an additive term on each step on the order of O ( η 2 α -2 ) which vanishes in the infinite batch limit B/N → ∞ . The error for value learning can be obtained from M n with L n = 1 N Tr M n ¯ Σ . Lastly, we note that we can express the formula for Q n ( t, t ′ ) entirely in terms of M n and 〈 w n 〉 . This

gives the lengthy expression

B.3 Final Result

Below we state in compact form the full final result for our TD learning curves. The below equations give the evolution of the first and second moments of w n obtained from the mean-field density of the previous section. Concretely, these moments obey dynamics

These equations can be solved iteratively for ¯ w n , M n , Q n . Finite dimensional versions of this result can be obtained by replacing α with B/N as written in the main text. The value estimation error is

glyph[negationslash]

B.4 Non-Zero Mean Feature

We can also simply modify the DMFT equations if the mean feature is nonvanishing µ ( s ) = 0 . In this case, when averaging over all possible trajectories through state space, there is a mean feature vector at each episodic time µ ( t ) . The above equations are exact for non-zero mean features if Σ ( t, t ′ ) is regarded as the (non-centered) correlation matrix 〈 ψ ( t ) ψ ( t ′ ) 〉 .

B.5 Action Dependent Rewards

B.5.1 Expected Q-Learning Reduces to Previous Model

In the case where we consider using features that depend on both states and actions ψ ( s, a ) then we can use expected value learning to identify the expected value of a state-action pair under policy π

This V function quantifies the expected reward associated with taking action a when in state s and subsequently following policy π . This problem is structurally identical to the state dependent case

by recognizing that state action pairs ( s, a ) act as new states ˜ s . As before, the policy defines the probability distribution over transitions on ˜ s . We can thus use Equation (4) to calculate the learning curve for this problem.

B.5.2 Action Dependence Generates Target Noise in State Dependent Value Learning

In the case where the rewards depend on both state and action R ( s, a ) but features only depend on state ψ ( s ) , we need a slight modification of our theory which models the reward at each state as a mean value (over actions) plus a noise. For each state, we decompose the reward function into mean and fluctuation

The function ¯ R ( s ) can again be decomposed into the basis of features ψ ( s ) . However, we need to consider the correlation structure of glyph[epsilon1] ( s, a ) .

glyph[negationslash]

The above average vanishes for t = t ′ since glyph[epsilon1] ( s t ′ , a t ′ ) is zero mean over a | s . We introduce the notation σ 2 t = Var a | s t R ( s t , a ) . Thus, we effectively have a model where our TD errors obey

The addition of this term leads to a simple modification of our Q ( t, t ) function

This change to the Q n ( t, t ′ ) correlation function alters the dynamics of M n . Lastly, our population risk for the value estimation takes the form

where 1 T ∑ t σ 2 t exactly quantifies the variance in rewards unexplained by state-dependent features.

B.6 Tracking Iterate Moments with Direct Recurrence Relation

In this section we give a direct calculation of the first two moments of w over the collection of randomly sampled features { ψ µ n ( t ) } and show which terms are disregarded in the proportional limit examined in the main text.

Letting A Σ -γ Σ , we note that the average evolution of w has the form

Thus, if we disregarded fluctuations in w n due to SGD, the model will converge to the correct fixed point. Next, we look at M n = 〈 ( w n -w TD ) ( w n -w TD ) 〉 . Under the Gaussian equivalence ansatz, we have

The mean field theory derived from saddle point integration consists of the first two terms in the final expression. Therefore mean field theory disregards the last term which computes cross time correlations of RPEs with features, effectively making the approximation

After making this approximation, we recover the learning curve obtained in the previous Section B.3. We show in our experiments that dropping this term does not significantly alter the learning curves.

B.7 Scaling of Asymptotic Fixed Points

To identify fixed points in the value error dynamics, we can seek non-vanishing fixed points for the weight error covariance M = 〈 ( w -w TD )( w -w TD ) 〉 . We note that 〈 w 〉 ∼ w TD asymptotically. Again, letting A = ¯ Σ -γ ¯ Σ + , we obtain the following fixed point condition for M under these assumptions

Where we used the formula for Q n ( t, t ′ ) from Appendix B.6, evaluated at 〈 w 〉 = w TD and used the fact that w R = w TD -γ ¯ Σ -1 ¯ Σ + w TD . The solution M = 0 is a valid fixed point for M in the η → 0 and B →∞ limits because the constant terms on the right-hand side vanish. Similarly, if γ = 0 (which corresponds to the standard supervised learning case), the right hand side is linear in M , allowing M = 0 to be a valid fixed point.

However, for finite B and non-zero η and γ , there exists a solution to the above fixed point equation. For small ηγ 2 B , we can deduce that M must satisfy a self-consistent asymptotic scaling of the form

implying an asymptotic value error scaling of L ∼ Tr M ¯ Σ ∼ O ( ηγ 2 B ) . These scalings are examined in Figure 3 where experiments obey the expected behavior.

C Reward Shaping

In this section, we consider the role of reward shaping on the dynamics of TD learning. As discussed in the main text, we consider potential based shaping with potential function decomposable in the features φ ( s ) = w φ · ψ ( s ) . We first describe the change to the average weight evolution 〈 w n 〉 and then describe the dynamics of the correlations. In potential based shaping, the TD errors take the form

Computing from the DMFT equations the evolution of 〈 w n 〉 we have

We see that including the reward shaping function φ offsets the fixed point of the algorithm to be w TD + w φ . This occurs precisely because the potential-based reward shaping generates an additive correction to the target value function by φ ( s ) [64]. When we predict value at evaluation, we use the reshifted value ˆ V ( s ) -φ ( s ) . The natural quantity to track at the level of the mean field equations is the adapted version of M n

This correlation matrix has dynamics

and the TD-error correlations Q n ( t, t ′ ) have the form

The value estimation error is again L n = Tr M n ¯ Σ . We see that the two primary ways that reward shaping alters the loss dynamics is

A change in the initial condition for M n to be M 0 = ( w TD + w φ )( w TD + w φ ) glyph[latticetop]

A change in the TD error covariance term Q n ( t, t ′ )

Both effects can generate significant changes in the dynamics and plateaus of the model.

D Non-Gaussian Features

The full non-asymptotic theory (no assumptions on N,B large) of TD dynamics with linear function approximation closes under the fourth moments of the features. In this setting we do not incorporate explicit factors of N in the defintion of the value estimator ˆ V ( t ) = w · ψ ( t ) . As before, we track the update to the M matrix

To calculate the last term, we introduce the following tensor of fourth moments

In the Gaussian case, this reduces to an expression involving only the correlations. For example, if the features are zero mean, then we can use Wick's theorem to obtain the decomposition

Now, using the fact that ∆ n ( t ) = ( w R -w n ) · ψ ( t ) + γ w n · ψ ( t +1) , we find

Putting this all together, we find the following recurrence for M n in the non-Gaussian case

where all traces are taken against the last two time and feature indices. The averages over the weights close in terms of the average 〈 w n 〉 and the covariance M n . This equation is exact for any feature distribution, but requires significant computational resources to evaluate and is less illuminating than the mean field limit analyzed in the previous sections.

D.1 Breakdown of Gaussian Theory in Low Dimension

In this section, we discuss the breakdown of Gaussian theory at low dimension N . In Figure D.1 we provide an example where the non-Gaussian distributions exhibit noticeably different learning curves than the Gaussian approximate theory (dashed black) and Gaussian samples with matching covariance (dashed color). We use the features

(((⊗}{∖⊔∐√√√]∐{

(((⊗}{∖⊔∐√√√]∐{

(((⊗}{∖⊔∐√√√]∐{

 

 

√√˜√√

 

 

 

∧∐⌈√˜(〉√√}√

⊗(((⊗}{∖⊔∐√√√]∐{

⊗(((⊔∐√√√]∐{

⊗(((⊗}{∖⊔∐√√√]∐{

⊗(((⊗}{∖⊔∐√√√]∐{

(a)

B

= 5

 

 

√√˜√√

 

 

 

∧∐⌈√˜(〉√√}√

(((⊔∐√√√]∐{

(b)

N

= 100

D.2 A Simple Solveable Example

We next examine a very simple case where we can exactly characterize the gap between the nonGaussian and Gaussian distributions. In this section, we examine the special case of T = 1 and look at features which are independent p ( ψ ) = ∏ N i =1 p ( ψ i ) (form a factor distribution). In this case, we obtain the following exact learning curve

which holds for any N,B . We note that in the limit where N,B →∞ with B/N = α , we see that the dependence on κ disappears and we arrive at the universal behavior

For example, we can consider vectors on the hypercube where ψ k ∈ {± 1 } with equal probability for k ∈ { 1 , ..., N } for the non-Gaussian distribution and compare to the Gaussian with identical covariance ψ ∼ N (0 , I ) .

The reason for the discrepancy between the Gaussian and Bernoulli/Hypercube loss curves is exactly the negative kurtosis of the hypercube features

κ

Gauss

= 0

An example of this result for low and high dimensions N with B = αN with α = 0 . 1 is provided in Figure D.2. In low dimension ( N = 10 ) the Bernoulli/Hypercube feature have noticeably different dynamics than the Gaussian features. In the proportional limit N,B → ∞ with α = B/N these learning curves are identical and all have the form L n ∼ [ (1 -η ) 2 + η 2 α ] n .











 

 

 

 

 

 

 

⊗

˜√{}√⌈⌈]

⊔∐√√√

√[˜}√glyph[arrowbt]

(a) Low Dimension











 

 

 

 

 

 

⊗

˜√{}√⌈⌈]

⊔∐√√√

√[˜}√glyph[arrowbt]

(b) High Dimension

E Tests on Other Feature Distributions

In this section, we include additional tests of our theory on alternative features with the same random walk policy as in Figure 1.

F Plateau Scaling in MountainCar-v0 Environment

We verified the results of the theory on the environment MountainCar-v0. First, we train a policy with tabular glyph[epsilon1] -greedy Q-Learning ( glyph[epsilon1] = 0 . 1 , γ = 0 . 99 , η = 0 . 01 ) to learn policy π . The position and velocity are discretized into 42 and 28 states, respectively. The learned policy π is not optimal but consistently reaches goal within 350 timesteps. Therefore, each episode is set to have a length of 350 timesteps. Next, we take π and evaluate it with TD learning.

Since MountainCar-v0 is a continuous environment, there is no closed solution to the ground truth of the value function. To estimate the ground truth value function, we ran TD learning with a small learning rate for 10M batches ( η = 0 . 01 , B = 1 , γ = 0 . 99 ) to obtain V π ≈ ˆ V 10 M .

10 0

10 1

10 2

10 3

Iteration

10 0

2×10 1

3×10 1

4×10 1

6×10 1

Value Error

Polynomial Features

degree = 4

degree = 6

degree = 10

(a) Polynomial features

10 0

10 1

10 2

10 3

Iteration

10

2

10

1

10 0

Value Error

Fourier Features

scale = 0.2

scale = 1.0

scale = 5.0

(b) Fourier Features

G Numerical methods and additional details

The code to generate the Figures is provided in the Supplementary Material as a Jupyter Notebook at the following Github repository https://github.com/Pehlevan-Group/TD-RL-dynamics . Here, we briefly highlight some of the parameter choices.

For Figures 3 and 4 we use diagonally decoupled, but temporally correlated power law features with Σ kglyph[lscript] ( t, t ′ ) = δ kglyph[lscript] k -1 . 2 exp( -| t -t ′ | /τ k ) with τ k = 10 k +1 and w R k = k -1 . 1 for k ∈ [ N ] with N = 300 . This type of feature structure is especially easy to evaluate the theoretical learning curves for. Unless otherwise stated, these figures used γ = 0 . 9 and batch size B = 10 .

For the 2D MDP grid world, we defined a discrete set of states on a 17 × 17 grid. The agent starts in the middle position and follows a random diffusion policy where each possible movement (up, down, left, right) is taken with equal probability. The features were generated as bell-shaped place cells (shown). We computed Σ ( t, t ′ ) for the theory by sampling 5000 random draws of length T = 50 . The Gaussian learning curve is obtained with TD learning with ψ G ∼ N (0 , Σ ) .

Numerical experiments were performed on a NVIDIA SMX4-A100-80GB GPU using JAX to vectorize repetitive aspects of the experiments. With the exception of the MountainCar-v0 simulations, the numerical experiments (both preliminary experiments and those presented in the paper) took around 1 hour of compute time.

0

10

20

v

0

5

10

15

20

25

30

35

40

x

40

30

20

10

0

0

10

20

v

0

5

10

15

20

25

30

35

40

x

80

60

40

20

0

0.025

0.050

0.075

0.100

0.125

0.150

0.175

0.200

Learning Rate

0

1000

2000

3000

4000

5000

6000

Convergence Value Error

=0.01

=0.02

=0.05

=0.1

=0.2

Linear fit

0.2

0.4

0.6

0.8

1.0

1/

B

200

400

600

800

1000

Convergence Value Error

B

=1

B

=2

B

=4

B

=8

Linear fit

(c) Scaling of value error with learning rate

(d) Scaling of value error with batch size

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn summary, they study how learning dynamics and plateaus depend on feature structure, learning rate, discount factor, and reward function in the case of batch TD(0) for policy evaluation.\n\nThis paper applies concepts from statistical physics to study typical case learning curves for TD learning in linear FAs with nonlinear but fixed features (specifically, their study is targeted at policy evaluation with batch TD(0)). They conjecture a Gaussian feature equivalence: roughly, having a high-dim feature vector, the learning curves of a TD learner are conjectured to be equivalent to that of one with Gaussian features having matching mean and correlations. Further, this view allows them to use a Spectral Perspective to find a proxy for what would constitute a harder reward function to optimize (showing alignment between learning rate progress vs. the measure; shaped reward being simpler than sparse reward).\n\nSTRENGTHS:\n- A step in an important direction: towards better understanding the learning dynamics of TD methods\n\n- Proposing a creative tool for the analysis of TD methods    \n\n- Good exposition of related works\n\n- Paper is well-motivated and clearly written \n\n- Claims are carefully stated and supported: no overstated claims \n\n- Experiments are pretty interesting and carefully designed\n\nWEAKNESSES:\n- Considering tasks have a finite horizon (fixed T), it has not been made clear if the experiments are run using features that have a good representation of the remaining time in them or not. Of course, this could be my misunderstanding (to be clarified by authors in the Questions section).\n\n- Not sure whether the comparison between fixed and annealing learning rates make too much sense in the context of this study (to be clarified by authors in the Questions section).\n\nQUESTIONS:\n1. Are the results of the Discount Factor analysis (Fig. 3b) for a dense reward scenario? Wouldn’t this be a very different case with sparse rewards due to the issues studied by van Seijen et al. (2019)?\n\n2. Isn’t annealing the learning rate a condition (even for tabular TD(0)) to guarantee asymptotic convergence to the optimal V with probability 1? If yes, what is the comparison of Fig. 5c vs 5d really showing us in the context of this paper?  \n\n\n- van Seijen *et al.* [NeurIPS 2019], \"Using a Logarithmic Mapping to Enable Lower Discount Factors in Reinforcement Learning\".\n\nLIMITATIONS:\nLimitations have been discussed sufficiently well.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper looks to introduce a new theory for the learning dynamics in the online batch policy evaluation setting for temporal difference learning. The paper introduces the Gaussian Equivalence Conjecture, which postulates that the learning curves in TD can be modeled by Gaussian features (with per-time-step mean and covariance over features and all trajectories). Along with this modeling assumption, the work does not prove this conjecture, but instead assumes this conjecture to be true to show empirical results in modeling policy evaluation learning curves analytically on a simple gridworld MDP with “place cell” features.\n\nUsing this assumption and the example MDP, the work first shows results fitting their model with the learning curve of online policy evaluation, which is interesting in itself. Using this modeling assumption, the authors provide insight into the “hardness” of certain policy evaluation tasks and how it relates to their TD learning model with an example comparing sparse vs dense reward functions in policy evaluation. Beyond this, the work also shows examples (on the same domain) of their theory of learning dynamics predicting the effects of batch size, discount factor, learning rate and learning rate annealing. Finally, the work also uses this model to explain and predict what happens to learning dynamics in the case of reward shaping. The work uses this proposed model to reshape rewards to improve timescales of convergence by rotating the reward-scaling weights to be more aligned with features of high variance.\n\nSTRENGTHS:\nOverall, I think the work is of great interest to the reinforcement learning community with a good amount of both theoretical and empirical results to back their proposed model of learning dynamics. The paper poses and answers well-founded questions, and is also well written. I only have a few (albeit important) questions I’ll pose below.\n\nWEAKNESSES:\nMy biggest issue with this paper in its current state is its set up. Currently, the proposed model is just a conjecture that seems to fit well with this one set of features (that are Gaussian it seems!) in this one gridworld environment. All the results seem to hinge on this conjecture being true, as evinced by the claim “We do not aim to provide a rigorous proof of this conjecture for TD learning but instead compute the learning curve implied by this assumption and compare to experiments on simple Markov Decision Processes”. Different forms of feature construction have been proposed throughout reinforcement learning history, including tile coding, polynomials, fourier basis etc. I’m still not convinced that this Gaussian model will carry over to learning dynamics with different features and different environments. While this is essentially my only concern with this paper (it’s very well written!), I think it is quite an important concern to address. Due to this concern, I have decided to only give this paper a weak accept. If more feature construction methods and more environments were included in these results, I would definitely be open to raising this score!\n\nQUESTIONS:\nThroughout reading this work, I only have a few section-by-section questions:\n\n******3.1******\n\nCould you give more specific details with regards to this environment? Maybe even in the appendix? I don’t exactly understand what you mean when you say ‘The feature map is parameterized by the bandwidth of individual “place cells”’.\n\n******3.2******\n\nFor Remark 2 and 5, could you elaborate a bit on “fully explainable by features”? Is this referring to the case of partial observability? or is it just the most general formulation of R as a function of (s, a, s’)?\n\nThe notation for the proof for Proposition 3.1 is a bit confusing. I would suggest either simplifying by removing the notation that’s not defined here in the main paper and moving this to the appendix, or moving more of the appendix here.\n\nLIMITATIONS:\nPlease see the general comment made in weaknesses to address limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work utilizes concepts from statistical physics to analyze the learning dynamics of TD(0) (policy evaluation) under the assumption of Gaussian equivalence, online batch update and linear function approximation. Using the theory, it demonstrates how learning rate annealing and reward shaping can improve learning in various aspects, based on multiple synthetic experiments.\n\nSTRENGTHS:\nThe paper studies the learning dynamics of TD(0) from a novel perspective, which could provide readers with new insights and understandings\n\nWEAKNESSES:\n1. The whole framework relies on an important hypothesis that is not sufficiently verified in the TD(0) context. The Gaussian equivalence conjecture is only justified using a toy MDP (Fig.1) without explaining how accurate this hypothesis is for general MDPs. The provided example is using RBF features and RBF reward, which could be favorable to the given theory.\n\n2. The connection to least-square TD methods is missing. In the same regime as the current paper, LSTD has been extensively studied for its convergence and learning dynamics (see, for example, Tagoti and Scherrer (2015) and Pan et al., (2017)). It does not rely on the Gaussian assumption and should be compared with the convergence result in the current paper.\n\nRef:\n- Tagorti, M. and Scherrer, B., 2015, June. On the Rate of Convergence and Error Bounds for LSTD (λ). In *International Conference on Machine Learning* (pp. 1521-1529). PMLR.\n- Pan, Y., White, A. and White, M., 2017, February. Accelerated gradient temporal difference learning. In *Proceedings of the AAAI Conference on Artificial Intelligence* (Vol. 31, No. 1).\n\n3. It is unclear what Sec.4 is trying to demonstrate. The matrix A naturally appears in LSTD and has nothing special w.r.t. the current Gaussian equivalence setup.\n\nMinor\n- L136: correlation here means autocorrelation instead of Pearson product-moment correlation\n\nQUESTIONS:\nQ1: Any comments on why the Gaussian equivalence assumption would hold for general MDPs?\n\nQ2: How are the current convergence results different from the typical results from the LSTD literature?\n\nLIMITATIONS:\nThe paper did not sufficiently explain its connection to the literature.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a theoretical model that predicts the dynamics of TD learning. The theory assumes that the distribution of feature vectors is, in some sense, equivalent to a Gaussian distribution and predicts the value estimate at each iteration. The theory reveals a rich set of phenomena such as plateaus in TD learning.\n\nSTRENGTHS:\nThe complete dynamics of TD learning have not been previously studied. And the paper provides a unique perspective on this problem.\n\nThe dynamics of TD learning according to the theoretical prediction are close to those in experiments, verifying the usefulness of the theory in these experiments.\n\nThe theory is also consistent with some existing tricks to improve TD learning such as reward shaping and step-size annealing.\n\nWEAKNESSES:\nThe paper does not mention the limitations of its main assumption: the Gaussian Equivalence Conjecture. It verifies it using experiments in some small MDPs but these experiments do not tell when this assumption holds.\n\nThe paper does not explain clearly its theoretical results, making it difficult to understand these results.\n\nThe writing of the paper needs to be improved. Specifically, many sentences and notations remain confusing to readers. See Questions for details.\n\ntypos:\n\nline 73: the features -> the vector of features\nline 78: the offline setting\nline 79: width -> with\nline 245: two \"learning\"s\n\nQUESTIONS:\nIs T a constant or a random variable? If T is a constant you are considering the finite-horizon setting and the value function is horizon-dependent. If T is a random variable you need to specify how it is defined.\n\n\"the algorithms bootstraps its current predictions to estimate future states\" Not sure what you mean here. I think you mean algorithm, not algorithms. And I think you mean bootstrapping from the estimated next state's value.\n\nline 134: what are \"the learning curves\"? \"high dimensional features\" how many dimensions are needed? What do you mean when you say a set of learning curves is equivalent to another set of learning curves?\n\neqn 3: what is the meaning of <x>_{y} in your notation?\n\nLine 138: \"higher order cumulants of the features is negligible in high dimensional feature spaces under the square loss\". Again, what do you mean by square loss. What do you mean by \"higher\" order \"cumulants\" of the features?\n\nFigure 1: what is this diffusion process?\n\nProposition 3.1: what is the meaning of <>_s in the definition of L_n?\n\nLine 191: this remark seems to be much more useful. Why not present your result without assuming that B -> infty? Could you briefly explain under what conditions the theory holds when B is not infinity?\n\nLine 195: this remark also seems to be much more useful. Again, why not present your result without assuming that the value function is representable?\n\nLine 203: how is this remark different from remark 2?\n\nLine 205: what is w (without _n)?\n\nLine 209: \"which can be learned easily and which require more sampled trajectories\" Why is it \"easily\" when it requires more sampled trajectories?\n\nLine 216: \"The theory predicts that,\" I do not see which part of your theory gives this result. Furthermore, how is this result different from the rate of convergence in a linear dynamical system?\n\nProposition 5.1: what is a fixed point of a learning curve?\n\nHow is eqn B.24 derived?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the concept of statistical physics to analyze reinforcement learning models. The authors propose a theory of learning dynamics for RL, with an emphasis on the role of linear function approximation. They investigate how strategies such as learning rate annealing and reward shaping can positively modify learning dynamics and surmount plateaus. The paper concludes by revealing new tools for developing a theory of learning dynamics in RL, thereby setting the stage for future research in this domain.\n\nSTRENGTHS:\n1. This paper introduces a novel method using statistical physics to formulate a theory of learning dynamics in RL.\n2. The authors derive an analytical formula for the typical learning curve, demonstrating how their theory can predict the scaling of both learning convergence speed and performance plateaus based on problem parameters. This predictive ability is a significant advantage of the paper.\n3. The paper outlines how this theory can assist in understanding and guiding design principles when selecting meta-parameters in RL algorithms. Such insights can enable practitioners to gain a deeper comprehension of how these factors influence the RL learning process.\n\nWEAKNESSES:\n1. The paper does have limitations regarding practical implications. Although it presents a theoretical framework and makes predictions about learning dynamics, the methods utilized, such as linear approximation, deviate significantly from current reinforcement learning (RL) methodologies.\n\nQUESTIONS:\nIs there a particular reason for choosing batch sizes that are not powers of 2, such as N=3, N=30, N=20, for the experiments? As far as I am aware, batch sizes that are powers of 2 are most commonly used for practical considerations.\n\nLIMITATIONS:\nNo concerns about negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}