Lecture 27:  Temporal Difference Methods with Function Approximators in Reinforcement Learning
Channel: Abhishek Gupta
Date: 2020-04-29

In this lecture of ECE 8851: Reinforcement Learning, we explore Temporal Difference (TD) methods with function approximators, which are widely used in reinforcement learning for approximating the value function. We start with an overview of TD learning and then move on to discuss TD with a linear function approximator, which is a popular and simple approach. Next, we explore TD with a nonlinear function approximator, such as a neural network, and the challenges that arise with such an approach. We then dive into advanced TD algorithms, such as GTD2 and TDC, which can handle more complex function approximators. Additionally, we discuss bootstrapping in TD learning, which can lead to faster convergence and better results. Finally, we explore the "deadly triad," a term used to describe the combination of function approximation, bootstrapping, and off-policy learning, which can cause instability in TD algorithms.