Lec 3 - Computing Optimal Policies using Value Iteration: Stochastic Shortest Path, Discounted Cost
Channel: Abhishek Gupta
Date: 2020-01-16

Welcome to ECE 8851: Reinforcement Learning, Lecture 3! In this video, we will continue our exploration of Markov decision processes (MDPs) and delve into the topic of value iteration, a key algorithm for solving stochastic shortest path problems and discounted cost MDPs.

We will start by discussing the space of value functions, and explore how value iteration algorithms can be used to determine the optimal value function and policy in an MDP. We will also examine the concept of monotone convergence, and discuss how this relates to the convergence of value iteration algorithms in MDPs. Additionally, we will explore the contraction mapping theorem and its relevance to value iteration.

Next, we will explore the topic of non-stationary policies, and discuss how these policies can be used to optimize decision making in MDPs. We will also examine the concept of average cost MDPs and the unit chain condition, which play a critical role in the convergence proof for value iteration algorithms. Finally, we will consider the concept of single and multi-class average cost MDPs, and discuss how these types of MDPs differ from other types of MDPs.

Overall, this lecture will provide a detailed and comprehensive overview of value iteration and its application to a variety of different types of MDPs.