Approximate Dynamic Programming: Solving the Curses of Dimensionality, 2nd Edition (Wiley Series in Probability and Statistics)
This book is a major revision of the first edition, with seven new or heavily revised chapters.  This edition starts clearly from a foundation in reinforcement learning (using classical RL notation and concepts), but continues to build a bridge to the types of high-dimensional problems familiar to operations research.  The book uses "a" for discrete actions, but switches to "x" for vector-valued decisions, common in the operations research community but largely ignored in computer science.The book begins with an overview of a wide array of problems (in chapter 2), with an introduction to classical Markov decision processes in chapter 3 (minor changes from the first edition).  Chapter 4 is a completely rewritten introduction to reinforcement learning using classical concepts, with one major exception.  It now provides an extended overview of the concept of the post-decision state variable, which is used throughout the book (because it avoids the imbedded expectation within the min/max operator).  The RL community is very familiar with the use of state-action pairs (in Q-factors), which is a clumsy form of post-decision state.  Chapter 4 gives a series of examples where the post-decision state can (but does not always) provide significant benefits.Chapter 5 is a minor revision of the old chapter 5, providing an in-depth discussion of how to model a dynamic program.The book brings together different fields within stochastic optimization by identifying (in chapter 6) four fundamental classes of policies:  1) myopic policies (which ignore the future), 2) lookahead policies (which optimize over a short horizon to determine the decision to be made now), 3) policy function approximations (analytic functions that return an action given a state), and 4) policies based on value function approximations.  Policy function approximations (a little-used term that I am promoting) and value function approximations both require some sort of method for approximating a function, of which there are three basic classes:  lookup tables, parametric models, and nonparametric models.Of course, you can build hybrid policies by mixing and matching.Chapter 7 provides an in-depth treatment of methods for optimizing policy function approximations (widely known as "policy search" in the RL literature).  In addition to classical material from stochastic search, it also includes a description of the knowledge gradient concept, which we developed as part of research to develop insights into the exploration-exploitation problem. Chapters 8, 9 and 10 provide a layered presentation of how to go about designing policies based on value function approximations.  Chapter 8 is an overview of popular methods for approximating functions.  Chapter 9 discusses methods for obtaining an update to a value function approximation for a fixed policy, and Chapter 10 describes the complex problem of approximating a value function approximation while simultaneously optimizing over policies.Chapter 11 provides an introduction to the fundamentals of updating value function approximations based on stochastic approximation methods, with an overview of stepsize formulas.  This is a revision of the old chapter 6, somewhat streamlined but with new insights into how to design stepsize rules for different algorithmic strategies (Q-learning, approximate value iteration, LSTD, LSPE) and a new, optimal stepsize rule designed specifically for methods based on bootstrapping (TD(0), Q-learning, approximate value iteration).Chapter 12 is a completely rewritten chapter on the exploration vs. exploitation problem, including a new algorithm for using the knowledge gradient idea (designed originally for pure learning problems) in the presence of a physical state.Chapters 13 and 14 retain the material from the first edition on designing value function approximations in the context of high-dimensional resource allocation problems which exploit convexity.Warren Powell